ICAROB2019PROCEEDINGS OF THE 2019
INTERNATIONAL CONFERENCE
ON ARTIFICIAL LIFE AND ROBOTICS
January 10-13, 2019
B-Con Plaza, Beppu, Oita, JAPAN
24th AROB International Meeting Series
Editor-in-Chief
Masanori Sugisaka
Editors: Yingmin Jia, Takao Ito, Ju-Jang Lee
ISBN 978-4-9908350-4-0

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Proceedings of The 2019 International Conference on

ARTIFICIAL LIFE AND ROBOTICS(ICAROB2019)

January 10-13, 2019
B-Con Plaza, Beppu, Oita, JAPAN
24th AROB International Meeting Series
Editor-in-Chief
Masanori Sugisaka
Editors: Yingmin Jia, Takao Ito, Ju-Jang Lee
ISBN 978-4-9908350-4-0©ICAROB 2019 ALife Robotics Corp. Ltd.

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Contents
1

Organization, etc.

1

2

Messages

93Time Table

14

4

Opening Ceremony

17

5Technical paper index

18

6Abstracts
PS abstracts
IS abstracts
OS abstracts
GS abstracts

39
40
42
87

7

Authors index

968

Conference room

107

6-1
6-2
6-3
6-4

©ICAROB 2019 ALife Robotics Corp. Ltd.

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

SPONSERED
ALife Robotics Corporation Ltd.

ORGANIZED BYInternational Steering Committee of International Conference on Artificial Life andRobotics (ICAROB)SUPPORTED BYIEEE Fukuoka Section (Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

-1-The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

CO-ORGANIZED BY
Chinese Association for Artificial Intelligence (CAAI, P. R. China)

ADVISORY COMMITTEE CHAIR
Moshe Kam (New Jersy Institute of Technology, Former IEEE President, USA)ADVISORY COMMITTEE
Adam Grzech (Wroclaw University of Technology, Poland)
Bruce Eisenstein (Drexel University, Former IEEE President, USA)
Hidenori Kimura (JST, Japan)
Jeffrey Johnson (The Open University, UK)
Jerzy Świątek (Wroclaw University of Technology, Poland)
Joshua M. Epstein (The Johns Hopkins University, USA)
Kai-Tai Song (National Chiao Tung University, Taiwan)
Masayoshi Tomizuka (University of California Berkeley, USA)
Moshe Kam (New Jersey Institute of Technology, Former IEEE President, USA)Paul Kalata (Drexel University, USA)Paul Oh (Drexel University, USA)
Steen Rasmussen (University of Southern Denmark, Denmark)
GENERAL CHAIR
Masanori Sugisaka(Alife Robotics Corporation Ltd., Japan)
CO-GENERAL CHAIRS
Yingmin Jia (Beihang Univesty, P. R. China)
Takao Ito (Hiroshima University, Japan)
Ju-Jang Lee (Honorary professor, KAIST, Korea)©ICAROB 2019 ALife Robotics Corp. Ltd.

-2-

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

VICE GENERAL CHAIR
Henrik. H. Lund (Technical University of Denmark, Denmark)John. L. Casti (International Institute for Applied Systems Analysis, Austria)
Jangmyung Lee (Pusan National University, Korea)
Luigi Pagliarini (Technical University of Denmark, Denmark)(Academy of Fine Arts of Macerata, Italy)
PROGRAM CHAIRMAN
Makoto Sakamoto (University of Miyazaki, Japan)
CO-PROGRAM CHAIR
Marion Oswald (Technische Universität Wien, Austria)
INTERNATIONAL ORGANIZING COMMITTEE
Akira Fukuda (Kyushu University, Japan)
Andrew Gadsden (University of Guelph, Canada)
Caoli Wang (University of Shanghai for Science and Technology, P. R. China)
Chan Gook Park (Seoul National University, Korea)
Eiji Hayashi (Kyushu Institute of Technology, Japan)
Evgeni Magid (Kazan Federal University, Russia)
Fuzhong Wang (Henan Polytechnic University, P. R. China)
Hazry Desa (University of Malaysia, Perlis, Malaysia)
Hidehiko Yamamoto (Gifu University, Japan)
Hiroki Tamura (University of Miyazaki, Japan)
Hiroshi Kage ((Mitsubishi Electric Corporation, Japan)
Hiroshi Matsuno (Yamaguchi University, Japan)
Jiwu Wang (Beijing Jiaotong University, P. R. China)
Jovana Jovic (CNRS-AIST JRL, Japan, France)
Junping Du (Beijing University of Posts and Telecommunications, P. R. China)
Katsunori Shimohara (Doshisha University, Japan)
Kazuo Ishii (Kyushu Institute of Technology, Japan)
Kevin Voges (Canterbury University, New Zealand)
Kuo-Hsien Hsia (Far East University, Taiwan)
Kuo-Lan Su (National Yunlin University of Science and Technology, Taiwan)
Kyungho Park (Micro-System Programs, RDECOM, ITC-PAC, U.S. ARMY, Japan and USA)
Masao Kubo (National Defense Academy of Japan, Japan)
Masanao Obayashi (Yamaguchi University, Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

-3-

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Maxim Sokolov (Innopolis University, Russia)
Mehta Rajiv (New Jersey Institute of Technology, USA)Peter Sapaty (Ukrainian Academy of Science, Ukraine)
Pierre Parrend (University of Strasbourg, France)
Qiang Cai (Beijing Technology and Business University, P. R. China)
Takashi Kohno (LIMMS/CNRS-IIS, Institute of Industrial Science, The University of Tokyo,
Japan)
Takashi Ogata (Iwate Prefectural Univerity, Japan)
Teruhisa Hochin (Kyoto Prefectural University, Japan)
Tetsuro Hattori (Kagawa University, Japan)
Thi Thi Zin (University of Miyazaki, Japan)
Toru Yamamoto (Hiroshima University, Japan)
Victor Berdonosov (Komsomolsk-on-Amur State University of Technology, Russia)
Yasunari Yoshitomi (Kyoto Prefectural University, Japan)
Yi Chai (Chongqing University, P. R. China)
Yo Horikawa (Kagawa University, Japan)
Yoshifumi Morita (Nagoya Institute of Technology, Japan)
Yoshiro Imai (Kagawa University, Japan)
Zengqiang Chen (Nankai University, P. R. China)
INTERNATIONAL PROGRAM COMMITTEE
Akira Nakamura (AIST, Japan)
Ali Selamat (University of Technology of Malaysia (UTM), Malaysia)
Andre Rosendo (ShanghaiTech University, P. R. China)
Bin Zhang (Beijing University of Posts and Telecommunications, P. R. China)
Arsit Boonyaprapasorn (Chulachomkloa Royal Military Academy, Thailand)
Dongmei Ai (University of Science and Technology Beijing, P. R. China)
Endra Joelianto (Bandung Institute of Technology, Indonesia)
Fengzhi Dai (Tianjin University of Science & Technology, P. R. China)
Haisheng Li (Beijing Technology and Business University, P. R. China)
Haruna Matsushita (Kagawa University, Japan)
Hidetsugu Suto (Muroran Institute of Technology, Japan)
Hiroyuki Iizuka (Osaka University, Japan)
Huailin Zhao (Shanghai Institute of Technology, P. R. China)
Hussein Abbass (University of New South Wales and ADFA, Australia)
Istvan Harmati (Budapest Institute of Technology and Economics, Hungary)
Ivan Tanev (Doshisha University, Japan)
Jiandong Zhao (Beijing Jiaotong University, P. R. China)
Joono Cheong (Korea University, Korea)
Kunikazu Kobayashi (Aichi Prefectural University, Japan)
Lin Li (University of Shanghai for Science and Technology, P. R. China)
Lin Zhao (Qingdao University, P. R. China)
Manabu Yamada (Nagoya Institute of Technology, Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

-4-

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Masahide Ito (Aichi Prefectural University, Japan)
Masayoshi Tabuse (Kyoto Prefectural University, Japan
Masaomi Hatakeyama (University of Zurich, Switzerland)
Norrima Mokhtar (University of Malaya, Malaysia)
Noritaka Sato (Nagoya Institute of Technology, Japan)
Palakorn Tantrakool (King Mongkut’s Institute of Technology Thonburi, Thailand)
Ping Wang (North China Electric Power University, P. R. China)
Qinglin Sun (Nankai University, P. R. China)
Satoshi Ikeda (The University of Miyazaki, Japan)
Sanjay S. Joshi (College of Engineering, University of California, USA)
Seong-Ik Han (Dongguk University, Korea)
Shingo Mabu (Yamaguchi University, Japan)
Shumin Fei (Southeast University, P. R. China)
Shyi-Ming Chen (National Taichung University of Education, Taiwan)
Takashi Kuremoto (Yamaguchi University, Japan)
Takayoshi Yamada (Gifu University, Japan)
Taishiro Kishimoto (Keio University, Japan)
Tetsuro Katayama (The University of Miyazaki, Japan)
Thunyaseth Sethaput (Thammasat University, Thailand)
Tieqiu Huang (Beijing Jiaotong University, P.R China)
Toshinori Nawata (Kumamoto National College of Technology, Japan)
Tsunehiro Yoshinaga (Tokuyama National College of Technology, Japan)
Weicun Zhang (University of Science and Technology Beijing, P. R. China)
Xiaoyan Fu (Capital Normal University, P. R. China)
Xuemei Ren (Beijing Institute of Technology, P. R. China)
Yunzhong Song (Henan Polytechnic University, P. R. China)
LOCAL ARRANGEMENT COMMITTEE
Makoto Sakamoto (University of Miyazaki, Japan)
Masanori Sugisaka (ALife Robotics Corporation Ltd., Japan)
Takao Ito (Hiroshima University, Japan)
HISTORY
The International Conference on Artificial Life and Robotics (ICAROB) resulted from the
AROB-symposium (International Symposium on Artificial Life and Robotics) whose firstedition was held in 1996 and the eighteenth and last edition in 2013. The AROB
symposium was annually organized by Oita University, Nippon Bunri University (NBU),
and ALife Robotics Corporation Ltd., under the sponsorship of the Science andTechnology Policy Bureau, the Ministry of Education, Science, Sports, and Culture(Monbusho), presently, the Ministry of Education, Culture, Sports, Science, andTechnology (Monkasho), Japanese Government, Japan Society for the Promotion of
Science (JSPS), the Commemorative Organization for the Japan World Exposition ('70),©ICAROB 2019 ALife Robotics Corp. Ltd.

-5-

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Air Force Office of Scientific Research, Asian Office of Aerospace Research andDevelopment (AFOSR/AOARD), USA. I would like to express my sincere thanks to not
only Monkasho (annually fund support from 1996 to 2013) but also JSPS, theCommemorative Organization for the Japan World Exposition ('70), and various otherJapanese companies for their repeated support. The old symposium (this symposium
has been held every year at B-Con Plaza, Beppu, Oita, Japan except in Oita, Japan (AROB
5th ’00) and in Tokyo, Japan (AROB 6th ’01).) was organized by the International
Organizing Committee of AROB and was co-operated by the Santa Fe Institute (USA),
RSJ, IEEJ, ICASE (Now ICROS) (Korea), CAAI (P. R. China), ISCIE, IEICE, IEEE (Japan Council),
JARA, and SICE. The old AROB-symposium expanded much by absorbing much newknowledge and technologies into it. This history and character of the former AROB
symposiums are passed on the current ICAROB conference and to this journal,
International Journal of Robotics, Networking and Artificial Life (JRNAL). From now on,
ALife Robotics Corporation Ltd. is in charge of management of both the conference andthe journal. The future of the ICAROB is brilliant from a point of view of yielding newtechnologies to human society in the 21st century. We also expect to establish aninternational research institute on Artificial Life and Robotics in the future with the helpof Japanese Government and ICAROB. This conference invites you all.
AIMS AND SCOPE
The objective of this conference is the development of new technologies for artificial lifeand robotics which have been recently born in Japan and are expected to be applied invarious fields. This conference presents original technical papers and authoritative stateof-the-art reviews on the development of new technologies concerning robotics,
networking and artificial life and, especially computer-based simulation and hardware
for the twenty-first century. This conference covers a broad multidisciplinary field,
including areas such as:
Artificial intelligence & complexity
Artificial living
Artificial mind research
Artificial nervous systems for robots
Artificial sciences
Bipedal robot
Brain science and computing
Chaos
Cognitive science
Computational Molecular biology
Computer graphics
Data mining
Disasters robotics
DNA computing
Empirical research on network and MOT
Environment navigation and localization©ICAROB 2019 ALife Robotics Corp. Ltd.

-6-

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Evolutionary computations
Facial expression analysis, music recommendation and augmented reality
Foundation of computation and its application
Fuzzy control
Genetic algorithms
Human-welfare robotics
Image processing
Insect-like aero vehicles
Intelligence in biological systems
Intelligent control
Management of technology
Medical surgical robot
Micro-machines
Multi-agent systems
Nano-biology
Nano-robotics
Networking
Neural circuits
Neuro-computer
Neuromorphic Systems
Neuroscience
Pattern recognition
Quantum computing
Reinforcement learning system & genetic programing
Robotics
Software development support method
System cybernetics
Unmanned underwater vehicles
Unmanned Aerial Systems Technologies
Unmanned Aerial Systems designing, controls and navigation
Unmanned Aero vehicles
Virtual reality
Visualization
Hardware-oriented submissions are particularly welcome. This conference will discuss
new results in the field of artificial life and robotics©ICAROB 2019 ALife Robotics Corp. Ltd.

-7-

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

COPYRIGHTS
Accepted papers will be published in the proceeding of The 2019 International
Conference on Artificial Life and Robotics (ICAROB 2019) by ALife Robotics Corp. Ltd.
Copyright belongs to ALife Robotics Corp. Ltd. Some of high-quality papers in theproceeding will be requested to re-submit their papers for the consideration of
publication in an international journal ROBOTICS, NETWORKING AND ARTIFICIAL LIFE
under agreement of both Editor-in- Chief Dr. Masanori Sugisaka and 3 reviewers. All
correspondence related to the conference should be addressed to ICAROB Office.
ICAROB Office
ALife Robotics Corporation Ltd.
4-Go, 8-Ban, Higshi 2Cyome, Handadai, Oita 870-1108, JAPANTEL/FAX：+81-97-597-7760
E-MAIL：icarob@alife-robotics.co.jp
Home Page：http://alife-robotics.co.jp/©ICAROB 2019 ALife Robotics Corp. Ltd.

-8-

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

MESSAGES
Masanori Sugisaka
General Chair of ICAROB

Masanori Sugisaka
General Chair(President, ALife Robotics Co.,
Ltd. Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

It is my great honor to invite you all to The 2019 International Conferenceon Artificial Life and Robotics (ICAROB 2019).
This Conference is changed as the old symposium from the first (1996) tothe Eighteenth (2013) annually which were organized by Oita University,
Nippon Bunri University(NBU), and ALife Robotics Corporation Ltd. under
the sponsorship of the Science and Technology Policy Bureau, the Ministryof Education, Science, Sports, and Culture (Monbusho), presently, theMinistry of Education, Culture, Sports, Science, and Technology(Monkasho), Japanese Government, Japan Society for the Promotion of
Science (JSPS), The Commemorative Organization for the Japan World
Exposition (’70), Air Force Office of Scientific Research, Asian Office ofAerospace Research and Development (AFOSR/AOARD), USA. I would liketo express my sincere thanks to not only Monkasho (annually fund support
from 1996 to 2013) but also JSPS, the Commemorative Organization for theJapan World Exposition (’70), Japanese companies for their repeatedsupport.
The old symposium was organized by International Organizing Committeeof AROB and was co-operated by the Santa Fe Institute (USA), RSJ, IEEJ,ICASE (Now ICROS) (Korea), CAAI (P. R. China), ISCIE, IEICE, IEEE (Japan
Council), JARA, and SICE. The old AROB symposium was growing up by
absorbing many new knowledge and technologies into it.
This history and character was inherited also from ICAROB2014(The 2014
International Conference on Artificial Life and Robotics, included a series of
ICAROB proceedings in SCOPUS and CPCI now. From now on, ALife
Robotics Corporation Ltd. is in charge of management. This year we have
The 2019 International Conference on Artificial Life and Robotics(ICAROB2019) (24th AROB Anniversary). The future of The ICAROB isbrilliant from a point of view of yielding new technologies to human society
in 21st century.
I hope that fruitful discussions and exchange of ideas between researchers
during Conference (ICAROB2019) will yield new merged technologies forhappiness of human beings and, hence, will facilitate the establishment of
an international joint research institute on Artificial Life and Robotics infuture.

-9-

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Yingmin Jia
Co-General Chair of ICAROB

Yingmin Jia
Co-General Chair(Professor, Beihang University,
R .P. China)©ICAROB 2019 ALife Robotics Corp. Ltd.

It is my great pleasure to invite you to The 2019 International Conferenceon Artificial Life and Robotics (ICAROB 2019), in B-Con Plaza, Beppu, Oita,
Japan from January 10 to 13, 2019.
ICAROB develops from the AROB that was created in 1996 by Prof.
Masanori Sugisaka and will celebrate her 24 th birthday in 2019. Doubtless,new mission and big challenges in the field of artificial life and robotics will
promote ICAROB to start a new stage and attract wide interests amongscientist, researchers, and engineers around the world.
For a successful meeting, many people have contributed their great efforts
to ICAROB. Here, I would like to express my special thanks to all authors
and speakers, and the meeting organizing team for their excellent works.
Looking forward to meeting you at ICAROB in Beppu and wishing you enjoy
your stay in Japan.

- 10 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Takao Ito
Co General Chair of ICAROB

Takao Ito
Co-General Chair(Professor Hiroshima
University, Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

It is my great honor to invite you all to The 2019 International Conferenceon Artificial Life and Robotics (ICAROB 2019). This Conference is changed
as the old symposium (ISAROB) from the first (1996) to the Eighteenth. I
am pleased to welcome you to The 2019 International Conference on
Artificial Life and Robotics in the wonderful city of Beppu, Oita city, Oita
Prefecture, Japan.
The ICAROB has its long history. The former organization of the ICAROB wasdeveloped under the strong leadership of the President, famous Professor
Masanori Sugisaka, the father of AROB. We gathered many researchers,
faculty members, graduate students from all over the world, and publishedmany high-quality proceedings and high-reputational journals every year.
Over the years, dramatic improvements have been made in the field ofartificial life and its applications. The ICAROB has becoming the unifying theexchange of scientific information on the study of man-made systems thatexhibit the behavioral characteristic of natural living systems including
software, hardware and wetware. Our conference shapes the developmentof artificial life, extending our empirical research beyond the territory
circumscribed by life-as-we-know-it and into the domain of life-as-it-could–
be. It will provide us a good place to present our new research results,
excellent ideas, and valuable information about artificial intelligence,complex systems theories, robotics, and management of technology.
The conference site is B-con Plaza, one of the most famous international
convention centers in Kyushu island, Japan. You can find many fantastic
scenic spots and splendid historical places in Beppu, Oita city. Enjoy yourstay and take your time to visit Beppu, Oita city.
I am looking forward to meeting you in Beppu, Oita city, during the ICAROB
2019 and to sharing the most pleasant, interesting and fruitful conferencewith you.

- 11 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Ju-Jang Lee
Co-General Chair of ICAROB

Ju-Jang Lee
Co-General Chair(Honorary professor, KAIST)©ICAROB 2019 ALife Robotics Corp. Ltd.

The First International Conference on Artificial Life and Robotics (ICAROB)
was held in Oita City, Oita, Japan from Jan. 11th to 13th, 2014. This
year’s Conference will be held amidst the high expectation of theincreasingly important role of the new interdisciplinary paradigm ofscience and engineering represented by the field of artificial life androbotics that continuously attracts wide interests among scientist,
researchers, and engineers around the globe.
Distinguished researchers and technologists from around the world arelooking forward to attending and meeting at ICAROB. ICAROB is becoming
the annual excellent forum that represents a unique opportunity for theacademic and industrial communities to meet and assess the latest
developments in this fast growing artificial life and robotics field. ICAROB
enables them to address new challenges, share solutions, discussresearch directions for the future, exchange views and ideas, view theresults of applied research, present and discuss the latest developmentof new technologies and relevant applications.
In addition, ICAROB offers the opportunity of hearing the opinions ofwell-known leading experts in the field through the keynote sessions,provides the bases for regional and international collaborative research,and enables to foresee the future evolution of new scientific paradigms
and theories contributed by the field of artificial life and robotics andassociated research area. The twenty-first century will become thecentury of artificial life and intelligent machines in support of humankind
and ICAROB is contributing through wide technical topics of interest that
support this direction.
It is a great honor for me as a Co-General Chair of the 6th ICAROB 2019
to welcome everyone to this important event. Also, I would like to extend
my special thanks to all authors and speakers for contributing theirresearch works, the participants, and the organizing team of the 6th
ICAROB.
I’m looking forward to meeting you at the 6th ICAROB in Beppu, Oita
Prefecture and wishing you all the best.

- 12 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

GENERAL SESSION TOPICS
GS1 Robotics I (3)GS3 Complexity (2)GS5 Others ( 8 )GS2 Robotics II (6)GS4 Poster (5)ORGANIZED SESSION TOPICS
OS1 Intelligent Systems and Control (8)OS3 Intelligent navigation & Robotics (4)OS5 Business Models and Management (4)OS7 Media Information Processing and Artificial
Intelligence (4)OS9 Natural Computing and Beyond (3)OS11 Recognition and Control (12)OS13 System and Control (10)OS15 Advanced Control (5)OS17 Multiagent and Complex Systems (3)OS19 Advanced Technology on Sensing Technology,
Devices, Application (3)OS21 Advances in Marine Robotics and It’s
Applications (5)OS23 Kansei and Information Engineering (4)OS25 High-Performance Computing for Interactive
Content Creation and Rendering (3)©ICAROB 2019 ALife Robotics Corp. Ltd.

OS2 Theory and Implementation of Neuromimetic
Systems (5)OS4 Service Robotics (7)OS6 Software Development Support Method (6)OS8 Computational intelligence and cognitive sciencefor human biosignals and human well-being (7)OS10 Mathematical Informatics (5)OS12 Robot Design and Simulation (5)OS14 Intelligent Control (4)OS16 New development about Control Engineering
Education (4)OS18 Data-Driven Control and Diagnosis Control
Systems (5)OS20 Advances in Field Robotics and Sensing System(8)
OS22 Robot Competitions for Social Contribution (6)OS24 Automated content generation for narrative andcognitive contents (5)- 13 -©ICAROB 2019 ALife Robotics Corp. Ltd.

- 14 -

Meeting Room 31: Committee waiting room and Rest room

15:50-17:50

15:30-15:50

14:30-15:30

OS4 Service Robotics (7)Chair: Evgeni Magid

OS2 Theory and
Implementation of
Neuromimetic
Systems (5)Chair: Timothée Levi

Coffee break

OS7 Media Information
Processing and Artificial
Intelligence (4)Chair: Yasunari Yoshitomi

OS18 Data-Driven
Control and Diagnosis
Control Systems (5)Chair: Takuya Kinoshita

OS3 Intelligent navigation &Robotics (4)Chair: Chan Gook Park

Coffee break

14:10-14:30
OS1-2 Intelligent Systems
and Control (4)Chair: Kuo-Hsien Hsia

Chair: Kazuo Ishii
Plenary Speech PS-2(Conference room)
Masanori Sugiyama

13:10-14:10

12:10-13:10

Chair: Kuo-Hsien Hsia (H. H. Lund), Chair: Jangmyung Lee (L. Pagliarini)
Invited Speech IS1-1, IS1-2(Conference Room)
Henrik Hautop Lund, Luigi Pagliarini
Lunch

OS21 Advances in Marine
Robotics and It’s
Applications (5)Chair: Keisuke Watanabe

Meeting Room 1

11:10-12:10

OS20 (8)Advances in Field
Robotics and Sensing
System (8)Chair: Eiji Hayashi

OS14 Intelligent Control(4)
Chair: Yingmin Jia

Registration (3F)

Meeting Room 33

Opening Ceremony (Conference Room)OS1-1 Intelligent Systems
and Control (4)Chair: Kuo-Hsien Hsia

Meeting Room 32

10:30-11:00

OS8 Computational
intelligence andcognitive science forhuman biosignals
and human wellbeing (7)Chair: T. Hiroyasu

Meeting Room 31

Coffee breakConference Room

10:15-10:30

9:00-10:15

8:40-

1/11(Fri.)TIME TABLE (1/11)

Farewell Party (Conference Site: 3F, Meeting Room 31)

ICAROB Secretariat

1/10(Thu.) - 1/13(Sun.)

1/13(Sun.)

Welcome Party (Hotel Shiragiku)

The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), B-Con Plaza, Jan. 10- 13, Beppu, Oita, Japan, 2019

1/10(Thu.) 17:30-19:30

OS13 System andControl (10)Chair: Qiang Wei

GS1 Robotics I (3)Chair: Jiwu Wang

OS11 Recognition andControl (12)Chair: Fengzhi Dai

Meeting Room 4©ICAROB 2019 ALife Robotics Corp. Ltd.

- 15 -

Meeting Room 31: Committee waiting room and Rest room

18:30-20:30

16:20-17:50

16:00-16:20
GS2 Robotics II (6)Chair: Jiwu Wang

OS25 High-Performance
Computing for Interactive
Content Creation and
Rendering (3) Chair: R.P.C.
Janaka Rajapakse

Coffee breakOS12 Robot Design and
Simulation (5)Chair: QinJian Zhang

Banquet: Hotel Shiragiku

OS15 Advanced Control (5)Chair: Yingmin Jia

14:10-14:30
GS4 Poster

Coffee break

13:10-14:10

14:30-16:00

Lunch
Chair: Akira Nakamura
Plenary Speech PS-1 (Conference Room)
Kensuke Harada

12:10-13:10

Meeting Room 1
OS9 Natural Computing
and Beyond (3)Chair: Marion Oswald

OS22 Robot Competitionsfor Social Contribution (6)Chair: Yasunori Takemura

11:10-12:10

OS23 Kansei andInformation Engineering (4)Chair: T. Hattori

Coffee break

GS3 Complexity (2)Chair: Manabu Sugii

Registration

Meeting Room 33

Coffee break

OS16 New development
about Control Engineering
Education (4)Chair: Yoshihiro Ohnishi

OS17 Multiagent andComplex Systems (3)Chair: Akihiro Yamaguchi

Meeting Room 32

TIME TEBLE (1/12)

Chair: Ju-Jang Lee (Y. I. Cho), Chair: Marion Oswald (K. Sumi)
Invited sessionIS-2, IS-3 (Conference Room)
Young Im Cho, Kaoru Sumi

GS4 Poster (5)Chair: Evgeni Magid

Meeting Room 31

11:00-11:10

10:00-11:00

9:45-10:00

9:00-9:45

8:40-

1/12(Sat.)

Conference RoomThe 2019 International Conference on Artificial Life and Robotics (ICAROB2019), B-Con Plaza, Jan. 10- 13, Beppu, Oita, Japan, 2019

OS19 Advanced
Technology on Sensing
Technology, Devices,
Application (3)Chair: Hiroki Tamura

Meeting Room 4©ICAROB 2019 ALife Robotics Corp. Ltd.

- 16 -

no presentation, only papers

GS5 Others ( 8 )Meeting Room 31

Chair: Jumpei Ono

Farewell Party (13:55-14:25)

Chair: Makoto Sakamoto

Meeting Room 31: Committee waiting room and Rest room

12:40-13:55

OS24 Automated content
generation for narrative andcognitive contents (5)Lunch
OS10 Mathematical
Informatics (5)11:40-12:40

Meeting Room 1

Chair: Takao Ito
Plenary Speech PS-3(Meeting Room 31)
Yingmin Jia

OS5 Business Models
and Management (4)Chair: Takao ItoRegistration

Meeting Room 33

10:40-11:40

OS6 Software
Development Support
Method (6)Chai: Tetsuro Katayama

Meeting Room 32

Coffee break

Meeting Room 31

10:30-10:40

9:00-10:30

8:40-

1/13(Sun.)

TIME TABLE (1/13)

The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), B-Con Plaza, Jan. 10- 13, Beppu, Oita, Japan, 2019

Meeting Room 4

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

The 2019 International Conference on
ARTIFICIAL LIFE AND ROBOTICS(ICAROB2019)
January 10 (Thursday)
17:30-19:30

Welcome Party (Hotel Shiragiku)

January 11 (Friday)
10:30-11:00

Opening Ceremony (Conference Room)

Chair: Marion Oswald (Technische Universität Wien, Austria)
Welcome Addresses
1. General Chairman of ICAROB
2. Co-General Chairman of ICAROB
3. Co-General Chairman of ICAROB
4. Co-General Chairman of ICAROB
5. Vice General Chair of ICAROB
6. Vice General Chair of ICAROB

Masanori Sugisaka (ALife Robotics Co., Ltd., Japan)
Yingmin Jia (Beihang University, P. R. China)
TaKao Ito (Hiroshima University, Japan)
Ju-Jang Lee (KAIST, Korea)
Henrik Hautop Lund (Technical University of Denmark,Denmark)Jangmyung Lee (Pusan National University, Korea)

January 12 (Saturday)
Banquet: Hotel Shiragiku
18:30-20:30
Chair: Takao Ito (Hiroshima University, Japan)
Welcome Addresses
Prof. Kensuke Harada (Osaka University, Japan)
Prof. Yingmin Jia ( Beihang University, P.R. China)
Prof. Henrik Hautop Lund (Technical University of Denmark, Denmark)Prof. Ju-Jang Lee (KAIST, Korea)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 17 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

TECHNICAL PAPER INDEXJanuary 11 (Friday)
8:40-Registration
Conference Room
10:30-11:00 Opening Ceremony
Chair: Marion Oswald (Technische Universität Wien, Austria)
11:10-11:40
Invited session IS1-1
Chair: Kuo-Hsien Hsia (Far East University, Taiwan)
IS1-1-1 Engineering Modular Playware
Henrik Hautop Lund, Morten Roed Frederiksen, Massimiliano Leggieri(Technical University of Denmark, Denmark)IS1-1-2 User-Friendly Robotics 3 – Playware
Henrik Hautop Lund, Luigi Pagliarini (Technical University of Denmark, Denmark)11:40-12:10
Invited session IS1-2
Chair: Jang-Myung Lee (Pusan National University, Korea)
IS1-2-1 User-Friendly Robotics 1 – AI Software
Luigi Pagliarini 1,2 , Henrik Hautop Lund 1(1Technical University of Denmark, Denmark, 2Academy of Fine Arts of Macerata, Italy)
IS1-2-2 User-Friendly Robotics 2– Interfacing Robotics with any user
Luigi Pagliarini1,2, Henrik Hautop Lund1(1Technical University of Denmark, Denmark) (2Academy of Fine Arts of Macerata, Italy)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 18 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 201913:10-14:10
Plenary Speech PS-2
Chair: Kazuo Ishii (Kyushu Institute of Technology, Japan)
PS-2 Toward Life with Partner Robots-Developing robots with the field trial toward the practical use
Masanori Sugiyama (Toyota Motor Corporation, Japan)
Meeting Room 31
9:00-10:45 OS8 Computational intelligence and cognitive science for human biosignals and humanwell-being (7)Chair: Tomoyuki Hiroyasu (Doshisha University, Japan)
Co-Chair: Hiroshi Furutani (Doshisha University, Japan)
OS8-1

Unified Approach to (1+1) EA on Discrete Linear Functions
Kenji Aoki, Makoto Sakamoto (University of Miyazaki, Japan)
Hiroshi Furutani, Satoru Hiwa, Tomoyuki Hiroyasu (Doshisha University, Japan)

OS8-2

A fNIRS study of brain state during letter and category fluency tasks
Akane Onishi, Satoru Hiwa, Hiroshi Furutani, Tomoyuki Hiroyasu (Doshisha University, Japan)

OS8-3

Construction of a meditation practice support system leads to a good meditation state: an fNIRS
study
Seika Fujii, Satoru Hiwa, Hiroshi Furutani, Tomoyuki Hiroyasu (Doshisha University, Japan)

OS8-4

Automated panoramic image creation system for corneal endothelial cells
Keitaro Kobayashi, Naoki Okumura, Noriko Koizumi, Satoru Hiwa, Hiroshi Furutani,
Tomoyuki Hiroyasu (Doshisha University, Japan)

OS8-5

An fMRI study of the inhibitory effects of the random stimulus-response compatibility task on
brain function
Kei Sahara, Satoru Hiwa, Hiroshi Furutani, Tomoyuki Hiroyasu (Doshisha University, Japan)

OS8-6

Measurement of brain activity and problem discovery during actual driving
Seishiro Nakamura, Satoru Hiwa, Kenya Sato, Hiroshi Furutani, Tomoyuki Hiroyasu(Doshisha University, Japan)

OS8-7

Control of driving simulator based on state detection of the driver using Electrocardiogrammeasurement
Koma Yoshioka, Satoru Hiwa, Kenya Sato, Hiroshi Furutani, Tomoyuki Hiroyasu(Doshisha University, Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 19 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 201915:50-17:50 OS20 Advances in Field Robotics and Sensing System (8)Chair: Eiji Hayashi (Kyushu Institute of Technology, Japan)
Co-Chair: Kazuo Ishii (Kyushu Institute of Technology, Japan)
OS20-1

Field Robot and Sensing system
Eiji Hayashi (Kyushu Institute of Technology, Japan)

OS20-2

Saliency Map Based on Maximization of Difference between Central and Surround Visions and
Its Relationship with Image Feature Points
Ruugo Mochizuki, Shinsuke Yasukawa, Kazuo Ishii (Kyushu Institute of Technology, Japan)

OS20-3

Dynamic Wireless Network System using Multiple Drones for Tunnel Applications
Raji Alahmad, Kazuo Ishii (Kyushu Institute of Technology, Japan)

OS20-4

Counting Crops under Cultivation using Drone
Yasunori Takemura1, Yusuki Hirata2, Eiji Mizoe3, Masao Tashiro4 and Yousuke Nagai5(1Nishinippon Institute of Technology, 2Kyushu Institute of Technology, 3Sky Canvas Co.
4
Joint Corp. Project Sky-eye, 5Aruku Aguriculture Service LLC, Japan)

OS20-5

Soil Compaction and Rolling Resistance Evaluation of a Locomotion System with Adjustable
Contact Patches for Applications in the Vineyard
Enrico di Maria, Kazuo Ishii (Kyushu Institute of Technology, Japan)

OS20-6

Robot Navigation in Forest Management Based on Graph
Ayumu Tominaga1, Ryusuke Fujisawa1, Eiji Hayashi1, Abbe Mowshowitz2(1Kyushu Institute of Technology, Japan, 2The City College of New York, USA)

OS20-7Development of Autonomous Moving System for Field Robot
Kengo Kawazoe, Eiji Hayashi, Ryusuke Fujisawa (Kyushu Institute of Technology, Japan)

OS20-8Development of the sense system that is combined force feedback and vision feedback-Improvement reproducibility of deformation simulation by using LEMTamon Shigeyama, Eiji Hayashi , Ryusuke Fujisawa (Kyushu Institute of Technology, Japan)

Meeting Room 32
9:00-10:00 OS1-1 Intelligent Systems and Control (4)Chair: Kuo-Hsien Hsia (Far East University, Taiwan)
Co-Chair: Chung-Wen Hung (National Yunlin University of Science & Technology, Taiwan)
OS1-1

A Vishay Supercapacitor based Fast Charge Battery
Chung-Wen Hung, Yi-Chang Yang, Chun-Chu Lin, Li-Sheng Jheng(National Yunlin University of Science and Technology, Taiwan)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 20 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS1-2

Multi-Motor Synchronous Control with CANOpen
Chung-Wen Hung, Roger CL Lee, Bo-Kai Huang, Shit-Ting Yu(National Yunlin University of Science and Technology, Taiwan)

OS1-3

Selection Strategy for VM Migration Method
Yan-Ren Chen, I-Hsien Liu, Keng-Hao Chang, Jung-Shian Li(National Cheng Kung University, Taiwan)

OS1-4

VM Migration Placement in Cloud Service
Yan-Ren Chen, I-Hsien Liu, Keng-Hao Chang, Chuan-Gang Liu, Jung-Shian Li(National Cheng Kung University, Taiwan)

14:30-15:30 OS1-2 Intelligent Systems and Control (4)Chair: Kuo-Hsien Hsia (Far East University, Taiwan)
Co-Chair: Chung-Wen Hung (National Yunlin University of Science & Technology, Taiwan)
OS1-5

Control System for Maintaining Safe Following Distance while Driving
Kuo-Hsien Hsia, Jia-Hong Cai, Shu-Li Pai (Far East University, Taiwan)
Evgeni Magid (Kazan Federal University, Russia)

OS1-6Markerless Indoor Augmented Reality Navigation Device Based on Optical-Flow-Scene Indoor
Positioning and Wall-Floor-Boundary Image Registration
Chian C. Ho, Bo-Kai Wang, Guan-Lung Liao
(National Yunlin University of Science and Technology, Taiwan)

OS1-7

Development of the handheld gas detector with IoT function.
Jr-Hung Guo, Kuo-Hsien Hsia, Kuo-Lan Su(National Yunlin University of Science and Technology, Taiwan)

OS1-8

Research on Employee Attribute Correlation of Information Security Awareness in Organization
Tse-Yao Wang (Air Force Institute of Technology, Taiwan)
Fu-Hsiang Wen (National Kaohsiung University of Science and Technology, Taiwan)

15:50-17:35 OS4 Service Robotics (7)Chair: Evgeni Magid (Kazan Federal University, Russia)
Co-Chair: Kuo-Hsien Hsia (Far East University, Taiwan),
Co-Chair: Kuo-Lan Su (National Yunlin University of Science and Technology, Taiwan)
OS4-1

Apply 2D Barcode Scanner for Mobile Robot Navigation in Checkerboard Mapping
Chun-Chi Lai1, Kuo-Lan Su1, Chia-Jen Lin1, Evgeni Magid²(1National Yunlin University of Science and Technology, Taiwan)(²Kazan Federal University, Russia)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 21 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS4-2

Development of the MyRio Based Mobile Platform
Bo-Jun Yang1, Sheng-Jie Liou1, Jr-Hung Guo1, Kuo-Lan Su1, Evgeni Magid²(1National Yunlin University of Science and Technology, Taiwan)(²Kazan Federal University, Russia)

OS4-3

Modelling Avrora Unior Car-like Robot in Gazebo Environment
Ksenia Shabalina1, Artur Sagitov1, Kuo-Lan Su², Kuo-Hsien Hsia³, Evgeni Magid1(1Kazan Federal University, Russia)(²National Yunlin University of Science and Technology, Taiwan)(³Far East University, Taiwan)

OS4-4Transfer of learned exploration strategies for a mobile robot from a simulated world to real
Environments
Artur Sagitov1, Tetsuo Takano2, Shohei Muto2, Evgeni Magid1(1Kazan Federal University, Russia) (2Kanazawa University, Japan)

OS4-5

Extending Gazebo simulator for surgical robotics: tissue and suture modeling
Artur Sagitov1, Hongbing Li2, Evgeni Magid1(1Kazan Federal University, Russia) (2Shanghai Jiao Tong University, China)

OS4-6

Experiences of Robotics students in Machine Vision course being taught in a foreign language:
comprehension, self-efficiency, and active learning strategies improvement
Tatyana Tsoy, Artur Sagitov, Evgeni Magid (Kazan Federal University, Russia)

OS4-7

Pilot Virtual Experiments on ArUco and AprilTag Systems Comparison for Fiducial Marker
Rotation Resistance
Aufar Zakiev1, Ksenia Shabalina1, Kuo-Lan Su², Kuo-Hsien Hsia³, Evgeni Magid1(1Kazan Federal University, Russia)(²National Yunlin University of Science and Technology, Taiwan)(³Far East University, Taiwan)

Meeting Room 33
9:00-10:00 OS14 Intelligent Control (4)Chair: Yingmin Jia (Beihang University (BUAA), China)
Co-Chair: Weicun Zhang (University of Science and Technology Beijing, China)
OS14-1

Flock Guiding of Hybrid Agents via Root Block
Yunzhong Song, Ziyi Fu, Fuzhong Wang (Henan Polytechnic University, China)

OS14-2Multiple Model Adaptive Control of Flexible Arm
Yingzhao Zhang1, Xiao Wang1, Handong Li1, Weicun Zhang2(1Guizhou University, 2University of Science and Technology Beijing, China)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 22 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS14-3Time-Varying Lyapunov Function for Mechanical Systems
Bin Zhang1, Yingmin Jia2
(1Beijing University of Posts and Telecommunications, 2Beihang University (BUAA), China)

OS14-4

Optimizing Control Parameters of Space Robot Manipulator for Pulsar X-ray Interference
Measurement
Qiang Chen1, Hengbin Zhang1, Xiaomin Bei1, Hanwen Zhang2(1Chinese Academy of Space Technology, 2Beijing Institute of Control Engineering, China)

14:30-15:30 OS7 Media Information Processing and Artificial Intelligence (4)Chair: Yasunari Yoshitomi (Kyoto Prefectural University, Japan)
Co-Chair: Masayoshi Tabuse (Kyoto Prefectural University, Japan)
OS7-1

Effectiveness of Data Augmentation in Automatic Summarization System
Tomohito Ouchi, Masayoshi Tabuse (Kyoto Prefectural University, Japan)

OS7-2

A System for Posting on SNS Portrait Selected Using Facial Expression Analysis While Writing
MessageTaro Asada1, Yuiko Yano2, Yasunari Yoshitomi1, Masayoshi Tabuse1(1 Kyoto Prefectural University, 2 Nara Guarantee Corporation, Japan)

OS7-3

A Method Using Wavelet Transform for Judging Character to be Inserted into Image
Yasunari Yoshitomi1, Takashi Sato2, Taro Asada1, Masayoshi Tabuse1(1 Kyoto Prefectural University, 2 Tokyo Institute of Technology, Japan)

OS7-4

An Authentication Method for Digital Audio Using Wavelet Transform and Fundamental
Frequencies
Yasunari Yoshitomi1, Shohei Tani2, Masaki Arasuna3, Ryota Kan4, Taro Asada1, Masayoshi Tabuse1(1 Kyoto Prefectural University, 2 Fukuchiyama City Hall, 3 Nissay Information Technology Co., Ltd.,
4
Shimazu Business Systems Co., Ltd., Japan)

15:50-17:05 OS2 Theory and Implementation of Neuromimetic Systems (5)Chair: Timothée Levi (The University of Tokyo, Japan)
Co-Chair: Yuichi Katori (Future University Hakodate, Japan)
OS2-1

Decoding spike patterns of auto-associative memory on spiking neuronal networks
Naoki Toyoshima, Takashi Kohno (The University of Tokyo, Japan)

OS2-2

Possible Mechanism of Internal Visual Perception: Context-dependent Processing by Predictive
Coding and Reservoir Computing Network
Hiroto Tamura1, Yuichi Katori1,2, Kazuyuki Aihara1(1The University of Tokyo, 2Future University Hakodate, Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 23 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS2-3

Snake robot controlled by biomimetic CPGs
Damien Blanchard1, Kazuyuki Aihara2, Timothée Levi2(1University of Bordeaux, France, 2The University of Tokyo, Japan)

OS2-4

Real-time implementation of ReSuMe learning in Spiking Neural Network
Xia Yang, Seiji Uenohara, Kazuyuki Aihara, Takashi Kohno, Timothée Levi
(The University of Tokyo, Japan)

OS2-5

Biomimetic spike-timing based ionic micro-stimulation for neuron culture
Stephany Mai Nishikawa1, Farad Khoyratee1,2, Soo Hyeon Kim1, Yoshiho Ikeuchi1, rawr
Kazuyuki Aihara1, Teruo Fujii1, Timothée Levi1(1The University of Tokyo, Japan, 2University of Bordeaux, France)

OS2-6

Dynamical network model for visual cortex: hierarchical reservoir computing with bidirectional
interactions (withdraw)
Shohei Takaichi1, Yuichi Katori2, Kazuyuki Aihara1(1The University of Tokyo, 2Future University Hakodate, Japan)

Meeting Room 1
9:00-10:15 OS21 Advances in Marine Robotics and It’s Applications (5)Chair: Keisuke Watanabe (Tokai University, Japan)
Co-Chair: Kazuo Ishii (Kyushu Institute of Technology, Japan)
OS21-1

System Development of AUV’s Sampling Device Controller Employing MATLAB/Simulink
Toolboxes
Takashi Sonoda1, Shinsuke Yasukawa2, Ahn Jonghyun2, Yuya Nishida2, Kazuo Ishii2(1Nishinippon Institute of Technology, 2Kyushu Institute of Technology, Japan)

OS21-2

Inspection System for Underwater Structure of Bridge Pier
Takumi Ueda1, Hiyoyasu Hirai1, Kazuki Fuchigami1, Ryoma Yuki1, Ahn Jonghyun1, Shinsuke
Yasukawa1, Yuya Nishida1, Takashi Sonoda1, Kazuo Ishii1, Katsunori Higashi2, Katsunori Tanaka2,
Tomomasa Ikeda2(1Kyushu Institute of Technology, 2West Nippon Expressway Engineering Kyushu, Japan)

OS21-3Three-dimensional measurement using laser pattern and its application to underwater scanner
Yuya Nishida, Tomoya Shinnoki, Shinsuke Yasukawa, Kazuo Ishii(Kyushu Institute of Technology, Japan)

OS21-4Development of Dam Inspection Underwater Robot
Hiroyasu Hirai, Kazuo Ishii (Kyushu Institute of Technology, Japan)

OS21-5

Conceptual Design of Small ROV for Sky to Water System
Keisuke Watanabe, Koshi Utsunomiya, Kazumasa Harada, Nakajima Shuhei(Tokai University, Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 24 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 201914:30-15:30 OS3 Intelligent navigation & Robotics (4)Chair: Chan Gook Park (Seoul National University, Korea)
Co-Chair: Jang-Myung Lee (Pusan National University, Korea)
OS3-1

Trajectory tracking control for a 7-arms robot manipulator
Wang Jie, Hyun-Hee Kim, Saad abbasi, Min-Cheol Lee (Pusan National University, Korea)

OS3-2

Estimation tire-rod friction coefficient based on mobile robot
Ji-Hyeon Kim, Zhang Hongyu, Jang-Myung Lee (Pusan National University, Korea)

OS3-3

Real-time reflection removal algorithm using stereo camera only
Do-Kyung Hwang, Jong-Woo An, Jang-Myung Lee (Pusan National University, Korea)

OS3-4

The Effect of Inertial Measurement Unit on Synthetic Aperture Radar Image Quality
Soo-Jeong Lee1, Yong-Gonjong Park1, Woo-Jung Park1, Chan Gook Park1, Jong-Hwa Song2(Seoul National University, Korea1, Hanwha Systems, Korea2)

15:50-17:05 OS18 Data-Driven Control and Diagnosis Control Systems (5)Chair: Takuya Kinoshita (Hiroshima University, Japan)
Co-Chair: Takao Sato (University of Hyogo, Japan)
OS18-1A Design Scheme of a Data-driven Predictive-PI Controller
Yoichiro Ashida, Shin Wakitani and Toru Yamamoto (Hiroshima University, Japan)

OS18-2

Improvement in Intersample Response of Multirate Regulation Control
Takao Sato, Natsuki Kawaguchi, Nozomu Araki, Yasuo Konishi (University of Hyogo, Japan)

OS18-3

Design of a PID Controller using a Fictitious Exogenous Signal for a Fluctuation System
Masatoshi Kozui, Takuya Kinoshita, Toru Yamamoto (Hiroshima University, Japan)

OS18-4

Izhikevich Model Based Self-Repairing Control for Plants with Sensor Failures and Disturbances
Masanori Takahashi (Tokai University, Japan)

OS18-5

Design of a Data-Driven Controller with Evaluating Controller Performance
Takuya Kinoshita and Toru Yamamoto (Hiroshima University, Japan)

Meeting Room 4
9:00-10:15 OS11 Recognition and Control (12)Chair: Fengzhi Dai (Tianjin University of Science and Technology, China)
Co-Chair: Yizhun Peng (Tianjin University of Science and Technology, China)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 25 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS11-1

Design of feed part control system for rectification process
Lingran An1, Fengzhi Dai1, Yujie Yan1, Zhongyong Ye1, Xia Jin1, Yiqiao Qin12, Chengcai Wang3,
Kaige Liu1 (1Tianjin University of Science and Technology, 2Tianjin Tianke Intelligent andManufacture Technology CO.,LTD, 3CETC Ocean Information Co., Ltd, China)

OS11-2Ultrasonic range finder for vehicle collision avoidance system
Ying Chen1, Guowei Yang1, Fengzhi Dai1, Yuxuan Zhu1, Di Yin1, Yasheng Yuan1,Yiqiao Qin12, Ce
Bian12, Chengcai Wang3, Xinyu Zhang1 (1Tianjin University of Science and Technology,
2
Tianjin Tianke Intelligent and Manufacture Technology CO.,LTD, 3CETC Ocean Information Co.,
Ltd, China)

OS11-3

Chaos synchronization method of Qi system and the circuit design
Xia Jin 1 , Fengzhi Dai 1 , Zhongyong Ye 1 , Lingran An 1 , Yujie Yan 1 , Chenglin Zhao 1 ,Yiqiao Qin 12 , Hao Li 3 ( 1 Tianjin University of Science and Technology, 2 Tianjin
Tianke Intelligent and Manufacture Technology CO.,LTD, 3 CETC Ocean Information
Co., Ltd,, China)

OS11-4

Design of air quality monitoring platform based on Internet of Things
Yujie Yan 1 , Fengzhi Dai 1 , Kailun Zhang 1 , Wei Wang 2 , Jialin Han 1 , Yang Li 3 , Tianyi Zhang 1( 1 Tianjin University of Science and Technology, 2 China Institute of Aerospace
Engineering , 3 CETC Ocean Information Co., Ltd, China)

OS11-5

Design of metal weld seam tracking equipment based on image processing
Yujie Yan 1 , Fengzhi Dai 1 , Lingran An 1 , Xia Jin 1 , Zhongyong Ye 1 , Yufan He 2 , Xinran Guo 3( 1 Tianjin University of Science and Technology, 2 CETC Ocean Information Co., Ltd,
3
Nanjing University, China)

OS11-6

The recognition and implementation of handwritten character based on deep learning
Zhongyong Ye 1 , Fengzhi Dai 1 , Xia Jin 1 , Yasheng Yuan 1 , Lingran An 1 , Yujie Yan 1 ,Yiqiao Qin 12 , Hao Li 3 ( 1 Tianjin University of Science and Technology, 2 Tianjin Tianke
Intelligent and Manufacture Technology CO., LTD, 3 CETC Ocean Information Co., Ltd,
China)

OS11-7

Cluster analysis of wine based on three-dim fluorescent spectra characteristic
Di Yin 1 , Fengzhi Dai 1 , Yuxuan Zhu 1 , Yasheng Yuan 1 , Tingting Zhang 1 , Ying Chen,
Yiqiao Qin 12 , Chengcai Wang 3 ( 1 Tianjin University of Science and Technology, 2 Tianjin
Tianke Intelligent and Manufacture Technology CO., LTD, 3 CETC Ocean Information
Co., Ltd, China)

OS11-8The control design of the planting device for the hydroponic vegetable
Yasheng Yuan 1 , Fengzhi Dai 1 , Zhongyong Ye 1 , Yuxuan Zhu 1 , Di Yin 1 , Tingting Zhang 1 ,Yiqiao Qin 12 , Yang Li 3 ( 1 Tianjin University of Science and Technology, 2 Tianjin Tianke
Intelligent and Manufacture Technology CO.,LTD, 3 CETC Ocean Information Co., Ltd,
China)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 26 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS11-9An anti -theft system based on the self-checkout
Yuxuan Zhu 1 , Fengzhi Dai 1 , Yasheng Yuan 1 , Di Yin 1 , Tingting Zhang 1 , Ying Chen 1 ,Ce Bian 12 , Yufan He 3 ( 1 Tianjin University of Science and Technology, 2 Tianjin Tianke
Intelligent and Manufacture Technology CO.,LTD, 3 CETC Ocean Information Co., Ltd,
China)

OS11-10

Case study on communication between embedded linux environment andmicrocontroller
Ziheng Gao, Yizhun Peng, Shuo Wang
(Tianjin University of Science and Technology, China)

OS11-11

Design and implementation of a baby care robot
Yizhun Peng, Zefan Ge, Kaixuan Geng, Zhou Yang,Rui Yang
(Tianjin University of Science and Technology, China)

OS11-12

Research on intelligent shopping service robot design
Shuo Wang, Yizhun Peng, Ruixiang Bai, Ziheng Gao(Tianjin University of Science and Technology, China)

14:30-15:15 GS1 Robotics I (3)Chair: Jiwu Wang (Beijing Jiaotong University, China)
GS1-1

System to decide unit’s layout of cell assembly machine by GA (Big Mutation in mutation process)Yoshiki Ito, Hidehiko Yamamoto, Takayoshi Yamada (Gifu University, Japan)

GS1-2

Autonomous decentralized FMS that adopts priority ranking structure based on AGV’s lies
Kento Uejo, Hidehiko Yamamoto, Takayoshi Yamada (Gifu University, Japan)

GS1-3

CULET: Cubelet Lego Turing Machine
Ricardo Q. Figueroa, Daniel A. Zamorano, Genaro J. Martínez (Instituto Politécnico Nacional,
México), Andrew Adamatzky (University of the West of England, UK)

15:50-17:50 OS13 System and Control (10)Chair: Qiang Wei (Jianghan University, China)
Co-Chair: Huailin Zhao (Shanghai Institute of Technology, China)
OS13-1

Research on a non-invasive measuring method of blood glucose concentration based on electrical
impedance spectrum
Tong Yin, Xiaoyan Chen, Meng Du, Hongyi Yin(Tianjin University of Science and Technology, China)

OS13-2Synchronization of Novel 4D Chaotic Systems with Different Control Laws
Hong Niu (Tianjin University of Science and Technology, China)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 27 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS13-3Chaotic characteristics analysis of fractional-order Liu system
Wengxin Shi, Hongyan Jia (Tianjin University of Science and Technology, China)

OS13-4

Design and analysis of the multi-robot grouping aggregation algorithm
Huailin Zhao, Zhen Nie (Shanghai Institute of Technology, China)

OS13-5

Research of the ARM-based multi-robot aggregation control
Huailin Zhao, Jingnan Zheng, Zhen Nie, Weiya Zhang, Zonglin Liu(Shanghai Institute of Technology, China)

OS13-6

Research on the Trajectory Planning Problem of Robots in Spherical Cutting
Jun Min, Wenping Jiang, Huailin Zhao (Shanghai Institute of Technology, China)

OS13-7

Emotional processing between artificial voices and human voices
Qiang Wei1, Xiaoyu Zhang1, Xiao Yang1, Yixin Lin2, Qingming Liu3
(1Jianghan University, 2Huazhong University of Science and Technology, 3Wuhan Rayson School,
China)

OS13-8How people are affected by emoticon icon: An ERP study
Qiang Wei1, Jingxuan Huang1, Yu Han1, Yixin Lin2
(1Jianghan University, 2Huazhong University of Science and Technology, China)

OS13-9Single Chinese character fragments: an ERPs study on orthographic neighborhood effect
Qiang Wei1, Yunfei Wang1, Chen Song1, Yixin Lin2
(1Jianghan University, 2Huazhong University of Science and Technology, China)

OS13-10

Research on embedded electrical impedance measurement system
Tong Yin, Xiaoyan Chen, Hongyi Yin (Tianjin University of Science and Technology, China)

January 12 (Saturday)
8:40-Registration
Conference Room
11:10-11:40
Invited session IS-2
Chair: Ju-Jang Lee (KAIST, Korea)
IS-2 A New Machine Learning Algorithm Applicable for Weather Visibility and Food Recognition
Young Im Cho (Gachon University, Korea)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 28 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 201911:40-12:10
Invited session IS-3
Chair: Marion Oswald (Technische Universität Wien, Austria)
IS-3 Affective Interactive Systems
Kaoru Sumi (Future University Hakodate, Japan)
13:10-14:10
Plenary Speech PS-1
Chair: Akira Nakamura (National Institute of Advanced Industrial Science and Technology, Japan)
PS-1 Motion-Data Driven Grasp/Assembly Planner
Kensuke Harada12, Natsuki Yamanobe2, Weiwei Wan12, Kazuyuki Nagata2
Ixchel G. Ramirez-Alpizar1, Tokuo Tsuji3 (1Osaka University, 2National Institute of Advanced
Industrial Science and Technology, 3Kanazawa University, Japan)
Meeting Room 31
10:00-16:00 GS4 Poster (5)Chair: Evgeni Magid (Kazan Federal University, Russia)
GS4-1

Relevance Research among Destination Image, Brand Trust, and Satisfaction: Tainan Festival in
Taiwan as Example
Shang-Hui Li, Shu-Fang Hsu, Yi-Tai Shang (Far East University, Taiwan)

GS4-2

Third-party Logistics Service Supply: Catering Service Trade as Example
Shu-Fang Hsu, Shang-Hui Li (Far East University, Taiwan)

GS4-3

Pedal Scale Control Device for Weighing Food Ingredients or Chemical Materials
Shang-Hui Li, Shu-Fang Hsu, Yi-Tai Shang, Kuan-Ying Chen (Far East University, Taiwan)

GS4-4

Convolution Neural Network Based Fault Diagnosis of Induction Motors
Jong-Hyun Lee, In-Soo Lee(Kyungpook National University, Korea)

GS4-5

Analysis of Value Chain on Food and Beverage Micro-enterprises: Mobile Diner as Example
Shu-Fang Hsu, Shang-Hui Li (Far East University, Taiwan)

Meeting Room 32
9:00-9:45 OS17 Multiagent and Complex Systems (3)Chair: Akihiro Yamaguchi (Fukuoka Institute of Technology, Japan)
Co-Chair: Saori Iwanaga (Japan Coast Guard Academy, Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 29 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS17-1

Different Type of Interaction or Decision Error contribute to Functional Differences
Saori Iwanaga1, Masao Kubo2(1Japan Coast Guard Academy, 2National Defense Academy of Japan, Japan)

OS17-2

Evaluation for the Synchronization of the Parade with OpenPose
Yohei Okugawa, Masao Kubo, Hiroshi Sato, Bui Duc Viet(National Defense Academy of Japan, Japan)

OS17-3Phase shift estimation of the bifurcating neuron from superimposed chaotic spike sequences
Akihiro Yamaguchi1, Yutaka Yamaguti1, Masao Kubo2(1Fukuoka Institute of Technology, 2National Defense Academy of Japan, Japan)

10:00-11:00 OS16 New development about Control Engineering Education (4)Chair: Yoshihiro Ohnishi (Ehime University, Japan)
Co-Chair: Kazuo Kawada (Hiroshima University, Japan)
OS16-1

Skill Model Estimation of Ability for Reading Drawings
Kazuo Kawada, Tsukasa Hiyama (Hiroshima University, Japan）

OS16-2

Learning simulation based on a computational model of neuromodulators
Yuki Moriguchi, Masayasu Nagamatsu(Hiroshima University, Japan)

OS16-3

Estimation of Programming Learning Achievement by Line Tracing Robot
Yoshihiro Ohnishi1, Teruyuki Tamai1, Shinnosuke Mori1, Kawada Kazuo2（1Ehime University, 2Hiroshima University, Japan）

OS16-4Development of Cultivate Computational Thinking using Finger Robot
Kaito Omata, Shinichi Imai (Tokyo Gakugei University, Japan)

14:30-15:45 OS15 Advanced Control (5)Chair: Yingmin Jia (Beihang University (BUAA), China)
Co-Chair: Weicun Zhang (University of Science and Technology Beijing, China)
OS15-1

Pulse Pose Follow Control and Simulation for a 4-DOF Pulse Diagnosis Robot
Qunpo Liu1, Guanghui Liu1, Hongqi Wang 1, Xianzhe Liu1, Hanajima Naohiko2(1Henan Polytechnic University, China, 2Muroran Institute of Technology, Japan)

OS15-2

A New Adaptive Control System Design Method Based on Neuronetwork
Prediction
Weicun Zhang, Sufang Wang, Yongnan Jia, Qing Li(University of Science and Technology Beijing, China)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 30 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS15-3

Mean-square quasi-composite rotating formation control of second-ordermulti-agent systems under stochastic communication noise
Lipo Mo1, Xiaolin Yuan1, Yingmin Jia2, Shaoyan Guo3
(1Beijing Technology and Business University, 2Beihang University (BUAA),
3
South China University of Technology, China)

OS15-4

Distributed Energy Resource Control Based on Multi-Agent Group Consensus
Yize Yang1,2, Hongyong Yang1, Fan Liu1, Yuling Li1, Yuanshan Liu1
(1Ludong University, China, 2The University of New South Wales, Australia)

OS15-5

Robust Adaptive Control of Air-to-air Refueling Boom with State-dependent Output Constraints
Liang Chang, Yingmin Jia (Beihang University (BUAA), China)

16:20-17:50 GS2 Robotics II (6)Chair: Jiwu Wang (Beijing Jiaotong University, China)
GS2-1

Automated Guided Vehicle System Analysis in Foam Manufacturing Plant Using Petri-net
Amornphun Phunopas1, Wisanu Jitviriya1, Noppadol Pudchuen1, Sutee Kumjaikong1,
Songklod Tunsiri2, Eiji Hayashi3 (1King Mongkut’s University of Technology North Bangkok,
Thailand, 2Urban Community, Development College, Thailand,
3
Kyushu Institute of Technology, Japan)

GS2-2

Preliminary Comparative Experiments of Support Vector Machine and Neural Network for EEGbased BCI Mobile Robot Control
Yasushi Bandou, Takuya Hayakawa, Jun Kobayashi (Kyushu Institute of Technology, Japan)

GS2-3

Position Tracking Error Constrained Super-Twisting Dynamic Surface Control with DisturbanceObserver for Robot Manipulators
Seong Ik Han (Dongguk University, Korea Republic)

GS2-4

Image Processing for Picking Task of Random Ordered PET Drinking Bottles
Chen Zhu, Takafumi Matsumaru (Waseda University, Japan)

GS2-5

The Use of Importance Ranks to Derive Suitable Timing of Visual Sensing in Manipulation Task
Containing Error Recovery
Akira Nakamura1, Kazuyuki Nagata1, Kensuke Harada2, Yukiyasu Domae1(1National Institute of Advanced Industrial Science and Technology (AIST),
2Osaka University, Japan)

GS2-6-1

Electronic Measurement and Gamification of Balance Tests
Yan-Xin Liu, Massimiliano Leggieri, Henrik Hautop Lund
(Technical University of Denmark, Denmark)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 31 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

GS2-6-2

Playful Body and Brain Test with the Moto Tiles
Yan-Xin Liu, Massimiliano Leggieri, Henrik Hautop Lund
(Technical University of Denmark, Denmark)Meeting Room 33
9:00-9:30 GS3 Complexity (2)Chair: Manabu Sugii (Yamaguchi University, Japan)
GS3-1

Extracting Co-occurrence Feature of Words for Mail filtering
Seiya Temma, Manabu Sugii, Hiroshi Matsuno (Yamaguchi University, Japan)

GS3-2

Learning Style Classification with Weighted Distance Grey Wolf Optimization
Duangjai Jitkongchuen, Piyalak Pongtawevirat (Dhurakij Pundit University, Thailand)

10:00-11:00 OS23 Kansei and Information Engineering (4)Chair: Chair: Tetsuo Hattori (Kagawa University)
Co-Chair: Yoshiro Imai (Kagawa University)OS23-1

Color Image Arrangement Based on Histogram Matching Using Smoothed
Brightness Histogram (I) --- Overall Smoothing --Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai, Koji Kagawa, Yo Horikawa (Kagawa University,
Japan), R. P. C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)

OS23-2

Color Image Arrangement Based on Histogram Matching Using Smoothed
Brightness Histogram (II) --- Piecewise Smoothing--Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai, Koji Kagawa, Yo Horikawa (Kagawa University,
Japan), R. P. C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)

OS23-3

Parameter Estimation Method for Compartment Model - PET Inspection Tetsuo Hattori, Yusuke Kawakami, Yoshiro Imai, Koji Kagawa, Yo Horikawa (Kagawa University,
Japan), Hiromichi Kawano (NTT Advanced Technology Company Ltd, Japan)

OS23-4

Application and Trial Evaluation of Document Writing Support System to Avoid Emotional
Misunderstanding
Shunsuke Doi, Yoshiro Imai, Koji Kagawa, Tetsuo Hattori, Yusuke Kawakami(Kagawa University, Japan)

14:30-15:45 OS12 Robot Design and Simulation (5)Chair: QinJian Zhang (Beijing Jiaotong University, China)
Co-Chair: Wei Liu (Beijing Jiaotong University, China)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 32 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS12-1

Design and Research of Navigation and Dynamics Co- Simulation Platform for Planetary Rover
Huang Tie-qiu, Zhang Bo-wen, Huang Qian-li (Beijing Jiaotong University, P.R China)

OS12-2

Research on Application of SFM Method in Virtual Reality Modeling
Jiwu Wang, Chenyang Li (Beijing Jiaotong University, China)
Min Li (Beijing Forestry University, China)

OS12-3Visual SLAM System Design based on Semantic Segmentation
Jiwu Wang, Yafan Liu, Qinjian Zhang (Beijing Jiaotong University, China)

OS12-4

Spherical Mobile Robot Designed with Single Omnidirectional Wheels
Wei Liu, Lian Luo, Jiwu Wang (Beijing Jiaotong University, China)

OS12-5

An augmented reality implementation method based on Unity3D
Weixin Zeng, Jiwu Wang (Beijing Jiaotong University, China)

16:20-17:05 OS25 High-Performance Computing for Interactive Content Creation and Rendering (3)Chair: R.P.C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)
Co-Chair: Chia-Chen Kuo (National Center for High-performance Computing (NCHC), Taiwan)
OS25-1High-Performance Computing for Visual Simulations and Rendering
Jasmine Wu, Chia-Chen Kuo, Shu-Hsin Liu, Chuan-Lin Lai, Chiang-Hsiang Lien, Ming-Jen Wang,
Chih-Wei Wang (National Center for High-performance Computing (NCHC), Taiwan)

OS25-2

Virtual Reality as an Art Form
R.P.C. Janaka Rajapakse, Yi-ping Hung (Tainan National University of the Arts, Taiwan)

OS25-3Using Quill as a Tool for Real-Time Rendering
Jasmine Wu, Chia-Chen Kuo, Shu-Hsin Liu, Chuan-Lin Lai, Chiang-Hsiang Lien, Ming-Jen Wang,
Chih-Wei Wang (National Center for High-performance Computing (NCHC), Taiwan)

Meeting room 1
9:00-9:45 OS9 Natural Computing and Beyond (3)Chair: Marion Oswald (Technische Universität Wien, Austria)
Co-Chair: Yasuhiro Suzuki (Nagoya University, Japan)
OS9-1

Tactile Score: Development and Applications
Yasuhiro Suzuki (Nagoya University, Japan)

OS9-2

Tactileology; Haptic Informatics by using Tactile bit, T-bit
Yasuhiro Suzuki (Nagoya University, Japan)

OS9-3

Natural Computing and Formal Computing
Yasuhiro Suzuki (Nagoya University, Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 33 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 201914:30-16:00 OS22 Robot Competitions for Social Contribution (6)Chair: Yasunori Takemura (Nishinippon Institute of Technology, Japan)
Co-Chair: Kazuo Ishii (Kyushu Institute of Technology, Japan)
OS22-1

Study on Position Estimation Using Small Size ZigBee Module
Atsushi Sanada (Nishinippon Institute of Technology, Japan)

OS22-2The 4th Tomato Harvesting Robot Competition
Takayuki Matsuo1, Takashi Sonoda2, Yasunori Takemura2, Kazuo Ishii3(1National Institute of Technology, Kitakyushu College, 2NishiNippon Institute ofTechnology, 3Kyushu Institute of Technology, Japan)

OS22-3End-effector for robotic harvesting of a tomato fruit with calyx
Binghe Li, Shinsuke Yasukawa, Takuya Fujinaga, Kazuo Ishii(Kyushu Institute of Technology, Japan)

OS22-4

Evaluation for Ball Dribbling Mechanism of RoboCup Middle Size League Robot in The World
Teams
Kenji Kimura, Shota Chikushi, Kazuo Ishii (Kyushu Institute of Technology)

OS22-5

Robot-control method based on personal space
Kota Kawamoto, Eiji Hayashi, Ryusuke Fujisawa (Kyushu Institute of Technology)

OS22-6

Recognition method of target objects for autonomous tomato harvesting robot
Keiji Kamei, Hiroyuki Moriyama (Nishinippon Institute of Technology, Japan)

Meeting room 4
9:00-9:45 OS19 Advanced Technology on Sensing Technology, Devices, Application (3)Chair: Hiroki Tamura (University of Miyazaki, Japan)
Co-Chair: Keiko Sakurai (University of Miyazaki, Japan)
OS19-1The Actual Car Driving Evaluation System using Combined with Eyeball and Face Angle
Keiko Sakurai, Hiroki Tamura, Koichi Tanno (University of Miyazaki, Japan),
Yukio Hattori (OFA SUPPORT INC, Japan)

OS19-2

A Study on Speaker Identification Approach by Feature Matching Algorithm using Pitch and Mel
Frequency Cepstral Coefficients
Barlian Henryranu Prasetio, Keiko Sakurai, Hiroki Tamura, Koichi Tanno(University of Miyazaki, Japan)

OS19-3

A Study on Breathing and Heartbeat Monitoring System during Sleeping using
Multi-Piezoelectric Elements
Praveen Nuwantha Gunaratne, Chika Yoshida, Keiko Sakurai, Hiroki Tamura, Koichi Tanno,(University of Miyazaki, Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 34 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

January 13 (Sunday)
8:40-Registration
Meeting Room 31
10:40-11:40
Plenary Speech PS-3
Chair: Takao Ito (Hiroshima University)
PS-3 Robust Consensus Control of Multi-vehicle Systems
Yingmin Jia (Beihang University, P.R.China)
Meeting Room 32
9:00-10:30 OS6 Software Development Support Method (6)Chair: Tetsuro Katayama (University of Miyazaki, Japan)
Co-Chair: Tomohiko Takagi (Kagawa University, Japan)
OS6-1

Prototype of an Animated Graphics-Based Training Support Tool for Bug Fixing of Extended
Place/Transition Nets
Tomohiko Takagi, Shogo Morimoto, Yuki Ue, Yoshiro Imai (Kagawa University, Japan)

OS6-2

Prototype of a Modeling Tool to Convert between Extended Place/Transition Nets and
VDM++ Specifications
Tomohiko Takagi, Ryo Kurozumi (Kagawa University, Japan)

OS6-3

Application of Pairwise Testing to Test Cases by Boundary Value Analysis in BWDM
Futa Hirakoba*, Tetsuro Katayama*, Yoshihiro Kita†, Hisaaki Yamaba*, Kentaro Aburada*,
Naonobu Okazaki*(*University of Miyazaki, †Tokyo University of Technology, Japan)

OS6-4

Development of Library Fescue Extracting Elements of Attributes and Operations of Class
Diagram in UML
Keisuke Mori*, Tetsuro Katayama*, Yoshihiro Kita†, Hisaaki Yamaba*, Kentaro Aburada*,
Naonobu Okazaki* (*University of Miyazaki, †Tokyo University of Technology, Japan)

OS6-5

Tamias: a Syntax File Checker for Parsing Expression Grammar
Toshihiro Miyaji*, Tetsuro Katayama*, Yoshihiro Kita†, Hisaaki Yamaba*, Kentaro Aburada*,
Naonobu Okazaki* (*University of Miyazaki, †Tokyo University of Technology, Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 35 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS6-6

Implementation of an Arduino Simulator to Support Circuit Design
Tatsumi Nishida*, Tetsuro Katayama*, Yoshihiro Kita†, Hisaaki Yamaba*, Kentaro Aburada*
And Naonobu Okazaki* (*University of Miyazaki, †Tokyo University of Technology, Japan)

12:40-13:55 OS10 Mathematical Informatics (5)Chair: Makoto Sakamoto (University of Miyazaki, Japan)
Co-Chair: Amane Takei (University of Miyazaki, Japan)
OS10-1

A Study on Tourism Support Application Using the Virtual Technology
Masamichi Hori1, Makoto Sakamoto1, Takahiro Ishizu1, Satoshi Ikeda1, Amane Takei1,
Takao Ito2, Yu-an Zhang3 (1University of Miyazaki, 2Hiroshima University, Japan,
3
Qinghai University, China)

OS10-2

Proposal for Interaction Techniques for Intuitive Virtual Objects Manipulation in Augmented
Reality
Takahiro Ishizu1, Makoto Sakamoto1, Takaaki Toyota1, Masamichi Hori1, Satoshi Ikeda1,
Amane Takei1, Takao Ito2, Yu-an Zhang3(1University of Miyazaki, 2Hiroshima University, Japan, 3Qinghai University, China)

OS10-3

Two-Dimensional Image Based Body Size Measurement and Body Weight Estimation for YaksZijie Sun1, Chen Zhang1, XiaoFeng Qin1, Yu-an Zhang1, Rende Song2, Makoto Sakamoto3(1Qinghai University, 2Yushu Prefecture Animal Husbandry and Veterinary Station, China
3
University of Miyazaki, Japan)

OS10-4

An efficient structure of organization with complete group guidanceSatoshi Ikeda1, Mamoru Yoshimura1, Makoto Sakamoto1, Takao Ito2
(1University of Miyazaki, 2Hiroshima University, Japan)

OS10-5

Parallel finite element analysis for hyperbolic problems
Amane Takei,, Makoto Sakamoto (University of Miyazaki, Japan)

Meeting Room 33
9:00-10:00 OS5 Business Models and Management (4)Chair: Takao Ito (Hiroshima University, Japan)
Co-Chair: Minoru Kumano (University of Miyazaki, Japan)
OS5-1

Determining the Key Factors of Michinoeki in Yamaguchi Area
Minoru Kumano1, Tsutomu Ito2, Takao Ito3, Toru Hiraoka4, Hirofumi Nonaka5(1University of Miyazaki, 2Hino Motors, Ltd., 3Hiroshima University,
4
University of Nagasaki, 5Nagaoka University of Technology, Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 36 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS5-2

Discovering the Characteristics of Michinoeki in Japan
Minoru Kumano1, Tsutomu Ito2, Takao Ito3, Toru Hiraoka4, Hirofumi Nonaka5(1University of Miyazaki, 2Hino Motors, Ltd., 3Hiroshima University,
4
University of Nagasaki, 5Nagaoka University of Technology, Japan)

OS5-3

An Analysis of Robotic Relationship between Transaction Network and Cross-shareholding
Network in Yokokai
Takao Ito1, Tsutomu Ito2, Rajiv Mehta3, Seigo Matsuno4, Makoto Sakamoto5, Satoshi Ikeda5(1Hiroshima University, Japan, 2Hino Motors, Ltd., Japan, 3New Jersey Institute of Technology,
U.S.A., 4Ube National College of Technology, Japan, 5University of Miyazaki, Japan)

OS5-4

Technological Discontinuities and the R&D Strategy of Automobile Companies
Yousin Park1, Iori Nakaoka2, Yun-ju Chen3 (1Prefectural University of Hiroshima, 2National Instituteof Technology, Ube College, 3Shiga University, Japan)

12:40-13:55 OS24 Automated content generation for narrative and cognitive contents (5)Chair: Jumpei Ono (Vocational School of Digital Arts Sendai, Japan)
Co-Chair: Hiroki Fukushima (Kyushu Womens’ University, Japan)
Co-Chair: Takashi Ogata (Iwate Prefectural University, Japan)
OS24-1

A Method of Haiku Generation Using Deep Learning for Advertising Generation
Jumpei Ono (Vocational School of Digital Arts Sendai, Japan)
Takashi Ogata (Iwate Prefectural University, Japan)

OS24-2The Usage Features of Onomatopoeias in the Recipes in Japanese
Hiroki Fukushima (Kyushu Womens’ University, Japan)

OS24-3

Storytelling in the Conversation of Aged People
Yuki Hayashi (Chiba University / Center for Advanced Intelligence Project, Riken, Japan)
Akinori Abe (Chiba University /Dwango Artificial Intelligence Laboratory, Japan)

OS24-4

Expression of the taste of Japanese sake and metaphor
Akinori Abe (Chiba University / Dwango Artificial Intelligence Laboratory, Japan)

OS24-5An Analysis on Advertising Techniques of Beverages Using Positive Factors of Evaluation
Database System
Yoji Kawamura (Kindai University, Japan)©ICAROB 2019 ALife Robotics Corp. Ltd.

- 37 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

GS5 Others (8)GS5-1

Suppression of Roll Oscillation in Turning of Quadruped Robot by Asymmetric Amplification ofCentral Pattern Generator Output Waveform
Makoto Kitani, Ryo Asami, Noritake Sato (Nagoya Institute of Technology, Japan),
Tomofumi Fujiwara, Takahiro Endo, Fumitoshi Matsuno (Kyoto University, Japan),
Yoshifumi Morita (Nagoya Institute of Technology, Japan)

GS5-2

Tumble avoidance system for rescue robot by estimating the contact points using a 3D depth
sensor
Noritaka Sato, Kotaro Ohshima, Yoshifumi Morita (Nagoya Institute of Technology, Japan)

GS5-3

Rule based Intrusion Detection System by Using Statistical Flow Analysis Technique forSoftware Defined Network
Mahnoor Ejaz, Osama Sohail, Talha Naqash, Zain ul Abideen, Sajjad Hussain Shah
(Bahria University, Pakistan)

GS5-4

Development of Testbed for SDN to Know Its Feasibility of Deployment on Access Layer
Rafay Shah, M Osama Shaikh, Talha Shaikh, Tanveer Hussain(Mehran University of Engineering & Technology, Pakistan)GS5-5Spectrum Sensing using Unsupervised Learning for Cognitive Radio
Asmara Shaukat, Danish Khan, Talal Arshad, Haseeb Ahmad (PAF-KIET, Pakistan)

GS5-6

Tracking Secondary User In Cognitive Radio For 5g Communication
Muhammad Adnan, Muhammad Umair, Sameed bin Junaid（PAF-KIET, Pakistan）

GS5-7

Wireless Power Transfer And Data Communication For Biomedical Applications
Hammad Saleem, Muhammad Awais, Siraj Din, Attequa(PAF-KIET, Pakistan)

GS5-8

Feasibility Study Of UAV Implementation In Route Surveying
D.Hazry, M A Azizan, Safwan Suhaimi, Zulaiha Ramli, Mohamad Syafiq A.K, Zainuddin Hat(Universiti Malaysia Perlis, Malaysia)

Farewell Party©ICAROB 2019 ALife Robotics Corp. Ltd.

- 38 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

PS abstracts
PS-1 Motion-Data Driven Grasp/Assembly Planner

Kensuke Harada12, Natsuki Yamanobe2, Weiwei Wan12, Kazuyuki Nagata2,
Ixchel G. Ramirez-Alpizar1, Tokuo Tsuji3
1
2( Osaka University, National Institute of Advanced Industrial Science and Technology,
3
Kanazawa University, Japan)

This paper proposes a robotic grasp/assembly planner partially using a motion
data. A motion data includes an information on skillful motion which cannot
simply be realized by using conventional grasp/assembly planners. We first
explain the structure of motion data. Then, we explain the proposed
grasp/assembly planner. Finally, we show a numerical example to confirm theeffectiveness of our proposed method.

PS-2 Toward Life with Partner Robots Developing robots with the field trial toward the practical use
Masanori Sugiyama (Toyota Motor Corporation, Japan)

In the near future, robots are expected to be utilized as partners to coexist withpeople for various social needs. TOYOTA is developing robots aimed to help
societies. As a first step, rentals of a rehabilitation robot for walking training at
medical institutions have been launched in 2017. Another robot, Human
Support Robot, which is aimed for use to support daily living, has been
adopted as a standard platform of the domestic environment league at the
RoboCup competition since 2017. For the robot development, it is importantto learn deeply about the field which the robot is to be actually used. In thispresentation, we will show the scene of the field trial toward the practical use
of our robots. Also, we will introduce a humanoid robot which is integrated
with advanced technologies that would benefit us in the future.

PS-3 Robust Consensus Control of Multi-vehicle Systems

Yingmin Jia (Beihang University, P.R.China)

Consensus control of multi-vehicle systems is to design a coordinated protocol
based on the communication topology, so that the multi-vehicle states canreach consensus rapidly and safely to achieve the desired cooperative tasks.
Considering parameter variations of vehicles, modelling errors and external
disturbances, the corresponding robust consensus control problems should be
investigated. To this end, this talk focuses on three basic issues, i.e.,single-vehicle maneuverability, multi-vehicle coordination, and disturbanceattenuation of control performance of vehicle systems in the uncertain
environment, and some recent advances are reviewed. In particular, it isfound that most of the existing results are very limited from a practicable pointof view because the traditional decoupling control methods are inapplicable tovehicles with varying velocity, the previous coordinated protocols requireconnectivity of dynamical topologies, and the robust consensus control ofmulti-vehicle systems with internal uncertainties and external disturbances has
not yet fully been studied so far. Doubtless, these limitations make theapplications of consensus control theory for multi-vehicle systems to actual
objects more difficult, and therefore, it is necessary to develop novel control
methods to remove these limitations and all these will be introduced in detail.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 39 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

IS abstracts
IS
abstracts
IS1-1-1
Engineering Modular Playware
IS1-1-1 Engineering
ModularHenrikHautopPlayware
Lund, Morten Roed Frederiksen, Massimiliano Leggieri
Henrik Hautop
Lund, Morten
RoedofFrederiksen,
Massimiliano Leggieri(Technical
University
Denmark, Denmark)(Technical University of Denmark, Denmark)In this paper, we describe the engineering of modular playware. The
In this paper,
we describe
the engineering
modularity
of constructing
playwaresystems outofofmodular
modules playware.
may allow The
themodularity
constructing
playwaresystemsout of modules
thesystem to beofeasilytransported
to be usedanywhere,to easily may
buildallow
different
systemto bebrains,
easilyandtransported
be usedanywhere,
to combine
easily build
bodies andto allow topeopleto construct
andthe different
modulesbodiesand creative.
brains, andto allow
people to constructand combine
the modules
to becomeHence,
the modularity
helps fulfilling
the PlaywareABC.
tocreative.
the modularity
fulfilling
the Playware
Webecome
exemplify
the Hence,engineering
challengeshelpsof thehardware,softwareABC.
andWe
exemplify theengineering
challenges
hardware, softwareandof handheld
communication
designof modules
throughofthethedevelopment
communication
designof modules
themodules
developmentof ahandheld
modules for playful
rehabilitation.
Thethroughhandheldallow forfun andmodulesfor playful
The handheld
modulesallow
fun andmotivational
trainingrehabilitation.
of upper extremities,
and canbe viewed
as foran aextension
motivational
upperextremity
extremities,
and canwhich
be viewed
as an extension
to the Moto training
Tiles foroflower
training,
have proved
to be atothe Moto
Tiles playware
for lowerfor
extremity
training,
which have proved
to be ahighlysuccessful
prevention
and rehabilitation
among seniors.
highlysuccessful
for prevention
andandrehabilitation
among toseniors.
The paper
outlinesplayware
the engineering
challenges
proposed solutions
makeThe
outlines
the engineering
challenges
and proposed
suchpaper
modularplaywarefor playful upper
extremity
training. solutions to makesuch modular playware for playful upper extremity training.

IS1-1-2 User-Friendly Robotics 3 – Playware
IS1-1-2 User-Friendly
Robotics
3 – Playware
Henrik Hautop
Lund, Luigi
Pagliarini (Technical University of Denmark, Denmark)Henrik Hautop Lund, Luigi Pagliarini (Technical University of Denmark, Denmark)The sustainable The design of playware based on the lessons learned fromThe
sustainable
The designof aplayware
on thelessonscriteria
learnedto fromuser-friendly
robotics
suggests
focus on based
a numberof design
help
user-friendly
roboticssystems
suggeststhat
a focusa numberdesigncriteria toto help
can onandwill beofusedaccordingtheachieving playware
achievingplaywaresystems that
can andanytime.
will be The
useddesign
accordingto arethePlayware ABC
by anybody,
anywhere,criteria
Playwareanybody,anywhere,anytime.
The designaredescribed ABC
in thisbypaper,and they
include:modularity,
explicit criteria
immediatedescribed
in this paper,and they
include:design.
modularity,
explicitisimmediate
feedback, robustness,
simplicity,
one-click,
Each criterion
described
feedback,
robustness,
one-click,design.robotics.
Each criterion
is describedbased on thelessons simplicity,
learned fromuser-friendly
We exemplify
the
based
the design
lessons criteria
learnedwith
fromtheuser-friendly
robotics.
exemplifytheuse of onthesedevelopmentof the We
playwaresystem,use
these The
designcriteria
theisdevelopment
the playware
system,MotoofTiles.
Moto
Tiles withsystemdesigned as aofplayful
prevention
andMoto
Tiles. The
MotoforTiles
systemdesigned
as nota playfulprevention
andrehabilitation
systemseniors,
whoisarenormally
exposedto such highrehabilitation
systemThefor paper
seniors,
who arehow
normally
not exposed
to such for
highatechnology systems.
discusses
the design
criteria allowed
technology
systems.
paper discusses
howwhichthe design
criteria
successful designof Thea user-friendly
system,is nowused allowed
daily by for
tensa
successful
designof alla user-friendly
system, which is now used daily by tensof thousands
seniorsover the globe.
of thousands seniors all over the globe.

IS1-2-1 User-Friendly Robotics 1 – AI Software
1,2
IS1-2-1 User-Friendly RoboticsLuigi
1 – AI
Software
Pagliarini, Henrik Hautop Lund1

Pagliarini, Henrik
Hautop of
Lund(1Technical University ofLuigi
Denmark,
Denmark)(2Academy
Fine Arts of Macerata, Italy)
1( Technical University of Denmark, Denmark) (2Academy of Fine Arts of Macerata, Italy)
1,2

1

In this paper we present the trajectory of decades of research in AI andIn
this paper
we present
the trajectory
decades
research
AI intoand
Robotics
by highlightthe continuityof ourofmethodthatofhasalways intaken
Robotics
by highlightthe continuity
of ofourview
methodhas always
taken
consideration
the common
user’s point
as a thatpriority.
We callsuchinto
anconsideration
the common
user’s pointof view
as unravela priority.
call suchouranapproach User-Friendly
Robotics,
and weherebyit byWe
describing
approach
User-Friendly
Robotics,
weits
herebyunravelit bypath step-by-step,
to seeon oneandside
evolution
andon describing
the other our
its
path
step-by-step,
to see onTherefore,
one side starting
its evolution
and earliest
on the AI-based
other its
methodologicalcontinuity.
from themethodologicalcontinuity.
Therefore,
starting
fromthe earliest
AI-based
application we slowly
move-uptowards
the first
Robotics
ones tofinally
application
towards We
thediscuss
first Robotics
to finallydescribe thewelastslowly
results move-up
of our research.
how thisones
User-Friendly
describe
last results
our research.
Weparticular
discuss how
this in
User-Friendly
Robotics theapproachleads ofto solutions
that fit
purposessociety andRobotics
approachleads to solutions
that fit particularpurposes in society andlead to usefuluser interaction
with technologicaldevelopments.
lead to useful user interaction with technological developments.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 40 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

IS1-2-2 User-Friendly Robotics 2– Interfacing Robotics with any user

Luigi Pagliarini1,2, Henrik Hautop Lund1(1Technical University of Denmark, Denmark) (2Academy of Fine Arts of Macerata, Italy)

In this paper, the second part out of three, we continue presenting the route of
our long experience in interfacing AI and Robotics with the public. This paper,in particular, highlights a crucial step we experimented when creating
computer based software architecture so to let the common user easily interact
with complex behaviour robotics. We define such an approach User-Friendly
Robotics, and we hereby describe the principles and the techniques we used to
evolve its methodology reporting here the most important case studies wewent through, as examples. We then analyse the impact of such method on
both science and educational field. Finally, we discuss how this User-Friendly
Robotics approach leads to solutions that fit particular purposes in society andlead to useful user interaction with technological developments.

IS-2 A New Machine Learning Algorithm Applicable for Weather Visibility and Food
Recognition
Young Im Cho (Gachon University, Korea)

The sustainable Smart City need many intelligent technologies. In this paper,
we implement a new Machine Learning algorithm for visibility of weather
images. The aim of the research is that to estimate weather visibility using a
new machine learning techniques. We use images taken from CCTV cameras
as inputs and deep convolutional neural network model to predict results. Our
new machine learning algorithm apply to smart life for food recognition, too.
We augment many images using a new machine learning algorithm. In thispaper, we will detail explain regarding an architecture of the ML model,
System Structure, and other essential details.

IS-3 Affective Interactive Systems

Kaoru Sumi (Future University Hakodate, Japan)

In this talk, I will present the effect of using emotions in humans and artifacts(computer systems, virtual agents, robots, etc.) interaction. As known asMedia Equation, the relationship between humans and artifacts is similar tothe social relationship between humans and humans. For example, when ahuman is helped by an artifact a human want to thank artifacts and returnsomething. Like humans, it is important for artifacts to recognize emotions
and express emotions in relation between humans and artifacts. In this talk, I
introduce a technique to recognize and express emotions in human interaction
with artifacts and show examples of how effective artifacts are when using
emotions.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 41 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS abstracts
OS1 Intelligent Systems and Control (8)OS1-1 A Vishay Supercapacitor based Fast Charge Battery

Chung-Wen Hung, Yi-Chang Yang, Chun-Chu Lin, Li-Sheng Jheng(National Yunlin University of Science and Technology, Taiwan)

A supercapacitor (SC) power supply management system based on is proposedin this paper. Due to the high charging current characteristics of the SC, means
short charging time, the proposed SC-base battery is suitable as a backup
battery. The proposed system includes the input voltage, current andtemperature protection implemented with analog integrated circuits (IC); andoutput current limitation circuit. In order to keep output voltage when the SC
voltage drops, a boost circuit is used to provide a stable voltage for the load.
The circuit design is detailed in this paper, and the experimental results show
the proposed system works well.

OS1-2 Multi-Motor Synchronous Control with CANOpen

Chung-Wen Hung, Roger CL Lee, Bo-Kai Huang, Shit-Ting Yu(National Yunlin University of Science and Technology, Taiwan)

A Brushless DC (BLDC) Motors synchronous control system based on the
CANopen protocol is proposed in this paper. Multi-motor control is popular in
Robot or automation system, and the synchronization is an important issue,
may be the key issue. The CANopen is a solution, due to Process Data Object(PDO) protocol. A Texas Instruments microcontroller TMS320F28069 is usedas master to control six BLDC Motors used as slave in this paper. The Master
sends synchronization object (SYNC) to slaves base on the communication
profile CAN in Automation (CiA) 301 and the motion control profile CiA402.
The system suns at 500Kbits/s and 1Mbits/s transmission rate by setting
process data object (PDO). The firmware details in this paper to show
six-motor synchronous control.

OS1-3 Selection Strategy for VM Migration Method

Yan-Ren Chen, I-Hsien Liu, Keng-Hao Chang, Jung-Shian Li(National Cheng Kung University, Taiwan)

Virtual machine (VM) live migration occurs frequently in cloud environments.
When it is necessary to migrate many VMs, minimizing the migration time isan important concern. Thus, choosing an appropriate strategy to perform VM
migration is essential. Accordingly, this paper discusses the advantages anddisadvantages of four different migration methods and evaluates theirrespective migration times and throughputs.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 42 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS1-4 VM Migration Placement in Cloud Service

Yan-Ren Chen, I-Hsien Liu,Keng-Hao Chang, Chuan-Gang Liu, Jung-Shian Li(National Cheng Kung University, Taiwan)

Smooth and transparent VM migration is essential in maximizing theutilization of cloud resources and ensuring user satisfaction. Previous studies
have focused on VM operational procedures, live migration models andefficient migration methods. By contrast, the paper focuses on thefundamental theorems of VM migration. The present study considers two
different VM migration problems and proves that they too are NP-complete.
In the first problem, subsets of the VM migration groups are migrated to aphysical host until their combined resource usage approaches the thresholdvalue of the host. In the second problem, the VM migration groups are
migrated simultaneously in such a way as to minimize the total migration
time.

OS1-5 Control System for Maintaining Safe Following Distance while Driving

Kuo-Hsien Hsia, Jia-Hong Cai, Shu-Li Pai (Far East University, Taiwan)
Evgeni Magid (Kazan Federal University, Russia)

Safety is the most important thing while driving. Many traffic accidents occur
because of insufficient safe distance. With the advancement of technology,
there are cars with a front distance safety warning system. The invention
relates to a driving safety distance control system. Some of them are with
fixed safe distance regardless of the speed of the car. Other advanced distancedetection systems switch between different modes such as high speed and low
speed. In this paper, we proposed a driving safety system which candynamically adjust the safety distance between vehicles depending on the
speed and weather conditions. This system can be used to determine andmaintain safe distances both in front and rear of a car. This idea has obtained
the invention paten of Taiwan. A car with this system can be driven moresafely.

.OS1-6 Markerless Indoor Augmented Reality Navigation Device Based on
Optical-Flow-Scene Indoor Positioning and Wall-Floor-Boundary Image Registration
Chian C. Ho, Bo-Kai Wang, Guan-Lung Liao
(National Yunlin University of Science and Technology, Taiwan)

For markerless indoor Augmented Reality Navigation (ARN) technology,
camera pose is inevitably the fundamental argument of positioning estimationand pose estimation, and floor plane is indispensably the fiducial target ofimage registration. This paper proposes optical-flow-scene indoor positioning
and wall-floor-boundary image registration to make ARN more precise,reliable, and instantaneous. Experimental results show that both
optical-flow-scene indoor positioning and wall-floor-boundary image
registration have higher accuracy and less latency than conventional
well-known ARN methods. On the other hand, these proposed two methods
are seamlessly implemented on the handheld Android embedded platform andare smoothly verified to work well on the handheld indoor augmented reality
navigation device.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 43 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS1-7 Development of the handheld gas detector with IoT function.

Jr-Hung Guo, Kuo-Hsien Hsia, Kuo-Lan Su(National Yunlin University of Science and Technology, Taiwan)

Gas detection is very cumbersome because the gas will diffuse and mix withother gases. Therefore, this paper is development a handheld gas detector
using a single chip controller. This detector can use a variety of different gas
sensors. And use the Fuzzy Analytic Hierarchy Process (fuzzy-AHP) and the
Adaptive Fusion Method to improve the correctness and recognition rate of
the sensor. In terms of communication, because this gas detection module ishand-held, we use Wi-Fi, LoRa, and wireless communication interfaces suchas 433/868/915 Mhz. Let this gas detection module still transmit data whilemoving. In addition, multiple gas detection modules can be used to form adetection network for large area detection. And we can use the built-in
algorithm to do the calibration data for the detection and isolation of network
module or sensor. Make this gas detection module more flexible in use.

.

OS1-8 Research on Employee Attribute Correlation of Information Security Awareness inOrganization
Tse-Yao Wang (Air Force Institute of Technology, Taiwan)
Fu-Hsiang Wen (National Kaohsiung University of Science and Technology, Taiwan)

Enterprises are attacked by phishing emails, causing significant
business losses. The information security protection strategies
adopted by enterprises, such as mail filtering, phishing attack
website screening and phishing website blacklists, cannot guarantee
complete defense. This study attempts to propose anotherperspective that can reduce the risk of corporate information
security. The quality of employees' information security used as anindicator, and the Logistic regression model used as a tool toanalyze the key factors that can be used to measure the quality of
information security awareness.

OS2 Theory and Implementation of Neuromimetic Systems (5)OS2-1 Decoding spike patterns of auto-associative memory on spiking neuronal networks
Naoki Toyoshima, Takashi Kohno (The University of Tokyo, Japan)

Spiking neuronal networks model the electrophysiological activities in
Spiking neuronal cells’ networks, in which spikes emitted by the neuronalcells transmit information. A well-known application of the spiking neuronal
networks is auto-associative memory, in which one of the pre-stored patterns
that is mostly similar to the input pattern is retrieved. It is known that the
spatio-temporal output spikes are converged to anti-synchronized two groups
that correspond to the retrieved pattern and its complement. The spikes ineach group are synchronized. For practical use of auto-associative memory on
the spiking neuronal networks, it is necessary to convert the spatio-temporal
output spikes to digital codes that can be utilized in ordinary digital computer
systems. In this manuscript, we propose methods to efficiently do thisconversion and extract some more information by additional neurons
receiving signals from the neurons in the all-to-all connected network which isa typical configuration for the auto-associative memory.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 44 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS2-2
OS2-2 Possible
Possible Mechanism
Mechanism ofof InternalInternal Visual
Visual Perception:
Perception: Context-dependent
Context-dependent Processing
Processing by
by
Predictive
CodingandReservoir
Computing
Network
Predictive Coding and Reservoir Computing Network1
1,2
1
HirotoHiroto Tamura
Tamura1,, Yuichi
Yuichi Katori
Katori1,2,, Kazuyuki
Kazuyuki Aihara
Aihara1
11
22( The University of Tokyo, Future University Hakodate, Japan)( The University of Tokyo, Future University Hakodate, Japan)

HowHow our
our brains
brains perceive
perceive visualvisual stimuli
stimuli andand generate
generate internalinternal visualvisual images
imageslargelyremains
unknown.
Here,
weproposea
network
model
bylargely remains unknown. Here, we propose a network model by combining
combiningthethe idea
idea ofof predictive
predictive coding
coding andand reservoir
reservoir computing
computing (PCRC).
(PCRC). First,
First, wewedefinedefine thethe context-dependent
context-dependent task
task byby simplifying
simplifying thethe processprocess ofof human’s
human’sinternal
internal visualvisual perception,
perception, andand thenthen train
train thethe PCRC
PCRC network
network forfor thethe task.
task. InIn
our
our model,
model, thethe context
context represents
represents thethe category
category ofof thethe visualvisual stimuli,
stimuli, andand thethe
network
network forms
forms different
different attractor
attractor landscapes
landscapes depending
depending on
on eacheach context.
context.
However,themismatchbetweenthecontext
andthesensory
stimuli
However, the mismatch between the context and the sensory stimuli induces
inducesthethe perceptual
perceptual error,
error, which
which corresponds
corresponds toto thethe symptoms
symptoms ofof thethe hallucination
hallucinationinin dementia
dementia with
with Lewy
Lewy bodies.
bodies. Furthermore,
Furthermore, we
we incorporate
incorporate thethe parameterparameter
representing
a
sort
of
neuromodulation
into
the
networkmodel
representing a sort of neuromodulation into the network model andand analyze
analyzethethe influences
influences ofof thethe neuromodulationneuromodulation toto thethe stability
stability ofof thethe attractor
attractor on
on thethenetwork
network dynamics
dynamics andand performance
performance ofof thethe visualvisual perception.
perception.

OS2-3
OS2-3 Snake
Snake robot
robot controlledcontrolled byby biomimetic
biomimetic CPGs
CPGs

1
2
2
DamienDamien Blanchard
Blanchard1,, Kazuyuki
Kazuyuki Aihara
Aihara2,, Timothée
Timothée Levi
Levi2
11
22
( University of Bordeaux, France, The University of Tokyo, Japan)( University of Bordeaux, France, The University of Tokyo, Japan)

Locomotion
Locomotion isis oneone ofof thethe most
most basic
basic abilities
abilities inin animals.
animals. Neurobiologists
Neurobiologists have
have
establishedthat
locomotion
resultsfromtheactivityofhalf-centerestablished that locomotion results from the activity of half-center oscillators
oscillatorsthatthat provides
provides alternation
alternation ofof bursts.
bursts. Central
Central Pattern
Pattern Generators
Generators (CPGs)(CPGs) areareneuralneural networks
networks capable
capable ofof producing
producing rhythmic
rhythmic patterned
patterned outputs
outputs without
withoutrhythmicrhythmic sensory
sensory oror central
central input.
input. We
We propose
propose aa network
network ofof several
severalbiomimetic
biomimetic CPGs
CPGs using
using biomimetic
biomimetic neuron
neuron model
model andand synaptic
synaptic plasticity.
plasticity.
This
This network
network is
is implemented
implemented on
on aa FPGAFPGA (Field(Field Programmable
Programmable Gate
Gate Array).
Array).
We
designedone
unsupervised
snake
robot
using
this
networkof
We designed one unsupervised snake robot using this network of CPG.
CPG. ItIt isiscomposed
of
one
head
wagon
and
7
slave
wagon.
Infrared
sensorsarealso
composed of one head wagon and 7 slave wagon. Infrared sensors are also
embeddedembedded inin thethe headhead wagon.
wagon. This
This robot
robot cancan reproduce
reproduce thethe locomotion
locomotion ofof oneonesnake.
The
hardware
robot
is
also
describedin
thispaper.
snake. The hardware robot is also described in this paper.

OS2-4
OS2-4 Real-time
Real-time implementation
implementation ofof ReSuMe
ReSuMe learninglearning inin Spiking
Spiking Neural
Neural Network
Network
Xia
Xia Yang,
Yang, Seiji
Seiji Uenohara,
Uenohara, Kazuyuki
Kazuyuki Aihara,
Aihara, Takashi
Takashi Kohno,
Kohno, Timothée
Timothée Levi
Levi
(The
UniversityofTokyo,
Japan)(The University of Tokyo, Japan)

Neuromorphic
Neuromorphic systems
systems areare designed
designed byby mimicking
mimicking oror being
being inspired
inspired byby thethenervoussystem,which
realizes
robust,autonomous,andpower-efficientnervous system, which realizes robust, autonomous, and power-efficient
information
information processing
processing by
by highly
highly parallel
parallel architecture.
architecture. Supervised
Supervised learninglearning
wasproposedasasuccessful
conceptofinformation
processing
was proposed as a successful concept of information processing inin neuralneuralnetwork.
network. Recently,
Recently, there
there has
has been
been anan increasing
increasing body
body ofof evidence
evidence that
that
instruction-based
instruction-based learninglearning is
is also
also exploited
exploited byby thethe brain.
brain. ReSuMe
ReSuMe isis aa
proposed
proposed algorithm
algorithm byby Ponulak
Ponulak andand Kasinski
Kasinski inin 2010.
2010. ItIt proposes
proposes aa supervised
supervised
learning
learning forfor biologicallybiologically plausible
plausible neurons
neurons thatthat reproduce
reproduce template
template signals
signals(instructions)(instructions) oror patterns
patterns encoded
encoded inin precisely
precisely timed
timed sequences
sequences ofof spikes.
spikes.
Here,
Here, we
we present
present aa real-timereal-time ReSuMe
ReSuMe learninglearning implementation
implementation on
on FPGA
FPGA
using
using LIF
LIF Spiking
Spiking Neural
Neural Network
Network (SNN).
(SNN). FPGAFPGA allows
allows real-timereal-time
implementation
implementation andand embedded
embedded system.
system. We
We show
show thatthat this
this implementation
implementation cancan
makesuccessful
thelearningonaspecificpattern.
make successful the learning on a specific pattern.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 45 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS2-5 Biomimetic spike-timing based ionic micro-stimulation for neuron culture

Stephany Mai Nishikawa1, Farad Khoyratee1,2, Soo Hyeon Kim1, Yoshiho Ikeuchi1, Kazuyuki Aihara1, Teruo
Fujii1, Timothée Levi1
1( The University of Tokyo, Japan, 2University of Bordeaux, France)
Neurodegenerative diseases influence cognitive and/or motor functions inmillions of people worldwide. Neuroprosthesis are used today to support thequality of life but have yet to improve in their power consumption andbiocompatibility issues. Here, we propose a novel microfluidic system toinvestigate the response of the neurons directly stimulated by the potassiumions in biomimetic timing. Unlike the conventional way of stimulatingneurons by pipetting or sending electrical stimulus, this biomimetic
stimulation system enables precise and rapid control of ion concentration by
utilizing microfluidic pneumatic valves, which mimics physiologicalstimulation in a body. To circumvent these limitations, we propose a new
bio-hybrid system. There are three parts that consists this neuro-hybrid
system: Spiking Neural Network, neuron culture and ionic micro-stimulation
device. By using the system, we demonstrated biomimetic spike-timing basedionic stimulation of the neuron cells with potassium ions and monitored
neural activities with a calcium-specific fluorescence indicator.

OS2-6 Dynamical network model for visual cortex: hierarchical reservoir computing withbidirectional interactions (withdraw)
Shohei Takaichi1, Yuichi Katori2, Kazuyuki Aihara1( The University of Tokyo, Japan, 2 Future University Hakodate, Japan)
1

In visual cortex, there is a feedback connection between high-layer andlow-layer. However, the effect of feedback remains poorly understood. We
describe a hierarchical neural network model for visual recognition using
reservoir computing. We adapt this model to predictive coding of video
information. Because input of the network is an image of each frame of video,
which is too high dimension for the reservoir computing, we divide the image
to patches and integrate divided information in high-layer and send it as
feedback.

OS3 Intelligent Navigation & Robotics (4)OS3-1 Trajectory tracking control for a 7-arms robot manipulator

Wang Jie, Hyun-Hee Kim, Saad abbasi, Min-Cheol Lee (Pusan National University, Korea)

This paper proposes a trajectory tracking control based TSMCSPO (terminal
silding mode control with sliding perturbation observer) for a 7-arms robot
manipulator. The 7-arms robot arm are designed to assemble small parts or
packaging. System dynamics, kinematics and inverse kinematics are
introduced at first. Because dynamics of this 7-arms robot are difficult todetermine precisely with the uncertainties and many nonlinear terms.
TSMCSPO are used as a robust controller with the observer for unknown
states and perturbation which can estimate the error in obtained system
dynamics.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 46 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS3-2 Estimation tire-rod friction coefficient based on mobile robot

Ji-Hyeon Kim, Zhang Hongyu, Jang-Myung Lee (Pusan National University, Korea)

It is very important to estimate the friction coefficient of tire-road for vehicle
dynamic control system. For the current dynamic control research, a large
amount of experimental data is needed. The theoretical tire model is used toestimate the friction coefficient based on the Gim tire model to improve thetire experience model. Simulation by MATLAB, the simulation results showthat the Gim tire model has high precision and can predict the mechanicalproperties of the tire well.

OS3-3 Real-time reflection removal algorithm using stereo camera only

Do-Kyung Hwang, Jong-Woo An, Jang-Myung Lee (Pusan National University, Korea1)

In this paper, we implemented a light reflection elimination algorithm which
is one of the main obstacles of various object recognition systems using visionsystems. In other words, this algorithm can be used to remove robotic
recognition obstacles from the factory and perform tasks such as locating
robots, autonomous vehicles, or unmanned aerial vehicles. This algorithm has
been studied with versatility in mind and does not require sensors other thanstereo cameras. A brief description of the algorithm is an algorithm that
intuitively interprets pixels in HSI space and processes pixels corresponding
to light. For intuitive analysis, take an angle, intensity, or other light source(sunlight, fluorescent light, candle) from the camera, then record each pixel
value and separate it from a white object (whiteboard, paper, etc.). After pixel
operation, object recognition is implemented using the CNN (Convolution
Neural Network) method using the YOLO_V2 algorithm

OS3-4 The Effect of Inertial Measurement Unit on Synthetic Aperture Radar Image Quality

Soo-Jeong Lee1, Yong-Gonjong Park1, Woo-Jung Park1, Chan Gook Park1, Jong-Hwa Song2 (Seoul National
University, Korea1, Hanwha Systems, Korea2)
This paper analyzes the effect of inertial measurement unit (IMU) on synthetic
aperture radar (SAR) image quality. While operating SAR, it is supposed thatan aircraft flies straight and level at a constant speed. Because of atmosphericturbulence and aircraft maneuvers, it deviates from its nominal trajectory.
With navigation solutions, the deviations can be compensated through motion
compensation procedure. However, uncompensated motion errors generated
by IMU errors degrade SAR image quality, such as broadening bandwidth andblurring the image. Since the residual errors depend on the degree of IMU
inaccuracies, SAR images and impulse response function (IRF) according to
IMU grades are compared with each other. Through this result, the IMU
specification is suggested to satisfy given SAR image quality.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 47 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS4 Service Robotics (7)OS4-1 Apply 2D Barcode Scanner for Mobile Robot Navigation in Checkerboard Mapping
Chun-Chi Lai1, Kuo-Lan Su1, Chia-Jen Lin1, Evgeni Magid²( National Yunlin University of Science and Technology, Taiwan)(²Kazan Federal University, Russia)
1

In this work, a mobile robot is equipped with an industrial barcode scanner
which can provide the pose information respect to the tag in the field of view(FOV). For real multiple mobile robot transportation applications, the mobile
robot navigation flow is considered to get the global checkerboard type path
planning from a remote master server as an input. For the local planner, eachrobot is applied with a simple path controller to track the global path. The
simulation and experimental results show that this implementation has good
feasibility and effectiveness for multi- robot co-working such as in a factoryarea.

OS4-2 Development of the MyRio Based Mobile Platform

Bo-Jun Yang1, Sheng-Jie Liou1, Jr-Hung Guo1, Kuo-Lan Su1, Evgeni Magid²(1National Yunlin University of Science and Technology, Taiwan)(²Kazan Federal University, Russia)

The paper develops the MyRio based mobile platform with a robot arm. The
structure of the mobile platform uses the Matrix elements. The Matrix element
set builds the robot arm with four degrees of freedom. The mobile platform
integrates some sensors, three DC servomotors, a RC servomotor, a MyRio
based controller, and a vision system. The core of the MyRio based controller
is the NI-Single-Board RIO 9606 module. The mobile platform embeds arobot arm on the front side. The driver device of the gripper is a RC
servomotor. The vision system of the mobile platform can search andrecognize the assigned shape and color that is on the front side of each box
using Otsu algorithm. The mobile platform uses the robot arm to catch theball, and moves to the assigned box simultaneously. In the experimental
results, the mobile platform moves to the assigned location from the startlocation, and uses the vision system to find out the assigned ball, and puts
down the ball on another box by the robot arm. Finally, the mobile platformmoves to the start location and stop.

.

OS4-3 Modelling Avrora Unior Car-like Robot in Gazebo Environment

Ksenia Shabalina1, Artur Sagitov1, Kuo-Lan Su², Kuo-Hsien Hsia³, Evgeni Magid1(1Kazan Federal University, Russia)(²National Yunlin University of Science and Technology, Taiwan)(³Far East University, Taiwan)

In order to conduct thousands of complicated experiments on autonomous car
navigation algorithms it is much safer and cheaper to start algorithm
verification within a simulation area given that we succeed to achieve a proper
model configuration, which preserves physical properties of underlying
objects. This paper focuses on model construction of Avrora Unior mobile
robot. Avrora Unior is a Russian car-like mobile robot with Ackermann drive
mechanism. For the robot model design we used open source Robot operating
system (ROS) and 3D simulator Gazebo. We describe the entire process ofmodeling and first steps of control implementation for the model.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 48 -.

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS4-4 Transfer of learned exploration strategies for a mobile robot from a simulated world toreal environments
Artur Sagitov1, Tetsuo Takano2, Shohei Muto2, Evgeni Magid1( Kazan Federal University, Russia) (2Kanazawa University, Japan)
1

Reinforcement learning based approaches show promises in various robotic
applications, but a significant amount of time and resources are required for arobot to learn optimal behavior. Using virtual environments, we could
significantly speed up and improve performance of a target task. We
implemented a reinforcement learning based exploration algorithm for amobile robot, training in Gazebo environment and transferring learned
strategy to a real robot. We show that it is convenient and appropriate to use
simulation to train strategies for mobile robots.

.OS4-5 Extending Gazebo simulator for surgical robotics: tissue and suture modeling
Artur Sagitov1, Hongbing Li2, Evgeni Magid1( Kazan Federal University, Russia) (2Shanghai Jiao Tong University, China)
1

Active use of a simulator as a training tool has proven to be advantageous to ahuman surgeon, but there is no open source and convenient universal surgery
simulation of a robot surgeon. This paper presents an extension of Gazebo
simulator for surgical robots using Robot Operating System. We present
software architecture that allows modeling robot interaction with different
types of tissue and suture. We plan to apply the resulting system for a surgicaltask of autonomous suturing with different techniques and algorithms to becompared in the same virtual environment.

.OS4-6 Experiences of Robotics students in Machine Vision course being taught in aforeign language: comprehension, self-efficiency, and active learning strategies improvement
Tatyana Tsoy, Artur Sagitov, Evgeni Magid (Kazan Federal University, Russia)

Historical development of Russia in the end of 20th century significantly
decreased the quality of engineering education. Now, in order to catch up withthe developed countries, at Kazan Federal University a novel 2-year Masterprogram in Intelligent Robotics was designed and implemented. The program,which curriculum is based on experience of world leading universities in thefield of robotics, targets to educate competitive specialists with competencesthat are required in the world labor market. Students with various technicalbackgrounds were enrolled in the program. To follow students’ progress incore robotics courses we conducted surveys in the beginning and by the end ofeach course. This paper presents surveys that were conducted in Machine
Vision course. The paper reports results of the survey analysis covering such
issues as English language comprehension, self-efficiency, and active learning
strategies in the context of Machine Vision.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 49 -.

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS4-7 Pilot Virtual Experiments on ArUco and AprilTag Systems Comparison for Fiducial
Marker Rotation Resistance
Aufar Zakiev1, Ksenia Shabalina1, Kuo-Lan Su², Kuo-Hsien Hsia³, Evgeni Magid1(1Kazan Federal University, Russia)(²National Yunlin University of Science and Technology, Taiwan)(³Far East University, Taiwan)

There exists a large number of fiducial marker system types and bothresearchers and industry have difficulties to select among this variety a single
system that could provide optimal behavior for a particular task. This paper
presents design and results of pilot virtual experiments that were conducted inorder to compare a performance of two marker systems, ArUco and AprilTag.
Experiments were designed to estimate and compare marker systems
resistance against rotation with regard to different principal axes in 3D space.
Pilot experiment design eliminates influence of external environment,
including light conditions, camera resolution, sensor noise, distance between
camera and marker, etc. Experiments were implemented in ROS/Gazebo
environment. In total over 300,000 virtual experiments were performed andanalyzed in order to collect statistically significant data amount.

OS5 Business Models and Management (4)OS5-1 Determining the Key Factors of Michinoeki in Yamaguchi Area

Minoru Kumano1, Tsutomu Ito2, Takao Ito3, Toru Hiraoka4, Hirofumi Nonaka5(1University of Miyazaki, 2Hino Motors, Ltd., 3Hiroshima University,
4
University of Nagasaki, 5Nagaoka University of Technology, Japan)

It is becoming an important issue to improve efficiency in Michinoeki. Thus tofind the key determinants is considered as the most urgent task for Michinoeki.
This paper proposed a new approach with two steps: to calculate the efficiency
using regression model after determing the relevant factors using DEA model.
The data are collected from Chugoku area, because Chugoku is the birthplaceof Michinoeki. The efficiency of all Michinoeki have been calculated, and theimplications have been discussed in this paper.

OS5-2 Discovering the Characteristics of Michinoeki in Japan

Minoru Kumano1, Tsutomu Ito2, Takao Ito3, Toru Hiraoka4, Hirofumi Nonaka5(1University of Miyazaki, 2Hino Motors, Ltd., 3Hiroshima University,
4
University of Nagasaki, 5Nagaoka University of Technology, Japan)

World Bank reported that four stages including identification, preparation,
appraisal and operation are necessary to improve Michinoeki. Thus to estimate
the value of Michinoeki is a crucial issue. As our best knowledge, no research
to study the whole Michinoeki although there are 1,107 Michinoeki by April,2017 in Japan. Nine areas: Hokkaido, Tohoku, Kanto, Hokuriku, Chubu,
Kinki, Chugoku, Sikoku, and Kyushu&Okinawa have been divided, elevenbasic elements are used as the determinants of the Michinoeki. This paper
examined the determinants of the nine areas and found that the key
determinants of those nine areas are totally different. It contributes not only tounderstand the status quo of Michinoeki, but also helpful to find the potential
possibility for further development.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 50 -.

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS5-3 An Analysis of Robotic Relationship between Transaction Network andCross-shareholding Network in Yokokai

Takao Ito1, Tsutomu Ito2, Rajiv Mehta3, Seigo Matsuno4, Makoto Sakamoto5, Satoshi Ikeda5( Hiroshima University, Japan, 2Hino Motors, Ltd., Japan, 3New Jersey Institute of Technology, U.S.A.,
4
Ube National College of Technology, Japan. 5University of Miyazaki, Japan)
1

After bubble economy, the relationship of all parts suppliers in the keiretsu
have been changed from strong tie to weak tie. One typical thing isrelationship of cross shareholdings does not contribute to the transactional
relationship. The authors once proved that the relationship between crossshare holdings and transactions still remained in Keiretsu of Kyohokai evenafter the bubble economy. This paper collected the data from Yokokai, akeiretsu of Mazda, and calculated the relationship between cross share holding
and transaction to prove the correlationship between cross share holdings andtransaction in Yokokai.

OS5-4 Technological Discontinuities and the R&D Strategy of Automobile Companies
Yousin Park1, Iori Nakaoka2, Yun-ju Chen3(1Prefectural University of Hiroshima,
2
National Institute of Technology, Ube College,
3
Shiga University, Japan)

This paper focuses on patterns of technological changes and the R&D strategy
of automobile companies. Tushman and Anderson (1986) pointed out that
there are two types of technological changes. One is incremental technologicalchange which is often generated by existing companies, and the other one isradical technological change which makes great extension in businessenvironment and destroys the order of existing companies. We use the patent
information of three automobile companies (Toyota, BYD, Tesla) as the caseson these changes. And we examine our propositions by social network
analysis and text analysis. The analysis in this paper include: 1) try to
distinguish between the radical and incremental technological changes fromR&D projects, and create heat-maps to visualize these changes, 2) makediscussion ontechnological discontinuities and the R&D strategy of
automobile companies. In this paper we suppose that patterns of corporate
R&D strategy cope with the threat of radical technological changes.

The Network of Patent
Applicant of Toyota

OS6 Software Development Support Method (6)OS6-1 Prototype of an Animated Graphics-Based Training Support Toolfor Bug Fixing of Extended Place/Transition Nets

Tomohiko Takagi, Shogo Morimoto, Yuki Ue, Yoshiro Imai (Kagawa University, Japan)

An EPN (Extended Place/transition Net) is an executable formal model withhigh representation power for specifications definition and test casegeneration, but it would not be so easy for most of software engineers to learn
and use the EPN efficiently. This paper describes an animated graphics-based
training support method and prototype tool for bug fixing of EPNs. The
prototype tool gives a trainee a faulty EPN and its animated graphics. The
motion of the animated graphics is synchronized with the motion of the faulty
EPN. Therefore, it is expected that a trainee can intuitively understand thefaulty EPN, and smoothly try to fix a bug (a difference between the faulty
EPN and software requirements). The result of bug fixing is checked by the
prototype tool.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 51 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS6-2 Prototype of a Modeling Tool to Convert
between Extended Place/Transition Nets and VDM++ Specifications

Tomohiko Takagi, Ryo Kurozumi (Kagawa University, Japan)

In software modeling, engineers can use EPNs (Extended Place/transition
Nets) to define the expected behavior of software. The guards and actions of
EPNs are written in VDM++, and thus EPNs can be straightforwardly
converted to VDM++ specifications. EPNs provide more abstracted andintuitive view than VDM++ specifications, and VDM++ specifications can beused to generate source codes and test cases. Therefore they give engineers apractical way for model-driven development. VDM++ specifications should
be updated along with EPNs, and vice versa. However, it is not so easy forengineers due to their limited time. This paper shows a prototype of amodeling tool that converts (and keep the consistency) between EPNs and
VDM++ specifications.

OS6-3 Application of Pairwise Testing to Test Cases by Boundary Value Analysis in BWDM
Futa Hirakoba*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, Naonobu Okazaki*(*University of Miyazaki, †Tokyo University of Technology, Japan)

In recent years, specifications using specification language become more important. It is necessary to test the developed software, but it takes much timeand effort to design test cases manually. So, we developed BWDM(Boundary
Value Vienna Develop Method). It generates test cases from VDM++
specifications. In BWDM, test cases are generated by boundary value
analysis. However, it is possible to cause a combination explosion by the
number of the test cases. In order to solve the above problem, this researchextends BWDM so that the pairwise testing can be applied to boundary value
analysis. The extended BWDM does not cause the combination explosion,
because the pairwise testing only generates test cases which satisfy
combinations of two pairs.

 Please attach one typicalfigure in your paper here.

OS6-4 Development of Library Fescue Extracting Elements of Attributes and Operations ofClass Diagram in UML
Keisuke Mori*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, Naonobu Okazaki*(*University of Miyazaki, †Tokyo University of Technology, Japan)

Many modeling tools and development support environments for UML havebeen developed. And some modeling tools have the function to convert thedescribed model into programs. However, no existing tools can describe all
the grammar defined in UML 2.0 specification. Or even if it can be described,
the existing tools cannot analyze its meaning correctly. This paper develops
library fescue (Feature Elements Section of Class in UML Extraction)
extracting elements of attributes and operations of Class Diagram in UML toexpand the analyzable range of the grammar defined in UML 2.0
specification. Fescue can parse character strings expressing attributes andoperations based on the grammar defined in UML 2.0 specification, and can
extract each element from the syntax analysis result. It is confirmed that
fescue can parse the components of attributes and operations in a practicaltime that the existing tools cannot parse.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 52 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS6-5 Tamias: a Syntax File Checker for Parsing Expression Grammar

Toshihiro Miyaji*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, Naonobu Okazaki*(*University of Miyazaki, †Tokyo University of Technology, Japan)

In recent years, Parsing Expression Grammar (PEG) is introduced by Bryan
Ford. PEG is not ambiguous because it has ordered choice property. However,it causes “prefix capture”. “Prefix capture” is a problem of hiding thelanguage to be accepted according to the order of choice. Checking syntax
files that contain such mistakes confirms the behavior of the parser. However,in confirming the behavior of the parser, it is possible to check only the top
level non-terminal symbols, and it is necessary to rebuild the parser for eachchange in the syntax file. This paper proposes Tamias: a syntax file checker tosupport checking the syntax files. Tamias has a PEG interpreter which cancheck all choices and any non-terminal symbols in production rules follows
PEG.

OS6-6 Implementation of an Arduino Simulator to Support Circuit Design

Tatsumi Nishida*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, and Naonobu Okazaki*(*University of Miyazaki, †Tokyo University of Technology, Japan)

Embedded technologies are used everywhere and are indispensable in ourdaily life. One of the educational materials to learn embedded technology is
Arduino. However, when a beginner designs circuit, it is possible to design
circuit that damages Arduino itself or the modules on it. Therefore, thisresearch implements an Arduino simulator that runs on iPad for supporting todesign circuit. In this simulator, the circuit on iPad is analyzed to detect thevoltage value on the circuit and the risk Arduino itself or the modules being
damaged. This feature allows you to visually notify the risk of breakage of
Arduino itself or the modules without damaging them. Consequently, thissimulator can support to design circuit.

OS7 Media Information Processing and Artificial Intelligence (4)OS7-1 Effectiveness of Data Augmentation in Automatic Summarization System

Tomohito Ouchi, Masayoshi Tabuse (Kyoto Prefectural University, Japan)

We propose a new data augmentation method in automatic summarization
system. A large corpus is required to create an automatic summarization
system using deep learning. However, in the field of natural language
processing, especially in the field of automatic summarization, there are notmany data sets that are sufficient to train automatic summarization system.
Therefore, we propose a new method of data augmentation. We use an
encoder decoder model with an attention model as automatic summarization
system. First, we determine the importance of each sentence in an article using
topic model. In order to extend the data, we remove the least importantsentence from an input article and use it as a new article. We examine theeffectiveness of our proposed data augmentation method in automatic
summarization system.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 53 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS7-2 A System for Posting on SNS Portrait Selected Using Facial Expression Analysis While
Writing Message
Taro Asada1, Yuiko Yano2, Yasunari Yoshitomi1, Masayoshi Tabuse1( Kyoto Prefectural University, 2 Nara Guarantee Corporation, Japan)
1

We have developed a real-time system for expressing emotion as a portrait
selected according to the facial expression while writing a message. The
portrait is decided by a hair style, a facial outline, and a cartoon of facial
expression. The user selects one hair style and one facial outline before
writing a message. The image signal is analyzed by our real-time system using
image processing software (OpenCV) and a previously proposed featureparameter. Then the system selects one cartoon expressing one of neutral andhappy facial expressions using the facial expression intensity thresholdpreviously decided between these facial expressions. We applied the system topost on an SNS a message and a portrait expressing the facial expression
while writing the message. The experimental and questionnaire results suggestthat our system can be useful for expressing emotions while writing messages.

Generated portraits.

OS7-3 A Method Using Wavelet Transform for Judging Character to be Inserted into Image
Yasunari Yoshitomi1, Takashi Sato2, Taro Asada1, Masayoshi Tabuse1( Kyoto Prefectural University, 2 Tokyo Institute of Technology, Japan)
1

There has been an increase in the number of images on the Internet that
contain private information and/or the URL of an illegal Web site. This
information might result in a violation of human rights and/or a crime. We
previously proposed a method for extracting from an image the region(s) that
contain characters. Based on the previously proposed method, we haveproposed a method for judging character to be inserted into image using thediscrete wavelet transform, the image compression characteristics, and theempirical knowledge that characters have strong vertical and/or horizontal
elements. The experimental results show the usefulness of our method.

Original Processed

JPG size
Upper:28→19(KB)
Lower:29→4(KB)

OS7-4 An Authentication Method for Digital Audio Using Wavelet Transform and Fundamental
Frequencies
Yasunari Yoshitomi1, Shohei Tani2, Masaki Arasuna3, Ryota Kan4, Taro Asada1,
Masayoshi Tabuse1 (1 Kyoto Prefectural University, 2 Fukuchiyama City Hall,
3
Nissay Information Technology Co., Ltd., 4 Shimazu Business Systems Co., Ltd., Japan)

Recently, several digital watermarking techniques for audio files have been
proposed for hiding data in order to protect their copyrights. In general, there
is a tradeoff between the quality of watermarked audio and the tolerance ofwatermarks to signal processing methods, such as compression. For
overcoming the inevitable tradeoff, we previously developed an authentication
method for digital audio using a discrete wavelet transform. In the present
study, we have improved the method by deciding the region to beauthenticated in the audio file using the fundamental frequency characteristics.
The experimental results show that the method has high tolerance ofauthentication to cutting small parts in the audio data.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 54 -

Fundamental
frequency (Hz)RockTime (s)The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS8 Computational intelligence and cognitive science for human biosignals and humanwell-being (7)OS8-1 Unified Approach to (1+1) EA on Discrete Linear Functions
Kenji Aoki, Makoto Sakamoto (University of Miyazaki, Japan)
Hiroshi Furutani, Satoru Hiwa, Tomoyuki Hiroyasu (Doshisha University, Japan)

We consider the runtime property of discrete linear functions in (1+1)
Evolutionary Algorithms. We analyze the process of evolution by means ofMarkov chain method. This study was motivated by the paper of Jansen
treating the runtime property of (1+1) EA on monotonic functions by meansof probabilistic theory. As linear functions are special case of monotonic
function, we analyze their behavior by Jansen’s model, PO-EA. We show thatlinear functions show almost the same runtime if mutation rate pm is small.
On the other hand, in the case of strong mutation, the runtime shows thefunction specific magnitude. This fact means that higher order mutations play
more important roles in the calculation process with stronger mutation.

PO-EA(1+1) EA
Markov Chain Analysis↓
Discrete Linear Functions.

OS8-2 A fNIRS study of brain state during letter and category fluency tasks

Akane Onishi, Satoru Hiwa, Hiroshi Furutani, Tomoyuki Hiroyasu (Doshisha University, Japan)
One of the practical issues concerning automated driving vehicles is
identifying the state of the driver when switching between manual andautomatic control. When switching, it is important to detect the wakefulness
and awareness of the driver. In this study, we controlled a driving simulator
based on driver state detection. Electrocardiogram (EKG) measurement wasused for driver state detection. Heart rate variability analysis (HRV) is a
widely known means for estimating sympathetic and parasympathetic activity.
We created a Simulink model to calculate SDNN, RMSSD, and LF/HF in realtime, which are indicators obtained from the HRV analysis. Based on themodel, the state of the driver during the driving simulation was detected. We
also created a Simulink model that switches between automatic operation andmanual operation of simulated vehicles. By applying these models,construction of a system that safely switches between automatic operation andmanual operation is expected.

FWE,
p

22

<0.05

17.6
13.2
8.8
4.3

OS8-3 Construction of a meditation practice support system leads to a good meditation state: an
fNIRS study
Seika Fujii, Satoru Hiwa, Hiroshi Furutani, Tomoyuki Hiroyasu (Doshisha University, Japan)

Meditation is one methods of realizing a state of mindfulness. By continuing
meditation, an improvement of concentration and work efficiency is expected.
However, when novices practice meditation, they find it difficult to judge
improvements in their practice. Therefore, in this study, a system forsupporting novices’ meditation practices was formulated. In the proposedsystem, quantification of and feedback regarding the meditation state based onbrain functional information was performed. Brain activity during meditation
was measured using functional near-infrared spectroscopy. The brain activity
could be measured using this method close to every day. A good brain state
during meditation was defined based on previous research. The measured
brain state was compared with this good state and the similarity wascalculated. The state was then judged on the basis of the similarity andfeedback was given when the meditator approached the goal state. The
effectiveness of the constructed system was examined.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 55 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS8-4 Automated panoramic image creation system for corneal endothelial cells

Keitaro Kobayashi, Naoki Okumura, Noriko Koizumi, Satoru Hiwa, Hiroshi Furutani, Tomoyuki Hiroyasu(Doshisha University, Japan)
The imaging area of the corneal endothelium imaged by current technology isnarrow, mainly observing the central portion. A system for automaticallycreating a panoramic image based on an imaging video of the corneal
endothelium of the mouse was developed. First, still images were extracted
from the imaging videos and were convolved with a Laplacian kernel. Withthe variance of the response as a threshold value, focused images wereextracted. By stitching the focus images, panoramic images of the corneal
endothelium were created. The reason this method works is that the Laplaciankernel highlights regions of an image containing rapid intensity changes. If thevariance is very high, there are huge edges in the image, that is, the image isfocused. In this paper, automatically created panoramic images were evaluatedby comparison with manually created panoramic images, which were definedas correct images. This system can be applied to the evaluation of the
progression of symptoms

OS8-5 An fMRI study of the inhibitory effects of the random stimulus-response compatibility
task on brain function
Kei Sahara, Satoru Hiwa, Hiroshi Furutani, Tomoyuki Hiroyasu (Doshisha University, Japan)

Our behavior is optimized by the brain to suppress unnecessary movement atparticular moments. Here, we investigated the differences in inhibition during
a spatial stimulus-response compatibility task. The stimulus-responsecompatibility task is used to investigate brain function while inhibiting
movements to accomplish the optimized movements during the compatible
and incompatible trials. A previous study investigated inhibitions in specificcompatible and incompatible tasks. However, these studies did not consider
the inhibition during random events, which is an everyday occurrence inhuman life. Therefore, in this study, we focused on the differences between
specific inhibition and random effect inhibition using three stimulus-responsecompatibility tasks, comprising the compatible, incompatible, and randomtasks. Twenty healthy participants were measured using 1.5-T functional
magnetic resonance imaging (fMRI) and the results were analyzed by using
activation, brain function network, and behavioral analyses.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 56 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS8-6 Measurement of brain activity and problem discovery during actual driving

Seishiro Nakamura, Satoru Hiwa, Kenya Sato, Hiroshi Furutani, Tomoyuki Hiroyasu(Doshisha University, Japan)

In this study, we focused on a system that provides driving support basedon a driver's biological parameters. Brain function is one such biologicalparameter and functional near-infrared spectroscopy (fNIRS) was used tomeasure brain function. fNIRS is less susceptible to body movements andeasier to measure than electroencephalography. In order to compare brain
activities during actual vehicle operation and simulated operation, a task
performed using a simple course was prepared. In order to quantify thechange in cerebral blood flow obtained from each channel, the fractional
amplitude low-frequency fluctuations (fALFF), which is an index of local
spontaneous brain activity, was calculated. Furthermore, it was ztransformed (zfALFF) to allow comparisons between subjects. In addition,
to confirm the relationship between driving performance and brain
activity, we compared the time series data for driving performance andcerebral blood flow change obtained from each channel. The measurementof brain activity during actual vehicle driving was performed and theresults were analyzed. In addition, discussions of the findings and future
directions are also included.

OS8-7 Control of driving simulator based on state detection of the driver using
Electrocardiogram measurement

Koma Yoshioka, Satoru Hiwa, Kenya Sato, Hiroshi Furutani, Tomoyuki Hiroyasu(Doshisha University, Japan)One of the practical issues concerning automated driving vehicles is
identifying the state of the driver when switching between manual andautomatic control. When switching, it is important to detect the wakefulness
and awareness of the driver. In this study, we controlled a driving simulator
based on driver state detection. Electrocardiogram (EKG) measurement wasused for driver state detection. Heart rate variability analysis (HRV) is a
widely known means for estimating sympathetic and parasympathetic activity.
We created a Simulink model to calculate SDNN, RMSSD, and LF/HF in realtime, which are indicators obtained from the HRV analysis. Based on themodel, the state of the driver during the driving simulation was detected. We
also created a Simulink model that switches between automatic operation andmanual operation of simulated vehicles. By applying these models,construction of a system that safely switches between automatic operation andmanual operation is expected.









©ICAROB 2019 ALife Robotics Corp. Ltd.

- 57 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS9 Natural Computing and Beyond (3)OS9-1 Tactile Score: Development and Applications

Yasuhiro Suzuki (Nagoya University, Japan)

Unlike music, which can be described by means of a musical score, tactile
sense does not have a standard method for transcription. Likewise, tactile
sense has no developed principle of composition. Hence, this study developed
a method for describing tactile sense, referred to here as the ‘tactile score’. We
developed a mutual transformation method between tactile score and haptic
vibrations, and then we verified that a hand massage performed according to atactile score and haptic vibration from a tactile score evoked similar brain
activities.

OS9-2 Tactileology; Haptic Informatics by using Tactile bit, T-bit
Yasuhiro Suzuki (Nagoya University, Japan)

Most of Internet of Things, IoT data is time series data, however the timeseries patterns of tactile sense have not been investigated. Tactile Score is a
novel method for describing time sequences of different tactile senses, such as
massages. It can not only describe the massages but also tactile senses in
multimedia data such as locution, vocalism or rhetoric, etc.; multimedia timeseries data can be transformed into Tactile Score and we can display tactile
senses denoted by Tactile Score by using Tactile Score players. Tactile Score
enables us to compose and edit tactile senses. We extract composing principleof time series patterns of tactile senses though analyzing massages; extracted
principle has been verified through examining biological responses. OS9-3 Natural Computing and Formal Computing

Yasuhiro Suzuki (Nagoya University, Japan)

In natural computing, computing substances, such as DNA molecules, cells or
ecosystems, observe the result of algorithm and verify it by themselves.
Hence, natural computing is primitive in computing and throughout of theevolution from natural computing, it develops “external formal” system toobserve and verify and evolve to formal computing system. In this talk, I will
briefly address the evolution of computing from natural computing to formal
computing and show a novel natural computing framework; thinking about
nature by natural computing. ©ICAROB 2019 ALife Robotics Corp. Ltd.

- 58 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS10 Mathematical Informatics (5)OS10-1 A Study on Tourism Support Application Using the Virtual Technology

Masamichi Hori1,Makoto Sakamoto1, Takahiro Ishizu1, Satoshi Ikeda1, Amane Takei1, Takao Ito2, Yu-an Zhang3(1University of Miyazaki, 2Hiroshima University, Japan, 3Qinghai University, China)
Currently, the tourism industry in Miyazaki prefecture has various problems,and various measures are taken. On the other hand, in 2016, virtualtechnology attracted much attentions such as VR and AR technology. We
thought that through the VR and AR technology experience, we could aim for
an increase in tourists. Therefore, we will create new applications for
smartphones using virtual technology and aim for tourism support.  We
propose unique features aimed at increasing tourists and creating repeaters. It
is a function that allows you to reproduce the scenery anywhere with VR, andto feel the sightseeing experience by AR close.

OS10-2 Proposal for Interaction Techniques for Intuitive Virtual Objects Manipulation in
Augmented Reality

Takahiro Ishizu1, Makoto Sakamoto1, Takaaki Toyota1, Masamichi Hori1, Satoshi Ikeda1, Amane Takei1,
Takao Ito2, Yu-an Zhang3 (1University of Miyazaki, 2Hiroshima University, Japan, 3Qinghai University, China)
Recently, studies for realizing interaction between users and virtual objects
using Augmented Reality (AR) have been actively conducted. However, few
previous studies have realized that a user intuitively manipulates virtual
objects like clay with their bare hands. In this study, we aim at user's
intuitive virtual objects manipulation by realizing basic manipulation on
virtual objects with their bare hands. In addition, it is considered that theuser feels uncomfortable in the manipulation of the virtual objects becauseof the occlusion problem in AR, and cannot perform an intuitive
manipulation. Therefore, in this study, we solve this problem by performing
appropriate hidden surface processing within the range of the user's
fingertips.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 59 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS10-3 Two-Dimensional Image Based Body Size Measurement and Body Weight Estimation for
Yaks
Zijie Sun1, Chen Zhang1, XiaoFeng Qin1, Yu-an Zhang1, Rende Song2, Makoto Sakamoto3(1Qinghai University, 2Yushu Prefecture Animal Husbandry and Veterinary Station, China
3
University of Miyazaki, Japan)

In stock raising, non-contact and non-stress measurement of animal’s bodysize parameters and body weight is a research highlight. Yak’s body size
parameters and body weight estimated based on body size can reflect yak’s
growth and development features, production performance and hereditarycharacters. Therefore, studying the body size measurement and body weight
estimation of yaks is an effective approach to promoting scientific breedingand enhancing culture benefits. Traditional manual methods of measuring
animal’s body size and body weight have such disadvantages as heavy
workload and potential safety hazards. This paper firstly acquires the color
image of the yak by taking pictures of yaks in Sanjiangyuan. Then, the yak is
extracted from the image with the method of foreground extraction based ongrey level classified process; after that, the profile curve of the yak is
extracted to obtain the temple point and ischium end of the yak with themethod of calculating the image edge curvature. Based on that, the marked
points for the yak’s body height and body dip length are determined; weka
based linear regression helps obtain the body weight estimation formula;
finally, estimation about the yak’s body weight is achieved. Experimentssuggest this algorithm is effective to non-contact body size measurement andbody weight estimation for yaks in Sanjiangyuan, Qinghai Province.

OS10-4An efficient structure of organization with complete group guidanceSatoshi Ikeda1, Mamoru Yoshimura1, Makoto Sakamoto1, Takao Ito2
(1University of Miyazaki, 2Hiroshima University, Japan)

The purpose of this paper is to show that the efficient organizational shapewhen the evaluation criteria is only one, is limited to the three types under theassumption of the communication style called the complete group guidance.

OS10-5 Parallel finite element analysis for hyperbolic problems

Amane Takei,, Makoto Sakamoto (University of Miyazaki, Japan)

In this research, we propose a parallel finite element analysis for hyperbolic
problems based on an iterative domain decomposition method. In a
microwave analysis, the simplified Berenger’s PML is developed in whichthese eight corners are given the average value of all PML’s layers. As for
accuracy verification of the analysis solver, the absorbing performance of the
PML is evaluated by using reflection coefficient based on S parameter. In theaccuracy verification, when the maximum side length is set to 1/20 of the
wavelength, the error rate of the numerical solution and the theoreticalsolution is about 1.70 [%] with the PML of 9 layers. So, we have found that
proposed method can calculate with high accuracy.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 60 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019OS11 Recognition and Control (12)OS11-1 Design of feed part control system for rectification process

Lingran An1, Fengzhi Dai1, Yujie Yan1, Zhongyong Ye1, Xia Jin1, Yiqiao Qin12, Chengcai Wang3, Kaige Liu1
(1Tianjin University of Science and Technology, 2Tianjin Tianke Intelligent and Manufacture Technology
CO.,LTD, 3 CETC Ocean Information Co., Ltd, China)
This paper completes the structure design of the feed process of the small
rectification experimental unit. According to the process requirements, we
design the overall scheme of the detection and control system of the main
parameters and select the type of the instrument.The system hardware designis based on the CPU222 of Siemens s7-200 PLC and the corresponding
input/output module. The lower computer adopts the ladder diagram tocomplete the program design such as parameter acquisition, scaletransformation and PID operation. The monitoring interface of the upper
computer adopts configuration the King monitoring software.

OS11-2 Ultrasonic range finder for vehicle collision avoidance system

Ying Chen1, Guowei Yang1, Fengzhi Dai1, Yuxuan Zhu1, Di Yin1, Yasheng Yuan1,Yiqiao Qin12, Ce Bian12,Chengcai Wang3, Xinyu Zhang1 (1Tianjin University of Science and Technology,
2
Tianjin Tianke Intelligent and Manufacture Technology CO.,LTD, 3CETC Ocean Information Co., Ltd, China)
In recent years, highway rear-end collision accidents have occurred
frequently, causing huge personal injury and property damage. Protective
measures such as seat belts and airbags only serve to reduce damage
afterwards and do not solve the problem from the source. Therefore, researchon the system of vehicle collision avoidance, which actively avoids accidents,
is of great significance. The ultrasonic range finder designed based on
AT89C51 single-chip microcomputer utilizes the principle of sound wave
reflection in the air. It can detect the distance between the vehicle and the
obstacle in real time, and immediately alarms beyond the set threshold.
Ultrasonic ranging is widely used in automotive anti-collision systems due toits non-contact detection method and simple structure.

OS11-3 Chaos synchronization method of Qi system and the circuit design

Xia Jin1, Fengzhi Dai1, Zhongyong Ye1, Lingran An1, Yujie Yan1, Chenglin Zhao1, Yiqiao Qin12, Hao Li3(1Tianjin University of Science and Technology, 2Tianjin Tianke Intelligent and Manufacture Technology
CO.,LTD, 3CETC Ocean Information Co., Ltd,, China)
Chaos is a kind of uncertain and random phenomenon that appears in thesystem. Its dynamic characteristics are rich and interesting. The study of the
dynamic characteristics and synchronization methods of Qi chaotic system has
fascinated many researchers. In this paper, the dynamic characteristics of Qi
chaotic system are analyzed, and the synchronization circuit of Qi chaotic
system is designed based on the driving-response chaotic synchronous systemand the single-variable feedback perturbation synchronous system, The
synchronization circuits of Qi chaotic system are designed respectively,
comparing the synchronization curves of the variables simulated by Multisim
and the numerical simulation curves obtained by MATLAB, verifying thecorrectness of the built circuit, and analyzing the advantages and disadvantages
between the above two synchronization methods.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 61 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS11-4 Design of air quality monitoring platform based on Internet of Things

Yujie Yan1, Fengzhi Dai1, Kailun Zhang1, Wei Wang2, Jialin Han1, Yang Li3, Tianyi Zhang1( Tianjin University of Science and Technology, 2China Institute of Aerospace Engineering, 3CETC Ocean
Information Co., Ltd, China)
1

The air quality monitoring platform based on the Internet of Things (IoT) uses
the STM32f407 main control module. Data such as temperature, humidity,
PM2.5 and carbon monoxide concentration in the atmospheric environment arecollected for a long time by each sensor module. Not only the system has the
LCD display function, users can also log on to the ESLINK IoT cloud platform
to remotely observe the data collection status. The upper limit of each parametercan be set in a specific environment. If the data monitored by the system
exceeds the upper limit, the system will alarm. The system has a good
application prospect in the environment of office, automobile, factory and so on.

OS11-5 Design of metal weld seam tracking equipment based on image processing

Yujie Yan1, Fengzhi Dai1, Lingran An1, Xia Jin1, Zhongyong Ye1, Yufan He2, Xinran Guo3
(1Tianjin University of Science and Technology,
2
CETC Ocean Information Co., Ltd, 3Nanjing University, China)

With the development of the intelligent industry, metal automatic welding
technology needs to be more popular. The metal weld seam tracking device
based on image processing includes two parts of image processing and motion
control. The image processing part is developed by the Visual Studio 2013
platform, and the motion control part is controlled by the PID algorithm.
Machine vision technology can be used to convert images captured by industrial
cameras into vector graphics, which in turn drive the welding table to track
metal welds. The device has good control effect and can be well applied insmall and medium-sized enterprises.

OS11-6 The recognition and implementation of handwritten character based on deep learning
Zhongyong Ye1, Fengzhi Dai1, Xia Jin1, Yasheng Yuan1, Lingran An1, Yujie Yan1,
Yiqiao Qin12, Hao Li3 (1Tianjin University of Science and Technology, 2Tianjin Tianke Intelligent andManufacture Technology CO., LTD, 3CETC Ocean Information Co. Ltd, China)
For recognizing handwritten characters, especially the handwritten Chinesecharacters, this paper uses deep learning techniques to construct convolutional
neural networks of different depths to identify character sets containing
handwritten digits and Chinese characters, and compares the performance of
networks with the same depth and different depths. In the end, the network
structure that can be used for identification is finally obtained, and ahandwritten character recognition system is implemented based on the network.
The recognizing accuracy of the paper reaches 99.18% for the MNIST dataset
and 92.02% for the subset of the Chinese dataset HWDB1.1. Compared withother handwritten character algorithms, this paper proves that the used network
structure has the application value.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 62 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS11-7 Cluster analysis of wine based on three-dim fluorescent spectra characteristic

Di Yin1, Fengzhi Dai1, Yuxuan Zhu1, Yasheng Yuan1, Tingting Zhang1, Ying Chen, Yiqiao Qin12, Chengcai
Wang3 (1Tianjin University of Science and Technology, 2 Tianjin Tianke Intelligent and Manufacture Technology
CO., LTD, 3CETC Ocean Information Co., Ltd, China)
The traditional sensory analysis method can not discriminate different kinds ofwine objectively and accurately. The three-dimensional (3D) fluorescence
technology has the advantages of strong selectivity, high resolution and directaccess to the fluorescence characteristics of the sample. In this paper, the 3D
fluorescence spectroscopy five brands of red wine samples. The fivecharacteristic parameters (the mean value, standard deviation, center of gravity
coordinates, first order center moment and correlation coefficient) of the 3D
fluorescence spectra of these five dry red wine samples are extracted
respectively. The results show that the difference of the samples can be
distinguished intuitively by extracting the 3D spectral parameters of thedifferent brands. On this basis, the clustering analysis shows that even the sameraw material brewing wine is different from the raw material origin and the soil
environment. The climate environment will cause an important impact on thecomposition of the wine.

OS11-8 The control design of the planting device for the hydroponic vegetable

Yasheng Yuan1, Fengzhi Dai1, Zhongyong Ye1, Yuxuan Zhu1, Di Yin1, Tingting Zhang1, Yiqiao Qin12, Yang Li3(1Tianjin University of Science and Technology, 2 Tianjin Tianke Intelligent and Manufacture Technology
CO.,LTD, 3CETC Ocean Information Co., Ltd, China)
The equipment is designed to household indoor small hydroponic vegetable
planting control device. The temperature, humidity, light and carbon dioxide
concentration sensors are integrated to the Arduino UNO R3 microcontroller. It
is used to detect various parameter data of planting environment. The WIFI
module can send the real-time environment variables detected by the sensor tothe mobile devices, such as the user's mobile phone, through the wirelessnetwork. Moreover, users can also change the corresponding control range ofvariables according to the planted vegetables through the WIFI module.

OS11-9 An anti -theft system based on the self-checkout

Yuxuan Zhu1, Fengzhi Dai1, Yasheng Yuan1, Di Yin1, Tingting Zhang1, Ying Chen1, Ce Bian12, Yufan He3(1Tianjin University of Science and Technology, 2Tianjin Tianke Intelligent and Manufacture Technology
CO.,LTD, 3CETC Ocean Information Co. Ltd, China)
Nowadays, the imperfection of the anti-theft system impedes the developmentof self-service cash registers. This paper designs an anti-theft system based on
self-service cash registers, which can greatly reduce the theft rate insupermarkets. This anti-theft system adopts Siemens’s LOGO!, which is thecore of the controller. Moreover, it has massive facilities of anti-theft protection,
including radio frequency detection, photoelectric counting, repeated scanningreminder, magnetic detection, video monitoring and anti-theft electronic door.
Through the above technologies, the supermarket that uses the self-service cash
registers can avoid to some problems, i.e. cash collection leakage, using low
price goods to replace that of high price. Multiple facilities are set to jointly
protect the safety of goods, effectively reduce theft and loss.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 63 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS11-10 Case study on communication between embedded linux environment andmicrocontroller
Ziheng Gao, Yizhun Peng, Shuo Wang
(Tianjin University of Science and Technology, China)

In many embedded development scenarios, we need to combine the real-time
microcontroller and non-real-time embedded Linux environment forcollaborative development. In a high real-time environment (such as: vehicles,
production lines), it can complete kinds of features relying on huge Linuxecosystem, such as hot fixable functional modules, improved network
communication, easier OTA firmware updates, more efficient algorithm
capabilities, and more objects. This paper aims to enumerate and explore
several methods that can be used to implement similar duplex communications.
For ease of understanding,a single-board computer called Raspberry Pi running
Raspbian Linux and an STM32F103 32-bit ARM microcontroller are used inexperiment.

OS11-11 Design and implementation of a baby care robot

Yizhun Peng, Zefan Ge, Kaixuan Geng, Zhou Yang,Rui Yang
(Tianjin University of Science and Technology, China)

Aiming at the shortcoming of single function of traditional pram, this paper
proposes an embedded multi-functional intelligent baby care robot. With baby
cradle as the carrier, the bionic cradle is realized by combining embedded
controller, drive module, motor and various sensors. The babysitting robot has
the functions of autonomous following and obstacle avoidance, voice play andreal-time picture return, which not only improves the convenience and securityof the baby carriage, but also can assist parents to take care of the baby , greatly
reducing the burden of parents' parenting.

OS11-12 Research on intelligent shopping service robot design

Shuo Wang, Yizhun Peng, Ruixiang Bai, Ziheng Gao(Tianjin University of Science and Technology, China)

With the rapid development of the social economy, it’s good to help customers
have more efficient and relaxed shopping experience through intelligent means
in large commercial centers. In this paper, a shopping service robot project thatuses machine vision, deep learning, and IoT is introduced. The robot can help
customers complete shopping and complete indoor positioning navigation andautonomous movement intelligently. The robot product is designed to help all
customers, especially people with mobility impairments. It will reduce theirburden of action, thereby enhancing the user experience and doing special
analysis of shopping behavior to help launch more efficient and personalized
market planning.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 64 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS12 Robot Design and Simulation (5)OS12-1 Design and Research of Navigation and Dynamics Co- Simulation Platform forPlanetary Rover
Huang Tie-qiu, Zhang Bo-wen, Huang Qian-li (Beijing Jiaotong University, P.R China)

In view of the difficulties in the design of the navigation control system for the
exploration of complex and passing missions on Mars and other planets, theoverall design idea of the co-simulation platform for the planetary rovers
navigation and dynamics is proposed. On the basis of MSC.ADAMS software,
the dynamic model of planetary rover in soft soil environment is establishedthrough two times of subprogram development. An interactive parallel
simulation system is established by using C++, and the cooperative invocation
dynamic model and navigation control model are solved to form the closed loop
system of co-simulation, and the navigation control system and the dynamic
model are realized. Seamless link and parallel efficient solution. Taking theco-simulation of Mars rover as an example, the simulation platform hascompleted the simulation of Mars rover cruising, crab line, steering, lifting
wheel, body lifting and so on in the Martian environment, and can show themoving state of the rover, the contact force of the rover and the driving torqueof each mechanism in real time. Due to the high efficiency of the wheel contact
model algorithm and the application of parallel multi task processing method,co-simulation can almost achieve real-time. The platform lays an important
foundation for the design of navigation control system.

OS12-2 Research on Application of SFM Method in Virtual Reality Modeling
Jiwu Wang, Chenyang Li (Beijing Jiaotong University, China)
Min Li (Beijing Forestry University, China)

Simulation technology has become an important part of design and training.
With the development of computer science, Virtual Reality（VR）technology canenable the experiencer to reach the immersion degree that other simulation
technologies cannot achieve, and become the focus of military simulation
research in various countries. But VR technology also has some shortcomings,
such as long modeling period, certain differences between models and real
objects. Based on the current technology, this paper proposes a VR technology
combined with SFM which can greatly reduce the modeling period and improve
the realism of the model to enhance the experience of simulation.

OS12-3 Visual SLAM System Design based on Semantic Segmentation

Jiwu Wang, Yafan Liu, Qinjian Zhang (Beijing Jiaotong University, China)

Visual SLAM is helpful for the development of unmanned platform. But only
perceiving the geometric information of the environment is not enough. It isnecessary to join the image semantic segmentation with SLAM, which helps
to comprehend the scenes and improve the accuracy of pose estimation.
Nowdays, the technique of deep learning brings new ideas into traditional
SLAM by modeling this problem with convolution neural network.In thisapproach, we propose a pose estimation technique based on pixel-level
multi-target detection, and 2D semantic information are transferred to 3D
mapping via correspondence between connective Keyframes. Meanwhile,thesemantic information is utilized in other modules for to optimize the visual
SLAM system.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 65 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS12-4 Spherical Mobile Robot Designed with Single Omnidirectional Wheels
Liu Wei, Lian Luo, Jiwu Wang (Beijing Jiaotong University, China)

This paper presents a new spherical mobile robot with a single
omnidirectional wheels locating at the ends of the inside vertical diameter ofthe sphere shell, and the two wheel axes being in orthogonal position. The
omnidirectional wheels can roll along the inner surface of the spherical shell,
and then drive the spherical shell rolling. The lower omnidirectional wheel is
responsible for walking drive, and the upper one for steering drive. The
prototype shows the spherical robot moving well.

OS12-5 An augmented reality implementation method based on Unity3D

Weixin Zeng, Jiwu Wang (Beijing Jiaotong University, China)

Augmented reality technology uses computer graphics and visual technology
to achieve the superposition of virtual objects in real scenes, presenting thesensory effects of real-time interaction. It has broad application prospects andapplication value, and its popularity has been increasing in recent years. At
present, there are two main ways to achieve augmented reality. One is theunderlying code, which combines OpenGL and Open CV libraries to realize
augmented reality. The second is the secondary development of related
content through the corresponding software and SDK package. In this article,
we will initially implement augmented reality through Unity 3D and
Qualcomm's Vuforia SDK.

OS13 System and Control (10)OS13-1 Research on a non-invasive measuring method of blood glucose concentration based onelectrical impedance spectrum
Tong Yin, Xiaoyan Chen, Meng Du, Hongyi Yin (Tianjin University of Science and Technology, China)
Different from the traditional measurement approach and instruments of blood
glucose concentration, we propose a novel measuring method related toelectrical impedance spectrum without invasion. A serial experimentations aredesigned and carried out to detect the electrical impedance amplitudes andphases under 1Hz-10MHz frequencies. Considering the skin contacting
impedance and cytomembrane conductivity feature, a simulate circuit isdesigned and connected in serial with the measuring loop. The measured data
were analyzed by an improved Cole-Cole model to derive the relationshipbetween the impedance value and the blood glucose concentration. The results
show that the electrical impedance is strong negative correlation to blood
glucose concentration, as the blood glucose concentration increases, theimpedance value decreases. The conclusion of this research provides atheoretical basis for non-invasive blood glucose surface-measuring.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 66 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS13-2 Synchronization of Novel 4D Chaotic Systems with Different Control Laws
Hong Niu (Tianjin University of Science and Technology, China)

In this paper, synchronization of novel four-dimensional(4D) autonomouschaotic systems is studied. Two different control laws are presented, which are
based on the strictly positive realness of transfer function matrix and thecenter translation method respectively. The characteristics of the two control
laws, as well as the differences between them, are proposed. Some relevant
numerical simulation results, such as the curves of the corresponding
synchronization state variables and the errors, are given to illustrate thefeasibility and effectiveness of the two control laws.

OS13-3 Chaotic characteristics analysis of fractional-order Liu system

Wengxin Shi, Hongyan Jia (Tianjin University of Science and Technology, China)
10
8
6
z

The paper analyzes a newly reported fractional-order Liu system based ontime-domain approximation and frequency-domain approximation,
respectively, and chaotic characteristics of this system are investigated by thephase portrait diagram, Lyapunov exponents diagram and bifurcation diagram.
The results of the two approximation methods are consistent, which not onlyshows chaotic dynamics of the system is reliable, but also is very convenient
to utilize chaotic dynamics of the system with fractional-order 0.9.

4
2
0
-10

-5

0
y

5

10

OS13-4 Design and analysis of the multi-robot grouping aggregation algorithm
Huailin Zhao, Zhen Nie (Shanghai Institute of Technology, China)

In this paper, the grouping aggregation problem of multi-robot in simple
environment is studied. The grouping algorithm and aggregation algorithm aremainly discussed. Based on two clustering algorithms, K-means andK-medoids, two grouping algorithms are designed. And the noise im-munityand grouping rapidity of these two grouping algorithms are analyzed.
Aiming at the problem of aggregation, three kinds of aggregation algorithms
are proposed. Taking the center aggregation control as an example, thecooperation and control of each group of robots are studied. Finally, thesimulation results of MATLAB shows that the multi-robot grouping andaggregation algorithm is effective.

OS13-5 Research of the ARM-based multi-robot aggregation control

Huailin Zhao, Jingnan Zheng, Zhen Nie, Weiya Zhang, Zonglin Liu (Shanghai Institute of Technology, China)
This paper mainly analyzed the basic aggregation problem of multi-robot
cooperative system. The algorithm design of multi-robot dynamic aggregation
control was discussed. The multi-robot aggregation path planning method in
simple environment was discussed. And the kinematics equation of theaggregation algorithm was given at last. Taking the center aggregation control
as an example, the cooperation and control of multi-robot are studied. Then
we designed a simulation control program on MATLAB for multi-robot
system and analyzed the simulation results. Finally, the center aggregation
experiment was carried out by using the three car robots. The results show thatthe multi-robot aggregation algorithm is effective.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 67 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS13-6 Research on the Trajectory Planning Problem of Robots in Spherical Cutting
Jun Min, Wenping Jiang, Huailin Zhao (Shanghai Institute of Technology, China)

For cylindrical shape cutting of spherical surface vessel a kind of 5-DOF
cutting method of cylin-drical shape for spherical surface is provided in thispaper. Based on the kinematics model of cutting robot the mathematic
relations between cutting tool cylindrical shape and spherical surface are built
using Euler transformation and the intersection line model of cylindrical andspherical surface is presented. The robotic cutting trajectory is computed
according to the position and attitude relation of robot and spherical sphereand simulations and experiments of cutting trajectory for robot are provided in
the end. Experiments results verified the rightness of robot kinematics andcylindrical shape cutting algorithm, and will promote the robot-is applicationin cutting field.

z3

y3

Joint 3
z2

x3

d3
y2

Joint 14
d4

Joint 2
z

d1

y

x2

Joint 5

y4

d5

z1

Pedestaly1

Joint 1

x1

Tool

z4
yt

Qiang Wei1, Xiaoyu Zhang1, Xiao Yang1, Yixin Lin2, Qingming Liu3
( Jianghan University, 2Huazhong University of Science and Technology,3
Wuhan Rayson School, China)
1

To evaluate the time course of changes in emotional processing between
artificial voices and human voices (happy, angry, neutral), the present study
examined behavior experience and event related potential component. The
emotion arouse of artificial human voice and brain signal were evaluatedwhile the presentation of audio from each conditions (happy, angry, neutral,
and artificial voices). The results showed that there were differences in theintensity and the late processing between emotional processing induced byartificial voices and human voices.

Qiang Wei1, Jingxuan Huang1, Yu Han1, Yixin Lin2
( Jianghan University, 2Huazhong University of Science and Technology, China)
1

The aim of this study was to investigate the ERP component evoked byemoticon with different attributes and the reaction times during the LDT task.
The behavioral results showed that participants demonstrated a positive
emotion with a shorter reaction time under both a positive emoticon and anegative emoticon compared to participants under a meaninglessemoticon.The results of ERPs indicated the relationship between three ERP
components and emotional processing of attention distribution, emotion
arousal and emotion valence. Based on these results, we provide evidence fora better understanding of how emoticons improve atmosphere during daily
communication.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 68 -

d6xt

d2

OS13-7 Emotional processing between artificial voices and human voices

OS13-8 How people are affected by emoticon icon: An ERP study

x4x5

zt

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS13-9 Single Chinese character fragments: an ERPs study on orthographic neighborhood
effectQiang Wei1, Yunfei Wang1, Chen Song1, Yixin Lin2
( Jianghan University, 2Huazhong University of Science and Technology, China)
1

The aim of study was to discuss family effects and brain activity during theprocessing of Chinese characters. The study observed the event-related
potentials (ERPs) elicited by fragments of Chinese single-character words in
Chinese character completion and naming experiment. The result showed thatthe “Multiple-solution” elicited greater N400 component than the“One-solution”, which index a inhabitation effect. Multiple-solution-wordanalogy to large families, they are more intense than the false-words andsingle-words, which are in line with the interaction activation model and havemore resources in the cognitive processing process. The glyph processing, theassociation process, and the memory phase of Chinese characters have an
impact. The N400 component may not only mean the integration of themeaning of the word, but may also be related to the process of adjusting theinformation to adapt to the task.

.

OS13-10 Research on embedded electrical impedance measurement system

Tong Yin, Xiaoyan Chen, Hongyi Yin (Tianjin University of Science and Technology, China)

This paper develops an impedance measurement system based on S3C6410
processor. An OK6410 host board as the controller platform, and an AD5933
chip is used to collect impedance data, and the data is exchanged with thecontroller with IIC bus protocol. Comparing with the measuring result of theWayne Kerr 6800B impedance analyzer with the accuracy of 0.05%. The
system has the behaviors as lower relative error 0.422% and higher
signal-to-noise (SNR) ratio 64.11dB. After calculation, the average absolute
error of the impedance phase of the system is 0.527°. The tests results verified
that the system is reliable and flexible with a wide application prospect.

OS14 Intelligent Control (4)OS14 -1 Flock Guiding of Hybrid Agents via Root Block

Yunzhong Song, Ziyi Fu, Fuzhong Wang (Henan Polytechnic University, China)

The convergent equilibrium of flocking agents is vital importance for theircollective behavior, where flock guiding of hybrid agents, which arecomposed by both first order and second order agents, can only be boiled
down to the root node agents, is touched upon here. Both the theoreticalanalysis and the simulation results are provided to verify the suggested
scheme.

2

5

1

3

4

)LJ1HWZRUNFRQVLVWVRIQRGHVOS14-2 Multiple Model Adaptive Control of Flexible Arm

Yingzhao Zhang1, Xiao Wang1, Handong Li1, Weicun Zhang2( Guizhou University, 2University of Science and Technology Beijing, China)
1

In this paper, dynamic model of flexible arm is built through Lagrange theory.
Then, based on the software SIMULINK, model of flexible arm is establishedand the state of flexible manipulator under multiple model adaptive control issimulated. Finally, comparing the control effect (like stability, and accuracy)of multiple model adaptive control method with other control methods,
Summarize the advantages of MMAC control method.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 69 -

state of flexible manipulator
under multiple model adaptive
control

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS14-3 Time-Varying Lyapunov Function for Mechanical Systems

Bin Zhang1, Yingmin Jia2
( Beijing University of Posts and Telecommunications, 2Beihang University (BUAA), China)
1

In this paper, a general method for constructing time-varying Lyapunov
functions is provided by using homogeneity theory. Sufficient conditions of
uniform asymptotical stability are established for mechanical systems.
Different from the existing results on this subject, we remove the periodic
restrictions and persistency of excitation restrictions in our work. Robustness
properties are also considered in this paper. It is shown that the mechanicalsystems remain stable after adding higher order perturbing terms. To illustrate
the utility of our results, we provide detailed applications on practical systems
and numerical results are given.

States of the Mechanical
Systems.

OS14-4 Optimizing Control Parameters of Space Robot Manipulator for Pulsar X-ray
Interference Measurement

Qiang Chen1, Hengbin Zhang1, Xiaomin Bei1, Hanwen Zhang2( Chinese Academy of Space Technology, 2Beijing Institute of Control Engineering, China)1

In paper, a pulsar X-ray observation plan of using two detectors installed in
two sides of space station is given. Dynamic model of the space robot
manipulators and an adapt control method are described. An orthogonal arrayexperimental design method is developed to optimize the control parameters.
Simulations are carried out by L81 (340) orthogonal array while fourteen
adaptive control parameters are selected. The optimization results show thatthe parameters expected value could be acquired and computationallyefficiency is promoted.

The composition of pulsar x-ray
measurement system

OS15 Advanced Control (5)OS15-1 Pulse Pose Follow Control and Simulation for a 4-DOF Pulse Diagnosis Robot
Qunpo Liu1, Guanghui Liu1, Hongqi Wang 1, Xianzhe Liu1, Hanajima Naohiko2(1Henan Polytechnic University, China, 2Muroran Institute of Technology, Japan)

Pulse diagnosis has been proved to have great practical value in the pastdynasties. The quality of pulse diagnosis is related to the effect of“syndrome differentiation and treatment” in various clinical departments. The
key of high-quality pulse diagnosis is how to dynamically adjust the vertical
relationship between the pulse and the sensor. The design adopts a four-degreeof freedom manipulator combined with a matrix sensor to form a diagnostic
manipulator, which dynamically adjusts the posture of the end sensor of themanipulator by using the flexibility of the manipulator, so as to achieve rapid
adjustment and keep the diagnostic pulse sensor perpendicular to the pulse. In
this paper, the method of Denavit-Hartenberg is used in kinematics analysis of
the manipulator, and Simulations are platformed based on the robot dynamics
simulation software V-REP. The simulation results show that the design is
feasible.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 70 -

Fig.1Pulse diagnosis robot

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS15-2 A New Adaptive Control System Design Method Based on Neuronetwork
Prediction
Weicun Zhang, Sufang Wang, Yongnan Jia, Qing Li(University of Science and Technology Beijing, China)

In this paper, a new control system design method is proposed, in which, thecontroller is designed according to a model with un-modeled uncertainties, anartificial neuronetwork (error back propagation neuronetwork) was adopted to
predict the model output error, and the predicted model output error isconsidered in the control law. This control system design can be used to bothlinear and non-linear systems. The simulation results verified the effectivenessof the proposed method.

yr

C

u

P

 (k)

y

M
ANN

Block diagram of the proposed
schemeOS15-3 Mean-square quasi-composite rotating formation control of second-ordermulti-agent systems under stochastic communication noise

Lipo Mo1, Xiaolin Yuan1, Yingmin Jia2, Shaoyan Guo3
( Beijing Technology and Business University, 2Beihang University (BUAA),
3
South China University of Technology, China)
1This paper mainly concerns the mean-square quasi-composite rotating formation
problem of second-order multi-agent systems (MASs) with stochastic
communication noises. Firstly, the definition of mean-square quasi-composite
rotating formation is proposed. Afterwards, a novel distributed control protocol
with time-varying control gains is designed. And then, the origin closed-loopsystem is translated into an equivalent closed-loop system by taking a proper
coordinate transformation. Under some mild assumptions, by means of
stochastic differential equation theory, sufficient conditions are deduced forachieving mean-square quasi-composite rotating formation. Eventually,
numerical simulations are provided to confirm the effectiveness of the proposed
theoretical results.

Fig. Local trajectories of theposition and moving rotating
centers of the MASs.

OS15-4 Distributed Energy Resource Control Based on Multi-Agent Group Consensus

Yize Yang1,2, Hongyong Yang1, Fan Liu 1, Yuling Li1, Yuanshan Liu1 (1Ludong University, China, 2The
University of New South Wales, Australia)
In this paper, a cooperative control strategy of distributed energy resources
based on the group consensus of multi-agent systems is proposed. Combining
with actual engineering needs, a multi-agent dispatch model of energy networks
is designed. Considering the possible adverse effects that may be encountered in
actual energy transportation, the matched disturbances and mismatched
disturbances are added to simulate the modeling. A group control strategy of
distributed energy resources applied to the distribution network is proposed toachieve precise control of energy distribution and safe transportation undermultiple disturbances. Finally, the numerical simulation results verify thevalidity of the model and algorithm.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 71 -

group consensus of distributed
energy resources

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS15-5 Robust Adaptive Control of Air-to-air Refueling Boomwith State-dependent Output Constraints

Liang Chang, Yingmin Jia (Beihang University (BUAA), China)

In this paper, a robust adaptive controller design method is presented for theuncertain air-to-air refueling boom system with output constraints and disturbance.
One advantage of this design method is that the output constraints arestate-dependent, which can extend the workspace of the boom while avoiding
collision between the boom and the receiver. Backstepping method and barrier
Lyapunov function(BLF) are used in this design method. Simulation results show
the effectiveness of the proposed controller design method.

Aerial Refueling Boom

OS16 New development about Control Engineering Education (4)OS16-1 Skill Model Estimation of Ability for Reading Drawings
Kazuo Kawada, Tsukasa Hiyama (Hiroshima University, Japan）
In Japan, students are studying various drawing educations in junior high school
technical education, high school industrial education and university engineering
education. However, it is very difficult to read the drawing without understanding
trigonometry. Moreover, it is also very difficult to draw wish parts. Because of this,
there are many research studies, but these are before-after analysis, and none
analysis by time-series data. Therefore, this study aims to develop a newquantitative evaluation method of 3D recognition ability.

OS16-2 Learning simulation based on a computational model of neuromodulators
Yuki Moriguchi, Masayasu Nagamatsu(Hiroshima University, Japan)

Reinforcement learning, and its biological bases are key to understand humanlearning. Here we illustrate learning processes of the tower of Hanoi based on thecomputational model of neuromodulators(Doya, K.2002). The model assumes
hypothetical roles for the following parameters of reinforcement learning.
1)Dopamine signals TD error, 2)Serotonin controls discount factor, 3)
Noradrenaline controls the inverse temperature, and 4) Acetylcholine controls
the learning rate. We have compared the two learning situations. Case a) the
reward is given at the goal state only, and Case b) the reward is divided and placedat the intermediate state and goal state. The resulting learning curves shows that b)
divided reward enhanced the learning. This result shows an example of learning
simulation with a variety of learners who have their own learning characteristics.

OS16-3 Estimation of Programming Learning Achievement by Line Tracing Robot
Yoshihiro Ohnishi1, Teruyuki Tamai1, Shinnosuke Mori1, Kawada Kazuo2（1Ehime University, 2Hiroshima University, Japan）
It is an important problem how to estimate the learning achievement of the
programming learning. However, the quantitative evaluation method according to
the learning achievement is difficult. This research considers estimation method forthe achievement value of the programming learning. The task is given as a courseof the line tracing robot. There are multiple courses to be passed. The difficult
program can go through a short route. In other words, it is possible to estimate thedifficulty level program created by the goal time.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 72 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS16-4 Development of Cultivate Computational Thinking using Finger Robot
Kaito Omata, Shinichi Imai (Tokyo Gakugei University, Japan)

Computational Thinking is one of the necessary skills for children who live inmodern times. Therefore, in this research, we aim to design lesson that cancultivate Computational Thinking. In this research, design a lesson that cultivates
Computational Thinking by moving a robot to the specified finger state using afinger robot. At that time the children can know that by using the notion of"binary" you can count many numbers by bending and extending your fingers. By
using a finger robot, it is thought that children will motivate themselves to teachand to understand easily.

OS17 Multiagent and Complex Systems (3)OS17-1 Different Type of Interaction or Decision Error contribute to Functional Differences
Saori Iwanaga1, Masao Kubo2( Japan Coast Guard Academy, 2National Defense Academy of Japan, Japan)
1

There are many situations where interacting agents can benefit from coordinating
their actions, that is, agents gain payoffs by taking the same action as others. Thesesituations can be modeled as coordination games. Another situations is modeled ascomplementary game, asymmetric coordination games or minority games. In
complementary game, agents gain payoffs by taking the different action fromothers. We deal with these coordination games and asymmetric coordination games
as type of interaction and each agent can change the interaction type. And eachagent can change the interaction partner. We show that population changes to amixed population comprised of 80% cooperative agents and 20% complementary
agents. A mixed population diversify the collective behavior, type of interaction andpreference. There, opposite type of interaction plays a role of small decision error,
and it is inefficient to adopt both.

OS17-2 Evaluation for the Synchronization of the Parade with OpenPose

Yohei Okugawa, Masao Kubo, Hiroshi Sato, Bui Duc Viet(National Defense Academy of Japan, Japan)

In National Defense Academy, parade training is one of our daily training, and isreviewed regularly. However, although the synchronization level of each cadet in
the parade is a major factor of the performance, we don’t have any objective
evaluation methods for the synchronization level. In this research, we propose an
objective method focusing on the synchronization level of the parade with
OpenPose. OpenPose is capable of measuring many kinds of postures from movies
simultaneously, so it is much easier to acquire data than conventional motion
analysis with motion capture, and it is suitable for parade analysis, which is acooperative operation of multiple people. In order to measure the synchronizationlevel, our method is based on the arm swing data of each cadet in the parade, andour results can contribute to objective evaluation methods for cooperative
operation, not only parade.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 73 -

Fig. 1. Analysing a Paradewith OpenPose.

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS17-3 Phase shift estimation of the bifurcating neuron from superimposed chaotic spikesequences
Akihiro Yamaguchi1, Yutaka Yamaguti1, Masao Kubo2( Fukuoka Institute of Technology, 2National Defense Academy of Japan, Japan)
1

For the bifurcating neuron, the phase shift value of background oscillation is one of
the typical parameters that determine the shape of dynamics (Fig.1 (a)-(c)). In thisresearch, we propose an estimation method of the phase shift value fromsuperimposed chaotic spike sequences generated by the bifurcating neuron. The
bifurcating neuron exhibits chaotic inter-spike interval dynamics that depends on
the phase shift value of the background oscillation. Therefore, if we correctly
estimate each phase shift values from the superimposed chaotic spike sequences,
we can characterize the dynamics of the bifurcating neuron and identify each spikesequences from the other superimposed spikes. Fig. 1 (d) shows an example ofestimated phase shift values for superimposed three chaotic spike sequences withdifferent phase shift values. In this case, three different phase shift values werecorrectly estimated and more than 90% of superimposed spikes were also correctly
identified.

Fig. 1. Estimation ofphase shift values.

OS18 Data-Driven Control and Diagnosis Control Systems (5)OS18-1 A Design Scheme of a Data-driven Predictive-PI Controller

Yoichiro Ashida, Shin Wakitani, and Toru Yamamoto (Hiroshima University, Japan)

PID controllers have been widely employed in real processes. However, forprocesses with long dead time, the control performance obtained by a PID
controller is limited. This is because prediction by a derivative element is not very
effective for the processes. For such processes, some predictive controllers havebeen proposed. Among them, a predictive PI(PIP) controller has only threeparameters. Although a PIP controller can be tuned by "trial and error", it isdesired to determine control parameters automatically. In this paper, a data-driven
design method for a PIP controller is proposed. In the proposed method, control
parameters of the PIP controller are calculated automatically from one set ofoperation data.

Fig. 1. Block diagram ofthe predictive PI
controller.

OS18-2 Improvement in Intersample Response of Multirate Regulation Control

Takao Sato, Natsuki Kawaguchi, Nozomu Araki, Yasuo Konishi (University of Hyogo, Japan)

This study discusses a multirate regulator system, in which the sampling interval ofa plant output is an integer multiple of the holding interval of a control input. A
multirate control system is optimized based on a quadratic performance function. In
order to improve intersample response, the derived control law is extended using
the null-space and/or steady-state constraint. Furthermore, the original performance
function is also modified, and the designed control systems are compared
quantitively.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 74 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS18-3 Design of a PID Controller using a Fictitious Exogenous Signal for a Fluctuation System
Masatoshi Kozui, Takuya Kinoshita, Toru Yamamoto (Hiroshima University, Japan)

A lot of industrial machines realize the desired motions by the PID controller. In
particular, it is possible to design PID parameters to achieve the desired behavior
by the Fictitious Reference Iterative Tuning (FRIT). However, in the steady state, if
characteristics of the target system are changed by the operating environment, thedesired control performance can not be obtained by the FRIT. In this paper, it isassumed that a disturbance is input when system characteristics are changed by thesystem switching. The Fictitious Exogenous Signal is calculated by the operational
data after the system change. A PID controller tuning by the Fictitious Exogenous
Signal is proposed. Moreover, the effect has been verified by numerical example.

Fig. Block diagram of
Fictitious Exogenous Signal.

OS18-4 Izhikevich Model Based Self-Repairing Control for Plants with Sensor Failures andDisturbances
Masanori Takahashi (Tokai University, Japan)

In the previous works, several types of the self-repairing control systems (SRCS)
have been developed against unknown sensor failures. The SRCS can automaticallydetect the failure, and replace the failed sensor with the healthy backup so as tomaintain the system stability. This paper presents a new SRCS, whose fault detector
is constructed based on a spiking neuron model proposed by E. Izhikevich. In thedetector of the SRCS, the sensor failure induces the spikes. Hence, only monitoring
the output of the detector makes it possible to find the sensor failure. Also, in thispaper, the robustness with respect to disturbances is theoretically analyzed, and it isshown that self-repairing control can be accomplished in the presence of unknown
disturbances.

OS18-5 Design of a Data-Driven Controller with Evaluating Controller Performance
Takuya Kinoshita, Toru Yamamoto (Hiroshima University, Japan)

Data-driven controllers have been proposed to achieve the desired control
performance without using any system identifications. The effectiveness of thesecontrol schemes has been shown through experimental results. For time-variant andnonlinear systems, it is important to evaluate the controller performance andredesign controller when the performance is poor. According to the proposed
scheme, the controller performance calculator and controller design are integratedusing only input and output data. Furthermore, the controller can be assessed in thetransient and steady state.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 75 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS19 Advanced
Advanced Technology
Technology on
on Sensing
Sensing Technology,
Technology, Devices,
Devices, Application
Application (3)(3)
OS19
OS19-1The
ActualCar
Driving
Evaluation
Systemusing
Combinedwith Eyeball
Eyeball andand Face
Face
OS19-1 The Actual Car Driving Evaluation System using Combined with
Angle
Angle
Keiko Sakurai,
Sakurai, Hiroki
Hiroki Tamura,
Tamura, Koichi
Koichi Tanno
Tanno (University(University ofof Miyazaki,
Miyazaki, Japan)
Japan)Keiko
Yukio
Hattori(OFA
SUPPORT
INC,
Japan)
Yukio Hattori (OFA SUPPORT INC, Japan)

The lacklack ofof automobile
automobile driving
driving standards
standards ofof persons
persons with
with brain
brain dysfunction
dysfunction andand thetheThe
increaseofcar
accidentsbytheelderlyareproblems.
Therefore,
researchongaze
increase of car accidents by the elderly are problems. Therefore, research on gazeestimation necessary
necessary forfor vehicle
vehicle driving
driving evaluation
evaluation isis rapidly
rapidly developing.
developing. In
In thisthisestimation
paper, wewe proposed
proposed thethe actual
actual carcar driving
driving evaluation
evaluation systemsystem which
which cancan bebe evaluatedevaluatedpaper,
by combining
combining eyeeye movements
movements using
using eyeeye tracking
tracking device
device (TalkEyeLite)(TalkEyeLite) andand facefacebymovement
estimatedusing
templatematching.
In
order
to
the
evaluateourproposal
movement estimated using template matching. In order to the evaluate our proposal
method, we
we carried
carried out
out thethe actual
actual carcar driving
driving experiments.
experiments. The
The subjects
subjects were
were oneone
method,instructor
of
the
car
drivingschool,andfivegeneraldrivers.
We
comparethe
gaze
instructor of the car driving school, and five general drivers. We compare the gaze
range ofof thethe five
five generalgeneral subjects
subjects andand thethe instructor.
instructor. As
As aa result,result, we
we confirmed
confirmed that
that
range
one
male
in
the
40s
and
one
elderly
narrowthegaze
range.
one male in the 40s and one elderly narrow the gaze range.

OS19-2 A
A Study
Study onon SpeakerSpeaker Identification
Identification Approach
Approach by
by Feature
Feature Matching
Matching Algorithm
Algorithm using
using Pitch
Pitch
OS19-2
andMel
Frequency
Cepstral
Coefficientsand Mel Frequency Cepstral Coefficients
Barlian Henryranu
Henryranu Prasetio,
Prasetio, Keiko
Keiko Sakurai,
Sakurai, Hiroki
Hiroki Tamura,
Tamura, Koichi
Koichi Tanno
Tanno
Barlian(UniversityofMiyazaki,Japan)(University of Miyazaki, Japan)

In thisthis paper,
paper, we
we grouping
grouping thethe words
words based
based onon thethe speakerspeaker inin aa sequence
sequence ofof speech
speechInin aa conversation.
conversation. There
There areare two
two speakers
speakers inin eacheach conversation.
conversation. The
The first
first speechspeechinassumed spoken
spoken by
by speaker-1.
speaker-1. In
In recognizing
recognizing thethe speakers,
speakers, we
we useuse pitch
pitch detection
detection
assumedand Mel
Mel Frequency
Frequency Cepstral
Cepstral Coefficients
Coefficients featurefeature extraction
extraction with
with 13
13 filters.
filters.
andFurthermore,weexaminethe
distanceofthesecondspeechvector
with
the
first
Furthermore, we examine the distance of the second speech vector with the firstFeaturematching
algorithm.
Previously,
wehadspeechvectorusing
thespeech vector using the Feature matching algorithm. Previously, we had
experimentedoneach
speaker
to
find
out
the
mean
and
varianceoftheFeatureexperimented on each speaker to find out the mean and variance of the Feature
matching. Based
Based onon thethe experimental
experimental results,
results, thethe variancevariance ofof Euclidean,
Euclidean,matching.
Mahalanobis
andMahattan
Distanceare0.0015,0.00064,0.0012.
Hence,
if thethe Mahalanobis and Mahattan Distance are 0.0015, 0.00064, 0.0012. Hence, if
Feature matching
matching value
value deviates
deviates is
is notnot moremore than
than variancevariance value
value then
then thethe speechspeech isisFeatureassume spoken
spoken by
by speaker-1.
speaker-1. Otherwise,
Otherwise, thethe speechspeech assume
assume spoken
spoken by
by speaker-2.
speaker-2.
assumeOS19-3 A
A Study
Study ononBreathing
Breathing andand Heartbeat
Heartbeat Monitoring
Monitoring System
System during
during Sleeping
Sleeping using
using
OS19-3
Multi-Piezoelectric Elements
Elements
Multi-Piezoelectric

Praveen Nuwantha
Nuwantha Gunaratne,
Gunaratne, Chika
Chika Yoshida,
Yoshida, Keiko
Keiko Sakurai,
Sakurai, Hiroki
Hiroki Tamura,
Tamura, Koichi
Koichi Tanno,
Tanno, (University(University ofof
PraveenMiyazaki,Japan)
Miyazaki, Japan)
We propose
propose aa method
method toto monitor
monitor breathingbreathing andand heartbeat
heartbeat during
during sleeping
sleeping using
using
We
piezoelectric elements.
elements. First,
First, thethe signal
signal acquired
acquired fromfrom thethe piezoelectric
piezoelectric elements
elements
piezoelectricduring sleeping
sleeping is
is separated
separated into
into aa breathingbreathing signal
signal andand aa heartbeat
heartbeat signal
signal using
using
duringbiomedical
signalprocessing
software.
A
methodisthenproposedtoquantify
biomedical signal processing software. A method is then proposed to quantify
breathing andand heartbeat
heartbeat by
by applying
applying fuzzy
fuzzy theory
theory toto thethe signals.
signals. This
This proposed
proposedbreathingprocedure.
In thethemethodisexperimentallydemonstrated
using
the
following
method is experimentally demonstrated using the following procedure. In
experiment,
twosubjects
slept
for
one
and
a
half
hours.
The
data
from
the
experiment, two subjects slept for one and a half hours. The data from thepiezoelectric elements
elements during
during sleep
sleep were
were extracted.
extracted. We
We compared
compared thethe true
true value
value with
withpiezoelectric
the count
count results
results fromfrom these
these data
data using
using thethe proposed
proposed method.
method. The
The error
error rate
rate ofof thethethebreathing count
count was
was 0–13%,
0–13%, andand thethe error
error rate
rate ofof thethe heartbeat
heartbeat count
count was
was 0–8.2%.
0–8.2%.
breathing
The results
results were
were non-uniform,
non-uniform, andand thethe error
error rate
rate depended
depended onon thethe subject.
subject.
The©ICAROB 2019 ALife Robotics Corp. Ltd.

- 76 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019OS20 Advances in Field Robotics and Sensing System (8)OS20-1 Field Robot and Sensing system

Eiji Hayashi (Kyushu Institute of Technology, Japan)

Labors in which forestry and agriculture and Factory, are working in severe
circumstance. So, such workers are necessarily declines and decreases. However,
it is important for us to keep going economy, life and nature. Therefore, to prevent adevastation of the forest, the safety and the lack of food and work in harshenvironments, we are thinking alternative technologies that a part of severe anddangerous works are taken robot instead of such workers．Something changes areneeded that they are seen innovation such as the generation of a new industry by
new technologies using AI and intelligent to improve how to recognize and express
more and more of the things that we have various desires.

OS20-2 Saliency Map Based on Maximization of Difference between Central and Surround
Visions and Its Relationship with Image Feature Points
Ruugo Mochizuki, Shinsuke Yasukawa, Kazuo Ishii (Kyushu Institute of Technology, Japan)

The Saliency map proposed by Itti is a human attention model considering thecolors, edge orientations and intensity and designed to have high values in the
curious area. In the model, the contrast of Saliency map is not so high and its
parameters are adjusted heuristically. The problems to distinguish the front object
from background is not suitable. In this research, a Saliency map is proposed tomaximize the difference of two images by changing Gaussian filter sizes for thecentral vision, i.e., the small size Gaussian filtered image, and the surround vision,i.e., the big size Gaussian filtered image. As one of image processing techniques,feature-points extraction methods such as SIFT, SURF are often introduced into
object detection and image mosaicking to generate the invariant features against
scale and rotation changes. However, these feature points extraction methods havethe effects of environments and lighting conditions. The relation between the
Saliency map and the feature points extraction is discussed.

OS20-3 Dynamic Wireless Network System using Multiple Drones for Tunnel Applications
Raji Alahmad, Kazuo Ishii (Kyushu Institute of Technology, Japan)

Tunnels requires inspection and maintenance as their age. Conventional methods
depend on special ground vehicles that have many limitations in mobility. UAVs, on
the other hand, can overcome these limitations. Communication is of the biggest
challenges for UAVs, due to signal attenuation. In this paper, a wireless network
system using multi-drones that act as repeaters is proposed. A number ofmeasurements for Wi-Fi signals is conducted to find out the effect of five different
scenarios on the signal in virtual tunnel and two scenarios in a real life tunnel. As aresult, the signal strength with -64 dBm can provide five Mbps of bandwidth, whichis recommended for video streaming. Also, placement of transmitter and receiver isessential for better connection.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 77 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS20-4 Counting Crops under Cultivation using Drone

Yasunori Takemura1, Yusuki Hirata2, Eiji Mizoe3, Masao Tashiro4, Yousuke Nagai5( Nishinippon Institute of Technology, 2Kyushu Institute of Technology, 3Sky Canvas Co. Ltd.,
4
Joint Corp. Project Sky-eye, 5Aruku Aguriculture Service LLC, Japan)
1

In recent years, agricultural workers in Japan are decreasing year by year. Also, rising
the average age of agricultural workers is a problem in Japan. As a solution to thisproblem, there is promotion of new agriculture (smart agriculture) that realizes
super-energy saving and high quality production by utilizing robot technology andIoT Technology. The authors are promoting technology by operating forest drone
competeton and tomato havevesting robot competition. Throu robot competition, weare currently paying attention to the use of drone in the agriculturel field. In thisresearch, we aim to automatic detect the number of agricultural crops by image
processing from image data of aerial photographs taken by drone. In this paper, we
will report a method to discriminate agricultural crops by color and to detect the
number.

OS20-5 Soil Compaction and Rolling Resistance Evaluation of a Locomotion System with
Adjustable Contact Patches for Applications in the Vineyard
Enrico di Maria, Kazuo Ishii (Kyushu Institute of Technology, Japan)

Soil compaction is a form of soil degradation which causes soil erosion, drainage
difficulties, hardpan production and loss of nutrients. In order to mitigate this issue,
different solutions have been proposed, such as: plough, low ground pressure tires,tracked tractors and controlled traffic farming (CTF). In this work the idea of using
a locomotion system with adjustable contact patch is proposed. Two configuration
modes are foreseen: wheel mode and track mode. The aim is to reduce thecompaction but also optimize the rolling resistance and energy consumption
depending on the payload of the robot and the soil conditions. The robot will be
used for transporting the grapes harvested in the vineyard. A finite element methodmodel is developed to evaluate the performance of the system in terms ofcompaction and rolling resistance and experiments in the field are carried out toconfirm the model itself.

OS20-6 Robot Navigation in Forest Management Based on Graph

Ayumu Tominaga1, Ryusuke Fujisawa1, Eiji Hayashi1, Abbe Mowshowitz2 (1Kyushu Institute of Technology,
Japan, 2The City College of New York, USA)This paper addresses the problem of using a mobile, autonomous robot to manage aforest whose trees are destined for eventual harvesting. We have constructed arobotic system enabling an autonomous robot to move between the trees without
damaging them and to cut the weeds as it traverses the forest. We proposes the
trajectory computation for guidance of the robot in the forest is carried out based ona weighted undirected graph G with trees as vertices. The line graph, L(G) of G will
represent the safety paths for guidance of the robot. It is shown that using trees in aforest as environmental landmarks is effective in navigation of forestry autonomous
robot.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 78 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS20-7OS20-7 Development
Development ofof Autonomous
Autonomous Moving
Moving System
System for
for Field
Field Robot
Robot

Kengo
Kengo Kawazoe,
Kawazoe, Eiji
Eiji Hayashi,
Hayashi, Ryusuke
Ryusuke Fujisawa
Fujisawa (Kyushu(Kyushu Institute
Institute ofof Technology,
Technology, Japan)
Japan)SLAM
SLAM technology
technology isis applied
applied toto resource
resource management
management inin forestry
forestry inin artificial
artificial forest.
forest.
From
thistechnology,
mappingoftrees
inartificial
forestandautomation
From this technology, mapping of trees in artificial forest and automation ofof growth
growthamountamount measurement
measurement ofof trees
trees have
have been
been realized.
realized. Meanwhile,
Meanwhile, forestry
forestry go
go intointo aadeclinedecline byby labor
labor shortage
shortage inin JapaneseJapanese forestry.
forestry. And,
And, itit hashas been
been expected
expected that
that
mechanization
mechanization andand automation
automation ofof work.
work. WeWe havehave been
been developing
developing autonomous
autonomous
movingmoving robot
robot with
with all-terrain
all-terrain vehicles
vehicles (ATV).
(ATV). And
And wewe designed
designed anan obstacle
obstacle
avoidance
avoidance behavior
behavior toto movemove aroundaround inin thethe forest.
forest. Moreover,Moreover, by
by addingadding aa pan
pan angle
angle
control
control actuator
actuator toto thethe visualvisual system,
system, we
we increased
increased thethe degree
degree ofof freedom
freedom ininexternal
external situation
situation recognition.
recognition. This
This mademade itit possible
possible thethe robot
robot toto thethe target
targetcoordinates
coordinates specified
specified onon thethe global
global map
map basedbased onon thethe obstacle
obstacle avoidance
avoidance behavior.
behavior.

OS20-8OS20-8 Development
Development ofof thethe sense
sense systemsystem thatthat isis combined
combined force
force feedback
feedback andand visionvisionfeedback-Improvement
reproducibilityofdeformation
simulation
byusing
LEMfeedback-Improvement reproducibility of deformation simulation by using LEMTamon
Tamon Shigeyama,
Shigeyama, Eiji
Eiji Hayashi
Hayashi ,, Ryusuke
Ryusuke Fujisawa
Fujisawa (Kyushu(Kyushu Institute
Institute ofof Technology,
Technology, Japan)
Japan)In
In thethe medical
medical treatment
treatment andand bio-technology
bio-technology field,
field, doctors
doctors andand researchers
researchers need
need
technologies
technologies thatthat can
can accurately
accurately performperform small-size
small-size work.
work. But
But there
there areare somesomeproblems.
problems. The
The operators
operators rely
rely only
only onon sight
sight throughthrough thethe microscope.
microscope. The
The operators
operatorscannot
feel
the
sense
of
force.
And
the
operatorscannotsee
theallcannot feel the sense of force. And the operators cannot see the all statesstates ofof thethesample.
Our
aim
is
to
developthe
combined
system
that
that
uses
both
sample. Our aim is to develop the combined system that that uses both force
force
feedback
feedback fromfrom thethe manipulator
manipulator andand visualvisual feedback
feedback on
on aa deformation
deformation simulation.
simulation.
By
reproducing
force
sense
and
visualsenseofasample,an
By reproducing force sense and visual sense of a sample, an operator
operator can
can feel
feel these
these
sense
sense ofof aa virtual
virtual object.
object. For
For this
this research,
research, focus
focus onon aa way
way toto produce
produce aa force
force
calculated
calculated inin simulation
simulation ofof deformation
deformation using
using aa haptic
haptic device.
device. As
As thethe first
first stage,stage, ouroursimulation
simulation ofof deformation
deformation was
was improved
improved byby using
using LEM
LEM (Long(Long Element
Element Method)
Method)andand evaluatedevaluated accuracy
accuracy ofof aa virtual
virtual object’s
object’s deformation.
deformation.

OS21
OS21 Advances
Advances inin Marine
Marine Robotics
Robotics andand It’s
It’s Applications
Applications (5)(5)
OS21-1OS21-1 System
System Development
Development ofof AUV’s
AUV’s Sampling
Sampling Device
Device Controller
Controller Employing
Employing
MATLAB/Simulink
Toolboxes
MATLAB/Simulink Toolboxes

1
2
2
2
2
TakashiTakashi Sonoda
Sonoda1,, Shinsuke
Shinsuke Yasukawa
Yasukawa2,, Ahn
Ahn Jonghyun
Jonghyun2,, Yuya
Yuya NishidaNishida2,, Kazuo
Kazuo Ishii
Ishii211
22( Nishinippon Institute of Technology, Japan, Kyushu Institute of Technology, Japan)( Nishinippon Institute of Technology, Japan, Kyushu Institute of Technology, Japan)We
We havehave developed
developed thethe sampling
sampling device
device control
control systemsystem forfor thethe AUVAUV “Tuna-Sand2”“Tuna-Sand2”
byusing
MATLAB/Simulink
andits
toolboxes.
The
toolboxesareby using MATLAB/Simulink and its toolboxes. The toolboxes are powerful,
powerful, useful,useful,andand functional
functional forfor developing
developing thethe robot
robot programming,
programming, especially
especially Robotics
Robotics System
SystemToolboxToolbox supports
supports thethe ROSROS soso thatthat thethe MATLAB
MATLAB andand ROSROS programs
programs areare available
available
forfor fast
fast developing
developing with
with highhigh functions
functions andand thethe cloudcloud data
data communication.
communication. Parallel
ParallelComputingComputing Toolbox
Toolbox can
can consist
consist aa multi-process
multi-process andand parallel
parallel computing
computing on
on script
scriptbase
base andand Simulink
Simulink model
model base
base program.
program. Stateflow
Stateflow is
is aa powerful
powerful tool
tool forfor design
design ofofstate
state transitiontransition control
control method.
method. The
The toolboxes
toolboxes areare employedemployed inin thethe sampling
sampling device
devicecontrol
control systemsystem ofof thethe AUV.
AUV. This
This method
method with
with using
using thethe toolboxes
toolboxes ofof
MATLAB/Simulink
realizetherapid
prototypingofthesystem
MATLAB/Simulink realize the rapid prototyping of the system designs,
designs,
developments
developments andand systemsystem test.
test. The
The developed
developed systemsystem mounted
mounted on
on an
an AUVAUV waswaspoweredpowered inin several
several real
real sea
sea areas.
areas.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 79 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS21-2 Inspection System for Underwater Structure of Bridge Pier

Takumi Ueda1, Hiyoyasu Hirai1, Kazuki Fuchigami1, Ryoma Yuki1, Ahn Jonghyun1, Shinsuke Yasukawa1, Yuya
Nishida1, Takashi Sonoda1, Kazuo Ishii1, Katsunori Higashi2, Katsunori Tanaka2, Tomomasa Ikeda2(1Kyushu Institute of Technology, 2West Nippon Expressway Engineering Kyushu, Japan)
Inspection of underwater infrastructure of bridge pier is important for bridge
maintenance, however, the underwater environment is dangerous and theirobservation is one of difficult problems. We have been developing the underwaterinspection system which consists of an unmanned surface vehicle (USV) and aremotely operated vehicle (ROV). The concepts of the system are (1) Operation is
executable with a small number of persons, (2) USV and ROV are controllable
from land or bridge, (3) Build 3D model and images for crack detection. The ROV
observes the status of whole infrastructure using cameras and control its postureusing a depth sensor and IMU sensor. The USV vertically lifts up and down ROV
by winch and remotely conduct ROV to infrastructure. USV is installed with GPS,LRF and IMU for horizontal positioning and direction control of the camera.
Inthis paper, we report the results of the inspection operation test by the robot andpresent issues.

OS21-3 Three-dimensional measurement using laser pattern and its application to underwater
scanner
Yuya Nishida, Tomoya Shinnoki, Shinsuke Yasukawa, Kazuo Ishii(Kyushu Institute of Technology, Japan)

There are many mineral resources on the seafloors energy resources such as
methane hydrate, and biological resources such as fish and crustaceans. The
accurate amount of the resources needs to be surveyed in detail for making
effectiveness use of the resources. In order to measure an accurate volume of theresources, the authors developed a new three-dimensional instrument consisting ofa laser projector and a camera. The laser projector irradiates a sharp
two-dimensional laser pattern independent the distance to the target object. Our
instrument is able to measure three-dimensional shape of the target object using thelaser pattern and the principle of light cutting method which is one of active
measurement methods.

OS21-4 Development of Dam Inspection Underwater Robot

Hiroyasu Hirai, Kazuo Ishii (Kyushu Institute of Technology, Japan)

Maintenances of social infrastructures such as dam, bridge are important subjects.
The more aging problem going on, the more the effective inspection methods havebeen expected. In the inspection of dams, divers observe the conditions of theirsurface manually, however underwater structure deeper than 30m is limited andtheir inspections are insufficient. In this paper, we propose the ROV whosefunctions are video capture, indirect measurement using laser, target depth keeping,heading keeping. The ROV is applied into the dam inspection and showed theeffectiveness to find cracks.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 80 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS21-5OS21-5 Conceptual
Conceptual Design
Design ofof Small
Small ROVROV forfor Sky
Sky toto Water
Water System
SystemKeisuke
Keisuke Watanabe,
Watanabe, Koshi
Koshi Utsunomiya,
Utsunomiya, Kazumasa
Kazumasa Harada,
Harada, Nakajima
Nakajima Shuhei
Shuhei(TokaiUniversity,
Japan)(Tokai University, Japan)

Long
Long term
term monitoring
monitoring ofof an
an areaarea ofof coralcoral reef
reef isis necessary
necessary forfor investigation
investigation ofof coralcoral
bleaching.
The
researchers
usuallyuse
underwater
camerastorecordthebleaching. The researchers usually use underwater cameras to record the reef’s
reef’spresent
present condition
condition using
using aa ROVROV oror diving
diving by
by themselves.
themselves. In
In thosethose cases,
cases, they
they need
need to
to
hire
a
boat
whose
port
is
not
alwaysnear
thetargetinvestigation
sitehire a boat whose port is not always near the target investigation site where
where
sometimes
sometimes near
near aa beach.
beach. The
The researcher
researcher diving
diving isis effective
effective fromfrom thethe viewpoint
viewpoint ofof
preciseprecise investigation
investigation ofof aa limited
limited spot,
spot, however,
however, it
it is
is impossible
impossible forfor aa diver
diver totoinvestigate
investigate wide
wide area
area simultaneously.
simultaneously. To
To breakthroughbreakthrough these
these limitations
limitations we
we areareconsidering
considering aa new
new method
method which
which uses
uses aa combined
combined systemsystem ofofmulticopter-USV(Unmanned
Surface
Vehicle)-ROV.
We
named
this
concept
multicopter-USV(Unmanned Surface Vehicle)-ROV. We named this concept as
as SkySky
toto Water
Water System(SWS).
System(SWS). In
In thisthis paper,
paper, we
we introduce
introduce thethe conceptual
conceptual design
design ofof our
our
Small
ROV
which
can
be
deliveredwith
its
motherUSV
byamulti-copter
from
Small ROV which can be delivered with its mother USV by a multi-copter from thethenearestnearest beach
beach andand explain
explain its
its hardware/software
hardware/software system.
system.

OS22
OS22 Robot
Robot Competitions
Competitions forfor Social
Social Contribution
Contribution (6)(6)
OS22-1Studyon
Position
Estimation
Using
Small
Size
OS22-1 Study on Position Estimation Using Small Size ZigBee
ZigBee Module
ModuleAtsushiAtsushi Sanada
Sanada (Nishinippon(Nishinippon Institute
Institute ofof Technology,
Technology, Japan)
Japan)The
The positional
positional information
information can
can bebe usedused forfor navigation
navigation inin controlling
controlling aa robot,
robot, andand it
it is
is
very
very important
important information.
information. In
In thisthis research,
research, we
we describe
describe thethe development
development ofof aa
position
position estimation
estimation systemsystem using
using aa ZigBee
ZigBee wirelesswireless module.
module. GPS
GPS isis often
often used
used for
for
position
detection,buthighprecision
measurementisdifficult.
And
also,
GPS
position detection, but high precision measurement is difficult. And also, GPS cannot
cannotbebe usedused indoors.
indoors. Furthermore,
Furthermore, forfor position
position detection
detection by
by thethe camera,camera, since
since it
it isisnecessary
toanalyzebycoordinatingapluralityofcameras,
much
cost
is
required.
necessary to analyze by coordinating a plurality of cameras, much cost is required. InInthisthis paper,
paper, wewe propose
propose aa position
position estimation
estimation systemsystem using
using aa low-cost
low-cost small
small wirelesswirelessmodule.
module. This
This isis toto estimate
estimate thethe position
position based
based onon thethe radio
radio field
field intensity
intensity with
withmultiple
multiple antennas.
antennas. Since
Since thethe radio
radio wave
wave intensity
intensity is
is nonlinear
nonlinear with
with distancedistance andand itit isisdifficulttocalculatethe
distance,aneuralnetworkisusedfor
distanceestimation.
difficult to calculate the distance, a neural network is used for distance estimation. AsAsaa result,result, it
it waswas confirmed
confirmed thatthat thethe distancedistance cancan bebe estimatedestimated correctly.
correctly.
thOS22-2OS22-2 The
The 44th Tomato
Tomato Harvesting
Harvesting Robot
Robot Competition
Competition

1
2
2
3 1
Takayuki
Takayuki Matsuo
Matsuo1,, Takashi
Takashi Sonoda
Sonoda2,, Yasunori
Yasunori Takemura
Takemura2,, Kazuo
Kazuo Ishii
Ishii3 ((1National
National Institute
Institute ofof Technology,
Technology,
22
33
Kitakyushu
College,
NishiNippon
InstituteofTechnology,
KyushuInstituteofTechnology,
Kitakyushu College, NishiNippon Institute of Technology, Kyushu Institute of Technology, Japan)
Japan)TomatoTomato isis oneone ofof important
important fruit
fruit vegetables
vegetables andand most
most tomatoes
tomatoes areare produced
produced inin thethegreenhouses,
greenhouses, oror large-scalelarge-scale farms,
farms, where
where thethe highhigh temperaturetemperature andand humidity,
humidity, andandlong
long harvest
harvest age
age force
force thethe farmer
farmer heavy
heavy works.
works. With
With anan aimaim toto promote
promote thetheautomationoftomatoharvesting,wehaveorganizedthetomatoharvesting
automation of tomato harvesting, we have organized the tomato harvesting robot
robot
competitioncompetition fromfrom 2014.
2014. The
The tomato
tomato harvesting
harvesting robot
robot competitioncompetition include
include twotwodivisionswhich
areseniordivisionandjuniordivision.
Inseniordivision,
divisions which are senior division and junior division. In senior division, free-stylefree-style
robotsrobots andand rail-stylerail-style robots
robots had
had matches.
matches. In
In juniorjunior division,
division, robot
robot competitioncompetition using
using
lego
lego mindstorm
mindstorm was
was hold.
hold. InIn thisthis paper,
paper, wewe report
report on
on thethe results
results ofof thethe 4th
4th tomato
tomatoharvesting
harvesting robot
robot competition.
competition.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 81 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS22-3 End-effector for robotic harvesting of a tomato fruit with calyx

Binghe Li, Shinsuke Yasukawa, Takuya Fujinaga, Kazuo Ishii(Kyushu Institute of Technology, Japan)

Tomato fruit harvesting is a labor-intensive process, and therefore robotic
automation is required. For tomato fruit harvesting robot, we developed three typesof end-effectors (EEs): scissors type, cylinder type and suction-cutting type. It isnecessary to harvest the tomato fruit with calyx especially in Japan. Therefore, theproposed EE has a cutter mechanism for cut the peduncle. The results show that thescissors type and the cylinder type EE can cut the peduncle; however, it is difficult
to separate one fruit from the cluster. On the other hand, the suction-cutting type canseparate and cut one fruit from the cluster and therefore it is useful for harvesting
tomato fruit with calyx.

OS22-4 Evaluation for Ball Dribbling Mechanism of RoboCup Middle Size League Robot in The
World Teams
Kenji Kimura, Shota Chikushi, Kazuo Ishii (Kyushu Institute of Technology)

RoboCup is a platform designed to promote such research fields as artificial
intelligence (AI) and robotics. In recent years, the Middle size league soccer robot
has a ball dribbling mechanism. This mechanism controls the ball with two drivingrollers and in order to play soccer games well, ball control capability and holding
capacity are indispensable. As the result of a survey at the 2017 World Cup, allteams in the world have determined the rollers arrangement heuristically, and nomathematical consideration has been made. In this research, we quantify theslippage between sphere and rollers in order to propose a mathematical modelthat allows for slippage and evaluate the optimality of the roller arrangementof the world teams.

OS22-5 Robot-control method based on personal space

Kota Kawamoto, Eiji Hayashi, Ryusuke Fujisawa (Kyushu Institute of Technology)

In order for a robot to interact with people, it is necessary for the robot to recognize
human emotions and express its own emotions by both verbal and nonverbal
communication. Furthermore, in our daily life, we unconsciously adjust ourpersonal space according to the intimacy with others. It is also said that personal
space is classified into non-verbal communication. In this research, we analyzed the
distance that a person can tolerate when the robot approaches to identify theminimum distance that the person is able to interact comfortably.

OS22-6 Recognition method of target objects for autonomous tomato harvesting robot
Keiji Kamei, Hiroyuki Moriyama (Nishinippon Institute of Technology, Japan)

Recently, the shortage of agricultural workers and depression of its production
efficiency have been said in Japan. To solve those problems, the researches that
apply the information and robot technology are studied actively. The aim of ourresearch, we focus on the autonomous tomato harvesting robot, and develop therecognition method for target objects. The color identification from a camera anddistance measurement between target objects and a robot are key points. To solve
those problems, we propose to identify the color based on HSV values andLook-Up-Table and measure the distance for target objects using Depth camera.
From the experiments on real field, we succeeded in acquiring the target object fromcamera and measuring the distance.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 82 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS23 Kansei and Information Engineering (4)OS23-1 Color Image Arrangement Based on Histogram Matching Using Smoothed Brightness
Histogram (I) --- Overall Smoothing ---Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai, Koji Kagawa, Yo Horikawa (Kagawa University, Japan), R. P.
C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)
Although we have already reported that automated color image arrangement by
Histogram Matching based on Gaussian Distribution (HMGD) gives good feeling
impression if the brightness histogram of the original image has a single peak.
However, if there are multiple peaks in the histogram, the HMGD processing does
not always bring good results. In this paper, we propose a Histogram Matching
method using overall smoothing over the brightness histogram of original image as
the reference one instead of using Gaussian distribution. Also in this paper, we
present the experimental results.

Left: original image
Right: processed image

OS 23-2 Color Image Arrangement Based on Histogram Matching Using Smoothed Brightness
Histogram (II) --- Piecewise Smoothing---

Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai, Koji Kagawa, Yo Horikawa (Kagawa University, Japan), R. P.
C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)
Although we have already reported that automated color image arrangement by
Histogram Matching based on Gaussian Distribution (HMGD) gives good feeling
impression if the brightness histogram of the original image has a single peak.
However, if there are multiple peaks in the histogram, the HMGD processing does
not always bring good results. In this paper, we propose a Histogram Matching
method using piecewise smoothing over the brightness histogram of original image
as the reference one, after Histogram Analysis by curvature computation. Also inthis paper, we present the experimental results.

2000
1800
1600

Frequency

1400
1200
1000
800
600
400
200
0
1

22 43 64 85 106 127 148 169 190 211 232 253
Lightness

OS23-3 Parameter Estimation Method for Compartment Model - PET Inspection Tetsuo Hattori, Yusuke Kawakami, Yoshiro Imai, Koji Kagawa, Yo Horikawa (Kagawa University, Japan),
Hiromichi Kawano (NTT Advanced Technology Company Ltd, Japan)
This paper proposes a new general method for parameters estimation incompartment model using DOC (Differentiation of Convolution). In this paper, weapply the DOC method to the parameters estimation in PET (Positron Emission
Tomography) inspection especially. Pet inspection is done for getting physiologicaland biochemical information of organs. The PET camera acquires the image ofradioactive distribution inside the human body. In this paper, we make amathematical analysis of the PET compartment model and describe the DOC
method.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 83 -Ci (t )K1

C p (t )k3
Ce (t )k2

Cm (t )

k4

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS23-4 Application
Application andand Trial
Trial Evaluation
Evaluation ofof Document
Document Writing
Writing Support
Support System
System toto Avoid
Avoid
OS23-4
Emotional
Misunderstanding
Emotional Misunderstanding

Shunsuke Doi,
Doi, Yoshiro
Yoshiro Imai,
Imai, Koji
Koji Kagawa,
Kagawa, Tetsuo
Tetsuo Hattori,
Hattori, Yusuke
Yusuke Kawakami
Kawakami (Kagawa(Kagawa University,
University, Japan)
Japan)
ShunsukeThis paper
paper describes
describes aa newly
newly developed
developed systemsystem ofof document
document designing,
designing, handling
handlingThisandself-checking
in
order
for
writers
to
avoid
their
readers'
unnecessary
emotional
and self-checking in order for writers to avoid their readers' unnecessary emotional
misunderstanding andand possible
possible repulsion.
repulsion. It
It is
is realized
realized as
as server-client
server-client computing
computingmisunderstanding
system.
The
servercanobtainuser
document
throughtheclient,scan
andseparatesystem. The server can obtain user document through the client, scan and separate
expressions,
check
each
word
while
the
regardingdocument
intoword-level
the regarding document into word-level expressions, check each word while
matching
it
with
our
sentiment
dictionary,calculatetherelevant
sentimental
valuesmatching it with our sentiment dictionary, calculate the relevant sentimental values
for thethe document
document andand thenthen generate
generate thethe corresponding
corresponding radar
radar chart
chart based
based ononforemotional
axis
such
as
delight,
anger,
sorrow,
pleasure
etc.
on
the
relevant
user'semotional axis such as delight, anger, sorrow, pleasure etc. on the relevant user'sWeb. Document
Document writers
writers utilize
utilize such
such aa systemsystem before
before transferringtransferring and/or
and/or archiving,archiving,
Web.
they
can
check
sentimental
valuesfortheirdocuments
andrecognize
how their
their
they can check sentimental values for their documents and recognize how
ones
have
a
lot
of
emotional
feelings.
With
thisfacility,
the
user
can
check
his/herones have a lot of emotional feelings. With this facility, the user can check his/her
document andand recognize
recognize whether
whether thethe document
document possibly
possibly generates
generates readers'readers'
document
unnecessary
emotional
misunderstanding
andmoreoverrepulsion
which
may
occur
unnecessary emotional misunderstanding and moreover repulsion which may occur
asaresult.
as a result.

OS24 AutomatedAutomated content
content generation
generation forfor narrative
narrative andand cognitive
cognitive contents
contents (5)(5)
OS24
OS24-1
A
Methodof
Haiku
Generation
Using
Deep
Learning
for
Advertising
Generation
OS24-1 A Method of Haiku Generation Using Deep Learning for Advertising Generation
Jumpei Ono
Ono (Vocational(Vocational School
School ofof Digital
Digital Arts
Arts Sendai,
Sendai, Japan)
Japan)
JumpeiTakashi Ogata
Ogata (Iwate(Iwate Prefectural
Prefectural University,
University, Japan)
Japan)
Takashi

We tried
tried various
various approaches
approaches toto generating
generating story.
story. One
One ofof themthem isis haiku
haiku generation
generation
Weusing deep
deep learning.
learning. The
The method
method wewe tried
tried has
has several
several tasks,
tasks, one
one ofof whichwhich isis that
that
using
only
the
word
time
series
data
is
used
as
learning
data.
Therefore,
theresultofonly the word time series data is used as learning data. Therefore, the result oflearning
can
be
a
model
similartogeneration
basedonsimplewordtransition
learning can be a model similar to generation based on simple word transition
probability. Therefore,
Therefore, we
we will
will organizeorganize thethe current
current tasks
tasks andand try
try toto study
study thetheprobability.
featuresusedforlearningandgeneratebylearning
results.
We
also
considerhow
features used for learning and generate by learning results. We also consider how
to useuse thethe method
method tried
tried inin thisthis paper
paper forfor advertisement
advertisement generation.
generation. In
In addition,
addition, we
wetoconsider how
how toto useuse thethe attemptattempt inin thisthis paper
paper forfor generating
generating different
different types
types ofof
considerstories.
stories.

OS24-2 The
The Usage
Usage Features
Features ofof Onomatopoeias
Onomatopoeias inin thethe Recipes
Recipes inin JapaneseJapanese
OS24-2
Hiroki Fukushima
Fukushima (Kyushu(Kyushu Womens’
Womens’ University,
University, Japan)
Japan)
HirokiIn thisthis study,
study, thethe author
author analyzes
analyzes thethe co-occurrenceco-occurrence tendency
tendency ofof onomatopoeias
onomatopoeias in
in
In
the
onlinerecipesinJapanese.
In
Japanese,
onomatopoeias
are
often
used
to
the online recipes in Japanese. In Japanese, onomatopoeias are often used todescribe faint
faint oror vaguevague senses.
senses. Especially,
Especially, onomatopoeias
onomatopoeias have
have considerable
considerable power
power
describe
to express
express thethe sense
sense ofof tastetaste oror sense
sense ofof odor.
odor. Focusing
Focusing co-occurrenceco-occurrence relationships,
relationships,
tothe author
author has
has revealed
revealed thethe roles
roles ofof onomatopoeias
onomatopoeias inin thethe tastingtasting description
description corpus
corpus
the
of
wine
and
Japanese
sake.
In
this
study,
applying
the“form-screening
method”,
theof wine and Japanese sake. In this study, applying the “form-screening method”, theauthorrevealesthesemantic
functionsofonomatopoeia
intherecipedata.
Theauthor reveales the semantic functions of onomatopoeia in the recipe data. The
protocol ofof form-screening
form-screening methodology
methodology is:
is: First,
First, extract
extract soundsound symbolic
symbolic words
words
protocolfromthecorpus.
Onthenextscreening
step,
all
the
words
extracted
in
the
first
step
from the corpus. On the next screening step, all the words extracted in the first stepischangedtoanscreening
word.
This
screening
word
must
be
the
same
word
class
is changed to an screening word. This screening word must be the same word classas extracted
extracted words.
words. And
And finally,
finally, analyze
analyze thethe co-occurrenceco-occurrence relationships
relationships ofof thetheasscreened words
words using
using aa coding
coding soft.
soft.
screened©ICAROB 2019 ALife Robotics Corp. Ltd.

- 84 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS24-3 Storytelling in the Conversation of Aged People

Yuki Hayashi (Chiba University / Center for Advanced Intelligence Project, Riken, Japan)
Akinori Abe (Chiba University /Dwango Artificial Intelligence Laboratory, Japan)

In the conversation of aged people, some people well spoke about theirexperiences, which was able to attract others. In this paper, we will analyze theirskills of storytelling from the viewpoints of the selection of theme, the constructionof their story and their vocabulary. In addition, it is also important whether they can
attract the other participants’ interests. So we will also analyze the transition of
topics during a conversation.

OS24-4 Expression of the taste of Japanese sake and metaphor

Akinori Abe (Chiba University / Dwango Artificial Intelligence Laboratory, Japan)

Seto explained “tasting expression” from the aspects of synesthetic expressions(both direction), which are metaphor, metonoymy, and synecdoche. When we
express the taste of Japanese sake, we sometimes use the expression “Refreshingscent such as the scent of muskmelon and pear is elegant...” It is rather easy to
express the taste by using the well known taste. However, how we controle the pathof the metaphor flow? For instance, we frequently use the expression “cheek likeapple.” Why we assume the colour is red? Similarly, for the taste of the sake, we
frequently use the expression “taste like apple.” Which apple we assume? In thispaper, we will discuss the flow of the metaphor from the aspect of discusion above..

OS24-5 An Analysis on Advertising Techniques of Beverages Using Positive Factors of
Evaluation Database System
Yoji Kawamura (Kindai University, Japan)

In the advertising business, the CM Research Institute has constructed an
exhaustive evaluation database system of commercial film representation withframework of positive factors. Several factors of positive factors are related tocommercial film techniques. By utilizing data on positive factors, it is possible toanalyze the outline of the commercial film techniques. In this research, based on
the viewing experiment on the communication effect (positive factors, interests,
willingness to buy) of 100 commercial films of beverages (can coffee, tea, beer),
the characteristics (by brand, by category, history transition, relation between
techniques and interest, relation between techniques and willingness to buy) ofcommercial film techniques are analyzed. This analysis method brings about a lotof knowledge and can be utilized for transient analysis and simultaneous analysis ofcommercial film techniques using the evaluation database system.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 85 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS25 High-Performance Computing for Interactive Content Creation and Rendering
OS25-1 High-Performance Computing for Visual Simulations and Rendering

Jasmine Wu, Chia-Chen Kuo, Shu-Hsin Liu, Chuan-Lin Lai, Chiang-Hsiang Lien, Ming-Jen Wang, Chih-Wei
Wang(National Center for High-performance Computing (NCHC), Taiwan)
Everyone who is involved in film or animation industry knows how rendering can
be a nightmare. National Center for High-performance Computing (NCHC) built a
render farm that provides a platform for the industry to render their work in a much
more efficient timeframe. Over 100 films in Taiwan was rendered using ourplatform. The newest Render Farm was launched in March of 2018, which allows
not only CPU rendering, but also GPU rendering. The throughput is also greatly
improved to support complex and large scale simulations. Our Render Farm iscurrently being used by animation students in Taiwan, as well as companies fromthe industry. The main goal of this session is to introduce scientific discoveries andtechnical innovation through prospective computing technology and platform.

OS25-2 Virtual Reality as an Art Form

R.P.C. Janaka Rajapakse, Yi-ping Hung (Tainan National University of the Arts, Taiwan)

After introducing content creations in the immersive environments, the VR content
creation process has been changed dramatically. It has enhanced the direct
engagements of the artists and reduces the time it takes to get a part from the idea
stage to the production stage. Time consuming VR content creation process has
been reduced to hours or days. Not only is the time reduced, but the cost is alsosignificantly reduced. The reduced time and cost allows artist to work in nearreal-time, with the design and production happening at nearly the same time. Thissection of the session is focused on VR content creation process and its artistic
perspectives. Especially, not only concerning VR contents creation in immersive
environment is an alternative to find new form of art. And the last part of the speech
concerns about the role of VR in the live performance and new form of theater.

OS25 -3 Using Quill as a Tool for Real-Time Rendering

Jasmine Wu, Chia-Chen Kuo, Shu-Hsin Liu, Chuan-Lin Lai, Chiang-Hsiang Lien, Ming-Jen Wang, Chih-Wei
Wang�(National Center for High-performance Computing (NCHC), Taiwan)
Unity and Unreal are game engines that renders in real-time, and the supercomputers at NCHC not only supports both, but it also presents a faster frame rate.
Rendering is indeed a time consuming procedure, the process of 3D animation
takes even longer with more men-power. Quill is a VR illustration and animation
tool for the Oculus Rift, which can shorten the traditional 3D animation process
done in 3D software, because its 3D models are painted rather than having to go
through the process of modeling, texturing, and rigging. This is a great tool tocreate content for Unity or Unreal, which can then be rendered in real-time by thesuper computers at NCHC. Another example to demonstrate NCHC’s real-timerender farm was by performance art. We have high data throughput that can help
the performer to do a large scale VFX simulation and to create new kinds ofcross-disciplinary performances. As we develop through time, more and more
applications can be seen in the near future.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 86 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019GS
GS abstracts
abstracts
GS1
GS1 Robotics
Robotics II (4)
(4)
GS1-1
GS1-1 System
System toto decidedecide unit’s
unit’s layout
layout ofof cellcell assembly
assembly machinemachine byby GA
GA (Big(Big Mutation
Mutation inin mutation
mutationprocess)
process)
Yoshiki
Yoshiki Ito,
Ito, Hidehiko
Hidehiko Yamamoto,
Yamamoto, Takayoshi
Takayoshi Yamada
Yamada (Gifu(Gifu University,
University, Japan)
Japan)The
The purpose
purpose ofof thisthis researchresearch isis toto develop
develop aa systemsystem toto decidedecide unit’s
unit’s layout
layoutwhich
automaticallydecides
to
place
a
lot
of
assembledparts,
jigs
which automatically decides to place a lot of assembled parts, jigs andand robot
robot
hands
hands ofof aa cell
cell type
type Assembly
Assembly Machine
Machine byby genetic
genetic algorithm
algorithm (GA).
(GA). TheThe
systemsystem consists
consists ofof twotwo modules,
modules, thethe conditions
conditions module
module andand GAGA module.
module. The
The
conditions
conditions module
module reads
reads work
work data
data forfor robot,
robot, andand sets
sets various
various parametersparameters
required
required forfor GA.GA
GA.GA module
module decides
decides thethe efficientefficient arrangement
arrangement place
place ofof parts,
parts,
jigs
jigs andand robot
robot hands
hands byby GA,
GA, andand outputs
outputs thethe acquired
acquired arrangement
arrangement images.
images.
This
This systemsystem proposes
proposes thethe concept
concept ofof Big
Big Mutation
Mutation thatthat changes
changes thethe mutation
mutation
rate
of
GA
every
certaingeneration.
By
introducing
Big
Mutationrate of GA every certain generation. By introducing Big Mutation ,, thethepossibility
possibility ofof searching
searching anan optimal
optimal unit
unit placement
placement increases
increases andand thethe systemsystemsimulation
simulation time
time isis shortened.
shortened. It
It is
is ascertained
ascertained thatthat thethe systemsystem toto decidedecide unit’s
unit’slayoutincludingBig
Mutationisuseful.
layout including Big Mutation is useful.

GS1-2
GS1-2 Autonomous
Autonomous decentralized
decentralized FMS
FMS thatthat adopts
adopts priority
priority ranking
ranking structure
structure based
based onon AGV’s
AGV’s
lies
lies
Kento
Kento Uejo,
Uejo, Hidehiko
Hidehiko Yamamoto,
Yamamoto, Takayoshi
Takayoshi Yamada
Yamada (Gifu(Gifu University,
University, Japan)
Japan)

We
We introduce
introduce thethe model
model ofof AGVAGV mindmind toto eliminate
eliminate path
path interferencesinterferences between
between
AGVs
AGVs when
when runningrunning an
an autonomous
autonomous decentralized
decentralized FMS
FMS andand carry
carry out
out
behavior
behavior controls.
controls. The
The conventional
conventional autonomous
autonomous decentralized
decentralized FMS
FMS believes
believesall
all information
information sent
sent byby agents
agents as
as correct.
correct. However,
However, thethe information
information sent
sent bybysomeagentsmaybeincorrectdue
tomalfunctions.
We
definethisincorrect
some agents may be incorrect due to malfunctions. We define this incorrect
information
information asas aa “lie”“lie” andand carry
carry out
out thethe AGVs
AGVs action
action controls
controls which
which doesn’t
doesn’tdecreasetheoutputsofautonomous
decentralized
FMS
even
if
decrease the outputs of autonomous decentralized FMS even if lies
lies occur.
occur. InIn
order
order toto do
do this,this, we
we propose
propose aa “priority“priority ranking
ranking structure”.
structure”. The
The proposed
proposedmethodmethod is
is ①to①to give
give aa ranking
ranking toto eacheach AGV
AGV according
according toto aa lie
lie number,number, ②to
②to
give
high
ranking
AGVs
to
share
jobs.
In
order
to
prove
the
give high ranking AGVs to share jobs. In order to prove the effectiveness,
effectiveness,
production
production simulations
simulations using
using priority
priority ranking
ranking structure
structure andand thethe conventional
conventional
research
research method
method were
were carried
carried out.
out. As
As aa result,result, thethe productivity
productivity ofof propose
proposemethodmethod was
was higher
higher than
than thatthat ofof thethe conventional
conventional researchresearch method.
method. It
It isis
ascertained
ascertained thatthat thethe priority
priority structure
structure isis usefulusefulGS1-3
GS1-3 CULET:
CULET: Cubelet
Cubelet Lego
Lego Turing
Turing Machine
Machine..

RicardoRicardo Q.
Q. Figueroa,
Figueroa, Daniel
Daniel A.
A. Zamorano,
Zamorano, Genaro
Genaro J.
J. Martínez
Martínez (Instituto(Instituto Politécnico
Politécnico Nacional,
Nacional, México)
México)AndrewAdamatzky(University
of
the
West
of
England,UK)
Andrew Adamatzky (University of the West of England, UK)In
In thisthis paper,
paper, wewe show
show thethe construction
construction ofof CULET:
CULET: aa robot
robot thatthat simulates
simulates thetheexecutionofany
2-symbols
Turingmachine.
CULET
isconstructed
execution of any 2-symbols Turing machine. CULET is constructed withwith
Cubelets
Cubelets (small(small autonomous
autonomous robot-cubes
robot-cubes used
used forfor teachingteaching basic
basic robotics
robotics andandprogramming
programming forfor kids)kids) andand Lego
Lego bricks.
bricks. Cubelets
Cubelets areare aa kind
kind ofof highhigh level
level
programming
programming where
where you
you dodo notnot useuse traditional
traditional code
code but
but rather
rather achieves
achieves
different
robotstroughtheconcatenationofblocks,i.e.
robotsdifferent robots trough the concatenation of blocks, i.e. robots assembled.
assembled.
CULET
CULET isis designed
designed toto performance
performance thethe readread andand write
write ofof aa Turing
Turing machine.
machine.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 87 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

GS2 Robotics II (5)GS2-1 Automated Guided Vehicle System Analysis in Foam Manufacturing Plant Using Petri-net
Amornphun Phunopas1, Wisanu Jitviriya1, Noppadol Pudchuen1, Sutee Kumjaikong1, Songklod Tunsiri2,
Eiji Hayashi3 (1King Mongkut’ s University of Technology North Bangkok, Thailand,2Urban Community
Development College, Thailand, 3Kyushu Institute of Technology, Japan)
The logistic robot is very popular nowadays. The robots run in autonomous,
reactive, and parallel situations. These kinds of work are defined as astochastic problem. We focus on task distribution, routing, scheduling, andqueueing. This paper presents a multi AGVs operational analysis in the foam
manufacturing plant using the Petri-net. Each robot is designed arole-assignment supervisor with the role of a robot fixed in a zone. According
to the work capacity, the supervisor assigns the role of high capacity work toanother robot to help in a new zone. To monitor the work situation, thesupervisor continually receives feedback information of the system. The
analysis is necessary for preventing any failure that can make the central
software go down. The results show system management for robotic assistantsin the foam manufacturing process.

Zone0

Zone1

*Special0

Zone5

Zone2
Zone3

Zone4

GS2-2 Preliminary Comparative Experiments of Support Vector Machine and Neural Network
for EEG-based BCI Mobile Robot Control
Yasushi Bandou, Takuya Hayakawa, Jun Kobayashi (Kyushu Institute of Technology, Japan)

The authors have been developing an EEG-based BCI for mobile robot
control, using neural networks that classifies their brain waves to recognize
intention of a mobile robot operator. In order to achieve practical classification
capability with them, we had applied some techniques, such as Bayesian
Optimization for hyperparameters, to the neural network training; however,satisfactory performance of the neural networks has not been obtained yet.
Support Vector Machine (SVM) is another machine learning method that
demonstrated its usefulness in lots of other studies. In this study, for further
development of practical EEG-based BCI for mobile robot control, we haveexperimentally employed SVM for brain wave classification and compared
the performance with the neural networks. This paper presents thecomparative results.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 88 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

GS2-3 Position Tracking Error Constrained Super-Twisting Dynamic Surface Control with
Disturbance Observer for Robot Manipulators
Seong Ik Han (Dongguk University, Korea Republic)

A tracking error constrained dynamic surface control (DSC) with asuper-twisting algorithm (STA) and a nonlinear disturbance observer is
proposed for robot manipulator systems, where the system dynamics are
partially known. The stabilizing controls of the DSC were designed by
combining the STA with the virtual tracking errors defined from the DSC
design concept. A nonlinear disturbance observer was designed to estimate
the unknown nonlinear function and external disturbance instead of usingparameterization of the nonlinear function or adaptive law in theconventional STA and DSC schemes. Moreover, a simpler tracking error
constraint method is also proposed. The proposed tracking error constrained
STA-DSC scheme combined with a nonlinear disturbance observer has
better tracking error performance and robustness to uncertainty of robot
manipulator system than control systems with a conventional DSC anddisturbance observer. The control scheme was evaluated throughsimulations with the articulated manipulator system.

GS2-4 Image Processing for Picking Task of Random Ordered PET Drinking Bottles
Chen Zhu, Takafumi Matsumaru (Waseda University, Japan)

In this research, six brands of soft drinks are decided to be picked up by a 6
degree of freedom robot with a monocular RGB camera. The drinking bottles
need to be located and classified with brands before being picked up. The
Mask R-CNN, a mask generation network improved from Faster R-CNN, istrained with COCO datasets to detect and generate the mask on the bottles in
the image. The Inception v3 is selected for the brand classification task.
Around 200 hundred images are taken or found at first; then, the images areaugmented to 1500 images per brands by using random cropping andperspective transform. The result shows that the masked image can be labeledwith its brand name with at least 85% accuracy in the experiment.

GS2-5 The Use of Importance Ranks to Derive Suitable Timing of Visual Sensing
in Manipulation Task Containing Error Recovery

Akira Nakamura1, Kazuyuki Nagata1, Kensuke Harada2, Yukiyasu Domae1( National Institute of Advanced Industrial Science and Technology (AIST), 2 Osaka University, Japan)
1

In general, a manipulation task can be composed of many skill primitives. It is
desirable to perform plural visual sensing in most skill primitives as geometricmodeling and task planning are performed before execution, and task
achievement is confirmed after execution. However, the performance ofsensing at all sensing points is difficult because of the restrictions of time andhardware. In this paper, we proposed the addition of importance ranks to theattribute of skill primitives to derive suitable timing to perform visual sensing.
The use of the proposed method eases the selection of skill primitives inwhich visual sensing should be performed. Furthermore, we showed that skill
primitives distinguished according to their high importance ranks considerably
correlate with those in which error recovery is considered in advance. This
characteristic can be used for selecting skill primitives in which error recovery
is planned in advance.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 89 -

Move-to-destination

Falling

Plan

Practice

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

GS2-6-1 Electronic Measurement and Gamification of Balance Tests

Yan-Xin Liu, Massimiliano Leggieri, Henrik Hautop Lund (Technical University of Denmark, Denmark)We gamified three common balance tests for fall risk diagnosis, Timed Up &Go (TUG), Chair Stand (CS), and Four Square Step Test (FSST), by using theMoto Tiles. The embedded pressure sensor of the Moto Tile was used to detect
the movement of subjects and to time the tests. The proposed Motomeasurement was compared with traditional stopwatch measurement. A totalof 51 samples were analyzed. The intraclass correlation coefficients of the
TUG and CS tests were very high (0.98 and 0.94). The FSST was analyzed by
Pearson’s correlation coefficient and the result also indicated strong
correlation (0.79). Based on the results, we concluded that the proposed Moto
tests are as reliable as traditional tests.

GS2-6-2 Playful Body and Brain Test with the Moto Tiles

Yan-Xin Liu, Massimiliano Leggieri, Henrik Hautop Lund (Technical University of Denmark, Denmark)We gamified three common balance tests for fall risk diagnosis, Timed Up &Go (TUG), Chair Stand (CS), and Four Square Step Test (FSST), by using theMoto Tiles. The embedded pressure sensor of the Moto Tile was used to detect
the movement of subjects and to time the tests. The proposed Motomeasurement was compared with traditional stopwatch measurement. A totalof 51 samples were analyzed. The intraclass correlation coefficients of the
TUG and CS tests were very high (0.98 and 0.94). The FSST was analyzed by
Pearson’s correlation coefficient and the result also indicated strong
correlation (0.79). Based on the results, we concluded that the proposed Moto
tests are as reliable as traditional tests. Further, we propose a novel playful
body and brain test. The test includes a series of standardized Moto Tiles
games, which cover both physical and cognitive abilities. Reference scores of
the tests were modeled by data collected from a large number of participantsof different ages. The reference scores can be applied to evaluate a user’s bodyand brain abilities in comparison with the average performance at the sameage.

GS3 Complexity (2)GS3-1 Extracting Co-occurrence Feature of Words for Mail filtering

Seiya. Temma, Manabu. Sugii, Hiroshi. Matsuno (Yamaguchi University, Japan)

Spam mail filters often take advantage of appearance frequency of words in atext for mail classification. However the appearance frequency is one of themost important attribute information with which the mail can be characterized,
not a few spam mails can not be distinguished with only the appearance
frequency of words. In order to search new attribute information tocharacterize and classify the mails, we analyzed relationship between words ina text of mails by text data mining. Also we visualized the word network by
the co-occurrence and multi-dimensional scaling analysis with the jaccard
coefficient in real mails. Co-occurrence network analysis showed important
word connections with noun and verb in the same kinds of mails.
Multi-dimensional scale analysis showed some word clusters extracted fromthe same kind of mails. We’ll plan to explore how to utilize the result of thisstudy for mail classification.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 90 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

GS3-2
GS3-2 Learning
Learning Style
Style Classification
Classification with
with Weighted
Weighted Distance
Distance Grey
Grey Wolf
Wolf Optimization
Optimization

DuangjaiDuangjai Jitkongchuen,
Jitkongchuen, Piyalak
Piyalak Pongtawevirat
Pongtawevirat (Dhurakij(Dhurakij Pundit
Pundit University,
University, Thailand)
Thailand)The
The multiclassmulticlass classification
classification seems
seems toto be
be difficult
difficult toto improve
improve thethe performance
performanceofof thethe built
built classification
classification model.
model. This
This researchresearch aims
aims toto improve
improve thetheperformance
performance ofof multiclassmulticlass classification
classification by
by using
using grey
grey wolf
wolf optimizer
optimizer (GWO)(GWO)
algorithm.
algorithm. But
But inin thethe original
original GWO,
GWO, thethe evolutionary
evolutionary information
information ofof
population
population has
has not
not been
been fully
fully utilized,
utilized, andand itit isis easy
easy toto fallfall into
into local
local optimum.
optimum.
The
The proposed
proposed algorithm
algorithm presents
presents aa solution
solution toto improve
improve thethe grey
grey wolf
wolf
optimizer
performanceusing
weighted
distanceandimmigration
optimizer performance using weighted distance and immigration operation.
operation.
The
The weightweight distancedistance isis usedused forfor thethe omegaomega wolves
wolves movement
movement is
is defined
defined fromfrom
fitness
value
of
each
leader(alpha,beta
anddelta).
The
proposedfitness value of each leader (alpha, beta and delta). The proposed technique
technique isisbased
based onon learninglearning style
style prediction
prediction which
which addresses
addresses multiclassmulticlass classification
classification
problem.
problem. The
The results
results showed
showed thatthat thethe proposed
proposed technique
technique obtained
obtained thethe higher
higheraccuracy
accuracy rate
rate than
than other
other classification
classification techniques.
techniques. This
This was
was implemented
implemented totoapplication
application ofof learninglearning style
style prediction
prediction which
which cancan provide
provide thethe course
course materials
materials
suitablesuitable forfor learners
learners regarding
regarding their
their learninglearning style.
style.

GS4
GS4 Poster
Poster (5)
(5)
GS4-1
GS4-1 Relevance
Relevance Research
Research among
among Destination
Destination Image,
Image, Brand
Brand Trust,
Trust, andand Satisfaction:
Satisfaction: Tainan
TainanFestivalin
Taiwanas
ExampleFestival in Taiwan as Example
Shang-Hui
Shang-Hui Li,
Li, Shu-Fang
Shu-Fang Hsu,
Hsu, Yi-Tai
Yi-Tai Shang
Shang (Far
(Far East
East University,
University, Taiwan)
Taiwan)In
In recentrecent years,
years, all
all local
local governments
governments ofof Taiwan
Taiwan have
have held
held sightseeing
sightseeing
festivals
toenhancetheirpopularity,
promotetheireconomy,festivals to enhance their popularity, promote their economy, toto establish
establish their
their
brand.
brand. Tainan,
Tainan, thethe second
second biggest
biggest city
city inin southernsouthern Taiwan,
Taiwan, also
also promotes
promotes thethevariousvarious agricultural
agricultural products
products andand sightseeing
sightseeing toto serve
serve customers
customers as
as thethe region's
region's
annualannual event.
event. This
This study
study took
took Tainan
Tainan Festival
Festival asas anan example
example toto explore
explore thetherelevanceofthedestination
image,
brandtrust,
andsatisfaction.
relevance of the destination image, brand trust, and satisfaction. The
The
participants
participants forfor those
those who
who took
took part
part inin 20172017 Tainan
Tainan Festivals,
Festivals, fromfrom May
May to
to
July,
were
the
main
subjectsonthisstudy.
500
questionnaires
wereissued,July, were the main subjects on this study. 500 questionnaires were issued, 465
465
valid
valid questionnaires
questionnaires collected,
collected, 93%.
93%. TheThe finding
finding results
results showed
showed thatthat thethedestination
imageofTainanfestival
activities
hadapositively
effects
destination image of Tainan festival activities had a positively effects ononsatisfaction,satisfaction, brand
brand trust
trust had
had aa positive
positive effects
effects on
on satisfaction,satisfaction, destination
destination
image
image had
had aa positive
positive effects
effects on
on brandbrand trust,
trust, brand
brand trust
trust had
had thethe intermediaryintermediaryeffecteffect toto thethe satisfaction
satisfaction andand destination
destination image.
image.

GS4-2
GS4-2 Third-party
Third-party Logistics
Logistics Service
Service Supply:Supply: Catering
Catering Service
Service Trade
Trade asas Example
ExampleShu-Fang
Shu-Fang Hsu,
Hsu, Shang-Hui
Shang-Hui Li
Li (Far(Far East
East University,
University, Taiwan)
Taiwan)With
With thethe development
development ofof third-party
third-party logistics,
logistics, refrigeration
refrigeration technology,
technology, andandtransportation
which
made
the
food
to
obtaineasily,andimprovetransportation which made the food to obtain easily, and improve thethe quality
quality ofoffood
food preservation.
preservation. Now,
Now, thethe catering
catering chefs
chefs just
just ordered
ordered his
his neededneeded foods
foods bybyline,
line, telephone,telephone, fax,
fax, oror fillfill thethe orderorder list
list etc.,
etc., toto inform
inform thethe third-party
third-party logistics
logistics
suppliers
suppliers andand there
there will
will bebe someone
someone assigned
assigned toto thethe designateddesignated location
location which
which
could
could save
save purchasingpurchasing time
time andand catering
catering chefs
chefs don’t
don’t need
need toto rushrush aroundaround totoprocureprocure foods.
foods. This
This study
study analyzed
analyzed thethe collectedcollected information
information throughthroughliterature,
literature, in-depth
in-depth interviews,
interviews, andand observation
observation methods.
methods. Third-party
Third-party logistics
logistics
suppliers
conducted
theinformation,
logistics,
and
cash
flow
suppliers conducted the information, logistics, and cash flow by
by thethe different
different
deal
deal duedue toto thethe different
different nature
nature ofof suppliers.
suppliers. Third-party
Third-party logistics
logistics suppliers
suppliers
couldbeexhaustiveon
thequality
management,
logistics
activities,
could be exhaustive on the quality management, logistics activities, interactive
interactivemode
mode ofof trust
trust mechanism,mechanism, which
which could
could notnot only
only win
win thethe good
good reputation
reputation by
by thethecatering
chefs,
buttheprofessional
core
competence.
catering chefs, but the professional core competence.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 91 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

GS4-3 Pedal Scale Control Device for Weighing Food Ingredients or Chemical Materials
Shang-Hui Li, Shu-Fang Hsu, Yi-Tai Shang, Kuan-Ying Chen (Far East University, Taiwan)

Safety and hygiene is one of the most important elements when weighing
ingredients or chemical materials. It may cause cross-contamination when
preparing them if the operators conduct the scale button and hold the foodingredients or chemical materials by using both hands. The invention relates toa pedal scale control device comprising a scale body and a foot unit. The
wireless signal on the foot unit can be connected to the scale body to zero andbuckle the function without using of both hands. This invention has obtained
Taiwan's invention patent, which is more hygienic and safety for the users
when weighing the food ingredients or chemical materials.

GS4-4 Convolution Neural Network Based Fault Diagnosis of Induction Motors
Jong-Hyun Lee, In-Soo Lee (Kyungpook National University, Korea)

Induction motors are one of the most important components for machine andindustrial equipment in modern industrial applications. Therefore, it isnecessary to develop a fault diagnosis system that detects the operating
condition and failure of the induction motor early. This paper presents a
induction motor fault diagnosis system using CNN(Convolution Neural
Network) model. In the proposed method, the fault diagnosis of the induction
motor is performed by using vibration signal data obtained from theinduction motor experiment as the input value of convolution neural networkmodel. And then fault diagnosis was made using convolution neural network.
In this paper, fault diagnosis of steady state, rotor and bearing failure ofinduction motor is performed. From the experimental results, it is confirmedthat the proposed method is suitable for diagnosis of rotor and bearing failureof induction motor.

Fig. 1. Experimental environment
for fault diagnosis ofinduction motors

GS4-5 Analysis of Value Chain on Food and Beverage Micro-enterprises: Mobile Diner as
ExampleShu-Fang Hsu, Shang-Hui Li (Far East University, Taiwan)

This study was based on the theory of value chain proposed by Porter, 1985, in
conjunction with the mobile diner industry from the value activities created in
the value chain. The main activity was to analyze the competitive advantage ofthe mobile diner and the added value to meet customers’ need. In this study,
the Delphi method was used, and then conducted with interview andobservation method on 10 homogeneous mobile diners to gain an in-depth
understanding of the development and current situation in southern Taiwan.
Exploring their competitiveness, and discovering that the businessmen
purchased the necessary food by themselves, which could ensure safety. In
terms of marketing and sales, due to the mobility and easy acceptance ofcustomization, the relative value of products and services are relatively
improved. This study could provide a practical reference formicro-entrepreneurs who want to invest in food and beverage in the future.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 92 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

GS5 Others (8)GS5-1 Suppression of Roll Oscillation in Turning of Quadruped Robot by Asymmetric
Amplification of Central Pattern Generator Output Waveform
Makoto Kitani, Ryo Asami, Noritake Sato (Nagoya Institute of Technology, Japan),
Tomofumi Fujiwara, Takahiro Endo, Fumitoshi Matsuno (Kyoto University, Japan),
Yoshifumi Morita (Nagoya Institute of Technology, Japan)

One of control methods for walking of a robot is a central pattern generator(CPG). Using the CPG for a legged robot, it can walk on rough terrain.
Kimura et al. developed a quadruped robot “Tekken”, which is controlled by
the CPG. Tekken can turn by controlling the hip yaw joint. However, when the
robot turns by controlling the hip yaw joint, there is a problem that thevibration in the roll direction is large. To solve this problem, we proposed amethod of asymmetric amplification of CPG output waveform. We
implemented the proposed method on the robot inside the simulator andverified the effectiveness at turning. As a result, we achieved a 43.7%
decrease in the roll angle of the robot body than the conventional method. It is
assumed that legged robots can turn stably using our proposed method.

GS5-2 Tumble avoidance system for rescue robot by estimating the contact points using a 3D
depth sensor
Noritaka Sato, Kotaro Ohshima, Yoshifumi Morita (Nagoya Institute of Technology, Japan)

Rescue robots are expected to perform works in hazardous areas. However,when the robot runs on a rough terrain, fatal rollover falling sometimes
occurs. Therefore, we propose a tumble-risk-assessment system, which
reconstructs the environment, estimates the contact points between the groundand the robot, and calculates a normalized energy stability margin (SNE) by
using a 3D depth sensor. Moreover, a control system to avoid the tumble
situation by moving the sub-crawlers automatically is proposed. In the control
system, the SNE is partially differentiated by the angles of the sub-crawlers
and the sub-crawlers move to increase the SNE. We implemented the
proposed systems to a real robot, carried out experiments, and confirmed theeffectiveness of the proposed method.

GS5-3 Rule based Intrusion Detection System by Using Statistical Flow Analysis Technique forSoftware Defined Network
Mahnoor Ejaz, Osama Sohail, Talha Naqash, Zain ul Abideen, Sajjad Hussain Shah
(Bahria University, Pakistan)

Network Security is a vast field making progress around the globe very fast.
In every progressing year, developers have implemented different tools, which
include Intrusion detection systems. Nowadays Intrusion Detection System(IDS) is one of the popular tools, which are drawing the attention of many
researchers. Applying it in Software Defined Networking (SDN) facilitates
network management and enables to enhance the productivity of network
monitoring. Classifying packets based on their statistics and separating theforward process of network packets from the routing process is the main
challenge. In this paper, rule-based classification is done in order todifferentiate between viruses and normal packets. Statistical analysis ofdifferent network traffic flows are done through which segregation is madeand intrusion is detected in Software Defined Networking. The proposedsystem is experimentally tested on UNB ISCX data,sets©ICAROB 2019 ALife Robotics Corp. Ltd.

- 93 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

GS5-4 Development of Testbed for SDN to Know Its Feasibility of Deployment on Access Layer
Rafay Shah, M Osama Shaikh, Talha Shaikh, Tanveer Hussain(Mehran University of Engineering & Technology, Pakistan)

Today, the network requirement of organizations includes enhancing
performance, realizing broader connectivity and security regulations. In order
to comply with all of these criteria, networking protocols have been evolving
significantly over the last few decades. Internet of Things (IoT) has shifted ITfrom server entity to the network entity. Software Defined Networking (SDN)
is a solution of these all problems found in traditional networking. SDN
separates control plane from data plane, which makes it easier for an IT
Administrator to easily deploy and update regulatory policies. In this project,
we have developed a testbed kit using Zodiac-FX Open- Flow Switch for
SDN and have tested SDN feasibility on Access Layer. We have also
simulated a network of Faculty of Electrical Electronics Computer
Engineering (FEECE) of Mehran University of Engineering and Technology(MUET) and have found that SDN performs better than traditional networkarchitecture of FEECE of MUET.

GS5-5 Spectrum Sensing using Unsupervised Learning for Cognitive Radio

Asmara Shaukat, Danish Khan, Talal Arshad, Haseeb Ahmad (PAF-KIET, Pakistan)Cognitive Radio is the type of wireless communication which has the ability
to learn from its surrounding and reconfigure its operating parameters. It uses
radio spectrum efficiently by detecting which communication channels are in
use and which are not, and instantly move into vacant channels while avoiding
occupied ones. This optimizes the use of available radio-frequency (RF)spectrum while minimizing interference to other users. Spectrum sensing is
one of the most important tasks performed by the CR. Spectrum sensing refers
to the ability of a CR to measure the spectrum activities due to ongoingtransmissions over different spectrum bands and to capture the relatedparameters, it tells us about spectrum usage and existence of primary users ina geographical area. In this work we perform Spectrum sensing using
unsupervised learning algorithm (K- mean clustering) and find vacant
spectrum.

GS5-6 Tracking Secondary User In Cognitive Radio For 5g Communication

Muhammad Adnan, Muhammad Umair, Sameed bin Junaid (PAF-KIET, Pakistan）

Cognitive Radio (CR) is a promising technology which resolves the spectrum
scarcity issues and allows secondary user (SU) to use the vacant spectrum
space. Many secondary users are allocated different spectrum bands toenhance spectrum utilization. SU co-exists with the PU in the CR network andcontinuously monitors PU activities in order to use vacant space of PU in the
spectrum. The movement of SU in the CR network is random and therefore itis necessary to track the position of SU and measure radio signal strength
indicator (RSSI) in order to avoid interference with the PU. In this work, we
present the implementation and comparison of two algorithms KALMAN
Filter and State Space Least Mean Square (SSLMS) to track the position of
SU in the CR network under log distance and log normal shadowing path loss
models. Both algorithms track the position of SU by measuring RSSI and
Signal to Noise Ratio (SNR) from the PU; however SSLMS gives better
performance as compared to KALMAN Filter.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 94 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

GS5-7
GS5-7 WirelessWireless PowerPower Transfer
Transfer andand Data
Data Communication
Communication For
For Biomedical
Biomedical Applications
Applications
HammadHammad Saleem,
Saleem, Muhammad
Muhammad Awais,
Awais, Siraj
Siraj Din,
Din, Attequa
Attequa (PAF-KIET,(PAF-KIET, Pakistan)
Pakistan)A
A technique
technique ofof wirelesswireless power
power transfer
transfer (WPT)(WPT) systemsystem forfor biomedical
biomedical
applications
withbackscatteringcommunication,arectifieruse
applications with backscattering communication, a rectifier use with
with active
activebias
bias mechanism
mechanism (dependent(dependent andand independent
independent stages)
stages) overcome
overcome thethe diodediodedevicedevice losses
losses has
has been
been presented
presented inin thisthis paper.
paper. Apart
Apart fromfrom conventional
conventional andandstaticstatic VthVth cancellation
cancellation technique
technique rectifiers,
rectifiers, it
it achieves
achieves more
more than
than double
doubleefficiency
efficiency andand achieves
achieves power
power loss
loss reduction
reduction inin bothboth forward
forward andand reverse
reversebiasedbiased conditions
conditions it
it decrease
decrease turn
turn onon voltage
voltage inin forward
forward bias
bias condition
condition along
along
with
with decrease
decrease inin reverse
reverse leakage
leakage current
current during
during reverse
reverse bias
bias condition.
condition. Under
Under
the
the input
input conditions
conditions (Vin
(Vin =1Vp-p=1Vp-p coupling
coupling coefficient
coefficient ‘k’=0.01‘k’=0.01 at
at 200MHz
200MHzfrequency
withtransmitterandreceiverinductanceof22nH),
activelyfrequency with transmitter and receiver inductance of 22nH), actively biased
biaseddifferential
differential drive
drive rectifier
rectifier isis ableable toto achieve
achieve DC
DC voltage
voltage ofof 880mv
880mv forfor 50kΩ
50kΩ
load
load resistanceresistance inin dependent
dependent stagestage andand achieves
achieves DC
DC voltages
voltages ofof 803.2mv
803.2mv forfor50kΩ
50kΩ loadload resistanceresistance in
in independent
independent stage.
stage. Regulated
Regulated output
output DC
DC voltage
voltage isisprovidedprovided under
under variablevariable coupling
coupling conditions
conditions andand becomes
becomes more
more efficient.
efficient.
Backscattering
Backscattering communication
communication has
has been
been performed
performed using
using switch
switch byby changing
changing
thethe resonanceresonance frequency
frequency ofof thethe receiver.
receiver.

GS5-8
GS5-8 Feasibility
Feasibility Study
Study ofof UAVUAV Implementation
Implementation In
In Route
Route Surveying
Surveying

D.Hazry,
D.Hazry, M
MAA Azizan,Azizan, Safwan
Safwan Suhaimi,
Suhaimi, Zulaiha
Zulaiha Ramli,
Ramli, Mohamad
Mohamad Syafiq
Syafiq A.K,
A.K, Zainuddin
Zainuddin Hat
Hat(Universiti(Universiti Malaysia
Malaysia Perlis,
Perlis, Malaysia)
Malaysia)

Unmanned
Unmanned Aerial
Aerial Vehicles
Vehicles (UAVs)(UAVs) areare widely
widely used
used inin numerous
numerous field
field andand thethetechnology
technology keeps
keeps growing.
growing. Generally,
Generally, conventional
conventional method
method use
use inin datadatacollection
collection forfor engineeringengineering work
work isis tedious
tedious andand requires
requires aa lot
lot ofof manpower.
manpower.
This
research
focusesontheinvestigationofthesuitabilityof
UAV
This research focuses on the investigation of the suitability of UAV forfor routeroute
survey.
There
were
three
stages:data
collection,
data
processing
andsurvey. There were three stages: data collection, data processing and data
data
analysis.
analysis. By
By using
using KAMA
KAMA BETA,
BETA, thethe data
data waswas collected.
collected. For
For data
data processing,processing,Pix4Dmapper
Pix4Dmapper used
used forfor point
point cloud
cloud andand AutoCAD
AutoCAD 20142014 forfor analysis
analysis stage.
stage. The
The
resultresult isis mainly
mainly focused
focused on
on extracting
extracting thethe roadroad profile
profile andand test
test thethe point
point cloud
cloud
data
data using
using RMSE.
RMSE. TheThe result
result fromfrom both
both method,
method, conventional
conventional andand UAVUAV fromfromthethe point
point cloud
cloud data
data using
using RMSE
RMSE show
show only
only thethe smallsmall difference.
difference.

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 95 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

AUTHORS INDEX
Notation of session name
PS: Plenary Session IS: Invited Session, OS: Organized Session, GS: General
Session,Note: 33/90 = (page no. in Technical Paper Index) / (page no. in Abstracts)[A]

Arasuna

Masaki

OS7-4

23/54

Abbasi

Saad

OS3-1

25/46

Arshad

Talal

GS5-5

38/95

Abe

Akinori

OS24-3

37/85

Asada

Taro

OS7-2

23/54

OS24-4

37/85

OS7-3

23/54

OS7-4

23/54

Abideen

Zain ul

GS5-3

38/93

AburadaKentaro

OS6-3

35/52

Asami

Ryo

GS5-1

38/93

OS6-4

35/52

AshidaYoichiroOS18-1

25/74

OS6-5

35/53

Attequa

GS5-7

38/96

OS6-6

36/53

Awais

Muhammad

GS5-7

38/96

Azizan

MA

GS5-8

38/96

Adamatzky

Andrew

GS1-3

27/87

Adnan

Muhammad

GS5-6

38/94

Ahmad

Haseeb

GS5-5

38/94

Ahn

JonghyunOS21-1

24/79

Bai

Ruixiang

OS11-12

27/64

OS21-2

24/80

BandouYasushi

GS2-2

31/88

OS2-2

23/45

Bei

XiaominOS14-4

23/70

OS2-3

24/45

Bian

Ce

OS11-2

26/61

OS2-4

24/45

OS11-9

27/63

OS2-5

24/46

Blanchard

Damien

OS2-3

24/45

OS2-6

24/46

Bui

Duc Viet

OS17-2

30/73

GS5-8

38/96

AiharaA. K.

Kazuyuki

Mohamad[B]

Syafiq[C]

AlahmadRaji

OS20-3

20/77

Cai

Jia-Hong

OS1-5

21/43

An

Jong-Woo

OS3-3

25/47

Chang

Keng-Hao

OS1-3

21/42

An

LingranOS11-1

26/61

OS1-4

21/43

OS11-3

26/61

Chang

Liang

OS15-5

31/72

OS11-5

26/62

Chen

Kuan-Ying

GS4-3

29/92

OS11-6

26/62

Chen

Qiang

OS14-4

23/70

Chen

XiaoyanOS13-1

27/66

OS13-10

28/69

Aoki

Kenji

OS8-1

19/55

Araki

NozomuOS18-2

25/74©ICAROB 2019 ALife Robotics Corp. Ltd.

- 96 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Chen
Chen

Yan-Ren
Yan-Ren

Chen
Chen

Ying
Ying

OS1-3
OS1-3

21/42
21/42

Fujii
Fujii

Seika
Seika

OS8-3
OS8-3

19/55
19/55

OS1-4
OS1-4

21/43
21/43

Fujii
Fujii

Teruo
Teruo

OS2-5
OS2-5

24/46
24/46

OS11-2OS11-2

26/61
26/61

Fujinaga
Fujinaga

TakuyaTakuyaOS22-3OS22-3

34/82
34/82

OS11-7OS11-7

26/63
26/63

Fujisawa
Fujisawa

Ryusuke
Ryusuke

OS20-6OS20-6

20/78
20/78

OS11-9OS11-9

27/63
27/63

OS20-7OS20-7

20/79
20/79

Chen
Chen

Yun-juYun-ju

OS5-4
OS5-4

37/51
37/51

OS20-8OS20-8

20/79
20/79

Chikushi
Chikushi

Shota
Shota

OS22-4OS22-4

34/82
34/82

OS22-5OS22-5

34/82
34/82

Cho
Cho

Young
Young Im
Im

IS-2
IS-2

28/41
28/41

[D]
[D]

Fujiwara
Fujiwara

Tomofumi
Tomofumi

GS5-1
GS5-1

38/93
38/93

Fukushima
Fukushima

Hiroki
HirokiOS24-2OS24-2

37/84
37/84

Furutani
Furutani

Hiroshi
Hiroshi

OS8-1
OS8-1

19/55
19/55

D.
D.

Hazry
Hazry

GS5-8
GS5-8

38/95
38/95

OS8-2
OS8-2

19/55
19/55

Dai
Dai

FengzhiFengzhiOS11-1OS11-1

26/61
26/61

OS8-3
OS8-3

19/55
19/55

OS11-2OS11-2

26/61
26/61

OS8-4
OS8-4

19/56
19/56

OS11-3OS11-3

26/61
26/61

OS8-5
OS8-5

19/56
19/56

OS11-4OS11-4

26/62
26/62

OS8-6
OS8-6

19/57
19/57

OS11-5OS11-5

26/62
26/62

OS8-7
OS8-7

19/57
19/57

OS11-6OS11-6

26/62
26/62

OS11-7OS11-7

26/63
26/63

OS11-8OS11-8

26/63
26/63

OS11-10OS11-10

27/64
27/64

OS11-9OS11-9

27/63
27/63

OS11-12OS11-12

27/64
27/64

[G]
[G]
Gao
Gao

ZihengZihengdidi Maria
Maria

EnricoEnricoOS20-5OS20-5

20/78
20/78

Ge
Ge

Zefan
Zefan

OS11-11OS11-11

27/64
27/64

Din
Din

Siraj
Siraj

GS5-7
GS5-7

38/95
38/95

Geng
Geng

KaixuanKaixuanOS11-11OS11-11

27/64
27/64

Doi
Doi

Shunsuke
ShunsukeOS23-4OS23-4

32/84
32/84

Gunaratne
Gunaratne

PraveenPraveen

OS19-3
OS19-3

34/76
34/76

Domae
Domae

Yukiyasu
Yukiyasu

GS2-5
GS2-5

31/89
31/89

Du
Du

Meng
Meng

OS13-1OS13-1

27/66
27/66

OS1-7
OS1-7

21/44
21/44

OS4-2
OS4-2

22/48
22/48

NuwanthaNuwantha
Guo
Guo

[E]
[E]
Ejaz
Ejaz

Mahnoor
Mahnoor

GS5-3
GS5-3

38/93
38/93

Endo
Endo

Takahiro
Takahiro

GS5-1
GS5-1

38/93
38/93

Jr
Jr Hung
Hung

Guo
Guo

Shaoyan
Shaoyan

OS15-3OS15-3

31/71
31/71

Guo
Guo

XinranXinranOS11-5OS11-5

26/62
26/62

Han
Han

JialinJialinOS11-4OS11-4

26/62
26/62

[H]
[H]
[F]
[F]
Figueroa
Figueroa

RicardoRicardo Q.
Q.

GS1-3
GS1-3

27/87
27/87

Han
Han

Seong
Seong Ik
Ik

GS2-3
GS2-3

31/89
31/89

Frederiksen
Frederiksen

MortenMorten Roed
Roed

IS1-1-1
IS1-1-1

18/40
18/40

Han
Han

Yu
Yu

OS13-8OS13-8

28/68
28/68

Fu
Fu

Ziyi
Ziyi

OS14-1OS14-1

22/69
22/69

Hanajima
Hanajima

NaohikoNaohiko

OS15-1
OS15-1

30/70
30/70

Fuchigami
Fuchigami

Kazuki
KazukiOS21-2OS21-2

24/80
24/80

HaradaHaradaKensukeKensuke

PS-1
PS-1

29/39
29/39

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 97 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

HaradaKensuke

GS2-5

31/89

OS8-4

19/56

HaradaKazumasa

OS21-5

24/81

OS8-5

19/56

Hat

Zainuddin

GS5-8

38/95

OS8-6

19/57

Hattori

TetsuoOS23-1

32/83

OS8-7

19/57

OS23-2

32/83

HiyamaTsukasa

OS16-1

30/72

OS23-332/83

Ho

Chian C.

OS1-6

21/43

OS23-4

32/84

Hori

Masamichi

OS10-1

36/59

OS10-2

36/59

OS23-1

32/83

HattoriYukio

OS19-1

34/76

Hayakawa

Takuya

GS2-2

31/88

Hayashi

Eiji

OS20-1

20/77

OS23-2

32/83

OS20-6

20/78

OS23-3

32/83

OS20-7

20/79

OS1-5

21/43

OS20-8

20/79

OS1-7

21/44

OS22-5

34/82

OS4-3

22/48

GS2-1

31/88

OS4-7

22/50

GS4-1

29/91

Horikawa

Hsia

Yo

Kuo-Hsien

Hsu

Shu-Fang

HayashiYuki

OS24-3

37/85

He

Yufan

OS11-5

26/62

GS4-2

29/91

OS11-9

27/63

GS4-3

29/92

GS4-5

29/92

Higashi

KatsunoriOS21-2

24/80

Hirai

Hiroyasu

OS21-2

24/80

Huang

Bo-Kai

OS1-2

21/42

OS21-4

24/80

Huang

JingxuanOS13-8

28/68

Hirakoba

Futa

OS6-3

35/52

Huang

QianliOS12-1

33/65

Hiraoka

Toru

OS5-1

36/50

Huang

TieqiuOS12-1

33/65

OS5-2

37/50

Hung

Chung-Wen

OS1-1

20/42

OS1-2

21/42

HirataYusukiOS20-4

20/78

Hiroyasu

Tomoyuki

OS8-1

19/55

Hung

Yi-ping

OS25-2

33/86

OS8-2

19/55

Hussain

Tanveer

GS5-4

38/94

OS8-3

19/55

Hwang

Do-Kyung

OS3-3

25/47

OS8-4

19/56

OS8-5

19/56

OS8-6

19/57

Satoshi

OS5-3

37/51

OS8-7

19/57

OS10-1

36/59

OS8-1

19/55

OS10-2

36/59

OS8-2

19/55

OS10-4

36/60

OS8-3

19/55

OS21-2

24/80

Hiwa

Satoru©ICAROB 2019 ALife Robotics Corp. Ltd.

[I]
Ikeda

Ikeda

- 98 -Tomomasa

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Ikeuchi

Yoshiho

OS2-5

24/46

OS14-3

23/70

Imai

ShinichiOS16-430/73

OS15-3

31/71

Imai

Yoshiro

OS6-1

35/51

OS15-5

31/72

OS23-1

32/83

Jia

Yongnan

OS15-2

30/71

OS23-2

32/83

Jiang

Wenping

OS13-6

28/68

OS23-3

32/83

Jin

Xia

OS11-1

26/61

OS23-432/84

OS11-3

26/61

OS20-2

20/77

OS11-5

26/62

OS20-3

20/77

OS11-6

26/62

OS20-5

20/78

Jitkongchuen

Duangjai

GS3-2

32/91

OS21-1

24/79

Jitviriya

Wisanu

GS2-1

31/88

OS21-2

24/80

JunaidSameed bin

GS5-6

38/94

OS21-324/80

OS21-424/80

OS22-2

34/81

Koji

OS23-1

32/83

OS22-3

34/82

OS23-2

32/83

OS22-434/82

OS23-3

32/83

OS10-1

36/59

OS23-432/84

OS10-2

36/59

Kamei

Keiji

OS22-6

34/82

OS5-1

36/50

Kan

Ryota

OS7-4

23/54

OS5-2

37/50

Katayama

Tetsuro

OS6-3

35/52

OS5-3

37/51

OS6-4

35/52

OS10-1

36/59

OS6-5

35/53

OS10-2

36/59

OS6-6

36/53

OS10-4

36/60

OS2-2

23/45

OS5-1

36/50

OS2-6

24/46

OS5-2

37/50

OS16-1

30/72

OS5-3

37/51

OS16-3

30/72

Ishii

Kazuo

IshizuTakahiro

Ito

Takao

Ito

Tsutomu[K]
KagawaKatori
KawadaYuichi
Kazuo

Ito

Yoshiki

GS1-1

27/87

Kawaguchi

Natsuki

OS18-2

25/74

Iwanaga

Saori

OS17-1

30/73

Kawakami

YusukeOS23-1

32/83

OS23-2

32/83

OS23-3

32/83

OS23-4

32/84

[J]
Jheng

Li-Sheng

OS1-1

20/42

Jia

HongyanOS13-3

28/67

Kawamoto

Kota

OS22-5

34/82

Jia

Yingmin

PS-3

35/39

Kawamura

Yoji

OS24-537/85©ICAROB 2019 ALife Robotics Corp. Ltd.

- 99 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS25-3

33/86

Chun-Chi

OS4-1

21/48

Lee

In-Soo

GS4-4

29/92

Lee

Jang-Myung

OS3-2

25/47

OS3-3

25/47

Jong-Hyun

GS4-4

29/92

Lee

Min Cheol

OS3-1

25/46

34/82

Lee

Roger CL

OS1-2

21/42

OS18-325/75

Lee

Soo-jeong

OS3-4

25/47

OS18-525/75

Leggieri

Massimiliano

IS1-1-1

18/40

OS6-3

35/52

GS2-6-1

31/90

OS6-4

35/52

GS2-6-2

32/90

OS6-5

35/53

OS2-3

24/45

OS6-6

36/53

OS2-4

24/45

OS2-5

24/46

KawanoHiromichi

OS23-3

32/83

KawazoeKengo

OS20-7

20/79

Lai

Khan

Danish

GS5-5

38/94

Khoyratee

Farad

OS2-5

24/46

Kim

Hyun-Hee

OS3-1

25/46

Kim

Ji-Hyeon

OS3-2

25/47

Lee

Kim

Soo Hyeon

OS2-5

24/46

KimuraKenji

OS22-4

Kinoshita

TakuyaKita

Yoshihiro

Levi

TimothéeKitani

Makoto

GS5-1

38/93

Kobayashi

Jun

GS2-2

31/88

Li

BingheOS22-3

34/82

Kobayashi

Keitaro

OS8-4

19/56

Li

Chenyang

OS12-2

33/65

Kohno

Takashi

OS2-1

23/44

Li

Hao

OS11-3

26/61

OS2-4

24/45

OS11-6

26/62

KoizumiNoriko

OS8-4

19/56

Li

Handong

OS14-2

22/69

KonishiYasuo

OS18-2

25/74

Li

Hongbing

OS4-5

22/49

Kozui

Masatoshi

OS18-325/75

LiJung-Shian

OS1-3

21/42

Kubo

Masao

OS17-1

30/73

OS1-4

21/43

OS17-2

30/73

Li

Min

OS12-2

33/65

OS17-3

30/74

Li

Qing

OS15-2

30/71

OS5-1

36/50

Li

Shang-Hui

GS4-1

29/91

OS5-2

37/50

GS4-2

29/91

KumanoMinoruKumjaikong

Sutee

GS2-1

31/88

GS4-3

29/92

Kuo

Chia-Chen

OS25-1

33/86

GS4-5

29/92

OS25-3

33/86

OS11-4

26/62

OS6-2

35/52

OS11-826/63

Kurozumi

Ryo

Li

[L]
Lai

Chuan-Lin

OS25-1©ICAROB 2019 ALife Robotics Corp. Ltd.

33/86

Yang

Li

Yuling

OS15-4

31/71

Liao

Guan-Lung

OS1-6

21/43

Lien

Chiang-OS25-1

33/86

Hsiang- 100 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Lien

Chiang-OS25-3

33/86

Hsiang

Mehta

Rajiv

OS5-3

37/51

Magid

Evgeni

OS1-5

21/43

Lin

Chia-Jen

OS4-1

21/48

OS4-1

21/48

Lin

Chun-Chu

OS1-1

20/42

OS4-2

22/48

Lin

Yixin

OS13-7

28/68

OS4-3

22/48

OS13-8

28/68

OS4-4

22/49

OS13-9

28/69

OS4-5

22/49

Liou

Sheng-Jie

OS4-2

22/48

OS4-6

22/49

Liu

Chuan-Gang

OS1-4

21/43

OS4-7

22/50

Liu

Fan

OS15-4

31/71

Martínez

Genaro J.

GS1-3

27/87

Liu

GuanghuiOS15-1

30/70

Matsumaru

Takafumi

GS2-4

31/89

Liu

I-Hsien

OS1-3

21/42

MatsunoFumitoshi

GS5-1

38/93

OS1-4

21/43

MatsunoHiroshi

GS3-1

32/90

Liu

Kaige

OS11-1

26/61

Matsuno

Seigo

OS5-3

37/51

Liu

Qingming

OS13-7

28/68

MatsuoTakayukiOS22-2

34/81

Liu

Qunpo

OS15-1

30/70

Min

Jun

OS13-6

28/68

Liu

Shu-HsinOS25-1

33/86

MiyajiToshihiro

OS6-5

35/53

OS25-3

33/86

Mizoe

Eiji

OS20-4

20/78

Liu

Wei

OS12-4

33/66

Mo

Lipo

OS15-3

31/71

Liu

XianzheOS15-1

30/70

Mochizuki

Ruugo

OS20-2

20/77

Liu

Yafan

OS12-3

33/65

Mori

Keisuke

OS6-4

35/52

Liu

YanxinGS2-6-1

31/90

Mori

Shinnosuke

OS16-330/72

GS2-6-2

32/90

Moriguchi

Yuki

OS16-2

30/72

Liu

YuanshanOS15-431/71

Morimoto

Shogo

OS6-1

35/51

Liu

ZonglinOS13-5

28/67

MoritaYoshifumi

GS5-1

38/93

Lund

Henrik

IS1-1-1

18/40

GS5-2

38/93

HautopLuo

LianMoriyama

HiroyukiOS22-6

34/82

IS1-1-2

18/40

Mowshowitz

Abbe

OS20-6

20/78

IS1-2-1

18/40

Muto

Shohei

OS4-4

22/49

IS1-2-2

18/41

GS2-6-1

31/90

GS2-6-2

32/90

Nagai

Yousuke

OS20-4

20/78

OS12-4

33/66

Nagamatsu

Masayasu

OS16-2

30/72

NagataKazuyuki

PS-1

29/39

GS2-5

31/89

[N]

[M]

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 101 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Nakajima

ShuheiOS21-5

24/81

Pai

Shu-Li

OS1-5

21/43

Nakamura

Akira

GS2-5

31/89

Park

Chan Gook

OS3-4

25/47

Nakamura

Seishiro

OS8-6

19/57

Park

Woo-Jung

OS3-4

25/47

Nakaoka

Iori

OS5-4

37/51

Park

Yong-Gonjong

OS3-4

25/47

Naqash

Talha

GS5-3

38/93

Park

Yousin

OS5-4

37/51

Nie

Zhen

OS13-4

28/67

Peng

YizhunOS11-1027/64

OS13-5

28/67

OS11-1127/64

OS11-1227/64

NishidaTatsumi

OS6-6

36/53

NishidaYuya

OS21-1

24/79

Phunopas

Amornphun

GS2-1

31/88

OS21-224/80

Pongtawevirat

Piyalak

GS3-2

32/91

OS21-324/80

Prasetio

BarlianOS19-2

34/76

Noppadol

GS2-1

31/88

Qin

XiaoFeng

OS10-3

36/60

Qin

YiqiaoOS11-1

26/61

Nishikawa

Stephany Mai

OS2-5

24/46

Niu

Hong

OS13-2

27/67

NonakaHirofumi

OS5-1

36/50

OS5-2

37/50

Henryranu
Pudchuen
[Q]

[O]
Ogata

TakashiOS24-1

37/84

OS11-2

26/61

Ohnishi

Yoshihiro

OS16-330/72

OS11-3

26/61

Ohshima

Kotaro

GS5-2

38/93

OS11-6

26/62

OkazakiNaonobu

OS6-3

35/52

OS11-7

26/63

OS6-4

35/52

OS11-8

26/63

OS6-5

35/53

OS6-6

36/53

[R]
Rajapakse

OS23-1

32/83

Okugawa

Yohei

OS17-2

30/73

Okumura

Naoki

OS8-4

19/56

Omata

Kaito

OS16-430/73

OS23-2

32/83

Onishi

Akane

OS8-2

19/55

OS25-233/86

OnoJumpeiOS24-1

37/84

Ramirez-Alpizar

Ixchel G.

PS-1

29/39

Ouchi

Tomohito

OS7-1

23/53

Ramli

Zulaiha

GS5-8

38/95

Artur

OS4-3

22/48

Janaka[P]
Pagliarini

R. P. C.

[S]
Luigi

IS1-1-2

18/40

IS1-2-1

18/40

OS4-4

22/49

IS1-2-2

18/41

OS4-5

22/49

©ICAROB 2019 ALife Robotics Corp. Ltd.

Sagitov- 102 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

OS4-6

22/49

Song

Chen

OS13-928/69

Sahara

Kei

OS8-5

19/56

Song

Jong-Hwa

OS3-4

25/47

Sakamoto

Makoto

OS5-3

37/51

Song

Rende

OS10-3

36/60

OS8-1

19/55

Song

Yunzhong

OS14-1

22/69

OS10-1

36/59

SonodaTakashiOS21-1

24/79

OS10-2

36/59

OS21-224/80

OS10-336/60

OS22-2

34/81

OS10-4

36/60

OS1-7

21/44

OS10-5

36/60

OS4-1

21/48

OS19-1

34/76

OS4-2

22/48

OS19-2

34/76

OS4-3

22/48

OS19-3

34/76

OS4-7

22/50

Sakurai

Keiko

Su

Kuo-LanSaleemHammad

GS5-7

38/95

Sugii

Manabu

GS3-1

32/90

SanadaAtsushiOS22-1

34/81

Sugiyama

Masanori

PS-2

19/39

Sato

HiroshOS17-2

30/73

Suhaimi

Safwan

GS5-8

38/95

Sato

Kenya

OS8-6

19/57

Sumi

Kaoru

IS-3

29/41

OS8-7

19/57

Sun

Zijie

OS10-3

36/60

GS5-1

38/93

Suzuki

Yasuhiro

OS9-1

33/58

GS5-2

38/93

OS9-2

33/58

OS9-3

33/58

OS7-1

23/53

Sato

Noritaka

Sato

Takao

OS18-2

25/74

Sato

Takashi

OS7-3

23/54

Shabalina

Ksenia

OS4-3

22/48

OS4-7

22/50

[T]
TabuseMasayoshi

Shah

Rafay

GS5-4

38/94

OS7-2

23/54

Shah

Sajjad

GS5-3

38/93

OS7-3

23/54

OS7-4

23/54

OS6-1

35/51

OS6-2

35/52

HussainTakagiTomohiko

ShaikhM Osama

GS5-4

38/94

Shaikh

Talha

GS5-4

38/94

Shang

Yi-Tai

GS4-1

29/91

Takahashi

Masanori

OS18-4

25/75

GS4-3

29/92

TakaichiShohei

OS2-6

24/46

Shaukat

Asmara

GS5-5

38/94

TakanoTetsuo

OS4-4

22/49

Shi

WengxinOS13-3

28/67

Takei

Amane

OS10-1

36/59

Shigeyama

Tamon

OS20-8

20/79

OS10-2

36/59

Shinnoki

TomoyaOS21-324/80

OS10-536/60

Sohail

Osama

GS5-3

38/93

OS20-4

20/78©ICAROB 2019 ALife Robotics Corp. Ltd.

Takemura- 103 -

Yasunori

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Wang

OS25-1

33/86

OS25-3

33/86

Fuzhong

OS14-1

22/69

Wang

HongqiOS15-1

30/70

34/76

Wang

jie

OS3-1

25/46

Wang

Jiwu

OS12-2

33/65

Takemura

YasunoriOS22-2

34/81

Tamai

TeruyukiOS16-3

30/72

TamuraHirokiOS19-1

34/76

Wang

OS19-2

34/76

OS19-3Chih-Wei

TamuraHiroto

OS2-2

23/45

TanakaKatsunoriOS21-224/80

OS12-3

33/65

Tani

Shohei

OS7-4

23/54

OS12-4

33/66

Tanno

KoichiOS19-1

34/76

OS12-5

33/66

OS19-2

34/76

OS25-1

33/86

OS19-3

34/76

OS25-3

33/86

OS11-1027/64

OS11-12

27/64

Wang

Ming-Jen

Wang

Shuo

TashiroMasao

OS20-4

20/78

Temma

Seiya

GS3-1

32/90

Tominaga

Ayumu

OS20-6

20/78

Wang

Sufang

OS15-2

30/71

Toyoshima

Naoki

OS2-1

23/44

Wang

Tse-Yao

OS1-8

21/44

ToyotaTakaaki

OS10-2

36/59

Wang

Wei

OS11-4

26/62

Tsoy

Tatyana

OS4-6

22/49

Wang

Xiao

OS14-2

22/69

Tsuji

Tokuo

PS-1

29/39

Wang

YunfeiOS13-928/69

TunsiriSongklod

GS2-1

31/88

Watanabe

KeisukeOS21-5

24/81

Wei

Qiang

OS13-7

28/68

OS13-828/68

OS13-9

28/69

[U]
Ue

Yuki

OS6-1

35/51

Ueda

TakumiOS21-2

24/80

Wen

Fu-Hsiang

OS1-8

21/44

Uejo

Kento

GS1-2

27/87

Wu

JasmineOS25-1

33/86

Uenohara

Seiji

OS2-4

24/45

OS25-3

33/86

Umair

Muhammad

GS5-6

38/94

Utsunomiya

Koshi

OS21-5

24/81

OS6-3

35/52

OS6-4

35/52

[Y]
YamabaHisaaki[W]
WakitaniShin

OS18-1

25/74

OS6-5

35/53

Wan

Weiwei

PS-1

29/39

OS6-6

36/53

Wang

Bo-Kai

OS1-6

21/43

GS1-1

27/87

Wang

Chengcai

OS11-1

26/61

GS1-2

27/87

OS11-2

26/61

Yamaguchi

AkihiroOS17-330/74

OS11-7

26/63

Yamaguti

YutakaOS17-330/74©ICAROB 2019 ALife Robotics Corp. Ltd.

Yamada- 104 -

Takayoshi

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Yamamoto
Yamamoto

HidehikoToru

Yin

Hongyi

GS1-1

27/87

GS1-2

27/87

OS18-1

25/74

OS18-325/75

OS18-525/75

YoshidaYin

OS13-1

27/66

OS13-1029/69

OS13-1

27/66

OS13-10

29/69

Chika

OS19-3

34/76

Tong

Yamanobe

Natsuki

PS-1

29/39

Yoshimura

MamoruOS10-4

36/60

Yan

Yujie

OS11-1

26/61

Yoshioka

Koma

OS8-7

19/57

OS11-3

26/61

Yoshitomi

Yasunari

OS7-2

23/54

OS11-4

26/62

OS7-3

23/54

OS11-5

26/62

OS7-4

23/54

OS11-6

26/62

Yu

Shit-Ting

OS1-2

21/42

Yang

Bo-Jun

OS4-2

22/48

Yuan

XiaolinOS15-3

31/71

Yang

GuoweiOS11-2

26/61

Yuan

Yasheng

OS11-2

26/61

Yang

Hongyong

OS15-431/71

OS11-6

26/62

Yang

Rui

OS11-1127/64

OS11-7

26/63

Yang

Xia

OS2-4

24/45

OS11-8

26/63

Yang

Xiao

OS13-7

28/68

OS11-9

27/63

Yang

Yi-Chang

OS1-1

20/42

Ryoma

OS21-2

24/80

Yang

Yize

OS15-4

31/71

Yang

Zhou

OS11-11

27/64

Yano

Yuiko

OS7-2

23/54

Zakiev

Aufar

OS4-7

22/50

Yasukawa

ShinsukeOS20-2

20/77

Zamorano

Daniel A.

GS1-3

27/87

OS21-1

24/79

Zeng

WeixinOS12-5

33/66

OS21-2

24/80

Zhang

Bin

OS14-323/70

OS21-3

24/80

Zhang

Bowen

OS12-1

33/65

OS22-3

34/82

Zhang

Chen

OS10-3

36/60

OS11-1

26/61

Zhang

HanwenOS14-423/70

OS11-3

26/61

Zhang

HengbinOS14-423/70

OS11-5

26/62

Zhang

Hongyu

OS3-2

25/47

OS11-6

26/62

Zhang

KailunOS11-4

26/62

OS11-8

26/63

Zhang

QinjianOS12-3

33/65

OS11-2

26/61

Zhang

TianyiOS11-4

26/62

OS11-7

26/63

Zhang

Tingting

OS11-7

26/63

OS11-8

26/63

OS11-8

26/63

OS11-927/63

OS11-9

27/63

Ye

Yin

Zhongyong

Di©ICAROB 2019 ALife Robotics Corp. Ltd.

Yuki
[Z]

- 105 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

Zhang

WeicunOS14-2

22/69

OS15-2

30/71

Zhang

Weiya

OS13-5

28/67

Zhang

XiaoyuOS13-7

28/68

Zhang

Xinyu

OS11-2

26/61

Zhang

YingzhaoOS14-2

22/69

Zhang

Yu-an

OS10-1

36/59

OS10-2

36/59

OS10-3

36/60

Zhao

ChenglinOS11-3

26/61

Zhao

HuailinOS13-4

28/67

OS13-5

28/67

OS13-6

28/68

Zheng

JingnanOS13-5

28/67

Zhu

Chen

GS2-4

31/89

Zhu

YuxuanOS11-2

26/61

OS11-7

26/63

OS11-8

26/63

OS11-927/63©ICAROB 2019 ALife Robotics Corp. Ltd.

- 106 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 107 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 108 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 109 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 110 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 111 -

The 2019 International Conference on Artificial Life and Robotics ( ICAROB2019),B-Con Plaza, Jan. 10-13, Beppu, Oita, Japan, 2019

©ICAROB 2019 ALife Robotics Corp. Ltd.

- 112 -

Motion-Data Driven Grasp/Assembly Planner
Kensuke Harada12, Natsuki Yamanobe2, Weiwei Wan12
Kazuyuki Nagata2, Ixchel G. Ramirez-Alpizar1, and Tokuo Tsuji3

1 Graduate School of Engineering Science, Osaka University, 1-3 Machikaneyama, Toyonaka 560-8531, Japan
2 Intelligent Systems Research Institute, National Institute of Advanced Industrial Science and Technology(AIST), 1-1-1
Umezono, Tsukuba 305-8568, Japan
3 Institute of Science and Technology, Kanazawa University, Kadoma-cho, Kanazawa 920-1192, Japan
E-mail: harada@sys.es.osaka-u.ac.jp, n-yamanobe@aist.go.jp, wan@sys.es.osaka-u.ac.jp, k-nagata@aist.go.jp,
ramirez@sys.es.osaka-u.ac.jp, tokuo-tsuji@se.kanazawa-u.ac.jp

Abstract
This paper proposes a robotic grasp/assembly planner using a database of complex and skillfull motion. In our
proposed planner, if a skillfull motion is included in the solution path, motion data included in the database is used.
In this paper, we first explain the three-layered structure of the motion data. Then, we explain the proposed
grasp/assembly planner. Finally, we show a numerical example to confirm the effectiveness of our proposed method.
Keywords: Robotic assembly, Motion planning, Skillfull motion, Database.

Especially, our motion generation tool utilizes thepreviously proposed three-layered structure of motion
data. In our proposed grasp/assembly planner, if thesolution path includes complex and skillfull motion, suchmotion is realized by using the database of skillfull
motion.
This paper briefly outlines our novel grasp/assemblyplanner using the motion data and shows a simple
numerical example. The rest of this paper is organized asfollows; we first explain the three-layered data structurein Section 2. The robotic grasp/assembly planner is
explained in Section 3. Application of motion data to thegrasp/assembly planner is explained in Section 4. A
numerical example is shown in Section 5.

1. Introduction
In factory environment, industrial robots are expected toassemble a product. However, robotic assembly isdifficult since a robot has to assemble a variety of objects.
Sometimes, a robot has to realize a very skillfull andcomplex motion.
On the other hand, this research aims to fully automate
the product assembly process and proposes a novel
robotic grasp/assembly planner. In conventional
grasp/assembly planners, we simultaneously planned thegrasping posture of a part, the assembly sequence and the
motion of a robot. However, conventional
grasp/assembly planners can just realize a series of pickand-place motion of a robot. For the purpose of realizing
more complex and skillfull motion by using agrasp/assembly planner, we consider incorporating thedatabase of skillfull motion. The motion data can begenerated by using several methods such as humanmotion capture and robotic motion generation methods.

2. Three-Layered Data Structure
In this section, we explain the structure of the motion data
and method for storing skillfull motion to the database[1].
An assembly task is composed of multiple task elements© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan1

Kensuke Harada, Natsuki Yamanobe, Weiwei Wan, Kazuyuki Nagata, Ixchel G. Ramirez-Alpizar, Tokuo Tsuji

where each task element is composed of a set of motion
sequence. According to this classification, the first, thesecond and the third layers are respectively named as"work", "task" and "action" layers. Fig. 1 shows anexample of assembly of a toy airplane. Since a single
action usually includes multiple objects to be assembled,
the object held by a robot is named as the "main object"
and the object assembled to the main object is named as
the "sub object". The following is a set of information
included in the database:・Motion strategy (A sequence of task/motion)
・Human (Identified name etc.)
・Robot (Name, type, hand)・Object Name
・Object category (e.g., JICFS code)
・Object shape
・ Object parameters (Mass, CoG, material, stiffness,
etc.)
・Grasping configuration (Main object)・Fixture configuration (Sub object)・Time series of the main object's pose w.r.t the sub
object・Control method and its parameters
・Force information applied by the hand
The data can be obtained by several methods such as ahuman motion capture and a motion generation tool. The
data is stored in the YAML format where some examples
are shown in Fig. 1.
On the other hand, we have constructed a software tool
manually constructing a skillfull motion and store it to
the database [1]. Among the three-layered structure of
database, our proposed grasp/assembly plannerespecially uses the action data corresponding to the edgeof the AND/OR graph of assembly which will be
explained in the next section.

Fig. 1 Three Layered Data Structure and Data Expression
using YAML Format

3. Grasp/Assembly Planner Definition
This section shows the overview of the grasp/assemblyplanner which has proposed in [2,3]. Let us consider aproduct composed of m parts P = (P1, · · · , Pm). Let us also
consider assembly of parts by C = (C1, · · · , C r). For example,
the assembly Cl composed of the parts Pu, P v , and Pw canbe defined as
Cl = {(Pu, P v , Pw), (T uv , T uw), (nuv, nuw)}(1)

where Tuv and nuv denote the homogenous matrix and theapproach vector, respectively, of Pv relative to Pu used foran assembly task.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan2

Motion-Data Driven Grasp/Assembly Planner

4.

Application of Assembly Motion Data

The grasp/assembly planner explained in the previous
subsection just can realize a series of pick-and-placemotion. We consider extending this grasp/assemblyplanner for more complex tasks by using the database ofskillfull assembly motion data. In the AND/OR graphshown in Fig.2, the wing is picked by the gripper and put
on the body, and then the cockpit can be picked and put
on the wing. However, the bolt cannot be assembled to
the wing just by using a pick-and-place motion. The
edges marked in red shows that the assembly motion data
is required for realizing this assembly sequence. We
consider associating the action Yaml file corresponding
to this assembly sequence. To obtain an assembly
sequence, we consider solving the AND/OR graph. If an
action Yaml file is associated to the solution edge, we
consider using the action Yaml file to generate the robot
motion. Otherwise, we consider constructing themanipulation graph and solving it to obtain the robot
motion.

Fig. 2 AND/OR graph of our assembly task where the redlines denote the edges with which the action data is
associated.
To assemble a product, a gripper has to grasp an assemblyof parts. We consider a set of n grippers H = (H1, · · · , Hn)
where Hj denotes the j-th gripper. We consider using agrasp planner to calculate a grasping posture where Gljk
denotes the k-th grasping posture of the assembly Cl by
the gripper H j . By iteratively calculating grasping posturewith changing the approach position, we consider
constructing a set of grasping postures Glj = (Glj1, · · · , Gljk)of the assembly Cl grasped by the gripper H j .
Every possible assembly sequence can be expressed byusing the AND/OR graph G(V, E) where the node V is
composed of the assembly of parts Cl, l = 1, · · · , r (Fig. 2).
In the grasp/assembly planner introduced in this section,
we assume that the AND/OR graph is defined in advance.
To obtain robot motion to assemble a given product, wefirst solve the AND/OR graph[4,5] and obtain a possible
assembly sequence. Then, we consider constructing themanipulation graph from the assembly sequence
including a set of grasping posture. By solving themanipulation graph, we simultaneously plan grasping
posture and robot motion. The method is detailed in ourprevious papers [2,3].

5. Result
We consider performing the robotic assembly of a toy
airplane. We prepared the skillfull motion of the robot
rotating the bolt stored in the database by the YAML
format. We first obtained the solution path of the
AND/OR graph. Since the solution path includes thescrewing motion of the bolt, we used the action YAML
data of the screwing motion. The result is shown in Fig.
3. As shown in the figure, when the robot assembles thebolt to the wing, the robot tries to rotate the bolt by using
the task motion data.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan3

Kensuke Harada, Natsuki Yamanobe, Weiwei Wan, Kazuyuki Nagata, Ixchel G. Ramirez-Alpizar, Tokuo Tsuji

Acknowledgements
This research was conducted under the support of NEDO
AI-Robotics Project.

References
1. N. Yamanobe, T. Tsuji, K. Harada, K. Nagata, R. Hanai,
and W. Wan, and I.G. Ramirez-Alpizar, ”Construction of
Cloud Database for Task Motion Generation”, Annual Conf. on
the Robotics Society of Japan, 2016.
2. K. Harada, K. Nakayama, W. Wan, K. Nagata,
N.Yamanobe, T. Tsuji, and I.G. Ramirez-Alpizar, ”Tool
Exchangeable Grasp/Assembly Planner”, Proc. of Int. Conf. onIntelligent Autonomous Systems, 2018.
3. W. Wan and K. Harada, ”Developing and Comparing
Single-arm and Dual-arm Regrasp,” Robotics and Automation
Letter, vol. 1, no. 1, pp. 243-250, 2016.
4. L.S.M. de Mello and A.C. Sanderson, “And/Or Graph
Representation of Assembly Plans”, CMU Research
Showcase, CMU-RI-TR-86-8, 1986.
5. L.S.M. de Mello and A.C. Sanderson, “Automatic
Generation of Mechanical Assembly Sequences”, CMU
Research Showcase, CMU-RI-TR-88-19, 1988.
6. T. Siméon, J.P. Laumond, J, Cortés, and A. Aahbani”,
Manipulation Planning with Probabilistic Roadmaps”, Int. J.
Robotics Research, vol. 23, no. 7-8, pp. 729-746, 2004.
7. K. Harada, T. Tsuji, and J.-P. Laumond, ”A Manipulation
Motion Planner for Dual-arm Industrial Manipulators”, Proc.
of IEEE Int. Conf. on Robotics and Automation, pp. 928934, 2014.

Fig. 3 Simulation result of toy airplane assembly

6. Conclusions
In this paper, we proposed a motion-data driven robotic
grasp/assembly planner. A motion data includes aninformation on skillfull motion which cannot simply berealized by using conventional grasp/assembly planners.
This paper first explained the structure of motion data.
Then, we explained the proposed grasp/assembly planner.
Finally, we showed a numerical example of a toy airplaneassembly.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan4

Toward Life with Partner Robots:
Developing robots with the field trial toward the practical use
Masanori Sugiyama

Toyota Motor Corporation, 1 Toyota-cho
Toyota, Aichi 471-8571, Japan
E-mail: masanori_sugiyama@mail.toyota.co.jp
www.toyota-global.com

Abstract
In the near future, robots are expected to be utilized as partners to coexist with people for various social needs.
TOYOTA is developing robots aimed to help societies. As a first step, rentals of a rehabilitation robot for walking
training at medical institutions have been launched in 2017. Another robot, Human Support Robot, which is aimed
for use to support daily living, has been adopted as a standard platform of the domestic environment league at the
RoboCup competition since 2017. For the robot development, it is important to learn deeply about the field which
the robot is to be actually used. In this presentation, we will show the scene of the field trial toward the practical use
of our robots. Also, we will introduce a humanoid robot which is integrated with advanced technologies that would
benefit us in the future.
Keywords: robot, rehabilitation, walking, training, humanoid© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan5

Masanori Sugiyama© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan6

Toward Life with Partner© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan7

Masanori Sugiyama© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

8

Engineering Modular Playware
Henrik Hautop Lund, Morten Roed Frederiksen, Massimiliano Leggieri

Center for Playware, Technical University of Denmark, Building 326, 2800 Kgs. Lyngby, DenmarkE-mail: hhl@playware.dtu.dk
www.playware.elektro.dtu.dk

Abstract
In this paper, we describe the engineering of modular playware. Constructing playware systems out of individual
modules allows the system to be easily transported to be used anywhere, to easily build different bodies and brains,
and to allow people to construct and combine the modules in a creative fashion. Thus the modularity helps fulfilling
the Playware ABC. We exemplify the engineering challenges of the hardware, software and communication designof modules through the development of handheld modules for playful rehabilitation. The handheld modules allow fora fun and motivational training of upper extremities, and can be viewed as an extension to the Moto Tiles for lower
extremity training, which have proved to be a highly successful playware for prevention and rehabilitation among
seniors. The paper outlines the engineering challenges and proposed solutions to make such modular playware for
playful upper extremity training.
Keywords: Playful technology, Playware, Personal Health Technology, Adaptive Games.

distributed and put to use anywhere, to easily build
different bodies and brains, and to allow people toconstruct and combine the modules to become creative.
Hence, modularity often helps when aiming to fulfill the
Playware ABC.

1. Introduction
Playware is defined as intelligent hardware and softwarethat creates play and playful experiences for users of allages [1, 2], and such playware has been developed for anumber of application areas including rehabilitation [3, 4,
5]. Within this area of application, the playware forrehabilitation can be viewed as games for health, though
playware puts primary focus on play rather than ongames. Playware objects are often designed to match theplug-and-play mentality of modular technology, sincesuch design strategies help facilitate the realization of theconcepts of Playware ABC. The Playware ABC entails
developing technology for Anybody, Anywhere,
Anytime by Building Bodies and Brains, whichfacilitates that users can Construct, Combine, and Create[6].

Modular playware units act as interactive nodes in a
network. The control of the modules can be either fully
contained in the embedded systems modules themselves,or the control can, for instance, be divided between onboard control on the embedded system modules andcontrol from a network connected smart device, e.g.
tablet or smartphone. In the latter case, communicationfrom the smart device to a server backbone may provide
the possibility of data collection and data analysis. In thework presented here, we investigate this latter case andthereby the engineering research examines the divisionof control between the individual modules and the smart
device, and the best strategy for robust communication
between them. The focus is working on low level

Approaching the constructing of playware systems witha focus on modularity, allows the systems to be easily© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan9

Henrik Hautop Lund, Morten Roed Frederiksen, Massimiliano Leggieri

software for controlling the embedded systems; alongwith the investigation of the on-board sensors andactuators of the boards with the objective of providing
playware experiences for the users.

methodology for designing and adapting playware gameactivities to motivate people to perform the desired
actions. The research methodology [8] is based on
iterative prototyping, along with prototyping and testingin laboratory conditions with external probands. Hence,
knowledge is built from iterations of synthesis andapplication. Here, Moto Tiles serves as an example at amuch later iteration stage than the hand-held modules at
an earlier iteration stage of rapid prototyping, illustrating
the research methodology and implementation.

One example of this for lower extremity interaction are
the Moto Tiles [4, 7]. For upper extremity interaction, wedevelop and investigate hand-held modular playware
devices. More specifically, for Moto Tiles using
NordicRF microchips for developing ANT+ radio
communication protocols, and for the hand-held devices
using ESP microchips, which provide WiFi connectivity
between devices as well as between devices and theAndroid device. The Internet of Things has opened a newworld of possibilities by using protocols that have beenaround us for a while, such as TCP/IP. The
abovementioned ESP chips currently have become themost inexpensive and most accessible choice for this kindof projects involving WiFi communications.
Furthermore, the amount of energy consumed by thedevice, places it as an efficient utility for low-energy andlow-cost working contexts. However, using the ESP
modules poses a challenge for both the embedded
software and the client systems that communicates withthem, as they can be error prone and sometimes proves
unreliable in fast connect-disconnect scenarios.

2. Rehab app with handheld modules
The goal of this application is to engage the elderly
generation in pulse-increasing activities while having fun.
It specifically targets the solar plexus area and focuses on
accurate arm movements in a fun and motivating setting,utilizing modules that can be used by Anybody,
Anywhere, Anytime according to the Playware ABC (seeFig. 1).
Hardware setup
The current setup consists of custom built ESP modules
running NODEMCU/Arduino firmware with custom
tailored components to facilitate communication over
WiFi. An external barometric pressure-meter has been
added to the board to approximate the pressure atdifferent altitude levels. The hardware sensors employedin this project consists of: raw accelerometer data froman onboard accelerometer, and barometric pressure data.
In details, we use:
• The ESP8266 Tetra module, which is a low-cost
Wi-Fi microchip at 80MHz with 32 kB memory
for instruction and 80 kB for user data.
• The Adafruit BME280 I2C Temperature
Humidity Pressure Sensor, which measureshumidity with ±3% accuracy, barometricpressure with ±1 hPa absolute accuracy (usedfor altimeter), and temperature with ±1.0°C
accuracy. It has a low altitude noise of 0.25m
and a fast conversion time.
• The Adafruit 9-DOF Absolute Orientation IMU
Fusion Breakout - BNO055, which provides a
MEMS accelerometer, magnetometer andgyroscope on a single die with a high speed
ARM Cortex-M0 based processor to digest all
the sensor data, abstract the sensor fusion andHence the following questions arise:• What are the software measures needed in orderof working around the unpredictability of using
inexpensive modules for swift real timecommunication?
• How can such communication between modular
devices be used to create a playware experienceby interacting with the users?
• Whichsensory
modalities
andHW
implementations can be used reliably to track
simple upper body movements in a manner asrobust as seen from the sensors of the Moto
Tiles in order to provide an equally robustinteractive playware e.g. for the elderly
performing upper extremity play?
By answering these questions through our research andachieving robust, simple fault tolerant communication
between hardware modules, we aim to develop a© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan10

Engineering Modular Playware

real time requirements away, and e.g. provide
sensor data such as Absolute Orientation (Euler
Vector, 100Hz), Three axis orientation data
based on a 360° sphere, Absolute Orientation(Quaterion, 100Hz), Angular Velocity Vector(100Hz), Three axis of gravitational
acceleration (minus any movement) in m/s^2.

in SWIFT for IOS devices. Both platforms performed
equally well in similar fashion.
Improving the level of fault tolerance
During the testing phase of the development of the
playware modules, various errors were encountered indifferent phases of the process flow. Randomly the ESP
modules they would fail to detect various onboard
sensors during the initialization startup phase. To counter
that, a measure was implemented to automatically restartthe device, for the state where not all hardware checks
had been passed. This could result in a never ending reset
loop, so a visual cue was added to indicate the device
startup phase (blinking led lights). This meant that a user
could manually switch the device off and on again if theautomatic restart did not solve the problem.

Software setup
ESP firmware
The current iteration of the ESP firmware is utilized
towards sending as much sensor data each second as
possible. Original versions of the firmware were
designed to be more versatile with incoming socket
connections to accept incoming messages. However, togain most of our sensors, the later versions skipped thispart and focuses solely on sending out as muchinformation as possible. The cycle is as follows:
1) Once turned on, the ESP module attempts toconnect to previously stored WIFI access point.
2) If it fails it will become a hotspot itself and canbe accessed at http:://192,168.4.1. Here it
attempts to discover any nearby WiFi accesspoints and lets the user select them from a given
list. Any selected point is stored along with auser-entered password.
3) When it successfully connects to an access point,
it broadcasts its services via MDNS/Bonjour
protocols and can be discovered by a client.
4) When a client connects via the MDNS address
to the outgoing TCP socket at port 8885, it sends
out all given sensor data until the socket
connection is broken.
5) The current format for the data transfer is JSON.

At the connection phase the modules would arbitrarily
fail to connect multiple times. The source of this error
might have been interference from other WiFi signals,
however this was not verified. As a simple solution, the
ESP firmware would allow two attempts at reconnecting
before initiating the state WiFi access point. Defining theamount of reconnect attempts that should be allowed, isa tradeoff between how quickly the user should be able
to reconfigure the device for a new working context, andhow robust the connection phase should seem for an
already configured device. Sometimes the ESP devices
would disconnect the socket connections without any
indication in the embedded software. To resolve thatproblem, an implementation of idle counters was added
to both client and embedded software parts to see
whether any data was transmitted or not. If no data wastransmitted above a specified threshold amount of times,
the ESP modules would enter idle mode and start reaccepting incoming socket connections. The client
software would in such case attempt to ping the device toinvestigate whether the device was still connected to thesame network. If the device answered, the client would
reconnect. If the device was unresponsive the client
notified the user that a module had disconnected.

Client software
The device(s) is discovered via MDNS browsing
software that basically scans the dans after “.local”
addresses that match the service profile. Any founddevice is passed off to a handler thread that establishes
the connection. The thread connects to the socket address
at port 8886 and starts reading data. Once a complete
JSON string is received it is parsed and added to a bufferqueue with sensor readings. The client software wasdeveloped both in JAVA for android devices as well as

3. Motion interpretation
The data gathered from the specific sensors are translated
into specific movement patterns in two isolated steps.
Some parts of the data received from the sensors are© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

11

Henrik Hautop Lund, Morten Roed Frederiksen, Massimiliano Leggieri

rather not precise and needs to be averaged before they
can be used. The two steps are detailed in the subsequent
sections.

SIDE and different movement patterns are expected foreach of them.
Example - target state UP:
For this state to trigger we expect the user to lift his orhers arm. This entails moving the module up a bit,rotating it a bit and keeping it still at a certain height.
These are the signs we look for. With each
iteration/measure, we see if the module has rotated. If it
has, we increment the rotation counter. Once this value
surpasses a fitting threshold, we set a HasRotated value
to true. The same goes for correct altitude changes (wemake sure the ball moves upwards). Once moved androtated we expect the user to hold the ball still at the samelevel.

Step 1 Averaging the data
We experimented with different buffer sizes to find asuitable match between speed and accuracy. We
measured single runs of each step of the algorithm andfound that each iteration takes about 20ms to complete
letting us read complete samples at 50hz. However, some
fitness arm movements are a lot swifter than a second so
we opted for a buffer of 40 samples with a moving frame
which seemed to fit our needs. All samples are averaged
and we calculate the following data points:• Average acceleration across x,y,z axis.
• Average rotation across roll, pitch, and yaw.
• Average altitude. Note that this number is alreadyfiltered on the ESP device as well using an average of 22
samples.
• A boolean value that determines whether a user defined
acceleration threshold has been reached.
• A direction of altitude movement UP, DOWN orLEVEL based on the last 25 calculated average altitudes.
If all but three values fall in one category it is determinedthat a movement in that direction happened.

When that has occurred we accept the target as reachedand resets the cycle with a new target state. Note that we
could employ pitch, yaw and roll values for each position
to assist the algorithm but we found that people shift theirwrists around too much when they hold objects, making
it impossible to determine if a target position has bereached using that specific data. It simply shifts around
too much. The algorithm works well when the values andthreshold have been fine tuned so the user don’t have tohold still in outer positions too long. In such cases where
the algorithm reacts slowly it gets hard to complete
longer workouts.

For the altimeter data, we employed low pass filtering on
all raw samples as well as the averages to hinder highly
fluctuating output. Note that the sensors are placed inside
a ball, why it makes sense to discard x,y, and z as isolateddata.

We acknowledge that such slowness holds a certain
fitness value, that might suit another perhaps younger
audience. All in all the algorithm seems to fit well for thetarget audience, in that it is very forgiving for people thatmight not be able to reach outer arm positions or lackstrength to move swiftly.

Step 2 Using the information in a health setting.
The system obtains information on roughly what altitude
the module is held (with occasional spiking errors), andif it is rotated or not. We employ all these features in asetting that captures the user’s imagination (They believe
they’re tracked 1:1) and at the same time allows a certain
movement freedom.

Step 3 game implementation
To make the game fun we have added a very simple
scoring system that increments a counter for each correct
position reached individually for each arm. As the gameis started a progress counter starts crawling across thescreen. In 30 seconds the player has to complete as manyarm positions as possible. The final score depicts anability to perform movements and stay coordinated in a
stressing situation. It gets increasingly harder when twocontrollers are used with different target positions for

A separate thread handles all movement verification toallow the fetcher thread to gather data as fast as possible.
With each looping round it fetches the calculated
movement status. A target arm positional state is
randomly selected for each controller: UP, DOWN or© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

12

Engineering Modular Playware

each arm. To facilitate training also the disabled, thegame works just as well with a single controller.

how easy it is to pickup and start using. A developer kit
has been made available for computer science classes todevelop their own applications with the module andclient software.

Fig 1. Playing the rehab game for upper extremity training withtwo playware modules in the form of handheld balls.

Fig 3. The tablet interface showing score.

References
1. H. H. Lund, and C. Jessen, “Playware - Intelligent
technology for children’s play,” Technical Report TR2005-1, June, Maersk Institute, University of Southern
Denmark, 2005.
2. H.H. Lund, T. Klitbo, and C. Jessen, “Playware
Technology for Physically Activating Play”, Artificial Lifeand Robotics Journal, 9:4, 165-174, 2005
3. H. H. Lund, “Play for the Elderly - Effect Studies ofPlayful Technology,” in Human Aspects of IT for the Aged
Population. Design for Everyday Life. (LNCS Vol. 9194,
pp 500-511, Springer-Verlag, 2015)
4. H. H. Lund, and J. D. Jessen, “Effects of short-term
training of community-dwelling elderly with modular
interactive tiles,” GAMES FOR HEALTH: Research,
Development, and Clinical Applications, 3(5), 277-283,
2014.
5. H.H. Lund, M.D. Pedersen, R. Beck, “Modular robotic
tiles: Experiments for children with autism”, Artificial Lifeand Robotics, 13, pp. 393-400, 2009.
6. H. H. Lund, H.H. “Playware ABC: Engineering Play for
Everybody.” Journal of Robotics Networks and Artificial
Life, 3(4), 2017
7. http://www.moto-tiles.com (checked: 6/12/2018)
8. H. H. Lund. “Playware Research–Methodological
Considerations”. Journal of Robotics, Networking andArtificial Life, 1, pp.23-27, 2014

Fig 2. The interface on the tablet when playing the rehab gamefor upper extremity training with two playware modules.

4. Future iteration and extension modules
With the calculated sensor data, there are no limits to thetype of fitness games that we can make using the relative
simple building stones of subsequently employed
movements. For this game a high score of multiplayercomponent would be ideal but the real strength lies in© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

13

User-Friendly Robotics 3 - Playware
Henrik Hautop Lund, Luigi Pagliarini

Center for Playware, Technical University of Denmark, Building 326, 2800 Kgs. Lyngby, DenmarkE-mail: hhl@playware.dtu.dk luigipagliarini@gmail.com
www.playware.elektro.dtu.dk

Abstract
The design of playware based on the lessons learned from user-friendly robotics suggests a focus on a number ofdesign criteria to help achieving playware systems that can and will be used according to the Playware ABC byanybody, anywhere, anytime. The design criteria are described in this paper, and they include: modularity, explicit
immediate feedback, robustness, simplicity, one-click, design. Each criterion is described based on the lessons learned
from user-friendly robotics. We exemplify the use of these design criteria with the development of the playware
system, Moto Tiles. The Moto Tiles system is designed as a playful prevention and rehabilitation system for seniors,
who are normally not exposed to such high technology systems. The paper discusses how the design criteria allowed
for a successful design of a user-friendly system, which is now used daily by tens of thousands seniors all over the
globe.
Keywords: Playful technology, Playware, Personal Health Technology, User-friendly robotics, Seniors.

playware, the technology acts as a play force, which
pushes the user into a play dynamics. It is the interactivity
with the physical device (in the form of immediate
explicit feedback), as learned from the user-friendly
robotics, which can be designed to become the play force.

1. Introduction
The studies of User-Friendly Robotics [1, 2] led tounderstanding of how to create interactive technology,
which is targeted everyday users who have no
prerequisite to understand technological subjects such as
robotics and AI. In a number of cases as with the LEGO
Mindstorms robots, these users were young children.
From a methodological point of view, the backgroundidea is that developing user-friendly systems for young
children also leads to an understanding on how to create
equally user-friendly systems for any other age group andany other user group with little knowledge of technology.

In the same way as in user-friendly robotics, where
simple control and understandable interactivity provides
the user with a sense of adequacy and achievement tofurther interact, the interactivity of the playware provides
the user with a push (a play force) for further interactivity
with the playware, i.e. the user enters into a play
dynamics.

Indeed, it is interesting how lessons learned fromdeveloping user-friendly robotics can be applied to otherareas for other users. Based on the developed knowledge,
we founded the area of playware. Playware is defined as
intelligent hardware and software that creates play andplayful experiences for users of all ages [3, 4]. With

Hence, the playware mediates a playful interaction. In
order to best possible take advantage from the learningfrom user-friendly robotics, we develop playware
according to the concept of Playware ABC. The
Playware ABC concepts entails developing technology
for Anybody, Anywhere, Anytime by Building Bodies© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

14

Henrik Hautop Lund, Luigi Pagliarini

and Brains, which facilitates that users can Construct,
Combine, and Create [5].

systems never leaves the research labs and the prototype
stage, because they fail to put emphasis on the robustness,and such robustness is of crucial importance for theinteraction with the user. If the system has just theslightest glitch in robustness, then the user will reject thesystem as not understandable, not user-friendly or as a
failure.

2. Design guidelines
The design of playware based on the lessons learned fromuser-friendly robotics suggests a focus on a number ofdesign criteria to help achieving playware systems that
can and will be used by anybody. These design criteria
are:
•
•
•
•
•
•
•

Simplicity
In order to achieve the crucial robustness, we havelearned to focus on simplicity. It is much better to design
a simple system that works with the user than to design amore complex technological system, which then may
only work 98% of the time. But designing a simple
system is not always easy. Actually, it is often moredifficult to create simplicity than to add more complexity
to the system. To quote Leonardo da Vinci: “Simplicity
is the ultimate sophistication”. The simpler the system is,the easier it is for any user to understand and to interact
with.

modularity
explicit immediate feedback
robustness
simplicity
one-clickdesignmotivation

Modularity
The modularity is based on the learning from distributedand parallel systems such as the artificial neural network
control and evolutionary processes. The distribution
becomes physical by allowing systems to be composedon many modules that interact with each other. The
overall output of the system becomes the emergent
behavior when all the modules are interacting with eachother. Essentially, the modules can be viewed as
intelligent LEGO bricks, as we designed such in the formof I-Blocks [6]. Later, we showed modules of various
types and forms, e.g. [7, 8].

One-click
Related to the simplicity, we have learned also to focus
on one-click interaction, both in terms of the physicalinteraction with robots and interactive playware, and interms of software interfaces (apps, web-interfaces, etc.).
Whenever possible, we recommend to reduce theclicks/interactions to one in order to facilitate theunderstanding for everybody, and to be able to provide
explicit, immediate feedback.

Explicit immediate feedback
In order to make interactivity understandable for any user,
it is important that there is a feedback from the interactionof the user, and that this feedback is immediate andexplicit. This can be a recognizable visual feedback, e.g.
a color change on a tile is touched, or a recognizable
auditory feedback, e.g. a music instrument change whena Music Tiles cube is turned.

DesignThe design of the system is of utmost importance, whenusers have to interact with the system. The choice of form,material, look, etc. will influence the user acceptance ofthe system and the user interaction with the system. It
should be clear that the physical design of the LEGO
invites certain physical interactions, as well as the designof Moto Tiles, Music Tiles cubes, etc. does. With afunctional design, some proposed actions by the user are
invited by the design.

Robustness
The work with Khepera robots, LEGO Mindstorms
robots, VIKI humanoids robots, etc. within user-friendly
robotics has shown us that it is of crucial importance todevelop systems with a robust hardware. Many robotic

Motivation
When designing systems that users should interact with,the lessons learned points to the insight that the first andforemost issue is the user motivation to interact with the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

15

User-Friendly Robotics 3 - Playware

system. If the user is not motivated to interact, then it
does not matter how good the technology and innovationis, so creating for the user motivation is essential. This is
the reason to look towards play. Play is defined as a freeand voluntary activity that you do for no other purpose
than the play itself and the enjoyment that it brings you.
Hence, people have an intrinsic motivation to play. Ifsuch intrinsic motivation can be seen also in theinteraction with the designed system, then people willinteract with the system out of their own free will and forthe pleasure that the interaction brings.

pressed. The sound can be a simple click sound, ananimal sound, a piano sound, a spoken word, etc. Hence,the system has been designed for different modalities ofimmediate feedback, which are explicit and make thesystem interaction easily understandable for the user.
Robustness
The Moto Tiles were developed through a series ofprototypes over a decade. Some of the prototypes used IR
communication between neighbouring tiles, but to ensure
robustness this technological solution was skipped in the
final product development due to robustness issue with
the IR neighbor communication sometimes failing. Evenif such failure would only happen rarely, it showed inextensive user tests with the prototypes to hamper theuser interaction. Likewise, later rounds of the productdevelopment introduced static electricity protection of
the tiles at a level much higher than the industrial
standard in order to provide further robustness of thesystem.

3. Experiments
In accordance with the design guidelines, we havedesigned a number of playware systems for sport, music,
play, education, and health. These include MusicTiles
and Music Tiles cubes, I-Blocks, intelligent playgrounds,
handheld rehab modules, Moto Tiles, etc.
As an example here, we will look more closely at one ofthese, namely the Moto Tiles [9, 10, 11, 12], since theMoto Tiles system has recently become quite successful
worldwide for prevention and rehabilitation among
seniors.

Simplicity
For achieving robustness of the Moto Tiles, the productdevelopment relied upon the principle of simplicity. For
instance, to make the system simple, the use of force
sensitive resistor (FSR) sensors was substituted withsimple membrane switches, providing an on/off pressfeature instead of the analog reading from FSR sensors.
Likewise, the simplification of not installing IR neighborcommunication in between the tiles removed thepossibility to do automatic topology recognition of thetiles layout, but provided a simpler solution. Further, thetablet interface was designed to make interaction as
simple as possible.

Modularity
The original idea of the Moto Tiles was to create the
LEGO bricks for playful physical interaction, so by its
nature, it would be modular. Indeed, a Moto Tiles system
is composed of 10 homogeneous modular tiles, which
can be attached together to form different patterns on thefloor. The modules are interchangeable, so system
support can for instance be as simple as to replace one
tile with another tile. Also, if one tile should break, themodularity ensures that the system will still work, in thiscase with 9 tiles. This is important since this ensures thattherapists and care workers are always sure to be able toperform a planned training session with the tiles.

One-click
All GUI interface for the Moto app was made to provide
a one-click solution, in order to facilitate that even care
workers and seniors would be able to use the systemeasily. Also the physical tiles were designed to allow
operation with a one-click strategy. Hence, to start
playing with the Moto Tiles, all the user has to do is 1)
turn on the tiles, 2) lay the tiles in a pattern on the ground,3) turn on the tablet and select the Moto app, 4) select agame and press play. All is optimized to provide a kindof one-click strategy on tiles and tablet.

Explicit immediate feedback
The Moto Tiles are designed to provide an immediate
feedback. When a tile is pressed, the LED ring on the tile
may change colour or turn off to jump to another tile.
Also, sound from the tablet provides an immediate
feedback that signals to the user that the tile has been

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

16

Henrik Hautop Lund, Luigi Pagliarini

DesignThe design of the Moto Tiles was done in collaboration
with designers from Henning Larsen Architects toprovide a minimal functional design, and designed for“small is smart”. This design gives a system, which iseasy to store away, transport and set-up anywhere. The
charging station is designed to allow tiles to slide easily
in and to be carried away. Further, the sounds for theMoto app are designed with sound designers from D4zed
for appropriate and understandable sound feedback.

significant effects are presented, it must be noted that theplayful aspect related to the motivation is of highest
importance in the design process, and as such the healtheffects are viewed as collateral effects.

References
1. L. Pagliarini and H. H. Lund “User-Friendly Robotics 1”
ibid
2. L. Pagliarini and H. H. Lund “User-Friendly Robotics 2”
ibid
3. H. H. Lund, and C. Jessen, “Playware - Intelligent
technology for children’s play,” Technical Report TR2005-1, June, Maersk Institute, University of Southern
Denmark, 2005.
4. H.H. Lund, T. Klitbo, and C. Jessen, “Playware
Technology for Physically Activating Play”, Artificial Lifeand Robotics Journal, 9:4, 165-174, 2005
5. H. H. Lund, H.H. “Playware ABC: Engineering Play for
Everybody.” Journal of Robotics Networks and Artificial
Life, 3(4), 2017
6. H. H. Lund "Intelligent artefacts." In Proceedings of the 8th International symposium on artificial life and robotics,
ISAROB, pp. I11-I14. 2003
7. H. H. Lund, P. Marti, and M. Tittarelli. "Remixing
playware." In Robot and Human Interactive
Communication, 2014 RO-MAN: The 23rd IEEE
International Symposium on, pp. 49-55. IEEE, 2014.
8. M. Pacheco, M. Moghadam, A. Magnússon, B. Silverman,
H. H. Lund, and D. J. Christensen. "Fable: Design of amodular robotic playware platform." In Robotics andAutomation (ICRA), 2013 IEEE International Conferenceon, pp. 544-550. IEEE, 2013.
9. H. H. Lund, “Play for the Elderly - Effect Studies ofPlayful Technology,” in Human Aspects of IT for the
Aged Population. Design for Everyday Life. (LNCS Vol.
9194, pp 500-511, Springer-Verlag, 2015)
10. H. H. Lund, and J. D. Jessen, “Effects of short-term
training of community-dwelling elderly with modular
interactive tiles,” GAMES FOR HEALTH: Research,
Development, and Clinical Applications, 3(5), 277-283,
2014.
11. H.H. Lund, M.D. Pedersen, R. Beck, “Modular robotic
tiles: Experiments for children with autism”, Artificial Lifeand Robotics, 13, pp. 393-400, 2009.
12. http://www.moto-tiles.com (checked: 6/12/2018)13. H. H. Lund. “Playware Research–Methodological
Considerations”. Journal of Robotics, Networks andArtificial Life, 1, pp.23-27, 2014

Motivation
We have designed more than 35 games for the Moto Tiles
system. All games are designed to be fun and engaging
for the users. The playfulness of the games is always theforemost design criteria for the games, and in an iterative
fashion [13], the games are developed and tested
extensively with seniors to verify whether they provide
enjoyment or if they need adjustments to do so. The
enjoyment of the seniors in actual training periods with
Moto Tiles is validated through qualitative interviews
and quantitative measurements in terms of adherence tothe training.

Fig 1. Moto Tiles use in group play amongst seniors in Finland.

4. Discussion
Apart from following the design guidelines, theimplementation of Moto Tiles in society demanded an
extensive effort in scientific effect studies of both thephysical effect and cognitive effect on seniors fromplaying with the Moto Tiles. Even though such important© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan17

User-Friendly Robotics 1 – AI Software
Luigi Pagliarini1,2, Henrik Hautop Lund1
1

Centre for Playware, Technical University of Denmark, 2800 Kgs. Lyngby, Denmark2
Academy of Fine Arts of Macerata, Via Berardi, 6, 405111 Macerata, Italy
luigipagliarini@gmail.com

Abstract
In this paper, the first part out of three, we present the trajectory of decades of research in AI and Robotics byhighlight the continuity of our method that has always taken into consideration the common user’s point of view asa priority. We call such an approach User-Friendly Robotics, and we hereby unravel it by describing our path stepby-step, to see on one side its evolution and on the other its methodological continuity. Therefore, starting from theearliest AI-based applications, which are mainly described in this paper, we slowly move-up towards the first
Robotics ones to finally describe the last results of our research. We discuss how this User-Friendly Robotics
approach leads to solutions that fit particular purposes in society and lead to useful user interaction withtechnological developments.

Keywords: Playful technology, Playware, User-friendly robotics, Artifcial Intelligence, Artificial Life.

specific market target, and has to accomplish more thana single goal.
Anyway, the present paper do not aims at being atheoretical one in which we claim a specific hypothesis
for HCI (i.e. Human-Computer-Interface) and, on thecontrary, it is an attempt for a reportage of ourinvestigations that, one after the other, empirically took
us to an successful applicative model of user-oriented
tools.

1. Introduction
Approximately a jubilee ago our path towards a userfriendly high-tech tools accidentally started and,basically, has never stopped evolving itself. We
improved the criteria of robustness, efficiency andfriendliness themselves, and enlarged the domain of
application from software to hardware. In the following
paper, we describe the essential steps we took, and thetechnology we developed along the way. We also
discuss and highlight the fundamental principles andcriteria that, in our opinion, make an AI based artefact, auser-friendly one.
Indeed, we believe that it might be quite hard to definesuch a field of competencies since it is very
multidisciplinary and it includes a quite large set ofdifferent aspects, such as user-interaction flow,
comprehensibility, simplicity, efficacy, efficiency,
aesthetics, and etc. Therefore, this on going processreaches and includes very many different aspects andperspectives that are not always easy to integrate in one
single artefact, especially if such an artefact is oriented
to more than a single activity, has to reach more than a

2. AI- based user-friendly software
In 1993 we were part of the research group on Artificial
Intelligence and Artificial Life that was commissionedby the Museum of Science of Naples to shape threesoftware that could explain concepts like Neural
Networks, Genetic Algorithms and Guided Evolution toa as large population of users as possible. The
production of such AI based software - named“Educational Artificial-Life Lab” - was an extraordinary
challenge because for the first time we were bringing© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan18

Luigi Pagliarini, Henrik Hautop Lund

brand new scientific concepts to the public and we had
to find an original way to interface with it.

Network it was thought to guide the users to idea of
Genetic Algorithms, therefore, the Darwinian laws. In
Pet Evolution the user has to develop the behaviour ofthe same simple neural network control system agent(i.e. 8 input neurons, and 2 output neurons) so that theagent navigates quickly within its environment without
running into the walls or the central obstacle.

3. Shape You Brain
The first one [1,2], Shape Your Brain (Figure 1),
consisted of a simple neural network control system
made of eight sensors as sensory input for 8 input
neurons, and having 2 output neurons that activate theagent In Shape Your Brain, the task was to develop acontrol system, a Neural Network in the shape of a
Perceptron, so that the agent navigates quickly aroundin its environment without running into the walls or thecentral obstacle.

Figure 2. Pet Evolution, software

Instead of a manual tuning of the NN, in this case, asimulated evolution process is used as a developmental
tool. A whole population of nine agents is constructed
randomly as an initial population. Each agent lives for anumber of time steps in the environment with walls andthe central obstacle, and the behaviours of all nine
agents are shown on the screen. Each agent moves
around in the environment leaving a tiny line on its
trajectory. From the initial population, the user has toselect the three agents that show the best behaviour
according to the task of moving around in theenvironment. Each of the three selected agents makes
three copies of its neural network control system, where
mutations are applied to a small number of theconnection weights. The total of nine mutated copies
constitutes a new generation that is permitted to live inthe environment. The user selects among these agents,
the selected agents reproduce a new generation, theagents of the new generation live in the environment,
the user selects again, etc. One experiences the game,
Pet Evolution, in a simple way. This tells the carefuluser that evolution is capable of finding adaptive
solutions to difficult problems by such methods as
selective reproduction and mutation only. Pet Evolution
was one of the first Artificial Life Games ever built in
Artificial Life history as well as one of the first
Edutainment Electronic Games.

Figure 1. Shape Your Brain, software

The user observes each single step of the agent and foreach step is allowed to change the connection weightsof the neural network that constitute the control system.
In this way, the user experiences how behaviours thatare appropriate under some circumstances (e.g., faraway from obstacles) are not appropriate under othercircumstances (e.g., close to obstacles). To succeed, theuser must construct a global behaviour that lets theagent behave appropriately under all circumstances.
This is a rather difficult task, since the user has to adjust
all connection weights in the neural network that control
the agent's. Probably, Shape Your Brain is one of thefirst Edutainment Electronic Games based on AI everbuilt.

4. Pet Evolution
The second AI-based software[3], named Pet Evolution(Fig. 2) , was similar to Shape Your Brain but, insteadof focusing on the direct manipulation of a Neural© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

19

User-Friendly Robotics 1 - Playware

5. Survive!

used guided artistic experience. The result was theArtificial Painter (AP, Fig. 4) a software package thateuses a Genetic Algorithm on Neural Networks to
produce artistic images. The evolution of pictures was
based on the user's aesthetic evaluation of a number of
pictures shown on the screen. The AP model originatesfrom an inspiration derived from bio-image techniquessuch as Computed Tomography (CT), Positron
Emission Tomography (PET), and Single Neuron
Records. All these techniques visualize physiologicalactivities by means of coloured pictures that are used inresearch and diagnostic [5,6].

The third software, called Survive! (Fig. 3), was a single
person game and consists of a little world where a few
artificial organisms and some food pieces are placed.
The organisms are small neural networks that navigate
in the environment. When an organism reaches a foodpiece, the food piece is ingested and disappears from theenvironment. An organism reproduces a child when acertain number of food pieces have been ingested. The
number of food pieces ingested by an organism isshown by the organism's size.
If an organism lives for a long period without ingesting
any food pieces, the organism dies. When the gamestarts the only thing a player can do is to administrate acontainer of food resources in such a way as to keep the
population alive as long as possible.

Figure 4. Artificial Painter, software

In a similar way, in AP the techniques of single neuron
records are applied on an Artificial Neural Network(ANN) to produce pictures. The Genetic Algorithm isused to lead an evolutionary process that gives the user
the possibility of choosing the 'desired' image, i.e.
"ANN's brain", amongst many. Chosen images (andunderlying ANNs) will be reproduced and, eventually,
mutated and the process goes on until the user has
reached a satisfactory aesthetical result. Artificial
Painter, represents one of the first examples of image
evolving simulators in the Electronic Art history (thefirst one based on neural networks). AP is also
considered a milestone in the ALife and AI fields
because of its original and innovative conception andfor its application in medical field [7].

Figure 3. Survive!, softwareThis process is made more difficult by the fact that the
population grows or risks extinction in relation to thequantity of food released and it is very hard to find theright rate of resources to be given. Obviously, this kindof problem-solving enables a child's mind to understandand cope with two important and up to date bioecological topics. The first is energy management; thesecond is the meaning of demographic aspects of life onearth. For these reasons, this game was considered ashighly educational in regard to encouraging respect for
wildlife and habitats, in such a way that can be termed"easy to understand" and "easy to interact" with [4].
Survive! was one of the first Artificial Life Games everbuilt in Artificial Life history , too, as well as one of thefirst Edutainment Electronic Games.

7. Face-It
One further step towards a user-friendly AI softwareexperience was determinate by the development ofFace-It, in 1995.
Face-It software package used a Genetic Algorithm toevolve pictures of facial expression. The evolution of
pictures was based on the user's evaluation of a numberof facial expressions shown on the screen. The Face-It

6. Artificial PainterAfter this experience, in 1994, we were financed to get
involved in a new challenge and design an AI-based and© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

20

Luigi Pagliarini, Henrik Hautop Lund

model originated from an inspiration of Leonardo Da
Vinci's Facial Studies and Artificial Life techniques ofUser Guided Evolution of Images, plus Cognitive
Science Studies of Recognition of Facial Expressions.

dynamics with human being, regardless of their age andlevel of competences [7], both in computer science andin general. It is self-evident that since the beginning weconcentrated on hiding the level of complexity targeted
application and focused on the users’ cognitive
approach to generic tools, trying to design
immediateness within our interactive model. As it will
be shown in the following parts we kept on the same
path while getting out of virtuality and interfacing withphysical agents, robots.
References
1. L. Pagliarini, Insegnare 'Biologia' attraverso l'uso di
software di Vita Artificiale. ECOSOFT International
Conference. Perugia, 15-19 Novembre 1994.
2. Miglino O., Calabretta R., Pagliarini L., Nolfi S., La vita
artificiale come strumento per la didattica delle scienze.
Technical Report, Reparto Sistemi Neurali e Vita
Artificiale, Istituto di Psicologia, Roma.1994
3. Pagliarini, L., Rubinacci, F., Lund, H., Sementina, C. I
modelli simulati aiutano ad apprendere le tradizionali
discipline scolastiche? Atti del Congresso della Sezione
di Psicologia Sperimentale (Proceedings of "ItalianCongress of Experimental Psychology), pp. 76-78,
Editrice Democratica Sarda, Sassari, Italy 2001.
4. Miglino, O. Rubinacci, F. Lund, H. H., and Pagliarini, L.
"Using artificial life to teach evolutionary biology", in
Cogn Process 5:123-129 (2004).
5. H. H. Lund, L. Pagliarini, O. Miglino Artistic Design
with Genetic Algorithms and Neural Network. In
Proceedings of First Nordic Conference on Neural
Network, 9-13/1/95, Vaasa, Finland.
6. H. H. Lund, L. Pagliarini, O. Miglino The Artificial
Painter. In Proceedings of ICANN 1995 International
Conference on Artificial Neural Network, 5-13/6/95,
Granada, Spain.
7. L. Pagliarini, H. H. Lund, O. Miglino and D. Parisi
Artificial Life: A new way to build educational andtherapeutic games. In Proceedings of ALife V
International Conference, NARA, 15/18-V-1996, Japan,
MIT Press(1997).
8. Pagliarini,L., Parisi, D., Il Progetto Face-It. XV
Congresso Nazionale della Ricerca di Base in Psicologia(A.I.P.), Capri, 30-IX/2-X 1996.
9. Rubinacci F, Paparone C, Pagliarini L, Matarazzo O,
Face-It: Uno strumento di Vita Artificiale per lo studiodella genesi delle espressioni facciali delle emozioni, Attidel Congresso Nazionale dell'AIP - Sessione diIntelligenza Artificiale e Modelli Connessionisti,
Bellaria-Rimini, 16-18 Settembre 2002 (abstract).

Figure 5. Face-It, software

A Genetic Algorithm works on a population of
genotypes. In Face-It, the genotype of each individual isrepresented by a bit string codifying the values of 33
continuous variables that describe 400.000 possible
expressions of, approximately, the same face. Face-It
constructs a whole population of individuals. Initially,
the genotype of each individual will be set randomly[8,9]. Each individual builds a face that is shown on thescreen in front of the user. According to the genotype
different faces will appear. The user selects some of theexpressions for reproduction by clicking them on withthe mouse. Each selected individual (i.e.: expression)
will be cloned a fixed number of times and a newgeneration of faces will appear on the screen. Eachclone will be mutated in some randomly chosen parts ofthe genotype (bit string). The user can decide theaverage number of genetic mutations. The selection andcloning process can continue until one or moresatisfactory expressions are obtained. Face-It, was aclear example of reducing an incredibly complex event,such as face expressions, towards an easy to understandand use artefact.
8. Discussion
This first set of studies clearly traced a way on how tointerface complex concepts and machine/software© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

21

User-Friendly Robotics 2– Interfacing Robotics with any user
Luigi Pagliarini1,2, Henrik Hautop Lund1
1

Centre for Playware, Technical University of Denmark, 2800 Kgs. Lyngby, Denmark2
Academy of Fine Arts of Macerata, Via Berardi, 6, 405111 Macerata, Italy
luigipagliarini@gmail.com

Abstract
In this paper, the second part out of three, we continue presenting the route of our long experience in interfacing AI
and Robotics with the public. This paper, in particular, highlights a crucial step we experimented when creating
computer based software architecture so to let the common user easily interact with complex behaviour robotics.
We define such an approach User-Friendly Robotics, and we hereby describe the principles and the techniques weused to evolve its methodology reporting here the most important case studies we went through, as examples. We
then analyse the impact of such method on both science and educational field. Finally, we discuss how this UserFriendly Robotics approach leads to solutions that fit particular purposes in society and lead to useful userinteraction with technological developments.

Keywords: Playful technology, Playware, User-friendly robotics, Artifcial Intelligence, Robotics Interfaces.

different aspects, such as user-interaction flow,
comprehensibility, simplicity, efficacy, efficiency,
aesthetics, lightness, and etc. Indeed, this process takes
into consideration very many different perspectives andaims at including many different features, not always
compatible or easy to integrate. Therefore, sometimes, it
is a big challenge, especially when one single artefact orinterface is oriented to more than a single activity, has
to reach more than a specific human target, or toaccomplish more than a single goal.
In this paper we present an intermediate step we took
when moving the idea of robotics as a field under thecontrol of very experienced users to directly interface
with children and not-experienced users. In suchpassage we inherited the experience maturated indesigning AI-based simulators and created different
methods to interface with robots in an easy and playful
way. We believe that such an example can be of great
inspiration for those who are investigating on how toconceive and build new and successful applicativemodel of user-oriented intelligent tools.

1. Introduction
It is now about 25 years that we are immersed in the
study of robots and dynamic systems interfaces. This
path almost accidentally started in 1993 [1] hasundergone continuous evolution going through ate least
three main steps, from full virtual world to mixed
realities, to the present where we put all of our effort in
hiding the software component as much as possible. At
the same time we in parallel enhanced the criteria ofrobustness, efficiency and friendliness of our method,and enlarged the domain of applications. In thefollowing paper, we describe the intermediate step wetook, when moving the control and the evolution ofrobots behaviour from the researcher/scientist to thecommon user, trying to depict the fundamental
principles and criteria that, in our opinion, make an AI
based artefact, a user-friendly one.
We strongly believe, indeed, that the future of high-tech
and AI based tools well be grounded on screen
disappearance. This consideration leads to an approach
toward the idea of “interface” that is necessarily
multidisciplinary and it includes a quite large set of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

22

Luigi Pagliarini, Henrik Hautop Lund

2. Interfacing robots with human beings

robot as a little pet moving from the belief that children(and, in general, human beings) would like to play withrobots in the same way they do with animals, using
direct interaction. Therefore, in our approach, thechildren-toybots interaction focused only on the"behaviour''. In order to permit children to build
interesting mobile robots, we tried to avoid any form of"information
processing
methodology''(i.e.:
programming). Artificial Life and Evolutionary
Robotics techniques represented a useful solution forour goals. For a specific task designed by the child, weallowed an evolutionary process to design theappropriate robot body plan that the child could
assemble. In co-evolution and/or through a user-guided
genetic algorithm, the controller of the robot wasevolved and downloaded to the robot. The user-guided
genetic algorithm consisted of representation of therobots' behaviours on the screen, and the user performed
the selection based on the user's own judgement. Retraining in the real world could be done throughreinforcement learning.

In 1997 we started to face a new challenge. The
continuous hardware evolution was about to bring into
the scientific community a new set of computer boards like Khepera, as first and LEGO MINDSTORMS andArduino, later – ideal to assemble autonomous andmobile robots in a relatively cheap way. It clearly was
the beginning of a sort of Cambrian age for robotics.
We, as part of this on going process started working at
the design of new algorithms and to implement newuser-oriented interfaces.
3. ToyBot
After such a deep experience in user-friendly AI-based
software, our focus moved to robotics applications
where an easy to understand and use tool could be
combined with robots in action. Since 1997, we
produced a quite a amount of artefacts and some ofthem can be considered exemplar studies that, in a more
direct or undirected way, have had a large number ofemulations all over the world. In here we will describe
the most important ones.

Figure 2. ToyBot, the Robot
The Toybots' behaviours were controlled by Neural
Networks, that model the brain of pets. At the end of
our project we had children that could develop simple
behaviours such as obstacle avoidance, following or
avoiding lines on the floor, attraction or fear of light, etc.
for LEGO robots. The ToyBot project was a real
pioneer in children-oriented games based on robotics.
Today a big share of the toy market is taken up by
similar approaches.

Figure 1. ToyBot, the Simulator for Robots
In 1997 [2] we developed the ToyBot project (Fig. 1 andFig. 2), a research collaboration between the Mobile
Robots Group at the Department of Artificial
Intelligence, University of Edinburgh (UK) and theInstitute of Psychology at National Research Council,
Rome (Italy). At that time, a strong limitation of
traditional robot kits led to the assumption that to build
an intelligent toy-robot (toybot), it was better toprogram the robot as a sort of "computer with wheels''.
On the opposite, ToyBot project was based on adifferent point of view: we considered a mobile toy

4. The LEGO Stadium
The LEGO Stadium, or Stade de Victor LEGO (Fig. 3
and Fig. 4), was a robot soccer show exhibited, in© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan23

User-Friendly Robotics 2 – Interfacing Robotics

occasion of RoboCup '98, in "La Villette", La Cite desSciences et de l'Industrie, Paris (France) [3]. This was
actually the very first time the LEGO MINDSTORMS
Robotics Invention System was shown in public.

MINDSTORMS RCX, had an autonomous behaviour
guided by a behaviour-based algorithm.
A placed from above camera was filmed the stadiumand downloaded the information to a colour-detector
card which, in turn, furnished information on eachplayer and ball positions to each robot that was thenable to take, step by step, its strategic decision. The
resulting soccer game was an exiting one full of action
and lots of goals. The LEGO Stadium show at "La
Villette" was a great media event and it is calculated
that it was shown to more the 200 million people allover the world, through television shows. A 'preview' of
this game was also given as part of a Stockholm
Cultural Capital '98 event called Global Tendency
Machine in Stockholm in May 1998.
5. Interactive LEGO FootballMoving from this experience we had the intuition thatsuch scheme for behavioural-based robotics could be
opened directly to the users/public and a great set ofgames or professional activity could have been designed.
Indeed in 1999, we started building the Interactive
LEGO Football platform (Fig. 5, 6, 7, 8), a set-up that
aimed to be a fully personalized and interactive robotics
experience for any user [5,6,7].

Figure 3. The Stade de Victor LEGO

The LEGO Stadium was fully built with LEGO pieces.
The LEGO Stadium consisted of: 1.500 LEGO
Spectators making the Mexican Wave, two LEGO
Cameramen Operators running up and down along thefield and filming the matches, an automatic LEGO
Score Board, Rolling Commercials, a synchronizedaudio system underlining the crucial events in thematches and two of the LEGO Soccer Players, thegoalkeepers, that were controlled by LEGO Barcodeswith pre-programmed behaviour [4].

Figure 5. Playing the game at MIT Medialab

Therefore, the Interactive LEGO Football (I.L.F.)
project was a one-on-one robot soccer game to beplayed with real robots. The robots use only simple
sensors and motors, but a special smart, electronic ballis used to allow the robots to detect it. The robots areprogrammed with a user-guided behavior-based
approach. I.L.F. robot soccer game for children consistsof an environmental set-up (field, ball, etc.), LEGO

Figure 4. The Robot players

Moreover, inside the soccer field we had four LEGO
Soccer Players, two for each team, that were able to"introduce" them selves and "sing" their own national
anthem. These players, controlled by LEGO

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan24

Luigi Pagliarini, Henrik Hautop Lund

Mindstorms robots of different colours,
programming environment.

andainto the robot and go and play the game against his/her
opponent.
The programming environment has different levels ofdifficulty that can be accessed accordingly with user’scompetences. The game was tested at both RoboCup'99
in Stockholm and at the MIT Media Lab MindFest '99,
and children from 7 years and up were able to maketheir own LEGO MINDSTORMS robot soccer players
to score within 20-30 minutes and ready to play in small
tournaments from within 30 to 60 minutes. I.L.F., under
the name of RobocupJunior [8], has become a milestone
in children oriented robot games. The same game has
also been used in the European RoboCup 2000 event in
Amsterdam (Holland) and, in a slightly different version,in RoboCup 2000 in Melbourne (Australia), as part of
the RoboCup Jr. setup. At the moment the game isprobably one of the most widespread platforms used for
educational purposes in the world [9].

Figure 6. A Robot player, downloading the control system via infrared

The LEGO robots are premade ones (with two lightsensors for ball detection and a switch sensor forcrashing detection) so that the children can concentrate
their attention on "programming" the behaviour. To letthem do so, we built up a behaviour-based algorithm,
shown on a very easy-to-understand interface, withphotographs and videos of the robot behaviours, where
the children have to choose the behavioural sequence
the robot will play.

Figure 8. RobocupJunior competitions, nowadays

This step represented a sort of breakthrough in robotics
since it empirically demonstrated to everybody that
there are methods to build and interface robots in such auser friendly way that anybody can access robot’s
programming and that such “access” can be placed at
different levels.
6. Discussion
This second set of studies evidently traced what we candefine a breakthrough method on how to interface
complex concepts and machine/software dynamics withhuman being. Since the beginning it become clear to apretty large number of researchers that suchmethodology was one of most innovative way to

Figure 7. The Behaviour-Based Interface

It is a lot like being the coach of the team and real
programming is not needed. Once the child has madehis/her choices, he/she can download his/her program© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan25

User-Friendly Robotics 2 – Interfacing Robotics

interface robots with people, regardless of their age andlevel of competences. Indeed, the ILF method was soon
inherited by so many research labs, school programs
and is, at the moment, one of the most widespread
concept for introducing children to robots programming.
The basic idea of hiding the level of complexity of thetargeted application and focus on the users’ cognitive
possibilities was, and still is, the right way to go when
designing a HCI model.
At this point, there were only a couple goals to beachieved:
A. To enlarge the domain of application of suchmethodology;
B. To reduce the number of iterations with a computer
screen as much as possible.
In the third part of this essay on User-Friendly Robotics
we will show the further steps taken towards thisdirection.
References
1.
2.

3.

4.

5.

6.

7.

8.
9.

L. Pagliarini and H. H. Lund “User-Friendly
Robotics 1” ibid
Miglino O, Lund HH, Pagliarini L, Toybots.
Allevare robot per apprendere a governare unprocesso evolutivo. In Tecnologie Didattiche ITD,
CNR (2002).
Lund, H. H., and Pagliarini, L. Robot Soccer with
LEGO Mindstorms. In Asada and Kitano (eds.)
RoboCup'98,
LNAI
1604,
Springer-Verlag,
Heidelberg.
Lund, H. H., Arendt, J. A., Fredslund, J., andPagliarini, L. Ola: What Goes Up, Must Fall Down.
In Proceedings of Artificial Life and Robotics(AROB'99), ISAROB, Oita.
Lund, H. H., and Pagliarini, L. RoboCup Jr. with
LEGO Mindstorms. In Proceedings of Int. Conf.
Robotics and Automation 2000 (ICRA2000), IEEE
Press, NJ.
Lund, H. H., and Pagliarini, L. (2001) Edutainment
Robotics: Applying Modern AI Techniques. In
Proceedings of International Conference on
Autonomous Minirobots for Research and
Edutainment (AMIRE-2001).
Lund HH, Pagliarini L, Edutainment and Robotic
Games for Children. In Proceedings of the 2nd
IFAC conference on Mechatronic Systems
Mechatronics 2002, Berkeley, California, USA,
December 9-11, 2002. Elsevier.
https://en.wikipedia.org/wiki/RoboCup_Junior
http://junior.robocup.org/events/© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan26

A New Machine Learning Algorithm for Weather Visibility
and Food Recognition
Young Im Cho1, Akmaljon Palvanov2
1, 2

Gachon University, Korea
1yicho@gachon.ac.kr
akmaljon.palvanov@gmail.com

2

Abstract
In this paper, we discuss two fundamentally different problems, such as visibility estimation and food recognition
using convolutional neural networks. Our focus was revealing limitations of deep convolutional neural networks andmethods to tackle with those drawbacks. We propose two dissimilar data-driven approaches for each tasks. The first
proposing model designed to estimate the visibility distance from images captured via digital camera. For this task,
we use images taken from CCTV cameras as inputs. The second model implemented to recognize dishes from limited
images. Additionally, we implemented a graphic user interface as well as special web page that allows users
flexibility to operate all operations in real-time. This paper describes both models and other essential details.
Keywords: atmospheric visibility, convolutional neural networks, CCTV, graphic user interface, recognition.

outputs. The key factors for this ability is large labeleddatasets, which support the learning process and novel
network architectures with hundreds of millions ofparameters. Nevertheless, it is difficult to find suitabledataset always. In this case, data augmentation techniques
are widely used. In this work, we study the most common
methods to enlarge a dataset consists of limited images.
In order to evaluate the CNNs effectiveness using
artificially extended dataset, we selected limited images of
Korean dishes. Since input images different in terms ofshape, texture, size and color as the Korean dishes lacks
any kind of generalized layout.
In contrast, to other types of food such as Indian or
Italian, it is more complicated recognizing the images of
Korean dishes as the following reasons: images in the same
category appear differently; it is complex for modeling dishimages because of noise, likely, various backgrounds andtextures; a number of high quality and proper images islimited.
We implemented a new deep CNN model for this task.
Our model has shown promise results even on a large anddiverse image datasets with non-uniform backgrounds.
In the next section, a brief summary of related studies isreviewed. Then, Section 3 presents details of both models
and experiments as well as achieved results is given in
Section 4. Finally, Section 5 concludes the paper.

1 INTRODUCTION
Visibility is a degree of ways nicely an observer canview a scene. This consists of how some distance you cansee within the surroundings as well as the ability to see thetextures and colors of the scene. Haze is the inverse ofvisibility that is the measure of the inability to view a scene.
Nobody would intentionally agree that poor air great is
acceptable. The advantages of correct visibility are apparent.
Visibility is a primary and enormously apparent indicator ofpreferred air excellent. Visibility is distinctly easy torecognize via public. Many humans regard suitablevisibility as a figuring out thing for the satisfactory of out ofdoors existence. As an end result, the community can alsodecide the effectiveness of environmental control policies
to improve air satisfactory by means of visibility.
Experts use special equipment to provide measurementof visibility. These tools themselves and their installation
processes are ample expensive and measuring range is also
limited. Additionally, the factors mentioned above strongly
affects the outcomes.
Hereby, we propose camera and convolutional neuralnetworks (CNN) based approach to estimate visibility as
combinations of these methods gives strong superiorities.
On the one hand, comparing to tools like visibility meter
camera-based visibility measuring is more close to visual
method. Also, the technique does not cost expensive,
weather conditions like dust and fog etc. can be artificial
removed while measuring and can be achieved very
accurately results.
CNNs are good at classification and recognizing images
as they can learn the relationships between inputs and2 RELATED WORK
Shengyan et al [1]. proposed visibility estimation basedon meteorological laws using digital images. They useddeep learning model to artificially extract features from
input images. Another approaches for estimating visibility© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan27

Young Im Chohello1, Akmaljon Palvanov

proposed by Sami et al [2] essential properties of thesestudies are using single camera and evaluate visibility on abasis of daytime light from the input images. In the field of
meteorology, one of the main law is Koschmieder’s law andmany researchers used this function to estimate visibility inboth cases, using digital images and using special
equipment for visibility estimation, these works have been
done in Du et al [3]. Since all these methods use images, as
an input quality of the images is vital to get better results,
means that degradation of images strongly restricts
accurately estimation. This requires essential extra stepprior estimate visibility the step is enhancement of thequality of input images. For this situation Dark Channel
Prior (DCP) method is widely used. For instance, in poor
visibility conditions enhancement of color images proposedby Keong et al [4], in this study, several visibility
conditions tested using synthetic images taken via airborne
camera. Another effective use of DCP can be seen in
Kaiming et al [5]. they used a haze-imaging model toestimate the thickness of the haze then recover quality of
the image. Convolutional neural network based dehazing
method Boyi et al [6] relies on the re-formulated
atmospheric scattering model. In next section we will
detailed explain our approach and our techniques.
There have proposed several CNN based food image
recognition algorithms. A system based on deep CNN
namely DeepFood [7] by Chang et al. analyzes the foodimage taken by mobile devices and the performanceoutperformed of previous proposed dietary assessments. In[8] by Yuzhen, five layer CNNs introduced to recognize
5822 food images. It has only 10 classes and overall 90 %accuracy obtained. Another model FoodNet [9] by Paritosh
et al. addressed food images recognition using multilayer
CNN using the large dataset ETH Food-101. All models
mentioned above take advantage of deep CNNs and requirelarge sets of images. Therefore, we addressed to the
problem of enlarging dataset from limited images.

Fig. 1. Samples from the visibility dataset
Atmospheric visibility measurements for those locations
was also available. The visibility values are register in therange from 0 to 20000 meters. We assigned 21 classes foreach 1 kilometer of visibility (including 0) by dividing thevisibility by 1000 and rounding the values. Because of thepoor image quality during the night, we decided to use only
images from daytime. Examples shown in the Fig. 1.
Limited dataset is a severe problem in case deep CNNs.
As we mentioned above, CNNs mostly data-driven andrequire proper set of images. To overcome this problem, weare proposing an algorithm, which takes single image andapplies several augmentation techniques sequentially inorder to gain multiple images. The proposing algorithm cangive desired number of unlike images with maximum
efficiency.
Initially, the system takes single or many images as
input(s). Then, normalizes and adjusts the contrast andbrightness of the input(s). The normalized image used forgenerating new images according to the proposing
algorithm (image generating functions can be chosen byusers).
Some generated inputs might lose their quality during
the second step. The reason for reducing the quality is thatimage resolution will change during the process andzoomed/cropped/scaled image need to quality enhancement.
After gaining desired outputs, some images will processusing auto-encoder model or image enhancement methods.
Pre-final, quality recovered images will be processedagain in order to reduce the similarity of images, to do so
we apply random Gaussian noise. In addition, it is donewith the contrast, sharpness, color and brightness changes
randomly. In the Fig. 2 the proposing dataset enlarger
algorithm described.

3 METHODS
3.1 Datasets
For the task of visibility estimation, we used a set of
CCTV images taken by stationary surveillance cameras
from four different points in South Korea, near the sea side.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan28

A New Machine Learning

Fig. 4. The logical structure of the CNN model for visibility
estimation

Fig. 2. The proposing dataset enlarger algorithm.
Since, there is no publicly available image datasets for
Korean dish recognition, to train our complex foodrecognition model we have collected a new dataset. The
resources for the dataset are taken photos and web images.
Our group in daily life collected some partial of the images
in this dataset. The Fig. 3 shows example images from thedataset. We selected 50 images for each 23 classes. Images
are artificially extended later using the proposed algorithm
above, resulting in totally 92000 images.

feature maps using kernel size 3x3. After the last maxpooling operation, feature maps are sent to the dense layer,
which includes 1024 neurons. Finally, the last layer isSoftmax layer that produces the output. Visibility iscomplex phenomena as dense fog or low-visibility
presented images have almost no features. Fog occupies
entire image space as a repercussion everything is abstract,features of objects hidden. In this case, deep CNNs strongly
overfits and produces wrong prediction. Therefore, we use
only two convolution layers to avoid overfitting.
Food recognition
Schematic architecture of the model is given in Fig. 5.
Initially, the first layer receives 150x150 image as an input
and produces 32 feature maps by convolving with thekernel size 9x9. Next, max-pooling operation reduces thedimensionality. The kernel size of 7x7 is used to convolvean output of the previous layer, resulting 64 maps offeatures. Third and fourth convolution layers produce 128
and 256 feature maps respectively, each followed by maxpooling operation. After the last max-pooling layer, w dense
layers employed sequentially each with 512 neurons.
Finally, there are 23 Softmax neurons in the output layer,
which correspond to the classes of dishes.

Fig. 3. Sample images from the food dataset
3.2 Networks
Visibility estimation
Our first model estimates visibility from day time
images. Especially, it can classify accurately if there is afoggy weather condition is illustrated. A logical structure of
the model for visibility estimation can be seen in Fig. 4.
The first convolutional layer produces 32 feature maps
using 5x5 convolutional kernels followed by max-pooling
operation. The second convolutional layer produces 64

Fig. 5. The logical structure of the CNN model for foodimage recognition© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan29

Young Im Chohello1, Akmaljon Palvanov

4 EXPEREMENTS AND RESULTS

After training the model, we exported the model into acompressed file for deployment. We developed a Java basedoffline graphic user interface as well as a Java EE based
web page for testing the model in real-time. A web based
application is designed to run on a remote server accessible
over a TCP/IP network such as LAN or the Internet. The
application takes a. jpeg or .png image as input, uploads itto the server, loads the trained neural network model andreturns the inference result to the clients’ web browser.

Our experiment performed on a high-end server with
64GB of RAM equipped with two Nvidia GeForce GTX
1080 Ti GPUs. We conducted the experiment using
Tensorflow machine learning framework. The training
process is computationally costly and takes much time. It
reads inputs from the memory then shuffles all images thenstarts training choosing random batches.
A training accuracy presents how perfectly fits every
input image to the training data whereas a loss measures theinconsistency between predicted value and actual label.
The evaluation of the accuracy, as well as the loss
function over the course of training, is given in figures
below.

5 CONCLUSION
To sum up, our studies showed that one of methods toestimate visibility can be done by deep CNN model. I
addition, we showed efficient methods to enlarge dataset oflimited image to train complex models. We implemented
the deep CNN model that recognizes food images andtrained it using extended dataset.

ACKNOWLEDGEMENT
This

paper

is

supportedby

NRF(no:

2018R1D1A1A09084151).

Fig. 6. The accuracy functions over the course of training.
The horizontal axis represents training epochs; the verticalaxis is the training accuracy functions

REFERENCES[1] Shengyan L, Hong F, and Wai-Lun L (2017),
Meteorological visibility evaluation on webcam weather image
using deep learning features. International Journal of Computer
Theory and Engineering[2] Sami V, Jari H (2015), Image based visibility estimation
during day and night. Computer Vision - ACCV 2014, pp. 277-289[3] Du K, Wang K, Shi P, Wang Y (2013), Quantification of
atmospheric visibility with dual digital cameras during daytime
and nighttime. Atmos, pp. 2121–2130[4] Keong, John P. Oakley (2002), Enhancement of color
images in poor visibility conditions. Proceedings 2000
International Conference on Image Processing, Canada[5] Kaiming H, Jian S, Xiaoou T (2011), Single image hazeremoval using dark channel prior. IEEE Transactions on PatternAnalysis and Machine Intelligence[6] Boyi L, Xiulian P, Zhangyang W, Jizheng W, Dan F (2017)
AOD-Net: All-in-one dehazing network. IEEE International
Conference on Computer Vision[7] Chang L, Yu C, Yan L, Guanling Ch, Vinod V, Yunsheng
M (2016), DeepFood: Deep Learning-Based Food Image
Recognition for Computer-Aided Dietary Assessment. ICOST[8] Yuzhen L (2016), Food Image Recognition by Using
Convolutional Neural Networks, CSE803 final project[9] Paritosh P, Akella D, Bappaditya M, Puhan N (2017),
FoodNet: Recognizing Foods Using Ensemble of Deep Networks.
IEEE Signal Processing Letters, pp. 1758-1762

As we can see from Fig. 6 during training, our accuracy
on random batches approaches 90% of successful visibility
classification based in image similarity.

Fig. 7. The evolutions of the loss. The horizontal axis
represents training epochs; the vertical axis is the loss
function
It is clear from the loss function that the loss
successfully reduced to approximately 20%. To perform theevaluation, we developed a java application that selects
images from the reference image database and performs anevaluation on each of them separately. The proposing
method achieves about 81% classification accuracy on
random sample images, which is quiet promising for thisproblem.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan30

Affective Interactive Systems
Kaoru Sumi

Future University Hakodate,
Hakodate,Hokkaido 041-8655, Japan*

E-mail: kaoru.sumi@acm.org
www.fun.ac.jp

Abstract
Known as Media Equation, the relationship between humans and artifacts is similar to the social relationship between
humans and humans. For example, when a human is helped by an artifact, the human wants to thank the artifact andgive something in return. Like humans, it is important for artifacts to recognize emotions and express emotions wheninteracting with humans. This talk introduces a technique to recognize and express emotions in human and artifact
interaction, and shows examples of how effective artifacts are when using emotions.
Keywords: affective computing, affective learning, human agent interaction

agent behaves in a disagreeable manner [3]. It is said thatthe relationship between humans and artifacts is similar
to the social relationship between humans and humans.
Because of this, for artifacts to be used successfully, it isimportant that the artifacts both recognize humanemotions and show emotion.
Therefore, our research group has been studying
interaction using emotions. If artifacts can recognize
human emotions dynamically, it will be possible toprovide very fine-grained services.
In recent years, we have become able to identyfy
emotions using various devices in various ways. For
example, devices which can detect things such as heartrate, skin potential, facial expression, brain waves, andgestures have become familiar. There are various
methodologies
foremotionrecognition,butmethodologies for emotional expression are limited. For
example, since many robots do not change theirThis talk introduces human and artifact interaction,
which occurs with computer systems, virtual agents,
robots, etc. What are the information elements that areimportant for us to consider in the relationship between
human and artifact?
In our previous study, it was shown that the expressions
and words of the character agent have an important rolein persuasive technology [1]. Persuasive technology [2]is technology that artifacts use to persuade humans. It issaid that if a user recognizes the presence of something
in a computer, he or she will respond to it according to
the normal social rules. Thus, it is important to present
emotions of artifacts in order to change the behavior ofhumans.
According to the concept “Media Equation”, people treat
computers, televisions, and new media as real people andplaces, thereby making the users uncomfortable if an© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

31

Kaoru Sumi

expressions, there is no flexibility in expressing emotions,so we often conduct experiments using character agents.
We have developed a counseling system [4][5], acustomer service learning system [6], a job interview
system [7], and a programming learning support system[8][9][10] in our research on interactive systems using
emotion. These studies were developed to examineinteractive systems between a human and an artifact.

References
1. Kaoru Sumi and Mizue Nagata: Evaluating a Virtual
Agent as Persuasive Technology, Psychology of
Persuasion, Janos Csapó and Andor Magyar eds., Nova
Science Publishers.2010.
2. B.J.Fogg: Persuasive Technology –Using Computers toChange What We Think and Do-, Elsevier, 2003.
3. Reeves, Byron and Clifford Nass The Media Equation:
How People Treat Computers, Television, and New Media
Like Real People and Places, Cambridge University Press,
1996.
4. Kaoru Sumi: Spoken Dialogue Agent Applications using
Emotional Expressions, Emotions and Technology:
Communication of Feelings for, with, and Through DigitalMedia, Elsevier (2015).
5. Kaoru Sumi: Human Computer Interaction via Affective
Computing, Journal of Robotics, Networking andArtificial Life, Vol. 3, Number 2, Atlantis Press (2016.9).
6. Kaoru Sumi, Ryuji Ebata: A Character Agent System for
Promoting Service-Minded Communication, Intelligent
Virtual Agents, Lecture Notes in Computer Science,
LNAI8108, pp.438, Springer (2013.8).
7. Kaoru Sumi, Kodai Morita: Spoken Dialogue Agent
System for Writing Resumes while Practicing Job
Interviews: Theory and Practice of Computation:
Proceedings of Workshop on Computation: Theory andPractice WCTP2015, World Scientific (2016)
8. Thomas James Z. Tiam-Lee and Kaoru Sumi: Analyzing
Facial Expressions and Hand Gestures in Filipino
Students' Programming Sessions, 2017 International
Conference on Culture and Computing, Culture andComputing 2017, Kyoto, Japan, September 10-12, 2017,
pp75-81,(2017.9).
9. Thomas James Tiam-Lee and Kaoru Sumi: Procedural
Generation of Programming Exercises with Guides Basedon the Student’s Emotion, IEEE SMC 2018 (2018.10).
10. Thomas James Z. Tiam-Lee and Kaoru Sumi: Adaptive
Feedback based on Student Emotion in a System for
Programming Practice, International Conference onIntelligent Tutoring Systems, Volume 10858 of the Lecture
Notes in Computer Science, pp.243-255, (2018.6)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan32

A Vishay supercapacitor based fast charge battery
Chung-Wen Hung1*, Yi-Chang Yang2, Chun-Chu Lin2, Li-Sheng Jheng1
1

Department of Electrical Engineering

National Yunlin University of Science and Technology
No. 123 University Rd. Section 3, Douliu, Yunlin 640, Taiwan
2Vishay (Taiwan) Ltd.

233 Baoqiao Rd., Xindian District, Sindian City District, New Taipei City 23145, Taiwan,
Corresponding author: *wenhung@yuntech.edu.tw

Abstract
A supercapacitor (SC) power supply management system based on is proposed in this paper. Due to the high charging
current characteristics of the SC, means short charging time, the proposed SC-base battery is suitable as a backup
battery. The proposed system includes the input voltage, current and temperature protection implemented with analog
integrated circuits (IC); and output current limitation circuit. In order to keep output voltage when the SC voltage
drops, a boost circuit is used to provide a stable voltage for the load. The circuit design is detailed in this paper, andthe experimental results show the proposed system works well.
Keywords: fast charge, supercapacitor; current limiting circuit.

system is turnoff and its electronic system parameters are
lost, it means long reboot time. A SC based battery
management system for backup function is proposed inthis paper, to maintain stable power supply when thebattery is replaced.

1. Introduction
Due to energy crisis and global warming issues, a global
ban on the sales of traditional internal combustion engine(ICE) vehicles is considered these years. [1] Thealternative solution is electric vehicle (EV) with moreenvironmentally friend, but the energy storage is an issue.
The disadvantages of the current electrochemical cellbattery solution are not only long charging times andshort battery life, but also easy damaged in nonmonotonic consumption. [2] A SC based battery is new
direction, because of almost no limited charging current,
and free non-monotonic consumption. However, the SC
based battery product for EV is not so popular, due to its
lower energy density. It is suitable used in high frequency
charging and discharging situation, or as a backup battery,such in automatic guided vehicle (AGV) and electricturret truck. When the battery under exchanging, aTable 1. Vishay SC specifications [3]
Item
Value
Category temperature range-40°C ～ +85°C
Rated voltage
IP (MAX. PEAK CURRENT)

2.3V ~ 2.7 V
35 A

The specification of Vishay SC used in paper isdifferent from normal capacitor and electrochemical cell,as shown in Table 1. [3] Vishay SC provide many
features, which is suitable for EV. Such as: “Double 85
degree”, Vishay is only supplier to provide temp /humidity 85-degree test of EDLC type SC. Moreover, itssafe function without bulging of sealing. The protection© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan33

Chung-Wen Hung, Yi-Chang Yang, Chun-Chu Lin, Li-Sheng Jheng

specification proposed in this paper is based on thesevalue, including: the temperature, voltage, chagrining
and discharging current protection.

shortened and even be damaged the circuit. Therefore,
installing overvoltage protection circuit is necessary.
2.3. Over-current protection circuit:
When the output current exceeds the SC withstandcurrent, SC life will be shorted. A shunt is used to transfer
output current to the corresponding voltage in overcurrent protection circuit. Similarly, the protection circuit
will immediately disconnect switch when Over-current.
3. Circuit Designed

Fig. 1. System architecture diagram.

3.1. Voltage Balance Method of SC

2. System Architecture

The voltage limitation of SC is extremely low, only 2.7V.
In order to provide 24 V output, 11 SCs are connected inseries, as shown in Fig. 2 (a). [4] The voltage balancemethod of the SC is necessary to keep the same voltageof each series SC. In high frequency charging anddischarging application, the parallel resistor method ismost common and simplest method. A resistor, 100KΩ,is paralleled with each SC, as presented in Fig. 2(a). Its
advantages are low-cost and simple, but the resistancewill continue to consume energy, means poor device
efficiency.

The system architecture is shown in Fig. 1. It includes
five blocks: front-end protection, current limiting circuit,
SC module, back-end protection and boost. The SC
module serves as a storage element, and the current
limiting circuit is adopted to limit the inrush charging
current, and the boost circuit is used to provide a stable
output voltage.
2.1. Over-temperature protection circuit:
When the SC is charged and discharged, the temperatureof the SC will gradually increase. In order to prevent
damage, a temperature sensor is installed in the SC block.
With operational amplifiers (OPA) comparing, when thetemperature voltage exceeds the threshold’s, the frontend protection circuit switch immediately disconnected
protection switch to perform the protection function.
Table 2. Current-limiting circuit component specifications
Item
Value
P-MOSFET1
J376
Capacitor C1
4400uF
Diode D1
RURG5060
R1
100K Ohm
R2
20k OhmR32 Ohm(a) Balancing circuit(b) Current limiting circuit
Fig. 2. Voltage balancing and Current limiting circuit

3.2. Current limiting circuit
The proposed Current limiting circuit is shown in Fig. 2(b). A diode, D and a P-MOSFET provide the charging
path. A Resister, R2, and a capacitor, C1, construct acharging circuit, and VC1 equal to VSG, what is used tocontrol SC charging current. In the initial stage, means
the voltage of SC is zero, the VC1 is charging slowly, andthe SC charging current is limited less than maximum.
The VC1 is charged to approach to the voltage divided by
R1 and R2, which is higher than threshold voltage and2.2. Over-voltage protection circuit:
In charging input, when the charging voltage of the SC
exceeds the SC maximum tolerable voltage, again, theprotection circuit will immediately disconnect protection
switch to prevent overcharging. If the input voltage
exceeds the limitation voltage of SC, its lifetime will be© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan34

A Vishay supercapacitor based

Table . Protection circuit specification
Item
Value
Overvoltage Protection
28 V
Overcurrent Protection
25 A
Current Limiting Protection
15 A
Temperature Protection
80°C

totally turns on the switch. Note, the ratio of R1 and R2
is fixed to 4.7 for keep the maximum turn on voltage.
Now, the SC is charged in low current, due to the high
VSC. Moreover, R3 and switch provide a discharging rout.
The Table 2. shows a typical parameter and value ofcomponents in the proposed Current limiting circuit.
3.3. Boost circuit:
The output boost circuit is shown in Fig. 3, and its
specification is described in Table 3. Note, the input
range is from 6 to 24V, because the SC voltage will
always decrease when discharging. The load current is
set into 2 and 4.5 A, which are required from users ofelectric turret truck for backup battery solution. Table 4.
shows the part number and value of circuit components
used in the boost circuit, these the part number and value
are calculated based on the specifications in Table 3.

(a) Series SC block(b) Control system blockFig 4. Photo of circuit

4.1. Experimental Circuit
The photos of system are present in Fig. 4, and left andright are SC and Control system blocks, respectively. In
the control system block, Fig. 4 (b), the upper left is thefront protection circuit, the upper right is the current
limiting circuit, the lower left is the back-end protection
circuit, and the lower right is the boost circuit.

Fig. 3. Boost circuit

4.2. Current Limit Test

Table 3. Boost circuit specifications
Item
Value
Input Voltage
6 V~24 V
Output Voltage
24 V
Load Current
2 A/4.5 A
Po_max108W
Frequency
200K Hz
Table 4. Boost component specifications
Item
Value or Part number
Inductor L1
25uH
Output filter capacitor C1
220uF
Power Switch Q1
IRF0642
Diode D1
F15S60S
Controller
LM3478 IC [5](a)Current limiting current 14.83A
C1 (5V / div), C2 (5V / div), C3 (5A / div), t: 10s /div

4. Experimental results.
Table 5. is the protection specification. Due the maximum
withstand voltage of a SC is 2.7V, the total voltage of 11series SC is 29.7V. The overvoltage protection value isset to 28V. The current limiting could be set less thanspecification’s value.

(b) Current limiting current 22.9A
C1 (5V / div), C2 (5V / div), C3 (5A / div), t: 10s /div
Fig. 5. Current Limit Circuit Experiment Section

With the components in Table 6, the operation of the
input charging current is captured, as shown in Fig. 5.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan35

Chung-Wen Hung, Yi-Chang Yang, Chun-Chu Lin, Li-Sheng Jheng

The channel C1, C2 and C3 is the VSG voltage, SC voltage,
and the input current, respectively. In (a)/(b), themaximum current about 14.83A / 22.9A, and T1 is initialcharging time is about 30 /16 seconds depended on
different RC combinations. The initial charging time isset by adjusting the resistor value ofR1 and R2 in Table
6, and the C is fixed to 4400uF. The target charging
current is limited to 15A, so choose R1 = 100kΩ and R2= 20kΩ.
R1(k Ω)
470
360
240
120
100
75
62
30
18
12

C1 (10V / div), C2 (10V / div), C3 (10A / div), C4 (5A / div),
t: 10s /div

Fig. 7 shows that the SC is output by the boost circuit tosupply the load. Again, channel C1, C2, C3 and C4
represent the SC voltage, the output voltage, the SC
current, and the output current. The target voltage of 24V
can be maintained for 19 seconds under 2A load current.
And, in Fig. 7 (b), output is 6.5A load first for T1,1
second, then the 2A output maintains about 13 seconds,
T2.
5. Conclusions.
A Vishay SC base fast charge battery is proposed in thispaper. 11 SCs are connected in parallel to perform a
backup battery. The over voltage protection, overtemperature protection, and SC voltage balance are
discussed. High input current limitation is implemented
to provide the high charging current and protect function.
To keep constant output voltage, a boost circuit isadopted. The experiment results show the proposedsystem workable. The capacity of energy storage could
be increased when series SCs are connected in parallel
and component parameters are modified.

Table 6. Current limiting circuit test
R2(k Ω)CURRENT(A)TIME(Sec)
100
7.53
83
75
8.2
67
48
10.35
55
24
14.57
38
20
14.83
30
15
15.75
27
12
16.66
21
6.2
20.1
18
3.6
22.9
17
2.4
23.0
16

4.3. Over-current test

Acknowledgements
This work is partially supported by the Vishay (Taiwan)
Ltd; and also by Ministry of Science and Technology,
ROC, under contract No. MOST 107-2221-E-224-054.
and 106-2622-E-224-017-CC3

Fig. 6. Over-current circuit test section
C1 (10V / div), C2 (10V / div), C3 (5A / div), t: 2s /div

Fig. 6 shows the over-current protection operation.
Channel C1, C2 and C3 represent the SC’s voltage, the DS
voltage, and the SC current, respectively. In the first, theSC is fully charged, and system discharges to a DC load.
When the protection current is exceeded, the postprotection switch is turned off. T1 in Fig. 6 show theswitch is turned off, due to discharging current larger than
25A. Moreover, T2 presents the power supporting timefrom fully charging SC to out of energy with output
current under control.

References
[1]

[2]

[3]
[4]

[5]
(a) Load 2A(b) Load 2A + 4.5A

A. Petroff, “These countries want to ban gas and diesel
cars,”
https://money.cnn.com/2017/09/11/autos/countriesbanning -diesel-gas-cars/index.html
L. Kouchachvili, W. Yaïci and E. Entchev, Hybrid
battery/supercapacitor energy storage system for theelectric vehicles, Journal of Power Sources 375, 2018,
pp. 237-248.
Vishay, “225 EDLC CAP™ Ruggedized Electrical
Double Layer Energy Storage Capacitors”, 225 EDLCENYCAP™ datasheet, 27, Mar. 2018.
C. Ionescu, A. Vasile and R. Negroiu, "Investigations onbalancing circuits for supercapacitor modules," 2016
39th International Spring Seminar on Electronics
Technology (ISSE), Pilsen, 2016, pp. 521-526.
TEXAS INSTRUMENTS, “LM3478 High-Efficiency
Low-Side N-Channel Controller for Switching
Regulator”, LM3478 datasheet, Dec. 2017.

Fig. 7. Test output voltage and current© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan36

Multi-Motor synchronous control with CANopen
Chung-Wen Hung*,1, Roger CL Lee2, Bo-Kai Huang1, Shih-Ting Yu1
1

Department of Electrical Engineering

National Yunlin University of Science and Technology
No. 123 University Rd. Section 3, Douliu, Yunlin 640, Taiwan
2

Naroller Electionics Co., Ltd.

6F., No578, Sec. 1, Minsherg N. Rd., Guishan Dist., Taoyuan City 333, Taiwan
Corresponding author: *wenhung@yuntech.edu.tw

Abstract
A Brushless DC (BLDC) Motors synchronous control system based on the CANopen protocol is proposed in thispaper. Multi-motor control is popular in Robot or automation system, and the synchronization is an important issue,
may be the key issue. The CANopen is a solution, due to Process Data Object (PDO) protocol. A Texas Instruments
microcontroller TMS320F28069 is used as master to control six BLDC Motors used as slave in this paper. The Master
sends synchronization object (SYNC) to slaves base on the communication profile CAN in Automation (CiA) 301
and the motion control profile CiA402. The system suns at 500Kbits/s and 1Mbits/s transmission rate by setting
process data object (PDO). The firmware details in this paper to show six-motor synchronous control.
Keywords: CAN bus, CANopen, Brushless DC Motors

mentioned in [4] and [5]. The Service Data Object (SDO)
is used to ask or set parameter for any single node. And
the PDO is the object communication code for instantcommunication and the SYNC synchronizationcommand. The target of the paper is the synchronous
control of six BLDC motors, through the basic
communication specification CiA 301 [6] and the motion
device specification CiA 402 [7]. The CAN peripheral of
TI TMS320F28069, master, is used to handle the CAN
communication, and the CANOpen protocol is peformedby program. The parameters of target speed, accelerationand deceleration on the Maxon EPOS4 motor drive
could be set and synchronous controled. The speed modeof the CiA 402 is used to achieves position control of the
motor. The updating rate for six motors synchronous
control will be discussed in this paper.

1. Introduction
Multi-motor synchronized control is a important and key
issue in Robot or automation system. How acommuncation protocal supports synchroniztion is
discuused in many literatures. [1] mentioned CAN
communication for complex in-vehicle networks can
reduce the cost of hardware configuration andmaintenance. [2] is similar to [1], but the CANopen
protocol is adopted and supports compatibility. The
master communicate with the devices form different
manufacturers. CAN is also widely used in the industrial
field due to the better error mechanism and hightransmission rate.[3] The CANopen protocol also defines
device specifications for industrial devices. This paper
will refer to the Network Management (NMT) Object,which CANopen basic communication specification is© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan37

Chung-Wen Hung1*, Yi-Chang Yang2, Chun-Chu Lin2, Li-Sheng Jheng1

followed by corresponding format specifications for thedata sections in the packet.

2. System Architecture
The system architecture is shown in Fig.1. The system
includes a mater (TI's TMS320F28069), CAN driver(Renesas' R2A25416SP) and motor drivers (Maxon's
EPOS4) with motors.

Fig. 2. Base of CAN data frame format
stuff bit
= 8 + 2 ∗ DLC(1)

3. Software Architecture

In the CANopen CiA 301 protocol, a master must arrange
communication relationships between a transmit- and a
receive-PDO with SDO. [6] is stipulated that the NMT
must be used to control the working state of the device
before using the SDO or PDO. Each device has a working
state machine. The master controls the working state of
the node through the NMT. As shown in Table 1, if
people like to use SDO, the node working state have be
set into the pre-operation mode. If PDO, the node inoperation mode is necessary.

Fig. 1. System architecture diagram

2.1. Master- TI TMS320F28069
The build-in peripherals of TMS320F28069, CAN
module and timer, are used in the CAN master. Due todigital format of CAN module signal, a driver isnecessary to convert signal to differential signals of
CAN-H and CAN-L. These two wires are connected
directly to CAN bus. The timer is sued to trigger CPU to
set the new target speed, acceleration and deceleration
values of the slaves every fixed time interval.

Table 1. States and Communication Objects [6]
NMT
PDO
SDO
SYNC

2.2. Slave- Maxon EPOS4 drive
The Maxon EPOS4 is adopted in system to drive motor,
the CAN communication interface is built in, and the CiA
301 and 402 object specifications are support by thid
driver. In this paper, it is selected as the slaves to receive
the target speed, acceleration or deceleration from themaster for motor control.

Pre-Operational√√√

Operational√√√√

Stopped√In the master, the timer is used to trigger master to update
the parameters of the PDO mapping to the device
periodically. These parameters are the motion device
state machine control object, such as the target speed
object, the acceleration object, and the deceleration
object. The data type of each object is also shown in
Table 2.

2.3. CAN Transport Packets
The CAN2.0A format is shown in Fig.2, what is used in
CANopen protocol. This format is also used in this paper,
it is the key of calculate communication capability, later.
However, the format includes stuffing bit, and this bit
stuffing mechanism is sued to detect short even. The
maximum of stuffing bits generated by the bit stuffing
mechanism is discussed in [8], as shown in Eq. (1). The
first 4 bits of the 11bits ID of the arbitration field in asingle complete packet are defined as the control code,and the else 7 bits specified in the following are the node
ID. These 11 bits are collectively referred to as thecommunication object identifier(COB-ID), and isTable 2. Data format of the object [7]ObjectData type6040h: Control word
60FFh: Target velocity6083h: Acceleration
6084h: Profile deceleration

UNSIGNED16 (2 Bytes)
INTEGER32 (4 Bytes)
UNSIGNED32 (4 Bytes)
UNSIGNED32 (4 Bytes)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan38

Multi-Motor synchronous control

3.1. Implementation Process data object

BLDC motor. Obviously, the CAN bus is sued to linkmaster and all salves in parallel.

In this paper, PDO is used for parameter setting andsynchronous control. The NMT object communication
code is used to set the working state of the node, and the
NMT object includes the single or broadcast mode. Due
to multiple nodes in the proposed system, the working
state of all nodes is set to the operation mode in the
broadcast mode. When the working state of node is thepre-operation mode, the node could process the SDO
object. It is necessary for the PDO usage, the PDO
mapping function of the node muse be set by SDO in
advance. The SDO is only use in a point-to-point scheme.
A node has to respond the SDO send from the master in
1 ms, as shown in Fig. 3(a).
After completing the PDO setting, the working mode ofa node needs to be set to the working mode, and then the
PDO function is valid. Note, it is important, when thenodes receive the PDO from the master, the nodes will
wait for the SYNC command from the master to start toachieve the command. Here, this SYNC is a broadcast
signal without data, and the all of motors can startsimultaneously, as shown in Fig. 3(b).

Fig. 4. SDO vs PDO [5]Fig. 5. Experiment system(a) SDO(b)PDO
Fig. 3. sequence in time

4.1. Method of updating speed commands

3.2. Comapre SDO with PDO for synchronized control
The Fig. 4 shows the comparison of SDO and PDO forsynchronized control with three parameters, which
lengths are 1, 2 and 4 bytes. Because the SDO has a
specific format in the data field when setting the object
parameter to a node. Three packets are necessary, due tosetting three object parameters in SDO mode. Aftersetting, the node will return 4 bytes of response to themaster. However, after PDO mapping function is
performed in SDO mode, only 8 bytes of one packet isrequired to set object parameters.

The system minimize requirement is that every slave isupdated the driver commands every 0.1 second forsynchronized control. These commands are the target
speed, acceleration, deceleration and control word, andthese four parameters are arranged into two groups of
PDO, means two PDO mapping. When the all slaves(named consumer in PDO mode) received all PDO
parameters from the master (named producer in PDOmode), and then the master sends a broadcast SYNC
signal to trigger all salves.

4. Experimental results

4.2. Node Capacity Calculation

The Fig. 5 presents the system diagram of the experiment.
As mentioned above, the master, TI TMS320F28069, and
CAN driver in red dash box. The blue dash box shows
the salve, and every slave includes a Maxon drive and a

To make sure how many slaves could be control
synchronously in the proposed system, the node capacity
calculation is necessary. As discussed previously,
CANopen follows the packet format of CAN2.0A, which
is present in Fig. 1. The length of the minimum packet© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan39

Chung-Wen Hung1*, Yi-Chang Yang2, Chun-Chu Lin2, Li-Sheng Jheng1

which never includes any data byte is 47 bits, which
includes the IFS (Inter-frame Space) field already. The
length of a packet depends on the object. The packet's
length without stuff bit can be calculated from Eq. (2).
Moreover, in the actual situation, the stuff bit must be
considered, and its length could get from Eq. (3).
CAN data bits
CAN data bits

= 47 + 8 ∗ DLC= 55 + 10 ∗ DLC

sane commands. They are synchronous stopped at thesame two positions. The results show the proposed
CANopen synchronous control is workable.
5. Conclusions
This paper proposes a multi-node synchronization
control system. The system includes the CiA 301 and theCiA 402 of the CANopen protocol and use the PDO andSYNC object for synchronization control. The protocol
is detailed and the capacity is calculated in the paper.
Finally, the experimental results shown the proposedsystem work well.

(2)
(3)

In SDO mode, the maximum number of nodes controlled
synchronously in 0.1 second is derived as (4). Note, the
8 ms is the extra inter-frame space requested by Maxonfor 8 packets. First, the SDO_Packet_Length equals tothe length of 4 parameter SDO packets plus 4 response
packets. Note, the parameter SDO packet includes theminimums length (55 bits), 4 bytes (CS, index, and subindex) and data bytes as shown in Table 2, and the exact
bit number can get from Eq. (3). Moreover, the response
packet is fixed to 4 bytes plus 55 bits. Finally, the SDO
packet is equal to 900 bits. And node number can be getfrom (4).
Node number=

0.1(s)
SDO_Packet_Length+ 8(ms)baud rate

Acknowledgements
This work is partially supported by the Vishay (Taiwan)
Ltd; and also by Ministry of Science and Technology,
ROC, under contract No. MOST 107-2221-E-224 -054.
and 106-2218-E-150-001References

(4)

[1]

On the other hand, the PDO equation is more complex,
as shown in Eq. (5). The one SYNC packet (55 bits) is
necessary for every 0.1 S, so is 1ms (extra inter-frame
space). Due to the two PDO mapping groups, the 2 ms isthe extra inter-frame space for two packets. Next,
PDO_Packet_Length includes 2 PDO packets. The dataof the first PDO pocket is 8 bytes, and second’s is 6 bytes.
In short, PDO_Packet_Length is equal to 250 bits.
Node numbersync− 1( )baud
rate=
PDO_Packet_Length+ 2(ms)baud rate
0.1(s) −

[2]

(5)

[3]

The calculation results are present in Table 3 for thedifferent communication mode and different baud rates
Table 3. Maximum number of nodes
SDO
PDO
500kbits/s
10
39
1Mbits/s
11
43

[4]

4.3. Experimental Results of Position Control

[5]

[6]
(a) Initial(b) Finish
Fig. 6. Position experiments[7]

The 4.3. Experimental Results of Position Control isshown in Fig. 6, and (a)/(b) is the initial/finish status.
This experiment run in the PDO mode. From left to right,
the first and third motors are controlled in the samecommands, and the second’s and fourth’s are in the other[8]D. Q. Wang, S. Gao, Y. Q. Chen, Y. Wang and Q. Liu,
"Intelligent Control system based on CAN-bus for car
doors and windows," 2009 3rd International Conference
on Anti-counterfeiting, Security, and Identification in
Communication, Hong Kong, 2009, pp. 242-245.
G. Livint, V. Horga, M. Ratoi, M. Albu and G. Chiriac,"Implementing the CANopen protocol for the distributed
control of a hybrid electric vehicle," 2009 8thInternational
Symposiumon
AdvancedElectromechanical Motion Systems & Electric Drives
Joint Symposium, Lille, 2009, pp. 1-6.
W. Li, G. Lijin and L. Zheyuan, "Design of STM32Based CANopen Motion Control Master in Transfer
Robot," 2013 Third International Conference on
Instrumentation,
Measurement,
Computer,Communication and Control, Shenyang, 2013, pp. 16091612.
Yang Fan, R. Chen and Z. Chen, "Communication of thewind turbine testing system based On CANopen
protocol," 2011 IEEE Power Engineering andAutomation Conference, Wuhan, 2011, pp. 162-165.
B. K. Huang, "Design and implementation of microstepping motor driver with CANopen" Master Thesis,
National Yunlin University of Science and Technology,
June 2017.
CANopen Application layer and communication profile,CiA DS301, Version: 4.0.2, Feb. 13, 2002, pp. 15-114.
CANopen Device Profile Drives and Motion Control,CiA DS402, Version 2.0, Jul. 26, 2002, pp. 14-174.
Holger Zeltwanger(2008), C. H. GE, “CANopen: dasstandardisierte, eingebettete Netzwerk(Chinese Edition),”Beijing University of Aeronautics and Astronautics Press,
Jul. 1, 2011, pp. 1-123.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan40

Selection Strategy for VM Migration Method
Yan-Ren Chen, I-Hsien Liu, Keng-Hao Chang

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University, Tainan City, 70101, Taiwan
Chuan-Gang Liu

Department of Applied Informatics and Multimedia, Chia-Nan University of Pharmacy and Science, Tainan City, 71710,
TaiwanJung-Shian Li

Department of Electrical Engineering, National Cheng Kung University, Tainan City, 70101, Taiwan
E-mail: {yrchen, ihliu, khchang}@cans.ee.ncku.edu.tw, chgliu@mail.cnu.edu.tw, jsli@mail.ncku.edu.tw
www.cans.ee.ncku.edu.tw

Abstract
Virtual machine (VM) live migration occurs frequently in cloud environments. When it is necessary to migratemany VMs, minimizing the migration time is an important concern. Thus, choosing an appropriate strategy toperform VM migration is essential. Accordingly, this paper discusses the advantages and disadvantages of four
different migration methods and evaluates their respective migration times and throughputs.
Keywords: VM migration, Sequential, Delay, Parallel, Simultaneous.

1.

migration method is essential to minimize the total
migration time under different VM memory loads,
migration link bandwidths, page dirtying rates, and soon.

Introduction

VM migration plays a critical role in cloud computing
in satisfying user demand for computing resources
while simultaneously meeting the needs of the operator
for energy savings, load balancing, periodic
maintenance, and so on. VM migration can be
performed using four different methods, namely
sequential, delay, parallel, or simultaneous. In
sequential migration, migration of a VM is performed
only once the migration of the previous VM on the same
server has been completed. In delay migration, the VMs
are located on different servers and a small delay is
introduced between their migration. In parallel
migration, multiple VMs located on the same physicalserver are migrated at the same time. Finally, insimultaneous migration, VMs located on different
physical servers are migrated at the same time. In
practice, the migration times of the four methods mayvary widely. Thus, the choice of an appropriate

2. Background and Related work
2.1. VM Migration Techniques
Virtualization platforms generally utilize either a precopy live migration approach [1, 2] or a post-copy live
migration approach [3, 4]. Of the two methods, the precopy method results in a shorter downtime, while thepost-copy method results in a shorter total migration
time. In [5], [6], the authors showed that the overall
migration time for Stages 0 ~ 5 of the pre-copy method
comprises four components, namely the Pre-Migration
Overhead (Stages 0 & 1), the Stage 2 Overhead, theStage 3 Overhead, and the Post-Migration Overhead(Stages 4 & 5). Among these components, the Stage
0/1/4/5 overhead time is static but not constant.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

41

Yan-Ren Chen, I-Hsien Liu, Keng-Hao Chang, Chuan-Gang Liu, Jung-Shian Li

Consequently, under the same VM migration conditions,the overhead times of the pre-and post-copy migration
methods are slightly different.

units (SRU) transmissions. However, the method
resulted in a low bandwidth utilization rate during theslow start cycle.

2.2. Migration Performance Indicators

3. VM Migration Methods and Window Size

Common migration performance indicators include thetotal migration time, the migration down time, and thetotal number of VM migrations. The total migration
time measures the elapsed time between the start of thefirst VM migration process and the end of the final VM
migration process. The authors in [5-7] investigated theeffects of various factors on the total migration time,
including the migration link bandwidth, the frequency
of dirty memory pages, the VM memory size, and so on.
The migration down time measures the time for which
service is interrupted during the migration process, andshould be minimized in order to maintain user
satisfaction. Finally, the number of VM migrations
indicates the total number of VMs migrated, and should
be reduced to the minimum number possible tominimize the impact of the migration process.

3.1. Sequential Method
Figure 1(a) shows a typical example of the sequential
VM migration process. Let bandwidth delay product(BDP) be the product of the link bandwidth and theround-trip time. Assume that VM1 and VM2 have thesame memory size and page dirtying rates (i.e., tss+t1 isconstant). Furthermore, assume that TCP ssthresh islarger than BDP. Thus, sendwin is equal to BDP in theslow start phase. The sum of sendwin for the two VMs
is then obtained as shown in Figure 1(b).
In the sequential migration method, there is no overlap
between the migrations of VM1 and VM2, respectively.
Consequently, the sum of sendwin in Figure 1(b) does
not exceed BDP. As a result, the throughput is notdegraded by overflows at the switch buffer. However,the bandwidth cannot be fully utilized during the slow
start phase and the migration overheads of the two VMs
cannot be processed at the same time. Consequently, thetotal migration time is increased. Furthermore, theoperator cannot explicitly configure the time at which
each VM migration task is to be performed. In otherwords, the sequential migration method lacks versatility.

2.3. TCP Window Size
VMware [8] prescribes the use of the transmission
control protocol (TCP) to establish a connection for VM
migration and transmit the corresponding migration
traffic. TCP is a synchronous transmission protocol withgood reliability and completeness. The TCP window
size used to transmit the migration packets is assigned aminimum value of (CWND, RWND), i.e.,
sendwin=min(cwnd,rwnd). The authors in [9] presented
a scheme for lowering the value of ssthresh in order toenter the congestion avoidance stage early. However,the system suffered an insufficient startup problem
accordingly. The authors in [10] considered a serialtransfer process rather than parallel server request

3.2. Parallel Method
As shown in Figure 2(a), in the parallel migration
method, VM1 and VM2 are located on the same server
and are transmitted at the same time over a single
connection. The sendwin size of each VM is thus
reduced to (1/2)BDP. Consequently, the sum of sendwin
is equal to BDP, as shown in Figure 2(b). It is seen in
Figure 2(b) that t’ss is less than tss. In other words, theFig.1. (a) Sequential Method (b)Sum of Sendwin

Fig.2. (a) Parallel Method (b)Sum of Sendwin© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

42

Selection Strategy for VM

parallel migration method improves the bandwidth
usage during the slow start phase. In addition, themigration overheads of the two VMs can be processedat the same time. Hence, the total migration time is
reduced.

Figure 4(b)), and therefore smooths the traffic volume
and improves the throughput. However, the total
migration time is usually much longer than that of thesimultaneous migration method.
4. Performance Experiments and Analysis

3.3. Simultaneous Method

4.1. Experiment Environment

Figure 3(a) illustrates the simultaneous migration
method, in which the migrations of VM1 and VM2,
which are located on different servers, are executed in a
many-to-one communication mode. Assuming that thelinks between the source servers and the switch have thesame bandwidth, the sum of sendwin reaches BDP after
just t’, as shown in Figure 3(b). Notably, t’ is less thantss. In other words, the migration process results in along overlap time between the transmissions of the two
VMs. Consequently, the switch buffer easily overflows;
resulting in packet drops and a significant reduction in
the throughput. Thus, the total migration time of thesimultaneous migration process is longer than that of theparallel migration process.

Migration experiments were performed using three HP
ProLiant DL380 G6 physical servers (two sources and a
destination) and hypervisor VMware vSphere 5.5
software. The experiments replicated the networkarchitectures and migration methods shown in Figs.
1(a)~4(a). Each VM was configured to have eightvirtual CPUs and 14 GB of virtual RAM. Memory
testing tool was used to increase the memory loads of
the VMs in the range of 2 ~ 11 GB. The total migration
time of the VMs under different loads was measured.
4.2. Experiment Results
Figure 5 shows the experimental results obtained for thevariation of the total migration time with the VM
memory load for each of the considered migration
methods. The results support the following main
conclusions:(1) The migration methods can be ranked in terms of thetotal migration time (shortest to longest) as follows:
Parallel < Simultaneous < Delay < Sequential.
(2) For a memory load of 11 GB, the total migration
time of the simultaneous method is close to that of thesequential method.
(3) The delay method generally results in a longer total
migration time than the simultaneous method. However,the performance of the delay method depends on the
delay time applied. For example, given a memory loadof 11 GB and a delay time of 120 seconds, the delay
method results in a shorter total migration time than thesimultaneous method.
Figure 6 shows the throughputs obtained for each of thefour migration methods given a VM memory load of 4
GB. The migration methods can be ranked in terms ofthe throughput performance (high to low) as follows:
Parallel ≈ Sequential > Delay > Simultaneous.

3.4. Delay Method
As shown in Figure 4(a), the delay migration method issimilar to the simultaneous migration method other thanfor the fact that a short delay is introduced between the
migrations of the two VMs. The delay method reduces
the overlap time of the multiple TCP connections (seeFig.3. (a) Simultaneous Method (b)Sum of Sendwin

4.3. Migration Selection Strategy
Fig.4. (a) Delay Method (b)Sum of Sendwin

Based on the experimental results presented above for© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan43

Yan-Ren Chen, I-Hsien Liu, Keng-Hao Chang, Chuan-Gang Liu, Jung-Shian Li

Acknowledgements
This work was supported by the MOST (Ministry of
Science and Technology), Taiwan under contracts
numbers MOST 107-2218-E-006-036- and MOST 1072221-E-006-140-.
References
1. Virtual Machine Live Migration Overview, Microsoft
TechNet,https://technet.microsoft.com/enus/library/hh831435(v=ws.11).aspx, 21 August. 2013.
2. Christopher Clark, Keir Fraser, Steven Hand, Jacob
Gorm Hansen, Live Migration of Virtual Machines,
https://www.usenix.org/legacy/event/nsdi05/tech/full_pa
pers/clark/clark.pdf
3. Hines, Michael R., and KartikGopalan, Post-copy based
live virtual machine migration using adaptive pre-paging
and dynamic self-ballooning, ACM SIGPLAN/SIGOPS
international conference on Virtual execution
environments. ACM, 2009, pp. 51-60.
4. Michael R. Hines, Umesh Deshpande, Kartik Gopalan,
Post-Copy Live Migration of Virtual Machines,
http://osnet.cs.binghamton.edu/publications/hines09postc
opy_osr.pdf
5. Sherif Akoush, Ripduman Sohan, Andrew Rice, Andrew
W. Moore and Andy Hopper, Predicting the Performanceof Virtual Machine Migration, IEEE International
Symposium on Modeling, Analysis & Simulation ofComputer and Telecommunication Systems (MASCOTS),
2010, pp. 37-46.
6. Mohamed Esam Elsaid, Christoph Meinel, Live
Migration Impact on Virtual Datacenter Performance :Vmware vMotion Based Study, International Conference
on Future Internet of Things and Cloud (FiCloud). 2014,
pp. 216-221.
7. Felix Salfner, Peter Troger, Andreas Polze, Downtime
Analysis of Virtual Machine Live Migration, The FourthInternational Conference on Dependability, ISBN: 9781-61208-149-6. 2011, pp. 100-105.
8. "VMware
vSphere5.1
vMotionArchitecture,
Performance and Best Practices." VMware TECHNICAL
WHITE PAPER, Available at: https://www.vmware.com/
iles/pdf/techpaper/VMware-vSphere51-vMotion-Perf.pdf.
9. A. Phanishayee, E. Krevat, Y. Vasudevan, D. G.
Andersen, G. R. Ganger,G. A. Gibson and S. Seshan,
Measurement and Analysis of TCP Throughput Col/apse
in Cluster-based Storage Systems, The 6th USEUNTX
Conference on File and Storage Technologies (FAST2008), 2008. pp. 12:1-12:14 [Online].
10. Shigeyuki Osada, Kazutoshi Kajita, Yukinobu
Fukushima and Tokumi Yokohira, TCP Incast Avoidance
Based on Connection Serialization in Data Center
Networks,
Communications,
19thAsia-PacificConference((APCC)), 2013, pp. 142–147.

Fig.5. Total migration times of different VM migration
methods.

Fig. 6. Throughputs obtained under different VM migration
methods.

the total VM migration time, the following migration
selection strategies are proposed:(1) Selection strategy 1: If the switch-server 3 link
bandwidth is much higher than that of the source-switch
links:
Selection order: Simultaneous -> Parallel -> Delay >Sequential method(2) Selection strategy 2: If the bandwidths of the links
between the switch and the servers (sources anddestination) are the same:
Selection order: Parallel -> Simultaneous -> Delay >Sequential method.
5. Conclusion
This paper has analyzed the migration time andthroughput performance of four different VM migration
methods (sequential, delay, parallel, and simultaneous)under different VM memory loads. In general, theresults have shown that when the link bandwidth of the
migration destination is much higher than that of thesource(s), the simultaneous method results in the
shortest migration time. By contrast, if the link
bandwidth of the migration destination is equal to thatof the source(s), the parallel migration method provides
the shortest total migration time.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan44

VM Migration Placement in Cloud Service
Yan-Ren Chen, I-Hsien Liu, Keng-Hao Chang

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University, Tainan City, 70101, Taiwan
Chuan-Gang Liu

Department of Applied Informatics and Multimedia, Chia-Nan University of Pharmacy and Science, Tainan City, 71710,
TaiwanJung-Shian Li

Department of Electrical Engineering, National Cheng Kung University, Tainan City, 70101, Taiwan
E-mail: {yrchen, ihliu, khchang}@cans.ee.ncku.edu.tw, chgliu@mail.cnu.edu.tw, jsli@mail.ncku.edu.tw
www.cans.ee.ncku.edu.tw

Abstract
The present study considers two different VM migration problems and proves that they too are NP-complete. In thefirst problem, subsets of the VM migration groups are migrated to a physical host until their combined resource
usage approaches the threshold value of the host. In the second problem, the VM migration groups are migrated
simultaneously in such a way as to minimize the total migration time.
Keywords: VM, Live Migration, NP-Completeness.

1.

Introduction

2. Background and Related work

Generally speaking, problems can be classified as either
Unsolvable, Intractable, Non-Deterministic Polynomial(NP), or Polynomial (P). This study focuses on theparticular case of NP problems. In other words, theproblem solution can be calculated in exponential timeand verified in polynomial time. When a problem
belongs to NP and some NP-complete (NPC) problems
can be solved by this problem, then the problem is alsoa NPC problem. NP-hard problems are one type of NP
problem for which it still has not found a polynomial
complexity of the decision algorithm, and polynomial
complexity of the decision algorithm does not exist. NPhard problems are a particular class of NP problem towhich every problem in NP can be translated, and eachsolution to such a problem gives a fast solution to every
other problem in NP. Problems which are both NP and
NP-hard are referred to as NP-Complete (NPC).

2.1. NP-CompletenessIn [1-3], the authors proposed various methods forproving that problems belong to NPC. These methods
typically involve three main steps:(1). Prove that the problem under consideration belongs
to an NP problem (i.e., B∈NP).
(2). Select an already known NPC problem. (i.e.,
EXACT-COVER, SAT, Bin Packing, and so on).
(3). Transform (reduce) the selected NPC problem tothe problem under consideration.
Common NPC problems include the Satisfiability
Problem [3], the 0-1 Knapsack problem [2], theHamiltonian Cycle, and so on. Taking the Knapsackproblem for illustration purposes, Lagoudakis [2]reviewed various algorithms for solving the 0-1
Knapsack problem and showed by reference to the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan45

Yan-Ren Chen, I-Hsien Liu, Keng-Hao Chang, Chuan-Gang Liu, Jung-Shian Li

EXACT-COVER and SAT problems that they all
belong to NPC.
The nature of the VM migration problem (i.e., NPcomplete or NP-hard) has attracted significant attention
in the literature. The authors in [4] proved that VM
migration flow over the delivery path is NP-complete byreference to the Hamiltonian Cycle problem. The
authors also proposed a method for minimizing thenumber of delivery flow under the migration process
and showed by reference to the Bin Packing problem
that this too was NP-complete. Kangkang et al. [5]reported that for both off-line and online scenarios, theauthor-defined total completion time minimization VM
placement problem was an NP-hard problem. However,no formal proof was provided to substantiate the claim.
The authors in [6] showed that the problem of placing
VMs on a physical host with both a minimal cost and aminimal response time is NP-hard. However, while aheuristic algorithm was proposed for its solution, theNP-hardness of the problem was not formally presented.
Zhang et al. [7] used a software-defined networking(SDN) technique to minimize the network flow
resumption time during VM migration. The flow
migration problem was transferred to a Disjoint Path
Problem and shown to be NP-complete. The authors in[8] considered a network-aware VM migration(NetVMM) problem and showed that it is a variant of
the well-known NP-complete multiple-knapsackproblem.

Fig.1. VM migration to destination host
Table 1 Notations used in Section 3.1
Notation

DescriptionC

CPU resources of physical host.

M

Memory resources of physical host.

c(u)CPU loads of migrated VMsm(u)Memory loads of migrated VMs

VM

Total VM migration group (i.e., VM = {vm1,vm2, ..., vmn-1, vmn} )C_ssthresh

Threshold value of CPU resources of physical host

M_ssthresh

Threshold value of memory resources of physicalhostof the host is an NP problem. In general, a nondeterministic algorithm is one which, for the same input,
a different result is obtained every time. For the VM
migration problem considered here, the total usage
resources of the VMs to be migrated must be less than
the remaining resources of the destination host before
migration. Furthermore, the migration process should
continue until the VM migration resource usage justapproaches (but does not exceed) the threshold value ofthe physical host. This problem can be verifiable inpolynomial time. Hence, the migration of a subset of the
VM migration groups to a physical host until theirresource usage approaches the threshold value of thisphysical host belongs to NP.
Let the migration problem be transformed using the
Sum of Subset problem (a known NPC problem). In the
Sum of Subset problem, there exist n integers, of whicha particular subset of n can be found for which the sumof the subset is equal to M. For example, given a set A ={a1, a2, a3, ..., ap}, the aim of the Sum of Subset problem
is to find the subset of A, denoted as Sub_A={ a1, a4, a5,a6 }, for which the sum of Sub_A is M. In the VM
migration problem, the host resources and the usage
resources of the VMs are both given. When Sub_A
contains a subset of A
usage resource of partial VM

3. Proposed Scheme
3.1. A subset of the VM migration groups is migrated
to a physical host until their resource usage
approaches the threshold value of this physical host
NPC problem
Assume that a subset of the VM migration groups on
multiple source hosts are to be migrated to a single
physical host, as shown in Figure 1. For the migration
process to be successful, the total resource requirementsof the selected subset of VM migration groups must be
less than the designated threshold resources of thedestination host. The related notations are listed inTable 1.
As discussed in the following, the migration of thesubset of VM migration groups to the physical host until
the usage resource load approaches the threshold value© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

46

VM Migration Placement inin this group that satisfies equation (1). Thus, the VM
migration problem can be formulated as the following
Sum of Subset problem:∑ ∈c(u) ≤ C_M_∗ C and ∑ ∈∗ M. (1)m(u) ≤

Assuming that |A| = n objects, then u = {ui, i = 1, ..., n}.
Let the selection factor have the following form:

if x = {. (2)Fig.2. Simultaneous migration of multiple VMs on different
hosts
Table 2 Notations used in Section 3.2

The resource usage of the VMs can thus be obtained as
VM_Resource_usage{ C , M }={ ∑ x × c(u ) , ∑m(u )}≈ {C_∗ C, M_∗ M} . (3)x ×Equation (3) gives the correspondence between A andVM and between VM and the load. The corresponding
complexity is O (np2) polynomial time. Thus, themigration problem belongs to NP-complete.

Notation

DescriptionmNumber of processors (i.e. m≧2)；expressed as
P=[p1,p2,…pm]

3.2. Minimizing the simultaneous migration time (i.e.,
non-preemptive schedule) is a NPC problemnNumber of tasks；expressed as J=[j1,j2,…jn]

C_T

Completion time valueT

Set of processing times of various tasks；expressed
as T=[t1,t2,…,tn]

VM migration can be performed in four different modes,
namely sequential, parallel, simultaneous, or delay. In
the simultaneous VM migration process, multiple VMs
are migrated simultaneously to different physicalservers, as shown in Figure 2. In many-to-one
communications, many TCP connections compete forthe same fixed resources of the switch and router during
the migration process, and thus buffer overflows readily
occur and cause packet losses. In order to facilitate thediscussion, assume that congestion and retransmission
do not occur and B>b1+b2+b3+…+bn, where B, b1, b2,…,
bn are the link bandwidths. Under this condition, theoverall migration time can be reduced by moving morethan one VM at the same time. However, when
consolidating the VMs at the source host side into
batches for migration, it is first necessary to check
whether the resources at the destination side aresufficient to enable the migration to be successfullycompleted. As described in Section 3.1, this problem isverifiable in polynomial time. Consequently, thesimultaneous migration of multiple VMs on different
hosts to multiple destination hosts belongs to NP.
Let the simultaneous VM migration problem shown in
Figure 2 be transformed to the Minimum Finish Time
Non-Preemptive Schedule problem (a known NPC
problem). The related notations are listed in Table 2.
The Minimum Finish Time Non-Preemptive Schedule
problem assumes that each task is not interrupted during

N

Number of VMs to be migrated

M

Number of physical hosts

VM_distDistribution of N VMs on physical host; expressed
as VM_dist =[N1,N2,…NM]

VM_migrate

Subset of VM_dist to be migrated

execution and seeks a schedule which achieves acompletion time less than or equal to a pre-set value.
For example, assume that the processors are capable ofperforming parallel processing (i.e., each processor iscapable of processing each task) and, in the case ofidentical processors, the tasks are performed in parallel.
For such a scheduling scenario, the completion time isequal to the time at which the final task is completed. In
other words, the completion time can be expressedsimply as
n tasks
k1 tasks
C_T= max ( ) ∣:k2-k1 tasks+ max ( ) ∣:n-kp tasks+…+max ( ) ∣:m processors

There exists a one-to-one mapping between the
Minimum Finish Time Non-Preemptive Schedule
problem and the VMs on the physical hosts. Supposethat VM_migrate is distributed as follows:© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan47

Yan-Ren Chen, I-Hsien Liu, Keng-Hao Chang, Chuan-Gang Liu, Jung-Shian Li

Physical Host 1

Physical Host 2

Therefore, the simultaneous VM migration problem isalso an NPC problem.

…..

VM_migrate={[Subset_1 of VM_dist], [Subset_2 of VM_dist],…….
Physical Host m

4. Conclusion[ Subset_m of VM_dist]}

This study has proven that VM migration is an NPC
problem by reference to two known NPC problems,
namely the Sum of Subset problem and the Minimum
Finish Time Non-Preemptive Schedule problem.

where Subset_1 of VM_dist , Subset_2 of VM_dist
and…Subset_m of VM_dist is VM_dist are the subsets of
VMs assigned to Physicals Hosts 1, 2 and m,
respectively.
In other words, the VMs are consolidated into several
batches, and all of the VMs in one batch are migrated tothe target host at the same time. The VM migration time
consists mainly of migration overheads and the transfer
time itself. Thus, for each batch of VMs, the migration
time can be expressed as
T_migration time_batch =max(each of VM migration time
(Overheads + transfer time))∣
The total migration time is equal to the sum of the
migration times of the individual batches, i.e.,
T_total_migration_time_batch = T_migration_time_batch 1 + T_migration_time_batch 2 +…+ T _migration_time_batch m
The correspondence between the minimum of thesimultaneous migration time and the Minimum Finish
Time Non-Preemptive Schedule problem is given by:processors↔physical host resource, tasks↔VMs (where
each VM cannot be interrupted as it migrates) ↔ eachtask cannot be interrupted as it executes. In other words,
the output of the minimum of the simultaneous
migration time problem can be transformed to the
output of the Minimum Finish Time Non-Preemptive
Schedule problem. Thus, the following relation equation
is obtained for the Minimum Finish Time NonPreemptive Schedule problem:
T’ = ∑ max (T ) ∣,…,Minimizing the simultaneous VM migration time issimilar to solving the non-preemptive schedule forprocessing a set of tasks in such a way as to achieve theminimum finish time. The Minimum Finish Time NonPreemptive Schedule problem is known to be NPC.
From the transform above, it is seen that thesimultaneous migration time for the migration operationof VM vMotion ≤p minimum finish time of the nonpreemptive schedule problem. In other words, theproblem of determining the simultaneous migration timefor VM vMotion is polynomial time reducible to theminimum finish time non-preemptive schedule problem.

Acknowledgements
This work was supported by the MOST (Ministry of
Science and Technology), Taiwan under contracts
numbers MOST 107-2218-E-006-036- and MOST 1072221-E-006-140-.
References
1. Robert J. FOWLER, Michael S. PATERSON and Steven
L. TANIMOTO, 1981. Optimal Packing and convering in
the
plane
are
NP-Complete,
INFORMATIONPROCESSING LETTERS, Vol. 12, Iss. 3, 1981, pp. 133137.
2. Michail G. Lagoudakis, The 0–1 Knapsack Problem--An
Introductory Survey, citeseer. nj. nec. com/151553.
html, 1996.
3. Jun Gu, Paul W. Purdom, John Franco, Benjamin W.
Wah, Algorithms for the Satisfiability (SAT) Problem: A
Survey,https://www.cs.toronto.edu/~chechik/courses03/csc2108/
algorithms-for-satisfiability.pdf, 1997.
4. Lei Cui, JinpengHuai, Jianxin Li, Chunming Hu, Bo Li,
TianyuWo, Hussain Al-Aqrabi, Lu Liu , VMScatter:
Migrate Virtual Machines to Many Hosts, In: VEE’13,
ACM 978-1-4503-1266-0/13/03,. 2013, pp. 63-72.
5. Kangkang Li, Huanyang Zheng, and Jie Wu, Migrationbased virtual machine placement in cloud systems, Cloud
Networking (CloudNet), IEEE 2nd International
Conference, 2013, pp 83-90.
6. Rajalakshmi Shenbaga Moorthy et al, An Effective
Mechanism for Virtual Machine Placement using Aco in
IAAS Cloud, IOP Conference Series: Materials Scienceand Engineering, 225012227, Vol. 225 , Issue 1, pp.
012227, 2017, https://doi.org/10.1088/1757-899X/225/
1/012227.
7. Sai Qian Zhang, Pouya Yasrebi, Ali Tizghadam, Hadi
Bannazadeh, Alberto Leon-Garcia, Fast Network Flow
Resumption for Live Virtual Machine Migration on SDN,
2015 IEEE 23rd International Conference on Network
Protocols, 2015, pp. 446-452.
8. 7. Weizhe Zhang, Shuo Han, Hui He, Huixiang Chen,
Network-aware virtual machine migration in an
overcommitted cloud, Future Generation Computer
Systems, 2016, pp. 428-442.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan48

Control System for Maintaining Safe Following Distance while Driving
Kuo-Hsien Hsia

Department of Electrical Engineering, Far East University
No.49, Zhonghua Rd., Xinshi Dist., Tainan City 74448, Taiwan (R.O.C.)
Jia-Hong Cai

Department of Electrical Engineering, Far East University
No.49, Zhonghua Rd., Xinshi Dist., Tainan City 74448, Taiwan (R.O.C.)
Shu-Li Pai

Global Learning and Assessment Development, Greater China
Room 402, Building 2, No. 78, Maizidian Street, Chaoyang District, Beijing 100020, China
E-mail: khhsia@mail.feu.edu.tw, vic1004210564@gmail.com, dosg1222@qq.com
www.feu.edu.tw
Evgeni Magid

Laboratory of Intelligent Robotic Systems (LIRS), Intelligent Robotics Department, Higher Institute for Information
Technology and Intelligent Systems, Kazan Federal University, Russia

Abstract
Safety is the most important issue while driving. Many traffic accidents occur because of insufficient safe distance.
With the advancement of technology, there are cars with a front distance safety warning system. Some of them arewith fixed safe distance regardless of the speed of the car. Other advanced distance detection systems switch between
different modes such as high speed and low speed. In this paper, we proposed a driving safety system which candynamically adjust the safe distance between vehicles depending on the speed and weather conditions. This system
can be used to determine and maintain safe distances both in front and rear of a car. This idea has obtained an invention
patent of Taiwan. A car with this system can be driven more safely.
Keywords: Driving safety, safe distance, driving speed, weather conditions.

(i)

Not maintaining the driving safety distance(50.5%)(ii) Not appropriately changing lanes or directions(12.8%)(iii) Not paying attention to the state in front of the car(10.3%)(iv) Other violations that caused accidents (8.0%)(v) Backing the car without accordance withregulations (2.2%)(vi) Wheel shedding or tire burst (1.8%)

1. Introduction
Living in this era, transportation is inseparable frompeople's daily lives. Many people often drive to work or
travel. For some people, driving is their work. So, driving
safety is an absolutely important part for them.
According to the annual report of the Freeway Bureau,
Ministry of Transportation and Communications(MOTC), Taiwan, the reasons for the top ten accidents
on the super highway of Taiwan in 2015 are as follows:1© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan49

Kuo-Hsien Hsia, Jia-Hong Cai, Shu-Li Pai(vii) Loading goods not safely fixed (1.3%)(viii) Drunk driving (1.1%)(ix) Vehicle parts falling (0.7%)(x) Not maintaining driving safety interval (0.6%)
Traffic accident analysis reports in many countries
have similar conclusions. Statistics of traffic accidents
show one of the most common causes of traffic accidents
to be the “failure to observe a safe distance behind
another vehicle.2” There are three safe distance requiredin car-following3: front safe distance, rear safe distance,and lateral safe distance, as shown in Fig. 1. Everybody
knows that keep safe distance in driving is very important.
Considering the three safe distances, the front distancecan almost completely controlled by the driver. Studies
have shown that leaving more space between vehicles
helps to reduce traffic accidents and allows for smoothertraffic.4
There are many factors that may have influence onsafe distance. For example, speed of the vehicle, driver’s
attentiveness, size and condition of the vehicle, roadsurface, and weather conditions are all factors of thedriving safety distance. An article of WHO (World
Health Organization) shows that speed has been
identified as a key risk factor in road traffic injuries,
influencing both the risk of a road crash as well as theseverity of the injuries that result from crashes.5 A goodsuggestion for keeping safe driving distance is “twosecond rule.” The rule is that a driver should ideally stayat least two seconds behind any vehicle that is directly infront of his or her vehicle.6 However, counting seconds
may cause distracting for the driver. Another guiding rule
about safe distance (in meter) on the super highway in
Taiwan is half of the speed (in km/h) for small cars andspeed minus 20 for large cars.
With the advancement of technology, there are cars
with a front distance safety warning system. Some ofthem are with fixed safe distance regardless of the speedof the car. Other advanced distance detection systems
switch between different modes such as high speed andlow speed. However, from earlier discussions we knowthat the safe following distance closely regards to the
speed of the vehicle. In this paper, we will propose adriving safety system which can dynamically adjust thesafe distance between vehicles depending on the speed
and weather conditions.

Fig. 1. Distances that should be concerned by the driver.

Fig. 2. System architecture of the proposed system.

Our system is primarily designed to alert drivers when
there is insufficient safe distance from the vehicle in front.
The system architecture is shown in Fig. 2. We modified
the rule of safe distance promulgated by the Freeway
Bureau, MOTC, Taiwan by considering the environment
parameters such as uphill/downhill, heavy rain, and thickfrog. Hence the driving safety distance can be adjusted
dynamically. The function of the safety distance in frontof the car is shown as Eq. (1).

df ( v, r , g , s )v* (1 r g s )for small car
2(v 20) * (1 r g s ) for large car

whe re 0(1)

r , g , s 1.

Here d is the safe distance, v is the speed of the car, r is
the status of rain, g is the status of frog, and s is the slopeof uphill or downhill. The parameter r is 0 when it is nota rainy day, and is 1 when it is heavy rain. The parametersof g and s are defined in similar way. For example, asmall car is driven on the highway at a speed of 110 km
per hour, with clear weather and non-uphill/downhill
sections, the safety distance should be 55 meters. For the

2. System Architecture© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

50

Control System for Maintaining

same speed but in the downhill section, the safety
distance should be more than 55 meters.
This system can be integrated into the driving
computer to form a complete driving safety system, or it
can be an independently-operated driving safety assistantsystem. For the integrated system, the system can remindthe driver to slow down or automatically activate thebrake system to force the deceleration when the safety
distance is insufficient. For an independently operated
system, the system can only alert the driver to slow down
without any mandatory action.
The alarm unit can include an audio alert and a lightalert. The sound or light warning in the car can remindthe driver to properly slow down to avoid the danger ofchasing. If there is a light warning at the rear of thevehicle, it can alert the driver of the rear car that the safety
distance to the front vehicle and the vehicle in front canbe reminded that it is insufficient. It is recommended thatthe rear vehicle should be properly decelerated.
This structure has been awarded the invention patentof Taiwan with certificate No. I594215 in August 2017.

(a)

(b)

(c)

Fig. 3. Car attitude. (a) Horizontal, (b) Downhill, (c) Uphill.

3. Experimental Simulation
To promote our patent concept, we constructed a simple
simulation experiment system. An HT66F50 is used as
the central processing unit of the system. An ultrasonicsensor is used as the front distance detector. We use asliding-type variable resistor to simulate the throttle
acceleration system. In order to detect that the car is on
an uphill or downhill state, we use two mercury switches
configured as inverted V-type, as shown in Fig. 3, todetect the attitude of the car. If the front switch is closed
and the rear one is open, then the car is on a downhill
slope. If both switches are closed, then the car is on ahorizontal road. However, due to the acceleration willaffect the action of the mercury switch, the mercury
switch is not suitable for use on actual cars to detect
uphill or downhill. An alternative way for the
uphill/downhill detection is to use the relationshipbetween throttle and speed changes. Our experimental
car for simulation is shown in Fig. 4. All circuits arepacked into the cabin of a model car. An LCD for display
the message of simulation result is mounted on the roofof the model car. Fig. 5 shows that the car speed is at 76
to 77 km/h. No rain and no frog is detected at that time.
By (1), we know that the safe distance should be 38
meters, as shown in the LCD display. Fig. 5 shows that

Fig. 4. Our experimental car for simulation.

Fig. 5. An experimental result of dynamic adjustment of safe
distance.

the detected front distance is 20 meters, and hence thealarm system will be activated.
4. Discussions and Conclusions
In this paper, we proposed a driving safety system which
can dynamically adjust the safe distance between
vehicles depending on the speed and weather conditions.
This system can be used to determine and maintain safe
distances both in front and rear of a car. Our technology
has the following characteristics:© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan51

Kuo-Hsien Hsia, Jia-Hong Cai, Shu-Li Pai

dynamics, Math. Problems in Eng. 2014(604023), 2014,
pp. 1-7.
4. S. H. Chacon, The Effect of Driver Behavior on Freeway
Traffic Flow, Dept. Operations Res. & Financial Eng.,
Princeton University, 2008.
5. Road Safety – Speed, available at: http://www.who.int/
violence_injury_prevention/publications/road_traffic/wor
ld_report/speed_en.pdf, cited: Sep. 20, 2018.
6. Two-second rule, available at: https://en.wikipedia.org/
wiki/Two-second_rule, cited: Sep. 20, 2018.

The safe distance will be automatically adjusted withthe speed and weather conditions. The faster the
speed or the poor weather conditions, the longer thesafe distance is.
The safe distance will be automatically adjusted withthe road slope. The safe distance of the downhill roadwill be automatically extended.
With the front distance detection system, the vehicle
can be alerted when the safe distance is insufficient,
to remind the driver to decelerate properly to avoid
the occurrence of chasing.
If the system is mounted with the automatic brakeassisting system, the brake deceleration can be
automatically started in the case of insufficient safe
distance to improve the road safety.
With the rear distance detection system, in the case of
insufficient safe distance of the rear vehicle, thedriver should promptly accelerate or change the laneto avoid collision in the case of inconspicuous driving.
The technology can also be applied to the automatic
following system, which can keep the distance ofvehicles in a safe region to improve the driver's
acceptance of the following system.
In addition to being integrated into the safety designof a new car to improve the safety factor of the vehicle,
the technology can also be used for retrofitting oldvehicles as a driving safety facility.
This technology is applicable to the automobile
manufacturing industry and the automobile
maintenance industry. The installation on the vehicle
can greatly improve the driving safety factor.
The technology can also be applied to locomotives orvarious types of moving vehicles to improve thesafety factor of the vehicle.
This idea has obtained an invention patent of Taiwan. We
deeply believe that a car with this system can be driven
more safely.
References
1. 2015 National Road Accident Review Analysis Report,
Freeway Bureau, MOTC, Taiwan, July 13, 2016. (inChinese)
2. L. Zemanek and J. Prnka, The issue of observance of safe
following distance between vehicles in Germany, Eur.
Scientific J. Special Edition, 2015, pp. 448-461.
3. D. Qu, X. Chen, W. Wang and X. Bian, Modeling of carfollowing required safe distance based on molecular© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan52

Markerless Indoor Augmented Reality Navigation Device Based on
Optical-Flow-Scene Indoor Positioning and Wall-Floor-Boundary Image Registration

Chian C. Ho, Bo-Kai Wang, and Guan-Lung Liao

Department of Electrical Engineering, National Yunlin University of Science and Technology
Douliou, Yunlin County 64002, Taiwan
E-mail: futureho@yuntech.edu.tw, m10312027@yuntech.edu.tw, b10012153@yuntech.edu.tw
www.yuntech.edu.tw

Abstract
For markerless indoor Augmented Reality Navigation (ARN) technology, camera pose is inevitably the fundamental
argument of positioning estimation and pose estimation, and floor plane is indispensably the fiducial target of image
registration. This paper proposes optical-flow-scene indoor positioning and wall-floor-boundary image registration
to make ARN more precise, reliable, and instantaneous. Experimental results show both optical-flow-scene indoor
positioning and wall-floor-boundary image registration have higher accuracy and less latency than conventional wellknown ARN methods. On the other hand, these proposed two methods are seamlessly implemented on the handheld
Android embedded platform and are smoothly verified to work well on the handheld indoor augmented reality
navigation device.
Keywords: augmented reality, indoor positioning, image registration, navigation.

1.

model navigation devices fail to be evolved into the evergrowing wearable see-through devices, like eyeglasses,
helmet, or goggles. Even worse, staring at 2D (birdview)or 3D (overlook) virtual model navigation devices ondriving or on foot might cause careless accidents or
dangerous occurrences.
Augmented Reality Navigation (ARN) technology isthe best alternative choice and becomes more and more
fascinating in ubiquitous computing applications since it
can lay real-world navigation directions over what users
are actually seeing in front of themselves in the true
world.1-8 In general, according to the object-tracking
principles, ARN technologies can be divided into threemain categories: 1) marker, 2) markerless, and 3)
positioning. The marker ARN technology is the mostreliable one tracking the registered objects and projects
the computer graphics onto the registered objects basedon distinctive markers or landmarks.1,2 Nevertheless, alarge amount of deployment of distinctive markers or

Introduction

With the rapid development and wide deployment ofemerging technologies in ubiquitous computing field,
personal navigation system that can accommodate toperform well outdoors and indoors have drawn more andmore interest. This is because personal navigation device
offering personal destination awareness anywhere andanytime is essential to versatile ubiquitous computing
applications, like outdoor and indoor path directions,outdoor and indoor information guide, outdoor andindoor marketing advertisement, outdoor and indoorsocial networking, and so on.
However, conventional 2D (birdview) or 3D(overlook) virtual-model navigation devices are notintuitive enough to guarantee the proper perceptionalignment between virtual-model navigation scenario
and real-world navigation situation. Besides,
conventional 2D (birdview) or 3D (overlook) virtual© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan53

Chian C. Ho, Bo-Kai Wang, and Guan-Lung Liao

(a)

(b)
Fig. 1. Flowchart of (a) typical and (b) proposed markerless indoor ARN technologies.

landmarks for marker ARN technology are very
expensive and unrealistic. The markerless ARN
technology simply on the cost-effective basis of naturalor plain features, like points, lines, corners, textures, isactually recognized as the most practical but difficult
one.3-5 The positioning (a.k.a. markerless) ARN
technology is the most common and used one depending
only upon the available location coordinate from outdoor
Global Positioning System (GPS) or specific indoor
Wireless Sensor Networking (WSN) infrastructure.6-8
Due to the markerless characteristic of main
playground of outdoor and indoor real-world
environments, most of recent research activities andinterests in ARN technologies focuses on markerlessissue, especially for indoor scenarios.9-11 Fig. 1 (a) shows
the algorithm flowchart of typical markerless indoor
ARN device. In Fig. 1 (a), first of all, the indoor ARN

device has to finish the estimation of location coordinate
through external radio positioning infrastructure (e.g.,
WiFi, Bluetooth LE, RFID) or internal inertial
positioning unit at the step of location estimation, in thestage of positioning estimation. Meanwhile, the indoor
ARN device also finishes the estimation of orientation
angle through fusion of digital compass and Inertial
Measurement Unit (IMU) sensors at the step oforientation estimation, in the stage of positioning
estimation. Then, the indoor ARN device matches theestimated location coordinate and orientation angle of thetargeted device onto the built 2D map at the step of map
matching, in the stage of positioning estimation. Next,
the indoor ARN device has to build the critical nodes on
the built 2D map at the step of node graph, in the stage ofpath planning, and has to finish the shortest or fastest
navigation path evaluation to the destination on the built© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

54

Markerless Indoor Augmented Reality

2D map at the step of path searching, in the stage of path
planning. After accomplishing the floor plane
registration (including camera pose initialization) at thestep of plane coordinate, in the stage of image registration,andHomography-based
2D-to-3D
projective
transformation of the map and navigation path at the stepof projective transformation, in the stage of image
registration, the indoor ARN device can accurately
project the ARN path directions onto the markerless realworld floor plane through elaborate floor region
segmentation at the step of directions placement, in thestage of image registration. Afterward, when the camera(viewing) pose of the indoor ARN device varies, thetranslation/scaling and rotation variations of camera(viewing) coordinate in real-world coordinate can beproperly evaluated at the steps of translation/scaling
estimation and rotation estimation, respectively, in thestage of pose estimation. Finally, the ARN pathdirections on the screen of the indoor ARN device can berotated and deformed properly in object (image)coordinate at the step of projective transformation, in thestage of image refreshing, so as to be closely stuck onto
the markerless real-world floor plane through elaborate
camera pose estimation at the step of directions
rerendering, in the stage of image refreshing.
Among stages of the algorithm flowchart of thetypical markerless indoor ARN device in Fig. 1 (a), theaccuracy and reality of the ARN path directions is
actually sensitive and vulnerable to these three stages of
positioning estimation, pose estimation, and image
registration, especially to these three steps of location
estimation, translation/scaling estimation, and plane
coordinate. For markerless indoor ARN technology,
camera pose is inevitably the fundamental argument of
positioning estimation and pose estimation, and floor
plane is indispensably the fiducial target of image
registration. Thus this paper concentrates to address on
issues of camera pose estimation and floor plane
registration in the subsequent sessions.
This paper is organized as follows. Section 2 reviews
pros and cons of conventional well-known camera pose
estimation methods for the markerless indoor ARN
device, like frame-to-frame planar homographies andfeature-based optical flow. In addition, Section 2 also
reviews pros and cons of conventional well-known floor
plane registration methods for the markerless indoor
ARN device, like Otsu binarization floor plane

registration and moving-average binarization floor plane
registration. Section 3 presents optical-flow-scene indoor
positioning and wall-floor-boundary image registration
to improve the accuracy and reality of the ARN pathdirections. Section 4 compares the experimental results
between these proposed two methods and conventional
well-known methods. Section 5 demonstrates theimplementation results of Android-based portable indoor
ARN device based on proposed optical-flow-scene
indoor positioning and wall-floor-boundary image
registration. Finally, Section 6 draws brief conclusions
and future work.
2. Conventional Methods
There are many conventional camera pose estimation
methods and floor plane registration methods applied tofor the indoor ARN device in the past 10 years. But theseconventional methods are rightly the dominant factors todegrade the accuracy and reality of markerless indoor
ARN technologies.
2.1. Conventional camera pose estimation
methods
Single planar homography can thoroughly interpret therelation between a real plane in 3D real-world coordinate
and its projective plane on the 2D image coordinate, so
frame-to-frame planar homographies10, 12-15 can estimate
the camera pose variation and some depth information
when the camera is rotated or translated. It is the mostpopular and reliable method for camera pose estimation.
However, the accuracy and reliability of camera pose
estimation of frame-to-frame planar homographies
depends deeply upon the accuracy and reliability offeature detection of the fiducial projection planar surface,e.g. floor plane. In addition, it is difficult to guarantee todetect correct and enough distinctive features on thefiducial projection planar surface frame by frame when
the camera is sometimes rotated or translated
dramatically.
Another widely-used camera pose estimation method
is feature-based optical flow. Feature-based optical flow
is more efficient and effective to track the distinctive
features and estimate the pose variation, but it can notestimate the depth information and viewpoint rotation.10,
13, 15© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan55

Chian C. Ho, Bo-Kai Wang, and Guan-Lung Liao

2.2. Conventional floor plane registration methods
All conventional floor plane detection or floor planesegmentation methods can be sorted into 2 maincategories: 1) Otsu binarization floor detection and 2)
moving-average binarization floor detection.15-18
The binarization threshold of Otsu binarization floor
detection is decided by the illumination statistical
distribution result of the scene. It can perform well andreadily on low-contrast scenes, but not on high-contrast
scenes. Because the illumination of the floor plane underhigh-contrast scenes is almost deeply dark.
On the other hand, the binarization threshold ofmoving-average binarization floor plane detection isdetermined by a series of averages of different lines on
the full-scale scenes. Although it can perform better onhigh-contrast scenes than Otsu binarization floor
detection, its computation complexity is too high to meetthe real-time requirement, especially on complicatedbackground scenes.

Fig. 2. Transformation model of camera pose estimation of
proposed optical-flow-scene indoor positioning method.

3. Proposed Methods
This paper proposes optical-flow-scene indoor
positioning and wall-floor-boundary image registration
to improve these three stages of positioning estimation,pose estimation, and image registration of typicalmarkerless indoor ARN technologies, especially to
improve these three steps of location estimation,translation/scaling estimation, and plane coordinate,illustrated in Fig. 1(b).

Fig. 3: Flowchart of proposed wall-floor-boundary image
registration method.

world coordinate, respectively. f implies the focal lengthof the camera. w denotes the height difference between
some optical flow pixel (P) and the middle point of
optical flow field on the present image plane, and w'
denotes the height difference between some optical flow
pixel (P') and the middle point of optical flow field on thedisplaced image plane, along Yi axis of 2D image-plane
coordinate. d indicates the real-world displaced distanceof the camera, that is the location difference between V
and V', along Yr axis of 3D real-world coordinate. The
unit of all variables above is centimeter.
Therefore, the real-world forward/backward
displaced distance of the camera can be elaborately
evaluated by (1)-(3), along Yr axis of 3D real-world
coordinate, while (V – C) is given from the difference
between known locations of the faraway wall (or obstacle)
and the present camera on the real-world map.

3.1. Proposed optical-flow-scene indoor
positioning method
Optical-flow-scene indoor positioning method with only
monocular RGB camera can solve the location estimationand translation/scaling estimation effectively andefficiently. Fig. 2 illustrates the transformation model ofcamera pose estimation from 2D image-plane coordinate
onto 3D real-world coordinate in proposed optical-flowscene indoor positioning method, and how to work out
the camera pose estimation issue. In Fig. 2, C means the
known location of the faraway wall or obstacle, along Yraxis of 3D real-world coordinate. h represents the height
difference between some real-world pixel and the middle
point of real-world field, along Zr axis of 3D real-world
coordinate. V and V' are the present and displaced realworld location of the camera, along Yr axis of 3D real-© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

56

Markerless Indoor Augmented Reality(V C )
h

f
w

h

w(V

C)

(1)

f

V' C f
hf
(2)
V' C
h
w'
w'
hf
f (V C )w
d V V ' (V C )(V C ) ( )()w (V C)(1
)
w'
w'
f
w'

(3)
As for the real-world horizontal or slightly verticaldisplaced distance of the camera can be proportional andreversely evaluated by the horizontal or slightly verticaldisplaced distance of some optical flow pixels on theimage plane.

Laboratory (11m)
MethodRFID
Wi-Fi
Feature-Tracking
Optical Flow
Optical Flow Field

Localization error
0.972 m0.712 mExecution time
0.524 sec
0.372 sec

0.773 m0.209 sec

0.463 m0.124 sec(b)
Corridor (14m)
MethodRFID
Wi-Fi
Feature-Tracking
Optical Flow
Optical Flow Field

3.2. Proposed wall-floor-boundary image
registration method
Fig. 3 illustrates the algorithm flowchart of proposed
wall-floor-boundary image registration. The detailed
descriptions on Fig. 3 will be explained step-by-step asfollows. Wiener Deconvolution filter is a key
preprocessing method for floor detection, because it canblur and simplify the background details. Then, strong
features of edges or contours can be preserved inherently.
Because the floor plane usually has fewer edges orcontours than the furniture or the wall decorations, edge
detector or contour detector can extract non-floor regions
smoothly and easily. Hough transform stands for
Probabilistic HoughLine Transform. It is an efficient andeffective way to detect the extremes of edge-lines. In
order to detect the representative edge-line features,
Directional Line filter is intended to reduce a number ofnoisy points or unconnected lines in a curve that is
approximated by a series of points. After acquiring theend dots (extremes) of edge-lines exactly, the contours
connected by every end dots are rightly the border offloor segmentation.
4.

Table 1. Localization error and execution time comparison
between proposed optical-flow-scene indoor positioning andconventional camera pose estimation methods under various
indoor spaces of (a) laboratory, (b) corridor, and (c) lobby.
(a)Localization error
0.532 m0.698 mExecution time
0.481 sec
0.371 sec

0.721 m0.198 sec

0.526 m0.112 sec(c)
Lobby (21m)
MethodRFID
Wi-Fi
Feature-Tracking
Optical Flow
Optical Flow Field

Localization error
0.482 m0.643 mExecution time
0.453 sec
0.312 sec

0.877 m0.203 sec

0.44 m

0.113 sec

Table 2. Accuracy and execution comparison between various
floor plane detection methods.
Floor Detection
Methods
Otsu binarization
moving-averagebinarizationwall-floor-boundary

Coverage
Accuracy
40%

Execution
Performance
7.19 fps65%

2.85 fps

88%

6.22 fps

latency than conventional camera pose estimation
methods, especially under high-obstructed environments.
Table 2 shows the coverage accuracy and execution
time comparison between conventional Otsu binarization
floor
detection,
conventional
moving-averagebinarization floor detection, and proposed wall-floorboundary detection. From Table 2, it is obvious that
proposed wall-floor-boundary detection has much higher
coverage accuracy and less execution latency thanconventional floor detection registration methods,
especially under high-contrast environments.

Experimental Results

Table 1 shows Localization error and execution timecomparison between conventional RFID indoor
positioning, conventional WiFi indoor positioning,
conventional feature-based optical flow indoor
positioning, and proposed optical-flow-scene indoor
positioning, under various indoor spaces of (a) laboratory,(b) corridor, and (c) lobby. From Table 1, it is evident
that proposed optical-flow-scene indoor positioning has
much higher localization accuracy and less execution

5.

Android Implementation

Android embedded platform with rich HW/SW features
is the best choice to implement and verify the portable© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan57

Chian C. Ho, Bo-Kai Wang, and Guan-Lung LiaoIEEE Journal of Biomedical and Health Informatics, vol.
18, no. 1, Jan. 2014, pp. 368–374.
8. B. H. Thomas and C. Sandor, What wearable augmented
reality can do for you, IEEE Pervasive Computing, vol. 8,
no. 2, Apr.–Jun. 2009, pp. 8–11.
9. A. I. Comport, E. Marchand, M. Pressigout, and F.
Chaumette, Real-time markerless tracking for augmented
reality: the virtual visual servoing framework, IEEE
Transactions on Visualization and Computer Graphics,
vol. 12, no. 4, Jul.–Aug. 2006, pp. 615–628.
10. T. Lee and T. Hollerer, Multithreaded hybrid feature
tracking for markerless augmented reality, IEEE
Transactions on Visualization and Computer Graphics,
vol. 15, no. 3, May–Jun. 2009, pp. 355–368.
11. Y.-G. Kim and W.-J. Kim, Implementation of augmented
reality system for smartphone advertisements,
International Journal of Multimedia and UbiquitousEngineering, vol. 9, no. 2, Sep. 2014, pp. 385–392.
12. G. Simon and M.-O. Berger, Pose estimation for planar
structures, IEEE Computer Graphics and Applications,vol. 22, no. 6, Nov./Dec. 2002, pp. 46–53.
13. S. J. D. Prince, K. Xu, and A. D. Cheok, Augmented
reality camera tracking with homographies, IEEE
Computer Graphics and Applications, vol. 22, no. 6,
Nov./Dec. 2002, pp. 39–45.
14. M. Maidi, M. Preda, and V. H. Le, Markerless tracking formobile augmented reality, in Proceedings of IEEE
International Conference on Signal and Image Processing
Applications, Nov. 2011, pp. 301–306.
15. G. C. Barcelo, G. Panahandeh, and M. Jansson, Imagebased floor segmentation in visual inertial navigation, inProceedings of IEEE International Instrumentation andMeasurement Technology Conference (I2MTC), May
2013, pp. 1402–1407.
16. Ma Ling, Wang Jianming, Zhang Bo, and Wang Shengbei,
Automatic floor segmentation for indoor robot navigation,
in Proceedings of International Conference on Signal
Processing Systems (ICSPS), vol. 1, Jul. 2010, pp. V1684–V1-689.
17. L. F. Posada, K. K. Narayanan, F. Hoffmann, and T.
Bertram, Floor segmentation of omnidirectional images
for mobile robot visual navigation, in Proceedings ofIEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), Oct. 2010, pp. 804–809.
18. F. G. Rodriguez-Telles, L. Abril Torres-Mendez, and E. A.
Martinez-Garcia, A Fast Floor Segmentation Algorithm
for Visual-Based Robot Navigation, in Proceedings ofInternational Conference on Computer and Robot Vision(CRV), May 2013, pp.167–173.

markerless indoor ARN device. The proposed opticalflow-scene indoor positioning and wall-floor-boundary
image registration methods are smoothly integrated into
the portable markerless indoor ARN device and arecertainly verified to work well.
6.

Conclusions

Although Visual Simultaneous Localization And
Mapping (VSLAM) also can improve the issues of
positioning estimation, pose estimation, and image
registration, this paper proposes and implements twomore effective and efficient methods, optical-flow-scene
indoor positioning and wall-floor-boundary image
registration, to make markerless indoor ARN device
more precise, reliable, and instantaneous.
Acknowledgements
This work was supported in part by Ministry of Scienceand Technology, Taiwan, under Grant MOST 106-2221E-224-053.
References
1. Jongbae Kim and Heesung Jun, Vision-based location
positioning using augmented reality for indoor navigation,
IEEE Transactions on Consumer Electronics, vol. 54, no.
3, Aug. 2008, pp. 954–962.
2. O. Mohareri and A. B. Rad, Autonomous humanoid robot
navigation using augmented reality technique, inProceedings of 2011 IEEE International Conference on
Mechatronics (ICM), Apr. 2011, pp. 463–468.
3. S. DiVerdi and T. Hollerer, Heads up and camera down: avision-based tracking modality for mobile mixed reality,
IEEE Transactions on Visualization and Computer
Graphics, vol. 14, no. 3, May–Jun. 2008, pp. 500–512.
4. H. Hile and G. Borriello, Positioning and orientation inindoor environments using camera phones, IEEE
Computer Graphics and Applications, vol. 28, no. 4, Jul.–
Aug. 2008, pp. 32–39.
5. T. Oskiper, M. Sizintsev, V. Branzoi, S. Samarasekera,
and R. Kumar, Augmented reality binoculars, IEEE
Transactions on Visualization and Computer Graphics,
vol. 21, no. 5, May 2015, pp. 611–623.
6. A. D. Cheok and Li Yue, A novel light-sensor-based
information transmission system for indoor positioning
and navigation, IEEE Transactions on Instrumentationand Measurement, vol. 60, no. 1, Jan. 2011, pp. 290–299.
7. R. Hervas, J. Bravo, and J. Fontecha, An assistive
navigation system based on augmented reality and context
awareness for people with mild cognitive impairments,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan58

Development of a Handheld Gas Detector with IoT Function

Jr-Hung Guo

Department of Electrical Engineering, National Yunlin University of Science & Technology
No. 123, University Road, Section 3, Douliou, Yunlin 64002, Taiwan
Kuo-Hsien Hsia

Department of Electrical Engineering, Far East University,
No.49, Zhonghua Rd., Xinshi Dist., Tainan City 74448, Taiwan
Kuo-Lan Su, Chien-Wei Liu

Department of Electrical Engineering, National Yunlin University of Science & Technology,
No. 123, University Road, Section 3,Douliou, Yunlin 64002, Taiwan
E-mail: g9710801@yuntech.edu.tw, khhsia@mail.feu.edu.tw, {sukl, m10512059}@yuntech.edu.tw

Abstract
Gas detection is very cumbersome because the gas will diffuse and mix with other gases. A handheld gas detector
with IoT function using a single-chip controller will be developed in this paper. This detector consists of a variety ofdifferent gas sensors. Adaptive fusion method is used to integrate sensing information from different sensors to
improve the correctness and gas identification of this detector. Since this gas detection module is hand-held, we use
wireless communication interfaces such as Wi-Fi and LoRa with bands at 433/868/915 MHz to enable this gas
detector to be able to transmit data while moving. In addition, multiple gas detection modules can be used to form adetection network for large area detection. A built-in algorithm can assist in the correction and isolation of data frommodules or sensors in the detection network. Therefore, the gas detecting module can be more flexible in use.
Keywords: Handheld gas detector, Gas detection, Adaptive Fusion Method.

technique to detect CO [3]. Sören Dierks and Andreas
Kroll used remote sensing and sensor data fusion to
detect methane gas leakages [4]. From these previous
studies we know that the detection of harmful gases isvery difficult and costly. In this paper, we will develop a
handheld gas detector with gas sensors easily bought inthe market and integrate multiple communicationinterfaces to form the architecture of Internet of Things(IoT). These sensors used in this paper and the gases thatthey can detect are summarized in Table 1.
All sensors except TGS825 can detect multiple gases.
Hence an output of a sensor may correspond to different
concentrations of different gases. How to distinguishdifferent gases from the output signal is the key of the

1. Introduction
Harmful gas detection is always a hot research topic. The
main reason is that the environment in which we live isfull of harmful gases from various industries or daily life.
Easy to diffuse or mix with other gases is an importantfeature of gas, so how to properly detect harmful gases isvery difficult. There have been many studies in this issue
in the past. For example, José Luis Herrero et al.
proposed the use of electronic nose and web architecture
to form a system for detecting harmful gases [1]. Jasmine
Pramila Devadhasan et al. developed a gas sensor arraythat can be connected to a smart phone [2]. Ayushi
Paliwal et al. developed a sensor for coating Au on a ZnO
surface using a surface plasmon resonance (SPR)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan59

Jr-Hung Guo, Kuo-Hsien Hsia, Kuo-Lan Su, Chien-Wei Liu

Table 1. Sensor characteristics table.

entire detection system. In this paper, the sensor signals
are distinguished by a curve fitting method. The
redundant management method is used to confirm anddetermine the gas concentration. Also the Adaptive
Fusion Method is used to improve the correctness and therecognition rate of the sensor. In addition, multiple
detectors can form a gas sensing network via wirelesscommunication interface and IoT technology. This canincrease the accuracy of the gas detection system andsolve the gas diffusion problem simultaneously.
2. System Architecture
The gas detector developed in this paper is based on an
8051 series single-chip, and is combined with the sensor
signal processing circuit, communication interface, andother peripheral circuits. The block diagram of the gas
detector is shown in Fig. 1. Up to 8 gas sensors can beconnected for this detector. The results sensed by eachsensor are first passed through the signal processing
circuit and then sent to the A/D converter in the single
chip for digitization before operation and processing.
Due to the handheld design, we integrate a433/868/915 MHz tri-band multi-channel RF interface, a
Wi-Fi interface and a LoRa interface for wirelesscommunication. With this design, the detector cantransmit the data or calculation results of the sensors to
the host computer during moving. In addition, there arealso wired communication interfaces such as UART
(Universal Asynchronous Receiver/Transmitter) and
Ethernet communication interface. Hence the applicationof the whole module can be more flexible.
In addition, we use uIP [5] in the detector to handle
the network communication of the detector. The goal ofuIP is to enable an 8-bit or 16-bit microcontroller tocommunicate over the network using the TCP/IP
protocol. It can provide basic UDP and TCP services.
The code size of uIP is only a few kilobytes, and it only
requires a few hundred bytes of RAM to run. Fig. 2 is a
block diagram of uIP.
In the detector we use the TCP and UDP protocols inuIP. These two protocols are the most commonly used in
network communications. So we can develop different
applications according to the needs, or use the TCP
protocol to directly store the data of the detector into the
database. In addition, we set up Web Server in thedetector, so users can directly use the browser to operate
the detector, such as controlling the I/O, or setting the

Fig. 1. Handheld Gas Detector Hardware Block Diagram.

Fig. 2. Stack Structure of uIP.

sensor type or threshold. This makes the detector very
convenient to use and manage.
3. Hardware and Software
The main board of this gas detector is shown in Fig. 3.
We use a modular design for this gas detector, so either
the communication interface, the display module or thehuman machine interface can be replaced as needed. In
addition, 10 I/Os are reserved for applications or
extensions. An expansion board for gas sensors, as shownin Fig. 4, are designed for easily replacing or installing
required gas sensors as needed. The well-assembled gas
detector with cover and LCD display is shown in Figure
5.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan60

Development of a Handheld

4. Experimental Results

The monitoring system is developed by VB.NET. It
can graphically display the real-time data of theconnected gas detector, and store the data in the databasesystem. In addition, this monitoring system can also be
used to set the gas detector I/O or gas sensor parameters
to make the use of gas detectors more convenient. The
monitoring system is shown in Fig. 6.

We developed a handheld gas detector using an 8051
single chip with multiple communication interfaces
combined with an easy to buy gas sensor. Two gases at
an environment of 26.2ºC and 57.8% humidity are tested.
We use the MQ2 sensor as an example to illustrate how
RO is obtained. The voltage on RO with RO and RL in
series connection in the air under known condition is 1V,
and the variable resistance RL of the gas sensor is 5.1K .
The voltage divider rule gives: 5V*5.1K/(RO+5.1K) =1V resulting in RO 20.4K. So RO is set as 20K for
simplicity in the experiment. Applying voltage divider
rule to RS and RL in series, the voltage on RS is5V*5.1K/(RS+5.1K). The A/D value can be obtained bymultiplying the voltage by 5.1. Table 2 summarizes thedata for MQ2. Data for other sensors can be obtained in
the same way.
For the first test gas, the measured data is summarized
as Table 3. From the characteristics of each sensor, we
can have the result as Table 4. It can be seen from Table

Fig. 3. Main board of the gas detector.

Table. 2. MQ2 Sensor characteristics table.
Fig. 4. Extension board for gas sensors.

Fig. 5. Well-assembled hand-held gas detector.

Table 3. Sensed data of Experiment 1.
(a) Handheld Gas Detector with Web Interface.

(b) Handheld Gas Detector I/O Status in Web Interface.
Fig. 6. Handheld Gas Detector Web Interface.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan61

Jr-Hung Guo, Kuo-Hsien Hsia, Kuo-Lan Su, Chien-Wei Liu

4 that the detection results of more than half of the gas
sensors indicate that the gas is most likely to be LPG.
Moreover, all of MQ2, MQ5, MQ6, and MQ9 can detect
LPG, and the sensed concentration is quite close to 5000
ppm except for MQ2. The rest of the gas sensors are notspecifically designed for LPG detection, and the detected
gas concentration is also quite different from them. So it
can be concluded that this gas is most likely to be LPG.
For the second test gas, the measured data issummarized as Table 5. By applying the same analysis
procedure as what had done for the previous test gas, wefound that the test gas should be alcohol.

Table 4. Gas analysis of Experiment 1.

Table 5. Sensed data of Experiment 2.

5. Conclusion
We developed a handheld gas detector with multiple
communication interfaces. This detector uses gas sensors
easily bought on the market. In addition, we designed asensor connector (extension board) to make this gas
detector more convenient to use. We use multiple
communication interfaces on the gas detector and use uIP
to enable the detector to communicate via TCP/IP and
UDP. Therefore, we use the TCP/IP protocol to establish
a Web Server on the detector, allowing the user tomonitor the status of the detector through the browser.
Finally, we developed a monitoring system to monitor
the gas detector. In the monitoring system, all connected
detectors can be monitored in real time, and the data ofthese detectors can be instantly stored in the database.
Hence the entire detection system can be used in a moreconvenient way. In the future, we will continue to reduce
the size of the detector and strengthen the function of themonitoring system. A web-based monitoring system will
also be developed, and it is hoped that it will be appliedin various fields requiring gas detection in the future.

4. S. Dierks and A. Kroll, Quantification of methane gas
leakages using remote sensing and sensor data fusion,2017 IEEE Sensors Applications Symposium (SAS),(Glassboro, New Jersey, 2017) 1-6.
5. A. Dunkels. Full TCP/IP for 8-bit architectures. The FirstInternational Conference on Mobile Systems, Applications,and Services (MOBISYS ‘03), (San Francisco, California,
2003) 85-98.

References
1. J. L. Herrero, J. Lozano, J. P. Santos, J. A. Fernández andJ. I. S. Marcelo, A web-based approach for classifying
environmental pollutants using portable E-nose devices,
IEEE Intell. Syst. 31(3) (2016) 108-112.
2. J. P. Devadhasan, D. Kim, D. Y. Lee and S. Kim,
Smartphone coupled handheld array reader for real-time
toxic gas detection, Analytica Chimica Acta 984 (2017)
168-176.
3. A. Paliwal, A. Sharma, M. Tomar and V. Gupta, Carbon
monoxide (CO) optical gas sensor based on ZnO thin films,
Sensors and Actuators B: Chemical 250 (2017) 679-685.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

62

Research on Employee Attribute Correlation of Information
Security Awareness in Organization
Tse-Yao Wang

Department of Aviation & Communication Electronic, Air Force Institute of Technology,
No. 198. Jieshou W. Rd., Gangshan Townwhip
Kaohsiung County, 820, Taiwan (R.O.C.) *u-Hsiang Wen

PHD program in Engineering Science and Technology,
Department of Engineering at National Kaohsiung University of Science and Technology,
1 University Rd., Yanchao,
Kaohsiung, Taiwan (R.O.C.)
E-mail: harry0716@gmail.com, rsbmb751@gmail.com
www.afats.khc.edu.tw, www.nkust.edu.tw

Abstract
Enterprises are attacked by phishing emails, causing significant business losses. The information security protection
strategies adopted by enterprises, such as mail filtering, phishing attack website screening and phishing website
blacklists, cannot guarantee complete defense. This study attempts to propose another perspective that can reduce therisk of corporate information security. The quality of employees' information security used as an indicator, and the
Logistic regression model used as a tool to analyze the key factors that can be used to measure the quality of
information security awareness.
Keywords:. Phishing, Logistic regression, Information security awareness.

1.

attacks. Although the email filtering strategy deployed in
the enterprise security protection framework, the successfactors of such attacks are often not the weakness of
information technology, but whether the employees in
the organization have a high level of information security
awareness.
In the context of armed phishing attacks where
technology cannot drip, the focus of corporate protection
should not be on the priority of technology but should
focus on providing the establishment of corporate
security culture. The information security awareness of
employees is the cornerstone of the enterprise's asset
security culture. Understanding the information security
awareness of employees has become an information
security issue worth studying.

Introduction

Information security threats have become an enormous
potential business risk for companies. Enterprises rely on
the Internet to support day-to-day operations, and the useof IT tools has become a target for potential attackers.
Regarding the most commonly used e-mail system, it isalready an essential daily application system for modern
enterprises. Implementing phishing email attacks against
email is the most commonly used attacking pipeline forcyber attackers. This low-cost and easy-to-implement
cyber attack technology can penetrate by the official
communication activities and private communication
activities of employees in the organization. Once theattacker successfully penetrates, it will launch further© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

63

Tse-Yao Wang , Fu-Hsiang Wen

2. Literature Review

the research finds that the picture presented. Better thanthe effect of a large number of text prompts [7].
Most of the subjects studied in the previous study wereanalyzed by students [5][4]. However, in recent studies,
the subjects of the survey are not limited to college
students [3][2][6], and the scope of sampling is closer toours. Real world observation. Also, the analysis methods
used for fishing sensitivity often use regression analysis[3] or chi-square independence test [5] to explain the
independent variables (such as gender, age) and phishingsensitivity. Whether there is a lack of significancebetween sexes, although the above methods can test the
crucial factors affecting fishing sensitivity, there is still
no further explanation for the interaction between
variables.

The primary method of phishing email attacks is to send
carefully designed emails to the victim's email address toentice the victim to click on the malicious URL link or
open the attachment file containing the malicious
program, causing the victim's computer. The malware is
implanted, and the attacker completes the purpose ofmanipulating the victim's computer [1]. Although thetechnical level of phishing attacks is not high, many
protection measures proposed for such attacks, such as
mail filtering and phishing blacklisting strategies.
However, it is still not useful to completely block suchattacks. Given this, some researchers have proposed toresearch the role of the injured party in the analysis of therole of the injured party in the form of data analysis. The
research on demographic characteristics analysis has
attracted much attention.
The most critical step in the successful implementationof the phishing attack is whether the target victim
chooses to trust the content indication of the email andcomplete the click action on the hyperlink provided by
the content. Therefore, individual differences in trust
relationships become essential indicators of observation.
A demographic analysis of victims of phishing attacks
from previous studies indicates that gender and age are
two significant predictors of observation [3]. Other
studies have found that it is easier to get a letter sent by
the opposite sex than a letter sent by the same sex [5], and
it is easier to submit personal information on the phishingwebsite. An analysis of the age of the victims indicatesthat ethnic groups between the ages of 18 and 25 havehigher rates of hooking than those of other age groups [3].
The general view is that information security curriculum
training can help to raise awareness of information
security. Some studies have looked at how responsive tophishing email attacks after training in information
security courses. The results of the study showed thatafter the implementation of the training course, thesubjects were more alert to phishing attacks [4]. Similaranti-phishing training studies have also pointed out thattraining courses are equally valid for increasing risk
awareness among men and women [6]. On the other hand,when the training system provides a presentation of
security cues, it will affect the participants' willingness toparticipate in the training (in the case of non-forced
training), and regarding the content of the training course,

3. Research methodology
In order to verify that information security awareness has
a positive impact on corporate information security
protection, we try to propose a phishing attack design
experiment to analyze the key factors affecting employee
information security awareness within the organization,and the quality of information security awareness of
employees within the organization. Can it become an
important indicator of corporate information security
risks? We also hope to explore further whether the impact
of information security awareness on information
security risks is a direct or indirect relationship. Throughthe research and analysis of the above issues, we canobtain a measurement method that can measure
information security awareness and the information
security risk of enterprises.

Fig. 1. Phishing attack test architecture.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan64

Research on Employee Attribute

4. Analytical Method
This study will use the Logistic regression model analysis
method to explore the relationship between candidate
auto-variables and dependent variables and identify the
key factors that lead to the quality of information security
awareness among employees.
Logistic regression formula:
x

x

....

x

p p
e 0 11 2 2E ( y)xx .... p x p
1 e 0 11 2 2.
References
1. Christian Ludl, Sean McAllister, Engin Kirda, andChristopher Kruegel, “On the Effectiveness of Techniques
to Detect Phishing Sites”, Detection of Intrusions and
Malware, and Vulnerability Assessment Lecture Notes in
Computer Science Volume 4579, 2007, pp 20-39.
2. Michael Workman,“wisecrackers: a Theory-Grounded
investigation of phishing and pretext social engineering
threats to information security”, Journal of the American
Society for Information Science and Technology, Volume
59, Issue 4, 15 February 2008, pages 662–674.
3. Steve Sheng, Mandy Holbrook, Ponnurangam
Kumaraguru, Lorrie Cranor, Julie Downs, “Who falls forphish? A demographic analysis of phishing susceptibility
and effectiveness of interventions”, Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems, CHI 2010, April 10-15, 2010, Pages 373-382 .
4. Ronald C. Dodge Jr., Curtis Carver, Aaron J. Ferguson,“Phishing for user security awareness”, Computers &Security, Volume 26, Issue 1, February 2007, Pages 73–
80.
5. Tom Jagatic, Nathaniel Johnson, Markus Jakobsson, andFilippo Menczer, “Social phishing”, Communications of
the ACM, Volume 50, Issue 10, October 2007, Pages 94100.
6. Ponnurangam Kumaraguru, Justin Cranshaw, Alessandro
Acquisti, Lorrie Cranor, Jason Hong, Mary Ann Blair,
Theodore Pham, ” school of phis：a real-word evaluationof anti-phishing training”, Symposium on Usable Privacy
and Security(SOUPS) 2009, July 15-17, 2009, Article No.
3.
7. Ponnurangam Kumaraguru, Steve Sheng, Alessandro
Acquisti, Lorrie Faith Cranor, Jason Hong ,“Teaching
Johnny not to fall for phish”, ACM Transactions on
Internet Technology (TOIT), Volume 10 Issue 2, May
2010, Article No. 7.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan65

Decoding spike patterns of auto-associative memory on spiking neuronal networks
Naoki Toyoshima

Graduate School of Engineering, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo113-8656, Japan
Takashi Kohno

Institute of Industrial Science, The University of Tokyo, 4-6-1 Komaba, Meguro-ku, Tokyo 153-8505, Japan
E-mail: tnaoki@sat.t.u-tokyo.ac.jp, kohno@sat.t.u-tokyo.ac.jp

Abstract
Spiking neuronal networks model the electrophysiological activities in neuronal cells' networks. One of theirapplications is auto-associative memory, in which one of the pre-stored patterns mostly similar to the input pattern isretrieved. In correctly retrieved states, neurons generate spikes periodically in two anti-synchronized groups that
correspond to the retrieved pattern. In this manuscript, we propose methods to evaluate the retrieved spatio-temporal
spike patterns by adding neurons that receive spikes from the original auto-associative memory network.
Keywords: spiking neuronal networks, auto-associative memory, synchronization, spatio-temporal spike patterns

Auto-associative memory is a well-known applicationof both SNNs and ANNs. In this task, one of the prestored patterns that is mostly similar to the input pattern
is expected to be retrieved. On SNNs, the retrieved
pattern is a spatio-temporal spike pattern in which thesynchrony between the spikes can be utilized to detect its
accuracy.2 Therefore, in order to realize practical SiNN
circuit modules for the auto-associative memory, it isnecessary to develop methods to interpret the spatiotemporal spike patterns as digital codes. Conventionally,some measures such as the overlap and phasesynchronization index have been used for theirevaluation, which require relatively complex calculation
including the exponent.
In this research, we propose methods to efficiently
interpret the spatio-temporal spike patterns in the autoassociative memory by some additional SNNs to theoriginal auto-associative memory SNN. By using SNNs,
we expect the interpretation will be realized powerefficiently in the future by implementation using ultralow
power analog SiNNs.
In the following sections, we first explain the SNN

Introduction
Researches aiming to realize information processing
systems that incorporate excellent characteristics of thebrain can be roughly classified into neuro-inspired andneuromimetic systems. In the neuro-inspired systems, thedynamical properties in the neuronal cells’ networks inthe brain is ultimately abstracted. For example, theconventional artificial neural networks (ANNs) are adiscrete time system. On the other hand, theneuromimetic systems try to mimic the dynamical
properties as precisely as possible.
Spiking neuronal networks (SNNs) are one of the neuromimetic systems. They model the electrophysiologicalactivities in the neuronal cells’ networks, in which spikes
emitted by the neuronal cells code information. They arecomposed of spiking neuron models which take thedynamics of the membrane potential of the neuronal cells
into account. Circuit implementation of SNNs that aims
to simulate them efficiently in terms of power
consumption and execution speed is called siliconneuronal networks (SiNNs)1.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan66

Naoki Toyoshima, Takashi Kohnoo(1 − )( > 0)(5),
− ( ≤ 0)where is the synaptic current, and are constants,
and is the membrane potential of the pre-synaptic
neuron. Figure 1 shows a sample waveform of membranepotential and corresponding synaptic current . It
models the transmitter release in chemical synapses.
in Eq. (1) is calculated by the
Stimulus current
following equation.
==

Fig. 1. A sample waveform of membrane potential
and corresponding synaptic current

whererepresents connection weight fromneuronneuron in the network, andrepresents antoarbitrary stimulus applied by the experimenter.

model used in this work. We describe two methods tointerpret the spatio-temporal spike patterns and theirsimulation results, and then conclusion.

Auto-associative memory
The auto-associative memory is a distributed memory
scheme in which one of the pre-stored patterns that ismostly similar to the input pattern is to be retrieved. In
this research, we use -dot binary (black/white) patterns
and a -neuron all-to-all connected network of DSSN.
Each pixel in the pattern corresponds to each DSSN.
Patterns are stored by the correlation learning5, in which
is determined as follows.

Model
Digital Spiking Silicon Neuron (DSSN) model
Generally, spiking neuron models represent the
dynamics of the membrane potential as differential
equations. DSSN model3 is one of the qualitative spiking
neuron models optimized for digital circuit
implementation. It qualitatively reproduces various
neuronal activities by relatively simple equations. It isrepresented as 2-variable differential equations asfollows.
=

where

( )≡

( ( )−

+

+

( +

) −

( < 0)=

1( ( ) − ),

),

=

1

( ≠ )

,

(7)

0( = )
whereis the value of the
pixel in thestoredpattern (black: +1, white: -1), and p is the number of thestored patterns.
In the associative memory on SNNs, it is known that the
spatio-temporal spike patterns converge to antisynchronized two groups that correspond respectively to
black and white pixels in the retrieved pattern when astored pattern is correctly retrieved2. The degree ofsynchronization depends on the input patterns. Example
patterns are shown in Fig. 2 (high synchrony) and Fig. 3(low synchrony). In these raster plots, a dot is placed at apoint that corresponds to the firing time and the neuron
index of each spike. Generally, input patterns farther
from the nearest stored pattern make the synchronization
more incomplete.

(1)

(2)

(3)
,
−
+ ( ≥ 0)( − ) + ( < )
( )≡
(4)
,
−
+ ( ≥ )
where, v represents the membrane potential and n
represents the abstract activity of the ionic channels.
represents a stimulus input from synaptic models. Theseequations are designed to reduce the number ofmultiplications which consume an amount of circuit
resources.
When
is low, membrane potential converges toconstant value (the resting potential). However, whenis higher than a particular threshold, the modelshows repetitive firing as shown in Fig. 1.
Our synaptic model4 is represented as a differential
equation as follows.
−

(6)

+ ,

Decoder network
In order to interpret the spatio-temporal spike patternsof the auto-associative memory, following two problems
have to be solved.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan67

Decoding spike patterns of

1.
2.

Fig. 2. Raster plot of a retrieved state with high synchrony

Fig. 4. Raster plot with dummy neuron (red dots)

Fig. 3. Raster plot of a retrieved state with low synchrony

Fig. 5. Raster plot with reference neuron (purple dots and lines)

Reference neuron

Which of the anti-synchronized neuron groups
corresponds to the black dots ?
To which group does each neuron belong ?

Another neuron is introduced to solve the second
problem. When the original network is in the antisynchronized states, the new neuron repetitively fires inthe blank periods between the two anti-synchronizedneuron groups. Therefore, the two groups can beseparated by the firing times of this neuron. We call thisneuron a reference neuron. Figure 5 is a raster plot where
the spikes of the reference neuron are represented bypurple dots and lines. We can see the two neuron groups
corresponding to black and white are correctly separated
by these lines.
The mechanism of the reference neuron is as follows. In
Fig. 6, the black and white plots representof theneurons in the network. In anti-synchronized states, theaverage of all neurons’ (the orange plot) has its peaks
at the centers of the two anti-synchronized groups and its
bottom at the blank period between them. Thus, thenegated average (the yellow plot) with a bias can be used
to stimulate the reference neuron so that it fires at theblank periods repetitively as shown in Fig. 7. The purple
plot is membrane potential v of the reference neuron.
Stimulus inputof the reference neuron is
calculated by the following equation.

Dummy neuron
We add a neuron to solve the first problem. The learning
process is modified so that the neuron is always
synchronized with the black group regardless of the input
pattern. We call it a dummy neuron.
The dummy neuron is introduced as follows. First, a
black pixel (a dummy pixel) is added to each of the stored
patterns each of which is composed of N pixels. These
patterns with (N+1) pixels are stored in a network of(N+1) neurons by ordinary correlation learning. The
dummy neuron is a ( + 1) neuron which corresponds
to the dummy pixel. The dummy pixel is also added tothe input pattern. The dummy pixel is black in all of thestored patterns, thus the dummy neuron is always
synchronized with the black group. The connection
weights from the original network to dummy neuron arecalculated by the following equation based on Eq. (7).
,

=

1

.

(8)

=

Connection weights from dummy neuron to the original
network are fixed to zero because it is not mandatory forthe retrieval process in the original network. Figure 4 is araster plot with the dummy neuron represented by red
dots. It is synchronized with the black group.

∙

−1

+ ,

(9)

where N is the number of neurons in the network andrepresents synaptic current ofneuron in the original
network. Positive constants A and B are appropriately
selected so that the reference neuron correctly works.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

68

Naoki Toyoshima, Takashi Kohnoo

Fig. 8. The stored pattern set used in the simulation

Fig. 6. of the neurons in the network (the black andwhite plots), the average of all neurons’ (the orange plot),
the negated average of all neurons’ (the yellow plot)
Fig. 9. Examples of the input patterns
based on the stored pattern 1

input patterns.
When error rate of the input pattern is higher than 30%,
the network sometimes converged to stable repetitive
firing states which could not be interpreted as any of thestored patterns as well as different from the original
pattern. As the future works, we will develop methods todetect such undesirable stable states by additional SNNs.
Fig. 7. of the neurons in the network (the black andwhite plots), the negated average of all neurons’
with a bias (the yellow plot), membrane potential of
the reference neuron (the purple plot)

Acknowledgement
This research was supported by JST SICORP and JSPS
Grants-in-Aid for Scientific Research “Fund for the
Promotion of Joint International Research”.

ResultWe executed simulations with many input patterns tocheck whether the proposed methods correctly work. The
number of pixels in the stored pattern (N) is 256. We usedfour patterns shown in Fig. 8 as the stored pattern set.
Any pair of the patterns in the set is orthogonal to eachother. We generated input patterns inserting errors byinverting randomly selected pixels in the stored patterns.
For each error rate of 10%, 20%, 30%, 400 input patterns(100 patterns for each stored pattern) were generated.
Figure 9 shows examples of the input patterns based on
the stored pattern 1.
With all of the input patterns, the network converged toanti-synchronized states and all of these states werecorrectly interpreted as one of the stored patterns by thedummy neuron and the reference neuron. However, whenthe error rate is 30%, some of the interpreted patterns
were different from the one from which the input pattern
was generated.

References
1.

2.

3.
4.
5.

Kohno. T, Li. J, and Aihara. K, Silicon neuronal
networks towards brain-morphic computers, Nonlinear
theory and its applications, IEICE, vol. 5, no. 3, (2014)
pp. 379-390
Li. J, Katori. Y, and Kohno. T, An FPGA-based siliconneuronal network with selectable excitability siliconneurons, Frontiers in Neuroscience, vol. 6, Article. 183,
pp. 1-13 (2012).
Kohno. T and Aihara, K, Digital spiking silicon neuron:concept and behaviors in gj-coupled network,
Proceedings of ISAROB, (2007) OS3–OS6.
Destexhe. A, Mainen. Z. F, and Sejnowski. T. J, Kinetic
models of synaptic transmission, Methods in Neural
Modeling, 2nd Edn (1998), pp. 1–25.
Hopfield. J. J, Neural Networks and Physical Systems
with Emergent Collective Computational Abilities, Proc.
Natl. Acad. Sci. USA, vol. 79, (1982) pp. 2554-2558

Conclusion
In this manuscript, we proposed two methods, thedummy neuron and the reference neuron, to interpret the
spatio-temporal spike patterns. We confirmed that thesemethods work correctly by simulations using many
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan69

Possible Mechanism of Internal Visual Perception: Context-dependent Processing by
Predictive Coding and Reservoir Computing Network
Hiroto Tamura

Graduate School of Engineering, The University of Tokyo*
Yuichi Katori

Future University Hakodate†Institute of Industrial Science, The University of Tokyo‡
Kazuyuki Aihara

Institute of Industrial Science, The University of Tokyo‡
E-mail: hiroto0324@sat.t.u-tokyo.ac.jp, katori@fun.ac.jp, aihara@sat.t.u-tokyo.ac.jp

Abstract
The predictive coding is a widely accepted hypothesis on how our internal visual perceptions are generated.
Dynamical predictive coding with reservoir computing (PCRC) models have been proposed, but how they work
remains to be clarified. Therefore, we first construct a simple PCRC network and analyze the nonlinear dynamics
underlying it. Since the influence of contexts is another important factor on the visual perception, we also construct
PCRC networks for the context-dependent task, and observe their attractor-landscapes on each context.
Keywords: visual system, perception, predictive coding, reservoir computing, context, nonlinear dynamics.

prediction made by the internal generative model. The
predictive coding also assumes that the generative modelis optimized to minimize the residual error between theprediction and the actual sensory input. In particular, thehierarchical predictive coding model1 postulates that the
top-down signals from the higher-order area carry thepredictions of lower-level neural activities, whereas thebottom-up signals from the lower-order area carry theresidual errors between the predictions and the actual
lower-level activities, so that the ascending signals havemuch less redundancy.
However, this model is only suitable for static visual
inputs and cannot deal with temporally changing visual
images, or movies. Then Fukino et al.2 proposed the

1. Introduction
It is widely known that what we see is not the visualsensory input as it is. Instead, our brains integrate thesensory inputs and reconstruct the internal image in themanner we can easily understand. For example, although
the actual visual input is 2D and received by both eyes,
what we see is the 3D vision as one image. However,exactly how the internal visual perceptions are generated
in the visual cortex has provoked much debate.
The predictive coding is one of the most acceptedhypotheses on the internal perception. In the predictive
coding framework, a perceived image is not merely the
integrated visual sensory input, but the result of the*

7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan
116-2 Kamedanakano-cho, Hakodate, Hokkaido 041-8655, Japan
‡
4-6-1 Komaba, Meguro-ku, Tokyo 153-8505, Japan© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan†70

Hiroto Tamura, Yuichi Katori, Kazuyuki Aihara

2.1. Network architecture and dynamics

predictive coding with reservoir computing (PCRC)
model, which can predict the temporally changing
auditory inputs, implementing the generative model by
the dynamical reservoir. Furthermore, the hierarchical
PCRC models for more complex auditory inputs were
proposed by Ara and Katori.3,4Here, the reservoir computing5 refers to a type of therecurrent neural network (RNN) approach with a simple
learning strategy. When the reservoir computing
networks are trained, only the output connections are
modified, and the recurrent and feedback connections are
fixed with randomly given values.
However, precisely how these PCRC models2,3,4
work largely remains to be clarified. Moreover, theseconventional models cannot perceive unlearned inputs.
In addition, they are not exactly driven by the prediction
error but by the sum of the error and their own prediction,
which is equal to the original sensory input.
In this study, therefore, we first modify them andconstruct a simple one-layer PCRC network exactly
driven by the prediction errors, which can perceive evenunlearned inputs. Then we analyze the nonlinear
dynamics underlying the trained network, in order toclarify the mechanism of the behavior.
The influence of contexts, which refers to situations,
goals, and relevant past experiences, is another importantfactor on the visual perception. For example, even
identical sensory stimuli can result in very different
perceptions depending on contexts. Indeed, RNN models
for context-dependent tasks have been proposed.6
Therefore, we also construct a PCRC network for a
simple context-dependent perception task. We analyze
the trained network again, in order to reveal how the
network perceives the sensory stimuli on each context.
We further construct a PCRC network which canperceive more high-dimensional visual inputs, in order toshow that the proposed network can be a possible
mechanism of the visual perception. We observe that themismatch between the context and the type of sensory
stimuli induces the perceptual error, which exhibits
complex visual features.

We use a leaky-integrator RNN, defined by the equation:
̇ = − +++( − ), (1)where is the membrane time constant, is the number( ), … , ( ) ∈ ℝ representsof neurons, ( ) ≔the membrane potentials or activities of the neurons attime∈ ℝ , and≔ ( ), … , ( ) ∈ ℝ
represents the firing rates of the neurons with ( ) ≔∈ℝ ×is a random recurrent
tanh( ) .
connectivity matrix, whose elements are sampled i.i.d.
is thefrom (0, / ) with the parameter ∈ ℝ.
dimension of the input and output. The output of the
network ≔∈ ℝ represents the prediction∈ ℝ × , whose
and is fed back through weights
elements are independently and uniformly sampled from[−1, 1]. The residual error between the target (or sensory
input) ( ) ∈ ℝ and the prediction ( ) is fed throughweights∈ℝ ×, whose elements are
independently and uniformly sampled from [−1, 1]. The
output weights∈ ℝ × are initially set to all zero,and modified during training. Note that the last term is
unique to the PCRC network. This network architecture
is illustrated in Fig. 1, though contexts are ignored in thissection.

Fig. 1. Schematic chart of the one-layer PCRC networkarchitecture. Only the output weights (red) are modified during
training.

In order to simulate this dynamics numerically, we
introduce the discrete-time version of Eq. (1), derived by
Euler method:

2. Simple PCRC

( + 1) =In this section, we construct a simple one-layer PCRC
network exactly driven by the prediction errors. We also
analyze the nonlinear dynamics underlying the trained
network to elucidate how it works.

+( ) + [– ( ) +( )+( ) − ( ) ],

( )

(2)

where ∈ ℤ is the discrete time step, is the small timeinterval, and other notations follow Eq. (1).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan71

Possible Mechanism of Internal

Throughout this paper, we use= 1000, =1.2, = 100 msec, and = 10 msec. In this section, we
use = 2 for visibility of the dynamics.
2.2. Task∈ ℝ in turn
We present the constant vectors , … ,as the sensory inputs to the network, whereis thenumber of trials. The network is trained to keep
until the next targetis
outputting the target
presented, at each th tial. Each sensory input
ispresented for 0.2 sec, and its elements are independently
and uniformly sampled from [1, 2].
Since the network actually receive the prediction
error − ( ) as the input, it is required to decode thiserror into the original sensory input . This corresponds
to the framework of the predictive coding.

Fig. 2. The response of the trained network in the test phase.
The 1st row represents the activities of the reservoir ( ), redplots in the 2nd and 3rd row represent the target ( ), green
plots in the 2nd and 3rd row represent the output ( ), and the
4th row represents the prediction error ( ) − ( ).

2.3. Learning rule
We trainby FORCE (Fast Order Reduced andControlled Error) learning algorithm,7 which is based on
the recursive least square filter. Its update rule is:( )≔( − Δ ) ( ) − ( ),( )≔ ( −Δ ) ( ) 1+

( ) ( −Δ ) ( )( )= ( −Δ )− ( )

(3)
, (4)

( ) ( − Δ ),(6)

where the initial value for ( ) ∈ ℝ × is given by

(7)

( )=

(0) =

( −Δ )− ( )( ),, (α ∈ ℝ).

In order to reveal the underling mechanism of thisbehavior, we analyze the nonlinear dynamics of the
trained network. In what follows, we regard the term( − ) as the external force and separate it from the
network’s own dynamics, because of its pulse-likebehavior. i.e., we here analyze the dynamics:(5)

̇ = − +

( )Throughout this paper, we use
2.4. Results and analysis

+

.

.

(9)

Following the approach of Sussillo and Barak8, we
define the scalar function ( ) ≔ | ̇ | /2, which is nearto zero if is an approximate fixed point, or a slow point.
Fig. 3A shows that almost all the values at the end oftrials are very low, and the corresponding slow points arelocated on a 2D-manifold in the phase space. Fig. 3A also
shows that at the beginning of each th trial, the pulselike prediction error − ( ) drives the trajectory out of
the 2D-manifold, but in the subsequent relaxation phase,the trajectory is attracted by the 2D-manifold, and the
projection of the trajectory onto the manifold
corresponds to the total movement −.
Furthermore, by analyzing the linearized system
around each slow point on the 2D-manifold, we uncover
the stability of this manifold. Linearizing Eq. (9) aroundthe slow point ( ( ) ≃ 0) , we obtain the dynamics
about the perturbation≔ − :In this algorithm, the inverse of ( ) is a runningestimate of the autocorrelation matrix of the firing rates( ) plus a regularization term:( )= ∫ ( )+(8)= 0.02 and Δ = .

We trained the network for 1000 trials. (i.e.,=
1000.) At each trial in the test phase, the sensory input
is presented for 1.0 sec. As shown in Fig. 2, thetraining resulted in almost perfect performance. Fig. 2
also shows that at the beginning of each th trial, theprediction error − ( ) is fed to the network as a sharp
pulse, but it immediately decays to zero and the network
settles into a fixed point
where ( ) ≡ .

wherė = [− + (
( )≔

+

( ).

) ( )]© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan72, (10)

Hiroto Tamura, Yuichi Katori, Kazuyuki Aihara

As for almost all the slow points, the linearized
systems around them have only eigenvalues with thenegative real part, as shown in Fig. 3B. This suggests that
almost all the slow points are locally stable, and the 2Dmanifold composed of them attracts any trajectories inthe vicinity of it. Nevertheless, this manifold attractor is
not fully continuous and there is a slow flow on it. Thenthe trajectory on the manifold is attracted by the specificslow point on the manifold where the output ( ) is nearto but not equal to the target , which leads to the little
prediction error shown in Fig. 2.

Fig. 4. The response of the trained network to a continuously
changing sensory input. (A) The activities of the reservoir ( )(the 1st row), the target ( ) (red plots in the 2nd and 3rd row),
the output ( ) (green plots in the 2nd and 3rd row), and theprediction error ( ) − ( ) (the 4th row). (B) The trajectoryof ( ) (orange) and the 2D-manifold composed of the stableslow points (blue) in 3D PCA space.

3. Context-dependent PCRC
In this section, we construct a simple PCRC network forthe context-dependent perception task. We also analyze
the trained network to elucidate how it switches theprocessing depending on contexts.
Fig. 3. (A) The locations of the slow points for the entire∈ [1,2] in 3D PCA (Principal Component
target range
Analysis) space. The color scale of each slow point represents
the value of ( ). The orange points represent an example ofthe trajectory of ( ) during a trial. (B) A typical example ofthe eigenvalue spectrum of the Jacobian at slow points on the
2D-manifold.

3.1. Network architecture and dynamics

Up to this point, the trained network has exhibited theperformance only on the discontinuously changing
sensory inputs. Here we show that the same network canalso perceive the continuously changing sensory inputs.
For example, Fig. 4A shows that the network output
succeeded in following the sinusoidal input. In this casethe network trajectory ( ) keeps travelling around the
2D-manifold in the phase space, as shown in Fig. 4B.
This behavior results from the balance between theattracting force from the 2D-manifold and the driving
force by the prediction error ( ) − ( ). Even in thegeneral case, the same mechanism enables the network toperceive the continuously changing input.
Throughout this section, we have shown the case of= 2 for simplicity, but the same scenario holds for thecase of general .

where the context ( ) ∈ ℝ is fed through weights∈ ℝ × , whose elements are independently anduniformly sampled from [−1, 1]. In this section we use= 4 and = 2 for simplicity, and the other settings
follow section 2.

We add to Eq. (1) the term of the context signal fromexternal modules:̇ = − ++( − )++,(11)

3.2. Task and learning rule
We present the constant vectors , … ,∈ ℝ in turnas the sensory inputs to the network, and train the
network to keep outputting each given constant vector
until the next target is presented. We also present thecontext ≔ (0, 1) during the 1st to /2 th trials, andthe context ≔ (1, 0) during the /2 + 1 th to th
trials, respectively. Each sensory input is presented
for 0.2 sec, and its elements are given as=, 1/ , , 1/on the context
and=, , /2 , /2 on the context , respectively. At
each trial
andare independently and uniformly© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan73

Possible Mechanism of Internal

sampled from [1, 2]. Note that the essential dimension of
the input is /2 on each context, but the trained network
is required to switch the type of processing depending oncontexts.
by FORCE learning algorithm used in
We trainsection 2.

match the context. In this case, we present each sensory
input
for 5.0 sec. Fig. 6 shows that the context
mismatch keeps the prediction errors apart from zero, sothat the network fails to perceive the sensory inputs, butnevertheless the network trajectories sometimes settle
into slow points.

3.3. Results and analysis
We trained the network for 1000 trials on each context= 2000.) At each trial in the test phase,and . (i.e.,the sensory input is presented for 1.0 sec. As shown in
Fig. 5A, the training resulted in almost perfectperformance, and the pulse-like prediction error drives
the network from one slow point to another, as with thecontext-free case in section 2. Fig. 5B shows that the twodifferent 2D-manifold attractors are formed for theand , respectively. This suggests that thecontexts
network switches its attractor-landscape depending oncontexts, and the same mechanism as section 2 enables
the network to perceive the sensory inputs on eachcontext.

Fig. 6. The perceptual errors induced by the context mismatch.
(Left:=, 1/ , , 1/on the context. Right:=, , /2 , /2 on the context .)4. Context-dependent PCRC for Visual Data

In this section, we construct a context-dependent PCRC
network which can perceive more high-dimensional
visual inputs, in order to demonstrate that the proposed
network can be a possible mechanism of the visual
perception. We also observe the complex features of theperceptual error induced by the context mismatch.
4.1. Network architecture, task, and learning rules
The network architecture and settings follow those of
section 3, except the dimension of the input and output:= 20.
We use the MNIST (Mixed National Institute ofStandards and Technology) data set, which is widelyused for handwritten numeral recognition tasks, as thehigh-dimensional visual sensory stimuli. As the
preprocessing, we first compress the MNIST data whoselabels are “0” or “1” into 20 dimension, using the NMF(Non-negative Matrix Factorization). We next randomly
choose one of the compressed MNIST data as the sensory
input and present it to the network for 0.2 sec at eachth trial. At the same time, we present the context if
has “0” label, and the context if has “1” label,
respectively. (i.e., each context represents the category of
the visual sensory input.) We train the network to keep
outputting the presented sensory input
during eachtrial. In the test phase, we present to the network
unlearned compressed MNIST data as the sensory inputs.

Fig. 5. The context-dependent response of the trained network.
(A) The activities of the reservoir ( ) (the 1st row), the target( ) (red plots in the 2nd and 3rd row), the output ( ) (green
and purple plots in the 2nd and 3rd row), and the prediction
error ( ) − ( ) (the 4th row). (Left: context. Right:
context .) (B) The locations of the slow points
for theentire target range,∈ [1,2] in 3D PCA space. (Blue:on the context . Red: on the context .)We next evaluate the performance of the trained
network when the type of the sensory input ( =, 1/ , , 1/or, , /2 , /2 ) does not© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan74

Hiroto Tamura, Yuichi Katori, Kazuyuki Aihara

by inversely transforming the output into the dimensionof the original MNIST data, using the matrix generated
in NMF. As a result, in the errored predictions, theoriginal sensory inputs and the predictions for the wrong
label MNIST image overlap each other, as illustrated inFig. 9.

The trained network is required to form slow points that
correspond to even unlearned MNIST data in its phasespace.
We use the FORCE algorithm again during training.
4.2. Results and analysis
We trained the network for 2000 trials on each context
and . (i.e.,= 4000.) At each trial in the test phase,
we present a randomly chosen unlearned MNIST data for
5.0 sec as the sensory input . Fig. 7A shows that thetraining network almost succeeded in perceiving
unlearned MNIST inputs, and the pulse-like prediction
error drives the network from one slow point to another,as with the case above. Fig. 7B shows that the twodifferent manifold attractors are formed for the “0” label
and “1” label MNIST inputs respectively, but in the 3D
PCA space we cannot observe the actual shapes of thesemanifolds.

Fig. 8. The perceptual errors induced by the context mismatch.
(Left: label “0” data are presented on the context . Right:
label “1” data are presented on the context .)Fig. 9. The examples of the visualized comparison between theand errored predictions on the context
original inputs
mismatch. (Left: label “0” data are presented on the context .
Right: label “1” data are presented on the context .)Fig. 7. The context-dependent response of the trained network
to unlearned MNIST inputs. (A) The activity of the reservoir( ), the output ( ) , the target ( ) and the prediction error( ) − ( ) . (Left: context . Right: context .) (B) The
locations of the slow points in 3D PCA space on the context(red) and on the context(blue), respectively.

5. Discussion
We first proposed the simple one-layer PCRC network
driven by the prediction error, which can perceive evenunlearned inputs. We analyzed the nonlinear dynamics
underling the trained network, and revealed that the
network perceives the sensory stimuli using the lowdimensional manifold attractor in its phase space. Since
low-dimensional manifold attractors have been also
observed in the trained RNNs in previous studies,6,8,9 it
can be a natural strategy for RNNs to use them for the
information processing.
Next, we constructed the simple PCRC network for thecontext-dependent task, and observed that the different

We next evaluate the performance of the trained
network when the label of the sensory input does notmatch the context. As shown in Fig. 8, the context
mismatch keeps the prediction errors apart from zero, butnevertheless the network trajectories settle into slow
points.
We further visualize these errored predictions at
slow points and compare them with the original inputs ,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan75

Possible Mechanism of Internal

5. M. Lukoševičius and H. Jaeger, Reservoir computing
approaches to recurrent neural network training, Computer
Science Review 3(3) (2009), 127-149.
6. V. Mante, D. Sussillo, K. V. Shenoy and W. T. Newsome,Context-dependent computation by recurrent dynamics in
prefrontal cortex, Nature 503(7474) (2013), 78-84.
7. D. Sussillo and L. F. Abbott, Generating coherent patternsof activity from chaotic neural networks, Neuron 63(4)(2009), 544-557.
8. D. Sussillo and O. Barak, Opening the black box: lowdimensional dynamics in high-dimensional recurrent
neural networks, Neural Comput. 25(3) (2013), 626-649.
9. C. Beer and O. Barak, Dynamics of dynamics: following
the formation of a line attractor, arXiv:1805.09603 (2018).
10. D. Collerton, J. P. Taylor, I. Tsuda, H. Fujii, S. Nara, K.
Aihara and Y. Katori, How can we see things that are notthere? Current insights into complex visual hallucinations,
Journal of Consciousness Studies 23(7-8) (2016), 195-227.

attractor-landscape is formed on each context.
Throughout this study, we used the PCRC networks with
only one layer and assumed the context signals to be fed
from the external module, for simplicity. However, thehierarchy plays a key role in the predictive coding
framework,1 and how the context signals are generated
remains to be clarified. Therefore, it is our future work tobuild the hierarchical PCRC model composed of the onelayer networks which are analyzed in this study, andincorporate the modules which generate the context
signals inside the model.
Finally, we constructed the context-dependent PCRC
network for the compressed MNIST data task, anddemonstrated that the proposed network can be a possible
mechanism of the visual perception. The perceptual
errors induced by the context mismatch exhibitedcomplex features, and interestingly, they share somecommon features with the symptoms of the hallucinationin dementia with Lewy bodies,10 in which the patients seeother people who are not there on the background which
actually exists there. It is also our future work to study
the relation between these perceptual errors.
Acknowledgements
The authors are grateful to T. Kohno for useful
discussions. This paper is partially supported by AMED
under Grant Number JP18dm0307009, NEC Corporation,
and JSPS KAKENHI Grant Number JP16K00246, andalso based on results obtained from a project subsidized
by the New Energy and Industrial Technology
Development Organization (NEDO).
References
1. R. P. Rao and D. H. Ballard, Predictive coding in the visualcortex: a functional interpretation of some extra-classicalreceptive-field effects, Nat. Neurosci. 2(1) (1999) 79-87.
2. M. Fukino, Y. Katori and K. Aihara, A Computational
Model for Pitch Pattern Perception with the Echo State
Network, in 2016 Int. Symp. Nonlinear Theory and Its
Application (Yugawara, Japan, 2016), 271-274.
3. K. Ara and Y. Katori, Hierarchical network model ofauditory information processing using dynamical
predictive coding, in 23rd Int. Conf. Artificial Life andRobotics (Beppu, Japan, 2018), 41-46.
4. Y. Katori, Network model for dynamics of perception withreservoir computing and predictive coding, Advances in
Cognitive Neurodynamics (VI), Chapter 11 (Springer
Nature Singapore Pte Ltd, Singapore, 2018).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan76

Snake robot controlled by biomimetic CPGs
Damien Blanchard

IMS, University of Bordeaux, France
Talence, 351 cours de la Libération, 33400, France
Kazuyuki Aihara

IIS, The University of Tokyo, JapanTokyo, 4-7-1 Komaba, Meguro-ku, 153-8505, Japan
Timothée Levi

IIS, The University of Tokyo, JapanTokyo, 4-7-1 Komaba, Meguro-ku, 153-8505, Japan

E-mail: damien.blanchard@u-bordeaux.fr, kohno@sat.t.u-tokyo.ac.jp, aihara@sat.t.u-tokyo.ac.jp, levi@sat.t.utokyo.ac.jp
www.iis.u-tokyo.ac.jp/ja/

Abstract
Locomotion is one of the most basic abilities in animals. Neurobiologists have established that locomotion resultsfrom the activity of half-center oscillators that provides alternation of bursts. Central Pattern Generators (CPGs) areneural networks capable of producing rhythmic patterned outputs without rhythmic sensory or central input. We
propose a network of several biomimetic CPGs using biomimetic neuron model and synaptic plasticity. This network
is implemented on a FPGA (Field Programmable Gate Array). We designed one unsupervised snake robot using thisnetwork of CPG. It is composed of one head wagon followed by seven slave wagons. Infrared sensors are also
embedded in the head wagon. This robot can reproduce the locomotion of one snake.
Keywords: snake robot, CPG, biomimetic, spiking neural network, locomotion.

swimming in salamander3 or lamprey4, and cardiac
pulsation system in leech5, 6.
Usually, in the field of robotics, CPGs are made using
simple neuron models7 or simple oscillators8, which are
not biological time scales and which are considered as
bio-inspired systems rather than biomimetic systems.
These models provide sinusoidal oscillations and cannot
reproduce biomimetic CPGs which have variability inperformance. As our aim is to replicate biological

1. Introduction
Locomotion is one of the most basic abilities in animals.
Neurobiologists have established that locomotion iscreated by Central Pattern Generator (CPG) activity that
produces alternating bursts1. Most rhythmic movements
are programmed by CPG (Central Pattern Generator)
networks2. CPGs are neural networks capable ofproducing rhythmic patterned outputs without sensory
input. CPGs can be found in animal locomotion such as© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan77

Damien Blanchard, Kazuyuki Aihara, Timothée Levi

behavior, we developed biomimetic CPGs using digitalneuromorphic system.

the Izhikevich Regular Spiking neurons, the activity ofHill's model was reproduced.

2. Biomimetic CPGs in digital neuromorphic
system
2.1. CPG neural network
The biomimetic CPG is based on the neural network
system that controls the heartbeat of a leech6. This simple
network uses 8 excitatory neurons with 12 inhibitory
synapses, making it an ideal candidate for ourapplications.

Fig. 2. Comparison of CPG activity between Hill's model(in blue) and our model (in black)

Even though the model was simplified, the same
behavior and characteristics as the biological CPGs was
successfully reproduced. The important point is to allow
a variability in the characteristics of the CPGs.
Percentages of variability can be controlled by adjusting
the parameters of short-term synaptic plasticity.
By changing the parameters of the inhibitory synapses, it
is possible to obtain different periods and cyclic ratios.
This is very important especially for closed-loop
experiments and to reproduce locomotion activities for
robotics. A single parameter pair manages themodification of the period and of the duty cycle. The
possible period interval in seconds is quite wide [0.3s;
35s].
2.3. Implementation in digital neuromorphic
systemFig. 1. Electrical activity of the neural system of theHeartbeat of the Leech. Neurons are represented by circles.
Inhibitory synapses are represented by black dots. A:
Electrical Activity from the Leech [HILL 2003]. B:
Diagram of an Elemental Oscillator of the Heart of theLeech. C: Diagram of the Segmental Oscillator of theHeart of the Leech.

Biomimetic CPGs12, 13, 14, 15 are implemented in a digitalplatform: CMOD A7-A35 Xilinx Artix-7 FPGA (Field
Programmable Gate Array). The neural network
implementation architecture operates on a single
computation core. This real-time digital system requires
few resources and low power consumption16, 17, 18, 19.
Table 1 summarizes the stated resources.

2.2. Simplification of the model
To design this CPG (Segmental Oscillator in Figure 1.C
C), Hill's neural network diagram6 was used.
The network designed by Hill uses the Hodgkin-Huxleymodel9 and complex equations for synapses. First work
was to simplify this model by applying Izhikevich
neurons10. Using the short-term synaptic plasticity11 and© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan78

Snake robot controlled by

Table 1. Utilization of the resources for the
implementation of biomimetic CPGs

Kintex-7
LUT
FF
BRAM
DSP

Utilization
7786
3299
50
40

Available
20800
41600
50
90

Percentage
37.43
7.93
100
44.44

Figure 3 describes the CMOD A7 FPGA board where
CPGs with different period are implemented.

Fig. 3. Three different CPGs with different period (1,7s,
3,6s and 5s). The output are from the scope.

Fig. 4. 3D view of the wagon design

As we implemented the biomimetic CPGs in the FPGA
board, we design the structure of the snake robot whichembedded the FPGA board.

All the distance between the different wagons and thepossible movement angle can be tuned with mechanicalstructure. Figure 5 shows how the tuning is possible.

3. Snake robot
3.1. Design
The snake robots is composed of 7 wagons (like spinalcord) and 1 locomotive (mimics the brain).
Biomimetic CPGs will control the different motors of thewagons. As the output of the FPGA board is 3.3V withlow current, a power driver for the motors and a VHDL
module for converting the CPGs to PWM (Pulse Width
Modulation) to control motor speed needs to be added.
The PWM frequency should be between 100 Hz and 200
kHz. The duty cycle of the PWM controls the speed ofthe motor. Pololu_DRV8835 was used for the power
driver. Motors are DG01D with a 0.8 kg.cm couple and ano load speed of 90 rpm.

Fig. 5. The orange screw allows the tuning of distance ofthe different wagons but also the angle for the movement.

The design of electrical connection and components forthe locomotive and for the wagon is described in figure
6.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan79

Damien Blanchard, Kazuyuki Aihara, Timothée Levi

Fig. 7. Forward locomotion with two examples of speed
using a chain of 7 CPGs for the 7 wagons. Time axis is one
second by division. Signal are the left neuron (N0, N2, N4,
N6, N8, N10 and N12) of each CPG.

The final snake robot is 59 cm long and works with realtime biomimetic CPGs and mimics the snake locomotion.
Figure 8 shows the final snake robot picture.

Fig. 6. On the left, the electrical circuit of the wagon. On
the right, the electrical circuit of the locomotive with the
FPGA board.

Fig. 8. Picture of the snake robot with the locomotive andthe 7 wagons.

The locomotive system has 3 IR sensors for detecting thedifferent obstacles and then send stimulus to CPGs formodifying duty cycle and period. The tuning of the duty
cycle allows the movement and the tuning of period
allows the speed or the stop of the robot.

4. Conclusion
This article describes the design of a snake robot. This
robot is controlled by real-time network of biomimetic
CPGs. This system can be used in robotic applications21
for a closer behavior to biological animals. It can also be
used in bio-hybrid robotics (biomimetic CPGs can be
replaced by in vitro cell culture). We previously showedthat our system can be connected to biological neurons22,
23. Another advantage is that the real-time systems cansimulate multiple CPG topologies and simulate different
hypothesis and protocol treatments for biomedical
applications such as spinal cord injury24 andneuroprosthetics25.

3.2. Results
Height CPGs were implemented (one for the locomotive
and 7 for the wagons). Enable input are connected to eachwagon to choose which ones are working for simulating
a lesion of the spinal cord.
The CPGs are connected for allowing forward
locomotion like we can find in animal locomotion20. Fig.
7 describes this CPG network and the forward
locomotion.

Acknowledgements
Authors thank Social Cooperation Program for BrainMorphic AI to Resolve Social Issues, IIS, U Tokyo.
References
1. T. Brown, On the nature of the fundamental activity of thenervous centres; together with an analysis of theconditioning of rhythmic activity in progression and atheory of the evolution of function in the nervous system.
Journal of Physiology, 48, 18-46, 1914
2. E. Marder and D. Bucher, Central pattern generators andthe control of rhythmic movements. Current Biology 11,
986-996, 2001
3. A. Ijspeert, A. Crespi, D. Ryczko, J. Cabelguen, Fromswimming to walking with a salamander robot driven by aspinal cord model. Science, 315, 1416-1420, 2007

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan80

Snake robot controlled by

4. AH. Cohen et al., Modelling of intersegmental
coordination in the lamprey central pattern generator for
locomotion. Trends in Neuroscience. 15, 434–438, 1992
5. GS. Cymbalyuk, Q. Gaudry, MA. Masino, R.L. Calabrese,
Bursting in Leech Heart Interneurons: Cell-Autonomousand Network-Based Mechanisms. J. Neuroscience, 22,10580-10592, 2002
6. AA. Hill, J. Lu, MA. Masino, OH Olsen, RL Calabrese, A
model of a segmental oscillator in the leech heartbeat
neuronal network, J Comput Neurosci., 10, 281-302, 2001
7. S. Amari, Characteristic of the random nets of analog
neuron-like elements, IEEE Trans. on System, Man andCybernetics, 2, 643-657, 1972
8. B. Van Der Pol, The heartbeat considered as a relaxation
oscillation, and an electrical model of the heart.
Philosophical Magazine, 6, 763–775, 1928
9. AL. Hodgkin, AF. Huxley, A quantitative description of
membrane current and its applications to conduction andexcitation in nerve, The Journal of Physiology, 117, 500544, 1952
10. E.M. Izhikevich, Simple model of spiking neurons. IEEE
Transactions on Neural Networks, 14, 1569- 1572, 2003
11. E.M. Izhikevich and G. Edelman, Large-scale model ofmammalian thalamocortical systems, Proceedings of the
National Academy of Sciences of the United States ofAmerica, 105, 3593-3598, 2008
12. J.H. Barron-Zambrano and C. Torres-Huitzil, FPGA
implementation of a configurable neuromorphic CPGbased locomotion controller, Neural Netw.45, 50–61, 2013.
13. C. Li, R. Lowe, T. Ziemke, Humanoids Learning to Walk :a Natural CPG-Actor-Critic Architecture, Frontiers in
Neurorobotics, 7:5, 2013
14. E. Donati, F. Corradi, C. Stefanini, G. Indiveri, A spiking
implementation of the lamprey's Central Pattern Generator
in neuromorphic VLSI, IEEE BioCAS 2014,2014
15. C. Qi et al., A real-time FPGA implementation of abiologically inspired central pattern generator network,
Neurocomputing, 244, 63-80, 2017
16. M. Ambroise, T. Levi, S. Joucla, B. Yvert, S. Saïghi, Realtime biomimetic Central Pattern Generators in an FPGA
for hybrid experiments. Frontiers in Neuroscience, 7, 215,
2013
17. M. Ambroise, T. Levi, Y. Bornat, S. Saïghi, Biorealistic
spiking neural network on FPGA, 47th Annual Conferenceon Information Sciences and Systems CISS, 1-6, 2013
18. F. Grassia, T. kohno, T. Levi, Digital hardware
implementation of a stochastic two-dimensional neuronmodel, Journal of Physiology Paris, 110:4, 409-416, 2016
19. T. Levi, Y. Guo, K. Aihara, T. Kohno, Study of real-time
biomimetic CPG on FPGA: behavior and evolution,
JRNAL, 4:4, 299:302, 2018
20. T. Matsuo, T. Yokoyama, D. Ueno, K. Ishii, Biomimetic
Motion Control System Based on a CPG for an
Amphibious Multi-Link Mobile Robot, Journal of BionicEngineering, 91-97, 2008

21. T. Levi, T. Nanami, A. Tange, K. Aihara, T. Kohno,
Development and Applications of Biomimetic Neuronal
Networks Toward BrainMorphic Artificial Intelligence,
IEEE Transactions on Circuits and Systems II, 65:5, 577581, 201822. S. Joucla et al.,, Generation of locomotor-like activity inthe isolated rat spinal cord using intraspinal electricalmicrostimulation driven by a digital neuromorphic CPG,
Frontiers in neuroscience, 10, 2016
23. T. Levi et al., Biomimetic neural network for modifying
biological dynamics during hybrid experiments, Journal
on Artificial Life and Robotics, May 201724. M. Capogrosso et al., A brain–spine interface alleviating
gait deficits after spinal cord injury in primates, Nature,539, 284-288, November 2016
25. T. Levi, P. Bonifazi, P. Massobrio, M. Chiappalone,
Closed-loop systems for next-generation neuroprostheses,
Frontiers in Neuroscience, 12-26, 2018

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan81

Real-time implementation of ReSuMe learning in Spiking Neural Network
Yang Xia*

Graduate School of Engineering, The University of Tokyo, JapanTokyo, 4-7-1 Komaba,Meguro-ku,153-8505, Japan
Seiji Uenohara

IIS, The University of Tokyo, JapanTokyo, 4-7-1 Komaba,Meguro-ku,153-8505, Japan
Kazuyuki Aihara

IIS, The University of Tokyo, JapanTokyo, 4-7-1 Komaba,Meguro-ku,153-8505, Japan
Timothée Levi

IIS, The University of Tokyo, JapanTokyo, 4-7-1 Komaba,Meguro-ku,153-8505, Japan
E-mail:xiayang@sat.t.u-tokyo.ac.jp, uenohara@sat.t.u-tokyo.ac.jp, aihara@sat.t.u-tokyo.ac.jp, levi@sat.t.u-tokyo.ac.jp
www.iis.u-tokyo.ac.jp/ja/

Abstract
Neuromorphic systems are designed by mimicking or being inspired by the nervous system, which realizes robust,autonomous, and power-efficient information processing by highly parallel architecture. Supervised learning was
proposed as a successful concept of information processing in neural network. Recently, there has been an increasing
body of evidence that instruction-based learning is also exploited by the brain. ReSuMe is a proposed algorithm byPonulak and Kasinski in 2010. It proposes a supervised learning for biologically plausible neurons that reproduce
template signals (instructions) or patterns encoded in precisely timed sequences of spikes. Here, we present a realtime ReSuMe learning implementation on FPGA using Leaky Integrate-and-fire (LIF) Spiking Neural Network(SNN). FPGA allows real-time implementation and embedded system. We show that this implementation can makesuccessful the learning on a specific pattern.
Keywords: Spiking neural network, ReSuMe, LIF, FPGA

robust, autonomous, and power-efficient information
processing by highly parallel architecture. There are threecommon methods to realize the neuromorphic circuits,
which are software1 2 3, analog hardware4 5 6 7 and digital

1. Introduction
Neuromorphic systems are designed by mimicking or
being inspired by the nervous system, which realizes© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan82

Yang Xia, Seiji Uenohara, Kazuyuki Aihara, Timothée Levi

hardware8 9 10 11 12. Software can implement simple
neuron model but a large scale neural network withcomplex neuron model cannot be realized in real-time.
The power consumption is also quite important (kW forsupercomputer).
For
hardwareimplementation,compared to analog circuits, digital implementations
consume more power but they are convenient to modify,
more portable and lower cost for implementation withFPGA devices.
Supervised learning was proposed as a successful
concept of information processing in neural network13.
Recently, there has been an increasing body of evidence
that instruction-based learning is also exploited by thebrain.
Remote Supervised Method (ReSuMe) is a newsupervised learning method for Spiking Neural Networks.
The main reason for the study of ReSuMe is the need toinvent an effective learning method to control the
movement of people with physical disabilities. However,
the in-depth analysis of ReSuMe method shows that thismethod is not only suitable for motion control tasks, butalso suitable for other practical applications, including
modeling, identification and control of various nonstationary and non-linear objects14 15.
In this paper, we present a real-time ReSuMe learning
implementation on FPGA using Leaky Integrate-and-fire(LIF) Spiking Neural Network (SNN). FPGA allows
real-time implementation and embedded system16.
We show that this implementation can make successful
the learning on a specific pattern.

where v(t) represents the membrane potential at time t,
is the membrane time constant and R is the membrane
resistance. This equation describes a simple resistorcapacitor (RC) circuit where the leakage term is due tothe resistor and the integration of I(t) is due to thecapacitor that is in parallel to the resistor. The spiking
events are not explicitly modeled in the LIF model.
Instead, when the membrane potential v(t) reaches a(spiking threshold), it iscertain thresholdinstantaneously reset to a lower value (reset potential)
and the leaky integration process described by Eq. (1)starts a new with the initial value .
Consider the case of constant input: I(t) = I. We assume= 0. The solution of Eq. (1) is then given by:

( )=

(2)

Eq. (3) describes the computing equation of dv, thensolution v = v + dv obviously.
2.2. Postsynaptic Potential and Spike Response
Model

By considering a single postsynaptic neuron i with amembrane potential
at time , a simplified SRM isdefined17.
( |Χ, ) ≔ ∑∑ ∈ ( − )(4)This section proposed three methods that applied to the
ReSuMe learning implementation on FPGA, which are
LIF-neuron model, Postsynaptic potential (PSP) andSpike response model (SRM), as well as ReSuMe
algorithm.

This SRM signifies a dependence of the neuron’smembrane potential on its presynaptic input pattern Χfrom synapses. An output spike occurs at a time .
The term of Eq. (4) describes a weighted summation of
the pre-synaptic input: thecorresponds to thesynaptic weight from a presynaptic neuron j, the kernelrefers to the shape of an evoked PSP. The PSP kernel
evolves according to

2.1. LIF neuron model
The LIF neuron is one of the simplest spiking neuronmodels. Due to the convenience with which it can beanalyzed, simulated especially implemented in digitalsilicon neural network, the LIF neuron is very popular17.
A neuron is modeled as a “leaky integrator” of its input
I(t):
( )

)

Here v(t) is in an exponential decay. In discrete digital
sequential circuit, a linear decay method is usually used
to optimize computing process for saving hardware
resources.
= [− + ](3)

2. Method=− ( )+

1 − exp(−( )= ∫exp(−) ( − ′)

′Θ( )

(5)

The term Θ( )is the Heaviside step function definedsuch that Θ( ) = 1 for ≥ 0 and Θ( ) = 0 . Here weapproximate the postsynaptic current’s time course by an
exponential decay18.

(1)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan83

Real-time implementation of ReSuMe( )=exp(− )Θ( )(6)

For a further simplified computer in digital circuit, theexponential decay ( ) substitute the PSP kernel
evolves ( ) approximately.

2.3. ReSuMe architecture and algorithm

An implementation of ReSuMe in the Liquid State
Machine (LSM) architecture is proposed as an example19.
The Liquid State Machine consists of a large, fixed“reservoir” network - the neural microcircuit (NMC)
from which the desired output is obtained by training thesuitable output connection weights.
In the implementation of ReSuMe method, the original
LSM approach has been modified. The modified
architecture consists of a set of input neurons, thewith a total
NMC structure, a set of learning neurons
number k and a corresponding set of teacher neurons(see Fig.1). NMC receives signal( ) fromandtransforms it into a vector of signals( ) which i ispresented to the learning neurons ∈ . The teacher
neurons
are not directly connected with any otherstructure.
Since we focus more on the ReSuMe learning
implementation itself on this paper, so we generated pre-Fig. 2. An example waveform of synaptic weight updating in
ReSuMe learning process.

3. Implementation
These section proposed implementation of above
methods we introduced above with results showed inwaveforms.
3.1. Implementation of LIF neuron
We implement LIF Neuron with VHDL language in
FPGA. By adjusting the size of dt and matching different
time constant, our LIF Neuron can work at very highclock frequency (10 kHz), which means that its
calculation accuracy is very high and the real-time
requirement is realized.

Fig. 1. ReSuMe implemented in the modified Liquid State
Machine Architecture.

synapse to the learning neuron as NMC output.
The modification algorithm, which adjusts weights
between pre-spike and post-neuron, is applied according
to the following simplified equation:

[

( )=[

+∫

(

( )−

)

( −

( )] ∗)]

Fig. 3. Stimulation of a LIF neuron by a constant input current:
the time-course of the membrane potential, v(t) with 15 mV
threshold and 20 mA current I (left); f-I curve for a LIF neuron(right).

(5)

The Fig.2 shows a specific weights update process.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan84

Yang Xia, Seiji Uenohara, Kazuyuki Aihara, Timothée Levi

3.5. Results and discussion

3.2. Implementation of PSP and SRM
As we introduced in 2.2, we use simplified exponential
decay to achieve RSM and PSP. Because post neurons
are connected to 500 pre-synaptic inputs, hardware
resources are still unacceptable if 500 exponential
operations (even in linear decay) are performed in thesame clock cycle. We adopt time division multiplexing,
and use two-stage pipeline to complete 500 sets of PSP
operations in 1000 clock cycles (actually 501 cycles,
remaining standby, theoretically supporting input ceilingof 999), just using one multiplier and one adder.

Fig. 6. Architecture of ReSuMe learning.

Firstly, we simulate the actual learning waveform of
ReSuMe. As shown in Fig. 7, we have completed thelearning of 500 input data containing target patterns andrandom patterns. Post neurons can spike the location ofeach pattern with a minimal delay in 2-4 time steps. Heretime step means cycle of computing and input patterns.
For example, if clock frequency is 100 MHz andcomputing needs 1000 cycles under our implementation,
the time step is 0.01 ms. Then we tested different input
pattern groups, and completed real-time test andverification on the FPGA.
4. Conclusion

Fig. 4. Waveform of PSP process in which indicates once presynaptic input spikes, the PSP starts an exponential decay.

3.3. Implementation of ReSuMe learning
ReSuMe we implemented includes an exponential
attenuation (linear attenuation) which attribute to changeof parameter k. Each time teacher input spike or postsynaptic neuron spike arrives, weight is updated. 500
exponential operations are performed. We use 500 clock
cycles, time division multiplexing to achieve this change,
and update weight in real-time so that PSP and RSM
modules are used to calculate the correct weight instantly.

This paper introduces the advantages of using devices
such as FPGA to realize the digital neural network.

3.4. Architecture of ReSuMe learning

Fig. 7. Waveform of different input patterns and their ReSuMelearning process. The third result shows that the minimum spikeinterval is 5 time steps.

Methods of LIF neuron, PSP and RSM module as well as
ReSuMe learning are described and illustrated on
hardware implementation. Then the overall framework of
ReSuMe learning is elaborated, with the different output
data due to different inputs analyzed.

Fig.5. Waveform of ReSuMe weight adjusting process which
achieves the ideal functions as showed in Fig.2

The overall hardware architecture of ReSuMe learning
is shown in Fig. 6. We use LIF Neuron as post-synaptic
neuron, equipped with ReSuMe learning module, and
500 pre-synaptic inputs are connected to post-synaptic
neuron. Each connection is operated by PSP andsummarized by RSM.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan85

Real-time implementation of ReSuMe

Acknowledgements
Authors would like to thank Prof. Takashi Kohno, from
IIS, the University of Tokyo.

15.

References
1. M.-O. Gewaltig and M. Diesmann, M., NEST (NEural
Simulation Tool), Scholarpedia 2:1430 (2007).
2. M.L. Hines and N.T. Carnevale, NEURON: a tool forneuroscientists, The Neuroscientist 7 (2001) pp. 123-135.
3. D. Goodman and R. Brette, The Brian simulator, Frontiersin Neuroscience 3(2) (2009) pp. 192-197.
4. A. Natarajan ans J. Hasler, Hodgkin-Huxley Neuron and
FPAA Dynamics, IEEE Trans Biomed Circuits Systems,
12(4) (2018), pp. 918-926.
5. F. Grassia, L. Buhry, T. Levi, J. Tomas, A. Destexhe and
S. Saïghi, Tunable neuromimetic integrated system foremulating cortical neuron models, Frontiers in
Neurosciences, 5:134 (2011).
6. N. Qiao, H. Mostafa, F. Corradi, M. Osswald, F. Stefanini,
D. Sumislawska, G. Indiveri, A reconfigurable on-line
learning spiking neuromorphic processor comprising 256
neurons and 128K synapses, Frontiers in neuroscience,
9:141(2015).
7. J. Schemmel, L. Kriener, P. Muller, K. Meier, An
Accelerated Analog Neuromorphic Hardware System
Emulating NMDA- and Calcium-Based Non-Linear
Dendrites, Proceedings of the International Joint
Conference on Neural Networks (2017)
8. R. Wang, G. Cohen, K. Stiefel, T. Hamilton, J. Tapson,and A. Van Schaik, A., An FPGA Implementation of a
Polychronous Spiking Neural Network with Delay
Adaptation, Frontiers in Neuroscience, 7:1–14(2013).
9. P. Merolla et al., A million spiking-neuron integrated
circuit with a scalable communication network andinterface, Science, 345:6197(2014) pp. 668-673.
10. M. Ambroise, T. Levi, S. Joucla, B. Yvert and S. Saïghi,Real-time biomimetic Central Pattern Generators in an
FPGA for hybrid experiments, Frontiers in Neurosciences,
7:215(2013).
11. Grassia F, Kohno T, Levi T, Digital hardware
implementation of a stochastic two-dimensional neuronmodel, Journal of Physiology Paris, 110:4(2016), pp. 409416.
12. Grassia F, Levi T, Doukkali E, Kohno T, Spike patternrecognition using artificial neuron and Spike-TimingDependent Plasticity implemented on a multi-coreembedded platform, Journal of Artificial Life and Robotics,
23:2(2018), 200-204
13. S. Ghosh-Dastidar, H. Adeli, A new supervised learningalgorithm for multiple spiking neural networks withapplication in epilepsy and seizure detection, J. Neural
Networks, (2009).
14. A. Kasin ́ski and F. Ponulak, Experimental demonstrationof learning properties of a new supervised learning method

16.

17.

18.
19.

for the spiking neural networks, Lecture Notes in
Computer Science, Vol. 3696, (2005), pp. 145–153.
M. Kraft, A. Kasin ́ski and F. Ponulak , Design of the spiking neuron having learning capabilities based on FPGA
circuits, Proceedings of the 3rd International IFAC Workshop on Discrete-Event System Design, (Rydzyna, Poland,2006),pp. 301–306.
L. Kechiche, L. Touil, and B. Ouni, Toward the
Implementation of an ASIC-Like System on FPGA for
Real-Time Video Processing with Power Reduction,
International Journal of Reconfigurable Computing, vol.
2018(2018)
W. Gerstner, and W. Kistler, Spiking Neuron Models:Single Neurons, Populations, Plasticity, Cambridge
University Press (2002).
B.Gardner and A.Gruning, PLOS ONE | DOI : 10.1371 /journal.pone.0161335(2016).
W. Maass, T. Natschlaeger, H. Markram. Computationalmodels for generic cortical microcircuits, Computational
Neuroscience: A Comprehensive Approach, (Chapman
and Hall/CRC, Boca Raton, 2004), pp. 575–605.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan86

Biomimetc Spike Timing Based Ionic Micro-Stimulation for Neuron Culture
Stephany Mai Nishikawa

Institute of Industrial Science, The University of Tokyo, Japan
Farad Khoyratee

IMS Lab, University of Bordeaux, France
Soo Hyeon Kim

Institute of Industrial Science, The University of Tokyo, Japan
Yoshiho Ikeuchi

Institute of Industrial Science, The University of Tokyo, Japan
Kazuyuki Aihara

Institute of Industrial Science, The University of Tokyo, Japan
Teruo Fujii

Institute of Industrial Science, The University of Tokyo, Japan
Timothée Levi

Institute of Industrial Science, The University of Tokyo, JapanIMS Lab, University of Bordeaux, France
E-mail: stephany@iis.u-tokyo.ac.jp, levi@sat.t.u-tokyo.ac.jp
www.iis.u-tokyo.ac.jp

Abstract
Neurodegenerative diseases are incurable and debilitating conditions that influence cognitive and/or motor functions
in millions of people worldwide. Neuroprosthesis are used today to support the quality of life but have yet to improvein their power consumption and biocompatibility issues. For the future realization of neuroprosthesis, understanding
the neurophysiological behaviors and the investigations on the interaction of neuronal cell assemblies is therefore
essential. Here, we propose a novel microfluidic system to investigate the response of the neurons directly stimulatedby the potassium ions in vitro in biomimetic timing.
Keywords: Ionic micro-stimulation, Neurons, Spiking Neural Network, Microfluidics

system which includes a real-time Spiking Neural
Network (SNN) and biomimetic ionic micro-stimulation
coupled to living ‘in vitro’ neuron culture. To simplify
the interactions with biological neurons and to reduce thebio-compatibility issues, we designed a biomimetic ionicmicro-stimulation using microfluidic techniques andionic exchange. Furthermore, to obtain a stimulation
close to biological one, a neuromorphic system called
Spiking Neural Network (SNN) is used. Firstly, the SNN
was integrated in a digital platform FPGA. Then, thespike-timing coming from SNN system is used fortriggering the microfluidic ionic micro-stimulation to
make the stimulation the closest to biological behavior.

1. Introduction
Most of neurodegenerative diseases are incurable anddebilitating conditions that influence cognitive and/or
motor functions in millions of people worldwide.
Neuroprosthesis are used today to support the quality oflife but have yet to improve in their power consumption
and biocompatibility issues. To realize a neuroprosthesisthat are close to biological behavior, understanding theneurophysiological behaviors and the investigations on
the interaction of neuronal cell assemblies is therefore
essential. Often being silicon based, this kind of
neuromorphic engineering is called biomimetic artificial
neural systems [1], [2], [3]. The main goal of such
systems is to design tools for biomedical applications
including neuroprosthesis and to understand the humannervous system. This project reveals a new bio-hybrid

2. Biomimetic spike-timing based microfluidic
system© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan87

Stephany Mai Nishikawa, Farad Khoyratee, Soo Hyeon Kim, Yoshiho Ikeuchi, Kazuyuki Aihara, Teruo Fujii, Timothée Levipneumatic valve to control the stimulation timing. The
channels are equipped with ion selective permeable
membrane (Sigma-Aldrich, Nafion® perfluorinated resin
solution 5wt.%) to mimic the chemical exchange of
biological neurons. It also has an integrated cell dish forthe placement of living neurons with a thinness of lessthan 5mm allowing the observation through microscopeduring calcium imaging.

There are three parts that compose the neuro-hybrid
system (Fig. 1): (1) Spiking Neural Network (SNN), (2)neuron culture and (3) ionic micro-stimulation in
microfluidic system. This system characterizes the neuralnetwork and its evolution by using biomimetic spiketiming based ionic stimulation. Each part will be
described in detail throughout this article.

Fig.2. Structure of ionic micro-stimulation device. Set of four
fluidic channels with under layer of pneumatic channel. The
crossing of these two different types of channels will allow tocontrol the input of solution in fluidic channel.

Fig. 1. Neuro-hybrid system with real-time digital SNN (green),
biomimetic ionic micro-stimulation in microfluidic system(blue) and biological neuron culture (pink). SNN electrical
output is converted into air pulse.

5. Neuron Culture
This bio-hybrid system contains ‘in vitro’ primarycultures of mice hippocampal neurons. Neurons were
carefully dissected from embryos and were prepared as
described [6]. Neurons isolated from embryonic day 17
ICR mouse were plated in Neurobasal medium. They
were used in this experiment after two weeks ofmaturation inside the cell dish of the microfluidic device.

3. Spiking Neural Network (SNN)The biomimetic SNN network is a neuromorphic systemthat is the closest detailed level of analogy to the nervous
system [4]. A network of silicon neurons is connected via
silicon synapses under plasticity rules. In neuro- hybrid
experiments, spike timing and shape of the action
potential (AP) must reproduce the same dynamics of abiological nerve impulse. This tunable biomimetic
system works in real time and is based on HodgkinHuxley (HH) formalism [5] which is the most bioplausible neuron model. We integrated the SNN in adigital platform FPGA. This spike timing-based system,
SNN, was connected to the air pressure controller totrigger ionic micro-stimulation on the microfluidic
device with stimulation closer to biological behavior.

6.

Results

6.1. Integration of Spiking Neural Network into
FPGA
Regular and simplified equations of biomimetic SNN are
implemented on a FPGA [8]. Electrical activities ofdifferent classes of cortical neurons can be simulated inbiological time scale with strong correlation between
hardware simulations (Fig.3. red) and software
simulations (Fig.3. blue). The Central pattern generators(CPGs) provide bursts of spikes with frequency,
variability and spike width in the ranges that arecommonly observed in nature [9]4. Ionic micro- stimulation in microfluidic
deviceThis device is structured with fluidic channels andpneumatic channels that allow the control of astimulation sent to the neurons. The structure is describedin Fig. 2 [7]. This device has four fluidic channels to input
potassium chloride (KCl) of 10mM with each having a© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan88

Biomimetic spike-timing based

Fig. 3. Comparison between model (blue) and FPGA (red).
FPGA implementation and Matlab software simulations arenearly similar.

Fig. 5. Stimulations were sent to the device with neuron culture.
After calcium imaging, the activity was seen accordingly to theprogrammed air pulse sent to the pneumatic valve.

6.2. Micro stimulation in the microfluidic system

After this successful stimulation, the device was
integrated with the Spiking Neural Network (SNN) to
acquire a more bio-inspired stimulation. This was doneby connecting the SNN implemented FPGA unit to thevalve of the air controller.

After two weeks of cell maturation, neuronal activities
were visualized by calcium imaging with Fluo-4 AM(Molecular Probes, United States). To observe the spatial
and temporal changes in calcium resulting fromspontaneous activity, the raw sequences were processedto observe changes in fluorescence intensity of when thebiomimetic ionic micro-stimulation is activated (Fig. 4).

6.3. Neural response to spike-timing based microstimulation

Fig.
4.
Bio-hybrid
experimentsusingbiomimetic ionic micro-stimulation. When micro-stimulation isON, the neuron culture is active. Calcium imaging technique isused for detecting neuronal activity.

When the 250ms length pulse of ON/OFF were sent to
the pneumatic valve, the ionic stimulation was received
by the neurons which in turn were stimulated bypotassium ion. This was confirmed by staining theneurons with calcium imaging. The neural activities(Fig.5.) successfully responded accordingly to theprogrammed ionic stimulation of 250ms. When thepneumatic microvalve is open, the fluorescent intensity
increases. On the other hand, when it closes, thefluorescent decreases. During when the pneumatic valve
is closed, the neurons activity returns back to its original
intensity range.

Fig.6. After calcium imaging, the neural activity without any
stimulation (top) and during the stimulation using SNN system(bottom) were recorded. The top graph shows the control
activity of neurons at resting state. The bottom graph shows theneural activity when CPGs-based burst-like stimulations weresent for 500ms. Images were taken at 200ms frame rate and thefluorescent intensity mean was plotted. The neural activity isvisualized when the florescent intensity increased
spontaneously.
Activity was seen accordingly to theprogrammed spikes with added delays of approx. 250ms each
spike.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan89

Stephany Mai Nishikawa, Farad Khoyratee, Soo Hyeon Kim, Yoshiho Ikeuchi, Kazuyuki Aihara, Teruo Fujii, Timothée Levi

The matured neurons were stained with calcium specificfluorescence to visualize the neural activity. The SNN
stimulation sent has 8 bust-like spikes for 500ms with
500ms of no signals between stimulation. Fluorescence
response observed shows a timely neural activity.
However, there were delays in the response time forneuron activity (Fig. 6.). This may have been caused by
the insufficient performance of the camera that was used.

7.

[4]

[5]

[6]

Conclusion

Our device can generate a biomimetic stimulationaccordingly to the sequenced air pressure programmedwith Spiking Neural Network (SNN). The activity ofneurons observed through calcium imaging showed
timely response to SNN signals sent with certain delays.

[7]

The neural response was as expected when triggering thestimulation manually and by programmed air pressure of
250ms with air pressure controller. During the 500ms of
OFF stimulation, the neural activities decreased as its
fluorescent intensity diminished. On the other hand, as
we integrated the Spiking Neural Network in the system,
delays in response time for the activity of neuron after
each stimulation were observed. However, during thisrecording, the computer had error for insufficientworking space (RAM) and the recording showed somelags. This might have caused the error of response timein neural activities.

[8]

[9]

Generators into FPGA for hybrid experiments,”
Frontiers in Neuroscience, 7, 215, 2013
M. Ambroise, T. Levi, Y. Bornat, S. Saighi,“Biorealistic spiking neural network on FPGA”,
47th Annual Conference on Information
Sciences and Systems (CISS), 1-6, Baltimore,
US, 2013
AL. Hodgkin and AF. Huxley, “A quantitativedescription of membrane current and its
applications to conduction and excitation in
nerve,” The Journal of Physiology, 117, 500544, 1952
G. Brewer, J. Torricelli, E. Evege, P. Price,“Optimized survival of hippocampal neurons in
B27-supplemented Neurobasal, a new serumfree medium combination,“ J Neurosci Res, 35,
567–576, 1993
S.M. Nishikawa, S.H. Kim, Z. Luo, T. Kirihara,
Y. Ikeuchi, T. Fujii and T. Levi, “Biomimetic
microfluidic
neuronsfor
Bio-hybrid
experiments”, Artificial Life and Robotics, 2018
T. Levi, F. Khoyratee, S. Saighi, Y. Ikeuchi,“Digital implementation of Hodgkin–Huxleyneuron model for neurological diseases studies”,
Artificial Life and Robotics, Artificial Life andRobotics 23 (1), 10-14, 2018
M. Ambroise et al., Biomimetic neural network
for modifying biological dynamics duringhybrid experiments, Journal on Artificial Lifeand Robotics, 22:3, 398-403, September 2017

In the future, we will try to record the neural activity witha camera capable of capturing higher frame rate persecond without having a lag during the recording of theneurons. On the other hand, the next step of our work is
to create a closed-loop bio-hybrid system where therecorded neuronal activity is used as a feedback for SNN.
References[1] M. Mahowald, R. Douglas , “A Silicon neuron,”,Nature, 354, 515-518, 1991
[2] G. Indiveri et al., “Neuromorphic silicon neuron
circuits ”, Frontiers in neuroscience, 5, 73, 2011[3] M. Ambroise, T. Levi, S. Joucla, B. Yvert, S.
Saighi, “Real-time biomimetic Central Pattern© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan90

Trajectory Tracking Control for a 7-Arms Robot Manipulator
Wang Jie

Department of Mechanical Engineering, Pusan National University,
Hyun-Hee Kim, Saad abbasi and Min-Cheol Lee*

School of Mechanical Engineering, Pusan National University,
635-4Jangjeon 2 dong, Geumjeong-gu, Busan, 609-735,South Korea*
E-mail: wj@pusan.ac.kr, saadjamshed93@gmail.com,mclee@pusan.ac.kr
me.pusan.ac.kr

Abstract
This paper proposes a trajectory tracking control method based SPO (sliding perturbation observer) for a 7-arms robot
manipulator. The 7-arms robot manipulator is designed to assemble small parts or packaging in industrial
manufacturing. System dynamics modeling is introduced at first. The dynamics of this 7-arms robot is difficult todetermine precisely caused by its uncertainties and many nonlinear terms through mathematical analysis. An
experimental identification process using signal compression method is applied to divide the linear term from
integrated dynamics with non-linear terms and uncertainties. Estimation of the non-linear terms and uncertainties forcompensating the real dynamics is obtained by SPO. Controller sliding mode control is designed based on estimation
perturbation.
Keywords: manipulator, robust controller, sliding mode control, perturbation observer.

1.

hydraulic manipulator with showing an excellent
performance in tracking. Before designing and applying
the above mentioned controllers, system dynamics isnecessary modeled at first. In this research, a 7 arms
manipulator is designed to simulate the assembling work
which mainly focus on trajectory tracking. The dynamics
of this manipulator is modeled using signal compression
method [3]. Signal compression method uses anequivalent impulse signal which is obtained from ansimilar impulse signal filtering by a time delay filter, thengiven as a tracking reference to the object system. This
identification process is introduced at first in this paper.
However, as same as mostly used identification method,signal compression method only estimates the divided
linear terms without any reflection from nonlinear terms
in original system dynamics. Therefore, compensation
for the nonlinear terms and uncertainties which are not

Introduction

Multi-axis robot arm are wildly used for industrial area,such as parts assembly, automatic welding in vehicle
manufacturing, pre-arrangement etc. Usually, thosekinds of robots are requested for an accurate trajectory
tracking. However, caused by those multi-axis arms the
traditional controller without consideration of dynamics
such as PID is not proper. Many robust controllers basedon system dynamics model are designed with robust
performance. In [1], a linear control logic LQR is
combined with PID to control a flexible manipulator.
Sliding mode control (SMC) is the most used one. The
designed nonlinear control input which restrains the error
states within sliding surface that ensures an outstanding
performance for a nonlinear system. In [2], a robust
controller SMC based SMCSPO is designed to control a© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan91

Wang Jie,Hyun-Hee Kim, Saad abbasi, Min-Cheol Lee

identified is necessary. Kalman fiter can estimate
uncertainties which is well known caused by its highaccuracy in low frequency area [4]. SPO has the similar
algorithm with kalman fiter, but derived from state space,which also has outstanding estimation performance in
low frequency area [5]. In this paper, we use the SPO toobserve the perturbation which contains the nonlinear
terms in dynamics, identification errors, uncertainties anddisturbance. The estimated perturbation is used fordesigning a robust controller such as sliding mode control.
This paper is organized as following: signal compression
method for system modeling is introduced in section 2,
the perturbation design is shown in section 3, weconclude this work in section 4.

Fig. 2. Expanded Impulse Signal.

The expanded signal is shown in figure.2. It obtained
from the designed impulse signal in Eq.(1) passed by antime delay filter with following function in Eq.(2):nP(n) = 60 exp[−( )12 ], 0  n  N / 2 − 1aP(n) = 0, n = N / 2(1)
P(n) = P( N − n), N / 2 + 1  n  N − 1

2. Dynamics ModelingSignal compression method is used for dynamics
identification. In this method, an equivalent impulse
signal is generated to implement on an objective system
as the desired trajectory. The equivalent impulse signal is
obtained by passing a time delay filter from a designed
impulse signal. The expanded impulse signal is available
to apply on the real system. Then, the response output isfiltered by the inverse of the time delay filter. The result
after the inverse time delay filter is supposed as the real
response which is from the origin impulse signal to thelinear system. The system can be modeled when wechange the model dynamics to match the correlation withthe output signal from real experiment. This logic isshown in figure. 1.

12n 2j ], 0  n  N / 2 − 1b
H (n) = 0, n = N / 2
H (n) = H ( N − n), N / 2 + 1  n  N − 1
H (n) = exp[−(2)

The robot manipulator are shown in figure.3. The
experiment of signal compression method is
accomplished with an simple P controller (Kp=10) which
logic is shown in figure(4).

Fig. 3. Real System .
Fig. 1. Signal Compression Method Logic.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan92

Trajectory Tracking Control for

Table 1. Result of MedelingFig. 4. Expanded Impulse Signals.

The output from compression signal of link7 is shown in
Fig .5. In Fig . 6, the result of real system output passedby the invers filter of Eq. (2) and model result with sameprocess are shown. All links modeling result are shownin Table.1 with the assumption of their dynamics are asimple 2 order linear system.
3.



n

1

0.642

6.864

2

1.521

2.937

3

0.324

7.628

4

0.61

5.521

5

0.346

12.469

6

0.37

13.21

7

0.35

12.502

2nd order.Model

Controller

In this section, we define the perturbation in amathematic equation. Before defining perturbation, thedynamics of a general robot system is normally definedasnx j = f j (x) + f j (x) +  [(b ji (x) + b ji (x))ui ] + d j (t )i =1

j = 1,..., n.

(3)




where x =[ X1 ...Xn ]T is the state vector and X j =[ x j , x j ]T . The
terms f j (x) correspond to linear driving terms whilef j ( x) corresponds to the nonlinear driving terms, theirparameters, dynamic modeling, and dynamic
uncertainties. The components b ji and b ji represent
the elements of the control gain matrix and theiruncertainties, while d j is the external disturbance andu j is the control input. It is assumed that the terms f j
and b ji can be obtained after system identification.
Perturbation is introduced simply in the introduction that
contain the nonlinear terms in dynamics, identification
errors, uncertainties and disturbance as shown in Eq. (4).

Fig. 5. Real System Response(link7).

n j (x, t ) = f j (x) +  [b ji (x)ui ] + d j (t ).
i =1Fig. 6. Correlation of Real System Response with Modeled
System Response Result(link 7).

The observer contains states are shown in Eq. (5).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

93

(4)

Wang Jie,Hyun-Hee Kim, Saad abbasi, Min-Cheol Leexˆ1 j = xˆ2 j − k1 j sat( x1 j ),ˆ ,xˆ2 j =  3u j − k2 j sat( x1 j ) + jxˆ3 j =  32 j (u j +  3 j xˆ2 j − xˆ3 j ),ˆ j =  3 j ( 3 xˆ2 j − xˆ 3 j ).

(5)

The stability of SPO is proved in [6]. The perturbation
estimation is used to design a robust controller. In here,
we use sliding mode control with its combination of SPO.
4.

Experiment and Conclusion

Fig. 9. Error in Tracking Result of link 2.

The experiment is designed for trajectory tracking on the
Windows based RTX software (real time operating
system). The application interface is made from MFC
API and shown in Fig. 7. It also can plot the movementof link 7.

Acknowledgments
This research was supported by the MOTIE (Ministry of Trade,Industry & Energy), Korea, under the Industry Convergence
Liaison Robotics Creative Graduates Education Programsupervised by the KIAT (N0001126) and the National Research
Foundation of Korea (NRF) grant funded by the Korea
Government (MSIP) (No.NRF-2012M2B2B 1055503).

References
1. Yun, Yuan, and Yangmin Li. "Active vibration control
based on a 3-DOF dual compliant parallel robot using
LQR algorithm." Intelligent Robots and Systems, 2009.
IROS 2009. IEEE/RSJ International Conference on. IEEE,2009.
2. Wang, Jie, Karam Dad, and Min Cheol Lee. "Bilateral
Control of Hydraulic Servo System Based on SMCSPO
for 1DOF Master Slave Manipulator." alife-robotics,(2017), International Conference on Artificial Life andRobotics.
3. Lee, Min Cheol, and Nobuharu Aoshima. "Identificationand its evaluation of the system with a nonlinear elementby signal compression method." Transactions of theSociety of Instrument and Control Engineers 25.7 (1989):
729-736.
4. Wan, Eric A., and Rudolph Van Der Merwe. "The
unscented Kalman filter for nonlinear estimation."
Adaptive Systems for Signal Processing, Communications,and Control Symposium 2000. AS-SPCC. The IEEE 2000.
Ieee, 2000..
5. Lee, Min Cheol, et al. "Reaction force estimation ofsurgical robot instrument using perturbation observer with
SMCSPO algorithm." Advanced Intelligent Mechatronics(AIM), 2010 IEEE/ASME International Conference on.
IEEE, 2010.
6. Jie, Wang, Karam Dad, and Min-Cheol Lee. "SMCSPO
based force estimation for a hydraulic cylinder." Control,
Automation and Systems (ICCAS), 2017 17th
International Conference on. IEEE, 2017.

Fig. 7. Real Time Tracking Result Using SMCSPO for link 7.

In Fig. 8, the reference trajectory of link 2 with blue line
and real trajectory with red line are presented. The error
between them is shown in Fig. 9.

Fig. 8. Tracking Result Using SMCSPO of link 2.

The tracking error shows within the limit about 0.3
degree. It verifies the sliding mode control with proposed
perturbation estimation has outstanding performance fora
multi-arm robot which dynamics is obtained by signal
compression method approximately.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan94

Estimation tire-rod friction coefficient based on mobile robot
Ji- Hyeon Kim

Dep. of Electronic Eng., Pusan National University, 2,Busandaehak-ro 63beon-gil Geumjeong-gu
Busan, 46241, Republic of Korea
Zhang Hongyu

Dep. of Electronic Eng., Pusan National University, 2,Busandaehak-ro 63beon-gil Geumjeong-gu
Busan, 46241, Republic of Korea\
Jang-myoung Lee

Dep. of Electronic Eng., Pusan National University, 2,Busandaehak-ro 63beon-gil Geumjeong-gu
Busan, 46241, Republic of Korea
E-mail : jihyeon1696@pusan.ac.kr, zhanghy0218@pusan.ac.kr, jmlee@pusan.ac.kr
www.pusan.ac.kr

Abstract
It is very important to estimate the friction coefficient of tire-road for vehicle dynamic control system. For the current
dynamic control research, a large amount of experimental data is needed. The theoretical tire model is used to estimate
the friction coefficient based on the Gim tire model to improve the tire experience model. Simulation by MATLAB,
the simulation results show that the Gim tire model has high precision and can predict the mechanical properties of
the tire well.
Keywords: simulation, dynamic control, Gim tire, frcition

important information for safe driving and dynamic
control of tire force, but they are limited by the friction
coefficient [1], and the tire model is roughly. It can bedivided into three types: empirical model, semi-empiricalmodel and theoretical model.[2,3] The empirical model
has a high amount of access to specific tires for semiempirical models, but often requires a large amount ofexperimental data, so the versatility is low, the cost is
high, and it brings great It is inconvenient to have awidely accepted model of the Magic Formula proposedby pacejka and a semi-empirical model of uniform tires
describing the 6-component force characteristics of tires
proposed by Academician Guo Konghui[4]. The
theoretical model is to establish the relationship between
the tire force and the parameters through the

1. Introduction
Logistics robots are growing in non-manufactured
environments due to the surge in online shopping.
Shipping robots, which are robots that deliver goods tocustomers indoors or within five kilometers of the city,
are growing exponentially. Among the major
technologies related to logistics robots such as hardware,
driving, awareness and operation, we are going to study
stability of driving when transporting goods.
We studied how safe it was to be off the road dependingon the various conditions in the city center. As theinterface between the vehicle and the road, the tire shapein different driving environments makes different
characters dynamic response in the vehicle. It is© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan95

Ji-hyeon kim, Zhang Hongyu, Jang-myoung Lee

mathematical description of the mechanical properties of
the tire. The most representative pure theoretical modelis the Gim tire model. The tire has a simple formula anddoes not require data fitting. And the characteristics ofthe tire force can be used to quickly and easily obtain thefriction coefficient of the tire.

Where: α is the wheel side angle, α = arctan(VY/VX );VY is the lateral speed of the wheel.
2.2 Model force characteristics under the
combined action of longitudinal force andlateral force
The tire-ground adhesion coefficient is related to the slip
rate of the tire surface relative to the ground. If the
combined effect of longitudinal and lateral forces is taken
into account, the integrated adhesion coefficient of thewheel is defined as(7)
0 (1 Assa )2. Gim tire model
The Gim tire model is based on the concept of an"interacting spring", which itself is considered to be aseries of three-dimensional elastic elements that transmit
forces in the radial, transverse, and longitudinal
directions, each of which has three 3D elastic elements.
The contact area between the tire and the ground, which
is orthogonal to the center axis, interacts with each other,can be assumed to be a rectangular area. [5,6]The
relationship between the length l of the grounding wire
and the free radius r of the wheel and the radial
deformation δ is

1
2

r2

Where: u0 is the static friction coefficient of the tire andthe road surface.
A is the adhesion coefficient reduction factor
A 1 1 / 0 / s1; ssa Synthetic slip rate to reflect thelongitudinal slip and lateral offset of the tire,ssa
ss 2 sa 2 , The two-point friction coefficient (0, μ0)and (s1, μ1) obtained by the test ,Synthetic slip ratio.

2

(r

)2

(1)

Because r ＞＞ δ, so

l

8r

(2)

2.1 Simulation Implementation of Gim Tire
Model
Regardless of the tire roll angle, depending on thelongitudinal and lateral forces of the tire, there are twoforms of braking and driving, respectively, which defines
the longitudinal slip ratio as

SX

VX VC
VX

0 (Braking situation)

(3)

Ss

VX VC
VX

0 (Drive situation)(4)

Fig. 1. Synthetic slip ratio
At the same time, considering the combined action of thelongitudinal force and the lateral force, the synthetic
adhesion coefficient μ can be decomposed into thelongitudinal adhesion coefficient x and the lateraladhesion(1 Ss ) tan a (Braking situation)
Sa

tan a

y

corresponding

to

the

longitudinal force and the lateral force using the friction
circle concept. From the concept of friction circle, we
know

Where: Vc is the circumferential speed of the tire; Vx isthe longitudinal speed of the wheel.
Define the lateral slip ratio asSa

coefficient

2

2
x

(5)

y

1

(8)

Then the longitudinal adhesion coefficient of the wheelis(6)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan96

Estimation tire-rod friction coefficient

ss

x

cos

ssa

velocity v, the tire side yaw angle, and the vertical load,
and the system output is the longitudinal and lateralground friction of the tire.

(9)

The lateral adhesion coefficient of the wheel is

sa

y

sin

ssa

(10)

Where: β is the slip direction angle and is a function of
the longitudinal slip ratio ss and the lateral slip ratio sa .
Contact area rolling-sliding critical point is

sn

1
3 FZ

(CS ss )2(Ca sa ) 2(11)Longitudinal critical slip rate of elastic deformation oftire

3.1 Slip rate subsystem

3 FZ
CS

ssc

Fig. 2. Gim tire model Simulink block

Vehicles on the road have brake and drive two patterns,
and both of the calculation of the slip rate varied,
therefore need to use the tire rolling radius R, and the
input of the wheel speed v, and the angular velocity w
logic judgment, the system uses two switch modules, todetermine the wheel in driving or braking mode, and thenthe slip rate is calculated using the corresponding formula.

(12)

Tire lateral critical slip ratio

ssa

CS
Ca

ssc 2

ss 2

(13)

Where: FZ is the wheel normal load; CS , Ca are the

3.2 The critical slip rate subsystemlongitudinal and lateral (lateral) stiffness of the tire.
Defined ln l sn , ln as the dimensionless value of the

The longitudinal and lateral friction coefficients of tires
are determined according to the given road surface
comprehensive adhesion coefficient and longitudinal andtransverse slip conditions.

tire grounding wire length, regardless of the tire returning
moment characteristics, the longitudinal force between
the tire and the road surface isFx

C S s S ln 2

Fz (1 3ln 2 2ln 3 )( ssFx
ssc )y Fz ( saxssc ) (14)

3.3 The road adhesion coefficient subsystem
This module is to calculate the tire ground wire length,
dimensionless value of tire and the longitudinal criticalslip ratio of the elastic deformation and lateral critical slip
ratio, the module switching function is used to determine
the lateral critical slip ratio, namely when the actual
wheel longitudinal slip ratio is greater than its criticalpoint indicates that the wheel is in pure sliding state,means the wheel lateral critical slip ratio of 0.

(15)

3. Simulation Implementation of Gim Tire
Model
Simulink is an important component of MATLAB. It
provides a dynamic system modeling, simulation andcomprehensive analysis integration environment, which
is widely used in the dynamic simulation and design ofcomplex simulation and design of control systems.
In the establishment of the Gim tire model, the tire model
is divided into four modules, namely four different
subsystems, namely the slip rate subsystem, the criticalslip rate subsystem, the road adhesion coefficientsubsystem, and the tire force system, the total system
input, the wheel angular velocity w, and the wheel linear

3.4 The tire force system
The module USES the calculated values to determine thelongitudinal and lateral forces of the final tire Fx andFy .

Because we only care about the change in the coefficientof friction in a straight line, we don't care about the thirdand fourth part.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan97

Ji-hyeon kim, Zhang Hongyu, Jang-myoung Lee

4. Model verification and simulation analysis
In order to validate the established model of tire featuresa description of the precision of this experiment in dry
concrete Liu Min (measured 0.8 road surface friction
coefficient, 0 = 1.14) on tire force characteristics inlaboratory test data to validate the theoretical tire model,
the experiment of tire in the case of side-slip Angle of 0
to determine the coefficient of friction, you known, themobile robot in the execution of the longitudinal force toperform the friction, so we only pay attention to thechange of the longitudinal coefficient of friction, We
carry the data measured by the mobile robot on the actualroad into the model for analysis.
Fig. 4. wet Asphalt road friction coefficient
In the figure above, the friction coefficient of the mobile
robot on the wet road surface is compared with the actual
value. It can be seen that this model can also measure thefriction coefficient well in the wet environment.
5. Conclusion
Tire has a very important role in vehicle dynamics
model, is an important part of vehicle dynamic control
system, this paper study was conducted based on
Simulink simulation, the simulation, compared to theestimated value and actual value of the Gim tire modelhas good performance ability, in the following study
will be in the Segway VDC (vehicle dynamic control
system) in the further research

Fig. 3. Asphalt road friction coefficient
From figure 3 estimates with the actual friction
coefficient of friction coefficient by contrast, one has
good matches, although have certain deviation estimatesand actual values, but estimates range is around the actualroad friction coefficient, estimate the tendency of thefriction coefficient and the actual value is the same, it isverified the model established in this paper, thecalculation of the tire force is credible, Gim tire model
can effectively predict the tire mechanical characteristics.
The possible reason for the data fluctuation is that themobile robot is light in mass and slow in speed, which isgreatly affected by the flatness of the ground.

Acknowledgements
This research was funded and conducted under 『theCompetency Development Program for Industry
Specialists』 of the Korean Ministry of Trade, Industry
and Energy (MOTIE), operated by Korea Institute forAdvancement of Technology (KIAT). (No. N0001126,
HDR program for Industrial convergence/connected for
creative robot human resource development)
References
1. Changsun Ahn Huei Peng, H.Eric Tseng. Estimation ofRoad Friction for Enhanced Active Safety System:
Dynamics Approach Proceedings of 2009 American
Control ConferencesR. Loren and D. B. Benson (eds.),
Introduction to String Field Theory, 2nd edn. (SpringerVerlag, New York, 1999).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan98

Estimation tire-rod friction coefficient

2. Pacejka H B, Bakker E.Magic formula tyre model [J].
Vehicle systems Dynamics 1993R. Loren and D. B.
Benson, Deterministic flow-chart interpretations, J.
Comput. System Sci. 27(2) (1983) 400–433.
3. Pacejka H B, Besselink I J M. formula tyre model withtransite properties [J]. Vehicle systems Dynamics 1997
4. Guo K H, Lu D, Ren. A unified non-steady non-linear tyremodel under complex wheel motion inputs including
extreme operating conditions [J]
5. Gim G, Nikravesh P. An analytical model of pneumatictyres for vehicle dynamics simulations. Part 1[J]. Pure
Slips, internationals Journal of Vehicle Design, 1990
6. Gim G, Nikravesh P. An analytical model of pneumatictyres for vehicle dynamics simulations. Part 1[J].
Comprehensive Slips, internationals Journal of Vehicle
Design, 1991

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan99

Real-time reflection removal algorithm using stereo camera only
Do-kyung Hwang

Dep. of Electronic Eng., Pusan National University, 2,Busandaehak-ro 63beon-gil Geumjeong-gu
Busan, 46241, Republic of Korea
Jong-woo An

Dep. of Electronic Eng., Pusan National University, 2,Busandaehak-ro 63beon-gil Geumjeong-gu
Busan, 46241, Republic of Korea
Jang-myoung Lee

Dep. of Electronic Eng., Intelligent Robot Laboratory, 2,Busandaehak-ro 63beon-gil Geumjeong-gu
Busan, 46241, Republic of Korea
E-mail: dokyung7379@pusan.ac.kr, jmlee@pusan.ac.kr
www.pusan.ac.kr

Abstract
In this paper, we implemented a light reflection elimination algorithm which is one of the main obstacles of various
object recognition systems using vision systems. In other words, this algorithm can be used to remove robotic
recognition obstacles from the factory and perform tasks such as locating robots, autonomous vehicles, or
unmanned aerial vehicles. This algorithm has been studied with versatility in mind and does not require sensors
other than stereo cameras. Algorithm that intuitively interprets pixels in HIS(Hue ,Saturation, Intensity) space andprocesses pixels corresponding to light. For intuitive analysis, take an angle, intensity, or other light source(sunlight, fluorescent light, candle) from the camera, then record each pixel value and separate it from a white
object (whiteboard, paper, etc.). After pixel operation, object recognition is implemented using theCNN(Convolutional Neural Network) method using the YOLO(You Only Look Once)_V2 algorithm[5]Keywords: Light less, Pixel analysis, Object detection, reflections removal, CNN, YOLO(You Only Look Once).

light reflected in water during rain[2], and removal ofafterimage using time difference motion in edge channel
are being studied. However, it is difficult to apply thesestudies in an industry where real - time image
recognition is required.

1. Introduction
One of the big problems in the field of image
recognition is the detection problem due to light. In
order to solve the problem of light recognition,
researches on shadow removal technique[3], removal of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan100

Do-kyung Hwang, Jong-woo An, Jang-myoung Lee

For example, removing shadow, removing afterimages
using time-lag motion in the edge channel results inelimination of objects (such as a person or a vehicle thatshould actually be collected) that should not be removed,
On the contrary, whe0n the light is removed in the rain,
the object is considered more. However, considering theobject, the light reflection component itself is notremoved much. In this study, considering themarketability, we came to the study with general
purpose in mind. This algorithm is divided into two
major stages.
First, white objects and light reflection components are
distinguished in the HSI channel through pixel analysis,
And then processes the distinct Light Pixel components.

In the case of Fig 2, I took a picture with a ZED stereo
camera when the sunlight penetrated strongly through

2. Pixel analysis
First, the first step is to analyze the pixel values in the
HSI channel [1]. In the HSI channel, the RGB values
are different depending on the original color, saturation,and brightness. Assuming that the y-axis reference isbrightness and the degree of brightness is expressed as 0
~ 128, RGB color is expressed as shown in Fig.1.

Fig. 2-

Fig. 1-

the window at 7 o'clock in the morning, and I analyzed
various situations in a single photograph by adding afluorescent light source. In Figure 2, light separates into
red boxes and white objects into green boxes..
The meaning of each point is P1, P2 is the light source(fluorescent light, sunlight) P3 Weak sunlight P4 is the
weak sunlight on the white wall P5 is the sunlight
passing through the plastic water bottle and G1 and G2
are white paper and white glass bottles.

In this paper, to derive the supernatural components oflight, several angles and various light sources arephotographed in the color space as shown in Fig. 1, andthe pixel values corresponding to the light are derived..
Then, layers are constructed based on the collected data,
and the pixel values extracted by the layer information
are processed by 0 to discriminate.
Fig. 2 show the result of light separation in a situation
where various objects are placed.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan101

Paper Title (First 4 Words)

Fig. Pixel
3.
Fig. 3 shows the HSI
values for the example Pointin Fig. 2. In the light sources P1 and P2, R channel
value is 255, G and B channel value are 0
It is possible to confirm that the R channel value of thelight material transmitted through the remaining object
is relatively high. On the other hand, G1 and G2, which
are white objects, have a relatively low R channel value
and relatively high G and B channel values.[4] The
method proposed in this paper distinguishes between
bright objects and white objects in the same way as theanalysis in Fig. 3, and shoots bright objects and white
objects with various angles and brightness to construct alarge number of Data tables and layers about light

Fig. 4.

3. Experiment
In order to test the performance of the algorithm, we
experimented with the recognition rate by placing an
object named H in front of the window in anenvironment where the window is very sunny at 2:00
pm. For object detection, YOLO_v2 algorithm was used
and total 800 pieces of learning data and 500,000 piecesof learning data were used, and in an additional situation,
the environment is configured so that the light noise canenter more intensely in the presence of a fluorescentlamp in a relatively bright corridor.
Fig. 5.

The y axis in Fig. 5 is the detection rate for the object,
and the x axis is the detection distance.
Fig. 5 shows the difference in detection rate anddetection distance when applying the algorithm (greenline) and when not applying it (red line).

The camera used in the experiment was ZED stereoCam, Increase the distance to H in the red arrow
direction as shown in Fig 4. We collected data about
when applying the algorithm presented in this paper andwhen not applying it.

4. Restore
However, if you treat the Pixel of light as '0', the object
may be processed as '0' when there is light on the object.
In this case, we present a method for restoring objects
using surrounding pixel information and color© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan102

Do-kyung Hwang, Jong-woo An, Jang-myoung Leeof trade, Industry and Energy (MOTIE), Korea Institutefor Advancement of Technology (KIAT) through the
Robot
Business
Belt
Development
Project(A012000009)

correction.[1] The basic perimeter Pixel information
usage method follows eq. (1).
x 1

y 1

n x 1

m y 1

p( x, y )countp(n, m)(1).

Fig. 6 below shows a restored object using surrounding
pixel information.

References
1. Taeha Um and Wonha Kim, Dehazing in HSI ColorSpace with Color Correction (JBE Vol. 18, No. 2, March2013) pp. 85–104
2. Sang-Woong Lee, Removing Lighting Reflection under
Dark and Rainy Environments based on Stereoscopic
Vision (Journal of KIISE Vol. 37, No. 2, February, 2010)
pp. 104–107
3. Byeong-Ju Han and Jae-young Sim, Single image based
shadow removal algorithm (IPIU 2015 Vol. 27 February,2015)
4. Bong-hwa Song and Hyung-Je Cho, Vehicle Recognitionin the Forward Direction and the Trace of Movements for
Collision Prediction (Institute of Image Culture andContent in Dongguk University 2008 Vol.1 2018)
pp.101–114
5. Joseph Redmon, Santosh Divvala, Ross Girshick and Ali
Farhadi, You Only Look Once: Unified, Real-Time
Object Detection

Fig. 6

5. Conclusion
Although this algorithm has been studied inconsideration of versatility, it is impossible to restore
the shape of objects such as small letters due to light.
Therefore, robots that perceive that the range of objects
is simply limited, or peripheral pixels may be applicablein simple fields. These problems will require more
research or results from some media that can block light.
Acknowledgements
This research was funded and conducted under 『theCompetency Development Program for Industry
Specialists』 of the Korean Ministry of Trade, Industry
and Energy (MOTIE), operated by Korea Institute forAdvancement of Technology (KIAT). (No. N0001126,
HDR program for Industrial convergence/connected for
creative robot human resource development)This research was financially supported by the Ministry© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

103

The Effect of Inertial Measurement Unit on Synthetic Aperture Radar Image Quality
Soojeong Lee1, Yong-gonjong Park1, Woo Jung Park1, Chan Gook Park1, Jong-Hwa Song2 and Chang-Sik Bae2
12

Department of Mechanical & Aerospace Engineering/Automation and Systems Research Institute, Seoul National
University, 1 Gwanak-ro, Gwanak-gu, Seoul 08826, Republic of Korea

Hanwha Systems, 491-23 Gyeonggidong-ro, Namsa-myeon, Cheoin-gu, Yongin-si, Gyeonggi-do 17121, Republic of
Korea

Abstract
This paper analyzes the effect of inertial measurement unit (IMU) on synthetic aperture radar (SAR) image quality.
While operating SAR, it is supposed that an aircraft flies straight and level at a constant speed. Because of atmosphericturbulence and aircraft maneuvers, it deviates from its nominal trajectory. With navigation solutions, the deviations
can be compensated through motion compensation procedure. However, uncompensated motion errors generated by
IMU errors degrade SAR image quality, such as broadening bandwidth and blurring the image. Since the residual
errors depend on the degree of IMU inaccuracies, SAR images and impulse response function (IRF) according to
IMU grades are compared with each other. Through this result, the IMU specification is suggested to satisfy given
SAR image quality.
Keywords: synthetic aperture radar (SAR), inertial measurement unit (IMU), SAR image quality, impulse response
function (IRF)

frequency (PRF) control and range shift compensates thedeviations, resulting in better image quality than without
MOCO.3 However, due to inertial measurement unit
(IMU) errors the difference between actual position andmeasured one is generated. This, uncompensated motion
errors degrade SAR image quality, for example,
broadening bandwidth and blurring the image.4, 5
MOCO such as PRF control is known to be essential
before SAR image processing to improve SAR image
quality. Therefore, researches about MOCO are widelydone.3, 6, 7 At the same time, the uncompensated motion
error is concerned.4, 8−11 Especially, the effect according
to the error term such as constant, linear, quadratic andcubic terms are studied.4, 9−11
Since residual errors depend on the degree of IMU
inaccuracies,8 in this paper SAR images and impulse
response function (IRF) according to IMU grades arecompared to each other. Through this result, the IMU

1. Introduction
Synthetic aperture radar (SAR) is a radar systemproviding high resolution images by synthesizing thereceived radar signals. During SAR operation, theaircraft is supposed to fly straight and level at a constant
speed. While moving, it transmits and at the same timecollects received signals through the whole time, called
as synthetic aperture time (SAT). By post-processing
such as range Doppler algorithm (RDA), the collectedsignals are synthesized to make the phases coherent.1, 2
Because of this step, SAR antenna can get the same effectas using longer antenna, meaning the higher resolution.
However, it is impossible for an aircraft to fly thenominal trajectory because of atmospheric turbulence
and aircraft maneuvers. Therefore, before image
formation the deviations should be measured andcompensated. With measured navigation solutions,motion compensation (MOCO) such as pulse repetition© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan104

Soojeong Lee, Yong-gonjong Park, Woo Jung Park, Chan Gook Park, Jong-Hwa Song, Chang-Sik Bae

specification is suggested to satisfy given SAR image
quality.
This paper is organized as follows. Section 2 explains
the navigation method before SAR operation and therelationship between phase error and position error.
Section 3 provides the specific values of IMU
specifications used in the simulation and compare theresultant images and IRF according to IMU grades.
Section 4 concludes the paper and suggests the IMU
specification to satisfy given SAR image quality.

quality can also be clarified. To obtain expression for that,several steps are carried out as below.
First, the relationship between the phase error and therange error is widely known as Eq. (1). Since the range
error here is motion error in range direction, it does notstand for the position error commonly mentioned. As in
Ref. 5, the range error can be expressed as Eq. (2).
4Δp ηLOSR

2. Residual Errors by IMU Inaccuracies

(1)

R

(2)

The error contributions in short term according toerror sources are summarized in Table 1, sorted into error
type.12 Then, the position error can be derived as Eq. (3).
Because in short term, quadratic error term is dominant,Eq. (3) can be simplified as Eq. (4). Lastly, Eq. (5) is
derived from Eq. (1), (2), (4) to represent the relationshipbetween phase error and IMU errors.

In this section, the navigation algorithms during
simulation and the relationship between phase error andposition error are explained.
2.1. EGI/IMU integrated navigation algorithm
Because of the aircraft structure, the SAR antenna andthe global positioning system (GPS) receiver cannot bemounted in the same position. IMU which is mountedwith the GPS receiver on the center of gravity isdesignated as IMU 1. The other IMU mounted on the
SAR antenna is designated as IMU 2. During SAR
operation, to compare SAR image quality according to
IMU inaccuracies, only inertial navigation system (INS)with IMU 2 is performed. Before SAT, to estimate thebias of accelerometer and gyroscope of IMU 2, EGI/IMU
integrated navigation algorithm is applied. EGI/IMU
integrated navigation is as follows: embedded GPS/INS(EGI) system is implemented with IMU 1. The
navigation solution from EGI system is considered as
GPS data in EGI/IMU integrated navigation algorithm.
Then, another EGI system is performed with IMU 2 andthe previous EGI data. However, since the IMU 2 ismounted in the other position, at the measurement updatein EGI/IMU integrated system, the lever arm effectshould be considered.5 Through this system, the bias ofaccelerometer and gyroscope of IMU 2 can be estimatedbefore operating SAT, resulting in the reduced residual
errors.

pN

pN
1
2

2

vN t
2

N

1
g
6

t2

1
g Nt2
2

2

2
E

D

t3

pN

1
2

g N

1

2

3

4

2

1
g N t3
6

2

2 2

t

N

1
2

(3)

1
2

2

g N

(4)

2

2 2
N

(5)

t

where is phase error,
is wavelength, R is range
error, Δp is position error vector, ηLOS is line-of-sightunit vector, and 1 , 2 , 3 , pN , pE , pD are elements ofvector ηLOS , Δp respectively. t is time, g is
gravitation constant, and E is Earth rotation rate. pN
is initial position error, vN is initial velocity error, N
Table 1. Position error due to each error source of navigation

2.2. Position error equation with initial errors
Phase error of the received signal causes the degradationof SAR image, such as defocusing. Therefore, if therelationship between phase error and position error isclarified, the effect of IMU inaccuracies on SAR image

Error type

Error source

Position error value

Constant

Initial position errorpN

LinearInitial velocity errorvN t

Quadratic

Initial tilt,
Accelerometer bias

1
1
g Nt2 ,
2
2

Cubic

Initial azimuth error,
Gyroscope bias

1
1
g E Dt 3 , g N t 3
6
6© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

105

N

t2

The Effect of Inertial

Fig. 2. Impulse Response Function

Fig. 1. Position error during SAR operation

is initial tilt, N is accelerometer bias,
azimuth error, and N is gyroscope bias.

D

is initial

3.2. IMU specifications

To compare the effect of IMU on SAR image quality,
simulations are performed. In this section, simulation
conditions, the chosen IMUs specification and results aredescribed. 30 cm resolution and peak sidelobe ratio(PSLR) are chosen as the requirements of SAR image
quality.

To compare the effects of IMU inaccuracies, different
grades of IMU are selected: microelectromechanicalsystems (MEMS) grade IMU, tactical grade IMU andnavigation grade IMU. For each grade of IMU, ADIS
16460, ADIS 16488A, HG 9900 are selected to obtainspecific values of IMU specifications and they are listed
in Table 2. Fig. 1 shows the position error during SAR
operation according to IMU.

3.1. Simulation conditions

3.3. Resultant images and IRF

The simulation is implemented for 200 seconds in 200
m/s at an altitude of 7,600 m and during the last 30
seconds SAR is operated. The sampling frequency of
GPS is 1 Hz and IMU 1, IMU 2 are 200 Hz, respectively.
GPS with 10 cm white Gaussian noise is used. IMU 1 inthe same position as GPS receiver has 10 μg
accelerometer bias and 0.005 deg/hr gyroscope bias. 50
Monte Carlo simulations are implemented. For image
formation, spotlight SAR mode is chosen and the PRF is
4,000 Hz.

By point target simulation, spotlight SAR is implemented.
Fig. 2 shows IRF according to IMU grades. IRF with
MEMS grade IMU is distorted and broadened much morethan tactical and navigation grade IMU. IRF with tacticalgrade is asymmetric and shifts from true value. IRF with

3. Simulation

Table 2. IMU on SAR antenna specifications
Accelerometer

GyroscopeBias[deg/hr]

Angularrandomwalk[deg/√hr]

IMU 2

Bias
[μg]

Velocity
randomwalk[μg]

ADIS 16460

200

20

8

0.12

ADIS 16488A

70

7

5.1

0.26

HG 9900

25

2.5

0.003

0.002

Fig. 3. SAR image with MEMS grade IMU© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan106

Soojeong Lee, Yong-gonjong Park, Woo Jung Park, Chan Gook Park, Jong-Hwa Song, Chang-Sik Bae

Table 3. SAR image quality
IMU Grade

Resolution
[cm]

PSLR
[dB]

MEMS

815

-12.14

Tactical

110-23.51

Navigation

14-28.71

Acknowledgements
This work was supported by a grant-in-aid of HANWHASYSTEMS (Project number 0418-20180054).
References
1. A. Moreira, P. Prats-Iraola, M. Younis, G. Krieger, I.
Hajnsek, and K. P. Papathanassiou, A tutorial on synthetic
aperture radar (IEEE Geoscience and Remote Sensing
Magazine, 2013), pp. 6–43.
2. M. Elhefnawy and W. Ismail, Fundamentals of synthetic
aperture radar systems (International Journal of
Emerging Technology and Advanced Engineering, vol. 5,2015), pp. 172–178.
3. G. E. Haslam and A. Damini, Specifying the allowablelatencies in the application of SAR motion corrections(Proc. EUSAR ’96, European Conf. on Synthetic apertureradar, Koenigwinter, Germany, 1996), pp. 26–28.
4. S. Buckreuss, Motion errors in an airborne synthetic
aperture radar system (European Transactions on
Telecommunications, vol. 2, 1991), pp.655–664.
5. A. Kusk, J. Dall, and E. L. Christensen, Estimating sensor
motion in airborne SAR (Ph.D. Thesis, Technical
University of Denmark, 2006).
6. E. Zaugg and D. Long, Generalized SAR processing andmotion compensation (2008).
7. D. H. Kim, S. H. Park, and K. T. Kim, SAR motion
compensation using GPS/IMU (The Journal of Korean
Institute of Electromagnetic Engineering and Science, vol.
22, 2011), pp. 16–23.
8. E. P. Velikanova, A. A. Geltser, Z. T. Erdyneev, and N. V.
Panokin, Analysis of accuracy requirements for an inertial
navigation system in synthetic aperture radars (Gyroscopy
and Navigation, vol. 8, 2017), pp. 129–135.
9. Y. Mao, M. Xiang, L. Wei, and S. Han, The effect of IMU
inaccuracies on airborne SAR imaging (Journal of
Electronics (China), vol. 28, 2011), pp. 409–418.
10. G. E. Haslam and B. Reid, Motion sensing requirements
for synthetic aperture radar (IEEE Proc. vol. 1, 1983), pp.
126–131.
11. G. Fornaro, G. Franceschetti, and S. Perna, Motion
compensation errors: effects on the accuracy of airborne
SAR images (IEEE Transactions on Aerospace andElectronic Systems, vol. 41, 2005), pp. 1338–1352.
12. D. H. Titterton, and J. L. Weston, Strapdown inertial
navigation technology – 2nd Edition (The Institution ofElectrical Engineers, 2004).

Fig. 4. SAR image with tactical grade IMU

navigation grade shifts less than IRF with tactical grade,
but it also has asymmetric sidelobe. Figs. 3–5 show
resultant SAR images. As expected with IRF results andTable 3, Fig. 3 is defocused and broadened much ascompared to Figs. 4, 5.
4. Conclusion
In this paper, the effects on SAR image quality according
to IMU grades are compared. To satisfy the given SAR
image quality requirements with pure navigation,navigation grade IMU is recommended. If tactical grade
IMU should be used, then auxiliary sensor has to be
added to improve the image quality. In future work, tosatisfy the given SAR image quality with only tacticalgrade IMU, the effect of autofocus on alleviating
quadratic and cubic term errors will be considered.

Fig. 5. SAR image with navigation grade IMU© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan107

Apply 2D Barcode Scanner for Mobile Robot Navigation in Checkerboard Mapping
Chun-Chi Lai

Bachelor Program in Interdisciplinary Studies, National Yunlin University of Science and Technology, 123 University
Road, Section 3,
Douliou, Yunlin 64002, Taiwan, R.O.C.
Kuo-Lan SU

Department of Electrical Engineering, National Yunlin University of Science and Technology, 123 University Road,
Section 3,
Douliou, Yunlin 64002, Taiwan, R.O.C.
Chia-Jen Lin

TECO Electric & Machinery Co., Ltd., 4F, No.3, Park St.,
Nan-Kang, Taipei 11503, Taiwan R.O.C.
E-mail: cclai@yuntech.edu.tw, jeromy@teco.com.tw
Evgeni Magid

Laboratory of Intelligent Robotic Systems (LIRS), Intelligent Robotics Department, Higher Institute for Information
Technology and Intelligent Systems, Kazan Federal University, Russia

Abstract

In this work, a mobile robot is equipped with an industrial barcode scanner which can provide the pose
information respect to the barcode tag in the field of view (FOV). For real multiple automated guided
vehicle (AGV) transportation applications, the mobile robot navigation flow is considered to get the
global checkerboard type path planning from a remote master server as an input. For the local planner,each robot is applied with a simple path controller to track the global path. The simulation andexperimental results show that this implementation has good feasibility for multi robot co-working in afactory area.
Keywords: Automatic Guided Vehicle, 2D Barcode Code Navigation.

over the 2D barcode tags, the proximity of the 2D
barcode tag can be detected and identified via the image
processing system that allow Kiva system to get highquality positioning information.
B. Dzodzo et al. [2] had applied the 2D bar codes
mounted on ceiling for requirements in facilities where
floors and walls are in constant damaging contact
with equipment and personnel. Also, there were many
researches and commercial approaches which placed
the 2D codes on the walls, ceilings as well as floors [3],[4], [5], [6].

1. Introduction
For a traditional automated guided vehicle (AGV)
control system that usually follows a magnetic tape forroutine tasks. At a later time, two-dimensional barcode
tags were researched as a popular AGV positioning
solution after Amazon acquired Kiva Systems [1] which
was the most notable commercial success of AGV fleet
management system. The Kiva robots are operated
within in a zone where the 2D barcode tags were affixed
to the ground as checkerboard map. Without driving© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan108

Chun-Chi Lai1, Kuo-Lan Su1, Chia-Jen Lin1, Evgeni Magid

In this work, a mobile robot is equipped with anindustrial barcode scanner which can directly provide
the pose information respect to the tag in the field ofview (FOV). For multiple AGV transportation
applications, the navigation flow is considered to get the
global checkerboard type path planning from a remotemaster server as an input. For the local planner, eachrobot is applied with a path controller to track the global
path. The simulation and experimental results show that
this implementation has good feasibility for multi robot
co-working.

Fig. 2. Automated guided vehicle with 2D barcode tag

zero point of the 2D tag to the motion controller.

2. System Description

3. Frame Transformation in ROS

Fig. 1 (a) shows a differential wheeled robot platformthat was built in our laboratory. The movement ofdifferential wheeled robot is based on two separately
servo driven wheels placed on either side of the robot.
The differential wheeled robot platform is the mostpopular type in AGV applied. Due to it can thus change
its direction by varying the relative rate of rotation of its
wheels and hence does not require an additional steering
motion.
Fig. 1 (b) shows an industrial optical scanner [7]. This
device will detect 2D barcode tags which are typicallyglued onto the floor in a grid as shown in the left of Fig.
2. The individual 2D barcode tags are numberedconsecutively and include position information. It canbe used together with a colored tap affixed to the floor
and code tags printed with 2D barcode.
In the right of Fig. 2 shows the 2D barcode tags of AGV
positioning system. A 2D barcode tag contains position
information in addition to a specific number. A cross inthe center of the 2D barcode tag marks the zero point.
The X and the Y axes are marked starting from the zero
point. The black arrow indicates the positive axis andthe white arrow indicates the negative axis. So the readhead reports the position of the AGV in relation to the

In this section we take an overview of Robot Operating
System (ROS Base on ROS framework and utilities,
we can integrate the control and positioning systemquickly. Besides, it is necessary to induce the coordinatetransform between tag in map (world coordinate) andthe odometry coordinate. So the transform tree in ROS
will be completed for AGV navigation.

(a)

3.1 Robot Operating System
Robot Operating System (ROS) is a software frame
work for robotics research and development. It applies apeer-to-peer topology for communication between robot
processes and it provides tools for robot softwaredevelopment. For more information about ROS, theonline wiki [8] is available. For the sake ofcompleteness, some common concepts are listed as
below:
Nodes are ROS processes that perform computation.
Nodes can communicate with each other using
messages.
Topics are named medium over which nodes
exchange messages. Multiple nodes can publish /subscribe to a predefined topic.
Subscriber is wrapped in a node which listens to themessages that are published to a topic.
Publisher is wrapped in node which sends to a topic
for other nodes can subscribe.
roscore is a collection of nodes that are prerequisites of a ROS-based system. roscore starts a
ROS master node. So we must have a roscore
running in order for nodes to communicate.
TF in ROS is a package that lets the user keep trackof multiple coordinate frames over time. TF
maintains the relationship between coordinate
frames in a tree structure buffered in time, and lets(b)

Fig. 1. (a) a differential wheeled robot platform equipped withan industrial PC (b) the optical reading head (PGV 100 serial)for barcode tag identification© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

109

Apply 2D Barcode Scanner

the user transform points, vectors, etc., between any
two coordinate frames at any desired point in time.
3.2 Coordinate Transform from Barcode ScannerFig. 3 shows the TF tree relationship in this work. The
barcode scanner is attached in the robot with a posetransformation respect to the robot center (base_link). In
general, the odometry will accumulate the encoder
pulses from wheels and transfer it as displacement androtation from the robot center respect the initial zero
odometry coordinate frame. If there is no wheel drift
error, the odometry coordinate frame will be the same as
the world map coordinate frame. However in realsituation, the wheel drift will accumulate to increase thepositioning error. So we need the barcode scanner to get
the real world’s position, and re-align the drift between
the odometry and world map.
In Fig. 4, the cross m’ presents the world’s coordinate
center and the cross o’ presents the odometry’s
coordinate center. The blue arrow presents the AGV’spose {xo, yo, Φo} in odometry coordinate frame. Whenthe barcode scanner attached in the AGV that identifies
the barcode tag on the floor, we will convert it as global
position information such as {xm, ym, Φm}. So we cancalculate the coordinate drift {xd, yd, } between theworld map and odometry frames.
R=+β = tan=Φm-Φo= cos( + )= sin( + )=−=−Fig. 4. Coordinate Transform between world (map) andodometry.

completion such as shown in Fig. 3.
4. Action Server Mechanism for Navigation
In ROS based system, if we would like to send a request
to a node to execute a task, and also receive a reply tothe request. This can currently be achieved via ROS
actionlib package. Fig. 5 shows the Action Server nodethat is designed in this work. The global path receiver isa template function to subscribe the goal messages fromthe Action Client. The goal would be a pose stamped
array message that contains the path information for
robot sequential moving in the world. When Action
Server has received the goal, the local path follower willcalculate the motion command and publish the cmd_vel
topic. And another servo control node will subscribe thetopic for handling AGV to follow the global path. Also
the Action Server will give the feedback as the state
reply to tell the Action Client about the incremental
progress of the goal. For moving the platform, this willbe the robot's current pose along the path. Finally, aresult is sent from the Action Server to the Action Client
upon completion of the goal. This is different thanfeedback, since it is sent exactly once.

(1)
(2)
(3)
(4)
(5)
(6)
(7)

From the above equation (3), (6) and (7) the coordinate
drift {xd, yd, } will be calculated if we have the input{xo, yo, Φo} and {xm, ym, Φm}. In ROS framework,
we need to create a transfer node to publish the /tf topic
between odomety and world map for the TF tree’s

Fig. 5. An Action Server framework for AGV navigation

Fig. 3. Coordinate transformation (TF) in ROS.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan110

Chun-Chi Lai1, Kuo-Lan Su1, Chia-Jen Lin1, Evgeni Magid

5. Fleet Management Framework
A fleet management system [9] primarily concerns itself
with managing a group of vehicles to meet the goals andobjectives obtained from an enterprise computer system.
In this paper, we primarily implement a simplified fleet
management system using ROS multi master system. To
implement a multi master system, a package called
multimaster_fkie [10] is needed and can be easily
installed. Fig. 6 shows the simple fleet management
framework in this study. The major task of the path
planning server is to handle all the AGV’s path planningand send the path message via wireless network to each
AGV.

Fig. 6. Fleet management framework

6. Experiment Results and Conclusions

3.

The experiment is firstly simulated and demonstrated in
ROS rviz user interface for verifying the system
integration. For convenient, a robot platform model
named (turtlebot-waffle) is pre-loaded as shown in Fig.
7 (a) and the blue line presents the global path which
was published from the remote ROS master. Fig. 7 (b)shows the Action Server was handling robot’s
orientation for global path following. Fig. 7 (c) showsthat robot had reached the goal.
In this work, we have demonstrated the methodology
and integration the mobile robot navigation incheckerboard mapping based on the 2D barcode scanner.
The direction for future work therefore includes optimal
management strategy to improve the efficient whennumerous robots are co-working in the same time.

4.

5.

6.

7.
8.
9.

7. References10.

1. Kiva system overview.
http://www.kivasystems.com/solutions/system-overview
2. B. Dzodzo et al., Realtime 2D code based localization for(a) robot received the global pathindoor robot navigation, in IEEE International
Conference on Robotics and Biomimetics (ROBIO), 2013.
p. 486-492.
J. Huh et al., Mobile robot exploration in indoor
environment using topological structure with invisible
barcodes, in ETRI journal , 2007, 29(2), p.189–200.
S. Zhao et al., Robot autonomous localization based on
single artificial landmark, in Journal of Shandong
University (Engineering Science), 2007, 37.4, p.39-44.
G. Lin, X. Chen, A Robot Indoor Position andOrientation Method based on 2D Barcode Landmark.
JCP, 2011, 6.6, p.1191-1197.
Kobayashi, A new proposal for self-localization ofmobile robot by self-contained 2d barcode landmark, inProceedings of IEEE SICE Annual Conference, 2012, p.
x2080-2083.
Position Guided Vision (PGV): https://www.pepperlfuchs.com/global/en/
Robot Operation System (ROS): http://wiki.ros.org/
A. Singhal et al., Managing a fleet of autonomous mobile
robots (AMR) using cloud robotics platformm. In IEEE
European Conference on Mobile Robots (ECMR), 2017,
p. 1-6.
Juan et al., Multi-master ros systems, Institut de Robotics
and Industrial Informatics, 2015.

(b) robot turned in the check point(c) robot reached the goal

Fig. 7. The robot executed the path following task which was published from remote server.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan111

Development of the MyRio Based Mobile Platform

Jr-Hung Guo*, Bo-Jun Yang and Kuo-Lan Su

Graduate of Electrical Engineering, National Yunlin University of Science & Technology, Yunlin 64002, Taiwan, ROC..
E-mail:g9710801@yuntech.edu.tw, sukl@yuntech.edu.tw
Evgeni Magid

Laboratory of Intelligent Robotic Systems (LIRS), Intelligent Robotics Department, Higher Institute for Information
Technology and Intelligent Systems, Kazan Federal University, Russia
Abstract
The paper develops the MyRio based mobile platform with a robot arm. The structure of the mobile platform uses
the Matrix elements. The Matrix elements build the robot arm with four degrees of freedoms, too. The mobile
platform integrates some sensors, four DC servomotors, two DC motors, two RC servomotors, a MyRio based
control box, and two vision devices. The core controller of the MyRio-1900 control box is the NI-Single-Board
RIO 9606 module. The mobile platform embeds a robot arm on the frond side. The driver device of the gripper is a
RC servomotor. The developed mobile platform uses ultrasonic sensors to detect the obstacles.

Trapezoidal acceleration and deceleration algorithm and Proportional-Integral-Derivative (PID)
algorithm are used for precise motion control of each DC servomotor. A vision device of the mobile
platform can search and recognize the shape and color of the assigned billiard ball. The other recognizes the symbolof each QR code. These vision devices are fixed on the frond side of the mobile platform, and recognize theassigned object using Otsu algorithm. In the experimental results, the mobile platform tests the positioning functionof the Proportional-Integral-Derivative (PID) algorithm for each DC servomotor. Then we implement the
movement precious of the mobile platform.
Keywords: MyRio, DC servomotors, NI-Single-Board RIO 9606 module, Otsu algorithm.

In the past literatures of the robotic research, many
experts research stable walking on uneven terrain. Su etal. have designed a mobile robot to solve the problems
such that the robots can move from the start point to thetarget point on uneven terrain. [3]. Peng et al. designed
a where/track mobile platform to search and rescue indangerous environment. The motion modes of themobile platform can be switched alternatively to adapt
on different ground situations [4]. Guo et al developed amobile platform, based on KNR controller. The mobile
robot embed a robot arm with four degrees of freedom,and used of light sensors and touch sensors for line
tracking and detects the initial location [5]. Su et aldeveloped a mobile based robot arm using KNRm
system. The developed mobile platform uses the vision
device to recognize the position of the assigned object,
and uses ultrasonic sensors to detect the distance ofobject and obstacle [6,7].

1. Introduction
An autonomous mobile platform usually works for apredefined task. The remote user can control the mobile
platform doing the assigned task, too. In practice, ahuman wants the mobile platform to do the assigned
task, such as catching an exploder and moving adangerous object. Many mobile platforms have been
widely applied in many fields, too. Such as factory
automation, dangerous environment detection, officeautomation, hospital, entertainment, farm automationand security management system. There are somesuccessful examples, such as ASIMO, PEPPER, NAO,QRIO and AIBO. We have been designed an intelligent
mobile robot to do patrol autonomously and autorecharging process [1,2]. In the paper, we design amobile platform to finish some assigned tasks with arobot arm using the MyRio control system, and test thefundamental functions on the motion control.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan112

Jr-Hung Guo, Bo-Jun Yang, Kuo-Lan Su, Evgeni Magid

The paper is organized as follows: Section II
describes the system architecture of the mobile platform
and the robot arm, and explains the functions of the
MyRio-1900 control box. Section III explains the
hardware devices of the mobile platform. Section IV
presents the experimental results of the motion control
for the mobile platform using PID control method.
Section VI presents the brief concluding remarks.

shown in Fig.2. The length, width and height are about
15cm, 10cm and 3cm. The control box displays thefunction of each connective pin. Users can select theneeded element to connect with the control box for theassigned task. In order to achieve the purpose of
accurate positioning, the core controller of the box is
Xilinx Zynq-7010 to be a SoC system (a ARM-CortexA9 and a FPGA chip). The driver device of each DC
servomotor uses the Tetrix DC motor expansion
controller.

2. System Architecture
The system architecture of the mobile platform isshown in Fig. 1. The system contains two parts; one isthe laptop and the other is a mobile platform. The laptop
can program motion trajectories and recognize color andsize of each object using NI LabVIEW software. The
monitor interface system is developed on the laptop.
The mobile platform contains five parts. There are
structure and motors, image system, sensory system,
power and drivers and control methods.

Fig. 2 MyRio-1900 control box
The control method uses PID control law andtrapezoidal acceleration and deceleration algorithm tocontrol each DC servomotor, and tune the mobile
platform to follow the programmed trajectories. The
controller of the KNRm system computes eachcompensator signals (P, I and D) according to the error
signals as following：
Fig 1. System architecture
The structure of the mobile platform is built by some
Tetrix elements. We use three type DC motors to drive
the mobile platform with a robot arm. One is four DC
servomotors to control the motion trajectories of themobile platform; two DC motors and two RC
servomotors to build the robot arm. The image system
uses WSR LifeCam Cinema Video Camera to search thepositions of selected billiard balls, and uses Logitech
C310 camera to recognize the style of each QR code.
The main element of the sensor system is someultrasonic sensors to be fixed on the frond side of themobile platform. The ultrasonic sensors can detect the
distance from the obstacle.
The main controller of the mobile platform is a
MyRio control box to program the control methods. The
hardware device of the MyRio-1900 control box is

e

SP PV

ut

Kc e

(1)

1 t
de
edt Td
0
Ti
dt

(2)

SP is desired value, and PV is measured value.
The e is the error value to be the desired value minus
the measured value. K c is the proportional constant, Ti
is integral time constant, and Td is derivative time
constant.
3. Mobile Platform
The structure integration of the mobile platform uses
Matrix element shown in Fig. 3(a). The mobile platform
can uses four DC servomotors to connect four mecanum
wheels moving on the programmed motion paths. The
MyRio system controls each DC servomotor through© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan113

Development of the MyRio

the DC servomotor driver is shown in Fig. 3(b), andprograms the rotation range according to the feedback
signal of the encoder sensor. The controller of the
MyRio box uses PID control law and trapezoidal
acceleration and deceleration algorithm to control each
DC servomotor, and tune the mobile platform to follow
the programmed trajectories, and finish the precision
position.

using LabView software shown in Fig. 5. We set therotation angle of the DC servomotor to be 180 degree.
Then we program the driver software of the DC
servomotor using LabVIEW language shown in Fig. 6.
In the experimental result, the DC servomotor makes
the red label on the top of the wheel shown in Fig. 7(a).
Then we run the programmed software to control thewheel turning 180 degree. We can see the label to go
down shown in Fig. 7(b).

The driver elements of the robot arm are two DC
motors and two RC servomotors. The robot arm fixes on
the frond side of the mobile platform, and is driven by
the MyRio controller shown in Fig. 4. The structure of
the robot arm is built using Tetrix elements. The image
system is embedded on the top of the gripper torecognize position and color of the billiard ball, andsearches the assigned billiard ball on the competition
playground. Then the mobile robot moves approach tothe assigned billiard ball, and catches the ball using thegripper of the robot arm, and moves the billiard ball to
put down on the assigned position of the QR code.

Fig. 5 PID control method program of LabVIEW(a)Fig. 3 The mobile platform(b)

Fig. 6. Driver program of LabVIEW

Fig. 4 The robot arm
4.

Experimental Results

We implement the experimental results on thecompetition playground using the mobile platform(length 4m and width 2m). The pre-processing programof the PID control method for each DC servomotor(a)Fig. 7. Experimental result(b)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan114

Jr-Hung Guo, Bo-Jun Yang, Kuo-Lan Su, Evgeni Magid

5. Conclusion

Then we test the motion precision of the mobile
platform, and program the software using the LabVIEW
system shown in Fig.8. We set the parameters of the
PID Control method K C 2.0 , Ti 0.0 , andTd 0.00085 shown in Fig. 9. Now we implement
these parameters to control the mobile platform moving
50cm. We can see the initial position to be zero (thecenter of the rear wheel) shown in Fig. 10(a). Then themobile platform moves to stop at the assigned position.
We can see the stop position to be about 49.7 cm shownin Fig. 10(b). The speed curve of the mobile platform
uses the Trapezoidal acceleration and deceleration
algorithm shown in Fig. 11.

We designed a mobile platform with a robot arm
using MyRio-1900 control box, and used the Tetrix
elements to construct the mobile platform. The core
controller of the MyRio box programs trapezoidal
acceleration and deceleration algorithm and PID
algorithm to control each DC servomotor of the mobile
platform. The platform used two vision devices. One is
WSR LifeCam Cinema Video Camera; the other is
Logitech C310 camera. In the experimental results, themobile platform can finished the precision position ofeach DC servomotor, and moved to the assigned
distance very successfully. In future, they will finish
various tasks with the image recognition function.
Acknowledgment
This work was supported by the Ministry of Scienceand Technology of Taiwan (MOST 107-2221-E- 224053).
References
1. K. L. Su, Y. L. Liao, J. H. Guo and C. Y. Chung,
Implement of the Auto-docking Processing for Mobile
Robots, The Innovative Computing, Information andControl – Express Letters (ICIC-EL), Vol.5, No.1, pp.3136, 2014.
2. S. H. Chia, J. H. Guo, K. L. Su and B. Y. Li, Team
Mobile Robots Based Intelligent Security System,
Applied Mathematics & Information Science, Vol.7,
No.2L, pp.435-440, 2013.
3. S. T. Su, C. H. Chen and K. L. Su, “Develop a Mobile
Robot Moving on Uneven Terrain,” The Innovative
Computing, Information and Control – Express Letters ,Part B, (ICIC-ELB), Vol.6, No.2, 2015, pp.459-464
4. A. Peng, Y. Zhou, J. Hu and Y. Ou, “Mechanical design
for wheer/track transform mobile platform-search andrescue robot,” IEEE Internation Conference on Robotics
and Biomimetics. P.1787, 2014
5. J. H. Guo, K. H. Hsia, K. L. Su and J. T. Wang,
Development of the Mobile Robot With a Robot Arm,
International Conference on Industrial Technology(ICIT),
Taipei, Taiwan, March 14-17, pp.1648-1653, 2016.
6. J. H. Li, K. H. Hsia, J. H. Guo and K. L. Su,“Development of the KNRm System Based Mobile
Robot,” The Innovative Computing, Information andControl – Express Letters, Part B, (ICIC-ELB), Vol.9,
No.3, 2018, pp.185-194.
7. J. H. Li, J. H. Guo and K. L. Su, “Implementation of the
Mobile Based Robot Arm for Image Recognition,”International Conference on Artificial Life and Robotics,
Beppu, Oita, Japan, Feb. 1-4, 2018, pp.58.

Fig. 8. Motion control program of LabVIEW

Fig. 9. Parameter values of PID Fig. 11. Speed curve(a)(b)
Fig. 10. Experimental result of motion control© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan115

Avrora Unior Car-like Robot in Gazebo Environment
Ksenia Shabalina1, Artur Sagitov1, Kuo-Lan Su2, Kuo-Hsien Hsia3, Evgeni Magid1
1

Laboratory of Intelligent Robotic Systems (LIRS), Intelligent Robotics Department, Higher Institute for Information
Technology and Intelligent Systems, Kazan Federal University, 35 Kremlyovskaya street, Kazan, Russia Federation2
Department of Electrical Engineering, National Yunlin University of Science and Technology, Taiwan
3
Department of Electrical Engineering, Far East University, Taiwan
E-mail: ks.shabalina@it.kfu.ru, sagitov@it.kfu.ru, sukl@yuntech.edu.tw, khhsia@mail.feu.edu.tw, magid@it.kfu.ru
http://kpfu.ru/robolab.html

Abstract
Experiments are valuable tool of robust control algorithms design, but experiments tend to be expensive. In order to
conduct thousands of complex experiments with autonomous car navigation algorithms it is safer and cheaper to startalgorithm verification within a simulation, relying on a proper robot model, which preserves physical properties ofunderlying objects. In this paper we present the design of Avrora Unior mobile robot model, which is a Russian carlike robot with Ackermann steering geometry, and describe the process of modeling its kinematics and dynamics.
Robot model was designed within open source robotics framework ROS for Gazebo simulator.
Keywords: car-like robot, non-holonomic robot, simulation, Avrora Unior, Gazebo, ROS.

UGVs2,3. It could be used as an environment for
performing various experiments: testing basic robot
motions, path planning and collaboration with otherrobots4, manipulation5, modeling USAR scenarios6 etc.
Other specialized simulators (e.g., UWSim7) could help
simulating water environments for AUV robots.
In our research we develop autonomous car
navigation and locomotion algorithms for mobile robot
Avrora Unior (Fig.1), and use Gazebo simulator toconstruct the robot model and to test control algorithms
before performing experiments with the real robot.

1. Introduction
In the past decades a broad use of simulations became an
essential part in robotics research field. We highlight thefollowing reasons of using simulation in robotics: realexperiments with robots are expensive while simulations
are cheap; simulation experiments take less time than realexperiments and are always safer than experiments in realworld; simulations allow testing novel concepts andalgorithms even if required hardware is not available fora user; simulations help to quickly detect and correct
gross errors in algorithms; finally, simulations could
provide fairly reasonable testing of experimental setups.
Those reasons apply to all robot types and tasks.
A simulator became a very progressive tool that canbe used to reproduce complicated environments (e.g.,water); they can use user-defined physics with dynamicchanges, which makes it possible to construct and use arobot model behaving very close to a real robot. Gazebo
is the one of the most popular 3D simulator of robots,
which was successfully used to simulate UAVs1 and2. Avrora Unior Car-like Robot
Avrora Unior robot is a car-like mobile robot that was
created by Russian company Avrora Robotics (Fig. 1).
Originally, this robot was designed for students’ training
in the field of autonomous driving, and such purpose
clearly influenced technical design of the robot. Due to asmall size and relatively low weight (~43,5 kg) of therobot, it is a safe solution for testing and validating© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan116

K. Shabalina, A. Sagitov, K. Su, K. Hsia and E. Magidalgorithms for autonomous driving both in indoor andoutdoor environments. As it is a small robot, it could be
easily moved around the laboratory room by two students.
Moreover, Avrora Unior provides Ackermann drive
mechanism that is used in real cars. Table 1 and Table 2
show linear dimensions and weight parameters of therobot, respectively. The robot consists of body panels(robot shell), chassis (metal backbone), power source(lead acid battery), sensors and control unit (laptop Acer
TravelMate P2). Another significant advantage of thisrobot is an open source software and off-the-shelf
hardware.
Hardware could be easily replaced, upgraded or new
elements could be retrofitted (e.g., sensors). The robot
has preinstalled Linux Ubuntu 16.04 LTS operating
system with ROS Kinetic. The vanufacturer also provides
open source steering and drive control packages.

Table 2. Weight of Avrora Unior robot parts.

Table 1. Robot dimensions.
Dimension (m)
0.204
0.26
0.11
1.112
0.57
0.65

Wheel
Shell of the robot
Cover
Hokuyo weight
Kinect weight
The entire robot

0.9
8.5
3.0
0.16
0.45
43.5

Gazebo Robot Model

This section describes the process of modeling. We
present the architecture of the model package, discuss
key elements and their characteristics, cover joint
geometry of the model and sensor specifications.

Fig. 1. Avrora Unior car-like robot.

Dimension

Weight (kg)Gazebo provides an open source 3D simulator
integration with ROS that can properly simulate a various
types of existing robot models (including AUVs, UGVs,
manipulators, etc.) and allows creating custom robots.
Gazebo evaluates world physics and a robot modelinteraction. Developers can test their algorithms with aproper robot model in a virtual environment a desired
number of times, modify the environment (e.g., add or
remove obstacles) to explore algorithm’s strengths andweaknesses before conducting field experiments withexpensive equipment. In addition, Gazebo provides
customizable plugins that simulate various types of
sensors. For our robot model we used standard sensor
plugins after a number of necessary adjustments. Gazebo
supports robot model in SDF and URDF formats andallows importing STL and DAE meshes for objects.
4.

Rim diameter
Wheel diameter
Wheel width
Lateral length of the entire robot
Robot height
Robot width

Element

4.1. Architecture
Using avrora_ros name for our metapackage wedecomposed the model into several individual packages:
avrora_description, avrora_gazebo, avrora_control.
This decomposition of the model provides a convenientand effective division of main functional parts of therobot model. Package avrora_description contains
general description of the model for ROS framework:
robot URDF (Universal Robotic Description Format)
representations, component meshes and materials of themodel. Package avrora_gazebo contains configurationsof the model that should be loaded into Gazebo simulator.
Package avrora_control contains configuration of themodel joints’ settings and geometry: joint type, PID
controller values and ROS controller settings.

3. Robot Operating System Environment
Robot Operating System (ROS) is a popular open source
framework for collaborative robot software development.
The framework is composed of atomic units (packages),which aim to solve particular problems in domain orprovide particular functionality for a certain robot(s). Allexisting packages in ROS could be improved by a user
and a new package could be added and described in a
special knowledge base.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan117

Avrora Unior Car-like Robot

Our project uses URDF model representation format,
which is a XML format file that describes a robot modelwith a help of special tags. The main tags, which are used
for describing virtual structure of a robot, are link andjoint tags. Links (robot elements) are connected with ajoints, generating a tree-structured representation.

Fixed joints attach rigidly two links with each other andin our model we used these joints to rigidly attach robot
sensors to the main body. Continuous type allows
rotation of a joint around a specified axis without upperand lower limits on the angle of rotation. Those type of
jointsinthemodel(left_steer_wheel_joint,
left_drive_wheel_joint,right_steer_wheel_joint,right_drive_wheel_joint) were used to connect thewheels to the chassis. Revolute joints allow constrained
rotation around a specified axis within upper and lower
limits. We used revolute type joints for steering joints(left_steer_joint and right_steer_joint), to implement
Ackermann steering geometry for model steering control(Fig. 3). The axis of rotation in both joints is Y-axis. Suchjoint configuration is a base for implementation ofAckerman drive mechanism for Avrora Unior model.

4.2. Core elements
We selected several link elements (i.e., core components)
to represent our model: base_link, right_steer_link,right_steer_wheel, right_drive_wheel, left_steer_link,
left_steer_wheel, left_drive_wheel. In order to make themodel visually similar to real Avrora Unior robot weapplied meshes for those links (the meshes are thecourtesy of Avrora Robotics company) withcorresponding inertial parameters (Fig. 2).
Link base_link represents the outer shell (or the body)of the robot. Links right_steer_link and left_steer_link
describe the parts of Ackermann steering mechanism
actuating the front wheels (right_steer_wheel and
left_steer_wheel, respectively). Links right_drive_wheel
and left_drive_wheel represents the rear driven wheels.
All links in the model were scaled to proportions anddimensions of the real robot. To adapt inertial
characteristics of the robot to the model, we calculated
simplified inertial tensor for all links. We substituted
compound meshes of a base_link with a solid cuboid
object, steer wheel links, rear and front wheels as a solid
cylinder to calculate approximated link inertia tensors.

steer wheels

wheel linkages

steer linkage

Fig. 3. Avrora Unior chassis scheme.

Fig. 2. Avrora Unior model in ROS/Gazebo environment.

4.3. Joints, motors and transmission

Fig.4. rqt_graph of Avrora Unior model topics (sensor’s
topics are not visualized).

URDF format provides several types of joints: prismatic,
revolute, continuous, floating and fixed. In our model weused three type of joints: fixed, revolute and continuous.

We approximate Ackermann geometry by moving
the steering pivot joints steering to angles (between© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan118

K. Shabalina, A. Sagitov, K. Su, K. Hsia and E. Magidsteering kingpins and the centre of the front axle) that arecalculated from a desired steer angle as Eq. 1:
⎧
⎨
⎩

=

∗

=с ∗

,

,

=

=

cot

cot

(

(

)−

)+

2

2

ℎ

ℎ

>0

<0

Table 3. Sensors and plugins.

(1)

where is the input angle, = 0.5, B is a half of thewheelbase, l1 is the wheel linkage length. As a result, wehave four continuous joints providing wheel rotation in
X-axis direction (linear motion on the plane) and two
revolute joints to replicate steering mechanism of
Ackerman drive system.
Since there are only two motor driven wheels in
Avrora Unior and a motor actuated steering mechanism,
we configured the following ROS control joint
controllers: position controllers SteerRight_controller
and SteerLeft_controller (ROS JointPositionController),effort controllers EffortDriveRight_controller andEffortDriveLeft_controller (ROS JointEffortController).
Controllers’ topic visualization is showed in Figure 4.

Sensor

ROS package

Gazebo plugin

LIDAR
Microsoft
KinectGPS
Proximity
IMU

hokuyo_node
freenect_stacklaser_controller
camera_plugin

ur_nmea_driver
ur_rangefinder_driver
myahrs_driver

novatel_gps_sim
Sonar
imu_plugin

Acknowledgements
Part of the work was performed according to theRussian Government Program of Competitive Growthof Kazan Federal University.
References
1. A., Sagitov and Y., Gerasimov, Towards DJI Phantom 4
Realistic Simulation with Gimbal and RC Controller in
ROS/Gazebo Environment. In Int. Conf. on Developments
in eSystems Engineering, (2017), pp. 262-266.
2. I., Afanasyev, A., Sagitov, E., Magid. ROS-based SLAM
for a Gazebo-simulated mobile robot in image-based 3Dmodel of indoor environment. In Int. Conf. on Advanced
Concepts for Intelligent Vision Systems, (Springer, Cham,
2015), pp. 273-283.
3. M., Sokolov et al., Modelling a crawler-type UGV for
urban search and rescue in Gazebo environment, In IEEE
Int. Conf. on Artificial Life and Robotics (2017).
4. W., Yao et al., A simulation system based on ros andgazebo for robocup middle size league, In IEEE Int. Conf.
on Robotics and Biomimetics, (2015), pp. 54-59.
5. W., Qian et al., Manipulation task simulation using ros andgazebo, In IEEE Int. Conf. on Robotics and Biomimetics,(2014), pp. 2594-2598.
6. S., Kohlbrecher et al., Robocuprescue 2014-robot league
team hector Darmstadt (Germany), RoboCupRescue(2014).
7. O., Kermorgant, A dynamic simulator for underwater
vehicle-manipulators, In Int. Conf. on Simulation,
Modeling, and Programming for Autonomous Robots(Springer, Cham, 2014), pp. 25-36.
8. A. Andreychuk and K. Yakovlev. Applying MAPP
Algorithm for Cooperative Path Finding in Urban
Environments. In Int. Conf. on Interactive Collaborative
Robotics, (2017), pp. 1-10.

4.4. Sensor Simulation
Table 3 presents the list of sensors of the real robot, theirimplementation in ROS framework and Gazebo
environment. The model sensors were placed in the same
positions as the real robot sensors in order to further
match real sensor data. All Gazebo sensor plugins wereconfigured to fit specifications of real robot sensors. We
visualize sensor data with Rviz software.
5. Conclusions and Future WorkIn this paper, we presented car-type Avrora Unior robot
modeling. We used ROS environment and Gazebo
robotics simulator. To construct the model, we used
URDF format, which is supported by Gazebo. Coreelements of Avrora Unior model were imported frommesh files provided by the manufacturer. Appropriate
joint controllers for Ackermann steering and sensor
simulations were added. As part of our future work, weare preparing a proper setup of the PID parameters of
joint controllers and developing an improved Ackermann
drive system. After successful modelling the cooperative
path finding tasks will be applied in simulated within
Gazebo urban environments8 prior to real world testing.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan119

Transfer of learned exploration strategies of a mobile robot from a simulated to realenvironments
Artur Sagitov1, Tetsuto Takano2, Shohei Muto3, Evgeni Magid1
1

Laboratory of Intelligent Robotic Systems (LIRS), Intelligent Robotics Department, Higher Institute of Information
Technology and Intelligent Systems, Kazan Federal University, 35 Kremlyovskaya street, Kazan, 420008, Russia
2
Department of Electronic Information Sciences, Kanazawa University, Kanazawa, Japan3
Department of Mechanical Science and Engineering, Kanazawa University, Kanazawa, Japan
E-mail: E-mail: sagitov@it.kfu.ru, shohei0625@stu.kanazawa-u.ac.jp, magid@it.kfu.ru
http://kpfu.ru/robolab.html

Abstract
Reinforcement learning based approaches show promises in various robotic applications, but a significant amount oftime and resources are required for a robot to learn optimal behavior. Using virtual environments, we could
significantly speed up and improve performance of a target task. We implemented a reinforcement learning based
exploration algorithm for a mobile robot, training in Gazebo environment and transferring learned strategy to a realrobot. We show that it is convenient and appropriate to use simulation to train strategies for mobile robots.
Keywords: navigation, algorithm, mobile robots, reinforcement learning, exploration.

obtaining information from real world trials to correct thesimulated environment.
One of the major frameworks for reinforcement
learning research is OpenAI Gym1. This frameworkprovides an easy way to debug and benchmark an
algorithm under a variety of different environments.
Zamora et al. extended OpenAI Gym functionality withinterfaces to Robot Operating System (ROS) and theGazebo simulator, to simplify the integration with therobotic hardware to validate existing reinforcement
learning algorithms in real environments2.
Our custom mobile robot was integrated into ROS
and Gazebo environment with control interfaces on both
real and virtual robots unified. The robot was trainedusing Q-Learning algorithm. Reward increased if it had
explored previously unsearched areas on the map anddecreased if it had a collision with the walls. With thetraining completed in the simulation, resulting strategy iseasily transferred to the real robot.

1. Introduction
Reinforcement learning is a field of machine learningwhere an agent learns a behavior by interactions within
particular environment. All interactions are being graded
and the goal of an agent is to achieve maximum possible
cumulative grade. Method is inspired by how humans aretaught in schools, i.e., taking tests and exams andreceiving positive or negative feedback. In robotics, suchmethod presents a way to design and implement complex
behavior that are hard to conceptualize. Our initial goal
was to build a simple mobile robot that autonomously
explores an environment based upon reinforced learningalgorithm, without any human interventions.
Major obstacle to be considered using reinforced
learning is an amount of time that should be spent forlearning optimal behavior, as real trials are slow andcostly. One of the possible solution to this problem is theidea of incorporating simulations of a real environment.
This process is self-correcting and can improve by© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan120

Artur Sagitov, Tetsuo Takano, Shohei Muto, Evgeni Magid

2. Related Work
Mathematical concepts of reinforcement learning began
when optimal control framework for Markov decision
process proposed by Bellman3 was reformulated by
Sutton4 and Watkins5. In recent years, reinforcement
learning has become an important method in robotics6. It
was applied to locomotion7,8, manipulation9-12 andautonomous vehicle control13. Combining reinforcement
learning with general-purpose neural networks shown
significant potential, including real-time control of 7
degree-of-freedom manipulators14-16. Using large anddeep neural networks have made it possible for robots tomaster complex manipulations with minimal manualengineering, though it is still unknown whether this canbe easily applied to an arbitrary task17-18.
In a study19 deep neural network (combination of
convolutional networks and a long short-term memory
network) learns to self-calibrate from a history ofpreviously set of actions and observations. Learning frommultiple simulated samples that had consisted of
trajectories and objectives, this network was able to learn
controlling a robotic arm successfully, achieving goals
being set from various frames of reference and using a
non-calibrated camera.

Fig. 1. Learning robot.

3. System Setup
Fig. 2. Virtual learning robot in Gazebo.

We used OpenAI gym-gazebo extension for a virtual
robot and a virtual environment. Gym-gazebo is acombination of OpenAI Gym, ROS and Gazebo. OpenAI
provides interface for implementation and testing of analgorithm that will controls real and virtual robot in the
Gazebo simulation20.

ROS package rosserial provided a ROS
communication protocol over Arduino's UART interface
and allowed publishing and subscribing to ROS
messages, publishing coordinate frame transforms andaccessing ROS system time. The robot control and sensor
processing was implemented using rosserial protocol bywrapping standard ROS serialized messages andmultiplexing multiple topics over to the microcontroller.
We implemented virtual representation of the learning
robot for the Gazebo simulator, including controllerinterfaces, controller managers, transmissions andhardware interfaces to match exactly with the real robot(Fig.2), using the same approach that we had exploited in
our previous research on crawler robot modelling21.

3.1. Learning Robot
The target for the experiments was a custom-build mobile
robot being controlled through a mounted Arduino Uno
microcontroller (Fig.1). This learning robot is a fourwheel mobile robot that was controlled by 4 DC motors(one per wheel, two motors on each side), with a springamortization on each wheel. Velocity controller was
implemented for each axis using ros_control package.
Robot uses differential steering for taking turns and canmake complete in-place (pivot) rotations by creating
difference in velocities of the left or right side wheels.
Sensory information was provided by Hokuyo LIDAR(UTM-30LX) mounted on the top of the frame.

3.2. Learning algorithm
We implemented multi-step Q-Learning method tofacilitate learning of exploration strategy. Q-learning is a

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

121

Transfer of learned exploration

widely used reinforcement learning method introducedby Watkins5 that uses formulated policies, constructing
robot strategy by defining action to take under whatcircumstances with probabilistic transitions and rewards.
We assumed exploration strategy as a finite Markov
decision process (FMDP). Using Q-learning we searched
for a strategy that is optimal in a sense that it maximizes
an expected value of a total reward (area explored) overall successive steps. Starting from an initial point indecision space, Q-learning can find optimal actionselection strategy given formalized FMDP. Reward wasused to provide the reinforcement and in our case itmeasured an added explored space after an action taken.
Learning robot have a set of states S, and a set of
possible actions in this state A. When action a∈A is
selected, the robot makes transitions from one state toanother state. With each action robot takes from aparticular state a reinforcing reward will be changed.
When a number of steps reaches a predefined limit or the
robot gets stuck we calculate a final reward.
The robot tries to maximize its total reward by adding
a maximum reward achievable from future states to thereward for achieving its current state, affecting current
action selection given a possible future reward. A total
potential reward, which is defined as a weighted sum of
expected values of rewards of all future actions startingfrom the current state, is taken as an objecting function.
At each step, we update the quality function:( , )← ( , )++ max (, )−( , ) , (1)where parameters α (learning rate) and γ (discountfactor) are hyperparameters that determine the influenceof new information and possible future rewards.
We implemented four possible actions in each state:
forward motion (both axes 0.2 m/s), left turn (left axis 0.1 m/s, right axis 0.1 m/s), right turn (left axis 0.1 m/s,
right axis -0.1 m/s) and backward motion (both axes -0.2m/s). The robot was severely penalized if it made acontact with a wall. As soon as a current action wasachieved, a new goal was provided and a reward wasissued based upon a change of explored territory area.

Fig. 3. Testing environment in ROS Gazebo.

LIDAR sensor to localize and map the environment,using no other positioning information.
4. Results
Our setup allowed us to get around 10 times faster thanreal time simulation; thus, our virtual robot could
perform ten times more learning trials than the real robot.
5000 trials were completed within around 6 hours. Figure
4 demonstrates that the robot exploration strategy
significantly improved over time, as the average total
reward over time increased.
Even with training performed entirely in a simulator
different from a real world, we have obtained exploration
strategies, which perform well on the physical robot. We
attribute successful transfer to randomizations ofsimulated environment and unified control interface
between real and virtual robot.
We observed several factors that contributed to thedifference between the simulation and the reality duringexecutions: the real robot exhibits a lateral and longitude
wheel slip, accumulates wheel velocity inaccuracies andis more prone to collisions when driving near the walls.

3.3. Exploration environment
As a test for our setup, we selected a standard gymgazebo labyrinth GazeboCircuit2TurtlebotLidar-v0. This
environment consists of a simple straight lined circuitwith five right turns and one left turn (Fig.3). We used a

Fig. 4. Reward after trials.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

122

Artur Sagitov, Tetsuo Takano, Shohei Muto, Evgeni Magid

5. Discussion and Future Work6. J.Kober, J.A. Bagnell, J. Peters, Reinforcement learning
in robotics: A survey, in The Int. J. of Robotics Research,
32, (2013), pp. 1238-1274.
7. N. Kohl and P. Stone, Policy gradient reinforcement
learning for fast quadrupedal locomotion, in Int. Conf. on
Robotics and Automation 3 (2004), pp. 2619-2624.
8. G. Endo, J. Morimoto, T. Matsubara, J. Nakanishi, and G.
Cheng, Learning CPG-based biped locomotion with apolicy gradient method: Application to a humanoid robot,
in Int. J. of Robotic Research, 27(2) (2008), pp. 213–228.
9. J. Peters and S. Schaal, Reinforcement learning of motor
skills with policy gradients, in Neural Networks, 21(4)(2008), pp. 682–697.
10. E. Theodorou, J. Buchli, and S. Schaal, Reinforcement
learning of motor skills in high dimensions, in Int. Conf.
on Robotics and Automation (2010) pp. 2397-2403.
11. J. Peters, K. Mulling, and Y. Altun, Relative entropy
policy search, in AAAI Conference on Artificial
Intelligence (2010), pp. 1607-1612.
12. M. Kalakrishnan, L. Righetti, P. Pastor, and S. Schaal,
Learning force control policies for compliant manipulation,
in Int. Conf. on Intelligent Robots and Systems (2011), pp.
4639-4644.
13. P. Abbeel, A. Coates, M. Quigley, and A. Ng, An
application of reinforcement learning to aerobatic
helicopter flight, in Advances in Neural Information
Processing Systems (2006) pp. 1-8.
14. P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal,
Learning and generalization of motor skills by learningfrom demonstration, in Int. Conf. on Robotics andAutomation (2009), pp. 763-768.
15. J. Schulman, S. Levine, P. Moritz, M. Jordan, and P.
Abbeel, Trust region policy optimization, in Int. Conf. on
Machine Learning (2015), pp. 1889-1897.
16. S. Levine, C. Finn, T. Darrell, and P. Abbeel, End-to-end
training of deep visuomotor policies, J. of Machine
Learning Research, 17(1) (2016), pp. 1334-1373.
17. M. Deisenroth and C. Rasmussen, PILCO: a model-based
and data efficient approach to policy search, in Int. Conf.
on Machine Learning (2011), pp. 465-472.
18. T. Moldovan, S. Levine, M. Jordan, and S. Abbeel,
Optimism-driven exploration for nonlinear systems, in Int.
Conf. on Robotics and Automation (2015), pp. 3239-3246.
19. F. Sadeghi, A. Toshev, E. Jang and S. Levine, Sim2Real
Viewpoint Invariant Visual Servoing by Recurrent Control,
in IEEE Conf. on Computer Vision and Pattern
Recognition (2018), pp. 4691-4699.
20. I., Afanasyev, A., Sagitov, E., Magid. ROS-based SLAM
for a Gazebo-simulated mobile robot in image-based 3Dmodel of indoor environment. In Int. Conf. on Advanced
Concepts for Intelligent Vision Systems, (Springer, Cham,
2015), pp. 273-283.
21. M., Sokolov et al., Modelling a crawler-type UGV for
urban search and rescue in Gazebo environment, In IEEE
Int. Conf. on Artificial Life and Robotics (2017).

There are several ways we are trying to improve thelearning process. In the order of priority, we list:
1. Verify the approach for more complicatedrobots.
2. Speed up simulation for robots using multiple
computers.
3. For complex environments, make simulation
run at a faster speed.
4. Further diversify environments and add
multiple robots.
5. Implement additional tools for calculating
performance metrics for different algorithms.
6. Make automatic recommendations from theresultant strategy.
6. Conclusion
Reinforcement learning plays a significant role in thegrowing field of machine learning and, in order toovercome difficulties, robotics simulator like Gazebo isshown saving costs and speeding up the learning process.
We have shown that reinforcement learning algorithms
are capable of learning complex exploration skills fromscratch and without purposefully designed trajectories,
but our method has a number of limitations, such as
simplified robot and test environment. We have plans toaddress these issues in the future.
Acknowledgements
This work was supported by the Russian Foundation for
Basic Research (RFBR), project ID 18-58-45017. Part of
the work was performed according to the Russian
Government Program of Competitive Growth of Kazan
Federal University.
References
1. G. Brockman, et.al., OpenAI gym (arXiv preprint,
arXiv:1606.01540, 2016).
2. I. Zamora, et.al. Extending the OpenAI Gym for robotics:
a toolkit for reinforcement learning using ROS and
Gazebo (arXiv preprint arXiv:1608.05742, 2016).
3. R. Bellman, A Markovian decision process, in Journal of
Mathematics and Mechanics (1957) pp. 679-684.
4. R.S. Sutton, Temporal credit assignment in reinforcement
learning, (PhD Thesis, University of Massachusetts,
Amherst, MA., 1984).
5. C.J.C.H. Watkins, Learning from Delayed Rewards (Ph.D.
thesis, Cambridge University, 1989).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan123

Extending Gazebo simulator for surgical robotics: tissue and suture modeling
Artur Sagitov1, Hongbing Li2, Natalia Schiefermeier-Mach3, Evgeni Magid1
1

Laboratory of Intelligent Robotic Systems, Intelligent Robotics Department, Kazan Federal University, Kazan, Russia
2
Department of Instrument Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
3
Health University of Applied Sciences Tyrol, Innsbruck, Austria
E-mail: sagitov@it.kfu.ru, natalia.schiefermeier-mach@fhg-tirol.ac.at, magid@it.kfu.ru
http://kpfu.ru/robolab.html

Abstract
Active use of a simulator as a training tool has proven to be advantageous to a human surgeon, but there is no open
source and convenient universal surgery simulation of a robot surgeon. This paper presents an extension of Gazebo
simulator for surgical robots using Robot Operating System. We present software architecture that allows modelingrobot interaction with different types of tissue and suture.
Keywords: robotic surgery, tissue modelling, suture modelling, Gazebo, ROS.

ways to address this is combining available separate
domain frameworks into a single modular package.
We used Gazebo simulation as a base for our surgery
robot simulation because of its support for robot
operating system (ROS)1, but a number of changes wererequired as Gazebo is not capable of deformable objects
simulation. Gazebo consists of two main components: aserver and a client. The server evaluates physics of theenvironment: entire state of each object being involvedin the simulation, forces and velocities of each object
and external controllers’ input. The client uses server
output to displays the environment in a graphical userinterface in order to provide visualization and control.
The server can use different physics’ engines each being
optimized for different use cases. We have selected
Bullet physics engine for existing soft body routines andhad to implement a new external plugin for Gazebo inorder to add soft body modeling to Gazebo.
Modern vector graphics editors and computer games
use a special data structure that represents all entities - a
scene graph, which contains different object
representations in the scene. In our simulation weorganize objects, their relations and algorithms in a

1. Introduction
Robotic surgery field experiences exponential growthand emerging new techniques present new challenges
for training of surgeons. We note the increase in qualityof robotic surgery simulations and incorporation of
those into modern surgical training curriculum.
Simulators have proven to be advantageous in a practice,
but they are designed for a human training and it is notpossible to use those for developing autonomous
procedures.
Surgery simulation is an interactive physicalsimulation that involves rigid and deformable objects.
Simulation development engages various appliedmathematics branches including numerical analysis,
geometric modeling, computational mechanics, collision
detection, and rendering, while their computational
efficiency is achieved with compound algorithms andmulti-threading that take full advantages of modern
hardware capabilities. Therefore, developing surgicalsimulations is a challenging task that requires anexpertise in all of the aforementioned areas. One of the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan124

Artur Sagitov, Hongbing Li, Natalia Schiefermeier-Mach, Evgeni Magidtensor pre-computation5, multi-grid solvers6 and domain
decomposition7 could be applied. FEM that is used inreal time surgical simulations often uses simplified
linearized equations for material law and strain
computation. These simplifications are applicable forsmall deformations and cause significant errors for large
deformations8. To correct for large scale deformations ageometric nonlinear tensor-mass model has been
applied9 but it imposes complicated computations using
quadratic formulations of strain. To address geometric
nonlinearity involving rotational deformations FEM
could be supplemented with a co-rotational
formulation10. In meshless method11 a deformation
could be evaluated without explicit node connections
with a penalty of additional computations of nodeadjacency metric at each iteration. Lack of explicit
topology is also an issue12. Formulation of Lagrangian
dynamics for FEM taking into account geometric andmaterial nonlinearities method13 depends upon constant
topology of an object as it uses precomputations, andhas shown numerical stability with only small time steps.
To address deformation nonlinearity method of
hyperelastic mass links a new approach14 was proposedthat provides fast computation, but as of now isapplicable only for homogenous objects15. Boundary
element method16 was unable to address objects
anisotropic and heterogeneous properties. Using
specialized hardware, such as Graphics Processing Unit
(GPU), could be used to handle computational load17.
Simulation framework SOFA18 was developed with afocus on deformable objects and complex interactions,and can simulate a large variety of models, including
rigid-bodies, MSM and FEM with hyperelastic masses.
Our implementation for Gazebo simulator is a somehow
intermediate approach between simple game physics
engines and a complicated FEM analysis.

hierarchical data structure that is similar to scene graphs.
Each object is decomposed into a set of properties: statevectors, mass, forces, constraints, topology, integration
scheme, and solving process. Simulation algorithms canbe customized for each component individually as well
as for physical models themselves. A physical object isrepresented as a combination of four models: internal
model, inertial model, collision model and visual model.
We propagate forces and movements between themodels concurrently during the simulation.
2. Related Work
In a field of rigid body simulations for robotics there isa number of different products, including an opensource software Gazebo2, proprietary V-REP,
WorkspaceLt, RoboticSimulation, Webots and others.
Those simulators rely on different physics engines (e.g.,Open Dynamics Engine (ODE), Bullets, DART or
NVidia PhysX) and are fast and efficient in rigid-bodysimulations and easy to use. The main disadvantage
from our perspective is that very few of them arecapable of modeling realistic soft bodies. Usually soft
object modelling involves a use of structural multiphysics tools such as ComSol. As those tools use
continuous mechanics algorithms the disadvantage ofthis method is a slow computational speed and therequirement of deep understanding of the physicalprocesses behind.
Modeling soft objects is an active field of research,
and a number of methods exist for soft tissue
deformation in a surgical simulation. Mass-springmodels (MSM) approximate a soft tissue as a set ofpoint masses being interconnected with massless elastic
springs. Advantages of MSM are their easy
implementation and efficient evaluation. Disadvantages
include difficulty in distribution of material propertiesand dependency of deformation upon a selecteddiscretization topology3. When deformation accuracy isa concern, a finite element method (FEM) is considered.
FEM represents a soft tissue model as a continuous
object and constructs built-in equations to model
complex mechanical behavior of soft tissues. Equations
approximating geometric and material properties being
involved in deformation are numerous, which makes
FEM computationally expensive; moreover, it canbecome numerically unstable. To address the issue ofthe computational efficiency of FEM model reductions4,

3. Tissue and suture models
Both tissue and suture are represented as soft
deformable bodies. Assuming suture simple geometry,
we directly use a default 2D discretization for solving
constitutive equations. The tissues are a 3D object andto construct a mechanical model we had to discretize thetissue domain. We use CGAL (an efficient geometricC++ library) to generate a hexahedral polygonal meshfrom 3D model. We adjusted discretization parameters
to have the best deformation accuracy given selected© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

125

Extending Gazebo simulator for surgical robotics

for discretization options, contact configuration andoptions for numerical integration and rendering. Using a
synchronized tissue model, we collect force vectors andcontact points from different representations to obtainan entire system. Applying our Gazebo plugin, we
connect the Gazebo world and the soft tissue to allow
interaction with other objects. Plugin synchronizes
locations of each of soft body nodes between different
representations. Rendering module of the plugin utilizes
the state of the physical simulation to render avisualization because Gazebo does not have deformablemeshes rendering capability. We render our meshes
using custom OGRE renderer as a plugin to Gazebo thatis implemented through rendering::visual class.
In our current configuration of the plugin we allow
interaction of a rigid body object (manipulator) with asoft tissue in a realistic manner. Behavior of thesimulated soft tissue is sensitive to parameter selection,mechanical parameters and tissue topology whichshould be specified manually.

limit of volume elements to address the performance.
Discretization example is presented in Fig.1.
Deformable objects’ behavior is a subject to thelaws of continuum mechanics for material modeling,
Lagrangian multipliers for constraints solving, andSignorini problem for contact resolution. Internal forcesof a simulated object are dependent on a current state
and a sum of all known external forces. Amount of
deformation depends only on external loads. We use
Hooke’s law for this dependency because ofcomputation time constraints. Deformation is treated as
purely elastic behavior. Each tissue type specifies
Young modulus and Poisson’s ratio as parameters.
When a potential contact with a robot end-effector has
been detected, we measure the distance between the
robot and an obstacle at a contact point. Using contact
mechanics, we find an area of the contact patch and finda displacement vector by resolving Signorini problem.

5. Examples of abdomen simulation
We are using abdomen models available from
OpenHELP phantom, which are designed with realistic
anatomy and material characteristics, for ourautonomous suturing system prototype testing19. Using
plugin for each of the tissue type, abdomen models were
imported to Gazebo with proper mechanical properties.
Model of the interaction is represented on Fig. 2.

Fig. 2. Integrated model in Gazebo
Fig. 1. Model discretization and triangulation

We used a custom manipulator to interact with thesoft tissue to observe the interaction results. Theinteraction simulation took around 5 seconds for 1 realtime second (time step of 5ms, 10000 mesh nodes)
using Haswell-E i7-5820K. The complete soft tissue
model therefore was successfully implemented for

4. Gazebo integration
A scene contains the robot and its environment and is
described within XML file. Soft tissues are definedusing an extension of URDF format with components© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan126

Artur Sagitov, Hongbing Li, Natalia Schiefermeier-Mach, Evgeni Magid

Gazebo, although the performance should be improved
further.

8. Y. Payan, Soft Tissue Biomechanical Modeling for
Computer Assisted Surgery, vol. 11. (Springer, Berlin,2012).
9. S. Xu, P. X. Liu, H. Zhang, A nonlinear viscoelastic
tensor-mass visual model for surgery simulation, in IEEE
Trans. Instrum. Meas.60, (2011), pp. 14–20.
10. R. Plantefève, I. Peterlik, N. Haouchine, S. Cotin,
Patient-specific biomechanical modeling for guidance
during minimally-invasive hepatic surgery, in Ann.
Biomed. Eng.44, (2016), pp. 139–153.
11. G.Y. Zhang, A. Wittek, G.R. Joldes, X. Jin, K. Miller, A
three-dimensional nonlinear meshfree algorithm forsimulating mechanical responses of soft tissue, in Eng.
Anal. Bound. Elem.42, (2014), pp. 60–66.
12. V.P. Nguyen, T. Rabczuk, S. Bordas, M. Duflot,
Meshless methods: a review andcomputer
implementation aspects, in Math. Comput. Simul.79 (3),(2008), pp. 763–813.
13. K. Miller, G. Joldes, D. Lance and A. Wittek, Total
Lagrangian explicit dynamics finite element algorithm
for computing soft tissue deformation, in Int. J. Numer.
Methods Biomed. Eng.23 (2), (2007), pp. 121–134.
14. F. Goulette, Z.-W. Chen, Fast computation of soft tissue
deformations in real-time simulation with Hyper-Elastic
Mass Links, in Comput. Methods Appl. Mech. Eng.295,(2015), pp. 18–38.
15. J. Zhang, Y. Zhong, and C. Gu, Ellipsoid bounding
region-based ChainMail algorithm for soft tissue
deformation in surgical simulation, in International
Journal on Interactive Design and Manufacturing, vol.
12, no. 3, (2018), pp. 903-918.
16. P.Wang, A.A. Becker, I.A. Jones, A.T. Glover. S.D.
Benford. C.M. Greenhalgh, M. Vloeberghs: Virtual
reality simulation of surgery with haptic feedback basedon the boundary element method, in Comput. Struct. 85(7–8), (2007), pp. 331–339.
17. S.F. Johnsen, Z.A. Taylor, M.J. Clarkson, J. Hipwell, M.
Modat, B. Eiben, L. Han, Y. Hu, T. Mertzanidou, D.J.
Hawkes, S. Ourselin, NiftySim: a GPU-based nonlinear
finite element package for simulation of soft tissue
biomechanics, in Int. J. Comput. Assist. Radiol. Surg.10(7), (2015), pp. 1077–1095.
18. F. Faure, C. Duriez, H. Delingette, J. Allard, B. Gilles, S.
Marchesseau, H. Talbot, H. Courtecuisse, G. Bousquet, I.
Peterlik, S. Cotin, SOFA: A Multi-Model Framework forInteractive
Physical
Simulation,
Soft
TissueBiomechanical Modeling for Computer Assisted Surgery,11, Studies in Mechanobiology, Tissue Engineering andBiomaterials, (Springer, 2012), pp. 283-321.
19. A. Sagitov, H. Li, E. Magid. Toward automated open
wound suturing using haptic feedback: detecting wounds
and planning the suture. Int. Conf. on Artificial Life andRobotics, (2018), pp. 208-211.

6. Conclusions
We developed and implemented a new soft tissue
Gazebo plugin with an emphasis on performance andhave achieved an acceptable level of usability. The
plugin demonstrates realistic interaction with soft tissue.
As a future work, we consider to improve performancewith a GPU-based approach and to allow for complex
elastic interactions’ simulation. We intend to turn thisplugin into a basis for long-term developments in robot
surgery automation.
Acknowledgements
This work was supported by the Russian Foundation for
Basic Research and the Government of the Republic of
Tatarstan, project ID 18-48-160037. Part of the work
was performed according to the Russian Government
Program of Competitive Growth of Kazan Federal
University.
References
1. A. Sagitov, T. Tsoy, H. Li, E. Magid. Automated open
wound suturing: detection and planning algorithm. J. of
Robotics, Networking and Artificial Life, Volume 5, Issue
2, September 2018, pp. 144 - 148
2. N. Koenig and A. Howard, Design and use paradigms forgazebo, an open-source multirobot simulator, in Proc. ofConf. on Intelligent Robots and Systems, vol. 3 (2004).
3. Y. Duan, W. Huang, H. Chang, W. Chen, J. Zhou, S.K.
Teo, Y.Su, C.K. Chui, S. Chang, Volume preserved
mass-spring model with novel constraints for soft tissue
deformation, in IEEE J. Biomed. Health Inform.20 (1),(2016), pp. 268–280.
4. C. Yang, S. Li, Y. Lan, L. Wang, A. Hao, H. Qin,
Coupling time-varying modal analysis and FEM for realtime cutting simulation of objects with multi-material
sub-domains, in Comput. Aided Geom. Des.43, (2016),
pp. 53–67.
5. S. Cotin, H. Delingette, N. Ayache, Real-time elastic
deformations of soft tissues for surgery simulation, IEEE
Trans. Vis. Comput. Graphics, 5 (1), (1999), pp. 62–73.
6. D. Weber, J. Mueller-Roemer, C. Altenhofen, A. Stork,
D.W. Fellner, Deformation simulation using cubic finite
elements and efficient p-multigrid methods, in Comput.
Graphics UK 53, (2015), pp. 185–195.
7. J. Huang, X. Liu, H. Bao., B. Guo, H.Y. Shumb, An
efficient large deformation method using domain
decomposition, in Comput. Graphics UK30 (6), (2006),
pp. 927–935.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

127

Experiences of Robotics students in Machine Vision course being taught in a foreign language:
comprehension, self-efficiency, and active learning strategies improvement
Tatyana Tsoy, Artur Sagitov, Evgeni Magid,

Laboratory of Intelligent Robotic Systems (LIRS), Intelligent Robotics Department, Higher Institute of Information
Technology and Intelligent Systems, Kazan Federal University, 35 Kremlyovskaya street, Kazan, 420008, Russia
E-mail: {tt, sagitov, magid}@it.kfu.ru
www.kpfu.ru/robolab.html

Abstract
Crisis in Russia in 1990s significantly decreased engineering education quality. To catch up with developed countries,
a novel 2-year master program in Intelligent Robotics was designed and implemented at Kazan Federal University
based on world leading universities’ experience. The program targets to educate competitive specialists withcompetences that are required by world labor market. To follow students’ progress in core robotics courses we
conducted surveys. The paper reports survey analysis that was conducted in Machine Vision course and covered such
issues as English language comprehension, self-efficiency, and active learning strategies.

Keywords: Engineering education, robotics education, postgraduate studies, master program, Machine Vision,
motivation.

As a result of these efforts, a new master program inIntelligent Robotics was launched in September 2017 atHigher Institute of Information Technology andIntelligent Systems of Kazan Federal University, and its
syllabus was designed to consider local employers’
requirements4. After establishing the pilot program, we
continue conducting student feedback surveys about ourcourses and to update the courses aiming for further
effectiveness improvement. This paper presents analysis
of Machine Vision course survey that was conducted inthe (first) Fall semester of 2017-2018 academic year.

1. Introduction
We are evidencing comprehensive digitalization andautomation of all aspects of our daily life, including
healthcare1, economics2, etc. Social and demographic
challenges that developed countries are recently facing
force us to develop technologies at accelerated pace inorder to replace human labor with robotic systems (RS).
Consequently, world demand for specialists in robotics is
growing rapidly and creating new challenges for up-todate engineering education.
According to Robotics Russian Association, robot
density in 2014 was 66 RS per 10 000 workers while indeveloped countries the automation level is significantly
higher, e.g., 478 RS in Korea, 314 RS in Japan, 292 RS
in Germany, 164 RS in US3. To catch up with developed
countries Russian government started long-term
investments into engineering education and establish new
higher education programs.

2. Postgraduate robotics studies
The master program targets to educate highly skilled
experts in the field of Intelligent Robotics using modern

set of robotic equipment that includes various types ofrobots and broad selection of sensors and cameras.
Along with technical skills in robotics we provide© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

128

Tatyana Tsoy, Artur Sagitov, Evgeni Magid

postgraduate students with such courses as Social
Robotics and Communication Skills as serious robotics
projects are always a team work and soft skills are a mustin order to successfully implement a project.
Before opening the master program in Intelligent
Robotics at KFU, we had started our surveys among
bachelor students in order to understand motivation,
evaluate environment of teaching in English and an
opportunity to allow complicated material explanations
in Russian considering the fact that all students were
Russian native-speakers4. Further to have a regular
update and improve our master program based on
feedback from students we continued conducting surveys
with students that are enrolled in Intelligent Robotics
master program on the core courses of our syllabus such
as Introduction to Robotics8, Sensors and Sensing, Robot
Operating System (ROS)5 and Machine Vision.

5. Stereovision: calibration, epipolar geometry,
homography, stereo disparity, structure from motion, andoptical flow.
6. Recognition: Machine learning in computer vision,neural networks, and image categorization.
By completing the course successfully, theparticipant was expected to possess the following skills:- understand theoretical and practical aspects of image
manipulation, formation, measurement, andanalysis;- implement methods of image matching andalignment;- determine geometric relations between 2D images
and the 3D environment;- know and able to apply object and scene recognition,
image categorization;- design architecture of computer vision applications.

3. Curriculum of Computer Vision Course

4. Research Method and Analysis

Machine Vision is one of the essential disciplines in themaster program curriculum as visual feedback is very
important for robots especially in industry where wide
range of visual systems is applied6. In the last decadevision systems become more available due to technological improvements and progressively decreasing costs,
which provided more facilities for developing new
applications and using robots in production. In our
curriculum we take into account that vision-assisted
robots taking over a number of manufacturing jobs suchas quality control, precision manufacturing and productsorting. Additionally, they also play a significant role
human-robot collaboration and multi-agent robotics.
According to survey results no respondent studied
Machine Vision subject before entering the program,either at educational organizations or by self-studying.
Before launching the program, we anticipated this factand designed Machine Vision curriculum assuming thatstudents do not possess any knowledge in this field. Thus,
the curriculum consisted of the following topics:
1. Image formation: optics, cameras, and representations.
2. Image transformations: sampling, color spaces,
convolutions, linear filters, and histograms.
3. Nonlinear filtering: median filter, Fourier and Laplacetransforms, frequency analysis, and transfer function.
4. Feature Detectors: feature points and matching,
gradient, edge and corner detection, Harris detector,
Canny and Sobel filters.

The surveys assist us to analyze attitude, motivation, andchallenges that students faced during Machine Visioncourse in the first semester of the master program, whichlasted 1.5 astronomical hours, and was run 18 weeks. We
conducted initial survey after the first class and a final
survey in the end of the course before a final test. The
initial survey included 49 questions that referred to
students’ expectations from the course and previous
experience, while the final survey included 39 questions
and questioned about new in-class experience. A numberof identical questions in both surveys allowed observing
dynamics of English language and motivation. The
surveys were conducted in Russian via on-line Google
forms. The questions contained statements, open-ended
and multiply choice questions. Each statement waspresented on a 5-point scale with optional answers:
strongly disagree (SD), disagree (D), no opinion (NO),agree (A) and strongly agree (SA), which appear along
X-axis in Fig. 1-6. Y-axis of Fig. 1-6 indicate percentageof the respondents that selected the corresponding
options. We applied the same research method that we
had used in our previous papers4,5,7,8. For the analysis weselected 9 respondents with technical background who
participated in both initial and final surveys.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

129

Experiences of Robotics students

4.1. English language comprehension

to be the most difficult one in both initial and final
surveys, 88,8% of them disagreed with the statement “I
think, I will not be able to learn the subject no matter how
much efforts I put”. Moreover, responding “If a lecture
content is difficult I would avoid learning the material”
statement, the students replied 66,6% SD and 33,3% D in
the beginning of the course and 77% SD, 11,1% NO,
11,1% A by the end of the course, which supports theconclusion that despite difficulty of the course more
students shifted to be more self-efficient while learning
the material. Figure 3 shows that during the course 77,7%of the students thought that they will pass the final
examinations successfully.

Machine Vision classes were taught in English to Russiannative speakers considering that students had somehistory of studying English language at school anduniversity. English language comprehension was
assumed the main milestone for effective teaching,
however, the students did not meet significant difficulties
with receiving information in English4,5. As Fig.1
demonstrates, the tendency of using English with lessanxiety was positively progressive by the end of thecourse. A special language environment was created forthe students to feel comfortable and perceive material
more effectively. Figure 2 depicts a reflection on theopportunity to ask for explanations in Russian if a student
could not understand the material in English, and thetendency was positive toward the course end.

Fig. 3 I am sure I will successfully pass Machine Vision course.

4.3. Active learning strategies

Fig. 1 I do not worry if I make mistakes while speaking English
during classes.

During the course the students not only passively
digested the material but also actively learned it using
extra sources (Fig. 4). One of the motivation components
to study the subject could be related to students’ plans
after graduation as it is shown in Fig. 5: 100% of thestudents (66,6% SA, 33,3%A) realized the importance ofstudying the subject since they believed that they aregoing to use this knowledge in a future job.
Moreover, 6 out of 9 respondents would like toconnect their future with robotics, while 3 did not decideyet. However, even with that high motivation the course
was not an easy one for the respondents as 77,7% (66,6%
SD and 11,1% D) disagreed with the statement that theywould study the course independently rather thanparticipate in the classes (Fig. 6), which adds credits to
self-efficiency where students considered the subject themost difficult among other courses.

Fig. 2 I am not afraid if I do not understand when the teacher
speaks English, because I can ask for explanations in Russian.

4.2. Self-efficiency
In the end of the course majority of students (11,1% SA
and 44,4% A vs. 44,4% NO) considered the course was
the most difficult for them in comparison with othercourses. Even though 55,5% of the students considered it© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

130

Tatyana Tsoy, Artur Sagitov, Evgeni Magidof the respondents associated their future career withrobotics. Next we plan to develop new surveys in orderto evaluate master program in Intelligent Robotics
courses content comprehension along with conducting
continuous surveys to track students’ needs and to follow
the robotics area global market employers’ requirements.
Acknowledgements
This work was supported by the Vladimir Potanin
Foundation, project ID GK180000602. Part of the work
was performed according to the Russian Government
Program of Competitive Growth of Kazan Federal
University.

Fig. 4 If I do not understand new material of Machine Visioncourse, I shall try to find extra sources myself to understand it.

References
1. A. J. Aljaaf, C. Mallucci, D. Al-Jumeily, A. Hussain, M.
Alloghani, J. Mustafina. A Study of Data Classificationand Selection Techniques to Diagnose. Applications of Big
Data Analytics: Trends, Issues, and Challenges, Springer(2018), p 121-134.
2. R. C. Brown. Made in China 2025: Implications of
Robotization and Digitalization on MNC Labor Supply
Chains and Workers' Labor Rights in China. Tsinghua
China L. Rev. (2016), 9, 186.
3. Russian National Association of Robotics Market
Participants. Analytical Research: World Market of
Robotics (January, 2016). http://www.robotunion.ru/ru/
4. T. Tsoy, L. Sabirova, & E. Magid, (2017). Towards
Effective Interactive Teaching and Learning Strategies in
Robotics Education. IEEE Int. Conf. on Developments ineSystems Engineering (2017), pp. 267-272.
5. T. Tsoy, L. Sabirova, & E. Magid. Effective robotics
education: Surveying Master Program Students
Experiences in Robot Operating System Course. IEEE Int.
Conf. on Developments in eSystems Engineering (2018).
6. L. Pérez, Í. Rodríguez, Í., N. Rodríguez, R. Usamentiaga,
R., & D. García, D. (2016). Robot guidance using machine
vision techniques in industrial environments: A
comparative review. Sensors, 16(3), 335.
7. T. Tsoy, L. Sabirova, M. Abramsky, E. Magid.
Establishing Effective “Teaching for Robotics: acomparison study of Bachelor students participated in
Introduction to Robotics course”. Int. Conf. on Artificial
ALife and Robotics, pp. 212-215, 2018.
8. T. Tsoy, L. Sabirova, E. Magid. “Effective robotics
education: Surveying Master Program Students
Experiences in Introduction to Robotics Course”, Int. Conf.
on Mechanical, System and Control Engineering, vol. 220,2018.

Fig. 5 I think that studying Machine Vision course isimportant, because I shall use this knowledge in my future job

Fig. 6 If I had an opportunity not to take Machine Vision class
I would have better learned it by myself.

5.

Conclusions

Analysis of surveys among postgraduate students of
Machine Vision course showed positive results in thecourse material comprehension in English while having
an opportunity to ask for additional explanations in
Russian. Despite the fact that the course was evaluated as
the most difficult one among others, the students applied
significant self-efficiency to learn the subject and used
active learning strategies while possessing highmotivation to study the course because the vast majority© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

131

Pilot Virtual Experiments on ArUco and AprilTag Systems Comparison for Fiducial Marker
Rotation Resistance
Aufar Zakiev, Ksenia Shabalina, Evgeni Magid

Laboratory of Intelligent Robotic Systems (LIRS), Intelligent Robotics Department, Higher Institute of Information
Technology and Intelligent Systems, Kazan Federal University, 35 Kremlyovskaya street, Kazan, 420008, Russia
E-mail: zaufar@it.kfu.ru, ks.shabalina@it.kfu.ru, magid@it.kfu.ru
http://kpfu.ru/robolab.html

Abstract
There exists a large number of fiducial marker system types and both researchers and industry have difficulties toselect among this variety a single system that could provide optimal behavior for a particular task. This paper presents
design and results of pilot virtual experiments that were conducted in order to compare a performance of two marker
systems, ArUco and AprilTag. Experiments were designed to estimate and compare marker systems resistance torotation with regard to different principal axes in 3D space. Pilot experiment design eliminates influence of external
environment, including light conditions, camera resolution, sensor noise, distance between camera and marker, etc.
Experiments were implemented in ROS/Gazebo environment. In total over 300,000 virtual experiments were
performed and analyzed in order to collect statistically significant data amount.
Keywords: robotics, fiducial marker system, recognition algorithms, experimental comparison, ROS, Gazebo.

task the FMS should be resistive to manipulator rotations
and partial occlusions being caused by marker overlap
with various objects (e.g., robot’s parts).
Early approach to systematically compare the FMSs
was conducted through multiple manual experiments inorder to estimate different markers’ resistance to rotationand overlapping (both systematic and arbitrary).
However, this approach has a number of significant
disadvantages, which make manual experiment results
hardly reproducible:
Overwhelming time consumption. Multiple
iterations are required in order to collect statisticallysignificant amount of data.
Complexity of experiments’ fairness control.
Multiple environment conditions, e.g., inclination
angle, marker position with regard to a camera,
lighting conditions, etc. are hard to monitor andcontrol.
Limited hardware choice. Hardware has unavoidable
noises and often do not possess desired properties,e.g., camera resolution, lens distortion level, opticalsensor sensitivity, etc.

1. Introduction
Fiducial marker systems (FMS) are systems of planargraphical markers that are designed to be detected bycorresponding machine vision algorithms. FMS are
widely used in physics, medicine, robotics, augmented
reality, metrology, robotics, etc. Broad range of robotictasks including navigation1, localization2, mapping3 andcamera calibration4 use FMS as a main element. Our
long-term goal is to calibrate several Russian robots,
including humanoid AR-601 robot (Fig. 1), and FMS
usage is way to accomplish this task in automated manner.
Modern FMSs have different designs and are developed
for various purposes: each of them has its own
advantages and drawbacks. Therefore, a suitable FMS
choice requires to compared systems for various criteria,
paying attention to criteria that are important for aparticular task of interest. Our goal is to auto calibrate
robot cameras: markers are placed on the humanoid
robot’s manipulator (e.g., palm) and the humanoidobserves this marker in order to estimate andprogrammatically eliminate camera distortions. For this

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan132

Aufar Zakiev, Ksenia Shabalina, Evgeni Magidconsists of nodes and topics between them forcommunication. Such distributed structure allows
creating various data and command flow schemas,
making sensor data analysis and robot motion control
easy. ROS is distributed in the form of minimallyfunctional units called packages.
The following FMS have their detection algorithms
encapsulated in ROS packages: AprilTag, ArUco, Alvar9
and ChiliTag10. Actually, ArUco is universal detection
library, that could be used to detect ArUco markers and,in addition, AprilTag, ARTag11,12, ARToolKitPlus13, andARToolKitPlusBCH (Binary Coded Hexadecimal).
Gazebo is 3D-simulator that could be integrated with
ROS as a tool to visualize simulations and apply realworld properties, including light and collision processing
to the objects within the simulation.

Fig. 1. Fiducial marker placement on robot manipulator(left); marker observation by robot (right).

This paper presents virtual experiments design andArUco5 and AprilTag3 systems comparison for rotation
resistance results. Section 2 presents previous research
results. Section 3 briefly describes ROS/Gazebo virtualenvironment used for experiments. Section 4 is dedicatedto experiment setup; Section 5 shows experiment results.
Finally, Section 6 concludes our work.

4.

Experiment Setup

We created two robots in virtual environment: a robotperformer with marker and R2D2-like robot with acamera. The robot-performer is designed to modify amarker appearance between distinct detections: it rotates
the marker for a predefined angle within user-definedangle limits (the scheme is shown in Fig. 2). The robot
with the camera simulates a static camera stand (Fig. 3).
Numerous experiment parameters were kept constantthrough all the experiments (see Table 1) in order toeliminate their influence on comparison results.

2. Related WorkAs it was mentioned, in our previous research6, 7 we had
conducted manual experiments with several fiducial
marker systems. We had chosen AprilTag, ARTag and
CALTag8 for comparison and designed our experiments
to validate their resistance to rotations, systematic
occlusions and arbitrary overlaps. These experiments
were conducted using Genius FaceCam 1000X webcamera first to get data about FMS applicability on lowcost equipment. Then experiments were continued with
AR-601 humanoid robot, using its integrated Basler
acA640-90gc cameras. The experiment results analysis
revealed that AprilTag and ARTag are resistive to marker
rotations, however, are very sensitive to marker edge
overlaps. This could be explained by the detection
algorithm sequence: one of the first steps is edge
detection, and if it fails, the entire detection process stops.
On the opposite, CALTag demonstrated high detection
rate on various rotation angles and different occlusions.

Fig. 2. Virtual experiment rotation scheme.

3. ROS/Gazebo environment
Robot Operating System (ROS) is a fast-growing
framework for robotics development. Its architecture© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan133

Pilot Virtual Experiments on

5. Experimental Results

Table 1. Virtual experiment constant parameters.
Parameter
Camera resolution
Camera distortion level
Camera noise level
Distance
Rotation range (X-axis)
Rotation range (Z-axis)
Marker side size
Light angle of incidence
Light spectrum
Light conditions

Value
640 x 480 px.
0 (ideal lens)
0 (ideal device)
2 m.
[ -180˚; +180˚ )[ -90˚; +90˚ ]
0.4 m
45˚
White light
Uniform at wholemarker area

All markers of a selected type are tested to collectstatistically significant amount of data: 100 ArUco (typeis 25h7) markers and 242 AprilTag (type is 25h7)
markers. Markers of this type have 25 encoding pixels
and Hamming distance between any of them is equal ormore than 7. Equal marker types eliminated difference
between FMS encoding properties and encoded dataamount. Experiments were conducted with each distinct
marker twice to collect reliable data about each detection
angle. Experiment results are presented in Table 2.

Rotation experiments flow works as follows:
1. The robots spawn at constant distance from eachother. Initially, the marker inclination angle
around a (particular) rotation axis is zero.
2. The logger waits for a half of second for themarker detection.
3. The logger logs an inclination angle and a resultof detection procedure.
4. The robot-performer rotates the marker for 1
degree around a user-defined axis (X-axis or Zaxis).
5. If the rotation limit is not reached, the algorithm
goes to step #2; else the experiment ends.
The logger logs all results twice: first output goes to a
console, second goes to a file with proper name including
tag family (e.g., ArUco), tag type (e.g., 25h7), tag ID, thedistance to the camera and the date of the experiment.

Table 2. Average detection rate through all themarkers in rotation experiments by FMS type androtation axis.
Rotation axis
Marker family
and type
AprilTag – 25h7
ArUco – 25h7xz99.94%
99.97%

69.96%
86.07%The experiment results allow concluding that AprilTagfamily markers are practically insensitive to X-axis
rotations; however, ArUco family markers havesignificantly better resistance to Z-axis rotations. In
addition, failed detections distributions for each marker
family are presented in Fig. 4 and Fig. 5.

Fig. 4. Virtual experiment results for X-axis rotation: detection
rates through all markers depending on rotation angle forAprilTag (top) and ArUco (bottom).

Fig. 3. Virtual experiment design: initial marker position (topleft); rotated 30 degrees around X-axis (bottom left); rotated 45
degrees around Z-axis (right).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

134

Aufar Zakiev, Ksenia Shabalina, Evgeni Magid

References
1. R. Kuriya, T. Tsujimura and K. Izumi, Augmented reality
robot navigation using infrared marker, in Proc. 24th IEEE
Int. Symposium on Robot and Human Interactive
Communication, (2015), pp. 450-455.
2. V. Dhiman, J. Ryde and J. J. Corso, Mutual localization:
Two camera relative 6-DOF pose estimation fromreciprocal fiducial observation, in Proc. IEEE/RSJ Int.
Conf. on Intelligent Robots and Systems, (2013), pp. 13471354.
3. E. Olson, AprilTag: A robust and flexible visual fiducial
system, in Proc. IEEE Int. Conf. on Robotics andAutomation, (2011), pp. 3400-3407.
4. M. Fiala, Comparing ARTag and ARToolKit Plus Fiducial
Marker Systems, in Proc. IEEE Int. Workshop on Haptic
Audio Visual Environments and their Applications, (2005),
pp. 148-153.
5. F. J. Romero-Ramirez, R. Muñoz-Salinas and R. MedinaCarnicer, Speeded Up Detection of Squared Fiducial
Markers, in Image and Vision Computing (2018).
6. A. Sagitov, K. Shabalina, R. Lavrenov and E. Magid.
Comparing fiducial marker systems in the presence ofocclusion, in Proc. Int. Conf. on Mechanical, System andControl Engineering, (2017), pp. 377-382.
7. K. Shabalina, A. Sagitov, L. Sabirova, H. Li, and E. Magid.
ARTag, AprilTag and CALTag Fiducial Systems
Comparison in a Presence of Partial Rotation: Manual andAutomated Approaches, Lecture Notes in ElectricalEngineering.
8. B. Atcheson, F. Heide, W. Heidrich. CALTag: High
Precision Fiducial Markers for Camera Calibration, in
Vision, Modeling, and Visualization Workshop, vol.10,(2010), pp. 41-48.
9. C. Woodward, S. Valli, P. Honkamaa and M. Hakkarainen,
Wireless 3D CAD viewing on a PDA device in Proc. 2nd
Asian International Mobile Computing (Langkawi,
Malaysia, 2002), pp. 83-87.
10. Q. Bonnard, S. Lemaignan, G. Zufferey, A. Mazzei, S.
Cuendet, N. Li and P. Dillenbourg, Chilitags: Robust
Fiducial Markers for Augmented Reality (Switzerland,2013).
11. M. Hirzer, Marker Detection for Augmented Reality
Applications, Seminar/Project Image Analysis Graz,(2008), pp. 1-25.
12. M. Fiala, ARTag, a fiducial marker system using digitaltechniques, Computer Vision and Pattern Recognition,
vol.2, (2005), pp. 590-596.
13. D. Wagner and D. Schmalstieg, Artoolkitplus for posetracking on mobile devices (2007), pp. 139-146.

Fig. 5. Virtual experiment results for Z-axis rotation: detection
rates through all markers depending on rotation angle forAprilTag (top) and ArUco (bottom).

Z-axis diagrams show a rather predictive FMS
behavior: increasing rotation angle decreases marker areathat is visible to the camera, leading to unsuccessful
detections. Both FMSs showed highest resistance torotations around X-axis: nearly ideal detection rate has
been logged for all rotation angles.
6. Conclusions and Future Work
This paper presents a set of pilot experiments with
AprilTag and ArUco marker systems in virtualenvironment and their performance comparison for
rotations. The created virtual environment could be used
as a framework for further comparative research because
new FMS addition to the project is easy. In addition, thevirtual environment experiments could be easily
distributed for multiple cores: this gives a chance toqualitatively peer-review and reproduce theseexperiments without limitations. Our future work
concentrates on different FMS comparison for theirresistance to occlusions (both systematic and arbitrary)and exploring dependence of maximum detection
distance on camera resolution.
Acknowledgements
This work was supported by the Russian Foundation for
Basic Research (RFBR), project ID 18-58-45017. Part of
the work was performed according to the Russian
Government Program of Competitive Growth of Kazan
Federal University.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan135

Determining the Key Factors of Michinoeki in Yamaguchi Area
Minoru Kumano

Faculty of Regional Innovation, University of Miyazaki
1-1 Gakuendai-Nishi, Miyazaki, 889-2192 Japan
E-mail: kumano@cc.miyazaki-u.ac.jp
Tsutomu Ito

Hamura Plant, Hino Motors, Ltd.,
5-2-1 Midorigaoka, Hamura City, Tokyo, 205-8601, Japan
E-mail: fw.eldorado.500cuin@gmail.com
Takao Ito

Graduate School of Engineering, Hiroshima University
1-4-1 Kagamiyama, Higashi Hiroshima, 739-7527
E-mail: itotakao@hiroshima-u.ac.jp
Toru Hiraoka

Department of Information Systems, University of Nagasaki
1-1-1, Manabino Nagayo, Nishisonogi, Nagasaki, 815-2195, Japan
E-mail: hiraoka@sun.ac.jp
Hirofumi Nonaka

Department of Information Management System Engineering, Nagaoka University of Technology
1603-1, Kamitomioka, Nagaoka, Niigata, 940-2188, Japan
E-mail: nonaka@kjs.nagaokaut.ac.jp

Abstract
It is considered as the most urgent issues for Michinoeki to find the key determinants and to measure its efficiency.
This paper proposed a new approach with two steps: to calculate the efficiency using DEA model after determining
the relevant factors using regression model. The data are collected from Yamaguchi area, because Yamaguchi is thebirthplace of Michinoeki in Japan. The contribution of this paper are: 1) identified the key factors of Michinoeki in
Yamaguchi area, and 2) calculated the efficiencies of the Michinoeki, and 3) proposed a four cell matrix model tomake judge of the position of Michinoeki for their further development. Based on the findings, the managerial
implications are discussed, the study limitations are identified and directions for further research are suggested.
Keywords: DEA model, Four cell matrix model, Michinoeki, Efficiency.

cooperation function, and disaster prevention function
have been published in the past decades. Based on ourinvestigation, the studies associated with efficiency
calculation to improve their management quality are still

1. Introduction
A plethora of studies of Michinoeki from the viewpointsof break function, information dispatch function, regional© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan136

Minoru Kumano, Tsutomu Ito, Takao Ito, Toru Hiraoka, Hirofumi Nonaka

sparse. This paper proposes a new approach with two
steps: to calculate the efficiency using DEA model after
determining the relevant factors using multivariate
regression model. The data are collected from theMichinoeki in Yamaguchi area, because Yamaguchi areais the exact birthplace of Michinoeki in Japan. The
contribution of this paper are: 1) identified the key factorsof Michinoeki in Yamaguchi area, and 2) calculated theefficiencies of the Michinoeki, and 3) proposed a fourcell matrix model to make judge of the position ofMichinoeki for their further development. Based on thefindings, the managerial implications are discussed, thestudy limitations are identified and directions for future
research are suggested.
This paper is structured as follows: Section 2
introduces the background of this research. In section 3,
the paper explicates data collection and the models ofmultivariate regression model and DEA model to identify
the key factors and efficiency of Michinoeki using
selected variables. Section 4 shows the results anddiscusses our findings. The conclusions and directions
for future research and managerial implications areproffered in the final section.

Michinoeki reached 23 as of July 21, 2017. They are
shown in Table 1.
Table 1 Michinoeki List in Yamaguchi Area.
1
2
3

Abu Town
Hagi Okan
Asahi

13
14
15

Niho no Sato
Hagi Seamart
Ganjoju Onsen

4

Aio

16

Uribo no Sato Katamata

5

Yutori Park Tamagawa

17

Hotaru Kaido Nishi no Ichi

6

Sazanseto Towa

18

Kirara Ajisu

7

Kikugawa

19

Hagi Sansan Sammi

8Happiness Fukue

20

Kitaura Kaido Hohoku

9

Chomonkyo

21

Solene Shunan

10

Pure Line Nishiki

22

Kaminoseki Kaikyo

11

Ofuku

23

Shiosai Ichiba Houfu

12

Mitou

24

Senzakitchen**Semzakitchen has been opened in 2018 fiscal year.

The geographic distribution map of Michinoeki in
Yamaguchi area is shown as in Figure 1.

2. Background
Michinoeki is well-known for its four-function model.
Today Michinoeki plays important roles in regional
economic revitalization in Japan. Recently many studies
of Michinoeki have been published. Regional economic
revitalization is main part of Abenomics [1]. Mitsuhashi
analyzed the economic background of Michinoeki [2].
Ogata et al. explained the relationship between
Michinoeki and regional revitalization based on the
viewpoint of business model [3]. Hiraoka et al. discussed
the relationship between the number of customers andinformation associated with agriculture, forestry, andfisheries of Michinoeki [4]. Furthermore, Kumano et al.
discovered successful determinants of Michinoeki in
Chugoku area using regression model [5]. All of theseresearches show that it is becoming an important issue tocalculate efficiency to improve the quality of Michinoeki.
Thus to find the key determinants and to measure its
efficiency are considered as the most urgent tasks for
Michinoeki. This paper will focus on the discovering thedeterminants, and calculating the efficiency ofMichinoeki in Yamaguchi area. The number ofFig. 1 Geographic distribution map of Michinoeki.

3. Data Collection and Models
The data in 2015 used in this paper are collected from thedatabase of the headquarter of Michinoeki to calculate
the efficiency of Michinoeki in Yamaguchi area. Under
the advices from the experts of Michinoeki, twelve
variables are selected as the determinants of theMichinoeki in this paper. They are: square meters of land
space (SMLS), weekdays’ traffic near the station(WTNS), holidays' traffic near the station (HTNS),
operating cost (OC), total parking number (TPN), total
number of the restroom (TNR), square meters of free rest
place (SMFRP), number of the selling items (NSI),
number of local products items (NLPI), number of the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan137

Determining the Key Factors

registered farmers (NRF), square meters of facilities
space (SMFS) and population of the city located (PCL).
The SMLS and OC could be considered as thedeterminants of variables of information dispatch
function, the WTNS, HTNS, NSI, NLPI, PCL, and NRF
are designed as the indexes of regional cooperation
function, the TPN, TNR, SMFRP and SMFS will play
important role in break function and disaster preventionfunction. Sales revenue and number of purchasers are
selected as the performance of Michinoeki. Thus, thisleads to the formation of the following regression
equation:
= 1
+ 6
+ 11

5
10

+ 2
+ 3
+ 7
+ 8
+ 12
+ꜫ

+ 4
+ 9

PCL

Subject to
∑

∑

∑

=∑

( = 1, 2, … , )≤ 1 ( = 1, 2, … , )≥ 0 ( = 1,2, … , )≥ 0 ( = 1,2, … , )

+
+
(1)

(2)
(3)
(4)
(5)

4. Calculation and Discussions
The relationship between the twelve variables and sales
revenue and the number of purchasers are calculated
respectively. The key factors are finally determined using
backward delamination method. The results are shown asin Table 2.
Table 2 Results of the regression models of sales andnumber of the purchasers.
Sales Revenue
coefficientStandardized

0.0006

0.946100.97644

multiple R2

0.97268

0.0.98815

Adjusted R2

0.90969

0.96604

AIC

445.97207.33

DW Ratio

2.28532.6311

1.2

1.2

1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2
0
2

3

4

6

7

8

10 11 12 13 14 15 16 17 18 19 21 22

Sales Revenue Efficiency

SMLS

-

-

WTNS

-

-

-

-

OC

-

-

-0.34020.0436TPN

0.648600.72710.0004

TNR

-

-

-

-

0.24120.0128

-

-

NSI

-

-

-1.28530.0107

NLPI--

1.43180.0059

NRF

-

-

-

-

SMFS

0.41650.01280.79970

SMFRP

0.0506

Number of Purchaser' Efficiency

Probability

coefficient
-

0

R2

1

Number of the Purchasers

Probability

0.13270.00110ModelsStandardized

0.00020

The determinants of those two models are different.
The adjusted R square of sales revenue and number ofpurchases are 0.9069 and 0.96604 respectively. Fourvariables and six variables are significant statistically.
The four variables in the model of sales revenue are: TPN,SMSRP, SMFS, and PCL, and the six variables ofnumber of purchasers are: OC, NSI, NPLI, SMFS, andPCL. Accordingly, regional cooperation function, breakfunction, and disaster function are becoming much
important rather than information dispatch function
because Yamaguchi is the birthplace of Michinoeki withlong history.
The efficiency of the 23 Michinoeki in Yamaguchi
area are calculated using DEA model based on our results.
The efficiency of Yutori Park Tamagawa, Chomonkyo,
Kitaura Kaido Hohoku, and Shiosai Ichiba Houfu can not
be calculated because of lack of data. It is illustrated as
Fig. 2.

After determining the key factors of the Michinoeki,
the efficiency will be calculated using the following
DEA model [6-7].

max

0.548

Intercept

Fig. 2. Efficiency of the 23 Michinoeki.

-

The correlation coefficient of sales revenue andnumber of purchasers is 0.318, not significant
statistically. Thus, no significant statistically correlation
between those tow results exits. In order to find the future
direction of these Michinoeki, a four cell matrix is
proposed based on the efficiency of sales revenue andnumber of purchasers in this paper. It is reported in Fig.
3.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

138

Minoru Kumano, Tsutomu Ito, Takao Ito, Toru Hiraoka, Hirofumi Nonaka

is required. Furthermore, only one-year Data set is not
sufficient. Much more data set and investigation isrequired.
References
1. Successful examples of regional revitalization, http://www.
kantei.go.jp/jp/singi/sousei/meeting/top_seminar/h29-0113-haifu4.pdf#search=%27%E5%9C%B0%E6%96%B
9%E5%89%B5%E7%94%9F+%E4%BA%8B%E4%BE%8B%27, retrieved April 20, 2018.
2. Masakuni Mitsuhashi, Why is regional revitalizationdifficult? http://news.yahoo.co.jp/feature/499, retrieved
September 20, 2018.
3. J. Ogata, D. Nozokito, H. Murata, and K. Yamamoto,Developing Michinoeki using Michinoeki: an analysis
based business model,
http://www.waseda.jp/seminoue/file/archives/2015_sotsuron_roadsidestation.pdf#se
arch=%27%5B5%5D+%E7%B7%92%E6%96%B9%E7%B4%94%E4%B8%80%E9%83%8E%E3%83%BB%E
8%8E%85%E6%88%B8%E5%A4%A7%E5%9C%B0%
E3%83%BB%E6%9D%91%E7%94%B0%E5%B8%86%E3%83%BB%E5%B1%B1%E6%9C%AC%E5%92%8
C%E5%B8%8C%EF%BC%8C%E3%80%8C%E9%81%
93%E3%81%AE%E9%A7%85%E3%80%8D%E3%81%AB%E3%82%88%E3%82%8B%E5%9C%B0%E5%9
F%9F%E6%B4%BB%E6%80%A7%E5%8C%96%E2%
80%95%E3%83%BC%E3%83%93%E3%82%B8%E3%
83%8D%E3%82%B9%E3%83%A2%E3%83%87%E3%
83%AB%E3%81%AE%E8%A6%B3%E7%82%B9%E3%81%8B%E3%82%89%E3%81%AE%E5%88%86%E6%9E%90%E2%80%95%27, retrieved September 10,
2018.
4. Toru Hiraoka, Shiori Nishimura, Hirofumi Nonaka, Takao
Ito, and Minoru Kumano, A Correlation Analysis between
the Number of Customers and Information Related to theAgriculture, Forestry, and Fisheries of Michi-no-eki inJapan, Journal of the Institute of Industrial Applications
Engineers, Vol.6, No.4, pp.173-178, Oct. 2018.
5. M. Kumano, T. Ito, T. Ito, T. Hiraoka, H. Nonaka, M.
Hirota, Discovering Successful Determinants of
Efficiency of Michinoeki in Chugoku Area, Proceedingsof The 2018 International Conference on Artificial Lifeand Robotics, 352-354, B-Con Plaza, Beppu, Oita, Japan,
February 1-4, 2018.
6. Farrell M. J., The Measurement of Productive Efficiency,
Journal of the Royal Statistical Society, Series A (General)
Vol. 120, No. 3, 253–290, 1957, Royal Statistical Society(Wiley).
7. Charnes A. Cooper W. W. Rhodes E., Measuring theefficiency of decision-making units, European Journal ofOperational Research, 2, 429–444, 1978, North-Holland
Publishing Company.

Fig. 3 The positions of the Michinoeki.

The Michinoeki located in the 4th quadrant could be
excellent because they have high efficiency of sales
revenue and the number of purchases both. The
Michinoeki located in the 2nd quadrant could beconsidered as sales oriented one because they have highefficiency of sales revenue even with low efficiency ofnumber of purchase. The Michinoeki located in the 3rd
quadrant could be poor because there are still rooms to
improve. The Michinoeki located in the 4th quadrant
could be called visitors oriented because the efficiencyof the number of purchasers are lower. Accordingly, Abu
Town, Hagi Seamart, Ganjoju Onsen, Hotaru Kaido
Nishi no Ichi, Kirara Ajisu, Solene Shunan, andSazanseto, Pure Line Nishiki, and Kikugawa areconsidered as excellent Michinoeki while Niho no Sato,
Uribo no Sato Katamata, Hagi Sansan Sammi, andHapiness could be the poor Michinoeki. Ofuku could besales revenue oriented Michinoeki while Hagi Okan,
Asahi, Mitou, and Kaminoseki Kaikyo could be visitors
oriented Michinoeki.
5. Conclusion
The relationship between twelve variables and sales
revenue and number of purchasers are calculated anddiscussed. Only four and six variables are significant
statistically for model of sales revenue and number ofpurchasers, respectively. The 18 Michinoeki could be
divided into four groups, Abu Town and other 8Michinoeki are excellent, and Niho no Sato and other 3
could be poor. Ofuku needs to improve its efficiency ofnumber of purchaser while Hai Okan and other3Michinoeki need to improve their efficiency of sales
revenue. More qualitative support for these conclusions© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

139

Discovering the Characteristics of Michinoeki in Japan
Minoru Kumano

Faculty of Regional Innovation, University of Miyazaki
1-1 Gakuendai-Nishi, Miyazaki, 889-2192 Japan
E-mail: kumano@cc.miyazaki-u.ac.jp
Tsutomu Ito

Hamura Plant, Hino Motors, Ltd.,
5-2-1 Midorigaoka, Hamura City, Tokyo, 205-8601, Japan
E-mail: fw.eldorado.500cuin@gmail.com
Takao Ito

Graduate School of Engineering, Hiroshima University
1-4-1 Kagamiyama, Higashi Hiroshima, 739-7527
E-mail: itotakao@hiroshima-u.ac.jp
Toru Hiraoka

Department of Information Systems, University of Nagasaki
1-1-1, Manabino Nagayo, Nishisonogi, Nagasaki, 815-2195, Japan
E-mail: hiraoka@sun.ac.jp
Hirofumi Nonaka

Department of Information Management System Engineering, Nagaoka University of Technology
1603-1, Kamitomioka, Nagaoka, Niigata, 940-2188, Japan
E-mail: nonaka@kjs.nagaokaut.ac.jp

Abstract
It is a crucial issue to estimate the Michinoeki today. As our best knowledge, no research to estimate the Michinoeki
using the data set for all Michinoeki although 1,107 Michinoeki have been established by April , 2017 in Japan. We
divided Japan into nine areas, and examined the determinants of the nine areas. We found that the key characteristicsof those nine areas are completely different. This paper contributes are: 1) understanding the status quo of Michinoeki,2) finding the potential possibility for their further development. Based on the findings, the managerial implications
are discussed, the study limitations are identified and directions for further research are suggested.
Keywords: Michinoeki, revitalization, sales revenue, number of purchasers.

reducing poverty because they address social issues andbenefit local residents directly” [1]. Today, Michinoeki
developed fast not only in Japan, but also in Kenya andChina. According to our investigation, no research toestimate the characteristics of Michinoeki using data set

1. Introduction
Michinoeki is considered as one of the important models
for revitalization of regional economy. Maryvonne once
indicated that “Michinoeki can be an effective tool for© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan140

Minoru Kumano, Tsutomu Ito, Takao Ito, Toru Hiraoka, Hirofumi Nonaka

of whole Japan although 1,107 Michinoeki have been
established by April, 2017 in Japan. Using data drawn
from 1,107 Michinoeki in Japan, this research attempts
to shed light on discovering the characteristics of theMichinoeki to improve the quality of management.
This paper is structured as follows: Section 2
introduces the background of this research. In section 3,
the paper explicates data collection and multivariate
regression model using specific eighteen variables
selected from the data set of whole Japan. Section 4shows the results and discussions on our findings. The
conclusions and directions for further research andmanagerial implications are proffered in the final section.

analyzed the relationship between the number ofcustomers and information associated with agriculture,
forestry, and fisheries of Michinoeki [4]. Furthermore,
Kumano et al. identified the determinants of Michinoeki
in Kyushu area, Japan [5]. As our best knowledge, noresearch to estimate the characteristics of all Michinoeki
although 1,107 Michinoeki have been established by
April, 2017 in Japan. In order to discover thecharacteristics of Michinoeki, we divided Japan into nine
areas, they are: Hokkaido, Tohoku, Kanto, Hokuriku,
Chubu, Kinki, Chugoku, Shikoku, and Kyushu &Okinawa. Furthermore, eighteen basic variables are
selected from the database as the determinants of theMichinoeki. This paper focused on examining thedeterminants of the nine areas and finding the key
characteristics of those nine areas.

2. Background
Michinoeki are located along major national highways
with four functions of 1) providing free parking space,
restrooms, 2) spreading information, 3) allying with
regional society, and 4) preventing disaster. Accordingly,
the four-function model of Michinoeki could be
illustrated in Fig. 1.

3. Data Collections and Model Building
The data of 2015 are collected from the database of theheadquarter of Michinoeki. In this paper, eighteen
variables are selected as the determinants to express thefour functions in Michinoeki. They are: 1) square metersof land space (SMLS), 2) number of standard-sized car
parking lot(NSSCPL), 3) number of large vehicles(NLV), 4) square meters of parking area (SMPA), 5) total
number of the restroom (TNR), 6) square meters of freerest place (SMFRP), 7) total seats of the free rest place(TSFRP), 8) weekdays' traffic near the station (WTNS),9) holidays' traffic near the station (HTNS), 10)
population of the city located (PCL), 11) operating cost(OC), 12) number of the agriculture and marine products(NAP), 13) number of the local products (NLP), 14)
number of the selling items (NSI), 15) number of original
products (NOP), 16) number of the registered farmers(NRF), 17) square meters of the facilities for marine
products (SMFMP), and 18) square meters of facilities
space (SMFS). SMLS, SMPA, SMFAP, SMFS and OC
could be considered as the determinants of the functionof spreading information and preventing disaster. NSSC,
NLV, TNR, SMFPR, TSFRP, WTNS, and HTNS are
designed as the indexes of providing free parking spaceand restrooms. PCL, NAP, NLP, NSI, NOP, and NRF
will play important role in allying with regional society.
Sales revenue and number of purchasers are selected as
the performance of Michinoeki. Thus, this leads to theformation of the following multivariate regression
equation:Fig. 1 Four-function model of Michinoeki.

A plethora of theories and analyses of Michinoeki
from the four viewpoints have been published in the past
decades. Ogata et al. explained the relationship between
Michinoeki and regional revitalization based on the
viewpoint of business model [2]. Kumano et al.
discovered successful determinants of Michinoeki in
Chugoku area using regression model [3]. Hiraoka et al.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan141

Discovering the Characteristics ofTable 1 Results of standardized coefficients of the eighteen variables and sales revenue.
Hokkaido

1

2

-0.28

0.30

3

4

TohokuKanto

2.07

Hokuriku
Chubu

-0.25

0.20

Kinki

-1.10

0.66

Chugoku

-0.45

0.14

0.17

6

7

8

0.19

0.42

-0.22

0.40

-1.30

-0.19

9

10

0.64

0.30

1.25

1.34

-0.16

0.17

-0.17

-0.64

-1.20

13

14

15
0.23

1.02

0.33
-21.82

1.18

0.57
0.35

1.30

12

-0.66

-1.02

-0.91

11

0.26

0.36

SikokuKyushu

-0.61

5

-0.20

-0.19

0.55

-0.19

0.54

22.33

0.12
0.49

-0.43

0.78

0.10

-0.78

0.83

-0.49

1.21

-0.49

0.58
-0.28

16

17

0.15

0.92

0.32

0.44

18

0.18
0.21

-0.44

0.38

0.12

-0.15

0.52

&
0.21

-0.50

0.55

0.40

-0.22

0.48

0.22

18

Okinawa

Table 2 Results of standardized coefficients of the eighteen variables and number of purchasers.
Hokkaido

1

2

-0.21

0.24

Tohoku

-0.27

Kanto

-1.19

3

4

5

0.30

-0.36

6

-0.24
3.52

Hokuriku

-1.89

-0.49

Kinki

0.53

0.36

-1.45

0.43

Chugoku
-1.08

0.54

9

10

1.44

0.35

0.73

1.26

-1.05

-3.21

3.03

0.87

12

13

14

1.00

15

16

17

0.33

0.20

0.56

0.35

0.42

0.56

-0.15

-0.31
-1.02

11

0.21

-0.64

0.17
1.09
0.16

Sikoku

8

0.47
0.42

-0.37

Chubu

7
0.39

0.27

0.49

0.29
0.83

-0.46

0.41

0.18

-1.06

1.36

0.38
-0.23

-0.32

0.32

0.33
-0.25

0.86

0.09

0.46

0.44

-0.16

Kyushu &
0.30

-0.42

0.42

0.21

0.82

Okinawa

= 1

+ 2

+
+
+
+
+

+ 3

6
9
12
15
18

+ 4
+ 5
+ a7 TSFRP + 8
+ 10
+ 11
+ a13 NLP + 14
+ 16
+ 17
+

provide free space for visitors. Second, the correlation
coefficients of 17) square meters of the facilities formarine products (SMFMP), and 18) square meters offacilities space (SMFS) are all positive. Thus, the scalesof the shop of marine products and the scale of the
facilities space contribute not only the sale revenue, butalso the number of purchasers. Third, 17) square metersof the facilities for marine products (SMFMP) has strong
impact on the performance of Hokkaido, Tohoku, andKyushu & Okinawa. Kyushu & Okinawa are strong
effected by not only 17) square meters of the facilities formarine products (SMFMP), but also 9) holidays' traffic
near the station (HTNS). Thus, it will be considered thatthe performance is effected by SMFMP and HTNS both,differs from Hokkaido and Tohoku. Fourth, the highest
value of Kanto is 2) number of standard-sized car parkinglots(NSSCPL), thus the traffic will be considered as oneof the characteristics of Kanto. Fifth, the key
determinants of Hokuriku, Chubu, Chugoku, and Kinki(1)

4. Calculation and Discussions
The standardized correlation coefficients between theeighteen variables and sales revenue and number ofpurchasers are calculated respectively. The results areshown as in Table 1 and Table 2.
Based upon Table 1 and Table 2, we find following
results. First, the correlation coefficients of 1) square
meters of land space (SMLS) are all negative. The
possible reason is that the scales of all Michinoeki are big
enough because the first purpose of Michinoeki is to© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

142

Minoru Kumano, Tsutomu Ito, Takao Ito, Toru Hiraoka, Hirofumi Nonaka

data set is not sufficient. Much more data set andinvestigation is required.

are 13) number of the local products (NLP), 14) numberof the selling items (NSI), 12) number of the agriculture
and marine products (NAP), and 9) holidays' traffic near
the station (HTNS). Kanto and Shikoku are effected by4) square meters of parking area (SMPA) and 2) numberof standard-sized car parking lot (NSSCPL). Thus
different area has its different specific characteristics.
The key determinants of each area are listed in Table 3.
Table 3 Key determinants of each Michinoeki.
Area
Hokkaido
Tohoku
Kanto
Hokuriku
Chubu
Kinki
Chugoku
SikokuKyushu &Okinawa

References
1. Guidelines for Roadside Stations “Michinoeki”, p.7,
http://documents.worldbank.org/curated/en/75305146813
7999706/pdf/356830Guidelin1de0stations01PUBLIC1.pd
f#search=%27woeld+bank+Michinoeki%27,
retrieved
November 10, 2018.
2. J. Ogata, D. Nozokito, H. Murata, and K. Yamamoto,Developing Michinoeki using Michinoeki: an analysis
based business model, http://www.waseda.jp/seminoue/file/archives/2015_sotsuron_roadsidestation.pdf#se
arch=%27%5B5%5D+%E7%B7%92%E6%96%B9%E7%B4%94%E4%B8%80%E9%83%8E%E3%83%BB%E
8%8E%85%E6%88%B8%E5%A4%A7%E5%9C%B0%
E3%83%BB%E6%9D%91%E7%94%B0%E5%B8%86%E3%83%BB%E5%B1%B1%E6%9C%AC%E5%92%8
C%E5%B8%8C%EF%BC%8C%E3%80%8C%E9%81%
93%E3%81%AE%E9%A7%85%E3%80%8D%E3%81%AB%E3%82%88%E3%82%8B%E5%9C%B0%E5%9
F%9F%E6%B4%BB%E6%80%A7%E5%8C%96%E2%
80%95%E3%83%BC%E3%83%93%E3%82%B8%E3%
83%8D%E3%82%B9%E3%83%A2%E3%83%87%E3%
83%AB%E3%81%AE%E8%A6%B3%E7%82%B9%E3%81%8B%E3%82%89%E3%81%AE%E5%88%86%E6%9E%90%E2%80%95%27, retrieved September 10,2018.
3. M. Kumano, T. Ito, T. Ito, T. Hiraoka, H. Nonaka, M.
Hirota, Discovering Successful Determinants of
Efficiency of Michinoeki in Chugoku Area, Proceedingsof The 2018 International Conference on Artificial Lifeand Robotics, 352-354, B-Con Plaza, Beppu, Oita, Japan,
February 1-4, 2018.
4. Toru Hiraoka, Shiori Nishimura, Hirofumi Nonaka, Takao
Ito, and Minoru Kumano, “A Correlation Analysis
between the Number of Customers and Information
Related to the Agriculture, Forestry, and Fisheries ofMichi-no-eki in Japan”, Journal of the Institute ofIndustrial Applications Engineers, Vol.6, No.4, pp.173178, Oct. 2018.
5. M. Kumano, T. Ito, and T. Ito, A Determinants Analysisof the Michinoeki in Japan, International Journal of
Economics and Management Systems, Vol. 2, pp.287-291,
November, 2017.

Key Determinants
Weekdays' traffic
Square meters of thenear the station
facilities for marine
products (SMFMP)(WTNS)
Square meters of the
Total seats of the
facilities for marine
free rest place
products (SMFMP)(TSFRP)
Number of standard-sized
car parking lot(NSSCPL)
Number of the local
products (NLP)
Number of the selling items(NSI)
Holidays' traffic near thestation (HTNS)Square meters of
Number of the agriculture
facilities spaceand marine products (NAP)(SMFS)
Square meters of parking
area (SMPA)
Number of the agriculture
Holidays' traffic nearand marine products (NAP) the station (HTNS)

5. Conclusions
The relationship between eighteen variables and sales
and number of purchasers are calculated and discussed.
Different area has its different specific characteristics.
For instance, the sale revenue of Hokkaido, Tohoku, andKyushu & Okinawa are all depend on the square meter ofthe facilities for marine products, but the influence of
Tohoku is weak than Hokkaido, Kyushu and Okinawa
are affected by the square meter of the facilities for
agriculture products and holidays traffic near the station
both. However, much more qualitative support for ourconclusion is required. Moreover, only one fiscal year

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan143

An Analysis of Robotic Relationship between Transaction Network and Cross-shareholding
Network in Yokokai
Takao Ito

Graduate School of Engineering, Hiroshima University
1-4-1 Kagamiyama, Higashi Hiroshima, 739-7527
E-mail: itotakao@hiroshima-u.ac.jp
Tsutomu Ito

Hamura Plant, Hino Motors, Ltd.,
5-2-1 Midorigaoka, Hamura City, Tokyo, 205-8601, Japan
E-mail: fw.eldorado.500cuin@gmail.com
Rajiv Mehta

School of Management, New Jersey Institute of Technology, University Heights,
Newark, New Jersey, 07102-1982, U.S.A.
E-mail: kumano@cc.miyazaki-u.ac.jp
Seigo Matsuno

Department of Business and Administration, NIT, Ube College,
2-14-1, Tokiwadai, Ube, Yamaguchi, 755-0096, Japan
E-mail: hiraoka@sun.ac.jp
Makoto Sakamoto and Satoshi Ikeda

Faculty of Engineering, University of Miyazaki,
1-1 Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
E-mail: sakamoto@cc.miyazaki-u.ac.jp,and bisu@cc.miyazaki-u.ac.jp

Abstract
Cross shareholdings is one of the typical characteristics of Keiretsu. It contributes to maintain good transaction
relationships among memberships. However, cross shareholdings as reflected by the percentage and amount of stock
cross ownership is being abandoned. As a result, carmakers are losing control of their parts suppliers. This paper
calculate the relationship between transaction and cross shareholdings, and tested the relationship between inter-firm
relationship and corporate performance. Based on the findings, the managerial implications are discussed, the study
limitations are identified and directions for further research are suggested.
Keywords: inter-firm relationship, transaction, cross shareholding.

being successful, Keiretsu is made-up of affiliatedorganizations that are ordered and characterized by longterm continuous rationality, trade exclusivity, andinformation concealment. After bubble economy,

1. Introduction
Keiretsu was considered as one of the useful models toimprove corporate performance. Widely accepted as© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

144

Takao Ito, Tsutomu Ito, Rajiv Mehta, Seigo Matsuno, Makoto Sakamoto, Satoshi Ikeda.

Keiretsu has been changed from trade exclusive tomarket competition. And cross shareholdings is no moreconsidered as one of the contribution resources oftransaction. Thus, the new changes begs the following
questions: What is the status quo of the inter-firm
relationships between cross shareholdings andtransaction? Does the robotic relationship between
transaction network and cross shareholding network
hold? Robotic relationship is defined if the crossshareholding relationships are automatically affected by
the inter-firm’s transaction relationships. It means thatcross shareholdings relationships continue contribute to
the interfirm transaction relationships. It will provide
managerial implications that can be used as guidelines informulating corporate strategy. Accordingly, this paper
makes a unique contribution to extant thought by: 1)
calculating the degree of all firms in Yokokai, 2)
analyzing the correlation between cross shareholdings
and transaction, 2) Discovering the relationship between
degree and corporate performance in order to shed lighton the robotic relationship between transaction and crossshareholdings.
This paper is structured as follows. Section 2 reviews
the literature focusing on network analysis. In section 3,
the paper explicates the calculation of degree andregression model. Section 4 discusses managerial
implication emerging from our findings. Section 5 will
mention limitations and provide directions for future
research.

this paper will shed light on the correlation between
networks of cross shareholdings and transactions, andcalculate relationship between network indexes andcorporate performance.
3. Data Collection and Model
To calculate the correlation between networks of crossshareholdings and transactions, data for fiscal year 1997
and 2004 were drawn from Mazda’s Yokokai keiretsu.
The detailed information of those two years are shown in
Table 1 and Table 2.
The network relationships among the companies wereidentified through graph modelling. A tie shows thepercentage of the transaction and/or cross shareholdings
between each pairs of firms.
The degree is calculated as follows.
(

)=∑

( ,

= 1, 2, … , ;)= 1,2, … ,

(1)

where
a(pi, pk)=1; if and only if pi and pk are connected by aline
= 0; otherwise
4. Calculation and Discussions
We first calculated the index of degree for all firms in
Yokokai in 1997 and 2004 respectively. And we
calculated correlation matrix between transaction andcross shareholdings. The results are shown in Table 3 andTable 4.
Table 3 and Table 4 show that all correlation
coefficients are significant statistically. Transaction
outdegree is negatively associated with transaction
indegree (-0.4691 in 1997 and -0.5652 in 2004) same as
the relation between transaction outdegree andtransaction indegree (-0.4648 in 1997 and -0.3268 in
2004). It means more selling, less purchasing holds. Bothof the correlation coefficients between transaction
indegree and cross shareholdings outdegree are negativein 1997 and 2004 (0.4013 in 1997 and 0.7368 in 2004).
It means more invest, more selling holds. The capital
relationship contributes its transaction still remains evenin 2004, this contradicted with the traditional viewpoint.
Thus, the robotic relationship between transaction andcapital holds. The correlation coefficients between
transaction outdegree and cross shareholdings indegree is

2. Background
Inter-firm relationship is one of main issues in corporate
management. Many researches have been publishedrecently. Takeshi and Noro published a paper onstrengths and weakness of Keiretsu [1]. Ts. Ito et al.
proposed a new approach called Fragility to calculate theinter-firm relationship in Yokokai [2]. Furthermore, Ta.
Ito et al. did a empirical research on inter-firm capital
relationship in Yokokai using IDE spatial model [3]. All
these researches focused on analyzing inter-firm within asingle network. As a well-known fact, all economic
behaviors are associated with each other. Keiretsu iscomposed of multiple networks, such as transaction
network and cross shareholdings network. All of these
networks are interactively work with each other. Basedon our investigation, the studies associated withinteraction among multiple networks is still sparse. Thus,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan145

An Analysis of Robotic

Table 1 Total number of Banks and Finance Institutions, and Suppliers including carmakers in Yokokai in 1997.
Total Number in Yokokai before Removing Singletons

Total Number in Yokokai after Removing Singletons

Total

Banks and FinanceSuppliers including

Total

Banks and FinanceSuppliers including

numberInstitutions

Car-makers

numberInstitutions

Car-makers

Transaction

201

0

201

122

0

122

Cross shareholding

254

53

201

122

27

95

Table 2 Total number of Banks and Finance Institutions, and Suppliers including carmakers in Yokokai in 2004.
2004

Total Number in Yokokai before Removing Singletons

Total Number in Yokokai after Removing Singletons

Total

Banks and FinanceSuppliers including

Total

Banks and FinanceSuppliers including

numberInstitutions

Car-makers

numberInstitutions

Car-makers

Transaction

188

0

188

91

0

91

Cross shareholding

224

36

188

101

36

65

Table 3 Correlation matrix of out-degrees and in-degrees of Transaction and Cross shareholdings in 1997.
Transaction
Outdegree
Transaction
Outdegree
Transaction
IndegreeCross shareholdings
Outdegree
Cross shareholdings
IndegreeCross shareholdings
IndegreeTransaction Indegree

Cross shareholdings Outdegree-0.4691-0.4648-0.3718000.001310.40130.8561

1-0.46910-0.46480.40130.0005010.294500.0005-0.37180.85610.29450.012

0.0013

0

0.012

1

Table 4 Correlation matrix of out-degrees and in-degrees of Transaction and Cross shareholdings in 2004.
Transaction Outdegree

Transaction Indegree

Cross shareholdings Outdegree

Cross shareholdings Indegree

Transaction
Outdegree

1-0.5652-0.3268

0.447500.00790.0002

Transaction
Indegree-0.565210.7368-0.380200.0018Cross shareholdings
Outdegree-0.3268

0.73681-0.21930.00790

Cross shareholdings
Indegree0.4475-0.3802-0.21930.00020.00180.07920-0.3718 in 1997 and 0.4475 in 2004 respectively. Itmeans “more capital accepted, less selling” in 1997 has
been change to “more capital accepted, more selling ” in2004. Furthermore, the correlation coefficients between
cross shareholdings-outdegree and cross shareholdings
indegree is 0.2945 in 1997 and -0.2193 in 2004

0.0792
1

respectively. It means “more invest, more capital accept”
in 1997 has been change to “more invest, less capital
accept ” in 2004.
Furthermore, we calculated the regression model
between transaction outdegree, transaction indegree,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan146

Takao Ito, Tsutomu Ito, Rajiv Mehta, Seigo Matsuno, Makoto Sakamoto, Satoshi Ikeda.

References

Cross shareholding outdegree, and cross shareholdings
indegree and sales revenue, one of the corporate indexes.
The four variables were all significant in 1997 while only
two variables of Transaction indegree and crossshareholding outdegree are significant statistically. The
adjusted R square of 0.95375 in 2004 is higher than
0.65336 in 1997. The standardized coefficients of areillustrated as in Fig. 2.

1. Akira Takeshi, and Yoshihisa Noro, Keiretsu Divergencein the Japanese Automotive Industry: Why Have Some,
But Not All, Gone? CEAFJP Discussion Paper Series 1704, http://ffj.ehess.fr/upload/Discussion/CEAFJPDP-1704.pdf#search=%27Keiretsu+Divergence+in+the+Japane
se+Automotive+Industry%3A%27 retrieved Oct. 10,2018.
2. Ts. Ito, K. Takahashi, K. Morikawa, Ta. Ito, R. Mehta, S.
Matsuno, and M. Sakamoto, Fragility in Network
Systems: An Empirical Investigation, Journal of Robotics,
Networking and Artificial Life, Vol.4, Issue 1, pp.81-86,
Atlantis Press, July 2017.
3. Ta. Ito, Ts. Ito, K. Takahashi, M. Sakamoto, S. Ikeda, and
R. Mehta: An Empirical Research on Inter-firm Capital
Relationship in Yokokai Using IDE Spatial Model,
Journal of Robotics, Networking and Artificial Life,
Volume 2, Issue 2, pp. 125-131, Atlantis Press, September
2015.

Fig. 2. Standardized Coefficients of outdegree and indegree oftransaction and cross shareholdings.

Fig. 2. provides that transaction indegree in 2004
contributes to its sales revenue positively while it was
negative in 1997. The more capital accept, the more sales
holds. It will be considered all investment are selected toinvest only for the powerful supplier. And consequently
cross shareholdings contributes to sales revenue is muchhigher.
5. Conclusion
In this paper, we calculated 1) degree, one of the network
indexes, 2) correlation coefficients of outdegree andindegree of transaction and cross shareholdings networks,
and verified the relationship between the four variables
and sales revenue using regression model. Based upon
our findings, more invest, more selling holds. Thus works
even in 2004. Accordingly, a certain robotic relationshipbetween transaction and cross shareholdings holds.
However, it is not sufficient to prove the robotic
relationship between transaction and cross shareholding.
Much more data sets are required to shed light on the real
relationship between the transaction and crossshareholdings and discover the mechanism of robotic
behavior in Keiretsu.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

147

Technological Discontinuities and the R&D Strategy of Automobile Companies
Yousin Park

Dept. of Business Administration, Prefectural University of Hiroshima,
1-1-71 Ujina-Higashi, Minami-ku, Hiroshima City, Hiroshima Pref., 734-8558, Japanecventure@pu-hiroshima.ac.jp
Iori Nakaoka

Dept. of Business Administration, National Institute of Technology, Ube College,
2-14-1 Tokiwadai, Ube City, Yamaguchi Pref., 755-8555, Japan
nakaoka@ube-k.ac.jp
Yunju Chen

Faculty of Economics, Shiga University,
1-1-1, Banba, Hikone City, Shiga Pref.,734-8558, Japan
yun-chen@biwako.shiga-u.ac.jp

Abstract
This paper focuses on patterns of technological changes and the R&D strategy of automobile companies. Tushman
and Anderson (1986) pointed out that there are two types of technological changes. One is incremental technologicalchange which is often generated by existing companies, and the other one is radical technological change whichmakes great extension in business environment and destroys the order of existing companies. We use the patent
information of three automobile companies (Toyota, BYD, Tesla) as the cases on these changes. And we examine
our propositions by text analysis and social network analysis. The analyses in this paper include the followings: 1)
To distinguish between the radical and incremental technological changes from R&D projects, and create heat-maps
to visualize these changes, 2) To discuss on technological discontinuities and the R&D strategy of automobile
companies. In this paper we suppose that patterns of corporate R&D strategy cope with the threat of radical
technological changes.
Keywords: technological discontinuities, radical technological change, R&D strategy of automobile companies,
patent analysis,

great extension in business environment and destroys theorder of existing companies.
Nowadays, traditional automobile industry is facedwith new challenges due to radical technological changesuch as the development of electric vehicle. New entrants,
such as Tesla and BYD, are competing with existing
companies in research development. We suppose that

1. Introduction
This paper focuses on patterns of technologicalchanges and the impact of technological breakthroughson environmental conditions. Tushman and Anderson(1986) point out that there are two types of technologicalchanges [1]. One is incremental technological change
which is often generated by existing companies, and theother one is radical technological change which makes© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan148

Yousin Park, Iori Nakaoka, Yunju Chen

2.2 The overview of automobile industry

some features of R&D management help these entrants
to keep their competitive advantage.
In this paper, we use the patent information of threeautomobile companies (Toyota, BYD, Tesla) as the caseson these changes. And we examine our propositions by
text analysis and social network analysis. The analyses inthis paper include the following: 1) To distinguishbetween the radical(product) and incremental(process)technological changes from R&D projects, and create
heat-maps to visualize these changes, 2) To discuss ontechnological discontinuities and the R&D strategy of
automobile companies. In this paper we suppose thatpatterns of corporate R&D strategy cope with the threatof radical technological changes.

Table 1 Market value of world top automobile
companies

Source: Factset (Jun 2017)
Although as a new entrant, Tesla gives confidence andexpectation to investors. Table 1 shows the market valueof world top automobile companies. From this table, we
can see that Tesla in the United States is valued higher in
corporate value than GM in the stock market in 2017. In
addition, China's BYD has attracted attention for the
global production of electric vehicles.
In this paper we discuss the patterns of R&D strategies
of Toyota, Tesla and BYD. Because the advent of electricvehicles in the automobile industry may be productinnovation and is also possible to become technologicalinnovation of competence-destroying discontinuities toexisting firms like Toyota. To test this hypothesis, we
make a comparison among R&D strategies of Toyota,
BYD and Tesla by social network analysis.

2. Background
2.1Technological discontinuities
Abernathy(1978) described that a dynamic model ofinnovation includes a pattern of sequential and crosssectoral change in product innovation, process
innovation, and organizational structure. Firms that are
new to a product area will exhibit a fluid pattern ofinnovation and structure. As the market develops, atransitional pattern will emerge. Finally, the market
stabilizes, fostering a specific pattern of behavior. [1].
Tushman and Anderson(1986) demonstrate thattechnology evolves through periods of incremental
change punctuated by technological breakthroughs
which enhance or destroy the competence of firms in anindustry. These breakthroughs, or technologicaldiscontinuities, significantly increase environmental
uncertainty and munificence. They point out that while
competence-destroying discontinuities are initiated by
new firms and are associated with increased
environmental
turbulence,competence-enhancing
discontinuities are initiated by existing firms and areassociated with decreased environmental turbulence.
Therefore, a radical innovation that can create new
businesses and transform or destroy existing ones [2].
Anderson and Tushman(1990) suggest that the key
punctuation points are technological discontinuities anddominant designs; these delimit eras of ferment and erasof incremental change [3]. Firms must develop diverse
competences both to shape and deal with technologicalevolution.

3. Methodology and data
In the following sections, the trends of R&D strategies
of automobile companies are analyzed. Patent documents
are archived in Patent integration. All patents areclassified according to three kinds of classification codes,
IPC (International Patent Classification), FI (File Index)
and F-term (File Forming Term). FI and F-term areclassification codes for segmentalizing IPC in Japan. IPC
codes in table 2 are the most frequently used in electricvehicle. This paper utilizes all patents (Japan, America,
China, and WIPO) of automobile companies and we pay
special attention to their IPC which is related to electricvehicle described in “IPC B60L”. Patents which are
based on F-term and IPC related to electric vehicle andpublished by each company are extracted and collected© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

149

Technological Discontinuities and the3.2. An approach by the text analysis.

Table 2 IPC of patents related to electric vehicle
IPC codes
Contents

We visualize their R&D strategies associated with coapplicant of patent publication by social network analysis
and calculate network centrality (degree, betweenness,closeness). Fig 2, 3, 4 show the network of patent coapplicant of Toyota, BYD and Tesla in 2016.

PROPULSION OF ELECTRICALLYPROPELLED VEHICLESSupplying electric power to auxiliary
equipmentofelectrically-propelledvehiclesElectrodynamic brake systems for vehicles
in general

B60L
B60L1/003/12
B60L7/0013/00
B60L15/0015/42

Methods, circuits or devices for controlling
the propulsion of electrically-propelledvehicles(Source) World Intellectual Property Organizationfrom the patent information. In order to examine target
companies’ R&D strategies and the change of coreresearchers in their R&D organizations, we visualize
their patent information in 3 steps, and then diagrams canbe created by these analyses.
First, count the number of patent publications related
to electric vehicle of each target company. Second,
conduct social network analysis and calculate network
centrality. Third, create heat maps of core member in
R&D projects.

Fig.2 Network of patent applicant of Toyota

3.1. An approach based on the number of patentpublications
Fig. 1 shows that Toyota is overwhelmingly more thanother two companies in the number of patent applications.
The reason is that the R&D investment amount of Toyota
is as 10 times as the other two.
11,120

1400
1200

8,126

1000

6,868

800
600
400
200

7,424

8,048

10,000
7,570
6,311
6,113
6,035
5,711
5,560

5,660
4,380

1,219
1,174

3,000
2,841

963

858

673
010 016 021 032 272
0 0

7 21 281,847

62 64 61 88 61 33
30541 49 42
156 161 96 92
133 131 238

8,000
6,000
4,000
2,000
0

2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017

0

12,000

10,220
9,398

TESLA

BYD

TOYOTA

Fig. 1 The number of patents applications of the threecompanies
Fig. 3 Network of patent applicant of BYD
Fig. 4 Network of patent applicant of Tesla© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan150

Yousin Park, Iori Nakaoka, Yunju Chen

3.3. An approach by core rigidities.

Fig. 5, 6, 7 illustrate the core rigidities of these threecompanies. The score of core rigidities means that core
members have been continuously active in R&D projects.
In pattern of heat maps, we find that Toyota has exploitedcore member every year and taken an improvement of
R&D capabilities. BYD and Tesla have changed core
member per 2-3 years and taken an exploration of R&D
capabilities.

4.

Conclusions

We discuss that patterns of corporate R&D strategy
cope with the threat of radical technological changes suchas electric vehicle. We find the similarities anddifferences in the R&D strategies of three companies.
Why has each company taken different R&D strategies?
For Toyota, whose core product has been gasoline engine,
has adopted R&D strategy that leveraged this. BYD
whose core competence has been battery technology,
developed FHV of world in 2009. Recently BYD has
focused on the development of EV. Tesla may have beenreorganized R&D strategy by mass production of Model
X since 2014 and development of Model S as a new car.
Therefore, the hypotheses on R&D strategy of eachfirms seem to be related to their business domain and corecompetence. In the case of business domain, we thinkthat the decision of vertical integration or horizontal
specialization affects a firm’s profit and competitive
advantage. The analyses we have made on Chinese ICT
companies, prove our hypotheses [4]. For automobile
companies, we should also pay attention to the decisionof vertical integration or horizontal specialization.

Fig. 5 Core rigidities of Toyota

Fig. 6 Core rigidities of BYD

References
1. W. J. Abernarthy, The productivity dilemma, Baltimore,
MD: Johns Hopkins University Press (1978).
2. M. L. Tushman and P. Anderson, “Technological
Discontinuities and Organizational Environments”
Administrative Science Quarterly, 31, pp. 439-465 (1986).
3. P. Anderson and M. L. Tushman, “Technologicaldiscontinuities and dominant designs: A cyclical model oftechnological change” Administrative Science Quarterly,
35, pp. 604-633 (1990).
4. Y. Chen, Y. Park and I. Nakaoka ”A Comparison Study on
the Vertical Integration and Horizontal Specialization ofChinese ICT Companies” Journal of Robotics,
Networking and Artificial Life, Vol.4, No.4, pp. 317-321(2018)

Fig. 7 Core rigidities of Tesla© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

151

Prototype of an Animated Graphics-Based Training Support Toolfor Bug Fixing of Extended Place/Transition Nets
Tomohiko Takagi

Faculty of Engineering and Design, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
Shogo Morimoto

Graduate School of Engineering, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
Yuki Ue

Graduate School of Engineering, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
Yoshiro Imai

Faculty of Engineering and Design, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
E-mail: takagi@eng.kagawa-u.ac.jp, s17g483@stu.kagawa-u.ac.jp,
s18g459@stu.kagawa-u.ac.jp, imai@eng.kagawa-u.ac.jp

Abstract
This paper describes an animated graphics-based training support method and prototype tool for bug fixing of EPNs
(Extended Place/transition Nets). The prototype tool gives a trainee a faulty EPN and animated graphics. The motionof the animated graphics is synchronized with the motion of the faulty EPN. Therefore, it is expected that a trainee
can intuitively understand the faulty EPN, and smoothly try to fix a bug. The result of bug fixing is checked by the
prototype tool.
Keywords: Training Support, Software Modeling, Place/Transition Net, Bug Fixing

tool to generate boundary value test cases from formal
specifications written in VDM++3.
In our previous study, an EPN (Extended
Place/transition Net) that is a combination of a PN
(Place/transition Net) and VDM++ was proposed andintroduced into test case generation4 and evaluation5. The
PN is a kind of Petri nets that can be used to modelsoftware from the viewpoint of its basic state transition.
The advantage of the EPN for the PN is that somecomplex and essential aspects such as guards and actions
on transitions can be written in VDM++, and a state of

1. Introduction
Formal models in graphical and mathematical languages
can give unambiguous abstracted representation of the
behavior of software. In software development, they areuseful to define specifications, generate program codes,
and generate test cases systematically. For example, inRef. 1, time Petri nets are used to model real-time
software and generate test cases that focus on its statetransition and timing properties. Ref. 2 shows a prototype© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan152

Tomohiko Takagi, Shogo Morimoto, Yuki Ue, Yoshiro Imai

software is closely expressed as a marking and values of
instance variables. The EPN is executable on a tool, andsoftware engineers can confirm the abstracted behaviorof software before beginning its implementation.
However, it would not be so easy for most of software
engineers to learn and use the EPN efficiently. In general,
a formal model with higher representation power requires
software engineers to gain higher degree of expertise andexperience, which will prevent the spread of the formal
model in actual software development. There are only alimited number of software engineers skilled in formal
modeling, and they are too busy to train unskilled
software engineers. Therefore, a training support tool
should be established to solve this problem.
In this study, we are planning to develop a tool tosupport the training of formal modeling using EPNs.
Different support methods should be constructed basedon the difference of trainees' achievement. First, weclassified the trainees' achievement into the following
two levels.
Level 1: A trainee has gained the ability tounderstand the behavior of software from a given
EPN, and fix a bug (that is, a difference between the
EPN and true software requirements).
Level 2: A trainee has gained the ability to construct
an EPN from given software requirements.
As the first step of this study, we have constructed an
animated graphics-based training support method andprototype tool to achieve the level 1. This paper shows
this method, functions of the prototype tool, anddiscussion about its effectiveness. The prototype tool
gives a trainee a faulty EPN and its animated graphics.
The faulty EPN is an EPN that includes an intended bug,and the animated graphics illustrate the behavior ofsoftware based on the faulty EPN. The motion of theanimated graphics is linked to (that is, synchronizedwith) the motion of the faulty EPN. Therefore, it isexpected that a trainee can intuitively understand thefaulty EPN, and smoothly try to fix a bug. The prototype
tool checks the result of bug fixing, and evaluates eachtrainee's achievement.
The rest of this paper is organized as follows. In
section 2, we propose an animated graphics-based
training support method. Section 3 illustrates thefunctions of the prototype tool that the core of the
proposed method has been implemented into. Section 4
gives discussion about effectiveness based on trial

application of the prototype tool, and finally section 5shows conclusion and future work.
2. Training Support Method
In this section, we propose an animated graphics-based
training support method for bug fixing of EPNs. Thismethod consists of (1) creation of exercises, (2) work on
exercises, and (3) evaluation.
2.1. Creation of exercises
A trainer, that is, a skilled software engineer creates
exercises for trainees. Each exercise consists of software
requirements that was written in a natural language, anoriginal EPN that was correctly constructed based on thesoftware requirements, a faulty EPN that was constructedby inserting an intended bug into the original EPN,
operations to fix the intended bug of the faulty EPN, andanimated graphics that illustrate the behavior of software
based on the faulty/fixed EPN.
An intended bug can be created by applying existing
mutation operators to an original EPN. A good exercise
includes an intended bug that cannot be easily found andis likely to be made in actual software development.
The operations for bug fixing are defined as a pair (orpairs) of a kind of existing mutation operators, and thenumber of times to apply the mutation operator to a faulty
EPN. For example, when (arc addition, 2) is defined as
operations, correct bug fixing can be achieved by adding
two arcs somewhere to a faulty EPN. In other words, theintended bug of the faulty EPN would be created by (arcdeletion, 2).
The motion of animated graphics needs to besynchronized with the motion of the faulty/fixed EPN
that is shown as the change of its current state (that is, its
current marking and values of instance variables) and thehighlighting of a recently fired transition. Trigger ofmotion in the animated graphics is the fire of a transitionin the faulty/fixed EPN; on the contrary, trigger of the fireof a transition can be the result of motion in the animatedgraphics. The definition of the trigger in an EPN and its
animated graphics is one of trainer's tasks.
2.2. Work on exercises
A faulty EPN and its animated graphics are given to a
trainee. As mentioned above, the motion of the animatedgraphics is linked to the motion of the faulty/fixed EPN.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

153

Prototype of an Animated

In more detail, the following procedure is automatically
performed based on the trigger definitions.
(i) In the faulty/fixed EPN, each transition is checked
whether it can fire or not. A transition is fireable, if
all from-places of the transition contain the required
number of tokens, and the guard of the transition istrue. If there are no fireable transitions, it will be a
failure that was caused by an intended bug or an
incorrect bug fix, and this procedure is terminated.
(ii) If a trigger event occurs in the animated graphics, atransition relating to the trigger event is identified in
the faulty/fixed EPN. If the transition is fireable,
corresponding motion in both of the faulty/fixed
EPN and the animated graphics is invoked, and thenthis procedure returns to (i). If the transition is not
fireable, it will be a failure that was caused by anintended bug or an incorrect bug fix, and thisprocedure is terminated.
(iii) A fireable transition is randomly selected, and thencorresponding motion in both of the faulty/fixed
EPN and the animated graphics is invoked. This
procedure returns to (i).
If the above-mentioned procedure is terminated, all
the motion is stopped, and then, the faulty/fixed EPN andthe animated graphics return to their initial states.
It is expected that a trainee can intuitively understanda faulty EPN from its animated graphics. If the trainee
has found the difference between the faulty EPN andgiven software requirements, he/she tries to remove an
intended bug from the faulty EPN. The trainee canmodify the faulty EPN by using only the operations forbug fixing that have been given in the exercise.
Modification to the faulty EPN is reflected on the
animated graphics, and thus the trainee can confirm theresult of his/her bug fixing by viewing the synchronous
motion of the fixed EPN and its animated graphics.

Fig. 1. Top page providing the list of exercises.

3. Prototype of a Training Support Tool
The core of the training support method proposed in theprevious section has been implemented as a prototype
tool. This section illustrates its functions.
The prototype tool has been constructed as a Web
application in order that trainees can use it easily. The
aim of this prototyping is to evaluate the effectiveness ofanimated graphics against understandability of EPNs,
and thus it does not include the implementation offunctions to help the creation of exercises. Also, only
model-based mutation operators6 are introduced toprepare faulty EPNs and operations for bug fixing.
First, a trainee needs to log on to the prototype tool
by using a Web browser, and select an arbitrary exercise
on a top page shown in Fig. 1. The top page provides theinformation about each exercise, including its title,
degree of difficulty that was supposed by a trainer, time
taken for the completion of correct bug fixing, andnumber of incorrect bug fixing.
Soon after clicking a title of a desired exercise on the
top page, a trainee receives a corresponding training page.
A training page consists of software requirements,
animated graphics, a faulty EPN, operations for bug
fixing, and some buttons (an "Answer" button, a "Retry"button, buttons for operations for bug fixing, and so on).
When selecting "Crossing Gate Control System"7 fromthe list shown in Fig. 1, the trainee receives the training
page shown in Fig. 2. The animated graphics are formed
from map chips (blocks to form a map, such as rails anda road) and motion objects on the map (such as a train

2.3. Evaluation
When a trainee has finished his/her bug fixing, the fixed
EPN is compared with the original EPN. If they are thesame, it is judged as correct bug fixing. If they are not thesame, it is judged as incorrect bug fixing, and the trainee
is suggested to retry the exercise.
The time taken for the completion of correct bug
fixing, and the number of incorrect bug fixing arerecorded in order to evaluate not only trainees'
achievement but also the quality of exercises.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan154

Tomohiko Takagi, Shogo Morimoto, Yuki Ue, Yoshiro Imai

Fig. 2. Training page (initial state).

Fig. 4. Training page (answering a correct bug fix).

Fig. 3. Training page (modifying a faulty EPN).

Fig. 5. Training page (answering an incorrect bug fix).

and a crossing gate). The operation for bug fixing in thisexercise is (arc deletion, 1), and therefore a "X" button to
apply a mutation operator of arc deletion only once isprovided on the pane that contains the faulty EPN of thecrossing gate control system. The trainee pushes the "X"
button, and clicks an arc of the faulty EPN to delete it. In
Fig. 3, the arc from T5 to P10 has been deleted from thefaulty EPN. The number on a button for operations forbug fixing tells how many times its mutation operator
needs to be applied to a faulty EPN, and it decreases withapplication, as shown in Fig. 2 and Fig. 3. A button for
operations for bug fixing becomes unavailable if its
number reaches 0. Also, a trainee can push the "Retry"button to undo all his/her operations at any time.

Soon after pushing the "Answer" button, the trainee
is notified whether his/her bug fix is correct or not.
Additionally, the training page shows the synchronous
motion of the fixed EPN and its animated graphics. The
motion can be controlled by using fast/normal/slow play
buttons and a stop button. Fig. 4 and Fig. 5 give the screen
shots of the training page on the subject of the crossing
gate control system, just after pushing the "Answer"
button. In Fig. 4, the trainee has successfully fixed theintended bug, and the synchronous motion of the fixed
EPN and its animated graphics shows that the crossing
gate closes with the approach of a train. On the other hand,in Fig. 5, the crossing gate does not close in spite of theapproach of a train, since the trainee has failed in bug© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

155

Prototype of an Animated

5. Conclusion and Future Workfixing. The trainee can undo his/her incorrect bug fixing,
and retry this exercise by pushing the "Retry" button.
Note that a trainee cannot view the synchronous
motion of a faulty/fixed EPN and its animated graphics
before pushing the "Answer" button in the prototype tool,
which will encourage a trainee to carefully view andthink about the EPN at first. When a trainee cannot
understand a given faulty EPN, he/she pushes the"Answer" button without any modification and can view
the synchronous motion.

In this paper, we have proposed an animated graphicsbased training support method for bug fixing of EPNs,
and showed a prototype tool that the core of the proposed
method has been implemented into. As a result of trial
application of the prototype tool, we found that animatedgraphics seem to accelerate trainee's understanding for
EPNs, but the cost to create good exercises is not small.
To improve this problem, we will construct a techniqueand tool to assist the creation of animated graphics andintended bugs. Also, we plan to develop a function tosuggest appropriate exercises automatically for eachtrainee, according to his/her achievement.

4. Discussion
This section gives discussion about effectiveness basedon trial application of the prototype tool.
We created six exercises on the subject of a crossing
gate control system and an elevator control system, andset them to the prototype tool. After that, a graduate
student who is studying a test case generation techniquebased on EPNs in our laboratory worked on the exerciseson the prototype tool. As a result of this trial application,
he succeeded in all bug fixing. His average time to finish
correct bug fixing is about 133 seconds, and his average
number of incorrect bug fixing is three. Finally, we andthe graduate student entered a free discussion, and thefollowing opinions were obtained:
The prototype tool is of high quality, and animatedgraphics seem to accelerate trainee's understanding
for EPNs.
However, some exercises have lower degree ofdifficulty, since same original EPNs and theiranimated graphics are reused to reduce the cost of
creating exercises.
Trainer's effort will be largely devoted to creating
animated graphics, and a technique and tool need to
be developed to support it.
The quality of intended bugs will depend on the
degree of expertise and experience of a trainer.
Heuristic approaches such as a genetic algorithm
will be able to be applied to support/automate thecreation of good intended bugs.
A trainee will not know which exercise to select. It
is better that the prototype tool can suggest
appropriate exercises for each trainee, according tohis/her achievement.
It will be better to allow a trainee to view thesynchronous motion from the first.
Some user interfaces should be improved.

Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP17K00103.
References
1. I. Ho and J. Lin, Generating Test Cases for Real-Time
Software by Time Petri Nets Model, in Proc. of 8th Asian
Test Symposium (ATS) (Nov. 1999), pp.295-300.
2. T. Katayama, H. Tachiyama, Y. Kita, H. Yamaba, K.
Aburada and N. Okazaki, BWDM: Test Cases Automatic
Generation Tool Based on Boundary Value Analysis with
VDM++, Journal of Robotics, Networking and Artificial
Life (JRNAL), Vol.4, No.2 (Sept. 2017), pp.110-113.
3. J. Fitzgerald, P.G. Larsen, P. Mukherjee, N. Plat and M.
Verhoef, Validated Designs for Object-Oriented Systems,(Springer-Verlag London, 2005).
4. T. Takagi and T. Katayama, Negative Test Case
Generation from an Extended Place/Transition Net-Based
Mutants, in Proc. of International Conference on Artificial
Life and Robotics (ICAROB) (Feb. 2018), pp.513-516.
5. T. Takagi, S. Morimoto and T. Katayama, Development ofa Tool for Extended Place/Transition Net-Based Mutation
Testing and Its Application Example, Journal of Robotics,
Networking and Artificial Life (JRNAL), Vol.4, No.2 (Sept.
2017), pp.168-174.
6. T. Takagi, R. Takata, Z. Furukawa, F. Belli and M. Beyazıt,
Metrics for Model-Based Mutation Testing Based on
Place/Transition Nets, in Proc. of Joint Conf. of 21st Int.
Workshop on Software Measurement and 6th Int. Conf. on
Software Process and Product Measurement (IWSMMENSURA) (Nov. 2011), pp.7-10.
7. N.G. Leveson and J.L. Stolzy, Safety Analysis Using PetriNets, IEEE Transactions on Software Engineering, Vol.13,
No.3 (Mar. 1987), pp.386-397.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan156

Prototype of a Modeling Tool to Convert
between Extended Place/Transition Nets and VDM++ Specifications
Tomohiko Takagi

Faculty of Engineering and Design, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, JapanRyo Kurozumi

Faculty of Engineering, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
E-mail: takagi@eng.kagawa-u.ac.jp, s15t225@stu.kagawa-u.ac.jp

Abstract
In software modeling, after converting an EPN (Extended Place/transition Net) to a VDM++ specification, engineers
will repeatedly modify the EPN or the VDM++ specification. However, it is not so easy for engineers to keep theconsistency between them, since it is a time-consuming task. To address this problem, we are developing a prototype
of a modeling tool including the generators between an EPN and a VDM++ specification. This paper shows functionsof the prototype, discussion about effectiveness, and so on.
Keywords: Software Modeling, Place/Transition Net, VDM++ Specification

thus an EPN can be straightforwardly converted to a
VDM++ specification.
Both of EPNs and VDM++ specifications are
abstracted, unambiguous, and executable requirements
specifications, but EPNs provide more abstracted andintuitive view than VDM++ specifications. Both can beused to create test cases1,3, and also can be converted tosource codes. Therefore, they will give software
engineers a practical way for model-driven development.
After converting an EPN to a VDM++ specification,
software engineers will repeatedly modify the EPN or the
VDM++ specification. In order to keep the consistency
between them, the VDM++ specification should be
updated along with the EPN, and vice versa. However, it
is not so easy for software engineers due to their limited
time. A technique and tool are needed to address thisproblem.
This paper shows a prototype of a modeling tool thatconverts and keeps the consistency between EPNs and1. Introduction
Software modeling is the activity to construct arequirements specification that represents the abstracted
structure and/or behavior of software to be developed,
and it plays an important role in the development of highquality software. In software modeling, software
engineers can use EPN (Extended Place/transition Net)1
that is a directed graph-based formal modeling language.
An EPN specification, that is, a requirements
specification that has been drawn based on EPN is also
called an EPN. EPN is useful to define the complex
behavior of software in which some modules work
concurrently. It has higher representation power than
traditional place/transition net, because of theintroduction of guards and actions. A guard and action
can be attached to each transition of an EPN, and they are
written in VDM++2 that is a text-based formal modelinglanguage in VDM (Vienna Development Method), and© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan157

Tomohiko Takagi, Ryo Kurozumi

t1 (nat1 money) [balance+money<=100000] /balance:=balance+money;t3 (nat1 money) [balance−money>=0] /balance:=balance−money;
instruct to withdraw

instruct to deposit

t1
acceptanceof deposit

p2

idle

t3
p3

p1

t2

confirmwithdrawalof deposit

t4

receive money

Fig. 1. EPN of simple account management.
Fig. 2. VDM++ specification based on an EPN.

VDM++ specifications. In section 2, we describe theoverview of software development using EPNs and
VDM++ specifications, and illustrate functions of the
prototype of the modeling tool. Section 3 gives
discussion about its effectiveness, and finally section 4shows conclusion and future work.

places, one token, four transitions, and eight arcs.
The transitions t1 and t3 have guards and actions. For
example, t1 can be fired, if the from-place p1 holds a
token and the guard "balance + money <= 100000"
is satisfied. Also, when it is fired, the action"balance := balance + money;" is executed.
"balance" is an instance variable that can beaccessed from all the guards and actions in the EPN.
"money" is an event parameter of nat1 type that
represents an input from a user, and it can beaccessed from the guard and action of t1 only.
Tools and techniques for EPN are used in thisprocess.
(ii) A VDM++ specification is constructed based on the
EPN in order to refine the requirements specification.
The guards and actions in the EPN have been written
in VDM++, and thus the EPN is straightforwardly
converted to a VDM++ specification. Fig. 2 shows
an example of a VDM++ specification that isconverted from Fig. 1. The refinement is performed
based on the converted VDM++ specification, andits results are reflected to the EPN. Software
engineers can return to the process (i) at any time.
Tools and techniques for VDM++ are used in thisprocess.
(iii) The implementation is performed based on the EPN
or VDM++ specification, that is, source codes are
written based on skeleton codes that are converted
from the EPN or VDM++ specification. Software
engineers can return to the processes (i) and (ii) at
any time, if they find any problems. Tools andtechniques for EPN or VDM++ are used to generate

2. Conversion between EPNs and VDM++
Specifications
In this section, we describe the overview of softwaredevelopment using EPNs and VDM++ specifications,and clarify the scope of this study. After that, we illustrate
functions of the prototype of the modeling tool.
2.1. Overview of software development using
EPNs and VDM++ specifications
The basic processes of software development we supposein this study are as follows.
(i) An EPN is constructed based on software
requirements. It consists of six kinds of element, thatis, places (representing the state of each module),
tokens (representing the current state of each
module), transitions (representing the state transition
that is triggered by an event), arcs (representing thechange of tokens by the firing of a transition), guards(representing the additional condition for a transition
to be fireable), and actions (representing the dataprocessing in a transition). A state of software isexpressed as a marking and values of instancevariables.
An example of an EPN that represents simple
account management is given in Fig. 1. It has three© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

158

Prototype of a Modelingthe skeleton codes, and common programming
environments are used to complete theimplementation.
(iv) Software testing is performed based on the EPN,
VDM++ specification and others. Software
engineers can return to the processes (i), (ii) and (iii)
at any time, if they find any problems. Tools andtechniques for EPN, VDM++ and others are used inthis process.
The scope of this study is the processes (i) and (ii).
2.2. Functions of the prototype of the modelingtool
We are developing a prototype of a modeling tool tosupport the processes (i) and (ii) that have been discussed
in section 2.1. The screen shot of the prototype is shownin Fig. 3. It consists of the following functions (including
functions under construction).

errors. The user can keep his/her VDM++ specification
on file.

Editor of an EPN

Generator from a VDM++ specification to an EPN

The EPN editor with GUI (Graphical User Interface)allows a user (that is, a software engineer) to intuitively
construct an EPN. The user puts places, tokens,transitions, and arcs on the drawing pane shown as (A) in
Fig. 3, and then edits their attributes including guards andactions on the attribute pane shown as (B) in Fig. 3. The
editor can highlight the parts that will include structure
errors, such as isolated places and transitions. The user
can keep his/her EPN on file.

An EPN can be generated from a VDM++ specificationthat has been constructed on the VDM++ specification
editor. The generator is invoked from the GUI of the EPN
editor, and it parses the VDM++ specification. Note thatthe generator can accept only a VDM++ specification
based on the coding pattern shown in Fig. 2. If thegenerator finds any problems on the VDM++
specification, it stops the generation and gives error
messages to the user. An EPN generated is sent to andshown in the EPN editor, and it can be modified on theeditor.

Fig. 3. Prototype of the modeling tool.

Generator from an EPN to a VDM++ specificationA VDM++ specification can be generated from an EPN
that has been constructed on the EPN editor. The
generator is invoked from the GUI of the EPN editor. The
generation is executed based on the coding pattern shownin Fig. 2. If the generator finds any problems on the EPN,
it stops the generation and gives error messages to theuser. A VDM++ specification generated is sent to andshown in the VDM++ specification editor discussed
below.

3. Discussion
This section gives discussion about the effectiveness ofthe prototype.
The key functions to achieve the aim of this study are
the generators between an EPN and a VDM++
specification. When a user has modified an EPN, he/she
can immediately update a VDM++ specification by using
the generator from an EPN to a VDM++ specification.
Also, when a user has modified a VDM++ specification,
he/she can immediately update an EPN by using thegenerator from a VDM++ specification to an EPN.
However, both of the generators are invoked by manual
operation. It is better that they are automatically invoked
in order to reduce software engineers' effort and surely
keep the consistency.

Editor of a VDM++ specification
The VDM++ specification editor with GUI allows a user
to refine a VDM++ specification given from the abovementioned generator. The editor can highlight somekeywords of VDM++ and parts that will include syntax© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

159

Tomohiko Takagi, Ryo Kurozumi

Mutants, in Proc. of International Conference on Artificial
Life and Robotics (ICAROB) (Feb. 2018), pp.513-516.
2. J. Fitzgerald, P.G. Larsen, P. Mukherjee, N. Plat and M.
Verhoef, Validated Designs for Object-Oriented Systems,(Springer-Verlag London, 2005).
3. T. Katayama, H. Tachiyama, Y. Kita, H. Yamaba, K.
Aburada and N. Okazaki, BWDM: Test Cases Automatic
Generation Tool Based on Boundary Value Analysis with
VDM++, Journal of Robotics, Networking and Artificial
Life (JRNAL), Vol.4, No.2 (Sept. 2017), pp.110-113.

The prototype does not provide a function to execute
an EPN and VDM++ specification on its GUI. As alreadymentioned, the advantage of EPNs and VDM++
specifications is to be executable, and EPNs provide
more abstracted and intuitive view than VDM++
specifications. Therefore, a function to execute an EPN
on the EPN editor should be developed so that a user cancheck a requirements specification effectively. It will be
useful also to support model-based testing in a software
testing process.
The quality of a requirements specification affects thequality of software as a final product. The requirements
specification in this study is written in EPN and VDM++,and thus techniques and tools for EPN and VDM++ need
to be introduced into the prototype in order to effectively
improve the quality of the requirements specificationfrom the viewpoint of both EPN and VDM++.
4. Conclusion and Future WorkIn this paper, we have discussed a prototype of amodeling tool that converts and keeps the consistency
between EPNs and VDM++ specifications. The
prototype has four functions, that is, (a) the editor of an
EPN, (b) the generator from an EPN to a VDM++
specification, (c) the editor of a VDM++ specification,and (d) the generator from a VDM++ specification to an
EPN. The key functions to achieve the aim of this study
are (b) and (d). When a user has modified an EPN, he/she
can immediately update a VDM++ specification by using(b). Also, when a user has modified a VDM++
specification, he/she can immediately update an EPN byusing (d).
In our future study, the prototype will be improved sothat (b) and (d) can be automatically invoked in order toreduce software engineers' effort and surely keep theconsistency. Also, we plan to develop a function to
execute an EPN on the EPN editor, which will be usefulto check a requirements specification.
Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP17K00103.
References
1. T. Takagi and T. Katayama, Negative Test Case
Generation from an Extended Place/Transition Net-Based© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan160

Application of Pairwise Testing to Test Cases by Boundary Value Analysis in BWDM
Futa Hirakoba*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, and Naonobu Okazaki**University of Miyazaki, Japan, †Tokyo University of Technology, Japan
hirakoba@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kitayshr@stf.teu.ac.jp,
yamaba@cs.miyazaki-u.ac.jp, aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp

Abstract
BWDM is a test case generation tool for the VDM++ specification. The existing BWDM could cause a combinatorial
explosion of the generated test cases. To reduce the number of test cases, there is PICT: a pairwise testing tool.
However, BWDM cannot call PICT library. Hence, we have developed PICT-wrapper. It is an interface to connect
PICT and BWDM. We extend BWDM in that PICT-wrapper is embedded. The extended BWDM eliminate thepossibility of the combinatorial explosion.
Keywords: software testing, boundary value analysis, pairwise testing, formal methods, VDM++, PICT

existing BWDM, this research extends BWDM so thatthe pairwise testing can be applied to boundary value
analysis in BWDM. In applying the pairwise testing, we
use PICT (Pairwise Independent Combinatorial TestingTool) developed by Microsoft Corporation.5 We havedeveloped PICT-wrapper. It is an interface to connect
PICT and BWDM. And, we have extended BWDM inthat PICT-wrapper is embedded.

1. Introduction
As the importance of software in society is increasing,specifications using specification languages become
more important.1 It is necessary to test the developed
software, but it takes much time and effort to design test
cases manually. So, we developed BWDM (Boundary
Value Vienna Develop Method).2, 3
In BWDM, test cases are generated by boundary
value analysis and symbolic execution from VDM++
specification. However, it is possible to cause acombinatorial explosion by the number of the test cases
because the existing BWDM generate by all
combinations when boundary value analysis. There is apairwise testing4 as an effective testing method to reduce
the total number of the combinations. According to Ref.
4, there are very few defects occurring in three or morefactors. Therefore, combination testing is effective fortwo factors. (To find the defects occurring in three ormore factors, you should test with other techniques.) And,to test combinations of only two factors is called pairwise
testing. In order to solve the above problem in the

2. The existing BWDM
BWDM has the following features.
Test cases generation by symbolic execution.2
Test cases generation by boundary value analysis.3
In symbolic execution, it is expected that the test cases
generated by symbolic execution can cover the all
execution flow. Test cases generated by boundary value
analysis can be used for boundary testing. By using
BWDM, efficiency of software development for VDM++
specification can be improved.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan161

Futa Hirakoba, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, and Naonobu Okazaki

Fig. 1. The flow of the extended BWDM.

PictAddParameter - Registering factors and levels to
PICT
PictGenerate - Generating of combination data
PictGetNextResultRow - Getting generated data

However, there is a problem with BWDM. In
generating test cases by boundary value analysis, thenumber of the generated test cases is determined bymultiplying values of each that the factors can take. For
example, in the case of factors (6, 6, 2, 4, 5, 7), BWDM
generates 10,080 test cases. That is, a combination
explosion may occur. In this paper, we extend BWDM tosolve this problem.

3.2. Application of pairwise testingFig. 1 shows the flow of the extended BWDM. In theboundary value analysis unit, it extracts boundary valuesof minimum and maximum values of types for eachargument accordance with conditional expression in thefunction of VDM++ specification. In the existing
BWDM, all combinations of boundary values generated
for each argument are generated as input data inboundary value testing.
In the extended BWDM, input data are generated byapplying the pairwise testing using the PICT-wrapper
described in Section 3.1. Specifically, it inputs
parameters that can be taken by factor obtained byboundary value generation into the PICT-wrapper. In the
pairwise testing, by focusing on a pair of two factors, acombination testing of all pairs is executed. Here, in the
PICT-wrapper by changing its setting, it is also possible
to make N pairs of factors.
The input data generation algorithm of the PICTwrapper that received boundary value data is shown
below. The flow of the PICT-wrapper using thisalgorithm is shown in Fig. 2. PICT-wrapper generates

3. The extended BWDM
We have developed PICT-wrapper. It is an interface toconnect PICT and BWDM. And, we have extended
BWDM in that PICT-wrapper is embedded. The details
are described below.
3.1. PICT-wrapper
Although PICT is a CLI tool, it also provides an API(PICT library), which can be used from C++ programs.
However, since the existing BWDM is written in Java, itcannot call the PICT library. Therefore, as a preparationfor extension of BWDM, we develop another PICT
library that can be called from Java programs using JNA(Java Native Access).6 We call it PICT-wrapper. The
functions mainly used in the PICT-wrapper are shown
below.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

162

Application of Pairwise TestingFig. 2. The flow of PICT-wrapper.
Table 1. Application result.
Number
1
2
3
~
38
39
40

Input
100
100
2,147,483,647~
2,147,483,647
2,147,483,647
2,147,483,647

-1
2,018
0
~
-1
-1
2,018

Expected output
11
Undefined Action
12“a is 100 or more and c is 12 or more”
4,294,967,295
Undefined Action~~-1
Undefined Action
4,294,967,294
Undefined Action
4,294,967,295
Undefined Action

test data, using factors and values of each factor, which
are given by boundary value analysis result, as arguments.
i.
ii.
iii.

iv.

specification, the factor is 3, the level is a function of (6,6, 6).
In the existing BWDM, 216 test cases weregenerated. In the extended BWDM, 40 test cases are
generated test cases. The number of the generated test
cases can be reduced. In addition, it was confirmed that40 test cases can cover all combinations of pairs of two
factors.

By using PictAddParameter, register factors andlevels in PICT.
By using PictGenerate, generate combination
data to which the pairwise testing was applied.
By using PictGetNextResultRow, acquire one
combination data generated by step ii. The
acquired data is an index of parameters that canbe taken by the factors.
Generate output data of PICT-wrapper usingparameters corresponding to the index of dataacquired by step iii.

5.

Evaluation

We confirm that the extended BWDM can reduce thenumber of test cases compared to existing BWDM. In
experiment of the comparison, we input function offactor 7 and level (6, 8, 6, 8, 8, 6, 6) into BWDM. A
comparison of generation results is shown in Table 2. The
execution environment is macOS 10.13.6 (CPU: Intel

4. Application example
The results of applying a VDM++ specification to theextended BWDM is shown in Table 1. In the VDM++© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan163

Futa Hirakoba, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, and Naonobu Okazaki

Table 2. Comparison of the generation result.
Number of generated test cases

Execution time(sec)663,55278

6.767146152
0.88973152

The existing BWDM
The extended BWDM

Core i5 2.3 GHz, RAM: 16 GB). The formula used forcomparison is shown below.
−(%) =× 100(1)possibility to cause the combination explosion of the test
cases generated from the boundary value analysis result.
Therefore, the extended BWDM is more highly practical.
Also, the extended BWDM is expected to improve theefficiency of the test process because the test cases for afunction including many factors and many levels havebecome a practical number.
The future issues are as follows.
Expanding an applicable range of BWDM.
Corresponding to a constraint of the specificcombination.
Corresponding to boundary value analysis ofconditional expression including two or morevariables.

where
A is Total test cases generated by the existing BWDM.
B is Total test cases generated by the extended BWDM.
From Table 2 and Eq. (1) the number of generated test
cases could be reduced by (663,552 - 78) / 663,552 × 100= 99.98(%). The existing BWDM generated a huge
number of test cases. But, the extended BWDM
generated a practical number of test cases. Therefore, theextended BWDM can eliminate possible to cause acombination explosion of test cases. Furthermore, test
case generation time could be shortened (see Table 2). As
a result, the extended BWDM is more highly practical.
However, the extended BWDM cannot use a part offeatures of PICT library. For example, the extended
BWDM cannot set a constraint of a specific combination.
The restriction of the specific combination means a
combination which must be tested or a combination
which does not need to be tested. In order to conduct
flexible testing, it is necessary to deal with it in future.
Also, since BWDM still has few VDM++ syntax that canbe applicable to it, it is necessary to expend its applicablerange, In the case of A boundary value analysis for theextended BWDM, it can't be analyze conditional
expression that contains two or more variables. By
solving these problems, BWDM will become morepractical.

References
1. Anne E. Haxthausen, An introduction to formal methods
for the development of safety-critical application, DTU
Informatics Technical University of Denmark, pp.1-32(2010).
2. Hiroki Tachiyama and Tetsuro Katayama, Structure
Recognition Method of If-Then-Else Expression in BWDM
using VDM++ Specification, IPSJ/SIGSE Software
Engineering Symposium, pp.130-137 (2017) (in Japanese).
3. Hiroki Tachiyama, Tetsuro Katayama, et al., Prototype of
Test Cases Automatic Generation Tool BWDM Based on
Boundary Value Analysis with VDM++, International
Conference on Artificial Life and Robotics, pp.275-278(2017).
4. D. Richard Kuhn, Dolores R. Wallace and Albert M. Gallo,
Software fault interactions and implications for software
testing, IEEE Transactions on Software Engineering, Vol.
30, pp.418-421 (2004).
5. Microsoft Corp., Microsoft/pict Pairwise Independent
Combinatorial Testing, https://github.com/Microsoft/pict(last accessed on 2018/11/07)
6. java-native-access/jna
Java
NativeAccess,https://github.com/java-native-access/jna (last accessed on
2018/11/07)

6. Conclusion
In this paper, we have extended BWDM. The purpose isto eliminate the possibility of the combinatorial
explosion occurring in the number of test cases generated
from boundary value analysis results. The extended
BWDM performs boundary value analysis on the
VDM++ specification, applies a pairwise testing, andautomatically generates test cases. As an evaluation
result, we confirmed the extended BWDM can eliminate© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan164

Development of Library Fescue Extracting Elements of Attributes and Operationsof Class Diagram in UML
Keisuke Mori*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, and Naonobu Okazaki**University of Miyazaki, Japan, †Tokyo University of Technology, Japan
hirakoba@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kitayshr@stf.teu.ac.jp,
yamaba@cs.miyazaki-u.ac.jp, aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp

Abstract
No existing UML tools can describe all the grammar defined in UML 2.0 specification. This paper develops library
fescue (Feature Elements Section of Class in UML Extraction) extracting elements of attributes and operations ofClass Diagram in UML to expand the analyzable range of the grammar defined in UML 2.0 specification. It is
confirmed that fescue can parse the components of attributes and operations that the existing tools cannot parse.
Keywords: UML, Class Diagram, Attribute, Operation, Syntax Analysis

Figure 1 shows the BNF (Backus Naur Form) of theattribute and the operation of the Class Diagram definedin the Specification. The attribute is defined as anonterminal symbol <attribute>, and the operation isdefined as a nonterminal symbol <operation>. The
attribute and the operation of Class Diagram are definedcomprehensively in the Specification, but someundefined items remain.
13 nonterminal symbols undefined in the nonterminal
symbols in Fig. 1 are as follows:nameparameter-nameattr-typetype-expression
return-type
attr-constraint
oper-constraint
value-specification
default
integer
attribute-nameoper-nameparam-property

1. Introduction
In object-oriented software development, UML1 (Unified
Modeling Language) is widely used as a standard model
notation for problem analysis and specification
description. Therefore, many modeling tools anddevelopment support environments for UML have beendeveloped, and some modeling tools have the function toconvert the described model into programs. However, no
existing tools can describe all the grammar defined in
UML 2.0 specification2 (hereinafter called theSpecification).
This paper develops library fescue (Feature Elements
Section of Class in UML Extraction) extracting elements
of attributes and operations of Class Diagram in UML for
expanding the analyzable range of the grammar definedin the Specification. The two functions to realize thispurpose are needed as follows:
Parsing grammar defined in the Specification
Extracting each element from parsing result
2. The Attribute and The Operation in Class
Diagram© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

165

Keisuke Mori, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

Fig. 2. Root nodes of defined grammar in ClassFeature.g4.

and spaces (e.g. space character and tab character) that
may be accepted as other nonterminal symbols.
3. The Grammar File
This paper describes the grammar file ClassFeature.g4
needed for ANTLR3 used in Section 4 based on thegrammar defined in Section 2. The described grammar
file is 320 lines. Figure 2 shows the root nodes of thedefined grammar in the grammar file. Also, Table 1
shows the relationship of nonterminal symbols in Fig. 1
and nonterminal symbols in Fig. 2.
From Table 1, it can be seen that all items of terminal
or nonterminal symbols of the attribute and the operationin Fig. 1 are defined by the respective nonterminal
symbols in Fig. 2.
4.

Fig. 1. The BNF of the attribute and the operation of Class
Diagram defined in the Specification.

FescueThis paper develops library fescue4 (Feature Elements
Section of Class in UML Extraction), which is an element
extraction of the attribute and the operation. Fescue isbased on the parser generated by ANTLR from thegrammar file described in Section 3. Fescue is written in
Java programming language.
Fescue instantiates each nonterminal symbol in thegrammar file described in Section 3 when fescue isentered and scans a character string representing an
attribute or an operation. The class structure in fescue isa tree structure because it is modeled on the structure of
the syntax tree generated by the parser. An instancecorresponding to a higher nonterminal symbol has afunction of extracting each instance corresponding to alower nonterminal symbol. Basically, fescue isdeveloped as the Java class file structure based on thenonterminal symbols defined in Section 2 and Section 3.
Here, due to restrictions of the programming language,
some structures are partially changed.

This paper defines all undefined nonterminal
symbols in the Specification to exactly accept theSpecification. In this paper, explanation about <name>and <parameter-name>, and definition of the grammar
are as follows.
In the Specification, <name> is defined as the nameof the attribute and the operation, and <parameter-name>is defined as the name of the parameter in the operation.
The name in the Specification is defined as the name of
the String type had by the NamedElement class, which is
the generalization relation between the Attribute classhaving <name> and the Parameter class having<parameter-name>. String type is one of the primitive
type in the Specification, and is not based on a specificcharacter and a character string length. In term of syntax
analysis, this paper defines <name> and <parametername> as an arbitrary character string excluding
characters such as symbols (e.g. colon and parenthesis)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

166

Development of Library Fescue

Table 1. Relationship of nonterminal symbols in Fig. 1 and nonterminal symbols in Fig. 2.
Nonterminal Symbol in Fig. 2

Nonterminal Symbol in Fig. 1

attribute<attribute>

operation

visibility
dividednameattrType
multiplicityRange
defaultValue
attrModifiers
visibility
nameparameterList
returnType
operProperty<operation><visibility>‘/’<name>‘:’ <attr-type>‘{‘ <multiplicity> ‘}’‘=’ <default>‘{‘ <attr-modifier> [‘,’ <attr-modifier>]* ‘}’<visibility><name>‘(‘ <parameter-list> ‘)’‘:’ [<return-type>]‘{‘ <oper-property> [‘,’ <oper-property>]* ‘}’

Fig. 3. Example: a statement of attribute of one line.

Fig. 4. Example: a statement of operation of one line.

Fescue is executed by the test code written in JUnit55
and JaCoCo6, and C1 coverage is measured in fescue. In
all classes generated by fescue, C1 coverage of class files
other than the syntax analyzer files automaticallygenerated by ANTLR is 100%. The number of items
being testing with the fescue library is 4000 or more. In
addition, automatic build and automatic test using Travis
CI7 are carried out, and now all test cases are passed. As
a result, it is confirmed that fescue can extract all terminal
symbols in the syntax analysis result (see Subsection 5.1).

Fig. 5. Syntax tree of the statement in Fig. 3.

For example, the attribute in Fig. 2 has sevennonterminal symbols, but Attribute class in fescue has arelationship with the next six classifiers.
Visibility enumeration
Type className classMultiplicityRange classDefaultValue class
Property interface
There is not a class corresponding to the nonterminal
symbol “divided”. This is the reason for a structure that
Attribute class has as a boolean “isDivided” field.
Furthermore, Attribute class has List type Property
interface as nonterminal symbol “attrModifiers”.

5. Evaluation
This paper evaluates the grammar file in Section 3, anddescribes related research. Hereinafter, a character string
representing an attribute is called an attribute sentence,
and a character string representing an operation is called
an operation sentence.
5.1. Evaluation of grammar fileAs an application example, an attribute sentence and an
operation sentence are entered to a grammar test which isa function of ANTLR, and its syntax tree is generated.
Figure 3 shows the applied attribute sentence, and Figure© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

167

Keisuke Mori, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

Fig. 6. Syntax tree of the statement in Fig. 4.

expanding the analyzable range of the grammar definedin the UML 2.0 specification. This paper has described
ANTLR grammar file which is based on the BNF of theattribute and the operation in the UML 2.0 specification.
It is confirmed that fescue can extract all terminal
symbols of attributes and operations which cannot extractby the existing UML tools.
Therefore, fescue developed in this paper can extract
all terminal symbols. If fescue is incorporated into theexisting UML tools, the incorporated tools are expected
to expand grammar which can analyze. As a result,fescue is useful because the incorporated tools are usefulfor converting the described model to programming
language.
Future works are as follows:
Application to definition grammar with UML 2.5.1
Acceptance of OCL grammar in prop-modifier of
attribute, oper-property, and param-property ofoperation
References

4 shows the applied operation sentence. In addition,
Figure 5 shows the syntax tree of the statement in Fig. 3,
and Figure 6 shows the syntax tree of the statement in Fig.
4.
Even if any existing tool is entered the character
strings in Fig. 3 and Fig. 4, it does not output correctparsing results (see Subsection 5.3). Looking at eachnode of depth one from the root nodes in Fig. 5 and Fig.
6, it is confirmed that each node corresponds with eachnonterminal symbol in Fig. 2. Similarly, it is also
confirmed that each node deeper than depth one
corresponds with each terminal or nonterminal symbol.
Therefore, it is confirmed that the correct syntax tree canbe generated for all terminal symbols in Fig. 5 and Fig. 6.
As a result, the grammar file in the paper is useful
because it can analyze attribute sentences and operation
sentences which cannot be analyzed by the existing tools.
5.2. Related research
BlueJ8 is an IDE (Integrated Development Environment)
for supporting to learn Java. It can execute coding Javaand editing Class Diagram simultaneously.
When comparing BlueJ and fescue, BlueJ cannot edit
attributes and operations with the character string definedin the Specification. BlueJ is an IDE targeting Java, andif you want to edit the attributes and operations of theClass Diagram, you must edit the Java code. Fescue canparse attribute sentences and operation sentences. Hence,
you can edit them in the Class Diagram.

1. Welcome To UML Web Site!, http://www.uml.org/(accessed 2018-11-12).
2. Object Management Group, The Unified Modeling
Language 2.0 Specification, Ohmusha, 2006 (in Japanese).
3. Terence Parr, The Definitive ANTLR4 Reference, The
Pragmatic Bookshelf, 2012.
4. fescue, https://github.com/morichan/fescue/ (accessed
2018-11-12).
5. JUnit 5, https://junit.org/junit5/ (accessed 2018-11-12).
6. EclEmma – JaCoCo Java Code Coverage Library,https://www.jacoco.org/jacoco/ (accessed 2018-11-12).
7. Travis CI – Test and Deploy Your Code with Confidence,https://travis-ci.org/ (accessed 2018-11-12).
8. Michael Kolling and Bruce Quig, The BlueJ system andits pedagogy, in Computer Science Education, Vol. 13, No.
4, pp. 208–213, (2014).

6. Conclusion
This paper has developed library fescue, which is anelement extraction of attributes and operations due to© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

168

Tamias: a Syntax File Checker for Parsing Expression Grammar
Toshihiro Miyaji*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, and Naonobu Okazaki**

University of Miyazaki, 1-1 Gakuen-kibanadai nishi, Miyazaki, 889-2192 JapanTokyo University of Technology, 1404-1 Katakuramachi, Hachioji City, Tokyo 192-0982, Japan
E-mail: miyaji@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kitayshr@stf.teu.ac.jp,
yamaba@cs.miyazaki-u.ac.jp, aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp†

Abstract
Parsing Expression Grammar (PEG) proposed by Bryan Ford has the higher expressive ability than traditional BNF,
but it also has problems such as prefix capture. “Prefix capture” is a problem of hiding the language to be accepted
according to the order of choice. To support checking syntax files including such mistakes, this paper proposes
Tamias: a syntax file checker to support checking the PEG syntax files. Tamias has PEG interpreter which can checkproduction rules of PEG. It can verify the behavior of production rules and measure the reach rate of choices.
Keywords: Syntax Analysis, Parser, Parsing Expression Grammar, Packrat Parsing

parser, it is possible to check only the top level nonterminal symbols, and it is necessary to rebuild the parser
for each change in the syntax file. To support checking
syntax files, this paper proposes Tamias: a syntax file
checker to support checking the PEG syntax files.

1. Introduction
BNF (Backus-Naur Form) can express type-2 grammar(CFG: Context-Free Grammar) in the Chomsky
hierarchy. Although BNF is traditionally used in thesyntax definition of programming languages, it ispossible to describe ambiguous grammar (e.g. danglingelse problem). The ambiguous grammar is not allowed
programming languages because the compiler wouldinterpret multiple languages. Unfortunately, it has been
proved that there is no algorithm to judge that the CFG
contains ambiguous grammar.1,2
On the other hand, Parsing Expression Grammar(PEG) introduced by Bryan Ford3 is not ambiguous
grammar because it has ordered choice property.
However, the ordered choice causes “prefix capture”.4“Prefix capture” is a problem of hiding the language tobe accepted according to the order of choice. Checking
syntax files that contain such mistakes usually confirms
the behavior of the parser generated by the parser
generator. However, in confirming the behavior of the

2. Parsing Expression Grammar and its parsing
techniqueParsing Expression Grammar (PEG) is a formal grammar,
introduced by Bryan Ford in 2004. 3 PEG is deterministic.
Therefore, it is suitable for the syntax definition of the
programming languages, and it was found that not only
the Context-Free Grammar (CFG) but also a part of theContext-Sensitive Grammar (CSG) can be expressed.
2.1. Definition of PEG
A PEG

is defined by the 4-tuples:=( , , , )

where
1.
is a finite set of non-terminal symbols.
2. Σ is a finite set of terminals (Σ ∩= ).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan169

Toshihiro Miyaji, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, and Naonobu Okazaki3.
is a finite set of production rules.
4.
is a parsing expression called the start expression.
The production rule is described in the form of “ = ”(where ∈ , is a parsing expression). A parsing
expression is an instruction for an input string given byusers and is described by a non-terminal symbol or aterminal symbol. A parsing expression can use operators.
When the input string
is applied to the parsing
expression , matches the substring of and indicates“success” or “failure”. has a pointer representing thereading position. If indicates success, consumes thesubstring of (i.e. advances by the length of thesubstring). Also, if e indicates failure, let applies thenext parsing expression.

extracts testable symbols from the grammar described in
the text editor area.
3.1. Production rules verification
Production rules verification can confirm whether the
input string is accepted by the production rule. In one
verification, production rules verification requires an
expected output in addition to one testable symbol andone input string. The users can select whether the
specified input string is accepted or rejected. Tamias
displays the comparison result between the expected
output and the output of PEG interpreter. If the expected
output and the actual output are equal, “Passed” isdisplayed. Otherwise, “Failed” is displayed.

2.2. Packrat parsing technique3.2. Reach rate measurement

Packrat parsing is one of parsing techniques that canparse PEG. 5 It is a top-down parsing technique proposedby Bryan Ford in 2002. Ford solved the problem that theparsing time of PEG including a backtracking increases
exponentially with linear time by memoization of theparsing result.

Reach rate measurement calculates the rate of success ofparsing in the ordered choice of the selected production
rule. The reach rate is obtained by assigning two or morecheck results in one production rule to the following
calculating formula:
× 100
(1)
Reach rate [%] =2.3. Prefix capture“Prefix capture” is a problem of hiding the language tobe accepted according to the order of choice. For example,
the grammar “A = (‘+’ / ‘++’)[a-z]” does not accept thelanguage “++i”. The reason is that parsing “[a-z]” fails
after parsing “+” of the language “++i”. To solve thisproblem, the order ‘+’ and ‘++’ is reversed. Thus, it isdifficult to understand prefix capture immediately.

3.3. PEG interpreter

PEG interpreter is an interpreter that executes parsing for
PEG described in text editor in Tamias. PEG interpreter
requires a testable symbol and an input string. PEG
interpreter shows the output of the parsing expression
after parsing. By selecting testable symbols, PEG
interpreter can check not only top level non-terminal
symbols.

3. Tamias
Tamias is a syntax file checker to support checking thesyntax files for PEG. Tamias has three areas: a text editorarea, a list of non-terminal symbols, and a production
rules check area. Tamias has PEG interpreter which cancheck all choices and any non-terminal symbols in
production rules according to PEG. Also, Tamias canparse without building a parser. A checking method usinga PEG interpreter can be selected from the production
rules check area. There are two methods of checking the
production rules, “production rules verification” and“reach rate measurement”. Both methods require one
testable symbol and one input string. A testable symbolrefers to a non-terminal symbol from which a terminal
symbol can be derived. Tamias recursively searches and4.

Implementation of PEG interpreter4.1. Data used in the PEG interpreter
Production rules
Production rules are rules obtained from nonterminal symbols that can be derived from testable
symbols. One production rule is expressed as a pairof a non-terminal symbol and a parsing expression.
Input string
Input string has a string and a pointer that expresses
a reading position.
Lookup table
Lookup table stores the reading position of the input
string. The row of the lookup table corresponds to© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

170

Tamias: a Syntax File

Fig. 2. Result of executing production rules verificationin the production rules check area. Next, we selected“accept” as the expected output in both languages. The
experimental results are shown in Fig.2.
As shown in Fig.2, the check result of the language“1+2*3” according to the grammar was “Passed”. On theother hand, the check result of the language “*2” notconforming to the grammar was “Failed”. Therefore, theproduction rules verification can check the grammar
correctly.

Fig. 1. PEG interpreter algorithm

the parsing processing, and the column corresponds
to the reading position of the input string.

5.2. Reach rate measurement

4.2. PEG interpreter algorithm

We confirm the reach rate measurement method by using
grammar “A = (‘+’ / ‘++’)[a-z]” including prefix capture.
However, since Tamias does not currently support inputof parsing subexpression, we write the following
grammar into the text editor area of Tamias.
A = Op Id
Op = ‘+’ / ‘++’Id = [a-z]Next, we measure the reach rate of “A” using two
languages “+i” and “++i”. Finally, we reverse the choices
of “Op” and measure the reach rate of “A” using the samelanguages. The experimental results are shown in Fig.3.
As shown in Fig.3, the reach rate is not 100% in thegrammar including prefix capture. In contrast, the reachrate is 100% in the grammar excluding prefix capture.
Therefore, from the formula of reach rate (section 3.2(1)):
Parsing success choice = All choices
it can be proved that prefix capture has not occurred for
all input strings.

The algorithm of the PEG interpreter is shown in Fig.1.
It recursively is executed until parsing of the terminal
symbol.
5. Experiment and Discussion
We confirm that PEG interpreter can parse correctly byusing grammar representing addition and multiplicationand grammar including prefix capture.
5.1. Production rules verification
We confirm the production rules verification method byusing grammar representing the addition andmultiplication below:
Sum = Product ‘+’ Product / ProductProduct = Value ‘*’ Value / Value
Value = [0-9]We write the grammar in the text editor area ofTamias and check the two languages “1+2*3” and “*2”© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

171

Toshihiro Miyaji, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, and Naonobu Okazaki

7. Conclusions
In this paper, we have proposed Tamias: a syntax file
checker to support checking the syntax files for PEG.
Tamias has PEG interpreter which can check all choices
and any non-terminal symbols in production rules
according to PEG. Tamias can check the production rules
and can measure reach rate of choices by using PEG
interpreter. In the production rules verification, Tamias
compares the expected output with the output of the PEG
interpreter and displays the result. In the reach ratemeasurement, Tamias can check that the grammar does
not contain prefix capture.
The experiment with a grammar representing addition
and multiplication showed that Tamias can compare theexpected output by users with actual output by PEG
interpreter without building a parser. The experimentwith a grammar including prefix capture showed that
Tamias can prove that prefix capture has not occurred forany input strings by measuring reach rate.
Therefore, Tamias can support checking PEG syntax
files. A future work is to improve Tamias to supportparsing subexpression described by parentheses.
References
1. David G Cantor, On the ambiguity problem of Backus
systems. Journal of the ACM (JACM), Vol.9, No.4,
pp.477-479, 1962.
2. Robert W Floyd, On ambiguity in phrase structure
languages. Communications of the ACM, Vol.5, No.10,
pp.526-534, 1962.
3. Bryan Ford, Parsing expression grammars: a recognitionbased syntactic foundation. ACM SIGPLAN Notice, Vol.
39, pp.111-122. ACM, 2004.
4. Roman R Redziejowski, Parsing expression grammar as aprimitive recursive-descent parser with backtracking.
Fundamenta Informaticae, Vol.79, No.3-4, pp.513-524,
2007.
5. Bryan Ford, Packrat parsing:: simple, powerful, lazy,linear time, functional pearl. ACM SIGPLAN Notices,
Vol.37, pp.36-47. ACM, 2002.
6. Shun Honda et.al., Implementing and Evaluating a
Debugger for Supporting Development of PEGs, JSSST,
Vol.32, pp.1-6, 2015. (in Japanese)
7. Kensuke Mori et al., Composition of parsers forincremental Parsing Expression Grammars., JSSST,
Vol.30, pp.641-650, 2013. (in Japanese)

Fig. 3. Two results of executing reach rate measurement

6. Related Research
Honda et al. proposed a PEG debugger to support
development of PEG. 6 The PEG debugger has functions
such as setting breakpoints, executing steps, displaying
stack trace, and so on. The PEG debugger can check the
behavior for one input and the occurrence of prefix
capture. However, the PEG debugger cannot prove thatprefix capture has not occurred for any input string.
Mori et al. proposed a parser generator that cancomposite parsers and implemented its prototype. 7 In the
traditional method, when part of the syntax in PEG waschanged, it was necessary to rebuild the corresponding
parser. The proposed method can generate a parser thatcan deal with the problem of the traditional method.
Tamias solves the problem of rebuilding the parser byimplementing a PEG interpreter.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

172

Implementation of an Arduino Simulator to Support Circuit Design
Tatsumi Nishida*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, and Naonobu Okazaki**

University of Miyazaki, 1-1 Gakuen-kibanadai nishi, Miyazaki, 889-2192 JapanTokyo University of Technology, 1404-1 Katakuramachi, Hachioji City, Tokyo 192-0982, Japan
E-mail: nishida@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kitayshr@stf.teu.ac.jp,
yamaba@cs.miyazaki-u.ac.jp, aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp†

Abstract
Embedded technologies are used everywhere. One of the educational materials to learn the embedded technologies
is Arduino. When a beginner designs circuit, it is possible to design the circuit that damages Arduino itself or themodules on it. Therefore, this research implements an Arduino simulator that runs on iPad for supporting to design acircuit. In this simulator, the circuit on iPad is analyzed to detect the value of voltage on the circuit and the risk that
Arduino itself or the modules are damaged.
Keywords: embedded technologies, Arduino, simulator, iPad

voltage on the circuit and the risk that Arduino itself or
the modules are damaged. These features can support todesign a circuit on Arduino.

1. Introduction
Embedded technologies are used everywhere and are
indispensable in our daily life.1 For example, there arecars, car navigation systems, air conditioners, televisions,and so on. As the demand for embedded software
increases, its development is also diversifying. One of theeducational materials to learn the embedded technologies
is Arduino. Arduino is used around the world as learning
kits.2,3 We can learn embedded technologies by designinga circuit and controlling the circuit by programming.
The learning kits that summarize sensors andjumper wires are also on the market, and Arduino has theadvantage that even a beginner who have never designed
a circuit can easily develop it. However, when a beginnerdesigns a circuit, it is possible to design the circuit thatdamages Arduino itself or the modules on it.
Therefore, this research implements an Arduino
simulator ADVIS (Arduino VIrtual Simulator) that runs
on iPad for supporting to design a circuit. In ADVIS, thecircuit on ADVIS is analyzed to detect the value of

2. Arduino Uno
Arduino has various kinds such as Arduino Uno, andArduino Leonardo, and so on. ADVIS developed in thispaper simulates Arduino Uno.
Fig.1 shows an overview of Arduino Uno. Arduino
Uno is a digital input / output device equipped with 8 bits
microcomputer. Arduino Uno itself has 32KB flash
memory. Therefore, even if it is turned off, the program
can be saved in it. Arduino Uno flows electric current of
20 mA of current from each output pin. Based on theabove specifications, the value of the electric current
flowing from each output pin of ADVIS is 20 mA.
In Arduino, depending on the circuit designed by auser, it is possible to design the circuit that damages
Arduino itself on it. Specifically, there are the followings.
A risk of overvoltage when output pins areconnected© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan173

Tatsumi Nishida, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, and Naonobu Okazaki

Fig. 2. An overview of ADVIS.

more flows, the LED lights up in green. The resistor has
a resistance value of 150Ω.
3.3. Menu partFig. 1. An overview of Arduino Uno

In the menu part, it is implemented the "CLEAR"
function, the "BACK" function, the "RUN" function, andthe "VOLT RUN" function.
Using the "Clear" function removes all modules
that are placed in the circuit design part and returns to theinitial state.
Using the "BACK" function returns the circuitdesign part to its previous state. Here, immediately afterusing the "CLEAR" function, you cannot use the"BACK" function.
Using the "RUN" function, the designed circuit isexecuted to detect the risk of module breakage. Fig.3
shows an example of the “RUN” function (Here, theimage is edited for easy-to-read). When the risk of amodule breakage is detected, highlight the input pin of
the module in red.
Using the "VOLT RUN" function displays the valueof voltage on the designed circuit. Fig.4 shows anexample of the “RUN” function (Here, the image isedited for easy-to-read).

A risk of damaging the blocking diode in theArduino itself when the voltage exceeding 3V is sent
to the input pin
A risk of overcurrent if the input pin receives anelectric current exceeding 200mA
In the implementation of ADVIS in this paper, only
the first and second risks as above can be detected.
3. Arduino simulator ADVISThe requirements of ADVIS is as follows.
iPad Pro 12.9 (2nd generation)
iOS 10+
Fig.2 shows an overview of ADVIS. ADVIS consists ofthe following three parts.
Circuit design part
Module selection part
Menu part3.1. Circuit design partIn the circuit design part, a circuit is designed using
jumper wires, LEDs, and resistors.

3.4. Function of ADVISIn ADVIS, in addition to the risk of damaging theArduino itself described in Section2, the risk ofdamaging the module can be detected. It’s target module
is only LED. LED is damaged if it receives voltage morethan 3V.

3.2. Module Selection partIn the module selection part, a module to be handled infor the circuit design part is selected. ADVIS can handle
jumper wires, LEDs, and resistors only. The LED has aresistance value of 100Ω, and when a voltage of 3 V or

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

174

Implementation of an Arduino

Fig. 3. Example of the “RUN” function

Fig. 5. Displaying the value of voltage on the circuit

Fig. 4. Example of the “VOLT RUN” function

Fig. 6. Displaying of value of voltage using resistor

From Fig.5, 5V is displayed on the circuit
connected to the 5V pin, and 10V is displayed on thecircuit connected to the Vin pin.
Next, the resistor and the 5V pin to the breadboard
are connected. Fig.6 shows the result that the "VOLT
RUN" function is executed.
From Fig.6, 5V is displayed on the circuit, and 2V,
which is calculated from Ohm's law ( = 0.02 × 150), isdisplayed on the circuit.
Therefore, it can be confirmed that ADVIS displays
the value of voltage.

4. Verification if ADVISIt is confirmed that ADVIS can simulates correctly thedesigned circuit. The followings are verified.
Displaying of the value of voltage on the circuit
Detecting a risk of Arduino itself being damaged
Detecting a risk of module being damaged
4.1. Displaying of value of voltage on the circuit
To display the value of voltage on the circuit, it can bedisplayed by using the “VOLT RUN” function. As an
example, connect 5V pin and Vin pin to the breadboard
respectively. Fig.5 shows the result that the “VOLT RUN”
function is executed.

4.2. Detecting a risk of Arduino itself being
damagedIn order to detect where there is a risk of corruption of
Arduino itself, it can be displayed by using the "VOLT
RUN" function. As an example, connect 5 V pin and Vin© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

175

Tatsumi Nishida, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, and Naonobu Okazaki

Fig. 7. Detecting a risk of Arduino itself being damaged

Fig. 8. Detecting a risk of the module being damaged

pin to input / output pins. In this circuit because thevoltage exceeding 3V is followed to the input pin, there
is a risk that the blocking diode of the Arduino itself will
be damaged. Fig.7 shows the result that the "VOLT
RUN" function is executed.
From Fig.7, ADVIS can display the Arduino input /output pins in red when the risk that the Arduino itself isdamaged is detected.

Save function of the designed circuit
Circuit control by programming
A function to automatically generate a program
based on the designed circuit
Implementation of resistors with various values andother modules
References
1. Masayuki Hirayama: Software Engineering for Embedded
System, Current State of Embedded Software
Development, IPSJ Magazine, Vol. 45, No. 7, pp. 677-681,
2004 (in Japanese).
2. P. Jamieson, "Arduino for teaching embedded systems. arecomputer scientists and engineering educators missing theboat?", Proc. FECS, pp. 289-294, 2010.
3. Mohammed El-Abd, A Review of Embedded Systems
Education in the Arduino Age: Lessons Learned andFuture Directions, International Journal of Engineering
Pedagogy, Vol. 7, No.2, pp.79-93, 2017.

4.3. Detecting a risk of module being damaged
In order to detect where there is a risk of corruption ofmodules, it can be displayed by using the "RUN"
function. As an example, connect the LED to the 5V pin.
In this circuit, there is a risk of damaging the LED
because it receives the value of voltage higher than 3V.
Fig.8 shows the result that the "RUN" function is
executed.
From Fig.8, ADVIS can display the input pin in red
when the risk that the module is damaged is detected.
5. Conclusion
This research has implemented an Arduino simulator
ADVIS (Arduino VIrtual Simulator) that runs on iPad forsupporting to design a circuit. The implemented ADVIS
has the following functions.
Designing a circuit
Visualizing the value of voltage in the circuit
Detecting a risk of Arduino itself being damaged
Detecting a risk of module being damaged
Consequently, ADVIS can support to design a circuit.
Future issues are as follows.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

176

Effectiveness of Data Augmentation in Automatic Summarization System
Tomohito Ouchi, Masayoshi Tabuse

Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Shimogamohangi-cho, Sakyo-ku, Kyoto 606-8522, Japan
E-mail:t_ouchi@mei.kpu.ac.jp, tabuse@kpu.ac.jp

Abstract
We propose a new data augmentation method in automatic summarization system. A large corpus is required tocreate an automatic summarization system using deep learning. However, in the field of natural language
processing, especially in the field of automatic summarization, there are not many data sets that are sufficient totrain automatic summarization system. Therefore, we propose a new method of data augmentation. We use an
encoder-decoder model with an attention mechanism as automatic summarization system. First, we determine theimportance of each sentence in an article using topic model. In order to extend the data, we remove the least
important sentence from an input article and use it as a new article. We examine the effectiveness of our proposed
data augmentation method in automatic summarization system.
Keywords: automatic summarization, data augmentation, encoder-decoder model, attention mechanism.

summarization. The nature of automatic summarizationis that there is little influence on the summary that is
generated whether there is redundant part or not. Further,in image processing, data may be expanded by changing
the background. This assumes that the background has
no direct impact on the target task. Therefore, we
focused on change of background which is one of data
expansion of image processing, and expanded data by
deleting redundant sentences of original article inautomatic summary.
Automatic summarization is classified into two
approaches. One is an extractive summarization inwhich words, phrases or sentences of input articles are
directly extracted and combined to create a summary.
The other is an abstractive summarization in which an
input article is temporarily converted to an intermediate
vector and then a summary is generated based on theintermediate vector. In extractive summarization, there
is a problem of how to connect extracted words, phrases
or sentences. And there is also the possibility that inextractive summarization it is unknown what the

1. Introduction
In recent years, the amount of information on the net of
the world has increased exponentially, and it is expected
to become 44 ZB in 2020[1]. Under the circumstance,
the technique of automatic summarization becomes
indispensable for efficient selection of information.
Therefore, we first tried to start studying converting
English articles that account for the majority of theinformation on the net into Japanese summaries.
However, there is no large corpus in which English
articles and Japanese summaries are set. Therefore, we
considered whether the technique of data augmentationcommonly used in image processing can be applied tonatural language processing.
Several studies have applied data augmentation tonatural language processing. A method of adding noise
to an intermediate vector[2], and a method of data
augmentation by generating a dialect sentence
correspond to each given standard sentence[3].
However, both methods are adapted to documentclassification, yet none have been adapted to automatic©The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan.10-13,B-ConPlaza, Beppu, Oita, Japan177

Tomohito Ouchi, Masayosi Tabusedirective word is pointing at. On the other hand, in
abstractive summarization, since it is generated from aninternal semantic representation, there is a possibility
that it becomes one integrated summary. It also learns
grammatical information, so it is easy to connect words
and phrases in contrast to an extractive summarization.
However, the abstract summarization needs to belearned with a large number of pairs of input articles
and generation summaries, which takes time and effort
to prepare the corpus.
In this research, we examine whether data
augmentation is effective in abstractive summary. As amethod of data augmentation, an article excluding thesentence with the lowest importance level in the input
article is also learned. Our abstractive summary is amodel obtained by adding an attention mechanism to theencoder-decoder model. The proposed model and themethod of data augmentation are described in Section 2.
Experiment and evaluation are described in Section 3,
discussion and summary are given in Section 4.

We put each word of the input article in RNN
sequentially and the hidden vector created. When thelast word of the input article is inputted, the encoder
outputs the intermediate vector and decoder outputs
word by word using the intermediate vector. After
inputting the word as a 1-of-k vector, we embed thevector to the lower dimensional vector with anembedding matrix. However, since the number of wordsof the target input article is several hundreds, theinformation of the word entered at the beginning is hard
to be transmitted to intermediate vector. Therefore, we
use LSTM unit, which is resistant to gradientelimination and gradient explosion. We make LSTM on
the encoder side bidirectional. As a result, it becomes
multi-layered LSTM. This model is shown in Figure2.
x k ,l represents the l-th word of the k-th sentence. n k
represents the number of words of the k-th sentence.
However, this model cannot reflect information wellSentencelevel

2. Our Model

Word
level

In this section, we describe the model used in thisresearch and the method of data augmentation.

LSTM1

LSTM1

LSTM1

LSTM1

LSTM1

x1,1

x1, 2

x1,n1

x2 , n 2

xk ,n kSecondsentence

Last
sentence

decoder

because the intermediate vector has a fixed number ofdimensions regardless of the length of the input article.
Therefore, we used the attention mechanism which uses
information of the input words when the systemgenerates words. Figure 3 shows attention-based model.

state h(t 1) at the previous time as inputs of RNN.

Uht 1

LSTM2

Fig.2 proposed model (encoder side only)

The basic form of the model used in our study is an
encoder-decoder model combining two RNNs. RNN isused to process time series data, and updates the hidden
state with input xt at a certain time t and the hidden

(Wx t

LSTM2

First sentence

2.1. Attention-based Encoder-decoder model

ht

LSTM2

b)

(1)
ĥl

W, U and b are parameters and they are the same at eachtime step, respectively.
is an activation function.
Figure 1 shows a simple encoder-decoder model.
y1

y2

cl
l

eos

h1

h2

s1

s2

hk

Sentence level

ĥl

sk

h1

hl

xn

y1

yl

Word level

x1

x2

xn

bos

Encoder

Fig.1 Encoder-Decoder model. xi , y i

y1

ym

x1

Decoder

encoderV
0,1 ,V is the

decoderFig.3 Attention-based model. State of generating
the (l +1)-th word on decoder side

number of vocabulary. xi and y i are vectors encoding of
1-of-k representation.

©The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan.10-13,B-ConPlaza, Beppu, Oita, Japan178

Tomohito Ouchi, Masayoshi Tabuse

Effectiveness of Data Augmentation

3. Experiment and Evaluation

The attention-based model is expressed by the following
formula.
lsoft max(h1 hl , h2 hl ,
k

cl

l ,K

(3)

3.1. Dataset(cl , hl ) WC(4)

arg max(soft max(hˆlWh ))(5)

We used CNN news story dataset [5]. This data set is a
total article number of 92,579 articles, multiple
summaries are attached to each article. In theexperiment, we used the first abstract as a reference
summary. Also, I tried learning with 45,000 articles and90,000 articles.

K 1

hˆl
yl 1

hK

In this section, we evaluate the effectiveness of the data
augmentation proposed in Section 2.

hk hl ) (2)Thl represents the hidden state of the l-th word of thegenerated summary. In equation (2), softmax function isapplied to the inner product of the hidden state of eachsentence and the hidden state of the generated summary
word, and creates a l indicating which sentence is how

3.2. Parameter settings

important. In equation (3) we take a weighted average(6)1-of-k vector of both encoder and decoder wasembedded to 300 dimensional vectors. So, the input of
LSTM at the word level encoder and the input of LSTMof the decoder were both 300 dimensions. Since the
LSTM at the word level encoder is bidirectional, theoutput is 600 dimensions. So, the input of LSTM at thesentence level encoder was 600 dimensions, and theoutput was 300 dimensions. The final output of theattention-based model was also set to 300 dimensions.
We used pre-learned word2vec [6] for the initial valueof the embedding matrix and a transformation matrix tobe transformed from the vector into a vector of
vocabulary number dimensionality. Adam was used as alearning algorithm. In addition, beam search was used
as a searching algorithm of generated sentences. The
maximum summary length was 50 and the beam size
was set to 16.
In the case of 90,000 articles for learning, it took 1.5
days without data augmentation and 3 days with data
augmentation.

(7)

3.3. Evaluation method

on it. In Equation (4), cl and hl are rearranged andconverted into a hidden vector ĥl . In Equation (5), ĥlis converted into a vector of vocabulary number
dimensionality, softmax is applied, and the word id
having the maximum value is returned.
2.2. Data augmentation method
We measure the importance for each sentence of the
input article and use the article generated excluding thelowest important sentence as new data.
Refer to [4] for how to determine importance for eachsentence in the input article. First, a topic model iscreated from the English Wikipedia full text, and thesentence weight is determined for each topic of the
input article. Then, the sum of sentence weights for all
topics is taken as the final weight of the sentence, and it
was taken as the importance of the sentence.

bt ,i

Wt ,i
Ni

bi

bt ,i
t

ROUGE [7] is used as a general evaluation method of
automatic summarization, which is evaluated bymatching of reference summary and generated summary
word. We used ROUGE-1, ROUGE-2, ROUGE-L,
ROUGE-W. ROUGE-1 and ROUGE-2 evaluate
unigram and bigram coincidence, respectively.
ROUGE-L evaluates matching the longest common
subsequence. ROUGE-W is also almost the same, but it
evaluates matching of continuing longest common
subsequence.

Firstly the weight bt ,i of the sentence i with respect tothe topic t is examined. In Equation (6), bt ,i multiplies
the sum Wt ,i of an importance of the words constituting
the sentence i by the inverse of the square root of thetotal number N i of words as a coefficient. Thiscoefficient is for weighting sentences that are notinfluenced by the sentence length. In Equation (7), we
let the sum of bt ,i be the final weight of sentence i.

©The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan.10-13,B-ConPlaza, Beppu, Oita, Japan179

Tomohito Ouchi, Masayosi Tabuse

3.4. Result

References

Evaluation of our data augmentation using by ROUGE
is shown in Table 1 and 2. f, r, p in the Table represent
F value, recall and precision, respectively.
Experiments were performed by recognizing 1,000 test
data not used for training data. The same test data wasused in all cases.

1. Data Growth, Business Opportunities, and the IT
Imperatives,
http://www.emc.com/leadership/digitaluniverse/2014iview/executive-summary.htm
2. Dongxu Zhang, Zhichao Yang, “Word Embedding
Perturbation for Sentence Classification” arXiv :1804.08166vl (2018).
3. I. Saito, J. Suzuki, K.Nishida, K. Sadamitsu, S.
Kobashikawa, Ryo. Masumura, Y. Matsumoto, J. Tomita“Improving Neural Text Normalization with Data
Augmentation at Character- and Morphological Levels”
Proceedings of the 8th International Joint Conference on
Natural Language Processing, 257-262 (2017).
4. H.Sigematsu, I.Kobayashi “Generation of abstracts
considering importance of potential topics” The
Association for Natural Language Processing (2012) (InJapanese).
5. How to Prepare News Articles for Text Summarization,
https://machinelearningmastery.com/prepare-newsarticles-text-summarization/
6. https://code.google.com/p/word2vec/
7. Lin C. W., “ROUGE: A Package for Automatic
Evaluation of Summaries”, Proceeding of the ACL-04
workshop, Vol.8 (2004).
8. F. Shinmura, Y. Kawanishi, D. Deguchi, I. Ide, H.
Murase and H. Fujiyoshi, “Pedestrian’s Inattention
Estimation based Recognition of Texting While Walking
from In-Vehicle Camera Images”, IEICE Technical
Report, 83–88 (2015).
9. A. Krizhevsky, I. Sutskever and G. E. Hinton, “ImageNet
Classification with Deep Convolutional Neural
Networks”, Advances in Neural Information Process.
Systems 25, 1097–1105 (2012).

original
extended

ROUGE-1 f ROUGE-1 r ROUGE-1 p ROUGE-2 f ROUGE-2 r ROUGE-2 p
0.10640.10500.1116
0.0436
0.0436
0.0438
0 .11 49
0 .1 13 8
0.1 19 7
0 .04 52
0 .0 45 1
0 .0 45 7

normalextended

ROUGE-l f
0.1571
0 .16 39

ROUGE-l r
0.15400 .1 61 2

ROUGE-l p ROUGE-w f ROUGE-w r ROUGE-w p
0.1641
0.09750.07820.13500.1 70 5
0 .10 08
0 .0 81 2
0 .1 38 6

Table 1 Results on 45,000 articles. “original” is original
data and “extended” is extended data.
normalextended

ROUGE-1 f ROUGE-1 r ROUGE-1 p ROUGE-2 f ROUGE-2 r ROUGE-2 p
0.13750.13480.14470 .07 74
0 .0 73 8
0 .0 75 6
0 .14 450.14 20
0.1 51 4
0.07240.07220.0730normalextended

ROUGE-l f
0.12530 .13 59

ROUGE-l r
0.12350 .1 34 5

ROUGE-l p ROUGE-w f ROUGE-w r ROUGE-w p
0.13020.07430.05990.1023
0.1 40 6
0 .07 95
0 .0 64 1
0 .1 09 1

Table 2 Results on 90,000 articles.

3.5. Discussion
Except for ROUGE-2 value of 90,000 articles, results of“extended” data were better than ones of “normal” data.
The reason for this result is that using the augmentation
data the system learn not only original articles but also
articles with less important sentences removed, so it
learn which sentences are important or not. Therefore,
the model with extended data learned more informationof sentences.
4. Conclusion
Although the degree of accuracy improvement by thedata augmentation was lower than the degree of
accuracy improvement by setting 45,000 articles to90,000 articles, it was able to confirm that there was aneffect. As a future task, since the rouge value in thismodel is still low, we want to improve it.

©The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan.10-13,B-ConPlaza, Beppu, Oita, Japan180

A System for Posting on SNS Portrait Selected Using Facial Expression Analysis
While Writing Message
Taro Asada1, Yuiko Yano2, Yasunari Yoshitomi1, Masayoshi Tabuse1

1: Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: t_asada@mei.kpu.ac.jp, {yoshitomi, tabuse}@kpu.ac.jp}
http://www2.kpu.ac.jp/ningen/infsys/English_index.html
2: Nara Guarantee Corporation,
163-2 Horencho, Nara 630-8668, Japan

Abstract
We have developed a real-time system for expressing emotion as a portrait selected according to the facial
expression while writing a message. The portrait is decided by a hair style, a facial outline, and a cartoon of facial
expression. The system selects one portrait expressing one of neutral, subtly smiling and smiling facial expressions
using two thresholds on facial expression intensity. We applied the system to post on an SNS a message and a
portrait expressing the facial expression.
Keywords: Facial expression analysis, Real-time system, Mouth area, Portrait, Writing messages, OpenCV.

smiling facial expressions using the facial expression
intensity thresholds previously decided for these facial
expressions. The portrait selection method used in thispaper is based on a previously reported method.4

1. Introduction
Recently, social network services (SNSs) have become
very popular as communication tools on the Internet. A
message, a static image, or a moving image can beposted on Twitter. However, information about our realemotions cannot be posted on Twitter while writing amessage. As a result, a message might be misunderstoodby a receiver. It was reported that such a
misunderstanding on a message on LINE caused
bullying at a school.1
In this paper, we propose a real-time system forexpressing three kinds of emotions as a portrait selectedaccording to the facial expression while writing amessage. For generating a portrait, a user selects one
hair style and one facial outline before writing amessage. The image signal is analyzed by our real-timesystem using image processing software (OpenCV)2 anda previously proposed feature parameter (facial
expression intensity)3. Then the system selects one
cartoon expressing one of neutral, subtly smiling and

2. Proposed System and Method
2.1. System overview and method outline
In this system, a webcam moving image captured inreal-time is analyzed via the following process.
The proposed method consists of (1) extracting
the mouth area, (2) calculating the facial expression
feature vectors, (3) determining the facial expression
intensity and calculating its average value while writinga message, (4) selecting hair style and facial outline ofportrait by a user, and (5) posting the message and an
automatically selected portrait for the message on
Twitter. The details of these steps are explained in thefollowing subsections.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan181

A System for Posting

2.2. Mouth area extraction
The mouth area is selected for facial expression analysis
because it is where the differences between neutral andsmiling facial expressions appear most distinctly. We
use the mouth area extraction method which is
described in our reported research5 and is briefly
reviewed in this subsection.
Moving image data while writing message arechanged from RGB to YCbCr image data, and then theface area is extracted from the YCbCr image as arectangular shape, and the lower 40% portion of the facearea is standardized. Next, the mouth area is extracted
from that portion.

Fig. 1. Selectable hair styles for female.

2.5. Posting a message and an automatically selected
portrait for the message on Twitter
Facial expression intensity is measured using ourpreviously discussed method.7 A portrait is generated by
the proposed system using a cartoon of facial expression,a hairstyle and a facial outline which are selected
beforehand by the user. The cartoon is automaticallyselected by comparing the average value of facial
expression intensity and two thresholds decided
experimentally beforehand. Then the message and theselected portrait are posted on Twitter when the userpresses a button of the proposed system. In this system,
three portraits, expressing neutral, subtly smiling andsmiling facial expressions, are used (see Fig. 2).

2.3. Facial expression intensity measurement
We use the facial expression intensity measurement
method which is described in our reported research5 andis reviewed in this subsection.
For the Y component of the selected frame, thefeature vector for the facial expression is extracted forthe mouth area. The extraction is performed by using a
two-dimensional discrete cosine transform (2D-DCT)
for each 8 × 8-pixel section. Two 8 8-pixel sections at
each of the left and right lower corners are not included
for this measurement, because these sections mightcause errors due to the appearance of the jaw and/or neckline(s) there.5
To measure the feature parameters of the facial
expressions, we select low-frequency components fromthe 2D-DCT coefficients for use as the facial expression
feature vector elements. However, the direct current
component is not included. In total, 15 feature vector
elements are obtained. The facial expression intensity isdefined as the norm difference between the facial
expression feature vectors of the reference and target
frames. In this study, the first 20 continuous frames ofmouth area data successfully extracted after the webcam
recording begins are treated as reference frame
candidates. The reference frame selection method is
explained in detail in Ref. 6.

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 2. Each example for male or female portrait
expressing neutral (a and d) , subtly smiling (b and e), andsmiling (c and f) facial expressions.

3. Experiment3.1. Conditions
The experiment was performed on a Dell XPS 9350 PC
equipped with an Intel Core i7-6560U 2.2 GHz central
processing units (CPUs) and 8.0 GB of random accessmemory (RAM). The Microsoft Windows 7
Professional operating system (OS) was installed on the
PC and Microsoft Visual C++ 2008 and 2013 Express
Editions were used as the development language.
CoreTweet,6 as a library, and the Twitter API were used
to post a message and portrait on Twitter.
First, the condition of the preliminary experiment4
for a preparation of this experiment is reviewed in thissubsection. We had performed experiments with one
male (subject A in his 20s) and five females (subjects B
to F in their 20s) under the two conditions listed below.

2.4. Selecting hair style and facial outline of portrait
For generating a portrait of a user, the user selects (1) one
gender (male or female), (2) one hairstyle among sevenones for each gender (for female, see Fig.1) and (3) one
facial outline among four ones for each gender, andregisters them beforehand in the proposed system.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

182

Taro Asada, Yuiko Yano, Yasunari Yoshitomi, Masayoshi Tabuse

As an initial condition in the experiment, the subjects
had been instructed to maintain a neutral facial
expression and face forward without speaking for about
five seconds just after the start of the experiment. Afterthe initial state of a neutral facial expression had been
terminated, the subjects had been requested tointentionally respond with one of two types of facial
expressions (Experiment P1, neutral; Experiment P2,
big smile) and write a message, '友達から貰ったこのぬいぐるみかわいくない' (in Japanese), which means,‘This stuffed toy, a present from a friend, is pretty, isn't
it?’ Experiments P1 and P2 had been performed twotimes for each subject. In each experiment, facial
expression intensity measurements had been performed
for each subject during writing a message for 35
seconds and then the average facial expression intensity
for that message had been calculated.
In this study, in order to distinguish between theneutral, subtly smiling and smiling facial expressions on
the basis of average facial expression intensity whilewriting message, a threshold between neutral and subtly
smiling expressions was set as the intermediate value of
12 average facial expression intensities of the six
subjects for neutral (Experiment P1), and then one
between subtly smiling and subtly smiling expressions
was set as that for big smile (Experiment P2).
In this study, three experiments (Experiments 1, 2,3) were performed with two males (subject G in his 20s,
subject H in his 40s) and one female (subject I in her
20s). After the initial state of a neutral facial expression
was terminated, the subjects were requested tointentionally respond with three types of emotions(Experiment 1, neutral; Experiment 2, subtly smiling;
Experiment 3, smiling) and wrote a same message asthat in Experiments P1 and P2. In each experiment,
facial expression intensity measurements were
performed for 35 seconds and then the average facial
expression intensity for that message for each subject
was calculated, after which both the message and theportrait expressing the facial expression while writing
the message were posted on Twitter.

range of 1 to 11 seconds just after starting theexperiment are shown. The intermediate values of facial
expression intensities in Experiments P1 (neutral) andP2 (big smile) were 0.37 and 1.37, respectively (table 1).
Therefore, the two thresholds for distinguishing
between the three types of facial expressions wererespectively determined as 0.37 and 1.37. Thus, in the
proposed system, a facial expression having an average
facial expression intensity under 0.37 is judged to be aneutral facial expression, one 0.37 or higher and under
1.37 is judged to be a subtly smiling expression, and one
1.37 or higher is judged to be a smiling expression.
Table 1. The average of facial expression intensity under thecondition of Experiment P1 (neutral) and P2 (big smile).4
SubjectA

BC

1st or 2nd

1

2

1

2

1

2

Neutral

0.38

0.44

0.36

0.17

-0.4

0.51

Big smile

1.48

3.1

0.29

1.01

0.68

3.4

Subject

DEF

1st or 2nd

1

2

1

2

1

2

Neutral

0.4

0.34

0.16

0.38

0.66

-0.06

Big smile

0.45

1.85

0.83

1.25

12.96

11.98

Tables 2 shows the average of facial expression
intensity under the condition of Experiments 1, 2 and 3
for each subject. The values in Table 2 were calculated
in the same manner as that for Table 1. Figs. 3 to 5
show changes in facial expression intensity of mouth areaand mouth image at the characteristic timing point foreach subject. As shown in Tables 2 and 3, our proposedsystem distinguished among the three types of facial
expressions (neutral, subtly smiling, smiling) at theaccuracy of (8/9). Our system misjudged as a smiling
facial expression for a subtly smiling one of subject I(Tables 2 and 3).
Table 2. The average of facial expression intensity under thecondition of Experiments 1, 2 and 3.

3.2. Results and discussion
Table 1 shows the average of facial expression intensity
under the condition of Experiment P1 and P2 for eachsubject. In Table 1, the values subtracted by the averageof facial expression intensity for 10 seconds in the time

Subject

G

H

I

Exp. 1

0.13

0.07

0.26

Exp. 2

0.87

1.26

1.78

Exp. 3

10.36

1.83

7.9

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan183

A System for Posting

Table 3. Generated portraits for subjects G and I under thecondition of Experiments 1, 2 and 3.
1

2

G

I

3

G

I

G

I3

I

Facial expression intensity

Exp.
Subject

Facial expression intensity

Portrait

Subject G

3

Subject H

Subject I

2and a portrait. The improvement of the system for a
subtly smiling judgement is one of the next targets.

I1

H1

G1

Subject G
Subject I

15

Subject H

G3

10
5

H3

0
-5

10

15

20

25

30

1

35
Time (s)

0
-1

10

15

20

G1

25

H1

30
35
Time (s)

G3

I1

G2

H2

References

I2

1. K. Takeuchi, Sumaho Jidai ni Taiou Suru Seitoshidou・
Kyouiku Soudan (Honnomori Shuppan, Tokyo, 2014). (inJapanese)
2. Open CV. http://opencv.org/ Accessed 13 December 2017.
3. T. Asada, Y. Yoshitomi, R. Kato, M. Tabuse, and J.
Narumoto, Quantitative evaluation of facial expressions
and movements of persons while using video phone, J.
Robotics, Networking and Artif. Life 2(2) (2015) 111–114.
4. T. Asada, Y. Yano, Y. Yoshitomi, and M. Tabuse, Portrait
selection system using facial expression analysis while writing
messages on SNS (in Japanese), in Proc. of Human Interface
Symposium 2018 (Ibaraki, Japan, 2018), pp. 219–222.
5. Y. Yoshitomi, T. Asada, K. Mori, R. Shimada, Y. Yano,and M. Tabuse, Facial expression analysis and its
visualization while writing messages, J. Robotics,
Networking and Artif. Life 5(1) (2018) 37–40.
6. R. Shimada, T. Asada, Y. Yoshitomi, and M. Tabuse,
Real-time system for horizontal asymmetry analysis on
facial expression and its visualization, in Proc. of The
2017 Int. Conf. on Artif. Life and Robotics (Oita, Japan,2017), pp. 120–123.
7. T. Asada, K. Mori, R. Shimada, Y. Yoshitomi, and M.
Tabuse, Facial expression analysis in real time while writing
messages (in Japanese), in Proc. of Human Interface
Symposium 2017 (Osaka, Japan, 2017), pp. 469–472.
8. CoreTweet. https://github.com/CoreTweet/ Accessed 20
December 2017.

2
1
0
-1

10

15

20

25

30

I3

Acknowledgements
We would like to thank the subjects for theircooperation in the experiments.

Subject G
Subject H
Subject I3

H3

Fig. 5. Changes in facial expression intensity of mouth area forsubjects G, H and I (upper graph). Mouth images are shown forthree moments during Experiment 3 (G3, H3, I3), as indicatedon the graph (lower images).

Fig. 3. Changes in facial expression intensity of mouth area forsubjects G, H and I (upper graph). Mouth images are shown forthree moments during Experiment 1 (G1, H1, I1), as indicatedon the graph (lower images).

Facial expression intensity

20

35Time (s)G2
H2
I2
Fig. 4. Changes in facial expression intensity of mouth area forsubjects G, H and I (upper graph). Mouth images are shown forthree moments during Experiment 2 (G2, H2, I2), as indicatedon the graph (lower images).

4. Conclusion
Herein, we have developed a real-time system forexpressing three emotions (neutral, subtly smiling,smiling) as the portraits generated according to thefacial expression while writing a message. We applied
the system to the posting on Twitter of both a message© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan184

A Method Using Wavelet Transform for Judging Character to be Inserted into Image
Yasunari Yoshitomi1, Takashi Sato2, Taro Asada1, and Masayoshi Tabuse1

1: Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: {yoshitomi, tabuse}@kpu.ac.jp, t_asada@mei.kpu.ac.jp}
http://www2.kpu.ac.jp/ningen/infsys/English_index.html
2: School of Engineering, Tokyo Institute of Technology,
2-12-1 Ookayama, Meguro-ku, Tokyo 152-8550, Japan

Abstract
There has been an increase in the number of images on the Internet that contain private information and/or the URLof an illegal Web site. This information might result in a violation of human rights and/or a crime. Based on apreviously proposed method, we propose a method for judging whether characters have been inserted into an image
using the discrete wavelet transform, the image compression characteristics, and the empirical knowledge thatcharacters have strong vertical and/or horizontal elements.
Keywords: Character insertion judgement, Image compression characteristics, Internet, Private information, URL ofillegal Web site, Wavelet transform.

1. Introduction

2. Wavelet Transform of Image Signals

There has been an increase in the number of images on
the Internet that contain private information and/or the
URL of an illegal Web site. This information could
result in a violation of human rights and/or a crime. It isvery time-consuming and inefficient to detect suchinformation by inspecting each individual Web page,
but there is no existing automatic method for checking
images for such information. To achieve such a method,
the first step is to extract from the image the regions that
contain characters. An algorithm for extracting suchregions has received considerable attention in the fieldof computer vision research,1-3 but it falls far short ofpractical application. For overcoming the problem, we
proposed a method for extracting from an image theregions that contain characters.4
In the present paper, based on the previously
proposed method,4 we propose a method for judging
whether characters have been inserted into an image.

The image is first decomposed into four subbands: 1LL,
1LH, 1HL, and 1HH. The subbands labeled 1LH, 1HL,
and 1HH represent the finest-scale wavelet coefficients.
Fig. 1 shows the level 1 decomposition, in which theimage is decomposed into four subbands that share thesame scale.

Fig. 1. Mallat division.

In general, the wavelet coefficients for these threedomains (HH, HL, and LH) are called the elements of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

185

Yasunari Yoshitomi, Takashi Sato, Taro Asada, Masayoshi Tabuse

the multiresolution representation, whereas the wavelet
coefficients of the other domain (LL) are called theelements of the multiresolution analysis. For further
information on the DWT, see Ref. 5.

Step 1: The RGB elements of the input image (Fig. 3)
are transformed into YCrCb elements (Fig. 4). This isfollowed by applying DWT to each element to obtainthe 1HL and 1LH elements, which correspond to thevertical and horizontal elements, respectively (Fig. 4).

3. Method for Character Domain Extraction
from Image
Fig. 2 shows the flowchart of our previously proposed
method,4 and Fig. 3 shows an example of input andoutput images for our system for character domain
extraction. The algorithm of our previously proposed
method is reviewed in this section because it is very
important for the present study.

Fig. 4. 1HL (top row center) and 1LH (top row right) elements
for the Y element (top row left); 1HL (middle row center) and1LH (middle row right) elements for the Cr element (middle
row left); and 1HL (bottom row center) and 1LH (bottom row
right) elements for the Cb element (bottom row left); allobtained from the input image (Fig. 3).4

Step 2: A segmentation is performed for both the 1HL
and 1LH elements, using each threshold (Fig. 5). Then,the scattered regions that have values of 255 in thebinary image for each of YCrCb elements arehorizontally merged by changing to the value 255 all the
pixels having the same horizontal coordinate as a pixel
already having the value 255.
Fig. 2. Flowchart of previously proposed method.4

Fig. 5. Binary images obtained for both 1HL and 1LHelements for the Y, Cr, and Cb elements of the related images
in Fig. 4; left: Y; center: Cr; right: Cb.4

Fig. 3. Input image (left) and output image (right).4© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan186

A Method Using Wavelet

Step 3: Isolated regions are erased and regions with thevalue 255 are expanded.
Step 4: A mask is generated for extracting from theinitial RGB image those areas that contain characters.
This is done by using a logical summation to merge theregions with the value 255 in the binary images
obtained for the Y, Cr, and Cb elements. Finally, theoutput RGB image is generated by changing to white
the color of any pixel for which the value of thecorresponding pixel on the mask image is 0; this leaves
only the regions containing characters on the RGB
image.

Fig. 6. JPEG format image (left) transferred from a BMP
format input image and JPEG format image (right) transferredfrom a BMP format output image with (bottom) or without(top) a character region.

The threshold in Step 2 was set to be half the minimum
value for each element, but not higher than -40. The
threshold is determined on the basis of the sharpness ofthe sharpest domain in the image. In cases of the input
image having character regions, the threshold might bedetermined on the basis of the sharpness of characters.
For further information on our previously proposed
method, see Ref. 4.

Table 1 Size of JPEG format image (Fig. 6) transformed from24-bit 256×256 pixel BMP form.
Character insertion
Input
OutputWithout
28
19
With
29
4
(KB)

4. Image Compression Characteristics of Output
Images of Our System
We have discovered that output images of our system4
for character domain extraction have a tendency to besmall after compression by the JPEG technique when
the input image has character regions. Fig. 6 shows anexample of compression characteristics of an output
image. The upper left image (Lena) in Fig. 6 wasselected from the Standard Image Database (SIDBA).
The lower left image was produced by writing several
black characters (“abc09056”) on the upper left image.
The background of the characters was white. As shownin Table 1, we can judge whether characters are written
on the image by focusing on the reduction in JPEG
format image size when applying our previously
proposed method4 to the image. Fig. 7 shows therelationship between the sizes of JPEG format input andoutput images transformed from 24-bit 256×256 pixel
BMP form. Several images used for creating Fig. 7 wereselected from SIDBA. Others were gathered from theInternet. Then, several black characters (“abc09056”)
were written on the original images in a way similar tothat shown in Fig. 6. In Fig. 7, images with character
regions appear below those without character insertion.

Fig. 7. Relationship between sizes of JPEG format input andoutput images.

5. Proposed Method for Character Insertion
Judgment
Firstly, an image database is produced by collecting
images with or without character insertion from a
database like SIDBA or the Internet. All images in theimage database are input into our system4 for characterdomain extraction, and then the input and output imagesof our system are transformed into JPEG format images.
The two-dimensional feature vector space whose
components are a pair of JPEG format sizes of the input© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan187

Yasunari Yoshitomi, Takashi Sato, Taro Asada, Masayoshi Tabuse

and output images of the system is prepared as teacher
data. A test image is judged as to whether it has
character regions as follows:in Step III described in Section 5, the k-nearest neighboralgorithm (k=3) was used.

Step I A pair of JPEG format sizes of input and output
images of the system is obtained for a test image. Go tostep II.
Step II When the size of the output image is the constantobtained in case of an all-white image, the test image isjudged as having no character regions. Otherwise, go tostep III.
Step III In the two-dimensional feature vector space
whose components are a pair of JPEG format sizes of
the input and output images of our system, the test
image is judged using teacher data and a patternrecognition technique as to whether it has character
regions.

According to Step II, one test image was correctly
judged as having no character regions because the sizeof the output image was the constant (1.61 KB) obtainedin the case of an all-white image. Then, according to thecriterion in Step III, 25 test images were correctly judged
as to whether they had character regions. The 5 othertest images were misjudged. As a result, the accuracy ofthe character insertion judgment by Steps II and III was
83.9% (26/31). The image compression characteristicsof the output images of our system are due to the JPEG
algorithm, mainly the Huffman coding.

6. Experiments

A method for judging whether characters have been
inserted into an image was developed based on ourpreviously proposed method;4 this method uses thediscrete wavelet transform and the empirical knowledge
that characters have strong vertical and/or horizontal
elements. In the proposed method, we utilize image
compression characteristics of output images of oursystem.4 Our experimental results show the usefulnessof the proposed method.

6.2. Results and discussion

7. Conclusion

6.1. Conditions
The standard images selected from the Standard Image
Database (SIDBA) and other images obtained from theInternet were prepared in 24-bit 256×256 pixel BMP
form for evaluation of the proposed method. The
numbers of teacher and test images as original images
are 15 and 16, respectively. Several black characters(“abc09056”) were written on the original images; thebackgrounds of the characters are white. We used the
Daubechies wavelets for the DWT. The experiment was
performed in the following environment: personal
computer, DELL OPTIPLEX790 (CPU: Intel(R)Core(TM)i7-2600 3.40 GHz; main memory: 4.00 GB);OS, Microsoft Windows 7 Professional; development
language, Microsoft Visual C++ 6.0.
In the process of segmentation in Step 2, which
was described in Section 3, the pixels in the binary
images were set to 255 when the values of the 1HL or
1LH elements were below their particular thresholds;
otherwise, the pixels were set to 0. In the process ofsegmentation in Step 3, as described in Section 3,
regions with 50 or fewer pixels were erased; thisexpansion of the larger regions was performed fivetimes.
By the proposed method described in Section 5, thetest data are judged as to whether the image has
character regions. As the pattern recognition technique

References
1. H. Hirano and M. Okabe, Detection of text area inserted
by image editing (in Japanese), in IPSJ SIG Notes, vol.
2012-CVIM-183, no. 9, (Japan, Koganei, 2012), pp. 1–8.
2. K. Ashida, H. Nagai, M. Okamoto, H. Miyao, and H.
Yamamoto, Extraction of characters from scene images (inJapanese), Trans IEICE J88-D-II(9) (2005) 1817–1824.
3. S. Masuko, T. Hoshi, and H. Hirano, A system forsearching texts in images by using reliability factor andsaliency (in Japanese), J. IPSJ 51(9) (2010) 1520–1529.
4. T. Taniguchi and Y. Yoshitomi, Method for characterdomain extraction from image using wavelet transform, J.
Robotics, Networking and Artif. Life 2(2) (2015) 103–106.
5. D. Inoue and Y. Yoshitomi, Watermarking using wavelet
transform and genetic algorithm for realizing hightolerance to image compression, J. IEEJ 38(2) (2009)
136–144.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan188

An Authentication Method for Digital Audio Using Wavelet Transform and Fundamental
Frequencies
Yasunari Yoshitomi1, Shohei Tani2, Masaki Arasuna3, Ryota Kan4, Taro Asada1, and Masayoshi Tabuse1

1: Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: {yoshitomi, tabuse}@kpu.ac.jp, t_asada@mei.kpu.ac.jp}
http://www2.kpu.ac.jp/ningen/infsys/English_index.html
2: Fukuchiyama City Hall, 13-1 Miki, Fukuchiyama, Kyoto Prefecture 620-8501, Japan
3: Nissay Information Technology Co., Ltd., 5-37-1 Kamata, Ohta-ku, Tokyo 144-8721, Japan
4: Shimazu Business Systems Co., Ltd., 1 Kuwahara-cho Nishinokyo, Nakagyo-ku, Kyoto 604-8442, Japan

Abstract
Several digital watermarking techniques for audio files have been proposed for hiding data in order to protect their
copyrights. There is a tradeoff between the quality of watermarked audio and the tolerance of watermarks to signal
processing methods, such as compression. In order to overcome the inevitable tradeoff, we previously developed anauthentication method for digital audio. We have improved the method by determining the region to beauthenticated in the audio data by using the fundamental frequency characteristics.
Keywords: Authentication, Audio, Copyright protection, Wavelet transform, Fundamental frequency.

the region to be authenticated in the audio data isdecided by the fixed length and the fixed starting timefrom the beginning of the audio data. Therefore, theauthentication tolerance to clipping of the audio data is
insufficient for practical use.
In the present study, to overcome this issue, wehave improved the method by determining the region to
be authenticated in the audio data by using thefundamental frequency characteristics.

1. Introduction
Recent progress in digital media technology anddistribution systems, such as the Internet and cellular
phones, has enabled consumers to easily access, copy,
and modify digital audio. Several digital watermarking(DW) techniques for audio files have been proposed forhiding data in order to protect their copyrights. In
general, there is a tradeoff between the quality ofwatermarked audio and the tolerance of watermarks tosignal processing methods, such as compression.
In previous research,1 to essentially overcome thisissue, we developed an authentication method for digitalaudio to protect the copyrights. In contrast to DW, no
additional information is inserted into the original audio
by the previously proposed method, and the digitalaudio is authenticated using features extracted using a
discrete wavelet transform (DWT) and characteristiccoding of the previously proposed method.1 However,

2. Observed Phenomenon Underpins theAuthentication Method
The procedure and algorithm of our previously
proposed method1 is reviewed in this section, because itis very important for the present study.
It has been observed that when a DWT is applied
to audio data, in the histogram of the wavelet
coefficients of the multi-resolution representation© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

189

Yasunari Yoshitomi, Shohei Tani, Masaki Arasuna, Ryota Kan, Taro Asada, and Masayoshi Tabuse(MRR), the center of the distribution is very close to
zero.2 We exploited this phenomenon in order todevelop an authentication method for audio data.1 For
further information on the DWT used, see Refs. 3 and 4.
3. Authentication Ratio
We set the authentication parameters as described
below.1,5
In Fig. 1, Th (minus) was chosen so that it
divides the nonpositive wavelet coefficients ( S m intotal frequency) into two equal groups, and similarly

Th ( plus ) was chosen so that it divides the positiveFig. 1. Three sets (F, G, and H) of MRR wavelet coefficientsused for authentication.1

wavelet coefficients ( S p in total frequency) into two
equal groups. Next, the values of the parameters T 1 –

T 4 , which control the authentication precision, were

1 and ViF

H , bi is set to 1.

When f i

0 , bi is set to 0.5.

Note that the value 0.5 can be chosen arbitrarily, since
the value of b i that is the bit for creating the code forchosen such that the following conditions weresatisfied:
1) T 1 T h ( minus ) T 2 0 T 3 T h ( plus ) T 4 .
2) The value of S T 1 , the number of wavelet coefficientsin (T 1 , Th ( minus)) , is equal to ST 2 , the number ofwavelet coefficients in [Th ( minus), T 2 ) , i.e., ST 1 ST 2 .
3) The value of S T 3 , the number of wavelet coefficientsthe original audio data1 does not influence the method's
performance. Finally, this sequence of bi values is used
to form the authentication code C′.
The authentication ratio AR (%) is defined as follows:
N

100
AR

in (T 3 , Th (plus)] , is equal to ST 4 , the number ofwavelet coefficients in (Th (plus), T 4 ) , i.e., S T 3 S T 4 .
4) ST 1 / S m

When f i

fi 1
i

bi

b i

1

,

N

(1)

fi
i

1

where N is the number of wavelet coefficients chosen
to create the authentication code for the original audio
data.1 As can be seen in equation (1), neither bi nor bi

ST 3 / S p .

In the present study, the values of both S T 1 / S mand ST 3 / S p are set to 0.3, which is the same setting used

influences the value of AR when f i

for creating the code for the original audio data.1 When
preparing the authentication codes, the wavelet coefficients
V for each MRR sequence are divided as shown in Fig. 1
into three sets, which are defined as follows:• F {V | V V AC ,V Th' (minus)}

when the corresponding Vi that is the wavelet
coefficient of the original audio is not selected forcoding in the original audio data.1
To use the proposed method, we need to store theflags f i and the original code C for each copyrighted

V AC , Th ' (minus ) V

file that we want to protect. When calculating (1) inorder to authenticate audio data, we do not use theoriginal audio data; instead, we use the flags f i and the• G {V | V• H {V | V

Th ' ( plus )}V AC , Th ' ( plus ) V } ,where V AC is the set of wavelet coefficients from thetarget audio data that is used to create the authenticationcode.
The wavelet coefficients V i are then classified

0 , which occurs

code C for that file.1
4. Fundamental Frequency Characteristics of
Audio Dataaccording to the following rules with the flags f i used

Fig. 2 shows the fundamental frequency of the first entry of
the rock music genre category in the music database RWC
for research purposes.6 As shown, several local maximums

in creating the original code C:
When f i 1 and Vi G , bi is set to 0.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan190

An Authentication Method fora DELL OPTIPLEX CF-SX1 (CPU: Core i7-2600 Duo,3.40 GHz; main memory: 4.0 GB); the OS was
Microsoft Windows XP; the development language wasMicrosoft Visual C++ 6.0.
Five music audio files, namely, the first entry ofeach of five genre categories—classical, jazz, popular,rock, and hiphop—in the music database RWC used forresearch purposes,6 were copied from CDs onto apersonal computer as WAVE files with the following
specifications: 44.1 kHz, 16 bits, and monaural. For
each music audio file selected from the database, 10 setsof 10-second clips of music audio were produced using
the proposed method described in Section 5.
For investigating the authentication tolerance toclipping, audio test data were produced by clipping one
region from each music audio file selected from thedatabase. The clipped regions were specified by all
combinations of the lengths 0.01, 0.1, 0.5, 1.0, 2.0, 3.0,
and 4.0 seconds and the starting times 0, 5, and 10
seconds (from the beginning of the original audio data),
resulting in 21 clipping conditions. Then, for each audio
test data file produced from each original audio data file,
10 sets of 10-second clips of music audio were
produced using the proposed method described in
Section 5, and the authentication procedure was
performed using the previously proposed method.1 The
highest value of the authentication ratio of the originalaudio data to the audio test data (hereinafter referred toas HAR) of AR as described in Section 3 among thoseof the 100 combinations of the 10 clips of the original
data and the 10 clips of test data was calculated.
As another method for comparison (hereinafterreferred as AMFC), we chose audio data from theclassical music genre category and produced one 10second clip whose starting time gave the highest
fundamental frequency for 0.01 seconds among those in
the audio data. Next, the clipping regions from the
beginning of the original audio data were specified by
the lengths 0.00001, 0.00005, 0.0001, 0.0002, 0.0003,
0.0004, 0.0005, 0.001, and 0.01 seconds. Then, thecalculation of AR for the original audio data and theaudio data after clipping by the above each length was
performed for one 10-second clip whose starting timefrom the beginning of the audio data was decided by theabove method for the original audio data. For the DWT,
we used Daubechies wavelets. Level 8 was chosen
based on an analysis of preliminary experiments.1

Fig. 2. Fundamental frequency of audio data.

of the fundamental frequency exist within the stream of themusic. We have proposed a method for determining theregion to be authenticated in the audio data by using local
maximums of the fundamental frequency of the audio data,
as described in the next section.
5. Proposed Method for Determining the Region
to be Authenticated in the Audio Data
The audio data clipping procedure for the authentication
is as follows:
Step 1: The fundamental frequency f (i) i 1,2,3 , n at
the start time index i is measured every 0.01 seconds
from the beginning to the end of T seconds of theoriginal audio data, i.e., n 100T . Then, the absolutevalue of difference ADF (i)f (i 1)f (i) is calculated

for all i i 1,2,3, , n 1 .
999

Step 2: The sum S i

ADF (ij ) is calculated for all

j 0

ii

1,2,3,, n 999 .

Step 3: The start time indexes i are determined bywhether the value of S i is among the top 10 of all
S k under the restriction i j 1000 for all j such that
Si S j .
Step 4: For the start time indexes i selected in Step 3,
10 seconds of audio data are clipped for theauthentication.

6. Experiments
6.1 Conditions
An experiment was performed in the following
computational environment: the personal computer was© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

191

Yasunari Yoshitomi, Shohei Tani, Masaki Arasuna, Ryota Kan, Taro Asada, and Masayoshi Tabuse

6.2 Results and discussion

When 5-second clip was used for narrowing the
10-second clip decided as the region to be authenticatedin the audio data, HAR was much improved, being
almost 100% for all the clipping conditions used in thisexperiment. Even for a clipping length of 4.0 seconds,
HAR improved to 100% except under one condition forclassical audio data: clipping from a starting time of 5
seconds from the beginning of the original audio data.
In the exceptional case, HAR was 76.25%.

Tables 1 and 2 respectively show the AR values for
AMFC and the HAR values for the proposed method. As
shown in Table 1, AMFC showed poor authentication
tolerance to clipping: clipping 0.01 seconds of audio
caused AR to decline to 65.88%. On the other hand, as
shown in Table 2, the authentication tolerance toclipping of the audio data was improved by adopting the
proposed method.

7. Conclusion

Table 1. AR with AMFC for clipping from the beginning oforiginal classical audio data.
Clipping length (s)0.00001
0.00005
0.00010.00020.00030.00040.00050.001
0.01

In general, there is a tradeoff between the quality ofwatermarked audio and the tolerance of watermarks tosignal processing methods, such as compression. To
overcome this inevitable tradeoff, we previously
developed an authentication method1 for digital audio
using a DWT. In the present study, we have improved
the method by determining the region to beauthenticated in the audio file by using the fundamental
frequency characteristics. The experimental results
show that the method has a high authentication
tolerance to clipping small parts from the audio data.

AR(%)
100
100
99.22
96.84
87.75
84.98
78.66
69.57
65.88

Table 2. HAR (%) with the proposed method under the 21
clipping conditions. Starting time for the test data from the
beginning of the original audio data: (a) 0, (b) 5, and (c) 10
seconds.
(a)length
(s)

0.01
0.1
0.5
1.0
2.0
3.0
4.0

classical
100
100
98.82
98.08
73.12
67.19
94.25

jazz
100
100
99.60
98.81
86.67
89.33
90.40

0.01
0.1
0.5
1.0
2.0
3.0
4.0

classical
95.62
96.16
82.75
91.78
86.03
79.73
64.38

jazz
98.82
94.12
89.02
80.63
74.67
72.19
71.76

0.01
0.1
0.5
1.0
2.0
3.0
4.0

classical
100
96.44
97.65
93.70
88.77
80.00
64.84

jazz
100
100
99.61
86.27
66.40
73.33
73.87

(b)

popular
100
100
99.17
92.24
99.17
100
95.01

rock
100
100
99.61
98.43
95.29
93.37
69.41

hiphop
100
85.49
78.04
80.63
80.63
80.63
71.76

rock
99.72
87.06
84.71
77.47
95.44
77.65
74.12

hiphop
94.90
90.91
83.40
75.29
73.52
86.17
73.12

rock
100
99.61
97.63
95.65
93.73
87.84
79.61

hiphop
100
100
99.60
94.47
88.54
91.70
76.28

Music

Clipping
length(s)(c)

popular
96.40
92.52
93.91
77.25
75.10
85.77
76.28

Music

Clipping(s)

1. Y.Yoshitomi, T.Asada, Y.Kinugawa, and M.Tabuse, An
authentication method for digital audio using a discrete
wavelet transform, J. Inf. Sec. 2(2) (2011) 59–68.
2. S. Murata, Y. Yoshitomi, and H. Ishii, Optimization ofembedding position in an audio watermarking method
using wavelet transform, in Abstracts of Autumn
Research Presentation Forums of ORSJ (Japan, Tokyo,
2007), pp. 210–211. (in Japanese)
3. D. Inoue and Y. Yoshitomi, Watermarking using wavelet
transform and genetic algorithm for realizing hightolerance to image compression, J. IEEJ 38(2)(2009)136–144.
4. T. Taniguchi and Y. Yoshitomi, Method for characterdomain extraction from image using wavelet transform, J.
Robotics, Networking and Artif. Life 2(1) (2015) 103–106.
5. R. Fujii, Y. Yoshitomi, T. Asada, and M.Tabuse, An
Authentication method using a discrete wavelet transform
for a recaptured video, J. Robotics, Networking and Artif.
Life 3(2) (2016) 107–110.
6. M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka,RWC music database: Database of copyright-cleared
musical pieces and instrument sounds for research
purposes”, Trans. IPSJ 45(3) (2004) 728–738. (inJapanese)Music

Clipping

lengthReferences

popular
100
100
98.42
96.08
91.70
80.29
79.45

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan192

Unified Approach to (1+1) EA on Discrete Linear Functions
Kenji Aoki

Information Technology Center, University of Miyazaki, 1-1 Gakuen-kibanadai-nishi
Miyazaki, 889-2192, Japan
Makoto Sakamoto

Faculty of Engineering, University of Miyazaki, 1-1 Gakuen-kibanadai-nishi
Miyazaki, 889-2192, Japan
Hiroshi Furutani

Innovative Computing Research Center, Doshisha University, 1-3 Tataramiyakodani
Kyoto, 610-0394, Japan
Satoru Hiwa and Tomoyuki Hiroyasu

Faculty of Life and Medical Sciences, Doshisha University, 1-3 Tataramiyakodani
Kyoto, 610-0394, Japan
E-mail: aoki@cc.miyazaki-u.ac.jp
www.miyazaki-u.ac.jp

Abstract
We consider the runtime property of discrete linear functions in (1+1) Evolutionary Algorithms, PO-mutation andJansen’s model, PO-EA. We analyze the process of evolution. This study was motivated by the paper of Jansen
treating the runtime property of (1+1) EA on monotonic functions by means of probabilistic theory. As linear
functions are special case of monotonic function, we analyze their behavior. We show that the (1+1) EA can obtainan optimum solution at a stable hitting time for discrete liner functions. When the mutation rate is weak, on the orderof 1 / l, most monotonic functions behave similarly and can be approximated well by the PO-mutation model.
Keywords: Evolutionary algorithm, Discrete linear function, (1+1) EA, PO-mutation, PO-EA

monotonic submodular functions have recently beenstudied extensively in computer science. This paper
focuses on the monotonic functions, leaving thesubmodularity for the future trial.
Jansen [3] introduced the partially ordered
evolutionary algorithm (PO-EA) model and analyzed its
performance in linear functions. Later, this algorithm was
expected to simulate the optimization of monotonic
functions [4, 5]. PO-EA is a pessimistic model of the true
optimization process. One may consider PO-EA as amodel to derive an upper bound on the expected hitting
time for monotonic functions.
In this study, we analyze the expected hitting timeof (1+1) EA, PO-mutation and PO-EA. Without loss ofgenerality, we assume the optimum binary string is {1}l,

1. Introduction
Discrete optimization covers one area of optimization forthe mathematical and applied aspects of integer
programming and combinatorial optimization. Usually,
most studies deal with each optimization problem
individually. However, there are some researchers who
try to capture several properties of optimization function
f(x) that make it tractable. In the continuous optimization,a function can be minimized efficiently, if it is convex [1].
On the other hand, a function is maximized efficiently, if
it is concave.
In the area of discrete optimization, there are suchproperties which make optimization tasks easier. One ofthem is submodularity, which has many interesting
properties [2]. Another one is monotonicity, and© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan193

Kenji Aoki, Makoto Sakamoto, Hiroshi Furutani, Satoru Hiwa, Tomoyuki Hiroyasu

and use EAs on discrete liner functions as arepresentative monotonic function. Colin et al. haveshown that PO-EA can be divided into two parts [5],which named PO-mutation and ZeroMax models. POmutation model contains mutations which increase thefitness of every monotonic function. By using computer
experiment, we calculate the expected hitting times of
optimum for (1+1) EA, PO-mutation and PO-EA on
discrete linear functions. By this analysis, we can see thecontributions of (1+1) EA, PO-mutation and PO-EA athitting time, and understand the mechanism of the
optimizing process of EAs.

Harmonic number. The runtime of this chain isapproximately given by l log(l) if we use pm = 1 / l.
3. Evolutionary Algorithms
3.1. (1+1) Evolutionary Algorithm
Evolutionary Algorithm (μ + λ) came from Evolutionary
Strategy developed by Rechenberg and Schwefel, where
μ and λ are numbers of parent solutions and offspring
solutions, respectively. We choose the parameters μ = 1
and λ = 1 for simplicity. Though (1+1) EA seems too
simple for an analysis, it has attracting properties. As
Wegener states that [2],(1) it is efficient for many problems,(2) it can not get stuck in a local optimum, and(3) the analysis of it reveals many tools that can be usedin more practical algorithms.
The algorithm of (1+1) EA is given by Algorithm
1.
Algorithm 1(1+1) EA
1: Initialize ∈ { , } uniformly at random.
2: Createby flipping one each bit in withprobability.
3: Select if ( ) ≥ ( ) then ∶= .
4: Go to 2 until a termination condition is fulfilled.

2. Mathematical models of EAs
In discrete optimization, mathematical models forevolutionary algorithms (EAs) have attracted the interestof many researchers [6]. For the theoretical analysis of
EAs, one usually chooses (1+1) EA as the first trial. A
detailed overview of these studies are presented in [7].
This paper treats the optimization of monotonic functions
within the framework of evolutionary algorithms.
If we are given a ground set L = {1, 2, …, l} and a
set function f: A → R for all subsets A ⊆ L, the monotonic
function has the property of f(A) ≤ f(B) if A ⊆ B and B⊆ L. This function covers a wide range of discrete
optimization. A set function can be equivalently
described by pseudo-Boolean function. For example,
with l = 5, a set A = {1, 2, 4} is equivalent to a binary
string (1, 1, 0, 1, 0). We use the notation of binary stringsin the following. OneMax function is one of mostfrequently studied monotonic functions. Mülenbein
carried out a pioneering study by using Markov chain foranalyzing the time evolution of states in EA [8]. He
derived the expected hitting time T of the optimum for
OneMax function on (1+1) EA. For large string length land mutation rate pm = c / l, the result is approximately
given by T ≈ ec / c ⋅ l ln(l / 2).
Furutani et al. also analyzed the expected hitting
times of EAs by applying Markov chain and random
local search (RLS) [9]. The behavior of this Markov
chain is completely determined by the transition matrix P
of an absorbing Markov chain [10]. The explicit form ofabsorbing Markov chain for OneMax problem ispresented in [9]. Using this expression of P, we showedthat the expected runtime of RLS algorithm is H1 / pm
steps. Here, pm is mutation rate, and H1 is the first

3.2. PO-mutation Algorithm
Colin et al. have shown that PO-EA can be divided into
two parts [5]. Ma et al. have named it PO-mutation and
ZeroMax models [11]. They treated two selection
conditions, ( ≥ ) and {( ≰ ) AND ( ( ) ≤( ))}, separately. We call the first condition as POmutation model, and the second condition as ZeroMax
model, respectively. PO-mutation has a role of driving
force for approaching the optimum solution. On thecontrary, ZeroMax model works as a resistance against
the move to the optimum. Please note that ZeroMax
model has a condition ( ≰ ), which makes its powersmaller than OneMax model. The algorithm of POmutation is given by Algorithm 2.
Algorithm 2
PO-mutation
1: Initialize ∈ { , } uniformly at random.
2: Createby flipping one each bit in withprobability.
3: Select if ≥ then ∶= .
4: Go to 2 until a termination condition is fulfilled.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan194

Unified Approach to (1+1)

It should be mentioned that the PO-mutation model
is designed to study the convergence properties of POEA. Since the selection process of Algorithm 2 assumes
that {1}l is the optimum string, this model cannot beapplied to real problems whose optimum strings are notknown.

s = 0.0
0.002(1+1)EA
PO-m

Probability

0.00153.3. PO-EA
Jansen introduced the partially ordered evolutionary
algorithm (PO-EA) model [3]. The algorithm of PO-EA
is given by Algorithm 3.
Algorithm 3
PO-EA
1: Initialize ∈ { , } uniformly at random.
2: Createby flipping one each bit in with.
probability
3: Select if ( ≥ ) OR{( ≰ ) AND ( ( ) ≤ ( ))} then ∶= .
4: Go to 2 until a termination condition is fulfilled.
The difference of three algorithms are Selection in
the step 3.

PO-EA

0.001

0.00050Generations

Fig. 1. Distribution of the first hitting time of optimum solutionin (1+1) EA, PO-mutation and PO-EA with s = 0.0. The solidline is the result of (1+1) EA calculation. The dotted line is theresult of PO-mutation calculation. The dashed line is the resultof PO-EA calculation.

4. Discrete Linear Function
As a test function, we apply discrete linear function f(x)( )=+s = 1.0

+ ⋯+

0.002

( )=∑
,
∈ {0,1},(1)= (1 + ) ,0 ≤ ≤ 1,
where x is a binary string of length l. We consider the
maximization problem of this function. The optimum
solution is xopt = {1}l. If = 0, then = 1 and ( ) =∑. That is to say OneMax function. If = 1, then
= 2 , i.e. binary number.
Since many studies suggested that the mutation
probability of pm = 1 / l may be the best choice, we carriedout our analysis using this value.

(1+1)EA
PO-m

0.0015

Probability

PO-EA
0.001

0.0005

5. Numerical Experiment0

In this section, we compare the first hitting time of thenumerical experiments of (1+1) EA, PO-mutation and
PO-EA with s=0.0, 1.0 and 0.5. We used the mutation
rate pm = 1 / l . The length of string is l = 100, and we
performed 10,000 runs for each parameter set, andaveraged over them.
Figure 1 shows the time dependence of probabilityof the first hitting time of optimum solution in (1+1) EA,
PO-mutation and PO-EA with s = 0.0. These lines are theGenerations

Fig. 2. Distribution of the first hitting time of optimum solutionin (1+1) EA, PO-mutation and PO-EA with s = 1.0. The solidline is the result of (1+1) EA calculation. The dotted line is theresult of PO-mutation calculation. The dashed line is the resultof PO-EA calculation.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

195

Kenji Aoki, Makoto Sakamoto, Hiroshi Furutani, Satoru Hiwa, Tomoyuki Hiroyasu

Fig. 1. The mean of the first fitting times in (1+1) EA,
PO-mutation and PO-EA are 1050, 1018 and 1280,
respectively.

s = 0.5
0.002(1+1)EA

Probability

0.0015PO-m

6. Summary

PO-EA

In this paper, we studied time behavior of (1+1) EA, POmutation and PO-EA to understand how evolutionary
computation works for discrete liner functions. We
demonstrate that the (1+1) EA can obtain an optimum
solution at a stable hitting time for discrete liner functions.
When the mutation rate is weak, on the order of 1 / l, mostmonotonic functions behave similarly and can beapproximated well by the PO-mutation model. Thesestudies can help to understand the working mechanism of
Evolutionary Algorithms, and give some suggestions todesign algorithms for other problems.

0.001

0.00050Generations

References

Fig. 3. Distribution of the first hitting time of optimum solutionin (1+1) EA, PO-mutation and PO-EA with s = 0.5. The solidline is the result of (1+1) EA calculation. The dotted line is theresult of PO-mutation calculation. The dashed line is the resultof PO-EA calculation.

1. Boyd S, Vandenberghe L (2004) Convex optimization.
Cambridge University Press, Cambridge
2. Fujishige S (2005) Submodular functions and optimization,
2nd edn. Elsevier, Amsterdam
3. Jansen T (2007) On the brittleness of evolutionary
algorithms. Foundations of Genetic Algorithms 2007.
Springer, Lecture Notes in Computer Science, New York,
pp 54–69 (4436)
4. Doerr B, Jansen T, Sudholt C, Winzen C, Zarges C (2013)
Mutation rate matters even when optimizing monotonefunctions. Evol Comput 21:1–12
5. Colin S, Doerr B, Férey G (2014) Monotonic functions in
EC: anything but monotone! In: Proceedings of the 2014
annual conference on genetic and evolutionary
computation, pp 753–760
6. Auger A, Doerr B (eds) (2011) Theory of randomized
search heuristics. World Scientific, Singapore
7. Oliveto PS, Yao X (2011) Runtime analysis ofevolutionary algorithms for discrete optimization, chapter
2. Theory of randomized search heuristics. World
Scientific, Singapore, pp 21–52
8. Mülenbein H (1992) How genetic algorithms really work
I. Mutation and hillclimbing. Parallel Probl Solving Nat
II:15–26
9. Furutani H, Tagami H, Sakamoto M, Du Y (2014) Markov
chain analyses of random local search and evolutionary
algorithm. J Robot Netw Artif Life 1:220–224
10. Iosifescu M (1980) Finite Markov processes and theirapplications. Wiley, New York11. Ma Q, Zhang Y, Yamamori K, Furutnai H, Hiwa S andHiroyasu T (2018) Markov chain analysis of evolutionary
algorithms for monotonic functions. Artif Life Robot

results of numerical calculation. The result is moving
averaged with window size of 30. The dotted line is theresult of PO-mutation calculation. The dashed line is theresult of PO-EA calculation. In Fig.1, the waveforms of(1+1) EA and PO-mutation are similar, but PO-EA isbehind the other. The mean of the first fitting times in(1+1) EA, PO-mutation and PO-EA are 1069, 1014 and1976, respectively.
Figure 2 shows the time dependence of probabilityof the first hitting time of optimum solution in (1+1) EA,
PO-mutation and PO-EA with s = 1.0. The dotted line is
the result of PO-mutation calculation. The dashed line is
the result of PO-EA calculation. In Fig. 2, the peak of thefirst hitting time is delayed in order of PO-mutation,(1+1) EA and PO-EA. The mean of the first fitting timesin (1+1) EA, PO-mutation and PO-EA are 1136, 1008
and 1295, respectively.
Figure 3 shows the time dependence of probabilityof the first hitting time of optimum solution in (1+1)
EA, PO-mutation and PO-EA with s = 0.5. The dottedline is the result of PO-mutation calculation. The dashedline is the result of PO-EA calculation. In Fig. 3, as in Fig.
1, the waveforms of (1+1) EA and PO-mutation are
similar. PO-EA is behind the other, but it is not as late as© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan196

An fNIRS study of brain state during letter and category fluency tasks
Akane Onishi,
Hiroshi Furutani, Tomoyuki Hiroyasu and Satoru Hiwa

Faculty of Life and Medical Sciences, Doshisha University, 1-3 Tatara Miyakodani
Kyotanabe-shi, Kyoto-fu 610-0394, Japanshiwa@mail.doshisha.ac.jp

Abstract
Verbal fluency tasks (VFT) can be categorized into the letter fluency task (LFT) that evaluates the ability to generatewords starting with a particular letter, and the category fluency task (CFT) that involves recall of words that belong
to semantic categories. In this study, we investigated the differences in brain activity during LFT and CFT in healthy
subjects. Brain activity in the frontal and the temporal region was measured using functional near-infrared
spectroscopy (fNIRS) with 40 channels. We observed more prominent brain activity when performing LFT than
CFT.
Keywords: Verbal fluency task, letter fluency task, category fluency task, fNIRS, brain activity

vocabulary search power, information processing speed,
and executive function3. There are several studies onbrain activity differences in LFT and CFT of healthy
subjects and patients with a mental illness4,5,6,7,8.
However, only a few studies compare LFT and CFT as
such. In this study, brain activities during LFT and CFT
were compared in healthy subjects. We hypothesized adifference in brain activity between LFT and CFT and
therefore, the purpose of the current study was toinvestigate the brain status of the two versions of the VFT
over a wide area of the frontal and bilateral temporalbrain regions using 40 channels of fNIRS.

1. Introduction
Functional near-infrared spectroscopy (fNIRS) is atechnique that measures metabolic changes inhemoglobin oxygen in vivo, using near-infrared light.
fNIRS detects changes in concentration of oxyhemoglobin, deoxy-hemoglobin, total hemoglobin, andmeasures alteration in cerebral blood flow caused byneural activity1,2. Brain function measurement using
fNIRS is noninvasive, has excellent temporal resolution,and is performed under routine conditions. In recent
times, fNIRS as a tool in advanced medicine is attracting
attention. fNIRS examination using verbal fluency task
(VFT) is widely being used in diagnosis of depression
symptoms. VFT is a task that requires to generate as
many words within a category or starting with a given
letter as possible within a time limit. In the letter fluency
task (LFT), words are recalled from particular letter. In
the category fluency task (CFT), words are recalled fromsemantic categories. The characteristics of brain activity
in VFT are different in depression, bipolar disorder andschizophrenia, therefore, diverse activating tasks are used
for different disorders. It has been pointed out that VFT
tests not only the semantic memory related to vocabulary
but also many cognitive functions such as effective

2. Material and methods
2.1. Study participants
Healthy subjects (average age 22.35 ± 0.93, 17 males, 3
females, right-handed) were recruited for this study. All
the participants performed LFT and CFT, and brain
activity during each task was measured using fNIRS.
The study was approved and conducted as per the ethicalguidelines of the Institue.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan197

All Author’s Full Names

2.2. Experiment procedure
In the experiment, a block-design VFT (letter andcategory version) was performed. The experimental
design is shown in Fig. 1. It consisted of 60 seconds ofpre-task rest followed by 60 seconds of task period.
Participants were initially instructed to watch a cross at
the center of the computer screen during the experiment.
The task section began with three tasks (a, ka, shi /animals, fruits, sports). This was explained to theparticipants by the computer as audible instructions. The
content of the task was determined from the frequency ofuse of letters and previous studies8,9. During LFT,
participants were instructed to generate as many words as
possible starting with the displayed letters, while during
CFT, participants were instructed to generate as many
words which would fit in the presented category. The twotasks were done consecutively and the participants wereexplained about the order at the beginning of theexperiment. The total number of correct words generated
during the task was recorded as an indicator of VFT
performance.
task a

rest
60s

task
60s

task karest
60s

task animal

task
60s
task fruit

Fig. 2. Experimental set-up

2.4. Data processing
fNIRS data was band-pass filtered using 0.01-0.1 Hz.
fNIRS data from 9 participants were excluded due to avariation of 0.01 mM*cm in 0.1 seconds. fNIRS data waswere band-stop filtered (physiological noise removal)using 0.12 - 0.35 Hz, and with 0.7 - 1.5 Hz after artifact
correction based on moving standard deviation and splineinterpolation. Next, trend removal was performed using
a high pass filter based on the discrete cosine transformset. General linear model (GLM) analysis was used forextraction of the active regions in the brain. Using thehemodynamic response function (HRF), a blood flow
alteration model was created as shown in Fig. 3.
Statistical significance in brain activity between the twogroups was examined using the regression coefficientobtained by regression analysis of blood flow alteration
model and measurement data. All the calculations andmodeling was done using the SPM-fNIRS toolbox.

task shi

rest
60s

task
60s
task sport

rest
60s
time

Fig. 1. Experimental design

2.3. NIRS instrument*

In this study, a 40-channel fNIRS device (LABNIRS,
Shimadzu Corporation, Kyoto) was used. The distance
between the irradiation probe and the light receiving
probe was set to 3.0 cm. In order to minimize the artifacts,
the participants were instructed to avoid voluntary
movements. The experimental set-up is shown in Fig. 2.
The sampling frequency was set at 37.037 Hz. After dataacquisition, three-dimensional (3D) coordinates of eachlight source and detector were measured using a 3D
digitizer (FASTRAK, Polhemus, Vermont, USA).
Subsequently, these were input to the SPM-fNIRS
toolbox for spatial registration, and the layout of eachchannel and Montreal Neurological Institute (MNI)
coordinates was created.

Experiment model=
Hemodynamic
response function

Blood flowchange modelFig. 3. Creation of blood flow alteration model

3. Results
Brain activation analysis was performed with 11 fNIRS
data. Regions with maximum activity for LFT and CFT
are represented in Table. 1 and 2, respectively. Groupanalysis results are shown in Table. 3 and 4. Brain
activity observed during LFT performance was higher
than during CFT.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan198

Paper Title (First 4 Words)

Table. 4. Group analysis result (CFT)(FWE, p<0.05)

Table. 1. Activated brain region (LFT)(FWE, p<0.05)
RegionRegionNumber of people

Precentral L
Precentral R
Frontal Sup L
Frontal Sup R
Frontal Mid L
Frontal Mid R
Frontal Mid Orb L
Frontal Inf L
Frontal Inf R
Postcentral L
Postcentral R
SupraMarginal L
SupraMarginal R
Angular R
Temporal Sup L
Temporal Sup R
Temporal Mid L
Temporal Mid R
Temporal Inf L

Frontal Inf Tri R
Caudate R

10
10
8
8
10
8
7
10
10
9
9
8
9
7
8
8
8
6
7

4. Discussion
In this study, brain activation was investigated based on
the change in hemoglobin concentration measured during
VFT performance in healthy subjects. An extensive
activation in the frontal and temporal brain regions
including Broca’s areas was observed during the VFT
performance. In particular, LFT augmented the brain
activation compared to CFT. During CFT, participants
would search for words by utilizing the concept ofassociations such as already existing vehicles. On theother hand, in LFT, words were searched from the notionof phonology which is rarely used in daily life. In otherwords, participants suppress the recall of related words
and perform complex word searches10. The results of thisstudy suggest that LFT requires more activation of brain
regions than CFT. As shown in Table 1 and 2, four
regions are activated exclusively during LFT. The
regions include angular gyrus R (R-ANG), middle
temporal gyrus R (R-MTG), superior temporal gyrus R(R-STG), and orbital part of inferior frontal gyrus L (LORB) as shown in Fig. 4. Amongst these four regions,
ANG is located at the junction of the occipital, temporal,
and parietal regions and is regarded as an importantinterface for communicating and integrating information
with various brain regions11,12. Several studies haveassociated ANG with memory, meaning, memory
consciousness, default mode network (DMN)13. MTG,
on the other hand, is activated with a more difficult
phonological task and plays an important role incontrolling semantic search14. Co-activation of ANG and
MTG observed during LFT suggests their role in word
searches that suppresses related word remembrance.

Table. 2. Activated brain region (CFT)(FWE, p<0.05)
RegionNumber of people

Precentral L
Precentral R
Frontal Sup L
Frontal Sup R
Frontal Mid L
Frontal Mid R
Frontal Inf L
Frontal Inf R
Postcentral L
Postcentral R
SupraMarginal L
SupraMarginal R
Temporal Sup L
Temporal Mid L
Temporal Inf L

7
7
7
6
7
7
7
7
7
6
6
6
6
6
6

Table. 3. Group analysis result (LFT)(FWE, p<0.05)
RegionRolandic Oper L
Temporal Sup R
Frontal Inf Tri L
Frontal Inf Tri R

Fig. 4. Brain regions activated only during LFT© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan199

All Author’s Full Names

5. Conclusion
Currently, several ongoing studies are investigating thedifference in brain activity between LFT and CFT in
healthy subjects and patients with a mental illness.
However, there are only a few studies that compare LFT
and CFT as such. Therefore, in this study, the brain
activities during LFT and CFT were compared only inhealthy subjects. The results from 11 healthy subjects
established an increase in LFT activation compared to
CFT. Our results suggest that co-activation of ANG and
MTG makes it possible to search words with related word
recall suppressed.

9.

10.

11.

6. References

12.

1. S. Heinzel, F.G. Metzger, A.-C. Ehlis, R. Korell, A. Alboji,
F.B. Haeussinger, K. Hagen, W. Maetzler, G.W.
Eschweiler, D. Berg, et al., “Aging-related corticalreorganization of verbal fluency processing: a functional
near-infrared spectroscopy study," Neurobiology of aging,vol. 34, no. 2, pp. 439-450, 2013.
2. G. Strangman, D.A. Boas and J.P. Sutton, “Non-invasive
neuroimaging using near-infrared light," Biologicalpsychiatry, vol. 52, no. 7, pp. 679-693, 2002
3. C.E. Bokat and T.E. Goldberg, “Letter and category
fluency in schizophrenic patients: a meta-analysis,"
Schizophrenia research, vol. 64, no. 1, pp. 73-78, 2003.
4. A.-C. Ehlis, M.J. Herrmann, M.M. Plichta and A.J.
Fallgatter, “Cortical activation during two verbal fluency
tasks in schizophrenic patients and healthy controls asassessed by multi-channel near-infrared spectroscopy,"
Psychiatry Research: Neuroimaging, vol. 156, no. 1, pp.
1-13, 2007.
5. K. Ikezawa, M. Iwase, R. Ishii, M. Azechi, L. Canuet, K.
Ohi, Y. Yasuda, N. Iike, R. Kurimoto, H. Takahashi, et al.,“Impaired regional hemodynamic response inschizophrenia during multiple prefrontal activation tasks:
a two-channel near-infrared spectroscopy study,"
Schizophrenia research, vol. 108, no. 1-3, pp. 93-103,
2009.
6. Y. Kubota, M. Toichi, M. Shimizu, R.A. Mason, C.M.
Coconcea, R.L. Findling, K. Yamamoto and J.R.
Calabrese, “Prefrontal activation during verbal fluency
tests in schizophrenia a near-infrared spectroscopy (nirs)
study," Schizophrenia research, vol. 77, no. 1, pp. 65-73,
2005.
7. K. Marumo, R. Takizawa, M. Kinou, S. Kawasaki, Y.
Kawakubo, M. Fukuda and K. Kasai, “Functional
abnormalities in the left ventrolateral prefrontal cortex
during a semantic fluency task, and their association withthought disorder in patients with schizophrenia,"
Neuroimage, vol. 85, pp. 518-526, 2014.
8. C.-J. Huang, P.-H. Chou, H.-L. Wei and C.-W. Sun,
“Functional connectivity during phonemic and semantic
verbal fluency test: a multichannel near infrared

13.

14.

spectroscopy study," IEEE Journal of Selected Topics inQuantum Electronics, vol. 22, no. 3, pp. 43-48, 2016.
T. Hatta, A. Kanari, M. Mase, Y. Nagano, T. Shirataki and
S. Hibino, “Strategy effects on word searching in japaneseletter fluency tests: Evidence from the nirs findings,"
Reading and Writing, vol. 22, no. 9, pp. 1041-1052, 2009.
S.V. Tupak, M. Badewien, T. Dresler, T. Hahn, L.H. Ernst,
M.J. Herrmann, A.J. Fall-gatter and A.-C. Ehlis,“Differential prefrontal and frontotemporal oxygenation
pat-terns during phonemic and semantic verbal fluency,"
Neuropsychologia, vol. 50, no. 7, pp. 1565-1569, 2012.
P. Hagmann, L. Cammoun, X. Gigandet, R. Meuli, C.J.
Honey, V.J. Wedeen and O. Sporns, “Mapping thestructural core of human cerebral cortex," PLoS biology,
vol. 6, no. 7, p.e159, 2008.
D. Tomasi and N.D. Volkow, “Association between
functional connectivity hubs and brain networks,"
Cerebral cortex, vol. 21, no. 9, pp. 2003-2013, 2011.
M.L. Seghier, “The angular gyrus: multiple functions andmultiple subdivisions," The Neuroscientist, vol. 19, no. 1,
pp. 43-61, 2013.
J. Davey, H.E. Thompson, G. Hallam, T. Karapanagiotidis,
C. Murphy, I. De Caso, K. Krieger-Redwood, B.C.
Bernhardt, J. Smallwood and E. Jefferies, “Exploring therole of the posterior middle temporal gyrus in semantic
cognition: Integration of anterior temporal lobe withexecutive processes," NeuroImage, vol. 137, pp. 165-177,
2016.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

200

Construction of a meditation practice support
leads to a good meditation state: an fNIRS study
Seika Fujii*
Satoru. Hiwa, Hiroshi. Furutani, Tomoyuki. Hiroyasu

Faculty of Life and Medical Sciences, Doshisha University, 1-3 Tatara Miyakodani
Kyotanabe-shi, Kyoto-fu 610-0394, Japan
tomo@is.doshisha.ac.jp

Abstract
In this study, we constructed a system to quantify a meditation state based on brain function information and feedback
that is based on the functional near-infrared spectroscopy (fNIRS). In the system, the meditation state is fed back to
the user in real time. The system informs the user via sound when it judges that a good meditation is being performed.
The meditation experiment was conducted using this system. As a result, user’s brain state at meditation werevisualized.
Keywords: meditation, neurofeedback, real-time, dynamic functional connectivity analysis, fNIRS.

meta-state is determined by measuring the time series
obtained from multiple channels made by the fNIRS,
obtaining the correlation of time series data between
channels, and finally expressing it in a matrix. A
meditation meta-state derived from the data of apractitioner is defined, the similarity is obtained by dataof the brain state under meditation and real-timeprocessing, and by definition that meditation is done
better if the similarity is high, meditation state is fed backin real time. By doing this, users can objectively
recognize their meditation state, and it becomes possible
to experience the feeling of meditation. Features of theconstructed system exist in this feedback method. The
system alerts the user when the meditation is being
performed well while the user is in the process of
meditating. In other words, the system praises the user.
Although this mechanism inhibits meditation, the act ofreturning from a wandering state to a mindful state is
essential in mindfulness training. At the same time, bygiving feedback to users immediately after a good
meditation state, the motivation to continue with theirpractice is maintained. In the following sections, we
explain the analysis method used to construct the system,

1. Introduction
Kabat-Zinn defined mindfulness as "to actively pay
attention and consciousness without value judgment at
this moment"1. Meditation brings a person to a state ofmindfulness, which typically has the effect of reducing
stress and improving concentration2. These effects havebeen neuroscientifically verified using non-invasive
brain function imaging equipment such as functional
magnetic resonance imaging (fMRI)3,4. On the other hand,
it is not easy for a novice who has just begun meditation
to judge whether they are achieving a good or bad stateof meditation, and it is thus difficult to experience thoseeffects mentioned above. It would be beneficial if there
were a system that could support novices' meditation
practices and improve the effects of meditation. In thisstudy, we constructed a system to quantify the meditation
state based on brain function information and feedback
that is based on functional near-infrared spectroscopy(fNIRS), which is one of the non-invasive brain function
imaging devices used to study the effects of meditation.
In our previous study, we defined the meditation metastate as one that is peculiar to meditation. This meditation© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan201

Seika Fujii, Satoru. Hiwa, Hiroshi. Furutani, Tomoyuki. Hiroyasu

the creation method of the meditation meta-state, and thespecifications of the proposed system.

and a corresponding matrix is created. The meditation
meta-state is used as a reference indicator. The procedure
for calculating the meditation meta-state is as follows. To
measure brain activity, the amount of CBF change across
116 channels was measured using the ETG-7100 fNIRS
system (Hitachi, Ltd.). Data for five meditation
practitioners (45.6 ± 0.7 years old, cumulative practicetime 3,910 ± 2,228 hours) were used for a meditation
meta-state. The meta-state was constructed from changes
in CBF during their five-minute meditation practice.
Using dynamic FC analysis, the FC commonly seenamong all five practitioners was defined as the meditation
meta-state. A region with meta-state Fig. 2 was shown.
From Fig. 2, it is suggested that the meditation meta-state
indicates one’s state during meditation.

2. Proposed System
2.1. Functional Connectivity Analysis
The fNIRS device measures changes in cerebral bloodflow (CBF). If the changes in CBF between two regions
are measured and found to be similar, it is considered thatthose regions are functionally coordinated and active. In
this way, cooperative and active regions are nodes, andthe cooperativeness of brain function connectivity can be
represented as a network that has a thick line to connect
regions with a high degree of cooperation. As shown in
Fig. 1, the network representation can be put into theform of a matrix. This series of analyses is called
functional connectivity (FC) analysis. In previous studies
related to meditation, FC analysis is often used5. The FC
analysis is summarized as follows. First, a correlation
coefficient of the time series data between two regions is
calculated and correlation coefficients are calculated forall measurement points, and a correlation coefficientmatrix is created. The correlation coefficient indicates thedegree of cooperation. Finally, thresholding is performedon this correlation coefficient matrix and a FC matrix thatshows a stronger degree of cooperation is obtained.
Furthermore, the brain state of the user during meditation
changes from moment to moment4. Dynamic FC analysis
is used to capture the changing brain state to construct thesystem. In dynamic FC analysis, a fixed-length windowof time is used, and the FC analysis is performed on theamount of CBF that changes within that window6.
Concerning the width of this window, a certain overlap is
kept, and the FC analysis is similarly performed whilemoving the window on the time axis. As a result, adynamic brain function network is observed.

Fig. 2 Brain regions containing the mediation meta-state

2.3. Estimation of brain state during meditation
The change in the amount of CBF obtained using thefNIRS was analyzed and the meditation state of the user
was determined. Fig. 3 shows the procedure used for
estimating the state of the brain during meditation. A
window of 60 seconds was used to analyze the change in
the amount of CBF. First, physiological noise wasremoved by the moving average and FC matrix was
calculated. Then, in the matrix, Fisher's Z transformation
was performed so that the correlation coefficient becamea normal distribution, and an edge density of 15% was set
as a threshold value of the correlation coefficient. The
number of connection was kept constant. This matrix isdefined as the state of the brain during the meditation ofa user, and the similarity between this matrix and themeta-state was calculated. The degree of similarity wasset as a ratio (occurrence) so that the connection of themeta-state is included in the FC of a user.

Fig. 1 Process of functional connectivity analysis

2.2. Definition of meditation meta state
To quantify the meditation state of a user, a referenceindicator is necessary. In this study, the brain state
peculiar to meditation is called the meditation meta-state,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

202

Construction of a meditation

Fig. 4 System overview

3. Materials & Methods
Fig. 3 The process of estimating the meditation state

3.1. Participants

2.4. Feedback

The participants comprised four male (with a mean ageof 23.0 years, SD 0.8 years) who were novices inmeditation. Each subject was evaluated every three days
for four days at a time (range 2018/6/13 - 6/25; 12: 00 17: 00). Furthermore, the subjects were randomly divided
into two groups (two in each group). One group received
feedback from the system based on changes in theamount of CBF, and one group received feedback at
random moments (two each).

The system feeds information regarding the meditation
state back to the user. Feedback is important in that it
informs you when you are in an excellent meditation state.
In other words, it praises the user. To realize this,feedback is made available to the user if the following
conditions are simultaneously satisfied at the point thatthe meditation meta-state is attained:⚫
The current occurrence is larger than the one thatwas calculated immediately before.
⚫
The current occurrence is 1st or 2nd is largest among
the last five.
When the occurrence is higher than the one immediately
preceding, it indicates that the meditation state isimproved. Furthermore, within the most recent (5 times)
occurrences, the taming with relatively high occurrenceis calculated. When these conditions are met, it is judgedthat the user's meditating brain state has approached thetarget state and a sound is heard. The user is thus
informed that their meditative state has improved.

3.2. Protocol
The experimental design is shown in Fig. 5.

Fig. 5 Experiment protocol

Subjects responded to a questionnaire before and after theexperiment. A psychomotor vigilance task (PVT) was
also conducted before and after the meditation
experiment to measure their attention control abilities.
After the PVT, the meditation experiment was started. In
the meditation experiment. Participants kept their eyes
closed and the breath counting method was used.
Counting one’s breath is one of the meditation practices
that novices can easily do. The feedback sound identified
the state in which the last meditation was done, andparticipants were instructed to maintain that state.

2.5. Real-time system
We assumed that it is essential to provide feedback of themeditation state during the user’s meditation practice.
For that purpose, the changes in CBF are processed inreal time and fed back to the user. In the constructedsystem, brain function is measured by the ETG-7100, andthe processes of sections 2.3 and 2.4 are performed in realtime. Feedback is given to the user according to themeditation state determined by the processing result ofsection 2.3.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan203

Seika Fujii, Satoru. Hiwa, Hiroshi. Furutani, Tomoyuki. Hiroyasu

3.3. fNIRS data and behavioral acquisition

5. Conclusion

Changes in CBF across the whole brain 116 CH were
measured using the ETG-7100. PVT was used for
examining changes in the attention control abilities ofparticipants with or without the system. When a whitecircle was displayed at the center of a screen, theparticipants were instructed to press a key quickly.
Reaction time (RT) is the amount of time between the
moments when a white ring is shown and when theparticipant presses the key.

By practicing meditation, the effects of reduced stressand improved concentration are experienced. However,for a novice, it is difficult to recognize and affect one’s
state during meditation. Therefore, we constructed asystem that feeds back the meditation state to the user inreal time by using fNIRS. In the system, the state of brain
function during meditation is expressed as a network bya dynamic FC analysis. The data of experts was used asa reference indicator of a good meditation state, and it
was defined as the meditation meta-state. For the brain
state of a user, the extent to which the meditation metastate is contained was defined as an occurrence. By
setting conditions on the occurrence, the reachability ofthe target state was evaluated. A feedback system wasbuilt to inform the user via an audible sound when it
estimates that a good meditation is being performed. The
meditation experiment was then conducted using thesystem. As a result, one’s state at rest and one’s state at
meditation were visualized. In this experiment, there wasno difference between the feedback group and the control
group obtained. The system has been further modified,and new validation experiments need to be done.
References

4.

Results & Discussion

4.1. Occurrence of meditation meta stateFig. 6 shows a graph of the transition of occurrences inboth the feedback group and the false feedback groupwithin 10 minutes of the meditation experiment.

Subject A (Feedback group) Subject C False Feedback group)
Fig. 6 Changes in the occurrence of subject A and subject C:
the first to fourth measurements

4.2. RT in the PVT
Fig. 7 shows the relationship between RT of each PVT.
As shown in Fig. 7, there was no difference between thegroups. Kaul et al. reported that RT in PVTs improved bydoing meditation for 40 minutes7. In other words,
attention control abilities may be enhanced by meditation
with or without feedback.

Fig. 7 The relationship between the RTs of pre- and post-PVT of all
subjects

1. M.A. Killingsworth and D.T. Gilbert, “A wandering mind
is an unhappy mind,” Science, vol. 330, no. 6006, pp. 932–
932, 2010.
2. J. Smallwood and R.C. O’Connor, “Imprisoned by the
past: unhappy moods lead to a retrospective bias to mind
wandering,” Cognition & Emotion, vol. 25, no. 8, pp.
1481–1490, 2011.
3. Kabat-Zinn, Jon and Hanh, Thich Nhat, “Full catastrophe
living: Using the wisdom of your body and mind to facestress, pain, and illness,” Delta, 2009.
4. Y.-Y. Tang, B.K. Hölzel and M.I. Posner, “Theneuroscience of mindfulness meditation,” Nature Reviews
Neuroscience, vol. 16, no. 4, pp. 213-225, 2015.
5. W. Hasenkamp, C.D. Wilson-Mendenhall, E. Duncan and
L.W. Barsalou, “Mind wandering and attention during
focused meditation: a fine-grained temporal analysis offluctuating cognitive states,” Neuroimage, vol. 59, no. 1,
pp. 750-760, 2012..
6. V.D. Calhoun, R. Miller, G. Pearlson and T. Adalı, “Thechronnectome: time-varying connectivity networks as thenext frontier in fMRI data discovery,” Neuron, vol. 84, no.
2, pp. 262–274, 2014.
7. P. Kaul, J. Passafiume, R.C. Sargent and B.F. O'Hara,“Meditation acutely improves psychomotor vigilance, andmay decrease sleep need,” Behavioral and brain
Functions, vol. 6, no.1, p.47, 2010.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan204

Automated Panoramic Image Creation System for Corneal Endothelial Cells
Keitaro Kobayashi, Naoki Okumura, Noriko Koizumi,
Satoru Hiwa, Hiroshi Furutani, Tomoyuki Hiroyasu

Doshisha University, 1-46 Tataratodani Kyotanabe-shi,
Kyoto, 610-0394, Japan
tomo@is.doshisha.ac.jp
www.doshisha.ac.jp

Abstract
In this study, we propose a method for observing the quality of corneal endothelial cells for the entire cornea by
capturing panoramic images from motion pictures. Panoramic images were generated by concatenating the focused
images extracted from the moving images using the Tenengrad focus evaluation function. These panoramic images
were evaluated to confirm the usefulness of this approach. As a result, Improvements in the capacity to observechanges in the state of these cells could be observed.
Keywords: Panoramic image, corneal endothelial cell, focused image extraction, Tenengrad,

the body before transplantation4-6 or the use of eye drops7.
Currently, medical research on the use of eye drops isassessed by observing changes to the number ofendothelial cells before and after treatment using a
specular microscope. However, the imageable range ofthese observations is narrow and only the central part ofthe cornea can easily be evaluated despite the eye drops
affecting the entire cornea8. This study proposes amethod for observing changes to the number corneal
endothelial cells for the entire cornea by capturingpanoramic images from motion pictures. The utility ofthis system for evaluating changes to corneal disorders is
also assessed.

1. Introduction
One cause of corneal disorders is endothelial cell
reduction1. These cells do not have a regenerative
function2. Therefore, if the number of these cells isdecreased through trauma or illness, a corneal disorderoccurs, which results in declining vision for the patient.
The primary treatment for this disorder is a corneal
transplants sourced from a donors. One problem,
however, is an insufficient number of donors. Primary
treatment for corneal disorders involves either thetransplantation of all corneal layers, or only theendothelium4. Both methods require a corneal donor. In
Japan, there are currently about 20,000 people with acorneal disorder but the annual number of corneal
transplants is limited to 2,800 and 1,200 of these use
imported corneas. Due to this chronic donor shortage,
there is an interest in treatments that regenerate corneal
endothelial cells, without the need for a donor3 This
involves either treatment by culturing these cells outside

2. Panoramic image creation
The proposed system captures panoramic images using
moving images of the corneal endothelial cells. This
makes it is possible to observe the condition of all cells.
Fig. 1 shows the procedure used. First, video images wereobtained of the corneal endothelium using a contact type© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan205

Keitaro Kobayashi, Naoki Okumura, Noriko Koizumi, Satoru Hiwa, Hiroshi Furutani, Tomoyuki Hiroyasu

2.2. Image stitching

specular microscope. Second, focused images wereextracted from the moving images. Finally, the images
were combined using software called AutoStitch™ 9 togenerate a panoramic image.

AutoStitch™ is free software that automaticallygenerates panoramic images from input images. The
Scale-Invariant Feature Transform (SIFT) was used to
extract features when aligning these images. There were
two options for the synthesis algorithm, including linear
or multiband blending. Multiband blending, which is apyramid-based method, was used for this study.
2.3. Experimental Method
Ten corneal endothelial videos (size: 720 × 480 pixels,
frame rate: 30 fps, magnification: 200 times) were
captured from mice provided by the tissue engineeringlaboratory of the Graduate School of Biomedical
Sciences, Doshisha University. These videos followed an
S-shaped path using a contact type specular microscope.
First, still images (size: 720 × 480 pixels) were extracted
from each video at 30 frames per second. The extracted
still images were then labeled using the Tenengrad focus
evaluation function and divided equally into 50 groups.
Images with the highest function value in each groupwere then extracted as focused images. A panoramic
image was then generated by linking these images with
AutoStitch™. This panorama was then observed toassess the usefulness of this approach.

Fig. 1. Proposed system
2.1. Image extraction
The panoramic image was generated by concatenating
the focused images extracted from the moving images. In
general, the edges of the captured objects in the focused
images were unclear due to the shaking of the camera,
which causes blurring. Therefore, images were extractedusing the Tenengrad focus evaluation function10 (Eq. (1)).
.
(, ) + (, )(1)
, = ∑( , )∈ ( , )Where, Gx and Gy are image gradients in the x and y
directions, respectively, calculated by convolving thetarget image with the Sobel operator. This filter
emphasizes specific areas of the image that undergo asudden change in brightness. Fig. 2 shows that a higher
value for the Tenengrad focus evaluation function relates
to images with clearer edges. In contrast, Fig. 3 showsthat the value of the function is smaller for blurred images.

2.4. Result and discussion
Figs. 4 and 5 show examples of panoramic images
generated by this method. In Fig. 4, it was difficult toobserve changes to the state of the cells in the upper left
section of the image. This was because an overexposed
image (Fig. 6) was extracted as one of the focused images.
Thus, a focused image with a lower Tenengrad focus
evaluation function value was substituted to generate a
new panoramic image (Fig. 7). This confirms that it ispossible to link alternative images if one is overexposed.
An image with ≥ 1000 pixels, having a brightness value≥ 250 was determined to be overexposed. However,difficulties still remained when attempting to observechanges to the state of the cells at the time of linking.
Thus, the image in Fig. 7 was flattened using histogram
equalization. This was shown to be useful for observing
the changes to the cells. Fig. 5 shows another example
where excluding the lower right of the imaged cornea
had been connected. It was confirmed using Fig. 9 thatthe state of the cell was only partially extracted in theFig. 2. Tenengrad focus evaluations for focused images

Fig. 3. Tenengrad focus evaluations for blurred images© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan206

Automated Panoramic Image Creation

focused image. Fig. 10 shows the same panorama
generated after manually deleting the focused image. The
improvement in the clarity of the cells that can beobserved when comparing Fig.8 and Fig.10 suggests that
this is a useful and necessary method for extracting
focused images that can observe changes in the state of
all cells.

Fig. 7. Panoramic image 1 after deleting theoverexposed image

Fig. 4. Panoramic image 1

Fig. 8. Panoramic image 1 after histogram equalization

Fig. 5. Panoramic image 2

Fig. 9. A partially focused image

Fig. 6. Overexposed image© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan207

Keitaro Kobayashi, Naoki Okumura, Noriko Koizumi, Satoru Hiwa, Hiroshi Furutani, Tomoyuki Hiroyasu

7

8

9.
10.

Koizumi, N. et al. Rho-associated kinase inhibitor
eye drop treatment as a possible medical treatment for
Fuchs corneal dystrophy. Cornea 32, 1167-1170,
doi:10.1097/ICO.0b013e318285475d (2013).
Tanaka, H. et al. Panoramic view of human corneal
endothelial cell layer observed by a prototype slitscanning wide-field contact specular microscope. Br
J Ophthalmol, doi:10.1136/bjophthalmol-2016308893 (2016).
M. Brown and D. Lowe, Automatic Panoramic Image
Stitching using Invariant Features, International
Journal of Computer Vision, 74(1) (2007) 59-73
J. Tenenbaum, Accommodation in Computer Vision.
Stanford University PhD. Thesis (1970)

Fig. 10. Panoramic image 2 after manually
deleting the partially focused image
3. Conclusion
In medical research examining the effectiveness of thetreatment of corneal disorders with eye drops, there is aproblem when attempting to evaluate images of sections
other than the central portion of the cornea. Therefore, amethod for capturing a panoramic image from moving
images was proposed. To confirm the usefulness of thisapproach, these panoramic images were evaluated.
Improvements in the capacity to observe changes thestate of these cells could be observed as a result. Thus, it
is concluded that this method is useful.
References
1

2
3
4

5
6

Gain, P. et al. Global Survey of Corneal
Transplantation and Eye Banking. JAMA Ophthalmol
134,
167-173,
doi:10.1001/jamaophthalmol.2015.4776 (2016).
Joyce, N. C. Proliferative capacity of the corneal
endothelium. Prog Retin Eye Res 22, 359-389,
doi:S1350946202000654 [pii] (2003).
Tan, D. T., Dart, J. K., Holland, E. J. & Kinoshita, S.
Corneal transplantation. Lancet 379, 1749-1761,
doi:10.1016/S0140-6736(12)60437-1 (2012).
Koizumi, N. et al. Cultivated corneal endothelial cellsheet transplantation in a primate model. Invest
Ophthalmol Vis Sci 48, 4519-4526, doi:48/10/4519[pii]10.1167/iovs.07-0567 (2007).
Okumura, N. et al. Rho kinase inhibitor enables cellbased therapy for corneal endothelial dysfunction. Sci
Rep 6, 26113, doi:10.1038/srep26113 (2016).
Kinoshita, S. et al. Injection of cultured cells with aROCK inhibitor for bullous keratopathy. N Engl J
Med 378, 995-1003 (2018).
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan208

An fMRI study of the inhibitory effects of
the random stimulus-response compatibility task on brain function
Kei Sahara, Hiroshi Furutani, Tomoyuki Hiroyasu, and Satoru Hiwa

Faculty of Life and Medical Sciences, Doshisha University, 1-3 Tatara Miyakodani
Kyotanabe-shi, Kyoto-fu 610--0394, Japanshiwa@mail.doshisha.ac.jp

Abstract
The Stimulus-Response Compatibility (SRC) task is typically used in a congruent control study. In this study, weused 3 conditions to investigate the differences between inhibition processes in incidental events. Congruent andincongruent conditions are common in SRC tasks. However, the incidental conditions are not investigated. We
hypothesize that the random condition has the strongest inhibition processes in congruent control processes. We used
fMRI to investigate brain function and analyzed the data using activation analysis and behavior analysis.
Keywords: Stimulus-Respond Compatibility task, functional Magnetic Resonance Imaging, Behavior analysis,
Activation analysis

activation analyses of SRC tasks. SRC tasks have several
styles, each using a different type of stimulus (e.g.,
numerical association target, pitch association target)5. In
this study, we used a spatial SRC task using a visual
stimulus. A spatial SRC is known as a psychologicalphenomenon in which an overlap of spatial properties
between stimuli and responses leads to faster and moreaccurate responses6. Furthermore, we used a 3rd condition,a random condition which contains both SRC task
conditions (congruent, incongruent), to investigate
inhibitions during incidental events. Therefore, in thisstudy, we hypothesized that the 3rd condition, the random
condition, has the strongest inhibiting effect in allconditions. We used behavior analysis, activation
analysis, and connecting analysis to investigate thedifferences among the three conditions.

1. Introduction
To successfully interact with our environment, we mustselect appropriate behavioral responses given our current
environmental situation and particular goals1. Cognitive
control processes include a series of mechanisms (e.g.,working memory, attentional control, inhibitory control,
cognitive flexibility, problem-solving, and planning) thatallow flexible and adaptive behavior in response toenvironmental demands by implementing specific goaldirected action2. The stimulus-response compatibility(SRC) paradigm is typically used in the study ofcognitive control3. SRC tasks usually require inhibitionof irrelevant features specified in task instructions, whichindicate the required response4. In general, the SRC task
contains 2 conditions: a congruent condition and anincongruent condition. There are a number of previous
studies investigating various behavior analyses and© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

209

Kei Sahara, Hiroshi Furutani, Tomoyuki Hiroyasu, Satoru Hiwa

2. Methods
2.1. Subjects
A total of 20 healthy young adults (age range: 21-25
years, average age: 22.3±1.05, 4 females) participated inthis fMRI study. One participant was excluded due toinaccurately responding during multi-tasks. Therefore,
the final number of participants was 19 (age range: 2125, average age: 22.26±1.07, 4 females). Eachparticipant’s dominant hand was established by the
Edinburgh handedness inventory (Oldfield, 1971).
Seventeen participants were right-handed and 2
participants were ambidextrous. Before participating inthe study, all participants received information andinstruction about the protocol.

Fig. 1 Experimental protocol: (A) Congruent andincongruent condition session, (B) Random condition

2.4. fMRI data acquisition
Images were acquired with a 1.5 Tesla MRI scanner(Hitachi Echelon Vega). Functional images were imagedby gradient-echoplanar imaging (EPI) pulse sequence(TR = 2500 ms, TE = 40 ms, FA = 85°). Images werepreprocessed by SPM12 (ASHBURNER, John, et al.
SPM12manual. Wellcome
Trust
Centrefor
Neuroimaging, London, UK, 2014.), which included
realignment for correction of head movements, slice
timing correction, co-registration for adjusting thefunctional and structural images, normalization to adjust
the EPI images into normalized MRI brain images, andsmoothing of activation images. The results were labeled
by Automated Anatomical Labeling (AAL).

2.2. Stimuli
There were 2 visual target stimuli, which consisted ofblue and red dots. The blue dot indicated the congruent
task, for which participants were instructed to respond to
the target stimulus with their ipsilateral hand. The red dot
indicated the incongruent task, for which participants
were instructed to respond to the target stimulus with
their contralateral hand. The target stimuli appeared on
both sides of the fixation stimulus during the experimentand the participants responded by pressing buttons on apad.

2.5 Behavior data analysis
The behavior data during the experiments were analyzed
in MATLAB (Guide, MATLAB User’s. "Themathworks." Inc., Natick, MA 5 (1998): 333.). We
compared the mean reaction times (RT) among the 3
conditions (congruent, incongruent, and random) byusing an ANOVA test (P<0.001) and Bonferroni test.

2.3. Experiment protocol
In this study, we had 3 tasks: the congruent task, theincongruent task, and the random task.. The congruent
and incongruent tasks contained only blue or red dots,
respectively, as the target stimuli. The random task
contained both blue and red dots as the target stimuli. The
3 tasks were divided into 2 sessions. Session 1 contained
6 task blocks, and only one task (congruent/incongruent)
was presented per block. The task appeared 3 times
randomly throughout the session. Session 2 contained 4task blocks, which contained all random tasks(congruent). In both sessions, there were 20 trials in eachblock, and the target stimuli were presented for 1000 ms,
with an inter-stimulus interval (ISI) of 1500 ms. During
the 30000 ms rest periods participants were instructed tostare at the fixation stimulus.

2.6 fMRI data analysis
The functional brain images were analyzed by SPM12.
For one level analysis we fit the preprocessed image data
into a general linear model (GLM) for fMRI model
specification. After model estimation, we compared thecongruent, incongruent, and random conditions to therest condition and each conditions. We then ran a 2 level
analysis to compare each condition to the others toestimate the differences of inhibition in each condition.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan210

An fMRI Study of

3. Results and Discussion s
3.1. Behavior data analysis
In the behavior data analysis, a one-way between subjects
ANOVA was conducted to compare the effect of mean
RTs among the congruent, incongruent, and random
conditions (Figures 2 and 3). There was a significant
effect of RT for all 3 conditions [F (2,187) = 65.75,
p<0.001].

congruent vs
rest

incongruent vs
rest

incongruent vscongruent

Precentral_R
Cerebellum_
Crus2_R
Occipital_Inf_L

Postcentral_R

Calcarine_R

Rolandic_Oper_L

Precentral_L

Postcentral_L

Frontal_Inf_Tri_R

Hippocampus_R

Heschl_R

Frontal_Sup_Medial_R

Rolandic_Oper_R Heschl_L
Postcentral_L

Calcarine_R

Occipital_Mid_L

Precentral_L

Heschl_R

Rolandic_Oper_R
Caudate_L
Insula_R
Temporal_Sup_L

Table. 1 The results of the 2 level (uncorrected, p<0.001)
activation analysis.

There were no signs of activation in random vs rest,
random vs congruent, or random vs incongruent
comparisons. We then checked the results of the 1 level
analysis to see if there was any region which wascommonly activated in all 19 participants. First, in the
random vs rest comparison, the most commonly activatedregion was the right postcentral gyrus (12 participants),
which is known as a primary receptor of general bodily
sesantion8. The second most common region was theright superior temporal gyrus (STG) (9 participants),
which is known as the main auditory region and alsofunctions in visual processing of moving stimuli9. The
medial frontal gyrus, middle occipital gyrus, middle
temporal gyrus (MTG), and superior frontal gyrus (SFG)
were commonly activated in 8 participants. The medial
frontal gyrus plays a role in performance monitoring,which allows for performance adjustments on subsequent
trials10. The middle occipital gyrus is involved in visual
processing11. The MTG has many functions, including
semantic retrieval, semantic memory, and functions as asemantic control network or as a default mode network(DMN)12. The SFG is reported to be involved in several
networks: a DMN, cognitive execution network, motor
control networks, and is also involved in execution ofworking memory and attention13. Second, in the randomvs congruent comparison, the most common region wasright MTG (11 participants). The second most common

Fig. 2 The results of mean RTs ANOVA in 3 conditions.

The Bonferroni post-hoc test indicated that the meanscore for the random condition (M = 535.06, SD = 8.59)
was significantly different from the congruent condition(M = 393.85, SD = 9.92) and the incongruent condition(M = 427.49, SD = 9.92). However, there was no
significant difference between the congruent condition
and the incongruent condition. The random condition had
the longest mean RT. This result suggests that we
observed a Simon effect in the random condition. The
Simon effect appears in incompatible trials and reflects
an interference from a proponent response based on thestimulus position4. Therefore, we show that the random
condition has the strongest inhibition, indicated by the
behavior analysis.
3.2. fMRI data activation analysis
The result of the image data analyzed in 2 level(uncorrected, p<0.001) had 3 comparisons: congruent vs
rest, incongruent vs rest, and incongruent vs congruent(Table 1).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan211

Kei Sahara, Hiroshi Furutani, Tomoyuki Hiroyasu, Satoru Hiwa

4.

regions were the right lingual gyrus, the right STG, andthe left MTG (8 participants). The lingual gyrus is
involved in higher order analysis of visual stimuli14.
Third, in the random vs incongruent comparison, themost commonly activated region was the left middle
frontal gyrus (MFG) (12 participants), and the second
most common region was the left MTG (9 participants).
The MFG is involved in a ventral frontoparietal network
for attentional processes15.

5.

6.

4. Conclusion
7.

In this study, we used a random condition to investigate
inhibition during incidental events during a spatial SRC
task. We analyzed behavioral and fMRI data from 19
healthy participants. The behavioral data show asignificant difference in the mean RT between the resting
condition and the congruent and incongruent conditions.
This result suggests that the random condition has thestrongest inhibition out of all the conditions. The
activation analysis showed only 3 comparisons having
differences in activated regions via the 2 level analysis,
which included the congruent vs rest, incongruent vs restand incongruent vs congruent comparisons. Therefore,
we went back to the 1 level analysis to observe which
regions were active in the rest comparisons: the random
vs rest, random vs congruent, and random vs incongruent
comparisons. In the 1 level comparisons, we could
observe differences in activation of functional regions
related to cognitive control processes, such as the medial
frontal gyrus, MTG, and MFG. The results suggest thatthe random condition has the strongest inhibition, asassessed by the behavior analysis, and we can observe
activity in regions that are related to cognitive control
processes.

8.
9.

10.

11.
12.

13.

14.

15.

References
1.

2.

3.

Eric H. Schumacher, Mark D’Esposito, Neural
Implementation of Response Selection in Humans as
Revealed by Localized Effects of Stimulus-Response
Compatibility on Brain Activation, Human Brain
Mapping 17:193-201 (2002)
Chan, R. C. K., Shum, d., Toulopoulou, T. &Chen, E.
Y. H. Assessment of executive functions: review of
instruments and identification of critical issues. Arch.
Clin. Neuropsychol. 23(2), 201-216 (2008)
Zhang, H. Zhang, J. & Kornblum, S. A parallel
distributed processing model of stimulus-stumulus andstumulus-response compatibility. Cogn. Psychol. 38,
386-432 (1999).

Jesus Cepon, Santiago Galdo-Alvarez, Fernando Diaz,
Cognitive control activity is modulated by themagnitude of interference and pre-activation ofmonitoring mechanisms, Scientific Report 6:39595(2016)
Weis, T., Estner, B., Krick, C. M., Reith, W., &Lachmann, T. (2015). SNARC meets SPARC infMRI—Interdependence of compatibility effects
depends on semantic content. Neuropsychologia, 77,
331-338.
Cieslik, E. C., Zilles, K., Kurth, F., & Eickhoff, S. B.
(2010). Dissociating bottom-up and top-down
processes in a manual stimulus–response compatibility
task. Journal of neurophysiology, 104(3), 1472-1483.
Oldfield, R. C. (1971). The assessment and analysis ofhandedness:the
Edinburgh
inventory.
Neuropsychologia, 9(1), 97-113.
Webb, W., & Adler, R. K. (2016). Neurology for the
Speech-Language
Pathologist-E-Book.
Elsevier
Health Sciences.
Xu, J., Wang, J., Fan, L., Li, H., Zhang, W., Hu, Q., &Jiang, T. (2015). Tractography-based parcellation of
the human middle temporal gyrus. Scientific reports, 5,
18883.
Ridderinkhof, K. R., Ullsperger, M., Crone, E. A., &Nieuwenhuis, S. (2004). The role of the medial frontal
cortex in cognitive control. science, 306(5695), 443447.
Zhang, B., He, S., & Weng, X. (2018). Localization
and Functional Characterization of an Occipital VisualWord form Sensitive Area. Scientific reports, 8.
Xu, J., Wang, J., Fan, L., Li, H., Zhang, W., Hu, Q., &Jiang, T. (2015). Tractography-based parcellation of
the human middle temporal gyrus. Scientific reports, 5,
18883.
Li, W., Qin, W., Liu, H., Fan, L., Wang, J., Jiang, T.,& Yu, C. (2013). Subregions of the human superior
frontal gyrus and their connections. Neuroimage, 78,
46-58.
Machielsen, W. C., Rombouts, S. A., Barkhof, F.,
Scheltens, P., & Witter, M. P. (2000). FMRI of visual
encoding: reproducibility of activation. Human brain
mapping, 9(3), 156-164.
Cieslik, E. C., Zilles, K., Kurth, F., & Eickhoff, S. B.
(2010). Dissociating bottom-up and top-down
processes in a manual stimulus–response compatibility
task. Journal of neurophysiology, 104(3), 1472-1483.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

212

Measurement of brain activity and problem discovery during actual driving
Seishiro Nakamura

Graduate School of Life and Medical Science, Doshisha University, 1-3 Tatara Miyakodani
Kyotanabe, Kyoto, Japan*
Satoru Hiwa, Kenya Sato, Hiroshi Furutani, Tomoyuki Hiroyasu

Doshisha University, Japan†
tomo@is.doshisha.ac.jp

Abstract
Currently, a system that performs driving support based on the biological information of a driver has attracted attention. Brainfunction is one of the sources of biological information, and we used fNIRS for brain function measurement. We used a simple course
to operate an actual vehicle, and fNIRS measured the brain activities during the driving. The measured cerebral blood flow change
data included artifacts of the driver's body motion and car body vibration. The removal of artifacts was performed using CBSI.
Keywords: fNIRS, CBSI, driving, brain activity

Another safety feature, Active safety, is a technology
to reduce the occurrence of accidents. Vehicle control
devices, such as the Anti-Lock Braking System (ABS)and Electronic Stability Control (ESC), are examples ofActive safety. ABS is a system that prevents the vehicle
from becoming uncontrollable by preventing the tirefrom locking during braking. Electronic stability control
technology was developed to make four-wheel
independent control possible. For example, the ESC
independently controls the braking devices of all thewheels and controls the generation and suppression of therotational motion of the vehicle body. With such a wheel
braking system, sudden skidding of the vehicle can beeliminated.
The application of existing technology to thedevelopment of new, preventive safety systems has
advanced. One of them is a driving support system. The
driving support system monitors the external
environment of a car and prevents accidents. The driving
support system may, however, cause malfunctions due tovarious factors. In order to further improve the driving
support system, it is necessary to have the system monitor
not only the external environment of the vehicle, but also

1. Introduction
Although the number of traffic accident deaths has
decreased, further reduction is expected. Reduction of
accidents has been realized by improved safety
performance of the car. There are two aspects to thesafety of a car: Passive safety and Active safety, as
discussed below.
Passive safety utilizes equipment to minimize injury
when an accident occurs. An example of Passive safety
is a seat belt or Supplemental Restraint System Airbag(SRS) Airbag. Early seat belt designs restrained thedriver in the vicinity of the waist, but the upper body was
not restrained. When an accident occurred, the upper
body of the driver hit the steering wheel and thedashboard, and the driver was more likely to be injured.
Subsequently, a three-point seatbelt was developed,
making it possible to reduce the movement of the upper
body. As another example of passive safety, structures
were designed to absorb the shock of collisions at thefront and rear ends of the car, making the safety of theoccupant's living space more robust, thus improving
safety of the driver.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

213

Seishiro Nakamura, Satoru Hiwa, Kenya Sato, Hiroshi Furutani, Tomoyuki Hiroyasu

the internal environment and the biological condition of
the driver. One cause of traffic accidents is the distractionof the driver. Therefore, by incorporating the biologicalinformation of the driver, it is expected that we canreduce the number of traffic accidents by detection ofchanges in driving behavior, and deterioration of physicalcondition. With regard to biological information, humanbehavior is controlled by the brain, so the interest in brain
function is increasing. Noninvasive brain function
imaging devices, such as functional magnetic resonanceimaging (fMRI), electroencephalogram (EEG), andfunctional Near infra-red spectroscopy (fNIRS), are used
to measure brain function. fNIRS observes changes inhemoglobin in the brain. Near-infrared light has
relatively high bio-permeability. fNIRS is lesssusceptible to body movements and more accessible thanEEG for measuring brain electrical activity.
Human "cognition," to obtain information with eyes
and ears, and "judgment," to process cognitive
information in the brain, results in movement of "hands
and feet" to "operate" the car. In car driving, cognition,
judgment, and operation are three essential elements. In
this study, the relationship between operation and brain
activity was examined. In the experiments, brain
activities were measured during driving of the vehicle,
and the discovery of the task and its examination werecarried out.

Fig. 1 NIRS device (OEG-16, Spectratech Inc.)

3. Method of removing artifacts of driver and car
body motion
In this experiment, the brain activity of the actual driverand the movement of the car were measured. Using theresults, we examined the effects of body movement andcar body vibration on cerebral blood flow. The cerebral
blood flow time series data were bandpass filtered as
preprocessing. A band of 0.008 - 0.09 [Hz] is considered
to be a frequency band related to the activity, and the
bandpass filtering was configured for this band. After
that, the correlation between OxyHb data of the cerebral
blood flow time series and DeoxyHb data was calculated.
In general, OxyHb data and DeoxyHb data derived frombrain function are anti-correlated, and when thecorrelation value is high, there is a high possibility thatdriver and car body artifacts exist in measurement data.
CBSI was adopted as a method to eliminate the influenceof body movement. At CBSI, artifacts of motion are
removed by the following method.

2. Brain function measurement by fNIRS
In this study, brain function measurement was performed
by fNIRS to acquire driver information. fMRI has a highspatial resolution, making it possible to observe specificbrain activity. However, the body must be kept
motionless at the time of measurement, and the flexibilityof the experiment is low. On the other hand, EEG is
widely used and has a high spatial resolution, but it issusceptible to electrical noise. Therefore, we concluded
that fMRI and EEG are not suitable for measuring brain
activities during car operation, especially incontraindicated cases. Since fNIRS can measure without
restraining the human, and uses near-infrared lightharmless to the living body, it is highly safe.
The wavelength of near-infrared light used for fNIRS is
700 to 1000 nm.
Near-infrared light has high bio-permeability.
Hemoglobin easily absorbs near-infrared light.
Oxygenated hemoglobin and deoxidized hemoglobin
have different absorption coefficients. fNIRS measures
oxyhemoglobin and deoxygenated hemoglobin usingnear-infrared light with two different wavelengths. In thisstudy, the frontal lobe brain function was measured by
the fNIRS device (Spectratech Inc.) composed of 16
channels. The equipment used is shown in Fig.1.

=

1
( −
2
1
=−

)

, , , are the measured OxyHb data, DeoxyHb
data, real OxyHb data, and DeoxyHb data. Through theabove processing, transients and artifacts of the car body
movement are removed.
4. Outline of the experiment
In this experiment, the brain activity of the driver and the
movement of the car were measured. The following
experiment was conducted.
4.1. Human subject of the experiment
The subject of this experiment was one graduate student
in his early 20 s, holding a regular driving license. His
physical and visual impairments were within thecorrectable range.
Prior to the experiment, theexperimental procedures and experimental protocol© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan214

Measurement of brain activity

details were explained to the subject. In addition, thesubject agreed to participate in the experiment.
4.2. Experimental devices
The vehicle used for the experiment was a compact, oneseater electric vehicle (ultra-small EV "COMS": Toyota
Auto Body、Fig. 2).

Fig.4 Driving Course Setting

We conducted questionnaires on subjects before andafter the experiment. Question items before theexperiment are age, sex, latest driving day, license
retention years, previous day's sleeping hours, dominanthandedness, corrective vision, pleasure/discomfort
degree, fatigue degree, and sleepiness degree. Question
items after the experiment were the four items ofpleasantness/discomfort degree, fatigue degree,
drowsiness level, and ease of driving. Three question
items related to ease of driving were ease of bending,braking effectiveness, and ease of acceleration. The
Visual Analog Scale (VAS) method was used for the
subject’s responses to the degree of pleasure/discomfort,
fatigue, and drowsiness, respectively.

Fig. 2 COMS (Toyota Auto Body)

Information, such as the steering angle and amount of
brake pedal stroke, was acquired from the Controller
Area Network (CAN) of this vehicle. The OEG 16(Spectaratech Inc.) was used to measure brain activity.
The sampling frequency was 1.53 [Hz]. The total
number of channels of brain activity in the forehead of
the subject was 16. The subject entered the vehicle
wearing the OEG16 sensors, and attached a seatbelt.
The OEG 16 controller was fixed to the car body so that
operations other than driving were not performed during
driving.

5. Results
The correlation value between OxyHb and DeoxyHb ineach channel is shown in Fig.5. Most of the correlation
values between OxyHb and DeoxyHb are close to 1.0(red part). This result shows that the time series ofoxygenated hemoglobin and deoxygenated hemoglobin
are largely similar. Therefore, this blood flow change
is not caused by brain activity. That is, it was a change
in blood flow caused by the driver's body movement andvehicle body movement.
In order to confirm this, an acceleration sensor was
attached to the subject's head to detect the motion of thesubject's head. When driving, the subject would always
points his / her face in the traveling direction. Also, thecar was shaking by the undulation of the road surface.

Fig.3 Channel placement

4.3. Experimental design
In this experiment, brain activity and driving operation
during driving of the car were measured. In the driving
task, the subjects drove the course, weaving in and out,
between the pylons.
The subject drove five turns of the circumferential
circuit as shown in Fig.4.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

215

Seishiro Nakamura, Satoru Hiwa, Kenya Sato, Hiroshi Furutani, Tomoyuki Hiroyasu

Fig.5 Correlation Coefficient Values between OxyHb
data and DeoxyHb before CBSI

Fig. 7 Cerebral blood flow change of channel 8

CHs was close to 1.0. This indicates that the measured
fNIRS data had artifacts of driver's body movement orvehicle body vibration. In this study, CBSI was applied
as a method of removing artifacts. As a result, theartifacts of body movement of the driver or vibration of
the vehicle body were removed.
In the future, we will further reduce these artifacts
during driving and measure brain activity during driving.

The possibility that this shake affected the motion of thesubject was confirmed.
For this data, CBSI was applied. Correlation between
OxyHb data to which CBSI was applied and DeoxyHb
data, to which only the bandpass filter was applied, wascalculated and the results are shown in Fig.6.As a result,the state in which the correlation value is close to 1.0
decreases.
Here, the cerebral blood flow change amount of CH8

References
1. National Police Agency, Traffic Accident Statistics（ 2016.10 ） ,"2016 ． http://www.estat.go.jp/SG1/estat/List.do?lid=000001163223
2. M. Tanishita, H. Miyoshi and M.Sano, Cost-benefit
analysis of automobile safety regulations, “Technicalreport, ITEC working Paper Series 06-16”, Doshisha
University Technology and International Cooperation
Center, 2006
3. S. Kusakabe, T. Kobuse, K. Okada and K. Nomura,
Performance evaluation of automobile, vol. 135, no. 2, pp.
99-102, 2015．
4. T. IINUMA, \MRI principle and equipment, “Clinicalsurgery”, vol. 43, no. 11, pp. 1569-1574, 1988．
5. Y. Inoue, About near-infrared spectroscopy," Medical
Image Information Society Journal, vol. 27, no. 4, pp. 98101, 2010．
6. S. Miyauchi, M, Misaki, S. Kan, T.Koike, K. Iwata, S.
Takahama, Simultaneous measurement of fMRI-EEG andsleep research, “Cognitive neuroscience”, vol. 9, no. 1, pp.
49-55, 2007．
7. Xu Cui, Signe Bray, Allan L. Reiss, Functional nearinfrared spectroscopy (NIRS) signal improvement basedon negative correlation between oxygenated anddeoxygenated hemoglobin dynamics, “Neuroimage”,
vol.49, no. 4, pp.3039-3046, 2010

Fig. 6 Correlation Coefficient Values between OxyHb
data and DeoxyHb after CBSI

is shown in Fig.7. Extreme blood flow change disappears
and the amplitude decreases. This result indicates that the
body motion of the driver and the vibration of the vehicle
body were removed.
6. Conclusion
In this research, a driving support system using biologicalinformation was developed. Utilizing a small electric car,the effectiveness of the developed system was examined.
In this experiment, problems were found when measuring
brain activity during driving. In car operation, thecorrelation value between OxyHb and DeoxyHb of all© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan216

Control of driving simulator based on state detection of the driverusing Electrocardiogram measurement
Koma Yoshioka

Graduate School of Life and Medical Science, Doshisha University, 1-3 Tatara Miyakodani
Kyotanabe, Kyoto, Japan*
Satoru Hiwa, Kenya. Sato, Hiroshi. Furutani, Tomoyuki. Hiroyasu

Doshisha University, 1-3 Tatara Miyakodani
Kyotanabe, Kyoto, Japan
tomo@is.doshisha.ac.jp

Abstract
One of the problems posed by automatic vehicles is the state of the driver at the time of switching between manualand automatic systems. In this research, the driving simulator was controlled based on detection of the driver’s stateby electrocardiogram measurements. A Simulink model that calculates the indicators of the driver’s state obtained
from real-time heart rate variability analysis was constructed in the driving simulator to assess the usefulness of the
proposed system.
Keywords: driver detection, ECG, heart rate analysis, stressduring levels 2 and 3, where a safe and smooth takeover[2] should be performed by the driver instantly. Thus, thestate of the driver at these levels is very important, and asystem capable of monitoring the driver’s state during
manual takeover will essentially lead to prevention of
accidents normally caused by dozing off, and mistakes
caused by excessive tension.
In this research, we developed a driver-state detection
system, via a real-time driving simulation, which iscapable of controlling the simulated vehicle based on thedriver state that is measured by the electrocardiogram(ECG). The system conducts an analysis of real-time
variations in heart rate (HR) from the ECG signal todetermine the state of the driver, and gains control of thedriving simulator (DS) based on the analysis.

1. Introduction
In a completely automatic vehicular operation, allconventional driving activities such as recognition,
judgment, and other manual operations are solely
performed by the system. This research adopts thedefinition of the Society of Automotive Engineers (SAE)
for an automatic vehicular operation, which ischaracterized by varying levels depending on technology.
SAE categorizes automatic operation in six levels— 0 to
5. At level 0, the system runs manually. At levels 1–2, thevehicle is currently in practical use. At level 3, allacceleration/deceleration, steering, and ambientmonitoring are performed by the system, and from level
4, the vehicle is in fully automatic operation. Automatic
operation is bypassed during emergencies, when thedriver is required to immediately assume manual controlof the vehicle (takeover) [1]. The critical point exists

2. Detection of the driver’s state© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

217

Koma Yoshioka, Satoru Hiwa, Kenya. Sato, Hiroshi. Furutani, Tomoyuki. Hiroyasu

In this section, we will first define the driver who

switching between automatic operation and manual
operation is performed.
2.2. State detection
The ECG instrument was used to detect the driver’s
condition during the simulation. ECG measurements canbe noninvasive and continuous. The autonomic nervous
activity of the driver was estimated through an analysis
of variations in HR shown by the ECG results.

operates the DS during the simulation.

2.1.1. ECG measurements

2.1. DS configuration

All ECG equipment were manufactured by g.tec, whose
products are easily operated via MathWorks Simulink.
The biometric instrument came from g.USBamp. A
g.GAMMA box was used as the active electrodeconverter with a sampling frequency of 256 Hz, and theg.GAMMA clip was the active electrode. A Kendall
electrode was the electrode used for placement on theskin. The three-point induction method was employed for
the ECG measurement.

Figure 2. The driving simulator (DS) FORUM8
UC-win/Road, by FORUM 8
Figure 1 shows the UC-win / Road of FORUM 8
company used in the simulation.
The DS is composed of 42-inch flat monitors installedin the front, left and right front, and left and right for a
total of five locations. In addition, there are smaller
monitors installed at eight more locations, three of whichwere at the side mirrors and the room mirror. During thesimulation, the image of the driver's seat viewpoint was
displayed. The driver's seat was equipped with handle
steering, accelerator, brake, handbrake, gear, and turnsignal where operations similar to real automatic cars
were possible. The overall system configuration is
illustrated in Figure 2.
The DS is controlled by a DS Controller (PC1) installedwith the UC-win / Road software. The automatic
operation control system is constructed in the Driving
Agent (PC 2), which performs automatic operation
control based on the DS vehicle information transmitted
by PC 1. In addition, PC 2 receives a processing signal
based on the driver’s biological information acquired
from a third computer (PC 3), or the system’s biomedical
information controller. For example, PC 3 controls PC 2
based on the biometric measurement results, and2.2.2. Analysis of R variations
The peak of the waveform obtained fromelectrocardiography is called an R wave. The instant at
which an R wave is generated can be likened to a momentof a heartbeat. The interval at which R waves are
generated is called the R – R interval (RRI). RRI is notconstant but is continually fluctuating, hence the term HR
variability. HR variability analysis is widely known as amethod of estimating the autonomic nervous activity of aperson. In general, autonomic nerves are divided into
sympathetic and parasympathetic nerves. Sympathetic
activities result to increased HR and decreased HR
variability, while parasympathetic activities lead todecreased HR and increased HR variability [3]. Time andfrequency domain analyses performed in this study werebased on HR variability. Time domain analysis is ananalysis in which constant RRI values of time series areapplied directly [4].
The HR, the standard deviation of the RRI (the standard
deviation of normal-to-normal RR intervals: SDNN), andthe root mean square of the difference between two
consecutive RRI values of successive differences(RMSSD) were used for the autonomic activity analysis.
HR increases when sympathetic nerves dominate, while
SDNN and RMSSD increase when the parasympathetic

Figure 1. Overall configuration of the system© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan218

Control of driving simulator

nerves dominate. In the frequency domain analysis, theestimated value of the spectrum is calculated for the datain which the RRI time series data is resampled at regularintervals. Estimation of power spectral density was
performed on the RRI data resampled at 1 Hz using theperiodogram method. The spectrum estimation value canbe roughly divided into a low frequency (LF) and a highfrequency (HF). The LF band is within the 0.05 – 0.15
Hz range, while HF band is within the 0.15 – 0.40 Hz.
Power was integrated into each band, and the ratio of thetotal value, LF / HF, was used as the activity index of thesympathetic nerve.

Figure 3. ECG real-time analysis model
manual operation is performed when the input is equal tozero. Otherwise, when the input is 1, the switch command
for automatic operation is transmitted to the system via a
UDP communication.
Along with the switch command, it is also possible tosimultaneously transmit a command limiting the vehicle
speed. The automatic driving system accepts data
encoded in ASCII and characterized in bytes.

3. DS control from ECG real-time analysis
The system, which switches the DS manual operation toautomatic operation, was developed based on ECG
results for estimation of the driver’s autonomic nerve
activity. The system is described in the succeeding
subsection.
3.1. The system and its functions
PC 3, the computer conducting the ECG measurement
and real-time analysis for the DS, was connected to PC 2
for control of the automatic operation system.
The state of the driver was monitored in the PC 3. WhenPC 3 detects a state of tension in the driver, it instantly
commands PC 2 to switch the DS to automatic operation.

Figure 4. Automatic operation instruction model

3.2. Simulink model
4.

Figure 3 illustrates the ECG real-time analysis model,
which consists of two functions.
The first function is ECG signal acquisition, achieved by
employing a g.USBamp block of the g.tec proprietarymodel. The Unbuffer block was used for the buffereddata, and a data type conversion block was used toconvert the data to double type.
The second function is the real-time analysis model. It
uses the raw ECG signal as input data to calculate the HR,
SDNN, RMSSD, LF / HF in real time.
From the calculated index, the state of the driver isestimated for a possible system switch from manual
operation to automatic operation. Estimation is done by
the LF/HF ratio. Figure 4 shows the automatic operationcommand transmission model, which sends the switch
command to the computer receiver when the specified
LF/HF threshold value is exceeded. From Figure 4 , theEvaluating the usefulness of the ECG realtime analysis model

To validate the usefulness of the real-time analysis
model, we compared the simulation results with those of
the batch process. R wave detection was performed byreal-time processing of the ECG data in a 3-minuteresting period.
The participants were 8 healthy males, all at 23 years ofage. After measurement, batch processing was employed
for R-wave detection. The correlation coefficient of the
RRI time series data detected in both real-time processing
and batch processing were compared. All of the 8 data inboth processes showed a correlation coefficient of 1.0.
From this, it was suggested that both models could
produce similar results.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan219

Koma Yoshioka, Satoru Hiwa, Kenya. Sato, Hiroshi. Furutani, Tomoyuki. Hiroyasu

The system was validated through a driving simulator
and real-time ECG measurements analysis. It was proventhat the real-time analysis model is useful for heart
variability analysis because it gives the same results as
for offline analysis.

5. Discussion on the driver-state estimation
methodThe procedures of the driver-state estimation are as
follows:① ECG measurement of the driver while at rest.
② Calculation of the average HR, LF/HF, SDNN, andRMSSD values at rest.
③ Comparison of each indicator by real-time analysis
during experiment with its average value at rest.
④ The output for each indicator whose real-time analysis
is sympathetically dominant over that at rest is 1.
⑤ The output 1 is added.
Because there are four indicators, the possible
maximum output value is 4, while the possible minimum
value is 0.
Figure 5 shows the value of the estimated time serieschange of stress for the driver. The peak, high-stress state

7. References
1. A. Eriksson and N.A. Stanton, “Takeover time in highly
automated vehicles: noncritical transitions to and frommanual control," Human factors, vol. 59, no. 4, pp. 689705,2017.
2. U U.E. Manawadu, H. Hayashi, T. Ema, T. Kawano, M.
Kamezaki and S. Sugano, “ Tactical-level input withmultimodal feedback for unscheduled takeover situations
in human-centered automated vehicles," in 2018
IEEE/ASME International Conference on AdvancedIntelligent Mechatronics (AIM)IEEE, pp. 634-639 2018.
3. G.G. Berntson, J.T. Bigger, D.L. Eckberg, P. Grossman,
P.G. Kaufmann, M. Malik, H.N. Nagaraja, S.W. Porges,
J.P. Saul, P.H. Stone, et al., “Heart rate variability: origins,
methods, and interpretive caveats," Psychophysiology, vol.
34, no. 6, pp. 623-648, 1997.
4. T.F.o.t.E.S.o.C.t.N.A.S.o.P. Electrophysiology, “Heartrate variability: standards of measurement, physiologicalinterpretation, and clinical use," Circulation, vol. 93, no. 5,
pp. 1043-1065, 1996.

Figure 5. Time series change of stress value
for the subjects occurred at around 150 seconds from theanalysis. This means that the subjects experienced
maximum tension at that time. The estimation method
gives an insight on the state of tension of the driver basedon behavior and objective data.
6. Conclusions
There have been a wide range of automatic driving
techniques developed in the recent years, which also
attracted vast research on automatic driving systems.
Because the switching of systems from automatic tomanual arises during critical levels of vehicular
operations, in this case within levels 2 – 3 of the SAE
definition, the issue is how the driver could ensure a safe
and smooth takeover to prevent pre-conceived accidents
or mistakes.
Realizing that the state of the driver is important during
the takeover, this study proposed a system that monitors
the driver’s state of tension or autonomic nerve condition
using heart rate variability analysis from a system’schannel, which sequentially communicates a switch
command to another channel for an effective manual-toautomatic operation shift.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan220

Tactile Score; Development and Applications

Yasuhiro Suzuki

Graduate School of Informatics, Nagoya University, Furocho Chikusa
Nagoya City, Aichi Prefecture 464 0814, JAPAN*
E-mail: ysuzuki@nagoya-u.jp
www.ysuzuki.info

Abstract
Unlike music, which can be decribed by means of a music score, tactile sense does not have a standard method fortranscription. Likewise, tactile sense has no developed principle of composition. Hence, this study developed amethod for describing tactile sense, referred to here as the ‘tactile score’. We developed a mutual transformation
method between tactile score and haptic vibrations, and then we verified that a hand massage performed according
to a tactile score and haptic vibration from a tactile score evoked similar brain activities.
Keywords: List four to six keywords which characterize the article.

its own aesthetic principles and a natural discerning
ability: […] the body (of another or even one’s own) can
provide beautiful sensory perceptions or (in Kant’s
famous terminology) repre-sentations.). Shusterman
pointed out that aesthetics is not a written knowledge but
the sum of body experiences:
When we are moved by a great artwork, it changes ourblood pressure, heart rate, brain activities, etc. When we
see a great artwork, if our body does not response to it,that means that the piece is not a great artwork for us
person-ally. In other words, our body experiences judge
the beauty of things. Such an enlarged view of aesthetics
would give more systematic attention to the body’s
crucial roles in aesthetic perception and experience,
including the aesthetic dimensions of body therapies,
sports, martial arts, cosmetics, etc., that re-main
marginalized in academic aesthetic theory.3

1. Introduction
Tactile sense1 is sometimes called ‘the first sense’; it is
the first sense formed in hu-mans during development asa foetus. Tactile sense is associated with intimacy andempathy. It plays an important role in the somatic
systems in its application of attention as a method ofchanging consciousness and improving knowledge.
However, whereas visual media are systematized withcolors and their combinational principles, such as color
circles, and music in auditory media is systematized withthe combinational principles of theories of harmonics,
there has been no such a combinational principle fortactile sense.
Since criticism by Kant2, many aestheticians havenearly given up thinking about beauty rationally althoughartists have been constructing beauty in various fields,such as music composition, dancing, and graphics.
Prominent aesthetician Richard Shusterman3 introduced
the concept of Somaesthetics based on Baumgarten’s
Aesthetica4, which proffered the belief that the body has© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan221

Yasuhiro Suzuki

2. Tactile Score

instance, draw a circle on the back of your hand using
your fingertip strongly and then draw a circle on the backof your hand using your palm with the same movement
velocity; if you massage with the same pressure, it will
not be comfortable, but if you massage more softly, it will
be comfortable. As S decreases and V remains constant,and P becomes stronger,
Const. = S × P × V is preserved; this will generate acomfortable massage. This experimentally obtained
principle has been confirmed through experiments andmathematical analysis using category theory.

The massage, which is composed of a sequence ofvarious tactile senses of movements of hands on the body,
is a form of haptic engineering that humankind has beendeveloping for more than 4,000 years. For this reason, we
chose to investigate the massage to devise a method of
describing the tactile sense.
Through our investigation, we found the massage tobe composed of pressure, area of contact or touching, andvelocity of the movement of the massager’s hands. We
modelled our tactile description on the musical score5. In
staff notation of the tactile note, we defined the third line
as the basic pressure—the basic pressure with which wemight hold a baby or delicate, breakable objects. Hence,
the basic pressure is not defined absolutely but may
change from person to person or for different types ofmassage. We also defined the parts of the hand used in
massaging and the kinds of stroke used when massaging.
For example, the fingertip to the first joint is 1, thesecond joint is 2, the third joint is 3, the upper part of thepalm is 4, the centre of the palm is 5, and the bottom of
the palm is 6. Thus, a flowing motion from a fingertip to
the third joint is denoted as ‘1–3’.
For massage strokes, we analysed the method of
massaging in face therapy and the stroking action of thehands’ movements. We assigned these as A, a, N, and n,
where A stands for the massage stroke of drawing a circle
on the cheek, and N stands for a massage stroke ofdrawing a line, a stands for a smaller circle on the checkand n also a shorter line than N. For example, A5
illustrates drawing a circle on the cheek with the centreof the palm.
1.1

2

Experimental Aesthetics of Tactile
Sense

To verify the obtained principle, we examined andcompared the effectiveness of massages that were, in
principle, correctly or incorrectly composed. We
analysed salivary alpha-amylase, because this protein
enzyme increases when a person suffers from stress. We
also examined the small face effect (improved elasticity
and smoothness) and improvements in the skin condition,
based on changes in the brightness of the skin.
We compared the values of the stress markers before
and after the massage. When the massage was principallycorrect, the values of the stress markers decreased, andwhen it was incorrect, the values increased. This finding
confirmed that people feel comfortable when a massage
is principally correct and can recognize a principally
incorrect massage.
We then set out to verify the effectiveness of themassages as beauty treatments. We took photos before
and after the principally correct massage and compared
the shape and brightness of the face using image
processing. The processed findings indicated that theshape of the face tended to shrink, especially around thecheeks, and that the brightness of the facial image tended
to be heightened. These results were obtained throughexperiments using a part of the same face therapy
massage that has been requested by and performed on
more than 200,000 customers in our beauty shops; its
effectiveness has therefore been validated by social
experiments. The human body distinguishes beautiful
and non-beautiful haptic sensations. People experience
the beauty of their own bodies with physical senses, as
proven by a range of indicators from the salivary alpha-Principle of Creating Tactile Sense of
Massaging

Having developed a workable method for describing
massages, we are now able to compose massages. We
have created various tactile scores and performed them,and have examined and determined the comfortablemassages. We synthesized these experiments andexperiences of massaging for customers and obtained thecommon characteristics of a comfortable massage asfollows:
Const.=S×P×V,where S is the area of touching, P is the pressure of themassage, and V is the velocity of hand movement. For© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan222

Tactile Score; Development andamylase levels6 to enhanced brightness or ‘glows’ toimproved elasticity, relaxation, and skin condition
improvements. If this appeal to the beauty of personal
experience seems strangely idiosyncratic, consider theremarks of Jean-Marie Guyau: To breathe deeply,
sensing how one’s blood is purified through its contact
with the air and how one’s whole circulatory systemtakes on new activity and strength, this is truly an almost
intoxicating delight whose aesthetic value can hardly bedenied (in Guyau, J-.M. (1884). Les problèmes del’esthétique contemporaine (11th ed.) (pp. 20–21).
The principle of creating pleasurable tactile sense,
which we extracted and codified, shows how to compose
beautiful haptic sensations in art and design.

3

this tactile score is transformed into the sequence of asine wave of 30 Hz (= 40 – 5 x 2 Hz) for 1 sec, 30 Hz for
1 sec, 50 Hz (= 50 + 5 x 2 Hz) for 1 sec, and 50 Hz for 1
sec.
Using this transformation, we can also obtain thetactile score from the sound input. When the first inputtedsound is analysed by Fourier analysis and frequencies,
we obtain the amplitudes and their durations; based on
the obtained results, we can define the frequency that
corresponds to the basic pressure and the amplitude,
which corresponds to the halftone.
For example, it is obtained as follows: (the highest
frequency – lowest frequency) / 12 = the pressure of half
tone, where we differentiate of the range of frequencies
into an octave (twelve half-tones). Using this algorithm,
we can create a tactile score automatically from music,
natural sounds, noise, etc. To verify the transformation
method from tactile score to haptic vibration, we
examined the effect of transformed vibrations fromtactile score on the human body. Preliminary
examinations showed that the human body distinguishes
the quality of vibrations, and this affects activities of thebrain.
Thus, the transformed vibration from a tactile score
may produce the same effect as a hand massage
according to the tactile score. We examined brain activity
using near infrared spectroscopy (NIRS)8 (specifically,
Pocket NIRS Duo, DynaSence, Co. Ltd.). We examined
how hand massages composed through tactile scores
affect brain activities and found that well-designed handmassages created through correctly composed tactile
scores, with the codified composing principle, evoke
brain activities higher than those produced by incorrectlycomposed massages.
The result of the experiment indicated that haptic
vibration transformed from a correctly composed tactile
score evoked brain activities that were statisticallysignificantly higher than sine waves and the control.

Mutual Transformation of Haptic
Tactile Scores and Haptic Vibrations

Haptic media, like a hand massage, is a personal media;
no one can massage thousands of people simultaneously,
display a massage in a gallery, or broadcast a massage
through television or over the internet and obtain thesame results as those for the individual receiving the
massage. This does not mean that tactile sensations
cannot be communicated. To do this, we attempted tobridge the gap between haptic sensation in the physicalenvironments and the digital world of bits and bytes.
To transform the haptic media of the massage(personal media) into mass media, we transformed tactile
scores into haptic vibrations7. There are several ways of
performing such transformations. In this work, we show
the case where only the pressure changes, and not the sizeof the contact area. Based on research on haptic
impressions and frequencies, stronger pressure
corresponds to a lower frequency, whereas weakerpressure, to a higher frequency relative to the frequencyof the aforementioned standard pressure, which we
signify with the third line in the tactile score.
For example, in transforming the following musicscore (Re, Re, So, So; with quarter notes) into tactile

4

Applications of Tactile Score

Construction of a transformation system from tactile
notation to vibrotactile has expanded related research on
the scores. For example, by using this technique toextract tactile notation from songs and convert it into
vibrotactile sense, it is possible to take out haptic sensescore, we set the sine wave at 40 Hz frequency and define
the difference of each half-tone as 5 Hz. When the tempoof the tactile score is one second for each quarter note,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan223

Yasuhiro Suzuki

inherent in music and obtain tactile sense corresponding
to songs. Unlike the traditional so-called "beat" bass
enhancement and tactileization (such as Body Sonic), theappeal to emotion increases in music without beats suchas Enka and majestic classic. This new music - haptic
media is of interest from acoustic experts, andcommercialization of major sound manufacturers andreproducers is under way (Patent No. 6322780, 6401758).
Also, at the request of NTT, I designed the vibration
tactile extracted from the music for the hearing-impaired
person (Tsukuba Technical University Dance Team) (forthe event "Nico-Nico Chokaigi, 2018"), and fromanimation and same touch It was also used for video workthat generated vibration tactile sense (exhibition Haptic
TV, ICC museum, Tokyo). This media has attracted
attention from music specialists and develops tocollaborative research and has created a technology tofoster presence by combining inaudible vibration haptic
sense with music and has started distribution business of
this media.
In addition, we have found that it is an active haptic
presentation device that can obtain various tactile
sensations when the user operates by combining newmaterial and vibrating tactile sense with a major
manufacturer of electronic component equipment. We
have started a collaborative research project (Vibration
Medicine project) that applies a series of vibrational
haptic technology to improve insomnia related to lower
brain dysfunction with major drug development
manufacturer. Depending on progress, we plan todevelop to demonstration experiments in cooperation
with local governments within a few years. Theinteractions in the information environment have been
digital (coupled or not) so far. What is important in the
future is the quality of interaction. For example, "Thankyou" for print is "Thank you" regardless of the depth ofgratitude. The quality of information changes as itchanges type / pictograph, handwritten character,telephone and interaction quality.
In the future, everything will be digitally linked byInternet of Things, IoT etc. This domain enables mutual
action of emotion and emotion in the information
environment by enriching the link quality while utilizing
this growing link as a social infrastructure, its
significance and ripple effect are great.

Moreover, even in the direct interaction between humans,
as well as playing instruments while watching the score,
you can perform a high quality massage by touching,combine tactile information technology and directinteraction, By monitoring speech and facial expressions
and smoothing human relationships, it is expected to
improve the environment of emotional labor (high nurse,
care giver, call center, etc) with high staff turnover rate.
In addition, by giving vibratory tactile sensation
generated from the talent of expert's massage inpreparatory research to the face, stress markers (amylase
concentration in saliva) decrease, blood flow increase at
the terminal (limb) Mire, a questionnaire suggests
improvement of sleep state. This result suggests thatvibratory tactile sensation can be improved by improving
lower brain dysfunction (disorders such as respiration,
sleep, mood) and exercise effects of patients who havebeen bedridden for a long time (from rising blood flow in
hands and feet) It is expected to have a high ripple effectas a new medical device that can be used cheaply andeasily at home.
Acknowledgement
This research has been supported by Grant in Aid for
Scientific Research (KAKENHI) No:15K00268 and
16H03093
References
1.
2.
3.
4.
5.

6.

7.

Serino, A. & Haggard, P. Touch and the body.
Neuroscience and Biobehavioral Reviews (2010).
doi:10.1016/j.neubiorev.2009.04.004
Kant. Crítica de la razón pura. critica de la razon pura(2005). doi:10.1109/ICRA.2013.6630904
Shusterman, R. Muscle memory and the somaesthetic
pathologies of everyday life. Hum. Mov. (2011).
doi:10.2478/v10038-011-0001-2
Gregor, M. J. Baumgarten’s Aesthetica. Rev.
Metaphys. (1983). doi:10.2307/20128010
Suzuki, Y., Watanabe, J. & Suzuki, R. Tactile score, aknowledge media of tactile sense for creativity. Smart
Innov. Syst. Technol. (2012). doi:10.1007/978-3-64229934-6_56
Nater, U. M. et al. Stress-induced changes in humansalivary alpha-amylase activity - Associations withadrenergic
activity.
Psychoneuroendocrinology(2006). doi:10.1016/j.psyneuen.2005.05.010
Suzuki, Y., Suzuki, R., Watanabe, J., Yoshida, A. &Shigeru, S. Haptic vibrations for hands and bodies.
SIGGRAPH Asia 2015 Haptic Media Contents Des.
(2015). doi:10.1145/2818384.2818389© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan224

Tactileology; Haptic Informatics by using Tactile bit, T-bit

Yasuhiro Suzuki

Graduate School of Informatics, Nagoya University, Furocho Chikusa
Nagoya City, Aichi prefecture 464-081, JAPAN*
E-mail:ysuzuki@nagoya-u.jp
www.ysuzuki.info

Abstract
With the progress information technology, IT our interaction has become dense through information space. On theother hand, the interaction of tactile "interaction" is getting loose. Therefore, we conceived an academic field that
integrates with the digital information space by systematically studying and informing the tactile interaction.
Therefore, we will create a tactile information processing system that outputs a sense of touch by inputting tactile,
converting it into a talent, converting it into a tactile score, using a method "touches" for describing tactile interactions
as an intermediate language. In this area, the tactile interaction which has been tacit knowledge so far is informed
and integrated with the existing academic arts and industrial infrastructure, enriching tactile interaction in the society
at large including the information space It is expected to become.
Keywords: Tactileology, Tactile bit, Tbit, Tactile Communicationsof electronic commerce such as Amazon, rationalizationof customer service and practical application of
automatic driving in public transportation. Through thedevelopment of IT, the interaction in the information
space becomes dense while the direct interaction in thereal space becomes sparse.
The disappearance of direct interaction between
individuals by IT also occurs in the medical field. In 2011,
Abraham Verghese2, Stanford University Medical
School professor gave a lecture entitled "Power ofDoctor's Hand" at Technology Entertainment Design,
TED Global. In the lecture he said that “ I joke, but I only
half joke, that if you come to one of our hospitals missing
a limb, no one will believe you till they get a CAT scan,
MRI or Orthopedic consult.” (CAT: Computed Axial
Tomography, MRI: Magnetic Resonance Imaging). And
he alarmed the present situation that only the data wasbiased in the medical field and the "interaction" between

1. Introduction
Developments of information technologies, IT such asSocial Network Services, SNS, Internet of Things, IoT1
and so on have transformed interaction space from realworld to cyberspace. Interactions in cyberspace do notlimit by time and borders; we will hesitate to make aphone all in midnight, while we will send e-mail in every
time and we have to pay international postal fee to senda letter to abroad, while we do not have to pay such fees
to send e-mail to abroad. In the past, individuals
interacted via mass media such as newspapers and TV.
SNS enabled individuals to interact directly.
Direct interaction allowed information andcapital to flow between individuals. This personal
information and the flow of capital have changed thesocial structure greatly. For example, there aredisappearance of personal shops due to the development© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan225

Yasuhiro Suzuki

the doctor and the patient is getting thin. He predicted that"[…] the most important innovation, I think, inmedicine to come in the next 10 years, and that is thepower of the human hand -- to touch, to comfort, todiagnose and to bring about treatment.” Although, ten
years have passed since this lecture, such medical
innovation as his prediction has not occurred during thistime.

massage, it has shown that massage gives effect on pain
relief18, mental health 19, infants 20 (weight gain) etc. In
addition, immune function of breast cancer patients 21,
the quality of life of diabetes patients 22, relationship withblood glucose level 23, etc. have been reported.
Regardless of domestic and overseas research,massage research is mainly clinical research, and studies
that have stepped into massage techniques are commonlydescribed in natural language. In nursing science,
Kawashima Midori (Nihon Red Cross Nursing
University) systemizes the hands, advocates "TE-ARTE"
and establishes associations and conducts practice anddissemination activities.
Conventional studies of time-varying tactile
sensations (massage), unified methods that can describe
arbitrary massages as tactile notes have not been
proposed, and the provision of technologies with highreproducibility with quality control is difficult.
We know the importance of touching directly; suchas shaking hands and hugs. The importance of the powerof human hands is a common recognition beyond ourtime and region among us. The sense that supports"people's contact" and "power of human hands" is"tactile".

1.1. Related works in Tactile Sense
Tactile sense has been investigated in cell biology(mainly tactile receptors)3, neuroscience4, psychology5,
cognitive science6, virtual reality (VR)7 , robotics8 and soon. In physiology and neuroscience, response pathways
and mechanisms for tactile stimuli have been studied 9.
However, there are no studies that designed the tactile
stimulus itself. In psychology studies are being
conducted on illusion10 and phantom limbs11 (pain inparts that should have been cut off), relationships withperception 12, emotion and relationships with affordance
13, and so on.
In informatics, Susumu Tachi et.al. (Keio University,
JAPAN) have been developed various tactile interfaces
and applied for Virtual Reality, VR system, such as thetele-existence system14 (system that makes people andobjects of remote areas feel as if they are nearby). They
also develop Tactile Toolkit 15 that records sound (suchas fricatives) when they are in contact with objects andreproduces them with a vibrotactile presentation device(using the same method for Nintendo-Alps Electrics
company). In overseas, Hiroshi Ishii (Massachusetts
Institute of Technology, MIT USA) et.al. have been
conducting research project "tangible bit"16 17 which canbe touched directly, and it is well known for many years.
Massage is a time-varying tactile used from theancient times (depicted on the mural paintings of theEgyptian era) over time and region. Hippocrates who is
one of the founder of Western medicine pointed out the
medical effect of massage, but massage is still only oneof alternative therapies, as compared with surgicalscience and internal medicine which was subsequently
established as Western medicine.
A lot of research has been conducted in nursing andphysiotherapy in Japan, and overseas the National Centerfor Complementary and Integrative Health (NCCIH) of
the National Institutes of Health has been leading
research on massage since 1991. In the review of

2. Tactile Score
Hence, I conceived a research that informs the tactileinteraction and integrates with tactile information andinformation environment. This research can utilize
traditional tactile research in information environment(nursing, physical therapy, engineering, etc.) information
environment. The sensitivity on frequency of sound oftactile sense is not high compared with auditory sense but
for tactile receptors sensing pressures and their patterns
are important to detect sharpness or roughness of object
to touch; of course, there are another important ability tosense e.g. temperature, pain, etc. However, in order todescribe tactile sense, not only pressures but also should
consider another characteristic; hence we have been
considered in massaging, which is the most common
products of haptic engineering and possesses long history.
To investigate the method of massage, we takemassaging to be composed of pressure, the area oftouching, and the velocity of the movement of hands. In
staff notation of the tactile note, we define the third line
as the basic pressure; the basic pressure is the pressure
when we hold a baby or an expensive jewel very carefully.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan226

Tactileology; Haptic Informatics by

inputs tactile information, performs
processing, and outputs tactile sense.

Hence, the basic pressure is not defined absolutely but
may change from person to person or for different typesof massage. We also define the part of the hand used in
massage and the kind of stroke when massaging. For
example, the fingertip to the first joint is 1, the second
joint is 2, the third joint is 3, the upper part of the palm is
4, the center of the palm is 5 and the bottom of the palm
is 6; when we flow from a fingertip to the third joint, thisis denoted as ``1-3''. For massage strokes, we analyze themethod of massaging and strokes of hands movements;
we symbolize each stroke as, etc. For example, thesymbol stands for the massage stroke of drawing a circle
on the cheek.

information

3.1. Information search by using T-bit

Fig. 1. Demonstration of information search by using T-bit, auser make a voice; when a user talks to the system, the user
have massage with the extracted vibrotactile sense (photographis J.Yamagiwa, the president of Kyoto University).

3. Tactile bit, T-bit
Tactile can be translated into language by using Tactile
Score. Tactile sensation is included in the senses. For
example, sculpture is called visual tactile, and has been
discussed in aesthetics, philosophy, and so on. The soundtexture is expressed tactilely like "soft voice" or "hardsound (heavy metal)". Also, tactile in taste is called
texture. Therefore, when describing the tactile sense
included in the five senses with Tactile Score, Tactile
Score becomes a sensory language.
Tactile Score describes discrete time change ofpressure. When describing five senses with Tactile Score,
only the time change of what corresponds to "pressure"in each sense is described. Therefore, even if the sense isdescribed by Tactile Score, not all tactile sense isdescribed. This is similar to music score in music. The
music score is a discretizing description of the temporalchange of the pitch. Therefore, the music score does notdescribe all of the music. However, music score is an
important way of describing music. By using music score
we can record, save, edit and distribute music, making
possible the birth of composers.
Tactile scores describe tactile symbols and quantify
them. Symbolized and quantified tactile sensations canbe processed with conventional information processing
systems and artificial intelligence systems. Therefore, thetactile information informed by Tactile Score is called
Tactile bit, T-bit. When tactile sense of five senses is
described by T-bit, that is, Tactile Score, T-bitized tactile
sense becomes input to the conventional information
system. Then, by outputting the result of information
processing as Tactile Score and converting Tactile Score
to tactile sense, it is possible to construct a system that

We constructed a haptic search system using T-bit.This
system uses only T-bitized tactile information without
using language information for information retrieval. The
retrieved information is also outputted as tactile sense.
This system processes tactile input and outputs tactile
sense, which is completely different from theconventional information retrieval system (Fig 1.). Thissystem converts the user's voice into T-bit and analyzes
its touch quality. Touch quality is a concept used in tactile
science, and it is defined by two-dimensional touch sensein the vertical direction and the horizontal direction. The
vertical direction is expressed as "hardness" and"softness", and the horizontal direction is expressed as"smoothness" and "roughness". This system evaluates as"Soft" if the variance of the Tactile Score used in the Tbitization (the strength of the pressure) is large, "Hard" ifnot, and if the envelop of the Tactile Score is smooth"Smooth", if it is jagged, it is evaluated as "rough". Byquantifying each tactile, the T-bit tactile is mapped on a
two-dimensional plane.
We have calculated the touch quality of hundreds ofT-bit music data and constructed a touch database. This
database stores music and its touch quality (this system
won the president of Kyoto University prize in Kyoto
University Interdisciplinary Research Idea Contest2017).
3.2. Toward Tactileology by using T-bit
Previous tactile studies have studied the tactile sensewhen integrating time. On the other hand, this researchcorresponds to tactile research when time isdifferentiated. In addition, tactile sense has been mainly
described in natural language, but in this research its© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan227

Yasuhiro Suzuki

described by symbol (score). Chemistry and mathematics
were once described in natural languages, but eventually
the description by chemical formulas and algebraic
symbols became to be used and it made a dramatic
progress. This research by Tactile Score makes dramatic
progress in tactile research and application.

12.

13.

Acknowledgement

14.

This research was supported by JSPS KAKENHI,
Grant in Aid for Scientific Research No. 15K00268
References
1.

2.
3.

4.

5.

6.

7.

8.

9.

10.

11.

15.

Saracco, R. The internet of things. in IEEE Technology
Time Machine Symposium on Technologies Beyond
2020,
TTM
2011
(2011).
doi:10.1038/scientificamerican1004-76
Verghese, A. A touch of sense. Health Aff. (Millwood).
(2009). doi:10.1377/hlthaff.28.4.1177
Lumpkin, E. A., Marshall, K. L. & Nelson, A. M. The
cell biology of touch. Journal of Cell Biology (2010).
doi:10.1083/jcb.201006074
Graczyk, E. L. et al. The neural basis of perceived
intensity in natural and artificial touch. Sci. Transl.
Med. (2016). doi:10.1126/scitranslmed.aaf5187
Bauer, A., Hagenburger, J., Plank, T., Busch, V. &Greenlee, M. W. Mechanical pain thresholds and therubber hand illusion. Front. Psychol. (2018).
doi:10.3389/fpsyg.2018.00712
Nonomura, Y. et al. How to identify water fromthickener aqueous solutions by touch. J. R. Soc.
Interface (2012). doi:10.1098/rsif.2011.0577
Kilteni, K., Groten, R. & Slater, M. The Sense of
Embodimentinvirtualreality.
Presence:
Teleoperators and Virtual Environments (2012).
doi:10.1162/PRES_a_00124
Kappassov, Z., Corrales, J. A. & Perdereau, V. Tactile
sensing in dexterous robot hands - Review. Rob. Auton.
Syst. (2015). doi:10.1016/j.robot.2015.07.015
Vallbo, Å. & Johansson, R. S. Properties of cutaneous
mechanoreceptors in the human hand related to touchsensation.
Hum.
Neurobiol.
(1984).
doi:10.1007/s00406-006-0654-6
Tsakiris, M., Prabhu, G. & Haggard, P. Having a bodyversus moving your body: How agency structures
body-ownership.
Conscious.
Cogn.
(2006).
doi:10.1016/j.concog.2005.09.004
Moseley, G. L. I can’t find it! Distorted body image
and tactile dysfunction in patients with chronic backpain. Pain (2008). doi:10.1016/j.pain.2008.08.001

16.

17.

18.

19.

20.

21.

22.

23.

Dargahi, J. & Najarian, S. Human tactile perception asa standard for artificial tactile sensing--a review. Theinternational journal of medical robotics + computerassisted surgery : MRCAS (2004). doi:10.1002/rcs.3
Lee, B. & Macgregor, C. G. The Emotional Impact ofSurface and Prototyping Properties in Product Design.
Proc. Hum. Factors Ergon. Soc. Annu. Meet. (2010).
doi:10.1177/154193121005402003
Tachi, S. Telexistence and Retro-reflective Projection
Technology (RPT). Proc. Virtual Real. Int. Conf.
(2003).
Pohl, I. M. . & Loke, L. . Touch toolkit: A method toconvey touch-based design knowledge and skills. in
TEI 2014 - 8th International Conference on Tangible,
Embedded and Embodied Interaction, Proceedings(2014). doi:10.1145/2540930.2540957
Ishii, H. & Ullmer, B. Tangible bits. in Proceedings of
the SIGCHI conference on Human factors incomputing systems- CHI ’97 (1997).
doi:10.1145/258549.258715
Ishii, H., Lakatos, D., Bonanni, L. & Labrune, J. B.
Tangible Bits: Beyond Pixels. Interactions (2009).
doi:10.1145/1347390.1347392
Frey Law, L. A. et al. Massage Reduces Pain
Perception and Hyperalgesia in Experimental Muscle
Pain: A Randomized, Controlled Trial. J. Pain (2008).
doi:10.1016/j.jpain.2008.03.009
Plews-Ogan, M., Owens, J. E., Goodman, M., Wolfe,
P. & Schorling, J. Brief Report: A pilot study
evaluating mindfulness-based stress reduction andmassage for the management of chronic pain. J. Gen.
Intern.
Med.
(2005).
doi:10.1111/j.15251497.2005.0247.x
Field, T., Diego, M. & Hernandez-Reif, M. Preterm
infant massage therapy research: A review. Infant
Behavior
andDevelopment(2010).
doi:10.1016/j.infbeh.2009.12.004
Hernandez-Reif, M. et al. Breast cancer patients have
improved immune and neuroendocrine functions
following massage therapy. J. Psychosom. Res. (2004).
doi:10.1016/S0022-3999(03)00500-2
Ezzo, J., Donner, T., Nickols, D. & Cox, M. Is
Massage Useful in the Management of Diabetes? A
Systematic Review. Diabetes Spectr. (2001).
doi:10.2337/diaspect.14.4.218
Sajedi, F., Kashaninia, Z., Hoseinzadeh, S. &Abedinipoor, A. How effective is Swedish massage on
blood glucose level in children with diabetes mellitus?
Acta Med. Iran. (2011).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

228

Natural Computing and Formal Computing

Yasuhiro Suzuki

Graduate School of Informatics, Nagoya University, Furocho Chikusa
Nagoya City, Aichi prefecture 464-0814, JAPAN*
E-mail: ysuzuki@nagoya-u.jp
www.ysuzuki.info

Abstract
The basis of natural calculation is considered. In order to that algorithm, programming, calculation are redefined.
From the redefinition, it is shown that the observation is indispensable for calculation. Considering from the new
definition, the calculation can be divided into Natural Computing and Formal Computing. Natural Computing is theprimitive computation and all computations occur as Natural Computing. And Natural Computing evolves to Formal
Computing. We reconsider the research on artificial intelligence using the results of this study and suggest a future
direction of progress.
Keywords: Foundation of Natural Computing, Observation of Computing, Formal Computing, Artificial
Intelligenceobservation, we can verify 10 is correct. In this example,
we used formal observation and verification that havegiven by mathematics.
However, most of computing in nature does not haveformal system to observer and verify. This isfundamental difference of formal computing from natural
computing. In natural computing, computing substances,such as DNA molecules, cells or ecosystems, observe theresult of algorithm and verify it by themselves. Hence,natural computing is primitive in computing andthroughout of the evolution from natural computing, it
develops “external formal” system to observe and verify
and evolve to formal computing system.

1. Introduction
Algorithm is an order of executions; so, there are manyalgorithms in nature. In general, elements in nature,including human, cannot predict the future, but we canpredict that DNA would replicate itself, growth of cellthrough cell cycles and how the ecosystem recover after
being destroyed by the flood.
Why can we predict them? Because we (not only
human) know algorithms and how they will work.
Computing is composed of algorithm, the observation ofcomputing and its verification; where observation is how“decode” the result of algorithm and the verification
guarantee the legitimacy of computing; for example,
result of algorithm (addition) of 1 + 1 can be 2 or 10, if
the method of observation is decimal system, we canobserve 2 and 10; if it is binary system, we can observe
10 but cannot 2; in decimal observation, we can verify 2
is correct, while 10 is not correct and in binary

2. What is Algorithm?
When you buy potatoes, carrots and onions in thegrocery store and make an account at the cash register. In
the register, the total is displayed as 0 yen at the
beginning, but the total amount changes as the price of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan229

Yasuhiro Suzuki

potatoes, carrots and onions are added, when all are
finally added and the consumption tax is added at the end,
the total amount will not change.
When the state change due to action becomes steady,that state is called normal form. As in this example, thenormal form includes not only the case where the statechange stops but also a dynamic steady state like thecomputing process of the circumference ratio.

computing entity, these will be entering at least in themaze.
On the other hand, there is no way to know whether
slime mold acted "unravel the maze" or not. Slime molds
are just a state change, not even more or less. Therefore,
the assertion that "slime mold has an intelligence to solve
the maze", "slime mold is clever" is an arbitrary part ofthe computing entity observing the behavior of slime
mold as such. If the correct answer set by the computing
entity is "the slime divides into two", this time, "regulardivision slime mold" will be selected for the normal typeset.
Although individuals reaching from the entrance tothe exit were chosen as correct candidates by observation,
since not all of them are passing through the shortest
route, it is necessary to verify the correctness of thecorrect candidate. For this purpose, the computing entity
investigates the shortest path length of the maze inadvance, and the individual that is the shortest path length
among the correct candidate is "correct answer".

2.1. Algorithm
In the case where the order of action is determined
beforehand when the state changes with action, or in thecase where the order of action occurs from the process ofchange, such order is called an algorithm.
In the grocery store example, the addition order of"potato + carrot + onion" corresponds to the algorithm.
In chemical reactions, the order of reaction speed occurs
relatively, and the state changes in that order, so the orderis the algorithm.
Changing the order of algorithms is called
programming. For example, if you buy 4 potatoes, 1
onion and 3 ginsengs, adding efficiency one by one is notefficient, so change the order of action with "potato x 4 +carrot x 3 + onion" this is programming. Also forchemical reactions, catalysts and enzymes are used tochange the order of reaction rates, which is also
programming.

2.3. Subjective verification and objective
verification
In the case where verification of validity is performed
outside the computing subject (the subject performing
computing), it is called "objective verification". In
addition, when the subject of verification itself conducts
validation, it is called "subjective verification".
"Objective" here is an objective to the computing entity.
The shortest path search using slime mold is acomputing using organisms of slime molds andverification is objectively performed (by the shortest path
length). Hamiltonian path search by DNA computing2
and DNA origami3 are also objective verification.
In addition, when the subject of computing is not
human, for example, in DNA replication system, thereplication result is subjected to objective verification by
thermodynamic calibration4, and even if protein isinactivated due to replication error, objective verification
is carried out by evolutionary selection. Meanwhile, ingraphics and composition using computers, since thework is produced by subjective observation by the drawer(verification of correct answer set) it is subjective
verification. Even with the same computing entity, there
are cases where objective verification and subjective
verification are used together by replacing theverification system.

2.2. Observation of Computing
When the way to observe the state change by thealgorithm is defined, the system becomes computing.
Observations are defined by predefined correct answer
sets and intersection sets of normal type sets. An element
included in this intersection set becomes a correct
candidate. For example, when searching for the shortest
path of a maze by using slime molds, the normal typegroup becomes the state of slime mold1 after a certain
period of time. In some cases, it may have reached fromthe entrance to the exit, various state changes such as
stopping on the way, splitting into multiple parts,
stopping at the entrance, etc. are all normal types.
Here, if the correct answer set is "an individual thathas reached from the entrance to the exit", the correct
answer candidate is "an individual that has reached fromthe entrance to the exit of the maze" among the various
regular forms. From the viewpoint of the observing© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan230

Natural Computing and Formal

3. Computational Aesthetics

4. Natural Computing and Formal Computing

How is "verification" in the case of subjective
verification carried out? In the case of objective
verification, verification is carried out by an externalmechanism such as mathematics outside the computing
entity, but subjective verification does not have a reason
like mathematics. In that case, it is beauty to become thebase of the verification. Here, beauty is the beauty in a
broad sense including arts, comfort, comfort, stability,
safety and other computing entities. All criteria that judge"good" are called beauty5.

Computing is "state change due to order of action and its
observation", which is not done only by computers or
human beings.
For example, natural selection in evolution is objective
verification. However, even without assuming Darwinianevolution, subjective verification is repulsion, attraction
and structural stability / instability in substanceinteraction. If the interaction between the substances isstable (aesthetic state) the substances constitute thestructure and if it is unstable (in a state not beautiful) the
structure cannot be maintained. Therefore, even if we do
not assume objective verification like Darwin evolution7,
the stability and instability of interaction between
substances becomes subjective verification.
Since the computational system that subjectively
verifies is more primitive, the computing that performs"state change by order of action and observation +subjective verification" is called natural computing. On
the other hand, computing that perform objective
verification are called formal computing.

3.1. Computational Aesthetics of Alberto
Giecometti
French artist, Giacometti (Alberto Giacometti: 1901 1966)6 is known for sculptural works sculpted to the limit,
his work is like a bone, but there is a rich and expansive
presence if fulfilled. Philosopher Isaku Yanaihara served
as a model for sculptural works of Giacometti and left aconversation with him during the work making a note.
In the process of making sculpture, Giacometti scrapes
extra clay, eventually all the clay is scraped and nothingis gone, and as Giacometti tries to capture Yanaihara witha graphic, there is still no satisfaction with the giant meter
Return to sculpture.
What is interesting in this case is that computing does
not halt because the subjective verification continues tobe incorrect, so that calculation continues even if thecomputing "case (canvas or clay)" eventually disappears.
There were no external noises (the date of Yanaihara's
return home was decided), etc. There was no shaky
subjective verification and the calculation continued until
the correct answer was reached.
If the computing entity is an electronic computer or arobot, calculation cannot be continued once the enclosureis exhausted. However, even if the chassis disappears, if
we can take the algorithm and the verification system toa moving case in some way, we can continue thecalculation while moving from the case to the case until
we get the correct answer. Such a calculation system has
already been realized and implemented in natural
systems. That is, we are biological systems.

5. Natural Computing and Formal Computing
We defined computing with subjective verification as
natural calculation. Natural systems including humanbeings in general, natural calculations are ubiquitous.
For example, it is said that wine8 called wine today is
discovered to have been naturally fermented by collapsed
grape where the grapes are clustered by chance, and that"delicious liquid" In order to reproduce the trial and error
it may have been repeated variously, such as crushing thegrapes etc. on the spot and adding water.
The trial and error of winemaking is "state change ofthe grape liquid by order of action", and the normal formis "the unnoticeable liquid" obtained as a result of thestate change. And the observation system is the correct
set of our taste. Therefore, winemaking can be regarded
as a computing system (wine computing system in thefollowing).
If the wine computing system is natural computing,even a suspicious liquid (correct answer candidate) thatcannot be said to be very wine, it becomes wine if it
becomes "correct answer" as a result of subjective
verification of the calculation subject.
However, that "liquid that the computing entity
verified with wine" would be considered "incorrect" from© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan231

Yasuhiro Suzuki

Acknowledgement

other subjective verification systems (i.e. those who tastethat wine).
When observing with many people, it is necessary tomodify (program) the algorithm so that wines that many
subjective verification systems observe as "correct
answers" can be brewed. The subjective verification
system becomes a group and constitutes objective, so
natural computing evolves to formal computing.
Attempts to change natural computing to formal
computing by evolving from a subjective verification
system to an objective verification system is a general
method of natural science.
In many cases, a big discovery comes from anirrational intuition such as intuition and experience by
one scientist. Objective verification is difficult when thenovelty of the discovery is high. Therefore, we repeatedly
make "a phenomenon that might be a big discovery" andmake efforts to enable objective verification.
There, attempts have been made to move from anatural calculation system to a formal calculation system,
and as a result, a perfectly rational form calculation
system is constructed (in many cases in the case ofmathematics or physics) or a consultative formal
computing is constructed (Wine computing).

This research has been supported by Grant in Aid for
Scientific Research (KAKENHI) No:15K00268 and
16H03093
References
1.

2.
3.

4.

5.

6.
7.

8.

6. Future Natural Computing and Artificial
Intelligence
Computer science started by fusion of formal calculationand natural calculation. Computer science andmathematics involve natural calculations (humanrecognition), so it is difficult to construct a completely
rational formal calculation system.
For example, research on artificial intelligence has
many difficulties. Many difficulties arise to develop
intelligence which is natural computing to formal
computing (such as frame problem9 and semantic
processing in natural language processing10)
In recent years, by changing the direction of artificial
intelligence research to controversial formal computing(statistics / use of corpus in natural language processing,etc.), a new development has begun to appear.
In the future, artificial intelligence will develop as
controversial formal computing that incorporates theinteraction with the environment, not rational formal
computing by "pure intelligence11".

9.

10.

11.

Nakagaki, T., Yamada, H. & Tóth, Á. Maze-solving
by an amoeboid organism. Nature (2000).
doi:10.1038/35035159
Adleman, L. Computing with DNA. J. Comput. Biol.
(1995). doi:10.1089/cmb.1995.2.1
Rothemund, P. W. K. Design of DNA origami. in
IEEE/ACM International Conference on ComputerAided Design, Digest of Technical Papers, ICCAD(2005). doi:10.1109/ICCAD.2005.1560114
Bennett, C. H. Dissipation-error tradeoff in
proofreading. BioSystems (1979). doi:10.1016/03032647(79)90003-0
Suzuki, Y. in SpringerBriefs in Applied Sciences andTechnology (2019). doi:10.1007/978-4-431-568445_1
Gaff, J. Alberto Giacometti. Artists in their time (2002).
Darwin, C. On the Origin of Species by Means of
Natural Selection. D. Appleton and Company (1859).
doi:10.1126/science.146.3640.51-b
Piškur, J., Rozpedowska, E., Polakova, S., Merico, A.
& Compagno, C. How did Saccharomyces evolve to
become a good brewer? Trends in Genetics (2006).
doi:10.1016/j.tig.2006.02.002
McCarthy, J. & Hayes, P. J. in Readings in Artificial
Intelligence (1981). doi:10.1016/B978-0-934613-033.50033-7
Ng, H. T. & Zelle, J. Corpus-Based Approaches to
Semantic Interpretation in NLP. AI Mag. (1997).
doi:10.1609/aimag.v18i4.1321
Searle, J. R. Chinese Room Argument. MIT Encycl.
Cogn. Sci. (1999). doi:10.4249/scholarpedia.3100© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan232

A Study on Tourism Support Application Using the Virtual Technology
Masamichi Hori1
1

Graduate of Engineering, University of Miyazaki,
1-1 Gakuenkibanadai-nishi Miyazaki-shi, Miyazaki, 889-2192, Japan
Makoto Sakamoto2*, Takahiro Ishizu1, Satoshi Ikeda2, Amane Takei2, Takao Ito3, Yu-an Zhang4
2

Faculty of Engineering, University of Miyazaki,
1-1 Gakuenkibanadai-nishi Miyazaki-shi, Miyazaki, 889-2192, Japan
3
Graduate School of Engineering, Hiroshima University,
1-3-2 kagamiyama Higashihiroshima-shi, Hiroshima, 739-8511, Japan
4

Department of Computer Technology and Application, Qinghai University,
251 Ning Da Lu, Xining, Qinghai, 810000, China*Corresponding Author
E-mail:sakamoto@cs.miyazaki-u.ac.jp
http://www.miyazaki-u.ac.jp/

Abstract
Currently, the tourism industry in Miyazaki prefecture has various problems, and various measures are taken. On theother hand, in 2016, virtual technology attracted much attentions such as VR and AR technology. We thought thatthrough the VR and AR technology experience, we could aim for an increase in tourists. Therefore, we will create
new applications for smartphones using virtual technology and aim for tourism support. We propose unique features
aimed at increasing tourists and creating repeaters. It is a function that allows to reproduce the scenery anywhere withVR, and to feel the sightseeing experience by AR close.
Keywords: Augmented reality, Virtual reality, Smartphone application, Traditional culture, Sightseeing.

world content and extends them. VR can gain unrealistic
experiences in a new world on computers. These arefound in a wide range of fields such as advertisement,
tourism, education and entertainment [2] [3]. In thisstudy, we will link virtual technology and tourism
industry and aim to solve problems related to sightseeingof Takachiho. Specifically, we provide functions thatmake it easy to experience images that reproduce
beautiful landscapes by VR technology, and that enable"take-away" of tourism experiences through AR
technology.

1. Introduction
Due to the popularization of mobile terminals and theemergence of interactive technology, the tourism
industry is changing drastically. In order to be a tourist
destination that is attractive and win the competition, it isessential to improve unique experiences and value-added
services. Miyazaki Prefecture · Takachiho Town is oneof the most famous sightseeing places in Japan, but it hasvarious problems and countermeasures have been taken[1]. Especially, it is urgent to introduce new technology.
In 2016 and 2017, the virtual technology called AR, VR,
MR showed great excitement and it was the year when it
became commonly recognized. AR associates with real-© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan233

Masamichi Hori, Makoto Sakamoto, Takahiro Ishizu, Satoshi Ikeda, Amane Takei, Takao Ito and Yu-an Zhang

1.1. BackgroundCentred Design" [7] is an AR application design with anemphasis on discussion with users. This study was
revealed what visitors really wanted in the AR display,and how the AR navigation should be displayed.
"Augmented Reality and Gamification in Heritage
museums" [8] conducted a definition and analysis of AR
communication model and noise at the museum, andapproached better experience. Furthermore, adding thegame elements revealed that the depth of the AR system
will increase.
As a study of the meta-viewpoint of introducing AR,
there is "How can Tourist Attractions profit from
Augmented Reality?" [9]. This study revealed the AR
profit model through collaboration with stakeholders in
the field and showed the possibility to introduce AR withminimal risk.
The impact on sightseeing in the AR application"Pokèmon GO" which became one of the keywords in2016 is great. According to the official report, thecollaboration event with Miyagi Prefecture in November
2016 mobilized about 100,000 people in 11 days, tourism
consumption amount of 2 billion yen [10], thecollaboration event with Tottori Prefecture in November
2017 mobilized about 89,000 people in 3 days, tourism
consumption amount of 1.3 billion yen and PR effect 500
million yen [11]. Players visit sightseeing spots triggeredby content capabilities, leading to attraction discovery
and economic activities. It is a characteristic result of AR
based on reality.
"Virtual Reality and Implications for Destination
Marketing" [12] is investigating how promotional
contents of VR technology will affect the image of tourist
spots. It is said that it will have a good influence on mostitems including the effectiveness of advertisement, also
highly likely to share experiences of destinations andadvertisements themselves.
"Virtual reality, presence, and attitude change: Empiricalevidence from tourism" [13] is a survey on VR tourist
content targeting 1,000 people, regarding the presence of
the VR experience, it showed that the attitude towards thedestination of the consumer is positive and VR iseffective.

The tourist site that is the target of this study is Takachiho
Town, Miyazaki Prefecture. Takachiho Town is locatedat the northernmost tip of Miyazaki Prefecture the centerof Kyushu. It has a beautiful natural treasure Takachiho
Gorge, a great change due to the four seasons, and it isblessed with wonderful nature. Takachiho Gorge andAkimoto shrine are shown in Figure 1. The history of thetown is old and there are reports that there were villages
since about 4000 BC [4]. It is a place where God
descended in Japanese mythology and is regarded as aland with considerably deep connection with the birth ofJapan. There are Kagura, born from the Amanoiwato
legend of Japanese mythology, and land that is related tomyths. In recent years as a power spot attracts attention
from a wide range of age groups.
Especially the Akimoto area called Okutakachiho
focuses on people living in the village. And they are
doing have tourists experience village-specificexperiences activities. Tourists can eat cuisine using
regional ingredients, and can visit shrines and sacredtrees standing in majestic nature. In 2016, it was selected
as a good example of the "treasure of Discover rural
mountain fishing village" designated by the Ministry ofAgriculture, Forestry and Fisheries, and it is thought that
tourists will further increase in the future [5].
However, as a change in the environment surrounding
tourism, Miyazaki prefecture states that the population
declines, the declining birthrate and aging, intensifiedcompetition, and the development of information andcommunication technology are left behind.
The number of sightseeing tourists is at a lower level [6]than at peak hours. Creating attractive sightseeing sitesthat want to visit many times, and sending out
information using media, ICT, and SNS are issues.

2. Method
We will explain the method used to create the application.

Fig.1 Takachiho Gorge(left), Akimoto shrine(right).

2.1. 3DCG modeling1.2. Literature review

About 3DCG model production cooperate withvocational schools in Miyazaki prefecture. The 3DCG
model to be produced is shown in Figure 2. It is a
reproduction model of God appearing in Kagura, theMajor use of AR in tourist spots is to combine withmobile terminals as a guide tool. It is AR use as a bridgeconnecting tourist spots with visitors. "ToARist: An
Augmented Reality Tourism App created through User-© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan234

A Study on Tourism

2.2.2. AR

character of Takachiho's special product, and hawk
inhabiting Takachiho.

The markerless type is inputting a natural image (animage having a use other than a marker). We'd like to run
it as an application alone, and implement it on a mobile
terminal, especially on a smartphone. When recognizing
the pamphlet of Kagura, God dances and whenrecognizing the "Chihomaro" package , the character isdisplayed. As "takeaway" of the unique function, whenrecognize the hand, dance Kagura on the palm of thehand. Figures 5 show the state of AR.

Fig.2 Bird’s eye view of Tadikarao (upper left), hawk (upper
right), “chihomaro” character (bottom).

2.2. System
We chose Unity 3d as a software that can produce
applications for mobile from the viewpoint of future
development and application creation that is not boundby OS. We used Blender for model creation. The device
used is an Android smartphone.

Fig.5 Pamphlet recognize (left), package recognize (center),“take away” hand dance (right).

2.3. Experimet
Application experience and questionnaire survey are
conducted for five 20's. The task is done for 3 minutes
and we respond to questionnaires about 6 items (scale of
1 to 5) and 1 free opinion. Items are about virtualtechnology and sightseeing, about contents, and so on.

2.2.1. VR
We use smartphone and VR glasses certified by googlecard board to realize VR function. We installed a simple
VR function by attaching it to a smartphone. Figure 3
shows the VR glasses. With this function, can see thestate of Takachiho reproduced with 3DCG. Figure 4
shows the state of the VR space.

3. Result and Discussion
In the item "Would you like to visit your destination after
experiencing virtual technology?", the score is 4.2 points.
By experiencing virtual technology, we think that
stimulating tourism motivation and being able to feeltourism more closely. "Would you like to share virtualtechnology contents?" is positive result. Virtualtechnology is generally known, furthermore we also
think that using a simple VR glasses or a markerless typeAR can easily produce a share effect. There is an opinion
that prioritize enjoy the sightseeing experience indestination over virtual technology. However, we think
can provide a special tourism experience by applying theguidance and shooting with the virtual character by AR.
"Is it easy to prepare for experiencing virtualtechnology?", "Is the operation of VR · AR good?","Contents fulfillment level" were negative point. It isbecause there were work to put a burden on the user suchas attaching VR glasses to devices and obtaining natural
images to be recognized by the AR. Also, about theFig.3 VR glasses.

Fig.4 VR takachiho.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan235

Masamichi Hori, Makoto Sakamoto, Takahiro Ishizu, Satoshi Ikeda, Amane Takei, Takao Ito and Yu-an Zhang

operation of the virtual technology, it was not possible tosmoothly read the hand at the AR, which led to a negative
opinion. It is necessary to revise the complete markerless
AR on the smartphone.

3. Zenji Nishikawa, Katsuomi Kobayashi, Yaseinootoko,izm, Kazuya Hiruma, “VR contents development guide”,
MDN Corporation, 2017, pp18-33.
4. Overview of Takachiho | Introduction / History of the town
| administrative information | Takachiho cho [Online]
Takachiho cho information official website.
http://www.town-takachiho.jp/administration/history/outl
ine.html (accessed 2018/11/1).
5. Takachiho Mura Council "Treasure of Discover Rural
Farm Village (Mura)" was selected. | What's New |Takachiho-cho Tourist Association [Online]
Takachiho cho information official website.
http://takachiho-kanko.info/news/detail.php?log=147968
8433 (accessed 2018/11/1).
6. Miyazaki Prefecture tourist statistics survey result inHeisei27. [Online]
Official website of Miyazaki Prefecture
https://www.pref.miyazaki.lg.jp/kanko-suishin/kanko/mir
yoku/documents/26823_20161226163703-1.pdf(accessed 2018/11/1).
7. Williams, M, Yao, KKK and Nurse, JRC et al.,
“ ToARist: An augmented reality tourism app createdthrough user-centred design”, 31st British Human
Computer Interaction Conference, 2017.
8. Ramy Hammady, Minhua Ma, Nicholas Temple,“ Augmented Reality and Gamification in Heritage
Museums. ” , Serious Games, Vol. 9894, pp 181-187,
Springer, Cham, 2017.
9. Eleanor E Cranmer, M. Claudia tom Dieck, Timothy Jung,“How can Tourist Attractions profit from Augmented
Reality? ” , Augmented Reality and Virtual Reality Empowering Human, Place and Business, pp 21-32,
Springer, Forthcoming, 2017.
10. About the number of tourists etc. of Pokèmon GO related
events - Official website of Miyagi Prefecture. [Online]
Miyagi Prefecture official website.
https://www.pref.miyagi.jp/release/ho20161219-6.html(accessed 2017/12/1).
11. Number of participants in "Pokémon GO Safari Zone in
Tottori Sand Dunes" and Economic Effect. [Online]
Tottori prefecture official website.
http://www.pref.tottori.lg.jp/271365.htm(accessed 2018/11/1).
12. Griffin Tom, Giberson Juleigh, Lee Seung Hwan (Mark),
Guttentag Daniel, Kandaurova Maria, Sergueeva Ksenia,
Dimanche Frédéric, "Virtual Reality and Implications for
Destination Marketing", Travel and Tourism Research
Association: Advancing Tourism Research Globally, 29,
2017.
13. Iis P. Tussyadiah, Dan Wang, Timothy H. Jung,
M.Claudia tom Dieck, “Virtual reality, presence, andattitude change: Empirical evidence from tourism” (2018).
Tourism Management Vol 66, Pages 140-154, 2018.

4. Conclusion
In this study, we created tourism support application
using two virtual technologies, VR technology and AR
technology. VR function can simulate the tour of
Takachiho. In AR technology, we implemented amarkerless type AR that displays 3DCG models byreading pamphlets and packages. Also, as a uniquefunction, the 3DCG model can dance on the palm bycomplete markerless AR. This function is called"takeaway" and is aiming for the share effect.
Based on the evaluation results, we found that byintroducing virtual technology to tourism support, we
will affirm the attitude towards the destination. Easy toshare with simple making, can aim to create new tourists
and repeat customers. There is a positive opinion in the
AR function at the tourist resort, there is a possibility thatit can provide a unique experience. As the contents
enhancement, improvement of interactivity whenexperiencing VR, and the AR function that takes acommemorative shooting with the 3DCG model. By
applying stamp rally type gamification, can aim not only
use applications before sightseeing and after sightseeing
but also use applications during sightseeing.
Acknowledgements
Miyazaki multimedia vocational school people are very
busy, thank you hearty thank you for your cooperation in
3DCG modeling and advice etc. Also, the Miyazaki
Tourism Convention Association, Inc., abc Corporation,Inc., Phoenix Seagaia Resort Co., Ltd., people in the
Akimoto district who took care of me in the interview,other prefectures within the prefecture We would like toexpress my gratitude to all concerned parties in eachmunicipality.
References
1. Heisei27 July Miyazaki Prefecture Tourism Promotion
Plan ~ Toward revival of tourism miyazaki ~. [Online]
Official website of Miyazaki Prefecture
https://www.pref.miyazaki.lg.jp/kanko-suishin/kense/shingikai/documents/37281_20180613084219-1.pdf(accessed 2018/11 / 1).
2. Takashi Kurida, Kiyoshi Kiyokawa, Takashi Okuma,"Fundamentals, Development and Practice of AR
Technology", Science Information Publishing, 2015, pp
377-385.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

236

Proposal for Interaction Techniques for Intuitive Virtual Objects Manipulationin Augmented Reality
Takahiro Ishizu, Makoto Sakamoto*, Takaaki Toyota, Masamichi Hori, Satoshi Ikeda and Amane Takei

Faculty of Engineering, University of Miyazaki,
Miyazaki-City, Miyazaki, Japan
Takao Ito

Graduate School of Engineering, Hiroshima University,
Hiroshima-City, Hiroshima, Japan
Yu-an Zhang

Department of Computer Technology and Applications, Qinghai University,
China
E-mail: hm14002@student.miyazaki-u.ac.jp, fruits2000jp@yahoo.co.jp*Corresponding Author
E-mail:sakamoto@cs.miyazaki-u.ac.jp
http://www.miyazaki-u.ac.jp/

Abstract
Recently, studies for realizing interaction between users and virtual objects using Augmented Reality (AR) have been
actively conducted. However, few previous studies have realized that a user intuitively manipulates virtual objects
like clay with their bare hands. In this study, we aim at user's intuitive virtual objects manipulation by realizing basic
manipulation on virtual objects with their bare hands. In addition, it is considered that the user feels uncomfortablein the manipulation of the virtual objects because of the occlusion problem in AR, and cannot perform an intuitive
manipulation. Therefore, in this study, we solve this problem by performing appropriate hidden surface processing
within the range of the user's fingertips.
Keywords: augmented reality, intuitive manipulation, virtual objects, gesture interaction, occlusion problem.

hand to manipulate virtual objects in the virtual space for
intuitive manipulation [1].
Kato et al. [2] introduced soft body virtual objects byusing a physics engine, reflected the action of crushingon the virtual object with the user's bare hand in thevirtual space, and realized the deformation manipulationof virtual objects by interaction with the user.
Suzuki et al. [3] realized a move manipulation of rigid
body virtual objects by interaction with the user.
However, it was not realized that after deforming thevirtual object, moving or join the virtual object while
maintaining that shape.

1. Introduction
Augmented reality (AR) is a technique of superimposing
information generated by a computer on perceptual
information that we receive from real space.
In recent years, technique for realizing interaction
between a user and virtual objects in AR has attracted
attention.
In interaction technology in AR, it is necessary for theuser to be able to intuitively manipulate the virtual
objects.
Since the main means of the interaction with theobjects in real life is hand, it should also be possible for© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

237

Takahiro Ishizu, Makoto Sakamoto, Takaaki Toyota, Masamichi Hori, Satoshi Ikeda, Amane Takei, Takao Ito and Yu-an Zhang

In this study, we realize to intuitively perform user's
manipulation such as deformation of virtual object,
maintenance of shape after deformation, joining, and soon, aiming for more advanced interaction between a user
and virtual objects.
We believe that this study is useful, for example, byapplying this study to modeling work of 3DCG, so that it
can be more intuitively.
2. Proposal
In this paper, we propose a method to create new virtual
objects and a method to distinguish whether the user
wants to grasp or wants to move the virtual object. In thisway, we realize that the user creates a virtual object,
deforms it, moves it while maintaining the deformed state,
and joins the two virtual objects.
Moreover, we perform a hidden surface processing toallow the user to recognize the context of the user's hand
and the virtual objects.

Fig.1. System Component

3.

Hidden Surface Processing

In AR, a virtual object superimposed on the image of thereal space later. For this reason, even when the virtual
object is located deeper than the real object, the virtual
object is not hidden by the real object, and it is displayed
as if the object is always in front . This is called an
occlusion problem in AR, and when the user manipulates
the virtual object, the user cannot recognize theanteroposterior relationship between the virtual object
and the user's hand correctly [6]. For this reason, we thinkthat the user feels uncomfortable visually and cannot
perform an intuitive manipulation. Therefore, in thispaper, we solve this problem by hidden surface
processing of hands and the virtual object in the virtual
space, using three-dimensional coordinates of fingers
obtained with the Leap Motion Controller andrecognizing hand region from skin color region in RGB
image obtained with Web camera

2.1. System Component
We make it possible to realize the interaction of the user
and the virtual objects by adopting the Leap Motion
Controller specialized for acquisition of threedimensional coordinates of fingers and introducing softbody virtual objects.
This system consists of a Web camera for real image
acquisition and marker recognition, the Leap Motion
Controller for obtaining three-dimensional coordinates offingers, and PC for performing arithmetic processing andvideo output. In this system, we use network
programming to prevent interference between the Leap
Motion Controller and the Web camera [4]. In this system,
we perform acquisition processing of three-dimensional
coordinates of fingers with the Leap Motion Controller
on the server side. And we perform acquisition
processing of image with the Web camera, physicalcalculation, and video output on the client side.
As shown in the Fig.1, we place the paper on which theAR marker is printed and the Leap Motion Controller,
and observe the paper with the Web camera. The virtual
objects are displayed by recognizing the AR marker.
In this system, we use virtual objects that can bephysically computed using the physics engine (Bullet
Physics) [5].

4.

Manipulation of Virtual Object

In this section, we will explain the manipulation of virtual
objects by bare hand to realize more intuitive interaction
between a user and virtual objects.

4.1. Generation of Virtual Object
The user can newly generate a “virtual object to be
manipulated”, which size is desired by user by gesture ofindex fingers of both hands (Fig.2).
The procedure will be described below.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan238

Proposal for Interaction Techniques

4.3. Joining of The Virtual Objects(i) This system acquires the three-dimensional
coordinates by putting the user's index finger of theright hand on the Leap Motion Controller.
(ii) This system acquires the three-dimensional
coordinates by putting the user's left index finger on
the Leap Motion Controller while keeping the stateof Step 1.
(iii) The user away the own left index finger with theLeap Motion Controller.
(iv) This system determines the size of the “virtual object
to be manipulated” based on the value when theindex finger of both hands comes closest andgenerate the object.

The user prepares two virtual objects in advance. The
user brings one “virtual object to be manipulated” closer
to the other “virtual object to be manipulated”.
Since collision determination between “virtual object
to be manipulated” is not performed in this system, whentwo “virtual object to be manipulated” continue closer toeach other, they eventually overlap each other. And whenreaching a fixed distance, the two virtual objects join
together (Fig.5).

4.2. Deformation and Moment of The VirtualObject
It is difficult to distinguish whether the user wants tograsp or wants to move the virtual object. Therefore, thissystem performs deformation of the virtual object when
the user's palm is downwards, and movement of thevirtual object when the user's palm is upward. Thissystem decides that the palm is upward when the user's
little finger, ring finger, middle finger, index finger,
thumb line up from the left of the Leap Motion Controller.
In this system, deformation and movement
manipulations on virtual object are performed with only
the right hand.
The user turns the palm downwards and crush the“virtual object to be manipulated” with the thumb andindex finger, whereby the user's “virtual objects attachedto the fingertips” and “virtual object to be manipulated”
collide with each other, and the “virtual object to be
manipulated” can be deformed (Fig.3).
The user can move “virtual object to for movement”
which hold the representative coordinate of “virtual
object to be manipulated” by turning the user's palm
upwards and moving his/her hand near the virtual object(Fig.4). Since “virtual object for movement” is attached
to “virtual object to be manipulated” in advance, “virtual
object to be manipulated” can be moved by moving“virtual object for movement”.
In the case where two virtual objects are generated,
this system obtains the Euclidean distance between thecenter of the hand and each “virtual object for movement”
of each of the two virtual objects and selects a closer“virtual object for movement” as the moving target.

5.

Fig.2. Create Object

Fig.3. Deform Object

Fig.4. Move Object

Fig.5. Join Objects

Evaluation

We investigated the interaction with the virtual object
using this system by questionnaire to evaluate whether it
is intuitive or not and to extract this system's
improvement point. We asked six university students ofsubjects to perform basic manipulations on virtual
objects shown in Fig.2 to Fig.5.
5.1. Experiment Method
After having the subject trial this system, we conducted
a question form questionnaire on each item of “goodpoint”, “difficult point” in manipulation, and“improvement point” in this system, and we evaluatedthis system.

5.2. Experiment Result
The questionnaire result was as follows.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan239

Takahiro Ishizu, Makoto Sakamoto, Takaaki Toyota, Masamichi Hori, Satoshi Ikeda, Amane Takei, Takao Ito and Yu-an Zhang

As for “good point”, there was some opinions such as“manipulation method is intuitive and easy tounderstand”, and “movements of the hands was reflectedin the system”.
As for “difficult point”, there are some opinions suchas “it is difficult to join two virtual objects”, “it is difficult
to grasp the sense of distance with the virtual object”, and“when the hand is manipulated at a low position from thedesk, the virtual object does not react in user's gesture”.
As for “improvement point”, there are some opinions
such as “I want feedback when crushing virtual objects”,“I want functions that can be transformed into arbitrary
figures such as cones and rectangular parallelepipeds by
gesture” and “I want a manipulation function to cut off
virtual objects”.

6.

7.

Conclusion

In this study, we realized a more intuitive manipulation
between a user and virtual objects by creating new virtual
objects and distinguishing whether the user wants tograsp or wants to move the virtual object.
In the evaluation experiment, we asked six subjects toperform the basic manipulation on the virtual objects andconducted a questionnaire survey. As a result of theexperiment, it was shown that by using this system, it ispossible to intuitively perform the interaction
manipulation with the virtual objects.
In our future work, we will address the issues of“difficult point” and add functions pointed out in“improvement point” so that it can more intuitively
interact with virtual objects. Furthermore, we think that
it is necessary to increase the precision of interaction,such as enabling finer deformation of virtual objects.

Consideration

References

First, we consider “good point”. From the “good point”,
we found that the interaction manipulation with thevirtual object incorporating the gesture is intuitive.
Therefore, we believe that it is possible to superimpose
the physical phenomenon such as deformation object byhand in the real space on the virtual objects.
Next, we consider “difficult point”. We think that thereason why the cause of the opinion that “it is difficult tograsp the sense of distance with the virtual object” is thatwhen the user manipulates the virtual object, an error
occurs due to a delay in the position information of thefinger acquired with the Leap Motion Controller, and theuser feels a sense of incompatibility.
We think that the reason why the cause of the opinion
that “it is difficult to join two virtual objects” is that inthis system, the coupling of virtual objects are joined
when the two virtual objects approach within a certain
distance, similarly to the reason that “it is difficult tograsp the sense of distance with the virtual object”, anerror occurs between the position of the finger on the real
space and the position of the finger on the virtual space.
We think that the reason why the cause of the opinion
that “when the hand is manipulated at a low position fromthe desk, the virtual object does not react in user's gesture”
is that when the hand is in a low position, the hand covers
the Leap Motion Controller, and the coordinates of thefingertips cannot be acquired we considered.

1. V. Buchmann, S. Violich, M. Billinghurst and A.
Cockburn.
“FingARtips-Gesture
Based
DirectManipulationin in Augmented Reality”.
Proc.
International Conf. on Computer Graphics andInterractive Techniques, pp.212-221, 2004.
2. S. Kato, S. Ishigaki, S. Suzuki and M. Sato．“A Study on
Presentation of Softness of Virtual Object in Gripping
Motion with Bare”．Journal of The Institute of Image
Information and Television Engineers，2017，VoL 71，
No. 3，pp.J121-J124．
3. H. Suzuki ， S. Suzuki and M. Sato ． “Visual
Improvement of AR System Focusing on Grasping Motion
with Bare Hands”. Journal of The Institute of Image
Information and Television Engineers，2014，VoL 68，
No. 7，pp. J299-J301．
4. U. Imamura, et al. “The proposal and development of thenovel technology with Augmented Reality”. Future
University Hakodate 2011 System Science Practice Group
Report, 2012．
5. F. Fujisawa. “Information media experiment, Application
development using physics engine”. [Online]
http://slis.tsukuba.ac.jp/~fujisawa.makoto.fu/lecture/iml/(accessed 2018-03-17).
6. K. Reiji and S. Makoto ． “Development of A Systemenabling Display of Gripping the AR Object Using Leap
Motion Controller”. The 29th Annual Conference of theJapanese Society for Artificial Intelligence, 2015．

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan240

Two-Dimensional Image Based Body Size Measurement and Body Weight Estimation for YaksZijie Sun

Department of Computer Technology and Applications
Qinghai University, 810016, Xining China
Chen Zhang

Department of Computer Technology and Applications
Qinghai University, 810016, Xining China
Xiaofeng Qin

Department of Computer Technology and Applications
Qinghai University, 810016, Xining China
Yu-an Zhang*

Department of Computer Technology and Applications
Qinghai University, 810016, Xining China
Rende Song

Yushu Prefecture Animal Husbandry and Veterinary Station
Yushu Qinghai, 815000, China
Makoto Sakamoto

Faculty of Engineering
University of Miyazaki, Miyazaki City 889-2192 Japan
Abstract
non-contact acquisition of the body size and weight of the animals is one of the hot research topics in the animalhusbandry sector. In this paper, the yaks were photographed in the Sanjiangyuan Region, and their body weight wasestimated by foreground extraction, identification of measuring points, linear regression and other methods.
Keywords: Sanjiangyuan; yak; image; foreground extraction; body size; body weight.

research, it is particularly important to measure yak
body size and body weight. In this paper, the color
images of the yaks were preliminarily processed, andthen the yak images were identified through gray scaleclassification. By identifying the processed images, 4
points for measuring the body height and body steep
length were marked based on curvature calculation
results. Finally, the actual body size of the yaks was
obtained based on the camera imaging principle. The
body weight formula for the yaks was given after linear
regression using weka, and the actual body weight of
the yaks was finally estimated.

1. Introduction
Yak is a kind of unique cattle which lives at an altitude
of above 3000m, mainly on the Qinghai-Tibet Plateau
and adjoining mountains and in the subalpine regions.
Yak can adapt to extreme ecological conditions such as
high & cold environment and oxygen-deficientenvironment[1], and is essential to animal husbandry
economy on the Qinghai-Tibet Plateau. China is thebiggest yak breeding country in the world, and accounts
for 92% [2]. Specifically, Qinghai Province raises 4.9
million yaks, and accounts for 38% in China [3].
With the improvement of information perception
technology and precision breeding level, the livestock
body size and body weight are more acquired by noncontact and high-precision methods [4]. As the yaks
occupy a special position on the Qinghai-Tibet Plateau,
yak research is particularly important. During yak*

2. Foreground Extraction
The experiments showed that the gray scale of the dark
yaks was the lowest in the gray-scale images, so the
pixel points of which the gray scale was greater than150 basically were not those of the dark yaks. Based on

Corresponding author: Yu-an Zhang,e-mail:2011990029@qhu.edu.cn© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

241

Zijie Sun,Chen Zhang,Xiaofeng Qin,Yu-an Zhang,Rende Song,Makoto Sakamoto

the yak images (Figure 1), Figure 2 gave the gray scalesof the images with different colors, with 50 gray scales
as a unit, in which the gray scale of the red part, yellow
part, blue part, white part and black part was [0,50],[51,100], [101,150], [151,200] and [201,255]
respectively.
It can be seen that the red part basically showed the
body type of the yaks. Based on this, the dark yak
images may be extracted from the color images.

on Formula 1 (standard deviation σ=1). Afterwards, thepictures were convolved by 3*3 Gaussian convolution
kernels after compression and graying for image
blunting and subsequent processing [5]. One thresholdvalue for background removal was determined, and the
pixels around the images were traversed. It wasdetermined whether the pixels were background pixels
based on the tolerance [6]. The background wasremoved if the answer was affirmative, otherwise
retained.

2.1. Background Removal
2.2. Classification

During image background removal, we firstly
passivized the images based on the Gaussian bluralgorism for which the formula was shown below:

f x ,yx2 y2

12

After image background removal, the gray scales of theimages were classified, and the yak center points in theimages were calculated. The images were continuously
iterated by the flood fill method, until the images
became smooth [7].
Figure 3 showed the classified images and center points.
The gray scale of the red part, yellow part, blue part andwhite part was [0, 50], [51,100], [101,150] and[151,200] respectively; the black part showed removed
background; the pink part showed the yak center points
pre-determined by the system in the images. Upon
determination, the overall shape of the yaks wasdetermined through extension from the center points.
After yak image foreground was extracted through

2

e

2

2

(1)

3*3 Gaussian convolution kernels were generated based

Fig. 1. Yak Image.

Fig. 3. Classified Image and Center Point.

classification, the extracted yak images perhaps were
unsmooth. Therefore, the images can be dilated and theneroded.
Final yak foreground images were shown in Figure 4.
Fig. 2. Gray Scale Classification of Yak Image.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

242

Two-Dimensional Image Based3.1. Identification of Point of Sideburns (Point T)
After the center points and image edges of the yaks
were determined, it can be determined that the point of
sideburns was at the upper left or upper right of thecenter point according to the head direction of the yaks.
Extensive experiments showed that the point of
sideburns of the yak was at the maximum point ofcurvature of the contour curve for the yaks which
formed an angle of 30°-60° to the yak center point in theyak curve contour map.
During curvature calculation, one triangle can beformed by one contour point of the area where thepoints of yak sideburns were located and two adjacentpoints, as shown in Figure 6. Three sides and perimeter
P of the triangle were calculated. The area S of thetriangle was obtained according to the Heron's formula,and the curvature K was calculated [11]. In Figure 6, thecalculation formula for P can be given according to thegeometrical relationships.

Fig. 4. Final Yak Foreground Image.
3. Identification of Measuring Point
According to traditional yak body size measurement
standards, the body steep length referred to the distancefrom the point of shoulder to the point of the buttocks,
as shown in TX-C Segment of Figure 5. The body steep
length was measured by a measuring stick. If the bodyweight was estimated according to the body size, thebody steep length may be measured by a tape. The bodyheight referred to the vertical distance from the point of
sideburns to the ground, as shown in T-G Segment of
Figure 5. The body height may be measured by ameasuring stick [8]. Once the point of buttocks, point ofshoulder and point of sideburns were accurately foundout, the body steep length and body height of the yaks
can be accurately acquired [9-10].

P

a

b

c

(2)

In the formula (2), P referred to perimeter, and a, b, creferred to three sides of the triangle. S may be
expressed as follows:

S

P(P

a )(P

b )(P

c)

abc
(3)
4R

Fig. 6. Determination of Point of Sideburns of Yak.

Fig. 5. Body Height and Body Steep Length of Yak.
After the binary foreground images of the yaks wereobtained, the edges of the binary yak images were firstly
extracted, and the yak center points were calculated.
After the standing direction of the yaks was known
through communication with the users, the images canbe divided into 4 parts, and points TX, C, T and G(Figure 5) were determined respectively. The bodyheight and body steep length of the yaks were finallyobtained.
When marking the points in the images, we firstly
extracted the edges of the images (Figure 4), and thencalculated the yak center points.

Fig. 7. Determination of Curvature.

In which, R referred to radius of circle.
In Figure 7, the calculation formula for curvature K wasshown below:

K

1

1
R

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

243

(4)

Zijie Sun,Chen Zhang,Xiaofeng Qin,Yu-an Zhang,Rende Song,Makoto Sakamoto

Final measuring points of the yak images were shown in
Figure 8, in which the point of sideburns (Point T),
lowest point (Point G), point of the buttocks (Point TX),
point of shoulder (Point C) and center point were shownin red, yellow, blue, black and pink respectively.

Thus:

K

4 P (Pa )( Pabcb )( Pc)(5)

Through the above steps, the curvature K of 3 noncollinear points can be obtained.
3.2. Identification of Lowest Point (Point G)
Upon determination of the center point, it can be
determined that Point G of the yaks was at the lower leftor lower right of the center point according to thestanding direction and center points of the yaks.
Afterwards, the lowest point can be determined only by
traversing the images.

Fig. 9. Camera Shooting Structure.

4. Estimation of Body size and Weight of Yaks

3.3. Identification of Point of Buttocks (Point TX)

The camera shooting structure was shown in Figure 9.
In Figure 9, H referred to actual object height; h referredto the object height on the image sensor; D referred tothe distance from the camera lens to the object; and freferred to lens focal length. Therefore, we can obtain
the actual height based on this principle.

Upon determination of the center points, it can be
determined that the point of the buttocks was at thelower left or lower right of the center point according to
the standing direction and center points of the yaks.
Extensive experiments showed that the point of the
buttocks of the yaks was at the minimum point ofcurvature of the contour curve for the yaks which
formed an angle of 10° - 40° to the yak center point in
the yak curve contour map[12].

4.1. Calculation of Body Steep Length and Body
Weight of Yaks
An image consists of pixels. Pixel is the smallest unit of
the image represented by a sequence of numbers. The
pixels can be deemed as indivisible units or elements inthe entire images [13]. Therefore, we can obtain theactual length H of the measured object according to
Formula 6.

3.4. Identification of Point of Shoulder (Point C))
Upon determination of the center points, it can be
determined that the point of shoulder was at the lower
left or lower right of the center point according to thestanding direction and center points of the yaks.
Extensive experiments showed that the point ofshoulder of the yaks was at the boundary point with anangle of 50° - 80° to the yak center point and with the

H

LvD
f

(6)

We shall obtain the sensor size and resolution of thistype of camera to calculate the actual length v
represented by each pixel. The area size of each pixel
can be calculated by dividing the sensor area of this typeof camera by its resolution. The actual length
represented by each pixel can be obtained throughsquare root calculation. Further, the actual body heightand body steep length of the yaks were obtained
according to Formula 6.
4.2. Calculation of Body Weight of Yaks
The actual body weight of the yaks can be estimatedaccording to the actual body height and body steep
length of the yaks.
The body height, body steep length and body weight of
597 yaks (0.5-5.5 years) were manually measured in the
Yushu Region, and linear regression was performed

Fig. 8. Final Yak Foreground Image.

shortest distance to the center point.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan244

Two-Dimensional Image Based7. Acknowledgements

using weka , so as to obtain the formula for estimationof the body height and body steep length of the yaks:
W=2.6123*H+1.3334*S-242.4778

This work is supported by Science and Technology
Project in Qinghai Province (No. 2017-ZJ-717); Modern
agriculture (cow calf) industrial technology systemproject (CRA-37); National natural fun projects(No.61640206);
References(7)

In Formula 7, W referred to body weight; H referred tobody height; S referred to body steep length.
5. Advantages & Disadvantages

1. Editorial Board of Chinese Yak Science, China Yak
Science (Chengdu: Sichuan Science and Technology
Press, 1989).
2. Wang, Q., Breeding Management Measures for
Domestication of Purebred Wild Yaks, Abstracts ofAnimal Husbandry & Veterinary Medicine(4), 18-18.
3. Yang Shi, B.J., Bao, S.K., Zhao, S.N., et al. Observationof Greenhouse + Supplementary Feeding Effects of Yaks
in North Pastoral Areas of the Qinghai Lake in Cold
Season, Heilongjiang Journal of Animal Science &Veterinary Medicine. 16 (2014) 124–124.
4. Zhang, L.N., Wu, P. Algorithm of sheep body dimension
measurement and its applications based on image
analysis. Computers and Electronics in Agriculture.
(2018) 33-35.
5. Réti, Z. Deblurring images blurred by the discrete
gaussian. Applied Mathematics Letters, 8(8), (1995) 2935.
6. María, M., & Lópezálvarez, M. A. Removing
Background Color from Blueprints. Color and Imaging
Conference. (Vol.volume 2011) pp.48-51(4)7. Kang, H., Lee, S. H., Lee, J. Image segmentation basedon fuzzy flood fill mean shift algorithm. Fuzzy
Information Processing Society (pp.1 - 6). IEEE (2010)
8. Measurement and Analysis of Body Weight and Body
Size of Yaks in Sanjiangyuan Region. (2017)
9. Blum, H., A transformation for extracting new
descriptors of shape. Models Percept. Speech Visual
Form 19, 362–380. China Cattle Science, (1967) 43(1),14-17.
10. Blum, H., Biological shape and visual science (part I). J.
Theor. Biol. 38 (2), (1973) 205–287.
11. Zhao, J.M., Zhao, Z.X., Li, Q., Kinect Sensor Based
Body Size Measurement Method for Sheep. Jiangsu
Agricultural Sciences, 43(11), (2015) 495-499.
12. Jiang, J., Yue, W., Cao, M.Z. Research on Machine
Vision Based Body Size Measurement Method for Sheep.
Journal of Inner Mongolia University of Science &Technology, 34(4), (2015) 322-327.
13. Zhang Z G, Dong F L, Zhang Q C, et al. Fabrication of
pixelated polarizer array and its application inpolarization enhancement. Acta Physica Sinica, (2014),
63(18): 184204(1-10).

According to this algorithm, two-dimensional images ofthe yaks were firstly identified based on classification
processing ideas, and the yak images were extracted
from the pictures by the flood fill method. Then, thecenter points and standing direction of the yaks were
determined, and the measuring points of the yak images
were finally obtained through curvature analysis. The
body height and body steep length of the yaks wereobtained based on the camera imaging principle. Afterthe formula for estimation of the body height and bodysteep length of the yaks was obtained by performing
linear regression using weka, the body weight of theyaks was finally estimated. Through detection, thisalgorithm had good effects for estimation of the bodyweight of the yaks. But under the impacts of various
external factors, following disadvantages possibly
existed during actual detection:1) During image identification, it needed to manually
determine whether the yak head faced left or right.
2) If the images were not accurately identified,
identification accuracy of the measuring points
would be affected.
3) If the measuring distance was inaccurate, theaccuracy of body weight estimation would be
affected.
4) This algorithm was affected by standing postures ofthe yaks. If the yak body was inclined in the yak
images, the final measurement accuracy would be
affected.
6. Conclusion
According to this algorithm, the gray scales were firstly
classified, and then the positions of the yaks in theimages were preliminarily determined by backgroundremoval, Gaussian blur, average processing and flood
fill and other methods. The yak images were extracted
from the pictures through extension from the yak center
point. Afterwards, the yak measuring points were
determined according to the center point and headdirection of the yaks as well as the curvature of the yakcontour curve map. Finally, the actual body height andbody steep length of the yaks were obtained based on
the camera imaging principle, and the body weight of
the yaks was calculated.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan245

An efficient structure of organization with complete group guidanceSatoshi Ikeda*

Department of Computer Science and System Engineering, Miyazaki University, Japan,
bisu@cs.miyazaki-u.ac.jp
Mamoru Yoshimura*

Graduate School of Engineering, Miyazaki University, Japan,
hf11031@student.miyazaki-u.ac.jp
Makoto Sakamoto*Department of Computer Science and System Engineering, Miyazaki University, Japan,
sakamoto@cs.miyazaki-u.ac.jp
Takao Ito†

Graduate School of Engineering, Hiroshima University, japan,
itotakao@hiroshima-u.ac.jp

Abstract
In this paper, we show model that an efficient organizational shape with one criterion can be classified into three
types by using a new mathematical.
Keywords: Hierarchical organization, Mathematical model, Discrete combination optimization

realistically[3,4]. Although the basic idea of the total
output to the outside of the organization as an evaluation
value has not changed, the major improvements in therevised mathematical model are the following two points.
(a) Contribution of the members for a given
organization can be categorized as the internal
contribution to maintain the organization and theexternal contribution to realize the purpose of theorganization in society.
(b) For any fixed organization, the sum of the effort
each person spends is constant, but its influence isdifferent depending on how to take a communication style.
The ratio of one's external contribution effort to thetotal of one's effort is called the external output
coefficient. In the mathematical model of [3,4] with one

1. Introduction
In [1,2], we proved using a mathematical model that an
efficient shape of a hierarchical organization with only
one criterion is limited to three types, see Fig.1 .

Fig.1. Three types of efficient trees when evaluation
criteria is one.
Subsequently, the mathematical model for evaluating
the shape of the organization has been generalized more*†

University of Miyazaki, 1-1, Gakuen Kibanadai Nishi, Miyazaki, 889-2192
University of Hiroshima, 1-4-1, Kagamiyama, Higashi-Hiroshima, 739-8527© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

246

Satoshi Ikeda, Mamoru Yoshimura, Makoto Sakamoto, Takao Itoevaluation criterion under the assumption that theexternal output coefficient takes a constant α ∈ [0,1]independent of member, previous results in [1,2]correspond to α = 0 and the communication style called"the complete group guidance".

deg ∗ =deg

deg − 1

if s ∈ { } ∪ ( ),otherwise.

For a rooted tree with ( ) = , we define natural
partial ordering $ ( , <)$ that so-called the tree-orderassociated with . That is, the root ∈ of is the least
element and the leaves of are the maximal elements inthis partial order. This ordering will be considered as theexpression ``depth": if < , we say lies below in .
Note that the root is the least element, and that theleaves of are its maximal elements in this partial order.
Suppose that = { , , ⋯ , }( ≥ 2) and(| |≧1) are finite sets. Through this paper, is the set of theevaluation measures. And is interpreted as the set ofmembers of a given organization, which consists of, , ⋯ , . There are many rooted trees with asvertices. After that, we identify each rooted tree as its
corresponding organization tree. Let (S) be the set ofrooted trees with as its elements.

Fig.2. A bruch graph.
When the number of evaluation criteria is one, in thepaper [3,4], we showed that the structure of an efficientorganization becomes pass-like shape in the case that
personal ability value of organization members is high.
This is an extension of the results of [1,2] to the case of
the model of [3,4] in part. However, in general it was notknown whether the results of [1,2] could be extended tothe case of the model of [3,4]. We expected that theresults could be extended in the general case, but inreality it was not. In [5], by simulation experiments,
Yoshimura et al. showed that an efficient tree is generally
a brush tree shape, see Fig.2, when the number of
evaluation criteria is one. After much consideration, we
concluded that the efficient tree shape as shown in Fig.1
may depend on the communication style.
The purpose of this paper is to show that theefficient organizational shape when the evaluation
criteria is only one, is limited to the three types shownin Fig.1 under the assumption of the communicationstyle called the complete group guidance.

2.2. External Contribution and Internal
Contribution
For a given and ∈ , we define { } ∈ by∶ →≡ { ∈ | > 0} for ∈We call ( ) the personal ability of ∈ with respectto ∈ . And we call (S, { } ∈ ) an evaluation system.
According to the idea of 1-3, the evaluation function of agiven evaluation system l (S, { } ∈ ) is defined asfollows. For ∈ , total output of ∈ that written by,( ) and s's→ is the product of s's personal ability
input that written by ←, . That is,,( ) ←, .
→ =2. Mathematical Model
2.1. Notations and Tree Graph as HierarchicalOrganization
Fig.3. Input and output for

Suppose that
= ( ( ), ( )) is a graph.
Throughout this paper, a graph is always finite,
directed and simple, with order = | ( )|( ≧ 2)
and size = | ( )|.
For ∈ ( ), by ( ) = { |{ , } ∈ ( )}, wedenote the set of vertices adjacent to , and calldeg( ) = | ( ) | the degree of ∈ ( ) . For an
arbitrarily fixed rooted treewith its root andleaves ( ), we denote∈For ∈ , s's total output →, is classified into s's
external output →,and s's internal output →, , see
Fig.3. That is,,+ →, .
→ = →For the subordinate ∈ who received instructions fromhis superior, it is necessary to transmit appropriate
instructions →, as superior to his own subordinates,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan247

An efficient structure ofwhile s as subordinate carries out the instructions →, ,see Fig.3.
We assume that the ratio of s's external output to s's
total output is a constant independent of ∈ . Throughto →, as α ∈this paper, we denote the ratio of →,[0,1] for any ∈ . That is,external output →,: internal output →,=α : 1− αfor any ∈ . We call α the external output coefficientand call 1 − α the internal output coefficient.

The case of (1) CIG is an organization like a secretsociety. Since a superior gives an instruction to his/her
subordinates individually, the members of theorganization do not know about each other's work. In
such organizations, when new member joins theorganization, or a member of the organization leave theorganization, its efficient organization can not be
constructed unless the existing organization is broken
once.
On the other hand, the case of (2) CGG is an openorganization. The characteristic of an organization withsuch a communication style is that it is stable in the sensethat there is no need to significantly change thecomposition of the whole organization when newmember joins it. Roughly speaking, the reason why CGG
has such a property is that when we convert to
′= −
+
for
, ’ ∈ (S), , , ∈ , ∈ ( ), > , theevaluation function is affected only by the part of the up
closure of , see Fig.4.

2.3. Weights' Policy

For a given organizational structure tree , we assume
that the value of the input for subordinate ∈ ( ) with> is obtained by multiplying its weight { } to,
→ , see Fig.3 again. Where≤10≤
for any ∈ ∖ ( ), > of
∈ ( ) and≤ deg*
1 ≤ ∑ ∈ ( ),for any ∈ ∖ ( ). Therefore, the total contribution of∈ ∖ ( ) for the organization can be expressed by,,∑ ∈ ( ),.
→
→ +We assume that ( ) ∈ ( ),for ∈ ∖ ( ) is asequence depending only on deg* and ∈ . That is,for , ′ ∈ ∖ ( ) s,s' and ∈ , deg* = deg* ′) ∈implies that (is a permutation of,() ∈ ( ), . Thus, for a fixed deg* and ∈ , theselection that we can do is which weight to assign whom.
We call the way of the determination a weights' policy.
For any weights' policy, we assume that if deg* ≥deg*s' for , ′ ∈ ,≥≥ ⋯ and≥≥ ⋯,) ∈ ( ),(
) ∈then (
satisfy,
for = 1,2, ⋯,deg* '.

Fig.4. Conversion fromto 'CIG and CGG are extreme cases, the general style isintermediate between them.
2.5. Evaluation Function
To summarize so far, for a given evaluation system(S,{ } ∈ ), a rooted tree with root and ( ) = , ∈, ∈and an external output coefficient α ∈ [0,1],
,
,
← and
→ denote the input and the output of∈ , respectively. And →,and →, denote theexternal and the internal outputs of ∈ for theorganization, respectively. Then we define the follows;≤

2.4. Communication Style,→

The weights' policy is a parameter determined by thecommunication style in the organization. Typical ones
are the following two cases;(1) Complete Individual Guidance (CIG)∑ ∈ ( ),
= 1 for any ∈ ∖ ( ).
(2) Complete Group Guidance (CGG)∑ ∈ ( ),
= deg* for any ∈ ∖ ( ).

,
←

=

( )

=
1

( )

,
( )→

if

,
←

= root,otherwise.

Wheredenote the weights from ∈ ∖ ( ) to
∈ ( ) with respect to ∈in anarbitrarily fixed .
And ( ) denotes the parent node (as superior) of ∈© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

248

Satoshi Ikeda, Mamoru Yoshimura, Makoto Sakamoto, Takao Ito

The typical form of an efficient tree with respect to some∈is a path graph or star graph as in Theorem1. In

on . Under the assumption of extenal and internal
output, we define
,
→

=

,
→

,
→

0
(1 − α)
=

=

α

,
→

,
→

,
( )→

,
→

(

if ∈ ( ),otherwise.
,
( )→

+

,

Theorem 2. Under the setting of CGG, suppose that
( , { } ∈ ) is an evaluation system. Assume that

For a given evaluation system ( , { } ∈ ) an external
output coefficient α ∈ [0,1] and a given weights' policy,
let be a rooted tree graph with ( ) = . Then we
will evaluate the rooted tree as organization model by(α )( α ,i )

(T )

(T ),

( α ,i )

(T )

max
i

{ wxy } i

i

max

T

( ,i )(S)(T ) is attained by( ,)we have the followings.
( ,)(a)is equal to 0 or 1.

( α ,i )f s out (T )()

for some ∈ , then,(b) Putting= { } whenthen we see that
( ,)
∈
>
=

Here
shall be taken about all the possiblecombinations under the given weights' policy. For a given(S,\{ { } ∈ ) and ∈ [0,1], we call Φ( ) ( ) the ability
( )

( ,)
( ,)

.

The general form of the most efficient tree
which theorem2 insists on is shown in Fig.6.

value of . Under a given weights' policy,
is anefficient tree for a given external output coefficient ∈[0,1], if max ( ) (T ) is attained by Φ( ) ( ( ) ). HereT),
general, we can show that the form of
becomes theform which seems to have put these two figures together.
For a given ( , { } ∈ ) and its organizational structure tree , let us define( ) = { ∈ | deg*(s) ≥ 2}

if ∈ ( ),otherwise,

= 1,
( ,)

(S )

(S) denotes the set of rooted tree graph with ( ) = .

3. Results

Theorem 1. Under the setting of Complete Group
Guidance, suppose that ( , { } ∈ ) is an evaluation
system such that = { , , ⋯ , } with( )≥ ( )≥⋯≥ ( )for some ∈ . Assume that max
T( ,i )(S)

Fig.6. General form ofReferences(T ) is attained, then we have the followings.
by(a) Asuume that( ) ≥ 2⁄(1 − ), then we see that( ,)one ofis the path graph in Fig.5(1).
(b) Asuume that ( ) ≤ 1⁄(1 − ), then we see that( ,)one ofis the star graph in Fig.5(2).

(1)

s1

sn

s2 s3

( ,)

( ,)

for some ∈.

1. S.Ikeda, An Efficient Structure of Organization,
International Interdisciplinary Workshop on Robotics,
Ecosystem, and Management(2010), pp.72-78.
2. S.Ikeda, T.Ito and M.Sakamoto, Discovering the efficientorganization structure: horizontal versus vertical, Artificial
Life and Robotics(2011), vol.15, 4, pp.478-481.
3. S.Ikeda, T.Ito, M.Sakamoto, A Mathematical Model of
Hierarchical Organization, Proceedings of theinternational conference on Pure Mathematics andApplied Mathematics(2015), pp423-432.
4. S.Ikeda, T.Ito, M.Sakamoto, T.Ito, An Evaluation Modelof Internal and External Contributions in HierarchicalOrganizations, International Journal of Mathematical
Models and Methods in Applied Sciences(2015), Vol.9,
pp.636-645.
5. M. Yoshimura, T. Ito, M. Sakamoto, T. Ito, Y. Shinano, S.
Ikeda, An efficient structure of organization with complete
individual guidance, Proceedings of the 2018
International Conference on Artificial Life andRobotics(2018), vol.23,pp.264-267.

( ,)

s1
s2
s3

( ,)

sn

as a path graph (2)as a star graphFig.5. An efficient organization structure.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan249

Parallel finite element analysis for hyperbolic problems
Amane Takei and Makoto Sakamoto

Department of Electrical and System Engineering, University of Miyazaki, 1-1 Gakuen Kibanadai-Nishi
Miyazaki, 889-2192, Japan
E-mail: takei@cc.miyazaki-u.ac.jp
www.miyazaki-u.ac.jp/english/

Abstract
With the expansion of electromagnetic field analysis using computers, large spaces that include complex shapes havealso become an analysis target, and the development of a high-accuracy analysis is required for these problems.
Therefore, in the present study, Berenger’s PML for hyperbolic problems, which is currently the most effective
absorbing boundary condition, is applied to the parallel finite element method based on the domain decomposition
method, which is an effective analysis method for the high-frequency band. As a basic study, we developed ananalysis solver using a parallel finite element method based on the iterative domain decomposition method. In
verifying the accuracy of the analysis solver, we analyzed TEAM Workshop Problem 29, which is a benchmark
problem, and confirmed that a highly accurate solution is obtained. Next, a model with Berenger’s PML added to thedipole antenna model is used as an analysis object, and the absorption performance of the PML is evaluated using areflection coefficient based on the S parameter. Moreover, the accuracy of the antenna analysis is evaluated bycomparing the directivity of the dipole antenna with the theoretical solution. As a result, the effectiveness of the
proposed method for high-frequency analysis is confirmed.
Keywords: Iterative domain decomposition method, Parallel finite element method, Berenger’s PML, Large-scale
analysis, Microwave analysis.

and evaluate the accuracy of the analytical solver. Next,a model with a PML added to the dipole antenna is usedas the analysis object, and the absorption performance of
the PML is evaluated using a reflection coefficient basedon the S parameter. Moreover, by comparing thedirectivity of the dipole antenna with the theoreticalsolution, evaluations of the accuracy and performance of
the antenna analysis are performed. As a result, we
confirmed the effectiveness of this method for
microwave analysis.

1. Introduction
In the present study, Berenger’s PML for hyperbolic
problems like a high-frequency electromagnetic field
analysis, which is currently regarded as the mosteffective absorbing boundary condition, is applied to theparallel finite element method, which is an analysis
method that is capable of computing large-scale
problems. We then show that the parallel finite element
method is effective as an analysis method for a
microwave band by calculating the problem dealing with
an open domain and evaluating the solution’s accuracy.
First, as a basic study, we developed a three-dimensional
electromagnetic field analysis solver using a parallel
finite element method based on the iterative domain
decomposition method. In order to verify the accuracy ofthe developed analytical solver, we calculate TEAM
Workshop Problem 29, which is a benchmark problem,

2. Numerical analysis
We analyze TEAM Workshop Problem 291 in order toverify the accuracy of the developed analytical solver.2
TEAM 29 is a benchmark problem and involves aresonator model. The resonator is cylindrical and has adiameter of 1.9 [m] and a height of 1.45 [m]. In theanalysis, a dielectric phantom having a relative© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

250

Amane Takei, Makoto Sakamoto

permittivity εr of 80 and an electric conductivity σ of 0.52
[S/m] is positioned, and the resonance state isinvestigated. The analysis domain boundary is a perfectconductor. The analysis model is shown in Fig. 1. Table
1 shows the specifications of the TEAM 29 model. The
highest calculation efficiency is achieved when thenumber of elements contained in one subdomain isapproximately 170, and the number of partial domains isdetermined such that the number of elements containedin one subdomain is equal to 170.

frequency step and the averagely used memory are shownin Table 2. Fig. 2 shows the frequency response of themagnetic field. The measured and calculated values areshown in Table 3.
Table 2. Specifications of the TEAM 29 model.
No. of Elements

DOFs

No. of Subdomains

121,277

149,668

100×7

Fig. 2. Numerical and measured frequency response
analysis results
Fig. 1. TEAM 29 cavity resonator model

Table 3. Resonance frequencies (Units: [MHz]. The

Table 1. TEAM 29 model data.

error rate [%] between measured data and thenumerical solution is shown in parentheses.)No. of Elements

DOFs

No. of Subdomains

121,277

149,668

100×7

The results of the analysis of TEAM 29 are subjected tofrequency response analysis in order to confirm theaccuracy of the developed analytical solver. In order todetect the resonant frequency and compare solutions withactual measurements, the frequencies of some range are
analyzed. The frequency band of 60 [MHz] to 140 [MHz]is calculated for 2-MHz steps, and the response for every
frequency step is investigated. In addition, calculations
near the resonance frequency are performed in 0.4-MHz
intervals. The computing environment used in the present
study is a 25-PC cluster equipped with Intel Core i72600K multi-core CPUs (total: 100 cores) and 32 GB
memory is used. The compiler used is the gcc. In addition,
Message Passing Interface (MPI) is used for theparallelization library. The average calculation time per

Resonance
mode

Measured
data

FDTD
25-mm
mesh

1st

68.6

67 (2.33)

2nd

110

110

3rd

134

134

Result65.6(4.37)
109.0(0.91)
134.4(0.30)As shown in Fig. 2, a solution resonating around theresonance frequency of the actually measured value isobtained. In the comparison of the measured andcalculated values, the error rate is 4.37 [%] in the 1stmode, 0.91 [%] in the 2nd mode, and 0.30 [%] in the 3rdmode. As the mode increases, the error rate decreases.
However, it is the same tendency as the analysis result
obtained by the FDTD method. Moreover, the error rateis less than 5 [%], and a solution with high accuracy is
obtained.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan251

Parallel finite element analysis( ) = 1 at the position of the (L-1)th layer, and ( )= L -1 at the position of the first layer.
Moreover, σmax is the maximum value of the electric
conductivity for the PML, and M is the degree
distribution of electric conductivity. This equation is used
to determine the electric conductivity of each layer of the
PML.
The parameters to be determined as the parameters of the
PML are the thickness ∆x of PML 1, the number L of
PML layers, the maximum electric conductivity σmax of
the PML, the degree M distribution of the electric
conductivity, the reflection coefficient R [dB] between
the PML of the outermost layer, and the perfectconductor wall. The reflection coefficient R isapproximated as follows:

Therefore, the solution obtained by the developed solver
is proven to have a sufficiently high accuracy. Moreover,in the analysis of the dipole antenna applying the PML
described in the following sections, the error tolerance
index is defined as 5 [%] in order to evaluate the accuracy.
3. PML3.1. Berenger's PML
The PML can be used to create an absorbing boundary bysurrounding the analysis domain with a PML. From theviewpoint of the accuracy of the obtained solution, thePML is currently the most effective absorbing boundary
condition. Although Berenger's PML is originallyproposed as an absorbing boundary condition for the
FDTD method, in the present study, we apply a finite
element method dealing with an unstructured grid, we
propose a simplified method omitting the directionalityof electric conductivity given to the PML and confirm its
effectiveness.
Berenger's PML stacks several PMLs outside the analysis
domain and gradually sets a large value of electric
conductivity according to the outer layer so that theoutermost wall can be surrounded with a perfectconductor wall without reflecting electromagnetic waves.
Fig. 3 shows a schematic diagram of Berenger's PML
absorbing boundary.

| ( )| ≅In this paper the distribution of the electric conductivity
for PML is expressed as follows:− ( ) ∆
∆

)

∆

cos

(2)

where ϕ is the incident angle of the electromagnetic wave,
and c is the speed of light. Since we cannot decide theincident angle for an arbitrary incident wave, ϕ = 0, a
reflection coefficient for perpendicular incidence is used
as a reference. Moreover, since the M that gives the
distribution of the electric conductivity causes thecalculation accuracy to deteriorate if the change of theelectric field in the PML is too steep, M is approximately
2 to 4. If the number of layers L is too large, morememory will be required, and if L is too small, it will not
function adequately as an absorbing boundary. There aremany cases where the concrete number of L is set to 4 to
16. The thickness ∆x of PML 1 is a constant thickness of
all layers.
We set the reflection coefficient R(0) according to therequired accuracy. Upon determining the above
parameters, the maximum electric conductivity σmax isgiven as follows:

Fig. 3. PML absorbing boundary

=

−(

=−

(

)

∆

ln| (0)|(3)

In the present study, we construct a PML using (1)through (3) with = 9, = 4, and ∆ = λ/10. However,in order to reduce the analysis scale, we examine theoptimum value of L in the next section.

(1)

where ∆x is the thickness of PML 1, L is the number oflayers of the PML, ( ) is a coefficient determined byposition x, and ( ) = 0 at the position of the Lth layer,

3.2. Numerical results
We assign the PML to the dipole antenna model. The
analysis domain is a cube of length 0.6 [m] so that the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan252

Amane Takei, Makoto Sakamoto

coefficient obtained using the S11 parameter3. The
observation point of the S11 parameter is on the x-axis 1
cm inside of the PML. The computing environment is thesame as in the section 2. Table 5 lists the reflection
coefficient, the CPU time, and the memory size.

distance from the antenna to the innermost PML matches
the wavelength. The current density is applied to theantenna as a current source as follows:
( )=

cos

∶− ≤

≤

(4)

Table 5. Results for reflection coefficient, CPU time, andmemory size

where = 0.08 [A/m2], is the wavelength, and is thelength from the feeding point to the antenna tip.
The analysis frequency is 1 [GHz], and the length of theantenna is 0.15 [m], which is the half wavelength. Here,mesh division is performed so that the maximum side
length of the element is 1/20 of the wavelength. The
analysis domain’s boundary is a perfect conductor. Fig. 4
shows a schematic diagram of the dipole antenna model.

Reflection
coefficient [dB]
CPU time [s]Memory size
[MB/core]

PML(0):
Perfect
conductor
wall

PML(9)

0-18.65

1,278

18,787

44.3

227.3

When the domain boundary is PML(0), i.e., when it is aperfect conductor wall, S11 = 1, so that the reflection
coefficient is 0 [dB]. On the other hand, when the domainboundary is PML(9), the reflection coefficient is -18.65[dB]. The design target reflection coefficient of theantenna, for example, is generally approximately -10 to 20 [dB], and in the present study, we use a reflection
coefficient of approximately -10 to -20 [dB]3. Thus,
PML(9) can obtain sufficient absorption performance.
On the other hand, in comparing with PML (0), PML (9)increases the amount of memory used and computation
time, depending on the absorbing layer applied. Fig. 7
shows a visualization diagram of the electric fieldobtained by analysis.

(a) Analysis domain(b) Antenna
Fig. 4. Dipole antenna model
We assign PMLs to the domain boundary as shown in Fig.
4(a). The plane portion of the PML at the domainboundary overlaps a number of flat plates according to
the number of layers, and the corner portion of the PML
is one rectangular parallelepiped or cube. The boundaryof the outermost layer of the PML is a perfect conductor
wall. We perform performance evaluation by setting the
thickness of one layer to be 0.03 [m] and the PML to have
= 9 (hereinafter a PML with L layers is abbreviated as
PML( )). Table 4 lists the number of elements and thedegree of freedom of the analysis model.
Table 4. Number of elements and DOFs of the dipole
antenna model
No. ofElements
DOFs

PML(0): Perfectconductor wall

PML(9)

4,669,759

26,899,669

5,506,368

31,703,550

Fig. 5. Visualization of the analysis result (electric field)(Left: PML(9), Right: PML(0) (perfect conductor wall))In (3), we set = 9, ∆ = 0.03, = 4, and (0) = -120[dB], which yields the maximum electric conductivity
to PML(9). In addition, we decide the electric
conductivity of each layer using (1). In this study, we set
the average value of each layer to the electric
conductivity of the corner portion. We evaluate theperformance of the PML based on the reflection

In Fig. 5, the left-hand side shows PML(9) at theboundary edge and the electric field propagates from thedipole antenna to the free space. On the other hand, theright-hand side of Fig. 5 shows the mode when the dipole
antenna is enclosed by a perfect conductor wall.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan253

Parallel finite element analysis

Next, we perform the directivity evaluation of the dipole
antenna by error evaluation using the theoretical solution
in the far field. The error evaluation of the far field uses
the E plane.
The theoretical solution3 of the far field of the E plane is
as follows:
= 60

∙

cos

cos
2
sin

=

(5)

No. ofelements
DOFs
Averageerror rate[%]
Reflection
coefficient[dB]
CPU time[h]No. ofiterations
Memorysize[MB/core]≤ 90 −arcsin

2

+1

× 100

[%]

(7)

PML(9)

PML(8)

PML(7)

26,899,669

24,184,687

21,533,641

31,703,550

28,506,352

25,383,890

1.70

3.81

12.87

-18.65-15.79-15.04

5.22

3.77

2.81

46,50837,75530,695

227.3

204.6

182.6

From Table 6, PML(9) is the case with the best far field
accuracy. When the allowable range of the error rate isless than 5 [%], which is the allowable range of numericalanalysis error, since PML(7) has a reflection coefficientof less than -15 [dB], the PML functions sufficiently.
However, the error rate exceeded the allowable range.
We can find that PML(8) is optimal because it has a better
calculation time and iteration count than PML(9).

The directivities of the numerical and theoreticalsolutions agree very well. The range of θ, which is the far
field far beyond the Fresnel’s region, can be expressed by(6). The lower limit
is arcsin( 2 ∕ ) + 90 ≅−57 [deg], and the upper limit
is 90 −
arcsin( 2 ∕ ) ≅ 53 [deg]. The average error ratein this range is obtained by (7). As a result, theaverage error rate is 1.70 [%], and it is shown that ahighly accurate solution can be obtained.
+ 90 ≤|

Table 6. Numerical model data and results

Fig.6. Numerical and theoretical solutions in the E plane

2

−

| −

In the calculations shown in Fig.5, we used a dipole
antenna model with PML(9). Here, we find the optimum
L from the average error rate in the far field and thereflection coefficient of PML(L) by a parameter study
using the number of PMLs. The computing environmentis the same as in the section 2. Table 6 shows the numberof elements for each L, the number of degrees of freedomof the edge, the error rate, the reflection coefficient, thecalculation time, and the number of iterations of the
COCG method applied to the interface problem.

where is the imaginary unit, is the current, and is the
distance from the feeding point. The approximate
distance to the far-field peak of the Fresnel’s region(2 / < ) is 0.250 [m], if the dimension l (= 0.150[m]) of the dipole antenna is not ignored. Moreover, isthe wave number and is given by = 2 ⁄ . The
directivity evaluation is performed by comparing thenumerical analysis solution with the theoretical solution
on the E plane. Fig. 6 shows a plot of the numericaland the theoretical solution
inanalysis solution
increments of 1 [deg].

arcsin∑4. Conclusion
In the present paper, we propose a simplified method thatomits the directionality to Berenger's PML hyperbolic
problems like a high-frequency electromagnetic field
analysis and gives the average value of the electric
conductivity of each layer at the corner of the model.
Performance evaluation reveale that sufficient absorption
performance can be obtained. In the accuracy verification
by directivity evaluation of the dipole antenna, when the
maximum element side length is set to 1/20 of the(6)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

254

Amane Takei, Makoto Sakamoto

wavelength and the PML to be given is set to 9 layers, theerror rate of the numerical solution and the theoreticalsolution is about 1.70 [%]. It is found that a highly
accurate solution can be obtained. In addition, when thetolerance range of the far-field error rate that is
considered to be sufficiently practical is set to less than 5[%], an eight-layer PML is found to be optimal. In
addition, the usefulness of the proposed method for afrequency band of 1.2 GHz or higher, which is used in
microphones and mobile phones, is demonstrated.

Acknowledgements
The present study was supported in part by a JSPS Grantin-Aid for Scientific Research (Basic Research (B),
17H03256). The computer environment used in thepresent study was supported in part through a JST
Adaptable and Seamless Technology transfer Program(A-STEP (Search Type), AS262Z02631H).
References
1. TEAM Workshop, Tucson, AZ, USA, 1998.
2. A. Takei, S. Sugimoto, M. Ogino, S. Yoshimura, H.
Kanayama: Full Wave Analyses of Electromagnetic Fields
with an Iterative Domain Decomposition Methodm, IEEE
Transactions on Magnetics, 46:8 (2010), 2860-2863.
3. Fujikura LTD. Homepage: http://www.fujikura.co.jp/eng/© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan255

Design of feed part control system for rectification process
Lingran An1, Fengzhi Dai1*, Yujie Yan1, Zhongyong Ye1, Xia Jin1, Yiqiao Qin12, Chengcai Wang3, Kaige Liu1
1
College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
2 Tianjin Tianke Intelligent and Manufacture Technology CO.,LTD, China;
3 CETC Ocean Information Co., Ltd, Beijing 100041, China

E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

This paper completes the structure design of the feed process of the small rectification experimental unit.
According to the process requirements, we design the overall scheme of the detection and control systemof the main parameters and select the type of the instrument. The system hardware design is based on the
CPU222 of Siemens s7-200 PLC and the corresponding input/output module. The lower computer adopts
the ladder diagram to complete the program design such as parameter acquisition, scale transformation
and PID operation. The monitoring interface of the upper computer adopts King View monitoring
software.
Keywords: rectifying column, feed, PLC, PID

1. Introduction

The design idea of feed part control system ofrectification process is as follows:Firstly, the technological process of rectifying column
and feed process is analyzed, and the process pipeline andcontrol flow chart are drawn. Then, the detection andcontrol points of the feed control system are determinedto achieve the ideal rectification effect, and the
instrument selection is completed.
Next comes the design of the hardware part of thecontrol system. We mainly complete the selection of PLC
modules and the hardware wiring of the system.
Finally, the design of the software part of the control
system. We use STEP7 programming software to write
the control program of the lower computer through theladder diagram programming language, so as to realize
the collection and processing of field data, and control
the communication between the field equipment and theupper computer. The upper computer adopts themonitoring software (such as King View) which is

The rectification operation makes use of the different
volatility of different components of the mixed liquid
under the same pressure, causes the partial vaporizationand condensation of different components in the column,
and finally the complete separation is obtained1.
This design is mainly aimed at the feed part of therectifying column. There are three feed states ofrectifying column: superheated steam, supercooled liquid
mixed in Vapor-liquid phase and saturated liquid. In
chemical research, how to select the feeding port ismainly studied. When the component becomes heavier,the feeding port should change downward to increase thereflux ratio, and when the component becomes lighter,the feeding port should change upward to reduce thereflux ratio2. Feed quantity, temperature, pressure andmaterial balance are the important factors that affect therectifying column3.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

256

Lingran An, Fengzhi Dai, Yujie Yan, Zhongyong Ye, Xia Jin, Yiqiao Qin, Chengcai Wang, Kaige Liucompatible with the selected PLC to monitor the control
process.

temperature will directly affect the vapor-liquid
equilibrium of the whole rectifying column. The
variation range of feed quantity should be controlled
within a reasonable range to achieve the ideal
rectification effect. In addition, the liquid level of the raw
material tank should be tested in order to add the raw
material in time. Therefore, detection points include feedflow (FIC101), feed temperature (TIC101) and feed tanklevel (LI101).
Next is the selection of the measuring instruments. For
the flow detection point FIC101, we choose differential
pressure flow transmitter, which is an intelligent
differential pressure flow meter based on the
monocrystalline silicon differential pressure sensor. It
can display feed flow in real time and realize remotetransmission, which is convenient for remote real-timemonitoring. For the temperature detection point TIC101,
we choose the Pt100 temperature sensor, which not only
has the analog signal output function of the transmitter,but also can increase the digital communication function.
For the liquid level detection point LI101, we choose theinput level transmitter, which is a pressure sensor tomeasure the liquid level. The essence of liquid levelmeasurement is to change the pressure differential into 4~ 20mA standard signal by sensor and amplifying circuitand then output it.

2. Overall scheme design
2.1. Analysis of process flow and control
The process pipeline and control flow chart of feeding
part are shown in fig.1. We add the mixture into the feedtank V102, open the valve VA107, and the feed pump
P102 feeds the raw material into the feed preheater E101
to heat the raw material. The feed flow meter FIC101
detects the feed flow in order to control its variation range.
We open the valve VA112 to allow the raw steam
generated by the feed preheater to enter the rectifying
column T101. TIC101 detects the feed temperature andcontrols the heating temperature of the feed preheater toobtain the target product. The raw materials and steam
entering the rectifying column are subsequently refluxand reboiler treated to produce light component products
and recombinant sub-products. After the rectification
experiment, the residual liquid in the raw material tank
was taken back to the tower kettle discharge tank V105
for recycling and utilization by vacuum pump P101 andvacuum buffer tank V101.

2.3. Determination of control scheme
According to the selected detection points, we
determined two control points. First is the flow control
point FIC101, which controls the feed flow bycontrolling the speed of the feed pump. Next is thetemperature control point TIC101, which controls thetemperature of the feed preheater by controlling theheating power of the heating rod.
Finally, the selection of actuators is completed
according to the selected control points. For FIC101, thefrequency converter is the one that controls the speed ofraw material pump, so we choose Siemens MM440
frequency converter. For TIC101, we chose solid statevoltage regulator to control the power of heating rod.

Fig.1. The process pipeline and control flow chart of feeding
part

2.2. Determination of detection scheme
First, we select the detection points. There are manyfactors influencing the feed in the process of rectification,
among which the temperature of feed and the flow rate offeed are the important factors influencing the wholeprocess of rectification. The variation range of feed

3. Hardware design
The input and output points of this control system include:
2 digital input points (start and stop switches), 2 digital© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

257

Design of feed part

output conversion subroutine is used to transform thedata after PID operation and output it to the executor. The
flow chart of the main program is shown in fig.3.

output point (2 control relays), 3 analog input points (feedtemperature, feed rate and feed tank level), 2 analog
output points (voltage of the solid-state voltage regulator
and frequency of the frequency converter). Therefore, theCPU222 module is selected, which has 8 native digital
inputs and 6 native digital outputs, allowing up to 2 I / O
expansion modules. One EM231 analog expansion input
modules and an EM232 analog expansion output
modules are selected as required. The hardware wiring
diagram is shown in Fig.2.

Fig.2. The hardware wiring diagram
Fig.3. The flow chart of the main program

4. Programming software design

4.2. PID operation subroutine design

The control layer of the control system of feed part ofrectification process is the PLC system. It is used toreceive on-site data and control the site equipment
according to the pre-designed control strategies, while
send the on-site data and equipment state to the PC
machine. This design selects the S7-200 series PLC as
the controller, and the programming software uses the
STEP 7-Micro/WIN software of the German SIEMENS
company.

In practical engineering, the ratio, integral anddifferential control is the most widely used regulator
control law, referred to as PID control, also known as PID
regulation4.
In S7-200CN, PID command is a method to realize
PID function5. In STEP7-Micro/WIN V4.0, we can use
the wizard to generate a PID instruction or program, andthen call it in the main program. After completing theconfiguration of the PID loop through the wizard, thespecified PID regulation subroutine PIDx_INIT andinterrupt program PID_EXE will be automaticallygenerated. The PIDx_INIT instruction performs the PID
function based on the inputs and outputs we havepreviously set in the PID wizard, and it can be invoked
every time the scan is performed. In this design, feedtemperature control system and feed flow control system
adopt PID wizard to generate the instructions6.

4.1. The main program design
The PLC program is designed in a structured way. The
main program completes all functions of the system, andcalls each module in turn according to the system
requirements and subroutine functions. The data
conversion and scale conversion subroutine is used toprocess and store the collected data. Some of the data andthe set value are compared to calculate the deviation,
which is used for the PID operation of the program
generated by the PID instruction wizard. The control© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

258

Lingran An, Fengzhi Dai, Yujie Yan, Zhongyong Ye, Xia Jin, Yiqiao Qin, Chengcai Wang, Kaige Liu

5. Monitoring software design

The detection and control points selected in this design
are for the feed part of the rectification process, and thecomplete rectification process should also include thereflux part and the reboiler part. Therefore, the selectionof parameters, algorithms, software and hardware isrelatively simple and cannot meet the complete
rectification control requirements. Adaptive control
algorithm, predictive control algorithm, inferential
control algorithm and robust control algorithm can be
developed for the complete rectification control, or
intelligent control methods such as neural network andfuzzy control can be considered for the rectification
column.

This design adopts kingview6.55 monitoring system
software. Generally, such a system can be divided into
three layers: control layer, monitoring layer andmanagement layer. PLC and PC are connected by PC-PPI
cable hardwire, and the monitoring software of PC iscommunicated with PLC through PPI7.
First of all, we complete the definition of the external
equipment and the communication between the
Kingview6.55 and the external equipment in the software.
Secondly, we define the variables in the control system.
Finally, we design the configuration screen, which
includes the drawing of the screen and the connection of
the animation, etc. Our configuration screen is divided
into three pages, namely, main interface for feed
monitoring, production report and data curve, and isequipped with the start and stop button, screen switch
button, user login and exit button, etc. The main
monitoring interface after login is shown in fig.4.

Acknowledgement
The research is partly supported by the “13th Five-Year”
comprehensive investment and basic research businessfee items of China (000040192).
References
1. Fuxue Liu, Research and application of rectification
column, Chinese Petroleum and Chemical Standards andQuality, 2011, 9: P199.
2. Chuanbo Yu, Correctly understand the proper feed
position and q value of rectification column, Higher
Education in Chemical Engineering, 2008, 2: P60-P68.
3. Kejing Li, Briefly discuss the design of process control
scheme of rectification column, Petrochemical Industry
Application, 2013, 9: P120-P124.
4. Xueyong Shi, Realization of PID control in the liquid levelof distillation tower kettle, Science and Technology
Economic Guide, 2017, 3: P72.
5. Jianfang Jiang, Siemens S7-200 PLC engineering
application technology tutorial, China Machine Press,
2010: P184-P197.
6. Huailin Zhao, Masanori Sugisaka, Leilei Cui, Guanglin
Shi, Research on the intelligent control algorithm for a soft
joint actuated by Mckibben muscles, Artificial Life andRobotics (ISSN1433-5298), Vol.14, No.1, 2009, P85-P88.
7. Yahui Mu, The practical technology of Kingview software,
Yellow River Water Conservancy Press, 2012: P34-P35.

Fig.4. The main monitoring interface after login

6. Summary and prospect
This design adopts DCS control system based onprogrammable controller Siemens S7-200. Its contents
mainly include the overall scheme design of the feed partof the rectification process control system and the designof hardware and software.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

259

Ultrasonic Range Finder for Vehicle Collision Avoidance System
Ying Chen1, Guowei Yang1, Fengzhi Dai1*, Yuxuan Zhu1, Di Yin1, Yasheng Yuan1,Yiqiao Qin12, Ce Bian12,Chengcai Wang3, Xinyu Zhang1

College of Electronic Information and Automation,
Tianjin University of Science and Technology,China

1

2 Tianjin Tianke Intelligent and Manufacture Technology CO.,LTD, China
3 CETC Ocean Information Co., Ltd, Beijing 100041, China)

E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

In recent years, highway rear-end collision accidents have occurred frequently, causing huge personal injury andproperty damage. Protective measures such as seat belts and airbags only serve to reduce damage afterwards and do
not solve the problem from the source. Therefore, research on the system of vehicle collision avoidance, which
actively avoids accidents, is of great significance. The ultrasonic range finder designed based on AT89C51 singlechip microcomputer utilizes the principle of sound wave reflection in the air. It can detect the distance between thevehicle and the obstacle in real time, and immediately alarms beyond the set threshold. Ultrasonic ranging is widelyused in automotive anti-collision systems due to its non-contact detection method and simple structure.
Keywords: Ultrasonic, Vehicle collision avoidance, AT89C51, Distance detection

to complete a specific complex task. 2 Research on the carcollision avoidance system that can play an active
protective role is very urgent. This design is mainly
applied to the anti-collision system based on the ultrasonicranging method. Ultrasonic ranging is not affected by light,
surrounding magnetic fields, or gas properties comparedto other methods. Especially in the air, due to the slow
circulation speed in the air, the echo signals contained in
the direction of structural information transmission areeasy to detect, the resolution is high, and the accuracy ishigher than other methods. Has a broad market
development prospects and practicality of life.

1. Introduction
Traffic accidents caused by vehicles are one of the mostinfluential factors in human society. On average, one
person in the world died of road traffic accidents every 60
seconds, and one person was injured in a road accident
every 5 seconds.1 Road traffic safety has become aconcern of all countries in the world.
When the whole world is threatened by road traffic
accidents, airbags, seat belts and other devices are
beneficial to reduce the harm of people in traffic accidents,
but they are passive anti-collision measures for emergency
protection after the event, and the effect is small, andPersonal injuries and economic losses will still occur after
the accident.

2. Hardware selection
The system is divided into ultrasonic transmitting
module, signal receiving module, single chip processing
module, LED display and sound and light alarm module.

Therefore, with the development of the computer andelectronical technology, it is an irresistible trend to makethe multiple intelligent agents cooperate with each other© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan260

Ying Chen, Guowei Yang, Fengzhi Dai, Yuxuan Zhu, Di Yin, Yasheng Yuan,Yiqiao Qin, Ce Bian, Chengcai Wang, Xinyu Zhanga basic RS flip-flop and discharge tube TD. The 555-time
circuit and external components form a 40kHZ multiresonant circuit, while the microcontroller controls theoutput of the NE555 to drive the ultrasonic transducer.
• Frequency 100kHz• Power supply voltage 4.5V~16V• Supply current 10mA
• Working temperature 0°C~70°C4
NE555 is shown in the Fig.2.

The ultrasonic ranging system adopts the transmission
and reception separation method. The transmission andreception separation have two advantages: one is that thetransmission and reception signals do not overlap, the purereflection signal is received by the probe; the second isthat the receiving probe is selected at a suitable position toprevent The loss and interference caused by the reflected
object on the ultrasonic surface enhances the reliability ofthe system. According to design requirements andcomprehensive factors, TCT40-16T/R ultrasonic distancesensor (T is the transmitting sensor and R is the receiving
sensor). At the same time, the main controller selects
AT89C51 microcontroller, LED digital display adopts
dynamic scanning mode.
2.1. Main control chip
The main chip uses the low-power, high-performance 8bit microcontroller AT89C51 chip as the main controllerof the ultrasonic ranging system. The chip contains
4kBytes ISP, which can repeatedly erase 1000 flash
memories. The main features are as follows.

Fig.2. NE555•

The 8031 CPU is compatible with the MCS-51;

2.3. CX20106A•

Full static operation: 0~33MHz;•

Three program memories;•

Serial channel;•

There are 32 programmable I/O lines;•

Contains 16 16-bit timer/counters;•

6 interrupt sources;•

Programmable 128 * 8-bit built-in RAM;3

Cx20106 is an infrared demodulation integrated circuit
produced by Sony Corporation of Japan. It is packaged in
8-pin single-row in-line plastic package. It contains
preamplifier, auto-biasing, passband filtering, peakdetection and Schmitt shaping output. The function is todemodulate the encoded signal and send it to themicroprocessor for processing.5 the parameters of the pins
and the transmission frequency of the correspondingtransmitting circuit. CX20106A is shown in Fig.3.

AT89C51 is shown in the Fig.1.

Fig.3. CX20106A

Fig.1. ILI9341 touch screen

2.2. NE555

3. System circuit module design

The 555 timer is composed of three 5KΩ resistors, avoltage divider and voltage comparators C1 and C2, andThe system circuit is mainly divided into 5 modules.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

261

Ultrasonic Range Finder for3.1. Single chip microcomputer and its peripheral
circuits

vfour pins, because the driving capacity of the P2transistor is limited, so the digital tube is not directly
connected to the public port P2, but can pass P2 The port
controls three transistors to turn the digital tube power onand off.

The main chip uses a single piece of AT89C51. The
AT89C51 chip and peripheral circuits form the smallest
system, which is the central module of the entire ultrasonicranging system. The 40kHZ transmission frequency isprovided to the software for processing by the
microcontroller P3.2. After the received echo is processedby the AT89C51 information, the measurement distance ison the LED display, and the data displayed by the P0 port
and the port 2 are LED display. The dynamic scanning
method is displayed. If the measurement distance is lower
than the safety distance set in advance (for example: 1m
or 0.5m), the alarm circuit is activated to issue an alarm
signal, and the alarm circuit is controlled by the P2.4
buzzer output port.

3.5. Ultrasonic receiver module
Connect the speaker, transistor and resistor to the P23
microcontroller pin, and the other end of the buzzer isgrounded to form an audible alarm circuit.
4. Test results and summary

3.2. Ultrasonic transmitting circuit
The ultrasonic transmitting circuit needs to output anelectrical signal during operation to enable the ultrasonictransducer to emit ultrasonic waves. In order to increase
the transmission capacity of the ultrasonic wave, it can betransmitted over a longer distance. It is necessary toincrease the voltage of the signal.6 Therefore, theultrasonic transmitting portion mainly includes a squarewave generator and a boosting circuit. At the same time,the microcontroller is used to control the NE555, and the
3-pin output is used to drive the ultrasonic transducer.

It can be seen that the ultrasonic can guarantee theaccuracy of the measurement within a certain
measurement range. As the distance of the obstacleincreases, the detection sensitivity of the ultrasonicranging system will decrease, and the measured distancewill have an error. After the actual test, the error range is
kept around 3 to 5 cm, which is negligible in theautomobile collision avoidance system.

3.3. Ultrasonic receiver module
The ultrasonic receiving circuit can amplify, filter andshape the weak electric signal received by the ultrasonicprobe, and output a step signal, prompting the AT89C51
to calculate the ultrasonic transmission time interval t in
the air. The echo signal shaping is output as a square wave
signal, and then input to the input terminal P3.2 of theexternal interrupt 0 of the microcontroller, and the specifictime of receiving the ultrasonic echo is calculated todetermine the propagation time t of the ultrasonic wave in
the air.

The ultrasonic sensor is used to convert the distanceinformation into an electrical signal and transmit it to theworking module for processing. Timely detection of the
distance between the vehicle and the obstacles in front of
the vehicle and behind the vehicle is a novel active safety
protection system. The NE555 timer and CX20106A
demodulator used in the design improve the measurement
accuracy and intelligence of the system. In addition, thealarm device is inexpensive and easy to install. The
designed ranging system has low cost, good performanceand practicality. It's handy to operate.

3.4. Display circuit

Acknowledgement

This design uses a four-in-one common anode digital tube
SM410564 as a display device. The most common one is
the P0 port, the common end of the P2 port P2.0 --- P2.3

The research is partly supported by the “13th Five-Year”
comprehensive investment and basic research business fee
items of China (000040192).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan262

Ying Chen, Guowei Yang, Fengzhi Dai, Yuxuan Zhu, Di Yin, Yasheng Yuan,Yiqiao Qin, Ce Bian, Chengcai Wang, Xinyu Zhang

4. Xinghua Peng, Research and application of ultrasonic range
finder (Hubei, Hubei University, 2013)
5. Chengjian Lu, Functional Model of 555 Timer and Its
Application (Yulin, Journal of Yulin Teachers College, 2006)
6. Derong Tan, Yuanjie Wu, Research on Key Technologies of
Vehicle Anti-collision Collision Active Safety System ( Taiyuan:
Taiyuan University of Technology, 1998)
7. Lei Li, Honghao Hu, Ultrasonic Sensor Design Based on
AT89C51 Single Chip Microcomputer (Shanghai: Fudan
University, 1999)
8. Guirong Qu, Principle and Maintenance of Infrared
Demodulator CX20106A (Beijing: Institute of Electronic
Design, University of Science and Technology Beijing, 1997)
9. Hongtao Tan, Xueping Zhang. The principle of single-chip
design range finder and its simple application, Journal ofComputer Applications, 2004, 18: p94 - p95.

References
1. Zuohan Wang, Reducing Traffic Accidents: More Active andPassive Safety Technologies Need to Be Adopted,
Commercial Vehicles, 2010, 4: p96 - p97.
2. Huailin Zhao, Yangguang Guo, Masanori Sugisaka, Design of
the Multi-Car Collaboration System, the 2017 International
Conference on Artificial Life and Robotics (ICAROB 2017),
Jan. 19-22, 2017, Miyazaki, Japan, p186 - p190.
3. Xuemei Dian, Application of Ultrasonic Ranging in
Automobile Reverse Collision Avoidance System, Artificial
Intelligence and Identification Technology, 2007, 10: p83～
p85.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

263

Chaos synchronization method of Qi system and the circuit design
Xia Jin1, Fengzhi Dai1*, Zhongyong Ye1, Lingran An1, Yujie Yan1,
Chenglin Zhao1, Yiqiao Qin12, Hao Li3
1

College of Electronic Information and Automation, Tianjin University of Science and Technology, China
2
Tianjin Tianke Intelligent and Manufacture Technology CO.,LTD
3 CETC Ocean Information Co., Ltd, Beijing 100041, China

daifz@tust.edu.cn

Abstract

Chaos is a kind of uncertain and random phenomenon that appears in the system. Its dynamic
characteristics are rich and interesting. The study of the dynamic characteristics and synchronization
methods of Qi chaotic systems has fascinated many researchers. In this paper, the dynamic characteristics
of Qi chaotic system are analyzed, and the synchronization circuit of Qi chaotic system is designed basedon driving-response chaotic synchronous system and single-variable feedback perturbation synchronous
system, The synchronization circuits of Qi chaotic system are designed respectively, comparing thesynchronization curves of the variables simulated by Multisim and the numerical simulation curves
obtained by MATLAB, Verifying the correctness of the built circuit., and analyze the advantages anddisadvantages between above the two synchronization methods.
Keywords: Qi system, Chaotic synchronization, Circuit simulation, DSP builder

1. Introduction
2. Chaotic synchronization
The term chaos originally represented the state of theuniverse when it was not formed. For the origin of theuniverse, many astronomers firmly believed that the chaostheory stated that the universe gradually evolved from theinitial chaotic state to the vast space full of laws andtheorems today. Chaos is a random phenomenon thatappears in a nonlinear system that determines the system.
Chaos is unpredictable and uncertain, but what isinteresting is that it can also be expressed in simple
formulas. More and more people begin to study its relatedcharacteristics and strive to apply it to real life. As chaos
is separated from physics into a single discipline[2], more
and more research results have enriched people's vision.
People are more convinced that chaos research will have apositive and far-reaching impact on human production andlife in the near future.

Chaotic synchronization[3], that is, two systems withdifferent initial conditions, whose orbits change with timeand merge into one orbit. Chaotic synchronization can also
be said to be a kind of chaotic control. So far, people havefound that a variety of synchronization methods have beenstudied, and these synchronization methods havecontinuously promoted the application of chaotic systems.
These methods can achieve good results in chaotic
synchronization. In this paper, the two methods control ofdriving response and univariate coupling control[1] areused to design the synchronization circuit of Qi chaotic
system. The correctness of the two synchronization
methods is proved by analyzing the simulation results.
And compare the two synchronization methods, and givethe method and theoretical support for the application
based on Qi chaotic system.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

264

Xia Jin, Fengzhi Dai, Zhongyong Ye, Lingran An, Yujie Yan, Chenglin Zhao, Yiqiao Qin, Hao Li

3. Driver response synchronization for Qi chaotic
systems驱 动 系 统 的 z（ 蓝 色 ） 与 响 应 系 统 的 z（ 红 色 ）
250

200

This section synchronizes the driving response of the Qi
chaotic system, and builds the modeling diagram basedon the DSP builder in the FPGA development module[7].
Let the continuous differential equations of the Qi chaotic
system be:

z

150

100

50

̇ = ( − )+
,
̇ =
− −
, （1）
̇ =
−
,
In this paper, the y variable is used as the driving signal,and the x variable is used as the driving response signal.
Use the DSP builder module to draw the modeling diagramin the Simulink template as follows:

0

0

1000

2000

3000

4000

5000

6000

7000

t

Fig.3. Chaos trajectory of variable Z in both systems
80
60
40
20
0
-20
-40
-60
-80
-80

-60

-40

-20

0

20

40

60

80

Fig.4.State synchronization diagram of variables X in both
systems
120

110

100

90

80

Fig.1.Synchronous modeling of driving response with Y as
the driving variable

70

The initial value of the drive system is (1, 1, 1) and theinitial value of the response system is (10, 10). The
modeling diagram is simulated to obtain the chaotic
trajectory curves of the two systems x and z variables asshown in the following figure.

60

50
50

40

x

20
0
-20
-40
-60
1000

1500

2000

2500

80

90

100

110

It can be seen from the above figure that the chaotic
trajectory[4] of the drive system and the response system issynchronized after a period of time, and thesynchronization time between the two system x variables
is very short, the state synchronization diagram shows acomplete 45 degree slash, and the two system z variables
Although it can be synchronized, but the synchronization
time is relatively long, the 45-degree diagonal line
displayed in the state synchronization diagram has aprominent part, which is incoherent, and the actualsynchronization effect is not as good as thesynchronization effect of the x variable. The
synchronization effect of the response system variable x is

60

500

70

Fig.5. State synchronization diagram of variables Z in both
systems

驱动 系 统 的 x （ 蓝 色 ） 与响 应 系 统 的 x （ 红 色 ）

0

60

3000

t

Fig.2. Chaos trajectory of variable X in both systems© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan265

Chaos synchronization method ofshorter than the synchronization effect of the variable z,
and the synchronization time is short and the precision ishigh. Therefore, if the variable y is selected as the driving
signal for the synchronization of the driving response of
the Qi system, the x variable is used as the effective signal
for practical application.

x

200
0
-200

0

50

100

150

100

150

100

150

t

y

100
0
-100

0

50
t

200

Univariate coupling synchronization for Qi
chaotic systems

z

4.

100
0

0

50
t

In this section, the univariate coupling synchronization[5]of the Qi chaotic system is performed, and the y variable
perturbation is used for synchronization.
The response system is:̇ = ( − )+
,
̇ =
− −
+ ( −
̇ =
−
,

Fig.7.Two systems X, Y, Z variable trajectory

According to the simulation diagram, thissynchronization method is effective, and the two systems
can be synchronized, the synchronization time is less than50, and the synchronization precision of the x variable ishigh, and the synchronization time is short. The
corresponding variables of the drive system are subtracted
from the corresponding variables of the response system,and the error graph and state synchronization diagram are
obtained for the circuit simulation. The results are shownin the following figure:),（2）

Design the analog circuit using Multisim as shown in the
figure below:

Fig.8.X1-x2 error graph

Fig.9.Y1-y2 error graphFig.10.Z1-z2 error graphFig.6.A circuit that synchronizes with a Y variable perturbation

The feedback gain is taken as 100, and the Matlab
simulation diagram is as follows:
Fig.11. State synchronization diagram for variable x1-x2© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

266

Xia Jin, Fengzhi Dai, Zhongyong Ye, Lingran An, Yujie Yan, Chenglin Zhao, Yiqiao Qin, Hao Li

5. Testing and conclusion
By comparing the driving response synchronization
method with and univariate coupling synchronization
method, it can be seen that the driving response[6]synchronization method is simple and easy to understand.
For the synchronization of the Qi chaotic system, theoperation is convenient and easy to succeed, and the final
result is also ideal, but in the face of other indecomposable
system is powerless and has certain limitations. The
univariate coupling synchronization method is applicablewidely, without decomposing the system, highsynchronization precision and high speed, but the
feedback coefficient k needs to be researched and obtained,
and at least two variables need to be feedback perturbed
when facing a hyperchaotic system. Increased thecomplexity.

Fig.12.State synchronization diagram of the variable y1-y2

Fig.13. State synchronization diagram of the variable z1-z2

It can be seen from the synchronization error graphbetween the three variables of the two systems that the
perturbation with the variable y can synchronize the two
Qi chaotic systems. By observing the state synchronization
diagram of the three variables in the dynamic process, thestate synchronization diagram of the x variable is uniformwith the state synchronization diagram of the other two
variables, and the 45degree diagonal line is uniform,
although there are errors at both ends of the oblique line,
but the oblique line Obvious and precise, it proves that thesynchronization effect is good. The 45-degree slash of they variable's state synchronization diagram is clearly visible
and the synchronization effect is also good. The statesynchronization diagram of the z variable has a very thinslash at the lower left end, but the upper right end is very
thick, the distribution is uneven, and the synchronization
effect is not good. The Lyapunov exponent calculated byMatlab is λ = −14.9805, λ = −28.3155, λ =−95.3522. The conditional Lyapunov exponent with all
negative values proves that the two chaotic systems canreach the synchronization state.

References
1. Zhong Q, Zhang C. Chaos Control of a New Chaotic
Dynamical System and Its Application. Lecture Notes inElectrical Engineering, 2014, 163:1465-1472.
2. Chen Y, Li S G, Liu H. Synchronization of fractional-orderchaotic systems based on adaptive fuzzy control. Acta
Physica Sinica, 2016.
3. Nijmeijer H J C H. Nonlinear chaos control andsynchronization. Handbook of Chaos Control, Second
Edition. Wiley‐VCH Verlag GmbH & Co. KGaA, 2007.
4. Yong-Jian L. Combined Synchronization Control of Hyperchaotic Systems. College Mathematics, 2011.
5. Zelinka I, Senkerik R, Navratil E. Investigation on realtime
deterministic chaos control by means of evolutionary
algorithms. IFAC Proceedings Volumes, 2006, 39(8):190196.
6. Senkerik R. Optimization of feedback control of chaos by
evolutionary algorithms. Analysis & Control of Chaotic
Systems. 2006.
7. Tuna M , Can B ü lent Fidan. Electronic circuit design,
implementation and FPGA-based realization of a new 3D
chaotic system with single equilibrium point. Optik International Journal for Light and Electron Optics, 2016,
127(24):11786-11799.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

267

Design of Air Quality Monitoring Platform Based on Internet of Things
Yujie Yan1, Fengzhi Dai1*, Kailun Zhang1, Wei Wang2, Jialin Han1, Yang Li3, Tianyi Zhang1
1College of Electronic Information and Automation,
Tianjin University of Science and Technology, China

2
Department of Electronic and Control Engineering,North China Institute of Aerospace Engineering , China
3

CETC Ocean Information Co., Ltd, Beijing 100041, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

The air quality monitoring platform based on the Internet of Things (IoT) uses the STM32f407 main
control module. Data such as temperature, humidity, PM2.5 and carbon monoxide concentration in theatmospheric environment are collected for a long time by each sensor module. Not only the system has
the LCD display function, users can also log on to the ESLINK IoT cloud platform to remotely observe
the data collection status. The upper limit of each parameter can be set in a specific environment. If thedata monitored by the system exceeds the upper limit, the system will alarm. The system has a good
application prospect in the environment of office, automobile, factory and so on.
Keywords:

Internet of Things, STM32f407, PM2.5, sensor

1. Introduction

function, and when a certain parameter exceeds the set
range, it can immediately alert the user. The system has
low cost, high precision, convenient use and certain
market application prospects.

With the development of industry, environmental
problems have become more and more serious, especially
air pollution has seriously affected people's lives.
Traditional meteorological monitoring stations are bulky,
susceptible to environmental and time constraints, andtheir communication networking costs are high anddeployment is difficult. At present, it is hoped that a
low-cost portable atmospheric monitoring device canobtain air quality information in real time. The
monitoring equipment can be accurately monitored in thehome, in the car, in the workplace1, etc.
The air quality monitoring platform developed by thisdesign can be applied to various occasions, and canmonitor the temperature, humidity, PM2.5, CO and otherconcentration changes in the air in real time, and canmonitor the current data information on the LCD screen
and terminal equipment. The system has an alarm

2. Hardware structure design
The hardware data acquisition part mainly uses eachsensor node to monitor the atmospheric environment
quality in real time, and transmits the detectedenvironmental data information to the network layer inreal time, and displays it on the LCD display in real time.
The environmental monitoring hardware system uses
STM32F407 as the main control chip for signal
processing, and each sensor module performs sensing
signal extraction. The signal conditioning circuit shapes,
amplifies, and filters the circuit waveform to condition
the data into a signal that the microprocessor can process.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

268

Yujie Yan , Fengzhi Dai , Kailun Zhang, Wei Wang, Jialin Han, Yang Li, Tianyi Zhang

2.3. Humidity sensor module

The A/D signal acquisition module performs signal
acquisition, and the MCU part performs signal processing
analysis. The overall hardware design of the system isshown in the Fig.1.

This design uses the DHT11 temperature/humidity sensor,
which is a temperature and humidity composite sensor
with a calibrated digital signal output. It uses dedicateddigital module acquisition technology and temperatureand humidity sensing technology to ensure highreliability and stability4. The sensor consists of a resistive
wet sensor and an NTC temperature sensor connected toa high performance 8-bit microcontroller. The calibration
coefficients are stored in the OTP memory as a program,and these calibration coefficients are called internallyduring the processing of the detection signal. The
single-wire serial interface makes system integration easy
and fast. The product is a 4-pin single-row pin package
for easy connection. The circuit principle of DHT11
interface is shown in the Fig.3.

Fig.1. Hardware design of the system

2.1. STM32F407 master chip
The main control chip adopts high-performance 32-bit
ARM M4 architecture series chip STM32F407. The
chip's clock speed is up to 168MHz, integrating
single-cycle DSP instructions and floating point units.
The processing capacity is up to 210DMIPS, which isabout 200 times that of a normal 8-bit microcontroller.

Fig.3. DHT11 hardware schematic

2.2. Wireless Communication Technology

2.4. Carbon monoxide collection module

The wireless communication technologies currently usedin the market include Bluetooth technology, Zigbee
technology, and WIFI technology. WIFI (WirelessFidelity) is a certification standard for wireless LAN
products consisting of Access Point and wireless network
cards2. WIFI technology is easy to integrate with existing
wired Ethernet networks3.
The ESP8266 is an ultra-low power UART-WIFI
pass-through module designed for mobile devices and
IoT applications. The user's physical device can beconnected to a WIFI wireless network for Internet or
LAN communication for networking. The hardware
schematic is shown in the Fig.2.

This design uses the MQ-7 carbon monoxide
concentration monitoring sensor, which is a dual signal
output (analog output and TTL level output). The TTL
output valid signal is low level (when the output is low,
the signal light is on, it can be directly connected to thesingle chip); the analog output is 0~5V, the higher theconcentration, the higher the voltage5. The sensor has
high sensitivity and good selectivity to carbon monoxide,and has a long service life and reliable stability. The
MQ-7 sensor hardware schematic is shown in the Fig.4.

Fig.4. MQ-7 sensor hardware schematicFig.2. WIFI hardware schematic© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

269

Design of Air Quality

2.5. Temperature acquisition module

serial port connection, SPI bus connection and SRAM
memory interface. The LCD panel of the project
development board uses the SRAM interface, which isequivalent to the LCD connected to the memory interfaceof the main chip. Therefore, accessing the LCD isequivalent to accessing the memory of the main chip byaccessing the three buses (data bus, address bus andcontrol Bus) implementation. The hardware schematic of
LCD9341 is shown in the Fig.7.

With the DS18B20 temperature sensor, only one port line
is required to connect the microprocessor to the two-way
communication between the microprocessor and theDS18B20. The working power supply is 3.0~5.5V/DC,no external components are needed in use; themeasurement results are serially transmitted in 9~12
digits; multi-point networking function is supported6. To
achieve multi-point temperature measurement, if thenumber is too much, the power supply voltage will be too
low, resulting in unstable signal transmission. The
hardware schematic of the DS18B20 temperature sensor
is shown in the Fig.5.

Fig.7. The hardware schematic of LCD9341

3. Software design
3.1. Temperature acquisition humidity realization
Fig.5. DS18B20 temperature sensor

The temperature sensor DS18B20 is a digital temperature
sensor that does not require AD conversion. The sensor
converts temperature to digital up to 93.75ms at 9-bit
resolution and temperature to up to 750ms at 12-bit
resolution for even more accurate results. This design
selects 12-bit data representation. The upper 5 bits are
sign bits, the lower 4 bits are fractional parts, and theothers are integer bits. After parsing this data, thetemperature is obtained. According to the read and write
timing of the DS18B20, the program realizes the reset,
initialization, read and write data of the sensor and theconversion function of the read data.

2.6. PM2.5 acquisition module
The GP2Y1010AU0F optical dust concentration
detection sensor is designed to sense dust particles in the
air. An infrared light-emitting diode and a phototransistor
are placed diagonally inside, so that it can detect thereflected light in the air. Even very small particles such as
tobacco smoke can be detected, and tiny particles of 0.8
micron or more can be measured to perceive phlegm andpollen produced by tobacco, house dust and the like. The
sensor has a very low current consumption and can beused up to 7VDC. The sensor output is an analog voltage
whose value is proportional to the dust concentration.
The hardware schematic is shown in the Fig.6.

3.2. Program
implementationacquisitionofhumidity

The humidity sensor DHT11 is also a digitaltemperature/humidity sensor that does not require AD
conversion. DHT11 should use the GPIO port of the
MCU for initial configuration, set the required GPIO portpin (PA4) to general push-pull output, no pull-up
pull-down, 50MHz output mode, so that the
DHT11-DATA pin of the MCU is set to none. Pull-up
pull-down input mode. The single-bus data format
communication between the microprocessor and the
DHT11 takes about 4ms. The data is divided into
fractional part and integer part. The complete data

Fig.6. GP2Y1010AU0F hardware schematic

2.7. LCD display module
The LCD9341 with 320*240 pixels displays the datacollected by each sensor. Generally, there are three ways
to connect the liquid crystal display to the main chip:© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan270

Yujie Yan , Fengzhi Dai , Kailun Zhang, Wei Wang, Jialin Han, Yang Li, Tianyi Zhang

transmission is 40bit, and the high position is first out.
After the user MCU sends a start signal, the DHT11
switches from the low power mode to the high speed
mode. After waiting for the host start signal to end, theDHT11 sends a response signal, sends 40 bits of data, andtriggers a signal acquisition, and the user can select toread part of the data.

4. Conclusion
After the system is running, the actual changes of eachparameter can be monitored on the IoT cloud platform.
Take the data waveform monitored by the temperature
sensor as an example, as shown in the Fig.8. It can beseen that the system can monitor temperature changes
and generate waveforms in real time over a certain periodof time.

3.3. Program implementation of carbon monoxideand PM2.5 acquisition
The mq-7 and GP2Y1010AU0F for collecting carbon
monoxide and PM2.5 are analog sensors, and the output
is analog, so the acquired analog signal needs to beconverted into a digital signal. There are 3 ADCs in the
STM32F407zgtx, a total of 24 channels, each ADC is 12
bits, corresponding to the converted digital range from 0
to 4095. In this project, for MP2.5, channel 10 of ADC1
is selected; Co selects channel 4 of ADC3. When
programming the ADC converter, it includes several
functions such as initialization, turning on AD conversion,
waiting for the end of AD conversion, and obtaining theconversion result.
3.4. Program
implementation
communication moduleof

Fig.8. Temperature data waveform

Acknowledgement
The research is partly supported by the “13th Five-Year”
comprehensive investment and basic research businessfee items of China (000040192).

wireless

References
1. Hongyu Xu, Huangfei Shen. Design of multi-pointtemperature detection system based on STM32. Electronic
Science and Technology, 2016, 5: P158-P161.
2. Jincheng Wu, Chengcheng Zeng, Jingrui Sun. PM2.5
Monitoring System Based on STM32F407 WirelessIntelligent Network. Journal of Tianjin Normal University,
2017, 6: P62-P66.
3. Huailin Zhao, Shunzhou Wang, Yunxiang Liu, Xuyao Hao,
Design of an Intelligent Housekeeping Robot Based on IOT,
Journal of Information and Communication Engineering,
Vol.2(2), 2016, P114-P118.
4. Na Liu, Temperature and Humidity Measurement andControl System Based on ZigBee Module Granary.
Microcontrollers & Embedded Systems, 2016, 5: P69-P72.
5. Xiaoping Ding. Design and Research of CO Detection andAlarm System Based on Wireless Network (Jiangsu, Suzhou
University, 2013)
6. Kai Li, Chen Xu, Research on temperature controller
design based on DS18B20. Science and Technology
Innovation, 2018, 7: P163-P165.

Software programs include WIFI initialization functions,connect WIFI, establish connections, disconnect, andsend data functions. A macro API_KEY defined in the
program is the connection key obtained after the cloud
platform is successfully registered. After the module
initialization is completed, the WIFI connection is
implemented by the customized esp8266_link_wifi()
function, where the first parameter of the function is the
network name to be connected, and the second parameteris the connection password. The status of thecorresponding device in the ESLINK IoT cloud platform
is obtained by the download_sensor_status() function.
The parameters of the function include the IP address ofthe connection route, one or more device numbers, and
API_KEY. Upload the data collected by each sensor tothe cloud platform through the upload_device_datas()function. The function has six parameters: the IP addressof the connection route, the device number, the
API_KEY, the array of sensor numbers, and the datacollected by each sensor. The array and size arecomposed.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

271

Design of Metal Weld Seam Tracking Equipment Based on Image Processing
Yujie Yan 1, Fengzhi Dai 1*, Lingran An 1, Xia Jin 1, Zhongyong Ye 1, Yufan He2, Xinran Guo3
1

College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
2

CETC Ocean Information Co., Ltd, Beijing 100041, China
3

School of Government, Nanjing University
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

With the development of the intelligent industry, metal automatic welding technology needs to be more
popular. The metal weld seam tracking device based on image processing includes two parts of image
processing and motion control. The image processing part is developed by the Visual Studio 2013
platform, and the motion control part is controlled by the PID algorithm. Machine vision technology canbe used to convert images captured by industrial cameras into vector graphics, which in turn drive thewelding table to track metal welds. The device has good control effect and can be well applied in small
and medium-sized enterprises.
Keywords: automatic welding, image processing, Visual Studio 2013, motion control, PID

1. Introduction

camera. A weld seam edge vector that can be recognizedby the control system is obtained by image processing
techniques. Finally, the motion control system controls
the welding table motion based on the weld edge
information to achieve automatic tracking and tracing of
the weld. The device has high control precision and canbe applied to small and medium-sized enterprises, andhas certain market prospects.

In the traditional welding industry, most welders use
manual welding methods. And because of the large
amount of smoke, arc and high temperature generated
during the welding process, this will directly damage thewelder's health. Moreover, the manual welding operationis labor intensive, and it is difficult for the welder toensure the welding quality under a long working
condition1.
The development of industry has promoted the rise andapplication of automatic welding technology. Currently,
teaching welding robots are the most widely used.
However, the welding process is easily affected by manyfactors, and the complex products processed by theteaching-type welding robot are difficult to meet theprocess precision. The intelligent weld seam tracking
device designed in this paper can collect the image
information of the workpiece to be welded by industrial

2. The hardware structure design
The overall frame of the automatic seam tracking
equipment is made of aluminum alloy and iron alloy, andthe equipment is equipped with hardware such asindustrial cameras and lenses. The control part iscomposed of STM32F407 main control board, three
stepping motors and motor drive. Three stepping motors
control the X, Y and R axes respectively. Two motors
control the X and Y axes for two-dimensional linear© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan272

Yujie Yan , Fengzhi Dai , Lingran An , Xia Jin , Zhongyong Ye , Yufan He, Xinran Guo

motion, and one motor controls the R axis for planarrotation motion to achieve three-degree-of-freedom
motion control. The mechanical structure model of theautomatic seam tracking equipment established by
SOLIDWORKS modeling software is shown in Fig.1.

Fig.2. DH-HV3151UC industrial camera

2.2. Camera lens module
The lens is very important to the camera, no lens camera
can not shoot any information. The lens is composed of aseries of prisms. The incident light passes through theprism group, and a series of transformations such as
convergence focus occurs, and finally an image is formedon the photosensitive element. The optical system of thelens is very important for the control of light, and thephotosensitive element is also imaged by the transformed
light. Therefore, the choice of lens will have a direct
impact on the quality of the image captured by the system.
When selecting the lens of an industrial camera, thefollowing parameters are mainly considered: focal length,
resolution, depth of field, aperture. In this design, theindustrial camera mainly collects images of metal
workpieces that need to be welded for subsequent image
processing. Therefore, the M0824-MPW2 fixed focus
lens is used here, as shown in Fig.3.

Fig.1. The design of the mechanical structure model

2.1. Industrial camera module
Industrial camera is one of the important modules ofimage acquisition system. Its performance will directly
affect the quality of the captured weld image, which willaffect the subsequent image processing. Industrial
cameras work by capturing a pattern with weld
information, converting the pattern signal into a digital
signal, and finally storing it on a computer as a digital
signal2.
Industrial cameras can be divided into CCD (Charge
Coupled Device) cameras and CMOS (Complementary
Metal Oxide Semiconductor) cameras. Among them,
CMOS cameras have a large light sensing range and low
cost, and have good guarantees in terms of safety,
radiation resistance and transmission speed. This designuses DH-HV3151UC industrial camera, as shown in
Fig.2. This camera belongs to the USB2.0 series ofcameras, with high resolution and clarity. It does not
require an additional image capture card during use, and
it is very convenient to communicate with the computer.

Fig.3. Camera lens module© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

273

Design of metal weld

2.3. Stepper motor selection• Bipolar constant current chopping mode with chopping
frequency up to 100KHZ.

The stepping motor converts the pulse signal into angulardisplacement or linear displacement, rotates according to
the set angle, and runs step by step according to thereceived pulse signal. When the motor is working
normally, only the number and frequency of the pulse
signals will affect its motion, regardless of the load. The
number of pulse signals affects the angular displacementof the motor, and the frequency of the pulse signal affects
the speed and acceleration of its rotation, thereby
regulating and positioning the system3.
According to the internal structure of the motor, it canbe roughly divided into three types: Variable Reluctance(VR), Permanent Magnet (PM) and Hybrid Stepping(HS). The unit uses a 36GP27B-42H250C13 two-phasehybrid stepper motor to adjust the motion of the entire
weld tracking device. Fig.4 shows the appearance of the
motor.

Fig.5. Motor drive module

2.5. Main control chip
The main control chip adopts high-performance 32-bit
ARM M4 architecture series chip STM32F407. The
chip's clock speed is up to 168MHz, integrating
single-cycle DSP instructions and floating point units.
The processing capacity is up to 210DMIPS, which isabout 200 times that of a normal 8-bit microcontroller5.
This allows the automatic seam tracking device toquickly process the acquired information and output theresults of the analysis to the actuator.
3. Image acquisition and processing
Before the weld seam is controlled and tracked, image
acquisition and processing are first required. Before this,the entire device needs to be initialized. After that, theindustrial camera is called to perform image acquisitionon the metal characters, and the captured image is readand judged by the computer. If the image fails to be read,the industrial camera needs to be recalled to capture theimage and read the image until the image is successfully
read.
After successfully reading the target image, it isnecessary to perform a series of processing on the image,and finally obtain the vector coordinates of the edge ofthe image weld. The image processing process mainly
includes image denoising of images, thresholdsegmentation of images, contour extraction and filteringof images, skeleton refinement and extraction of image
edges, and final bitmap vectors.
In this design, the collected image is filtered anddenoised by a median filtering method, and the denoised
welded image is subjected to threshold processing using
the Otsu method. The Suzuki85 algorithm is selected to
extract the edge of the pre-processed weld image. After

Fig.4. Two-phase hybrid stepper motor

2.4. Motor drive module
The stepper motor must be matched to the corresponding
motor drive for precise motion. The motor driver divides
its motion step angle into several small steps bycontrolling the phase current of the input motor to realize
the subdivision drive4. This design uses the matching
M420B stepper motor driver, as shown in Fig.5 below.
This type of stepper motor driver has the following
features:• High performance, low price, compact form factor andeasy installation;• The power supply voltage is 18~40VDC, and thedriving current is 0.25~2A;• Automatic semi-flow function; maximum dynamic 128
subdivision;• Photoelectric isolation signal input, motor noise
optimization function;© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

274

Yujie Yan , Fengzhi Dai , Lingran An , Xia Jin , Zhongyong Ye , Yufan He, Xinran Guo

Fig.6. Distribution of each motion axis

the edge image of the weld image is obtained, theskeleton extraction of the image is required, and the
Zhang fast parallel refinement algorithm is selected6.
Since the ultimate goal of image processing is to provide
the subsequent weld tracking control system with control
information that it can recognize, the edge image of theweld image is finally bitmap vectorized.

Acknowledgement
The research is partly supported by the “13th Five-Year”
comprehensive investment and basic research businessfee items of China (000040192).
References

4. Weld seam tracking system workflow

1. Zhiwei Luo, Lixiu Li. Research on automatic welding
guidance of industrial robot based on vision. Welding
Technology, 2018, 8: P69-P73.
2. Wenju Zhou, Research and application of online
high-speed detection and precise control based on machine
vision (Shanghai, Shanghai University, 2014)
3. Xiaolei Du. Design of electric lifting window based on
AT8989S52 single chip microcomputer. Mechanical &Electrical Technology, 2018, 5: P19-P21.
4. Qi Shimeng, Yan Huasong, Huang Wenhui. Design of
Robot Gripper Control System Based on Stepper Motor.
Instrument Technique and Sensor, 2018, 9: P61-P65.
5. Hao Zeng, Wei Zhang, Sikai Zheng. Image Remote
Acquisition Terminal Based on STM32F407. Industrial
Control Computer, 2014, 11: P81-P83.
6. Fengzhi Dai, Jinbin Zhao, Huailin Zhao. The applicationof resonance algorithm for image segmentation. Applied
Mathematics and Computation, Vol.194 (2), 2007, P453–
P459.

After image processing, the control systemdetermines the starting point of the weld to track theweld by identifying and calculating weld vector
information. And by controlling the motion trajectoryof each coordinate axis, the welding workbench is
moved to achieve the purpose of tracking anddescribing the weld seam. During the seam tracking
process, the arm above the device is fixed. During the
movement, the welding station can perform thethree-axis linkage of the X-axis, the Y-axis and the
R-axis according to the motion information provided
by the control system. The X-axis controls the left andright movement of the welding table, and thetheoretical stroke can reach 1200 mm; The Y axis
controls the back and forth movement of the welding
table, the theoretical stroke is 1600mm; the R axis
rotates by driving the indexing plate. The welding
table is driven to rotate, and the indexing accuracy
can reach 0.2°. Fig.6 below shows the distribution ofeach motion axis on the plane of the welding table.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

275

The recognition and implementation of handwritten character based on deep learning
Zhongyong Ye1, Fengzhi Dai1*, Xia Jin1, Yasheng Yuan1, Lingran An1, Yujie Yan1, Yiqiao Qin12, Hao Li3
1
College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
2 Tianjin Tianke Intelligent and Manufacture Technology CO.,LTD, China;
3 CETC Ocean Information Co., Ltd, Beijing 100041, China

E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

This paper mainly focuses on the recognition of handwritten characters, especially handwritten Chinesecharacters. Using deep learning technology constructs a deep convolution neural network to identify thecharacter set of handwritten Chinese characters, compares the performance differences of the same depth
network, and finally gets the network structure which can be used for recognition, and realizes therecognition system of handwritten characters based on the network. By comparing with other hand writing
characters, the engineering application value of the network structure used in this paper is proved, andfinally the handwriting character recognition system based on this model also embodies the feasibility of
the network structure in this paper.
Keywords: Deep learning, Machine learning, Pattern recognition, Handwriting character recognition, Convolution
neural network

1. Introduction

character characters has always been a hot research
problem in the field of machine learning.

Deep Learning is a deep-seated network structure based on
multiple hidden layers proposed in recent years, which isused to study and deal with some popular problems in thefield of machine learning, such as image retrieval andimage recognition1.
At present, although there are more application of deeplearning technology to solve the application of handwritten
digital character recognition, but in the field ofhandwritten Chinese character recognition, it is more
traditional method based on artificial feature extraction.
Because Chinese character strokes are more complex thanother common characters English letters or Arabic
numerals, and handwritten Chinese characters are morevaried because of the different styles and habits of personal
writing, Therefore, the recognition of handwritten Chinese

2. Application of deep convolution neural networkin handwritten Chinese character recognition
2.1. Data preparation and preprocessing
Because this paper is aimed at the recognition of
handwritten Chinese characters, the sample data using
handwritten Chinese characters set is also used from the
HWDB1.1 database to obtain some of the data samples. In
this experiment, the sample data HWDB1.1 divided into 4
sub-datasets, that is, Ten classification, Hundredclassification, Thousand classification, 3755 classificationof the dataset will be named Set1, Set2, Set3 and Set4
respectively. The composition of each dataset is as follows
in table 1.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

276

Zhongyong Ye, Fengzhi Dai, Xia Jin, Yasheng Yuan, Lingran An1, Yujie Yan, Yiqiao Qin, Hao Li

be grayscale to get a single-channel picture, it does nothave the same as high-resolution natural images, in the
RGB three channels have a wealth of information.
2. Change the number of convolution cores in the first
convolution layer from the original 96 to 60, and thecorresponding pooling layer changes accordingly.
3. Remove the first fully connected layer and connect
the final output layer with only one full-connection layer.
4. Change the number of output nodes of the finalclassifier to 10,100,1000 and 3755, respectively, that is,the datasets used are Set1, Set2, Set3, and Set4,
respectively. The structure obtained by the network
through the above 1,2,3 step is named CNNet, and because
its final output node is slightly different, it is named
CNNet1, CNNet2, CNNet3 and CNNet4 respectively.

Table 1 Composition of the data sets

In table 1, set name represents the data name of fourchild datasets, Writer1 that the number of people writing
training samples, that is, the sampling source of the
training sample, came from a different 240 persons,Writer2 represents the number of people who wrote the test
sample, The training column represents the total numberof training samples in the dataset, Testing column
indicates the total number of samples used for testing in
the dataset, and the last column represents the total numberof samples used to experiment with the dataset.

3. Experimental results and analysis
3.1. The influence of calculating mean image on the
result in preprocessing

2.2. Model structure and its improvement method

In this experiment, in order to verify whether the mean
processing of images in preprocessing will have an impact
on the training of the network, under the condition that therelevant parameters of the network are unchanged, thenetwork CNNet3 obtained by the improvement of theAlexnet network are preprocessed by mean calculation
Set3 Training and Training sample SeT3 without meancalculation preprocessing are trained, and the relevant
results of the two training models are shown in Fig.2.

The network used in the design of this paper is improvedon the basis of Alexnet deep convolution neural network.
It contains five convolution layers, 3 of which are followed
by a pool layer with a maximum sampling, followed by 3
full-attached layers, and the last Oftma classifier with 1000
output nodes, and if the input layer is not included, the total
number of layers of the network reaches 8 layers, thenetwork structure is shown in Fig.1.

Fig.1. Alexnet Network structure diagram

Fig .2 Relationship between calculating mean value andnetwork convergence in preprocessing

In this paper, the main improvement of the Alexnet
network is to adjust the number of convolution nuclei inthe convolution layer, reduce the number of convolution
layer, and adjust the number of the full connection layer,
according to the performance of their own computer toreadjust the network training parameters, the specificimprovement is as follows:
1. Reduce the dimension of the input layer from 3 to 1,
because for handwritten Chinese characters, the image can

The horizontal coordinates in the figure represent the
number of iterations of the network, that is, the number oftraining times, the longitudinal coordinates represent theloss value of the network model.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

277

The recognition and implementation

3.2. network related training parameter settings

weight needs to be used in the sample batch that isbatch_size also larger. Coupled with the need for all thetest sample data after each 10,000 iterations to verify theclassification accuracy of the network, it takes nearly 4days for the network cnnet to be fully trained on the
training Set 4, so that the time spent on each parameter isenormous, This is also a major disadvantage for deep
convolution neural networks compared to simple
convolution neural networks.

Using the strategy of the relevant parameters in Le Net5,
the learning rate is increased from 0.01 to 0.02, which isequivalent to increasing the step length, The decrease of
the gradient descent method is increased, and the network
oscillation is avoided, thus crossing the local minimum
point and approaching the larger extremum point. Finally,
it is proved by experiments that the value of batch_size is
reduced to 64 o'clock, the network can converge at a faster
speed, and the classification accuracy is high, which
proves that the network also achieves the optimal solution,
so this paper sets the Batch_size to 64 and the learning rate
to 0.01.

4. System design and implementation
4.1. System structure and design
1. System Framework and related modules
This system architecture adopts BS structure as shown in
Fig.3, that is, users only need to have simple interaction
with the browser, most of the core business of the system
is on the server side. The system of this paper uses Pythonbased Tornado web framework:

3.3. Experimental results and analysis of four data
sets and mnist in Cnnet
After adjusting the relevant parameters of the network sothat the network can converge, the four sub-datasets Set1,
set2, set3 and Set4 of HWDB1.1 handwriting characters
are trained with four sub-networks of Cnnet respectively.
For cnnet networks with higher complexity (moreconvolution nuclei and deeper network structures), therecognition rate of handwritten Chinese characters ishigher, and their ability to express the sample features ismuch higher than that of the simpler convolution neuralnetwork Net2, that is, more sample categories can beclassified.
Although the convolution network with shallow
accuracy has increased more, the complexity of cnnet
network is higher on the network convergence, and thefollowing table 2 is the comparison between Net2 andcnnet networks in convergence.

Fig.3. BS architecture diagram

According to the different division of functions, the system
can be divided into the following modules: input module,
model information loading module, model classification
module, result mapping module and the final output
module, the system flow chart composed of these modules
is shown in Fig.4.
Input Module

Table 2 Comparison of convergence times between Net2 andCnnet

Model Information
Loading Module

Model Classification
ModuleOutput Module

Result Mapping
ModuleFig 4 Overall module flow chart of the system

2. Core class Design
The main modules of this system are model loading, model
classification and result mapping module, so each module
corresponds to design a corresponding core class,complete the main functions of the module.
(1) Load model module; (2) Model classification
module; (3) Result mapping module.

Under the same experimental conditions6 7, cnnet ismuch higher than the Net2 network in the number of
iterations required to make the network converge. Taking
the training of DataSet Set4 as an example, the number ofsamples used for training has reached more than 900,000,
and each image size is larger, the network each training© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

278

Zhongyong Ye, Fengzhi Dai, Xia Jin, Yasheng Yuan, Lingran An1, Yujie Yan, Yiqiao Qin, Hao Li

4.2. Data-related preprocessing operations

1000 classes of the DataSet as shown in the map. As shownin the figure, class_000 represents the first classification of
the sample set, in which the folder is the same character
dataset, corresponds to the first character " 丁 " in the
mapping file and marks it as n1001.

The model used in this system is CNNet3, the input image
is a single channel grayscale image, and the test picture
which has been processed does not need to be grayscale,but if the color image of RGB is entered, it needs to be
preprocessed by grayscale.
There are many ways to grayscale, such as thecomponent method, which uses the pixel value of one of
the channels of RGB as the grayscale value; Mean method,
the average of three channels is used as the grayscalevalue; Weighted average method, that is, according to acertain weight to the RGB three components weighted, thispaper uses the third Way, which represents the grayscale
image, respectively, the input image of three sub-vectors,
the grayscale formula is as follows:

Acknowledgement
The research is partly supported by the “13th Five-Year”
comprehensive investment and basic research business fee
items of China (000040192).
References
1. Rong Zhang, Weiping Li, Tong Mo, Review of Deep
Learning, Information and Control, 2018, 5: P385-P397.
2. Bing Shun, Dehui Kong, Wenhui Zhang, Wenhao Jia,
Survey on Human Action Recognition From Depth Maps,
Journal of Beijing University of Technology, 2018, 10:
P1353-P1366.
3. Shengyi Zhou, Techniques Review of Deep Learning,
Information Technology Exploration, 2018, 10: P116-P117.
4. Anqi Liu, Shuai Zhang, Xinfang Chang, Bingkun chen, Real
time human action recognition and monitoring method
based on deep learning, Heilongjiang Science, 2018, 10:
P14-P16.
5. Kai Yang, Youchun Xu, XiangBi An, Yonglei Li, Kunpeng
Liu, Vehicle Detection Method Based on Deep Learning,
Army Military Transportation Academy, 2018, 7: P58-P61.
6. Youwen Huang, Chaolun Wan, Human behavior
recognition algorithm based on deep learning, Artificial
Intelligence, 2018, 10: P1-P5.
7. Qicheng Mei, Wenge Lv, Research on Commodity Image
Recognition Based on Deep Learning, Mechanical andElectrical Engineering technology, 2018, 9: P27-P151.
8. Huailin Zhao, Zheng Wu, Xiaoxing Wang. A Multi-robot
Rescuing System. The 2018 International Conference on
Artificial Life and Robotics (ICAROB 2018), Feb.1-4, 2018,
Beppu, Japan, P582-586.

f ( x, y ) 0.30( x, y ) 0.59( x, y ) 0.11( x, y ) (1)The G component weight is set to the highest, the lowest
part of B component is the human eye to green sensitivity
is the highest, and the blue sensitivity is the lowest. The
input of the system is a multi-formatted character picture,and the correct characters in the picture are obtained byentering the picture. However, in the network model, theinitial result of the output is the corresponding character
category in the training sample data, and the correspondingtransformation is carried out in the mapping module.
Therefore, the mapping relationship between charactersand categories must be established first in theimplementation of this system, as shown in Fig.5.

Fig.5 Mapping relationships for datasets

The CNNet3 network model is used in the system, thatis, the output node is 1000. You need to establish the first© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan279

Cluster Analysis of Wine based on Three-dim Fluorescent Spectra Characteristic
Di Yin 1, Fengzhi Dai 1*, Yuxuan Zhu 1, Yasheng Yuan 1, Tingting Zhang 1, Ying Chen,
Yiqiao Qin 12, Chengcai Wang 3
1

College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
2 Tianjin Tianke Intelligent and Manufacture Technology CO.,LTD
3 CETC Ocean Information Co., Ltd, Beijing 100041,
China E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

The traditional sensory analysis method can not discriminate different kinds of wine objectively andaccurately. The three-dimensional (3D) fluorescence technology has the advantages of strong selectivity,
high resolution and direct access to the fluorescence characteristics of the sample. In this paper, the 3D
fluorescence spectroscopy five brands of red wine samples. The five characteristic parameters (the meanvalue, standard deviation, center of gravity coordinates, first order center moment and correlation
coefficient) of the 3D fluorescence spectra of these five dry red wine samples are extracted respectively.
The results show that the difference of the samples can be distinguished intuitively by extracting the 3D
spectral parameters of the different brands. On this basis, the clustering analysis shows that even thesame raw material brewing wine is different from the raw material origin and the soil environment. The
climate environment will cause an important impact on the composition of the wine.
Keywords: Characteristic parameters; Clustering analysis; Wine; Three-dimensional fluorescence spectroscopy

1. Introduction

appearing constantly, although which can meet theneeds of wine detection on a certain basis, there are also
some shortcomings such as complex processing,consuming measurement, high cost, sensitivity andinaccuracy. Fluorescence spectrometry is an efficientand accurate analytical method, Because of thecharacteristics of high measurement accuracy, small
sample size and high resolution, it is very convenient tosolve the shortcomings of the traditional methods, suchas many and tedious analysis process.
In this paper 3 kinds of different brands of Wine as
the research object, carries on the analysis using
three-dimensional fluorescence spectra of highsensitivity. Wine contains many complex components.
It is difficult to study certain ingredients. Therefore, thefluorescence spectra of wine under different excitation

Wine has always been popular with people and isfamous for its elegance and noble. In recent years,
because some non-standard productions, wine tracecomponents exceed the standard, so infected andinvaded by some harmful elements that are harmful tohuman body, such as biogenic amine, ochratoxin andother harmful substances. These harmful substances cancause headaches, abnormal heart rate, and may evencause cancer.
Different brands of wine have different tracecomponents. With the development of chemical analysis
techniques, some analytical techniques such as feeling
evaluation, gas chromatography, ultraviolet absorption
spectrometry and infrared spectrometry have been

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan280

Di Yin, Fengzhi Dai, Yuxuan Zhu, Yasheng Yuan, Tingting Zhang,Ying Chen, Yiqiao Qin, Chengcai Wangwavelengths are determined in this paper, and thethree-dimensional fluorescence spectra are constructedaccording to the original data. The three-dimensional
fluorescence spectra of five different wine samples fromthree brands were studied, and three characteristic
parameters were extracted. The clustering method wasused to analyze the data. On this basis, thethree-dimensional fluorescence spectra characteristics ofwine samples were extracted to effectively distinguishdifferent brands of wine.
Fig.1 Great Wall Dry Red Wine

2. Experiment2.1. Experimental device
This experiment uses the LS-55 spectrofluorimeter of
PerkinElmer company of America, and the light sourceis pulsed xenon lamp. The cuvette is a 1cm quartz liquidcell with stopper. This consists of 5 units, namely, lightsource, monochromator, sample cell, detector anddisplay.
2.2. Experimental method

Fig.2 Great Wall Cabernet Dry Red Wine

Because of the fluorescence quenching of five samples
from three different brands, the selected wine samples
need to be diluted before they can be measured in a
cuvette. The dilution of the wine samples in thisexperiment is 20 times. In this paper, the excitation
wavelength is set from 250nm and the interval is 5nm to350nm ,the emission wavelength ranges from 200 to
700 nm, and the scanning speed is 1000 nm/min. Under
these conditions, two-dimensional fluorescence data are
obtained.
2.3. Data processing and analysis
The measured two-dimensional fluorescence spectrum
data are normalized, smooth filtering, synthesized
three-dimensional
fluorescence
spectraandthree-dimensional contour map, and curved surface
fitting and characteristic parameters are made by using
the program of MATLAB (number of fluorescent peaks,
position, peak wavelength of main fluorescence，optimal
excitation wavelength).

Fig.3 Changyu Dry Red Wine

2.4. Three dimensional fluorescence spectra ofwines of different brands
Fig.4 Changyu Dry Red Wine (premium wine)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan281

Cluster analysis of wine

is because the grapes detected in the brand wine are madefrom the same grape variety Cabernet Sauvignon grapes.
Although the location of wave peaks is the same, their optimal
excitation wavelengths are quite different. It is due to thedifferent content of trace fluorescent substances in wines fromdifferent regions. It is mainly distinguished by the location offluorescence peaks. According to the intensity and number offluorescence peaks, the characteristic substances of different
brands of wine can also be distinguished.

The difference of brands can not be seen from theatlas, so it can not be directly distinguished by the map.
Therefore, we need to extract the characteristic
parameters of the map to see the differences, and thenclassify them by cluster analysis.

Fig.5 Arabella Dry Red Wine

Table1 Characteristic parameters of 3D fluorescence
spectra of samples
samplewave

peak

fluorescence

optimum

peak

position

intensity

excitation

number
1

2

3

4

5

3

3

3

3

3

3. Clustering analysis

wavelength
260/376.5

619.6

290/375680.3

325/418.5

637.2

265/372.5

941.3

290/374.5

908.5

330/429567.6

265/372.5

710.5

290/373625.7

335/423483

290/374651.2

325/419.5

783.4

265/375587.2

265/376.5

757.3

290/374.5

772.8

335/418.5

455.1

(/nm)

3.1. The concept of cluster analysis

290“Birds of a feather flock together”. As an importantbranch of statistics, clustering analysis has been widely
applied in various fields. This is mainly based on
distance based clustering analysis. The software used inthis experiment is SPSS 19. To analyze the tracecomponents of different brands of wines and calculate
the similar distance between wines by various methods.
After repeated tests and combined with the data of thissubject, the data measured by Euclidean distanceanalysis can be used to obtain better clustering results.

265

265

325

3.2. Feature data extraction
Table2 Extraction of characteristic parameters of 3D
fluorescence spectra

290

These figures show three-dimensional fluorescence
spectra of five kinds of wines: Great Wall Dry Red Wine,
Great Wall Cabernet Dry Red Wine, Changyu Dry Red
Wine, Changyu Dry Red Wine(premium wine) and Arabella
Dry Red Wine. From the comparison of the spectra of thesamples, there is an obvious fluorescence peak in the range of
260 nm/375 nm, and the band of 260 nm/375 nm has the best
excitation wavelength.

3.3. Clustering results
The purpose of cluster analysis is to cluster different
brands of wines using extraction parameters. And
accurate classification of different brands of wines. Fivekinds of wine data were used in this paper. The distance
between samples was Euclidean distance andhierarchical clustering method was used for clustering.

From table 1, we can see the corresponding
comparison, Changyu Dry Red Wine and Great Wall
Cabernet Dry Red Wine are the best excitation wavelengths of
265nm. However, Great Wall Dry Red Wine and Arabella Dry
Red Wine are the best excitation wavelengths of 290nm. This

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan282

Di Yin, Fengzhi Dai, Yuxuan Zhu, Yasheng Yuan, Tingting Zhang,Ying Chen, Yiqiao Qin, Chengcai Wangwine is different from the raw material origin and thesoil environment. The climate environment will cause
an important impact on the composition of the wine.
Acknowledgement
The research is partly supported by the “13th Five-Year”
comprehensive investment and basic research businessfee items of China (000040192).
Fig.6

References

Dendrogram

1. Masanori Katsuyama,Nobuhito Ohte. Determining thesources of stormflow from the fluorescence properties ofdissolved organic carbon in a forested headwater
catchment[J]. Journal of Hydrology,2002,268(1).
2. Yawei Hou, Huailin Zhao. Handwritten Digit
Recognition Based on Improved BP Neural Network.
The 13th Chinese Intelligent Systems Conference(CISC2017), Oct.14-15, 2017. Mudanjiang, China.
P63-70.
3. Jianlei Yang,Tuo Zhu. Study on discrimination system of
distilled spirits based on three-dim fluorescent spectra
technique. (Jiangnan University, 2009)
4. Yuan Yan, Jin-Gou, Xu,Zhu-Guang Lin,Yi-Bin
Zhao,Le-Tian Wang,Guo-Zhen Chen. Study on totalluminescence spectra. Application of the Monte-Carlo
method to three-dimensional synchronous fluorescence
spectrometry.(Analytica Chimica Acta,1995)
5. Jiaman Wang, Liang Ma, Yuhao Zhang, Tao Jiang.
Investigation of the interaction of deltamethrin (DM) withhuman serum albumin by multi-spectroscopic method.
Journal of Molecular Structure,(2017)
6. Yuan Yan, Jingou Xu, Guozhen Chen, Total integration
method of three-dimensional fluorescence spectrometry(Chinese Science Bullentin,1997)
7. Jian Wang,Renguang Zuo,Jef Caers. Discovering
geochemical patterns by factor-based cluster analysis.
(Journal of Geochemical Exploration,2017)

It can be seen from Figure 6 that the Euclidean
distance between two samples of Changyu is small, andthe difference between them is small. The sample 1,5,2
has a larger Euclidean distance, which indicates that thedifference between samples is large.
4. Conclusion
In this paper, the 3D fluorescence spectroscopy fivebrands of red wine samples. It was found that the 3D
fluorescence spectra of wine brewed with Cabernet
Sauvignon were basically the same, and could not bedirectly distinguished by spectral comparison. The fivecharacteristic parameters (the mean value, standard
deviation, center of gravity coordinates, first ordercenter moment and correlation coefficient) of the 3D
fluorescence spectra of these five dry red wine samples
are extracted respectively. The results show that thedifference of the samples can be distinguishedintuitively by extracting the 3D spectral parameters ofthe different brands. On this basis, the clustering
analysis shows that even the same raw material brewing© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

283

The Control Design of the Planting Device for the Hydroponic Vegetable
Yasheng Yuan1, Fengzhi Dai1*, Zhongyong Ye1, Yuxuan Zhu1, Ying Chen1, Di Yin1, Tingting Zhang1,
Yiqiao Qin12, Yang Li 3
1

College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
2

Tianjin Tianke Intelligent and Manufacture Technology CO.,LTD;
3

CETC Ocean Information Co., Ltd, Beijing 100041, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

The equipment is designed to household indoor small hydroponic vegetable planting control device. The
temperature, humidity, light and carbon dioxide concentration sensors are integrated to the Arduino
UNO R3 microcontroller. It is used to detect various parameter data of planting environment. The WIFI
module can send the real-time environment variables which detected by the sensor to the mobile devices,such as the user's mobile phone. And, users can also change the control range of variables according to
the planted vegetables through the WIFI module.
Keywords: WIFI, remote control and monitoring, hydroponic vegetable, single chip

1. Introduction

Arduino Extended Edition to connect the sensors andcontrollers. In addition, this design adopts the WIFI
module, and the WIFI main module connects to theserver through the MQTT protocol, which can control
the controller and collect the environmental
information3. This design has high practical value insmall hydroponic vegetable planting.

Planting Hydroponic vegetables use the certain nutrient
solution, rather than use the soil. With the expansion of
the city and the reduction of agricultural land1, thetechnology of Hydroponic vegetables is more and more
important.
Compared with traditional farming, hydroponic
cultivation of vegetables can greatly reduce the waste oflabor force in terms of physical strength, and the crop ofhydroponic cultivation can grow rapidly, the matrix andmaterial of hydroponic vegetables can reduce the harmof plant bacteria2. The nutrient solution of hydroponic
vegetables can be recycled, thus reducing environmental
pollution.
In conclusion, the equipment is designed to create asmall hydroponic vegetables control device which can
automatically detect environmental parameters andmake adjustment, which can enrich our life and free ourhands. This design adopts the Arduino UNO R3 and the2. The hardware structure designIntelligent devices are mainly responsible for datacollection and the control of environmental parameters,
so the size is better to be small. Therefore, this design
adopted the Arduino Sensor Shield extension plate toplace the single chip4, the various sensors and a series ofcontrollers.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan284

Yasheng Yuan, Fengzhi Dai, Zhongyong Ye, Yuxuan Zhu, Ying Chen, Di Yin, Tingting Zhang, Yiqiao Qin, Yang Li

2.2. WIFI module
The design adopts the WIFI module of ESP8266, and itcomes with a core MCU. The MCU has the built-in
RAM, ROM, support RTOS, and reaches the maximum
clock speed of 80 MHz. The WIFI module has thefollowing characteristics:
Built-in 10-bit high-precision ADC, with acomplete TCP / IP protocol stack.
Supports Cloud Server Development / Firmware
and SDK for fast on-chip programming.
Support the UART/GPIO data communicationinterface.
Support Smart Link intelligent networking.
The WIFI module is shown in Fig.3.

Fig.1 Arduino Sensor Shield Functional Diagram

2.1. Hydroponic box design
In this device, the hydroponic box is designed by thesoildworks software. The hydroponic box can beplanting six vegetables, the upper cover can be opened
to changing vegetables and replace the nutrient solution,
the fan and light supplement lamp are installed on thewall, and the humidification device is installed into thebox.
the design of hydroponic box is shown in the Fig.2.

Fig.3. WIFI module

2.3. Main control chipIn this design, the external MCU is Arduino UNO R3,
and the Arduino board is based on Atmel's ATMEGA
328 MCU. It can be simply fixed the sensors andvarious electronic components. It has the following
characteristics:
Input voltage: there is no need for external power
supply or external connect USB 5.0V~9.0V voltage
input.
Output voltage: 5.0V voltage output, 3.3v voltage
output and external power supply input.
Support ICSP downloads and TX/RX.
Operating temperature range: - 40 ° C to + 85 ° C.
The design of the main control chip is shown in Fig.4.
Fig.2. hydroponic box design© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

285

The Control Design of

3.3. Design of connection circuit for ambient lightsensor module
The design adopts the ambient light sensor BH1750. It
is a digital ambient light sensor. It is possible to detect
wide range at high resolution. (1 ~ 65535Lux). BH1750
has five pins, one pin is the IIC bus clock pin, one pin is
the IIC bus data pins, one pin is the IIC device addresspin, the other two pins are connected to VCC and GND.
3.4. Design of connection circuit for carbon
dioxide module
Fig.4. ATMEGA 328P chip

The design adopts the carbon dioxide sensor MG811. It
using solid electrolyte battery principle to measurecarbon dioxide, it have simulate output and TTL level
output, and better sensitivity of carbon dioxide, it is also
can better identify the content of carbon dioxide in theair, carbon dioxide gas sensitive element test
concentration in the range of 350 ~ 10000 parts per
million, working conditions for the environment
temperature: - 40 ℃ ~ 70 ℃, humidity: ≤95%RH.

3. System circuit module design
In the circuit design, this design adopts a boost module,
temperature and humidity module, carbon dioxide
module and ambient light sensor module. These
modules greatly improve the function of the device.
3.1. Boost module
The universal input voltage of boost module is 3V ~32V, and the better working voltage range is 5V ~ 32V.
The universal output voltage is 5V ~ 35V. It built-in 4A
high efficiency MOSFET, which can make full use of
conversion efficiency. The high switching frequency is
400KHZ, so it can achieve a better effect5.

4. Introduction of functional module
With the functional modules, we can easily control andmonitor the hydroponic vegetable planting equipment.
4.1. The carbon dioxide regulator design
We adopt the ventilation fan as the carbon dioxide
controller. When the concentrations of carbon dioxide in
the device is too high, the ventilation fan working,which can effectively adjust the concentrations ofcarbon dioxide in this device.

3.2. Design of connection circuit for temperatureand humidity module
The design adopts the temperature and humidity
compound sensor DHT11. It has humidity measurement
elements to measure humidity and NTC temperature
measurement elements to measure temperature, and its
correction factor is stored in OTP memory by program6.
DHT11 has four pins, one pin is the temperature andhumidity data output pin, the other three pins areconnected to VCC, GND and floating7. The DHT11 is atemperature and humidity sensor, and its measurement
accuracy is very high. The accuracy of temperature
measurement is about 2℃, and the accuracy of humidity
measurement is about 5%. Its signal transmission
distance can reach to 20 meters.

4.2. The humidity regulator design
We adopt the atomized tablets to increase the humidity
and supplement the nutrient solution. The atomized
tablet has low price, low driving voltage and highconversion frequency8. It adopts atomized tablet
intermediate spray, 5um ultra-fine spray aperture and740 ultra-dense spray holes. Most importantly, it is easy
to operate.
4.3. The ambient light regulator design
We adopt the LED grow light as the light regulator
design. It can promote photosynthesis and promote
plant growth.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

286

Yasheng Yuan, Fengzhi Dai, Zhongyong Ye, Yuxuan Zhu, Ying Chen, Di Yin, Tingting Zhang, Yiqiao Qin, Yang Li

4.4. The temperature regulator design
The temperature regulation of this equipment is mainly
divided into two parts: the temperature increasing
device and the temperature decreasing device.
The temperature increasing device adopts the ceramicheating slice. Ceramic heating plate is a temperature
heating element, which has the features of simple use,quick temperature rise, more secure, uniform heating,long life and high heating temperature. The temperature
decreasing device is a fan. It has the following
characteristics:
The air flow which near the outlet of the fan ismostly flocculated, which can rapidly cool down
the air flow through the tuyere if there is more heat.
When the fan starts blowing, according to thedirection of the wind, positive pressure can beformed around the air outlet in the closed
environment, which can avoid the external
environment dust entering the device.
4.5. Wireless module design
This design uses a wireless Mesh network for ad hoc
network communication. With the mobile-phone APPNetAssist, the WIFI module of this design can access to
the Internet. We can send different instructions tochange and adjust the growth environment of different
vegetables. It can also query the environment
parameters of the current device in real time.
5. Testing and conclusion
By the test of the hydroponic vegetable planting control
device, the following data are obtained: WIFI module
can through the APP to choose the environment
parameters of plants. The above data shows that thedevice can basically meet the needs of device to control
different kinds of hydroponic vegetables.
By using temperature and humidity module，carbon
dioxide module and ambient light sensor module,
environmental parameters such as temperature,humidity, light and carbon dioxide concentration can be
can be real-time transmitted to the user.

By using controller, the equipment can be controlled
when the equipment detects the environmental
parameters not conform to the setting range.
The tests included the stability and speed of the WIFI
module, the sensitivity of the controller, and theaccuracy of the sensor. The test results show that thedevice can basically meet the user's control andmonitoring.
Acknowledgement
The research is partly supported by the “13th Five-Year”
comprehensive investment and basic research businessfee items of China (000040192).
References
1. Hannes Jochen König, Sandra Uthes, Johannes Schuler, etc.
Regional impact assessment of land use scenarios indeveloping countries using the FoPIA approach: Findings
from five case studies. Journal of Environmental
Management,2013,12.
2. Jane Tavares, Koon-Hui Wang, Cerruti R.R. Hooks. An
evaluation of insectary plants for management of insect pests
in a hydroponic cropping system. Biological Control,2015,9.
3. Huailin Zhao, Shunzhou Wang, Yunxiang Liu, Xuyao Hao,
Design of an Intelligent Housekeeping Robot Based on IOT.
Journal of Information and Communication Engineering.
Vol.2(2), 2016, P114-118.
4. Hanafy M. Omar. Enhancing automatic control learning
through Arduino-based projects[J]. European Journal ofEngineering Education,2018,4.
5. Linlin Tang. Design and implementation of intelligent
monitoring system for vegetable greenhouses (Shandong,
Shandong university, 2010).
6. Di Li, Based on Arduino's smart greenhouse surveillancesystem design, Journal of electronic science and technology.
2017,1: P81-P85
7. Jean-Christophe Cuilliere, Vincent Francois, Remi LacroisLi,
A new approach to automatic and a priori mesh adaptation
around circular holes for finite element analysis,
Computer-Aided Design, 2016.
8. V. Vanhoorne, E. Peeters, B. Van Snick, J.P. Remon, C.
Vervaet. Crystal coating via spray drying to improve powder
tabletability. European Journal of Pharmaceutics andBiopharmaceutics,2014.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan287

An anti-theft system based on the self-checkout
Yuxuan Zhu1, Fengzhi Dai1*, Yasheng Yuan1, Di Yin1, Tingting Zhang1, Ying Chen1, Ce Bian12, Yufan He 3
1
College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
2

Tianjin Tianke Intelligent and Manufacture Technology CO.,LTD;
3

CETC Ocean Information Co., Ltd, Beijing 100041, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

Nowadays, the imperfection of the anti-theft system impedes the development of self-service cash registers. This
paper designs an anti-theft system based on self-service cash registers, which can greatly reduce the theft rate insupermarkets. This anti-theft system adopts Siemens’s LOGO!, which is the core of the controller. it has massive
facilities of anti-theft protection, including radio frequency detection, photoelectric counting, repeated scanningreminder, magnetic detection, video monitoring and anti-theft electronic door. Through the above technologies,
the supermarket that uses the self-service cash registers can avoid to some problems, i.e. cash collection leakage,using low price goods to replace that of high price. Multiple facilities are set to jointly protect the safety of goods,
effectively reduce theft and loss.
Keywords: LOGO!, anti-theft system, repeated scanning reminder, radio frequency detection

1. Introduction

anti-theft system can save time for consumers andreduce manual charges for supermarkets. The
development of relevant apps can provide better
publicity to the supermarkets, which can not only
benefit the supermarkets but also facilitate thecustomers.

During the holidays, there are many customers in the
supermarket. When they finish shopping and decide tocheck out, they need to wait in long lines.
This phenomenon makes low efficiency. Therefore, theself-checkout plays an important role in the supermarket.
But at present, only a small number of self-checkout
systems have been accessed in China. Even though theself-checkout is becoming more and more mature inforeign countries, the manufacturing cost is very
expensive and the security system of self-checkout is
unperfect. Theft often occurs. The imperfection of theanti-theft system hinders the development of theself-checkout.
There is not a self-service cash register with perfectanti-theft function in domestic and foreign supermarkets.
Using a self-service cash register with a sophisticated

2. Function introduction
The article describes an anti-theft system based on the
self-checkout which can well solve the theft behaviors
in supermarket that uses the self-service cash register.
This anti-theft system adopts Siemens’s LOGO!, whichis the core of the controller. Moreover, it has massive
facilities of anti-theft protection, including radio
frequency detection, photoelectric counting, repeatedscanning reminder, magnetic detection, video
monitoring and anti-theft electronic door. Some thefts,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

288

Yuxuan Zhu, Fengzhi Dai, Yasheng Yuan, Di Yin, Tingting Zhang, Ying Chen, Ce Bian, Yufan Hei.e. cash collection leakage, using low price goods toreplace that of high price, can start the alarm equipment.
At the same time， through the thermal infrared bodysensor to connect the LOGO! which can control
conveyor to work or stop. These functions caneffectively save power resources. Moreover, multiple
facilities are set to jointly protect the safety of goods,
effectively reduce theft and loss.
Self-service cash register has three modules: cashiercover, scanning and payment. The anti-theft system iscomposed of electric control cabinet, control equipment,
camera and anti-theft detection door. The working
process of the system is to turn on the switch and press
the button to start work. If the buzzer alarms, thesecurity personnel will verify the situation and shut
down the system to restart. If the alarm lamp alarms, themonitoring room staff will verify the situation and thealarm lamp will be delayed to close.
The design of the anti-theft system based on the
self-checkout is shown in Fig.1.

device is activated, and the security personnel comes tocheck. At the same time, it will send the instructions to
the video monitoring room to make the monitoring
personnel pay attention. If the goods is scanned, thecount is cleared. This method can found the situationthat the customer takes the goods without scanning thecode
3.2. The reminder of rescanning
When the barcode reader in the cash register detects thesame item, it will send the signal to the video
monitoring room, which will attract the attention of thestaffs in the monitoring room, and check whether thecustomer has scanned the barcode of the same item, andwhether there is an undesirable phenomenon of low
price goods replacing the high price.
3.3. radio-frequency technique
The anti-theft system designed in this paper improves
the decoder and combines the decoder with the radio
frequency decoder. When the bar code is swept, thedecoder works automatically to fuse the rf label. The
radio frequency detection door is placed in the channel.
If the radio frequency signal is detected, the alarm
device on the radio frequency detection door will sendout an alarm to alert the security guards near the door.
3.4. Detection of magnetic
The demagnetization plate is added into the conveyor
belt, which can automatically demagnetize the goods onthe conveyor belt. The magnetic detection door is also
installed in the passageway to prevent customers fromcarrying the goods that haven't been paid for.

Fig.1.The anti-theft system based on the self-checkout

3. Function module introduction
This anti-theft system adopts Siemens’s LOGO!, whichis the core of the controller. It has massive facilities ofanti-theft protection, including radio frequency
detection, photoelectric counting, repeated scanningreminder, magnetic detection, video monitoring andanti-theft electronic door.

3.5. Smart conveyor belt
Using thermal infrared human induction technology, it
can detect if there is someone before the cash register,and send the detection signal to the LOGO! to control
the conveyor belt work or stop, effectively saving power
resources.

3.1. Count detection
The LOGO! controls the work of the photoelectric
sensor probe. When the goods is sanned and passed
through the conveyor belt, the photoelectric counteradds. Through the LOGO! determine whether the goods
has been scanned by the photoelectric sensor. If thegoods is not scanned, the counter is added and the alarm

4. Operating principle
There are three parts in the anti - theft system includes
mechanical, monitoring and electronic. The mechanicalpart includes conveyor belt; The monitoring part

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan289

An anti-theft system

includes the camera, monitor and radio frequency
detection door; The electrical component includes
switches. There are five inputs and three outputs. The
input variables include the buttons of start and stop,
infrared detector, code scanner and photoelectric sensor.
Moreover, output variables include conveyor belt switch,buzzer alarm and alarm lamp. Above is the control coreof this security system.
The actual working principle is when the customer
walks into the checkout, the infrared detector detects the
input signal and controls the conveyor belt to open.
After the customer scans the goods, the counter will add
one. When the customer puts the goods on the conveyor
belt and the goods pass the photoelectric sensor, thecounter will add another and reset. If someone steals
goods, that is, goods are not scanned, the buzzer alarm
begins to alarm. When the customer repeatedly scans
the same bar code, the alarm light in the monitoring
room will alarm, reminding the monitoring personnel tomonitor whether there is any behavior of replacing thehigh-priced goods with cheap goods. The scanningdevice is equipped with a rf decoder. Valuable goods
with rf soft label can only pass through the rf detection
door after the scanning soft label is invalid. The
magnetic device is installed under the conveyor belt, sothat the original magnetic goods in the supermarket
must be sent out through the conveyor belt to prevent
theft.

sensorPhotoelectric

Infrared

code

stop

start

Q2

Q3

buzzer alarm

conveyor

At present, the anti-theft system of self-service cash
registers at home and abroad is very imperfect, withmany loopholes, which has caused great losses tosupermarkets and affected the development ofself-service cash registers. The anti-theft systemdeveloped based on the self-service cash registers ofsupermarkets. With the application of new technology
in the field of anti-theft, it can well improve thephenomenon of serious loss, promote the promotion of
cash registers, and at the same time, take an important
step towards the realization of supermarket automation.
To this end, the system has the following highlights.
6.1. The low rate of theft
In order to reduce the theft rate, multiple anti-theft
detection is adopted, such as code sweeper,
photoelectric sensor, radio frequency detection,
degaussing plate, synchronous count, to prevent theleakage of goods; When the same bar code is scanned
repeatedly, the alarm light in the monitoring room will
alarm, reminding the monitoring personnel to monitor
whether to replace the high-priced goods with cheap
ones. The rf decoder is combined with the decoder.
Only when the rf soft label fails when the decoder isscanned, it can pass the rf test door. Installing

Table1 Inputs

I5

Q1
alarm light

6. Conclusion

5.2. I/0 list

I4

buttonFig.2. The control principle of the logo!

Turning on the switch and press the open button. If thesignal is detected by the infrared detector, the conveyor
will turn on. When the goods are scanned, the counter
will be increased by one. When the goods pass thephotoelectric sensor, the counter will be increased by
one again. At that time, the alarm will not alarm and thecounter will reset. When the counter is not added or
reset normally, the alarm system will be activated toalarm the theft.

I3

buttonThe control principle of the logo! is shown in Fig.2.

5.1. Control requirements

I2

scannerTable2 Outputs

5. Technical implementation

I1

detector© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan290

Yuxuan Zhu, Fengzhi Dai, Yasheng Yuan, Di Yin, Tingting Zhang, Ying Chen, Ce Bian, Yufan He

Acknowledgement

demagnetizer at the conveyor belt for anti-theft
detection.

The research is partly supported by the “13th Five-Year”
comprehensive investment and basic research businessfee items of China (000040192).

6.2. Humanized design
The appearance and structure of the self-service cashiersystem is similar to the traditional cashier desk. The
open touch screen panel is installed to display thecommodity information. The external structure of theanti-theft system is mainly composed of conveyor belt,
which will not bring inconvenience to customers but
help them to deliver the goods.

References
1. Meng Liu, Kai Fu, Analysis and design of self-service
checkout terminal in large supermarkets, Industrial
design, 2018, 1: P47 – P48.
2. Yingang Fu, Design and implementation of information
appliance control platform (Shangdong, Shandong
University of Science and Technology, 2007)
3. Sinuo Liu, Analysis and design of self-service shopping
cashier system, Modern industrial economy andinformatization,2017,7: P18- P20.
4. Yaling Tang, Ling Ma, Haitao Song, Analysis and designof supermarket self-service cash register system,2017,4:P128.
5. Xiaona Kong, Design and implementation of self-pay
system based on RFID technology, Computer-Aided
Design, 2016.
6. Shunzhou Wang, Huailin Zhao, Weiren Wang, Huijun Di,
Xueming Shu. Improving Deep Crowd Density
Estimation via Pre-Classification of Density. The 24th
International Conference on Neural Information
Processing (ICONIP2017), Nov.14-18, 2017, Guangzhou,
China. P260-269. (EI)
7. Huailin Zhao, Yangguang Guo, Sugisaka Masanori,
Design of the Multi-Car Collaboration System, the 2017
International Conference on Artificial Life and Robotics(ICAROB 2017), Jan. 19-22, 2017, Miyazaki, Japan,
p186-190.

6.3. Energy conservation and low consumption
The main electric consuming device of the equipment is
the conveyor belt. The system is detected by the infrared
detector, which makes the conveyor belt turn off when
no one uses it, greatly saving electricity.
6.4. Stable operation
Siemens’s LOGO! has high reliability andextreme reliability, with data maintenance function, canensure that when the equipment suddenly power down,
the data is saved safely.
6.5. Convenient equipment installation
Siemens’s LOGO! has almost no wiring requirements,
and it only needs to press the button to write and change
the program, which is highly portable and can adapt todifferent requirements.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

291

Case Study on Communication between Embedded Linux Environment and Microcontroller
Ziheng Gao, Yizhun Peng *, Shuo Wang

College of Electronic Information and Automation,Tianjin University of Science and Technology,
China
Binhai International Advanced Structural Integrity Research Centre

Tianjin University of Scienceand Technology,China
E-mail: * pengyizhun@tust.edu.cn

Abstract

In many embedded development scenarios, we need to combine the real-time microcontroller and non-real-timeembedded Linux environment for collaborative development. In a high real-time environment (such as: vehicles,
production lines), it can complete kinds of features relying on huge Linux ecosystem, such as hot fixable functional
modules, improved network communication, easier OTA firmware updates, more efficient algorithm capabilities,
and more objects. This paper aims to enumerate and explore several methods that can be used to implement similar
duplex communications. For ease of understanding, a single-board computer called Raspberry Pi running Raspbian
Linux and an STM32F103 32-bit ARM microcontroller are used in experiment.
Keywords: communication, embedded, real-time, Raspberry Pi, ARM

using two controllers to process real-time and non-realtime tasks separately is a good solution.
Communication is an inevitable problem between twocontrollers. This article focuses on how to build a complete
and reliable communication solution. We ran the Raspbian
Linux distribution of the single-board computer Raspberry
Pi and the 32-bit ARM microcontroller-STM32F103 as an
example. It is worth mentioning that Raspbian is acustomized version of Debian. It was created toaccommodate the ARM commands of the Raspberry pi.
Essentially, it's a full-featured Linux distribution.

1. Introduction
IoT (Internet of things) has become closer to people’s daily
life, such as smart speakers and remoting control table
lamps. As the working basis of the intelligent equipment
to meet the products’ demands, embedded development ismore complicated. Lots of terminal equipment need toconnect the Internet to run high computational tasks andhigh IO tasks. They will be used to develop deep learningmodels online for predictions.
Excellent native support for huge ecosystems andasynchronous tasks in an embedded Linux environment
can accommodate all of the above. In addition, native
compatibility of different types of scripting languages
optimizes workflow.
However, traditional embedded development must
operate in a real-time environment to meet reliability andother requirements. Obviously, the contradiction between
them cannot be solved in the same environment. Therefore,

2. Choices
The communication protocol is an inevitable part whenit comes to communication. The protocol is also divided
into application layer protocols and underlying protocols
according to the level. Firstly, we will consider the mostcomprehensive multi-layer communication protocol stack,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan292

Ziheng Gao, Yizhun Peng, Shuo Wang

such as the TCP/IP1 stack, which is used widely in Internet,
and the CAN bus, which is commonly used in industrial
control. This article will introduce four communication
protocols.

UART is a two-wire, full-duplex, asynchronous serial
communication bus. There are only two lines, one forsending and one for receiving. The timing requirements forboth parties are relatively strict, and the communication
speed is not very fast. However, it is simple and easy, and
it is suitable for transmitting data between two betterperforming controllers.

2.1 Ethernet protocol
Ethernet is one of the most popular computer LAN
technologies2. The advantage is that the ecosystem andprotocol stack is mature. For example, TCP/IP is wellknown, but its standard topology is the bus topology.

3. CASE

2.2 CAN bus

3.1 Choose plan

The Controller Area Network is a feature-rich automotive
bus standard based on the Broadcast Communication
Mechanism. According to the content of the message,
message Identifier is used to define the priority of content
and messages for delivery. It is a multi-master serial bus
standard for connecting ECUs4. The CAN network
includes multiple ECU nodes which can be input/output
devices, embedded devices that include CAN
interchangers or gateways. It is concluded that CAN bus
communication is generally applicable in large industrial
equipment, but the disadvantages in the underlying twomachine communication are similar to Ethernet.

The use of serial communication UART to implement thetwo-machine communication we describe is a very costeffective opinion. Below, we will explain how to make thissolution practical by designing a simple two-machinecommunication protocol. We will use the abovementioned single-board computer Raspberry Pi and
STM32F103 microcontroller to demonstrate.
3.2 Implementation plan
3.2.1 Configuration
On the Raspberry Pi8, we will use the highly acclaimed
Python to finish the programming. The Raspberry Pi 3
Model B has two sets of UART serial ports, one hardware
serial port for Bluetooth, and one software serial port for
GPIO pins. For performance considerations, we have to
exchange them. Then, we need bind the GPIO pins to thehardware serial port instead of BT. Specific steps are as
follows9:
A. Turn off the onboard Bluetooth feature. The steps are as
described above.
B. Restore the serial port and set it as a universal serial port.
Edit document: /boot/config.txt at the end of the document,
add a statement: dtoverlay=pi3-miniuart-bt .

2.3 I²C bus
Inter-Integrated Circuit is a serial communication bus thatuses a master-slave architecture. For simple two-wirecommunication, serial data line (SDA) and serial clock
line (SCL) are recommended to use when the amount ofdata is not large and data structure is not responsible.
2.4 Serial communication bus: SPI and UART
Serial communication is essentially on the bus and otherdata channels and continuously performs thecommunication process of the above single process. The
corresponding method is parallel communication. This
article only describes two examples of 5SPI and UART.

Edit document:/boot/cmdline.txt,next step replace the2.4.1 Serial Peripheral Interface

contents of the document with the following:

SPI is a high-speed, full-duplex, synchronous, serial
communication bus, working in master-slave mode,
independent transceiver. The SPI bus consists of SCLK
(serial clock), SDI (serial data input) and SDO (serial data
output). CS.SPI is a protocol that allows a master device
to initiate a synchronous communication with serval slave
devices. The data lines of the SPI for input and output are
independent, so it is allowed to complete the input andoutput of data at the same time.

dwc_otg.lpm_enable=0
console=ttyl
root=/dev/mmcblk0p2
rootfstype=ext4
elevator=deadline
fsck.repair=yes rootwait
Above we have completed the basic configuration of theserial port.

2.4.2 Universal Asynchronous Receiver Transmitter© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

293

Case Study on Communication

A. After solving the problem of how to initiate
communication on both sides, it is necessary to design a
set of application layer communication protocol to unify
the standard of multi-party hardware communication. In
order to make the communication content can berecognized bilaterally. Also, easy to increase or decrease
the modification function. This article thinks that the
protocol standard needs to meet the following.
The unit of communication content is a message
frame, including the frame header, the message body,and the end of the frame, all of which are ASCII
characters.
The message body includes a variety of data, each ofwhich has its corresponding flag, data body, separator
between the data and the data body, and a separator
between the data.
Each type of data is distinguished by a identifier, andthe meaning and data type of its representative arenegotiated in advance.
The final assembled message frame can be divided,
recognized, converted, and processed in the form ofa string or an array of characters as both ends.
B. As an example, this article will introduce a set of
application layer communication protocols designed in a
vehicle project.

3.2.2 Programming
A. Reference the pyserial6 library for serial
programming. It has been shown in the figure1.
PySerial encapsulates the serial communication moduleand supports different platforms such as Linux, Windows
and BSD. Also, it is a python support module. This module
encapsulates the access rights of the serial port. The
module named “Stand” will automatically select theappropriate backend.
B. On the STM327, the communication part code is as
figure 2.

Fig1. pyserial library for serial programming

Basic definition of interface:a. Serial communication using the UART protocol, thebaud rate is 115200.
b. In the form of a packet, the header is defined as*(ASCII code 0x2A) and the trailer is defined as # (ASCII
code ox23)
The data in data packet must have an identifier,a)Which is any uppercase letter in ASCII, followed by
the data body.
b) The data identifier is separated from the data body by‘:’ (ASCII code 0x3A)c) Multiple data is separated by ‘,’ (ASCII code 0x2C)
in the same package.

Figure 2: the communication part code

3.2.3 Design© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan294

Ziheng Gao, Yizhun Peng, Shuo Wangtransport layer protocol to design the application layer
protocol.

Figure 3: The custom identifier

Data return confirmation and synchronization: In order tomake the communication reliable, two ends can send thepacket back to the secondary station after receiving thedata that needs to be consistent at both ends. Besides, aim
to synchronize the upper and lower data, the logical state
can still obtain the underlying state after the restart, and thebottom layer should send all the latest data to the logicalend the synchronization data frame flag ‘Z’ should be
added to the header of the post back message. The certain
example is as figure 4.

5. Acknowledgement
This research was partially supported by Student’s
Platform for Innovation and Entrepreneur Training
Program, the Ministry of education of China（201810057019）. We are also very grateful to Yang
Jiasheng for his help in the revision of the manuscript.
REFERENCE:
1. LiMing. Research and development of embedded TCP/IP
protocol stack. Computer engineering and Application, 2002,(8): 18-19
2. IEEE 802.3-2015 Ethernet protocol official document
3. CJ1W Defined CAN Unit Operational Manual
4. Wang Weixin. Principle and interface technology of single
chip microcomputer: China Agriculture Press,2013.08
5. Song Qinhua. Shipboard Electronic Countermeasure.

Comparison of online programming mechanism
between SPI and UART, 2015, (10): 66-67
6. pySerial
3.0
documentation.
https://pythonhosted.org/pyserial/.
7. Wang Wei. Design of communication system without
host computer based on STM32F103. Tianjin
Polytechnic University,2017.
8. Raspberry
official
technical
documents.
https://www.raspberrypi.org/blog/.
9. THERASPBERRYPI
UARTS.
https://www.raspberrypi.org/documentation/configurat
ion/uart.md.

Figure 4.Certain example of programming

4. Conclusion
The above is an implementation scheme that we proposedand completed. We use the UART serial port as the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

295

Design and implementation of a baby care robot
Yizhun Peng, Zefan Ge *, Kaixuan Geng, Zhou Yang, Rui Yang

College of Electronic Information and Automation, Tianjin University of Science & Technology, China
Binhai International Advanced Structural Integrity Research Centre，Tianjin University of Scienceand Technology,China* gezefan@mail.tust.edu.cn
www.tust.edu.cn

Abstract

Aiming at the shortcoming of single function of traditional pram, this paper proposes an embedded
multi-functional intelligent baby care robot. With baby cradle as the carrier, the bionic cradle is realized
by combining embedded controller, drive module, motor and various sensors. The babysitting robot has
the functions of autonomous following and obstacle avoidance, voice play and real-time picture return,
It not only enriches the function of pram, but also improves the convenience and safety of pram, and can
assist the parents to take care of the baby, which greatly reduces the parents' burden of parenting.
Key words: robot, autonomous follow, obstacle avoidance, Voice play interface, real-time picture transmission

1. Introduction

and so on to accompany them in early childhood
education2. Car-mounted cameras can send pictures ofbabies back to the computer or mobile terminal in realtime, so that parents can see the situation of babies with
their mobile phones whenever they are cooking or busy.
At the same time, the baby carriage also has the functionof autonomous follow and obstacle avoidance. It not only
improves the convenience and safety of the baby carriage,
but also helps parents to take care of the baby, and greatly
reduces the burden of parenting3.
This product breaks through the capability range of theexisting baby carriage with artificial intelligence
technology, so that parents can take care of the baby
without affecting their work. It also enables parents tohave more independent space and free time, so that
intelligent childcare can lead the trend of the times 4.

Now China has become the largest child carrier producer
in the world 1. But with the rapid development of scienceand technology, the traditional baby carriage with single
function can not meet the needs of parents very well. In
order to relieve the pressure of parents to take care ofchildren, an embedded intelligent child-care system isdesigned and embedded in the baby carriage, whichmakes the existing ordinary baby carriage become an
intelligent baby-care robot. It can help parents to takecare of children, Parents spend less time in taking care ofbabies and making babies grow healthier and healthier.
For example, when the baby is awake and crying, thetraditional baby carriage must have someone waiting
beside the baby carriage to cradle and coax the baby tosleep at all times, so it can not do other things. The
simulation automatic cradle realized by the motor controlof this product can automatically complete the workwithout delaying the work of parents; When the baby
wakes up, the baby needs to be accompanied, cars canplay parents' recordings,Tang poems, children's songs

2.

The overall scheme design

The modular embedded intelligent child-rearing system isbased on Arduino single-chip computer, using baby
cradle car as the carrier, combined with drive module,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

296

Yizhun Peng, Zefan Ge, Kaixuan Geng, Zhou Yang, Rui Yangmotor and a variety of sensors. As shown in Fig. 1 is ahigh-definition camera, which is used to monitor the
baby's condition and upload the video to the terminal; 2,3, 4, 5 and 6 are photoelectric sensors, in which 2, 3, 4
and 5 are placed in the upper, middle and lower layers infront of the baby carriage respectively to avoid obstacles;
6 are placed above the rear of the baby carriage forindependent follow-up of the baby carriage; 7 aremotor-driven. Module, drive motor to provide power forbaby carriage. The sensor information transmission signal
is analyzed to drive the circuit to control the motor
movement.

microcontroller with ATmgea328p kernel as thecontroller of the robot.
The Arduino ecosystem includes a variety ofdevelopment boards, modules and expansion panels.
Among them, UNO R3 is the most suitable entry and themost functional Arduino development board. It is a
microcontroller board based on ATmega328P. It has 14
digital input/output pins (6 of which can be used as PWM
output), 6 analog inputs, 16MHz crystal clock, USB
connection, power jack, ICSP connector and reset button.
It only need to connect computer through USB data line
to power supply, program download and data
communication.The main control chip is shown in the Fig.
2.

Fig.1. overall layout of the baby care robot(1. Hd camera, 2-6. photoelectric sensor, 7. motor
drive module)

Fig.2. The Arduino UNO R3 real figure

2.1. Main control chip

2.2. The Photoelectric sensor

ATmgea328p MCU of Atmel Company has theadvantages of high performance, high cost performance,rich peripherals and low power consumption5.The
microcontroller based on Atmel 8-bit AVR RISC
combines 32 kB ISP flash memory and read-write
capability, 1 kB EEPROM, 2 kB SRAM, 23 general I/O
lines, 32 general working registers, 3 flexible
timers/counter of compare mode, internal and externalinterrupts, serial programmable USART, byte-oriented
2-wire serial interface, SPI serial port, 6-channel 10-bit
A/D converter (8 channels encapsulated by TQFP and
QFN/MLF), Programmable Watchdog Timer withinternal oscillator, and five software alternative
power-saving modes. The device operates between
1.8-5.5 volts. The throughput of the device is close to 1
MIPS per MIPS. This design uses the Arduino serial

This design uses photoelectric sensor to realize automatic
following and obstacle avoidance. The sensor model is
E18-D80NK, as shown in Fig.3.

Fig.3. The photoelectric sensor physical map

E18-D80NK infrared photoelectric switch, also known
as diffuse reflection obstacle avoidance sensor module© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

297

Design and implementation ofproximity switch, is a photoelectric sensor which
integrates transmission and reception. The emitted light ismodulated and sent out, and the receiver demodulates thereflected light. The interference of visible light is avoided
effectively. The use of lenses also makes it possible forthe sensor to detect up to 80 cm distances (due to thecharacteristics of infrared light, different colors of objects
can detect different maximum distances; white objects
are farthest, black objects are nearest). The sensor has
many advantages, such as long detection distance, lessvisible light interference, low price, easy assembly andeasy use. It can be widely used in many occasions such as
obstacle avoidance of robots, pipeline parts and so on. Its
working voltage is 5V DC power supply, driving current
is 100mA, working environment temperature is - 25 C 55 C, standard detection object illumination requirements
are below 1000LX, incandescent lamp 3000LX, effective
detection distance for opaque objects is 3 - 80cm,the
distance of the obstacle can be detected by adjusting theadjustable potentiometer at the end of the sensor.The
working principle of photoelectric sensor is to control by
converting the change of light intensity into the change ofelectric signal. When the sensor detects obstacles in front,
the red indicator lights up at the rear of the sensor, andthe return value of the single chip computer is 0, which
outputs low level. When there are no obstacles in front of
the sensor, the single chip computer outputs high level 1.
The single chip computer determines whether there are
obstacles in front of the sensor by collecting the return
value of the sensor. The sensor has three lines: brown,
blue and black. The Brown connects to 5V pin of MCU,
the blue connects to the ground pin of MCU, and theblack connects to the digital pin of MCU. The return
value of the sensor can be read out by the serial portmonitor of MCU to judge whether there is an obstacle infront of it.

acquisition system. Using DTAS (direct analog storage)
technology, the recorded and played information can bedirectly recorded in EEPROM inside the chip, so that theexisting analog data can be well preserved. Effective
components, reduce the distortion of sound quality,
improve the quality of recording and playing, and get thenatural and realistic sound reduction effect. The working
principle of this module is to decode the read signal
through the audio decoding circuit to get the digital signal.
Then the digital signal is converted into analog signal by
the digital-to-analog converter. The analog audio is
amplified by the audio power amplifier circuit. Finally,
the low-pass filter is sent to the headphone output port.

2.3. The recording and playback module

2.5.Motor control system

The nested recording and playback module in the baby
carriage plays the role of early education. This module
can not only record parents'voices and play them toinfants, but also play children's songs, Tang poems, fairy
tales and so on. It can accompany infants all the time andteach them early6.
This module is composed of speech recording circuit,audio power amplifier circuit and audio decoding circuit.
Language recording circuit is essentially an analog data

The system consists of Arduino main control board, drive
circuit, motor and coder. The system block diagram isshown in Fig. 4.

2.4. The video monitoring module
The video monitoring module is composed ofhigh-definition camera, ESP8266 control board andmobile phone (computer) APP7. The camera is inside thecradle car to capture the baby's status in real time andtransmit the image to the connected control board. Finally,
we can see the video taken by the camera through themobile phone (computer) terminal to understand the
baby's status. . This module has two working modes:
when the parents are close to the baby stroller, forexample, when the baby sleeps in the bedroom in thesame house and the parents cook in the kitchen, we cancreate hot spots through the WIFI module of ESP8266
main control board and transmit picture data under thesame LAN. When parents are far away from the baby
stroller, for example, when parents want to see the baby's
condition at work, they can also view it through theterminal, but they need a cloud server. First, the masterboard uploads the data collected by the camera to thecloud server, and then the parents download the videofrom the cloud to watch locally using the mobile
terminal8.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

298

Yizhun Peng, Zefan Ge, Kaixuan Geng, Zhou Yang, Rui Yang

Plan 1:use
threaded fittingto link
eachcomponent.
Plan 2:usebucklestructure to linkallparts.

Fig.4. The motor control system

The main control board transmits one-wave signal tothe motor driving circuit, and the motor driving circuitchanges the voltage at both ends of the motor according
to the square-wave signal, so as to control the motor
speed. Then the motor speed is monitored in real time by
the encoder and the pulse count is transmitted back to themain control to realize the closed-loop feedback
system,and then the PID algorithm is used to adjust the
duty ratio to achieve the set speed of the motor.

Plan 3:use thepin link
schemeTable 1 structure plan and analysis
The material is easy to obtain,Adv
convenient to use, simple inanta
structure, reliable in link andges
convenient in assembly anddisassembly.
Disa
The joint is easy to cause carelessdvant scratches, low pressure, and easy toages
rot and leak at the thread.
The assembly is simple, fast, saving
Advatime, saving material cost, and thentage
finished product is not easy to besdamaged artificially.
Disa
Customerdisassembly
time
dvant
consuming.
ages
Adva
Links are simple and easy tontage
replace.
s
Disa
The reliability of the link is notdvanthigh enough.
ages

In order to achieve the purpose of good strength andlight weight in the selection of mechanical structural
materials, we analyzed several schemes for the selectionof materials, and finally selected the first one, as shownin Table 2.

2.6 Data acquisition and processing
In this module, ADC is used to collect sensor data, andthe data collected directly from the sensor is not accurate
because it is susceptible to external interference.
Therefore, according to different types of sensor data,mean filtering, weighted recursive average filtering,
median filtering, Kalman filtering, dithering filtering andfirst-order lag filtering are carried out to improve the data
accuracy.

Plan1:
use 3D to
printmaterials.
Plan
2:
use
of
aluminum metalmaterials
Plan
3:
use
of
carbonfiber
materials

3. Mechanical structure design
We use embedded mechanical structure to combine
intelligent modules with baby carriage, which makes the
traditional baby carriage multi-functional and intelligent,and optimizes the mechanical structure of baby carriage
on the original basis.

Table 2 material selection plan and analysis
Strong development and short R
Advantages& D cycle.
Disadvant The production cycle is long and theages
R & D cost is high.
Advantages
Disadvantages
Advantages
Disadvantages

The technology is mature, and thecost of mass production is low.
The metal itself is heavier.
The quality is light and the intensity
is good.
The price is high.

3.2. Structural design of each module
In view of the monitoring, real-time and reasonablestructure of some sensors located in the baby carriage, wehave set up a special slot for the sensor, and use theembedded installation method to ensure the ease of use
and convenience of the sensor above. Because the sensors
in the lower part need to take into account thecharacteristics of safety and intelligence, a surroundmechanical structure is specially designed to ensure that

3.1. Interactive part embedded structure
In the mechanical part of human-computer interaction, inorder to achieve the purpose of easy use and simplicity,
we analyzed a number of solutions for structural design,and finally chose the first one, as shown in Table 1.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

299

Design and implementation of

they have the function of safe escort while having an
intelligent experience.
In order to make the display part run stably and copewith the unfavorable weather environment in general, aset of protective structure with waterproof and preventing
accidental impact damage is designed for the display andits surrounding components. On the premise of satisfying
the function, the control part is specially designed tomake it more ergonomic and bring users a better
experience.

N-F-0321
Gear
motor

5.1
6M*
m

146g

AB42
Planetary
reducer

10N
*m

400g

HB860H
stepper
motor

8.5
N*m

400g

Under the condition of synthetically analyzing the load
and torque that the stator rotor and shaft of the motor
need to bear and reducing the cost as far as possible, theN-F-0321 deceleration motor is chosen as the powersource of the traction part of the baby carriage9.
The motor operation scheme generally includes gear
drive, belt drive and direct traction drive. Through theanalysis of table 4, we finally choose the motor direct
traction mode of scheme 3.

3.3. Motor performance analysis and drag structure
designIn order to ensure the mechanical properties andstructural stability of the traction part of the motor,
ANSYS is used to carry out mechanical analysis of themechanism. After analyzing various schemes, the direct
traction structure with the best performance is selected.
The traction mechanism diagram is shown in Fig. 5.

Plan

Table 4 motor drive scheme and analysis
The advantages and disadvantages

Plan 1:use
thegear drive
scheme

advan
tages

disad
vanta
ges

Fig.5. Motor drive mechanism diagram

Then, the performance of the motor was analyzed, andmultiple types of motors were selected for comprehensive
comparison analysis to test the motor's mechanicalperformance. The analysis results are shown in table 3.

Name
42BYGH
40-1704A
steppermotorLight weight, high torque,
good for dragging.
External encoder control isrequired.
Light weight, high torque,
good for dragging.
External encoder control isrequired.
Large torque can travel oncomplex ground.
The price is high and thenoise is high.

Plan 2: use
belt drive
schemeTable 3 motor performance analysis
Advantages andtorq Motordisadvantagesueweight
The amount of movement
is accurate, the speed is
0.4
uniform, the structure is
8N* 360g stable, and the noise islow.
mTorque is small and drag isdifficult.

advan
tages

disad
vanta
ges

Gear transmission is stable,transmission ratio is accurate,
reliable, high efficiency, longservice life, using a wide range ofpower, speed and size.
Heavy weight, noise, transmission
distance is small, often needlubrication, and most of the baby
carriages on the market can notdirectly match the gears.
The belt has good elasticity, it canrelieve impact and vibration inwork, smooth movement and nonoise. When the load is too large,
the belt slips on the wheel, so it can
prevent other parts from being
damaged and play a role of safety
protection. The belt is anintermediate part. It can select thelength according to the need in acertain range to adapt to the larger
working conditions of the center
distance requirement. The structure
is simple, easy to manufacture, easy
to install and maintain, and low in
cost.
Transmission by friction can nottransmit high power, and cause
sliding loss and delay loss easily.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan300

Yizhun Peng, Zefan Ge, Kaixuan Geng, Zhou Yang, Rui Yang

Plan
3:
scheme ofdirecttractionwith motor

advan
tages
disad
vanta
ges

Simple and easy to use, flexible
structure, can be applied to a varietyof complex environments.

intelligent system of baby care robot meets the needs ofinfant growth.

The traction tire of motor is easy toslide on the smooth road whenrotating.

4.1. Bionic automatic cradle function
When a baby is awake and crying, parents usually use
rocking cradle to coax the baby to sleep or stop crying,
but this requires parents to stay beside the baby carriage
and not be able to do their own thing, and sometimes
distracted by urgent things. We designed a bionicautomatic cradle for this situation, the model. The block
adopts fully automatic technology, which can
automatically swing with the most suitable frequency andamplitude, simulate the scene of parents shaking the baby
to sleep, help the baby sleep better and more comfortably,and parents can also take time to do other things.

4. function introduction
Modular embedded intelligent child care system iscomposed of two main functions: safety monitoring andself motion. The product is nested on a baby carriage to
be a baby care robot. The robot's physical diagram isshown in Fig. 6 below.

4.2. Selffunctionsfollowingandobstacle

avoidance

The photoelectric sensor is used to realize the function of
independent following and obstacle avoidance of thebaby carriage. The photoelectric sensor is installed at therear of the baby carriage. The algorithm is used to realize
the function of automatic following. When the rear sensor
detects the parents, the baby carriage begins to move
forward. When the left sensor detects the parents, thebaby carriage turns right differently, and when the rear isright. When the side sensor detects the parent, the baby
carriage turns left. In the process of movement, if thefront sensor of the baby carriage detects obstacles, thebaby carriage immediately stops moving. The
photoelectric sensor in front of the baby carriage isdivided into three layers: upper, middle and lower, which
can detect obstacles of different heights. This function
can very well assist parents to take baby strollers out,
especially for the elderly to take children out, which canprovide great convenience and safety.

Fig. 6. The robot physical map

Baby care robot is a multi-functional baby carriage,
which can not only take care of the baby carefully, butalso ensure the safety of the baby efficiently. It can
provide a safe and comfortable environment for infants.
The system consists of several modules, such as bionicautomatic cradle module, automatic tracking module for
infant parents, intelligent autonomous obstacle avoidance
module and early education module, which caneffectively, comprehensively and real-time protect infants
and provide them with a comfortable environment.
Environmental Science. Its mechanical aspect uses a new
type of mechanical structure, through the motor rotation
to provide power for the baby carriage, can effectively
reduce the pressure of the cart. Baby care robot, the firstcar in the baby's life, gentle, considerate, intelligent andomnipotent, all-round care of the baby's safety. The

4.3. The function of early childhood education
An early education module is embedded in the baby
carriage, which includes recording and playback, poems,
children's songs, fairy tales and other functions. When thebaby is awake, we can not only play the audio recorded inadvance by parents to accompany the child, but also playchildren's songs, poems, fairy tales and so on to enlighten
the infant's early education. At the same time, the baby is
not alone when it is alone.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

301

Design and implementation of

4.4. The real time video return function

References

The baby carriage is equipped with a high-definition
camera facing the baby. We can view the video capturedby the camera in real time from the terminal of themobile phone or computer, and we can see the baby's
condition anytime and anywhere. Whether you arecooking in the kitchen, going out to buy food, or missing
your baby, you can always pick up your mobile phone tocheck the baby's condition. This function has brought
great convenience to parents.

1. Wenjie Wang. Design and research of a new type of baby
carriage. Tianjin University of Science and Technology,
2017.
2. Yanhui Chen, Xiaoqi Li. Design of an intelligent baby
carriage. science and technology innovation guide, 2014,11(27): 53-54.
3. Zhihan Li, Yicheng Shi, Ying Wang. Design and Researchof Multifunctional Intelligent Baby Car. Digital
Technology and Applications, 2018,36 (03): 174-175.
4. Saishai Li, Yue Qiu,Bowen Sun . Research on Intelligent
Baby Car Design Based on Emotional Design Theory.
Industrial Design, 2018 (08): 30-31.
5. Mengting Liu, Fan Wu, Zhiqin Chen. Design of aerial
reconnaissance six-axis rotary-aircraft based on ATmega328p and Android.Electronic world,2016(18):147-148.
6. Qiuming Wang, Chaoyang Zhu, Qiuyu Xu, Yuxuan Sun.
Intelligent baby carriage design. Information and computer(theoretical version), 2017 (21): 110-111.
7. Binqi Liu, Haitao Liu. Design of Intelligent and Safe Baby
Car for Data Communication Based on STM32. Scienceand Technology and Innovation, 2018 (16): 62-63+67.
8. Qijun Zou, Sunjia Wei, Shiji Yu, Huijin Chen. An
intelligent design of baby carriage. Popular Science andTechnology, 2015, 17 (06): 5-6+12.
9. Sen Li, Yicai Tian, Ming Tan, Fanghui Gao, Wanzhao
Zhao. Smart Following Baby Cart Design Based on
STM32. China Science and Technology Information, 2018(09): 81-82+12.

5. summary
With the change of time, technological means are also
developing rapidly. More and more behaviors and work
can be replaced by robots, which is an irresistible trend.
More detailed escort robots will be developed. With thedevelopment of artificial intelligence, robots are
becoming more and more intelligent, and they can domore and more things. Service-oriented robots will be
full of people in all aspects. By that time, care robots
need to be more capable and more humane.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan302

Research on Intelligent Shopping Service Robot Design
Shuo Wang1,2, Yizhun Peng *1,2, Ruixiang Bai1, Ziheng Gao1,2

1.College of Electronic Information and Automation,Tianjin University of Science and Technology, China
2.Binhai International Advanced Structural Integrity Research Centre
Technology,China

Tianjin University of Science and* pengyizhun@tust.edu.cn

Abstract

With the rapid development of the social economy, it is good to help customers have more efficient andrelaxed shopping experience through intelligent means in large commercial centers. In this paper, a
shopping service robot project that uses machine vision, deep learning, and IoT is introduced. The robot
can help customers complete shopping and complete indoor positioning navigation and autonomous
movement intelligently. The robot product is designed to help all customers, especially people withmobility impairments. It will reduce their burden of action, thereby enhancing the user experience anddoing special analysis of shopping behavior to help launch more efficient and personalized market
planning.
Keyword: feature recognition, Artificial neural networks, indoor positioning navigation, machine vision(3) Artificial neural networks (ANN) are used to train
the contours of customers. The color block recognition
feature in machine vision is used to reduce the weight of
the training model to improve the accuracy of object
tracking.
The above scheme can realize automatic tracking ofcustomers, indoor positioning navigation and autonomous
movement. Robots can combine recommendations andsystemstocreateclear,
personalized
route
recommendations to reduce barriers to customers'
shopping experience.

1. Introduction
With the in-depth development of artificial intelligence,the current research on smart shopping carts (robots) isalso deepening. There are some innovative designs such asa large shopping cart based on traditional ergonomics, asmart shopping cart system that can automatically follow
up, a smart shopping cart based on RFID technology, etc13.
This paper aim to design a shopping service robot to
optimize shopping and entertainment experience. The
main designing features are as follows.
(1) The robot is designed to result in implementing
object follow with the use of machine vision to identify
feature description.
(2) The inertial navigation system module uses the samefeature color block identification method. The INS(Inertial Navigation System) module is calibrated to
improve indoor positioning accuracy under known indoor
map conditions.

2. Design
2.1. Feature recognition module
2.1.1 Color feature
Color feature is a significant and intuitive underlying
feature of the image with RST (rotation, scale andtranslation) invariance. This article designs a method forcolor feature resolution and image feature extraction. The© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan303

Shuo Wang, Yizhun Peng, Ziheng Gao

coordinate system7, as shown in Figure 2. Figure 3 is aflow chart showing the speed and position of the robot
body in the navigation coordinate system by solving thequaternion attitude.

specific method is to use a small unit method combining
different characteristic color values to form a characteristiccolor block. The color feature4 mainly uses the HSV space.
The HSV is the representation method of the point of the
RGB color space in the inverted cone. The method canachieve simple, fast linear transformation and better
clustering5 effect.

Fig 2. Navigation coordinate system

Fig 1. HSV space

2.1.2 Image color feature extraction
This design uses feature color blocks as visual recognition
objects to achieve robotic automatic tracking accuracy
improvement and correct indoor positioning navigation. In
addition, the convolutional neural network is used to
identify and optimize the recognition object inidentification of the feature color block. thereby it
improved the recognition speed and accuracy. We design
to use the feature extraction of the neural network to
identify the feature values of the color blocks and thenobtain the image information according to the arrangement
and combination of the color feature values in the image6.
In this section, image color recognition is used to
identify and analyze preset different color feature blocks.
HSV color space and color image advantages are used to
improve the accuracy of recognition resolution. CNN
network training is used to identify recognition models
under different actual conditions to improve recognition
efficiency.

Fig 3. Navigation coordinate system by solving the quaternion

2.2.1 Error analysis
Since the accelerometer and gyroscope in the mems
inertial device are essentially sensors, there is a large error
in the actual use of the device8. The calibration principleof accelerometer and gyroscope is as follows.
1) Accelerometer calibration principle
a’ = a * scale + bias

2.2. Indoor position navigation module(1)(a’ is the measured value of the accelerometer, a is the ideal
value, scale is the accelerometer scale factor, and bias is
the zero-point decimation)2) Gyro calibration principle

The design uses the combination of inertial navigation
system and machine vision to improve the accuracy ofindoor positioning on inertial navigation. The basic
principle of inertial navigation is to read the linear
acceleration and angular velocity by accelerometer andgyroscope respectively and establish the navigation

gyro’ = gyro * scale + bias

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

304

(2)

Research on Intelligent Shopping

2.2.2 Actual error correction scheme

characteristics in eight direction.
b.

Actually, the robot cannot automatically recalibrate atregular intervals. As time pasting, temperature drift will
become more and more obvious. These factors will
increase the positioning error9. So, we set up I²C
communication between MPU6050 and Raspberry Pi, andthen configure the I²C address of Raspberry Pi and
MPU6050. We can read and write the MPU6050 registerby reading and writing the MPU6050 register. We also use
the quaternion algorithm to convert the acceleration andangular velocity values of the three axes into Euler angles
and establish the inertial navigation coordinate system toobtain the speed and position of the robot body .

The RGB and Ga or features of the image are used as
input values, and the contour of the human body is
calculated using the expectation maximization
algorithm to obtain a human body mask.

c.

Perform morphological processing such as noise
removal and open- loop calculation on the output ofstep b to obtain a more accurate human mask.

d.

The mask generated in step c is phase-operated withthe original image to extract the contour features of
the customer image.

2.3.2 Use CNN for identification training
By consulting a large number of references10,11,12,13, weknow that methods such as learning classifiers, using alarge number of feature blocks and probability
enhancement trees and using sparse coding gradientfeatures to detect and identify contours have been used.
We use Convolutional Neural Networks (CNN) to identify
the contours of the characters that have been extracted andthe color block features. Firstly, we design to construct a
Fig 4. Hardware connection diagram

14four-layer neural network, where the size of the input

We use the CNN image DHF feature extraction algorithm
in machine vision to identify the preset feature patches. We
obtain the image feature informationaccording to thecorresponding unique coordinate information. We store itin the memory of the Raspberry pi and replace thecoordinate information in the mpu6050 with it. Thisupdates the coordinates of the robot, improving theaccuracy of navigation.

layer is 45 × 45 × 3, and the size of the four-layer
convolutional layer is 45 × 45 × 32, 22 × 22 × 48, 11 ×
11×16 and 5×5×128. The first fully connected layer size is
128 and the last layer size is 101. Among them, fourconvolutional layers and two fully connected layers areused for parameter learning and the remaining layers are
randomly initialized15. Its structure is shown in Figure 5.

2.3. Human contour recognition moduleThis paper mainly adopts the way of extracting humanbody contour from sequence image as the main recognition
model of robot recognition customer and add color blockfeature to improve the recognition accuracy. The dualfeature recognition training mode has certain robustness.
2.3.1 Extraction of human contour features
This article uses Gabor feature information to extract thecharacter(customer)contour in the image. The steps are as
follows:
a. The input tested picture calculates the Gabor wavelet

Fig 5: CNN structure diagram© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan305

Shuo Wang, Yizhun Peng, Ziheng Gao

2.3.3 Experimental effect training

References

We use the specific person pictures collected in the
supermarket scene as the data set to extract the contour
features and color block recognition features of thecustomers in the image. There are 500 images in total, of
which 200 are for training, 200 for testing and 100 for
inspection. In order to evaluate the classification results, acorrect recognition rate (PCR) is introduced as anevaluation index, which is defined as follows

1. JiangChao. The principle and algorithm of automatic
following system for Internet supermarket shopping cart.
Electronic technology and software engineering, 2018, (11):1-2
2. WangChongning. Analysis of innovative design of large
supermarket shopping cart based on Ergonomics. Industrial
Design, 2016, (1): 33-35
3. LiYun. Research and design of intelligent shopping cart basedon RFID Technology. Internet of things technology, 2014, (7):18-19
4. PanYongsheng. A new multi feature fusion method for image
retrieval. Science and technology and Engineering, 2014,14(15):219-223
5. ZhengQiang. A new color feature extraction method in image
retrieval. National Conference on new computer technology
and computer education. 2009.
6. LiQin. Image depth level feature extraction algorithm. Patternrecognition and Artificial intelligence, 2017, 30(2): 127-136.
7. HeChao. Error compensation of MEMS sensor for Strapdown
Inertial Navigation System. Harbin Institute of Technology.
Master's degree thesis, 2010.
8. YuRongnan. Design of two order complementary filter
attitude solver. Journal of zhejiang university oftechnology,2017,45(4):417-420.
9. FengXinyu. Design of four rotor UAV. Beijing: Tsinghua
University Press, 2017.
10. SunDong. Recognition of human contour features based on
support vector machine. Scientific and technologicalinformation, 2009, (8): 53-55
11. D.R.Martin,C.Fowlkes, D.Tal,and J.Malik.A database ofhuman segmented natural images and its application toevaluating segmentation algorithms and measuring ecologicalstatistics.In Proc.ICCV,pages 416-425,2001.
12. P.Dollar,Z.Tu,and S.Belongie.Supervised learning ofedges and object boundaries. In Proc.CVPR,volume 2,pages
1964-1971,2006.
13. Z.Tu.Probabilistic boosting-tree: Learning discriminativemodels for classification, recognition, and clustering. In Proc.
ICCV, volume 2, pages 1589-1596. IEEE,2005.
14. Alex Krizhevsky. ImageNet Classification with Deep
Convolutional Neural Networks
15. X.Ren and L.Bo.Discriminatively trained sparse code
gradientsfor contour detection. In Proc. NIPS.

=

× 100%(3)

T is the number of correctly identified samples and N is
the total number of test samples.
If the CNN only recognizes the contour features of thecustomer, the average correct recognition rate is 80% after
multiple training verifications. If the recognition of thecolor patch feature is added, the correct recognition rate of
CNN can reach 90%, and the training effect of the model
is significantly improved. Moreover, the recognition
effect of CNN is more ideal than the single feature
recognition method in the environment of uneven
illumination, which shows that the dual feature recognition
enhances the robustness of CNN.
3. Conclusion
In a separate module test, the basic expected effect can beachieved in an ideal environment. Due to the large amountof computation, it is impossible to put all the functions on
the Raspberry Pi processor. So, we plan to use cloudservices to solve the problem and move some of theoperations to the cloud.
Acknowledgement
The study was partially supported by Student’s Platform
for Innovation and Entrepreneur Training Program, theMinistry of education of China(201810057019). We thank
Wu Peishu for his help in greatly improving themanuscript. We are also very grateful to Yang Jiasheng forhis help in the revision of the manuscript.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

306

Design and Research of Navigation and Dynamics
Co- Simulation Platform for Planetary Rovers
Huang Tieqiu

School of Mechanical, Electronic and Control Engineering
Beijing Jiaotong University
Beijing, China
Zhang Bowen

School of Mechanical, Electronic and Control Engineering
Beijing Jiaotong University
Beijing, China
Huang Qianli
The High School Attached to Beijing Jiaotong University,
Beijing, China

Abstract
In view of the high demand for the design of navigation control system for the complex mission of Mars and otherplanets which will be carried out in our country in the near future, the overall design idea of the combined simulation
platform for navigation and dynamics of the planet vehicle is put forward. On the basis of MSC-ADAMS software,
the dynamic model of planetary rover in soft soil environment was established through the secondary developmentof subroutine. C++ was used to establish an interactive parallel simulation system, and the dynamic model andnavigation control model were co-invoked for parallel calculation to form a co-simulation closed-loop system, so as
to realize seamless connection and parallel efficient calculation between the navigation control system and the
dynamic model. Because of the high efficiency of the wheel soil contact model algorithm and the adoption of parallel
multitasking method, the co-simulation can reach almost real time. The platform lays an important foundation for thedesign of navigation control system.
Keywords: planetary rover; co-simulation; dynamics of terrain-vehicle; dynamics of multibody systems

research institutions aiming at planetary wheel soil action.
Yen [1] and Jain [2]et al. developed a ROAMS (RoverAnalysis, Modeling and Simulation) planetary vehicle
virtual simulation platform, and had been applied to the
US Mars exploration program, which contains multiple
sub-modules; Patel[3] et al. developed RMPET(Rover
Mobility Performance Evaluation Tool) simulation
software on the basis of Bekker[4] theory, which is used
to study the rotations of the planet vehicle and calculate
the settlement amount, movement resistance, slip rate,
hook traction and so on of the planet rover; Bauer[5-6] etal. based on Bekker theory, developed a RCAST(Rover

1. Introduction
Mars terrain is more complicated than the moon, and therequirements for the passing of the planetary rover are
getting higher and higher. At the same time, the design ofnavigation and control system has put forward with ahigher demand. There is an urgent need to develop aplanetary vehicle simulation platform with the ability toestablish a planetary vehicle dynamics model and to beefficient, real-time and stable.
At present, a variety of simulation tools for planetary
vehicles have been developed by relevant foreign© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan307

Huang Tieqiu, Zhang Bowen, Huang Qianliconceptual, Analysis and Simulation Tool) tool with theAS2TM model of soil rotation; Feng[7] et al. havedeveloped
Artemis(ADAMS-based
Rover
Terramechanics and Mobility Interaction Simulator).
In China, Huang Tieqiu[8] et al. proposed a modular cosimulation model of lunar rover based on Adams andC/C++ language of multi-body dynamics software. Jiang
Lei [9] et al. constructed the simulation analysis software
for the 3D visualization of the lunar rover with Pro/E
secondary development technology, the Vortex physics
engine and the OSG graphical visualization object; Xu
Yawei[10] and Shao Zheng[11] have established a lunar
rover simulation platform with Bekker theory and
AS2TM wheel-soil interaction model, and Ding
Liang[12] has designed a comprehensive planetary
vehicle simulation system through Matlab and SpaceDyntoolkits. However, the seamless link and the parallel
efficient calculation between navigation control systemand dynamic model are not realized.
In order to solve the problems in the design of planetary
vehicle navigation system, this paper puts forward theoverall design idea of the navigation and dynamics cosimulation platform on planetary vehicle. On the basis ofdynamic simulation software MSC.ADAMS, wedeveloped subroutines with the formula of Bekker wheelsoil action, and a virtual prototype of the planetary
vehicle with the interaction of the soil contact was
programmed to simulate the vehicle to obtain moreaccurate results of the performance evaluation of thevehicle. By studying the interface between command anddata flow between dynamics and navigation control
system, an interactive parallel simulation control system
was established by C++, which seamlessly linked thenavigation control system and dynamics model of theplanetary vehicle. Then it can form a co-simulation
platform to realize efficient co-simulation of navigation
control and dynamics.

Fig.1 Architecture of Simulation Platform

2.1. Rover Model
By consulting relevant literatures, the active suspension
planetary vehicle configuration of the article[13] isselected. Traditional rocker-steering frame suspension of
the planet vehicle has good obstacle performance, but can
not cross the obstacles and subsidence. The active
suspension Planet vehicle is based on the traditional
rocker-steering frame suspension planetary vehicle toeliminate a certain degree of freedom in the suspension
and can adjust the suspension geometry. T The active
suspension structure not only inherits the good passthrough peace and smoothness of the rocker-steering
frame suspension, but also realizes the working
conditions such as lifting the vehicle body, lifting wheels
and the creep.
2.2. Wheel-Soil Interaction Model
Bekker theory model is widely used. Based on Bekker's
theory of ground mechanics, the research on the passingof American planetary vehicle is not universal, so it needs
to be rationalized and modified according to different
application background conditions.
During the rotation of the wheel around the axis, there isa trend of movement relative to the ground, and theground produces friction on the wheel, which is thefundamental reason for the wheel to move forward, also
known as adhesion. The magnitude of the wheel isclosely related to the movement trend of the wheel
relative to the ground, and the motion of the wheel
relative to the ground is described by the slip ratio. The
solution formula of adhesion force P is as follows:

2. Architecture of Simulation Platform
Fig. 1 is the general framework of the simulation
platform, including the planetary vehicle multi-bodydynamics model, the wheel soil action model, theinteractive parallel simulation system and the 3D
dynamic image and data display system.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

308

Design and Research of

2.3. Dynamic Model of Planetary Vehicle Drive
Systemslk
P (c A W tan ) [1
(1 e k )] (1)00slWhere c0 is soil cohesion, A is plate contact area, W
normal load on wheel, φ0 is soil internal friction angle, k
is Bernstein–Goriatchkin’s parameter of a load–sinkage
curve, s is slip ratio, l is wheel length of a contact area.
Since the slip ratio is 0 or tends to 0 when the wheel isstationary, the adhesion formula (1) is not usable at thistime. For this reason, the formula for correcting theadhesion is obtained by the formula (2) within the range
permitted by the guaranteed error:Pk( c A W tan ) [1
(1 e sl / k )]
0
0
sl
k

( c A W tan ) [1
0
0
s
l
min

(1 e sl / k )]

(s s
)
min
s
s
min

(s s
)
min

ADAMS subroutine is an effective tool to extend
dynamic model capability. Through the secondary
development of subroutine, the modeling and calculationof motor control torque and MOTION drive unit can berealized.
PID algorithm is a widely used motor drive control
algorithm, based on PID algorithm for the secondary
development of SFOSUB subroutine, can achieve wheel
drive modeling simulation while monitoring motor
current, voltage and so on.
Because the method of establishing MOTION driving
unit is adopted to realize the modeling and simulation ofactive suspension, according to the working mode ofactive suspension, the motion-controlling subroutine
MOTSUB is redeveloped to realize the dynamic
modeling and simulation of active suspension drive.

(2)

Where smin is correction factor of slip ratio.
The Bekker theory uses the following formula to describe
the external (due to soil compaction) motion resistances:

2.4. Natural Terrain

2n 2(3W ) 2n 1

Rc
(3

2n 2
n) 2 n 1(n 1) (kcb k1
) 2n 1n 1
D 2n 1(3)

Natural terrain is an essential part of the simulation
platform, this paper constructs the natural terrain
environment of the platform through the C + + language
subroutine. Natural terrain environment is established by
dividing grids, as shown in Fig.2 :(a) The terrain isdivided into square grids with equal spacing;(b) The
vertices of each lattice give the coordinate value torepresent the height of the terrain point;(c) The threedimensional vertices of the lattice are connected
diagonally.

Where n is Bekker sinkage coefficient, D is wheel overall
diameter, b is wheel width, kc 、 kφ are Bekker’s
cohesive and frictional moduli of soil deformation of aload–sinkage curve.
However, when the speed of wheel increases to a certain
value, the external (due to soil compaction) motion
resistances does not increase with the increase of speed.
In order to reflect this phenomenon and ensure thecontinuity of force, the resistance Rc is correlated withthe velocity, and the formula is obtained by modifying
the formula (3).

Fig.2 Natural Terrain

2n 2(3W ) 2 n 1
2n 2

1(3 n ) 2 n 1 ( n 1) ( k
Rc

c

n 1

(1 a s )( v v )10b k ) 2 n 1 D 2n 1

2n 2
(3W ) 2 n 1
2n 2(3 n ) 2 n 1 ( n 1) ( k

1
c

n 1

b k ) 2 n 1 D 2n 1

2.5. Interactive Parallel Simulation System
The simulation platform should make up the closed-loop
simulation with the planetary vehicle navigation system,
which has high requirements for the real-time simulation.
A "bridge" is established between the platform and thenavigation system to realize the closed-loop simulation,and the interactive parallel simulation system transmits
the command of the navigation control system to themulti-body dynamics model of the planetary vehicle.
Then the simulation data is transferred to the navigation
control system to realize the seamless link between the(4)v(1 a s )(v v )
10v
0Where a1 is correction coefficient of slip ratio, v0 is
judgment factor of velocity.
Drawbar pull DP is actually the real power of the wheel.
The formula is:
DP P Rc(5)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

309

Huang Tieqiu, Zhang Bowen, Huang Qianlinavigation control system and the dynamic model of theplanetary vehicle.
Real-time simulation is one of the most difficult
problems in platform building. How to improve theefficiency of program execution and satisfy the real-time
performance of simulation is an urgent problem to besolved in establishing an interactive parallel simulation
control system.
Through the research, it is found that the efficiency of
program execution can be greatly improved by using
multithreading parallel mode, that is, instruction andsimulation data transmission and state display operation
are independent. In order to solve the data transmission
between different threads, the memory mapping method
is adopted to map the shared data to a piece of memory
space, and the data is read by the token method to ensure
the integrity and real-time of the data.
To establish an interactive parallel simulation system, it
is necessary to clarify the interface between each moduleand the navigation control system under the functions ofestablishing the body system of the planetary vehicle, thesimulation of the contact dynamics of the wheel soil, thesimulation of the drive of the planet car, the measurement
and output of the dynamics and kinematics results, andto establish the data communication.

k
1.78 cmkc
0.14 N/cm(n+1)
kφ0.82 N/cm(n+2)
The active suspension motion mode is achieved by the
combination of the angle adjustment mechanism, theclutch and the drive mechanism, and the steering
mechanism maintains the zero position.
As shown in Fig.3 is the lifting condition of the planetary
vehicle body. Lifting wheel, creep and other working
conditions are based on the vehicle body lift to achieve.

Fig.3 Body lifting

When a wheel of a planetary vehicle fails or needs tocross an obstacle, the wheel can be lifted as shown in
Fig.4. During the simulation, we found that when theheight of the car body is low, the body may touch theground, so the height of the body needs to be raised

3. Rover Simulation
before lifting the wheel.

By using the soil characteristic parameters shown in
Table 1[14], an accurate control strategy is developed tosimulate the various working conditions of the planet
vehicle in the simulation of Mars surface environment.
are three core technologies of augmented reality
technology, including three-dimensional registration
technology, display technology and intelligent
interaction technology[5].

Fig.4 Wheel lifting

The peristaltic mode of the planetary vehicle is a complex working condition. In the process of lifting andlifting the body, the wheels move together to make theplanet rover wriggle, as shown in Fig.5.

Fig.5 Creep of rover

4. Conclusion
Aiming at the dilemma of the design of navigation
control system in the near future exploration of Mars andother planets, this paper puts forward the overall designidea of the co-simulation platform of planetary vehicle
navigation and dynamics. Through the development of
MASC.ADASM software two times, the efficient joint
simulation platform of planetary vehicle navigation

Table 1: Soil Properties

Soil Propertiesnc0
φ0

Figure1.0
0.017 N/cm2
35°© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

310

Design and Research of[12] Ding
Liang.
WHEEL-SOIL
INTERACTIONTERRAMECHANICSFOR
LUNAR/PLANETARYEXPLORATIONROVERS:MODELINGANDAPPLICATION[D]. Haerbin: Harbin Institute ofTechnology, 2010(in Chinese).
[13] Gao Haibo, Zheng Junqiang, Liu Zhen, et al. Xu Yawei,
Chen Jianxin, Liu Liangdong, et al. A Simulation Tool for
Dynamics and Control of a Lunar Rover in Soft Soil
Environment[J]. ROBOT, 2017, 39(2): 140-150(inChinese).
[14] Liu Xingjie, Su Bo, Jiang Lei, et al. Research on Soil
Mechanical Properties of Martian Surface Soil[J].
Manned Spaceflight, 2016, 22(4): 459-465(in Chinese).

control system and dynamics is established by using
multithreaded Parallel computing mode, which
successfully combines the contact Dynamics model ofwheel soil, the dynamic model of multi-rigid body ofplanetary vehicle and the navigation control system.
References[1]

Yen J, Jain A, Balaram J. ROAMS: Rover Analysis,
Modeling and Simulation[C]//Proceedings of the Fifth
International Symposium on Artificial Intelligence,
Robotics and Automation in Space, Paris: European
Space Agency, 1999, 249-254.
[2] Jain A, Balaram J, Cameron J, et al. Recent developmentsin the ROAMS planetary rover simulation
environment[C]//IEEE
Aerospace
Conference
Proceedings, Big Sky: IEEE, 2004, 861-876.
[3] Patel N, Ellery A, Allouis E, et al. Rover mobility
performance evaluation tool (RMPET): a systematic tool
for rover chassis evaluation via application of bekker
theory[C]//Proceedings of the 8th ESA Workshop on
Advanced Space Technologies for Robotics andAutomation, 2004, 251-258.
[4] Bekker M G. Introduction to terrain-vehicle systems[M].
Translation Group of Introduction to terrain-vehicle
systems. Beijing: China Machine Press, 1978(in Chinese).
[5] Bauer R, Leung W, Barfoot T. Development of a dynamic
simulation tool for the ExoMars rover[C]//Proceedings of
the 8th International Symposium on Artificial Intelligence,
Robotics and Automation in Space, Paris: European
Space Agency, 2005, 1-8.
[6] BAUER R, BARFOOT T, LEUNGW, et al. Dynamic
Simulation Tool Development for Planetary Rovers[J].
International Journal of Advanced Robotic Systems, 2008,
5(3): 311-314.
[7] ZHOU F, ARVIDSON R E, BENNETT K. Simulations ofMars Rover Traverses[J]. Journal of Field Robotics, 2013,
31(1): 141-160.
[8] Huang Tieqiu, Xing Yan, Teng Baoyi. A New Mode of
Co-Simulation Based on Adams and C/C++ Language[J].
Aerospace Control, 2011, 29(1): 63-66(in Chinese).
[9] Jiang Lei, Guo Jianjuan. Simulation Algorithm of Lunar
Rover Based on Vortex[J]. Transactions of the Chinese
Society for Agricultural Machinery, 2012, 43(s1): 352356(in Chinese).
[10] Xu Yawei, Chen Jianxin, Liu Liangdong, et al. A
Simulation Tool for Dynamics and Control of a Lunar
Rover in Soft Soil Environment[J]. Aerospace Contrd andApplication, 2008, 34(6): 52-57(in Chinese).
[11] Shao Zheng. Study on Virtual simulation Platform of
Tyre-soil Interaction for Lunar Rover Based on Virtual
Reality Technology[D]. Shenyang: Northeastern
University, 2008(in Chinese).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan311

Research on Application of SFM Method
in Virtual Reality Modeling
Jiwu Wang

School of Mechanical, Electronic and Control Engineering
Beijing Jiaotong University
Beijing, China
Chenyang Li

School of Mechanical, Electronic and Control Engineering
Beijing Jiaotong University
Beijing, China
E-mail: jwwang@bjtu.edu.cn, 17121244@bjtu.edu.cn
www.bjtu.edu.cn

Abstract
Virtual Reality (VR) technology has been widely used in digital cities, industrial simulation, training, etc. And virtualenvironment modeling is one of the most important parts in the process. When modeling virtual environment basedon reality scene, using conventional methods to modeling is complicated and the model was made is different fromthe real scene due to the high complexity of reality scene. Applying SFM method to VR environment modeli- ng
based on reality scene was proposed in this paper, which can improve the realism of the VR environment, reduce themodeling steps and shorten the modeling time.
Keywords: Virtual Reality, SFM method, modeling, Unity.

is very complex and the realism of the obtained modelcannot be guaranteed.

1. Introduction
Virtual Reality (VR) is an artificial media space built bycomputer, which is virtual with a sense of reality
although. VR enables people to enter a virtualenvironment through multimedia sensor interaction
devices with an immersive feeling. With external devices,such as helmets and data gloves, users see an interactive
dynamic 3D view formed by multi-source information.
VR has been applied in many fields such as aviation,
aerospace, automobile manufacturing, education, etc.
VR technology uses computer to generate a virtual world
with realistic three-dimensional visual, auditory, tactile
and other sensory forms. The first problem to be solved
during the process is virtual environment modeling, thatis, the construction problem of virtual world. Using
conventional methods to modeling, the modeling process

2. Introduction of conventional virtual reality
simulation modeling methods
The Conventional VR environment modeling has been
carried out in many research, which are shown as
follows:(1)Using VB, C++, OpenGL graphics library and othertools to develop, in which the process is complex and theworkload is large. These methods are not verysatisfactory as the processes are not convenient and theobtained models are not realistic enough.
(2)Using VR software (such as WTK and VRML) tomodeling complex 3D graphics is very difficult and timeconsuming, the obtained model is also not satisfied [1].

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

312

Jiwu Wang, Chenyang Li(3)Considering these methods above, Bing Dang [2] used
Multigen Creator simulation modeling software tocomplete the creation of virtual campus environment
through five stages: data acquisition, data pre-processing,create model, model optimization and scene integration.
This method uses digital technology to obtain geographic
coordinates from the plan of AutoCAD format, obtain theheight and 3D geometric feature data of each building
through architectural design, and finally collect thecampus texture information by ground photography tocomplete the establishment of 3D model. The model isbuilt as shown in Fig 1.

the image sequence, eliminating the step to pre-calibrate
the camera. Moreover, the robustness of the motion
method is also extremely strong due to the advancementof various feature point extraction and matching
techniques. Another great advantage of the SFM method
is that it can reconstruct large-scale scenes and thenumber of input images can reach million level, which isvery suitable for 3D reconstruction of natural terrain andurban landscape.
Using the SFM method for 3D reconstruction, images orvideos captured by various devices can be used as input,
and multiple images captured from different perspectives
can be restored to their three-dimensional structure.
Taking a bear model as an example, the main processing
flow is shown in Fig 2.

Fig. 1. 3D model of campus

It can be seen that the modeling of the real scene
completed by the method basically restores the actual
campus scene. But the model is relatively simple and not
realistic enough so the user cannot feel sufficient
immersion. It is not appropriate to modeling virtualenvironments based on reality scene.
3. Application of SFM Method in VR Modeling
Major headings should be typeset in boldface with thefirst letter of important words capitalized.
3.1. Instruction of SFM
Fig. 2.the process of SFM

The SFM (Structure from Motion) method is a method ofrecovering camera parameters and three-dimensional
information with numerical methods by detecting
matching feature point sets in multiple uncalibrated
images. The SFM method detects the set of feature points
to be matched in the image to restore the positional
relationship between the cameras firstly with low image
requirement. At the same time, the camera sequence canbe self-calibrated during the reconstruction process using

As you can see from Fig 2, SFM is mainly divided into
four steps:(1)Input and align photos: input the images of thecaptured models or scenes into the SFM program tocomplete the alignment of the images. In the process ofalignment, the feature points in each photo are extracted
firstly, and the feature points are extracted, where theshape or texture information is more prominent in the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan313

Research on Application ofphoto, and then the feature points between the photos arematched. After the matching is completed, the positional
relationship between the cameras is calculated by thedifferent positions of the feature points in the photo.
(2) Generate sparse point cloud: The three-dimensional
coordinates of each feature point in the three-dimensional
space can be solved by the basic principle of stereo vision
based on the positional relationship between the featurepoint and each camera. Then, according to the texture
information of each point, a coefficient point cloud withcolor information is generated.
(3) Generate dense point cloud: According to theobtained sparse point cloud, by using the PMVS tool toperform interpolation between points, a new point isgenerated in the sparse point cloud, thereby generating a
dense point cloud.
(4) Complete the 3D reconstruction: According to theresult of the dense point cloud, the triangulation of thepoint cloud is performed. The adjacent three points areused to form a facet, before which the outliers in thegenerated dense point cloud need to be performed. The
culling operation ensures that the final 3D reconstruction
has no defects, and facets generated by the triangulation
process are connected to form the final 3D reconstruction
model.
SFM method restored the shape and texture informationof the model very well with high realism, which only
requires little operation in the generation process of the
3D model. Modeling can be done manually, which
reduces the modeling process and time and increases therealism of the model.

Fig. 3.the material information of model.

Fig. 4.the texture information of model.

It can be seen that the model basically restores therealistic three-dimensional information, but he number of
texture files can only be one due to the limitation of theUnity software in the subsequent work. Therefore, thederived model only contain positive texture information,
which is also the shortcoming of the current SFM methodmodeling and needed to be improved in the future. This
shortcoming only has an impact on the modeling of themodel, while modeling the scene can get all the texture
information.
The files obtained in the above steps can be imported into
the Unity3D[3] software successively to complete thesubsequent edit of the VR. The imported model is shownin Fig 5.

3.2. Application of SFM method in VR modeling
Please After the modeling is completed, the material
information can be exported through a type 3D format
file such as OBJ, and the texture file of the model is also
exported as a picture, and the exported model and texture
information are respectively shown in Fig 3, Fig 4.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

314

Jiwu Wang, Chenyang Li

2. Bing Dang. Research on virtual scene generation in
VirtualReality[D].Beijing:
Beijing
Jiaotong
University,2010.
3. Unity3D [EB/OL], http://unity3d.com/.

Fig. 5.the model in Unity3D.

4. Conclusion
The application of the SFM method to reality-based VR
environment modeling is discussed in this paper.
Compared with the conventional modeling method, theadvantages of SFM method are shown as flowing:(1). Reduce the difficulty of modeling with less operation.
Using the SFM method to modeling only needs to take amulti-angle photos of the real environment in the earlystage, and then input photos into the program, in which
the modeling can be operated step by step. Comparedwith the conventional methods, the operation is very
simple and requires little on modeling skill in SFM
method(2). Reduce the modeling cycle with less modeling steps.
The SFM method needs less operation steps than theconventional method, and the time period is also shorter,
which can be even faster on a computer with a highperformance GPU.
(3).The realism of obtained model is higher. Compared
to conventional method modeling, the texture
information modeled by the SFM method comes from a
photo rather than a manual selection of tones, so a higher
degree of realism can be obtained when the texture is
finally mapped onto the model.
There is also a disadvantage to improve in the future of
the SFM method: only the positive information of themodel can be mapped on the model as the number
limitation of input texture file of Unity.
References
1. Fengwei Yuan, Biwen LI, Bin He. Accurate Modeling of
VR Based on SolidWorks-VRML [J]. MECHANICAL &ELECTRICAL
ENGINEERINGMAGAZINE,
2007,24(10):103-105.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

315

Visual SLAM System Design based on Semantic Segmentation

Jiwu Wang

School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University
Beijing, China
Yafan Liu

School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University
Beijing, China
E-mail: jwwang@bjtu.edu.cn, 17121246@bjtu.edu.cn
www.bjtu.edu.cn

Abstract
Visual SLAM is helpful for the development of unmanned platform. But only perceiving the geometric informationof the environment is not enough. It is necessary to join the image semantic segmentation in SLAM, which helps tocomprehend the scenes and improve the accuracy of pose estimation. Nowadays, the technique of deep learning
brings new ideas into traditional SLAM by modeling this problem with convolution neural network.In our approach,we propose a pose estimation algorithm based on pixel-level multi-object detection, and 2D semantic information aretransferred to 3D mapping via correspondence between connective Key frames. Meanwhile,the semantic information
is utilized in other modules for to optimize the visual SLAM system.
Keywords: Multi-object detection, Semantic segmentation, Pose estimation, Semantic mapping

problem with convolution neural network. From theperspective of algorithm, there are still three difficulties
in the implementation of traditional visual SLAM.
First, the accuracy of feature point extraction andmatching algorithms needs to be improved. In fact, there
is no feature extraction and matching algorithm withgood performance and efficiency. As we all known,
feature point extraction algorithm can consider
SIFT>SURF>ORB>FAST in performance, and
FAST>ORB>SURF>SIFT in efficiency.2 Furthermore,
the Direct method is less tolerant to illumination due tothe strong assumption of gray value, and the application
scenario is relatively simple. The Direct method can

1. Introduction
SLAM (Simultaneous Localization and Mapping), itmeans that the robot acquires effective information of thesurrounding environment via the sensors carried by the
robot in the process of continuous movement, andcompleting map construction and real-time pose
estimation.1 The contribution of traditional SLAM is that
“Where am I now?”; “What is the structure of myenvironment?”. But the traditional SLAM can't answer
that: “What are the characteristics of the objects in thesurrounding environment?”; “How can I get that target
position? At present, the technique of deep learning
brings new ideas into traditional SLAM by modeling this

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan316

Jiwu Wang, Yafan Liu

hardly perform loop closing because it does not extract
environmental features.
Second, reducing back-end optimized data is also a
difficulty for SLAM. The relationship between the image
coordinates and the spatial coordinates of the featurepoints is non-linear. These matches also introduce anumber of constraint relationships that complicate thecalculation of the variables to be estimated, while also
controlling the computation time to the allowable range.
Third, loop closing part is often prone to loss matching(called False Negative) or False Positive. When a false
negative occurs, the purpose of the loop closing partcannot be achieved (a closed loop cannot be formed) ;when a false positive occurs, the problem is more
complicated and directly leads to a loop-back error (therobot loses itself).
acing the challenges of these problems, we believe thatthe future development of SLAM is either to combinewith state-of-art deep learning or to adopt new sensors(such as Event Camera: the difference between the Event
Camera and the traditional camera is that it records
changes in the scene, not a fixed scene ). Therefore, inthis paper, our approach abandons the inherent idea of the
traditional SLAM, proposes a pose estimation algorithm
based on multi-object detection, and implements anenvironmental semantic map that combines the semantic
information of key frames with the 3D point cloud map.
The core technology adopted is the semantic
segmentation and deep-learning object detection
technology .

Fig. 1. Semantic SLAM Overview

algorithm is used to detect objects on RGB sequence
images. The unsupervised semantic segmentation
algorithm is used to semantically segment selected key
frames. At the same time, Depth information, object
detection results and semantic segmentation information
are simultaneously transmitted into the pose calculation
module for current Camera pose estimation. Secondly, a
locally sparse 3D point cloud mapping is performed
according to the obtained continuous camera pose.
Finally, the 3D point cloud map is merged with thesemantic information on the 2D key frames to construct
a globally consistent semantic map. At the same time, theresults of object detection and the semantic informationof key frames will also optimize the back-end andcomplete loop-back detection to achieve a more perfect
SLAM system.
3. Pose Estimation
The deep learning algorithm is the mainstream target
recognition algorithm in the field of computer vision. It
relies on multi-layer neural network to learn thehierarchical feature representation of images. Comparedwith the traditional object recognition method, it canachieve higher accuracy. At the same time, deep learning
can also semantically segment images, build a semantic
knowledge base of the environment, and improve robot
service capabilities and human-computer inter-action
capabilities.
In the field of visual SLAM, a hierarchical image feature
extraction method based on deep learning has appearedin recent years,3 and it has been successfully applied to
SLAM pose estimation and closed-loop detection.
However, most of them do not perform as well as ORBSLAM. Therefore, this paper breaks through the traditionand proposes a pose estimation method based on pixel-

2. Semantic SLAM Overview
Based on the traditional feature point ORB-SLAM
framework, this paper applies the image information
obtained by object detection and the semantic
information of key frames to pose estimation andsemantic map construction in visual SLAM system.
The overall system can be divided into four modules:
object detection, key frame semantic segmentation,
camera pose estimation and semantic map construction ,as shown in Figure 1. The image acquisition device is the
RGBD sensor (KINECT 2), which can obtain RGB
image and Depth image of surrounding environment.
Firstly, the Single Shot Multi-Box Detector (SSD)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

317

Visual SLAM System Designlevel multi-object detection combined with object
recognition and ORB feature points. The specificimplementation methods are as follows:(i) Using the state-of-art deep learning algorithm toperform object recognition and semantic segmentation on selected key frames in the image sequence.
Then, the semantic information is weighted to eachpixel in the image to filter the effective pixels thathave a positive effect on the object detect-ion,thereby improving the accuracy of the object
detection.
(ii) The system adopts different pose estimation schemes depending on the working environment. In thescene where the identifiable object is rich, thecamera pose estimation will be directly based on thecenter point of the detected objects; and in the scene
where the identifiable object is less, the feature point
matching will be accelerated by the semantic
information to calculate the camera pose.

object detection algorithm to be adopted by this moduleis based on Single Shot Multi-box Detector(SSD)
algorithm.4 As one of the main target detect-ion
algorithm frameworks, SSD object detection network has
obvious speed advantage compared with Faster R-CNN,5and has obvious accuracy advantage compared with You
Only Look Once(YOLO).6
3.2. Sparse scene pose estimation
As mentioned above, the implementation of the simplest
P3P algorithm requires at least four feature points, sopose estimation based on multi-object detection requires
determining at least four targets in the environment. But
sometimes the environmental information is relatively
simple, such as corridors, walls without textures, and soon. Therefore, when there are less than 4 identifiableobjects in the environment scene, we propose a featurepoint matching acceleration algorithm based on semantic segmentation.
In the semantic SLAM system, we know that the featurepoints are mostly concentrated on the surface of theobject with texture. When each frame of the image issemantically segmented, it is equivalent to the classify
the feature points of ORB. Therefore, when the brute
force matching algorithm is applied to the feature point
set with the same semantic label, not only can the
searching range of force matching be reduced, but also
the matching accuracy can be improved. On the other
hand, according to the positional relationship of the same
object in the image between adjacent frames, we caneasily eliminate the mismatched points and improve theaccuracy of matching. Therefore, for an unknown scene
where the identifiable object is sparse, we can use the
ORB feature point matching algorithm with semanticlabel to achieve the desired effect.7
In summary, the pose estimation algorithm based on
multi-object detection combines the continuity of ORBSLAM with SSD convolutional neural network to
improve the accuracy of semantic segmentation. Then, inthe case of crossing the low-level features, the pose
estimation of the camera is performed directly based on
the identifiable objects in the scene.

3.1. Multi-object scene pose estimation
A flowchart of a method for performing pose calcul-ation
based on identifiable object semantic information isshown in Figure 2.
First, the target detection module extracts the positionand category of the identifiable object in each frame ofthe sequence. Second, the extracted object center module
extracts a center point from the semantic pixel

Fig. 2. Pose estimation algorithm flow chart

recognition frame of each object in the image, and thedepth information of the center point in thecorresponding depth image can be easily obtained. If
there are enough similar objects between the front andback frames, and the pixel coordinates and depth
information of the center point of the object between thetwo frames in the respective camera coordinate system
are known, the pose estimation between the two frames
can use the Perspective n Points(PnP) algorithm. The

4.

Semantic Mapping

At the front-end of SLAM, Our approach is to use
unsupervised convolutional neural network algorithm to© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

318

Jiwu Wang, Yafan Liu

obtained semantic information of every frame which areprojected to globally consistent 3D map from a real-time
SLAM system. as shown in Figure 3.
The 3D map is constructed by a series of keyframes
combined with the depth information acquired by the
RGBD camera. At the same time, the semantic information obtained by keyframes in this process is also

Fig. 4. Overview of Semantic mapping

paper points out the technical difficulties of the
traditional SLAM, and summarizes the development
prospects of the traditional SLAM method and the deeplearning SLAM method. Facing the current challenges,
this paper proposes a simultaneous location and mapping program based on semantic segmentation. our
method combines pose estimation, object detection andsemantic segmentation by using convolutional neuralnetworks algorithm, and then makes great progress.

Fig. 3. Semantic segmentation

integrated into the 3D map. Since the 3D map shouldhave globally consistent depth information, it will be reoptimized based on the geometry information in theenvironment.8
The specific implementation of our method is to create a
3D scene map with consistent label information online bymoving the robot with the RGBD camera in an unknown
scene. The method is divided into two separate processes,
as shown in Figure 4. First, a valid keyframes is selectedfrom the sequence of image frames captured by the
RGBD camera, and the 3D map is reconstructed using
the selected keyframes in conjunction with the pose map
generated by SLAM process. Secondly, the 2D semantic
segmentation pro-cess predicts pixel-level classification
on key frames, and uses this classification to create alocal optimal depth estimate for each key frame, andobtains the correspondence between the pixels andvoxels marked in the 3D point cloud. Our approach canultimately achieve global optimal 3D semantic
segmentation and real-time updates of 3D point clouds togenerate glob-ally consistent 3D maps.
5.

References
1. Petillot Y, Salvi J, et al. The SLAM problem: a survey [M].
AMSTERDAM: IOS PRESS, 2008:363.
2. Cadena C, Carlone L, Carrillo H, et al. Past, Present, andFuture of Simultaneous Localization and Mapping:
Toward the Robust-Perception Age[J]. IEEE Transaction
on Robotics, 2016,32(6):1309-1332.
3. Han J, Zhang D, Cheng G, et al. Advanced Deep-Learning
Techniques for Salient and Category-Specific Object
Detection: A Survey[J]. IEEE Signal Processing
Magazine, 2018, 35
4. Jia Y, Shelhamer E, Donahue J, et al. Caffe: Convolutional Architecture for Fast Feature Embedding[J]. 2014.
5. Ren S, He K, Girshick R, et al. Faster R-CNN: Towards
Real-Time Object Detection with Region Proposal
Networks[J]. IEEE Trans Pattern Anal Mach Intell,
2017,39(6):1137-1149.
6. Redmon J, Divvala S, Girshick R, et al. You Only Look
Once: Unified, Real-Time Object Detection[M]//IEEE
Conference on Computer Vision and Pattern Recognition.
2016:779-788.
7. ChaoHua YAN. Deep Semantic Simultaneous Localization and Mapping[D]. Zhejiang University,2018．
8. Li X, Belaroussi R. Semi-Dense 3D Semantic Mappingfrom Monocular SLAM[J]. 2016.

Conclusions

SLAM is a outstanding topic in the field of computer
vision. The study of SLAM technology combined withdeep learning is the emerging direction in the future. This
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

319

Spherical Mobile Robot Designed with Single
Omnidirectional Wheel Method
Liu Wei

School of Mechanical, Electronic and Control Engineering
Beijing Jiaotong University
Beijing, China
Lian Luo

School of Mechanical, Electronic and Control Engineering
Beijing Jiaotong University
Beijing, China
Jiwu Wang

School of Mechanical, Electronic and Control Engineering
Beijing Jiaotong University
Beijing, China

Abstract
This paper presents a new spherical mobile robot driven by single omnidirectional wheel. The physical model isthat an arc rail is built inside the sphere shell, and the arc rail can rotate around a spherical diameter freely. It
supports one driving omnidirectional wheel and keeps the driving omnidirectional wheel touching the inside of thesphere shell. The touching friction force drives the sphere rolling forward or backward. Steering drive can drive theomnidirectional wheel moving along the arc rail, that causes the spherical mobile robot steering left or right. The
omnidirectional wheel and the steering drive working together can realize the omnidirectional movement in any
direction. The structure of the spherical mobile robot is described in detail, and its motion and dynamics models arebuilt. The preliminary analysis is carried out by dynamic software and shows that the spherical mobile robot canachieve omnidirectional motion well.
Keywords: Spherical mobile robot, Omnidirectional wheel, Omnidirectional motion.

position attitude is in stable, and when the mass centroid
away from the center, the sphere has a strong tendency toself-stabilization, etc. The characteristics of spheroid
inspire the enthusiasm of scholars to invent many kindsof spherical robots. Since the spherical robot concept first
appeared, it was considered as a new technology to leadthe industrial revolution like implementation in planetmoon exploration, military usage, and inevitably arising
the demand of the toy entertainment industry[1, 2].

1. Introduction
The sphere is concise shape with the smallest surface area,
the largest volume, and rolling in all direction. It is themost excellent shape existing in the universe, and the best
rolling body that every animal likes to play with. The
typical characteristics of the sphere are: 1) when nofriction and air resistances, an ideal sphere with uniform
density will be kept moving forward straightly andsmoothly by the inertia force and moment of inertia; 2)
when the mass centroid close to the sphere center, the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan320

Liu Wei, Lian Luo, Jiwu Wang

Through literature and patents review, there are several
types of representative spherical robots, and according totheir physical driving principles, the current sphericalrobots with omnidirectional motion performance could
be divided into two typical categories: 1) Inertia moment
driving type [3, 4, 5, 6], the principle is to rely on the rotating
inertia moment to drive the omnidirectional rolling of aspherical robot, but the inertia moment needs the rapid
rotation like spinning gyroscope to acquire enoughdriving force. 2) Eccentric torque drive type, its principle
is to rely on the motion of eccentric mass centroid
producing the eccentric torque on the fulcrum, and theeccentric torque drives the spherical robot rolling. The
invention of drive mechanism with eccentric torquedriving the spherical robot has been very rich[7, 8, 9,]. Oneof the earliest and concise spherical robots is invented
and researched by Halme etc.(1996) [5]. Its driving
mechanism is made up of a driving wheel, a steering shaft
and a balance wheel. One of recent and most concise
spherical robots is reported by Liu Wei etc.(2018).It has
two independent driving omnidirectional wheels locatedat each end of the inside vertical diameter of the sphereshell, and the two wheels’ axes are in orthogonal position.
Both omnidirectional wheels can roll on the inner surfaceof the spherical shell by their friction forces. The lower
omnidirectional wheel is responsible for walking drive,and the upper one for steering drive.
Although there are many mechanisms to realize theeccentric torque drive principle, their physical model canbe described briefly as shown in Figure 1. Supposing theonly resistance is the rolling resistance torque Tf,, thetorque T caused by mass centroid is considered to driveor steer the robot rolling. The distance r which from thesphere center to the mass centroid m directly determines
the quantity of the driving torque T, that is T=m.g.r.sin(a).
When T > Tf, the spherical robot will start to move andincrease speed. When T = Tf, the spherical robot will
move uniformly. When T<Tf, the spherical robot will
reduce speed until stop.

Figure 1 Principle of eccentric torque driveThis paper proposes a novel and concise spherical robot
mechanism with only one omnidirectional wheel to drive
the eccentric torque overcome the rolling resistancetorque. The spherical robot can realize theomnidirectional motion in any direction. The omni wheel
is arranged by a series of small rollers around thecircumference which are perpendicular to the driving
direction. When omni wheel drives, its rollers can slide
laterally on the wheel periphery. The lateral rolling of
small rollers does not interfere its circumferential driving
function. The mechanism is shown in Figure 2. The
spherical robot driven by one omni wheel will beanalyzed as following in several aspects as mechanism
composition, motion and dynamics analysis etc.

Figure 2 Omni wheel

2. Mechanism
The spherical mobile robot includes an omnidirectional
wheel, driving motor, arc rail, chassis, support wheels,
steering motor, side wheels, springs and spherical shell,as shown in Figure 3. The lower end is the Driving
omnidirectional wheel, and it produces friction at the
inner surface of the spherical shell under the co-functionof two forces of gravity and spring pre-pressure. The
upper end is the steering omnidirectional wheel, and it
produces friction at the inner surface of the sphericalshell with the function of the spring pre-pressure. The
sphere centroid deviates from the center of the sphere(distance r). When driving the omnidirectional wheel,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

321

Spherical Mobile Robot Designed

the generated rolling friction is transmitted to the frame,
which causes the frame swing forwards or backwards.
The steering omnidirectional wheel rolls sideways (leftor right direction), and it does not hinder the driving of
the driving, so that the generated eccentric torque drives
the spherical shell to roll backwards or forwards.
Similarly, when driving the upper omnidirectional wheel,
the generated rolling friction is transmitted to the frame,causing the frame to swing towards left or right. At thistime, the driving omnidirectional wheel rolls sideways
without hindering the steering drive, thereby generating
an eccentric moment to drive the sphere shell moveleftwards or rightwards.

the height of centroid, the control system, the power
supply and the drive motor of the upper omnidirectional
wheel are all arranged away from the center of the sphere(sphere centroid?). In order to ensure that the upper andlower omni-direction wheels are always on the vertical
diameter, an important task needs to be completed before
the frame components are installed into the sphericalshell - static balance.

Figure 4 Run pose and steering pose

4. Conclusion
The spherical robot's driving scheme makes use of thecharacteristics of the omnidirectional wheel to simplify
the driving structure, and it can better increase thedistance from the center of mass to sphere center so as togenerate a maximized eccentric driving torque. Thisorthogonalarrangement
makes
the
walking
omnidirectional wheel and steering omnidirectional
wheel independent at both force direction and motion
direction, which is very conducive to the control system.
(1). Reduce the difficulty of modeling with lessoperation. Using the SFM method to modeling only
needs to take a multi-angle photos of the realenvironment in the early stage, and then input photos into
the program, in which the modeling can be operated stepby step. Compared with the conventional methods, theoperation is very simple and requires little on modelingskill in SFM method

Figure 3 Drive mechanism of biorthogonal omni wheels

3. Function analysis
First, the coordinate system G-XYZ is established on the
ground, and the coordinate system O-xyz is establishedon the spherical robot. As shown in Figure. 4, since thespherical shell always rotates with respect to the frame,
the O-xyz coordinate system is fixed to the frame of thesphere centroid. Parameters are set as follows: m is themass of all the parts or mechanisms in the spherical shell,r is the distance between this mass and the spherecentroid, M is the mass of the spherical shell, t is the wall
thickness of the spherical shell, frr is the rolling resistanceof the spherical shell, far is the air resistance, V is the
speed of the sphere, w is the angular velocity of thesphere. Now we suppose that the rolling contact of thewhole system is pure rolling (that is, temporarily
excluding the influence of sliding), so that a series ofsimplified dynamic analysis can be performed. Here, amodel could be constructed for the motion of thespherical robot, and it could be decomposed into thefollowing phases: start-up phase, uniform phase, obstacle
phase, take-off phase and landing phase.
The sample mechanism uses two motors to control theforward/backward and turn right/left. In order to reduce

References
1. Rhodri H A，Julian F v. Rolling in nature and robotics: a
Review[J]. Journal of Bionic Engineering, 2006,
3(4):195-208.
2. Deng Zongquan, Yue Ming. Overview of the developmentof spherical robots[J]. Robotics and applications, 2006（3）,27-31.

3. Bhattacharya S, Agrawa S K. Spherical rolling robot: adesign and motion planning studies, Proceedings of theIEEE International conference on Robotics and© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan322

Liu Wei, Lian Luo, Jiwu Wang

4.

5.

6.

7.

8.

9.

Automation[C]. San Francisco ， CA ， April ， 2000 ，
835~839.
Joshi V A，Banavar R N，Hippalgaonkar R，Design
and analysis of a spherical mobile robot [J]，Mechanismand Machine Theory，2010，45(2) :130-136
Halme A, Schonberg T, Wang Y. Motion control of aspherical mobile robot. 4th IEEE International Workshop
on Advanced Motion Control, Mie, Japan, 1996, 1,259–
264.
Javadi A H A, Mojabi P. Introducing august: A novel
strategy for an omnidirectional spherical rolling robot.
IEEE International Conference on Robotics andAutomation, Washington, DC, USA, 2002, 3527–3533.
Batten C. Wentzlaff D. Kickbot ， A Spherical
Autonomous Robot. Massachusetts Institute of Technology people.csail.mit.edu/wentzlaf/documents/Batten.
2001. Class.Kickbot.pdf，2006，3，20
Jaimez M，Castillo J J，Garcia F，et al，Design andModelling of Omnibola © ，A Spherical Mobile Robot[J]，Mechanics Based Design of Structures and Machines，
2012，40 (4): 383-399
Otani T et a1，Position and attitude control of a sphericalrolling robot equipped with a gyro[A] ， International
Workshop on Advanced Motion Control[c]，Istanbul，Turkey：IEEE，2006，416—421© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan323

An Augmented Reality Implementation Method
Based on Unity3D
Jiwu Wang

School of Mechanical, Electronic and Control Engineering
Beijing Jiaotong University
Beijing, China
Weixin Zeng

School of Mechanical, Electronic and Control Engineering
Beijing Jiaotong University
Beijing, China
E-mail: jwwang@bjtu.edu.cn, 17121253@bjtu.edu.cn
www.bjtu.edu.cn

Abstract
Through computer graphics and visual technology achieving the superposition of virtual objects in real scenes,
augmented reality technology is able to present the sensory effects of real-time interaction. It has broad application
prospects and application value. At present, there are two main ways to achieve augmented reality. One is theunderlying code, which combines OpenGL and OpenCV libraries to realize augmented reality. The second is thesecondary development of related content through the corresponding software and SDK package. In this article,
augmented reality will initially be completed through Unity 3D and Qualcomm's Vuforia SDK.
Keywords: Augmented Reality, Vuforia SDK, Modeling, Unity3D.

superimposed on the TV screen through augmented
reality, providing more information to the audience. In
the field of education, augmented reality technology
will change the traditional education model and create amore vivid and interesting learning scene for students.
Augmented reality has broad application prospects andgreat application value, therefore it has very significant
research significance[2].

1. Introduction
Augmented reality technology is a technology
developed on the basis of virtual reality technology,
which uses computer graphics and computer vision tosuperimpose virtual objects into real scenes, allowing
users to perceive a real scene and virtual scenes
seamlessly. Meanwhile, with the new environment, theuser is presented with a sensory effect of real-time
interaction[1].
Augmented reality was first applied in the military field.
But with the continuous development in recent years, it
has been widely applied to more fields. In the medical
field, doctors can use augmented reality technology toaccurately locate the patient's lesions for easy surgery.
In the field of television broadcasting, important data on
the sports field in the live sports competition can be

2. Background
Many research institutes at home and abroad haveachieved certain results in the direction of augmented
reality. The first pioneer in the world to study
augmented reality was Ivan Sutherland[3]. In 1992,
Steven Feiner published a paper on the prototype of theAR system at the International Conference on Graphic© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

324

Jiwu Wang, Weixin Zeng

Images, which was subsequently widely cited by theAmerican Computer Society[4]. In 1999, the Universityof Washington released the open source augmented
reality development kit-AR Tool Kit. With the releaseof this development kit, a large number of augmented
reality applications are beginning to emerge. In 2014,
Google launched "Google Glass". In addition to thefunctions of sending messages, taking photos, talking,viewing maps, video chats, setting calendar reminders,etc. through voice control, its most prominent feature is
the use of augmented reality technology to attach
environment information on display devices. In 2015,
Microsoft released the AR head-up Microsoft HoloLens,
which is not subject to any restrictions - no cables,
earpieces, and no need to connect to a computer.
Microsoft HoloLens has the features of holograms,
high-definition lenses, stereos, and more, allowing you
to see and hear holograms around you. Appleannounced the ARKit at WWDC 2017, which helps us
implement AR technology in the easiest and fastest way.

virtual objects under the camera image through OpenGL
to realize augmented reality. As shown in FIG.1, thefeature points in the real-time scene are extracted byusing a related feature point extraction algorithm. Then,the relevant feature point data is obtained after detection
and description, and the Hamming distance is used formatching, and the RANSAC algorithm filters thematching points to obtain the final matching point data.
Then, according to the conversion of the coordinate
information, the position of the virtual information in
the real scene is determined, thereby realizing theaugmented reality[6].

3. Implementation Methods
There are three core technologies of augmented reality
technology, including three-dimensional registration
technology, display technology and intelligent
interaction technology[5].
Three-dimensional registration. In order to realize
the combination of virtual and real, the position of
the augmented reality real-time tracking camera iscalculated for the position of the camera image andthe position of the virtual image appearing in thereal scene.
Display technology. Depending on the real world,
virtual images or models present a virtual and fusedworld to the user.
Real-time interaction. This refers to an interaction
between the user and the virtual object, and theinformation flow and feedback can be timely andeffective
between
the
two.
The
main
implementation methods of augmented reality are
two categories in the general direction. The first is
through the underlying code, and the second is thesecondary development of related content throughthe corresponding software and SDK package.

Fig. 1. Underlying implementation

3.2. Secondary Development
With some mainstream AR SDKs, such as Easy AR,
Vuforia[7], Metaio, AR Kit and so on, theimplementation of augmented reality is achieved
through the related AR engine. Users of the AR SDK
need to know the functions that the SDK can implement,
the platforms they support, and the stability of
implementation. Secondary development is currently amainstream choice for AR development, not only toachieve faster results, but also to build practicalapplications.
4. Achievement Based on U3D
Augmented Reality have been achieved throughUnity3D[8] and Qualcomm's Vuforia SDK.
4.1. Unity3D

3.1. Underlying Code

Unity3D is not only a 3D animation software, but also aa very compatible platform. It is capable of making
games as well as realizing virtual reality and augmented

Relatively speaking, OpenCV realizes the recognitionand positioning of Marker, and then superimposes© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan325

An Augmented Reality Implementation

reality. We chose it because it is compatible with many
development environments and offers a variety ofplugins that are necessary for later higher levels ofdevelopment.

4.2. 3ds ModelingFig. 3. Identify map

5. Experiment and Results
This experiment uses Win10 system, and the graphics
card model is GTX960. The software is Unity3D2018.2.10-f1, and the development kit is Vuforia-unity6-2-10.
5.1. Building Environment
Fig. 2. 3D model of Knight

Figure 2 shows the 3D model named Knight. Using 3Dmodeling software called 3ds Max to build 3D model, it
can be exported to a format file that Unity3D can open.

In our research, we first import the Vuforia SDK into
U3D and then apply the AR Camera and Image Target.
When we get the KEY under Vuforia's official website,
we make a recognition map and import it, and thenimport the 3D model generated by 3ds Max.

4.3. AR Engine

5.2. Results

At this point, we should focus on implementing
augmented reality with Unity3D. In some SDKs such as
AR Kit, Smart AR, Easy AR and Vuforia, we choose
Vuforia to create a location-based AR application withno markup type, which can meet the requirements ofUnity3D. This SDK can provide a AR-camera which
could realize the superposition of virtual objects in realscenes. Meanwhile, the SDK can make identify map
which is able to be identified by the AR-camera. Basedon this SDK, we will add location-based functions andother functions. Figure 3 shows one of the identify map.

Fig. 4. Displaying the Knight Model by AR

Figure 4 shows the experimental results. U3D is able todisplay the Knight model above the recognition map© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

326

Jiwu Wang, Weixin Zeng

and let the model do some action. Once the natural
image inside the camera being read, the model willcontinue to be displayed even if the position changes orthe angle changes. In addition, the Knight is able tocomplete some designed actions when the Knight isdisplayed. However, when the recognition map is too
close to the camera, the model will appear outside thescreen and cause invisibility.
6. Conclusion and Future WorkIn this study, we tried to make an application using
Unity3D. Future tasks include advanced application
features such as adding instruction windows,
implementing some instructions, the creation of othermodels, and some of the problems actually used. We
hope to achieve more functionality in the future work.
References[1] T Ronald, Azuma. A survey of augmented reality Presence,
J. 6(4) (1997)355-385.
[2] Xiyu Wang and Fengjun Zhang, Research status anddevelopment trend of augmented reality technology, J.
Science and technology guide, 36(10) (2018)75-83.
[3] I E Sutherland, The ultimate display, Proceedings of IFIP,
J. 65(2) (1965)506-508.
[4] S Feiner, B Macintyre and D Seligmann, Annotating thereal world with knowledge-based graphics on a seethrough head-mounted display, J. Conference on
Graphics Interface, (1992)78-85.
[5] Yin Hou, Augmented reality technology review, J.
Computer measurement and control, 25(2) (2017)1-8.
[6] Yue Wang and Shusheng Zhang, Model-based augmented
reality unmarked three-dimensional registration tracking
method, J. Journal of Shanghai Jiaotong University, 52(1)(2018)83-90.
[7] Vuforia [EB/OL], http://developer.vuforia.com/.
[8] Unity3D [EB/OL], http://unity3d.com/.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

327

Research on a Non-Invasive Measuring Method of Blood Glucose Concentration Based on
Electrical Impedance Spectrum
Tong Yin 1, Xiaoyan Chen 1*, Meng Du1, Hongyi Yin1
1

College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
E-mail: * cxywxr@tust.edu.cn
www.tust.edu.cn

Abstract
Different from the traditional measurement approach and instruments of blood glucose concentration, we propose anovel measuring method related to electrical impedance spectrum without invasion. A serial experimentations aredesigned and carried out to detect the electrical impedance amplitudes and phases under 1Hz-10MHz frequencies.
Considering the skin contacting impedance and cytomembrane conductivity feature, a simulate circuit is designed
and connected in serial with the measuring loop. The measured data were analyzed by an improved Cole-Cole model
to derive the relationship between the impedance value and the blood glucose concentration. The results show thatthe electrical impedance is strong negative correlation to blood glucose concentration, as the blood glucose
concentration increases, the impedance value decreases. The conclusion of this research provides a theoretical basis
for non-invasive blood glucose surface-measuring.
Keywords: blood glucose concentration, impedance spectroscopy, non-invasive glucose measurement

spectroscopy, far infrared (FIR) spectroscopy, opticalrotation of polarized light, and electrical impedance
spectrum (EIS) etc. The first three techniques are withrelative weak changes, limited resolution and insufficientprecision due to glucose imprinted to the signals
registered[4]. EIS has been tested over a period of time for
non-invasive glucose concentration monitoring.
EIS has a unique standing among numerous modern
methods used for physical and chemical analyses ofmaterial, because it can investigate the relaxation
processes of complex systems in an extremely wide rangeof characteristic times from 10-12 to 104s. EIS isespecially sensitive to intermolecular interactions and, isable to monitor cooperative processes[5].
This aim of this research is to explore a new way that
detecting the blood glucose concentration non-invasively
by EIS method on the surface of body. The concentration

1. Introduction
The blood glucose concentration is one of the key
indicators reflecting the condition of diabetic patients. At
present, the most common testing method is acupuncture
measurement before and after meal invasively, whichbring great pain to the patients with potential dangers of
infection[1]. Compared with the common invasive
method, non-invasive blood glucose concentration
measuring has the advantages of no acupuncture and nopains with simple and quick operation. Non-invasive
blood glucose concentration measuring has an important
application value of monitoring the blood glucose change
in time and of visualizing the abnormally glucose
metabolism rapidly[2].
There are four categories of non-invasive glucose
monitoring technologies involving electromagnetic
waves in development[3]: near infrared (NIR)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan328

Tong Yin , Xiaoyan Chen , Meng Du, Hongyi Yinelectrode of 16 cm2 is selected for simulation test, soCmem, 0=16μF. Bring Cmem, 0 into formula (2) to obtaincell membrane capacitance values corresponding toblood glucose levels of different concentrations, as
shown in Fig. 1. The specific values are shown in Table
1.

of electrolytes in the blood and tissues changes withglucose concentration, which in turn causes an imbalancein electrolyte concentration between blood and tissue
fluid. This causes the ions to move in a directionalmanner, and the electrical properties of the cell
membrane change with the concentration of ions passing
through the cell membrane. From a macro perspective,
the impedance of the human body will also change
accordingly. On the biological principle, it indicates ahopeful direction for non-invasive blood glucose
measuring.
2. The relationship deduction of glucose and cell
membrane capacitanceAn increasing concentration of glucose in blood induces
significant reactions in blood and human tissues. It is
known that the concentration of sodium decreases andpotassium increases due to the active transport of ionsand water movement from the tissue and blood cells, suchas erythrocytes, into the vascular system. It can beunderstood on the cellular level that glucose uptake
through membrane transporters (GLUTs) requires
controlling the electrolyte balance to maintain osmolality
and cell sizes[5].
Ping. S proposed and verified the relationship between
cell membrane capacitance (Cmem) and blood glucose
concentration (Cglu) by Michaelis-Menten equation as
=

1+

,

,

Fig. 1. The relationship between Cglu and Cmem. As the Cglu
increases, Cmem shows an upward trend.
Table 1. Corresponding values between blood
glucose concentration (Cglu) and cell membrane
capacitance (Cmem)

Cglu(mmol/L)
0
1
2
3
4
5
6
7
8
9
10
11

(1)

Major headings should be typeset in boldface with thefirst letter of important words capitalized.
When the blood glucose concentration is zero, the cell
membrane capacitance is defined as Cmem, 0. According
to the experimental data in[6], when the blood glucose
concentration of the human body is 5 mmol/L, Cmem=1.12
Cmem, 0; When the blood glucose concentration of thehuman body is 20 mmol/L, Cmem=1.37 Cmem, 0.
Substituting this result into formula (1), the relationshipbetween cell membrane capacitance and blood glucose
concentration is
,

=1+

.

.

Cmem(μF)
16.0000
16.8861
17.6045
18.1988
18.6985
19.1245
19.492119.8125
20.0941
20.3438
20.5665
20.7665

From Fig. 1, at the change stage of 3 to 10 mmol/L inhuman blood glucose concentration, the cell membrane
capacitance changes very rapidly, and this change ismore obvious than that caused by other substances to thecell membrane capacitance. When the cell membrane
capacitance changes, the influence of other substances
can be ignored. Therefore, the impedance index can beused to indicate the blood glucose reasonably.

(2)

According to the experimental data in[7], the cell
membrane capacitance value is 1 μF/cm2 when the blood
glucose concentration is zero. In this paper, a rectangular© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan329

Research on a Non-Invasive

3. Equivalent circuit model=

3.1. Establishment of human body equivalentcircuit model
To simulate the effects of different blood glucose
concentrationson
impedance
spectroscopy
measurements, we established a human equivalent circuit
model. In an alternating electric field, the biologicaltissue has greater impedance at the lower frequency. The
capacitive impedance characteristic of biological tissues
was discovered by scholar Philippson in 1920, whoestablished a corresponding bio-impedance equivalentcircuit model.
The single cell equivalent circuit model of biologicaltissue is shown in Fig. 2(a). Ri, Rm and Re are theequivalent resistances of intracellular fluid, cell
membrane and extracellular liquid respectively. Ci, Cm
and Ce are theirs equivalent capacitance. According to theexisting research theory, at low frequencies (generally<1MHz), the values of Ci and Ce are small, so thecapacitive reactance is large and can be treated as an open
circuit; at this time, Rm is large and can also be regarded
as an open circuit. Through the above-describedconversion processing, a simplified biological tissue cell
equivalent circuit model as shown in Fig. 2(b) can beobtained. Since a large number of cells are combined ina certain manner, biological tissues or organs are formed.
So biological tissue can be called a collection of multiple
cells. Therefore, the equivalent circuit model shown in
Fig. 2(b) can also represent the entire biological tissue,and Ri, Re, and Cm represent the internal liquid resistance,
the external liquid resistance, and the membrane
capacitance of the entire biological tissue, respectively.
This is named the three-element circuit model ofbiological tissue [8].
Due to the detecting electrodes and skin contacting, it isalso necessary to consider the effect of contact
impedance. Therefore, we established a novel humanequivalent circuit model as shown in Fig. 3.
Rd represents the contact impedance, Ri is the internalliquid resistance of the human body, Re is the liquid
resistance, and Cmem is the cell membrane capacitance.
According to the experimental data in[9], we set Rd=1kΩ,
Ri=500Ω, Re=500Ω.
According to the Laplace transform in the circuit theory,the corresponding real and imaginary parts of the
impedance can be obtained as:

(

(a)

∗

∗

=

∗

∗

∗
∗ ∗

∗(

∗(

)∗

∗

∗(

)

)

(3)
(4)

)

(b)

Fig. 2. Biological tissue equivalent circuit modelFig. 3. Human equivalent circuit model

3.2. Simulation of the equivalent circuit model
In this paper, the impedance of the human bodyequivalent circuit model is completed under twelve
different equivalent blood glucose concentrations(Cglu=0,1,2,3,4,5,6,7,8,9,10,11). Spectral simulation witha measurement frequency range of 0Hz-100MHz. A
curve of blood glucose concentration and impedance
spectrum as a function of frequency is then obtained.
Impedance measurement at different blood glucose
concentrations was simulated by the human equivalentcircuit model of Fig. 3.
The situation shown in Fig. 4 is Cglu = 5 mmol/L. It canbe seen by Fig. 4 that above 100Hz the impedance keep
at 1250Ω stably. Therefore, in order to show the results
clearly, the high frequency band is out of consideration,and the change segment is displayed.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan330

Tong Yin , Xiaoyan Chen , Meng Du, Hongyi Yin(a)Fig. 5. The relationship between frequency and impedance at
different blood glucose concentrations.

(b)Fig. 4. (a) The curve of the model impedance value withfrequency when Cglu=5 mmol/L, the ordinate is the impedancevalue, and the abscissa is the frequency of the logarithmic
display; (b) The curve of the model Phase value withfrequency when Cglu=5 mmol/L, the ordinate is the phasevalue, and the abscissa is the frequency of the logarithmic
display.

Fig. 6. The relationship between frequency and phase atdifferent blood glucose concentrations.

results. With the change of blood glucose concentration,
the phase value is negatively correlated with the phasesize before the corner frequency; after the cornerfrequency, it is positively correlated with the phase size.
These trends indicate it is unreliable to measure theimpedance or phase values of human tissue individually.
Because the impedance of a single frequency point
carries less physiological and pathological information,
there are many factors affected.

The corresponding values of Cmem and Cglu in the model
simulation are shown in Table 1.
The data obtained by simulation in MATLAB, for eachCmem, 0 we obtain the corresponding impedance andphase values at different blood glucose concentrations asshown in Fig. 5 and Fig. 6.
As it can be seen from the above figures, as the blood
glucose concentration increases, the impedance value atthe same frequency decreases. However, as the blood
glucose concentration increases, the speed of impedancevalue decreases slowly. At the same time, the change ofimpedance value is not obvious in the blood glucose
range of the human body. The impedance changes
reflecting the changes of blood glucose, which are highly
demanding on the accuracy of the device, and someinfluencing factors can cause errors in the measured

3.3. Simulation experiments
Use the Wayne Kerr 6500B impedance analyzer to test
the human equivalent circuit model shown in Fig. 7. Set
Cmem as 20, 21, 22μF separately. The blood glucose
concentration corresponding to the measured capacitance© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

331

Research on a Non-Invasiveis shown in the following Table 2. The frequency range
is 20Hz-10MHz.

The Cmem value is brought into the human bodyequivalent circuit model shown in Fig. 3, and theimpedance measurement is simulated at different blood
glucose concentrations. The impedance value (a) and thephase value (b) thus obtained are as shown in thefollowing Fig. 8.

Table 2. Blood glucose concentration (Cglu)
corresponds to cell membrane capacitance (Cmem)

Cmem(μF)20
2122Cglu(mmol/L)
7.651786
12.31322
20.73387(a)Fig. 7. The relationship between frequency and impedance at
different Capacitance value.

In order to reflect the trend change, the following Fig.
intercepts the data before 1 MHz. It can be seen that as
the blood glucose concentration increases, the impedancevalue decreases.
4.

(b)Fig. 8. The relationship between frequency, impedance andphase under diagnostic criteria.

Data processing and result analysis

4.2. Analysis based on Cole-Cole theory

Table 3. Diagnostic criteria of blood glucose values and corresponding Cmem before and after meal

Cglu
minimum(mmol/L)

Cglu
maximum(mmol/L)
4.6

Cmem
minimum(μF)

Cmem
maximum(μF)
18.9619

Normal glucose

4.6

7.8

18.961920.0406

Abnormal glucose metabolism

7.8

11.1

20.0406

20.7854

Diabetes

11.1

zHypoglycemia

20.7854
In order to accurate describe the equivalent circuit modelof biological tissues, Cole KS and Cole RH haveconducted in-depth research in this regard. They summed
up a set of actual biological tissue electrical impedancetheory, the Cole-Cole theory [9-10], and used it to describe
the dielectric model of biological tissue. The basic

4.1. Blood glucose diagnostic standard simulation
Different blood glucose concentration standards
corresponding to different diagnostic results are obtained
and shown in Table 3. According to formula (2), thecorresponding Cmem values are calculated.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan332

Tong Yin , Xiaoyan Chen , Meng Du, Hongyi Yincontent of the Cole-Cole theory is that the trajectory of
the electrical tissue's electrical impedance in the complex
plane is a circular arc in the fourth quadrant, and thecenter of the circle is not on the real axis but in the first
quadrant.
The measurement of bio-impedance is essentially toobtain the characteristic parameters in the impedance
trajectory map. It can be seen from the above analysis thatas long as an excitation electric signal of a certain
frequency is applied to the biological tissue and theoutput signal flowing through the biological tissue issimultaneously measured, the impedance amplitude at
the frequency can be calculated by the relevant
demodulation algorithm. If the excitation power signal isoperated in a sweep mode, the entire impedance circle
can be obtained by curve fitting. On the impedance plot,
the impedance value at any frequency point can beobtained. Based on this theory, the Cole-Cole chart andits electrical characteristic parameters of the humanequivalent circuit model are obtained from the real andimaginary parts of the impedance and the phase andimpedance values. Corresponding Cole-Cole biologicaltissue electrical impedance characteristic equation is.
Z=

∞+

∞

higher blood glucose concentration, the smaller value offc corresponding to the impedance spectrum.
A polynomial fit was performed by MATLAB to obtaina correspondence between Cglu and fc:= −0.001+ 0.0316− 0.4229+ 9.7647 (6)The coefficient of determination R2 = 0.9999, whichmeans that the fitting effect is excellent. The
characteristic frequency fc in the impedance spectrum canreflect the change of human blood glucose concentration
to a certain extent, and it is possible to realize noninvasive blood glucose monitoring based on impedance
spectrum method.

Fig. 9. The Cole-Cole impedance circle
Table 4. Impedance spectrum characteristic parameters andcorresponding blood glucose concentration(5)

The Im axis represents the imaginary part of the
impedance, and the Re axis represents the real part value.
fc represents the frequency value when the imaginary partis maximum. R0 represents the electrical impedance of
biological tissue at a frequency of zero. R∞ represents theelectrical impedance of biological tissue at infinite
frequency. These scatter points in the figure arecalculated at frequencies from 10 Hz to 20 MHz, in which
the impedance changes from 150Ω to 1250Ω. Fitted intoa circle by these scatter plots and it can be seen that it isnormative. According to the measured real and imaginaryparts of the impedance, the Cole-Cole circle is fitted
through Python using the weighted average method, andthe following Fig. 9 shows the Cole-Cole circle fitted at
Cglu=5 mmol/L.
The change trend of the characteristic parameter fc has agreat correlation with the change of the blood glucose
concentration. From the data in the Table 4, themagnitude of fc has an inverse correlation with themagnitude of the blood glucose concentration. The

Diagnosisfc(Hz)

Cglu(mmol/L)

Hypoglycemia
Hypoglycemia
Normal glucose
Abnormal glucose metabolism
Abnormal glucose metabolism

9.04
8.511
8.165
7.92
7.738

2
4
6
8
10

Diabetes

7.597

12

5. Conclusion
As the glucose concentration changes, the binding of
glucose to the transporter changes the capacitance of thecell membrane. The transport of glucose into cells bytransporters is a macroscopic change in the impedance ofthe human body. Considering the skin contacting
impedance and cell membrane conductivity feature, asimulate circuit is designed and connected in serial withthe measuring loop. The measured data were analyzed by
an improved Cole-Cole model to derive the relationshipbetween the impedance and the blood glucose
concentration. The results show that the electrical
impedance is strong negative correlation to blood glucose
concentration, especially the characteristic frequency fc,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan333

Research on a Non-Invasive

volume fraction and medium composition[J]. European
Biophysics Journal, 1994, 23 (3), 207-215.
11. K. S. Cole, R. H. Cole, Dispersion and absorption indielectrics I. Alternating current characteristics[J]. The
Journal of chemical physics, 1941, 9(4), 341-351.
12. K. S. Cole, R. H. Cole, Dispersion and absorption indielectrics II. Direct current characteristics[J]. The Journalof Chemical Physics, 1942, 10(2), 98-105.

as the blood glucose concentration increases, theimpedance value decreases. The conclusion of thisresearch provides a theoretical basis for non-invasive
blood glucose surface-measuring.
Acknowledgment
The authors would like to thank the EIT research groupfrom Tianjin University of Science and Technology. This
paper was supported by a grant from the National Natural
Science Foundation of China (Grant No.61301246) anda grant from the Science and Technology Project ofTianjin (12JCYBJC19300). It was also supported by the
Reform in Education from Tianjin Municipal Education
Commission of China (171005704B).
References
1. A. G. A. Aggidis, J. D. Newman, G. A. Aggidis,
Investigating pipeline and state of the art blood glucose
biosensors to formulate next steps[J]. Biosensors andBioelectronics, 2015, 74, 243-262.
2. D. Klonoff, Noninvasive blood glucose monitoring.
Diabetes Care 20, 1997, 433-443.
3. T. Koschinsky, L. Heinemann, Sensors for glucose
monitoring: technical and clinical aspects. Diabetes Metab,
2001, 17, 113-123.
4. I. Harman-Boehm, A. Gal, A. M. Raykhman, et al.,Noninvasive glucose monitoring: a novel approach[J].
Journal of Diabetes Science and Technology, 2009, 3(2),253-260.
5. A. Caduff, E. Hirt, Y. Feldman, Z. Ali, L. Heinemann,First human experiments with a novel non-invasive, nonoptical continuous glucose monitoring system[J].
Biosensors and Bioelectronics, 2003, 19(3), 209-217.
6. Y. Hayashi, L. Livshits, A. Caduff, et al., Dielectric
spectroscopy study of specific glucose influence on humanerythrocyte membranes[J]. Journal of Physics D: Applied
Physics, 2003, 36(4), 369.
7. P. Shap, X. D. Zhang, X. C. Zhang, Biological Tissue
Impedance Model for Blood Glucose Monitoring (Acta
Laseｒ Biology Sinica, China). 2016, 25(03), 250-256.
8. L. Livshits, A. Caduff, S. Talaryms, The role of GLUT1 in
the sugar-induced dielectric response of humanerythrocytes[J]. The Journal of Physical Chemistry B,
2009, 113 (7), 2212-2220．
9. Y. D. Chen, Y. T. Cao, Y. M. Gao, S. H. Pun, P. U. Mak,
M. I. Vai, Y. R. Li, M. Du, Experimental study on contact
impedance of intra-body communication transmitter[J].
Journal of Biomedical Engineering Research, 2010, 29(1),13-17.
10. H. Beving, L. E. G. Eriksson, C. L. Davey, et. al.,
Dielectric properties of human blood and erythrocytes atradio frequencies (0.2-10 MHz); dependence on cell

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan334

Synchronization of Novel 4D Chaotic Systems with Different Control Laws
Hong Niu

College of Electronic Information and Automation,
Tianjin University of Science & Technology, China
E-mail: spots@163.com
www.tust.edu.cn

Abstract
In this paper, synchronization of novel four-dimensional(4D) autonomous chaotic systems is studied. Two different
control laws are presented, which are based on the strictly positive realness of transfer function matrix and the centertranslation method respectively. The characteristics of the two control laws, as well as the differences between them,are proposed. Some relevant numerical simulation results, such as the curves of the corresponding synchronization
state variables and the errors, are given to illustrate the feasibility and effectiveness of the two control laws.
Keywords: novel 4D chaotic system, chaos synchronization, strictly positive realness of transfer function matrix,
center translation method

1. Introduction

where a = 25, b = 3, c = 18, m = 19 and h = 14. The phase
portraits of the system (1) are shown in Fig. 1(a)-(f).

Chao synchronization has been one of the major subjects
in nonlinear dynamics.1-4 It has great engineeringsignificance and application prospect in signal processing,secure communication and so on.
In this paper, the synchronization of two identicalnovel 4D chaotic systems with different initial values isstudied via two different control laws based on thestrictly positive realness of transfer function matrix andthe center translation method.
2. Review of the Novel 4D Chaotic System

Fig. 1. Phase portraits of the novel 4D chaotic system: (a) x-y;(b) x-z; (c) x-w; (d) y-z; (e) y-w; (f) z-w

The novel 4D chaotic system has been discussed in Ref.
5. It is formulated as

3. Chaos Synchronization Based on Strictly
Positive Realness of Transfer Function Matrixx

a y x ,yc x

z

mx y hz ,w

xy bw,

y3.1. Corollary and Theoremz xw,

Corollary 1.1-3 Consider the system

1x

Ax Bu,

2

y

C x.

3

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

335

Hong Niu

Let G(s) = C(sI-A)-1B be a p×p transfer function matrix,
where I is a n×n identity matrix, and (A,B) is controllableand (A,C) is observable. Then, G(s) is strictly positive
real if there exist matrices P = PT > 0 and Q = QT < 0such thatbe the synchronization error and us = 0. Then the error
system is formulated ase1

a e2 e1

uc1 ,e2

c e1 e2e3

x2 w2

AT P PA Q,

4e3

me1 e2 he3 uc 3 ,

T

5

e4

x2 y2

B P C.

e

Ae B u uc ,

6

y

Ce ,

7

uc

Ky

KCe,

u

8

A

where u is the nonlinear part which is packaged as theexternal input of the system (6) and uc is the controller tobe designed. The system (6)-(7) is asymptotically stable
at the origin e = 0 if there exist matrices K, P = PT > 0
and Q = QT < 0, such that G(s) = C[sI-(A-BKC)]-1B isstrictly positive real.

BucConsider the system (1) as the drive system. The response
system is given by

x2

a y2

x2

us1 uc1 ,

y2

c x2

y2

z2

10

z2

mx2

y2 hz2 us 3 uc 3 ,w2

x2 y2 bw2 us 4 uc 4 ,

0

x2 w2

xw

0 x2 y2

a

a

0

0

c
m

c
1

1
h

0
,
0

0

0

0

b

T

xy

,

I,

uc1 uc 2

uc 3

uc 4

T

KCe

k1e1

k2 e2

k3e3

k4 e4

T

, 11

where k1, k2, k3, k4 ≥ 0. Then,x2 w2 us 2 uc 2 ,

A BKC

9

where

us 2

be4 uc 4 .

xywhere I is a 4×4 identity matrix. Let P = PT = I. From (5),C = BTP = I. Obviously, ((A-BKC),B) is controllable and((A-BKC),C) is observable.
Let the synchronization controller uc be

3.2. Design of Synchronization Controller

u s1

uc 2 ,

Hence,

Theorem 1.1-3 Consider the system

us

xw

us3

us 4

T

uc 4

T

T

P P A BKC

f11

f12

f13

0

f 21
f 31

f 22
0

0
f 33

0
0

0

0

0

f 44

F,

where

and

uc1 uc 2

uc

uc 3

are structure compensator and synchronization controller
to be designed. The function of the structure compensator
us is to compensate the structure difference between thedrive and the response systems so that the two chaotic
attractors can be synchronized under the control of thesynchronization controller uc.
Let

e

e1 e2

e3

e4

x2

y2

y

x

T

z2

z

w2 w

f11

2 k1 a ,f 22

2 k2 c ,f 33

2 k3 h ,f 44

2 k4 b ,

f12

f 21

a c,

f13

f31

m.

Comparing with the matrix Q in Ref. 5 which isformulated as

T© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

336

Synchronization of Novel 4DQ

q11 q12
q21 q22

q13
q23

0
0

q31
0

q33
0

0
q44

q32
0

axis t expresses the solution interval of differential
equations, so the variable t is a dimensionless quantity.
The definition of t is all the same in this paper. From Fig.
2 and Fig. 3, it can be seen that the errors e1, e2, e3 and e4
converge to zero asymptotically and rapidly and the statevariables are synchronized well. It demonstrates that the
response system (9) and the drive system (1) have been
synchronized under the control of the synchronization
controller uc.

,

where

q11

2n1 k1 a ,q22

2n2 k2 c ,q33

2n3 k3 h ,q44

2n4 k4 b ,

q12

q21

an1 cn2 ,

q13

q31

mn3 ,

q23

q32

n2 n3 .

It is easy to find that the matrix F is totally equal to thematrix Q, when n1 = n2 = n3 = n4 = 1. It has been proventhat the matrix Q is negative definite when
2

k2

h a c
4ah m 2

2

a c
4a

c

c

while k1 = k3 = k4 = 0 under the condition that n1 = n2 = n3
= n4. Therefore, it can also make the matrix F negative
definite. From Corollary 1 and Theorem 1, the transfer
function matrix of system (10) G(s) = C[sI-(A-BKC)]-1B
is strictly positive real, and the error system (10) will be
asymptotically stable at the origin e = 0 such that thestates of the response system (9) are synchronized withthose of the drive system (1) under the control of thesynchronization controller uc.
Let k2 = 45 and k1 = k3 = k4 = 0, and substitute them into
the synchronization controller uc, thenuc

uc1 uc2
0

45e2

uc3

uc4
T

Fig. 2.

Curves of errors

T

0 0 .

12

3.3. Numerical Simulation

Fig. 3.

Remark 1. The initial values of the drive system (1) andthe response system (9) are (x0, y0, z0, w0) = (1, 1, 1, 1)
and (x20, y20, z20, w20) = (5, 4, 7, 8) respectively in thispaper.

Curves of synchronized state variables

4. Chaos Synchronization Based on CenterTranslation Method
4.1. Formulation of Error System

The curves of the errors and the synchronized statevariables are shown in Fig. 2 and Fig. 3. The horizontal

The error system is formulated as3-4© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

337

Hong Niue1

a e2 e1

uc1 ,e2

c e1 e2

e3 e1e4 uc2 ,e3

me1 e2 he3 uc3 ,e4

e1e2 be4 uc4 ,compensator us and the synchronization controller uc.
Comparing Fig. 4 and Fig. 5 with Fig. 2 and Fig. 3, it canbe found that the convergence time in Fig. 4 and Fig. 5 isa little longer than that in Fig. 2 and Fig. 3.

13

where the synchronization controller uc is the same asthat in (11). In fact, a structure compensator us is added
to the response system (9), such that the error system (13)is equivalent to the response system (9) whose center is
moved to X = [x y z w]T.
4.2. Design of Synchronization Controller
Comparing the error system (13) with the controlled
system (2) in Ref. 5, it can be found that theirformulations are identical. It has been confirmed in detailin Ref. 5 that the system, such as the error system (13), is
globally asymptotically stable at the origin as long as
2

k2

h a c
4ah m 2ca c4aFig. 4.

2

Curves of errorsc,while k1 = k3 = k4 = 0. Thus, the synchronization controller
uc in the error system (13) is the same as that in (12).
4.3. Design of Structure Compensator
From (1), (9) and (13), the structure compensator us,which is difficult to find directly, can be calculated asusT

0 x2 w xw2 2 xw 0 2 xy x2 y xy2 .

In fact, from the construction procedure of the error
system (13), it can be found that the design of thestructure compensator can be omitted whatever the
structure of the drive system is. Thus, if the statevariables of the drive system and their derivatives are
known or can be estimated via state observer, thesynchronization controller uc can be designed from thestructure of the response system. It simplifies thesynchronization procedure.

Fig. 5.

Curves of synchronized state variables

5. Conclusion
The most difference between the two control laws is theglobal asymptotic stability of error system. The control
law based on the center translation method makes theerror system globally asymptotically stable at the origin,while the control law based on the strictly positive
realness of transfer function matrix only makes the error
system asymptotically stable at the origin. Whether the
nonlinear part of the error system could be packaged as
the external input, as described in Theorem 1, still needs
to be analyzed and proven mathematically. Meanwhile,

4.4. Numerical Simulation
The curves of the errors and the synchronized statevariables are shown in Fig. 4 and Fig. 5. The two figures
show that the errors e1, e2, e3 and e4 converge to zero
asymptotically and rapidly and the state variables aresynchronized well. It demonstrates that the response
system (9) and the drive system (1) have also been
synchronized under the control of the structure© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

338

Synchronization of Novel 4D

References

the proof of the control law based on the centertranslation method is rigorous and its design procedure ismuch more straightforward. However, a nonlinear
structure compensator is hidden in the formulation of theerror system of the control law based on the centertranslation method. It makes the circuit implementationof this control law complex and difficult. In contrast, themost advantage of the control law based on the strictly
positive realness of transfer function matrix is that there
is no structure compensator and the synchronization
controller is linear, so the circuit implementation of thiscontrol law is direct and easy. In brief, there are bothadvantages and disadvantages in the two control laws.
More researches would be established to conquer the
disadvantages of the two control laws.

1. Hong Niu and Guoshan Zhang, Synchronization of chaotic
systems with variable coefficients, Acta Physica Sinica,
62(3): 105-115, 2013.
2. Hong Niu, Guoshan Zhang and Jiankui Wang,
Synchronization of Chua's circuits based on strictly
positive realness, 26th Chinese Control and Decision
Conference, Changsha, China, 2014.5.31-6.2.
3. Hong Niu, Guoshan Zhang and Jiankui Wang, Chaos
synchronization of Chua's circuit and Lorenz system basedon strictly positive realness, Proceedings of the 33rd
Chinese Control Conference, Nanjing, China, 2014.7.287.30.
4. Hong Niu, On synchronization of 5D hyperchaotic systemand 3D chaotic system with variable coefficient,
Dynamical Systems and Control, 5(2): 41-47, 2016.
5. Hong Niu, Analysis and control of a novel 4D chaotic
system, The 2017 International Conference on Artificial
Life and Robotics, Miyazaki, Japan, 2017.1.19-1.22.

Acknowledgements
The research is supported by the Innovation Fund for theYoung Teachers of Tianjin University of Science &Technology.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

339

Chaotic characteristics analysis of fractional-order Liu system
Wengxin Shi *, Hongyan Jia

College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
E-mail: * 10161740075@qq.com

Abstract

The paper analyzes a newly reported fractional-order Liu system based on time-domain approximationand frequency-domain approximation, respectively, and chaotic characteristics of this system areinvestigated by the phase portrait diagram, Lyapunov exponents diagram and bifurcation diagram. The
results of the two approximation methods are consistent, which not only shows chaotic dynamics of thesystem is reliable, but also is very convenient to utilize chaotic dynamics of the system withfractional-order 0.9.
Keywords: Chaotic characteristic, fractional-order system, time-domain approximation,

frequency-domain approximation
1. Introduction

secrets of secure communication besides the parameters
and initial values of the chaotic systems 14-16. Therefore,
the research on fractional-order chaotic systems not only
help researchers to find more chaotic dynamics andenrich the theory of chaos, but also provide more suitablephysical model or tools for application such as securecommunication.
Generally speaking, computing the solutions offractional-order systems is very difficult, so someapproximation methods are always used to investigate thedynamics of fractional-order systems. So far as we know,
the time-domain approach and the frequency-domain
approach are two kind of common approximation
methods which are always adopted. In literature, somefractional-order systems have been investigated based on
frequency-domain
approach2,5,7,17,18
and
other
fractional-order systems have been investigated based ontime-domain approach4, 19-24. However, it is seldom that afractional-order system is simultaneously investigated toshow chaotic dynamics based on both of theapproximation methods. We find that it is insufficient toinvestigate chaotic characteristics of the fractional-ordersystems in either, because each of approximation

Since chaotic behavior was found to exit infractional-order Chua’s circuit in 19951, the theory andapplication of fractional-order chaos has been attracting
more and more interest of research in recent two
decades2-6. According to Poincare-Bendixon theorem, it
is impossible for a continuous autonomous integer-ordersystem to be found to be chaotic when the order of thesystem is less than three7-8. However, for many
fractional-order systems, the obvious advantage is thatchaotic dynamics can occur when the order of the system
is less than three, such as the fractional Rössler system 9,
the fractional modified Duffing system10, and so on 11-13.
Which provide an opportunity for researchers to findmore chaotic dynamics in some lower dimensional
systems. In addition, when studying securecommunications based on integer-order chaotic systems,
the parameters and initial values of the chaotic systems
are generally selected as key secrets of securecommunication. However, for fractional-order chaotic
systems, the remarkable advantage is the order offractional-order chaotic systems can also be used as key© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

340

Wenxin Shi, Hongyan Jia,

methods has own merits and demerits. Therefore, it isnecessary to use these two approximation methods toanalyze and display chaos for a fractional-order system.
Recently, a fractional-order system named as thefractional-order Liu system is reported, and its chaotic
attractor is found by phase portraits and Electronic
Workbench Basics simulation25-27. In this paper, thefractional-order Liu system is further discussed, thedifference from the research in25-27 is that chaotic
characteristics of the fractional-order Liu system areanalyzed in detail by showing the Lyapunov exponents
diagram and bifurcation diagram, based on time-domain
approximation and frequency-domain approximation,
respectively. The results of the two approximation
methods are consistent.

frequency-domain approximation when showing chaotic
behaviors of the fractional-order systems. Therefore, inorder to further analyze the chaotic characteristics, thesystem (1) is exclusively investigated based on bothtime-domain approximation and frequency-domain
approximation.
2.1 Bifurcation analysis based on time-domain
approximation
In this paper, the advised Adams-Bashforth-Moulton
method in Ref 4 and 24 is adopted to numerically analyze
chaotic behavior of system (1), and thus for the initialvalues x0 , y0 , z0 and step length h , on the basis of theabove method, the fractional-order four-wing system (1)can be written as following discrete form
xn 1

2. Bifurcation analysis for the fractional-orderLiu system

q

yn 1
zn 1

Recently, a fractional-order system was reported byZhang et al, and Chen et al, in 2007 and 2008, in Ref25-27,
respectively. The fractional-order Liu system is described
as
dqx
dt
dqy
dt
dqz
dt

f1 ( x, y , z )

a( y
bx

cxz ,

f 3 ( x, y , z )dzgx 2 ,y0z0nf ( x j , y j , z j )],j, n 1 1
j 0nh[ f 2 ( xnp 1 , ynp 1 , znp 1 )( q 2)j 0

hq[ f 3 ( xnp 1 , ynp 1 , znp 1 )
(q 2)

j 0

j, n 1f 2 ( x j , y j , z j )],(2)nf ( x j , y j , z j )],j, n 1 3

Where

x ),

f 2 ( x, y , z )hq[ f1 ( xnp 1 , ynp 1 , z np 1 )( q 2)

x0

(1)

xnp 1

x0

1 hq n(( n 1(q ) q j 0

j)q

(n

j ) q ) f1 ( x j , y j , z j ),

ynp 1

y0

1 hq n(( n 1(q ) q j 0

j)q

(n

j ) q ) f 2 ( x j , y j , z j ),

znp 1

z0

1 hq n(( n 1(q ) q j 0

j )q

(n

j ) q ) f 3 ( x j , y j , z j ),j ,n 1nq 1( n q )( n 1) q ,(nj 2) q 1

1,(nj )q 12(nj 1) q 1 ,

j

0

1

j

j

n 1

n ， i 1, 2,3

here the error for the time-domain approach is aboutO ( h r）, r min(2, 1 q) .
Fixing parameters b 40 , c 10 , d 2.5 , g 4 ,initial values x0 1, y0 1, z0 1 , and fractional-order
q 0.9 , based on time-domain approach, the Lyapunov
exponents diagram and bifurcation diagram of system (1)are obtained, as shown in Fig.1(a) and Fig.1(b),respectively. It can be seen that chaos exists in the system(1) with fractional-order q 0.9 for the parameter intervala [2.3, 20] in Fig.1. For a=6 , based on time-domain
approach, the phase portraits of chaotic attractor are
obtained to further show chaotic dynamics of system (1),as shown in Fig.2.

Where a, b, c, d , g R , ( x, y, z ) R 3 and 0 q 1 ,parameter q is called as the fractional order.
Although its chaotic attractor has been found by
discussing phase portraits and Electronic Workbench
Basics simulation in Ref25-27, it is insufficient to analyze
the chaotic dynamics underlying the system, therefore,
here we further discuss its chaotic characteristic.
As described above, when studying somefractional-order systems numerically, the time-domain
approximation and the frequency-domain approximation
are two kinds of mainly methods to be adopted toinvestigate chaotic dynamics, and each of them has its
own merits and demerits. Generally, compared with thetime-domain approximation, the frequency-domain
approximation is more practical to implement and utilize
the fractional-order systems. However, the time-domain
approximationismore
accuratethanthe© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan341

Chaotic characteristics analysis of

2.2 Bifurcation analysis based on frequency-domain
approximation2
Lyapunov exponents

L

1

1

L2

0

In this paper, the frequency-domain approach 7 is adopted
as well, which was also used in literatures 25-27, and theapproximation function with an error of 2 dB is listed asfollows

-1
-2

L

3

1
s 0.9

-3
-4
0

5

(a)

10
a

15

20

(3)

Under the same condition, b 40 , c 10 , d 2.5 ,g 4 , initial values x0 1, y0 1, z0 1 , andfractional-order q 0.9 , based on frequency-domain
approach, the Lyapunov exponents diagram andbifurcation diagram of system (1) are also obtained, as
shown in Fig.3(a) and Fig.3(b), respectively. It can bealso seen that chaos exists in the system (1) withfractional-order q 0.9 for the parameter intervala [2.8, 20] in Fig.3. For a=6 , based on
frequency-domain approach, the chaotic attractor has
been obtained to further show chaotic dynamics in thesystem (1), as shown in Fig.4.

Lyapunov exponents diagram(b) Bifurcation diagram
Fig.1 Lyapunov exponents diagram and bifurcation diagramof system (1) based on time-domain approach

Lyapunov exponents

4
L

1

2

L

2

0

-2

L

-4
0

5

(a)
(a) Phase portrait in x

2.2675( s 1.292)( s 215.4)( s 0.01292)( s 2.154)( s 359.4)

10
a

3

15

20

Lyapunov exponents diagram

y plane(b) Bifurcation diagram
Fig.3 Lyapunov exponents diagram and bifurcation diagramof system (1) based on frequency-domain approach(b) Phase portrait in y z plane
Fig.2 Chaotic attractors of system (1) based on time-domain
approach© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan342

Wenxin Shi, Hongyan Jia,

Acknowledgements

10

The research is partly supported by the Research Fund for
the National Natural Science Foundation of China (Grant
No.61573199). It is also supported by the Young
Scientists Fund of the National Natural Science
Foundation of China (Grant No.11202148).

5

y

0

-5

References
-10
-5

0
x

(a) Phase portrait in

5x

1. Hartley T T, Lorenzo C F, Killory Qammer H. Chaos in a
fractional order Chua's system [J]. IEEE Transactions on
Circuits & Systems I Fundamental Theory & Applications,
1995, 42(8): 485-90.
2. Grigorenko I, Grigorenko E. Chaotic dynamics of thefractional Lorenz system [J]. Physical Review Letters,
2003, 91(3): 034101.
3. Deng W H, Li C P. Chaos synchronization of the fractional
Lü system [J]. Physica A Statistical Mechanics & Its
Applications, 2005, 353(1): 61-72.
4. Diethelm K, Ford N J, Freed A D. A predictor-corrector
approach for the numerical solution of fractional
differential equations [J]. Nonlinear Dynamics, 2002,
29(1-4): 3-22.
5. Lu J G, Chen G R. A note on the fractional-order Chensystem [J]. Chaos Solitons & Fractals, 2006, 27(3): 685-8.
6. Dadras S, Momeni H R. Control of a fractional-ordereconomical system via sliding mode [J]. Physica A
Statistical Mechanics & Its Applications, 2010, 389(12):2434-42.
7. Ahmad W M, Sprott J C. Chaos in fractional-orderautonomous nonlinear systems [J]. Chaos Solitons &Fractals, 2003, 16(2): 339-51.
8. Lu J G. Chaotic dynamics of the fractional-order Lüsystem and its synchronization [J]. Chinese Physics B,
2006, 59(2): 301-5.
9. Li C, Chen G. Chaos and hyperchaos in thefractional-order Rössler equations [J]. Physica A Statistical
Mechanics & Its Applications, 2004, 341(1–4): 55-61.
10. Ge Z M, Ou C Y. Chaos in a fractional order modified
Duffing system [J]. Chaos Solitons & Fractals, 2007,
34(2): 262-91.
11. Huang X, Zhao Z, Wang Z, et al. Chaos and hyperchaos infractional-order
cellularneuralnetworks[J].
Neurocomputing, 2012, 94(3): 13-21.
12. Hosseinnia S H, Magin R L, Vinagre B M. Chaos infractional and integer order NSG systems [M]. Elsevier
North-Holland, Inc., 2015.
13. Aghababa M P. Chaotic behavior in fractional-orderhorizontal platform systems and its suppression using afractional finite-time control strategy [J]. Journal of
Mechanical Science & Technology, 2014, 28(5): 1875-80.
14. Sheu L J. A speech encryption using fractional chaotic
systems [J]. Nonlinear Dynamics, 2011, 65(1-2): 103-8.

y plane

10
8

z

6
4
2
0
-10

-5

0
y

5

10

(b) Phase portrait in y z plane
Fig.4 Chaotic attractors of system (1) based on
frequency-domain approach

By comparing the numerical results from thefrequency-domain approach and those from thetime-domain approach, respectively, we conclude thatchaos indeed exists in the chaotic dynamic of system (1)with fractional-order 0.9, no matter whether adopting
frequency-domain approach or time-domain approach.
What’s more, the numerical results from the two
approaches are very coincident except for some tiny
difference.
3. conclusion
The paper analyzes a newly reported fractional-order Liu
system based on time-domain approximation andfrequency-domain approximation, respectively, andchaotic characteristics of this system are investigated by
the phase portrait diagram, Lyapunov exponents diagram
and bifurcation diagram. And the results of the two
approximation methods are consistent. The work in thispaper not only show chaotic dynamics of system (1) isreliable, but also is very convenient to utilize chaotic
dynamics of system (1).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan343

Chaotic characteristics analysis of

21. Deng W H. Numerical algorithm for the time fractional
Fokker-Planck equation [J]. Journal of Computational
Physics, 2007, 227(2): 1510-22.
22. Zhou P, Wei L J, Cheng X F. A novel fractional-order
hyperchaotic system and its synchronization [J]. Chinese
Physics B, 2009, 18(7): 2674-9.
23. Petras I. Chaos in the fractional-order Volta's system:
modeling and simulation [J]. Nonlinear Dynamics, 2009,57(1-2): 157-70.
24. Li C P, Peng G J. Chaos in Chen's system with a fractional
order [J]. Chaos Solitons & Fractals, 2004, 22(2): 443-50.
25. Chen X R, Liu C X, Wang F Q, et al. Study on thefractional-order Liu chaotic system with circuit experimentand its control [J]. Acta Physica Sinica, 2008, 57(3):1416-22.
26. Lu J J, Liu C X. Realization of fractional-order Liu chaotic
system by circuit [J]. Chinese Physics B, 2007, 16(6):1586-90.
27. Xu Z, Liu C X. Realization of fractional-order Liu chaotic
system by a new circuit unit [J]. Chinese Physics B, 2008,17(11): 4033-8.

15. Kiani-B A, Fallahi K, Pariz N, et al. A chaotic securecommunication scheme using fractional chaotic systems
based on an extended fractional Kalman filter [J].
Communications in Nonlinear Science & Numerical
Simulation, 2009, 14(3): 863-79.
16. Sheu L J, Chen W C, Chen Y C, et al. A two-channel
secure communication using fractional chaotic systems [J].
World Academy of Science Engineering & Technology,
2010, 72(1): 248.
17. Jia H Y, Chen Z Q, Qi G Y. Topological horseshoe
analysis and circuit realization for a fractional-order Lüsystem [J]. Nonlinear Dynamics, 2013, 74(1-2): 203-12.
18. Charef A, Sun H H. Fractal system as represented bysingularity function [J]. IEEE Transactions on AutomaticControl, 2002, 37(9): 1465-70.
19. Tavazoei M S, Haeri M. Limitations of frequency domain
approximation for detecting chaos in fractional ordersystems [J]. Nonlinear Analysis Theory Methods &Applications, 2008, 69(4): 1299-320.
20. Tavazoei M S, Haeri M. Unreliability of frequency-domain
approximation in recognising chaos in fractional-ordersystems [J]. Let Signal Processing, 2007, 1(4): 171-81.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan344

Design and Analysis of Multi-robot grouping aggregation algorithm
Huailin Zhao

School of electrical and Electronic Engineering,Shanghai Institute of technology, No. 100, HaiquanRoad, Fengxian District Shanghai 201418,China
Zhen Nie

School of electrical and Electronic Engineering,Shanghai Institute of technology, No. 100, HaiquanRoad, Fengxian District Shanghai 201418,China
E-mail: zhao_huailin@yahoo.com, hbniezhen@foxmail.com
www.sit.edu.cn

Abstract:
In this paper, the grouping aggregation problem of multi-robot in simple environment is studied.
The grouping algorithm and aggregation algorithm are mainly discussed. Based on two clustering algorithms, K-means and K-medoids, two grouping algorithms are designed. And the noise
immunity and grouping rapidity of these two grouping algorithms are analyzed. Aiming at the
problem of aggregation, three kinds of aggregation algorithms are proposed. Taking the center
aggregation control as an example, the cooperation and control of each group of robots arestudied. Finally, the simulation results of MATLAB shows that the multi-robot grouping andaggregation algorithm is effective.
Keywords: multi-robot; clustering; grouping aggregation; swarm intelligence

1.Introduction
The multi-robot collaboration system is a frontier
topic in the field of artificial intelligence [1]. Fromthe control point of view, the multi-robot collaboration system is a typical distributed control system[2-4], and its goal is to build large complex systems
which included software and hardware systems into
small, easy-to-manage systems that communicate
and coordinate with each other. The research ofmulti-robot collaboration system involves theknowledge, goals, skills, planning of robots andhow to make robots coordinate actions to solve
problems, and the idea of swarm intelligence is345emphasized. The multi-robot collaboration systemforms a complex system through communication,
coordination, and cooperation among autonomous
robots. The major applications include formation
flying of multi-UAV, clusters of warehousing logistics robot, competitions of robot team, large-scalerobotic rescue systems, etc. Some Examples in nature are dancing fish, flying birds, and so on.
Aggregation is the basic problem of multi-robot
system consistency [5-6], and it is also an optimizing
measure [7], and the grouping aggregation is a special case of aggregation problem. Group aggregation is a motion planning of the robots randomly
distributed in space, which are divided into many

Huailin Zhao, Zhen Nie

groups according to a specific grouping rule, andthen each group of robots adjust the motion states
through communication and cooperation between
each other, finally they gather together, or in thedesired area to complete the aggregation. In the end,
all the robots formed a multi-group gathering situation, the idea of swarm intelligence is emphasized.
After the comparative analysis of two clustering
algorithms, K-means and K-medoids, an efficientgrouping aggregation algorithm is designed basedon K-means algorithm. The feasibility of multi-robot grouping aggregation is verified by
MATLAB simulation experiment. And the multi-robot distributed coordination and grouping aggregation control are realized.
2.Analysis and design of grouping algorithm
Grouping aggregation is divided into two steps,
grouping task and aggregation task. The groupingtask is to complete the group division of each robot
according to a specific grouping rule. This study
designed and analyzed a grouping algorithm basedon two clustering algorithms, K-means andK-medoids.
(1) The K-means algorithm is a clustering algorithm
which is the simplest and most commonly used inunsupervised learning. It is a center-based clustering algorithm [8-9]. The K-means algorithm completes the grouping task by minimizing the distance
between each sample and the center of the cluster.
The initial centers of each cluster were randomly
selected.
In the Euclid space, a set of n-dimensional samples
X
x1 , x2 , x3 xi xn ( representing n robots) is
given, where xi
R n . The K-means algorithm
dividesallsamplesintokclustersC
ci ,i 1,2, ,k . The Euclidean distance ischosen as the criterion for the distance, the equation
is expressed as follows.

d ( a , b)

a b

(1)

where a,b represents two points in the Euclid space,which are two n-dimensional data.
In determining the similarity of the samples, thesamples are divided into the clusters with the highest similarity according to the following equation(the similarity is based on the distance between thesamples and the centers, and the smaller the distance, the higher the similarity).

k

arg minci

i 1 xj X

C

xj

μi

2

(2)

Where i is the center of ith cluster.
The k-means algorithm needs to re-divide the cluster and update the centers iteratively. When the
Square Error Sum is converged, the K-means algorithm ends. The equation of Square Fitting Error
Sum is given by

J

Where dijkn

i 1j 1

dij x j1, x jci

0, x j

ci

2
i

(3)

。

(2) The K-medoids algorithm uses the point closest
to the cluster center as the center of the cluster [10-11].
There are many forms of K-medoids algorithm, themost classic one is the PAM algorithm. The basic
idea is:Firstly, k samples are randomly selected from nsamples as k initial centers; then Euclidean distanceis calculated according to the equation (1), and theinitial clustering is performed according to the principle of minimum distance; Finally, center points
are replaced by non-center points repeatedly to improve the clustering quality. The post-replacement
clustering quality is estimated by a cost function (4),which measures the average dissimilarity degree
between the object and its reference object. In theprocess of substitution, the least cost is selected for
replacement, and cluster again until the cluster nolonger changes. The cost function is expressed asfollows:

TCih

n
j 1

C jih

(4)

Where TCih is the total cost after the center point oi
replaced by non-central point oh ; C jih is the cost ofo j after oi is replaced by oh . If the total cost is negative, it means that the average dissimilarity degree
within the clusters becomes smaller after the replacement. So oi can be replaced by oh . Otherwise, it cannot be.
Under the requirement, to achieve the grouping aggregation, the grouping is the premise of the aggregation, and the quality of the grouping directly affects the aggregation of the robots. Therefore, thetwo grouping algorithms are further studied, mainly
analyzing the noise immunity and the calculation
rapidity.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

346

Multi-robot grouping aggregation algorithm(1) The analysis of the noise immunity
The noise immunity is the first factor to be considered in system design, and it is the key to systemsecurity under abnormal conditions. In order to test
the noise immunity of the two algorithms, 20 samples were set to simulate 20 robots randomly placedin space, and they were divided into two groups by
K-means and K-medoids respectively. Another abnormal sample was added as noise.
As mentioned earlier, the K-means algorithm is a
center-based clustering algorithm, so the position of
the center is an important criterion for the quality of
the grouping. The experimental results are shown in
Fig.1. Fig.1(a) shows the results of grouping without noise. The samples were well divided into twogroups by the k-means algorithm, and the grouping
effect was well. Fig.1(b) shows that when the noise
point c appeared, although the grouping task wascompleted, the center was obviously deviated. If thenoise is more intense, the center deviation would belarger. Due to the deviation of the center, thegrouping of the two samples, a and b, has changed.
If the noise is more intense, the grouping results ofmore samples would be affected. It can be seen that
the K-means algorithm is sensitive to noise. If thesample contains abnormal points, it would seriously
affect the grouping result.

result with noise

The K-medoids algorithm uses the point closest to
the cluster center as the center point of the cluster,thus effectively avoiding the phenomenon that thecenter of the K-means algorithm changes due to thedeviation of the noise. The simulation of grouping
results is shown in Fig.2. Fig.2(a) shows thegrouping result without noise, and the grouping
result was good. Fig.2(b) shows the grouping result
when the noise point c appeared, and the groupingtask was still completed. Because the center was
instead of the sample closest to the center. Therefore,
there is no center deviation, and the grouping result
was not affected. It can be seen that the K-medoids
algorithm has strong noise immunity.

(a)

c

a
b

(b)
(a)

Fig.2. The test of noise immunity of K-medoids algorithm:(a) the grouping result without noise; (b) the grouping
result with noise

c

a
b

(b)
Fig.1. The test of noise immunity of K-means algorithm:(a) the grouping result without noise; (b) the grouping(2) The analysis of the calculation rapidity
The K-means algorithm a center-based clustering
algorithm. In the calculation process, only the average of the robot coordinates should be calculated.
The time complexity is O nkt for k<<n, t<<n.
Where n is the number of samples, k is the numberof clusters, and t is the number of iterations. The
K-medoids algorithm uses the point closest to thecluster center as the center of the cluster. In the calculation process, the distance and total cost between
any two points are constantly calculated. The timecomplexity is O n2 kt for k<<n, t<<n. Where n is

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

347

Huailin Zhao, Zhen Nie

the number of samples, k is the number of clusters,
and t is the number of iterations. Obviously, thetime complexity of the k-medoids algorithm isgreater than k-means algorithm. When a large
number of samples or a large k should be deal with,the calculation speed will be slow, which seriously
affects the grouping efficiency.
Through the comparison and analysis, the K-means
algorithm is simple and fast. Although the noise
immunity is not as good as the K-medoids algorithm, when the sample is large, a few noise has
little effect on the K-means algorithm. The
K-medoids algorithm has strong noise immunity,
but when there are a large number of samples, thecalculation speed is slow due to the large time complexity. Since this grouping aggregation will processa large amount of data, the K-means algorithm wasdecided.

munication, and the result of the formation control
is not satisfactory.
3) Selecting the center of group as the reference
point, as shown in Fig.3(c). This selection method iscalled center algorithm. During the aggregation
process, multiple robots form a geometric figure.
According to the geometric figure, the geometriccenter is calculated. Each robot uses takes the centeras a reference point to adjust the direction and speed,
and move to the reference point. During the aggregation movement, the system constantly updates thereference points, and finally all robots reach a same
point. the aggregation task is end. This algorithm issimple and practical. The system is convergent. The
advantages of the distributed control system arefully reflected.

3.Analysis and design of aggregation algorithm
As mentioned earlier, the aggregation problem isthat multi-robots randomly distributed in the spaceadjust the motion states through communication andcooperation between each other, finally they gather
together, or in the desired area to complete the aggregation. Aiming at the aggregation task requirements of this topic, three aggregation control algorithms were designed and analyzed in this paper:
1) Selecting a robot from each group as leaders,
as shown in Fig.3(a). The basic idea of this algorithm is to pre-set a leader robot, and other robots
are called followers [12-13]. In the process of aggregation, the leader controls the motion path, and thefollowers keep a certain angle and distance with theleader, finally all robots aggregate at the position of
the leader. After careful analysis, the aggregation
has some defects. On the one hand, because thealgorithm doesn’t make full use of the advantagesof distributed systems, when the leader robot goes
wrong, it will cause all robots fail to aggregate. On
the other hand, because the control of the system isrelatively simple, the follower or leader robot may
be lost in the process of motion in the complicatedreality environment. The aggregation task failed.
2) Selecting the neighboring robot as the reference point, as shown in Fig.3(b). Each robot finds
its own target position in the whole aggregation
movement through the position information ofneighboring robots. The limitation of this method isthat a large amount of communications is required,and each robot needs to spend more time for com-

(a)

(b)

(c)
Fig.3. Three different aggregation algorithms:(a) Selecting a robot from each group as leaders; (b) Selecting the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

348

Multi-robot grouping aggregation algorithm

neighboring robot as the reference point; (c) Selecting thecenter of group as the reference point.

Comparing the three aggregation algorithms, thisstudy intends to use the center algorithm for aggregation control. The center calculation equations are
expressed as follows:
n

n

xc

xi / n ， y ci 1yi / n(5)i 1

Where（ xi , yi ) is the 2-D coordinate of the current
position for the ith robot, and ( xc , yc ) is the center
coordinate.
During the aggregation process, the system periodically updates the center position and notifies all
robots. Each robot makes motion planning according to its current position and new center position.
The motion planning equation is given by

X i (k1)

X i (k)

vi (k

1) T

(6)

Where T is the sampling period, Xi is the motion
state of ith robot, vi is the velocity of ith robot.
4.Calculation process of grouping aggregation
algorithm
As mentioned earlier, to complete the packet aggregation task, the first thing should be done isgrouping operation, and the next is the aggregation
operation. The specific process is as follows.
According to the grouping requirements, the robot
updates the position of the center of the group inreal time, and calculates the distance to the centroid,grouped according to the nearest distance principle.
The specific algorithm flow is as follows:(1) choose k robots from all robots randomly as
the initial group center;(2) The Euclidean distance between each robot
and the group center are calculated, and assign themto the group with the closest center according to thenearest distance principle;(3) Calculate the mean of all objects in each groupas the new center, and calculate the Square Error
Sum. Repeat from step 2 until the Square Error Sum
is converged.
After the grouping operation is completed, the aggregation operation is started. According to the requirements of center algorithm, the center positionof each group should be real-time updated and notified each robot in the group. Each robot adjusts the
speed and direction according to its current positionand new center position, then moves to the center.
The specific algorithm flow is as follows:(1) Each robot initializes its position, and reads
and saves the current position;(2) The robots in the group communicate witheach other to inform the position information. The
center coordinates are calculated according to theposition coordinates of all robots. Then each robot
calculates its own speed and direction of motion
based on its own coordinate and center coordinate.
Finally, the robots move to the center of the group.
(3) Repeat from step 1 until al robots in eachgroup enter a small range.
5.Simulation analysis
According to the grouping aggregation control algorithm, the research group carried out simulation
analysis on the MATLAB. In order to facilitate theobservation of the simulation results and analysis,
the simulation environment which is a
two-dimensional with a length of 100 and a widthof 100 was first configured. 60 robots were randomly set up in the grid. It was intended to divide
them into 4 groups and achieve group aggregation.
The specific two-dimensional grid and the initial
position of the robot are shown in Fig.4.

Fig.4. The specific two-dimensional grid and the initial
position of the robot

The multi-robot system is first divided into fourgroups according to the K-means algorithm, and theaggregation motion starts after the grouping is completed.
During the grouping process, the centers are updated after each grouping and grouped again according
to the nearest distance principle. Fig. 5(a) showsthat the four sample points were randomly selected
as the initial centroid, and the first grouping wascompleted. Fig.5(b) and Fig. 5(c) were the repeatedgrouping process of the robot, and the diamonds
showed the real-time position of the centers. The
grouping result was as shown in Fig. 5(d), and there© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

349

Huailin Zhao, Zhen Nie

was a good effect

In the process of aggregation control, the robots ofeach group moved from the initial position to thecenter of each group. During the aggregation motion, the centers were updated in each sampling
period (the sampling period T was set 1 second).
The Fig.6(a) to Fig.6(e) correspond to a sampling
period in the aggregation process, respectively.
Fig.6(d) shows that the robots have gathered withina small range. Fig.6 (e) shows that the motion stateof the robots was slightly adjusted relative toFig.6(d), but it was still within a small range, so thatthe aggregation task was completed.
(a)

(a)
(b)

(c)

(b)
（c）

(d)
Fig.5. The simulation of grouping process© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan350

Multi-robot grouping aggregation algorithm

(d）

（e）
Fig.6. The Simulation of aggregation process

6. Conclusion
This topic takes multi-robot system as the research
object, and takes grouping aggregation as the maintask, and realizes the grouping and aggregation
control of multi-robot in simple environment. The
research group designed the center grouping algorithm and the center aggregation algorithm based on
K-means algorithm. The MATLAB simulation
analysis shows the detailed process of grouping
operation and aggregation operation, which verifies
the feasibility of the grouping aggregation algorithm.
The algorithm embodies the swarm intelligence in acertain degree.

Technology, 2011.
4.Dimos V. Dimarogonas, Kostas J. Kyriakopoulos
Connectedness. Preserving Distributed Swarm Aggregation for Multiple Kinematic Robots[C]. IEEE
Transactions on Robotics, VOL. 24, NO. 5, OCTOBER 2008, 1213-1223.
5.Huailin Zhao, Wei Ren. Consensus Problem of Distributed Multi-agent Systems[C], Proceedings of the2015 International Conference on Artificial Life andRobotics (ICAROB 2015), 2015.01, Japan. 201-206.
6.Huailin Zhao, Zheng Wu, Xiaoxing Wang. A Multi-robot Rescuing System[C]. The 2018 International
Conference on Artificial Life and Robotics (ICAROB
2018), 2018.02, Japan. 582-586.
7.Zhu Guoqing. Study on Control on strategy for
Self-Organized Formation of Swarm Robotics Systems[D]. Shandong University, 2017.
8.Wang Qian, Wang Cheng, Feng Zhenyuan, Ye Jinfeng. Review of K-means clustering algorithm [J].
Electronic Design Engineering: 2012, 4:21-24
9.Yang Ping, Guo Chunyang, Li Cuiming. Swarm
Robots Aggregation Formation Control Based on
K-means Clustering Algorithm[J]. Modular Machine
Tool & Automatic Manufacturing Technology, 2012,
8:89-92.
10. Wang Zhongqiang, Li Gui, Han Ziyang, Zheng
Xinlu. Research and Implementation Based on
K-medoids Clustering Algorithm[J]. Silicon Valley:2011,1:80-81
11. Xia Ningxia, Su Yidan, Yu Xi. Efficient K-medoids
clustering algorithm[J].Application Research ofComputers:2010,12:4517-4519
12. Wang Wenming, Zhou Shuai. Research on Multi-Robot Cooperative Control and Formation Method[J]. Netinfo Security: 2014, 6:59-66.
13. Luo Jing, Liu Chenglin, Liu Fei. Pilot-following
formation and obstacle avoidance control of multi-mobile robots[J]. CAAI Transactions on Intelligent
Systems, 2017, 12(2): 202-212.

References
1.Huailin Zhao, Yangguang Guo, Masanori Sugisaka.
Design of the Multi-Car Collaboration System[C],the 2017 International Conference on Artificial Lifeand Robotics (ICAROB 2017), 2017.01, Japan.
186-190.
2.Wei Tingting. Research on the surround control
problem of multi-agent system [D]. University ofElectronic Science and Technology of China, 2014.
3.Shi Xiaoxing. Research on Formation and Coordination of Group Robots [D]. Lanzhou University of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan351

Research of the ARM-based Multi-robot aggregation Control
Huailin Zhao

School of electrical and Electronic Engineering, Shanghai Institute of technology, No. 100, HaiquanRoad, Fengxian District, Shanghai 201418,China
Jingnan Zheng

School of electrical and Electronic Engineering, Shanghai Institute of technology, No. 100, HaiquanRoad, Fengxian District, Shanghai 201418,China
Zhen Nie

School of electrical and Electronic Engineering, Shanghai Institute of technology, No. 100, HaiquanRoad, Fengxian District, Shanghai 201418,China
Weiya Zhang

School of electrical and Electronic Engineering, Shanghai Institute of technology, No. 100, HaiquanRoad, Fengxian District
hanghai 201418,China
Zonglin liu

School of electrical and Electronic Engineering,Shanghai Institute of technology, No. 100, HaiquanRoad, Fengxian District Shanghai 201418,China
E-mail: zhao_huailin@yahoo.com, hbniezhen@foxmail.com
www.sit.edu.cn

Abstract
This paper mainly analyzed the basic aggregation problem of multi-robot cooperative system.
The algorithm design of multi-robot dynamic aggregation control was discussed. The multi-robot
aggregation path planning method in simple environment was discussed. And the kinematics
equation of the aggregation algorithm was given at last. Taking the center aggregation control as
an example, the cooperation and control of multi-robot are studied. Then we designed a simulation
control program on MATLAB for multi-robot system and analyzed the simulation results. Finally,
the center aggregation experiment was carried out by using the three car robots. The results showthat the multi-robot aggregation algorithm is effective.
Keywords: Multi-robot; Aggregation Control; Motion planning; Swarm intelligence.

1. Introduction
The multi-robot collaboration system is a frontier
topic in the field of artificial intelligence [1]. Fromthe control point of view, the multi-robot collaboration system is a typical distributed control system [24], and its goal is to build large complex systems
which included software and hardware systems into
small, easy-to-manage systems that communicate andcoordinate with each other. The research of multirobot collaboration system involves the knowledge,
goals, skills, planning of robots and how to make robots coordinate actions to solve problems, and theidea of swarm intelligence is emphasized. The multirobot collaboration system forms a complex systemthrough communication, coordination, and cooperation among autonomous robots. The major applications include formation flying of multi-UAV, clusters

352

of warehousing logistics robot, competitions of robot
team, large-scale robotic rescue systems, etc. Some
Examples in nature are dancing fish, flying birds, andso on.
Aggregation is the basic problem of multi-robot system consistency [5-6], and it is also an optimizing
measure [7]. The aggregation problem is that the robots are distributed in space adjust the motion states
through communication and cooperation between
each other, finally they gather together, or in the desired area to complete the aggregation. The researchteam has realized the cooperation and aggregation of
two robots [8]. However, the Wi-Fi communication
between the two robots was indirectly realized by thehost computer, otherwise the bidirectional communication cannot be realized. So the cost of the system ishigh. Because there are only two robots, there is notmuch swarm intelligence. This topic intends to designHuailin Zhao, Jingnan Zheng, Zhen Nie, Weiya Zhang, Zonglin liu

and utilize three intelligent cars based on ARM, and a
distributed control system is formed by these cars.
This system does not require host computers as transfers, and any two machines can communicate directlywith each other. The project intends to design an effective algorithm of aggregation control to achieve
multi-machine distributed coordination and aggregation, which reflects certain swarm intelligence andlays a foundation for the research of large-scale multirobot distributed control system.
2. Analysis and design of aggregation algorithm
As mentioned earlier, the aggregation problem is thatmulti-robots randomly distributed in the space adjust
the motion states through communication and cooperation between each other, finally they gather together, or in the desired area to complete the aggregation. For a multi-machine system consisting of n robots. For a multi-machine system consisting of n robots, a typical kinematics equation of aggregate control is expressed as follows.

xi (t) ui (t), i 1, ,npoint, as shown in Fig.1(b). Each robot finds its own
target position in the whole aggregation movement
through the position information of neighboring robots. The limitation of this method is that a large
amount of communications is required, and each robot needs to spend more time for communication, andthe result of the formation control is not satisfactory.
3) Selecting the center of group as the reference
point, as shown in Fig.1(c). This selection method iscalled center algorithm. During the aggregation process, multiple robots form a geometric figure. According to the geometric figure, the geometric centeris calculated. Each robot uses takes the center as a reference point to adjust the direction and speed, andmove to the reference point. During the aggregation
movement, the system constantly updates the reference points, and finally all robots reach a same point.
the aggregation task is end. This algorithm is simple
and practical. The system is convergent. The advantages of the distributed control system are fully reflected.

(1)

Where xi(t) is the state variable of ith robot, ui(t)is the control input of ith robot.

u (t)
i

n
j 1

aij (t) x j (t) xi (t)(2)

Where aij(t) is coefficient of the system.
It can be seen from equation (2) that the farther the
robot is from other robots, the greater the control effect. Based on the above basic theory, three aggregation control algorithms were designed and analyzed
for the aggregation task requirements of this topic:
1) Selecting a robot from each group as leaders, as
shown in Fig.1(a). The basic idea of this algorithm isto pre-set a leader robot, and other robots are called
followers [9-10]. In the process of aggregation, theleader controls the motion path, and the followers
keep a certain angle and distance with the leader, finally all robots aggregate at the position of the leader.
After careful analysis, the aggregation has some defects. On the one hand, because the algorithm doesn’t
make full use of the advantages of distributed systems,
when the leader robot goes wrong, it will cause all
robots fail to aggregate. On the other hand, because
the control of the system is relatively simple, the follower or leader robot may be lost in the process of
motion in the complicated reality environment. The
aggregation task failed.
2) Selecting the neighboring robot as the reference

(a)

(b)

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan353

ARM-based Multi-robot aggregation Control(c)Fig.1 Three different aggregation algorithms:(a) Selecting
a robot from each group as leaders; (b) Selecting theneighboring robot as the reference point; (c) Selecting thecenter of group as the reference point.

Comparing the three aggregation algorithms, thisstudy intends to use the center algorithm for aggregation control. The center calculation equations are expressed as follows:
n

xc

n

xi / n , y ci 1yi / n

(3)

WIFI

i 1

Where（ xi , yi ) is the 2-D coordinate of the current
position for the ith robot, and ( xc , yc ) is the center
coordinate.
During the aggregation process, the system periodically updates the center position and notifies all robots. Each robot makes motion planning according to
its current position and new center position. The motion planning equation is given by

X i (k1)

is an intelligent individual. When cooperation isneeded, every robot communicates with each other.
Otherwise, each robot independently completes the
motion planning. The distributed control of the wholesystem is realized.
The main controller of each robot uses the STM32 [1112], and the WIFI module is additionally configured torealize the communication cooperation. The GPS
module is configured to read the real-time location.
The INS is configured to achieve azimuth adjustment.
Fig. 2 shows the robot motion control system.

X i (k)vi (k1 ) T (4)Where T is the sampling period, Xi is the motion stateof ith robot, vi is the velocity of ith robot.
3. Calculation process of center aggregation
algorithm
According to the requirements of center algorithm,
the center position of each group should be real-time
updated and notified each robot in the group. Each robot adjusts the speed and direction according to its
current position and new center position, then moves
to the center. The specific algorithm flow is as follows:(1) Each robot initializes its position, and reads andsaves the current position;(2) The robots communicate with each other to inform the position information. The center coordinates
are calculated according to the position coordinates of
all robots. Then each robot calculates its own speed
and direction of motion based on its own coordinate
and center coordinate. Finally, the robots move to thecenter of the group.
(3) Repeat from step 1 until al robots in each group
enter a small range.
4. Robot motion control system
In the multi-machine cooperation system, each robot

Coordinates of
Accumulation point

controllers

Left/rightmotors

cars

GPS

INS

Fig.2 The robot motion control system

As mentioned earlier, any two robots in the system
can communicate directly with each other by Wi-Fi,
and no longer require the host computers as transfers.
The reason is that we have configured two Wi-Fi
modules for each car, one dedicated to sending signal
and the other dedicated to receiving signal. In thisway, the two-way communication of each robot is realized, and the host computers are no longer require,
thereby greatly reducing the system cost.
5. Simulation analysis
According to the grouping aggregation control algorithm, the research group carried out simulation analysis on the MATLAB. The simulation environment
which is a two-dimensional with a length of 100 anda width of 80 was first configured. The initial positions of three robots were randomly set, and their coordinates are (1,1), (100,50), (5,80). The specific twodimensional grid and the initial position of the robot
are shown in Fig.3(a).
In the process of aggregation control, the robots
moved from the initial position to the center. During
the aggregation motion, the centers were updated ineach sampling period (the sampling period T was set
1 second). The Fig.3(b) to Fig.3(f) correspond to asampling period in the aggregation process, respectively. Fig.6(e) shows that the robots have gathered
within a small range. Fig.6 (f) shows that the motion
state of the robots was slightly adjusted relative to© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan354

Huailin Zhao, Jingnan Zheng, Zhen Nie, Weiya Zhang, Zonglin liu

Fig.6(e), but it was still within a small range, so thatthe aggregation task was completed.

(d)

(e)
(a)

(f)
(b)

Fig.3 The Simulation of aggregation process

6. Experiment
Based on the theoretical basis of the multi-robot system and the simulation analysis, the research teamcarried out an experiment on campus. The three intelligent cars we used are shown in Fig. 4.

(c)Fig.4 The intelligent cars used in the experiment.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan355

ARM-based Multi-robot aggregation Control

Before the experiment, three intelligent cars were randomly placed on the road at a certain distance. Then
start each car program at the same time to observe themotion situation of the multi-car system. The intelligent cars were constantly aggregate to the center. Finally, the cars were very close to each other, and nolonger have obvious displacement, it means that theaggregation is successful. Fig.5 shows the aggregation process

and speed of aggregation, which means the initialstate has a direct impact on the aggregation results.
7. Conclusion
This topic takes multi-robot system as the research
object, and takes aggregation as the main task, andrealizes the aggregation control of multi-robot in simple environment. After the specific centroid aggregation control algorithm was designed, not only the simulation analysis was carried out, but also the experimental results were carried out in the actual environment. The expected aggregation effect is achieved,
which verifies the feasibility of the multi-robot system. The topic embodies the swarm intelligence.
References

(a)

(b)
Fig.5 The aggregation process of the intelligent cars:(a)the initial position of the three cars; (b) the aggregation
position of three cars

Through the experiment, some conclusions are put
forward as follows:(1) Regardless of the initial placement of the threecars, they can aggregate successfully, which shows
the stability of the system.
(2) The initial position of three smart cars will affectthe experimental results, including the final location

1. Huailin Zhao, Yangguang Guo, Masanori Sugisaka. Design of the Multi-Car Collaboration System[C], the 2017
International Conference on Artificial Life and Robotics(ICAROB 2017), 2017.01, Japan. 186-190.
2. Wei Tingting. Research on the surround control problem
of multi-agent system [D]. University of Electronic Science and Technology of China, 2014.
3. Shi Xiaoxing. Research on Formation and Coordinationof Group Robots [D]. Lanzhou University of Technology,
2011.
4. Dimos V. Dimarogonas, Kostas J. Kyriakopoulos Connectedness. Preserving Distributed Swarm Aggregation
for Multiple Kinematic Robots[C]. IEEE Transactions
on Robotics, VOL. 24, NO. 5, OCTOBER 2008, 12131223.
5. Huailin Zhao, Wei Ren. Consensus Problem of Distributed Multi-agent Systems[C], Proceedings of the 2015
International Conference on Artificial Life and Robotics(ICAROB 2015), 2015.01, Japan. 201-206.
6. Yang Ping, Guo Chunyang, Li Cuiming. Swarm Robots
Aggregation Formation Control Based on K-means
Clustering Algorithm[J]. Modular Machine Tool & Automatic Manufacturing Technology, 2012, 8:89-92.
7. Zhu Guoqing. Study on Control on strategy for Self-Organized Formation of Swarm Robotics Systems[D].
Shandong University, 2017.
8. Huailin Zhao, Zheng Wu, Xiaoxing Wang. A Multi-robot Rescuing System[C]. The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018),
2018.02, Japan. 582-586.
9. Wang Wenming, Zhou Shuai. Research on Multi-Robot
Cooperative Control and Formation Method[J]. Netinfo
Security: 2014, 6:59-66.
10. Luo Jing, Liu Chenglin, Liu Fei. Pilot-following formation and obstacle avoidance control of multi-mobile
robots[J]. CAAI Transactions on Intelligent Systems,
2017, 12(2): 202-212.
11. Fengzhi Dai, Yuxing Ouyang, Yiqiao Qin, BaochangWei, Yujie Yan. The Design of Intelligent Vehicle System Based on STM32. [C], Journal of Advanced Control, Automation and Robotics 4 (1): 145-148, April 5,
2018.
12. Fengzhi Dai, Yuxing Ouyang, Yiqiao Qin, Ce Bian,
Baochang Wei, Shengbiao Chang, Bo Liu. Development of Integral Smart Home Appliances[J]. Journal of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

356

Huailin Zhao, Jingnan Zheng, Zhen Nie, Weiya Zhang, Zonglin liu

Robotics, Networking and Artificial Life, Vol 4, No. 4(March 2018), 291-294.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan357

Research on Trajectory Planning Problem of Robot in Spherical Cutting

Min JunSchool of electrical and Electronic Engineering,Shanghai Institute of technology,
No. 100, Haiquan Road, Fengxian District，Shanghai 201418,China
E-mail: 15205182192@163.com
Jiang Wenping

School of electrical and Electronic Engineering,Shanghai Institute of technology,
No. 100, Haiquan Road, Fengxian District，Shanghai 201418,China
E-mail: jiangwenping@sit.edu.cn
Zhao Huailin

School of electrical and Electronic Engineering,Shanghai Institute of technology,
No. 100, Haiquan Road, Fengxian District，Shanghai 201418,China

E-mail: zhao_huailin@yahoo.com

Abstract
For cylindrical shape cutting of spherical surface vessel a kind of 5-DOF cutting method of cylindrical shape for spherical surface is provided in this paper. Based on the kinematics model of cutting
robotthe mathematic relations between cutting tool cylindrical shape and spherical surface are built
using Eulertransformation and the intersection line model of cylindrical and spherical surface ispresented. The roboticcutting trajectory is computed according to the position and attitude relationof robot and spherical sphereand simulations and experiments of cutting trajectory for robot areprovided in the end. Experiments resultsverified the rightness of robot kinematics and cylindricalshape cutting algorithm,and will promote the robot-is application in cutting field.
Key words: spherical surface;Kinematics;cylindrical shape;cutting;Trajectory planning;roboof high labor intensity, dependence on operator
experience, low precision and poor consistency.
Although the latter greatly improves cutting
quality and work efficiency, it still exists. Some
disadvantages, such as the complicated structure of
the cutting aid, the large footprint of the cutting
machine, and poor cutting suitability and flexibility.
Compared to CNC cutting machines, cutting
robots offer greater flexibility and adaptability[1].
analytical method to study the problem oftheorthogonal intersecting line between thecylinder and the sphere. However, this algorithm islimited to the orthogonality of the cylinder and the

1. Introduction
Metal pressure vessels are widely used in chemical,
petroleum, mechanical, metallurgical, nuclear,aerospace and aerospace sectors, and are essential
equipment in the production process. The can bodycontainer is generally composed of a cylinder, aspherical surface, and a head, and the cylindricalsurface is usually required to be cut in the
manufacturing process. At present, most of China's
spherical cutting uses manual operation or CNC
cutting machine. The former has the disadvantages© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan358

Min Jun, Jiang Wenping, Zhao Huailin

spherical surface. It can not solve the intersectionof the cylinder and the spherical surface in any
direction, and lacks versatility. At present, thealgorithm for solving the intersecting line between
the cylinder and the spherical surface is more
focused on mathematical theory. The algorithm ismore complicated and is not suitable for theengineering application of the cutting robot.
Therefore, based on a 5-DOF robot, this paper
proposes a cylindrical cutting scheme for the
spherical surface of the robot. Based on the
kinematics model of the robot, the cutting
trajectory planning algorithm for the sphericalsurface and the arbitrary position and direction
cylinder is designed, and the robot The cutting
algorithm has been simulated and can be adapted
to practical engineering applications[2].

The structural diagram of the robot is shown in
Figure 2, where X, Y, and Z are the basic
coordinate systems centered on the robot base, x1 ,y1 , z1 、 x2 , y2 , z2 、 x3 , y3 , z3 、 x4 , y4 , z4 andx5 , y5 , z5 are the coordinate system of the cutting
robot joint coordinates 1, 2, 3, 4, 5 respectively,
x1 , y1 , z1 is the robot end tool coordinate system,
d1 ，d 2 ，d 3 ，d 4 ，d 5 ，d 6 is the length of eachrod of the robot[4].
z3y3

Joint 3
z2

x3

d3
y2

Joint 14
d4

Joint 2x2z

2. Cutting robot motion model

Joint 5

x4x5

y4

d1

d5

z1

y

d6
xt

The research object of this paper is a 5-DOF robot,
which adopts Cartesian coordinates. The robot
with this structure has the characteristics of good
rigidity, high precision and low cost. Among them,
the first joint is a moving joint along the X axis, thesecond joint is a moving joint along the Y axis, thethird joint is a moving joint along the Z axis, thefourth joint is a rotating joint, and the fifth joint isa swing joint.
When the robot uses the laser cutting method to
cut the spherical crown, the robot moves along theintersecting line of the cylinder and the cylinder,
and the end posture is consistent with the cylinder
axis, so that the straight and oblique cutting of thecylinder and the sphere can be completed. The
cutting process of the robot is shown in Figure 1[3].
Process
analysisy1

Joint 1

Pedestal

x1

Tool

z4
yt

zt

Figure.2. Schematic diagram of the robot structure

2. 1 Robotic Kinematics
It is assumed that the motion variables of the joints
are X , Y , Z , 1 , and 2 are the motion
variables of the first joint, the second joint, thethird joint, the fourth joint, and the fifth joint of therobot, respectively. Obviously, d 2 X , d 3 Y ,d4

Z , so that the coordinate systemtransformation matrix of adjacent joints can beobtained.
0

T1 Trans( X ,0,0)

Cutting point P0
Bearing point P1

CAD drawing

Position calibration
between robot andsphered2

1

T2

Trans (0,0, d1 )2

Trans (0, Y ,0)

T3

Cylindrical and sphericalintersecting line
trajectories and poses

3

T4

Rot ( x,180)Tran(0,0, Z ) Rot ( z , 1 )4

T5

Robot cutting track and attitude

5Tt

Rot ( x,90) Rot ( z , 2 )Rot ( x, 90)Tran(d 5 ,0,0)Trans (0,0, d 6 )Robot kinematics

Robot cutting

Figure.1. Schematic diagram of the robot cutting
process© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan359

Research on Trajectory Planning

The transformation matrix As consists of a rotation
matrix Rs and a translation matrix Ts , which can beobtained from Figure 3.

Thus, the pose matrix of the robot end cutting toolin the basic coordinate system can be obtained,justlike formula (1).

(1)

2. 2 Robot inverse kinematics
Assume that the end pose matrix of a 5-DOF
robot
is:

(2)

Figure.3. Schematic diagram of the intersection
the spherical surface and the cylinder surface

Here, the rotation matrix Rs is obtained by the

According to equations (1) and (2), the inverse
solution of the robot can be obtained according to
the principle that the corresponding elements of thematrix are equal:
12

equivalent
axis
angle
transformation
representation. It can be seen from the robot
coordinate system transformation that the twocoordinate systems can be obtained by rotating acertain angle around an Euler axis. For theconvenience of calculation, the coordinate system
origin os is translated to the p0 point, and thearctan 2( ox , o y )arctan 2( nz , a z )px cos 1 cos 2 d5 cos 1 sin 2 d 6XY

p y sin 1 cos 2 d 5 sin 1 sin 2 d 6

Z

d1

of

equivalent rotation axis K is perpendicular to
the z s and z s axes, so that the angle between

pz sin 2 d 5 cos 2 d 6

thez s and the z s axis is the equivalent rotationz s axis coincides with theaxis of the cutting cylinder, in the xs , ys , z s
[6]

angle

3. Rrobot cutting algorithm
The intersection of the sphere and the arbitrary
orientation cylinder is shown in Figure 4. Amongthem, xs , ys , z s is the spherical coordinate systemwith the spherical center as the origin, xs , ys , z s
is the coordinate system with the spherical centeras the origin, and the z s axis coincides with the axisof the cutting cylinder. xs , ys are respectively xs ,ys Rotated around the Euler axis. p0 ( x0 , y0 , z0 )is the intersection of the cylinder axis and thespherical surface, p1 ( x1 , y1 , z1 ) is an arbitrary
point in the direction of the cylinder axis, K is theequivalent rotation axis, andis the rotation[5]
angle around the equivalent axis .

. Since thecoordinate system, thezsvector can be

expressed as:zs

vx vy vzx0 x1 y0 y1 z0 z1
p1 p2
p1 p2
p1 p2

T

among them,

p1 p0

( x0

x1 ) 2 ( y0y1 ) 2 ( z0 z1) 2

z s axis vector as 0 0 1 T , thenK (k x , k y , k z ) is:take theK

xs , ys , z s and xs , ys , z s coordinate system
conversion matrix As

3.1

zs zs

vy

vx

0

T

further can get its standard form:© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan360

Min Jun, Jiang Wenping, Zhao Huailin

K

vy

vx

vx2 v y2v x2 v y2cos

obviously,

According to the conversion relationship between
the xs , ys , z s and xs , ys , z s coordinate0vz . Therefore, the rotation

coordinate system：( x0 , y0 , z 0 ,1)TRs can be obtained:matrix

kxk yvc

kx k yv

kz s

kxkzv

kys

kx kz vkx kz vkz s
kys

ky kyvk ykzvckxs

k y kz v
k z kzv

kx s
c

sin

1 cos

transformation matrix

As

Rs

Ts

0

1

2

zs

2

R2

x0 ) 2y0 ) 2

( ys

z

（3）

cylinder andequationintersecting

（5）

r2

（6）

x

y

value of s must be known, and s and s
respectively have two sets of values, which aresolved. select. Therefore, the geometric projection
method is used here toolve. In Figure 5, the cylinder is projected on thexs , ys plane. Consider the value of the entire
intersecting line[8]. Here, for equation (6), take:

line

xs
ys

xs , ys , zs coordinate system, the zs axis

x0 r cosy0 r coszs

xs , ys plane is a circle.
p0 ( x0 , y0 , z0 ) is the

R

2

（7）

（0 ，
360 ）, so you can get:Among them,coincides with the cylinder axis, and theintersection of the sphere and the cylinder becomes
orthogonal. As shown in Fig. 4, the projection of
the cylinder in theysAccording to equations (5) and (6), theintersection line data of the sphere and the cylinder
can be obtained. To solve the above equations, theAs :xs , ys , zs lower（4）

At the same time, the geometric equation of thecylinder can be obtained：. Thus obtains the pose

sphere2( xs

3.2 Coordinate system

In thexs

1 cos ,
2

As 1 ( x0 , y0 , z0 ,1)TAccording to Figure 5, the equation of the spherecan be obtained：

R s Rot( K , )in the formula, v

p0 can be obtained in the xs , ys , zs

systems,

xs

2

ys

2

（8）

3.3 Coordinate line position of the cylinder
and the sphere under the coordinate
system xs , ys , z s

Among them,intersection of the cylinder axis and the sphericalsurface, and the radius of the sphere is R , and theradius of the cylinder is r [7].

After obtaining the data ( x p , y p , z p )intersecting line in theof thexs , ys , zs coordinate

system, according to the conversion relationshipbetween the xs , ys , z s and the xs , ys , z scoordinate(xp , yp , z p )system,ofthe( x p , y p , z p ) data in the xs , ys , z s coordinate
system can be obtained:( x p , y p , z p ,1)TAs ( x p , y p , z p ,1)T（9）

3.4 Intersecting line cutting posture undercoordinate system xs , ys , z s
Figure.4. Schematic diagram of spherical intersectionin theWhen the robot performs a cylindrical cut on thespherical surface, the cutting tool space vector isxs , ys , zs coordinate system© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan361

Research on Trajectory Planning

the axis of the cylinder and remains unchanged.
Therefore, the robot end tool coordinate system
can be regarded as rotaing degree around theAtz s axis, so that the attitude transformation matrixof the intersecting line equation in thexs , ys , z scoordinate system can be obtained:Ry

Rot( z, a)

,

cos

sin

0

sin

cos

0

0

0

1

(0 ,360 )The data for points

（10）

（11）

(12)
In the robot cutting process, firstly, the mounting
position of the robot base and the ball is calibrated
to obtain its position and attitude transformation
matrix At , and the movement trajectory of theis the attitude matrix shown inPp

p0 and p1 can be obtained

In this paper, based on a 5-DOF cutting robot, thekinematics model of the robot is established, andthe forward and inverse kinematics algorithm of
the robot is given. According to the cylindricalcutting process requirement of the spherical tool at
any orientation, the equivalent axis Euler
transform method is used to establish the posetransformation matrix of the coordinate system andthe spherical coordinate system which coincides
with the axis of the cutting cylinder, and then thecylinder is solved according to the geometricprojection method. The orthogonal intersecting
line equation of spherical cutting is given, and theattitude planning algorithm of spherical cutting isgiven, so that the intersecting line model andcutting attitude of the spherical and arbitrary
azimuth cylinders in the spherical coordinate
system can be obtained. On this basis, based on thepose transformation matrix of the robot and thesphere, the robot's cylindrical cutting trajectoryalgorithm for spherical surface is designed. Finally,
the simulation experiment of the cutting trajectory
and the kinematics model of the robot is carried out.
The experimental results verify the correctness ofthe spherical cutting algorithm of the cutting robot.

3.5 Robot intersection line cutting position
and attitude calculation

equation ,

500
300
200
1

5. Conclusion

obtained:robot is:Where R p00
10from the CAD drawing parameters of the sphere[10].

coordinate systems, the posture of the cutting toolin the xs , ys , z s coordinate system can be

Rs R y0100Note that during the cutting process, the Z-axis
direction of the robot's cutting tool attitude isdetermined by p0 p1 and remains unchanged.

Thus, according to the conversion relationshipbetween the xs , ys , z s and xs , ys , z s

Rp

1
0
0
0

( x p , y p , z p )T [9].

4. ExperimentThis paper takes 5 degree of freedom cutting robot
as the research object，d1 =800mm，d 5 =100mm，
d 6 =200mm. Take the radius of the ball
R=1000mm，cutting cylinder radius r=150mm，Point p0 =（14.88，-4.83，-98.77）mm on thesphere, point p1 =（24.88，-24.83，-128.77）mm
in the direction of the cylinder axis. In the robot
cutting process, take the robot and sphere
coordinate system conversion matrix:

References
1. Liao Weidong, Li Chengneng, Wang Qiang, Liao Wei.
Kinematics Modeling and Simulation of 6R Robot
Tool End[J]. Mechanical Design & Manufacturing,
2018(09): 108-111.
2. Wang Yubao, Wang Shiyu, Li Beibei, Guo Fangda. A
Time Optimal Trajectory Planning Algorithm for
Industrial Robots with Improved Particle Swarm[J].
Small Computer Systems, 2018, 39(08): 1878-1881.
3. Zhang Yonggui, Li Hailong. Research on TrajectoryPlanning of Milling Robot to Remove Cracks in
Forgings[J]. Combined Machine Tool and Automatic
Machining Technology, 2018(07):1-4+9.
4. Wang Jun, Chen Zhilong, Yang Zhiyong, Qin© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan362

Min Jun, Jiang Wenping, Zhao Huailin

Zhengzheng, You Ying, Wei Qiong. Time Optimal
Trajectory Planning of Industrial Robot Based on
Improved DE Algorithm[J]. Combined Machine Tool
& Automated Machining Technology, 2018(06): 4246.
5. LIANG Zilong, DING Yufeng. A trajectory planning
method for free-surface polishing robots [J/OL].
Mechanical Science and Technology: 1-7[2018-1009].https://doi.org/10.13433/j.cnki.10038728.20180071.
6. Chen Xuefeng. Research on trajectory planningalgorithm of palletizing
robot[J].
ControlEngineering,2018,25(05):925-929.
7. Liao Weiqiang, Zhao Changming, Ma Wenbiao,
Zhang Tie. Trajectory Planning of Industrial Robots inFive-Star Welding Process[J]. Machinery Design &Manufacture, 2018(04):231-234.
8. LI Li, SHANG Junyun, FENG Yanli, HUA Yawen. A
review of trajectory planning research for articulatedindustrial robots[J]. Computer Engineering andApplications, 2018, 54(05): 36-50.
9. FANG Jian, SONG Yu, ZHU Maofei, ZHU Dequan,
LIU Yongbo, JIAO Jun. Time-optimal trajectory
planning for palletizing robots[J]. Control
Engineering, 2018, 25(01):93-99.
10. Dong Jiajia, Wang Taiyong, Dong Jingchuan, Zhang
Yongbin, Tao Hao. Improved B-spline curve applied
to 6R robot trajectory optimization[J]. China
Mechanical Engineering, 2018, 29(02): 193-200.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan363

Research on Trajectory Planning Problem of Robot in Spherical Cutting

Min JunSchool of electrical and Electronic Engineering,Shanghai Institute of technology,
No. 100, Haiquan Road, Fengxian District，Shanghai 201418,China
E-mail: 15205182192@163.com
Jiang Wenping

School of electrical and Electronic Engineering,Shanghai Institute of technology,
No. 100, Haiquan Road, Fengxian District，Shanghai 201418,China
E-mail: jiangwenping@sit.edu.cn
Zhao Huailin

School of electrical and Electronic Engineering,Shanghai Institute of technology,
No. 100, Haiquan Road, Fengxian District，Shanghai 201418,China
E-mail: zhao_huailin@yahoo.com

Abstract
For cylindrical shape cutting of spherical surface vessel a kind of 5-DOF cutting method of cylindrical shape forspherical surface is provided in this paper. Based on the kinematics model of cutting robot the mathematics
relations between cutting tool cylindrical shape and spherical surface are built using Euler transformation and theintersection line model of cylindrical and spherical surface is presented. The robotic cutting trajectory is computed
according to the position and attitude relation of robot and spherical sphere,and simulations and experiments of
cutting trajectory for robot are provided in the end. Experiments results verified the rightness of robot kinematics
and cylindrical shape cutting algorithm,and will promote the robot-is application in cutting field.
Key words: spherical surface;Kinematics;cylindrical shape;cutting;Trajectory planning;robot

intensity, dependence on operator experience, low
precision and poor consistency. Although the latter
greatly improves cutting quality and work efficiency, it
still exists. Some disadvantages, such as the complicatedstructure of the cutting aid, the large footprint of the
cutting machine, and poor cutting suitability andflexibility. Compared to CNC cutting machines, cutting
aid, the large footprint of the cutting machine, and poor
cutting suitability and flexibility. Compared to CNC
cutting machines, cutting robots offer greater flexibility

1. Introduction
Metal pressure vessels are widely used in chemical,
petroleum, mechanical, metallurgical, nuclear, aerospace
and aerospace sectors, and are essential equipment in the
production process. The can body container is generallycomposed of a cylinder, a spherical surface, and a head,
and the cylindrical surface is usually required to be cut inthe manufacturing process. At present, most of China's
spherical cutting uses manual operation or CNC cutting
machine. The former has the disadvantages of high labor© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan364

Min Jun, Jiang Wenping, Zhao Huailin

and adaptability[1].
At present, there are few researches on sphericalcutting of robots, mainly focusing on theintersection line solving algorithm of cylinderand spherical surface. Xu Yuanlong used theanalytical method to study the problem of theorthogonal intersecting line between the cylinder
and the sphere. However, this algorithm islimited to the orthogonality of the cylinder andthe spherical surface. It can not solve theintersection of the cylinder and the sphericalsurface in any direction, and lacks versatility. At
present, the algorithm for solving theintersecting line between the cylinder and thespherical surface is more focused onmathematical theory. The algorithm is more
complicated and is not suitable for theengineering application of the cutting robot.
Therefore, based on a 5-DOF robot, this paper
proposes a cylindrical cutting scheme for the
spherical surface of the robot. Based on the
kinematics model of the robot, the cutting
trajectory planning algorithm for the sphericalsurface and the arbitrary position and direction
cylinder is designed, and the robot The cutting
algorithm has been simulated and can be adapted
to practical engineering applications[2].

Process
analysis

Cutting point P0
Bearing point P1

CAD drawing

Position calibration
between robot andsphereCylindrical and sphericalintersecting line
trajectories and poses

Robot cutting track and attitude

Robot kinematics

Robot cutting

Figure.1. Schematic diagram of the robot cutting
process

The structural diagram of the robot is shown in
Figure 2, where X, Y, and Z are the basic
coordinate systems centered on the robot base,x1 , y1 , z1 、x2 , y2 , z2 、x3 , y3 , z3 、x4 , y4 , z4
and x5 , y5 , z5 are the coordinate system of the
cutting robot joint coordinates 1, 2, 3, 4, 5
respectively, x1 , y1 , z1 is the robot end tool
coordinate system, d1 ， d 2 ， d 3 ， d 4 ， d 5 ，
d 6 is the length of each rod of the robot[4].

2. Cutting robot motion model
The research object of this paper is a 5-DOF
robot, which adopts Cartesian coordinates. The
robot with this structure has the characteristics ofgood rigidity, high precision and low cost.
Among them, the first joint is a moving joint
along the X axis, the second joint is a moving
joint along the Y axis, the third joint is a moving
joint along the Z axis, the fourth joint is arotating joint, and the fifth joint is a swing joint.
When the robot uses the laser cutting method to
cut the spherical crown, the robot moves along
the intersecting line of the cylinder and thecylinder, and the end posture is consistent withthe cylinder axis, so that the straight and obliquecutting of the cylinder and the sphere can becompleted. The cutting process of the robot isshown in Figure 1[3].

z3y3

Joint 3
z2

x3

d3
y2

Joint 14
d4

Joint 2x2

z

Joint 5
d5

z1

y

x4x5

y4

d1

d6
xt

d2
Pedestaly1

Joint 1

x1

Tool

z4
yt

zt

Figure.2. Schematic diagram of the robot structure

2. 1 Robotic Kinematics
It is assumed that the motion variables of thejoints are X , Y , Z , 1 , and 2 are themotion variables of the first joint, the second
joint, the third joint, the fourth joint, and the fifth
joint of the robot, respectively. Obviously,
Z , so that thed 2 X , d3 Y , d 4coordinate system transformation matrix of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan365

Research on Trajectory Planning

the origin, xs , y s , z s is the coordinate systemadjacent joints can be obtained.

T1 Trans( X ,0,0)

with the spherical center as the origin, and thez s axis coincides with the axis of the cutting
cylinder. xs , y s are respectively xs , y s Rotated

0
1

T2

Trans (0,0, d1 )2

Trans (0, Y ,0)

T3

3

T4

T5

Tt

intersection of the cylinder axis and the sphericalsurface, p1 ( x1 , y1 , z1 ) is an arbitrary point in

Rot ( x,180)Tran(0,0, Z ) Rot ( z , 1 )45around the Euler axis. p0 ( x0 , y0 , z0 ) is thethe direction of the cylinder axis, K is theis the rotationequivalent rotation axis, and[5]angle around the equivalent axis .

Rot ( x,90) Rot ( z , 2 )Rot ( x, 90)Tran(d 5 ,0,0)Trans (0,0, d 6 )3.1

Thus, the pose matrix of the robot end cutting
tool in the basic coordinate system can beobtained,just like formula (1).
1

2

c 1c

2

3

4

5

s

1

system conversion matrix As
The transformation matrix As consists of a

T 0T 1T 2T 3T 4T 5Tt
s 1c 2
s 2
0

c 1s

c 1
0
0

2

c 1c

s 1s 2c 2
0d 5 c 1s2

xs , ys , z s and xs , ys , z s coordinate

rotation matrix Rs and a translation matrix Ts ,d6 X

which can be obtained from Figure 3.

2

s 1c 2d 5 s 1s 2d 6 Y
s 2d 5 c 2d 6 Z d 1
1(1)
2. 2 Robot inverse kinematics
Assume that the end pose matrix of a 5-DOF
robot is:

P

nx

ox

ax

px

ny
nz
0

oy
oz
0

ay
az
0

py
pz
1

（2）
Figure.3. Schematic diagram of the intersectionof the spherical surface and the cylinder surface

According to equations (1) and (2), the inverse
solution of the robot can be obtained according
to the principle that the corresponding elementsof the matrix are equal:
1

equivalent
axis
angle
transformation
representation. It can be seen from the robot
coordinate system transformation that the twocoordinate systems can be obtained by rotating acertain angle around an Euler axis. For theconvenience of calculation, the coordinate
system origin os is translated to the p0 point,

arctan 2( ox , o y )XYarctan 2( nz , a z )p x cos 1 cos 2 d 5 cos 1 sin 2 d 6
p y sin 1 cos 2 d 5 sin 1 sin 2 d 6

Z

d1

2

Here, the rotation matrix Rs is obtained by theand the equivalent rotation axis K isperpendicular to the z s and z s axes, so thatp z sin 2 d 5 cos 2 d 6

the angle between the

3. Rrobot cutting algorithmz s and the z s axis is

the equivalent rotation angle

The intersection of the sphere and the arbitrary
orientation cylinder is shown in Figure 4.
Among them, xs , y s , z s is the spherical[6].

z s axis coincides with the axis of the
cutting cylinder, in the xs , y s , z s coordinate
system, the z s vector can be expressed as:
Since thecoordinate system with the spherical center as© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

366

Min Jun, Jiang Wenping, Zhao Huailinzs

x0 x1 y0 y1 z0 z1
p1 p2
p1 p2
p1 p2

T

vx vy vz

among them,p1 p0x1 ) 2 ( y0( x0y1 ) 2 ( z0 z1) 2Tz s axis vector as 0 0 1 , thenK (k x , k y , k z ) is:take the

K

zs zs

vy

vx

0

T

further can get its standard form:

vy

K

v

2
x

obviously, cos
matrix

vx
v

2
y

v

2
x

v 2y

Figure.4. Schematic diagram of spherical intersectionin the xs , y s , z s coordinate system

0

According to the conversion relationshipbetween the xs , y s , z s and xs , y s , z s

vz . Therefore, the rotation

xs , ys , z s coordinate system：

R s Rot( K , )kxk yvc

kx k yv

kz s

kxkzv

kys

kx kz vkx kz vkz s
kys

ky kyvk ykzvckxs

k y kz v
k z kzvkx s
cin the formula, v

sin

1 cos

transformation matrix

As( x0 , y0 , z0 ,1)T

Rs

Ts

0

1

xs

2

( xs

As :

ys

2

zs

2

R2

（5）

x0 ) 2 ( y sy0 ) 2

r2（6）

According to equations (5) and (6), theintersection line data of the sphere and thecylinder can be obtained. To solve the above
equations, the value of z s must be known, and（3）xs and ys respectively have two sets ofvalues, which are solved. select. Therefore, thegeometric projection method is used here tosolve. In Figure 5, the cylinder is projected on
the xs , y s plane. Consider the value of thexs , ys , z s lower

cylinder and sphere intersecting line
equationIn the（4）

At the same time, the geometric equation of thecylinder can be obtained：. Thus obtains the pose

3.2 Coordinate system

As 1 ( x0 , y0 , z0 ,1)TAccording to Figure 5, the equation of the spherecan be obtained：

1 cos ,
2p0 can be obtained in thecoordinate systems,

Rs can be obtained:

xs , ys , z s coordinate system, the z s

entire intersecting line[8]. Here, for equation (6),take:axis coincides with the cylinder axis, and theintersection of the sphere and the cylinder
becomes orthogonal. As shown in Fig. 4, theprojection of the cylinder in thexsysxs , ys planex0 r cosy0 r cos

Among them,is a circle. Among them, p0 ( x0 , y0 , z0 ) is
the intersection of the cylinder axis and thespherical surface, and the radius of the sphere is
R , and the radius of the cylinder is r [7].

zs

R2

（7）

（0 ，
360 ）, so you can get:

xs

2

ys

2

（8）

3.3 Coordinate line position of the cylinder
and the sphere under the coordinate
system xs , y s , z s

After obtaining the data ( x p , y p , z p ) of the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

367

Research on Trajectory Planning

robot as the research object， d1 =800mm， d 5=100mm， d 6 =200mm. Take the radius of theball R=1000mm ， cutting cylinder radius
r=150mm，Point p0 =（14.88，-4.83，-98.77）mm on the sphere, point p1 =（24.88，-24.83，-128.77）mm in the direction of the cylinder
axis. In the robot cutting process, take the robot
and sphere coordinate system conversion matrix:

xs , ys , z s coordinate

intersecting line in thesystem, according to the conversion relationshipbetween the xs , y s , z s and the xs , y s , z scoordinate(xp , yp , z p )system,(xp , yp , z p )

of

the

xs , y s , z sdata in thecoordinate system can be obtained:( x p , y p , z p ,1)TAs ( x p , y p , z p ,1)T（9）

3.4 Intersecting line cutting posture undercoordinate system xs , y s , z sAtWhen the robot performs a cylindrical cut on thespherical surface, the cutting tool space vector isthe axis of the cylinder and remains unchanged.
Therefore, the robot end tool coordinate system
can be regarded as rotaing degree around thez s axis, so that the attitude transformation
matrix of the intersecting line equation in thexs , ys , z s coordinate system can be obtained:Ry

Rot( z, a)

cos

sin

0

sin

cos

0

0

0

1

(0 ,360 )

,

（10）

（11）

3.5 Robot intersection line cutting position
and attitude calculation
In the robot cutting process, firstly, the mounting
position of the robot base and the ball iscalibrated to obtain its position and attitudetransformation matrix At , and the movement
trajectory of the robot is:

Where

At

Rp
0

Pp
1

（12）

R p is the attitude matrix shown inequation ,

Pp

500
300
200
1

In this paper, based on a 5-DOF cutting robot,
the kinematics model of the robot is established,
and the forward and inverse kinematics
algorithm of the robot is given. According to thecylindrical cutting process requirement of thespherical tool at any orientation, the equivalent
axis Euler transform method is used to establish
the pose transformation matrix of the coordinate
system and the spherical coordinate system
which coincides with the axis of the cutting
cylinder, and then the cylinder is solved
according to the geometric projection method.
The orthogonal intersecting line equation of
spherical cutting is given, and the attitude
planning algorithm of spherical cutting is given,so that the intersecting line model and cutting
attitude of the spherical and arbitrary azimuth
cylinders in the spherical coordinate system canbe obtained. On this basis, based on the posetransformation matrix of the robot and the sphere,
the robot's cylindrical cutting trajectoryalgorithm for spherical surface is designed.
Finally, the simulation experiment of the cutting
trajectory and the kinematics model of the robot
is carried out. The experimental results verify thecorrectness of the spherical cutting algorithm of
the cutting robot.

obtained:

Ar

0
0
1
0

5. Conclusion

coordinate systems, the posture of the cutting
tool in the xs , y s , z s coordinate system can be

Rs R y0100Note that during the cutting process, the Z-axis
direction of the robot's cutting tool attitude isdetermined by p0 p1 and remains unchanged.
The data for points p0 and p1 can beobtained from the CAD drawing parameters of
the sphere[10].

Thus, according to the conversion relationshipbetween the xs , y s , z s and xs , y s , z s

Rp

1
0
0
0

( x p , y p , z p )T [9].

References

4. Experiment

1. Liao Weidong, Li Chengneng, Wang Qiang, Liao
Wei. Kinematics Modeling and Simulation of 6R

This paper takes 5 degree of freedom cutting© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

368

Min Jun, Jiang Wenping, Zhao Huailin

Robot Tool End[J]. Mechanical Design &Manufacturing, 2018(09): 108-111.
2. Wang Yubao, Wang Shiyu, Li Beibei, Guo Fangda.
A Time Optimal Trajectory Planning Algorithm for
Industrial Robots with Improved Particle Swarm[J].
Small Computer Systems, 2018, 39(08): 1878-1881.
3. Zhang Yonggui, Li Hailong. Research on TrajectoryPlanning of Milling Robot to Remove Cracks in
Forgings[J]. Combined Machine Tool andAutomatic Machining Technology, 2018(07):1-4+9.
4. Wang Jun, Chen Zhilong, Yang Zhiyong, Qin
Zhengzheng, You Ying, Wei Qiong. Time Optimal
Trajectory Planning of Industrial Robot Based on
Improved DE Algorithm[J]. Combined Machine
Tool & Automated Machining Technology,
2018(06): 42- 46.
5. LIANG Zilong, DING Yufeng. A trajectory
planning method for free-surface polishing robots[J/OL]. Mechanical Science and Technology:
1-7[2018-10-09].https://doi.org/10.13433/j .cnki.10
03-8728.20180071.
6. Chen Xuefeng. Research on trajectory planningalgorithm
ofpalletizing
robot[J].Control
Engineering,2018,25(05):925-929.
7. Liao Weiqiang, Zhao Changming, Ma Wenbiao,
Zhang Tie. Trajectory Planning of Industrial Robots
in Five-Star Welding Process[J].Machinery Design& Manufacture, 2018(04):231-234.
8. LI Li, SHANG Junyun, FENG Yanli, HUA Yawen.
A review of trajectory planning research forarticulatedindustrial
robots[J].
ComputerEngineering and Applications, 2018, 54(05): 36-50.
9. FANG Jian, SONG Yu, ZHU Maofei, ZHU Dequan,
LIU Yongbo, JIAO Jun. Time-optimal trajectory
planning
forpalletizing
robots[J].Control
Engineering,2018,25(01):93-99.
10. Dong Jiajia, Wang Taiyong, Dong Jingchuan,
Zhang Yongbin, Tao Hao. Improved B-spline curve
applied to 6R robot trajectory optimization[J].
China Mechanical Engineering, 2018, 29(02):193-200.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan369

Emotional prosody processing between artificial voices and human voices
Qiang Wei1*, Xiaoyu Zhang1, Xiao Yang1, Yixin Lin2, Qingming Liu3
1

School of Education, Jianghan University, China;

2 School of Energy and Power Engineering, Huazhong University of Science and Technology, China；
3 Wuhan Rayson School, China

E-mail: * weiqiang@jhun.edu.cn
Abstract

To evaluate the time course of changes in emotional prosody processing between artificial
voices and human voices (happy, neutral), the present study examined behavior experience andevent related potential component. The emotion of artificial human voice and brain signal were
evaluated while the presentation of audio from each conditions (happy-natural, neutral-natural,and synthetic speech). The results showed that there were differences in the time-course
distribution of P800 and the intensity of amplitude between emotional prosody processing
induced by artificial voices and human voices.
Keywords: P800, artificial voice, synthetic speeches, human voice, emotional prosody processing, event
related potential

1. Introduction

interactive experience, the developers of synthetic speech
and researchers are devoted to the study of therelationship between emotion, natural voice and artificial
voice.
There are many factors would affect the emotional
information in speech, such as volume, pause, syntax andso on. Among them, it is generally considered that one of
the most influential factors is prosody3. The semantic is
related to the content, grammar and context of thesentence itself, and often has an impact on the prosody4.
In order to exclude the semantic influence, this study
chooses those speech materials which have the samegrammar and have no semantic tendency in their own
content.
At present, many of the methods to evaluate theemotional information in synthetic speech are often
compare the synthetic speech build by new technology
with the real recording models, using EMOS, for example.
But there is little evaluation based on cognition
processing of the consumers, users of SPIN or audiences.
In this study, we analyzed the EEG recorded duringlistening to natural and artificial speech materials, inorder to understand the differences in the processing of

Human emotional voice materials are generally divided
into three categories: natural speech, performance speech
and induced speech. Natural speech refers to the naturalexpression of the speaker's emotions in real situations.
Compared with the latter, the emotional information ofnatural speech is real and direct. Researchers generally
use the speaker's speech to simulate the pronunciation ofa specific emotional state in the laboratory as the natural
voice material1.
Speech Interface (SPIN) is also expected to have thefunction of transmitting emotional information. There are
two forms of speech interface, one is real recording, theother is synthetic speech that is kind of artificial voice.
Synthetic speech refers to the use of human vocal channelsound mechanism, resonance frequency or the resonancepeak produced by the "voice" to simulate the humanpronunciation of the speech. Compared with the real
recording, the synthetic speech generated by the
text-to-speech technology has the advantages of lessresource consumption, flexibility, objective reality and soon2. In order to bring users a more real and effective370

Qiang Wei1*, Xiaoyu Zhang1, Xiao Yang1, Yixin Lin2, Qingming Liu3

natural speech and synthetic speech. Furthermore, it
provides a standard for evaluating the emotional
information of synthetic speech and promotes thedevelopment of affective artificial voice.
2. Experiment2.1. Stimuli
Total 150 sentences were on the same syntactic pattern(subject + predicate + object). Every single sentence wasmoderated in length (6 ± 1 Chinese characters) andwithout semantic orientation in emotion.
The natural voice is 100 speeches chose fromCHEAVD (Chinese natural emotional audio-visual
database) developed by the Institute of Automation of theChinese Academy of Sciences. Half of the speeches werelabeled as happy, and others were neutral.
The artificial voices were 50 synthetic speeches build by
text-to-speech conversion software. After denoise methods,
the total speeches (128bps) had same decibel quantity.
To verify the validity of speeches of natural voiceclassification, 35 students without dysaudia participatedin evaluation test. Participants were required to score theemotion in speeches when the 150 speeches were random
broadcasted, using the 7-point Likert scale. 33 data sets
were tested by One-way ANOVA. The result shows
significant difference between Happy-natural speech andNeutral-natural speech (p=0.032<.05), Happy-natural
speech and Synthetic speech (p=0.027<.05). But
Synthetic speech wasn’t different from Neutral-natural
speech. (p=0.827＞.05).
2.2. Participants
29 right-handed undergraduate students were participatedthe experiment. After data analysis, there are still data of
19 participants (10 male, 9 female; average age 19.9
years) can be used.

key (1 (very angry), 2 (angry), 3 (neutral mood), 4(happy), 5 (very happy)) with right hand.
The EEG was recorded from 64 scalp electrodes
located at standard International 10/20 system by Brain
Amp (Brain Products, Germany) for 2200ms/trial (500
Hz sampling rate; 0.05-100 Hz band-pass), and was
re-referenced by FCz. EEG recording started 200ms
before the presentation of first word in speeches. The
impedances were below 5kΩ. And the positions ofelectrodes were located by the left and right mastoids.
The vertical and horizontal electrooculograms (IO) wererecorded to exclude the amplifier saturation, ocular
artifacts, and movement artifacts from averaged ERP oftrials.
2.4. Data analyses
The Independent Component Analysis (ICA) was applied
and returned the artifacts induced by eye-blinks andmuscular movement. Averaging the EEG amplitude that
responded correctly by participants in different three
conditions (happy-natural, neutral-natural, and synthetic
speech).
Descriptive statistics was applied to five point scales.
According to the averaged ERPs, the research pickedfollowing electrodes: FCz, FZ, Cz, F3, F4, FC1 and FC2
as one of the within-subject factors, and the time point of
the vertex between 400-1000ms as independent variable.
Than Root Mean Square (RMS) of voltage value at the
sampling time points (400-1000ms) recorded from theseven electrodes (FCz, FZ, Cz, F3, F4, FC1 and FC2)
was computed in three conditions (happy-natural,
neutral-natural, and synthetic speech). After averaging
the RMS of electrodes, a repeated measures ANOVA was
performed to compare the averaged RMSs in three
conditions.
3. Results

2.3. Procedure and data acquisition

3.1. Behavioral data

Using Stereo speakers, participants were presented
aurally with speeches in three conditions (happy-natural,
neutral-natural and synthetic). The 150 speeches weredivided into three blocks of 50 trials in random order.
Participants were required to score every speech afterof which presentation was finished by pressing numeric

The descriptive statistics was applied to present the meanand standard deviation (SD) of the emotion evaluationcollected from participants in three conditions(happy-natural, neutral-natural, and synthetic speech).
The descriptive statistics is shown in Table 1.

371

Emotional processing between artificial

Happy-natural
Neutral-natural
Synthetic

3.2.2. RMS analysis
The results showed that the time-course differences alsopresented in averaged RMSs of voltage value in three
conditions (see Fig.4). Comparing the RMSs throughone-way repeated ANOVA, the significant difference
existed in three conditions (F(1,15)=28.95, p<0.05).

score
3.94±0.57
2.97±0.33
2.98±0.14

Table 1 descriptive statistics of the emotion evaluation

3.2. Electrophysiological results
Starting from the first word onset, ERPs had covered all
emotional processing. Averaging the ERP trace in three
conditions (happy-natural, neutral-natural, and synthetic
speech), from which the emotion processing between
different conditions had time course difference (seeFig.2).

Fig .4 averaged RMS in three conditions

4. Discussion
The results showed that there were differences in theintensity and the late processing between cognitive
processing induced by artificial voices and human voices,which provided the proof that the ERPs can be used toevaluate the emotion information included by the prosodyof synthetic speech.

Fig. 2 the averaged ERPs in tree conditions of a typicalelectrode

3.2.1. Time-related characteristics of vertex
There is a positive amplitude peak presenting between
400-1000 ms in three conditions. The time point of vertex
between 400-1000 ms from the electrodes (FCz, Fz, Cz,
F3, F4, FC1 and FC2) of 15 participants in three
conditions (happy-natural, neutral-natural, and synthetic
speech) was computed as independent variable. Two
factors repeated measures ANOVA taking conditions andelectrodes as repeated measures were applied to verify
the interaction and significant effects.
No interaction effect between conditions andelectrodes (F(12,7)=.83; p=.63＞05). The main effect ofelectrodes wasn’t significant (F(6,13)=1.38; p=.29＞.05).
But the main effect of three conditions (happy-natural,
neutral-natural, and synthetic speech) is significant(F(2,17)=52.70; p=.00<.05), and theresults of paired
comparison for three conditions was shown in following
Table (see Table 3).
Mean
SD
p
Happy -Neutral
49.13
5.90
.00*
Synthetic–Happy 30.61
6.12
.00*
Synthetic–Neutral 79.74
7.75
.00*

4.1. Comparison
betweenbehavioral
electrophysiological results

andThe results of five point scale for emotional evaluation
showed that the self-report can distinguish the neutral
speech from happy speech, but can’t tell the differences
between emotional information communicated by neutral
natural speech and synthetic speech. ERP covered theshortage of behavioral evaluation by a different time
distribution of positive amplitude peak in which thevertex of neutral natural speech was closer to the onset
than synthetic speech.
The comparison proved the rationality of theclassification that affective synthetic speech can bedivided into emotional speech and neutral speech.
4.2. Positive amplitude peak presenting between
400-1000ms
Electrophysiological results demonstrated that positive
amplitude peak presenting between 400-1000ms had

Table 3 main effect of three conditions

372

Qiang Wei1*, Xiaoyu Zhang1, Xiao Yang1, Yixin Lin2, Qingming Liu3

made the time-course difference in three conditions(happy-natural, neutral-natural, and synthetic speech)significant. To better understand the function of thepositive amplitude peak during emotion processing, we
compared the component with other typical positive
component related to emotion and language processing.
Because of the experimental paradigm that the onset
was the beginning of the first word in each sentence andparticipants should react after the end of whole sentence,
it is hard to judge the exact time position of positive
amplitude peak presenting between 400-1000ms. So
first at all, we take it as P600, a ERP component elicitedby syntactic errors, which didn’t square with the fact thatthe syntactic factor was controlled4.5.6.
Closure Positive Shift (CPS), a late positive component
was taken into consideration, who is related to theprosodic boundary showing in long sentence withcomplexity of syntactic. And it also was usually foundedin the interaction of syntactic and prosodic in sentencecomprehension. Comparing with the CPS, the positive
amplitude peak in our research was founded in sentencewith the same simple structure (subject + predicate +object), hardly needed using prosodic boundary4.5.6.
P800 is an important signal of prosodic processing,founded in cross-splicing paradigm of semanticallyincongruous and prosodically incongruous. But it can
only be induced in the situation where the participants
were required to pay attention to prosodic of sentences.
Some of the researchers take it as a component that is areal-time conversion from neutral to emotional
stimulation4. But it was also founded in the incongruousof modality prosody (assertion- question)7.
Controlling the syntactic and structure of sentence, ourresearch asked participants focused on the emotion
information conveyed by speech, which may lead to theattention of participants paid on intonation that is a maincommunicated way for prosody. So the positive
amplitude peak can be P800 the component demonstrated
the attention processing of prosodic, emotional prosodic
especially.
It would be doubt that P800 who is a late positive
component didn’t show at 800ms in our research.
However, considering the difference of paradigm that ourparticipants were asked to focused on the emotion ofwhole sentence while the participants of other research
reacted in the middle of material presentation, it isreasonable there was a hurry presentation of peak.

The time-course difference of P800 distribution inthree conditions suggested that:
The synthetic speech had deficiency of emotional
prosody, which didn’t trigger the attention of emotional
prosody earlier than natural voice. And there may was anegative correlation between task difficulty of attention
and the presented time of P800, because the P800 in
neutral-natural speech condition showed earlier than
happy-natural speech.
4.3. RMS reflexed the intensity of attention
While P800 component demonstrated the attention
processing of emotional prosody, the RMS difference
may show the intensity difference of attention triggeredby processing of emotional prosody. The RMS
differences were significant which proved that ERP canbe used to evaluate the synthetic speech.
5. Conclusion
The results of present study suggested a new way using
ERP to evaluate the emotion information included byvoice, especially the intonation difference induced byemotional prosody in speech. In this way, the synthetic
speech can be tested by audiences and users of speechinterface. P800 can be a successful standard to classify
emotional prosody.
References
1. Xiao Yang. ERP study on the difference of emotion
perception induced by natural voice and artificial voice [J].
Journal of Wuhan Vocational and Technical College, 2017,16(5):84-90.
2. Jianxiong Chen, Youhong Jian, Shimiao Wang, a study on
emotion conveyed in synthesized voice (Shandong,
Shandong University of Science and Technology, 2007)
3. Zhuangluan Su. Affective Speech Synthesis[D]. Universityof Science and Technology of China, 2006: P19-P25.
4. Aish Chen, Xuhai Chen, Yufang Yang. The Time Courseof Verbal Emotional Prosody Processing[J]. Progress in
Psychological Science, 2009, 17(6):1109-1115.
5. Ze Yu, Guiqin Ren. Overview on ERP Research of
Prosody Processing[J]. Journal of Liaoning Normal
University (Social Science Edition), 2012, 35(2):201-205.
6. Xuhai Chen, Xiaohong Yang, Yufang Yang.
Neurophysiological Mechanism of Implicit Processing ofVocal Emotion Transition [J]. Acta Psychologica Sinica,
2013, 45(4):416-426.

373

Emotional processing between artificial

7. Corine Astésano, Besson M , Alter K . Brain potentials
during semantic and prosodic processing in French[J].
Cognitive Brain Research, 2004, 18(2):172-184.
8. Pihan, H. Cortical activation patterns of affective speechprocessing depend on concurrent demands on the subvocal
rehearsal system A DC-potential study[J]. Brain, 2000,
123(11):2338-2349.
9. Dutoit T . An Introduction to Text-to-Speech Synthesis[M].
Kluwer Academic Publishers, 2001.

374

How people are affected by emoticons: An ERP study
Qiang Wei1*, Jingxuan Huang1, Yu Han1, Yixin Lin2
2

(1 School of Education, Jianghan University, China
School of Energy and Power Engineering, Huazhong University of Science and Technology, China)*weiqiang@jhun.edu.cn

Abstract

The aim of this study was to investigate the ERP component evoked by emoticons with different
attributes and the reaction times during the LDT. The behavioral results demonstrated the significant
promoting effect of emoticon for language processing with shorter reaction time under both a positive
emoticon and a negative emoticon compared to participants under a nonsense emoticon. The greater
amplitude on N170, P300 provided more evidence for the promoting effect of emoticons.
Keywords: emoticon, Lexical decision task, ERP, language processing

1. Introductionthat it is an stimulus that attracts more attention andexpresses more emotion-related information than normal
text stimuli[2]. On the other hand, it can be easily
retrieved, summarized and analyzed within different
social networks and forms.
The LDT is a common instrument to measure implicit
emotional processing which commonly used in affective
priming paradigms to evaluate the emotional state ofsubjects.Previous studies showed shorter RT ofparticipants for emotion-related targets when compared toneutral targets.
In this study, we used the emoticons of different
attributes for prime stimuli on Lexical Decision Task, inorder to reveal a corresponding relationship between
emoticons and behavioral results of LDT. And then,reveal the specific impact of emoticons during language
processing by compare the amplitude and regions within
EEG data.

Emotion is an essential project in psychological research.
It is generally considered to be
the subjectively
experienced and conscious mental reaction of individuals
to the relationship between external events and internal
needs which usually directed toward a specific object
and typically accompanied by psychological andbehavioral changes in the body.[1]Furthermore, in thefield of Man-machine dialog and network information
mining, the emotions of network users and publishers
have been trying to be predicted by researchers throughseveral different means such as models establishment andtext analysis. According to this, effects arising fromemotion-related information in the network gain more
and more interests.
Emoticons, as a new kind of emotion-related text
punctuation, became increasingly popular and werecommonly used on internet forums with the development
and widespread of internet. Different from stylized
pictures (such as Emoji, QQ face) , emoticons are consistof characters including punctuation marks, numbers andletters to express a person's feelings or mood. This
resulted in that emoticon combined advantages ofstylized pictures and text messages.On the one hand, itholds a natural salience as face pictures did, which means

2. Experiment2.1. Participants
Seventeen right-handed healthy undergraduates (9 males
and 8 females) from Jianghan University were recruited
to participate in the study. All of the participants were

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan375

Qiang Wei, Jingxuan Huang, Yu Han, Yixin Lin

native Chinese speakers free from any history of mental
diseases or neurological disorders. Their visual acuity
was normal or corrected-to- normal. Their age ranged
from 18 to 22 years with mean age of 19.42 years. All
participants provided written informed consent and werecompensated for their participation.

appeared for 800 ms again. This experiment consisted of
300 trials and include 4 breaks after each 60 trials.

2.2. Stimulus materials
The entire experiment consisted of 150 emoticons as
prime stimuli and 300 words as target stimuli. The
emoticons contained three conditions: 50 emoticons ofpositive mood (e.g., ヽ ( ￣ ▽ ￣ ) ﾉ ), 50 emoticons ofnegative mood (e.g.,(╥╯^╰╥)) and 50 randomcharacters without any mood (e.g.,(=’+) ).For the LDT a
set of 300 target stimuli were used. 150 neutral words
were extracted from theModern Chinese
Dictionary(e.g., “桌子(table)” “药品(medicine)”). 150
Pseudo-words were also generated out of a words listed
which composed of those neutral words in this study.
Here, neutral words with a length of two Chinese
ideograph were manipulated by changing single Chinese
ideograph with another word(e.g., “桌药”).Additionally,
the selected emoticons in this study were based on the
SOHO Pinyin IME 7.5.
The stimuli and recording triggers were presented
using the E-prime 2.0 software package.

Fig.1.Experimental design

2.4. Electroencephalogram(EEG) recording andanalysis
An EEG was continuously recorded for each participant
during the task. The reference electrodes were placed on
the left and right mastoid, whereas the ground electrodewas placed on the center point between Fpz and Fz.An
electrooculogram (EOG) was recorded from electrodes
placed at 10 mm from the lateral canthi of right eye.Only
when the electrode impedances were maintained below 5kΩ would this experiment started. The high pass filter ofthe EEG signal was set to 0.5 Hz, and the low pass filter
was set to 100 Hz with a sample rate of 500Hz. Data
were transferred off-line and vertical electrooculogramand horizontal electrooculogram were corrected .
EEG data sets were segmented for the epoch from 200
ms before the onset of the target appearing on the video
monitor to 1000 ms after the onset, with the first 200 ms
pre-targets as a baseline. Trails containing amplifierclipping, bursts of electromyography activity, or
peak-to-peak deflection exceeding ±50μV were excluded.

2.3. Procedure
The subjects were asked to sit in a sound-attenuated room
200 cm away from a computer-controlled screen, on
which the stimuli were presented centrally. Before the
ERP experiment formally started, they were requested tolook at a fixation cross in the centre of the screen.
As shown in Fig.1, each trial began with a fixation
cross against a black background for 1000 ms, followed
by a black screen for 800ms.Then, a emoticon withdifferent moods appeared for 500ms followed by ablack screen for 300 ms. In this step, there is no need to
do any button pressing for subjects but staring at thescreen. The written stimuli was then shown, and theparticipants had to determine whether that written
stimulus was a Real-Word or not by pressing a keypad(key “Z” for Real-words, key “X” for Pseudo-words )within 3000ms.The stimuli would then disappear as soon
as the subjects pressed button. After that, a black screen

3. Results
3.1. Behavioral results of LDT
The results of 3×2 two-way ANOVA on emoticons(positive/negative/random characters) and written
stimuli(real-word/pseudo-word) showed a significant
main effect of the emoticons [F(2,5) = 3.302; p < 0.05]; asignificant main effect of the written stimuli. The [F(1,2)= 531.471; p<0.0001]; and no significant emoticons ×
written stimuli interaction [F(2,2) = 0.748; p>0.05]. The
results of ANOVA on emoticons revealed shorter RT forthe subjects under the positive stimuli(p=0.017<0.05) and© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

376

How people are affected

the negative stimuli(p=0.047<0.05) compared to randomcharacter. However, there are no significant difference
between the RT of the positive stimuli and the negative
stimuli(p=0.56).

positive condition and negative condition(p=0.586 ﹥0.05).
Further analysis showed no significant differences
between the amplitude of responses to positive-moodtarget, negative mood target and nonsense target in
300ms-330ms time window at frontal region.

3.2. Electrophysiological data
There were clearly N170 component at all positions ,P300 and LPP(Late Positive Potential) at centro-parietal
region, N300 component at frontal region during the
LDT.
According to the GFP (Global field power) curve on
Figure 2, the event-related potential data at the three time
windows including: 140ms-180ms, 230ms-300ms,
300ms-330ms were analyzed. The value of global fieldpower were obtained by computing the quadratic sum ofelectricalactivityoverchosenelectrodes[Centro-parietal(Cz,CP1,CP2,P3,P4);Frontal(Fp1,Fp2,Fz)]at the time windows.
A repeated measures ANOVA on mean amplitude for140-180ms time window at centro-parietal region showeda significant greater amplitude for positive-emoticons andnegative-emoticons when compared to random characters.
[positive(p=0.000<0.0001), negative(p=0.000< 0.0001)].
However, there are no significant differences between thepositive condition and negative condition.(p=0.115 ﹥0.05)

4. Discussion
In this study, the participants browsed emoticons that
expressed either the positive, negative or neural mood,and then recorded their reaction time and EEG data oflexical decision task. The behavioral results of LDT
showed that the participants had a significant longer
response time for nonsense target compared toemotion-related targets.
The difference of the reaction time on LDT could befurther interpreted by evidences from EEG data. In the140-180ms time window identified as N170 component,
difference be detected and emotion-related target elicited
significant larger amplitude responses than nonsense
target, whereas the similar pattern of results was observedin the 230ms-300ms time window.
The results suggests that emotion-related target exert
an influence to the visual-orthographic processing andworking memory in Lexical decision task3, which may berelated to more effort have been devoted in these twoprocessing4. Compared to the nonsense characters,
emoticons hold more emotional information and semantic
information1, which might initiate affective priming
effect or semantic priming effect.
Different emotional types showed a general consistent
trend in these two part. More research is needed to further
detect the relationship between emotion types andvocabulary processing.
References
1. Lane R D , Reiman E M , Bradley M M , et al.
Neuroanatomical correlates of pleasant and unpleasantemotion[J]. Neuropsychologia, 1997, 35(11):1437-1444.
2. Wagenbreth C , Rieger J , Heinze H J , et al. Seeing
emotions in the eyes – inverse priming effects induced byeyes expressing mental states[J]. Frontiers in Psychology,
2014, 5(3):1039.
3. L. Carretié, et al. "N300, P300 and the emotional
processing of visual stimuli." Electroencephalography andClinical Neurophysiology 103.2(1997):0-303
4. Van H S , Seyffarth S , Pexman P M , et al. An ERP
Investigation of Vertical Reading Fluency in Scrabble(®)
Experts.[J]. Brain Research, 2017, 1667:1-10.

Fig.2 Global field power and amplitude topographies(Cz,Cp1,Cp2,P3,P4) for positive emoticons, nonsense
emoticons and negative emoticons.

A repeated-measures ANOVA on mean amplitude for230-300ms time window showed a significant greater
amplitude for positive-emoticons than randomcharacters[p=0.009<0.05], a significant greater amplitude
for negative-emoticons than random characters(p=0.040< 0.05) and no significant differences between the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

377

Single Chinese Character fragments: An ERPs Study On Orthographic Neighborhood Effect
Qiang Wei 1*, Yunfei Wang1, Chen Song 1, Yixin Lin2
1

1 School of Education, Jianghan University, China;

2 School of Energy and Power Engineering, Huazhong University of Science and Technology;

E-mail: * weiqiang@jhun.edu.cn

Abstract

This study is aimed at discussing Orthographic Neighborhood Effect and brain activity during theprocessing of Chinese characters. In this experiment, event-related brain potentials (ERPs) triggered by
Chinese single-word fragments in Chinese character completion and naming experiments were observedand compared. The results showed that the "multiple solutions" caused N400 to be larger than "onesolution." Multiple-solution -word are analogized to large families, they are more intense than the
Pseudo-words and single-words, which are in line with the interaction activation model and have moreresources in the cognitive processing process. The glyph processing, the association process, and thememory phase of Chinese characters have an impact.The N400 component may not only mean the
integration of the meaning of the word, but may also be related to the process of adjusting theorthographic of Chinese character fragments.
Keywords: Chinese character fragments, orthographic neighborhood effect, ERP

1. Introduction

for the input and recognition of Chinese characters incomputers.
The orthogonal neighborhood effect of words is an
important evidence to support parallel processing inhuman Chinese character recognition mode. However,
the orthographic neighborhood effect is more thoroughly
studied in the phonetic alphabet, and the conclusions of
the opposite are found in the existing Chinese. Therefore,
the orthographic neighborhood effect of studying Chinesecharacters can not only be explained the cognitive
processing of Chinese characters and can explore thedifferences between phonetic and ideograms. This study
is aimed to explores the influence of the size of theorthographic neighborhood effect on the extraction ofChinese characters through the naming experiment of theresidual word, and use Event-related potentials (EPRs) to
explore the time course associated with glyph extraction.

How to input text into the computer more accurately andefficiently makes the computer able to quickly andcorrectly identify the input text has become an importantresearch issue. In general, Chinese characters containmore visual information than alphabetic writing. Due tothe particularity and complexity of Chinese characters, itcannot be directly and accurately recognized bycomputers like alphabetic writing. When there is damage
or stain, it is difficult for the computer to recognize
Chinese characters quickly and accurately by the current
method. The recognition of words can still be improvedin accuracy. On the contrary, humans tend to have higher
accuracy
when
performing
Chinesecharacterrecognition.For this reason, exploring the neurosciencemechanism of the human body in the process of Chinesecharacter recognition may helps us to provide new ideas

2. Experiments© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan378

Qiang Wei, Yunfei Wang, Chen Song , Yixin Lin

2.1. Subjects

Participants were asked to report their answers as soon as
possible when they saw the Chinese characters displayedon the screen.

17 undergraduate students (10 males, no disease, normal
vision after correction, no history of physical or mental
illness, all right-handed, 18 to 22 years old, with anaverage age of 19.11 years old 10 females participated in
the experiment. They will be given some compensation
after participating in the trial.

Fig.2.

Experimental procedure

2.4. ERP

2.2. Stimulate
In this experiment, 100 words were selected from theappendix of the Modern Chinese Dictionary "7000 HighFrequency Words" as materials, including Multi-words,
pseudo words and single words. Multi-word means that a
residual word can be complemented into multiple
Chinese characters. Multi-word control position changes.
A single word means that the word can only becompleted as one Chinese character. Pseudo-words
cannot make up for common Chinese characters but
conform to Chinese characters’ orthographic Chinesecharacters. The specific information of these materials areshown in Fig.1. There is no statistical difference between
single word and multi-word, number of residual words,
number of pens, and total number of pens. The residual
word is produced by Microsoft's own software Educed. It
is presented in the PowerPoint by Song, the 34th word isdisplayed in 960×720 pixels, 96dpi×96dpi, and the BMP
format is exported.

recordings
and analysis
EEG was recorded by the ERP recording and analysis
system of Brain Products, Germany, in accordance withthe 64-electrode cap of the International 10-20 system.
The left and right mastoids are used as referenceelectrodes. When measuring, In addition, the recording of
the horizontal electrocochleogram (HEOG) was recordedon the outside of the eyes, and the verticalelectrocochleogram (VEOG) was recorded under theplacement electrode and the right eye. The scalp
resistance of each electrode was kept below 5 kΩ. The
filter band passes through 0.05 to 100 Hz and thesampling frequency is 500 Hz. After continuousrecording is completed, the data is processed offline, and
VEOG and HEOG are automatically corrected to exclude
other artifacts.
The analysis time course is 1000ms after theoccurrence of the residual word, and the baseline iscorrected for 200ms before the occurrence of the residual
word. Data with artifacts such as blink, eye movement,
and electricity were excluded, and the exclusion criterion
was ±50 μV.
The total average amplitude of ERP and thedifference topographic map of multi-word, pseudo-wordand single-word conditions showed that there weresignificant differences between the left and righthemisphere midline and brain forehead conditions, andO1, C3, Cp1, Cz, Fc1, Fz, Pz and FCz performs average
volatility statistics.
Three-factor GLM Repeated Measures for ERP
wave-forms with SPSS 17.0 statistical software, factors
including type (multi, pseud word and single), electrodeposition (7 electrodes selected) and 18 time windows(100ms-1000 ms) Two-way ANOVA was then performedon different time periods and three types of materials

Fig.1. Examples of character Stimuli

2.3. Procedure
Experimental materials were presented by E-prime 2.0
from Psychology Software Tools, Inc. There were 5
experimental groups in the experiment. Pilot study
contained two types of two Chinese characters, and four
formal experiments randomly appeared one of 300 three
types of residual characters.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan379

Single Chinese Character fragments

conditions of the Occidental lobe during the 200-250 ms
period (p<0.01), and the the “Multiple-solution” cancaused a larger negative wave than the Pseud word andthe “One-solution”.

according to different electrodes. Finally, one-way
ANOVA was performed for the three conditions of eachtime period.
3. ERP Results
The three-way ANOVA analysis between the condition ×electrode point × time period was performed on theexperimental data. The main effect of the condition was
found to be significant (F(2,102)=4.249, P=0.014); themain effect of the electrode was significant (F(6,34) =62.553, p< 0.001); the main effect of time was significant(F(17, 12) = 35.478, p< 0.001). At the same time, theconditional and electrode interaction effects weresignificant (F(12,204)=3.622, p<0.001); time andelectrode interaction were significant (F(102,204)=4.180,
p<0.001), conditions and time The interaction was not
significant; the three interactions of condition, electrodeand time were not significant.Differential topographic
map of three conditions is shown in the Fig.3.

Fig.5.the averaged ERPs in tree conditions of Pz channel

In this study, the difference between the Pseud
words and the true words in the 300ms-400ms parietal
electrodes (P3/P4, P7/P8) was significant (p<0.05).
In this study, multi-words stimulated more negative
waves, in the 400-550 ms time window, compared tosingle words. Moreover, the P600 component appeared inthis experiment, and the component was most obvious at
the Fz point, and the multi-word and the single word
were significantly different (p<0.05).
After the P600 component, there is a gentle slow
wave with an upward trend, which can be regarded as an
LNC component. In this experiment, the three types ofconditions are significantly different in the window of
750ms-800ms. Multi-words lead to larger amplitude LNC
components.

Fig.3. Differential topographic map of three conditions
4

Fcz

2

Multi-words
Pseud words
Single-words

4. Discussion

0

There were no significant differences in the three types ofmaterials in the early compositions. Explain that the true
word and the Pseud word are combined with the positive
character, and it does not cause the difference caused by
the non-conformity. In the subsequent of 200-250ms
window, the the “Multiple-solution” caused morenegative waves than the “One-solution” and the Pseud
words. It can be understood that since the Chinesecharacters are stored according to the template, the morenegative the collision occurs, the more negative waves
are generated. strong.

-2-4-200-100 0 100 200 300 400 500 600 700 800 900
Fig.4. the averaged ERPs in tree conditions of FCz channel

3.1. Time-related characteristic
An N1 component appeared during the early 100-200 ms
window, which has the largest amplitude in the unilateral
Occidental region and also in the posterior and frontal
scalp. There was a significant difference between the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan380

Qiang Wei, Yunfei Wang, Chen Song , Yixin Lin

Donchin proposed an updated theoretical model ofthe psychological mechanism of P300 component. That is,when a certain information appears, it is necessary tocorrect the original psychological representation to adapt
to the strategy of the future task. The process of thiscorrection produces P300 component, the greater themagnitude of the correction, the more inconsistent withthe original cognitive representation, the greater theamplitude1. In this experiment, the difference between
the Pseudo-words and the true words in the 300ms-400ms
parietal electrodes (P3/P4, P7/P8) was significant(p<0.05). This result is consistent with Wei Jinghan'sresearch2, P300 appeared only after the subjects
completed the classification or evaluation.
In this study, multi-words stimulated more negative
waves, in the 400-550 ms time window, compared tosingle words. The waveform and latency of thiscomponent are consistent with previous studies of theN400 component3. However, it is not consistent with theHolcomb’s study. In his study, the N400 effect can last
up to 1000ms, but there was a P600 component appearedin Huang’s study and this study. This may be due to adifferent type of task.The P600 component was originallycalled syntactic drift. It is related to the difficulty ofsyntactic integration. Recently, some studies havesuggested that the P600 component is greatly affected bysemantic or lexical processing mismatch. And this effectis often accompanied by the emergence of the N400. In
Wang's research, the type of task does not involve thesemantics of vocabulary, but the glyph. Holcomb's
experiments involve vocabulary judgment. In thisexperiment, the P600 effect was most obvious at the Fz
point, and there was a significant difference between themulti-word and the single word (p<0.05).

similar to the residual is extracted and updated, given the
hypothesis.A Pseud word will cause a larger positive bias
volatility than a true word. The glyph information
obtained before is adjusted in the vicinity of the frontal
lobe and the parietal lobe between 400 ms and 600 ms to
conform to the task. Multi-word cause a larger negative
wave than a single word. During the 600ms-1000ms
window, the obtained Chinese character voice
information is maintained on the vicinity of the parietal
lobe and the frontal lobe. Multi-words are morevolatizing in window time than Single words and Pseud
words. The conclusions of the study support aninteractive activation model in which multi-word
processing appears to induce a larger N400.
At the same time, the results of this study also
remind us to pay attention to the influence oforthographic neighborhood effect on Chinese character
recognition in the future. When designing the computer
Chinese character recognition program, the difficulty of
the “Multiple-solution” retrieval is also should beconsidered to improve the recognition efficiency.
References
1. Fabiani M, Karis D, Donchin E. Effects of mnemonicstrategy manipulation in a Von Restorff paradigm[J].
Electroencephalography and clinical Neurophysiology,
1990, 75(1): 22-35.
2. Wei Jinghan, Pei Peiyu, Zhang Dongsong, Pan Yitian.
ERP Features of the Meaning of Chinese Characters in
Full-view Chinese Characters and ERP Screening ofChinese Character Recognition[J]. Acta Psychologica
Sinica,1995,04:413-421.
3. Wang Q, Zhang J. N400 solution effect of Chinesecharacter fragments: An orthographic neighborhood size
effect[J]. Brain research bulletin, 2011, 86(3): 179-188.
4. Jinrui Zhang, Research and implementation of intelligent
transportation system gateway based on Internet of things(JlLin, Changchun University of Science and Technology,
2014)

5. Conclusion
The results of this experiment can basically outline thetime course of Chinese character processing in the brain.
Between the 100ms-200ms window, the Chinesecharacters near the parietal lobe undergo preliminary
visual feature processing, and the residual characters aredecomposed into feature information according to theorthographic rule. At this time, semantic processing is not
involved. Between the 200ms-400ms window, thecharacterization of the working memory is performed in
the parietal and frontal lobes, i.e., the characterization© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

381

Research On Embedded Electrical Impedance Measurement System
Tong Yin 1, Xiaoyan Chen 1*, Hongyi Yin1
1
College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;

E-mail: * cxywxr@tust.edu.cn
Www.tust.edu.cn

Abstract

This paper develops an impedance measurement system based on S3C6410 processor. An OK6410 hostboard as the controller platform and an AD5933 chip is used to collect impedance data, and the data is
exchanged with the controller with IIC bus protocol. Comparing with the measuring result of the Wayne
Kerr 6800B impedance analyzer with the accuracy of 0.05%. The system has the behaviors as lower
relative error 0.422% and higher signal-to-noise (SNR) ratio 64.11dB. After calculation, the averageabsolute error of the impedance phase of the system is 0.527°. The tests results verified that the system isreliable and flexible with a wide application prospect.
Keywords: impedance measurement, embedded system, AD5933, S3C6410

the degree of obstruction of an object by current, and is
the ratio of voltage U to current I, expressed in ohms (Ω),
represented by the letter Z. In general, the impedance is
a complex, the real part representing the resistancefeatures, labeled by R; the imaginary part represents
feature, labeled by X. Z can be expressed as follow:

1. Introduction
1.1. Impedance measurement technology
Impedance measurement technology is an important partof electronic measurement technology.
In the field of industrial production, the oil moisture
content can be measured by measuring the impedance ofmulti-phase flow [1-2]. In the field of biomedicine, thepulmonary respiratory process can be visualized andmonitored by measuring the impedance information ofhuman lungs [3-4].
It is widely used in biomedical, electrochemical,
industrial control, power grid control, etc.

Z

U
I

R

jX

Z

(1)

Where, is the amplitude of impedance, is the phase angle,
and the formulas for calculating and are according to Eqs.
(2) and (3).

Z

R2

X2

1.2. Principle of impedance measurement

arctanImpedance plays a very important role in the related
research on the electrical properties of materials. Fromthe impedance, a large number of electrical parameters
can be deduced to comprehensively analyze thesubstances. In physical electronics, impedance indicating

X
R

(2)
(3)

The conversion formulas of the real part and theimaginary part are using Eqs. (4) and (5).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan382

Tong Yin, Xiaoyan Chen, Hongyi Yin

R
X

Z cos
Z sin

2.2. DC-blocking drive circuit(4)

When measuring an object which unknown impedance,
the AD5933 will output a sinusoidal excitation signal
with a significant DC component, which will cause
polarization and affect the measuring precision in
TABLE 1. It shows the output impedances at different
gains and frequencies at 5V and 2.7V. In addition, theinternal amplifier of the AD5933 has a high output
impedance, which can affect the output signal adversely.
The DC bias and output impedance of different output
voltages under the voltage supply of 3.3V are shown in
Table 1.
The output voltage of this system is 1.98V. The DC offset
is fixed to 1.48V, and the DC bias voltage of the receiving
end of AD5933 is 1.65V. There is a potential differenceof 0.17V, which is easy to cause polarization occurs at(5)1.3. Measuring system frame
In the research, an OK6410 board with embedded
processor S3C6410 is responsible for running theoperating system and measuring programs. The dataacquisition and signal conditioning are carried out by
AD5933 chip. When it working, the instructions are sent
to the AD5933 chip through the analog IIC bus, whichmake the measurement circuit [5-6] simple. The AD5933
transmits a measured signal to the host board. After aseries of conversions and measurements, the signal isreturned to the host board and a measurement is finished.

Fig. 1. Hardware system frame.

the output and receiver.

Finally, the controller output the processing results to theoperating platform. The system frame is shown as Fig. 1.

Table 1. DC bias and output impedance ofeach excitation voltage.

2. Hardware design

Output excitation
voltage
1.98V
0.97V
383mv
198mv

2.1. Data collection
We use a direct digital frequency synthesizer (DDS) to
generate an alternative exciting current with shortconversion time, high frequency resolution and flexible
output. The signal is sampled using a high-speed analog
to digital converter (ADC). Then, the digital signal
processor (DSP) inside the AD5933 demodulate thesignals BY Fourier transform, the real and imaginary
information of the signals are extracted from themeasured impedance [7-8].

DC bias voltage
1.48V
0.76V
0.31V
0.173V

Output impedance
typical
200Ω
2.4kΩ
1.0kΩ
600Ω

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan383

Embedded electrical impedance measurement

To eliminate the 1.48V DC bias, a 47nF capacitor andtwo 50kΩ resistors are used to obtain a DC bias VDD/2= 1.65V, equivalent resistance R = 25KΩ. The circuit isshown in the Fig. 2. The cut-off frequency is set to 135
Hz, which is much lower than the lowest systemmeasuring frequency 10 kHz. The interference to theexcitation signal can be ignored. Resetting the DC offset
reduces the polarization and improves the accuracy of thedevice.

Fig. 3. AD5933 receiver circuit.

2.4. Feedback resistor
The measured signal is transferred to the ADC for
analog-to-digital conversion. In this step, the input
voltage of the ADC needs to be adjusted by some setting
parameters such as feedback resistor and gains to ensure
it within the linear input range of the ADC. If the input
voltage is too large, the ADC can be saturated and errors
occurs. If the input voltage is too small, it cannot carryenough measuring information and is susceptible to noiseand reduces measurement accuracy.
Due to the requirements of the measurement parameters
and the several factors affecting the output voltage, theexcitation voltage and the PGA gain can be fixed to asetting point. The feedback resistor needs to be properly
selected according to the magnitude of the unknown
impedance to ensure the input voltage within the range of
the ADC. The feedback resistor needs to select a
precision resistor with high precision, capacitive andinductive impedance.
A series of feedback resistor is considered to support
100Ω-200kΩ measurement listed in Table 2, and theoverall design of the data acquisition circuit is shown as
Fig. 4.

Fig. 2. Signal excitation circuit.

2.3. Receiving circuit
The current-voltage amplifier of the AD5933 issusceptible to the bias current, offset voltage, and CMRR
of itself, which resulting in inaccurate measurements. By
setting an amplifier with excellent characteristics such as
low offset voltage and low bias current at the receiving
port of the AD5933, the adverse effects of the selfintegrated amplifier can be eliminated and themeasurement accuracy can be improved. In this design,an AD8606 chip is adopted to build the excitation circuit.
The AD8606 has dual channels that can be designed as a
voltage follower and the required external converter
simultaneously on a single chip, which can reduce thecircuit complexity, save the manufacturing costs, andavoid the dual channels unbalance. It also can avoid

Table 2. Feedback resistance and its impedance measurement range.
Measurement Range
100Ω-1kΩ
1kΩ-10kΩ
2kΩ-20kΩ
10kΩ-100kΩ
20KΩ-200kΩ

Feedback Resistor
100Ω
1kΩ
2kΩ
10kΩ
20kΩ

inconsistencies caused by the using different amplifiers.
The circuit is shown in Fig.3.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan384

Tong Yin, Xiaoyan Chen, Hongyi Yin

The relative error and SNR ratio at each frequency arecalculated according to Eq. (6) and (7).

= −20

=

× 100%
∑ (

−

(6)
) /

(7)

is the average value of the measured impedance at theith frequency,is the actual value measured by theis theWayne Kerr 6500B impedance analyzer,
measured impedance at the ith frequency, E is the relative
error, and SNR is the signal noise ratio, n is the numberof measurements.
The Wayne Kerr 6500B impedance analyzer measures
the actual impedance of the resistor with an accuracy of
0.05%. Then the average relative error and signal to noise
ratio of the feedback path are calculated. The other four
feedback paths are tested in the same way as above. The
performance specifications of the five feedback paths areshown in Table 3.

Fig. 4. Data acquisition circuit. Including AD5933 chip andsignal conditioning circuit, DC amplifier circuit, receivercircuit and feedback resistor

In measuring, one of the feedback resistor is selected toguarantee most approximate to the measured object. The
hardware module is shown as Fig. 5.

Table 3. Relative error and snr ratio of
each of the five feedback paths.
Testedresistance
100
1k
2k
10k
20k

Feedback
resistance
330
4.7k
4.7k
51k
51k

Relative error
(%)
0.56
0.50
0.52
0.34
0.19

SNR(dB)
68.14
65.63
70.55
54.59
61.66

The average error of the system as 0.422%, and the SNR
ratio is 64.11dB. The results show that the system has
higher accuracy and signal to noise ratio.
3.2. Phase Accuracy Test
The system excitation voltage to 1.98 V. The test begins
at 10 kHz with 1 kHz incremental till 100 kHz and there
are 91 testing values acquired. The testing is repeated 100
times. After taking the average of the measured values,
the absolute error of each frequency point is calculated.
The Wayne Kerr 6800B impedance analyzer with anaccuracy of 0.05% measures the phase data used forcalibration. The calculated phase absolute error is shownin Table 4, where P indicates parallel and S indicatesseries.

Fig. 5. Hardware module. Including operation platform, dataacquisition module, and fixture.

3. System test
3.1. Accuracy and SNR
Taking 2kΩ testing as an example, we select 4.7kΩ as the
feedback resistor, the excitation voltage is 1.98 V, and
the PGA gain is 1. The test begins at 10 kHz with
1kHzincremental increase until 100 kHz. There are 91
testing values acquired. The testing is repeated 100 times.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

385

Embedded electrical impedance measurement

(a)

(b)

Fig. 6. Comparing the measurement results of this system with the measurement results of the impedance analyzer,
the black line is the result of the system, and the red line is the Wayne Kerr 6800B impedance analyzer result. Flow(a) is impedance data, and (b) is phase data.
Table 4 Absolute phase error of each r-c circuit.

Reform in Education from Tianjin Municipal Education
Commission of China (171005704B).

Absolute error
(°)
1.664

R(Ω)

C(nF)

Connect type

330

28

Parallel

330

32

Series

0.266

4.7k

3

Parallel

1.048

4.7k

3

Series

0.229

4.7k

16

Parallel

0.441

4.7k

16

Series

0.222

20k

3

Parallel

0.754

20k

3

Series

0.132

20k

16

Parallel

0.382

20k

16

Series0.132

References
1. Paolinelli LD, Yao J, Rashedi A. Phase wetting detection
and water layer thickness characterization in two-phaseoil-water flow using high frequency impedance
measurements [J]. Journal of Petroleum Science &Engineering, 2017, 157: 671-679.
2. Gao ZK, Dang WD, Xue L. Directed weighted network
structure analysis of complex impedance measurements
for characterizing oil-in-water bubbly flow [J]. Chaos,
2017, 27 (3):74-81.
3. Liu N, GJ Saulnier,JC Newell. A multichannel synthesizer
and voltmeter for electrical impedance tomography [J].
International Conference of the IEEE Engineering in
Medicine & Biology Society, 2003, 4:3110-3113 Vol.4.
4. Saulnier GJ, Blue RS, Newell JC, Isaacson D. Electrical
impedance tomography [J]. Signal Processing Magazine
IEEE, 2006, 18 (6):231-43.
5. Hoja J, Lentka G. Interface circuit for impedance sensors
using two specialized single-chip microsystems [J].
Sensors and Actuator A: physical, 2010, 163(1): 191-197.
6. Hoja J, Lentka G. Portable analyzer for impedance
spectroscopy [C]// XIX IMEKO World Congress on
Fundamental and Applied Metrology, Lisbon: 2009:497502.
7. Kohout J, George AD. A high-performance
communication service for parallel computing on
distributed DSP systems [J]. Parallel Computing, 2003, 29(7): 851-878.
8. Witte RA. Electronic Test Instruments: Theory andApplication [M]. Prentice Hall 1993, 186-203.

Finally, calculate the average phase absolute error of thesystem. The applied experiments on 4.7kΩ and 3nf series
R-C circuits, and use origin 8.5 to compare the two data,is shown in Fig. 6.
After calculation, the average absolute error of the
impedance phase of the system is 0.527°, which indicatesthat the system has good measurement performance forphase.
Acknowledgements
The authors would like to thank the EIT research groupfrom Tianjin University of Science and Technology. This
paper was supported by a grant from the National Natural
Science Foundation of China (Grant No.61301246) anda grant from the Science and Technology Project ofTianjin (12JCYBJC19300). It was also supported by the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan386

Flock Guiding of Hybrid Agents via Root Block
Yunzhong Song†, Ziyi Fu, Fuzhong Wang

College of Electrical Engineering and Automation, Henan Polytechnic University,
2001 Century Avenue, Jiaozuo 454003, P.R.China
E-mail: songhpu@126.com, fuzy@hpu.edu.cn, wangfzh@hpu.edu.cn
www.hpu.edu.cn

Abstract
The convergent equilibrium of flocking agents is vital importance for their collective behavior, where flock guidingof hybrid agents, which are composed by both first order and second order agents, can only be boiled down to theroot node agents, is touched upon here. Both the theoretical analysis and the simulation results are provided toverify the suggested scheme.
Keywords: Flock guiding, matrix block, hybrid order agents, collective behavior

order and second dynamics, under auspices ofaugmented matrix operation3, could be assumed unified.
In spite of collective behavior of hybrid agents has
been explored so deeply, there are still some rooms to
be filled in. Among those, the ultimate goal of collective
behavior will stand out, for its essential value inplanning of agent goal. Generally speaking, all themembers of the agents should and could contribute theirroles, however, counter intuitively, in this paper, in
directed networks of hybrid agents, only the root block
member agents can dominate the final equilibrium of
the flock. And this finding seems promising in today’s
real world, for the authorized agents, their number
should be made as less as possible in case of security.
To be distinguished from the already existed
results, our points here have at least three standing out
viewpoints. At first, block guiding which extends theconcept of leader guiding in flocking control has been

1. Introduction
The collective behavior of agents has been a long andlasting spotlight in systems and control field. The
underpinning knowledge was introduced by Murray1,
which including graph theory, adjacent matrix as well as
consensus concepts, made clear intuitively. Leader
follower agent systems, which have been a focus bymany experts, were still being improved up to now. The
networks of networks scheme of leader follower
systems was made possible by Song2, which
demonstrating interdependence of agents, turned into
reality via socialized status of the agents. For theengineering complexity, the agents should be different,
and only by this difference, the engineering systems canfunction well, and in this kind of sense, hybrid agents,
especially hybrid agents with heterogenous dynamics,
were explored in detail. Hybrid agents with both first†

Corresponding author© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

387

Yunzhong Song, Ziyi Fu, Fuzhong Wang

Proof: Since ξ is the left eigen vector corresponding toreported so less often. And secondly, that ultimate goal
position is invariant from the leaf block points will beawarded these days for security of the networks. And
the third, network of hybrid agents are borrowed toverify the suggested scheme, is much more promising,for the first order agents can be anchor points
sometimes and the second order agents can be themobile agents, which will be guided by the anchor
agents.
The structure of this paper is arranged as follows.
At first, problem formulation, together with the main
results as well as some comments on the main results
will be presented in the second part of this paper. Thenfollowed that will be some conclusions, which will be
the final part to end up this paper.

eigen value zero of matrix A, then we can get A T ξ = 0 ,and this can be translated into the following form

A1T
0

BT
DT

ξ1
=0
ξ2

(1)

To be concrete, we can have

A1T ξ1 + B T ξ 2 = 0(2)DT ξ 2 = 0

For D is invertible, ξ 2 must be zero vector. And in thissense, we can get A1T ξ1 = 0 , in case of symmetry of A1,
we have A1ξ1 = 0 . Referring that row summation of A1

2. Flock Guiding of Hybrid Agents, 1]T ,

is zero, so there must exist vector ξ1 =[1, 1,that makes

This part will start with the flock guiding of hybrid
agents, which will be covered by subsection 2.1, thetheorem will be demonstrated at first, and then somenotes about it. Then the main results will be followed at
2.2, both the topology of the agent networks and thesimulation results will be provided, and after that somecomments will be made ready to clarify the idea further
in 2.3.

A1 ξ 1 = 0(3)

Notes: Theorem1 points out that if sub-matrix A1 has no
input degrees and is symmetric, then its corresponding
nodes can be named as root nodes and the ultimate stateof the networks can be determined by those sub blocknodes; if sub matrix D is invertible, and it does not have
output degrees connected with the other sub blocks, thenodes corresponding to D can be named as leaf nodes,
leaf nodes can not affect the ultimate state of the
networks.

2.1. Problem formulation
For the directed networks, it is composed of root blocknodes and leaf block nodes. For the root block nodes, asa whole, they only have output degrees to connect withthe other blocks. Intuitively, the nodes of leaf block, asa whole they only have input degrees to connect withthe other blocks.

2.2. The main results
Consider the network, which consists of 5 agents,
among which node 1 and node 2 are root block nodes,
while node 3, node 4 and node 5, are leaf block nodes,
this can be referred to Fig.1.

Theorem 1 Suppose that the coupling matrix A can beturned into the following form:

1

A 0
A= 1
, and the row summation of A1 be zero, D
B D
is non-singular. And we also assumed the left eigen
vector corresponding to eigen value zero of A
be ξ = [ξ 1 ξ 2 ]T , then ξ 2 will be vector 0. And going
into further, if sub matrix A1 is symmetrical, then, we
can get ξ = [ξ 1 ξ 2 ]T =[1， 1， ， 1， 0， ， 0]T .

3

4

2

5
Fig.1 Network consists of 5 nodes© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

388

Flock Guiding of Hybrid

To be further, the nodes 1 and 2 are assigned as the firstorder agents, while nodes 3 to 5 are assigned as thesecond order agents, respectively.
The coupling matrix of Fig.1 is easily to be gotten as-1 1 0 0 0

A

1
1-1 0 0
0 -3 1

0
0

0
1

1
1

proportional factor a, and the second term is velocity
damping part, which is also scaled by proportional
factor , and the scaled value can be varied from a, and in
this case we named the factor as b. The consensus term
plus the damping term collaboratively turn the flock
guiding of hybrid agents possible. Among which, weassigned a=1 and b=2, respectively. The simulation
results can be seen as Fig.2.

0
1

-1 0
1 -3

The left eigen vector to the zero eigen value of A is [1/2
1/2 0 0 0]T. The dynamics of the system can be
described as

xi
xj
n

ui，i

1, 2,

x j1

x j2

xj2

uj

2, m

n 1, N,n ;,jm, m 1,,N ;(4)
Fig.2 Flock guiding of 5 hybrid agents

5.

During simulation, the initial values of the agents werelisted as follows
x 0 [ x10 x20 x310 x320 x410 x420 x510 x520 ]T

Which can be ensemble as compact as
0 0 0 0 0 0 0 0
1 0 0 0 0x

0 0 0 0 0 0 0 0

0 1 0 0 0

0 0 0 1 0 0 0 0

0 0 0 0 0

0 0 0 0 0 0 0 0
0 0 0 0 0 1 0 0

0 0 1 0 0x

0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 0 1 0

0 0 0 0 0 0 0 1

0 0 0 0 0

0 0 0 0 0 0 0 0Wherex [ x1 x2 x31 x32 x41 x42

0 0 0 0 1x51x 0 [3 1 -1 2 3 -2 -2 4]T
From Fig.2, the ultimate flock guiding position is 2, it is
the average of the initial values of agents 1 and 2. So, in
this respect, only the root block agents have their role
being played in the ultimate goal of flock guiding. It ismuch more interesting about the second sub variables of
agents 3, 4 and 5, all of them were rendered to zero
because of the damped factor of theirs. It seems
frustrated that, the velocity signals of the agents at leaf
block are not available sometimes. However, all of thesecond sub system states can be observed, it is astandard way to design observers for them. And in thispoint, this should not be an excuse to devalue ourcontribution here. And this is customary, because for the
fixed ultimate goal position, the velocity sub variablemust be ceased. Of course, the ultimate goal can be
adjusted if we like, we can regulate the ultimate goal
position by manipulation the coupling coefficients ofroot block nodes.

ux52 ]T ,u [u1 u2 u3 u4 u5 ]T .
To be concise, the control action can be expressed asu1
a( x2 x1 )

u

u2
u3
u4
u5

a( x1 x2 )a( x1 x41 x51 3x31 )-bx32
a ( x31 x41 )-bx42
a ( x2 x41 x31 3x51 )-bx52

The introduced control action for agents 1 and 2 areeasily to be understood, for agents 1 and 2, they makeup of the undirected sub network of root block, and theytend to be convergent to each other. Control action
added for agents 3, 4 and 5 can make a little nervous at
first glimpse, but they still can be made tractable in thisway, first we decomposed the added control function
into two different parts, and the first part is employed torealize the consensus among themselves and theirneighbours, and these consensus actions are scaled by

2.3. Some comments
Flock guiding of hybrid agents via root block has its
potential contributions at the following respects: First ofall, in the easy to get and easy to use style of thecontemporary network world, privacy protection
problems are becoming more and more pressing, inhelps of our idea here, the key information are strongly
suggested to be kept in root block agents. Secondly,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

389

Yunzhong Song, Ziyi Fu, Fuzhong Wang

infrastructure cyber physical system like smart grid
networks, transportation networks, can be kept immune
from the injected attacks if only the small number of the
key root block nodes are kept shield from the hackers.
And the third, root block agents and leaf block agents
have different dynamics, can conveniently be specified,paired connections of root block nodes and leaf blocknodes can be tagged into block chain, which thenstrengthened the safety further. Pros of our suggested
scheme are so obvious, and without doubt, we should
continue our pursuing to this amazing topic. And somecons should be overcome in the near future, such as theinjected invaders in the leaf nodes could not change theultimate goal position indeed, but they could make theprocess to the ultimate goal position so difficult, via
disturbing the topology of the leaf block agentsconnected graph. Problems like this one would be kept
as our focus.

system, but it should be kept in mind at first, for the firststep we should know the system locally in detail inorder to grasp the whole system globally, and in thisway, refined analysis and control of the complex
systems are necessary. Refined control of leader
follower system according their roles of the leader agent,follower agents that have connected path with theleaders, the follower agents that are failed with theconnection of the other agents, and the targeting agent
which represent the ultimate goal position arerecommended also7. So, flock guiding of hybrid agents
via root block, the suggested strategy here together withthe other champion achievements of complex systems
can see its prosperity in the near coming years.
Acknowledgements
This work is partially supported by NSFC Grant(61340041 and 61374079) and the Project-sponsored by
SRF for ROCS, SEM to Yunzhong Song as well as
NaturalScience
Fundof Henan
Province(182300410112). The root block and leaf block flock
guiding idea is benefited from the monograph, written
by Prof. Junan Lu, et al.8

3. Conclusion
In conclusion, in order to realize the security of theflock guiding, the ultimate goal of the flock, should be
exposed to only several key agents, and they can be
assigned as the root node agents, as to the leaf agents,
for their popularity is so massive, they should be kept
from the secrets of the ultimate goal. Flock with secrets
keeping is made possible here with root node and leaf
node decomposition.
System complexity, which is so elusive, could not
be easily explained if we are unwillingly exposed to theconcrete systems. The road to clarify the complexity canbe made tractable through the following exemplified
engineering systems as well as the theoretical
exploration. Instance like high performance vehicle
driving, which was spearheaded in Jia4 and thereafter
individualized designed in Jia5, can be the justified pilot
scale systems. As to the large quantities complex
systems, not only the object itself, but also the enduredhostile environment, will be an impediment for the
successful manipulation of them. Time varying delays,
uncertain and unreliable information, annoying
disturbances, and multi-inputs multi-outputs, can also be
the adversary factors. Adaptive scheme was born to fix
them, and this can be referred to Jia6. Reductionism
should not be the only way to deal with the complex

References
1. R. Olfati, R. Murray, Consensus problems in networks ofagents with switching topology and time delays, IEEE
Transactions on Automatic Control, 49(2004) 1520-1533.
2. Y. Song, Z. Fu, F. Wang, Socialized multi-agent systemrendezvous via networks of networks, Journal of Robotics,
Networking and Artificial Life, 3 (2016) 136-139.
3. Y. Song, Consensus of agents with mixed linear discrete
dynamics, International Journal of Control, Automation and
Systems, 14 (2016) 1139-1143.
4. Y. Jia, Robust control with decoupling performance forsteering and traction of 4WS vehicles under velocity-varying
motion, IEEE Transactions on Control Systems Technology,8(2000) 554-569.
5. Y. Jia, Alternative proofs for improved LMI representations
for the analysis and the design of continuous-time systems
with polytopic type uncertainty: a predictive approach, IEEE
Transactions on Automatic Control, 48(2003) 1413-1416.
6. Y. Jia, General solution to diagonal model matching control
of multi-output-delay systems and its applications in adaptive
scheme, Progress in Natural Science, 19(2009), 79-90.
7. Y. Song, W. Zhao, Multi-agent system rendezvous via
refined social system and individual roles, WSEAS
Transactions on Systems and Control, 9(2014),526-532.
8. J. Lu, H. Liu and J. Chen, Synchronization in Complex
Dynamical Networks (Higher Education Press, Beijing, 2016).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan390

Multiple Model Adaptive Control of Flexible Arm
Yingzhao Zhang*College of electrical engineering, Guizhou University
Guiyang, 550025, China
Weicun Zhang

School of Automation and Electrical Engineering, University of Science and Technology Beijing
Beijing, 100083, China
Xiao Wang, Handong Li

College of electrical engineering, Guizhou University
Guiyang, 550025, China
E-mail: 15600233121@163.com, weicunzhang@ustb.edu.cn
http://www.gzu.edu.cn/

Abstract
In this paper, dynamic model of a flexible arm is built through Lagrange theory. Then, based on the platform of
SIMULINK, model of the flexible arm is established and the states of the flexible manipulator under multiple
model adaptive control are simulated. Finally, to compare the control effect (like stability and accuracy) of multiple
model adaptive control method with other control methods, some simulation results are given.
Keywords: Flexible arm, MMAC(Multiple Model Adaptive Control), nonlinear system, system identification

infancy, so it is very important to realize the precise
control of flexible manipulators 1.
In the classical control field, the change of internal
characteristics and the influence of external disturbances
can be restrained by feedback control. In the design ofcontrollers, the parameters are always fixed, so if sensor
faults, characteristics changes, sudden disturbances andother conditions occur during the operation of thesystem, the overall performance of the system will
decline, or even be unstable. Because of thecharacteristics of flexible manipulator's material
structure and external disturbance, there are someproblems in its mathematical model, such as parameter
jump. In order to produce better control effect for the

1. Introduction
Most of the traditional industrial robots are made ofrigid materials, with huge mass and heavy weight. With
the continuous progress of science and technology, there
are higher requirements for robots. Many industries also
require intelligent robots with light weight, highprecision, high efficiency and high adaptability.
Therefore, the concept of flexible robot was put forward,
which is a kind of flexible manipulator with light weight,
less energy consumption and small inertia. Flexible
manipulator has gradually become an importantresearch object in control field in recent years. At
present, the research on flexible robots is still in its*

Corresponding author© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

391

Yingzhao Zhang, Weicun Zhang, Xiao Wang, Handong LiTcontrolled object when the parameters jump, this paper
adopts multi-model adaptive control method to control
the flexible manipulator and studies it with othercontrollers. The characteristics of multi model adaptive
control are summarized.

M r (q) is generalized inertia matrix of rotor. J is rotor
inertia matrix of motor rotor, s (q) is inertia matrix of

2. The Main Text

coupling part. The potential energy equation is asfollows:

2.1. Modeling

V p (q, )

T

H (q)

1
(
2

M

r

(q )

S (q )TJ

S (q )q )T K (q(2)

q ) Vg ( q )q

T (q , q )q

g (q ) K (
K(
q)

q)

ext
m

(4)

H (q ) is the inertia matrix of the model, g (q ) is thegravitational function.
It is important to note that we set up the flexible
manipulator under the following assumptions:Hypothesis 1: Consider the rotor of the motor as awhole with the axis as the center, avoiding the rotation
process of the rotor. It has a bad effect on motor rotation.
Hypothesis 2: Consider only the mechanical dynamics
of flexible manipulator system; assume that the motor
dynamics is fast enough to be ignored. Adverse effects
on the system is treated as a whole.
Hypothesis 3: A virtual spring is used to simulate joint
flexibility, spring elasticity and sum in the derivation ofmathematical model. The flexible deformation of thejoint achieves a linear relationship, and thecorresponding proportional coefficient is the elastic
coefficient of the joint. In addition, the spring has
damping force, and the deformation velocity is linearly
proportional to the damping force.
Finally, by simplification, we can get the following
kinetic equations:Fig. 1. Simplified model of flexible joint
Aiming at the simplified model mentioned above, thispaper uses Lagrange-Euler method to solve the dynamic
equation of flexible manipulator. The first thing to do isto write the expression of potential energy and kinetic
energy of the manipulator system. When formulating
the potential energy expression of flexible manipulator,
not only the gravitational potential energy but also theelastic potential energy produced by joint flexibility
should be considered.

1 T
q M 1 (q) q
2( ,q,q)q(3)
Substituting Eq. (2) and Eq. (3) into Lagrange's kinetic
equation we can get the following formula:

Generalized flexible manipulators are mainly divided
into two kinds: one is that the link of the manipulator ismade of elastic material, and the link itself has
flexibility, i.e. distributed flexible manipulator; the otheris that the link of the manipulator is rigid, and theflexibility mainly exists in the joint, which is called
flexible joint manipulator 2. The object of this paper isthe second flexible joint. The reason for the flexibilityof robots is that harmonic gear reducers and moment
sensors are widely used in the transmission system ofrobots. These components are flexible, so the joint ofrobots is flexible. We can imagine a flexible joint as aspring so that a simplified model of the flexible joint
manipulator can be obtained.

T1 (q, q )r

1
2

M ( q ) q C ( q, q ) q g ( q )
J

K(

q)

m

K(

q)

ext

(5)

2.2. Control System(1)

2.2.1. Control structure

Eq. (1) is the kinetic energy equation, q is the
position vector of the connecting rod. M 1 (q ) is
inertia matrix of connecting rod.
Converting Eq. (1) into vector situation

M i i 1,2, , n represents a model set with
M i as its element, which can be understood as a

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

392

Multiple Model Adaptive Control

generalized model set in which M i can represent not
only the transfer function model, but also state feedback
matrices.
C U i i 1,2, , n C is a controller set designed,
according to, Ci is a controller corresponding toeach time and adjusts the subset dynamically at any
time so that it can cover the real model parameters of
the controlled object.
2.2.3. Control strategy

M i . The overall controller can be expressed as
U sys f (U1 , U 2 , , U N , )There are two different control methods, one isweighted MMAC, one is switched MMAC. In this paper,
we use switched MMAC. The principle of switched
MMAC is to find the model closest to the controlled
object based on the performance index function at eachsampling time and switch the controller based on thismodel to the current controller. This multi-model
controller can ensure the stability of the controlled
system when switching for finite or infinite times 4.

Fig. 2. Multi model adaptive control structure diagram
is a
In this paper, f represents a nonlinear function.
parameter vector. Switching mechanism represents theswitching function. According to the error between the
output of the models and the outputs of actual plant, theswitching mechanism is used to determine the current
effective model and the corresponding controller 3.

2.2.4. Switching function
The core of the switching mechanism is theperformance index function. The performance index isused to judge the approximation degree between eachmodel and the actual state of the controlled object at thepresent stage, and the most suitable model is chosen as
the approximation model at the present stage. The
establishment of the performance index function is notunique, the performance index function in this article is2.2.2. Model set
MMAC is based on multiple models to approximate theuncertainties of the system and establish controllers on
the basis of multiple models. Therefore, the model set
and the number of element models will directly affectthe accuracy and performance of the control. According
to the nature of the model, the model set can be divided
into the following types: 1. all fixed models; 2. multifixed models and one adaptive model; 3. all adaptivemodels.
In practical problems, because the external
environment of the controlled system and the changes ofsystem model parameters are not described by a few
models, such problems arise. Without a large number ofmodel systems, it is impossible to describe themaccurately. If there are too many models, a large part ofthe model and the current system will be centralized ineach step of the control process. The real models are far
from each other, which not only waste the computationof the controller, but also reduce the performance of thecontroller by excessive competition of the redundant
models. In order to avoid the above problems, this paper
uses Moving Back method, which constructs thecontroller based on only one subset of the model set at

J i (t )ande 2j (t )t0e

are constants.

(t

) 2
j

e ( )d(6)

is the forgetting factor, ande j is the output error between the j model and theactual system. The controller which has the minimum
value of switching function will be selected as aneffective controller.
3.

Simulation Results and Conclusion

Fig. 3. Simulation diagram© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan393

Yingzhao Zhang, Weicun Zhang, Xiao Wang, Handong Li

Fig. 4. Control effect using three models.

Fig. 6. Control effect using twenty models.

Fig. 5. Control effect using ten models.
Fig. 3 is the basic structure of the simulation system.
The control effect of the multi-model control system isstudied below. In the following experiments, the onlychange is the number of centralized models and thecorresponding controllers. Parameter jumps occurred at
3 second of the simulation time.
From the above pictures, we can see that with theincrease of the number of models in the control system,
the control performance is getting better. In this control
system, when the number of models reaches 40, thejump of parameters has little effect on the control
performance.

Fig. 7. Control effect using thirty models.
References
1. H. Huang, Research on Modeling and control of flexible
joint manipulator (Hunan University of Technology,
Hunan, 2014).
2. Y. H. Wang, H. S. Li, W. Zhang, Sliding mode variable
structure control of flexible joint manipulator, J.
MMTAMT. 10(1) (2017) 91–95.
3. X. L. Li, W. Wang, W. Sun, Multi model adaptive
control, J. CONTROL AND DECISION. 15(4) (2000)
390–394.
4. W. R. Li, Multiple Model Adaptive Control andApplication for a Class of Nonlinear System (China
Jiliang University, Hangzhou, 2015).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

394

Time-Varying Lyapunov Function for Mechanical Systems
Bin Zhang*

School of Automation, Beijing University of Posts and Telecommunications, Beijing, 1008760, China
Yingmin Jia

The Seventh Research Division and the Center for Information and Control, School of Automation Science andElectrical Engineering, Beihang University (BUAA), 37 Xueyuan Road, Haidian District, Beijing, 100191, China
E-mail: zb362301@126.com, ymjia@buaa.edu.cn
www.bupt.edu.cn, www.buaa.edu.cn

Abstract
In this paper, a general method for constructing time-varying Lyapunov functions is provided for mechanical systems.
Sufficient conditions of uniform asymptotical stability are established. Different from the existing results on thissubject, we remove the periodic restrictions and persistency of excitation restrictions in our work. It is shown that the
time-varying mechanical systems remain uniformly asymptotically stable with the control input provided in our work.
Keywords: Mechanical systems, time-varying systems, strict Lyapunov function, subsystems.

kinds of weak-invariance principles for nonlinear
switched systems were developed and accurate
convergent regions were obtained. The readers can refer
to Refs. 7 and 8 for more detailed discussions on this
topic and a list of related references.
In recent decades, it is recognized that non-strict
Lyapunov functions suffer from some essential
disadvantages9. On the one hand, non-strict Lyapunov
functions are not well suited to robustness analysis, since
their negative semi-definite derivatives along trajectories
could become positive under arbitrarily small
perturbations of the dynamics. On the other hand, nonstrict Lyapunov functions are usually insufficient toanalyze nonlinear time-varying systems, since theirderivatives depend on the time parameters.

1. Introduction
Controller design for time-varying mechanical systems
has emerged as a hot research area. A lot of theoreticaland practical results for controller design have been
proved based on Lyapunov methods. Several approaches
for construction of strict Lyapunov functions have been
proposed under various frameworks and control
objectives1,2.
In Ref.3, a new method for estimation the domain of
attraction was shown by using non-strict Lyapunov
functions. In Ref.4, an extension of invariance principlewith non-strict Lyapunov functions was provided for
power systems with transmission losses. In Ref.5, strict
Lyapunov functions were established for finite-time
control problem of robot manipulators. In Ref.6, two*Corresponding author.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan395

Bin Zhang, Yingmin Jia

Motivated by the above considerations, we provide ageneral method for constructing time-varying Lyapunov
functions for mechanical systems in this paper. We
divide the closed loop system into two subsystems. Timevarying Lyapunov function is established for one of thesubsystems. Based on the derivatives of the time-varying
Lyapunov function along different subsystems, we
provide the time-varying Lyapunov function for theclosed loop system. It is shown uniform asymptoticalstability can be obtained with the control input provided
in this paper.

n

ai bi

(

i 1

n

1
p

p
i

a ) (

i 1

Lemma 3. 10 For xi| x1 |m| xn |m (| x1 |Lemma 4. For xi

R, i 1, 2,..., n, and 0

(1)
and q

q(t ), q(t ), q(t ) R n represent the position,

R n n isis a positiveof M and

i 1

t

0

1
p

where

ab
Lemma 2.
and q

Let ai

0, bi

1
p

1
1 , thenq

1 . If(7)| xn |) mq(8)

and k 0 are control parameters. Let q

x1 andx2 . Then the closed-loop system (1) and (8) can be

written in a compact form as

x1

x2

x2

(k (t ) k0 ) M 1 x1
1

bq
.
q

)

Mq Cq k0 q

(2)

1
1 , thenq

10

1

For the uniform asymptotic stability problem of timevarying mechanical system (1), we propose the following
control input

M.

Let a, b, p, q be positive real numbers. If

app1
1 m 1 m3. Subsystems

Before presenting the main results, we give somelemmas first.
Lemma 1.

nwhich completes the proof.

where k̂ is a positive constant.

10

1
,
m

i 1

n1 m (| x1 |Property 2. The derivative of k (t ) is bounded such that| k (t ) | kˆ

1, i 1, 2,..., n, p(| xi | ) ) ((max as the maximum eigenvalue

min as the minimum eigenvalue of

(6)

1
m m m

n

( ) is a positive definite function related to theStribeck effect; k (t ) denotes a time-varying spring
stiffness. For system (1), it is assumed that the following
properties hold.
Property 1. The matrix M is positive definite.

| xn |) m .

1. Then based on Lemma 2, we obtain1 m| x1 |m| xn |mconstant;Moreover, we definem 1,then the following inequality holds:

ai | xi |m , bi

the positive definite inertia of the system;| xn |)m .

(5)

Proof. Let

velocity, and acceleration, respectively; Mm 1,then the following inequality holds:

Consider the following mechanical system with timevarying friction
where

.

R, i 1, 2,..., n, and 0| xn |m n1 m (| x1 |) sat (q) k (t ) q

b )

(4)

| x1 |m
(q)

1
q q
i

i 1

2. Problem Statement

Mq Cq ( Fc (t ) Fs (t )en

M ( Fc (t ) Fs (t )ex2( x2 )(9)) sat ( x2 ) .

We define(3)

f (t , x)0, i 1, 2,..., n, p 1 ,where function f (t , x )x2
f (t , x)( k (t ) k0 ) M 1 x1

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

396

(10)

x2

Time-Varying Lyapunov Function( x2 )

M 1 ( Fc (t ) Fs (t )e) sat ( x2 ) .We also definex2

f1 (t , x)

and

x2

0
1

M ( Fc (t ) Fs (t )eThen, we can see that f (t , x )thissubsection,x

f1 (t , x) and x

we( x2 )) sat ( x2 )f1 (t , x)consider.

xsubsystems

Similarly, the derivative of V (t , x) along subsystemtwo

f 2 (t , x) , respectively.

xf 2 (t , x) shows:Vf 2 (t , x)x
2 x2T M 1 ( Fc (t ) Fs (t )eA((k (t ) k0 ) x1T x1 x2T Mx2 ) x1T x2
0 is a positive constant. In this paper, we
1
1
max{
,
}.
ko
min k1

1

2 n

Mx2 )

x1T ((k (t ) k0 ) M 1 x1

x2 )((k1 k0 )

1

(12)

n

Fc (t ) Fs (t ) x1

1

Fc (t ) Fs (t ) ( x1

x2 )

min

2 n

1

1

Fc (t ) Fs (t ) V 2 .

min

f1 (t , x)

In the following, we define time-varying parameter

w(t )

n

1

Fc (t ) Fs (t ) .

min

x2T x24. Main Results
From (13) and (14), we can see that V (t , x) is not a
Lyapunov function for system (1), because it is not easy

(13)

to verify the negativity of the derivative of V (t , x)along system xAkˆ) x1T x1

max(2 AM 1) x2T x2

(14)

min

shows:

2 Ax2T ((k (t ) k0 ) x1

) sat ( x2 )Fc (t ) Fs (t ) x2
1n

A((k (t ) k0 ) x1T x1 x2T Mx2 )VVV (t , x)
f1 (t , x)tx
ˆ T x 2 A( k (t ) k ) xT x
Akx
1 1
0
1 2

( x2 )) sat ( x2 )min

A((k (t ) k0 ) x1T x1 x2T Mx2 ) x1T x2

1 T( x1 x1 x2T x2 )2
1 T
( x1 x1 x2T x2 ).
2
The derivative of V (t , x) along subsystem x( x2 )

x1T M 1 ( Fc (t ) Fs (t )e

It can be verified directly that

V (t , x)Akˆ andf 2 (t , x) . In(11) where A
choose A

12

3
1 . Therefore, we can see that if we choose221
and k0 (Akˆ) max k1 , subsystem
32
f1 (t , x) is uniformly asymptotically stable.

l2

Inspired by the results in Ref.7, we consider thefollowing Lyapunov function

V (t , x)1(k1 k0 )max(k (t ) k0 ) M 1 x1

f 2 (t , x)

where we have l1

V (t , x) f (t , x )| x1T x2 |f (t , x) , which is shown as follows
VV( f1 (t , x) f 2 (t , x)). (15)tx

Therefore, motivated by the results in Ref.7, we provide
the following time-varying Lyapunov function

l1 x1T x1 l2 x2T x2© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

397

Bin Zhang, Yingmin Jia

V # (t , x)1ds
0 (1 s ) 0.5
V

1 Tw(v)dv w( ))d ln(1 V ) (16)t0 T
where T , are positive constants. It can be verifiedtasymptotically. The simulation clearly illustrate theexactness of our results.

(

directly that if k0 andthere exist T and

V # (t , x) f (t , x )6. Conclusions

are chosen large enough, thenIn this paper, the stability problems of nonlinear timevarying mechanical systems have been studied by using
strict Lyapunov functions. Control input and sufficientconditions for uniform asymptotical stability have been
provided. A numerical example has been presented forillustration.

such that

V
t
0.

V
( f1 (t , x)x

f 2 (t , x))(17)

Therefore, we conclude with the following theorem.
Theorem 1. Consider time-varying system (1). If thecontrol input is chosen as (8), then the closed loop system
is uniformly asymptotically stable. Moreover, (16) is astrict Lyapunov function.
Proof. This result can be verified by taking the derivative

Acknowledgements
This work is supported by National Natural Science
Foundation of China (Grant No. 61603050).
Referencesof V # (t , x ) given in (16) directly. The detailed process

1. H. K. Khalil, Nonlinear Systems (Upper Saddle River, NJ:
Prentice-Hall, 2002).
2. A. Bacciotti and L. Rosier, Liapunov functions andstability in control theory (New York: Springer Verlag,2005).
3. D. Han, A. Guindy and M. Althoff, Estimating the domain
of attraction based on the invariance principle, inProceedings of the 2016 IEEE Conference on Decision
and Control, (Las Vegas, USA, 2016), pp.5569-5576.
4. N. G. Bretas and L. F. Alberto, Lyapunov function for
power systems with transfer conductance: Extension of theinvariance principle, IEEE Transactions on Power
Systems, 18(2) (2003) 769-777.
5. Y. G. Hong, Y. S. Xu and J. Huang, Finite-time control for
robot manipulators, Systems and Control Letters, 46(4)(2002) 243-253.
6. B. Zhang and Y. M. Jia, On weak-invariance principle fornonlinear switched systems, IEEE Transactions on
Automatic Control, 59(6) (2014) 1600-1605.
7. M. Malisoff and F. Mazenc, Constrictions of Strict
Lyapunov Functions (New York: Springer Verlag, 2009).
8. F. Mazenc, Strict Lyapunov functions for time-varying
systems, Automatica, 41(11) (2005) 1973-1978.
9. P. Giesl and S. Hafstein, Review on computational
methods for Lyapunov functions, Discrete andContinuous Dynamical Systems-Series B, 20(8) (2015)2291-2331.
10. B. Zhang, Y. M. Jia, J. P. Du and J. Zhang, Finite-timesynchronous control for multiple manipulators with sensor
saturations and a constant reference, IEEE Transactions
on Control Systems Technology, 22(3) (2014) 1159-1165.

is omitted here.
5. A Numerical Example

Fig. 1. The state trajectories of system.

We give a numerical example to demonstrate theeffectiveness of the proposed theoretical results under
different initial values. Two different initial values arechosen as x(0)(0.5, 0.3) and x(0) ( 0.1, 0.05) ,respectively. By using the control input (8), we can seethat the state trajectories converge to the origin© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

398

Optimizing Control Parameters of Space Robot Manipulator
for Pulsar X-ray Interference Measurement
Qiang Chen, Hengbin Zhang, Xiaomin Bei

Qian Xuesen’s Lab of Space Technology, Chinese Academy of Space Technology, Beijing, 100190, China
Hanwen Zhang*

Beijing Institute of Control Engineering
104# Youyi Road 104, Haidian district, Beijing, 100190, China
Email:cq_01@sina.com, zhanghengbin@qxslab.cn,beixiaomin@qxslab.cn,hanwen96888@163.com
www.qxslab.cn

Abstract
In paper, a pulsar X-ray observation plan of using two detectors installed in two sides of space station is given.
Dynamic model of the space robot manipulators and an adapt control method are described. An orthogonal arrayexperimental design method is developed to optimize the control parameters. Simulations are carried out by L81 (340)orthogonal array while fourteen adaptive control parameters are selected. The optimization results show that theparameters expected value can be acquired and computationally efficiency is improved.
Keywords: Control parameters optimizing, coordinated control, attitude and position control, space robot
manipulator, pulsar x-ray measurement.

detectors installed in two sides of space station is given.
Dynamic model of the space robot manipulators and an
adapt control method is described in detail. Twomanipulators are used to set the relative attitude andposition of two detectors respectively and coordinately
while observing jointly. The attitude of two detectors arecontrolled for pointing to the same pulsar dynamically.
And the relative position of two detectors, as the intensity
interference baseline ranged from 10m to 100m, should
be controlled precisely. In fact, attitude and position
control system of the space robot manipulators is
nonlinear and uncertain. Some control parameters are noteasy to obtain or design precisely, especially the couple
parameters, so that it is necessary to acquire thecoordinated control system parameters of two
manipulators by mathematic test. There was a lot of
researchers who studies about the control method for

1. Introduction
Pulsars as sources of X-ray are observed with detectdevices in spacecraft1,2,3. The time of arrival (TOA)
measurement of their periodic pulse could be applied todeep space navigation in future. In 2017 NASA has
carried out a test named the Neutron star InteriorComposition ExploreR Mission (NICER) with a payload
in international space station (ISS) using space robot
manipulator to make the detectors point to the pulsars.
From the pulsar navigation formula it can be proved thatthe navigation precision is influenced by the angle
position of target pulsars. The measurement precision ofpulsar angle position is important and could be greatly
improved with intensity interference method by two
devices observing a pulsar jointly at the same time4. In
this paper, a pulsar X-ray observation plan of using two*

Corresponding author© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan399

Qiang Chen, Hengbin Zhang, Xiaomin Bei, Hanwen Zhang

space robot5,6,7,8, especially Chen9 has given a control
method for space robot with two manipulators andWang10 has proposed an adaptive Jacobian control
method for free floating space robot. In this paper anorthogonal array experimental (OAE) design method isdeveloped to optimize the parameters. Simulations arecarried out by L81 (340) orthogonal array while fourteen
adaptive control parameters are selected randomly.

M ( q ) q C ( q, q ) q

The torque vector τ is used to represent the actuator
input. τ R 3 6
Since the space station is free floating with microgravity,
the above formula can be separated into two parts.
(3)
M bb qb + M bm qm + Cbb qb + Cbm qm = 0
T
M bm
qb

M mm qm Cmb qb Cmm qm

The pulsar X-ray interference measurement system
consists of space station, two- joint manipulator B1, twojoint manipulator B2, X-ray detector D1 and X-ray
detector D2. Each part is connected by a rotating joint.
The system centroid is C, and the centroid of every
component is Ci. An inertial coordinate system isestablished (O-xyz) for the system. The mass of two partof every manipulator are mb11, mb12 and mb21, mb22. The
moment of inertia are Jb11, Jb12, Jb21, Jb22. The mass of two
detectors are md1 and md2, the moment of inertia are Jd1
and Jd2. The vector of the component centroid Ci relative
to the origin O of the inertial coordinate system is Pi. The
vector of the total centroid C relative to the origin O is Pc,
as shown in Fig.1.

C (q , q )Space StationyFig.1. The configuration of pulsar x-ray measurement system

In the formula,

Vector qb is used to describe the attitude of the spacestation, and qm is used to describe the angle of the hingeof two manipulators and the attitude of the detector.

Assumed thatMˆ bb qbr Mˆ bm qmr

T
s3

11

12

13

21

22

Cmb

Cmm

R9 9

R3 6

The Design of Adaptive Controller

3. Dynamics Model of Two Manipulators Space
Robot

s2

Cbm

Vector qmd R 6 is used to describe the desired rotation
angle of the manipulators. qmd is the velocity and qmd isthe acceleration. qmd , qmd and qmd are bounded in value.
qbr R 3 is the reference angular velocity of spacestation, and qbr (0) qb (0)(5)
qm qm qmd , qmr qmd
0 qm

z

s1

(4)

M mm ──Inertia matrix of two manipulators, M mm R 6 6
Cbb ──Centrifugal force and Coriolis force matrix ofspace station, Cbb R 3 3
Cbm ──Centrifugal force and Coriolis force matrix fromspace station to two manipulators, Cbm R 3 6
Cmb ──Centrifugal force and Coriolis force matrix fromtwo manipulators to space station, Cmb R 6 3
Cmm ──Centrifugal force and Coriolis force matrix oftwo manipulators, Cmm R 6 6
τ m ──Drive torque of the joint of two manipulators,
τm R6

4.

T

Cbb

M bb ──Inertia matrix of space station, M bm

x

qb qm

τm

M (q ) ──The total Inertia matrix
C (q, q ) ──Centrifugal force and Coriolis force matrix
M bb M bm
M (q )
R9 9
T
M bm
M mm

2. The Pulsar X-ray Interference Measurement
System

q

(2)

τ

0

is a positive constant. Thenqmr

In the formula Kb

(1)

qmd

R

(6)

qm

Cˆ bm qmr

Cˆ bb qbr
3 30

K b (qbqbr ) (7)is a symmetric semi-positive

matrix.

23

The system dynamic equation can be written as
Lagrangian form as follows.

Definite that qr

qbrT

T
qmr

T

.

Definite a matrix Yb (q, q, qr , qr ) , Ym (q, q, qr , qr ) as that

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

400

Mathematic Simulations of SystemMˆ bb qbrMˆ bm qmr

Cˆbb qbrCˆbm qmr

Yb (q, q, qr , qr )ad
T
M bm
qbr

M mm qmr

Cmb qbr

The desired rotation angles of the manipulators are(8)

Cmm qmrqmd 1 (t )qmd 2 (t )(9)

Ym (q, q, qr , qr )adSetad is the unknown dynamic parameter vector of free
floating manipulator.
So that the controller is designed as(10)τm
K m sm Ym (q, q, qr , qr ) ad
The kinetic parameter update law is selected as
T
(11)
aˆ d
d Y (q , q , q r , q r ) s
In the above formula,the(1 cos( t ))3
5
(1 cos( t ))18
Minimum

ITAE

performance index.
The controller parameters isT0

t e(t ) dt

0

astarget10I , K m

20 I ,

K b 20 I
The simulation results show that the two manipulator canbe controlled as expected curves and the parameters
expected value can be acquired, as Fig.2, Fig.3, and Fig.4.

Y (q, q, qr , qr ) [YbT (q, q, qr , qr ) YmT (q, q, qr , qr )]T
is a symmetric positive matrix of appropriateddimension.

s

sbT

smT

T

(qb qbr )T(qm qmr )TT

It can be proved that the tracking errors of free-floatingmanipulators converges to zero asymptotically by thecontroller (10) and the parameter update law (11).
， qm
When t0 ， qm0
5.

Fig.2. ad parameter estimating

System Mathematic Simulations and The
Controller Parameters Optimizing

The physical parameters of the space free floating
mechanical arm are shown in the following table.
Tab.1 Physical parameters of free-floating space
manipulator

Space Station
B11
B12
B21
B22
D1
D2

mi (kg )I i (kg m 2 )li (m)ri (m)

59000
6.3
5.1
6.3
5.1
7.6
7.6

2220000
1.11909
0.88128
1.11909
0.881280.22800
0.22800

70
10
5
10
5
0.30
0.30

15
0.72
0.71
0.72
0.71
0.60
0.60

Fig.3. Tracking errors of the manipulator

According to the data in the above table, the real value ofthe dynamic parameters of the space manipulator can becalculated.
ad [13.1746 19.3587 9.4486 1.3261

0.6473 77.4449 47.1541 22.3990]TAssumed that initial value of ad is

ad 0[1 1 1 1 1 160 82 40]T

Fig.4. Control moments of the manipulator© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan401

Qiang Chen, Hengbin Zhang, Xiaomin Bei, Hanwen Zhang

Tab.2 Optimized results with orthogonal array design
Parameters

Manipulator 1
BeforeAfter
optimizing
optimizing

5～15

01

14.9219～15

orthogonal array experimental design method. The
method is useful and effectively to apply to offline
optimizing control parameters of complex systems.

Manipulator 2
BeforeAfter
optimizing optimizing

5～15

5～5.0781

Acknowledgements

14.9219～15

This work was supported by the National Key Researchand Development Program of China (Grant No.
2017YFB0503300, 2017YFB0503304).

02

5～15

9.9219～10

5～15

K m1

15～25

24.9219～25

15～25 24.9219～25

Km2
K b1
Kb2

15～25

24.9219～25

15～25 24.9219～25

15～25

24.9219～25

15～25 24.9219～25

15～25

19.9219～20

15～25 24.9219～25

References

d1

15～25

15～15.0781

15～25 19.9219～20

d2

15～25

15～15.0781

d6

15～25 19.9219～20
18.3594～15～25 15～15.0781 15～25
18.4375
15～25 19.9219～20 15～25 19.9219～20
23.0469～
15～25 17.4219～17.5 15～25
23.1250
15～25 24.9219～25 15～25 24.9219～25

d7

15～25

24.9219～25

15～25 24.9219～25

d8

15～25 24.9219～25

1. A.G. Lyne, Francis Graham-Smith. Pulsar Astronomy.
Cambridge University Press.2006.
2. Sheikh. The use of variable celestial X-ray sources forspacecraft navigation. University of Maryland.2005.
3. P. Shuai, M. Li, S.L. Chen, Z. Huang. Principle andmethod of X-ray pulsar navigation system. China
Aerospace Publishing Press, Beijing, 2009.
4. Q. Chen, S.L. Chen, X.M. Bei, H.W. Zhang, H.B. Zhang.
Satellites formation flying for pulsar X-ray interference
measurement. 4th IAA conference on Dynamics andControl of Space Systems (DYCOSS2018), Changsha，Hunan，China，2018．
5. W. Michael, M. Walker. Adaptive Control of Space-based
Robot Manipulators. IEEE Transactions on Robotics andAutomation. 1992, 7(6):828-835
6. B.L. Ma, W. Huo. Adaptive control of space robot system.
Control theory and the application，1996，13 (2) ：19l197．
7. Y.M. Jia. Robust Control with Decoupling Performancefor Steering and Traction of 4WS Vehicles under
Velocity-Varying Motion. IEEE Transactions on Control
Systems Technology. 2000(3): 554-569.
8. Y.M. Jia. Alternative Proofs for Improved LMI
Representations for the Analysis and the Design ofContinuous-Time Systems with Polytopic Type
Uncertainty: A Predictive Approach. IEEE Transactions
on Automatic Control. 2003(8): 1413-1416.
9. L. CHEN ， X.T. TANG. Robust Variable Structure
Control for Free-floating Space Robot System with DualArms in Joints Space. Proceedings of the 26th ChineseControl Conference ， Zhangjiajie ， Hunan ， China ，
2007．
10. H.L. Wang. Adaptive Jacobian control for robot. Beijing
Institute of Control Engineering. Beijing, 2009.

d3
d4
d5

15～25

19.9219～20

ITAE
MAX

3.612994

0.809579

3.481480

0.909619

ITAE
MIN

0.994762

0.797814

1.076619

0.897425

Tab.3 Comparison between optimizing results withorthogonal array experimental method and optimizing on
the full scale

6.

Optimizing withorthogonal arrayexperimental method

Optimizing on thefull sacle

Times of ITAE
function

81

314=4782969

loop

7

7

All times

567

33480783

Conclusion

From Tab.2 to Tab.3, the optimization results show thatthe expected value of the parameters are acquired and thetest times decreased rapidly in approaching theappropriate value of every control parameter with the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

402

Pulse Pose Follow Control and Simulation for a 4-DOF Pulse Diagnosis Robot
Qunpo Liu*, Guanghui Liu, Hongqi Wang, Xianzhe Liu

School of Electrical Engineering and Automation, Henan Polytechnic University, 2001 Century Avenue
Jiaozuo, (454003), Henan, P.R.China,
Hanajima NaohikoCollege of Information and Systems, Muroran Institute of Technology, 27-1 Mizumoto-cho,Hokkaido
Muroran-shi, Hokkaido (050-8585), Japan
E-mail: lqpny@hpu.edu.cn, 211807020075@home.hpu.edu.cn, wanghq2013@yeah.net,
311508070330@home.hpu.edu.cn, hana@mondo.mech.muroran-it.ac.jp
www.hpu.edu.cn, www.muroran-it.ac.jp

Abstract
Pulse diagnosis has been proved to have great practical value in the past dynasties. The key of high-quality pulse
diagnosis is how to dynamically adjust the vertical relationship between the pulse and the sensor. The design adopts
a four degree of freedom manipulator combined with a matrix sensor to form a diagnostic robot, so as to achieve
rapid adjustment and keep the diagnostic pulse sensor perpendicular to the pulse. Kinematics analysis and simulations
are performed and the results show feasibility.
Keywords: Pulse diagnosis, robot, follow control, simulation

1. Introduction

2. Composition of the Pulse Diagnosis Robot

For more than two thousand years, pulse diagnosis has
played a very important role in the practice of diagnosis
and treatment1. The theory of pulse diagnosis has been
through the physiology, pathology, diagnosis, treatment
and other aspects of Chinese medicine. The quality ofpulse diagnosis is related to the effect of "syndrome
differentiation and treatment" in all clinical departments,
which is of great significance to the guidance of TCM
theory in medical practice. Since the 1960s, with thedevelopment of modern science and technology, many
domestic and foreign researchers have been committed topromoting the modernization of TCM pulse diagnosis
and expected to improve the situation of "difficult to
identify" by using modern medical instruments2.
*

Fig.1 The robot is made up of a four DOF manipulator and apulse diagnosis sensor matrix.

Corresponding author© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan403

Q. Liu et al

The key of high-quality pulse diagnosis is how todynamically adjust the vertical relationship between thepulse and the sensor (only the vertical relation between
sensor and pulse can guarantee accuracy of pulse
diagnosis). The robot is made up of a four degree of
freedom (DOF) manipulator and a pulse diagnosis sensor
matrix, as shown in Fig.1 and Fig.2.

rod offset di: along the Zi-1 axis, the distance between theXi-1 axis and the Xi axis is moved. Joint angle θi: around
the Zi-1 axis, rotate the Xi-1 axis to the angle of the Xi axis.
3.1. Forward kinematics of the manipulator
Tab.1 Parameters of D-H

i
1
2
3
4

αi
-π/2
0
0
0

ɑi
0
ɑ2
ɑ3
0

di
0
0
0
0

θi
θ1
θ2
θ3
θ4

According to the standard D-H parameter method, thegeneral equation of the link homogeneous transformation
matrix is:

Fig.2 The pulse diagnosis matrix sensor is composed of 5pressure sensors in a row. The matrix sensors are fitted on theend of the manipulators.

i 1
i

T

3. Kinematics of the Four DOF Manipulator
According to the standard D-H parameter table method,
the coordinate system of the manipulators as shown in
Fig.3. The coordinate system 1 is coincide with the originof the base coordinate system 0, and the origin of thecoordinate system of the pulse sensor coincides with theorigin of the wrist coordinate system. The D-H
parameters are obtained as shown in Tab.1.
The length of the connecting rod is ɑi: the distance fromthe Zi-1 axis to the Zi axis is moved along the Xi axis. The
angle of the connecting rod is αi: the angle of rotation of
the Zi-1 axis to the Zi axis around the Xi axis. Connecting

c θi
s θi

c αi s θi
c αi c θi

s αi s θi ai c θi
s αi c θi ai s θi

0
0

s αi
0

c αi
0

di
1

(1)

Where, c=cos, s=sin.
From the Eq. (1) and the established D-H parameter
table, the homogeneous transformation matrix equationof each adjacent joint can be obtained:

04

T 01T21T23T34T

c1c234s1c234c1s234 s1 a3 c1c23 a2 c1c2
s1s234 c1 a3 s1c23 a2 s1c2

s234
0

c234
0

0
0

a3 s23 a2 s2
1(2)

Where, c234 = cos (θ2+θ3+θ4), and the others are similar.
3.2. Inverse kinematics of the manipulator
The common methods of inverse kinematics solving aremainly include algebraic method and geometric method3.
Compared with the algebraic method, the geometricmethod have advantages of simple calculation and avoids
the discussion of range of values by using the constraint
relationship between the joints4. In this paper, the inverse
kinematics of the 4-DOF pulse diagnosis robot is solved
by geometric method.
Before solving the kinematics with the geometricmethod, the first rule is that θ1 has rotated to a position
parallel to the target position. According to theestablishment rule of D-H, it can be obtained of theFig.3 Manipulators structure and its coordinate systems ofjoints.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan404

Pulse Pose Follow Control

schematic diagram of the plane based on the geometricmethod as shown in Fig.4.
Assume that the position coordinates (xp, yp) of theend point P and the azimuth angle γ of the point P are
known. According to the geometric relation, it can beobtained:

xo3

xp

The pulse diagnosis matrix sensor adopted in thissimulation is composed of 5 pressure sensors in a row,(3)a 4 cos γ(a) Pulse sensor model(b) Pulse model
Fig.5 Pulse sensor and pulse model in V-REP

which is shown in Fig.5 (a). Each point of the sensors cansense the corresponding pulse. In the simulation, thepulse sensor is exposed to the pulse at different positions
to obtain the simulation data. By analyzing these data, therelative pose of sensors with respect to the pulse can be
drawn, which will be used to adjust the posture of thejoints.
Pulse model is also established with similar dynamic
properties to the human pulse, as shown in Fig.5 (b). The
width of the pulse model is set as 3.6 *3.6mm2 according
to average adult’s size.

Fig.4 Geometric relationship of three joints in XY plane

yo3

yp

a 4 sin γ(4)4.2. Pulse detect simulation

By using the cosine theorem in △o1o2o3, it can be
draw that:32

arccosxo3

arc tan xyoo 33

2

2

yo3
2a2a3

xo 3

2

yo3

2

2 a 2 xo 3

4

4.

a2

2

2

a3

a2
2

2

yo 3

2

a3
2

(5)
2

(6)
(7)

3

Modeling and Simulation

Fig.6 The first sensor is aligned with the pulse

4.1. Modeling in V-REP

In the V-REP simulation, the pulse beat curve at different
positions is measured with changing the relative positionof the pulse matrix sensor relative to the pulse model.
When the first sensor of the pulse matrix sensor is alignedwith the pulse model, the pulse curve obtained is shownin Fig.6. It is very obvious from the figure that the valueof sensor 1 is the largest, and the center of the pulse
matrix sensor is not perpendicular to the pulse model.

The robot model is established in V-REP as shown in
Fig.1. Set the parameters of the robot model: a2 = 0.08 m,
a3 = 0.1 m, ɑ1 = - 90 °. The initial posture of the robot canbe calculated by assigning the Angle of each joint to 0°:04

T initial position

1
0
0
0

0
0
-1
0

0
1
0
0

0 . 18001(8)This data is identical to the data obtained in the v-rep
simulation.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan405

Q. Liu et al

When the third sensor of the pulse matrix sensor isaligned with the pulse model, the pulse curve is shown in
Fig.7. It is clear that the intensity of sensor 2, sensor 3
and sensor 4 are same to each other. When the curve
similar to Fig.8 is obtained, the pulse matrix sensor canbe considered as perpendicular to the pulse, and the third
sensor is aligned with the pulse.

proposed robot can find the position of pulse, and adjust
the joints to the posture as expected according to the
feedback information from the pulse diagnosis matrix
sensor.

Fig.7 Beating curves of sensors when the third sensor is alignedwith the pulse

Fig.9 Curve change of θ2, θ3 and θ4 during pulse pose following
control

Acknowledgement
This work is partially supported by Innovation Scientists
and Technicians Troop Construction Projects of Henan
Province (Grant No. CXTD2016054).

Fig.8 Curve change of each sensor point during pulse pose
following control

References
1. Y. Zhang, H.Y. Wang, Z.l. Lv, Q. Zhang, C.M. Jia, Y.
Wang. Research on automatic identification system of
traditional Chinese medicine constitution based on
pulsewave. Modernization of Traditional Chinese
Medicine and Materia Medica-World Science andTechnology (J.). 19(10) (2017) 1687-1691.
2. H. Chang and JX. Chen. Piezoelectric pulse diagnosis
transducer of 9x9 sensing arrays and pulse signal
processing, in Proc. Int. Conf. Applied Informatics andCommunication (2011).
3. Ghosal A. Manipulator Kinematics. (In: Nee A. (eds)
Handbook of Manufacturing Engineering and Technology.
Springer, London 2015).
4. W.l. Li and N. Li, Kinematics analysis and inverse
kinematics solution of a 5-DOF manipulator, Mechatronics(J.). 23(11) (2017) 3-7&14.

4.3. Pulse pose follow control simulation
Fig.8 shows that the posture curves of the pulse sensors.
The values changes related to the relative position
between sensors and the pulse, until the matrix sensor isperpendicular to the pulse, and sensor3 is aligned with thepulse. Fig.9 shows that the joints were adjusted to movefrom sensor5 to sensor3, and finally sensor3 isperpendicular to the pulse as expected.
5.

Conclusion

In this paper, a diagnostic robot with a four degree of
freedom manipulator and a pulse diagnosis matrix sensor
is introduced. The kinematic was analyzed and the model
is simulated on V-REP. Simulation results show that the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan406

A New Adaptive Control System Design Method Based on Neural Network Prediction
Weicun Zhang*, Sufang Wang, Yongnan Jia, Qing Li

School of Automation and Electrical Engineering, University of Science and Technology Beijing
30 Xueyuan Road, Haidian District, Beijing, China
E-mail: weicunzhang@ustb.edu.cn, s20170610@xs.ustb.edu.cn
www.ustb.edu.cn

Abstract
In this paper, a new control system design method is proposed, in which, the controller is designed according to amodel with un-modeled uncertainties, an artificial neural network (error back propagation neural network) was
adopted to predict the model output error, and the predicted model output error is considered in the control law. This
control system design can be used to both linear and non-linear systems. The simulation results verified theeffectiveness of the proposed method.
Keywords: Adaptive control, BP artificial neural network prediction, linear and non-linear.

In Fig. 1, yr is reference input of system, y is output
value of system, C is controller, P is controlled plant, M
is system model, ANN is an Artificial Neural Network.
(k) is the difference between the output value of thecontrolled plant and the output value of the model, whichis called model output error or parameter tracking error.

1. Introduction
Many contributions have been made to adaptive control1. But there are still some problems in this important
control field, such as, for a general nonlinear system, it isdifficult to design an adaptive controller with guaranteedstability and performances5-8. In this paper, we present a
new frame as a trial to address the problem. In this frame,a simple self-tuning or non-adaptive controller and an
artificial neural network are adopted to form a parallel
adaptive control system, as shown in Figure 1.
4

yr

C

u

2. The Details of the Controller
Consider a plant to be controlledA( q 1 ) y (t )q d Bu (t ) C ( q 1 )e(t )

(1)

where

P
(k)

y

M

A(q 1 ) 1 a1q 1

an q nB (q 1 )bm q mb0

b1q 1

C (q 1 ) 1 c1q 1

ANN

dFig. 1 The parallel adaptive control system structure

1, b0

cl q l(2)0

Suppose we have the following on-line estimated model*

Corresponding author.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza,Beppu, Oita, Japan407

W. Zhang et al(3)
Am ( q 1 ) y (t ) q d Bm u (t ) C m ( q 1 )e(t )According to this model, we can design a minimum
variance controller to minimize
E [ y (k ) yr (k )]2JSet

prediction model can be established. The model can be in
the form of an AR model or an ARMA model.
The following is described in conjunction with a
specific controlled object and a corresponding parallel
adaptive control system. Consider the following plant:

Cm (q 1 )y (t ) 1.5 y(t 1) 0.3 y(t 2) 0.04 y(t 3)(12)u (t 1) 0.5u (t 2) e(t ) 0.4e(t 1) d (t )Am (q 1 ) F (q 1 ) q d G(q 1 ) (4)Where

F (q 1 ) 1

f1q 1

f d 1q d 1

G (q 1 ) g 0 g1q 1

g n 1q n 1

Where e(t) is white noise with a zero mean and avariance of 0.01; d (t) is a constant interference, taken(5)d (t) 1 . Using low-order modeling:

Then we obtaindeg Am (q 1 )Gy (k ) Bm Fu (k )Cmym ( k d )Fe(k d ) (6)deg Bm (q ) 1
1

deg Cm (q )Accordingly, we get the optimal predictionym ( k d / k )Gy (k )Cm

Bm F
u (k )CmG
y (k )
FBm

(k )

Cm
yr ( k d )FBmym ( k d / k )0l1 (k 1) l2 (k 2) e(k )(14)In the two parts, the parameters of the model are
estimated by recursive least square method.

(8)

Next, we consider a general case, i.e. to control anarbitrary non-linear plant, then there must exists an error
signal:(k ) y (k ) ym (k )(9)Then we havey (k d / k )(13)

The prediction model is taken as the AR(2) model:(7)

Then we have the control law

u (k )

21

ˆ (t )

ˆ (t 1) P (t ) (t ) y (t )
1
1
1

1

P1 (t ) 1 (t )

T
1
T
1

(t 1) ˆ1 (t 1)(t ) P1 (t 1) 1 (t )1

1 P1 (t ) 1 (t ) 1T (t ) P1 (t 1)

P1 (t )(k d / k ) (10)T
1P1 (t 1) 1 (t ) 1

T
1

(t )

a1 ,

, an , b0 , b1 ,y (t 1), y(t 2),, bm, y(t n),, u (t m)

Accordingly the control law (8) can be modified byu (k )Gy (k )FBm

Cm[ yr ( k d )FBm(k d / k )]

ˆ (t )(11)2

ˆ (t 1) P (t ) (t )22
2

P2 (t ) 2 (t )Generally speaking, (k d / k ) can be obtained by any
methods, in this paper, we give two approaches togenerate (k d / k ) and we will give the simulation
result with BP neural network prediction approach.

P2 (t )P2 (t 1) 2 (t ) 1

T
2

In this section we first consider the case where thedifference between the system output and the model
output is a stationary time series. In this case, an adaptive

T
2

T
2

(t 1) ˆ2 (t 1)(t ) P2 (t 1) 2 (t )1 P2 (t ) 2 (t ) 2T (t ) P2 (t 1)
T
2

3. Simulation Results

(t )

(t )

l1 , l2(t 1), (t 2)Control strategy is developed as follows:© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

408

1

Adaptive Control System Designu (k )G (q 1 )y (k )FBm

Cm
yr ( k d )FBm(k d / k )Secondly, a three-layer B-P neural network can be used
to approximate any continuous function as long as there
are enough hidden layer nodes. The structure of the threelayer B-P neural network is shown in Fig. 2.

Fig. 3 Simulation result of B-P neural network
prediction (plant with noise)

Fig. 2 The structure of B-P neural network
Then, when we consider controlling a complex non-line
ar system, we may use a B-P neural network to predict(k d / k )In the formula (12), consider d (t) 0.2sin(0.5 t) ,reference input y * (t) 1 , neural network parameters are
selected as the following:
Input node
20
Hidden layer node10
Output layer node3
The initial weights are set as random numbers whose
absolute values are selected less than 0.3.
The first 80 steps of the system are used to train the neuralnetwork, and then to start the prediction. The simulation
result is shown in Fig. 3. If the noise item in the formula(12) is removed, the simulation effect is shown in Fig. 4.
From the simulation results, it can be seen that the effectof the neural network prediction is significant.
From the simulation results (especially the result of Fig.
4), it can be seen that the system performance using theparallel adaptive scheme based on neural network
prediction is significantly improved compared to system
performance of the traditional adaptive (minimum
variance self-tuning) scheme.

Fig. 4 Simulation result of B-P neural network
prediction (plant without noise)

4.

Conclusions

In this paper, a new adaptive control system design
method based on time series prediction or B-P neuralnetwork prediction is proposed to expand the capabilityof adaptive control. From the effects of simulation
experiments, the use of neural network predictors caneffectively suppress nonlinear interference, thereby
improving the performance of the adaptive control
system.
Acknowledgements
This work was partly supported by National Natural
Science Foundation of China (No. 61520106010;
61741302).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan409

W. Zhang et al

References

5. P. V. Kokotovic, Adaptive Control Design and Analysis.,
Wiley-Interscience [John Wiley &Sons], Hoboken, NJ,
60(3)(2003) 211.
6. Bar Kana I, Parallel feed-forward and simplified adaptive
contro, International Journal of Adaptive Control & Signal
Processing, 1(2), (2010) 95-109.
7. Alvarez ‐ Ramírez J, Adaptive control of feedback
linearizable systems: amodelling error compensation
approach, International Journal of Robust & Nonlinear
Control,9(6), (2015) 361-377.
8. L. Xing, C. Wen and H. Su, A new adaptive control
scheme for uncertain nonlinear systems with quantized
input signal, Journal of the Franklin Institute,352(12),(2015) 5599-5610.

1. Z. D. Yuan, Adaptive Control Theory and Its Application,
East China Normal University Press, Shanghai, (1988).
2. R. H. Middleton, G. C. Goodwin, D. J. Hill and D. Q.
Mayne, Design issues in adaptive control, IEEE
Transactions on Automatic Control (Volume: 33, Issue: 1,
Jan. 1988), pp. 50-58
3. C. C. Fu, H. Khalil, Adaptive control of nonlinear systems
using neural networks, International Journal of Control,55(6), (1992) pp. 1299-1317.
4. K. S. Narendra, S. Mukhopadhyay Adaptive control usingneural networks and approximate models, Neural
Networks IEEE Transactions on, 1997, 8(3), pp.475-485.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan410

Mean-Square Quasi-Composite Rotating Formation Control of Second-Order Multi-Agent
Systems under Stochastic Communication Noises
Lipo Mo, Xiaolin Yuan

School of Science, Beijing Technology and Business University, Beijing, 100048, China
Yingmin Jia*

The Seventh Research Division and the Center for Information and Control, School of Automation Science andElectrical Engineering, Beihang University (BUAA), 37 Xueyuan Road, Haidian District, Beijing, 100191, China
Shaoyan Guo

School of Intelligent Engineering, South China University of Technology, Guangzhou, 510641, China
E-mail: beihangmlp@126.com, 1171681540@qq.com, ymjia@buaa.edu.cn, sanj8023@163.com
www.btbu.edu.cn, www.buaa.edu.cn, www.scut.edu.cn

Abstract
This paper mainly concerns the mean-square quasi-composite rotating formation problem of second-order multiagent systems (MASs) with stochastic communication noises. Firstly, the definition of mean-square quasi-composite
rotating formation is proposed. Afterwards, a distributed control protocol with time-varying control gains is designed.
And then, the origin closed-loop system is changed into an equivalent closed-loop system by taking coordinatetransformation. Under some mild assumptions, sufficient conditions are deduced. Eventually, numerical simulations
are provided to confirm the effectiveness of the proposed theoretical results.
Keywords: Quasi-composite, rotating formation, multi-agent systems, stochastic communication noises.

process. Recently, to imitate or explain such kinds ofmotions, the collective rotating motions3-5, compositerotating consensus6 and formation control7 of secondorder MASs were investigated. Moreover, notice that thecommunications among different agents may beinterfered by stochastic communication noises in reality,
it is necessary to design proper control protocol such thatthe stochastic communication noises can be countered,and numerous of good results have been reported8, 9.
However, up to date, it is a pity to find that there is no

1. Introduction
Over the past few decades, distributed coordination
control of MASs is a research focus and attracts lots ofattentions from various of fields, such as Refs. 1, 2. It is
common to find that there always exist lots of collective
motions in nature, such as the distributed formation flightof satellites around the moon while the moon movingaround the earth. Meanwhile, groups of agents may berequired to maintain desired formation during this*Corresponding author.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

411

L. Mo et al

4. Main Results

result on composite-rotating formation control of MASs
with stochastic communication noises.
Motivated by the above discussions, the mean-squarequasi-composite rotating formation control of secondorder MASs with stochastic communication noises is
considered in this paper, by using which the actual
physical system can be described more precisely. In
contrast with Ref. 10, where the quasi-composite rotating
formation control of MASs was solved without
communication noises, the results of this paper are more
realistic. Besides, the definition of the mean-squarequasi-composite rotating formation of MASs is first
proposed. Moreover, a novel distributed control protocol
by local information with a time-varying control gain isdesigned, by using which the stochastic communicationnoises can be attenuated, and all agents will surround amoving point with a desired structure and a constant> 0 while the moving point
angular velocity
surrounds the origin with a constant angular velocity> 0 asymptotically. Finally, the theories of stochastic
differential equation and proper coordination transformation are used to prove that under some mild conditions,the desired formation can be achieved.

In this section, the mean-square quasi-composite rotating
formation control of MASs (1) and (2) is investigated.
( )), ( ) = ( ) +( ( )+Let ( ) =( ), i = 1,2 … , n denotes the relative velocity andjthe center of the rotation of the ith agent respectively.
Suppose that the information exchange among theagents is affected by the stochastic communication noises.
The information of the ith agent received from its
neighbor agents is defined as follows:
( )+
( )
=
,
(3)
( )
( )+
}, i, k = 1,2 … , n are independwhere k ∈ , {,} are the finite noise
ent standard white noises, {,intensities.
Definition 1. Let ℎ = (ℎ , ℎ , … , ℎ ) = (,,…,) ∈ ℂ denotes the desired formation,
where > 0, ∈ [0,2 ), = 1,2 … , . The MASs (1) and (2) are said to achieve mean-square quasi-composite
rotating formation if for each , = 1,2 … , .
lim E‖c (t) − c (t)‖ = 0,
(4)
→

lim E

→

2. Preliminaries

( )

()

−

= 0,

lim E‖v̇ (t) − jω v (t)‖ = 0,→

Theories of algebraic graph . Let ( , ε, ) be an
undirected graph with n nodes,= {1,2, … n}represents the set of nodes, and ε ⊆ ×represents
the set of edges.
=
represents the weighted
adjacency matrix. Let= { ∈ : (j, i) ∈ } denotes
the neighbors set of node i, ℒ = [ ] denotes the
Laplacian of graph . Graph is said to be connected if
there exists a path between every two distinct nodes.
Lemma 1. 12 If is a connected undirected graph, thenthe Laplacian ℒ of has a simple zero eigenvalue, andis the corresponding eigenvector. Additionally,
0 = (ℒ) ≤ (ℒ) ≤ ⋯ ≤ (ℒ).
11

andlim E‖ċ (t) + jω c (t)‖ = 0,→→

[ ( )] < ∞ ,

(5)

(6)
(7)

[ ( )] < ∞ , for

→

given
> 0 and> 0.
In this paper, the following distributed control protocol
is proposed:( )u ( )= ( − ) ( )−
−a(t)

−a(t)
−

3. Model and Problem Statement[ ( )−

∈

∈
(

)

( )+
(

[ ( )−)( ))],( ( )−
( )+

( ))]

( ( )

(8)

where a(t) > 0 is the time-varying consensus control
gain. To continue, we need the following assumptions.
Assumption 1. The communication topology graph isconnected.
Assumption 2. ∑≠ 0 when | − |=0 or| − | = , , = 1,2 … , .
Assumption 3. 8 ∫ ( ) = ∞.

Consider a multi-agent system with n agents, thedynamics of each agent are given as follows:̇ ( ) = ( ),(1)
̇ ( ) = ( ),(2)where i = 1,2 … , n, vectors ( ) ∈ ℂ, ( ) ∈ ℂ, ( ) ∈ℂ respectively denotes the position, velocity and control
input of the ith agent.

Assumption 4.8 ∫

( )

< ∞.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

412

Mean-Square Quasi-Composite Rotating Formation

Combining the above equations, then we arrive atv̇ ( ) =

− ( )
+

+

1+

1+

( )− ( )∈( )( )] − j

+

j
1+

∈

( )

̇( )=−

j
+
1+

( )−

(1 +

( )−j

)

+
( )−

(1 +

∈

(

∈

+

( )]

( )
(

)

(

)

)

ℒ⊗

)

(

)

(

)

(

)

)

(

,…,

⎛
⎜j

+

−

×

,Ω =

(

)

(

)

j

−

+

(9)
( )

+

)
(

+ℒ⊗

(

(13)

+ℒ⊗

=−

,

)×(

(14)

×

(15)
with
(16)

)

where Λ ∈ ℂ
is a Jordan matrix. From
Lemma 3, all eigenvalues of Λ have negative real parts.
( ) , ζ(t) = ( ( ), ( ), ̅ ( )) ,Let ζ(t) =( )) . Then the origin system canζ̅(t) = ( ( ), … ,be decoupled into the following system:̇( )=( ) + a(t)Ω ,(17)
̇( )=−( ) + a(t)Ω ,
(18)
̇(t)
̅ = [j ( ⊗ ̅) + a(t)Λ] (̅ ) + a(t)Ω .
(19)

)

(

= 0,

ℒ⊗ +ℒ⊗= 0.
Take invertible matrix U = [ , , ] ∈ ℂ= [ , , ] ∈ ℂ × such thatℒ⊗ +ℒ⊗={0,0, Λ},(10)
Let δ(t) = ( (t), (t), … , (t), (t)) , then theclosed-loop system (9) and (10) can be rewritten asδ̇(t) =
⊗ + ( )ℒ ⊗ + ( )ℒ ⊗δ(t)+ ( )Ω ,
(11)
0
−1
0
where A =
,
=
,
=
−
0
0 −
0
−1
,ℒ=
ℒ ∗
, F = diag{ , … , },0 −H = diag{,…,}, η = ( , … , ) ∈ ℂ ,) ∈ ℂ ,Ω =where = (,,…,,diag{Ω , … , Ω } ∈ ℂ+ℒ⊗⊗[ ( )−( )∑ ∈

a(t) ∑ ∈( )[ ( )−

(

(

∈

( )

∈

rotating formation control of MASs (1) and (2) can beachieved by taking the control protocol (8).
1
0
=
⊗
,
=
⊗
, it
Proof. Let01
is clear that⊗ +ℒ⊗ +ℒ⊗=,
(12)

In view of the theory of stochastic differential equation,
the solution of (17) (18) and (19) is
( )=

( )=

(̅ ) =(0) + ∫∫ [a(τ)Ω(0) + ∫( ⊗ ̅)

̅

( ) ]

( ),

a(τ)Ω

(̅ 0)

(20)

( ),

(21)

a(τ)Ω
( ),
+∫ ∫ [ ( ⊗ ) ( ) ]( ),( ), … ,where ̅ = jA, W(t) = (

⎞
) ⎟),

⎝
⎠
i = 1,2 … , n.
Lemma 3. Under Assumptions 1, 2, ℒ ⊗ + ( )ℒ ⊗has zero eigenvalue with algebraic multiplicity 2, andall other eigenvalues have negative real part.
Proof. The proof is similar to the proof in Ref. 10, so
we omit it here.
Theorem 1. Suppose that Assumptions 1, 2, 3 and 4 are
fulfilled. The desired mean-square quasi-composite

(22)

( ),
( ),
( ), … ,( ),(t)) ,,(t) … ,( ),( ) , i, k = 1,2 … , n are standard
andBrownian motions. Therefore, we havelim ‖ (̅ )‖
→

≤ lim
→

+ lim
→

[‖ (̅ 0)‖[(2 − 2) ∫(( )( )) ∫(( )]( )) ∫( )d( ΩΩ∗ ∗ ).
×
(23)
Together Lemma 3 with Assumptions 3 and 4, we havelim ‖ (̅ )‖ = 0.
(24)→

Together ( ) = Uζ(t) with (20) and (21), we have( )−( )‖ = 0.
lim ‖ ( ) −(25)→( ), ( ), … , ( ) , c(t) =Let v(t) =( ( ), ( ), … , ( )) , then we can obtain that
lim

→

( )−

(0)

= 0,

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

413

(26)

L. Mo et al

lim

→

( )−

(0)

= 0,

6. Conclusions(27)

The mean-square quasi-composite rotating formation
problem of second-order MASs with stochastic
communication noises was investigated in this paper. A
novel distributed control protocol with a time-varying
control gain was designed. And then, corresponding
sufficient conditions were deduced. Besides, theeffectiveness of the proposed theoretical results was
confirmed by providing a numerical example.

which indicates that conditions (4-7) are satisfied, i.e.,the desired formation control of MASs (1,2) is achieved.
5. A Numerical Example

Acknowledgements
This work is supported by National Natural Science
Foundation of China (Grant No. 61304155).

Fig. 1. The communication topology graph.

References
1. R. Olfati-Saber, J.A. Fax, and R.M. Murray, Consensus
and cooperation in networked multi-agent systems, Proc.
IEEE, 95(1) (2007) 215-233.
2. Y. Jia, Robust control with decoupling performance forsteering and traction of 4WS vehicles under velocityvarying motion, IEEE Trans. Control Syst. Technol, 8(3)(2000) 554-569.
3. P. Lin and Y. Jia, Distributed rotating formation control ofmulti-agent systems, Systems Control Lett, 59(10) (2010)587-595.
4. L. Mo and Y. Yu, Finite-time rotating encirclement control
of multi-agent systems, Acta Automat. Sinica, 43(9)(2017) 1665-1672.
5. Y. Li, Y. Huang, P. Lin, and W. Ren, Distributed rotatingconsensus of second-order multi-agent systems withnonuniform delays, Systems Control Lett, 117(2018) 1822.
6. P. Lin, W. Lu, and Y. Song, Collective composite-rotatingconsensus of multi-agent systems, Chin phys B, 23(4)(2014) 040503.
7. F. Xiao, L. Wang, J. Chen, and Y. Gao, Finite-timeformation control for multi-agent systems, Automatica,
45(11) (2009) 2605-2611.
8. T. Li and J. Zhang, Mean square average-consensus undermeasurement noises and fixed topologies: necessary andsufficient conditions, Automatica, 45(8) (2009) 1929-1936.
9. L. Mo, S. Guo, and Y. Yu, Mean-square compositerotating consensus of second-order systems with
communication noises, Chin phys B, 27(7) (2018) 070504.
10. L. Mo, X. Yuan, and Y. Yu, Quasi-composite rotating
formation control of second-order multi-agent systems,(submitted).
11. C. Godsil and G. Royle, Algebraic graph theory. (NewYork, Springer Verlag, 2001).
12. W. Ren and R. Beard, Consensus seeking in multiagent
systems under dynamically changing interaction topologies, IEEE Trans. Autom Control, 50(5) (2005) 655-661.

Fig. 2. Whole trajectories of the position and moving rotating
centers of the MASs.

Fig. 3. Local trajectories of the position and moving rotating
centers of the MASs.

To demonstrate the effectiveness of the proposed
theoretical results, consider MASs with 6 nodes. The
communication topology graph of the MASs is shown in
Fig. 1, which is connected. Assuming that the weight ofeach edge is 1. Consider the MASs (1) and (2) with the
distributed control protocol (8). The initial conditions of
the system are taken as (0) = −2.8 − 4.8 , (0) =1.7 + 4.2 , (0) = −1.1 + 6.2 , (0) = 3 + 5 , (0)= −1 + 6 , (0) = 3 + 5.3 , (0) = 2.1, (0) =2.8, (0) = −0.9, (0) = −2, (0) = −1, (0) =−2.2. Takea(t) =,= 0.8,= 0.02, =, = 1,2, … ,6 and choose= 2, = 0.5, =1.5, = 2.5, = 1, = 2.5. We could see the whole(Fig. 2) and local (Fig. 3) trajectories of the position andmoving rotating centers of the MASs for more intuitive,
which implicates that Theorem 1 is effective.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan414

Distributed Energy Resource Control Based on Multi-Agent Group Consensus
Yize Yang

School of Electrical Engineering and Telecommunications, The University of New South Wales
Sydney, Australia
School of Information and Electrical Engineering, Ludong University
Yantai 264025, China
Hongyong Yang*, Fan Liu, Yuling Li, Yuanshan Liu

School of Information and Electrical Engineering, Ludong University
Yantai 264025, China
E-mail: yangyz1994@126.com,hyyang@ldu.edu.cn
www.ldu.edu.cn

Abstract
In this paper, a cooperative control strategy of distributed energy resources based on the group consensus of multiagent systems is proposed. Considering the disturbance effects in actual energy transportation, a multi-agent
dispatch model of energy networks with the matched disturbances and mismatched disturbances is designed. The
control of energy distribution and safe transportation with multiple disturbances is achieved for a group distributed
energy resources. Finally, the numerical simulation results verify the validity of the model and algorithm.
Keywords: Energy resource; distributed control; group consensus; matched disturbances; mismatched disturbances.

With the increasing complexity of networked systems,
the research on the group movement of multi-agent
systems has attracted much attention. Supposing thecomplex network system with multiple subnets, theconvergence of all the agents is achieved by thecommunication protocol, while the different subnets
converge to different states. In Ref. 1, iterativealgorithms for reaching acceptable levels of consensus
in group decision making is investigated. The
heterogeneous agents are governed by the EulerLagrange system and the double-integrator system is

1. Introduction
As an important part of the smart grid, distributed
energy resources play an irreplaceable role in alleviating
grid power supply pressure, balancing peak power loads
and storing excess energy. However, distributed energy
resources also have hidden dangers because of large
quantities, and random geographical locations. The
group cooperative control strategy of multi-agent
systems provides a new idea for solving the technicaldifficulties of the distributed energy grid connection.

*

Corresponding Author© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan415

H. Yang et. al.

investigated in Ref. 2. Group consensus for multi-agent
systems with switching topologies and time-varying
delays is addressed in Ref. 3. Group consensus problem
with discontinuous information transmissions among
different groups of dynamic agents is investigated inRef. 4. A group consensus problem for a multi-agent
network with time-varying topologies is concerned inRef. 5. Many results of multi-agent systems can be
consulted the Ref. 6-9 and their references.

the connected two-bipartite graph. Consider thefollowing form of the second-order multi-agent dynamic
equation:xi t vi t di1 t

V , , A be a weighted undirected

graph of nodes n . V

1, 2,

vertices (or nodes).

V V is a collection of edges.

R

n n

A

aij

aii

0 ).If i, j,nis a collection of

0 ， otherwise aij

j V i, j. Let D

diag d1 , d 2 ,

xi t

,n

(1)

For a dynamic system (1), the disturbances can be
estimated by the observers, for i 1, , n .
dˆl (x p )0.

The Neighbor collection of node i is defined as

Ni

yi t

di 2 t , i 1,

3.1. Design for disturbance observers

is the node adjacency matrix ( i V ,， so aijui twhere xi t is the position states of the agent i , vi t
is the speed states of the agent i , ui t is the control
input of the system, yi t is the control output of theand di 2 t
are the mismatch
system, di1 t
disturbances and the matching disturbances in thesystem respectively.

2. Preliminaries
Let the graph G

vi t

i1

, d n be adegree matrix. The analogous Laplacian matrix of the
diagram G is defined as L D A R n n ,where D isthe degree matrix corresponding to the topologicalstructure between the intelligent bodies, and A is thecorresponding adjacency matrix.
Definition 1. A network with topology
G1 V1 , 1 , A1 is said to be a sub-network of a
network with topology G V , , A if (i) V1 V ,(ii) 1and (iii)the weighted adjacency matrix A1
inherits A . Correspondingly, we call G1 a sub-graph of
G . Furthermore, if the inclusion relations in (i) and (ii)
are strict, and 1
vi , v j : i, j v1 , vi , v j, we
say that the first network is a proper sub-network of thesecond one. Correspondingly, we call G1 a proper subgraph of G .
Definition 2. For a given initial condition
x 0
x1 0 , , xn 0 , where xi 0 R . There is a
X 1 , X 2 , , X m satisfy (i) X i X j
andcollection
m
(ii)
Xk X
x1 , , xn . If the states of agentsk 1
lim xi t x j t
0satisfy(i)
,
iˆ ˆj
and
t
ˆ
ˆ
0 , i j . It is said to solve a(ii) lim xi t x j ttgroup consensus problem asymptotically.

i1

i

pi1
dˆ

vi

dˆi1

pi 2
where dˆuii2

li 2 (vii1(2)

pi 2 )dˆi 2and dˆ

the estimated vectors ofi1
i 2 aremismatched disturbances and matching disturbances inthe system respectively, pi1 and pi 2 are intermediate
vectors, li1 and li 2 are positive observational gains.
Let the error estimation be edik (t ) d ik (t ) dˆik (t ) ,k 1, 2 . The estimation error system is obtained fromthe system (3) and the observer (4).

edi1li1edi1

di1

edi 2

li 2 edi 2

di 2, i 1, 2,,n(3)Assumption 1. The disturbances dik (t ) and dik (t )are limited. That is, there are d ik* supt 0 d ik (t ) andsupt 0 dik (t ) .
ikAssumption 2. The disturbance of dynamic system(3) is lim di (t ) 0 , i 1, 2, , n .
tRemark 1. Assumption 2 is used to prove that the
disturbance estimation error system (3) is
asymptotically convergent. However, if the observer isdynamic enough, the disturbance observer (2) can also
estimate the fast time-varying disturbances by selecting
the appropriate observation gain.
Assumption 3. Each agent can receive theinformation of state and estimated disturbance from its
neighbors.

3. Group Consensus of Distributed Energy
Resources with Mismatched Disturbances
Assuming a networked system composed of n agents,
the network topology of n agents can be represented as© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

416

Distributed Energy Resource Control

Theorem 1. (1) If Assumption 1 is established, theobserver (4) estimates the error is limited. That is, there
are ed*ik sup{| edik (t ) |} and | edik ( ) | ik lik ,i 1, 2, , n t ,0 k 1, 2 .
(2) If Assumption 1-2 is established, the error
estimate dˆik will asymptotically converge to the
disturbance d ik of the dynamic system (1).
Proof. The proof is omitted.
□

Proof. The proof is omitted.
4. Numerical Simulation

Consider a topological structure consisting of 7
distributed energy resources, as shown in Fig. 1.
Assuming that the weights of all the lines in the
topology diagram are 1, the system's analogous
Laplacian matrix is
3 0 0 1 1 0 1
0 2 0 1 0 1 03.2. Design of control protocol for dynamic system
Based on the design of the disturbance observer, thissection will design the compound control protocol
combined with the sliding mode control, to drive thedynamic system (1) to achieve group consensus
flocking.
Based on the disturbance estimation of disturbanceobserver (2), a compound control protocol is designed:nc2 1ki sgn

ui

aij sic2 1 c1 vi

sj

j 1

dˆi1

□

0 0 2
L= 1 1 0
0 0 1
0 1 0
1 0 1

0
2
0
0
0

1
0
1
0
0

0
0
0
1
0

1
0
0
0
2

(4)

dˆi 2 , i 1,..., nwhere c1 , c2 0 ， ki (c1 c2 li1 )ed*i1 c2 ed*i 2i ，
ˆ and dˆ are respectively the estimated
0
，
d
i
i1
i2
disturbance values of the disturbance observer (2). The
dynamic sliding mode surface equation is designed:
si

n

c1

aij ( xi

xj )

j Ni
n

c2

(5)
aij

vi

dˆi1

vj

Fig.1. Graph of multiple distributed energy resources.

dˆ j1

j Ni

To verify the validity of the conclusion, it isassumed that all the agents are affected by different
disturbances. The disturbances of the agent 1 are
d1,1 4tanh 2t , d1,2 2sigmoid t . The disturbancesof the agent 2 are d 2,1 tanh t , d 2,2 sigmoid t .
Thedisturbancesof theagent 3ared 3,1 3sigmoid t, d 3,2 2sigmoid 2t . The
disturbances of the agent 4 are d 4,1 5sigmoid t ,d 4,2 tanh t . The disturbances of the agent 5 are
d5,1 4tanh 2t , d5,2 2tanh t . The disturbances of
the agent 6 are d 6,1 2tanh t , d 6,2 3tanh t . The
disturbances of the agent 7 are d 7,1 sigmoid (t ) ,d 7,2 sigmoid (2t ) . Where di1 and d i 2 are respectively
represented as mismatched disturbances andmatching disturbances.
The parameters in the control protocol are set to
k 50 , c1 10 , c2 8 . For the topology shown in

Theorem 2. Considering the formation of asecond-order dynamic multi-agent system (1), it isassumed that the network topology composed of n
agents can be represented as the two-bipartite graph
G1 ,V1 , A1
and G2 ,V2 , A2 . If Assumption 1,
Assumption 2, and Assumption 4 are established for the
dynamic system with the composite control protocol (4),there are many results as follows:(1) The multi-agent systems eventually converge toa limited area. That is, there are H1 0 and
n

n

lim
t

aij ( yiy j )2H1 .

i 1 j 1

(2) The system output consistency is asymptoticallyachieved. That is lim( yi y j ) 0 , i V1 , j V2 . thetmulti-agent system can finally realize the groupconsensus flocking movement.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan417

H. Yang et. al.

5. Conclusion

Fig. 1, we assume that the initial state of 7 agents are
x 0
5,8, 6,12, 4, 2, 3 , the initial speeds arev 0
5,8, 6, 9, 4, 2, 3 . The multi-agent dynamic
system equation (1) and the control protocol (4) areapplied, and the simulation results of the system
motion are shown in Fig.2 and Fig.3. The
observational estimation errors of the disturbanceobserver are calculated in Fig.2. The mismatch
disturbances can be estimated quickly. The transport
status of each distributed energy resource finallyconverges to the two-equilibrium state in Fig.3.
Multi-agent systems achieve the group consensus,
which verifies the effectiveness of the theoreticalresults. accurately.

A group cooperative control strategy is proposed forcooperative optimal control of multiple distributed
energy resources. Considering the situation of multiple
disturbances in the actual power supply system, adisturbance observer and a state observer are designed
to stabilize load fluctuations and resist external
disturbances in the energy supply system.

Acknowledgements
This paper is supported by the National Natural Science
Foundation of China (61673200, 61771231), Natural
Science Foundation of Shandong Province of China(ZR2018ZC0438, ZR2017MF010, ZR2017PF010).
References
1. Z. Xu and X. Cai, Group consensus algorithms based on
preference relations. Information Sciences, 181(1)(2011)
150-162.
2. H. Hu, W. Yu W, Q. Xuan, et al. Group consensus forheterogeneous multi-agent systems with parametric
uncertainties. Neurocomputing, 142(1) (2014) 383-392.
3. Y. Gao, J. Yu, J. Shao, et al. Group consensus forsecond-order discrete-time multi-agent systems with
time-varying delays under switching topologies.
Neurocomputing, 207(2)( 2016) 805-812.
4. H. Hu, L. Yu, W. Zhang, et al. Group consensus in multiagent systems with hybrid protocol. Journal of theFranklin Institute, 350(3) (2013) 575-597.
5. J. Yu and L. Wang, Group consensus in multi-agent
systems with switching topologies. Systems & Control
Letters, 59(6) (2010) 340-348.
6. H. Yang, F. Wang, F. Han, Containment control offractional order multi-agent systems with time delays.
IEEE/CAA Journal of Automatica Sinica, 5(3)(2018)
727–732.
7. H. Yang, Y. Yang, F. Han, et.al., Containment control ofheterogeneous fractional-order multi-agent systems.
Journal of the Franklin Institute, (2017).
https://doi.org/10.1016/j.jfranklin.2017. 09.034
8. H. Yang, F. Han, M. Zhao, et.al., Distributed containment
control of heterogeneous fractional-order multi-agent
systems with communication delays. Open Physics, 15(2017) 509–516.
9. H. Yang, Z. Zhang and S. Zhang, Consensus of secondorder multi-agent systems with exogenous disturbances.
International Journal of Robust and Nonlinear Control,21(9) (2011) 945–956.

Fig. 2. Errors of mismatched disturbancesFig. 3. Group flocking motion states of multi-agent system.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan418

Robust Adaptive Control of Air-to-air Refueling Boomwith State-dependent Output Constraints
Liang Chang, Yingmin Jia*

The Seventh Research Division and the Center for Information and Control, School of Automation Science andElectrical Engineering, Beihang University (BUAA), 37 Xueyuan Road, Haidian District, Beijing, 100191, China
E-mail: chlchang01@163.com, ymjia@buaa.edu.cn
www.buaa.edu.cn

Abstract
In this paper, a robust adaptive controller design method is presented for the uncertain air-to-air refueling boom
system with output constraints and disturbance. One advantage of this design method is that the output constraints
are state-dependent, which can extend the workspace of the boom while avoiding collision between the boom andthe receiver. Backstepping method and barrier Lyapunov function(BLF) are used in this design method. Simulation
results show the effectiveness of the proposed controller design method.
Keywords: Air-to-air refueling, robust adaptive control, output constraints, barrier Lyapunov function.

constraints, unknown parameter and disturbance. Compared with other output constraints, the state-dependent
output constraints extend the workspace of the boom
while avoiding collision between the boom and thereceiver, furthermore, the constraints can be unknown inadvance.
This paper is organized as follows: Section 2
formulates the control problem studied in this paper;
Section 3 solves this control problem and presents thecontroller design method; an illustrative example ispresented in Section 4 and Section 5 has the conclusions.

1. Introduction
During aerial refueling operation, the boom is too close
to receiver aircrafts1, thus collision avoidance problem is
one of the most important problem in air-to-air refueling.
The most effective method to deal with the collision
avoidance problem is output constrained control method.
There have been plenty of works about this method. Ref.
2 employed a log-type BLF to ensure the output constraints never be violated and a tan-type BLF was presented
in Ref. 3. However, the output constraints in these works
are constants, which will limit the aerial refueling boom’s
workspace. Ref. 4 presented a neural network controller
to solve the robotic manipulator control problem with
time-varying output constraints, but the constraints
require to be known in advance.
In this paper we concern the control problem of airto-air refueling boom with state-dependent output*

2. Problem Formulation
2.1. Aerial refueling boom model
The aerial refueling boom is shown in Fig. 1, which isfitted to the rear of the tanker. Its azimuth and elevation
can be adjusted arbitrarily, and it can be extended to make

Corresponding author© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

419

L. Chang, Y. Jia

contact with the receptacle of the receiver. Thus, theaerial refueling boom can be regarded as a 3-DOF robotic
manipulator. Setting d d i d as the disturbance,
where d 0 is a known constant, qdi as the desired
position of the ith joint (i.e. the joint position when theboom is contacted with the receptacle), which is second
order differentiable. Define q [q1 , q 2 , q3 ]T as joint
variables vector, qd [ q d 1 , q d 2 , q d 3 ]T as joints desired
position vector,[ 1 , 2 , 3 ]T as control input vector,Td [ d1 , d 2 , d 3 ] as disturbance vector, y q qd
Ty1 , y2 , y3 as output vector, then the model of theaerial refueling boom can be expressed as
M (q )q

C (q, q)q

y

q

G (q)

Fig. 2. The aerial refueling boom may cause collision with thereceiver if the output y is not constrained.

process may cause collision if the trajectory of the boom
relative to the receptacle is not constrained (or, equivalently, the output y is not constrained), as shown in Fig.
2, thus the control problem of air-to-air refueling boom
should consider output constraints.
Instead of adopting the traditional output constraints,
we propose a set of novel state-dependent output constraints (as shown in Fig. 3) in this paper as follows:

d,

(1)

qd .

Notice that the elevation of the boom is controlled bycontrol surfaces (See Fig. 1), thus model (1) should be
rewritten as
M (q)qwhere lC (q, q)q

G (q )

y

qd ,

q

L

k oi
k o 3ld,yi k oi ,i 1,2y3 k o 3 r ,

(3)

where

(2)

k oi

0 is a unknown constant, L diag {l ,1,1} ,M diag ( M 11 , M 22 , M 33 ), C diag (C11 , C 22 , C33 ), G [G1 ,G 2 , G 3 ]T , the specific expressions of these matrices are

k o 3r(k oi
C1

k oi ) y32 k o23l
k oi ,
0, k o3l

C2k oi ,

y3
y3

0
i 1,2
0

0,

(4)
(5)

not given due to space limitations.

C 1 , C 2 , k oi , k oi are known constants. Compared with

2.2. State-dependent output constraints

those traditional output constraints, the state-dependent
output constraints extend the workspace of the boom
while avoiding collision between the boom and thereceiver.
Thus, the problem under study in this paper is asfollows: find a controller
such that the closed-loopsystem (2) achieves the following two objectives:

During aerial refueling operation, the boom is contactedwith the receptacle to refuel receiver aircrafts. This

Fig. 3. State-dependent output constraints of air-to-air refueling boom.

Fig. 1. The aerial refueling boom and its mechanical structure
model.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan420

Boom Robust Adaptive Control(i) All the signals in the closed-loop system are global
uniformly bounded with unknown parameter l anddisturbance d.
(ii) The state-dependent output constraints (3)-(5) are
not violated.
The following lemma is needed in the proof of ourmain results.
, the following
Lemma 1.5 For any
0 andinequality always holds,

0where k p

kp ,

tanh

13

k13 z13

qd 3 ,

(6)

2
oi

k
1 2
ln 2
2 i 1 k oi z 12i

qd .

k
1
q ( z 13 ) ln 2
2
k o3r

M

2
o3r

q ( z13 )z130

0. z130z (zq ( z13 ) 13 232
k o 3r

13

(8)

ui

u3

qd 3 )
2
13

z

Lˆinv

k o23l0.

Design virtual controller

,

,

z13

0

z13

0

1

.

(12)

(13)

d

L )

1

).

(14)
(15)

G

T

M u1 , u 2 , u 3 ,z1i

k 2i z 2 i

1ikk 23 z 23

13

(

2
oi

z12i

) M 1i1 d , i 1,2

q ( z 13 ) z 13
k o23 r z 132(1

q ( z 13 )) z 13
)
k
z 132
2
o 3l

) M 131 d ,diag {lˆinv ,1,1} , lˆinv is an estimate of 1/l, k2i ,arepositive design parameters. By using Lemma 1, we have2q( z )
k1i z12i
k13 ( 2 13 2
V1 V2
kp
2
2
z1i
k o 3 r z13
i 1 k oi(16)3
1 q ( z13 ) 2
~
2
1
)
z
k
z
l
l
z
M
.
13
2i 2i
11 1
inv 21
k o23l z132
i 1
For the estimate parameter lˆinv , the adaptive law iswhere

k oi ) z13 z13

L )

1 T
z2 z2 ,
2tanh( 3 z 23 M 131 d

(9)

z132
13

d

tanh(3 z 2i M 1i1 d

qd 3 )z (z(1 q ( z13 )) 13 23 2
k o 3l( Cq G

Cq

Then the time derivative of V1 is as follows
2
z1i ( z 2 iq di ) z12i z13 k oi z13
1iV1
k oi2 z12i
( k oi2 z12i ) k oi
i 1

( k oiz 2 is given by

where

z 132

where

1,

1

(11)

We design the controller
as:
ˆ
ˆ
Linv
Linv [ 1 , 2 , 3 ]T ,

(7)

k2
1
(1 q ( z 13 )) ln 2 o 3l 2 ,2
k o 3l z 132

z13 z 23 ).

z 2T ( M 1 ( Cq G

V2

Choose the following BLF

k oi
z13

1 q ( z13 ))( k13 z132
k o23l z132

2o 3r

and its time derivative is:controller to be designed later. The dynamics of z1 isgiven by
1

(10)

i 1,2
q ( z13 )
k
z132

V2

[ 11, 12 , 13 ]T is a virtual

z2

z1i ,

Then consider the following Lyapunov function:

The backstepping design approach is adopted to design
our controller. Define z1 y [ z11 , z12 , z13 ]T , z2 q 1

q qd

k o 3l k oi

(

i 1

z2

1

k oi ) z132

(k oi

k1i z12i z1i z 2 i
k oi2 z12i

2

The dynamics of[ z 21 , z 22 , z 23 ]T , where2

0 are parameters. Therefore Eq. (9) becomesV1

3. Controller Design

V1

qdi

where k1i

0 . 2785 .

z11i

k1i z1ichosen aslˆz 21 M 111 1 k l (lˆinv l inv 0 ),where kl , linv 0 are positive design parameters.
inv

1 as

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

421

(17)

L. Chang, Y. Jia

meters are k1i

Then we can present our main results.
Theorem 1. Consider the closed-loop system (2)(10)(15),and parameter update law (17). For any positive parameters k1i , k2i , , k l , linv 0 , the following properties hold:

k 2i

kl

1 , linv 0

1/ 8 ,

0.1 .

Fig. 4 shows the output of the air-to-air refuelingboom. The solid lines are the output and the dashed lines
are state-dependent output constraints. We can find that
under the proposed controller (10)(15) the statedependent output constraints are not violated withunknown parameter l and disturbance d, and the statedependent output constraints provide a larger workspace
for the boom than constant output constraints.

(i) All the signals in the closed-loop system are global
uniformly bounded with unknown parameter l anddisturbance d.
(ii) The state-dependent output constraints (3)-(5) are
not violated.
Proof. Consider the following BLF:~
V V1 V2 (l 2) linv2 ,(18)~
where linv 1 / l lˆinv . And its time derivative is:

5. Conclusions

Then from Eq. (19) we can have the conclusion that thetwo properties above hold.

The control problem of air-to-air refueling boom withstate-dependent output constraints, unknown parameterand disturbance has been addressed in this paper.
Backstepping method and BLF are used for the statedependent output constraints problem, the unknown
parameter and disturbance are dealt with by adaptive
control method and hyperbolic functions. Simulation
results show the effectiveness of the proposed controller
design method.

4.

Acknowledgements

V

CV C0 ,C

min( 2 k1i , 2 k 2 i , k l ),C0

kp

( k l 2 )l (1 / l(19)linv 0 ) 2 .

Simulation

The joints desired position qd 1
/2 + ( /9)sin(t )， qd 3

/8 + ( /18)sin(t ), qd 2
13 + 0.5sin(t )，the disturbance d iThis work was supported by the NSFC (61327807,
61521091, 61520106010, 61134005) and the National
Basic Research Program of China (973 Program: 2012C
B821200,2012CB821201).

2000 sin(t )，d 2000 ，the constraints parameters ko1
6 , ko1
4 , ko 2
18 , ko2
18 , C1 1 , C2 6 ,the initial condition is q (0) [0, /3,8 ]T , the design para-

References
1. J. Doebbler, T. Spaeth et al., Boom and receptacle
autonomous air refueling using visual snake optical sensor,
J. Guid. Control Dyn., 30(6) (2007) 1753–1769.
2. K. P. Tee, S. S. Ge and E. H. Tay, Barrier lyapunov
functions for the control of output-constrained nonlinear
systems, Automatica. 45(1) (2009) 918–927.
3. Z. L. Tang, K. P. Tee and W. He, Tangent barrier lyapunov
functions for the control of output-constrained nonlinear
systems, in 3rd IFAC Int. Conf. on Intell. Control andAutom. Sci., (Cheng-du, China, 2003), pp. 449-455.
4. H. F. Huang, W. He and S. Zhang, Neural network control
of a robotic manipulator with time-varying output
constraints by state feedback, IFAC-PapersOnLine, 50(1)(2017) 8969-8974.
5. P. Li and G. H. Yang, Fault-tolerant control of uncertain
nonlinear systems with nonlinearly parameterized fuzzy
systems, In Proc. of 2009 IEEE Control Appl., (CCA) &Intell. Control, (ISIC), (Saint-Petersburg, Russia, 2009),
pp. 382–387.

Fig. 4. The output of the air-to-air refueling boom.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan422

Skill Model Estimation of Ability for Reading Drawings
Kazuo Kawada

Department of Technology and Information Education, Hiroshima University,
1-1-1 Kagamiyama, Higashi-Hiroshima, Hiroshima 739-8524, Japan
E-mail: kawada@hiroshima-u.ac.jp
https://www.hiroshima-u.ac.jp
Tsukasa Hiyama

Graduate School of Education, Hiroshima University,
1-1-1 Kagamiyama, Higashi-Hiroshima, Hiroshima 739-8524, Japan
E-mail: m160404@hiroshima-u.ac.jp
https://www.hiroshima-u.ac.jp
Teruyuki Tamai

Faculty of Education, Ehime University,
3 Bunkyo-Cho,Matasuyama, Ehime 790-8577, Japan
E-mail: tamai.teruyuki.xq@ehime-u.ac.jp
https://www.ehime-u.ac.jp

Abstract
In Japan, students are studying various drawing educations in junior high school technical education, high school
industrial education and university engineering education. However, it is very difficult to read the drawing without
understanding trigonometry. Moreover, it is also very difficult to draw wish parts. Because of this, there are many
research studies, but these are before-after analysis, and none analysis by time-series data. Therefore, this study aims
to develop a new quantitative evaluation method of 3D recognition ability.
Keywords: List four to six keywords which characterize the article.

In order to effectively acquire skills by a teachingguidance, proper support through modeling of learning isimportant. If it is possible to evaluate the learning processfrom initial learning performance, the learning support
according to individuals is possible. One of the authors
proposed a method to regard the skill acquisition process
in the model between teacher and student as "first-order+ time delay" system based on the control engineering
approach1. However, in past studies to evaluate thelearning process, exponential functions or exponentialfunctions are used as learning curves.

1. Introduction
In Japan, technical education of junior high schools,
industrial education of high schools, industrial collegesof technology, and engineering department of
universities, the engineering drafting education isconducted. If students can not understand trigonometry
handled in engineering drawing, they can not read or
draw drawings. From this, there are many efforts toimprove the read ability and drawing ability. However,these efforts were carried out only before-and-after
evaluation, not the one handling the learning curve (timeseries evaluation).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

423

Kazuo Kawada, Tsukasa Hiyama and Teruyuki Tamai

Therefore, we propose a new method to evaluate thereading skill of engineering drawings.

Initial individual population
P1

2. Proposed Reading Skill Modela bc

P2 a b c

A power function and an exponential function of typical
2 equations are often used as a learning curve in the skill
evaluation.
The power function is following equation.
𝑇𝑅 = 𝑎 + 𝑏 𝑁 −𝑐(1)where 𝑇𝑅 denotes the response time, 𝑎 denotes theasymptotic value, 𝑏 denotes the difference between
initial and asymptotic performance, N denotes thenumber of trials, and 𝑐 denotes the learning rate
parameter.
The exponential function is following equation.
𝑇𝑅 = 𝑎 + 𝑏 e−𝑐 𝑁(2)where 𝑎 again denotes the asymptotic value, 𝑏 denotes
the amount that learning can reduce 𝑇𝑅 , 𝑐 denotes the rate
at which asymptotic level performance is approached asa proportion, and N again denotes the number of trials.
We employed the Eq.(2) for evaluating the reading skillof engineering drawings. However, in order to evaluate
skill based on the real learning time rather than theamount of learning (practice), we proposed following
equation by changing from N to real time 𝑅𝑇 (total
learning time) in Eq.(2).
𝑇𝑅 [𝑛] = 𝑎 + 𝑏 e−𝑐 𝑅𝑇 [n](3)where the relationship between 𝑇𝑅 and 𝑅𝑇 when theamount of learning is 𝑛 is as follows.
𝑅𝑇 [𝑛] = ∑𝑛𝑖=0 𝑇𝑅 [𝑖](4)P3 a b c
P4 a b c

PNp a b c

Selection

fitness function

P9 a b c

0.9

P6 a b c

0.5

P1 a b c

0.4

PNp a b c

0.2

P3 a b c

0.1

f ( P)
highnextgeneration

low

Crossover

3. Skill Model Parameters Estimation using a
Real-Coded Genetic Algorithm
The parameters of skill model 𝑎, 𝑏 and 𝑐 are arranged ascells included in a string. These parameters included in
the string are given by real values. The real-coded GA isemployed, which is explained as follows.
(i) Initialization
The generation number 𝐺 is set, and the initialindividuals are produced with random real-codes
within the initial domain which is set in advance.
Here, the number of population is set as 𝑁𝑃 .
(ii) Selection
The fitness value 𝑓(𝑙) is calculated which is
given byparents

children

Pa

Pc

Pb

Pd

Mutation
Pkchild a b c

β percent cells are replaced with random values.

Fig. 1. Schematic figure of GA flow chat.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan424

Skill Model Estimation oft endf (l ) = 1 /{1 + {TˆR (k ) − TR (k )}2 .

the drawing. At this time, the time (response time)required for matching work was measured.
The procedure from (1) through (2) was repeated for 15
times.

(7)t =1

where T̂R and TR respectively denote the learning
time and the estimated learning time by theparameters of skill evaluation model a , b and c .
Each individual Pl is arranged in order, based on
the fitness value. Then,  percent individuals
with superior fitness values are selected, andsaved in the next generation.
(iii) Crossover
The (100 − α ) percent remaining are generated
by the crossover. Two individuals, Pa and Pb
are chosen from among the superior α percent,and new individuals Pc and Pd are generated
by employing the following procedure:

Pc (i ) = Psup (i ) −Pd (i) = Psup (i) +Pa (i ) − Pb (i)4
Pa (i ) − Pb (i),4Fig. 2. 3D models of 30 mm square.

(8)
(9)

where Psup in Eq.(8) and Eq.(9) refers to theindividual with the superior fitness value, i.e., Pa
or Pb . Note that this procedure is employed forevery cell included in Pa and Pb .
(iv) Mutation
Of all individuals which are randomly selected
and given by the crossover, β percent are chosen
and replaced with randomly determined values
within the initial domain.
(v) Update
The procedure from (i) through (iv) is repeated forgenerations.
This procedure is summarized in Fig. 1.

Fig. 3.

Drawing of 3D model.

The total learning time 𝑅𝑇 and the response time 𝑇𝑅 ofparticipants (A and B) measured by experiment are
shown in Table 1. Figures 4 and 5 are graphs of Table 1.
From these total learning time 𝑅𝑇 and response time 𝑇𝑅 ,the red lines in Figs. 4 and 5 were estimated the reading
skill model by using the real-coded GA. Then, estimatedparameters (𝑎, 𝑏 and 𝑐) were showed in Table 2.
It is clear that the estimated read skill models of A (Fig.
4) and B (Fig. 5) fit almost to the response time.
Furthermore, the reading skills of A and B can beunderstood from the parameters of Table 2. Specifically,
although A has a shorter response time than B, theresponse times of A and B are almost equal. This showsthat B has a higher learning rate than A. From the above,it was found that the proposed reading skill model
suggested individual characters.

4. Evaluation of Proposed Reading Drawings
Skill Model
The effectiveness of the proposed reading drawings skill
model was evaluated from the relationship between thelearning time and the reading time (response time) of
drawing. 2 university students who learned the basics ofengineering drawing were the research participants.
The procedure of the experiment is explained as follows.
(1) 17 kinds of 3D models of 30 mm square shown in
Fig. 2 and drawings by third angle projection method
shown in Fig. 3 are prepared.
(2) The participants randomly showed the drawing, andworked to match the three-dimensional model and© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

425

Kazuo Kawada, Tsukasa Hiyama and Teruyuki Tamai

350

References

Response Time [s]300

1. M. Nagamatsu, T. Usuzaka, K. Kawada and T. Yamamoto,Consideration on a Model Between Teacher and Student
Based on Control Engineering Approach, IEEJ Trans. EIS,
134(10) (2014), pp.1537-1542. (in Japanese)
2. A. Rosenbaum, R. A. Carlson, and R. O. Gilmore,
Acquisition of Intellectual and Perceptual-Motor Skills,
Annual Review of Psychology, 52 (2001) , pp.453-470.
3. W. Schneider and R. M. Shiffrin, Controlled andAutomatic Human Information Processing: I. Detection,
Search, and Attention, Psychological Review, 84(1)(1977), 1-66.
4. J. Cohen, K. Dunbar and J. L. McClelland, On the control
of automatic processes: A parallel distributed processing
model of the Stroop effect, Psychological Review, 97(3)(1988), 332-361.
5. I. G. Dobbins, D. M. Schnyer, M. Verfaellie and D. L.
Schacter, Cortical activity reductions during repetition
priming can result from rapid response learning, Nature,428 (2004), 316-319.
6. A. Newell and P. S. Rosenbloom, Mechanisms of Skill
Acquisition and the Law of Practice, Cognitive Skill and
Their Acquisition, J. R. Anderson (Ed.), Erlbaum,
Hillsdale, N. J. (1981).

250
200
150

100
50
0

0

200 400 600 800 1000 1200 1400 1600 1800
Total Learning Time [s]Fig. 4.

Learning curve of A.

350

Response Time [s]

300
250
200

150
100
50

0
0

200

400

600 800 1000 1200 1400 1600 1800
Total Learning Time [s]Fig. 5.

Learning curve of B.

Table 2. The parameters of drawing skill model.

a

b

c

A

40

379

0.0018B

46566

0.0023

5. Conclusions
We proposed a model to evaluate reading skills of
drawings and verified its effectiveness. Specifically, thereading ability of the two participants was evaluatedindividually by using the proposed skill model. In the
future, we plan to develop a learning support systemusing this reading skill model.
This work was supported by JSPS KAKENHI Grant
Number JP16K12760.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

426

Learning Simulation based on a Computational Model of Neuromodulators

Yuki Moriguchi

Hiroshima University, 1-1-1 Kagamiyama,
Higahsi-hiroshima, Hiroshima 739-8524, Japan
Masayasu Nagamatsu

Hiroshima University, 1-1-1 Kagamiyama,
Higahsi-hiroshima, Hiroshima 739-8524, Japan
E-mail: m176972@hiroshima-u.ac.jp, nagamatu@hiroshima-u.ac.jp

Abstract
Reinforcement learning, and its biological basis are key to understand human learning. Here we illustrate learning
processes of the tower of Hanoi based on the computational model of neuromodulators (Doya, K. 2002). The model
assumes hypothetical roles for the following parameters of reinforcement learning. 1) Dopamine signals TD error, 2)
Serotonin controls discount factor, 3) Noradrenaline controls the inverse temperature, and 4) Acetylcholine controls
the learning rate. We have compared the two learning situations. Case a) the reward is given at the goal state only,
and Case b) the reward is divided and placed at the intermediate state and goal state. The resulting learning curves
shows that b) divided reward enhanced the learning. This result shows an example of learning simulation with avariety of learners who have their own learning characteristics.
Keywords: Tower of Hanoi task, Neuromodulator, Reinforcement Learning,

1. Introduction

2. Simulation

Dopamine reward system plays an essential role for aconstellation of findings on human behaviors including
learning processes, addiction, and adaptation to unknown
environments1-8. In artificial systems, recent seminal
papers on complex cognitive tasks indicate the possibilityof such algorithms for modelling the human behavior 9-13.
One of the remaining problems is the modelling of thevariety of human learning characteristics for a variety oftask structures. As for learning characteristics, three basic
parameters need to be considered.
First, credit
assignment problem, which is called the “notorious”
technical problem for problem solving and cognitive skill
acquisition14. Second, the trade-off between explorationand exploitation. To explore all possibilities, we requirethat the behavior policy be soft (i.e., that it select all
actions in all states with nonzero probability), while
exploiting the past15. Third, the balance between memorystorage and renewal, which corresponds to the most basic
parameter of learning.

2.1. Reinforcement Learning
In reinforcement learning, a learner in some state take(1)

Fig. 1. Agent-Environment interaction in Reinforcement Learningsome action, and in typical case, proceed to the next stepand some fortunate learners get some reward(Fig.1). The
goal of the learners is to maximize the total reward. The
typical algorithm is to model the value of possible actions
at each state (Eq.1).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan427

Yuki,Moriguchi, Masayasu, Nagamatsu

α is learning rate, and γ is discount rate parameter whichmodels the effect of future reward.
By Eq.2, the action with highest preference for eachstate are given highest probability. This policy
parametrization is called soft-max16. β is inverse
temperature.

2.4. Result
Figure3 shows effect of reward design on different
learners. In Fig3(a) best performance is achieved at 25
trials. But the performance become worse at later trial.
In Fig 3(b), learning is enhanced in distributed rewardcondition. But the effect of reward design is very(2)
2.2. Roles of the neuromodulators
With the parameters α, β, γ in Eq.1 and Eq.2, we canmodel the characteristics of heterogeneous agents.
Doya17
proposed
thefunctionalmodelofneuromodulators listed below.
1. Dopamine represents the global learning signal for
prediction of rewards and reinforcement of actions.
2. Serotonin controls the balance between short-termand long-term prediction of reward.
3. Noradrenaline controls the balance between wide
exploration and focused execution.
4. Acetylcholine controls the balance between
memory storage and renewal.
These functions have one to one correspondence to theparameters in Eq.1 and Eq.2.

Fig. 3. (a) Learning curves (α = 0.5, β = 5, γ = 0.9)

2.3. Tower of Hanoi task(a) Initial state S(b) Goal state G(c) State space
Fig. 2. State space of 3-disk Tower of Hanoi.

The state space of three-disc tower of Hanoi is shown in
Fig.2. Agents of Q-learning start from state S, and moves
to G. The number of moves is counted. As a reward
design, (a)the reward is given at goal state only(Concentrated reward), and (b)reward is divided andadditional reward is given at intermediate state S8(Distributed reward). The condition is given by all
possible combination of α (0.1, 0.3, 0.5, 0.7, 0.9), β (0.5,
1, 5, 10, 50), γ (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9).
The number of combination is 225. The number of agent
for each combination is 100. The pattern of reward is two.
In sum, total of 45000 independent learners are tested.

Fig. 3. (b) Learning curve (α = 0.5, β = 10, γ = 0.5)

different by the individual learners. Effect sizes (Hedges’d) of different reward plan are shown in Fig.4. Error bars
indicate 95% confidence interval. Distributed reward
plan is effective for learners who have large β(tendency
to choose the same action) and small γ(leaners who tend
to neglect the future reward). But this reward design is
not effective for learners of another learning
characteristics© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan428

Learning Simulation based on

2. Schultz, W., Neuronal Reward and Decision Signals :From Theories to Data, Physiol. Rev. 95 (2015) 853–951.
3. Schultz, W., Dopamine reward prediction error signaling:a two-component response, Nature Reviews Neuroscience,
17 (2016) 183-195.
4. Baldassarre, G. and Mirolli, M., Intrinsically MotivatedLearning in Natural and Artificial Systems, (Springer
2013)
5. Todd M. Gureckis and Bradley C. Love, Computational
Reinforcement Learning, In Oxford handbook ofcomputational and mathematical psychology, 99-117,(Oxford University Press 2015)

3. Conclusion
Based on the computational model of neuromodulators(Doya, K. 2002), learning processes of the tower ofHanoi task for different learning characteristics are
simulated. The same reward design had different effecton different learners. This is an example of learning
simulation for biologically inspired numericalexperiments.
References

Fig. 4. Effect sizes for various learning characteristics.
1. Barto, AG., and Sutton, RS., Reinforcement Learning 2nd,ed. (MIT Press, Cambridge MA, 2018), 341-458.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan429

Yuki,Moriguchi, Masayasu, Nagamatsu

6. Bromberg-Martin, ES., Matsumoto, M. Hikosaka, O.,
Dopamine in Motivational Control: Rewarding, Aversive,
and Alerting, Neuron 68(5) (2010) 815-834.
7. Westbrook, A., and Todd S. Braver, TS., Dopamine Does
Double Duty in Motivating Cognitive Effort, Neuron 89,(2016) 695-710.
8. Shenhav, A., Musslick, S., Lieder, F., Kool, W., Griffith,
TL., Cohen, JD., and Botvinick, MM., Toward a Rationaland Mechanistic Account of Mental Effort, Annu. Rev.
Neurosci. 40(2017) 99–124.
9. Silver,D. Huang, A. Maddison,CJ, Guez,A, Sifre, L. George
van den Driessche, Schrittwieser, J, Ioannis Antonoglou,
I., Panneershelvam, V., Lanctot, M, Dieleman, S, Grewe,D., Nham, J, Kalchbrenner, N, Sutskever, I, Lillicrap, T,
Leach, M, Kavukcuoglu, K, Graepel, T & Hassabis, D.,
Mastering the game of Go with deep neural networks
and tree search, Nature 529 (2016) 484-489.
10. Silver, D., Huang, A., Maddison, C., Guez, A., Sifre,
L.,van den Driessche, G., Schrittwieser, J., Antonoglou,
I.,Panneershelvam, V., Lanctot, M. , Dieleman, S., Grewe,
D.,Nham., Kalchbrenner, N., Sutskever, I., Lillicrap, T.,
Leach, M., Kavukcuoglu, K., Graepel, T. and Hassabis, D.,
Mastering the game of Go without Human Knowledge,
Nature 550(2017) 354-359
11. Yamins, DLK, DiCarlo, JJ., Using goal-driven deeplearning models to understand sensory cortex, Nature
Neuroscience,19(3) (2016) 356-365.
12. Kumaran, D., Hassabis, D., McClelland, JL., What
Learning Systems do Intelligent Agents Need?
Complementary Learning Systems Theory Updated,
Trends in Cognitive Sciences, 20(7) (2016) 512-534.
13. Hassabis, D., Kumaran, D., Summerfield, C., Botvinick,M., Neuroscience-Inspired Artificial Intelligence, Neuron,95(2) (2017) 245-258.
14. VanLehn, K., Problem solving and Cognitive Skill
Acquisition, In M.J. Posner (ed.) Foundations of Cognitive
Science, MIT Press (1989) 529-579.
15. Barto, AG., and Sutton, RS., Reinforcement Learning 2nd,ed. (MIT Press, Cambridge MA, 2018), 91-103.
16. Barto, AG., and Sutton, RS., Reinforcement Learning 2nd,ed. (MIT Press, Cambridge MA, 2018), 321-338.
17. Fum, D., Missier, F.D., & Stocco, A.: Why a model issometimes better than 10000 words, Cognitive Systems
Research vol.8 (2007) 135-142 .

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan430

Estimation of Programming Learning Achievement by Line Tracing Robot

Yoshihiro Ohnishi

Faculty of Education, Ehime University, 3, Bunkyocho
Matsuyama, Ehime 790-8577,JAPANE-mail: ohnishi@ehime-u.ac.jp,
www. ehime-u.ac.jp*
Teruyuki Tamai

Faculty of Education, Ehime University, 3, Bunkyocho
Matsuyama, Ehime 790-8577,JAPANE-mail: tamai.teruyuki.xq@ehime-u.ac.jp
www. ehime-u.ac.jp
Kazuo Kawada

Graduate School of Education, Hiroshima University, 1-1-1, kagamiyama
Higashi-hiroshima, Hiroshima 739-8524,JAPANE-mail: kawada@hiroshima-u.ac.jp
www. hiroshima-u.ac.jp

Abstract
How to estimate learning outcomes of programming learning is an important problem. However, quantitativeevaluation method according to learning outcome is difficult. On the other hand, "work" of a control program
influences the control performance of the plant by petrochemical industry, and to have an influence on the operation
cost, Improvement of a program and am planning for productivity improvement are suggested by evaluating thecontrol performance quantitatively. In this paper, evaluation of the program itself is attempted with hints on thisconcept of control performance assessment. In particular, the higher the degree of difficulty of the program is raised,
the better the result is given to the learner. From this result, a method that can estimate the degree of difficulty of the
program and the achievement of learning is considered. The teaching material is a robot that performs line tracing.
Depending on the degree of difficulty of the program, use a route whose traveling route is different and whose goaltime changes.
Keywords: programming learning, control performance assessment, learning achievement, line tracing robot

the plant. And it also influences on the operation cost.
Therefore, the idea of control performance assessment(CPA) [1]-[6] is becoming important in the processcontrol area. In other words, in the process control, CPA
is used as a program evaluation. CPA is to calculate thequantitative evaluation of "performance of control",which tends to be subjective to the conventional operator,
as "performance index". This method defines the control

1. Introduction
How to estimate learning outcomes of programming
learning is an important problem. However, quantitativeevaluation method according to learning outcome isdifficult. On the other hand, in the process control area
represented by petrochemical industry, "work" of acontrol program influences the control performance of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

431

Ohnishi, Tamai and Kawada

performance related to economic indicators such as
product quality and expenditure of the plant as an index
and defines that a good program efficiently earns money.
In other words, the "effect" by the program is"visualized" as a continuous number "performanceindex".
Technical education at junior high school is expected toincorporate "problem solving" activities into
programming learning. In programming learning,
evaluation criteria tend to be whether a task has beencleared. With this, it is impossible to grasp how much thelearner is approaching the problem solving. This is afactor that makes learning difficult for both learners andinstructors.
In this paper, evaluation of the program itself is attemptedwith hints on this concept of control performanceassessment[7]. In particular, the higher the degree ofdifficulty of the program is raised, the better the result isgiven to the learner. From this result, a method that canestimate the degree of difficulty of the program and theachievement of learning is considered. The teachingmaterial is a robot that performs line tracing. Dependingon the degree of difficulty of the program, use a route
whose traveling route is different and whose goal timechanges.

Fig. 2. Diagram correlating program achievement level andgoal time (performance index).

in this research is an index that can express "work" of aprogram on some measure. As an example, consider arobot teaching material that performs line tracing with an
infrared sensor as shown in Fig. 1 as a clue. The program
following the course on the outside like the trajectory
shown in A performs only the right turn. This program
cannot be expected the good(short) goal time. Therefore,
if a program that follows inner courses like the trajectory
shown in B and can turn right and to the left, the goal time
can be shorten a little more. Furthermore, if the program
which uses a lightness sensor mounted on the robot isused and put in a function that runs in a bright place andenables shortcuts that shown in C, the time can be further
shortened. In this task, the time required until the goal is"performance index", which means a program with ashort time. At this time, there is a limit to the goal timewith the A type program, and in order to obtain a shorter
goal time, a type B program is required. In order toachieve a shorter time, a C type program is required. In
other words, as shown in Fig. 2, if the goal time is shownas the "performance index", the learner's achievementlevel is estimated which shown as Fig.2.

2. Basic conceptThis research develops teaching materials that can beevaluated with "performance index" which quantifies the"work" of the program. "Performance index" considered

3. Practice result
This chapter examines the concepts introduced in
Chapter 2. The undergraduate students were charged withthe task of programming the robot to trace the line in
Figure 1. Undergraduate students who attend this practicebelong to the teacher training course and do not receive
specialized programming education. The objective robot
is shown in Fig. 3, has ‘Arduino’ compatible ‘studuinomini’ [8] as the control board, and has two running

Fig. 1. Line trace course that allows to select a route according
to difficulty level of the program.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

432

Estimation of Programming Learning

Fig. 5. Diagram visualized by program difficulty based on
Table 1.

and if it gets brighter, it must take the right turnmovement. Fig. 4 shows the line tracing run of the robot.
Table 1 shows the results of students' line tracing robot.
Also, a diagram visualized by difficulty level based on
Table 1 is shown in Fig. 5. Comparing Fig. 5 with Fig. 2,
rough trends are similar. In the line trace course in Fig.1,
it can be said that it is possible to roughly estimate thedegree of achievement of programming learning with thegoal time as the performance index. However, some data
reverses between difficulty and goal time. It is necessary
to proceed verification with more data.

Fig. 3. Line trace robot with control board and sensor.

4. Conclusions
In this paper, to estimate the achievement ofprogramming learning was considered. It was challenged
to estimate the degree of difficulty the goal time as aperformance index using the line tracing robot
programming. Better goal time was obtained by more
complex movements and programming with multiple
sensors, was confirmed. In the future, verification will
be carried out at the actual school site.

Fig. 4. Line tracing by Robot.

motors, an infrared sensor and a lightness sensor. The
students who made the program were able to select thethree routes shown in Fig.1. In other words, route A is thesimplest program, route B in the next simple program,and route C in the most difficult program. In the route A,
only the right turn is performed, but in the route B, it isnecessary to perform left and right turns. In order to passthrough Route C, not only the robot recognizes the blackline under the robot, but also recognizes the brightness,

Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP18K02980.
References(1) B. Huang and S. L. Shah : Performance Assessmentof Control Loops: Theory and Applications, Springer,
London (1999)(2) M. Jelali : An overview of control performanceassessment technology and industrial applications,Control Engineering Practice, Vol.14, pp. 441-466(2006)(3) M. Jelali : Control Performance Management inIndustrial Automation ,Springer, London (2013)

Table 1. Measurement result of the goal time divided foreach degree of difficulty.
A
B
C

41.82
35.10
19.95

1’24.63
43.18
20.00

-1’04.32
22.83--23.80© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

433

Ohnishi, Tamai and Kawada(4) T. J. Harris: Assessment of Closed Loop
Performance, Canadian Journal of Chemical
Engineering, Vol.67, pp. 856-861 (1989)(5) M. J. Grimble : Controller performancebenchmarking and tuning using generalized
minimum variance control。 Automatica, Vol. 38,
pp.2111-2119 (2002)(6) L. Desborough and T. J. Harris : Performanceassessment measures for univariate feedback control,
The Canadian Journal of Chemical Engineering,
Vol 70, pp.1186-1197 (1992)(7) Y. Ohnishi: Control Performance Assessment
Method as Assessment of Programming Learning
Achievement,
Proceedings
of
The
2018
International Conference on Artificial Life andRobotics (2018)(8) https://www.artec-kk.co.jp/studuino/en/(Access
Nov.10, 2018)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan434

Development of Cultivate Computational Thinking using Finger Robot
Kaito Omata

Faculty of Education, Tokyo Gakugei University,
4-1-1, Nukuikita-machi, Koganei, Tokyo, 184-8501, Japan.*
Shinichi Imai

Graduate school, Tokyo Gakugei University,
4-1-1, Nukuikita-machi, Koganei, Tokyo, 184-8501, Japan.
E-mail: a151404s@st.u-gakugei.ac.jp, shimai@u-gakugei.ac.jp
http://www.u-gakugei.ac.jp/

Abstract
Computational Thinking is one of the necessary skills for children who live in modern times. Therefore, in thisresearch, we aim to design lesson that can cultivate Computational Thinking. In this research, design a lesson thatcultivates Computational Thinking by moving a robot to the specified finger state using a finger robot. At that time
the children can know that by using the notion of "binary" you can count many numbers by bending and extending
your fingers. By using a finger robot, it is thought that children will motivate themselves to teach and to understandeasily.
Keywords: Computational Thinking, Programming Education, Binary number, Robot, CS Unplugged

Japan 2, 3. According to a lesson using a robot, a robot candraw interest even if the degree of interest andunderstanding of the scientific concept of the child is nothigh. It is also excellent for thinking trial & error. And it
is reported that children can work on programming withenjoying.
Many of the robots used in programming lessons aremobile robots. The mobile robot has the advantage thatthe operation at the time of program execution can beobserved as the behavior of the robot and it is possible tounderstand the control structure and its necessity instages by repetition or case classification at the time ofcollision.
However, there are problems that teachers who can
instruct robot programming are few, and it is difficult forchildren to understand and program sensors and motors.

1. Introduction
In modern times, it is said to be called the fourthindustrial revolution. In Japan, the National Curriculum
was revised in 2017, and in the elementary school a newnational curriculum will be implemented from FY 2020.
In the new national curriculum, programming education
at elementary school level is compulsory. Worldwide, inEngland the subject "Computing" has already been
compulsory in the primary education stage since 2014 1.
From the world viewpoint, it can be seen that knowledge,
skills, qualities and abilities of computers andprogramming are considered necessary in future times.
Towards 2020, lessons using visual programming
languages, lessons using robots, etc. have been tried aspractices of programming education in various regions of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

435

Kaito Omata, Shinichi Imai

By developing robot teaching materials that are simple inprogramming and operate without having a deep
understanding of sensors and motors, robot programmingwith many advantages in programming education
becomes easy.
As an approach to teaching information science toprimary school students, there is "COMPUTERSCIENCE Unplugged" (Next, CS Unplugged) developedby Dr. Tim Bell of New Zealand and others 4.
CS Unplugged is a method that learns basic principles
of computers easily in elementary school students.
Examples include binary numbers, image representations,text compression, and the like. In the world of computers,
information is represented in binary notation. In theprocess of learning programming, learning the structureof the computer makes it possible to deepen theunderstanding of computers and programming.
It is thought that educational materials that can learn
programming and basic principles of computers
simultaneously are necessary in the future. In thisresearch, we develop robot teaching materials that cansimultaneously learn binary numbers and learn
programming.

Fig. 1. Finger robot(a)Digit correspondence(b) An example

Fig. 2. Concept of expressing binary numbers with fingers

2. Educational robot teaching materials

Until now, efforts to teach elementary school students
binary numbers using CS unplugged have been done.
What has been used in CS unplugged is a method of
expressing binary numbers on the front and back sides ofa card on which points corresponding to the decimal
number are drawn. In the CS unplugged, the back of thecard corresponds to "0" and the front corresponds to "1".
In the finger robot, the front and back of the card are
replaced by bending and extending the fingers. Also, by
moving the finger of the finger robot, the decimal number
stretched on the front of the finger robot can be seen /disappeared. This also corresponds to representing on thefront and back of the card.

Fig. 1 shows the appearance of the robot teachingmaterial (Next, finger robot). The finger robot uses
ARTEC Corporation's parts of ARTEC Robo. Finger
robot model human hands. By programming, children
can move each finger. By associating the hand shape ofthe finger robot with the binary number representation,
the child can know the binary number representation. A
decimal number stretched over the front of a finger robot
is an aid for associating the shape of a hand with a binary
number and a decimal number. Children associated with
finger shape and binary numbers can also associate
binary numbers with decimal numbers.
2.1. Binary number
Binary number is a number expressed using two numbersof "0" and "1". In the method of counting binary numberswith fingers, "0" and "1" correspond to the bending andstretching of the finger 5. Fig. 2 shows the idea of binary
numbers using fingers. Fig. 2 (a) shows thecorrespondence between fingers and digits, and (b)shows an example of representing binary numbers with
fingers.

2.2. Hardware
The hardware configuration is shown in Fig. 3. The
finger robot uses four AAA batteries as a battery. The
computer uses Studuino which is Arduino compatible
base. Studuino is connected to a personal computer (PC)to use. As shown in Fig. 4, the PC is used to create a
program to transfer to Studuino and to display the statusof the finger robot.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan436

Development of Cultivate Computational

Fig. 3. System diagram

Fig. 4. Program creation screen

Fig. 6. Program creation flowchart

Software, a visual programming software based on
Scratch. As shown in Fig. 4, Studuino Software creates a
program by combining blocks. Therefore, evenelementary school students can easily experience
programming. Although it is written in English in the
figure, when used by a child, it is used in Japanesenotation. The boxed portion in Fig. 3 is a space called"stage". The decimal number displayed on the stage
changes the appearance of the number corresponding to
the state of each finger of the finger robot. In Fig. 4, "4"and "1" are small. This indicates that the finger robot isbent with fingers corresponding to "4" and "1". In
addition, under the decimal number, the value
corresponding to the state of the finger and the notationof the binary number is displayed. "0" is displayed below"4" and "1" in Fig. 4 so that it can be seen that the current
finger state is "11010" when expressed in binary.
Children use Studuino Software for programming. As
shown in Fig. 6, the child creates a program for moving
the finger portion of the robot to a designated state (eg, astate in which only the ring finger and the thumb are bent).
In the program that moves the finger part, we usefunctions prepared beforehand by the authors. As an
example, the function to move the thumb is shown in
Fig.7. This function is a function that changes the value

Fig. 5. Block that specifies the angle

In the finger robot, one servo motor is used for eachfinger, and a total of five servo motors are used. Human
hand fingers, the thumb has two joints, and the otherfingers have three joints. However, as it is considereddifficult for elementary school students to think ofmultiple joints when programming, only one joint of afinger robot is considered. By connecting the terminal of
the servo motor to the port of Studuino, it becomes
possible to move the servomotor. To control the servo
motor, usually adjust the pulse interval. In Studuino, asshown in Fig. 5, the servomotor can be moved by directly
inputting the angle (arc degree). There are two states ofthe finger part of the finger robot, that is, a bent state andan extended state. In the case of Fig. 1, the fingers(thumb) at the right end and the third finger (middle
finger) from the right are bent and the other fingers are ina stretched state. The servo motor is in a state in which
the finger is bent by 180 ° and the state in which the finger
is extended by 90 °.
2.3. Software
In the finger robot, each finger moves independently. The
software that moves the finger robot uses Studuino© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

437

Kaito Omata, Shinichi Imai

Fig. 8. Program(left) Before robot(above) After robot(under)

5. Conclusion
In this research, we developed a teaching material calleda finger robot which can be used for programming
education in elementary school. Elementary school
students can use the finger robot to know the idea ofcounting binary numbers with fingers while experiencing
programming. Children can deepen their understandingof correspondence between binary numbers and decimal
numbers by associating the state of the finger robot withthe representation of binary numbers and decimal
numbers. In the future, we plan to conduct durability tests
assuming the use of children and setting tasks to copewith various developmental stages.

Fig. 7. Function example: To move thumbof a variable when called. The variable has two variables,a variable representing the state of the servomotor and avariable representing whether the finger portion
represents binary "0" or "1". The child does not create thecontents of the function. The servomotor can be moved
by merely using the function.
3.

Activity

In lesson, children use finger robots in pairs of two child.
The problem is presented from the teacher to move thefinger robot to the specified state. The question ispresented in four kinds of represents. Four kinds ofproblems are problems of fingers represented byphotographs, letters, binary numbers, and decimal
numbers. To these problems, consider a program in pairs.
To get to the state expressed by the problem, think about
which finger to move. From the current state to the stateof the problem, execute the program once and transformit. Therefore, instruct that every time you make a mistake,the current state will not change.

References
1. QCA, National curriculum in England: computing
programmes of study, 2013.
2. Katsuyuki HAGIWARA, Robot Programming for
Elementary School Children, Mie University Faculty of
Education Departmental Bulletin Paper, Vol. 68, 2017, pp.
307-315.
3. KOUJI KAI, YOSHIHIKO KIMURO, YOSHIFUMISAKAGUCHI, HIROTO YASUURA, An Educational
Method of Computer Principles for Elementary and JuniorHigh School Students, Transactions of Information
Processing Society of Japan, Vol.43, No.4, 2002, pp.11211131.
4. Tim Bell, Ian H.Witten, Mike Fellows, COMPUTERSCIENCE unplugged, 2015.
5. Frederik Pohl, The best of Frederik Pohl, Ballantine Books
Co., Ltd., 1975.

4. Results of using teaching materials
The correspondence between the program and the
movement of the finger robot is shown in Fig. 8. In thestate shown in Fig. 8, when the program on the left ofFig. 8 is executed, the state is changed to the state shownin the lower part of Fig. 8.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

438

Different Type of Interaction or Decision Error Contribute to Functional Differences
Saori Iwanaga

Department of Safety Technology, Japan Coast Guard Academy, 5-1 Wakaba
Kure, Hiroshima 7378512, Japan
Masao Kubo

Department of Computer Science, National Defense Academy of Japan, 1-10-20 Hashirimizu
Yokosuka, Kanagawa 2398686, Japan
E-mail:s-iwanaga@jcga.ac.jp, masaok@nda.ac.jp

Abstract
There are many situations where interacting agents can benefit from coordinating their actions, that is, agents gain
payoffs by taking the same action as others. These situations can be modeled as coordination games. Anothersituations is modeled as complementary game, asymmetric coordination games or minority games. In complementary
game, agents gain payoffs by taking the different action from others. We deal with these coordination games andasymmetric coordination games as type of interaction and each agent can change the interaction type. And each agent
can change the interaction partner. We show that population changes to a mixed population comprised of 80%
cooperative agents and 20% complementary agents. A mixed population diversify the collective behavior, type ofinteraction and preference. There, opposite type of interaction plays a role of small decision error, and it is inefficientto adopt both.
Keywords: Collective behavior, decision error, diversity, payoff matrix, type of interaction, preference.

1.

rarely reproduce emerged diversity in population.
There are many situations where interacting agentscan benefit from coordinating their actions, that is, agents
gain payoffs by taking the same action as others.
Examples where coordination is important include tradealliance and the choice of compatible technologies orconventions, such as the choice of a software or language.
These situations can be modeled as coordination games7,
8. Complementary games that have received attention
recently are asymmetric coordination games or minority
games. In these games agents gain payoffs by taking thedifferent action from others. Examples whereasymmetric coordination is important is division of labor,
traffic congestion or site selection in opening shops. The
Santa Fe bar problem and the minority games are alsomodeled in the same manner.

Introduction

Studies into multi-robot and multi-agent simulations
began in the late 1980s. Reynolds1 shows that collective
behavior occurs when three rules are used simultaneously
by Boids and that the collective behavior is a complex
system. Sims' study2 of virtual block creatures andTerzopoulos' study3 of artificial fish provide examples of
emerged physical or functional characteristics. Kuboet.al4 demonstrates the emergence of unique individuals
in a competition for food between ant colonies. However,individual heterogeneity is often ignored in most studies5.
For example, a recent study investigated the formation bymulti robots. In a multi-robot system, if heterogeneous
robots exist, such robots are removed6. There are limits
to the adaptation performance of multi robot systems
with heterogeneous robots, and computer simulations© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

439

Saori Iwanaga, Masao Kubo

We deal with these coordination games and asymmetriccoordination games as type of interaction. In this study,
we deal with population in which each agent can change
the interaction type. We show that small decision error
makes collective behavior efficient. We also show thatwith without error is good for population in which eachagent can change the interaction type.

expected utilities by S1 and S2 at time t, which is given
as follows,
Uc (S1, t) = pi (t) (1- θi), Uc (S2, t) = (1- pi (t)) θi.
(1)
Agent Ai chooses S1 if Uc (S1, t) > Uc (S2, t), or chooses
S2 if Uc (S1, t) < Uc (S2, t) at next time t +1. The decision
rule of agent Ai at time t +1 is described as follows. Agent
Ai chooses10,
S1 if pi (t) > θi, S2 or if pi (t) < θi.
(2)That is the best response rule and the decision
depends on the parameter θi. When pi (t) = θi, we assume
that the agent chooses the same behavior as at time t. We
define the parameter θi as threshold of agent Ai.
According to the threshold, behavior of the agentsdifferentiate from each other. Moreover, the proportionof agents having chosen S1 among the partners pi (t) ofagent is different from each other. Then, the decision of
the agent different from each other. That is, the thresholdand social network of agent effects on the behavior of theagent, they form the collective behavior, which is the
behavior of the whole population. We define thecollective behavior p(t) as the proportion of agents
having chosen S1 in population at time t. We call theagent with payoff matrix of Table 1 as cooperative agent.

2. Model2.1. Social Network
We consider the lattice structure for social network ofagents and each agent interacts with only its neighbors.
We deal with the von Neumann neighborhood model, in
which each agent interacts with its nearest four neighbor
agents all at once. Lattice model is used to make theecological characteristic of living beings clear 9.

Fig.1. von Neumann neighborhood model.

Table 12.2. Type of Interaction

The payoff matrix of agent Ai (cooperative).
Behavior of Other Agent

There are two behavior, S1 and S2, for each agent. We
adopt the payoff matrix given in Table 110, which
describes the outcome of the interaction between agents.
In this matrix, if two agents chose the same behavior,agent Ai can receive a positive payoff 1- θi or θi, or agent
Ai receive payoff 0 where 0 ≤ θi ≤ 1. This is coordination
game. Contrastingly, there is another game, if two agents
chose a different behavior, agent Ai can receive a positive
payoff. This is a complementary game. The payoff matrix
is given by Table 211.

Coordination Game

Behavior of
Agent Ai

S1

S2

pi (t)

1- pi (t)

S1

1- θi

0

S2

0

θi

Table 2 The payoff matrix of agent Ai (perverse).
Behavior of Other Agent

Complementary
Game
Behavior of
Agent Ai

3. Behavior of Agents
3.1. Decision of the Choices: Best Response RuleS1

S2

pi (t)

1- pi (t)

S1

0

θi

S2

1- θi

0

In contrast, Agent Ai in a complementary gamecalculates the expected utilities by S1 and S2 at time t,which is given as follows,
Ua (S1, t) = (1- pi (t)) θi, Ua (S2, t) = pi (t) (1- θi).
(3)
11Then, agent Ai chooses ,S1 if pi (t) < θi, S2 or if pi (t) > θi.
(4)When pi (t) = θi, we assume that agent chooses the same
behavior as at time t. We call the agent with payoff matrixof Table 2 as perverse agent.

Agents are not aware of the behavior the neighbor agentswill take, but know the last behavior the neighbor agentstook. We define the proportion of agents having chosen
S1 at time t as pi (t) (0 ≤ pi (t) ≤ 1) in four neighbors ofagent Ai. Each agent acts to maximize the utility andtakes the best response against the behavior which theneighbor agents took by calculating the expected payoff.
In a coordination game, rational agent Ai calculates the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan440

Different Type of Interaction

3.2. Decision Error
We set very small probability an agent will make a
mistake. If agents are not absolutely rational, there arepossibility to make mistakes, which we call decision
error. We define the probability as ε12. And each agent
make decision depending on (2) or (4) with probability
1- ε. With probability ε, each agent make decision
opposite from (2) or (4).

Agent Ai chooses cooperative game if Uc (S1, t) > Ua
(S1, t), or chooses complementary game, or if Uc (S1, t)< Ua (S1, t) at next time t +1. The decision rule of agent
Ai at time t +1 is described as follows. Agent Ai chooses
cooperative game if pi (t) > θi,complementary game or if pi (t) < θi.
(6)When pi (t) = θi, we assume that agent chooses the same
game as at time t.

3.3. Movable for New Neighbors

2.5.2. When agent chooses S2 at time t

If an agent wants to receive a high payoff, it is aneffective method to move to another site and change its
partners with which it interacts7, 13. If the average payoffof agent Ai received through and interaction with fourneighbor agents is above the threshold μ, agent Ai stays
the site. Or agent Ai moves to another site. In cooperative
or complementary games, the expected utilities by S1 andS2 at time t, which are given as (1) or (3). The rule ofchange neighbor agents is as follows, where v = a, c andk = 1, 2. Then, at next time t +1 agent Ai13,
stays at the site if Uv (Sk, t) ≥ μ,moves to another site if Uv (Sk, t) < μ.
(5)Previously, we showed that movability of agent iseffective for collective behavior13.

In contrast, agent Ai chooses cooperative game if Uc (S2,t) > Ua (S2, t), or chooses complementary game if Uc (S2,t) < Ua (S2, t) at next time. The decision rule of agent Ai
at time t +1 is described as follows. Agent Ai chooses
cooperative game if pi (t) < θi,complementary game or if pi (t) > θi.
(7)
When pi (t) = θi, we assume that agent chooses the same
game as at time t.
4. Simulation settings
We arrange agents for an area of 50×50 (2500 agents)
with no gap, four corners and ends of an area connect itwith an opposite side. First, we set the same threshold θi=0.5 for all agents. The Initial collective behavior p(0),which is the proportion of agents having chosen S1 in
population, from 0.0 to 1.0 at intervals of 0.125, where
we assume that each agent chooses at random. Each agent
makes its decision depending the best response rule (2)(4) each time step, and then collective behavior p(t)changes. We simulate 10,000 time steps for a trial. We
simulate 100 trials per one Initial collective behavior andinvestigate Final collective behavior p*. We set threecases, in which agents make decision with no noise, withnoise 0.001 and with noise 0.01.
The rule of moving agents is given by (5). For eachcase, we set the initial ratio of agents adopting acooperative game, 100%, 50% and o%. Each agent canselect the game (6) (7). We set parameters as follows
based on preliminary experiments, the threshold to move
μ as 0.3, interval of self-reinforcement T as 10, increasing
delta ∆ as 0.01 and interval of value change R as 20.
Increasing delta ∆ must be slower than decision speed for
emergence of preference diversity. Similarly, we set agreater interval of value change R than an interval of selfreinforcement T for emergence of type of interaction
diversity. We evaluate the average utility at last time step.

3.4. Self-Reinforcement of Preference
Each agent can update the payoff parameter throughinteraction. We define this as self-reinforcement ofpreference. As for the coordination game in Table 1, if
Agent Ai13 has been chosen S1 for T times in succession,
the agent increases the payoff 1- θi for S1 by delta ∆ anddecreases the payoff θi for S2 by delta ∆. Or if agent Ai
has been chosen S2 for T times in succession, the agent
increases the payoff θi for S2 by ∆ and decreases thepayoff 1- θi for S1 by ∆.
In contrast, in the complementary games in Table 2,
if Agent Ai has been chosen S1 for T times in succession,
the agent increases the payoff θi for S1 by ∆ anddecreases the payoff 1- θi for S2 by ∆. Or if agent Ai has
been chosen S2 for T times in succession, the agent
increases the payoff 1- θi for S2 by ∆ and decreases thepayoff θi for S1 by ∆.
3.5. Type of Interaction
3.5.1 When agent chooses S1 at time t© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

441

Saori Iwanaga, Masao Kubo

5. Simulation resultschanges to a mixed population comprised of 80%
cooperative agents and 20% complementary agents. A
mixed population diversify the collective behavior, typeof interaction and preference. There, opposite type ofinteraction plays a role of small decision error, and it isinefficient to adopt both.

Simulation results are shown below. The results are when
initial collective behavior p(0) was 0.5 as represents. In
these figures, x-axis is the average final collective
behavior p* and y-axis is the average utility. Simulation
result when agent makes decision error is shown in Fig.2.
All results show that about 50 % of agents choose S1 atlast. Without error, in population with 100% or 0% ofcooperative agents, which described by "co" or "per", theaverage utilities with error 0.001 are higher than thatwithout errors or with error 0.01. But, "50% mix"
population of both types shows the opposite result.
Simulation result when agent can change interaction
type is shown in Fig.3. Without error, the average utilities
are higher than that of with decision error. There, the ratioof cooperative agents become about 80% at last.

References
1. C. Reynolds, Flocks, Herds, and Schools: A Distributed
Behavioral Model, Computer Graphics, in SIGGRAPH '87
Proceedings of the 14th annual conference on Computer
graphics and interactive techniques, ACM (USA, New
York, 1987), pp.25-34.
2. K. Sims, Evolved Virtual Creatures, Computer Graphics,
in SIGGRAPH '94 Proceedings of the 21st annual
conference on Computer graphics and interactive
techniques, ACM (USA, New York, 1994), pp.15-22.
3. D. Terzopoulos, X. Tu and R. Grzeszczuk, Artificial
Fishes: Autonomous Locomotion, Perception, Behavior,and Learning in a Simulated Physical World, Artificial Life.
1(4) (1994) 327-351.
4. M. Kubo and Y. Kakazu, Evaluation of CoordinatedMotions of Multi-Agent Systems on Competition for Foodbetween Ant Colonies, IPSJ Journal (in Japanese). 35(8)(1994) 1555-1566.
5. R. Axelrod and M.D. Cohen, Harnessing Complexity(Free Press, New York, 2001).
6. M. Rubenstein, A. Cornejo and R. Nagpal, Programmableself-assembly in a thousand-robot swarm, Science, 345(6198) (2014) 795-799.
7. T. Schelling, Micromotives and Macrobehavior (Norton,
New York, 1978).
8. B. Huberman and N. Glance, Diversity and Collective
Action, in Interdisciplinary Approaches to Nonlinear
Complex Systems, Springer Series in Synergetics 62,(Springer, Berlin, 1993), pp.44-64.
9. M.A. Nowak, Evolutionary Dynamics: Exploring theEquations of Life (Harvard University Press, Cambridge,2006).
10. S. Iwanaga and A. Namatame, The Complexity ofCollective Decision, Journal of Nonlinear Dynamics andControl. 6(2) (2002) 137-158.
11. S. Iwanaga and A. Namatame, Emergent Division ofLabor and Evaluation of Its Efficiency and Equity,
International Journal of Knowledge-Based Intelligent
Engineering Systems. 7(3) (2003)115-123.
12. R. Selten, Reexamination of the perfectness concept forequilibrium points in extensive games, International
Journal of Game Theory. 4(1) (1975) 25–55.
13. S. Iwanaga and A. Namatame, Influence of Partner
Selection on Functional Differentiation: Emergence ofDiversity by Isolated Interaction and Preference Change,
Journal of Robotics, Networking and Artificial Life. 3(4)(2017) 257-264.

Fig.2. Effect of decision error.

Fig.3. Effect of different type of interaction.

6. Conclusion
We deal with these coordination games and asymmetriccoordination games as type of interaction and each agent
can change the interaction type. We show that population© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

442

Evaluation for the Synchronization of the Parade with OpenPose
Yohei Okugawa

Dep. of Computer Science, National Defense Academy of Japan
Yokosuka, Kanagawa, 239-8686, Japan
Masao Kubo

Dep. of Computer Science, National Defense Academy of Japan
Yokosuka, Kanagawa, 239-8686, Japan
Hiroshi Sato

Dep. of Computer Science, National Defense Academy of Japan
Yokosuka, Kanagawa, 239-8686, Japan
Bui Duc Viet

Dep. of Computer Science, National Defense Academy of Japan
Yokosuka, Kanagawa, 239-8686, Japan
E-mail: em57037@nda.ac.jp, masaok@nda.ac.jp
www.nda.ac.jp/cs/

Abstract
In military organization, the synchronization in the parade is important for showing their majesty. However, we don’t
have any quantitive evaluation methods for the parade. In this research, we propose an evaluation method focusing
on the synchronization level of the parade with OpenPose. In order to measure the synchronization level, our method
is based on the timing of arm swing data in the parade, and the result shows our method is appreciable for measuring
the synchronization level.
Keywords: OpenPose, Unity, Parade, Synchronization, Multi-person Pose Estimation, Multi-Agentand it is suitable for parade analysis, which is acooperative operation of multiple people (Fig. 1).

1. Introduction
In the military parade, the synchronization level of eachmember is a major factor of the performance. Shimohagi
et al. conducted scientific analysis for thesynchronization of the parade[1],[2],[3], however,measuring synchronization level of the parade requires alarge number of posture data at same time and is difficult.
We don’t have any quantitive evaluation methods for thesynchronization. In this research, we propose theevaluation method focusing on the synchronization levelof the parade with OpenPose. OpenPose [4] can measure
many kinds of postures from movies simultaneously,
therefore it is much easier to acquire data thanconventional motion analysis with motion capture [5],[6],Fig.1 Military Parade with OpenPose(https://www.youtube.com/watch?v=zaviumhF7aA)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan443

Yohei Okugawa, Masao Kubo, Hiroshi Sato, Bui Duc Viet

As a first step of the research, in order to measure thesynchronization level, we focus on the arm swing data ofeach cadets in the parade and analyze "Marking Time".
Marking Time is a military step in which soldiers marchin place, moving their legs as in marching, but without
step forward.

because the values and accuracy of the posture data aredifferent depending on the position and body size.
Therefore, it is necessary to use a trend which is not
influenced by position or height as an index.
3.

Proposed Method

In this section, we propose the measurement method
using indicators focusing on inflection point of
movement. In the following, we will explain the
proposed method in order.

In this paper, we analyze the synchronization of two
members. When we evaluate the synchronization level ofthe two members appropriately, in the case of multiple
people, we evaluate it in combination of two members.

3.1. Focus Points

This paper is organized as follows. After introducing
OpenPose in the next section, we describe the dataprocessing method from OpenPose. In the section 4, we
propose the evaluation method for the synchronizationlevel. After explaining the two experiments with 3D CG
and real behavior of Marking Time in the section 4, weshow our results from the experiments. Considering theresults, our evaluation method is appreciable for real
Parade.

Focusing on the synchronized arm swing of the left handduring the parade, we obtain the data of each shoulder (5in Fig. 3), elbow (6 in Fig 3) and perm (7 in Fig.3). We
converted the data into 4 angles: φ, ψ, λ, θ (Fig. 3)

2. OpenPose
OpenPose represents simultaneous posture recognition ofmultiple people on single images. The left side of Fig.2
shows an example of pose estimation with OpenPose. If
the movements of the two are synchronized, it can be
expected that the same trend data can be obtained.

Fig 3. Focus Points of OpenPose

3.2. K-means Clustering
In order to get time series data of each person’s arm
swing angle in the image, we conduct the clustering using
K-means method (1) for the angle data. In this section,
we analyze the Marking Time movies for two members,
thus we set K=2 for the cluster analysis using K-means
method.
arg minFig.2 Multi-person Pose Estimation with OpenPose

,

∑

min

−

(1)

3.3. Smoothing

The right side of Fig. 2 shows the points of posture data.
] onℎAs a three-dimensional data of [ , ,the image according to the number of points, we canobtain 85 points of data for one piece of posture
information. It is difficult to measure the synchronization

In order to read the inflection point of the time series dataof arm swing angle accurately, we conduct the smoothing
process for the angle data (2).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan444

Evaluation for the Synchronizationm=+ 0.5 × (

)+∑

+

+

4.1. Experiment 1: The Accuracy of Our Method(2)

For measuring the effectiveness of our method, weanalyze the 3DCG models. Since we can adjust thesynchronization level between two models, it can be an
ideal analysis target. Evaluation using 3DCG created by
Unity is appropriate because we can completely control
human-like model. In the 3D CG, the two models are
doing the Marking Time (Fig. 5).

Fig. 4 Data Smoothing Before and After

Fig. 4 shows an outline of the data before and after thesmoothing process. It can be seen that the outline issmooth, and the inflection points are easy to read.
3.4. Evaluation
We get the time series data of arm swing angle (3).
⃗=[

,

]

,⋯,

Fig. 5 An example of 3D CG Models with OpenPose(3)

The arm swing angle of the models during the Marking
Time was implemented according to the simple vibration
equation (5). In this section, the phase difference of thearm swing angle is expressed by changing the initialcondition ofas 0, π / 8, π / 4, π / 2 and π.

⃗ is the time series data of arm swing angle of member
A in a image, andis the arm swing angle ofmember A at the first frame of the movie.
From each time series data of the arm swing angle, we
use correlation coefficients for evaluation.
We define the synchronization level r as below,
r=

∑

∑

(

(

̅)

̅ )(

∑

(

)

.

x = A sin(

+

)

(

=2 )

(5)

(4)

)

We use Pearson’s moment correlation coefficient as thesynchronization level. If r approaches 1, it issynchronized. If r approaches -1, it is expected that thephase will be in opposite.
4. ExperimentFig. 6 The Synchronization Level r of 3D CG

In this chapter, we conducted two kinds of experiments.
In Experiment 1, we examine measurement accuracy andevaluation index of our method using 3D CG models.
After confirming the effectiveness of the evaluation
method, in Experiment 2, we evaluate thesynchronization level r in real Marking Time behavior
with our method.

Fig. 6 shows the synchronization level r in Eq. (4) in timeseries angle data of two models. As the initial value ofshifts, the value of r decreases. Also, when the initialvalue ofis π, that is, each arm swing angle is in
opposite phase, it approaches -1. From the result, the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

445

Yohei Okugawa, Masao Kubo, Hiroshi Sato, Bui Duc Vietsynchronization level r is monotonically decreasing
according to the value of . Therefore, it was confirmedthat our evaluation method is appreciable as a measuring
the synchronization level in the 3D CG Marking Timemovie.

decreases. This shows the same trend as the result ofExperiment 1, and it is our evaluation method isappreciable for measuring the synchronization in real
Parade. In addition, although it was suggested thatmeasuring the synchronization level with armpit angle
has good sensitivity for evaluation, further considerationis needed as future task.

4.2. Experiment 2:The Evaluation of Real Parade
Since we can show our evaluation method is appreciable,
we experiment for real Marking Time with our method
which is appreciable for evaluation the synchronization.
As in Experiment 1, we analyze Marking Time for two
members. We experiment for two cadets of the National
Defense Academy, which have parade training every day,and their height is 10 cm different (Fig. 7).

5. Conclusion
We propose the evaluation method for thesynchronization level of the Parade with OpenPose. We
focus on the arm swing data and analyze the Marking
Time for two members. Through the simulation
experiment using 3DCG, it was confirmed that our
method based on time series angle data is appreciable formeasuring the synchronization level. Moreover, throughthe real Marking Time of Parade, our evaluation method
is appreciable for measuring the synchronization level. In
addition, it was suggested that measuring thesynchronization level with armpit angle has goodsensitivity for evaluation.

References
1. Takato Shimohagi (2017), Parade Analysis based on
Network System, In Proceedings of National Defense
Academy of Japan Graduation Thesis, pp. 61-62(inJapanese).
2. Tento Ishihara (2018), Development of VR parade training
system, In Proceedings of National Defense Academy ofJapan Graduation Thesis, pp. 47-48(in Japanese).
3. Yohei Okugawa, Masao Kubo, Takato Shimohagi, Tento
Ishikawa, and Hiroshi Sato (2018), Analysis of a Paradewith the Kuramoto Model -For Better Performance
without Trainings-, In Proceedings of IEEE SMC 2018.
4. Z. Cao, T. Simon, S.E. Wei, and Y. Sheikh (2017),
Realtime multi-person 2d pose estimation using partaffinity fields. In Proceedings of the 2017 IEEE
Conference on Computer Vision and Pattern Recognition.
5. Kentaro Inada, Takaaki Ishikawa and Hiroshi Watanabe(2018), Coincidence Evalustion of Multiple People’sDance Using OpenPose, In Proceedings of the 80th
National Convention of IPSJ.
6. Masato Nakai et al. (2018), Prediction of Basketball Free
Throw Scoring by OpenPose, In Proceedings of the 32nd
Annual Conference of the Japanese Society for Artificial
Intelligence.

Fig. 7 A Capture of Real Marking Time with OpenPose

For Experiment 2, we have three types of arm swing:
Phase Synchronization, Phase Difference and Phase
Opposition. Fig 7 shows a snap shot of Phase Opposition.
We can see both cadet arm swing is opposite in Fig. 7.

Fig. 8 The Synchronization Level r of Marking Time

Fig. 8 shows the synchronization level r of Marking Time.
As shown in Fig. 8, we can confirm that as the differenceof the arm swing angle shifts, the synchronization level r© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

446

Phase shift estimation of the bifurcating neuron from superimposed chaotic spike sequences
Akihiro Yamaguchi*, Yutaka Yamaguti

Department of Information and Systems Engineering, Fukuoka Institute of Technology,
3-30-1 Wajiro-higashi, Higashi-ku, Fukuoka, 811-0295, JAPANMasao Kubo

Department of Computer Science, National Defense Academy of Japan,
1-10-20 Hashirimizu, Yokosuka, Kanagawa, 239-8686, JAPANE-mail: *aki@fit.ac.jp

Abstract
For the bifurcating neuron, the phase shift value of background oscillation is one of the typical parameters that
determine the shape of dynamics. In this research, we propose an estimation method of the phase shift values fromsuperimposed chaotic spike sequences generated by the bifurcating neuron. The proposed method was examined by
the numerical experiments for the estimation of the phase shift values and the identification of superimposed chaotic
spike sequences up to 12 spike sequences under additive noise.
Keywords: bifurcating neuron, neural coding, background oscillation, chaotic dynamics

the bifurcating neuron model. In Sec. 3, we propose an
estimation method of the phase shift value. In Sec. 4, thenumerical simulation is performed to evaluate
performance of our proposed method. Then, wesummarize results.

1. Introduction
In the view point of neural coding, temporal structure of
spike firing is regarded to play important role forrepresentation of neural information in the brain1. In ourprevious research2,3, authors have studied thedecomposition method of the superimposed chaotic spikesequences and selective synchronization for the phaseshift value of the bifurcating neuron. The bifurcatingneuron was introduced by Lee and Farhat4 and exhibits
chaotic inter-spike interval dynamics that depends on thephase shift value of the background oscillation. Therefore,
if we correctly estimate each phase shift values from thesuperimposed chaotic spike sequences, we cancharacterize the dynamics of the bifurcating neuron andidentify each spike sequence from the othersuperimposed spike sequences.
In this study, we propose an estimation method of thephase shift value from superimposed spike sequences
generated by the bifurcating neuron. In Sec. 2, we explain

2. Bifurcating neuron
The bifurcating neuron was introduced by Lee andFarhat4. We define the bifurcating neuron as a form of the
spike response model2,5. Let ( ) be the -th bifurcatingneuron, where = 0,1, ⋯ , − 1 and is the number ofneurons. The dynamics of the -th bifurcating neuron isdefined as follows:
()

()

( )=

+

()

()

( )+
()

()

,

(1)

where
,
,
and
denotes the internal()potential of, the resting potential, the kernel functionof internal dynamics, and the external noise, respectively.
The kernel function ( ) is defined as© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

447

Akihiro Yamaguchi, Yutaka Yamaguti, Masao Kubo
()

( )=

−

()

()

+

( , )=

(2

,

()

(2)

;

+ ),

(3)

is the time constant of the kernel ( ) , and ( ) is
the last firing time of ( ) . The internal potential ( ) is
linearly increasing by the kernel ( ) . The neuron ( )fires when ( ) exceeds the threshold , and then ( ) is
reset to the initial potential given by the back groundoscillation ( , ). The constants , , and ( ) is theamplitude, the frequency, and the phase shift value of
background oscillation, respectively. The bifurcatingneuron exhibits various chaotic dynamics of the interspike interval. The phase of the next firing time isdetermined by the chaotic one dimensional map of the
where

()

phase of the last firing time4. Let
firing time

()

()

.
()

()

=

()

is defined as()

Fig. 1. The bifurcation diagram for the amplitude of the
background oscillation and the Lyapunov exponents of the thephase map of the bifurcating neuron.

be a phase of the(4)

mod 1/ ,(a)andis determined by the chaotic one
anddimensional map such as

and
()

where
;

()

()

= (

()

= (

()

()

= +

;
−

;

()

),

(5)

) mod 1/ ,

(6)

−

= /3

(b)

=

(c)

= 5 /3Fig. 2. Examples of the one dimensional chaotic map of thephase of firing time.

3.1. Superimposition of spike sequences

,

()

.

()

()

()

,
, ⋯ } be a set of firing time
Let the set ( ) = { ,()of the neuron ( ) , whereis the firing time of -th()
spike of. The superimposition of spike sequences
is defined as a set of firing time of the all bifurcatingneuron such that

(7)

The bifurcation diagram of the map of the firing phase(Eq.(6)) and the Lyapunov exponents of this map areshown in Fig. 1.The shape of chaotic map Eq.(6) isdetermined by the phase shift value ( ) . Examples ofthis chaotic map is shown in Fig. 2.

The elements ofif < .

()

=

.

(8)

are denoted { , , , ⋯ } and<3.2. Direct estimation of the phase shift value

3. Phase shift estimation()

For the neuron ( ) , the timeis determined by the()previous firing time
as mentioned in the previous()()
section. If successive firing times
andare given,
we can solve Eq.(5) and directly obtain the phase shift
value ( ) such thatIn this section, we explain our proposed method of phaseshift estimation from the superimposed spike sequences
for two cases. One is the case that the phase shift values
are completely unknown. The other is the case that thecandidates of phase shift values are given in advance andselect the optimal one by the estimation.

()

where

= sin ( ( ,
( ,

()

)=( −

)/

()

)− 2

− (

−

,

(9)

)) .

(10)

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

448

Phase shift estimation of

For the superimposed spike sequences, we have noknowledge about the source neuron of each spike.
Furthermore, the inverse function of sin has two values
within the range from 0 to 2 . Therefore, we estimate
candidates of the phase shift value for each pair of firing
times ( , ) in the superimposition of spike sequences
as follows:( , )=
2 + sin ( ( ,)) − 2mod 2 ,( , )=− sin ( ( ,)) − 2mod 2 ,

and= {( , ⋯,)| ≤,∈ ,| ( ,

)| ≤

}.

(14)

The candidates of the phase shift value is given as
Φ={

,

⋯,

,⋯,

; )=

},

(15)

where is the number of candidates.
The distance between the candidate
and theestimated phase shift value from -tuple of firing times
is defined as

(11)

( ,

(12)

min

,

,

where − /2 ≤ sin (⋅) ≤ /2. From Eq. (5),if the pairof the phase of firing timeand is the successive
()
()
and
, thenfiring timeandsatisfy thecondition
| ( ,

)|

For the specified firing timeof the candidate such as(13)

where noise less dynamics is assumed in Eq.(1). By using
this condition, we could restrict pairs of firing timeandfor the estimation of the phase shift values(Eq.(11) and (12)).
The direct estimation method of the phase shift values
is as follows:(1) Choose all possible pairs of firing times , ∈that satisfy the condition Eq.(13).
(2) Calculate
andfrom chosen pairs ( , ).
(3) Make a histogram of the calculated phase shift
values.
(4) Obtain estimated phase shift values that correspondto local peaks of the calculated histogram.
If the pair of firing times , is the successive firing
time of a certain neuron, one of the phase shift values
andis correct and the other false one widelydistributes within the range from 0 to 2 . Therefore, thefrequency of the correct phase shift values become higher
than the others.

(,( ; )=

min

, )∈

∗(

, Φ) = arg min

,⋯,

( ,

−

(16)

.

∈ , we calculate the score

,⋯,

, ;

).

(17)

Then we obtain the optimal phase shift value ∗ ( , Φ)that gives the minimal score for the given firing time .
The definition of ∗ ( , Φ) is∈( ; ).

(18)

By using this method, it becomes to be possible toidentify spike sequence which generated by the sameneuron from the other superimposed spikes.
4. Numerical experiments
4.1. Experimental setting
We numerically simulate
bifurcating neurons andconstruct the superimposition of chaotic spike sequence(Eq.(8)). The phase shift value of each neuron is assigned
as()3.3. Selection of the optimal phase shift value= 2 ( + 0.5)/( = 0, ⋯ ,

− 1).

(19)

We examine the direct estimation method of the phaseshift value and evaluate a performance of selection
method of optimal one. Numerical simulation was
performed with the following setting: = 100 , =−30 ,= −70,= 21.5 , and= 1 . The
distribution of external noise in Eq. (1) was the Gaussian
distribution with the average = 0 and the variance= 0.3.

In this section, we propose a selection method of the
optimal phase shift value for each spike firing time fromgiven candidates. In order to select the optimal phaseshift value, we evaluate the distance between theestimated phase shift value and candidates for the
successive spike firing times. A set of the possible tuples of spike firing times is defined as© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

449

Akihiro Yamaguchi, Yutaka Yamaguti, Masao Kubo

(a)in Table 1. Here, for each firing time in the superpositionof spike sequences, the source neuron was identified by
the estimated phase shift value. More than 90% of spikes= 4 . The maximal
are correctly identified up tocorrect identification ratios are highlighted by italic boldstyle. Although these results also depends on theamplitude of the external noise, our proposed selection
method based on the successive
firing times iseffective when the number of superimposed spikesequences is increased.

=3

5. Conclusion
(b)

= 6,

(c)

= 12

For the bifurcating neuron, the estimation method of thephase shift value was proposed and the identification
performance of superimposed spike sequences based on
the estimated phase shift value was evaluated. Numericalresults show correct estimation of the phase shift valuesof superimposed spike sequences and good performancefor the identification of them. Since the performance of
estimation and identification depend on the amplitude ofexternal noise, analysis of relation between the noise
amplitude and the performance of estimation is one of
our future works.

Fig. 3. Histograms of the directly estimated phase shift valuesfor (a) = 3, (b) = 6, and (c) = 12.

Table. 1. The ratio of correct identification of the superimposedchaotic spike sequences.
=298.2±0.5%=396.3±0.6%=493.9±0.7%=591.3±0.9%486.1±0.8%90.4±0.8%87.7±1.0%

84.5±1.2%580.4±1.0%87.2±1.0%

86.5±1.3%

83.9±1.6%672.8±1.4%

83.5±0.9%83.4±1.0%

82.1±1.5%954.1±1.3%72.5±1.4%75.0±1.3%75.5±1.4%1241.0±1.8%60.3±1.4%

66.0±1.2%

67.7±1.8%

2391.6±0.8%92.1±0.9%88.9±1.2%

Acknowledgements
This work was partially supported by MEXT KAKENHI
Grant Number 15H05878, 16K00409 and 17H01794.

86.5±1.5%

References
1. Fuji, H., Ito, H., Aihara, K., Ichinose, N., and Tsukada, M.,
Dynamical cell assembly hypothesis – Theoreticalpossibility of spatio-temporal coding in the cortex, Neural
Networks, Vol.9, No.8, pp.1303-1350, 1996．
2. A. Yamaguchi, Y. Yamaguti, and M. Kubo,
Decomposition of superimposed chaotic spike sequencesby using the bifurcating neuron, Advances in cognitive
neurodynamics (VI), Springer, pp.3-9, 2018.
3. A. Yamaguchi, Y. Yamaguti, and M. Kubo, Selective
synchronization of coupled bifurcating neurons for phaseshift of background oscillation, J. Robotics, Networking
and Artificial Life, Vol.5, No.2, pp. 79-82, 2018.
4. Lee G., and Farhat, N.H., The Bifurcating Neuron
Network 1, Neural Networks, Vol. 14, pp. 115-131, 2001.
5. Gerstner, W., and Kistler, W., Spiking Neuron Models:Single Neurons Populations Plasticity, Cambridge
University Press, 2002.

4.2. Experimental results
Results of the direct estimation of the phase shift values
are shown in Fig. 3 for the cases = 3, = 6, and= 12 as examples. For these three cases, histogramsof the estimated phase shift values have significant peaks
that correspond the correct phase shift values. The
valiance of the estimated phase shift values depends on
the amplitude of the external noises. When the varianceof external noise decreases, the variance of the estimatedphase shift value also decreases.
Next we evaluated the performance of selection
method of the optimal phase shift value. The candidates
were given as Φ = { , , ⋯}. Results are shown© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan450

Design of a Data-driven Predictive-PI Controller
Yoichiro Ashida

Graduate School of Engineering, Hiroshima University,1-4-1 Kagamiyama
Higashihiroshima city, Hiroshima, Japan
Shin Wakitani

Graduate School of Engineering, Hiroshima University,1-4-1 Kagamiyama
Higashihiroshima city, Hiroshima, Japan
Toru Yamamoto

Graduate School of Engineering, Hiroshima University,1-4-1 Kagamiyama
Higashihiroshima city, Hiroshima, Japan
E-mail: ashida-yoichiro@hiroshima-u.ac.jp, wakitani@hiroshima-u.ac.jp, yama@hiroshima-u.ac.jp
http://www.cse.hiroshima-u.ac.jp/

Abstract
PID controllers have been widely employed in real processes. However, for processes with long dead time, thecontrol performance obtained by PID controllers are limited. The predictive PI controller can control suchprocesses efficiently and have only three parameters. However, it is desired to determine the parameters
automatically. In this paper, a data-driven design method for a predictive PI controller is proposed. The
effectiveness of the proposed scheme is evaluated by a simulation example.
Keywords: Predictive control, predictive PI controller, data-driven, process control, long dead time.

proposed. Among them, a predictive PI controller2-4 has
only three parameters, and the controller can be tuned
manually like a PID controller. Although a predictive PI
controller can be tuned by "trial and error", it is desired
to determine control parameters automatically. The
authors have proposed a data-driven design method of
PID controllers5. This method can be employed indesigning predictive PI controllers.
In this paper, the properties of a discrete predictive PI
controller are explained. In addition, a data-driven design
method for a predictive PI controller is proposed. In the
proposed method, control parameters of the predictive PI
controller are calculated automatically from one set ofoperation data. The effectiveness of the proposed scheme
is evaluated by a simulation example.

1. Introduction
PID controllers1 have been widely employed in realprocesses. Especially in chemical processes, 90% of thecontrol loops are controlled by PID controllers. However,for processes with long dead time, the control
performance obtained by a PID controller is limited. Thisis because prediction by a derivative element is not very
efficient for such processes. Furthermore, the derivative
part is sometimes switched off because the part amplifies
the influence of noise, and it is not easy to tune thederivative gain suitably. Nevertheless, PID controllers
are main controllers because PID controllers only havethree parameters and can be tuned by "trial and error". To
derive good control performance from the processes withlong dead time, some predictive controllers have been

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

451

Yoichiro Ashida, Shin Wakitani, Toru Yamamoto

(

)=

1+

−1+

{ (

+

2. Predictive PI controller

+

−1 −

)

A control law of a PID controller is written as follow:
( )
( )=
( )+
(1)
+ Δ ( ),
Δ
where ( ) and ( ) denotes a controlled error and input,
respectively. Δ denotes the differencing operator defined, where z is the shift operator which
by Δ ≔ 1 −( ) = ( − 1). The last term is a derivativemeans
element, and it compensate a dead time. However, thederivative does not effective for a long dead time. In
addition, the derivative amplifies high frequency signals
such as sensor noise. Therefore, PI controllers are
sometimes used.
To tackle this problem, a predictive PI controller has
been proposed as follows:
( )
( )=
( )+
Δ
=

Δ

{ ( − 1) − ( −[ ( )+

( )
Δ

− 1)}

+

)

(

=−

)−

,

=

+

1

(

.

)

(6)
(7)

3. Tuning of the control parameters3.1. Design of the predictive PI controller
If the system parameters are known, control parameters
can be determined as discussed above. However, they are
rarely known. In this section, a data-driven design
method is proposed.
E. q. (2) is rewritten as follows:1{ ( − 1) − ( − 1)}( )=Δ ( )+++(2){ ( − 1) − ( − − 1)} + ( ). (10)+Augmented output ( ) is defined as follows:( ) = Δ ( ) + { ( − 1) − ( − 1)}(11)+ { ( − 1) − ( − − 1)} + ( ),where coefficients , , andare defined as follows:
1
≔
, ≔
, ≔
. (12)+++
From e. q. (10) to (12), the following relationship can beobtained:( ) = ( ).
(13)
The evaluation function is defined as follows:+

2.2. Properties of the predictive PI controller
The controlled system is assumed to be the following
stable first order system with a dead time:
(

(

When
is set as= 1 + , the following
relation is obtained because the system is stable:) ( )(1 + )(1 +()=+
1+( + )
(1 + )
(
)
(9)
=
.
1+
Therefore, the predictive PI controller can control a firstorder and dead time system.

1{ ( − 1) − ( − − 1)}], (3)Δ
where denotes a dead time, and a prediction term of thepredictive PI controller is based on a smith predictor.
This controller has only 3 parameters.
+−}

By using these parameters, e. q. (6) can be rewritten as
follows:
( )=
(1 +
) ( )
. (8)
1+
−1+
+
−1

2.1. Definition of the predictive PI controller

+

+

)=

(4)
,
1+
and a reference model is designed as follows:(1 + )
(
)=
(5)
.
1+
The closed-loop transfer function obtained by thepredictive PI controller is shown as e. q. (6). The
influence of the dead time appears at the third and fourthterms of the denomination of e. q. (6). The control
parameters should be determined to remove the influence.
Therefore,
and should be determined as follows:

=

1

() ,

(14)

where denotes the number of data and augmented error( ) is defined as follows:( ) ≔ ( )−( ) ( ).
(15)
By minimizing the evaluation function , the following© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan452

Design of a Data

=

,

=

1−

,

3.2. Extend the design method

=

.

u

y

relationship can be obtained:() ( ) → ( ).
(16)
From e. q. (11), parameters to be optimized are ( =1, 2, 3) . When the minimization has been finished, it
leads the following relationship:( )→( ) ( ).
(17)
Therefore, the reference response is realized by using the
optimized . can be converted to control parameters
using the following equations:
1−
(18)
= , =
,
= ,
(19)

If the system is strictly first order and the dead time is
known, the above discussions are directly applicable.
However, most controlled systems are high order andhave unknown dead times. Then, the design method is
extended in this section.
In the previous design method, the augmented error isdefined as e. q. (15). As a result, a purpose of theminimization of e. q. (14) is exact model matching.
However, sometimes control parameters are calculated
too large when the system is a high order or the dead timeof the controller is incorrect.
To tackle this problem, e. q. (13) is rewritten as
follows:( )≔ ( )−() ( )+( )Δ ( ). (20)
The third term denotes a variation of the input signal. λis a weight of the variation of the input, and it is a userspecified parameter. E. q. (16) is rewritten as follows:( )≔ ( )−( − ) ()( ){ ( − 1) − ( − 1)}−( ){ ( − 1) − ( − − 1)}. (21)−
By using e. q. (21), control parameters are as follows:
1−
=
, =
,
=
, (22)+++

Fig. 1. Control result using PID and predictive PI controllers.

relationships are obtained by substituting e. q. (18) to theterms of e. q. (6):(24)
− ( + )=0⇔
− = 0,
(25)
+
=0⟺
−
= 0.
When e. q. (22) is substituted instead of e. q. (18), theterms are rewritten as follows:(26)− ( + )=,

From e. q. (24) to (27), the third and fourth terms of thedenomination of e. q. (6) are zero regardless of the valueof λ.
Hence, λ does not affect the performance to compensate
for the influence of the dead time.
4.

Simulation example

In this section, a simulation example is shown by using
the following controlled system.
18 + 27( )=(28).
10 + 30 + 100 + 1
The system was discretized by sampling time = 1 s,
and the following discrete system is obtained:
0.38+ 0.04− 0.02()=. (29)
1 − 0.57− 0.37− 0.05
A Gaussian white noise with zero means and 0.1
variance was added in the simulation.
The control result is shown as Fig. 1. There are two
results using PID control with Chien, Hrones, andReswick method and the proposed control. An initial
operating data is required to apply the proposed method.
The operating data obtained by PID control was used as
the initial data.

(23)
, =
,
=
.
+
1−
Normally, the weight λ is set as λ ≥ 0. From e. q. (23),affects only . This means that by tuning λ, an entire
gain of a controller is tuned. When the system is a highorder or dead time is unknown, λ should be large value
and a conservative controller should be designed.
The influence of λ for dead time compensation isconsidered below. The third and fourth terms of thedenomination of e. q. (6) must be zero. By using e. q.
(18), the terms become zero, and the following=© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

453

Yoichiro Ashida, Shin Wakitani, Toru Yamamoto=0= 10
Reference trajectory

100

60

Minimized J

y

80

40
20
0

0

200

400

600

800

1000

1200

1400

1600

1800

2000

40
=0
= 10

u

20

Fig. 3. The minimized evaluation values corresponding to thedead time of the predictive PI controller when λ = 0.

0

-20

0

200

400

600

800

1000

1200

1400

1600

1800

At last, the minimized evaluation values corresponding
to the dead time of the predictive PI controller when λ =0 are shown as Fig. 3.
The minimum evaluation value is calculated when deadtime is 100 steps, and 100 steps are the same as the true
dead time of the system. Then, the dead time can be
determined based on the evaluation function. However,in the actual systems, it is considered that exact dead timecannot be calculated because of various reasons likenonlinearity. Therefore, it is recommended that λ is set as
a nonzero value.

2000

Time [step]

Fig. 2. Control result of predictive PI controller designed as λ =0, and λ = 10.

In the proposed method, the reference model was madeas
= −0.98 with known dead time, and λ = 0. The
optimization method is fminserch function of MATLAB
R2017b. Control parameters were as follows:
(30)
PID:
= 0.02, = 0.0002,= 1.11,= 0.07, = 0.0007,
Predictive PI:= 0.02. (31)
By using the proposed predictive PI control, the moreaggressive response was realized than PID control andtracked the reference trajectory. In addition, thepredictive PI controller did not amplify the influence ofnoise in contrast to the PID control. This is because there
are no derivative elements in the predictive PI controller.
Next, the dead time used in designing predictive PI
controller was set as 85. The other parameters were set as
the same as the previous result, and the initial data wasalso the same. The control results by setting λ = 0, and
λ = 10 are shown as Fig. 2. When λ = 0, the closed looptransfer function became unstable. In contrast, when λ =10, the transfer function was stable while the control
performance was worse than the reference model.
The calculated control parameters were as follows:
λ = 0:= 0.35, = 0.003,= 0.14, (32)λ = 10:= 0.08, = 0.0008,= 0.03. (33)
The parameters are rewritten to the form of e. q. (19) as
follows:(34)
λ = 0: = 0.35, = 102.5,= 0.39,λ = 10: = 0.07, = 102.5,= 0.39, (35)
It is clear that only
was varied by changing λ. This
result confirms the discussion at 3.2.

5. Conclusions
In this paper, a consideration of a predictive PI
controller is presented, and a data-driven design method
is proposed. The controller is effective for systems withlong dead time. The effectiveness of the proposed
method is confirmed by a simulation example.
References
1. K. J. Åström and T. Hägglund, Advanced PID Control,(International Society of Automation, North Carolina,2006).
2. T. Hagglund, A predictive PI controller for processes withlong dead times, IEEE Control Systems Magazine 12(1)(1992) 57–60.
3. T. Hagglund, An industrial dead-time compensating PI
controller, Control Engineering Practice 4(6) (1996) 749–
756.
4. P. Airikka, Robust Predictive PI Controller Tuning, IFAC
Proceedings Volumes 47(3) (2014) 9301–9306.
5. Yoichiro Ashida, Kayoko Hayashi, Shin Wakitani, andToru Yamamoto, A novel approach in designing PID
controllers using closed-loop data, Proc. of the American
Control Conference 2016 (2016) 5308–5313.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan454

Improvement in Intersample Response of Multirate Optimal Control
Takao Sato, Natsuki Kawaguchi, Nozomu Araki, Yasuo Konishi

Graduate School of Engineering, University of Hyogo
2167, Shosha, Himeji, Hyogo, 671-2280, Japan

Abstract
In this study, the dual-rate sampled-data control system is designed, where the sampling interval of a plant output isan integer multiple of the holding interval of a control input. Because the discrete-time linear quadratic integrator(LQI) is used for optimizing the discrete-time control performance, intersample output might oscillate. In order toeliminate the oscillation of the intersample ripple caused by the input oscillation, the designed LQI controller isextended. Finally, the original LQI control law and the extended control laws are quantitatively compared.
Keywords: Sampled-data control system, dual-rate system, intersample response, discrete-time response, null-space,redundancy.

1. Introduction

2. Controlled Plant ModelIn the sampled-data control system1, the continuous-time
plant is controlled using the discrete-time controller. In
the single-input/single-output system, there are twointervals: the holding interval of a control input and thesampling interval of a plant output. This study discusses
a design method for controlling a dual-rate control
system, where the sampling interval is longer than theholding interval.
In such dual-rate control systems, intersample ripples
might occur even if the sampled output converges to thereference input because the control input can be changed
between the sampling intervals2. Therefore, the condition
for eliminating the intersample ripples caused by the
input oscillation has been proposed3.
This study proposes two design methods for eliminating
the intersample ripples: one is an extension method basedon the null-space4, and the other is to weight on the input
deviation.
Finally, the effectiveness of the proposed method isdemonstrated through numerical examples, and thecontrol performances are quantitatively evaluated basedon the numerical results.

A controlled plant is an unknown linear continuous-timesingle-input single-output time-invariant system given as
follows:
̇( ) =

( )+

( )=

( )

( )

(1)
(2)

is the state variable, ( ) ∈ is thewhere ( ) ∈control input, and ( ) ∈is the plant output,
respectively.
∈ × is a coefficient matrix, and∈and∈are coefficient vectors. Because thesampled-data control system is designed, a continuoustime plant output is sampled, and a discrete-time control
input is held. Since the sampling interval of the plant
output is an integer multiple of the holding interval of thecontrol input, the following lifted dual-rate model isobtained.
( + )=( )=[ ( )

⋯

( )=

( )+

( )

( + − 1)]

( )© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

455

(3)
(4)
(5)

Takao Sato, Natsuki Kawaguchi, Nozomu Araki, Yasuo Konishi( )=−where is a positive integer. The dual-rate model isassumed to be known, and a sampled-data control system
is designed based on this model. It is assumed that( , ) is stable.

( )+

( )

(17)

) ( )+

( )+

( )
(18)

where ( ) ∈is a newly introduced vector.
The closed-loop system using the extended control law(11) is obtained as follows:( + )=(

3. Optimal Controller Design

The control system is designed using the following state
feedback control law:−

(8)

From the definition of , the last-term in the right-handside of Eq. (18) is ( )× . As a result, the closed-loopsystem is not affected by ( ). In this study, ( ) isdesigned so that all the elements in ( ) are equivalent.

where ( ) ∈ is the reference input to be followed by
the plant output, and denotes the sampling interval of
the plant output. ∈ ×( ) is the feedback gain, and
it is designed using the linear quadratic integrator (LQI)5:

In the other extension method, the deviation of thecontrol input is suppressed in LQI. To this end, thedifference between ( + ) and ( + + 1) ispenalized as follows:

( )=−

( )

( )=[ ( )( + )=

=∑

( )]

( )+

( )

(6)
(7)

( )− ( )( )+ ( )( )4.2. Input-weighting method(9)( + − 1) − ( + )where ∈ ( )×( ) is a positive definite matrix,and ∈ × is a positive semi-definite matrix. The
performance index is rewritten as follows:
=∑

=

+

(10)

( )

=∑

[ ( + − 1)

( )

( )

(12)

The closed-loop system using control law (6) isobtained as follows:( + )=(

=

−

0
,
1

−

=

) ( )+
×

,

=

( )

=∑

∑

(14)

To eliminate the intersample ripples caused by the input
oscillation between the sampling instants, two extension
methods are proposed.

( )=Using the defined
follows:

∈

)=
|

( )

=

, ,

≠

(20)

( ) ( + Δ) ( )(21)( + − 1) − ( + )= ( ) Δ ( )(22)

5. Discussion on Discrete-time and Intersample
Control Performance

4.1. Null-space method

(

−1
1

In the proposed weighting method, must be redesigned.
Therefore, the Riccati equation is resolved by replacing
Eq. (12) by Eq. (21).

4. Two Intersample Suppression Methods

One extension method is based on the null-space.
∈ ×( ) is defined as follows:1−1where Δ is a matrix which satisfies the next equation:

(13)
×

Δ =

(19)

where
∈ is a non-negative weighting factor. To
evaluate the weighted input deviation in LQI, in Eq.
(12) is replaced with the following criterion:

(11)

( )

( + )]Δ=( + − 1)( + )In the dual-rate sampled-data control system, thediscrete-time model is available, and the continuoustime model is not always obtained. In such a case, when
the discrete-time performance is optimized, theintersample ripples might be caused by the input
deviation between the sampling instants, although thediscrete-time performance is optimized.
In this study, the discrete-time performance is optimized
based on the known lifted dual-rate model, and thedesigned control law is extended so that the intersample

(15)
(16)

, the control law is extended as© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

456

Improvement in Intersample Response

performance is improved. In the discrete-time
performance evaluation, because the discrete-time
performance function consists of and , two indices
are quantitatively evaluated.
In the first extension method based on the null-space,
the intersample ripples can be eliminated by introducing
the new design parameter. Further, the parameter isdesigned independent of the discrete-time closed-loopsystem. Therefore, is maintained and is changed.
Hence, the optimization on the control input is not
maintained
In the second extension method, since the difference ofthe control input is always penalized, the input deviation
is suppressed. As a result, the intersample ripples causedby the input deviation is eliminated. However, bothand are affected by the weighting factor .

amplitude of the difference in the control input is
attenuated since the control input deviation between thesampling instants is penalized. The larger , thesmaller the amplitude of intersample ripples.
For comparing the controllers, the performance indices
are quantitatively evaluated in Table 1, where theevaluation length is 50 steps. Table 1 shows that theindexof the original LQI is best, and that of theextended methods are increased, although theintersample ripples are suppressed. In the index of state,, the original LQI is best and it is the same as theextended method of 4.1 because the original closed-loopsystem is maintained in the extended controller
designed in 4.1.
Controller
Original
4.1
4.2( = 10 )4.2( = 10 )6. Numerical Example
Consider a continuous-time system given as follows:
( )=

(23)

The sampling interval and the holding interval are 2s
and 1s, respectively, and the following lifted dual-rate
system is given:( + 2) = 0.51 ( ) +

0.72
1.0

( ) = 0.28 ( )( )Table 1. Index values(× 10 )
3.029
93.82
3.029
96.38
3.034
95.49
3.037
96.35

3.122
3.125
3.130
3.133

7. Conclusion
This study has discussed the design methods forcontrolling a dual-rate sampled-data control system.
Firstly, an optimal control law is designed using LQI.
Secondly, the designed control law is extended using
two methods: one is based on the null-space, and theother is to weight the input deviation between thesampling instants. Finally, the effectiveness of the
proposed method is demonstrated through the numericalexamples, and the control performances arequantitatively evaluated.

(24)
(25)

The feedback gain is obtained by solving the LQI
problem with = = . The reference input is a unit
step function.
The control result using the original feedback control
law (6) is shown in Fig. 1. This figure shows that the
sampled output converges to the reference input at the
sampling instants. However, the control input oscillates,
and the intersample output hence oscillates. In the LQI
design, the amount of the control input is evaluated, but
the intersample output is not evaluated. Therefore, thedeviation of the control input is not attenuated, and theintersample ripples occur.
To eliminate the intersample ripples caused by the input
deviation, the extended controllers are applied. The
result using the extended controller designed in 4.1 isshown in Fig. 2. The lower figure shows that the control
input is constant between the sampling instants, and theupper figure shows that the intersample ripples are
eliminated. Furthermore, the sampled output response is
the same as Fig. 1 even though the control input isredesigned.
The control results using the extended controller
designed in 4.2 are shown in Fig. 3 and Fig. 4, whereis set to 10 and 10 , respectively. In both figures, theAcknowledgements
This study was supported by JSPS KAKENHI Grant
Number 16K06425.
References
1. T. Chen and B. Francis, Optimal Sampled-Data Control
Systems (Springer-Verlag, London, 1995).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan457

Takao Sato, Natsuki Kawaguchi, Nozomu Araki, Yasuo Konishi

2. A.K. Tangirala, D. Li, R.S. Patwardhan, S.L. Shah and T.
Chen, Issues in Multirate Process Control, in Proc. of theAmerican Control Conference (1999), pp.2771–2775.
3. A.K. Tangirala, D. Li, R.S. Patwardhan, S.L. Shah and T.
Chen, Ripple-Free Conditions for Lifted Multirate Control
Systems, Automatica, 37(10) (2001), pp.1637–1645.
4. L. Zaccarian, Dynamic Allocation for Input RedundantControl Systems, Automatica, 45 (2009), pp.1431–1438.
5. P. C. Young and J. C. Willems, An approach to the linear
multivariable servomechanism problem, Int. J. Control
15(5) (1972), pp.961–979.

Fig. 3. Simulation result using the extended
method given 4.2 with δ = 10

Fig. 1. Simulation result using the original state
feedback control law (upper: output trajectory,
lower: input trajectory)

Fig. 4. Simulation result using the extended
method given 4.2 with δ = 10

Fig. 2. Simulation result using the extended
method given 4.1© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

458

Design of a PID Controller using a Fictitious Exogenous Signal
for a Fluctuation System
Masatoshi Kozui

Graduate School of Engineering, Hiroshima University
1-4-1, Kagamiyama, ,Higashi-Hiroshima, Hiroshima, Japan
E-mail: kozuim@hiroshima-u.ac.jp
Takuya Kinoshita

Graduate School of Engineering, Hiroshima University
1-4-1, Kagamiyama, ,Higashi-Hiroshima, Hiroshima, Japan
E-mail: takuya-kinoshita@hiroshima-u.ac.jp
Toru Yamamoto

Graduate School of Engineering, Hiroshima University
1-4-1, Kagamiyama, ,Higashi-Hiroshima, Hiroshima, Japan
E-mail: yama@hiroshima-u.ac.jp

Abstract
It is possible to design PID controller to achieve the desired behavior of industrial machines by the FRIT. However,in the steady state, if the system characteristics are changed, the desired control performance can not be obtained by
the FRIT. In this paper, it is assumed that a disturbance is input when system characteristics are changed. A PID
controller used the Fictitious Exogenous Signal calculated by the operational data is proposed. The effect has beenverified by numerical example.
Keywords: PID Control, Fictitious Exogenous Signal, Fictitious Reference Iterative Tuning

obtained. Recently, the Fictitious Reference Iterative
Tuning (FRIT) method which calculates PID parameters
directly using the closed loop data obtained by a set ofoperational data attracts attention.1 The method can beapplied, even when the characteristic of the system are
unknown. The fictitious reference input is calculated
from acquired system input and output, and input toreference model. It is a method of designing an optimum
controller by adjusting the PID parameters so that theoutput obtained from the reference model is closer to the
output of the experiment data. However, in manyindustrial machines, especially the system characteristics
of machines operated by human are often changed as the

1. Introduction
A lot of industrial machines are controlled by a PID
control in order to obtain the desired performance. In
particular, the response of the system output is important.
It is necessary to set appropriate PID parameters for agood characteristic. However, it is difficult to decidethese parameters with trial and error. Therefore, the CHR
method can be mentioned as a conventional method for
determining these parameters. In this method, theproportional gain, the integral gain and the differential
gain are respectively determined based on the systemparameters to be controlled. However, this method can beapplied only when the system parameters is known, anddesired response characteristics can not necessarily be© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

459

Masatoshi Kozui, Takuya Kinoshita, Toru Yamamoto

2.2. Control law
The controllers for system1 and system2 use thefollowing I-PD controller.
∆ ( )=( )−∆ ( )−

∆

( )

(1)

( ), ( ) and ( ) are control input, control error andsystem output.
,
andare proportional gain,
integral gain and differential gain, respectively. The. Furthermore, thedifference operator is ∆≔ 1 −control error ( ) is calculated as following equation.
( ) is reference.

Fig.1 Switching system( )= ( )− ( )(2)2.2.1. FRIT method

Fig.2 Block diagram of FRIT.

The controller1 for system1 is set as,
andby
the FRIT method. The FRIT method can be calculated
the controller parameters directly from one experimental
data, and block diagram is shown in the Fig.2. ( ) and( ) are system input and system output in closed loop().
data. ( ) is the output of the reference model̃ ( ) indicates the fictitious reference input to the().
( ) andreference model̃ ( ) aredefined as the following equations.

operating environment changes. When such system
fluctuation occurs, it is necessary to change the controller
before and after the system fluctuation.
In this paper, it is assumed that the system to becontrolled is switched in a steady state, and the controller
is constituted by a PID controller. PID parameters before
system switching are set by the FRIT method. Aftersystem switching in the steady state, the method setting
PID parameters by the Fictitious Exogenous Signal is
proposed. Specifically, it is regarded system fluctuationin the steady state as disturbance like impulse virtually
inputted. At this time, the transfer function from thedisturbance to the control error is constructed. The
optimum controller is designed by adjusting PID
parameters so that the characteristic of the closed loop
system approaches a reference model. Genetic algorithm
was used to calculate PID parameters. Moreover, theeffectivity of the proposed method has been verified bynumerical examples.

( )=

and

(

̃( ) =(=As shown in Fig.1, the control objective is a switchingsystem in which the system changing occurs in the steadystate. Initial state is system1, and System2 is afterswitching. It is assumed that the switching of the systemoccurs at the known time. Controllers for these systems
are Controller1 and Controller2, respectively. PID
parameters are set for these controllers.

∆

( )

(

∆

( )
)

̃( )

( )

∆

+

(3)
(4)
( )

(5)

are designed by the following equations.2=

2.1. Control Objective

)=1+

( )+

= −2

2. Control System) ̃( ) =

−

−

cos

= 0.25(1 − ) + 0.51

(6)

σ is the rising characteristic of control, δ is the parameter
related to the attenuation characteristic, and can bearbitrarily set by a designer.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan460

Design of a PID

N1 and N2 indicate the total number of steps ofexperimental data in the system1 and the system2. In thesystem1, a desired response characteristic can beobtained by defining the difference between the reference
model output ( )and ( ) as an evaluation function
and setting the controller parameters so as to minimize
the evaluation value. When the evaluation value is low,
the fitness is high and PID parameters is better. The
system2 is also the same. First, initial individuals with, andas genes are randomly generated. Next,
initial evaluation is performed, and elite selection,
crossover, tournament selection, and mutation are
performed based on the evaluation values. These
procedures are repeated, and PID parameters are obtained
as the gene of the individual with the highest fitness inthe final generation. In this paper, the GA was calculatedwith 200 individuals and 200 generations.

Fig.3 Block diagram of Fictitious Exogenous Signal.

2.2.2. Fictitious Exogenous Signal method,
andby
The controller2 for system2 is set asa method using the Fictitious Exogenous signal. The
proposed method can be calculated the controller
parameters directly from a set of experimental data
similar to the FRIT method, and block diagram is shownin the Fig.3. As shown in Fig.3, the reference model() is constructed by the transfer characteristicfrom disturbance to control error when ( ) is the inputand ( ) is the output. The ( ) is a Fictitious
Exogenous signal, and the output of the reference model() when this is input is ̃ ( ) . The Fictitious
Exogenous signal ( ) and the reference model() can be expressed by the following equations.3( )=̃( ) =( )−( − 1) +∆ ( )−

( )=−

(

( )−

( )

∆

( )
)

∆

( )

3. Numerical Example

In order to verify the effect of the proposed method,
the switching system of the first order lag systems
were set the following equations and a simulation iscarried out.
【System 1】【System 2】

(7)

(11)

=

(12)

3.1. Conventional method (CHR method)(8)

Above equations are discretized at the sampling time
= 1 [sec], and following equations are obtained.
Furthermore, in 250 step, the system1 is switched to thesystem2. The effect of control in responsiveness has beenverified. Reference is ( ) = 10.

The setting method of ( ) is the same as the FRIT
method, and parameters different from system1 can be
set for σ and δ.
2.3. Genetic Algorithm( ) = 0.980 ( − 1) + 0.0990 ( − 3)

The genetic algorithm(GA) was used to minimize theevaluation function value.4,5 The evaluation function isdefined by the following equations.

( ) = 0.984 ( − 1) + 0.0165 ( − 3)= ∑( )− ( )(9)= ∑( ) − ̃( )

(10)

(13)
(14)

First, a simulation was performed on the system1 with
the PID gain calculated by the CHR method. The result
was shown in Fig.4, and the set PID parameters wereshown in Table 1.
As shown in Fig.4, it took long time to follow the target
value, and the responsiveness of the system1 is no good.
The output variation after switching to the system2 was【System 1】【System 2】=© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

461

Masatoshi Kozui, Takuya Kinoshita, Toru Yamamoto

Table 2. PID gain tuning by the FRIT and
Fictitious Exogenous Signal.

Table 1. PID gain calculated by CHR method.

System1&2

Kp

Ki

Kd

3

0.06

3

Kp

Ki

Kd

System1

1.22

0.05

1.21

System2

15.9

0.60

20

Fig.4 Simulation result by using CHR method.

large and recovery time is long, the control performance
is no good.

Fig.5 Simulation result by using the FRIT and
Fictitious Exogenous Signal.

3.2. Proposed method

Exogenous Signal calculated from one experimental data
is input to the reference model. A controller with a good
disturbance responsiveness can be obtained by adjusting
the PID parameters as approximating the reference model
output to the control error ( ).
In the future, the proposed method will apply to the actualmachine and confirm the effect.

Next, the result of the proposed method using the FRIT
method for the system1 and the method of controller
adjustment with the Fictitious Exogenous Signal for thesystem2 is shown in Fig.5. The control result by the CHR
method was used as the operation data ( ) and ( ).
Table 2 shows the set PID parameters. As shown in Fig.5,
the response of the system1 is faster than theconventional method, and the desired output is obtained.
In addition, the output variation after switching to thesystem2 is less than 1/4 as compared with theconventional method, and good control performance isobtained.
It was confirmed that responsiveness after switching of
the system is improved by considering the system
fluctuation as disturbance input and applying theadjustment law using the Fictitious Exogenous Signal.
4.

Reference
1. O. Kaneko, K. Yoshida, K. Matsumoto and T. Fujii, A
New Parameter Tuning for Controllers Based on LeastSquares Method by using One-Shot Closed Loop
Experimental Data: An Extension of Fictitious Reference
Iterative Tuning, in ISCIE Journal, 18(11) (2005), pp.400409.
2. K. Hayashi and T. Yamamoto, Evolutionary Design of a
PID Controller Using Closed-loop Data, Trans. of IEEJ C,
131(4) (2011), pp.794-799.
3. S. Masuda and K. Takeda, An FRIT Method forDisturbance Attenuation Using Input-Output Data
Generated from Disturbance Responses, Trans. of IEEJ C,
131(4) (2011), pp.788-793.
4. D.E.Goldberg, Genetic Algorithms in search, optimization,and machine learning (Addison-Wesley Publishing
Company, 1989)
5. K. Kawada, H. Sogo, T. Yamamoto and Y. Mada, Robust
PD Sway Control of a Lifted for a Crane Using a Genetic
Algorithm, Trans. of IEEJ D, 123(10) (2003), pp.10971103.

Conclusion

In this paper, a method of controller adjustment law using
the Fictitious Exogenous Signal for a system thatswitched characteristics in steady state has been proposed.
The system fluctuation is regarded as the input of the
disturbance, and the transfer function from thedisturbance ( ) to the control error ( ) is configured( ) . The Fictitious
as the reference model© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

462

Izhikevich Model Based Self-Repairing Control for Plants with
Sensor Failures and Disturbances
Masanori Takahashi

Department of Electrical Engineering and Computer Science, Tokai University
9-1-1 Toroku, Higashi-ku, Kumamoto,862-865, Japan
E-mail:masataka@ktmail.tokai-u.jp

Abstract
In the previous works, several types of the self-repairing control systems (SRCS) have been developed against
unknown sensor failures. The SRCS can automatically detect the failure, and replace the failed sensor with the healthy
backup so as to maintain the system stability. This paper presents a new SRCS, whose fault detector is constructedbased on a spiking neuron model proposed by E. M. Izhikevich. The sensor failure induces the spikes in the detection
filter. Hence, only counting up the number of spikes makes it possible to find the sensor failure. Also, in this paper,
the robustness with respect to disturbances is theoretically analyzed, and it is shown that self-repairing control can beaccomplished in the presence of unknown disturbances.
Keywords: Self-repairing control, sensor failure, fault detection, dynamic redundancy, spiking neuron model.

stability, which claims that control systems should beconstructed by stable elements3.

1. Introduction
Stabilities of control systems are guaranteed under thenatural assumption that the feedback loops are healthy.
Obviously, if just one loop has the broken sensor, thenthe control system would lose its stability. Hence, failure
detection for sensors plays an important role inmaintaining the control system. In the previous works,
several types of the self-repairing control systems(SRCS) have been developed against unknown sensor
failures1,2. The SRCS can automatically detect the failure,
and replace the failed sensor with the healthy backup so
as to recover the system stability. Compared withexisting active fault tolerant controls, the SRCS has thefollowing advantages: (1) the structure of the detection
filter can be simple independently of the mathematicalmodel of the plant, and (2) the maximum time ofdetection can be specified arbitrarily in advance, that is,
early fault detection can be accomplished. However, inorder to detect failures, an unstable detection filter has
been utilized1, but this is contrary to the idea of the strong

In this paper, for the SRCS against sensor failures, a new
design method for the detection filter is presented basedon a simple spiking neuron model by E. M. Izhikevich4,
and also a concrete detection of failures by counting thenumber of spiking waves from the filter is shown. Thismethod is solid for strong stability concept, because
boundedness of the filtered signal is always guaranteed.
Furthermore, this paper shows the high-gain feedback
controller stabilizing both the plant and the abovementioned detection filter. It is shown that the overall
control system could have robustness with respect todisturbances in spite of its simple structure.
Throughout this paper, with ∈ ℝ, we define the ‘sgn’
function by1 ( ≥ 0)sgn[ ] =−1 ( < 0)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

463

Masanori Takahashi

Notice that this is slightly different from the ordinary one.

This result shows the ‘bursting’ pattern, which will be
exploited in fault detection of the proposed SRCS.

2. Spiking Neuron Model

3. Problem Statement

The spiking neuron model proposed by E. M. Izhikevich4
is represented as,̇ = 0.04 + 5 + 140 − +̇ = ( − )(1)←if
≥ 30 [mV] then(2)
← +where ∈ ℝ is the membrane potential of the neuron,∈ ℝ is the recovery variable, ∈ ℝ is the stimulus,and ∈ ℝ and ∈ ℝ are the parameters for the recovery
dynamics. The auxiliary after-spike resetting is expressedby (2), where∈ ℝ and∈ ℝ are the reset values.

Consider the following linear time invariant system withunknown disturbances.
+++Σ ∶ ̇=
̇ =++(3)
where ∈ ℝ is the actual output, ∶ ℝ → ℝ is theis the state. Also,∈ℝ
control input, and ∈ ℝand∈ℝ
are unknown but bounded disturbances.
Here, it is assume that the high frequency gain ∈ ℝ is
positive, and that
∈ ℝ( )×( ) is the stable matrix(i.e., all eigenvalues lie in ℂ ).
For measurement of the output , the two sensors areprepared. One is the primary sensor #1, and the other isthe backup #2 for occasion of failure. Then, the feedback
signal ∶ ℝ → ℝ can be expressed as follows.
( ) ( ≤ )( )=(4)( ) ( > )where ∈ ℝ is the detection time, whose detail will be
discussed later. Each ∈ ℝ, ∈ {1, 2} is the output of
the sensor # . Obviously, the healthy output is = .
Based on dynamic redundancy (4), we usually use theprimary sensor #1, but switch to the backup when the
failure of the primary one is detected.

Fig. 1 shows the numerical example for the parameters= 0.02, = 0.2,
= −50,= 2, the initial values(0) = −50, (0) = 2 and the stimulus = 10 [mV].
In the figures, the point ‘A’ indicates the first spike in the
bursting pattern, and the red two lines are nullclines.

The failure scenario to be consider here, is expressed asfollows.
( )= , ≥(5)
where ∈ ℝ is the unknown failure time, and ∈ ℝ
is the unknown stuck value.
The problem is to design the SRCS, which can replace
the failed sensor with the backup to maintain the control
stability and guarantee the convergence property of :lim sup| ( )| ≤(6)→

for arbitrarily given∈ℝ .

4. Control System DesignFirst of all, the detection filter is introduced based on the
spiking neuron model (1) and (2).
)+Σ ∶ ̇ = sgn[ ](+ −)+ ( ̇ +

Fig. 1. A ‘bursting’ pattern by Izhikevich spiking neuron model:
the time history (top) and the trajectory ( , ) of the −plane (bottom).

̇ = ( sgn[ ] +− )© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

464

(7)

Izhikevich model based self-repairing← sgn[ ](8)← +where the parameters,∈ ℝ, ∈ ℝ, ∈ ℝ,∈ℝ, ∈ ℝ, ∈ ℝ and ∈ ℝ are positive constants.
Also,∈ ℝ is the threshold for auxiliary resetting. For
stability, the resetting rule (8) should be invalid before
the steady state.

1
+ ‖ ‖
2
Solving the differential inequality (13), we have

Next, the high-gain feedback controller is designed byΣ ∶ =− ( + )−−(9)
where ∈ ℝ is the feedback gain to stabilize the plant
and the detection filter.

lim sup| ( )| ≤ lim sup 2 ( ) ≤ifsgn[ ] ≥

then

= 2+

( ) ≤ (0)

+

,

∈ [0,

)

(14)
(15)

Therefore, we can conclude that all the signals in thecontrol system are bounded although the disturbances
exist. Moreover, if no sensor fails, that is, = ∞, then
→

→

2

(16)

For sufficiently small , we have 2 / < . Thismeans that the inequality (6) holds if no failure occurs.
The overall control system is illustrated in Fig. 2.

Then, we have the following representation of the overall
control system with the healthy sensor, i.e., = .
̇ = −( − ) −
−
−
+
+
̇=
+
+
̇ = −(− ) + sgn[ ]−)++sgn[ ]( −++̇ = ( sgn[ ] + − )(10)Here, consider the positive definite function ∶ ℝ →
ℝ as,
1
1
:=
+
+
+
(11)
2
where ∈ ℝ(
which satisfies
definite ∈ ℝ(
gives
1
̇≤−
2
+

where

)×(

2+

)×(

+

)

+

is the positive definite matrix= −2 for any positive). Taking the time derivative of
‖ ‖ +

1
+ ‖ ‖
2

=

1
−2 −‖ ‖ − ‖

=

−

=2

[ ]−3− ̅−−

1−1 − ‖ ‖ −

4

+
‖ −

1
2
−

1
̅

Fig. 2. The block diagram of the proposed SRCS with the
Izhikevich spiking neuron model.

(12)

5. Fault Detection
This section shows the concrete detection method byusing the detection filter by (7) and (8).
Refer to the ‘bursting’ as shown in Section 2. Roughly
speaking, when the two nullclines in (1) run apart fromeach other by the large stimulus , the bursting pattern
appears. Then, suppose that the parameters in (7) arechosen so that the following two nullclines run with asufficient distance each other.
0=
+
+ −
0=
+ −
(17)

− ̅

( ) , ̅ ≥ ‖ ( )‖. Choosing sufficientlyand ̅ ≥
large , we have all > 0, = 1,2,3. Hence, from (12),it follows that
̇( ) ≤ − ( ) + ,∈ [0, )
(13)
where
1
= min
,
,
,
[ ]
2

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

465

Masanori Takahashi

If the sensor fails, then the bursting pattern appears justlike Fig. 1. Such a pattern does not occur as far as thesensor is healthy. It does not depend on the disturbances.
Therefore, counting the number of the spikes in the
bursting pattern, makes it possible to find the sensor
failures. Thus, the detection time is defined by}≔ min{ | ( ) ≥(18)
where ∈ ℕ is the counted number of the spikes in the∈ ℕ isfiltered signal from the detection filter, andthe specified minimum number of spikes.

early fault detection can be achieved by using the spiking
neuron model.

After replacing the failed sensor, the boundedness of all
the signals in the control system are guaranteed again,
and the convergence (6) can be obtained.
6. Numerical Examples
To confirm the effectiveness of the proposed method, thenumerical simulation is explored.
Consider the following plant with disturbances.
̇ = − + + + 0.5 + sin(0.01 )̇ = −2 + + cos(0.1 )The initial values are (0) = 1 and (0) = −1.

The failure scenario (5) is supposed that= 25 [ ], = ( )Of course, these are assumed to be unknown.

(19)

Fig. 3. Simulation results: the measured output and the actual
output (top) and the filtered signal (bottom).

7. Conclusions(20)

In this paper, a new SRCS has been developed that has
the detection filter based on the spiking neuron model by
E. M. Izhikevich. In this method, the sensor failure canbe found by counting the spikes from the filter. The
applications to nonlinear systems with noise, MIMO
systems and so on are still left in the future works.

For the above plant, the parameters for the detection filter
are selected as follows.
= −0.06, = −0.6, = 4, = = 1= 0.02, = −6, = 20
Also, the parameters for resetting are= 1,= 0.2,=2
The specified number of the spikes in the bursting patternis supposed to be five, that is,= 5 within almost 1 [s].
In order to stabilize the plant and the detection filter
mentioned above, the controller gain is chosen as= 30

Acknowledgements
This work is supported by JSPS KAKENHI Grant
Number JP16K06429.
References
1. M. Takahashi: Self-repairing control against unknown
sensor failures and biases, Asian J. Control, Vol. 15, Issue4, pp. 1215-1223 (2013).
2. M. Takahashi: Self-repairing PI/PID control against
sensor failures, Int. J. Innovative Computing, Information
and Control, Vol. 12, No. 1, Feb. pp. 193-202 (2016)
3. A. Yanou, M. Minami, and T. Matsuno: “Strong Stability
Rate for Control Systems Using Coprime Factorization”,
Trans. of the Society of Instrument and Control Engineers,
Vol.50, No.5, pp.441–443 (2014) (in Japanese)
4. E. M. Izhikevich: Simple model of spiking neurons, IEEE
Trans. on Neural Networks, Vol. 14, No. 6, pp. 1596-1572(2003)The simulation results are shown in Fig. 3. In this figure,the measured output , the actual output (top) and thefiltered signal (bottom) are shown. From this result, it
is clear that the control system can be well stabilized inspite of existence of disturbances, and the actual output
converges to a very small ball before and after the
failure. The self-repairing control can be accomplished,
and the failed sensor is replaced at ≅ 26 [s] that is,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

466

Design of a Data-Driven Controller with Evaluating Controller Performance
Takuya Kinoshita

Graduate School of Engineering, Hirosihma University,
1-4-1, Kagamiyama, Higashihiroshima city, Hiroshima, Japan
Toru Yamamoto

Graduate School of Engineering, Hirosihma University,
1-4-1, Kagamiyama, Higashihiroshima city, Hiroshima, Japan
E-mail: takuya-kinoshita@hiroshima-u.ac.jp, yama@hiroshima-u.ac.jp
http://www.hiroshima-u.ac.jp

Abstract
Data-driven controller systems have been proposed to achieve the desired control performance without using any
system identifications. The effectiveness of these control schemes has been shown through experimental results. For
time-variant and nonlinear systems, it is important to evaluate the controller performance and redesign controller
when the performance is poor. According to the proposed scheme, the controller performance calculator andcontroller design are integrated using only input and output data.
Keywords: PID controller, controller assessment, data-driven controller, least squares method.

The schematic figure of the proposed control system
(
),
(
),
is shown in Fig. 1. ( ),) are the controller, controlled system,and (reference model and estimated system model,
respectively.
The purpose of the proposed control system is toachieve the desired control performance by minimizing
the following criterion :
1
(1)
( )
=
2
( ) = ( ) − ( ),(2)where ( ) and ( ) are the control output andreference output, respectively.
Another purpose is to design the estimated system) by minimizing the following equation:model (
1
(3)
( )
=
2
(4)
( ) = ( ) − ( ),where ( ) is the estimated output. In the proposed
scheme, ( ) can be introduced by using thecontroller parameters and reference model parameters

1. Introduction
In most industries, it is very important to get desired
control performance. Data-driven control schemes1, 2, 3
have been proposed to achieve the aforementioned
desired control performance without using any system
identifications. For time-variant systems, performancedriven controller4 has been proposed to improve thesteady state control performance only when theperformance is poor. However, performance-driven
controller cannot evaluate the transient state.
In this paper, the scheme of evaluating controller
performance without using any system identifications is
proposed. According to the proposed scheme, the control
performance is evaluated system output error includingtransient state. Furthermore, the controller performance
calculator and controller design are integrated using only
input and output data.
2. Schematic figure of the proposed scheme© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan467

Takuya Kinoshita, Toru Yamamoto

The controlled system ( ) is expressed as follows byusing the aforementioned equation:
(
)
(10)
(
)= ∗
.
( ){1 −()}
Here, the following estimated system model ( ) is)
) instead of ∗ (defined using a controller (because it is difficult to obtain the optimized controller
C∗(
):
(
)
(11)
(
)=
.
(
){1 −
(
)}
(
) is
Note that the estimated controlled system) and reference
expressed by using a controller (( ) without any system identifications.
model() equals to ( ) when the optimized) is obtained. It mentions that ( ) =controller ∗ (() = ( ) . Therefore, the0 because of( ) = 0 and) achives
optimized controller ∗ (( ) = 0 simultaneously.

Fig. 1. Schematic figure of the proposed control
system.
without any system identifications. Therefore, controller
and estimated system model can be designed
simultaneously.
In addition, user set a desired reference model
expressed by following equation:( )=( ) ( )
(5)
(
)
(1)
(6)
( )≔
( )
) is userwhere ( ) is the reference signal and (specified polynomial. ( ) is designed based on thereference design5 as follows:
(
) ≔1+
+
(7)
4 −1
⎧
= −2exp −
cos
2
2
⎪
⎪
= exp −
,
(8)
⎨
⎪
≔
⎪
⎩
≔ 0.25(1 − ) + 0.51
where is a parameter related to the rise-time and is aparameter related to the damping oscillation. User set
them arbitrarily. denotes the time when output reaches
about 60% of the step reference value. Moreover, is set
between 0 ≤ δ ≤ 2.0 desirably. In particular, = 0
indicates the response of Butterworth model and = 1.0
indicates the response of Binominal model.

4.

Evaluation of the controller performance

The optimized controller ∗ () makes ( )()
equals to zero. In contrast,
becomes large when theperformance of the controller ( ) is poor. Hence, thispaper considers the performance of the controller ()
based on ( ).

5.

Tuning scheme of the PID gains

In this paper, the controller is utilized as following IPD controller:( )− Δ ( )− Δ ( )(12)
Δ ( )=( ): = ( ) − ( ),(13)
where , andare the proportional gain, integral
gain and derivative gain, respectively.
) is expressed by
The estimated system model (using I-PD controller is as follows:
(
)
( )=
,
(14)
− ( ) ( )where

3. Relationship between controller andestimated system model

Δ+

+

Δ

(15)
.
Δ
The estimated output is calculated as follows:
(
)
( )=
( ),
(16)
) (
)
− (
where ( ) is control input. In order to apply the leastsquares method, ( ) = 0 is considered and thefollowing equation is derived by using Eq. (2) and (18):(

The following equation can be obtained byintroducing the optimized controller ∗ ( ) which
achieves ( ) = 0:( ) ∗()(9)().
=
1 + ( ) ∗())=© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

468

u

Design of a Data-Driven

Fig. 3. Control result of the proposed scheme without
controller retuning.

Fig. 2. Control result by using the initial PID gains.

=

(

) ( )

−

+

+

(

)

( )

(17)

{1 −
(
)} ( )−((18)) () ( ).
+
Therefore, the PID gains are calculated by using thefollowing least squares method:
(19)
=(
)
,
(
)
(
)
−Δ
⎧
( ) = {1 −()} ( )
⎪
⎪
(
) ( )
−Δ
. (20)[(1),(2),⋯ , ( )]⎨ =⎪( ) (1), ⋯ ,( ) ( )]
⎪ =[]⎩ =[ , ,() ( )=

Fig. 4. Trajectory of the estimated output ( ) corresponding
to Fig. 3.

6. Numerical example

B)

The controlled system ( ) is given as follows:1( )=( < 100)
1 + 10
.
(21)
2
( )=
( ≥ 100)
1 + 20The parameters of the reference model() are set
as follows:= 3.0, = 0.
(22)
Finally, the initial PID gains are set as follows:= 1.0, = 1.0,
= 1.0.
(23)
In this section, the following three simulations areshown.
A) Fig. 2: Control result by using initial PID gains of
Eq. (23).

C)

Fig. 3: Control result of the proposed scheme without
controller retuning.
Fig. 5: Control result of the proposed scheme with
controller retuning.

Control results
Fig. 2 shows the control result by using initial PID
gains of Eq. (23). The control performance is poor
because control output ( ) does not track to reference
output ( ).
Fig. 3 shows the control result of the proposed scheme.
The following PID gains were calculated applying Eq.
(19) by using the data between t = 0 to 100 [step] in Fig.
2:
= 6.74, = 2.49,= 0.0.
(24)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan469

Takuya Kinoshita, Toru Yamamoto

In Fig. 3, the control performance is good even though
the system () is changed at = 100 [step]. It
mentions that it is difficult to detect the systemparameters are changed. On the other hand, Fig. 4 shows
the trajectories of the estimated output ( ). It easier to detect
the system parameters are changed than Fig. 3.
Finally, Fig. 5 shows the control result with controller
retuning at 120 [step]. Fig. 6 shows the estimated output( ), and Fig. 7 shows the trajectories of the PID gains
corresponding to Fig. 5. In Fig. 7, each PID gains areslightly adjusted, however, the estimated output ( ) of
Fig. 6 is significantly improved. Therefore, the proposed

scheme can diagnose the system parameters are changed
strictly.

Fig. 5. Control result of the proposed scheme with controller
retuning at 120 [step].

7. Conclusions
This paper has proposed the design of data-driven
control system with evaluating controller performance
The features of the proposed scheme are as follows:
Controller and estimated system model can be
designed simultaneously.
Satirical diagnosis of that system parameters arechanged.
The proposed scheme has been verified by numericalexamples.
References
1. S. Soma, O. Kaneko and T.Fujii, “A New Approach to
Parameter Tuning of Controllers by Using One-Shot
Experimental Data: A Proposal of Fictitious Reference
Iterative Tuning (in Japanese)”, Trans. of the Institute ofsystems, Control and Information Engineers, 17(2) (2004)
528–536.
2. T. Yamamoto and K. Takao and T. Yamada, “Design of a
Data-Driven PID Controller”, IEEE Trans on control
systems Technology, 17(1) (2009) 29–39.
3. M.C. Campi, A. Lecchini and S.M. Savaresi, “Virtual
Reference Feedback Tuning: A Direct Method for the
Design of Feedback Controllers”, Automatica, (38) (2002),
1337-1346.
4. T. Kinoshita, Y. Ohnishi, T. Yamamoto and S. Shah,“Design of a Performance-Driven Control System basedon the Control Assessment”, Proceedings of the IECON
2018 - 44th Annual Conference of the IEEE Industrial
Electronics Society, Washington D.C., (2018).
5. T. Yamamoto and S. L. Shah, “Design and Experimental
Evaluation of a Multivariable Self-Tuning PID Controller”,
IEE Proc. Control Theory and Applications, 151(5) (2004)
645–652.

Fig. 6. Trajectory of the estimated output ( ) corresponding
to Fig. 5.

Fig. 7. Trajectories of the PID gains corresponding to Fig. 5.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

470

The Actual Car Driving Evaluation System using Combined with Eyeball and Face Angle
Keiko Sakurai

Faculty of Engineering, University of Miyazaki,
1-1,Gakuen-kibanadai-nishi, Miyazaki-shi ,Japan
Hiroki Tamura

Faculty of Engineering, University of Miyazaki,
1-1,Gakuen-kibanadai-nishi, Miyazaki-shi ,Japan
Koichi Tanno

Faculty of Engineering, University of Miyazaki,
1-1,Gakuen-kibanadai-nishi, Miyazaki-shi ,Japan
Yukio Hattori

OFA SUPPORT INC, 3535-2,Shimanouchi, Miyazaki-shi, Japan
E-mail: sakurai.keiko.u6@cc.miyazaki-u.ac.jp, htamura@cc.miyazaki-u.ac.jp,
tanno@cc.miyazaki-u.ac.jp,

Abstract
The lack of automobile driving standards of persons with brain dysfunction and the increase of car accidents by theelderly are problems. Under these circumstances, to evaluate brain dysfunction and driving ability of elderly people
is needed. So, gaze estimation research is rapidly developing. In this paper, we proposed the actual car driving
evaluation system which can be evaluated by combining eye movements using eye tracking device (TalkEyeLite)and face movement estimated using template matching. In order to the evaluate our proposal method, we carried out
the actual car driving experiments. The subjects were one instructor of the car driving school, and 6 general drivers(three of the elderly, three of 40 people). We compare the gaze range of the 6 general subjects and the instructor. As
a result, we confirmed that one male in the 40s and one elderly narrow the gaze range.
Keywords: actual car driving evaluation system, eye tracking, template matching.

pose detection method by using camera [3] in previous
research. Further, we proposed a system that selects the
RGB-D sensor to determine the direction of the head,
estimates both directions simultaneously using the RGBD sensor and the EOG sensor, and estimates highprecision gaze [4][5]. However, this proposed system had
not high accurate that could be used for car driving
evaluation. In addition, the existing system is necessary
to place the RGB-D sensor, there is a problem that is
specialized for use in a limited space.

1. Introduction
Development of a system for evaluating the driving
ability of a car is proceeding from the necessity ofevaluating the traffic accident of elderly people and theresumption of operation of patients with higher brain
functions.When a person sees a specific direction, in
most cases the position of the head and eyes
simultaneously moves [1] [2], so to consider the
movement of the head and eyes in evaluating the drivingof the car is necessary. Therefore, we proposed the facial© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan471

Keiko Sakurai, Hiroki Tamura, Koichi Tanno, Yukio Hattori

prepare three templates (Marker1,2 and 3) as shown Fig.2.
with downward triangle and upward triangle and circle
on the front glass of the car. Fig. 3 shows the coordinate
position of each markers and the range of calibration.

In this paper, we propose the car driving evaluation
system using gaze by using combined eyeball and faceangle. We use the eye tracking device (TalkEyeLite [6])to get the eyeball angle and the image processing
technique to get the face angle. In this image processing
technique, we proposed the face angle estimate method
using template matching method. We have constructed asystem that can be evaluated by actual car driving using
eye tracking device and proposed face angle estimation
method. In this paper, we consider the results of actual
car driving evaluation for one instructor at a driving
school and 6 general subjects (three elderly people andthree men in their forties). By using eye tracking device
and proposed face angle estimate method, we built asystem that can be evaluated by actual car driving. In thispaper, we consider about the results of actual car driving
evaluation.

Fig. 2. Types of markers.

Fig. 3. The markers positions and the range of calibration

We perform the template matching method using thesethree template images and estimate the face angle.
The inclination of the face has three tilt axes, the yaw axis,
the pitch axis, and the roll axis, but when the roll axis istilted, the error occurs in the yaw angle and the pitch
angle. In order to solve this problem, the coordinates
calculated by the template matching are converted into
the coordinates corresponding to the change of the rollaxis, and the angle of the movement of the face isestimated from the coordinates.

2. Measurement System
2.1. Measurement system using TalkEyeLite[6]TalkEyeLite [6] is a wearable eye movement
measurement system that uses a USB camera for eyeballdetection and visual field and connects directly to theprocessing computer. Fig. 1(a) is a picture of
TalkEyeLite goggles including a view camera andeyeball camera which is a goggle type device. The gazeof both eyes is blue cross points of Fig.1(b).

(a) TalkEyeLite Goggles

3. Proposal Driving Evaluation Method
In this section, we introduce the evaluation method of thissystem. The fixation point is obtained by adding theestimation result of the line of sight and the angle of theface estimated from TalkEyeLite. Fig. 4 shows theanimation of the analysis result after performing thetemplate matching.

(b) The operation imageof .

Fig. 1. The measurement system of TalkEyeLite

Through USB camera, TalkEyeLite is able to track the
pupil of subject, therefore through the eye motion
analysis software we can able to know which target the
subject was looking at. The angle of convergence canpossibly be calculated with angle data of both eyes.

Fig.4. The display of our proposal system.

To evaluate the performance of this system, an actual
vehicle experiment was conducted at the driving school.
The subjects were one driving school instructor and 6
general subjects (Subject A: 60s, Subject B: 60s, Subject
C: 60s, Subject D: 40s, Subject E: 40s, Subject F: 40s).
The driving school instructor ran four times on actualcars, and 6 general subjects each ran on an actual car once.
The data of the driving school instructor was taken as
teacher data. In order to evaluate, we compare teacher
data and 6 general subjects in the time in case of faceangle, eyes movement angle and gaze. Fig.5 and 6 are

2.2. The face angle estimation method using
template matching method
We proposed the face angle estimate method using
template matching method. We use the viewing cameraof TalkEyeLite to obtain the head movement. The faceangle estimate method using template matching method
does not need to set up the camera. Therefore, there is a
merit that it is easily usable in various environment. We© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

472

The Actual Car Driving

histograms of the face angle, the eye movement, and gaze
during car driving, which means the face angle, the lineof sight, and the line of sight are distributed in the area
during running. The vertical axis represents the numberof times, the horizontal axis represents the X direction(left and right: plus is right, minus is left), and the depth
is Y direction (upper and lower: plus is upper, minus isdown).
Fig. 5 is an instructor of driving school, and Fig. 6 issubject B(60s). The numbers of data differ depending on
subjects. Also, the hatched portion on each graph is therange during running calculated based on the result ofGaussian fitting with the confidence interval set to 95%.

In other words, we assumed that the range of 5% or lessis not seen. The maximum value in the X direction is Xmax,
the minimum value is Xmin when the threshold value is
5%, and X is defined as Xmax-Xmin. Similarly, Y is YmaxYmin in the Y direction.

Fig.6. Histograms during car driving (Subject B(60s))

Fig.5. Histograms during car driving (Instructor)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan473

Keiko Sakurai, Hiroki Tamura, Koichi Tanno, Yukio Hattori

number of subjects is small, our system is greatly
influenced by the individual’s driving. Therefore, we
consider that the investigation on the gaze point
movements and the collection of actual cars driving databy expert drivers, healthy people and people with higher
brain dysfunction are required.
Acknowledgements
Funding and support from the OFA SPPORT INC isgratefully acknowledged.

Fig.7. Compare the X*Y value

References

4. Conclusion

1. C.J.S.Collins and G.R.Barnes, “Independent control ofhead and gaze movements during head-free pursuit inhumans,” Journal of Physiology, February 1999; Volume515, Part1, pp 299-314.
2. S. Ron, A. Berthoz,”Eye and head coupled and dissociatedmovements during orientation to a double step visualtarget displacement,” Experimental Brain Research, May
1991, Volume 85, Issue 1, pp 196–207.
3. Min Gyo CHUNG, Jisook PARK and Jiyoun DONG,” A
Simple Method for Facial Pose Detection”, IEICE
TRANSACTIONS on Fundamentals of Electronics,
Communications and Computer Sciences Vol.E87-A
No.10, October 2004, pp.2585-2590.
4. Keiko Sakurai , Mingmin Yan, Kazuhiko Inami, Hiroki
Tamura and Koichi Tanno, “A study on human interface
system using the direction of eyes and face”, Artificial Lifeand Robotics, December 2015, Volume 20, Issue 4, pp
291-298.
5. Keiko Sakurai, Mingmin Yan,Hiroki Tamura and Koichi
Tanno,” Comparison of two techniques for gazeestimation system using the direction of eyes and head”,
2016 IEEE International Conference on Systems, Man,
and Cybernetics (SMC), October 2016, DOI:
10.1109/SMC.2016.7844609.
6. Takei Scientific Instruments Cooperation: URL
http://www.takei-si.co.jp/en/ (access 2018.9.10)

In this paper, we introduced the actual car driving
evaluation system using gaze by using combined eyeballand face angle during car driving. We extended previous
proposed method and constructed the car driving
evaluation system by actual car. We also investigate thesystem.
Previous studies, we could not measure the range ofaccurate gaze during actual cars driving, but in this paper,
we measured the gaze while driving with one instructor
and 6 general subjects on actual car. And compare withthe teacher data, we constructed a system that canaccurately show the range of gaze during driving. In theevaluation, we found that subject B (60's) and subject E(40's) had narrower gaze range than the instructor. By
performing Gaussian fitting, a certain driving evaluation
may be possible by comparing the gaze range of
instructors and general subjects.
In addition, the scores of each subject's car driving
evaluation system (Smart Driving Assessment Program:
SDAP) are as follows. Subject A (60s): -840, Subject B(60s): -887, Subject C (60s): -720, Subject D(40s): -770,
Subject E(40s): -808, Subject F(40s): -405. The scores
are calculated by subtraction method. Therefore, lower
score means better result. Experimental results in thispaper suggest that subjects with poor SDAP scores may
have narrow field of view.
5. Future Work
The final objective of this paper is to construct a car
driving evaluation system which is the standard forresuming the driving of people with higher brain
dysfunction. For that purpose, to find clear differences
between the driving of healthy people and people withhigher brain dysfunction is necessary. However, since the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan474

A Study on Speaker Identification Approach by Feature Matching Algorithm using Pitch and
Mel Frequency Cepstral Coefficients
Barlian Henryranu Prasetio

Interdisciplinary Graduate School of Agriculture and Engineering, University of Miyazaki, 1-1 Gakuen Kibanadai-nishi,
Miyazaki-shi, 889-2192, Japan
Keiko Sakurai

University of Miyazaki, 1-1 Gakuen Kibanadai
Miyazaki-shi, 889-2192, Japan
Hiroki Tamura

University of Miyazaki, 1-1 Gakuen Kibanadai
Miyazaki-shi, 889-2192, Japan
Koichi Tanno

University of Miyazaki, 1-1 Gakuen Kibanadai
Miyazaki-shi, 889-2192, Japan
E-mail: barlian@ub.ac.id, sakurai.keiko.u6@cc.miyazaki-u.ac.jp, htamura@cc.miyazaki-u.ac.jp, tanno@cc.miyazakiu.ac.jp

Abstract
In this paper, we grouping the words based on the speaker in a sequence of speech in a conversation. There are two
speakers in each conversation. The first speech assumed spoken by speaker-1. In recognizing the speakers, we use
pitch detection and Mel Frequency Cepstral Coefficients feature extraction with 13 filters. Furthermore, we
examine the distance of the second speech vector with the first speech vector using the Feature matching algorithm.
Previously, we had experimented on each speaker to find out the mean and variance of the Feature matching. Basedon the experimental results, the Standard Deviation of Euclidean, Mahalanobis and Manhattan Distance are 0.0383,
0.0254, and 0.0341. Hence, if the Feature matching value deviates is not more than variance value then the speech
is assume spoken by speaker-1. Otherwise, the speech assume spoken by speaker-2.
Keywords: Speaker identification, Pitch, MFCCs, Euclidean, Mahalanobis, Manhattan.

under stress. It makes relatively difficult to recognize
stress.
Speech is one of methods that can be recognizing a
stress.3 It focuses on speech features in the frequency
domain. Human voice signals have a very high level ofvariability.4 A speech signal issued by different speakers
produces different speech patterns. It makes a problem
when we recognize a stress in a conversation. As weknow, the conversation is a sequence of words spoken
by more than one speaker. Therefore, in this paper, we

1. Introduction
Stress is a mental disorder that occurs in a person due topressure.1 Stress is one of emotion. Emotions divided
into two types, conscious and unconscious emotions.2
Conscious emotions are emotions that we can feel likeanger, sadness and happiness. Unconscious emotions
are emotions that we cannot feel like stress anddepression. It means, we cannot know when we are© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

475

Barlian Henryranu Prasetio, Keiko Sakurai, Hiroki Tamura, Koichi Tanno

2.2. Feature extraction

present a simple method of identifying the speaker in aconversation that indicated to have a stress speech on
the speaker.

2.2.1. Pitch
The Pitch is the fundamental frequency of the vocal
cord vibration (called F0) followed by formants
bandwidth at higher frequencies.11 Typically, the malevoice pitch is around 85-155 Hz and the female is about
165-255 Hz.12

2. Materials and proposed method
2.1. Proposed method
In this paper, we used a sequence of words that has been
segmented from two-pilot conversation in an Apachehelicopter cockpit.5 This conversation is recorded andcollected by the Linguistic Data Consortium (LDC) in
the Speech Under Simulated and Actual Stress(SUSAS) database.6 This conversation indicates thestress condition of the speaker. The speaker
identification consists of two stages, feature extraction
and feature matching.7 The process of our proposed
method can be seen in Fig. 1.
The Fig. 1 shows that the speech conversations
consisting of sequence words. The first speech extracted
the frequency fundamentals pitch and Mel Frequency
Cepstral Coefficients (MFCCs). Furthermore, we do thesame thing in the second speech. We assume that thefirst speech spoken by the speaker-1. Furthermore, we
examine the distance between the first speech and thesecond speech vector using the feature-matchingalgorithm (Euclidean8, Mahalanobis9 and Manhattan10).
In the preliminary experiment, we have analyzed thestandard deviation of each speaker. Therefore, if the
distance between the two vectors is less or equal to thestandard deviation, we decide that the second speech
spoken by the speaker-1, otherwise spoken speaker-2.

(1)=

Where: L is length of vocal folds
σ is longitudinal stressρ is Tissue density
2.2.2. Mel Frequency Cepstral Coefficients (MFCCs)
The MFCC can use as a characteristic vector torepresent human sound and musical signals. The soundanalysis on Mel-Frequency based on the perception ofhuman hearing,13 because the human ear has observed tofunction as a filter at a certain frequency. The filters
have a frequency response forming a triangle, and thespace between their bandwidth determined by a constantmel-frequency interval.14
The triangular filters applied to computing filterbanks.15 We used 13 filters to extract frequency bands
on a Mel-scale to the power spectrum. The model offilter bank on a Mel-Scale can be express as:

( )=

0,
⎧ ( (
⎪ ( )
⎨ ( (
⎪ (
⎩

)

(

0,

)

))

)
)

( )

,
,

(

< (

− 1) ≤

( )<

− 1)

≤ ( )

≤ (

> (

+ 1)

+ 1)

(2)

Where: m is the number of filters we want, f() is the listof m+2 Mel-spaced frequencies.
The frequency band of the filter16 shows in Table 1.
Table 1. The frequency band of the filter.
Filters
Filter1
Filter2
Filter3…
Filter12
Filter13

Passband Edges (Hz)[133 267][200 333][267 400]…
[867 999][933 1071]

Fig. 1. Proposed method process.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan476

A Study on Speaker

2.3. Feature Matching Algorithm

100

2.3.1. Euclidean Distance95

The Euclidean distance between points x and y is thelength of the line segment connecting ( ). In Cartesian
coordinates, if x= (x1, x2,..., xn) and y = (y1, y2,..., yn) are
two points in Euclidean n-space, then the distance (dED)from x to y, is given by the Pythagorean formula:1790( , )=

2.3.2. Mahalanobis Distance

∑

(

−

)

Euclidean Distance
Mahalanobis Distance
Manhattan Distance

96.94

97.76

91.58

90.46

89.23

88.69

87.58
85.42

85

83.77

80
Pitch

MFCC

Pitch+MFCC

Fig. 4. The system accuracy.

(4)

The Mahalanobis distance is a measure between two
samples point. The Mahalanobis distance between avector x and y with covariance σ is18
( , )=

2.3.3. Manhattan Distance∑(3)

The distance between two points measured along axes atright angles. In a plane with x at (x1, x2) and y at (y1, y2),
it is:19( , )=∑

3. Results and Discussions

|

−

|

(5)
Fig. 2. The feature vector sample. The utterance is “break”.
Pitch features is 10 feature vectors and 13 feature vectors for
MFCC

In this work, we used three conversations between two
speakers. Therefore, we have from six speakers feature
vector data. Furthermore, we calculate the standard
deviation for each speaker in each distance algorithm, as
follows:
Euclidean Distance = 0.0383,
Mahalanobis Distance = 0.0254,
Manhattan Distance = 0.0341.
This standard deviation value is the threshold for
determining speaker identification.
In the features extraction, each speech extracted from its
pitch and MFCC features. The feature vector sample
from feature extraction can be seen in Fig. 2.
Then on, we calculated the vector distance both ofspeech vector. Feature matching of each speech in thefirst conversation can be seen in Fig. 3.
Finally, we compare the number of speeches that havegrouped with the actual number of speeches for eachdistance algorithm to determine the accuracy of our
proposed method. The accuracy based on its features
extraction. The system accuracy can be seen in Fig. 4.

In Fig. 4 can be seen that in general the accuracy
reaches above 80%. Accuracy for multi-feature
extraction is better than single features, above 90%.

Fig. 3. The distance value on feature matching process ofeach speech in the first conversation.

Euclidean distance is better for single feature extraction.
Mahalanonbis distance is better on multi-features.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

477

Barlian Henryranu Prasetio, Keiko Sakurai, Hiroki Tamura, Koichi Tanno

4.

Conclusion
11.

The experiment involves three conversations between
two speakers. The system accuracy rate calculated fromthe number of words clustered in both speakers bycomparing pitch feature, MFCC feature andcombination of pitch and MFCC with self-calculation.
The experimental result shows that Euclidean Distanceis better for single feature extraction and Mahalanobis
Distance is better for multi-features extraction.

12.
13.

14.

Acknowledgements
Thank you for Tamura Laboratory that has supported us
in this study. Thank you to LDC that has given us the
SUSAS Data.

15.

References

16.

1. N. Schneiderman, G. Ironson and S. D. Siegel, Stress
And Health: Psychological, Behavioral, and Biological
Determinants, Annu Rev Clin Psychol, 2005(1) (2005)
607-628.
2. A. K. Tiwari, Non-Verbal Communication-an Essence ofInterpersonal Relationship at Workplace, SMS Varanasi
Management Insight, 11(2) (2015) 109-114.
3. N. R. Prakash and J. Kaur, A Study on PhysiologicalParameters Used To Monitor Stress in Experimentally
Induced Stimuli, International Journal of Computer
Science and Information Technologies, 6(6) (2015) 52445246.
4. M. Latinus and P. Belin, Human voice perception,
Current Biology, 21(4) (2011) 143-145.
5. J. H. L. Hansen, Composer, SUSAS Transcripts
LDC99T33. [Sound Recording] (Philadelphia: Linguistic
Data Consortium. 1999).
6. J. H. L. Hansen, Composer, SUSAS LDC99S78. Web
Download. [Sound Recording] (Philadelphia: Linguistic
Data Consortium. 1999).
7. S. S. Tirumala, S. R. Shahamiri, A. S. Garhwal and R.
Wang, Speaker identification features extraction
methods: A systematic Review, Expert Systems With
Application 2017(90) (2017) 250-271.
8. A. . S. Thakur and N. Sahayam, Speech Recognition
Using Euclidean Distance, Proc. International Journal of
Emerging Technology and Advanced Engineering, 3(3)(2013) 587-590.
9. G. Aradilla, J. Vepa and H. Bourlard, Using PosteriorBased Features in Template Matching for Speech
Recognition, in Proc. Interspeech ICSLP (Pittsburgh,
Pennsylvania, 2006).
10. M. D. Malkauthekar, Analysis of euclidean distance andManhattan Distance measure in face recognition, in Proc.
Third International Conference on Computational

17.
18.

19.

Intelligence and Information Technology (Mumbai, India,
2013).
N. Peleg, Homepage of Audio-Video Compression
Course,2
2010.
[Online].
Available:
http://cs.haifa.ac.il/~nimrod/Compression/Speech/S1Basi
cs2010.pdf. [Accessed 23 3 2018].
N. Zivic, Modern Communications Technology,
Oldenbourg (De Gruyter Studium, 2016).
L. Muda, . M. Begam and I. Elamvazuthi, Voice
Recognition Algorithms using Mel Frequency Cepstral
Coefficient (MFCC) and Dynamic Time Warping (DTW)
Techniques, Journal of Computing 2(3) (2010) 138-143.
A. V. Bhalla and S. Khaparkar, Performance
Improvement of Speaker Recognition System,
International Journal of Advanced Research in Computer
Science and Software Engineering 2(3) (2012) 82-87.
H. Fayek, Haytham Fayek, 21 4 2016. [Online].
Available: http://haythamfayek.com/2016/04/21/speechprocessing-for-machine-learning.html. [Accessed 21 3
2018].
L. R. Rabiner and S. Ronald W, Theory and Applicationsof Digital Speech Processing (Upper Saddle River:
Pearson, 2010).
H. Anton, in Elementary Linear Algebra (7th ed.) (John
Wiley & Sons, 1994), pp. 170-171.
R. Wicklin, SAS Blog, The DO Loop , 15 2 2012.
[Online].
Available:
https://blogs.sas.com/content/iml/2012/02/15/what-ismahalanobis-distance.html. [Accessed 24 10 2018].
P. E. Black, xlinux, NIST, 31 5 2006. [Online].
Available:
https://xlinux.nist.gov/dads/HTML/manhattanDistance.ht
ml. [Accessed 21 10 2018].

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan478

A Study on Breathing and Heartbeat Monitoring System during Sleeping using MultiPiezoelectric Elements
Praveen Nuwantha Gunaratne

University of Miyazaki, 1-1 Gakuen Kibanadai-nishi, Miyazaki-shi, 889-2192, Japan
Chika Yoshida

Graduate School of Engineering, University of Miyazaki, 1-1 Gakuen Kibanadai-nishi, Miyazaki-shi, 889-2192, Japan
Keiko Sakurai

University of Miyazaki, 1-1 Gakuen KibanadaiMiyazaki-shi, 889-2192, Japan
Hiroki Tamura

University of Miyazaki, 1-1 Gakuen KibanadaiMiyazaki-shi, 889-2192, Japan
Koichi Tanno

University of Miyazaki, 1-1 Gakuen KibanadaiMiyazaki-shi, 889-2192, Japan
E-mail: praveennuwantha@gmail.com, hi13049@student.miyazaki-u.ac.jp, sakurai.keiko.u6@cc.miyazaki-u.ac.jp,
htamura@cc.miyazaki-u.ac.jp, tanno@cc.miyazaki-u.ac.jp

Abstract
We propose a method to monitor breathing and heartbeat during sleeping using piezoelectric elements. The signal
acquired from the piezoelectric elements during sleeping is separated into a breathing signal and a heartbeat signal
using biomedical signal processing software. In the previous experiment, two subjects slept for one and a half hours.
The data from the piezoelectric elements during sleep were extracted. We compared the true value with the countresults from these data using the proposed method. The error rate of the breathing count was 0–13%, and the error
rate of the heartbeat count was 0–8.2%. In this paper, we show the proposal system to apply the diagnosis of sleep
Apnea-syndrome. From this experiment, our proposal method clearly makes sense given that on diagnosis of sleep
Apnea-syndrome and the developed system has already been able to verify the 80% of non-breathing/stoppedbreathing time intervals.

Keywords: Piezoelectric Element, Breathing Waveform, Heartbeat Waveform.

persons in bed and evaluation of the sleeping state. Its
basic elements are electroencephalography, eyemovement, and mentum electromyography, and it cansimultaneously record respiration, electrocardiography,

1. Introduction
Polysomnography is considered the gold standard forroutine monitoring of physiological parameters of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan479

Praveen Nuwantha Gunaratne, Chika Yoshida, Keiko Sakurai, Hiroki Tamura, Koichi Tanno

oxygen saturation, pretibial electromyography, snoring,
body position, motion, and temperature, and otherparameters; however, it requires the attachment of many
sensors to the individual in an array designed for
performance of the measurements, and thus, its routine
use is generally not practicable.
In the present study, we constructed and assessed theaccuracy of a system that incorporates thin, highsensitivity, piezoelectric elements to obtain real-time
output and analysis of respiration and heartbeat counts
and information on body movement without imposing
movement restrictions on the subject or invading theirprivacy. We already found experimentally that the
proposal method using fuzzy logic can yield respirationand heartbeat counts with mean error rates of 0-13%
and 0-8.2%, respectively [1]. In this paper, we show the
proposal system to apply the diagnosis of sleep Apneasyndrome.

Hz [2], and a commercial power noise frequency of 60
Hz. With this information, the digital signal processing
to extract the respiration signal from the signal obtainedby the elements is performed in the following processflow.
The signals obtained by the six elements undergo 10 Hz
cutoff low-pass filtration, followed by 0.8 Hz cutoff
low-pass filtration, and then 0.05 Hz cutoff high-passfiltration. The respiration signal is extracted in this
digital signal process of respiration extraction signal
filtering. The respiration extraction signal ischaracterized by periodic increases in amplitude, as
illustrated by the example in Fig. 2, which shows asignal after respiration extraction filtering.
To extract the heartbeat from the signal obtained by the
PE element, the digital signal was next subjected to thefollowing process flow using software. The signals
obtained by the six elements undergo 7 Hz cutoff lowpass filtration, followed by 5 Hz cutoff high-passfiltration, full-wave rectification processing, and finally,
2.5 Hz cutoff low-pass filtration.The heartbeat signal is
extracted in this digital signal process of heartbeat
extraction signal filtering. The heartbeat extraction
signal is characterized by periodic increases inamplitude, as illustrated by the example in Fig. 3, whichshows a signal after heartbeat extraction filtering.

2. Measurement system
2.1. System
In the present study, respiration and heartbeat were
measured using six PE elements, which were placed
between the bed and a bed mat, to facilitate acquisition
and transmission of their voltage signals. They areplaced so that the subject’s chest area lies verticallyabove them and thus close to the chest region tofacilitate acquisition of respiration signals by relatedchanges in the position of the chest surface andheartbeat signals by fine heart pulsations. Six voltage
signals underwent analog-to-digital (A/D) conversion
by the A/D converter followed by recording on a
personal computer (PC). The sampling frequency for the
A/D conversion was 1,000 Hz. In the present study,
respiration and heartbeat were thus measured from thevoltage signals of the six elements in the above layout
and stored in the PC.

Fig. 1. Sample raw waveform

2.2. Acquisition of respiration and heartbeat signals
For an adult at rest, respiratory movement occurs
approximately 13 to 20 times per min with a respiration
signal frequency of 0:23–0:33 Hz, and heartbeat occurs
approximately 50–90 times per min at a frequency of
0.83–1.5 Hz. As shown in Fig. 1, signals obtained usinga PE element contain a respiration signal frequency of
0.23–0.33 Hz, a heartbeat signal frequency of 0.83–1.5

Fig. 2. Breathing waveform© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan480

A Study on Breathing

the obtained bio-signal data will be directed through acommercially available A/D convertor device before theprocess of signal processing is being executed and afterthat heart-rate & breathing waveforms will be extracted
mainly based on their frequency characteristics.

Fig. 5. Basic system configuration process flow

Fig. 3. Breathing waveform

3. EXPERIMENT AND RESULTS
3.2. Testing

3.1. Proposed bio-signal measurement method

In accordance with the proposed bio-signal
measurement method, the experiment on selected test
users was performed. The total experiment time per one
test user was 180 sec. and within that period, preselected 15 sec. time intervals have been considered as
non-breathing/stopped breathing for the waveformdifferentiation. 3 time intervals starting from 70th
second of the experiment time, were considered as the
stopped-breathing and test users were instructed toperform accordingly. 70-85, 100-115 & 130-145 were
the pre-set stopped-breathing time intervals taken while
performing the experiment. Further we managed to
execute all these experiments at a steady state, withminimal body motion in order to minimize any
distraction.

Fig. 4. Sensor arrangement of the prototyped sensory bed

Measurement and performance evaluation wereconducted using a prototyped sensory bed arrangement
and 5 persons, having different age, gender as well as
physical height & weight were subjected to theexamination process. In the development of sensory bed
arrangement, 6-piezoelectric sensors have been placedon a mattress by considering the most reliable contact
areas during the state of sleeping position (Backposition with arms down) of an average person for
precise bio-signal data acquisition. Fig. 4 depicts thesensor placements that have been used for dataacquisition.
Fig. 5 portrays the system configuration of themeasurement system. In the signal acquisition process,3.3. Results
Fig. 6 presents a sample raw waveform acquired
through a piezoelectric sensor in the examination of biosignals during sleep test of 1st test user. The acquired
test result shows a significant variation in the raw
waveform during the above mentioned nonbreathing/stopped breathing time intervals. Further it
depicts considerably similar waveform during the rest of
the time. Figs. 7(a) and 7(b) respectively portray the
extracted breathing and heart-beat waveforms based on
their frequency bands.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

481

Praveen Nuwantha Gunaratne, Chika Yoshida, Keiko Sakurai, Hiroki Tamura, Koichi Tanno

Fig. 6. Sample raw waveform of 1st test user
Fig. 7(b). Heart-beat waveform (70[sec] to145[sec])of 1st test user

The visual inspection of these extracted waveforms
clearly shows the successful and effective capture ofbreathing and heart-beat waveforms. This approachclearly makes sense given that on diagnosis of sleep
Apnea-syndrome and the developed system has alreadybeen able to verify the 80% of non-breathing/stoppedbreathing time intervals of the above mentioned
examination. Apart from that, the measurement result
has showed that the pressure waves acquired at thecustom designed piezoelectric transducer can be used toobserve any slight change in heart-beat and body
motions as well.

4.

Conclusion

In the present study, we have proposed and investigateda method of using multiple PE elements for respirationand heartbeat measurements during sleep. This
proposed method clearly makes sense given that on
diagnosis of sleep Apnea-syndrome and has alreadybeen able to verify the 80% of non-breathing/stoppedbreathing time intervals. The dismissal of systemconstraint of state of sleeping position and further
improvement on accuracy of diagnosis of sleep Apneasyndrome can be obtained by adopting a piezoelectrictransducer array, which is a task for future work.
.
References
1. Kiyotaka Ho, Naoki Tsuchiya, Hiroshi Nakajima, Kei
Kuramoto, Syoji Kobashi and Yutaka Hata, Heartbeat
Rate and Respiratory Rate Detection based on Fuzzy
Logic by Air Pressure Sensor, Japan Society for Fuzzy
Theory and Intelligent Informatics, Fussy Symposium,
25(0), 54-54, 2009
2. Hiroki Tamura, Hiroaki Matsumoto, Chika Yoshida,Nobuya Takahashi, Breathing and Heartbeat Monitoring
during Sleeping using Piezoelectric Elements, JapaneseSociety for Medical Engineering, Medical Engineering
Symposium, 2P-1-4,2016

Fig. 7(a). Breathing waveform of 1st test user

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

482

Field Robot and Sensing system
Eiji Hayashi*

Mechanical Information Science and Technology, Kyushu Institute of Technology
680-4, Kawazu, Iizuka, Fukuoka, 820-8502, Japan
haya@mse.kyutech.ac.jp

Abstract
Labors in which forestry and agriculture and factory, are working in severe circumstance. So, such workers arenecessarily declines and decreases. However, it is important for us to keep going economy, life and nature. Therefore,
to prevent a devastation of the forest, the safety and the lack of food and work in harsh environments, we are thinking
alternative technologies that a part of severe and dangerous works are taken robot instead of such workers. Somethingchanges are needed that they are seen innovation such as the generation of a new industry by new technologies using
AI and intelligent to improve how to recognize and express more and more of the things that we have various desires.
Keywords: List four to six keywords which characterize the article.

The technology of robotics has provided efficient
productivity, quality and safety in factory as well as
improved operability so far. The developing robots inforestry, agriculture and food factory are active
nowadays since AI and robotics technologies has
progressed rapidly. Robot in the future has to have thework capacity or the co-operation cooperativenessequivalent to the skilled worker. It is too difficult to makeup for the labor shortages only in such robotics
technologies at the present.
This paper reports on field robot and sensing system asapplications of the robotics and AI to the agriculture, theforestry, the food factory and future industrial factory.

1. Introduction
Japan’s population is decreasing after a peak in 2008 due
to aging of the population and declining birthrates. Suchshrinking population will be having a major impact tobuild a sustainable economy and society over a long-term
period, but almost all people feel that the effects for ashort term are small and negligible. Japan’s economy andsociety will require more development in technologies
such as artificial intelligence and robots to make up for a
decline in the labor force. However, the decline of thelabor force already invites the labor shortages in theindustries. Therefore, the Japanese government aims tointroduce a new residence status that the foreign workers
are accepted in the fourteen business sectors which arerestaurants, hotels, nursing care, building cleaning,
agriculture, fishery, food and beverage, materials
processing, industrial machinery, electronics and electricmachinery,
construction,
shipbuilding,vehiclemaintenance, and airport ground handling and aircraft
maintenance.

2. Agriculture
A decline of Japan’s Agriculture has 2 big problems
related to work in the severe environmental conditions
such as hot summer climate and labor shortage. Foreign
agriculture employs a large-scale management
represented by United states, Australia and Netherlands© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan483

Eiji Hayashi
I aready am engaged in Agriculture
4%2% No Comment
4%
I hope to be engaged
as a part-time farmer

70

61.1%

60
50
40

No
as a hobby

58%

30

32%

20

24.9%
22.0% 23.7%
16.6%
12.8%

10

グ

26.6%
25.3%
27.2%
25.7 %
23.0% 24.8%
24.9%
20.1%
16.8%
19.2%
16.9% 14.8%
16.1%

10.4%

FR
仏

独

UK
英

9.3%
4.1 %
2.9 %

8.0%

0

Fig.1 Question : Do you hope to be engaged in the
agriculture?

DE

22.6%

US
米

JP
日

A 35‐44歳
UA
N Age 65
Under age 35歳未満
35Age
35-44 Age
45‐54歳45-54 55‐64歳
Age 55-64
65歳以上
Over
Source : 3 rd. meeting's materials of Cabinet Secretariat of Governmentof Japan, No.4, 2015
https://www.cas.go.jp/jp/seisaku/kataro_miraiJPN/dai3/gijisidai.html

using the agricultural machines or the environmental
control technology introduced the latest technologies.

Japanese farmers are more likely to be “mainly andsignificantly aged farmers”

2.2. Present State of Japan
4.42 million ha of The Japanese are of the cultivated landin 2018 is estimated to be 54.4 % of paddy fields, 45.6 %exclusive use for meadows and 6.3 % land underpermanent crops of the total amount of the fields.
A survey results are described in the meeting's materials
of Cabinet Secretariat of Government of Japan as isshown in Fig.1.
The famers who mainly engaged in farming is estimatedto be 1.50 million persons, average age:66.7 in 2017,
compared to 2.05 million persons, average age: 66.7 in2010. Taking Japan’s rapidly decreasing working
population to consideration, it will be too severe in realty
for the future agriculture to substantially decrease theformers. The age structure of farmer in each country
shown in Fig.2 reveals that the Japanese farmers of thiskind are more likely to be “mainly and significantly aged
farmers” than other countries.

light, temperature, moisture, and carbon dioxide
concentrations all year around. The producing crops
almost are vegetables which are divided into of a fruit
vegetable such as tomato, strawberry, cucumber or
eggplant, and a green vegetable such lettuce, spinach,green onion and herbs.
Such a plant factory has the issues as follows.
Almost all work is done from raising seedling toharvesting by hand.
Initial cost and running cost

2.3. Agricultural Technology
Nowadays it is important to improve the laborproductivity and the local adaptability so that theagriculture product can be produced in quantity. The
technology in the agriculture has promoted theintroductions of management and automation system 1-2,
enviromental control for the green house like a plantfactory, especially in Netherlands. The plant factory
cultivates plant in the facility utilizing artificial control of
Fig.2 The age structure of farmer in each country. The

Fig.3 The tomato harvesting robot picks a tomato. This
robot can pick one of the tomatoes around 20 seconds.
2.4. Automation
All work from seedling to harvesting is difficult toautomatically do by robotic technology because theinfluence of light in the various situation. 3-6 It is© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan484

Field Robot and Sensing system

important for robot to reduce and suppress the influenceof light including infrared. The influence of light occurs
due to regular reflection light bounces off from thesmooth surface. If the intensity of the regular reflection
increases, a color of object is recognized as different
color. Human usually sees a color of the smooth surface
from an angle occurring no regular reflection to reduceand suppress.
This phenomenon has various influences on therecognition for degree of mutilation of crop, controlling
pests and inspect, crop disease diagnosing, robot’s
posture and travelling for the harvesting as shown in
Fig.3.

occupies 42.5 % (1,158 ha) of 0 – 6 degree from flat toundulating, 19.4% of 6-12 degree.
In case of SD2 (1699 ha) in Sepang Estate, the mature oilpalm area is broken into 9 blocks to maintain theharvesting rounds within approximately 10 days.
The robot 7 has to be aimed autonomously 2 - 2.5 hours /task (including traveling time) , depending on a terrain
and weather. If such robot can achieve, dozens of work
rate and people would be reduced, and would be provided
more efficient to the plantation.
3. Forestry
Japan has one of the world’ largest forests, and theamount of the forests area accounts approximately 25
million ha and 66 % of Japan's land area of 37.8 million
ha. However, the labor force population of Japan declines
with the aging over the long term, and is about 45,000
people in 2015.
The forests in Japan are almost the plantation forests
made by planting the seedings which cover the area of 10
million ha of the forests in Japan, and are consisted ofJapanese cedar, Japanese cypress which are a type ofleaved tree.
Although the forestry cycle is plant, glow and harvest,
nowadays the Japanese government has promoted thecyclical use of the forests as circulative resources in Fig.6
in facing the all-out harvest of the plantation forests that
it will cause the vicious cycle in large scale logging aswell as deterioration of the forest.

2.4. Malaysia
In mainly Malaysia and Indonesia, the palm oil industry
has developed more active with the palm plantation
because oil palm is a kind of tropical plants is grown on
a large scale as the commercial crops.

Fig.4 The plantation and Dropped fruit bunches in Sime
Darby’s plantation
The infield FFB in the plantation of Sime Darby in Fig.4
has 2724 ha in Sepang Estate. This area’s acclivity

3.1. Forestry Technology
The forestry is need as the follows

Fig. 5 The cyclical use of the forests as circulative resources© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan485

Eiji Hayashi

5. References
forest information system to measure the timber

1. K. Imou, “Automatization and Robotization ofAgricultural Machinery,” J. of the Robotics Society ofJapan, Vol.35, No.5, pp. 376-378, 2017.
2. H. Sori, H.Inoue , H. Hatta and Y. Ando, Effect for a
Paddy Weeding Robot in Wet Rice Culture, Journal of
Robotics and Mechatronics, Vol.30 No.2, pp198-205,
2018.
3. W. Nakagawa, “Robots of Agricultural Implement andMachinery. Current State, Problem, and View in the
Future,” J. of the Robotics Society of Japan, Vol.35, No.5,
pp. 387-389, 2017.
4. R. Fukui, K. Kawae and S. Warisawa, Development of a
Tomato Volume Estimating Robot that Autonomously
Searches an Appropriate Measurement Position – Basic
Feasibility Study Using a Tomato Bed Mock-Up –,Journal of Robotics and Mechatronics, Vol.30 No.2,
pp173-179, 2018.
5. A. Shinjo et al., “The practical use of IT in agriculture:
The movement into high-value-added crops and integrated
solutions,” The J. of the Institute of Electronics,
Information and Communication Engineers, Vol.96, No.4,
pp. 280-285, 2013 (in Japanese).
6. R. Loren and D. B. Benson (eds.), Introduction to String
Field Theory, 2nd edn., Springer-Verlag, New York, 1999.
7. T. Fujinaga, S. Yasukawa, B. Li, and K. Ishii, Imagecs
Mosaicking Using Multi-Modal Images for Generation ofTomato Growth State Map, Journal of Robotics andMechatronics, Vol.30 No.2, pp187-197, 2018.
8. A. Mowshowitz, A. Tominaga and E. Hayashi, Robot
Navigation in Forest Management, Journal of Robotics
and Mechatronics, Vol.30 No.2, pp223-230, 2018.

volume and the height of trees and to analyze the tree
community in the forest
Carrying system of the pot for the seedings using thedrone etc.
Weeding robot 7 system for a steep mountain slope
Since the Japanese mountain has the steep slope, theworkers are too taught to operate the planting and theweeding under the blazing the sun as well as the forcing
various painful work postures.
4. Conclusions
Recently, although there are various issues in global,
robotics has potential for development and improvement
to solve them using networks and AI. This paper is
described that there are only a few example of the
agricultural and the forestry robots and systems as thefield robotics and sensing system. As the related issues,
the robotic technologies are needed the robotic arm forfactory automation or food factory with the teaching lessand the autonomous motion using AI processing, theautonomous moving type robot for iron ores and cokes inironworks, cleaning robot for the drifted garbage on theseashore and so on.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

486

Proposition of Saliency map Based on the Maximization of Center-Surround Difference*
Ryuugo Mochizuki†

Graduate School of Life Science and Systems Engineering, Kyutech, 2-4 Hibikino Wakamatsu-ku
Kitakyushu-City, 808-0135/Fukuoka, Japan‡
Shinsuke Yasukawa§

Graduate School of Life Science and Systems Engineering, Kyutech, 2-4 Hibikino Wakamatsu-ku
Kitakyushu-City, 808-0135/Fukuoka, Japan**Kazuo Ishii

Department of Human Intelligence System., Kyutech, 2-4 Hibikino Wakamatsu-ku
Kitakyushu-City, 808-0135/Fukuoka, Japan
E-mail: mochizuki-ryuugo@edu.brain.kyutech.ac.jp, s-yasukawa@brain.kyutech.ac.jp, ishii@brain.kyutech.ac.jp
www.lsse.kyutech.ac.jp

Abstract
Saliency is a property of images which attracts human attention. The tendency is expressed as distribution in asaliency map. Traditionally saliency map is calculated by obtaining and merging Center-Surround Differences, where,
the Center and Surround are calculated through the convolution by the finer, coarser Gaussian Filter respectively.
However, Saliency depends on the filter scales, furthermore, requires adjustments of Gaussian Filter scales. Ideal
Saliency is difficult to define. We propose to maximize the Center-Surround Difference to calculate saliency.

Keywords: Saliency map, Maximization of Center-Surround Difference, Image Feature, Feature Stability(the Surround). Itti’s method1 is typical of the example.
They proposed a method of creating saliency map byutilizing Gaussian Pyramid, and calculated image
saliency by convolution of different scales of filters, thenoperated across-scale differences with the smoothed
images and merged them to form saliency map. The maps
created under the process is categorized as a bottom upsaliency map, which was inspired by a model originatedfrom biological visual mechanism. Including the method1,
the response of saliency depends on the filter size or scale.

1. Introduction
In the domain of image processing, researches of finding
salient features are ongoing. Saliency means the tendencyof features in images to attract human attentions. One ofa method of creating saliency map utilizes Gaussian
Filter. In the example, saliency is evaluated according toacross-scale difference (Center-Surround Difference) 1,
that is, the difference of images which were smoothed byfiner filters (the Center) and by coarser smoothing filters© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

487

Ryuugo Mochizuki, Shinsuke Yasukawa, Kazuo Ishii

As the result, trial and error is required for smoothingfilter selection. Selecting appropriate filters is a problem
to obtain desirable saliency. Under fixed scale, thesaliency varies by changing object size in the scene.
We propose to select filter automatically so as tomaximize the Center-Surround Difference. Filter scale isdynamically adjusted according to the frequency, then,unstable saliency by various object size and distance canbe prevented.

the object (=Lenna) size, when the image was smoothedwith fixed scale of Gaussian Filters in conventionalsaliency map generation. The cause of the variation ischanging frequency band of the image emerged from theresize. The characteristics of Gaussian Filters differ in anine levels of Gaussian Pyramid. Thus, the differences of
the responses by each filter reflect to MSal. In case of theapplication of saliency map into object detection withrobot vision, oversight may be occurred depending on
distance differences between camera and objects. On theother hand, proposed method can highlight the object
uniformly also in the case of distance change. Robust
object detection can be realized under the condition.

2. Related Research
Itti proposed a method of obtaining saliency map, byadopting Gaussian Pyramid, whose filters were fixed tothe power of two (s, c=20, 21, 22, …), and, input image
was smoothed with different scales of Gaussian Filter1.
Here, s and c correspond to the filter scales for theCenter and the Surround, respectively.
Since then, saliency map has been developed so that
objects in scenes are highlighted in intact shapes2-6. For
example, arbitral scale selection of Gaussian Filters is a
strategy to the purpose. R. Achanta stated that highresolution saliency map can be created with enhanced
object edges, where objects were monotonously
highlighted by finely tuning the frequency of band pathfilters2. S. Frintrop enabled free ratio s/c selection ofGaussian Filters’ scales for the Center and Surround as
VOCUS2(Visual Object detection with a CompUtational
attention System)6. However, from the point of view, thesaliency changes according to the scale ratio, then, ideal
ratio is difficult to determine.
We propose to adopt maximization of the CenterSurround Difference by enabling dynamic adjustment of
the filter sizes according to frequency components inlocal regions in an image, that is, maximization ofCenter-Surround Difference. The function differs fromthe examples enumerated above. Thus, empirical filter
selection can be omitted.

1(a)Input Image(b)Itti’s Model(c)Proposed

Fig. 1 Examples of saliency map generations(Upper row: Resized to70% Lower row: Resized to 40%)

We propose to introduce Center-Surround Difference
Maximization into the algorithm for saliency map
generation.
The flow of proposed method is indicated in Fig. 2.
Firstly, input image IIM is converted from RGB to
CIELab system, and distributed to Intensity Channel IL,
Color Channel Ia (Red-Green), Ib (Yellow-Blue). Ia and Ib
indicate the contrast of Red and Green, Yellow and Blue,
respectively. We use Integral Image5 to obtain the
Surround and the Center. Integral Image is obtained bysumming up pixel values from the origin to arbitrallocation (x, y). Integral Image is prepared for eachchannel, then, the calculation of the Center and theSurround is done using Integral Image together with Box
Filter. Frequency component differs according to local
regions in the input image. We adjust the filter size while
scanning the filter so as to maximize the Center Surround

3. Proposed Method
3.1. Outline
We show some examples of saliency maps MSal in Itti’sand proposed method, in Fig. 1, which were generated
from two Lenna Images (Size: 512x512 [Pixels]). The
Lenna was resized in the black background each other.
The variation of saliency emerged under the variation of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

488

Proposition of Saliency Map
𝑦 ′ =𝑦′′ ′𝐼𝐼𝑁𝑇,𝑖 ′ (𝑥, 𝑦) = ∑𝑦′ =0 ∑𝑥𝑥 ′=𝑥=0 𝐼𝑅𝑒𝑡,𝑖 ′ (𝑥 , 𝑦 )Difference. The difference is reflected to Feature Maps
MI for Intensity Channel, MCa, MCb for Color Channels.
These Fearute Maps are normalized and merged to form
saliency map MSal.

(1)

Fig. 3. Smoothing with Integral Image and Box Filter

Pixel value at (x, y) in a smoothed image is denoted asEq.(2) shows.
𝐼

𝐼𝐵𝑙𝑟,𝑖 ′ (𝑥, 𝑦) = 𝐼𝑁𝑇,𝑖′ (𝒑𝑐 )−𝐼𝐼𝑁𝑇,𝑖′ (𝒑𝐷 )−𝐼𝐼𝑁𝑇,𝑖′ (𝒑𝐵 )+𝐼𝐼𝑁𝑇,𝑖′ (𝒑𝐴 )

𝑊𝐹 𝐻𝐹

(2)

𝑖′ ∈ {𝐿, 𝑎, 𝑏}
Fig. 2. Procedure of proposed saliency map generation method

With the help by Integral Image, only five arithmetic
operations are required to obtain a pixel value of asmoothed image.

3.2. Procedure
3.2.1. The center and the surround
IIM is converted from RGB to CIELab System, then,decomposed into three components, that is, Luminance L,
Color (Red-Green) a, (Yellow-Blue) b, to handle colors
easily to compute Center-Surround Difference in the
Color Channel. The contrasts of a, b are the same as the
Color Channel in Itti’s method1. Integral Image LINT isgenerated, then, region p (Wpmax x Hpmax [pixel]) is
prepared. Two box filters FBc(for the Center), FBs(for the
Surround) whose centers are common to p (center pp) areprepared (Fig. 3). The size of FS is optimized within p so
as to maximize Center-Surround Difference at pp.
Integral Image was adopted for the face recognition
method proposed by Viola8 to speed up the calculation of
Haar-Like feature. S. Frintrop also adopted Integral
Image for fast calculation to obtain the Center and theSurround9. Smoothing by convolution requires large
computational cost. The cost rises up proportionally tothe square of filter width. For this reason, we utilize the
combination of Integral Image and box filter forsmoothing. Integral Image is obtained by Eq.(1).

3.2.2. Maximization of center-surround difference
We state the process of Center-Surround Maximizationin the section. The filter sizes are WBc xHBc, WBs xHBs,
respectively. The matrice are shown in Eq. (3), (4). We
show matrix fully one as WBc xHBc,[1] etc.
𝐹𝐵𝑐 =
𝐹𝐵𝑐 =

1
𝑊𝐵𝑐 𝐻𝐵𝑐
1
𝑊𝐵𝑠 𝐻𝐵𝑠∗

𝑊𝐵𝑐 ×𝐻𝐵𝑐

[1]

(3)

∗

𝑊𝐵𝑠 ×𝐻𝐵𝑠[1](4)

The two Box Filters are centered with pp = (x, y), and FBs
is magnified up to WPmax xHPmax at maximum. The size ischanged depending on the spectrum at region P. For thereason, FS is altered to FS (pp) and expressed in Eq. (5)(7). Thus, we select WBs so that Center-Surround
Difference is maximized at each pp. The maximization isplotted to feature map MI(pp), MCa(pp), MCb(pp) at each pp.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan489

Ryuugo Mochizuki, Shinsuke Yasukawa, Kazuo Ishii

𝑀𝐼 (𝒑𝑝 ) = max |𝐼𝐿 (𝐹𝑠 , 𝒑𝑝 ) ⊖ 𝐼𝐿 (𝐹𝑠 (𝒑𝑝 ), 𝒑𝑝 )|(5)

𝑀𝐶𝑎 (𝒑𝑝 ) = max |𝐼𝑎 (𝐹𝑠 , 𝒑𝑝 ) ⊖ 𝐼𝑎 (𝐹𝑠 (𝒑𝑝 ), 𝒑𝑝 )|(6)

𝑊𝑠 (𝒑𝑝 )𝑊𝑠 (𝒑𝑝 )magnified the cropped saliency map to original size.
Thus, we evaluated ||MSal||, respecting conventional andproposed saliency map by varying WObj and changing theratio of C/S for each Gaussian Filters in Twin Pyramidin VOCUS26. Fig. 5 and Eq. (10)-(12) explain theprocedure to obtain |MSal| and ||MSal||.

𝑀𝐶𝑏 (𝒑𝑝 ) = max |𝐼𝑏 (𝐹𝑠 , 𝒑𝑝 ) ⊖ 𝐼𝑏 (𝐹𝑠 (𝒑𝑝 ), 𝒑𝑝 )| (7)𝑊𝑠 (𝒑𝑝 )𝐻‖∆𝑀𝑆𝑎𝑙 ‖ =3.2.3. Merging feature maps
MCa(pp), MCb(pp) are merged to MC(pp) as shown inEq.(8).
1

𝑀𝐶 (𝒑𝑝 ) = (𝑀𝐶𝑎 (𝒑𝑝 ) + 𝑀𝐶𝑏 (𝒑𝑝 ))2

𝑊𝐼𝑀 ∑ 𝐼𝑀 |∆𝑀∑𝑦=1
𝑠𝑎𝑙 |(𝑥,𝑦)𝑥=1

𝑊𝐼𝑀 𝐻𝐼𝑀|∆𝑀𝑠𝑎𝑙 | =

(8)

∑𝑁−1
𝑛=1 |∆𝑀𝑠𝑎𝑙,𝑛 |𝑁−1|∆𝑀𝑠𝑎𝑙,𝑛 | = |𝑀𝑠𝑎𝑙,𝑛+1 − 𝑀𝑠𝑎𝑙,1 |

(10)

(11)
(12)

Only for the generation of saliency map MSal, MI and MC
are normalized in order to prevent extra extension of
luminance amplitude when each of MCa(pp), MCb(pp) is
extremely narrower to the other in amplitude.
1

𝑀𝑆𝑎𝑙 (𝒑𝑝 ) = (𝑓𝑁𝑜𝑟𝑚 (𝑀𝐼 (𝒑𝑝 )) + 𝑓𝑁𝑜𝑟𝑚 (𝑀𝐶 (𝒑𝑝 )))2(9)

We just magnify the amplitude of the feature map to255(unsigned 8bit) to simplify the calculation. The
normalization is denoted with fNorm

4.

Fig. 5 Calculation of |MSal|

Experiment

In the experiment, as described in Fig. 4, we evaluatedthe effect to saliency map as saliency variation |MSal| (inimage) and ||MSal|| (in scalar) caused by changing object
size Wobj x Hobj (Wobj = Hobj), respective Itti’s method1,
VOCUS26, and proposed method.

4.1. Effect for saliency by resized object
4.1.1. Experimental condition
Fig. 6 shows an example of changing object (=Lenna
Image) size WObj x HObj[Pixel] (WObj = HObj) in the input
image (size: WIM x HIM[Pixel]). The object size ischanged equally Rs times in its height and width. Rs is
calculated based on WIM x HIM[Pixel]. The object size isshown as Eq. (13)
𝑊𝑂𝑏𝑗 × 𝐻𝑂𝑏𝑗 = 𝑅𝑆 𝑊𝐼𝑀 × 𝑅𝑆 𝐻𝐼𝑀

Fig. 4 Influences for MSal

In the event of changing WObj, we cropped the area inMSal, where object (Lenna image) was located, and,Fig. 6 Input image of Lenna© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

490

(13)

Proposition of Saliency Map

the other hand, under VOCUS2, ||MSal|| rose up as s
increased.

In this experiment, Rs is varied as following, (RS=RS, 1,
RS, 2, RS, 3, RS, 4, RS, 5, RS, 6)=(1.00, 0.90, 0.80, 0.70, 0.60,0.50), Number of the variation NS is 6. We used thesource code from7 to use Itti’s method.

4.1.3. Discussion
The transition of ||MSal,n || is shown in Fig. 7. And theprocedure to calculate |MSal| (i.e. Difference image ofMSal) is shown in Fig. 8, 9 and 10 under VOCUS2 (c:s=1:10), (c:s =5:10) , and proposed

4.1.2. Results
Table 1 shows the variation of MSal, that is, ||MSal||.
Proposed method marked the smallest ||MSal|| than other
Table 1. Comparison of ||MSal|||MSal||Saliency Method

VOCUS2 (c: s)=(1:10)

50.00

VOCUS2 (c: s)=(3:10)

57.22

VOCUS2 (c: s)=(5:10)

63.81

Itti

44.91

Proposed WPMax=0.25WIM

32.30

Proposed WPMax=0.50WIM

32.30

Fig. 7 Transition of ||MSal||

conventional methods. The table indicates that MSal in
our method is the steadiest against changing WObj. On

𝐼𝑐

𝐼𝑐

𝐼𝑠

𝐼𝑐

𝐼𝑠

𝑀𝑆𝑎𝑙,1(𝑅𝑆 = 1.0)

𝑀𝑆𝑎𝑙,4(𝑅𝑆 = 0.7)

𝐼𝑠𝑀𝑆𝑎𝑙,6(𝑅𝑆 = 0.5)

𝐼𝑐𝐼𝑠

𝑀𝑆𝑎𝑙,1(𝑅𝑆 = 1.0)

𝐼𝑐

𝐼𝑐

𝐼𝑐

𝑊𝐵𝑆

𝑊𝐵𝑆

𝑊𝐵𝑆

𝐼𝑠

𝐼𝑠

𝐼𝑠

𝑀𝑆𝑎𝑙,1(𝑅𝑆 = 1.0)

𝑀𝑆𝑎𝑙,4(𝑅𝑆 = 0.7)

𝑀𝑆𝑎𝑙,6(𝑅𝑆 = 0.5)

𝐼𝑐𝐼𝑠

𝑀𝑆𝑎𝑙,4(𝑅𝑆 = 0.7)

𝐼𝑐𝐼𝑠

𝑀𝑆𝑎𝑙,6(𝑅𝑆 = 0.5)||∆𝑀𝑆𝑎𝑙,3 ||=48.46 ||∆𝑀𝑆𝑎𝑙,5 ||=59.79||∆𝑀𝑆𝑎𝑙,3 ||=65.41 ||∆𝑀𝑆𝑎𝑙,5 ||=83.97

Fig. 8 Transition of |MSal,n|(VOCUS2 1:10)

Fig. 9 Transition of |MSal,n|(VOCUS2 5:10)||∆𝑀𝑆𝑎𝑙,3 ||=38.80 ||∆𝑀𝑆𝑎𝑙,5 ||=32.99
Fig. 10 Transition of |MSal,n|(Proposed Method WPmax=0.5WIM)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

491

Ryuugo Mochizuki, Shinsuke Yasukawa, Kazuo Ishii

method respectively. Ic and Is correspond to the image ofthe smallest filter scale in Gaussian Pyramid. According
to the transition of ||MSal,n||, commonly to all saliency
maps, ||MSal,n|| increased in the surrounding area ofLenna, as Rs becomes farer from 1.0, because, theluminance gradient became less steep as the Lenna image
with black background is smoothed. In the case of Rs =1.0, without background, the attenuation of theluminance does not occur near the border of Lenna. The
difference of the attenuation is one of the reason forincreasing ||MSal,n||. As WObj becomes smaller, the areaof the black background is larger. Thus, the luminancegradient becomes milder. That is why ||MSal,n|| increases
as Rs decreases. Next, we explain the cause of thedifference in ||MSal|| between in the case of increasing
the ratio of c under VOCUS2 method, and, applying
proposed method. When c:s was set to 1:10, highfrequency spectrum remained more in Ic than under thecondition of (c:s)=(5:10). When Rs decreased from 1.0
to 0.7, only high-frequency component was cut. On theother hand, when (c:s) was equal to (5:10), much more
frequency component was cut in Ic. The effect wasserious since Rs was set to 0.7. Downsizing an image
corresponds to the increase of high-frequency component.
However, the higher frequency was cut by larger c ofGaussian Filter. Thus, MSal is affected by various ratio of
c in VOCUS2. While, when proposed method wasapplied, FBs was optimized so that the contrast between
the Center and the Surround would be maximized. As
Lenna was downsized and the background becomeslarger, FBs was enlarged. For the reason, the surroundingarea can be affected to lose intensity by the background.
Particularly, the variation of |MSal| was large when Rs
was 0.5. The contrast is relatively small only considering
on inside Lenna image, compared to the image withbackground. However, in proposed method, FBs was
optimized also inside Lenna, so that the contrast between
the Center and the Surround can be well enhanced by the
maximization of the Center-Surround Difference. Thisphenomenon is opposite to the situation that larger c ofGaussian Filter is cutting high-frequency component in
Lenna. The maximum difference between the Center andthe Surround is reflected to MSal. Thus, MSal is lessaffected by the combination of fIM c and s.

5.

Conclusion

We proposed to introduce smoothing filter optimization
into saliency map creation, that is adaptive to various
image frequency. In experiment, we compared thesaliency variation under conventional and proposed
methods. The maximization of the Center-Surround
Difference was effective to suppress the variation.
However, proposed method has the limitation to enlarge
the Box Filter FBs. Our method is not completely free ofthe constraint by the filter size.
References
1. L. Itti, C. Koch, and E. Niebur, “A Model of Saliencybased Visual Attention for Rapid Scene Analysis”, IEEE
TPAMI 20(11)(1998), pp.1254-1259.
2. M. Cheng, N. J. Mitra, X. Huang, P. H. S. Torr, S. Hu,“Global Contrast based Salient Region Detection”, IEEE
TPAMI 37(3)(2015), pp.569-582.
3. G. Yildirim, S. Susstrunk, “FASA: Fast, Accurate, andSize-Aware Salient Object Detection”, In Proceedings ofthe 12th Asian Conference on Computer Vision,(Singapore, 2014), pp. 515-528.
4. F. Perazzi, P. Krähenbühl, Y. Pritch, and A. Hornung,“Saliency Filters: Contrast Based Filtering for Salient
Region Detection”, In IEEE Conference on Computer
Vision and Pattern Recognition (Providence, Rhode
Islande, USA, 2012), pp. 733-740.
5. R. Achanta, S. Hemami, F. Estrada, and S. Seusstrunk,“Frequency-tuned Salient Region Detection”. the 2009
IEEE Computer Society Conference on Computer Vision
and Pattern Recognition, (Miami, Florida, USA, 2009),
p.1597–1604.
6. S. Frintrop, T. Werner, and G. M. Garcia, “Traditional
Saliency Reloaded: A Good Old Model in New Shape”,
Proceedings of IEEE Conference on Computer Vision and
Pattern Recognition, (Boston, Massachusetts, USA, 2015)
pp. 82-90.
7. Caltech HP, Introduction of Software”Simpsal”
http://www.vision.caltech.edu/~harel/share/gbvs.php
8. P. Viola, M. Jones, “Rapid Object Detecton Using a
Boosted Cascade of Simple Features”, Proc. IEEE
Computer Society Conference on Computer Vision and
Pattern Recognition, (Kauai, Hawaii, USA, 2001), pp.
511-518.
9. S. Frintrop, M. Klodt, and E. Rome, “A Real-time Visual
Attention System Using Integral Images. ”, InInternational Conference on Computer Vision
Systems(Bielefeld, Germeny, 2007).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan492

Evaluation of Multi-Drone Wireless Network Properties for Tunnel Inspection
Raji Alahmad

Department of Human Intelligence System, Graduate School of Life Science and Systems Engineering,
Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Kazuo Ishii

Department of Human Intelligence System, Graduate School of Life Science and Systems Engineering,
Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
E-mail: alahmad.raji941@mail.kyutech.com , ishii@brain.kyutech.ac.jp
www.lsse.kyutech.ac.jp

Abstract
Communication is one of the challenges for UAVs during operations inside tunnels. In this paper, a wireless network
system using multiple drones functioning as repeaters is proposed. A number of measurements for Wi-Fi signals are
conducted to confirm the effect of different scenarios on the signal in virtual tunnel and real life tunnel. As a result,
RSS with -64 dBm can provide five Mbps of bandwidth.
Keywords: Tunnel inspection, wireless network, multi-drones.

1.

as distance, free-space loss, absorption, reflection andinterference. Furthermore, the waveguide effect plays animportant role in influencing the signal propagation due
to receiving different number of reflected rays of signalthat cause multipath propagation which lead tointerference phenomenon.4
There are many documented studies aboutcommunication in subways and tunnels see Refs.5-11.
However, none of the previous researches and methods
have tackled the problem of communicating with therobots for long distances inside the tunnels.
In this paper, a number of measurements for Wi-Fi
signals are conducted to confirm the effect of different
scenarios on the signal.

Introduction

Since all infrastructures are gradually damaged by time,environment, and human factors that cause deteriorationin the underpinnings, the need for inspection andmaintenance is very important. Humidity, dust, andreduced visibility are the main defects that operators facein such areas. In addition, the collapses, accidents, andnatural disasters arose the exigency of sending robots for
performing the inspection tasks. Conventional methods
depend on special ground vehicles that have many
limitations in mobility see Refs.1,2. Unmanned aerial
vehicles (UAVs) on the other hand, can overcome theselimitations due to their small size and highmaneuverability.3
Communication is of the challenges for UAVs, due tosignal attenuation, which is a result of many factors such

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan493

Raji Alahmad, Kazuo Ishii

2.

Concept of multi-drone network

We propose providing a communication link to create adynamic wireless network using multiple drones. The
basic idea is to distribute a number of supporting drones
equipped with signal transmitters and receivers, which
will act as repeaters. Each drone will receive thetransmitted signals and retransmit it with more power. As
a result, we can transmit and receive the signals between
the base station (at the gate of the tunnel) and the maindrone, which is supposed to perform the main task. The
network topology is illustrated in the figure 1.
The procedure of building a dynamic network can be
described as follows:(i) Start with sending the group of all drones together
inside the tunnel.
(ii) When the drones face a weak received signal
strength (RSS) from the base station, one of the
supporting drones (repeaters) will stop and the others
will continue. In a similar manner, the second
supporting drone will stop when the signal, repeatedby the first repeater, is recognized as weak, and soon.
3.

Fig. 1. Wireless Network topology using multiple drones.

Fig. 2. Nonrealistic Tunnel shape and dimensions.
Table 1. Experimental setup in Virtual Tunnel

Experiments and Results

In order to find out the characteristics of wireless signals
in the tunnel, a number of measurements for Wi-Fi
signals is conducted with several scenarios.
3.1. Experimental setup
In nonrealistic tunnel with square shape and dimensions
as illustrated in the figure 2. five different scenarios are
implemented to find out their effect on the RSS.
Aterm WG 1800 HP2 is used as a transmitter, with Builtin antenna (transmission 3×reception 3), and frequency
band is 2.4 GHz. For receiving terminal, Raspberry Pi 3
model B is used with built-in antenna.
More details for experimental setup are illustrated intable 1.
3.1.1.

Parameter

Equipment / Value

Tunnel length
Tunnel width
Tunnel HeightTransmitter
ReceiverRx speed
Sampling freq.
Frequency
Tx altitude
Rx altitudeSampling freq. of BW

65 meter
2.05 meter
2.78 meter
Aterm WG 1800 HP2
Raspberry PI 3 model B - AWUS036h
0.3 - 1.93 m/sec
450 -750 - 1073 sample/pint
2.4 GHz
0.2 – 1 meter
0.2 – 1 – 2 meter
60 sample/point

mentioned above affecting signal propagation andcausing signal attenuation.
3.1.2.

With 90 degree of curvature

In order to check out the effect of curvature on the signal,at the distance of 50 meter, we turned by 90 degrees andcontinue measuring the average of signal strength. Figure
3 illustrates RSS before and after 90 degree of curvature.
As clearly shown in the figure, the RSS curve slopes
rapidly and the signal is lost after 15 meters. Which cansee the big impact of curvature on the signal strength.

Straight caseWe measured RSS versus the distance in the case ofstraight tunnel. At each meter we calculate the averagesof measured signal strength for 65 points as shown in the
figure 3. We can see that as distance increases, RSS
decreases, which is the result of the factors that have been

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan494

Evaluation Multi-Drone Wireless Network

50 m

50 m

Fig. 3. RSS VS Distance in case of straight and 90° of
curvature.
Fig. 6. Bandwidth VS RSS.

3.1.3.

Tx-Rx altitude difference

Figure 4 shows RSS versus distance while changing thealtitude between the transmitter Tx and the receiver Rx,
and measuring the signal strength at each 15 meter andthen calculate the average of RSS, the figure shows that
Tx-Rx altitude difference has a very small effect on RSS.

3.1.4.

Different speed of receiver

To confirm the moving speed effect of Rx on RSS, twodifferent speeds, 0.3 m/s and 1.9 m/s, are implemented
using Alfa AWUS036h Wi-Fi adapter. Figure 5 shows
RSS versus the distance for different Rx speeds. We cansee that Rx speed has almost no effect on the RSS.

Fig. 4. RSS VS Distance with Tx-Rx altitude difference.

3.1.5.

Bandwidth VS RSS

Iperf tool is used to see the relationship between
bandwidth BW and RSS. This tool sends and receives
packets between the server and the client to measure thebandwidth. As it is illustrated in Figure 6, as RSS
decreases, BW also decreases. In addition, the point with
-64dBm can provide 5Mbps, which is recommended for
HD video streaming.
4.

Fig. 5. RSS VS Distance with two different speeds ofreceiver.

Discussion

Based on the experiments results with several scenarios,
we could make the following discussion:© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan495

Raji Alahmad, Kazuo Ishii

•

•

•

•

•

5.

5. Guan, K., Ai, B., Zhong, Z., López, C. F., Zhang, L.,
Briso-Rodríguez, C., ... & Tang, T. "Measurements andanalysis of large-scale fading characteristics in curved
subway tunnels at 920 MHz, 2400 MHz, and 5705
MHz." IEEE Trans. Intell. Transp. Syst. 16.5 (2015):
2393-2405.
6. Val, I., Arriola, A., Rodriguez, P. M., Gonzalez-Plaza, A.,
Alonso, D., Zhang, L., ... & Goikoetxea, J. "Wirelesschannel measurements and modeling for TCMS
communications in metro environments." Antennas andPropagation (EUCAP), 2017 11th European Conference
on. IEEE, 2017.
7. Jinxing, L., Youping, Z., Jing, Z., Rui, J., Cheng, T., &Zhenhui, T. "Radio channel measurements and analysis at
2.4/5GHz in subway tunnels." China Commun. 12.1
(2015): 36-45.
8. Guan, K., Zhong, Z., Alonso, J. I., & Briso-Rodríguez, C.
"Measurement of distributed antenna systems at 2.4 GHz
in a realistic subway tunnel environment. " IEEE Trans.
Veh. Technol. 61(2), 834-837.
9. El-Maghrabi, H. M., Attiya, A. M., & Hashish, E. A.
"Indoor Wave Propagation Prediction for Corridors
Segments with Partially Reflecting Walls by Using 3D
Waveguide Modal Analysis." Appl Comput Electromagn
Soc J, 31(3). (2016).
10. Hrovat, A., Kandus, G., & Javornik, T. "A survey of radio
propagation modeling for tunnels." IEEE Commun.
Surveys Tut. 16(2), 658-669. (2014).
11. Nurminen, H., Dashti, M., & Piché, R. "A survey on
wireless transmitter localization using signal strength
measurements." WIREL COMMUN MOB COM, 2017.
(2017).

Signal strength: Distance and waveguide effectare the important factors effect on the signals.
The drops and increasing of RSS curve accrues
due to receive different number of reflected raysof transmitted signal.
Curvature: the curvature has a big effect on thesignal strength since it blocks the maintransmitted ray, and then the reflected rays fromthe tunnel’s walls will be dominant.
Rx speed: different speeds of receiving terminal
does not change the RSS. The different could be
happened depending on the receiver sensitivity.
Different altitude of Rx: we found that different
height of receiver with constant height oftransmitter causes a relatively small difference
in received signal strength due to the
propagation models.
Bandwidth: With the experiment conditions andequipment, we found the point with -64 dBm of
signal strength met 5 Mbps. However, using
different equipment and different environments
the signal strength that can provide HD video
streaming might be different.
Conclusion

In this paper, we proposed a dynamic wireless network
system using multiple drones in order to communicate
with the drone inside a tunnel during an inspection task.
We discussed the effects of speed, altitude, and curvatureon the received signal strength. The results show that forour specific communication equipment in a nonrealistic
tunnel, placing a repeater at the point where signal
strength is -64dBm allows a longer distancecommunication.
References
1. Lee, S. Y., Lee, S. H., Shin, D. I., Son, Y. K., & Han, C. S.
(2007). "Development of an inspection system for cracks
in a concrete tunnel lining." CAN. J. CIVIL. ENG. 34.8
(2007): 966-975.
2. Minichan, R., Fogle, R., & Marzolf, A. "H-Canyon Air
Exhaust Tunnel Inspection Vehicle Development." (No.
SRNL-STI-2011-00078). SRS. 2011.
3. Mitchell, J., & Marshall, J. A.. "Design of a Novel AutoRotating UAV Platform for Underground Mine Cavity
Surveying. " (2017).
4. Goldsmith, Andrea. "Wireless communications. "Cambridge university press, 2005.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

496

Counting Crops under Cultivation using Drone
Yasunori Takemura

Department of Engineering, Nishinippon Institute of Technology, 1-11 Aratsu, Kanda
Miyako-gun, Fukuoka, 800-0394, Japan
Yusuke Hirata

Graduate School of Life Science and System Engineering, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu
Kitakyushu, Fukuoka, 808-0196, Japan
Eiji Mizoe

Sky Canvas Co. Ltd. , 3B 3F, 1-9-10, Sakaimachi, Kokurakita-ku
Kitakyushu, Fukuoka802-0005, Japan
Masao Tashiro

Project Skyeye LLC. , 2243-55 Kamishinyu
Nogata, Fukuoka 822-0033, Japan
Yousuke Nagai

Aruku Agriculture LLC, 415-1, Oaza Hirosue, Chikujo-machi
Chikujo-gun, Fukuoka, 829-0100, Japan
E-mail:takemura@nishitech.ac.jp, hirata-yusuke@edu.brain.kyutech.ac.jp,
m.1978.eiji@ked.biglobe.ne.jp, contact@arknougyou.com

Abstract
In recent years, agricultural workers in Japan are decreasing year by year. Also, rising the average age of agricultural
workers is a problem in Japan. As a solution to this problem, there is promotion of new agriculture (smart agriculture)that realizes super-energy saving and high quality production by utilizing robot technology and IoT technology. The
authors are promoting technology by operating forest drone competition and tomato harvesting robot competition.
Through robot competition, we are currently paying attention to the use of drone in the agricultural field.
In this research, we aim to automatic detect the number of agricultural crops by image processing from image dataof aerial photographs taken by drone. In this paper, we will report a method to discriminate agricultural crops bycolor and to detect the number.
Keywords: Field Robotics, Sensing System, Agriculture robot, Drone, Image processing

elderly population aged 65 and older is flat. It is expectedthat this trend will continue in future, and problems inagriculture are labor shortage and reduction in production
efficiency.
The actual condition of the agricultural field is severe.
A lot of works are relied on manually. Although the

1. Introduction
In recent years, agricultural workers in Japan are
decreasing year by year (Fig.1) [1]. Also, rising the
average age of agricultural workers is a problem in Japan.
As shown Fig.1, agricultural workers are declining, the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

497

Yasunori Takemura, Yusuke Hirata, Eiji Mizoe, Masao Tashiro, Yousuke Nagai

Fig.1

Population

of

the

agricultureinnumber of settled plants after planting crops. The reason
is that counting the number of crops manually is too time
consuming and laborious. For that reason, it is very
difficult for farmers to make a profit plan. Farmers
usually ship to wholesalers after harvesting. However,wholesalers do not buy everything they harvested.
Therefore, we usually discard the remaining crops.
Also, the price of the crop changes according to theharvest time of the crop. How much crops can be
harvested at any time is one of the important factors inplanning income and expenditure. Knowing the numberof settled crops makes it easier to predict the number of
harvests and make it easier for business planning. When
it is possible to plan a processing plan assuming thenumber of agricultural crops considered to be unsold,loss due to disposal will decrease, it is thought that thiswill lead to improvement of production efficiency andstabilization of management.
As an example of previous research in agriculture, there
is a study that explores the possibility of some trait
evaluation by processing the image of the field from thedrone aerial image [7]. In this research, he used a drone
equipped with a camera with eight rotating blades with asize of 100 [cm] x 100 [cm] x 30 [cm] and a mass of 1.6[kg]. Drone flying automatically with a pre-programmed
flight path of 4 hectares of field. In this research,
photographs of the whole field are created bysynthesizing about 60 images taken from 100 m above
the sky. Growth data such as sprouting timing, initial
growth rate, and dead time can be obtained from thisgrowth amount, which measures the growth amount of
the crop from the time series image taken regularly by thesame flight route. In this way, Drone's aerial image is said
to be suitable for data collection by regular photography
because it has high definition and low operation cost.
In this hearing survey and previous research, we thoughtthat we could shoot color images taken of agricultural
crops using drone to count the number of planted crops.
We also thought that we can grasp the number of settled
plants and the growth situation of crops from the area ofcrops extracted by image processing.

japan

mechanization of agriculture is depending on item,
horticultural crops such as vegetables and fruit trees rely
on manual works such as harvesting and management.
In this situation, securing new workers is very important.
However, current agriculture requires a lot of hard workthat depends on hands, takes time to acquire skills, andmany farming techniques are difficult to manualize.
Normally in a crop, a worker can only experience all thework one a year. It takes a long time for new workers toacquire known-how like experienced workers [2].
As a solution to this problem, there is promotion of new
agriculture (smart agriculture) that realizes super-energy
saving and high quality production by utilizing robot
technology and IoT technology [3,4,5]. The authors arepromoting technology by operating forest drone
competition and tomato harvesting robot competition [6].
Through robot competition, we are currently paying
attention to the use of drone in the agricultural field.
In this research, we aim to automatic detect the numberof agricultural crops by image processing from image
data of aerial photographs taken by drone. In this paper,
we will report a method to discriminate agricultural crops
by color and to detect the number.

3. Algorithm of counting crops

2. Field Survey and Previous Research inAgriculture

Figure 2 shows the algorithm for counting crops using
image processing. First, the drone flies the field
automatically and takes pictures of the crop. At this time,the drone moves along the ridge if keeping a certain
altitude. The photograph is taken offline into a personal
computer and processed as image data.

The author conducted a survey on issues and work on
agricultural sites at a farmer who is a coauthor in working
in Aruku agriculture service.
When surveying, one of the biggest problems is income
instability. Because, regular farmers do not count the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

498

Counting Crops under Cultivation

Fig.2 Algorithm for image processing
Fig. 3 histogram of image processing binarized object

Because the read data is RGB image, it converts to HSV
hue system. Because, this time, identification of crops isdone with emphasis on hue data. Next, binary conversion
is performed by determining the threshold value by hue.
Filter the binarized image to identify the size of the crop.
In the filter, filling up the hole is done to grasp the areaof the object to be crop, before removing the noise. Somenoise or crops or unknown things remain in the binary
image detected as crop even if noise removal processing
is done. Therefore, in order to make the number of crops
accurate, labeling is performed on candidate images anda histogram is created with area.
Figure 3 shows the histogram example of result. The
horizontal axis is the object area (number of pixels) andthe vertical axis is the number of objects (pieces) Notethat the number of classes in the histogram is the Sturges’
formula. Sturges' formula [8] is expressed by thefollowing equation. N is the sample size, and k is thenumber of classes.
=+1
The formulas determine the appropriate number ofclasses in the histogram. As can be seen from Figure 3,
the first class has concentrated areas that are not crops in
the field, so treat the class width as a threshold and treat
it as noise. Figures 4 shows the processed images ofimage processing. Figure 4 (a) shows the raw image ofpicture. (b) shows after the binarize of image. (c) shows
after the filtering process, and (d) shows the end of image
processing.

4.

(a) raw imgae(b) after binalize(c) after noise reduction(d) counting result
Fig.4 Image processing process

Fig.5 DJI Drone and experimental view

obtaining the relative error between the number (truevalue) from the actual image and the number calculated
from the program.

Experiment of counting crops of Lettuce field

4.1. Usede equipment

This experiment was conducted to examine the validityof the program to calculate the number of plantings fromimages of agricultural crops taken with drone. In theexperiment, the accuracy of the program is evaluated by

For this experiment, "Phantom 3 Advance" which is adrone of DJI company was used (Fig.5). The
specifications of the camera mounted on Phantom 3 can© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

499

Yasunori Takemura, Yusuke Hirata, Eiji Mizoe, Masao Tashiro, Yousuke Nagai

As a result of the experiment, it was found that the
number of settled plants can be calculated from theprogram with an error of a few percent. However, somealgorithms such as removal of weeds and others by noise
elimination algorithms remove some crops from theimage. Therefore, as a future works, we need to improve
the algorithm for improvement of accuracy. For example,
we try to find ridges and weeds.

Acknowledgements
This works is supported by COC (Center of Community)
Project in Nishinippon Institute of Technology and Sky
Canvas Co. Ltd., Aruku Agriculture LLC., and Project
Skyeye LLC.

Fig.6 Experimental result
be taken with a maximum still image of 4000 x 3000
[pixcel].
4.2. Experiment method and result

References

Experiments were conducted by changing the altitude ata fixed point. By changing the altitude of the drones therange of shooting widens and the shooting abilitychanges. At that time, we conducted an experiment toverify to what extent the height can be measured to see if
it is appropriate. Height was changed 5 m at a time andshooting was performed from 5 [m] to 50 [m]. In theexperiment, 41 lettuces could be taken for shooting at 5[m], and 1384 lettuces could be taken when shooting at
50 [m].
Figure 6 shows the experimental results. Changing theheight, errors increased as the number of settled lettuce
increased. However, the relative error was within 4% at
the most. When the height is high, the relative error tends
to be in the minus direction. Changing the height makes
crops smaller. Therefore, it is difficult to discriminate
from noise.
At about 50 [m], it was found that the relative error was
not greatly affected. In addition, it was found that a minus
error of several percent is generated than the actual
planting number. In the future, we need to make
improvements to distinguish it from weeds as a measure.
In the situation where photography was actually done,
there were no weeds in the ridge, and there were many
weeds on the sidewalk. Therefore, we believe that
improvement of accuracy can be expected by adding
improvements such as finding ridges.

1. Homepage in Ministry of Agriculture, Forestry and
Fisheries, “Statistics of Agricultural Workers”, inJapanese,“http://www.maff.go.jp/j/tokei/sihyo/data/
08.html”
2. Sumito Yasuoka, “Promotion of “Smart Agriculture””,
Journal of Robot Society Japan, Vol35, No.5, pp.2-5, 2017,
in Japanese
3. JETRO Homepage, “Market Report: Smart agriculture”,
https://www.jetro.go.jp/ext_images/en/invest/attract/pdf/
mr_smartagri_en.pdf”, 2017
4. E.Morimoto, K.Hayashi, “Design of Smart Agriculture
Japan Model”, Advances in Animal Biosciences, Vol.8,
Special Issue 2), pp.713-717, 2017
5. T.Fujinaga, S.Yasukawa, B.Li and K.Ishii, “Image
Mosaicing Using Multi-Modal Images for Generation ofTomato Growth State Map”, Journal of Robotics andMechatronics, Vol.30, No.2, 2018, pp187-197
6. K.Ishii, Y.Takemura, T.Matsuo and T.Sonoda, “Tomato
Harvesting Robot Competition”, pp.537-542, SCIS-ISIS,
2016, Hokaido
7. Ryo Sugiura, “High-Throughput Field Phenotyping Using
Aerial Imagery from Drone”, Journal of Robotics Society
Japan (JRSJ), Vol.35, No.5, pp.369-371, 2017
8. H.Sturges, “The choice of a class-interval”, Journal of
American Statistical Assoc., Vol.21, pp.65-66, 1926

5. Summary
In this research, we aim to automatic detect the numberof agricultural crops by image processing from image
data of aerial photographs taken by drone.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan500

Soil Compaction and Rolling Resistance Evaluation of a Locomotion System with an
Adjustable Contact Patch for a Grape Transporting Robot
Enrico di Maria

Department of Human Intelligence System, Graduate School of Life Science and Systems Engineering,
Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan

Kazuo Ishii

Department of Human Intelligence System, Graduate School of Life Science and Systems Engineering,
Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
E-mail: e.dimaria.srt@gmail.com , ishii@brain.kyutech.ac.jp
www.lsse.kyutech.ac.jp

Abstract
Soil compaction is a form of soil degradation which causes soil erosion, drainage difficulties, and loss of nutrients.
To mitigate this issue, in agricultural farms several methods have been adopted. In this paper the preliminary studyof a locomotion system with adjustable contact patches is proposed. Aim is to reduce the compaction and optimize
the rolling resistance of a grape transporting robot employed in vineyard. A Finite Element Method model supported
by experimental results is developed to study the system.
Keywords: Soil Compaction, Rolling Resistance, Finite Element Method, Cone Penetrometer.

interventions, such as subsoiling and plough, or by
avoiding compaction using low ground pressure tyres,tracked tractors or controlled traffic farming (CTF).
Avoiding the compaction resulted to be the best solution
in terms of cost benefits. With CTF, not only compaction
itself is avoided, but energy consumption is reduced
because of the lower rolling resistance of vehicles on
permanent traffic lane, 2. Different levels of
mechanization can be adopted in the farms, and soil
compaction is often neglected.
Target of this work are the canopied vineyards, in which
the trees form a roof and the small and closed
environment makes more difficult a full mechanization.
Here harvester machines are rarely employed, unlike theespalier vineyards where big harvester machines arewidely used. Usual tractors don’t improve the harvesting

1. Introduction
Soil compaction of agricultural soil is a result of theincrease in mechanization and the usage of heavy
vehicles exerting high forces onto the soil. As a porous
material, soil consists of a solid part which includes porescontaining air or water. Excessive compression causes
the reduction of pore space and the loss of air and water,
important for plants roots growth, and determines
infiltration difficulties, loss of nutrients and soil erosion.
Soil compaction resulted to be related to a significant
reduction in the yield, and it was estimated a crop
yielding reduction of 25-50% in some districts of Europe
and North America, 1. To solve these issues manymethods have been proposed, with different results anddifferent costs, 2. The problem can be addressed by direct© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

501

Enrico di Maria, Kazuo Ishii

usage of the Cone Penetrometer (CP) is preferred toestimate the rolling resistance and performance of offroad vehicles,5. CP is a portable device to measure thestrength of the soil. It consists of a circular cone at theend of a metal stick and a load cell for measuring theforce required to push the cone into the ground. The
parameter obtained is the Cone Index (CI) [kPa]. Basedon the CI the following empirical equations for
estimating the RR on soft terrains have been proposed for
track vehicles, 6:

Fig. 1. System configurations in track and wheel mode.

process, but yet they cause a huge compaction.
The system proposed here consists of an autonomous kart
aimed to optimize the rolling resistance while avoiding
the compaction. It consists of four small track modules
which can be partially lifted so as to reduce their contact
patch and act as a wheel, depending on the soil conditionsand vehicle payload, Fig. 1. Tracks are more suited forsoft or wet soils, while wheels show better performanceon hard soils.
In this paper the system is studied in terms of rolling
resistance (RR) and compaction, evaluated by empiricalequations and a Finite Element Method model (FEM).
FEM simulations have been widely employed for RR
evaluation on soft and hard soils, 3. Experiments in thefield are carried out to find some of the soil parameters.
The empirical equations for RR evaluation, derived from
Terramechanics, are considered in the second section. In
the third section the experiments carried out in the field
are explained. These results, together with a non-linear
regression equation, are used in section 4 for evaluating
soil Young moduli and modelling the soil. Finally, in
section 5, static simulations to determine the soil
deformation and stress are carried out and explained.

𝐶𝑛 =𝐶𝐼 × 𝑏 × 𝑙
0,45, 𝑅 = 𝑊(+ 0,045).
𝑊
𝐶𝑛

(2)

And for wheel vehicles,7:
𝐶𝑛 =𝐶𝐼 × 𝑏 × 𝐷
1,2, 𝑅 =𝑊(+ 0,04).
𝑊
𝐶𝑛(3)

Where Cn is the track numeric, b and l the contact patch
width and length respectively, D the wheel diameter, W
the weight supported from each wheel or track, and R therolling resistance. As a first tentative choice for thedesign, the product HS-200 developed by Shangai Puyi
Industrial Co. is considered. Main dimensions and load
bearing are shown in Table 1:
Table 1. HS-200 Track module
Ground contact length
400 mmWidth
200 mmOuter sprocket diameter
370 mmLoad bearing
100 kgConsidering this track module and a vehicle weight of
400 kg, for a CI between 500 kPa and 7000 kPa thecomparison in terms of RR between track mode andwheel mode is shown in Fig. 2.
As expected, on soft soil tracks outperform wheels, whileon stiff soil wheels have better performance.

2. Rolling resistance on soft soils
Rolling resistance on soft terrains can be seen as the work
done in compacting the soil and making a rut of area A
and depth z,4, as in Eq. (1):
𝑧0

Track

250

Wheel

230
210
190
170
7000

6500

6000

5500

5000

4500

4000

3500

3000

2500

2000

150
1500

Where z is the depth, p the pressure, b the contact patch
width, l the contact patch length of the track and R therolling resistance. In case of wheel, the length consideredis the distance traveled by one revolution. The pressuresinkage relation proposed by Bekker requires a series of
empirical parameters, such as: the cohesion modulus kc,
the frictional modulus kɸ and the exponent of sinkage n.
These parameters are determined by means of theBevameter test, 4. Because the equipment required is big
and the interpretation of the results is not trivial, often the

270

500

0

1000

(1)

Rolling Resistance [N]𝑅𝑙 = 𝑏𝑙 ∫ 𝑝𝑑𝑧.

Cone Index [kPa]

Fig. 2. Rolling resistance comparison between wheel mode andtrack mode for different soil strength.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan502

Soil Compaction and Rolling

The difference between the two modes depends on track
module sizes and vehicle weight, and in the design a setof best parameters has to be decided in order to maximize
this difference.
3. Experiments in the vineyard
Soil strength measurements have been carried out in avineyard in the South of Italy, at Latiano in Puglia region.
Hand-held digital cone penetrometer SC900 developedby Spectrum Technologies was employed. It allows toinvestigate the soil until a depth of 45 cm and a CI of7000 kPa. Moisture sensor VG Meter 200 has been used
for measuring the volumetric water content percentage.
Moisture content affects soil behavior and rolling
resistance considerably, and for this reason it ismonitored in the analysis.
The measurements were taken with sets of 9 points fromrow to row every 6 m as shown in Fig. 3, so as to get a
profile of compaction, for a total of 63 measurements.
The distance between the trees is 2 m, while the spaceamong each measurement point is of 22 cm. The points
are in the middle between the trees for avoiding the roots.
Because the average CI for the first 15 cm is widely used
for rolling resistance evaluation, 5, a depth of 15 cm isconsidered here. On the wheel tramlines some of themeasurements could not be completed because theresistance exceeded the limit of the sensor. In the analysis
a value of 7000 kPa (sensor limit) is considered for thosepoints.
The results in terms of range of CI measured and profile
along the road are reported in Fig. 4 and Fig. 5
respectively. Moisture content average was of 6,5 % in arange of [2% - 17%]. High variability in the overall CI
results is due to the higher compaction at the wheel
tramlines, as it can be seen from the two peaks in thestrength profile.

Cone Index [kPa]

Fig. 4 Cone Index overall results.
3500
3000
2500
2000
1500
1000
500
0
1

2

3

4

5

6

7

8

9

Measurements

Fig. 5. Soil strength profile AVG 15 cm

4. Soil modelling and simulations
Soil under stress presents an elastoplastic behavior.
Drucker-Prager criterion (DP), outer cone approximationof the Mohr-Coulomb criterion, is used in this work, 3.
A method adopted for developing a FEM soil model is todivide it into layers. Based on the experimental results, 3
layers have been identified. First 2.5 cm with low
strength, 2.5 cm to 7.5 cm with medium strength andfrom 7.5 cm to 50 cm with high strength. Density andYoung Modulus E (in case of soils also called Modulus
of deformation) are different in each layer. The relation
between CI and E for cohesive and non-cohesive soils
was found from Melnikov,8, by the non-linear regression
equation shown in Eq. (4), based on the soil classes
classification proposed by Robertson,8, using a soil typefunction Ic. From the average CI of the experiments andconsidering a clay soil class, the Young moduli for the 3layers were estimated.
𝐸 = 24,60 − 6,005 × 𝐼𝑐 − 0,5811 × 𝐼𝑐 2 +(1,084 − 1,511 × 𝐼𝑐 + 1,090 × 𝐼𝑐 2 ) × 𝐶𝐼

Fig. 3. Experimental layout.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

503

(4)

Enrico di Maria, Kazuo Ishii

Fig. 7. Contact patch in track mode and wheel mode.

equations and a FEM model were evaluated, and theresults in line with the expectations. The aim of the study
is developing a tool for analyzing the performance ofsuch system by identifying the soil conditions in which
one mode outperforms the other, and finding the bestparameters for the final design, such as track module
dimensions and vehicle weight. After performing otherexperiments in the field, more complex analyses, alsowith multiple passes of the vehicle, will be carried out indifferent soil conditions, with different vehicle weights.

Fig. 6. CI comparison between experiments and FEM model.
At first soil is described as an isotropic elastic medium,as indicated in 9, with a specific Young Modulus andPoisson ratio. With these assumptions the CP test was
simulated by FEM with the Young moduli found by Eq.
(4), slightly adjusted for a better match with theexperimental results, as shown in Fig. 6. DP model wascompleted with the soil data available at 10. Then two
static simulations were carried out to evaluate: soil
deformation, to estimate the rolling resistance, and soil
stress, as a measure of the compaction. Track modules
are considered as stiff bodies and their deformation andcontribute to the total RR is neglected in this study.
5.

Acknowledgements
I thank my friend Daniele Ciccarese for allowing to carryout the experiments in his farm and for his support.
References

Discussion

1. J. Eriksson et al., 1974. The Effect of Soil Compaction on
Soil Structure and Crop Yields. Bullettin 354. Swedish
Institute of Agricultural Engineering, Uppsala. R.
2. W.C. Tim Chamen, et al., Mitigating arable soil
compaction: A review and analysis of available cost andbenefit data, Soil & Tillage Research 146 (2015) 10-15.
3. Sally A. Shoop, Finite Element Modeling of Tire-Soil
Interaction, US Army Corps of Engineers, 2001.
4. J.Y. Wong, Theory of Ground Vehicles, (John Wiley &sons, Inc).
5. G. L. Mason et al., Improved sinkage algorithm for
powered and unpowered wheeled vehicles operating onsand, J. Terramechanics 67 (2016) 25-36.
6. S. Al-Milli et al., Track-terrain modelling andtraversability prediction for tracked vehicles on soft terrain,
J. Terramechanics 47 (2010) 151-160.
7. H. Taghavifar, A. Mardani, Off-road Vehicles Dynamics,(Springer)
8. A.V. Melnikov, G.G. Boldyrev, Correlations between
CPT resistance and Young Modulus of soils, Soil
Mechanics and Foundation Engineering, 2015.
9. P. Defossez, G. Richard, Models of soil compaction due totraffic and their evaluation, Soil & Tillage Research 67(2002) 41-64.
10. www.jsg.utexas.edu, Some Useful Numbers on theEngineering Properties of materials (Geologic andOtherwise).

With a sinkage of 4.3 mm and 13.8 mm, RR for the track
and wheel mode were 43 N and 47,6 N respectively. Withan average CI of 1800 kPa the two RR are almost thesame as expected, and the values lower than those fromempirical equations. In this analysis only the soil
deformation due to a static vertical load is considered,
while the deformation due to shear forces, the slip and thebulldozing effects are neglected.
Further experiments in the field and more complex
analyses are required to improve and refine the modeland consider all the contributes.
Regarding the stress results, in track mode the peak was
13,3 kPa while for wheel mode it was 60 kPa. Values arehigh only for the first layer, while the stress propagatedto the deeper layers decreases rapidly in both cases, andthe difference in the value is not high. As from otherresearch,2, contact pressure affects mainly the first layers
while the load determines the stress reached at the deeper
ones. Contact patch shapes are shown in Fig. 7.
6.

Conclusions and future works

In this paper the preliminary study for the developmentof a locomotion system with an adjustable contact patch
was presented. Soil stress and RR by means of empirical© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan504

Robot Navigation in Forest Management Based on Graph
Ayumu Tominaga, Ryusuke Fujisawa and Eiji Hayashi

Department of Computer Science and Technology, Kyushu Institute of Technology, 680-4,
Iizuka,, Fukukoka, 820-8502, Japan
tominaga@mmcs.mse.kyutech.ac.jp, Fujisawa@mmcs.mse.kyutech.ac.jp, haya@mse.kyutech.ac.jp
Abbe Mowshowitz

Department of Computer Science, The City College of New YorkNew York, NY10031,USA
abbe@cs.ccny.cuny.edu

Abstract

This paper reports on an application of robotics to the management of a trees intended for eventual harvest.
Such management requires periodic cutting extraneous growth around the trees. We proposes the
trajectory computation for guidance of the robot in the forest is carried out based on a weighted undirected
graph G with trees as vertices. The graph to be used in the navigation system is ( ), the line graph of G.
The vertices of ( ) represent the midpoints of the edges of and the midpoints of each tree pair. The( ) would be safety paths for the robot pass through. Our robot was able to follow a part of the trajectorycomputed in experimental forest, thus demonstrating the feasibility of forest maintenance by anautonomous.
Keywords: Forestry Robotics, Autonomous Robot, Graph, Navigation

This paper reports an application of robotics to themaintenance of trees and forest intended for eventual
harvest. The weeding robot our focused is required tohave a mechanism for weeding, and traveling in the forest
without damage to the tree. Navigation is mostchallenging part of this process.
A navigation system is requires 1) localization, 2) path
planning and 3) path-following with obstacle avoidance.
This paper assumes that the localization is achieved byusing GPS, and mentions regard for the path planning
method and the path-following with obstacle avoidance
behavior. In the path planning, we propose using trees
consisting a forest as a landmark. Trees are resources thatthe subjects of management, its location will unchanged
in long period of time. Trees are resources that the
subjects of management, its location will unchanged inlong period of time. Using a SLAM techniques or aidentify locations of each tree by GPS will allow for
utilizing in a navigation system. There are not many

1. Introduction
The application of robotics in management of forest
resources is well-established [1]. Especially,
improvements in Simultaneous Localization and
Mapping (SLAM) techniques allow for the automatic
creation of a precise forest inventory that includes
individual tree locations trunk diameters, and tree height[2,3]. Such these applications have been introduced toincrease productivity, enhance safety and in some cases,
to compensate for the lack of available forest workers.
However, there are not so many discuss the automation
about a work with travelling on a rough terrain.
Particularly, eliminate weed plants work aimed atpromoting the growth of trees must be periodic
performed. Currently, it have been done manually. Suchwork have being placed large labor on the employees, so
it is expected that introducing the autonomous land
vehicle in forest management. [4].

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

505

Ayumu Tominaga, Ryusuke Fujisawa, Eiji Hayashi, Abbe Mowshowitz

changes in short term, so the landmark based map would
be created at low labor. Therefore, in this research, we
made the landmark based map in advance of the robot
entrances into the forest. Therefore，in this research, we
made the landmark based map in advance of the robot
entrances into the forest. The map contains only position
information of each tree without other objects (i.e. rocks
or any obstacles). The path planning is performed by agraph consisted by the individual trees as vertices. We
have been developed the robot having reactive pathfollowing with obstacle avoidance behavior system. We
show how it can be done and report on an experimentusing our method in the setting of a small experimental
forest located on the campus of the Kyushu Institute ofTechnology.

would allow for simplifying the computation of therobot’s trajectory. We created a set of graphs containing
several trees as vertices, and created a high-level graphconsisted these as vertices, as shown in Fig.1.

2. Computing the trajectory based on graph

3. Forestry Robotic System “SOMA”

Computation of a trajectory in a forest to make it follow
the robot would be facilitated by treating the tree asvertices in a graph. Trees can be represented by a graph= ( , ), where is the set of vertices and is the setof edges [5]．A vertex ∈ represents an individual
joining vertices andtree, and and edge ∈represents a path between trees．The graph used in the
robot’s trajectory is ( ), the line graph of . A line
graph is a graph having vertices that correspond one-toone to edges in G, and connected two vertices of line
graph corresponding to adjacent two edges [6] ． In
constructing the trajectory, we superimposed the naturalgeometry over the line graph, considering the position of
the vertices of L(G) to be the midpoints of the edges of
G．If the case of G with an Eulerian property, all vertices
in G have even degrees, and an Euler circuit can be found.
In addition, a Hamilton circuit can be used to represent
the vertices in the line graph corresponding to the orderto follow the edges in the Eulerian graph [6, 7].
To ensure the existence of Hamiltonian circuit in ( ),the graph will be made to be Eulerian. Any line of sightpath between two trees can define an edge of so there
are sufficiently many choices of edges to allow for
obtaining an Eulerian graph. Every pair of trees between
which there is an unobstructed path in the forest thatwould allow the robot to navigate safety could contributean edge.
In large scale forest applications it might be advisable
to divide the trees into sections and treat each one
separately. Criteria for subdividing the forest would
naturally include section sizes, location of robot staging
areas, and terrain issues such as steepness and presenceof potential obstacles (e.g., rocks). For example, there
were 3000 trees in the forest, it could be subdivided into
150 clusters each containing 20 trees. Such a subdivision

The autonomous robot “SOMA” shown in Fig.2 is built
on the platform of an All-Terrain Vehicle (ATV). An
ATV was chosen for its robust design for outdoor use inrough terrain. The power source is 90cc gasoline engine.
The throttle, brake lever and steering are controlled by
DC motors to propel the robot. The weeding operation isrealized by a specialized mechanism attached to the frontof the ATV. Included sensors are RTK-GPS for
localization，IMU in order to measurement the robot’s
heading，rotary encoder to measure the velocity，and
RGB-D camera is to recognize an surrounding externalsituation．The final dimensions of the robot are 0.9×1.4
× 1.2m (Width × Length × Height). It is sufficiently
smaller than a standard planting interval (2.5m), andallows for the robot to enter the interior of a forest.

Fig. 1. Concept of two layered graph structure fortrajectory in large forest

Fig. 2. Forestry robotic system “SOMA”© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan506

Robot Navigation in Forest

Fig. 3. System block diagram

4.

Autonomous Moving System
The overview of the system is shown in Fig.3. In thisresearch, the robot moves on two dimensional plane,
and its state vector in the discrete time is represented= [ , , ]. Here, , represent a locationbyof the robot obtained by RTK-GPS, andis therobot’s heading measured by IMU. The control input
is represented by = [ , ], is a steering angle
and is a moving velocity．Each actuator control
systems are built on another CPU in order to improve
development efficiency. The navigation system isconsisted by three subsystems as follows.
External situation recognition
Search secure trajectories
Determine the control input for path following
Each subsystem are described below.

Fig. 5. Search secure trajectory based on State SpaceSampling

each point, finally, obstacle boundaries approximated torectangular are obtained.
4.2. Search secure trajectories based on StateSpace Sampling

4.1. External Situation Recognition
The robot recognizes the external situation via thepoint cloud obtained from the RGB-D camera. We areusing R200 (Intel Corporation) for our vision system.
Figure 4 shows the procedure from the point cloud to the
obstacle recognition around the robot. First, after
acquiring the point cloud, performing the noise remove
processes. Then each three-dimensional point is
projected on the xz plane. Next, the clustering process ofpoints is performed based on euclidean distance between

After recognition the external situation, searching secure
trajectories capable avoiding collision with obstacles isperformed based on State Space Sampling method. StateSpace Sampling method creates a set of terminal statevector on an arc around the robot, and a set of control
input allowing for reach to these state vectors are also
computed [10]. The searching space of it is defined by ashape parameter. The details of the procedure ofsampling the terminal state vectors andare see [8].
In this research, as shown in Fig.5, The set of control= [ , , … , ] that ensured safety is
input
extracted via applying State Space Sampling to the local
coordinate of the robot.
4.3. Determine the control input for pathfollowing
Pure Pursuit algorithm [9] is used make the robot follow
the desired path based on graph described in Section 2.
In each discrete time, the destination point is set on anedge in desired paths. The control input
which theFig. 4 External situation recognition procedure© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

507

Ayumu Tominaga, Ryusuke Fujisawa, Eiji Hayashi, Abbe Mowshowitz

robot could follow the path is obtained, however it is notalways output because of the obstacle avoidance
behavior is high priority more than path-following. In
many situations, obstacles exist surrounding the robot.
Therefore, the robot should be took detour and return topaths. In the system, the final control input to propel therobot is determined by the cost function shown in Eq. (1)and (2).
Fig. 6. Desired graphs=,

5. Experimental results

=

,

|

−

∈

(1)
(2)

The experimental forest on the campus of the Kyushu
Institute of Technology consists of 39 trees spread over a
35×35 meter square. There are no objects other than trees.
The maximum degree of slope is about 5°.
Based on these dimensions and the width of the path
between trees, an undirected graph was developed, as
shown in Fig.6. The adjacencies represent pairs of trees
at a line-of-sight distance from each other. We createdthree graphs using by trees having label shown in Table.1
as for graphs of lower layers.

Fig. 7. The experimental results

References
1. R. Parker, K. Bayne, and P. W. Clinton, “Robotics inForestry”, New Zealand Journal of Forestry, No.60, pp.814, 2016.
2. J. Tang, et al., “SLAM-Aided Stem Mapping for Forest
Inventory with Small-Footprint Mobile Li-DAR”, Forests,
Vol.6, No.12, pp.4588-4606, 2015.
3. T. Ritter, et al., “Automatic Mapping of Forest Stands
Based on Three-Dimensional Point Clouds Derived fromTerrestrial Laser-Scanning”, Forests, Vol.8, No.8, pp.1-19,
2017.
4. T. Tatsuo, “Recent Research Issues in Forestry Robotics”,
JRSJ, Vol.12, No.7, 1994.
5. F. Harary, “Graph Theory”, Reading MA:AddisonWeslay, 1969.
6. F. Fujie and P. Zhang, “Line Graphs and Power ofGraphs”, in Covering wlks in graphs, Springer, 2014.
7. S. Skiena, “Line Graph”, in Implementing Discreate
Mathematics: Combinatorics and Graph Theory with
Mathematica, Addison-Wesley, 1990.
8. T. M. Howard, C. J. Green, and A. Kelly, “State SpaceSampling of Feasible Motions for High-Performance
Mobile Robot Navigation in Complex Environments”,
Journal of Field Robotics, Vol.25, pp.325-345, 2008.
9. M. Samuel, M. Hussein and M. B. Mohamad, “A Review
of Some Pure-Pursuit Based Path Tracking Techniques forControl of Autonomous Vehicle”, International Jounal ofComputer Applications, Vol.135, No.1, 2016.

Table 1. Trees used in each graph
Graph
1
2
3

Used trees
22,23,25,24,29
33,38,39,40,44
27,34,35,36,37

As shown the experimental results (Fig.8), the robot
traverses around each graph in low layer, and performed
transition between the graph and adjacent graph of its.
These suggests it is feasible The result suggest it isfeasible that the robot will be able to traverse a whole offorest by subdividing to small scale region and based onreactive motion control.
6. Conclusion
In this paper reported we have demonstrated thefeasibility of designing an autonomous forestry robot toservice trees meant for eventual harvest. A paths for anautonomous robot to follow can be computed based ongraph from initial geographic map of trees in a forest. In
the future work, we will improve the accuracy of pathfollowing to decline the error between trajectories anddesired paths.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan508

Development of Autonomous Moving System for Field Robot
Kengo Kawazoe, Ryusuke Fujisawa, Eiji Hayashi

Department of Mechanical Information Science and Technology, Kyushu Institute of Technology
680-4, Kawazu, Iizuka-City, Fukuoka, 820-8502, Japan
E-mail: kawazoe@mmcs.mse.kyutech.ac.jp, fujisawa@ces.kyutech.ac.jp, haya@mse.kyutech.ac.jp
Abstract
Forestry go into a decline by labor shortage in Japan. We have been developing autonomous moving robot with allterrain vehicles (ATV). And we designed an obstacle avoidance behavior to move around in the forest. Moreover,by adding a pan angle control actuator to the visual system, we increased the degree of freedom in external situation
recognition. This made it possible the robot to the target coordinates specified on the global map based on the obstacle avoidance behavior.

Keywords: Autonomous Moving System, Forest

Introduction

The Robot Platform

SLAM (Simultaneous Localization and Mapping) technology which simultaneously performs self-location estimation and construction of environmental map is applied
to resource management for forestry activity in artificial
forest[1][2]. As a result, the mapping of the trees in theartificial forest and the automation of the growth amountmeasurement of the trees have been realized. However,the development of a robot that make it possible to circulate inside an artificial forest instead of a worker by utilizing such a technique has not advanced. This is due tothe fact that artificial forests are developed in steep middle mountainous areas, and their planting density is also
high[3]. On the other hand, in the forestry industry in Japan, the decline of forestry due to labors has decreased,
and mechanization and automation of forestry work arerequired.
In this research, we developed an autonomous moving
robot with platform of all-terrain vehicle with high moving ability in rough terrain. With this robot, we aim toautomate the planting work and resource management
within the forest. We designed an obstacle avoidance behavior to move around in the forest. Moreover, by adding
a pan angle control actuator to the visual system, we increased the degree of freedom in external environment
recognition. The robot achieves to guide to the target on
the global map, and to cover the work area with the obstacle avoidance behavior by this function and mechanism.

Fig. 1. Outline of robot appearance

Figure 1 shows the robot developed in this study. We will
operate the robot on a steep slope in the actual mountainhillside, and the road is rough. Thus, the robot is basedon an ATV (Kawasaki, Inc.) to handle on the rough terrain. The size of robot is following; length: 1400 mm,
width: 900 mm, height: 1200 mm. In the Japanese forestry work, the planting interval is approximately 2.5 m,
which is narrow compared to forestry regulations in othercountries. As shown in Fig. 1, the size of our developed© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan509

Kengo Kawazoe, Eiji Hayashi, Ryusuke Fujisawa

RealSence R200, and filtering processing is performedon the point cloud. Next, three-dimensional coordinatetransformation is performed to make a top-down view
image. By labeling processing on this top-down view image, the size and position of each object in front of therobot is recognized. Then, a distance between each object
is calculated. Let the midpoint of this segment be the target point of the robot at time t. Finally, steering angle isobtained, and actuator control is performed. Filtering
processing and target point determination are describedas following.

robot is sufﬁciently small compared to the planting interval that it can easily travel into the native forest. The ro-3.1. Filtering Process
By filtering processing on the obtained three-dimensional
point cloud data, the obstacle is recognized accurately.
The plane detection as a ground surface and removal on
three-dimensional point cloud data was implemented using RANSAC (Random Sampling Consensus) algorithm.
Fig. 3 shows raw data obtained from RealSence R200,
and the result after the plane removal.
By the plane filtering processing, it is possible to remove the point cloud data of the robot ground contact
surface and to extract only the point cloud data of the obstacle at every moment.

Fig. 2. Outline of obstacle avoidance

bot has four wheels, the front two wheels are turning andthe rear two wheels are driving. The external environment is recognized by RealSence R200 (Intel, Inc.) whichis constructed with a depth sensor and a RGB-CCD camera, and equipped on the ATV. The depth sensor can acquire an object’s position as three-dimensional (3D)
point-cloud data via an infrared laser measurement. The
maximum measurable depth direction of the sensor is 15
m. In addition, the robot has an inertial measurement unit
(IMU) for pose detection and RTK-GPS for global localization. The power source is gasoline engine. The brakelever, accelerator lever, and steering are all controlled by
DC motors. A weeding task is performed by a weeding

3.2. Determination of Target Point
There are a lot of obstacles in traveling in the forest, therobot needs to travel like sewing between obstacles andtravel with straightness. Fig. 4 shows the flow of processing for determining the target point.

Fig. 3. Raw point cloud data (left) and after removing plane(right)Fig. 4. Flow-chart of processing for determining target pointmechanism that has multi blades and attached on thefront of the ATV.
External Environment
Obstacle Avoidance

Recognition

First, we calculate the midpoint between obstacles
based on the results after labeling processing in the topdown view shown in Fig. 2. Next, the route of the robot
to the midpoint of the obstacle is determined, and a collision with an obstacle is judged. When there are multiple
midpoints that can go through, the target point is the midpoint with the shortest distance between the robot and theandOutline of obstacle avoidance is shown in Fig. 2. First,
three-dimensional point cloud data is acquired from the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

510

Development of Autonomous Moving

midpoint. When the robot cannot go through between all
obstacles, it sets a target point to the wider space which
is selected the left or right of the nearest object. As a process after determining the target point, there is determination of the target point and turning point and the travel
route of the robot are shown in Fig. 5.
If an operator has a destination on the global map, theoperator cannot go to the final destination simply by going around avoiding obstacles. Thus, it is necessary to approximate the attitude angle of the robot to the current
attitude angle of the robot. As a result, obstacle avoidancewith straight traveling is accomplished. The target point
coordinate is ( , ), the turning point coordinate is( , ), and , is obtained by using the equation (1,2) .
= ⁄2.0(1)= ⁄2.0(2)

Fig. 5. Determination of target point and turning point andtravel route of robot

Steering angle for heading to the turning point, travel
distance . The rotary encoder judges arrival at the turning point. After arrival of the turning point, the robot runs
to the target point.
Moving Method in Work Area

The robot goes outside the work area because it justavoids obstacles. As a solution to this problem, we set up
a work area using RTK-GPS data and developed a system
towards the center point within that area using IMU and
RTK-GPS when a robot comes out from that area.
For determination of whether the robot is located within
the area or outside the area, the immediate result of RTKGPS is used. First, a rectangular work area is set. Whenexisting from the work area, control the pan rotation angle of the sensor so that the field of vision of the sensor
faces the direction closer to the directional data of the robot from the IMU, the center point in the work area andthe robot. By orienting the visual field range of the sensor
toward the center point side of the work area, and as aresult, the robot can return to the work space.

Fig. 6. Obstacle avoidanceFig. 7. Obstacle avoidance in work area

Experiment
For the experiment of the autonomous moving system,
we carried out the robot autonomously at the experimentsite in our university. A map showing the position of thetrees was prepared beforehand in the experimental fieldof 35 m2 and the x and y coordinates of each tree were
measured by RTK-GPS. Fig. 6 shows the travel route of
the robot at each time obtained from RTK-GPS. The result of autonomous movement with work area is shownin Fig. 7. A rectangle in the map indicates a work area.

Conclusion & Future work
In this study, we constructed an obstacle avoidance behavior based on a lot of obstacle information in autonomous moving robot that do weeding and investigate in
the forest.
With the constructed obstacle avoidance behavior, therobot did not proceed to escape to the outside of the experiment site, resulting in running with improved© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

511

Kengo Kawazoe, Eiji Hayashi, Ryusuke Fujisawa

straightness. In addition, when the robot is located outside the work area, it is possible to return to the work area.
In the future, when we reach the target point we will improve the method of determining the route to the target
point to control the posture of the robot.
References
[1]

[2]

[3]
[4]

H.-G. Maas, A. Bienert, S. Scheller, and E. Keane, "Automatic forest inventory parameter determination fromterrestrial laser scanner data," International Journal of
Remote Sensing, 29 (5), pp.1579-1593, 2008.
G. Brolly, and G. Király,. "Algorithms for stem mappingby means of terrestrial laser scanning," Acta Silv. Lign.
Hung. 5, pp.119-130, 2009.
新エネルギー・産業技術総合開発機構, “ロボット白書 2014”, p5-38, 2014.
Martin A. Fischler, and Robert C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to image Analysis and Automated Cartography”,
ACM, Vol. 24, No. 6. (June 1981), pp. 381-395© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

512

Development of the Sense System That Is Combined Force and Vision Feedback-Improvement Reproducibility of Deformation Simulation by Using LEMTamon Shigeyama, Ryusuke Fujisawa, Eiji Hayashi

Department of Mechanical Information
Science and Technology
Kyushu Institute of Technology
680-4, Iizuka City, Fukuoka, Japan, 820-8502
E-mail: shigeyama@mmcs.mse.kyutech.ac.jp, fujisawa@ces.kyutech.ac.jp, haya@mse.kyutech.ac.jp

Abstract
The purpose of this research is to develop a combined senses system that uses both force and visual feedback on a
deformation simulation. Especially, focus on a way to produce a force that is calculated in simulation of deformation
using a haptic device. As the first stage, our simulation of deformation was improved by using LEM (Long Element
Method) and evaluated accuracy of a virtual object’s deformation.
Keywords: force feedback, haptic interface, simulation, LEM

calculated by the spring-mass-damper model in previous
studies.1 However, it is difficult to construct the model ofa sample object with complicated shape, and measurespring-damper coefficient between mass points. Thus,
the modeling of an object is performed by LEM(LongElement Method)2. LEM is an object filled withincompressible fluid is modeled based on Pascal’s
principle and volume preservation and deriving
deformation results of semi-static state. LEM suitable forreproducing deformation of living tissue like surgicalsimulation. As the first step, we implemented simulation

1. Introduction
In the medical treatment and bio-technology field,
doctors and researchers need technologies that canaccurately perform microscale works, like a cell injection.
But there are some problems, the operators rely only onsight through the microscope, and cannot feel the senseof forces. And the operators cannot see the all states ofthe samples. As a result, the operators have to work for along time, and need expert skills for microscale works.
With the improvement of virtual reality technology
and development of haptic devices, we can obtain haptic
and visual of virtual object on computers. And
implementation of force and visual feedback can improve
efficiency and reproducibility in microscopic operations.
Our aim is to develop the combined system that uses
both force feedback from the manipulator and visualfeedback on a cell’s deformation simulation. By
reproducing force and visual sense of a sample, anoperator can perceive these sense of a virtual object. For
this research, we focus on a way to produce a force whichcalculated in simulation of deformation using a haptic
device. Deformable virtual objects simulation, model ofa sample object and reproduction of the deformation were

Fig. 1 The system structure© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

513

Tamon Shigeyama, Ryusuke Fujisawa, Eiji Hayashi

method which compute deformation of samples using
LEM and evaluate accuracy of a virtual object’s
deformation.

magnet. When electric current flows through the coil, thecoil generates a Lorentz force which towards according
to Fleming's left hand rule. The rotor moves depend on a
waveform input, less than 36 Hz. An operator holds amoving part like a pen and receives the vertical force.
Hereinafter, the moving part is called a pen slide. Figure
3 shows the connection between the haptic device and a
microcomputer. The haptic device is controlled bycommand signals from the microcomputer process.

2. System Structure
Figure 1 shows the system structure of our research. The
main computer also sends parameters to the haptic device.
The parameters are obtained by measuring dynamic
characteristics of a sample using microscope. Thissystem consists of a force feedback part and a simulated
virtual visual feedback part. At a force feedback part, thissystem is equipped with the haptic device and a
microcomputer for controlling the haptic device. At asimulated virtual visual feedback part, it shows asimulated video from a main computer.

4.

Simulation Interface

In this study, we integrate the working system using a
microscope, a haptic device and a simulation. A
fundamental element was simulating the deformation ofsample object. Figure 4 shows the GUI of theimplemented simulator. A sample object model isrendered using OpenGL.

3. Haptic Device
Figure 2 shows a diagram of the haptic device. It consists
a rotor, a laser module, and a PSD module. The angle of
the rotor can be measured by the laser and the PSD. The
haptic device has a coil on the rotor with a polarity
Top view

Side view

Magnets
Microcomputer
OUTPUT

DAC

AmplifierHapticdeviceCutoff Frequency
1000Hz

PSD
Laser

Rotor
Arm
Pen slideFig.2. The structure of the haptic device

Microcomputer
INPUT

ADC

LPF

PSD

Fig. 3 The connection between thehaptic device and a microcomputer

Fig.5. LEM modelFig.4. Simulator

Fig.6. 3-dimensional LEM model© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

514

Development of the Sense

5. Modeling by Using LEM
The deformation of the object is also reproduced using
LEM. Fig. 5 shows a method of reproducing thedeformation. The object is divided into LE(LongElement), and the surface shape is reproduced by the setof deformation of each LE. Each LE deforms only in one
axis direction. Modeling is divided by LE for eachsurface to be deformed. In 3-dimensional deformation,
divide 5 faces by LE. Fig. 6 shows the 3-dimensional
deformation.

7. Evaluation Experiment
We experimented to evaluate the simulation. We
deformed the gel by stick’s pushing the top surface andmeasured the displacement of side surface. We extracted
deformation by image processing software formeasurement. Figure 8 shows an example of theobserved image. Figure 9 and Figure 10 show measured
and simulation values (stick diameter 10 mm and 18 mm).
These values were compared and the simulation was
evaluated.

6. Deformation simulation
We reproduced the deformation of silicon gel. Table 1
shows the size of the silicon gel (Taika α gel ® Co., Ltd.)and the necessary physical properties for modeling.
Figure 7 shows the results of the deformation simulation.
Table 1 Parameter of the silicon gel
Parameter
Explanation
Value
Size[mm]Width×Depth×
20×20×Height10
E[kPa]
Elasticity
28.9
d[kg/ ]
Density980(a)Front view(b)Bottom viewFig.8 Deformation of the silicon gel
1mm
2mm
3mm
4mm
5mm
6mm
7mm

5

20
15

Width[mm]Height[mm]Simulation

10

0

0.5

10

1

1.5

Displacement[mm]

2

0

2.5

0.5

1

1.5

2

2.5

2

2.5

Displacement[mm]2015Width[mm]Height[mm]500Measurement

10

5

0
0

0.5

1

1.5

Displacement[mm]From Side

2

10
5
0

2.5

0

0.5

Fig.9 Experiment result (Diameter 10mm)1

1.5

Displacement[mm]From Bottom© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

515

Tamon Shigeyama, Ryusuke Fujisawa, Eiji Hayashi

20

Height[mm]Simulation

1mm
2mm
3mm

5

4mm
5mm

15

Width[mm]

10

10

0
0

1

2

Displacement[mm]

0

3

0

1

2

3

1

2

3

Displacement[mm]2015Width[mm]Height[mm]10Measurement

5

5

10
5
0

0
0

1

2

Displacement[mm]From Side03

Fig.10 Experiment result (Diameter 18mm)

Displacement[mm]From Bottom

with the haptic device. Such a system would make itpossible to test smaller samples easily.

8. Discussion
The comparison result is different depending on the
stick diameter. The stick diameter is 10 mm, the edge
is drawn inward by pushing the top side. But the stick
diameter is 18 mm, the area which the top surface ispushed is large so that it isn’t drawn inward. In themeasurement result of From Side, the displacementof top surface’s edge is smaller than other points, andthe peak value is lower than middle. However, thesimulation value has no tendency and it can’t be
reproduced.
Moreover, the results of From Bottom,
measurement value of edge’s displacement is smaller
than simulation. This is due to the gel deformation ofedge is deformed in the oblique direction, but thisdeformation can’t be reproduced in LEM.
From these facts, it was found that LEM simulation
deformation of the edge is a problem.

References
1. T.OKADA, Re-creation of a membrane puncture’s senseof an object constituted of liquid and an outer membrane
by a haptic device and a deformation simulation of thevirtual objects (International Conference on Artificial Lifeand Robotics (ICAROB2017), Jan 19-22, Miyazaki, Japan,2017).
2. Costa IF, Balaniuk R, LEM-An Approach for Real Time
Physically Based Soft Tissue Simulation (Proc of IEEEICRA 2001, Korea, pp. 2337-2343, 2011).

9. Conclusion
In this study, modeling an object and reproducing thedeformation of a sample is performed using by LEM.
The deformation of silicon gel was reproduced. And we
conducted an evaluation experiment to evaluate thesimulation. Since the error is large, it is necessary to
improve it.
Future research should focus on calculating force
from equations of LEM and generating calculated force© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

516

System Development of AUV’s Sampling Device Controller
Employing MATLAB/Simulink Toolboxes
Takashi Sonoda*

Faculty of Engineeing, Nishinihon Institute of Technology, 1-11, Aratsu, Kanda-machi Miyako-gun,
Fukuoka, 800-0344, Japan
Shinsuke Yasukawa, Jonghyun Ahn, Yuya Nishida, Kazuo Ishii
Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: sonoda@nishitech.ac.jp, s-yasukawa@brain.kyutech.ac.jp, ahn@brain.kyutech.ac.jp,
ynishida@lsse.kyutech.ac.jp, ishii@brain.kyutech.ac.jp

Abstract
We developed an AUV’s sampling device control system. It is employed the any toolboxes provided in
MATLAB/Simulink by Mathworks. The toolboxes are powerful, useful, and functional. “Robotics System Toolbox”
in theirs supports the ROS middle ware. It provides the cloud data communication. “Parallel Computing Toolbox”
can struct a multi process and parallel computing on script base and Simulink model base program. “Stateflow” is a
powerful tool for design of state transition control method. The toolboxes are employed in the sampling device control
system of the AUV. This method with using the toolboxes of MATLAB/Simulink realize the rapid prototyping of
the system designs, developments and system test. The developed system mounted on an AUV was powered inseveral real sea areas. Report on the results and problems when operating in real environments.
Keywords: System development, AUV, Sampling device.

1. Introduction
We have developed a sampling device that can bemounted on the AUV (Autonomous Underwater Vehicle)
in order to accomplish sampling missions of livings on
the ocean floor1-5. In order to operate it, we need toassume a collision with the undersea surface and various
unexpected situations. For robots working in anunderwater environment, the development of a systemthat controls the robot and some devices is very important.
In development of the system, it is necessary to consider
not only safety and certainty but also development speed,
reusability and maintainability. Our development teamhas challenged system development based on MATLAB/ Simulink. The toolboxes provided in MATLAB /Simulink are powerful, useful, and functional. In addition,“Robotics System Toolbox” in theirs supports the ROS
middle ware. ROS provides many software which are
Fig. 1. Photograph of AUV “Tuna-Sand2” with the sampling
device (manipulator type).
*

Corresponding author© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

517

Takashi Sonoda, Shinsuke Yasukawa, Jonghyun Ahn, Yuya Nishida, Kazuo Ishii.

Fig. 2. The 2nd type of the sampling device (Slurp-gun andmanipulator type) on the AUV.

Fig. 3. The 3rd type of the sampling device (Slurp-gun type)on the AUV.

regarding to robotics such as navigation, manipulation,
machine vision etc. It is possible to achieve both system
extensibility and development speed by linking
MATLAB / Simulink toolboxes and ROS.
We applied this system development method to thecontrol system of the sampling device mounted on the
AUV "Tuna-Sand 2" (see Fig. 1). The developed system
was mounted on the AUV which operated in several realsea areas. In this paper, we discuss about the results andproblems when operating in real environments.
2. Sampling device structure
There are several types of the sampling device
developed. Fig. 1 is “Manipulator type” of the sampling
device. Fig. 2 is “Slurp-gun and manipulator type”. Fig.
3 is “Slurp-gun type”. Fig. 4 shows the schema diagramof the sampling device. The control computer is built into
the control hull.

Fig. 4. Schema of the sampling device. The device has thecontroller hull including a computer and microcomputers. And
there are hydraulic actuation system and a slurp-gun module.

opening of joint angle. For the control system,
monitoring lock status is very important. The type is very
difficult to catch the livings.

2.1. Manipulator typeIn this type, it has a 3 DOF manipulator with a robotic
hand. The joints are actuated by hydraulic cylinders3.
The hydraulic system is structed by 2 types of cylinder.
The slave cylinder is pushed and pulled by the mastercylinder. The joint angles are measured by magnetic
encoders. However, there are no slurp-gun. The
manipulator was designed for underwater conditions only.
To launching, the manipulator must lock to avoid

2.2. Slurp-gun and manipulator typeIn this type, it has the manipulator and a slurp-gun
module. The slurp gun module is constructed by a suction
hose, a canister and a suction device. Sampling bysuction is easier than hand. However, this type requires along suction hose. The suction performance of this is notefficient.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

518

1st simulink model

AUV

Action planning system

Communication
to AUV main sys.

ROS topics
Computervision system

CameraAcoustic data Manipulatortransmission + Slurp-gun
control sys.
systemAcoustic
modem

flow chart editing functions. “Stateflow” is a powerfultool for design of state transition control method. The
model controls and monitor the action states.
3.2. 2nd Simulink model (Computer vision)
3rd simulink model

2nd simulink model

System Development of AUV’s

The 2nd Simulink model is connected to a GIGE visionsystem. In the model, computer vision system recognizes
and trach some living objects on underwater surface.
Data related to the captured object is transmitted as amessage of the ROS topic.
3.3. 3rd Simulink model (Commuication to AUV,
Acoustic data transmission, Manipulator andslurp-gun control)

Manipulator+ Slurp-gun

This Simulink model controls the hardware devices(AUV, Acoustic modem, manipulator and slurp-gun)other than the vision camera. There is the communication
block to AUV main system with TCP/IP. Acoustic
modem for data transmission is employed to send image
taken by vision system. The manipulator and the slurpgun connected to a microcomputer and two motor drivers
with serial communications.

Fig. 5. The system diagram of the sampling device. The system
has three Simulink models which are separated by category ofworks. Each model can communicate by ROS blocks which aresupported in Robotics System Toolbox.

2.3. Slurp-gun typeIn this type, it has the slurp-gun only. The suction hose
is devised so that it can also be used as a canister. Thisconfiguration realizes a strong suction force. However, it
is difficult to control while fine-tuning the position of thetarget aimed at.

4.

System operation in real environments

We operated this system in several real sea
environments (Okinawa, Shizuoka, Hokkaido etc.).
4.1. Data visualization

3. System structure

We developed a data visualization model using
Simulink. Fig. 6 shows the log file viewer using the log
file when operated in real environment at Hokkaido offsea. Simulink can save time-series log data. The viewer
can load the time-series log data. Because we can’t use
communication by radio waves in underwater situation,
visualization tools that easy to understand is important.
However, as a problem, the log file is generated
independently from each model and the start time isdifferent. Therefore, it takes time and effort to adjust thestart time of the time.

Fig. 5 shows the system diagram of the sampling device.
The sampling device is optional equipment for the AUV.
It can’t control the AUV and it does not have control
authority. The system of the sampling device can justsend any request commands to the main system of AUV.
The system has three Simulink models that run parallel.
“Parallel Computing Toolbox” can struct a multi process
and parallel computing on script base and Simulink
model base program. Each model can communicate by
ROS topics. It provides the cloud data communication on
ethernet in the AUV. We developed the system based on
Simulink that is visual programing. This can easily be
expressed as a Simulink software component.

4.2. Flexibility to change missions
In a real environment, the situation changes frommoment to moment. When changing the configuration of
the system, Simulink makes it easy to avoid mistakes
because it can be changed on a block basis. It is the same
when developing software components. Since the Action

3.1. 1st Simulink model (Action plan)
The 1st Simulink model has an action plan system using“Stateflow” toolbox that provides state transition and© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

519

Takashi Sonoda, Shinsuke Yasukawa, Jonghyun Ahn, Yuya Nishida, Kazuo Ishii.

Fig. 6. Developed log file viewer for underwater cruises of AUV with sampling mission. In a deep sea, we can’t see the states of the
AUV and sampling device easily. Data visualization is important in term of understanding the behavior of systems in underwater. This
viewer has the 4 view windows. (The upper right is camera view. The red line in the view shows a control direction and amplitude. The
bottom right is the sampling target view found by machine vision system, the yellow circle is the target. The upper left is the locationand trajectory view from surface of sea. The bottom left is the zoom view by the 3D model.)References

planning system can be executed separately from othersystems, offline checking of mission tasks is also
possible. However, as a matter of value, you need to open
the block with the value of the property, and you need tobe careful when changing property values not describedin the value change script.
5.

1. J. Ahn, S. Yasukawa, T. Sonoda, Y. Nishida, K. Ishii, T.
Ura: Image Enhancement and Compression of Deep-Sea
Floor Image for Acoustic Transmission, in in Proc.
OCEANS 2016 MTS/IEEE (2016).
2. Y. Nishida, S. Yasukawa, T. Sonoda, J. Ahn, K. Ishii, T.
Ura: Development of an Autonomous Underwater Vehicle
with Human-Aware Robot Navigation, in Proc. OCEANS
2016 MTS/IEEE (2016), pp.1-4
3. K. Ishii, A. A. F. Nassiraei, I. Godler, T. Sonoda, T.
Weerakoon: Development of a Hydraulic Underwater
Manipulator for Deep-Sea Survey AUV, The 2018
International Conferrence on Artificial ALife and Robotics,
OS7-1 (2018).
4. K. Ishii, T. Sonoda A. Nishjima, K. Watanabe: Development of End-effector for Sampling-AUV “TUNASAND2”, The 2018 International Conference on Artificial
Life and Robotics, OS7-6 (2018).
5. J. Ahn, S. Yasukawa, T. Sonoda, T. Ura, K. Ishii:
Enhancement of deep-sea floor images obtained by anunderwater vehicle and its evaluation by crab recognition,
in Journal of Marine Science and Technology (2017), pp
1–13.

Conclusions

I explained the sampling device I developed. In thissystem, it is based on MATLAB / Simulink and can beextended by ROS. As a result, it became a system thatcan be operated under the actual environment. However,
initial cost and delays in sampling time are future tasks.
Acknowledgements
This research is supported by JST CREST“Establishment of core technology for the preservation
and regeneration of marine biodiversity andecosystems”.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan520

Inspection System for Underwater Structure of Bridge Pier
Takumi Ueda, Hiyoyasu Hirai, Kazuki Fuchigami, Ryoma Yuki, Ahn Jonghyun,
Shinsuke Yasukawa, Yuya Nishida, Kazuo Ishii

Dept. of Human Intelligence Systems, Kyusyu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, 808-0196, Japan
Email: {ueda.takumi964, hirai.hiroyasu467}@mail.kyutech.jp
ynishida@lsse.kyutech.ac.jp, {ahn, s-yasukawa, ishii}@brain.kyutech.ac.jp
Takashi Sonoda

Faculty of Engineering, Nishinippon Institute of Technology
1-11 Niizu, Kanda-cho, Kyoto-gun, Fukuoka, 800-0394, Japan
Email: sonoda@nishitech.ac.jp
Katsunori Higashi, Katsunori Tanaka, Tomomasa Ikeda

West Nippon Expressway Engineering Kyushu, Japan
1-2-22 Maizuru, Chuo-ku, Fukuoka, Fukuoka, 810-0073, Japan
Email: {k.higashi.a, k.tanaka.a, t.ikeda.a}@w-e-kyushu.co.jp

Abstract
Inspection of underwater infrastructure of bridge pier is important for bridge maintenance, however, the underwater
environment is dangerous and their observation is one of difficult problems. We have been developing the underwaterinspection system which consists of an unmanned surface vehicle (USV) and a remotely operated vehicle (ROV).
The concepts of the system are (1) Operation is executable with a small number of persons, (2) USV and ROV are
controllable from land or bridge, (3) Build 3D model and images for crack detection. The ROV observes the statusof whole infrastructure using cameras and control its posture using a depth sensor and IMU sensor. The USV
vertically lifts up and down ROV by winch and carries ROV to the infrastructure. USV has GPS, LRF and IMU forhorizontal positioning and pan-tilt control of the camera. In this paper, we discuss the results of the inspection
operation test by the robot.

Keywords: Infrastructure inspection, ROV, USV,

Especially the inspection of underwater parts of theinfrastructure facilities are not conducted as thedangerous area, or done by divers in the emergency cases.
However, divers cannot work for a long period of time
due to the high risk which in fact is critical to theefficiency of the work. It is difficult to continuously
observe a specific damaged portion in order to predict
degradation. Therefore, inspection of underwater
infrastructure facilities using underwater robots has been
attracting attention as one of new methods[2-6].
One of the difficulties is position estimation of theunderwater robot due to the unavailability of GPS(Global Positioning System). Most of self-localization
method use acoustic positioning. In the operating around

1. Introduction
Most of Japan’s infrastructure were built during the higheconomic growth period, which brings a concern of aging
problems of the structures to be over 50 years within thenext 20 years. In order to extend the lifetime ofinfrastructure facilities, it is important to understand thecurrent damages and the deterioration progress byinspections. Then, it is effective to decide the repair
countermeasure method from future prediction of
degradation progression estimated from the inspection
data[1].

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

521

Takumi Ueda, Hiyoyasu Hirai, Kazuki Fuchigami, Ryoma Yuki, Ahn Jonghyun, Shinsuke Yasukawa, Yuya Nishida, Kazuo Ishii, Takashi Sonoda,
Katsunori Higashi, Katsunori Tanaka, Tomomasa Ikeda

underwater infrastructure facilities, acoustic positioning
becomes difficult due to the sound reflection problems
such as multi-path and water surface.
We introduce a system that involves an ROV (Remotely
Operated Vehicle) for underwater survey and inspection
and an USV (Unmanned Surface vehicle) for the ROVtransport towards the underwater infrastructure, which isequipped with GPS and LRF for self-positioning
estimation.
In this paper, we report on the results of the operation
confirmation test and current problems.

1

Start Up
Infrastructure

Inspection

GroundStart Up
Return2. Operation and required specification

2

3

Fig.1 Flow of Infrastructure inspection

The underwater infrastructure inspection robot system
proposed in this paper consists of a ROV for camera
observation attached to USV acting as an auxiliary boatthat guides ROV towards the infrastructure and isequipped with remote control system.
Following is the development concept of the system.
Minimum number of operator required
Ability to generate panorama image withabnormality detection information attached
Ability to send data to an operator on site
The flow of inspection using the underwater
infrastructure inspection system proposed in this paper isshown in Fig.1. The method of our inspection consists of(1) deploy / navigation, (2) investigation, (3) return.
After deployment of robot from (1) in the figure, theoperator navigates the auxiliary boat remotely towards
the target bridge to be inspected.
At this time, the auxiliary boat performs assistance inorientation and position control from the information of
GPS (Global Positioning system), IMU (Inertial
Measurement Unit) and LRF (Laser range finder) to theoperator.
After approaching to the bridge, underwater robot islowered by a winch from auxiliary boat as shown infigure 2 to lower the underwater robot to surveil theinfrastructure and washing condition using Front andBottom Camera. The operator checks the image sentfrom the camera robot in real time and performs
inspection. The auxiliary boat and the camera robot cankeep the position from the bridge piers constant by theguide rollers.
After the survey of the underwater infrastructure
equipment, the auxiliary boat returns back to injection
point and recovers the robot.

3. Underwater infrastructure inspection system
3.1 Overview
Table 1 shows the specification of the support boat
developed.
Table 1 The specification of the support boat
Voltage
AC240VSize
970*1030*1140 mm
Mass
64.0 kg
Buoyancy
89.0 kg
Propeller
300W Thruster*4
GPS
Observation
IMU sensor
equipment
Laser range finder
Winch

The support boat is equipped with Inertial Measurement
Unit (IMU), GPS, LAF for recognizing the operating
environment as well as a winch for depth control of ROV
and four 300W thruster for maneuver.
Table 2 shows the specification of the Camera robot
developed.
Table 2 The specification of the Camera robot
VoltageSize
Mass

AC240V
477.5*510*551.1mm
17.0kgPropeller
Underwater
weight90W Thruster*2

Observationequipment

4.5kg
Front camera
Bottom camera
Depth sensor
IMU sensor

As for image observation robot has an IMU, and a Depth
sensor for recognizing the operating environment. Frontand bottom camera are mounted to bridge piper© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan522

Inspection System for Underwater

occurring in underwater infrastructure increase and thecondition degrades over time. Due to this fact, it isimportant to monitor and recognize any micro cracks thatcurrently exist. Table 3 shows the necessity of repair foreach environment [7].

inspection. In order to make the captured image of theimage observation robot more stable, two Thruster (90W) are used as a thruster for sticking to the wall.
Figure 2 shows the image of infrastructure inspection
system. The robot uses AC240V. This power is also thesource of communication through PLC (Power Line
Communication), making it possible to send the images
from the two cameras to the operator on ground stationin real time.

Table 3 Necessity of repair for crack scale
Environment
Hard
NormalSoft
Need repair[mm]
0.4
0.4
0.6
Not need repair[mm]
0.1
0.2
0.2

The environment which is considered to be the most
severely degraded refers to under severe rain, dry
moisture, repeated wet and freezing, or in a corrosive
environment. Under such circumstances, there is apossibility that concrete and reinforcing bars will crack,
rust and peel off where it is thought that the frequency ofinspection and the level of degrade increase. The
underwater infrastructure equipment covered in thispaper includes ports, estuaries, coastal areas wherechloride ion is present in the air these areas require afrequent inspection.
The image observation robot developed in this paper
observes cracks that are already getting serious as well as
micro cracks. The minimum inspection target was
defined as a hair crack with 0.1 mm in width at which
may become more severe in the future.

Fig. 2 System diagram of the bridge inspection system

4. Experiment and results
In order to confirm the effectiveness of hair crack
recognition by the proposed method, a resolution
evaluation experiment was conducted. For the subject,
we used resolution chart ISO 12233 shown in Fig.5 and
HD Pro Web Camera C 920 for camera. The resolutionof the captured image is 5168 × 2907 pixels. Experiments
were conducted in water, and the distance between thesubject and the image observing robot was fixed at 120mm.
Fig.6 shows the images taken from experiments and theanalysis results. The image in Fig. 6 was taken of region
A in Fig. The result in Fig. 6 (a) is obtained by plotting
pixel values after grayscale conversion of thephotographed image. From this result, the influence ofaliasing increases from the vicinity of Fig. 6 (d), and theoriginal waveform can not be recognized.
Fig.7 shows the image of area B in Fig.5, which is takenof a continuously changing scale. The results in Fig. 7
show that the original waveform can be seen in thesections 1 to 5, but it can be confirmed that the sections
after the section 7 are collapsed.

Fig.3 The camera robot on of bridge inspection system.

Fig.4 The support boat on of bridge inspection system.

3.2 Using underwater camera
Necessity of repair may vary depending on theinspection site and the position. Generally, cracks© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan523

Takumi Ueda, Hiyoyasu Hirai, Kazuki Fuchigami, Ryoma Yuki, Ahn Jonghyun, Shinsuke Yasukawa, Yuya Nishida, Kazuo Ishii, Takashi Sonoda,
Katsunori Higashi, Katsunori Tanaka, Tomomasa Ikeda

6. Conclusion
In this paper, we propose a new underwater
infrastructure inspection system and conducted anevaluation experiment using resolution chart ISO 12233.
Firstly, we describe required specifications of underwater
infrastructure inspection, and defined the targeted haircracks in the proposed system. In order to investigate
whether it is possible to shoot according to the defined
hair crack width, evaluation experiment using resolution
chart ISO 12233 was carried out and showed a range
strongly affected by aliasing.
Based on the results obtained in this experiment, weplan to further develop underwater infrastructure
inspection system corresponding to finer cracks andapply it to inspection of underwater infrastructure in the
future.

Fig.5 Using ISO12233 test image

7. Reference(1) Ministry of Land, Infrastructure, Transport and Tourism,“Present state and future of aging of social capital”
Ministry of Land, Infrastructure, Transport andTourism,2016(2) Tetsuya Sumi , “Introduction to On-site Demonstrations of
Next-generation Robotic Technologies for Social
Infrastructures, Organized by Advisory on Underwater
Structures’ Maintenance” Journal of the Robotics Societyof Japan., Vol.34, No.10, pp.7-12. 2016(3) Hiroshi Shiramizu, Shinya Kusuhara, “Underwater
infrastructure inspection system using image sharpening
technology” Journal of the Robotics Society of Japan,
Vol.34, No.8, pp.503-504. 2016
(4) Taro Furutono, Atsuo Oono, “Maintenance Management
and Inspection Using 3D Laser Scanner and 3D Multibeam
Scanning Sonar”, Journal of the Robotics Society of Japan,
Vol.34, No.8, pp.509-510. 2016.
(5) Hideki Sugimoto, Tetsuya Ogasawara, Youichi Moriya,“Underwater Structure Diagnosis System by Remotely
Operated Vehicle”. Journal of the Robotics Society ofJapan, Vol.34, No.8, pp.505-506. 2016.
(6) Hideki Sugimoto, Tetsuya Ogasawara and Yoichi
Moriya ”Underwater Structure Diagnosis System by
Remotely Operated Vehicle“， Journal of the Robotics
Society of Japan,Vol.34, No.8, pp.15-16.
(7) Japan Concrete Engineering Society, “Survey of cracks inconcrete, repair and reinforcement guidelines -2003-,
Japan Concrete Engineering Society.

(a)(b) Point A(c) Point B(d) Point C (e) Point D
Fig.6 Obtained images of A

(A)
(B)
Fig.7 Obtained images of B

5. Discussion
From the results of the experiment in Fig. 7, thediscriminable pattern is section 6 in Fig. 7 (b), and thepattern width at that time is 0.3 to 0.4 mm. Observing
cracks that need repair is It is possible. In order to observe
cracks of 0.3 mm or less where follow-up observation isrequired, it is necessary to make the angle of view smaller
than the current angle and enlarge the drawing distanceper pixel.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan524

Three-dimensional Measurement Using Laser Pattern
And Its Application to Underwater Scanner
Yuya Nishida, Tomoya Shinnoki, Shinsuke Yasukawa, Kazuo Ishii

Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kita-kyushu, Fukuoka 808-0196, Japan
E-mail: ynishida@lsse.kyutech.ac.jp, n237038t@mail.kyutech.jp, s-yasukawa, ishii@brain.kyutech.ac.jp

Abstract
In order to measure an accurate volume of the resources, the authors developed a new three-dimensional instrument
consisting of a laser projector and a camera. The laser projector irradiates a sharp two-dimensional laser pattern
independent the distance to the target object. Our instrument was able to measure three-dimensional shape of thetarget object with maximum error of 5% in the water, at one scan
Keywords: Three-dimensional measurement, structured light, Underwater instrument, Resource investigation

independent the distance to the target object. The
instrument can measure three-dimensional shape of thetarget object using the laser pattern of several color beamat one scan. Thus, measurement accuracy of the
instrument doesn’t depend on position accuracy ofmounted vehicle. This paper explains the principle ofthree-dimensional measurement of the instrument, andshows the results of its experiment for evaluating
accuracy.

1. Introduction
Japan’s exclusive economic zone is in rich in resources
including mineral such as hydrothermal deposit, energy
such as methane hydrate and biological resources [1]. For
measurement of accurate volume of those resources,
many institutes and have researched underwater
instrument which is able to measure three-dimensional
shape of target objects. A sonar such as synthetic apertureand multi-beams sonar measure sea bottom topographicalof wide area with the resolution of a several meters, andis suited for survey of large resources [2]. An optical
instrument using underwater camera measures object
shape with high resolution, and its swath width is thesmaller than the sonar. Light cutting method using acamera and a sheet laser [3], and stereo vision using two
cameras [4], are able to measure three-dimensional shapeof millimeter order. Because measurement accuracy of
two methods depend on accuracy of self-localization ofunderwater vehicle with instrument, those are not suitedfor measurement of moving objects such as fish.
In order to measure actual volume biological resources,
this research developed a new underwater optical
instrument consisting of a laser projector and a camera.
The laser projector irradiates a sharp laser pattern

2. Principle of three-dimensional measurement
2.1. Measurement model
Our three-dimensional consist a laser projector inside a
PC hull and a network camera inside a camera hull, andthe position of the place where the laser beam hits ismeasured in camera coordinate. Although basic principle
is the similar to general light cutting method,measurement area of the instrument is larger than thegeneral by multi colors laser beam from the projector.
Camera model in the instrument is shown in Fig.1, and Z
axis direction of the camera coordinate which is locatedon the optical center inside the camera is same as thecamera direction. The projector which is located y axis in
the camera coordinate be inclined to pitch direction of φp© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

525

Yuya Nishida, Tomoya Shinnoki, Shinsuke Yasukawa, Kazuo Ishii

Fig.2 Schematic diagram of light path
2.2. Water influenceFig. 1. Measurement model in our instrument

In our three-dimensional instrument, the network camera
is installed inside acrylic housing for water proof, and it
take underwater image from in air. Light refracts basedon Snell’s law when the medium though which lightpasses changes. In our instrument, light goes from air toacrylic, from acrylic to water, shown in Fig.2. Based on
Snell’s law, the view angle of the camera in air and water
is related as following equation.

degrees, and n color beams be irradiated from the
projector. The position qi = [ui vi]T in image coordinate
denotes the position pi = [xi yi zi]T where laser beam
inclined at θj degree hits on the target object. Let thetarget object and its photo image be related by pinhole
camera model. xi and yi in pi are expressed by following
equation:

=

=

(1)

(3)

= sin

sin

(5)

(6)

Because va and vc are a parameter determined by physicalproperty values, ψc is obtained from ψa by using equation(6).

yL denotes the position of y axis of the projector. Then,equation (1) is substituted for equation (3), and zi isrepresented by following equation.

==

ψa, ψb and ψc denote the view angel in air, acrylic andwater, and va, vb and vc are light velocity in air, acrylic
and water. The angle in water is obtained by solving
equation (6) for ψc.

(2)where f is focal length, and s denote the size per one pixel
in image coordinate respectively. zi is geometricallycalculated using the projector position and beam angle.

==

Not only the view angle, but optical center also is affectedby water. Using the distance da between optical centerand acrylic and acrylic thickness db, the offset do of z axisof optical center is expressed by following equation.

(4)

The camera capture photo image of laser reflection, andthe position of each laser reflection in image coordinate
is detected by binarization of HSV model. The shape ofthe target object can be measured by applying equation(1), (2) and (4) to all the pixels of the detected laserreflection.

=

+

−

ψb in equation (6) is obtained in the same way as ψc.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

526

(6)

Three-dimensional Measurement Using Laser

Fig.4 The position of laser reflection

Fig.3 Experimental setup
3.

Experiment and evaluation

3.1. Dry test
To evaluate the performance of our three-dimensional
instrument, the shape of a box which 55.4 mm wide,
150.6 mm length and 30.8 mm height was measured
using setup shown in Fig.3. The box was putted on thebottom which 1,137 mm distance from the optical centerof the center, and the projector inclined at 60 degrees islocated at 600 mm distance from the camera. The
projector irradiated 6 color beams including red, yellow,
green, light blue, blue and purple to target object formeasurement. Figure 4 shows the position of laserreflection calculated using equations in chapter 2.1. The
instrument measured the distance to the bottom less than
5% error rate. As the box dimension was estimated basedon measurement results, error rate of wide, length andheight were 7%, 5% and 5% respectively.

Fig.5 The distance measured with each laser beams
4. Conclusions
We developed new three-dimensional instrument that canmeasure the shape of target located in 1,000m away withless than 5% error rate, in air. The instrument measured
the distance to target with high accuracy.
References
1. Yuya Nishida, et al., Autonomous underwater vehicle“BOSS-A” for acoustic and visual survey of manganesecrusts, Journal of Robotics and Mechatronics, Vol.28,
No.1, pp.91-94, 2016.
2. Akira Asada, et al., Expanded interferometry and synthetic
aperture applied to a side scanning sonar for seafloor
bathymetry mapping, Proc. Of International Symposiumon Underwater Technology, 8281046, Taipei, 2004.
3. Blair Thornton, et al., Instruments and methods foracoustic and visual survey of manganese crusts, Journal of
Oceanic Engineering, Vol.38, No.1, pp.186-203, 2013.
4. Mathew Johson-Roberson, et al., Generation andvisualizationoflarge-scale
three-dimensional
reconstructions from underwater robotic surveys, Journalof Field Robotics, Vol.27, Issue 1, pp.21-51, 2009.

3.2. Wet test
The distance to the bottom was measured in water by the
setup same as Fig.3, and equations considering water
influence shown in chapter 2.2 were used. Then, actual
distance between the optical center and the bottom was
1,096 mm. Figure 5 shows measurement results witheach laser beams. The instrument measured the distancein water with less than 5% error rate in each laser beam.
Even though parameters and lens distortion had wascalibrated before wet test, measurement results depended
on the position of x axis. This cause is considered that the
distortion of the projector lens was not calibrated.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

527

Development of dam inspection underwater robot
Hiroyasu Hirai

Dept. of Human Intelligence Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: hirai.hiroyasu467@mail.kyutech.jp
Kazuo Ishii

Dept. of Intelligence Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka, Japan
E-mail:ishii@brain.kyutech.ac.jp

Abstract
Maintenances of social infrastructures such as dam, bridge are important subjects. The more aging problem going on,
the more the effective inspection methods have been expected. In the inspection of dams, divers observe theconditions of their surface manually, however underwater structure deeper than 30m is limited and their inspections
are insufficient. In this paper, we propose the ROV whose functions are video capture, indirect measurement using
laser, target depth keeping, heading keeping. The ROV is applied into the dam inspection and showed theeffectiveness to find cracks. In this study, The Remotely Operated Vehicle (ROV) is developed with the aim ofpractical application of technologies capable of replacing and supporting dam inspection by diver. Using thedeveloped ROV, we conducted in the 'Verification field of the next generation social infrastructure robot' which was
publicly offered by the Ministry of Land, Infrastructure and Transport in FY 2015 and examined its usefulness. In
the verification on-site, it was able to satisfy about half of the required inspection items.
Keywords: Underwater robot, dam inspection, underwater robot positioning, remotely operated vehicle

out sufficient confirmations[2]. To solve such problems,
inspection methods utilizing underwater robots havebeen developed and researched[3].
The author have developed and researched Remotely
Operated Vehicle (referred to as ROV) with the aim ofpracticing technologies capable of replacing andsupporting diver inspection work by a diver. The
developed ROV has functions such as underwater
moving image photographing function by the camera,
object size indirect measurement, depth and maintaining
the distance and direction with respect to the dam body.
Also accompanied by a floating unit for measuring thehorizontal coordinate position during ROV diving.
In this paper, the authors report on the results ofunderwater examination using ROV at actual dam

1. Introduction
Social infrastructures such as dams and bridge is an
important existence that becomes the foundation of thestate, and it requires a great deal of cost and time forrestoration and conservation[1]. However, many are built
during the high economic growth period and aging due toaging is progressing[1]. Meanwhile, efficiency ofmaintaining and managing social infrastructure isrequired by the declining population and the aging of the
population. Among them, in the underwater inspection
work of the dam, dam damage and the function of thedrainage gate are inspected by visual confirmation by adiver but at the depth of more than 30 m depth, the diving
work is restricted for safety so have not been able to carry© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

528

Hiroyasu Hirai, Kazuo Ishii

2.2. ROV

participating in 'Verification field of the next generationsocial infrastructure robot' publicly invited by Ministryof Land, Infrastructure and Transport in FY 2015..

ROV has azimuth sensor, depth sensor, USB camera.
The control of ROV is carried out by built-in one-board
computer, numerical value acquisition from sensor,
operation of motor driver and LED projector by PWM
output, transfer of operation video from USB camera andphotographing of still picture.

2. Dam inspection underwater robot
2.1. Configuration of ROV system
In development of ROV, we focused on not being able
to take time and effort for lightweight preparation, on thepremise of operation with two people. Fig.1 present theappearance of the ROV system and Fig.2 shown in thesystem configuration diagram. In Fig.2 shows threeelements that make up the system. The Ground Station
operated on the ground and the underwater ROV areconnected by a neutral buoyancy cable and power istransmitted. Moreover, communication is performed via
power line by using Power Line Communication.
Floating unit operates with batteries and is connected to
Ground Station wirelessly using ZigBee. Use thecontroller for operations such as movement of ROV,
lighting of LEDs, shooting still images.

2.3. Floating Unit
The appearance of the floating unit is shown in Fig.3

Fig.3 Overview of Floating Unit

Floating unit is fixed to ROV via fishing line. By
sending out the fishing line according to the movementof the ROV and keeping a constant tension, it moving in
the direction corresponding to the horizontal position of
the ROV. In addition, it incorporates a GPS and transmits
the output value to the Ground Station via Zigbee.

Fig.1 The dam inspection underwater robot

2.4. Ground Station
Fig. 4 shows the display contents of Ground Station.
The contents displayed are the image of the monitor
camera installed in the ROV, the image of the camera ,the shipboard camera, the output values of the sensor
and the current direction of the ROV drawn on the map,the direction facing it. Automatically start recording of
the display contents simultaneously with powering on
the system.

Fig.2 System architecture of ROV© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan529

Development of dam inspection

Fig.5 Calculated object size result

3.4. Stabilize the attitude
In order to maintain the positional relationship with theobject to be photographed, feed-back control is used tostabilize the attitude of the ROV. Control consists ofdepth, azimuth, and horizontal distance from the dam
wall. PID control was used for feedback control, and thecontrol gain was adjusted by actual movement in pooland dam. In the maintenance control of the horizontal
distance from the dam body, since it is necessary for theunderwater distance sensor to be opposite to the dam
body, it is performed simultaneously with themaintenance control of the azimuth.

Fig.4 Display contents of Ground Station

3. Underwater infrastructure inspection method
3.1. Outline of inspection method
In the inspection of the dam using the ROV, although
the dam body is continuously photographed using thebuilt-in camera, it is necessary to simultaneously record
the coordinates of the object at the time of photographing.
Furthermore, it is desirable to be able to measure the sizeof the object to be imaged, and estimation was done bypost processing. Also, in order to photograph a wide
range continuously and smoothly, position keeping of arobot by feedback control was mounted on ROV andverified.

4.

Evaluation
inspectionofsystem

using

real

dam

To confirm the effectiveness of the developed ROV, we
conducted a test at the Amagase Dam in Kyoto Prefecture
on November 24, 2015. Amagase dam is an arched
concrete dam. The state of the test at the Amagase dam isshown in Fig.6.

3.2. Estimation of shooting coordinates
The output values of the GPS mounted on the floating
unit and the depth sensor, orientation sensor, underwater
distance sensor mounted on the ROV are recorded at thesame time stamp and the coordinates of the object at thetime of camera shooting are estimated by post processing.
3.3. Estimation of size of photographing object
Estimate the size of the object indirectly by using the
fixed laser mounted on the ROV. The fixed laser wascomposed of two pieces, and the interval distance was set
to 300 mm. Fig. 5 shows the result of calculating the sizeof the object to be photographed from the photographed
image. We calculated the size of the object by deriving
the size of each pixel in the photographed image by using
the distance between two points of the fixed laser shownin the image.

Fig.6 Amagase dam and boat

The inspection target in the test is to check the position
coordinates, the surface shape, and the size by
photographing the concrete block installed in the
preliminary conduit gate in a state of falling into thewater. In the test, the ROV dropped from the boat was
brought close to the dam wall and the submarine began© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan530

Hiroyasu Hirai, Kazuo Ishii

using the guide rail of the preliminary conduit gate
exposed on the water as a marker. Fig. 7 shows the stateof the boat and ROV at the start of the inspection.

Table 1. Concrete block position and difference

Block

Position(JGD2011 VI plane rectangular co-ordinates)
x[m]

y[m]

measured -124157.849
exact-124157.410-15709.042

7measured -124156.738
exact-124155.226
9 measured -124155.630-124155.139
exact
977 measured -124156.738
exact-124156.131
77-15709.955

7-15710.195

55Difference[m]0.470-15709.042
0.470-15709.039-15709.383-15709.955

0.600

0.650

-15709.188

5. Equations
Fig.7 Approach to dam with ROV

In this paper, in order to confirm whether ROV
developed and researched can be used for efficientunderwater inspection work, the result of participating in
the verification test at actual dam was described.
In the verification test, We were able to estimate thecoordinates of the shooting position by still image of theconcrete block installed in the dam body by ROV withina predetermined time and checking with the sensor
information based on the shooting time .In addition, We
were able to estimate the size of 12 simulated cracks fromthe captured image, respectively.
As a result, it was possible to show that there ispossibility of efficient inspection of the dam by ROV.

In Fig.7, we see ROV and Floating unit on the water
surface indicated by red circle. We tried diving by
manual operation and attempted maintenance byautonomous control about heading and horizontal
distance from the dam body. As a result of the test, we
succeeded in photographing four concrete blocks with aworking time of about 3 hours. Concrete blocks were notinformed of the number, position and current in advance,
but the total was 4 and it was possible to discover
everything, and on the concrete block surface slits andprotrusions imitating cracks were printed I was able toconfirm the numbers. The four concrete blocks found in
Fig. 8 are shown.

Acknowledgements
This research is promoted by the Nissui Marine Kogyo
Co., Ltd.
References
1. MLIT, “White paper on land, infrastructure, transport andtourisminJapan2013”(Japan,2013),pp.28-30.
Available:http://www.mlit.go.jp/common/001063075.pdf.
2. National Institute of Occupational Safety and Health,Japan.,JICOSH,Available:
https://www.jniosh.johas.go.jp/icpro/jicoshold/japanese/country/japan/laws/03_rel/08_selectedchemi
cals_reg/index.html
3. N. A. Cruz, A. C. Matos, R. M. Almeida, B. M. Ferreira,and N. Abreu, TriMARES – A hybrid AUV/ROV for dam
inspection, OCEANS2011 MTS/IEEE(Waikoloa, Hawaii,2011), pp. 1-6.

Fig.8 Inspection target concrete block and dummy clacks

In Fig. 8, it is understood that the numbers 5, 7, 9, 77
are printed on the concrete block.
Table 1 shows the calculation results of the position
coordinates of the four concrete blocks from the sensor
output values at the time of shooting. As shown in theerror of Table 1, although it is larger than the visualobservation, it got results that endure use.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

531

Conceptual Design of Small ROV for Sky to Water System
Keisuke Watanabe*

School of Marine Science and Technology, Tokai University, 3-20-1 Orido Shimizu-ku
Shizuoka-shi, Shizuoka, Japan†
Koushi Utsunomiya

School of Marine Science and Technology, Tokai University, 3-20-1 Orido Shimizu-ku
Shizuoka-shi, Shizuoka, Japan
Kazumasa Harada

School of Marine Science and Technology, Tokai University, 3-20-1 Orido Shimizu-ku
Shizuoka-shi, Shizuoka, Japan
Shuhei Nakajima

School of Marine Science and Technology, Tokai University, 3-20-1 Orido Shimizu-ku
Shizuoka-shi, Shizuoka, Japan
E-mail: keisuke@tsc.u-tokai.ac.jp

Abstract
Long term and wide range monitoring of coral reef is necessary for the research of coral bleaching. Researchers
diving for investigation, they need to hire a boat whose port is not always near the target site where is sometimes neara beach. We are developing a new method which uses a set of Multicopter-USV-ROV. We named this concept as
Sky to Water System. In this paper, we introduce the conceptual design of the SWS and its Small ROV.
Keywords: Sky to Water System, Coral bleaching, Unmanned Surface Vehicle, RTK GNSS, Multicopter, Floating
LBL

limited spot, however, it is impossible for divers toinvestigate such wide area like the Great Barrier Reef
simultaneously. Though satellite image can be used tomonitor the vast area, needless to say, it is difficult for us
to measure the underwater environmental parameterssuch as time, conductivity, pH, color of reefs etc.
To breakthrough these limitations we are developing
a new method which uses a combined system ofmulticopter-USV(Unmanned Surface Vehicle)-ROV.
We named this concept as Sky to Water System(STWS).

1. Introduction
Long term and wide range monitoring of an area of coral
reef is necessary for investigation of coral bleaching. The
researchers usually use underwater cameras to record thereef’s present condition using a ROV or diving by
themselves. In those cases, they need to hire a boat whose
port is not always near the target investigation site where
sometimes near a beach. The researcher diving iseffective from the viewpoint of precise investisation of a*

Corresponding author© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan532

Keisuke Watanabe, Koushi Utsunomiya, Kazumasa Harada, Shuhei Nakaji

In this paper, we introduce the conceptual design of SWS,
the conceptual design of the Small ROV which can bedelivered with its mother USV by a multi-copter from thenearest beach, and some experimental results of precise
RTK GNSS positioning which is essential for
determination of the underwater coordinate of ROV by
our developing floating LBL system.
2. STWS concept
STWS consists of a multi-copter for USV/ROV systemtransportation, multi-copters for taking pictures from thesky, USVs, ROVs or AUVs and the floating LBL system1
for determination of underwater vehicle’s position.2 Each
module is delivered by the multi-copter from a mothervessel or seashore.3 The ROV or AUV is docked to theunmanned surface vehicle (USV) when the multi-coptertransports it. The USV has wireless LAN so that we cancontrol the USV and ROV remotely and monitor theunderwater images which the ROV films by its video
camera. The USV also equips a kind of winch system torelease and wind up the umbilical cable attached to the
ROV. If an AUV is deployed, the winch system can beomitted. Each USV has ultrasound ranging module, RTK
GNSS and communication device (XBee module). The
communication device is used to communicate between
USVs for collision avoidance and transmission of
acquired data to calculate the base line distance orunderwater coordinate of the ROV or AUV.

Fig. 1. Principal dimension of designed ROV.

3. Conceptual Design of ROV
Our ROV is delivered by a multicopter UAV with aninterface USV. So it must be designed as light weight as
possible. The schematic view and drawing are shown in
Fig.1 and Fig.2 respectively. The equipment of ROV and
its architecture is shown in Fig.3. The total weight in air
is around 3.5kgf.
The ROV has its own battery inside to be able to be
used for a kind of AUV in the next experiment. A single
board computer controls sensor/actuator systems. We
implemented a video camera to capture the underwater
images of corals and the acryl pressure hull is hired forthis purpose. The supposed water depth is up to 100m
from the strength estimation using its thickness of thewall. At this stage, conductivity, temperature and depth(CTD) sensors are implemented for recording ofenvironmental condition data that should be recorded

Fig. 2. Principal dimension of designed ROV.

with the picture taken.
Six thrusters are hired to control its underwater
position. The underwater position is determined by thefloating LBL using ROV’s transponder ultrasound signal.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

533

Conceptual Design of Small

Rover1
Rover2

Base

Fig. 4. RTK GNSS distance measurement experimentFig. 5. Base-rover1 distance case 2[m].

Fig. 3. Equipments of ROV.

4.

GNSS baseline detection experiments

In our floating LBL system, the baselines are measured
using each USV’s GNSS (N,E) coordinate. Recent
progress in GNSS receiver enables us to measure thedistance with centimeter accuracy. The accuracy ofbaseline distance is crucial in our underwater position
calculation, we confirmed the accuracy of RTK GNSS isenough to develop our floating LBL system.
The experiments were carried out on our school
field as shown in Fig.4. GNSS base station is fixed as theorigin of the coordinate. Two rover antennas are arrangedon the orthogonal lines and each distance from the basestation is changed from 2m, 5m, 10m, 20m, 30m.
Figures 5-7 show the time series of measured
distance between base-rover1, base rover2, rover1rover2, when the setup distance is 2[m].

Fig. 6. Base-rover2 distance case 2[m].

In these experiments, the relative height of each GNSS
module was almost zero, so we calculated the distance asfollows.
disance = 1852.196 × ( 1 − 2) × ( 1 − 2) ×
cos(
)
(1)
Here, ( N1, E1 ) shows longitude and latitude© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan534

Keisuke Watanabe, Koushi Utsunomiya, Kazumasa Harada, Shuhei Nakaji

Fig. 7. Rover1-rover2 distance case 2[m].

Fig. 8. Base-rover2 distance case 10[m].

Figures 8-10 show the time series of measured distance
between base-rover1, base rover2, rover1-rover2, when
the setup distance is 30[m]. From these experiments, we
confirmed we can implement the RTK GNSS modules in
our floating LBL baseline measurement system in very
good accuracy.
5. Conclusion
We designed small ROV concept for STWS. The
fabrication is ongoing and we will carry out first sea
experiment in the near future. The underwater position
determination is essential for this ROV and the baseline
accuracy of the floating LBL is desirable within several
centimeters order. The RTK GNSS experimental results
showed our current system can measure relative lengthfrom 2[m] to 30[m] less than 2[cm]. Distancemeasurement using ultrasound transponder is our next
step.

Fig. 9. Base-rover2 distance case 10[m].

Acknowledgements
We sincerely thank to our laboratory members to help
field experiments. This research is supported by the
Tokai University Supporters Association Research andStudy Grant. ext.

Fig. 10. Rover1-rover2 distance case 10[m].
3. K. Watanabe, K. Utsunomiya, K. Harada, Y. Watanabe,“A Concept of Coastal Sea Monitoring System from Sky
to Water”, Proc. of the 3rd World Congress on Civil,
Structural, and Environmental Engineering (CSEE’18)(2018), Paper No. ICESDP 136
4. K. Watanabe, K. Mitsumura, K. Utsunomiya, K. Harada,“An Experiment of Multicopter and Small ROV
Combined Monitoring System for Coastal Area
Environment”, Proc. of the 28th International Ocean andPolar Engineering Conference, (2018), pp.373-378

References
1. K. Watanabe, K. Ishii, “Floating LBL concept forunderwater position detection”, Proc. of the 2017
International Conference on Artificial Life and Robotics,
Miyazaki, ICAROB, OS7-2(2017), pp.245-249
2. K. Watanabe, K. Mitsumura, K. Utsunomiya and S.
Takasaki, “Experiments of Floatable UAV for Wave
Dissipating Block Inspection”, Journal of Robotics,
Networking and Artificial Life, Vol.4, No.3(2017), pp.234238© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

535

Study on Position Estimation Using Small Size ZigBee Module
Atsushi Sanada

Nishinippon Institute of Technology,
1-11, Aratsu, Kanda, Miyako-gun, Fukuoka, 800-0934, Japan
E-mail: sanada@nishitech.ac.jp

Abstract
Position information is very important information that can be utilized in various fields. In this research, we propose
a location estimation system using low cost ZigBee compact radio module. The position is estimated based on theradio wave intensity for the plurality of antennas. Since the radio wave intensity is nonlinear with distance, thedistance was estimated using a neural network. As a result, it was confirmed that the distance can be estimatedcorrectly.
Keywords: Position Estimation, Small Size Wireless Module, TWELITE, Neural Network

1. Introduction
Table 1. Comparison of position detection methods

In recent years, services based on location information
have become widespread, and are used in all situations oflife. For example, the navigation system1 is installed not
only in automobiles but also in smartphones, indicating
an efficient route to the destination. For this purpose,
position information by GPS is mainly used. Additionally,
Bluetooth is installed for near distance search in GPS tags
for loss prevention. In addition to location services using
satellite radio waves and radio waves like this, a systemused in sports such as football that detects and displays
the positions of players and balls is realized by analyzing
a plurality of camera images set around the field. As
described above, location services are deployed byvarious technologies, but as shown in Table 1, each has
advantages and disadvantages.
GPS can be used mostly outdoors and has a very wide
detection area, but it cannot be used indoors. Further, if
it is tried to obtain accuracy of cm unit, an auxiliary
device such as a fixed station is required. The camera ishighly accurate if the shooting range is the detection area
and the detection area is narrow, but it is impossible todetect a wide range while maintaining accuracy with one

Position Detection
MethodGPS
CameraRadio Wave
Proposed Method

Area
◎
×
△
○

IndoorUse×○○○

Accuracy

Cost

○
◎
△
◎

○
×
○
◎

camera. In order to detect a wide range, it is necessary tocooperate with plural cameras, resulting in high cost.
Wireless radio waves are used for rough location
detection2,3 when ZigBee, Bluetooth or WIFI is used, andnot until detection in cm. This is because it is difficult tosimply obtain the distance from the radio wave intensity
because the radio wave changes due to fading caused byreflection on a wall or the like. Considering the influenceof this fading, the accuracy of position detection using© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

536

Atsushi Sanada

radio waves should improve. In this research, we aim to
improve position detection accuracy by learning the radio
field strength including fading with a neural network.

Receiver

Receiver

Receiver2. System overview
2.1. Transmitter and Receiver

TransmitterAs shown in Fig. 1, the present system is composed of areceiver set at known coordinates and a transmitter thatmoves. ZigBee is used as a wireless system. Signals are
periodically transmitted from the transmitter, and thesignals are received by the respective receivers. At thereceiver, a value of LQI (Link Quality Indicator)
indicating the quality of the received signal is obtained
when the signal is received. Since the approximate valueof the radio field strength can be obtained from LQI using
Eq. (1), LQI is used in this system. In the equation, PdBm
represents radio field intensity, and Q represents LQI.

Fig. 1. System overview(1)

The LQI information is gathered in one antenna, and theposition of the transmitter is estimated using a neuralnetwork learned beforehand.
2.2. Neural Network

Output: xy coordinate

7Q-197020

Receiver

ReceiverInput: Each LQI value

PdBm =

ReceiverAs shown in Fig. 2, the neural network used in thissystem consists of an input layer, two intermediate layers,
and an output layer. The number of input layers is 4
according to the number of receivers used in later
experiments, the number of intermediate layers is 500
and 100, respectively, and the output layer outputs twocoordinates because it outputs coordinates.

Intermediate layer2: 100
Intermediate layer1: 500

Fig. 2. Neural Network

3. Experiments and results
3.1. Confirm directionality
Experiments were conducted to confirm that neither thetransmitter nor the receiver had directivity. The
transmitter and the receiver were installed at a distanceof 10 m, and the radio field strength was measured whilerotating the transmitter. The experimental results areshown in Fig. 3. From the figure, the measured value isshifted in the direction of 315 °, but this is considered tobe due to a small inclination of the receiver, and it was
confirmed that neither transmitter nor receiver has
directivity.

Fig. 3. Directionality© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

537

Study on Position Estimation

3.2. Relationship between the distance and theradio wave intensity

LQI [-]In order to confirm the relationship between the distanceand the radio wave intensity, the radio field intensity wasmeasured every 1 m from 1 m to 100 m for the transmitterand the receiver. The measurement result is shown in Fig.
4. The LQI value decreases as the distance increases, andit decreases while bouncing. It is thought that this iscaused by fading due to reflection to the ground. Sincefading occurs not only due to the ground but also due towalls, it is difficult to express it by mathematicalformulas including all causes. Therefore, we decided toestimate the position by learning the radio field strength
including the influence of fading using the neuralnetwork.

180
160
140
120
100
80
60
40
20
0
0

10

20

30

40

50

60

70

80

90 100Distance [m]Fig. 4. Relationship between the distance andthe radio wave intensity

3.3. Position estimation
As shown in Fig. 5, the receivers (R1 to R4) were installedat the four corners of the half of the field, the fields weredivided into meshes with intervals of 10[m], and the radio
field intensity at each point was measured. Since themeasurement range is bilaterally symmetrical, it is set as
a half area surrounded by a broken line. 50 samples weretaken at each point. The relationship between LQI andcoordinates was learned using the neural network shownin Fig. 2. Normalization was performed by dividing by
the value (160) which is larger than the maximum valueof the measured LQI by using the LQI of the signal
received by each receiving antenna. The coordinates
were also divided by 70 to normalize. Therefore, theoutput is multiplied by 70 to output coordinates. 40
samples at each point were used for learning and theremaining 10 samples were used as unlearned data to
confirm the learning result. Therefore, since there are 24
measuring points, 960 samples of learning data and 240
samples of unlearned data are obtained. In addition, thelearning rate was 0.01, learning was done with batchlearning, and learning was done using 10 random
samples. The transition of the learning error is shown in
Fig. 6. The CPU of the used PC is Core i5-4300 U (1.9

Learning error

Fig. 5. Receiver position

Number of training
Fig. 6. Transition of the learning error

Table 2. Maximum error and average error
Numberof training

x-axis error(maximum) [m]y-axis error(maximum) [m]x-axis error(average) [m]y-axis error(average) [m]50000
100000500000

4.79
4.01
1.46

2.51
2.67
1.47

0.99
0.53
0.22

0.85
0.58
0.38

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

538

Atsushi Sanada

GHz - 2.5 GHz). Also, the learning time was 951 seconds
at 500,000 times. We obtained the result of inputting
unlearned data to the learned neural network. Fig. 7
shows the results when the learning number is 500000
times. In the figure, the true value is indicated by a blue
square and the estimated value is indicated by an orange
triangle. Also, the dispersion at each point is indicated bya bar. The dispersion tended to increase at the center ofthe field far from the receiving antenna. Table 2 shows
the maximum error and average error depending on thenumber of learning. It is understood that the error
decreases as the number of times of learning increases.
Also, the average error is within 0.5m. In this experiment,
we cannot verify the interpolation by the neural network,
but not only the measured points but also the coordinates
between the measured points should be able to be
estimated, so confirmation is necessary.

Fig. 7. Experiments result

4. Conclusion
In this research, the location was estimated from radio
field intensity using a compact radio module. By using aneural network for estimation, we were able to estimate
the position without concern for complicated fading
effects. Good results were obtained, such as estimation
with an error within 0.5 m with a learning number of500000, using a 3-layer neural network with anintermediate layer of about 500. In addition, since we
conducted experiments only in a quarter of the football
field this time we want to do it in the whole area next time.
And, I would like to improve position estimation
accuracy by increasing the number of receivers. When
this system is moved to another field, the influence offading varies, so it may be considered that it does notwork well. However, we believe that the neural networkthat has learned sufficiently can shorten the additional
learning time in the destination field. I would like toexamine the shortening of learning time from now on.

References
1. Robert Bosch GmbH, BOSCH Automotive Handbook,p.1116, 2007.
2. Erin-Ee-Lin Lau and Wan-Young Chung, Enhanced
RSSI-Based Real-Time User Location Tracking System
for Indoor and Outdoor Environments, 2007 International
Conference on Convergence Information Technology,
2007.
3. Oziel Hernandez, Varun Jain, Suhas Chakravarty andPrashant Bhargava, Position Location Monitoring Using
IEEE® 802.15.4/ZigBee® technology, Beyond Bits, Issue
2 Summer 2009, pp. 67-73, 2009.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

539

Report of the 4th Tomato Harvesting Robot Competition
Takayuki Matsuo

National Institute Technology, Kitakyushu College
2-20-1, Shii, Kokuraminami-ku, Kitakyushu-shi, Fukuoka, Japan
Takashi Sonoda and Yasunori Takemura

Nishinippon Instituite of Technology
1-11Aratsu, kanda-tyo, Miyako-gun, Fukuoka, Japan
Masanori Sato

Nagasaki Institute and Applied Science
536, Aba-matchi, Nagasaki-Shi, Nagasaki, Japan
Kazuo Ishii

Kyushu Institute of Technology
2-4, Hibikino, Wakamatsu-ku, Kitakyushu-shi, Fukuoka, Japan

E-mail: matsuo@kct.ac.jp, sonoda@nishitech.ac.jp, takemura@nishitech.ac.jp, ishii@brain.kyutech.ac.jp
http://www.lsse.kyutech.ac.jp/~sociorobo/ja/tomato-robot2017

Abstract
Tomato is one of important fruit vegetables and most tomatoes are produced in the greenhouses, or large scale farms,
where the high temperature and humidity, and long harvest age force the famers heavy works. In this paper, we reporton the results of 4th tomato harvesting robot competition.
Keywords: robots for socio synthesis, robot competition and agriculture robot

method using image processing and visualfeedback[2][3]. Kondo et. al. proposed the method toimprove the success ratio of tomato harvesting andspeed-up technique[4][5]. Ota et. al. proposed the path
planning method to pick up tomato[6].
We organized the tomato harvesting robot competition
for an aim to promote the automation of tomato
harvesting until 2014. In this paper, we will report on the
result of the 4th tomato harvesting robot competition.

1. Introduction
Recently, the aging and depopulation of famers growworse in Japanese agriculture, as results the shortages of
future famers and manpower become big problems.
Tomato is one of important fruit vegetables and most
tomatoes are produced in the greenhouses, or large-scale
farms, where the high temperature and humidity, andlong harvest age force the farmer heavy works.
As the research of tomato harvesting robots in Japan,
Kawamura et. al. developed a mobile robot withmanipulator[1] and proposed the tomato harvesting© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

540

Takayuki Matsuo, Takashi Sonoda, Yasunori Takemura, Masanori Sato, Kazuo Ishii

2. Competition regulations
The tomato harvesting robot competition consists of two
leagues which are he Senior League and the Junior
League. The target competitors for Senior League aresupposed to the team with automated and remotecontrolled robots, and the Junior League are for highschool or junior high school students who build robot
using LEGO Mindstorm. In this section, competition
regulations of both reargues will be introduced.

Table 1 Specifications of a robot
W:800mm D:800mm(H: No limitation)
Equipment
Emergency stop switch
Size

WeightMotor Power

Max. 50kg (recommend)
Max. 70W (recommend)

2.1. Senior League
The Senior League supposes that teams composed ofundergraduate, graduate students or developers join thecompetition with their tomato harvesting robots. Twokind of competition field are designed, the one is railstyle area and the another one is fee-style area as shownin Fig.1 and Fig.2. The rail-style area is designed to havethe similar environment with the tomato factory. Freestyle area is for the robots of general tomato fields in
outdoor environment. Tomatoes are set on height from800mm to 1200mm. The required specification of tomato
harvesting robot is shown in Table 1. In first stage, atomato is suspended as shown in Fig.3 and the team isable to advance to the second stage when an end-effecterof the robot is able to touch the tomato. In second stage,a bunch of tomato is suspended as shown in Fig.4. Teams
compete with each other for the harvesting number of thetomato. Even if the robot can harvest many tomato, thescore of the team is added to the score when the tomato
has no damage. However, When the tomato harvested by
the robots are not redder than a tomato shown by an
executive committee, and the basic point of the team isdecided. Final score of second stage is the score
multiplied the basic score and the coefficient decided by
a combination of choice between the control method
which are remotely and autonomous control and the areas
which are the rail area and the free style area. The fivehighest team can advance to the final stage. In final stage,
the robots harvest tomato from plant body as shown in
Fig.5.

Fig.1 Free-style area

Fig.2 Rail-style area

Fig.3 1st Stage

Fig.4 2nd Stage

Fig. 5 Final Stage

2.2. Junior League
In Junior League, we have selected an autonomoustransport system of tomato harvesting robot as the subjectof Junior League. Students should develop the robot
using LEGO Mindstorm. Competition subjects include
Line Trace Challenge, Color Identify Challenge,
Mechanism Design and Control Challenge and Object
Detection Challenge. In Line Trace Challenge, robots
should detect white line in the competition area and movealong the line using a color sensor. The robot starts fromthe starting point. In the middle of the course, the tomatobox- harvesting field (harvest field) exists, where tomato© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

541

Report of the 4thboxes are arranged. The robot must move to the harvest
field in order to get the tomato boxes. In Color
Identifying Challenge, robots should explore andrecognize color signs in the middle of the course and thesame color of three boxed tomatoes (tomato box). As
guidance to harvest field, red, yellow and blue lines are
drawn in the field. It is necessary for the robots to detect
guidance line by color sensor. The robot gets the tomato
boxes of the same color. Along the white line for line
trace, each color is signed. The robot can move to theharvest field from the lone detecting the color. In
Mechanism Design and Control Challenge, robot should
manipulate a tomato box using manipulator and carry to
the storage location. To pick up the tomato box by using
a manipulator equipment which is made by each team,participants are expected to design and make a device to
get a tomato box on their idea. The robot is required tostore, transport and relocation depending on tomato box
colors. After picking up the box, the robot should returnto the course. Then, the robot carries the box to thespecific storage location of the tomato box. In Object
Detection Challenge, robots should detect a battery
charging station and stop there. Each team consists of 3
or 4 students and make a robot using LEGO Mindstorms
EV3 and also do robot programming for solving thecompetition subjects. The basic specification for robot isthat the size of the robot is within 300mm x 300mm on
ground.

Table 2 Final result of the Senior League
RankingTeam
Overall winner

HAYASHI-LAB

Overall runner-up

Hibikino-Tom’s R

Table 3 Result of rail-style division
RankingTeam
1st place
2nd place

HAYASHI-LAB
Hibikino-Tom’s R

3rd place

KPC2017 A

Table 4 Result of free-style division
RankingTeam
1st place
2nd placeNagasaki GANBARANBA-

3rd place-

Table 5 Result of rail-style division
RankingTeam
1st place

Ohnishi lions (Fukuoka Technical
Hischool
Ooita kogyo C(Ooita Technical HighSchool)NiAScience (High school attached
Nagasaki Institute of Appied Science)

2nd place
3rd place

Table 6 Special award
RankingTeamth

3. Result of 4 tomato harvesting robot
competition
Teams which could proceed to final stage are 5 teams
including Hibikinno Toms’s R (Category T5, Kyushu
Institute of Technology), Nagasaki GANBARANBA(Category T4, Nagasaki Institute of Applied Science),
KPC2017A, KPC2017B (Category T5, Kyushu
Polytechnic college) and HAYASHI-LAB (Category T5,
Kyushu Institute of Technology). 1st place of rail-styledivision is HAYASHI-LAB, 2nd place is Hibikino Tom’s
R and 3rd place KPC2017A 1st place Nagasaki
GANBARANBA as the result of free-style. 2nd and 3rd
are no eligible teams. Finally, Overall winner is
HAYASHI-LAB and Overall runner-up is HibikinoTom’s R.In Senior League, number of the participatedteams were 18 teams and the result is Table 5 and Table
6

Best
presentation
award
Special Jury Award

J-Tech2 (Joto high school)

Challenge award

Team Daiichi (Fukuoka Daiichi
High School)

Team ROSA (Ooita Technical HighSchool)4. Discussion
In Senior League of 4th tomato harvesting robot
competition, scores are evaluated as shown in Eq.(1)𝑃 = 𝐶(2𝛼 + 𝛽) − 2(𝛾 + 𝛿) − 𝜀(1), where P is score, C is coefficient of magnification
depending on selecting class as shown in Table Ta, 𝛼isthe number of tomatoes which is no damaged and correct© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

542

Takayuki Matsuo, Takashi Sonoda, Yasunori Takemura, Masanori Sato, Kazuo Ishii

color, 𝛽 is the number of damaged tomatoes, 𝛾 is thenumber of drop tomatoes and 𝛾 is the number ofdamaged tomatoes which are not harvested. 𝜀 is adeducted point when robots damage stalks of tomato
plants and the point is deducted in Final Stage. However,if the teams select high class such as T5 and T6, Total
score P is almost not influenced from 𝛾, 𝛿 and 𝜀. Thismeans that teams selected high class can get high score if
robot damage tomatoes and drop tomatoes. In actual
tomato factory, damaged tomatoes and drop tomatoes
have no commercial value. Therefore, famer need totreats tomatoes with care. In next competition, we will
employ concept of carefully dealing with tomatoes. We
proposed new equation for next competition as shown inEq.(2).

Acknowledgements

Equation (2) is employed harvest rate and if robot
damage and drop tomatoes magnification C isdecreased.

This works was supported by Kyushu Institute ofTechnology, Center for Socio-Robotics Synthesis, The
University of Kitakyushu, Nippon Bunri University,
Nishinippon Institute of Technology(This support was
including The Ministry of Education, Culture, Sports,
Science and Technology(MEXT), Center of Community),
Kitakyushu College of Technology, Hibikinada-Saien,
City of Kitakyushu, The Kitakyushu Chamber ofCommerce and Industry, Institute of Systems,
Information Technologies and Nanotechnologies (ISIT),
Japan Chapter of IEEE Society on Systems, Man, andCybernetics, The Japan Society of MechanicalEngineers(JSME), The Robotics Society of Japan(RSJ),
Kitakyushu Foundation for the Advancement of Industry
Science and Technology (FAIS), IT and Mechatoronics
Chapter and Kyushu Chapter of Japanese Society ofAgricultural Machinery and Food Engineers, The
MathWorks Co. Ltd., and Ministry of Agriculture,
Forestry and Fisheries(MAFF).

Table 7 The class number and coefficient at choosing
remote control

References𝑷=𝜶
𝜶+𝜷+𝜸+𝜹𝑪(𝟐𝜶 + 𝜷) − 𝟐(𝜸 + 𝜹) − 𝜺

Method of View
Area
Number of Category
Coefficient C

Directly
Rail
Free
T1
T2
1
2

(2)

Indirectly
Rail
Free
T3
T4
2
4

1. Kawamura et. al. Study on agricultural robot (1st report),
The journal of JASM, vol. 46 No.3, pp.353-358, 1984 in
Japanese
2. Kawamura et. al. Study on agricultural robot (2nd report),
The journal of JASM, vol.47 No. 2, pp.177-182, 1985 inJapanese
3. Kawamura et. al., Fruit harvesting robot, The journal of
JASM, vol.47 No.2, pp. 237-241, 1985 in Japanese
4. Kondo, et.al., A Machine Vision System for Tomato
Cluster Harvesting Robot, Engineering in Agriculture,
Environment and Food, Vol. 2, No. 2, pp.60-64, 2009
5. Kondo, et.al, Development of an End-Effector for a
Tomato Cluster Harvesting Robot, Engineering inAgriculture, Environment and Food, Vol. 3, No. 1, pp.2024, 2010
6. Ota et. al., Development of a Tomato Harvesting Robot
with a Vision System Using Specular Reflection (Part 1),
the journal of JASM, vol. 72 No.6, pp.595-603, 2010
7. Ota et. al., Development of a Tomato Harvesting Robot
with a Vision System Using Specular Reflection (Part 1),
the journal of JASME, vol. 72 No.6, pp.587-594, 2010

Table 8 The class number and coefficient at choosing
autonomous control
Area
Number of Category
Coefficient C

5.

Rail
T5
8

Free
T6
16

Conclutions

In this paper, Tomato robot competition for robot sociosynthesis was introduced. The tomato harvesting robot
competition has two leagues which are Senior League
and Junior League. In Junior League, number of theparticipated teams were 18 teams. In Senior League,
number of participated team 9 team. In next competition,
concept of carefully dealing with tomatoes will be
employed and we proposed new equation employed
harvest rate and if robot damage and drop tomatoes
magnification C is decreased.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

543

End-effector for robotic harvesting of a tomato fruit with calyx
Bingh Li

Dept. of Human Intelligence Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: li-binghe@edu.brain.kyutech.ac.jp
Shinsuke Yasukawa

Dept. of Human Intelligence Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: s-yasukawa@edu.brain.kyutech.ac.jp
Takuya Fujinaga

Dept. of Human Intelligence Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: s-yasukawa@edu.brain.kyutech.ac.jp
Kazuo Ishii

Dept. of Human Intelligence Systems, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, Japan
E-mail: ishii@brain.kyutech.ac.jp

Abstract
Tomato fruit harvesting is a labor-intensive process, and therefore robotic automation is required. For tomato fruit
harvesting robot, we developed three types of end-effectors (EEs): scissor type, cylinder type and suction-cutting
type. It is necessary to harvest the tomato fruit with calyx especially in Japan. Therefore, the proposed EE has acutter mechanism for cut the peduncle. The results show that the scissor type and the cylinder type EE can cut thepeduncle; however, it is difficult to separate one fruit from the cluster. On the other hand, the suction-cutting type
can separate and cut one fruit from the cluster and therefore it is useful for harvesting tomato fruit with calyx.
Keywords: Tomato-Harvesting Robot, End-effector, Cutting, Suction

technologies such as robot to the primary industry has
become essential, and has high expectations fromagricultural site. Among various agricultural crops,
tomatoes require a long harvesting work time and highlabor load. Therefore, automation for harvesting by
robots is desired. In this research, we develop a tomato
harvesting robot aiming at practical application at
Hibikinada vegetable garden Co., Ltd. for the first time
at a tomato production facility with large infrastructure.

1. Introduction
Population of workers in agricultural field in Japan has
been drastically decreasing each year from 260,600 in2010 and 2097,000 in 2015[1]. The primary industries
have also been suffering from population aging due tohigh rate of departure of young workers. Various
agricultural machines have been introduced to solve thelabor shortage and the increase in burden on agricultural
workers so far. In recent years, the introduction of new© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan544

Bingh Li, Shinsuke Yasukawa, Takuya Fujinaga, Kazuo Ishii

In this paper, we focus on the tomato harvesting end
effector and describe the development of a harvestingmechanism capable of harvesting tomato containing thecalyx.

2. Related Researches
In the fruit harvesting robot, the end effector makes
direct contact to fruit. Various types of mechanisms arebeing introduced to safely harvest, without damaging
both the stem and the fruit. For example, there is amechanism that cuts fruit pattern by electric burning
method and harvests [2]. Some mechanisms utilize thecharacteristics of tomatoes taken out from abscission
layer and pick off from the cluster [3]. Suction method
can be added to pull a single fruit and pick off with
fingers [4]. Other mechanisms cut each tassel with ablade [5] [6], which can also be combined with suction[7]. From these studies, the harvesting mechanism canbe divided as follows, and the end effector is developedwith reference to these methods.
1) Fruit harvesting method: peeling type, cutting type.
2) Fruit guiding method: finger type, sucker type,suction type.

Fig. 1 Hibikinada vegetable garden Co., Ltd. and harvesting
tomato with calyx

fruit one by one. In countries that produce a large
amount of tomato such as Netherlands, many tomatoes
are sold in clusters. However, in Japanese market, single
fruits with the calyx attached are preferred. In fact, thecondition of the calyx indicates freshness of the fruit.
Therefore, it is essential to preserve each fruits’ calyx
when harvesting.
4.

Harvesting robot and End-effector system

4.1. Harvesting robot system
The developed robot is shown in Fig. 2 The robot
consists mainly of a manipulator, a moving cart, acamera, an end effector and a control PC.
In the harvest flow, the carriage moves on the rail until adesired tomato is detected from the image obtained by
the camera. When finding the tomato it stops, themanipulator moves to the measured tomato position,
and a single fruit is harvested from the cluster by theend effector.

3. Work condition and Target tomato
3.1. Work condition
Hibikinada vegetable garden Co., Ltd. is a large-scaleindoor vegetable garden (Fig.1). Tomatoes arecultivated in a system that can be harvested for a longtime. The harvestable tomatoes are adjusted to about 0.8
m to 1.2 m in height and the leaves between them areremoved for easy access to the fruits. There is a rail
between tomato plants where workers performs work
while using dollies. The target tomato in this study is
medium size tomato.

4.2. End-effector system
Following is considered to be requirement of end
effector for harvesting tomato.
① All tomato must include the calyx in order tomaintain its full value.

3.2. Tomato with calyx
There are two types of harvesting tomato. One is toharvest a cluster of tomato and another to harvest each

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

545

End-effector for robotic harvestingFig. 2 Overview of tomato harvesting robot and End-effectors② A single frit must be harvested at a time.
③ The mechanism must not damage any part of theplant or fruit during and after harvesting.
As a way to realize ①, there is picking off and cuttingof peduncle. The picking off method is designed toseparate fruit and peduncle at the part of abscission
layer, but there is a possibility of damaging calyx.
Therefore, cutting is desirable to ensure the harvestingof fruits with a calyx. Since there are several methods torealize ② , we developed three types of prototype
harvesting mechanism.

Fig. 3 Overview of scissor type and mechanism

4.3. Scissor typeThe appearance and operation principle of the scissor
type are shown in Fig.3. This end effector consists offingers and rotary blades, a DC motor and a servomotorthat drive them. The rotary blade rotates from the DC
motor via a pulley and a timing belt. The fingers are
opened and closed by a servomotor using a linkmechanism.
The fingers are divided into top and bottom, during
which there is a gap through which the rotary blade canenter when the finger is closed. The upper finger ismade into a shape that makes it easier to take thepeduncle up to the position of the rotary blade. The
lower finger can grasp the peduncle with the base partfrom the moment of cutting.

Fig. 4 Overview of Cylinder type and mechanism

As shown in the Fig.4, there is a cutting area in theupper part of the cylinder, and it has a shape that canguide the peduncle to the cut section. By pressing thecylinder against the cluster, the tomato fruit enters thecylinder. At that time, since the peduncle also enters the
cutting area, it is possible to harvest the fruit by cutting
the peduncle by rotating the cylinder.
4.5. Suction-cutting typeThe appearance and operating principle of the suctioncutting type are shown in Fig.5 and Fig.6. This typeconsists of a suction port, a finger with a U-shaped
blade, a suction hose and a suction device.
EDF Ducted fan for radio control is used for suction
device. It has an inner diameter of 69 mm and amaximum of approximately 2.65 kg. The EDF and the4.4. Cylinder typeThe appearance and operation principle of the cylinder
type are shown in Fig.4. This type consists of a fixedcylinder and a rotating cylinder. The rotary cylinder has
a function of moving by the motor and cutting thepeduncle.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

546

Bingh Li, Shinsuke Yasukawa, Takuya Fujinaga, Kazuo Ishii

Fig. 5 Overview of Suction-cutting typeend effector are connected by a hose. The diameter of
the tomato to be harvested was about 65 mm at the
maximum, and the suction part at the tip was set to 72
mm, which is the size that can tomatoes can be drawn.
This system utilizes suction force to separate a single
fruit from a cluster and hold it in position for cutting. A
stopper is installed in the suction port to prevent
separation of the fruit and its calyx and stopping it fromentering further into suction hose.
When the tomato is in place, the peduncle is cut byrotating the U-shaped blade-attached finger as shown in
Fig.6.

Fig. 6 Suction-cutting mechanism

References
1. http://www.maff.go.jp/j/tokei/index.html
2. Kollenburg-Crisan, L. M. V, et al, Robotic systems for
cutting fruit and leaves in greenhouse, Proceedings ofWorkshop on European Scientific and Industrial
Collaboration, pp.113~122, Netherlands (2001)
3. Shumpei Uegaki, Hidekazu Araki, Ryo Toshima, Makoto
Shinzaki, Daisuke Ueta, Ryuji Yamazaki, TomatoHarvesting Robot using AI for Environment Recognition,
Panasonic Technical Journal, Vol. 64, No. 1, May 2018
4. M. Monta, et.al., “End-Effectors for Tomato Harvesting
Robot”, Artificial Intelligence Review, Vol. 12, Issue 1,
pp.11-25, 1998
5. Kondo, N., Yata, K., Iida, M., Shiiji, T., Monta, M.,
Kurita, M., Omori, H., Development of an End-Effector
for a Tomato Cluster Harvesting Robot, Engineering inAgriculture, Environment and Food, 3(1), (2010), 20-24
6. Qingchun Feng, Wei Zou, Pengfei Fan, Chunfeng Zhang,
Xiu Wang, Design and test of robotic harvesting systemfor cherry tomato, International Journal of Agricultural
and Biological Engineering, vol 11, No 1(2018), 96-100
7. Arima, S. et al. Harvesting robot for strawberry grown ontable top culture. (Part 1) Harvesting robot using 5 DOF
manipulator. J. SHITA, 13(3), 159-166, 2001

5. Conclusions
In this paper, we described three prototypes of end
effectors. All three types of mechanism are proven to
be able to harvest the tomato fruit with the calyx
attached. However, figure type requires more accurate
image processing and precision manipulation due to thedifficult in pinpointing the exact place on peduncle fromcluster of tomatoes. It is relatively easier to harvestusing cylinder type, however, some tomato may fail toenter into the cylinder due to the structure of the cluster.
Therefore, we found that a mechanism that can actively
separate tomatoes like suction cut type is necessary.
For further studies, it is essential to examine each typeof end effectors and consider the optimal harvest
approach with consideration of their characteristics.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

547

Evaluation for Ball Dribbling Mechanism of RoboCup Middle Size League Soccer Robotin The World Teams
Kenji Kimura, Shota Chikushi, Kazuo Ishii

Department of Human Intelligence Systems
Kyushu Institute of Technology
2-4 Hibikino ,Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196 Japan
E-mail: kimura_kenji@edu.brain.kyutech.ac.jp,

Abstract
RoboCup is a platform designed to promote such research fields as artificial intelligence (AI) and robotics. In recent years
the Middle size league soccer robot has a ball dribbling mechanism. This mechanism controls the ball with two drivingrollers and in order to play soccer games well, ball control capability and holding capacity are indispensable. As the resultof a survey at the 2017 World Cup, all teams in the world have determined the rollers arrangement heuristically, and nomathematical consideration has been made. In this research, we focus the sphere slip velocity and derive relationshipsphere slip speed and sphere mobile speed. and evaluated the roller arrangement in the world teams.
Keywords: The ball dribbling mechanism, sphere slip velocity, dynamical friction

1. Introduction
RoboCup is an international project that promotes AI,
robotics, and related domains. It is an attempt to develop
AI and autonomous robotics research by providing afundamental problem where a wide range of technologies
can be integrated. Even among then, Middle Sized
League soccer competition
makes a goal in the
dynamical environment where many robots corporate
with the human in same space.
In recent years, the world teams have mounted aball dribbling mechanism for controlling the rotationof the ball. Their performances have to be good inorder to win in competition. The rollers are arranged on
the upper half of the ball, enabling them to keep pullingon the ball when driving reverse. A friction between theball and rollers generated by spring in lever supportingroller get the ball toward the robot. When the robot isdribbling with the ball, it is required that the roller rotate
in the ball has approximately the same speed and moves
in the same direction as the robot.
The authors conducted an investigation of the ball
dribbling mechanisms of the world teams (2017
Nagoya in Japan). Table 1 shows the investigation
results such as team name, symbol (pair of figure
and team’s initial letter), roller types and roller(a) Case of |𝛼𝑖 | = 0°(b) Case of |𝛼𝑖 | > 0°
Fig. 1 Two rollers arrangement in ball dribblingmechanism

Table 1

Roller type and angle in world teams

Team name

SymbolRoller type| 𝛼𝑖 |

RV-infinity[1]■Rconstraint

0°

Tec United Turtles[4]●Tconstraint

10°

Falcons●F

constraint

10°

Hibikino-Musashi[5]▲Hconstraint

20°

NuBot[6]▲N

constraint

20°

Water

◆W

constraint

30°

CAMBADA[3]—unconstraint

50°

arrangement angle. Fig. 1 shows rollers axis
arrangement patterns with respect to plane including
two contact points and center of sphere. As show in© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

548

Kenji Kimura, Shota Chikushi, Kazuo Ishii

Fig.1(a), RV-Infinity[1] is adopted non-slip
roller arrangement as |𝛼𝑖 | = 0° . B e c a u s e i t i s
condition that two rollers axes are on the sameplane (See[2]). On the other hand, as show in Fig.1(b), the others teams are adopted slip roller
arrangement as |𝛼𝑖 | > 0°. Because it is conditiont h a t two rollers axes are not on the same plane(See[2]). But, CAMBADA[3] doesn’t cause slip byusing unconstraint rollers(omni-rollers).
Most of the teams including Tec United Turtles[4],Hibikino-Musashi[5] and NuBot[6] and focus ball
Handling force and discuss roller arrangement.
However, they discuss it without slip and determine
roller arrangement by experiments, heuristically. Slip
can cause sphere mobile speed efficiency loss. But,
it can cause friction and improve ball holding force
and sphere rotational stability. In this way, both are
related to transactions and slip thought to beimportant factor for ball holding force.
In this study, we focus to sphere slip velocity
with respect to rollers and reveal relationship sphereslip speed and sphere mobile speed on horizontal planein backward movement and evaluate the roller
arrangement in the world teams by experiment using
device robot.
2. The sphere rotational motion by two constraintrollers
In this chapter,using sphere angular velocity vector
allow for slip caused by two constraint rollers, we
calculate sphere slip velocity and sphere mobile
speed.
2.1 The sphere angular velocity vector by two
constraint roller allow for slip
As show in Fig. 2, the sphere which have radius r andcenter 𝑶 is fixed as origin of coordinate system 𝛴 − 𝑥𝑦𝑧.
i-th constraint roller is contacted with sphere at theposition vector 𝑷𝒊 , and i-th roller is arranged such thatmass of roller, 𝑷𝒊 and 𝑶 are on the same line. 𝝎 denotes
the sphere angular velocity vector. 𝜼𝒊 denotes the unit
vector along i-th roller’s axis. 𝛼𝑖 ( −90° ≤ 𝛼𝑖 ≤ 90° )denotes roller arrangement angle between 𝜼𝑖 and𝐬𝐩𝐚𝐧{𝑷𝟏 , 𝑷𝟐 } (see Fig.1). When peripheral speed
𝜈1 and 𝜈2 are given. 𝝎 can be represented by eq.(1).
𝝎=1√𝑟 𝟒 − 〈𝑷1 , 𝑷2 〉𝟐

Fig. 2 Variables related to sphere rotational movement

Where
𝑷𝒊 = 𝑟[cos𝜃1,𝑖 cos 𝜃2,𝑖 , sin 𝜃1,𝑖 cos 𝜃2,𝑖 , sin 𝜃2,𝑖 ](2)

2.2 The sphere slip velocity and sphere mobile speed
In each roller, slip occurs when two contacting
surfaces (roller and sphere) move such that the roller
velocity 𝒗𝑹𝒊 and the sphere velocity 𝒗𝑺𝒊 at the tangent
plane at point 𝑷𝒊 are different (See Fig. 2). 𝒆𝑖 denotes
unit vector along roller velocity. Using 𝝎, sphere slip
velocity 𝜻𝒊 which is relative speed with respect to 𝒗𝑹𝒊 canbe represented by eq.(3).
𝜻𝒊 = 𝒗𝑺𝒊 − 𝒗𝑹
𝒊

(3)

𝑺
𝒗𝑹
𝒊 = 𝜈𝑖 𝒆𝒊 , 𝒗𝒊 = 𝝎 × 𝑷𝑖

(4)

Where
𝑿𝒊 is tangential unite vector of great circle 𝑪𝑮 (passthrough 𝑷1 and 𝑷2 ), 𝒀𝒊 is unit normal vector of𝐬𝐩𝐚𝐧{𝑷𝟏 , 𝑷𝟐 } . 𝜻𝒊 can be represented as linear
combination of 𝑿𝒊 and 𝒀𝒊 as follow.
𝜻𝒊 = 𝑆𝑖 𝑿𝒊 + 𝑇𝑖 𝒀𝒊(5)𝑆𝑖 = 〈𝜻𝒊 , 𝑿𝑖 〉, 𝑇𝑖 = 〈𝜻𝒊 , 𝒀𝑖 〉 [m/s]

(6)

Where

Thus. ‖𝜻𝒊 ‖ can be calculated by eq.(7).
‖𝜻𝒊 ‖ = √𝑆𝑖2 + 𝑇𝑖2 [m/s](7)And. Using components 𝜔𝑥 , 𝜔𝑦 of 𝝎 , norm of spheremobile velocity 𝑽 on the 𝑥𝑦-plane (horizontal plane) canbe represented by eq.(8).

[(𝜈2 cos 𝛼2 )𝑷𝟏 − (𝜈1 cos 𝛼1 )𝑷𝟐 ]𝜈1 sin 𝛼1 + 𝜈2 sin 𝛼2 𝑷1 × 𝑷2+
‖𝑷1 × 𝑷2 ‖2𝑟

𝑇‖𝑽‖ = 𝑟√𝜔2𝑥 + 𝜔2𝑦 [m/s](1)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, Beppu, Oita, Japan

549

(8)

Evaluation for Ball Dribbling

Table 2 Target values in 𝜈1 = 𝜈2 = −0.91 [m/s]
|𝛼𝑖 |

0°

10°

20°

30°

𝑆1 [m/s]

0

-0.16

-0.30

-0.41

𝑇1 [m/s]

0

0

0

0

𝑆2 [m/s]

0

0.16

0.30

0.41

𝑇2 [m/s]

0

0

0

0

‖𝜻𝟏 ‖ [m/s]

0

0.16

0.30

0.41

‖𝜻𝟐 ‖ [m/s]

0

0.16

0.30

0.41

‖𝑽‖ [m/s]

1

0.98

0.93

0.81

(a) Shpere slip velocity (left side roller)

Fig. 3

Roller positions on sphere, existence of sphere slip
vector and dynamical friction in backward movement(b) Shpere slip velocity (right side roller)(c) Shpere mobile speed and slip speed

Fig. 4 Comparison theoretical value and experimental value case of |𝛼𝑖 | = 0°(a) Shpere slip velocity (left side roller) (b) Shpere slip velocity (right side roller) (c) Shpere mobile speed and slip speed
Fig. 5 Comparison theoretical value and experimental value case of |𝛼𝑖 | = 10°(a) Shpere slip velocity (left side roller)(b) Shpere slip velocity (right side roller)(c) Shpere mobile speed and slip speed

Fig. 6 Comparison theoretical value and experimental value case of |𝛼𝑖 | = 20°

3. Experiment𝜃1,1 = 215°, 𝜃1,2 = 325°, 𝜃1,2 , 𝜃2,2 = 60°, r = 0.1[m].
Five experiments were conducted, each at the same four
different degrees angles ( 𝛼𝑖 0° , 10° , 20° , 30° ) . As
shown in Table 2, 𝑆𝑖 , 𝑇𝑖 , ‖𝜻𝒊 ‖ and ‖𝑽‖ are ideal target
values calculated using 𝜈1 = 𝜈2 = −0.91[𝑚/𝑠] f r o meq.(6)(7)(8). Fig . 4 -6 are indicated theoretical data:
𝑆𝑖𝑚 , 𝑇𝑖𝑚 , ‖𝜻𝒊 ‖𝑚 ‖𝑽‖𝑚 and experimental data:
𝑆𝑖𝑒 , 𝑇𝑖𝑒 ,‖𝜻𝒊 ‖𝑒 , ‖𝑽‖𝑒 (calculated in interval 0-8[s]).
In the evaluation of case ( |𝛼𝑖 | = 0°, 10°, 20°), 𝑆𝑖𝑒 , 𝑇𝑖𝑒 ,In this chapter. Using device robot, we experimented
to measure sphere mobile speed, sphere slip velocity toverify theoretical formula and consider distribution
sphere slip velocity in backward movement. Finally, weevaluate world team’s roller arrangement.
3.1 Verify theoretical formula
As show in Fig. 3, the conditions are given as follows:© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan550

Kenji Kimura, Shota Chikushi, Kazuo Ishii(a) End points of 𝜻𝟏 on span{𝑿𝟏 , 𝒀𝟏 }(b) End points of 𝜻𝟐 on span{𝑿𝟐 , 𝒀𝟐 }Fig. 7 The distribution of end points of sphere slip velocity‖𝜻𝒊 ‖𝑒 and ‖𝑽‖𝑒 are close to 𝑆𝑖𝑚 , 𝑇𝑖𝑚 , ‖𝜻𝒊 ‖𝑚 and‖𝑽‖𝑚 , respectively (see Fig. 4-6).
In the evaluation of |𝛼𝑖 | = 30°, By limited motor
drivers and influence of intense dynamical friction
which caused between roller and sphere and roller
surface could have heat. So, we had stopped the secondand subsequent experiments. However, 𝑆𝑖𝑒 , 𝑇𝑖𝑒 , ‖𝜻𝒊 ‖𝑒and ‖𝑽‖𝑒 are close to 𝑆𝑖𝑚 , 𝑇𝑖𝑚 , ‖𝜻𝒊 ‖𝑚 and ‖𝑽‖𝑚 .
Therefore. The validity of theoretical formulas isverified.
3.2 Consideration of sphere slip velocity
Theoretically, In case of 𝜈1 = 𝜈2 = −0.91[𝑚/𝑠], 𝜻𝟏and 𝜻𝟐 are tangent to great circle 𝑪𝑮 pass thorough 𝑷𝟏
and 𝑷𝟐 on sphere from 𝑇1 , 𝑇2 = 0 in Table 2). The
validity of this fact is verified by experiment as follow.
Fig 7 shows distribution end point of sphere slip
velocity 𝜻𝒊 on 𝐬𝐩𝐚𝐧{𝑿𝒊 , 𝒀𝒊 }. The horizontal axis and thevertical axis show 𝑆𝑖 and 𝑇𝑖 , respectively. All thevalues show mean value of 6-8[s]. □◯△ indicated ascoordinate by using target values in Table 3. ■●▲indicated as coordinate by using experimental meanvalues calculated in interval 6-8[s](See Table 1).
1) horizontal coordinate
In ■R , 𝑆𝑖𝑒 are almost ±0.04[m/s]. In ●T and ●F, 𝑆𝑖𝑒
are almost ±0.16[m/s]. In ▲H and ▲N, 𝑆𝑖𝑒 are almost±0.30[m/s].
2) vertical coordinate
In all dates, 𝑇𝑖𝑒 are almost 0.04[m/s] and close to zero.
From 1), 2), 𝜻𝟏 and 𝜻𝟐 are almost equal to tangential
vector on great circle and turn round to outward (SeeFig. 3). Therefore, the validity of the sphere slip
velocity was verified. And, dynamical friction 𝑭𝒊 are
generated by 𝜻𝒊 in opposite direction. These can exert
like grabbing ball (See Fig. 3(b)).
3.3 Evaluation of world teams’ rollers arrangement

Fig. 8 Relationship sphere slip speed
and sphere mobile speed

Fig. 8 shows relationship sphere slip speed and spheremobile speed. The horizontal axis and the vertical axis
show ‖𝜻𝒊 ‖ and ‖𝑽‖ , respectively. Dotted line istheoretical curve.
1) vertical coordinate (mobile speed efficiency)
In ■R, ●T and ●F, Sphere mobile speed ‖𝑽‖𝑒 arealmost distributed in the range of 0.95 to 0.97 [m/s]. In▲H and ▲N, sphere mobile speed ‖𝑽‖𝑒 are distributedin the range of 0.89 to 0.90 [m/s]. Thus. ■R, ●T and●F have sphere speed efficiency better than ▲H and▲N.
2) horizontal coordinate (sphere slip speed)
In ■R, ‖𝜻𝒊 ‖𝑒 are almost distributed in the range of 0 to
0.05[m/s]. Thus, dynamical friction 𝑭𝟏 and 𝑭𝟐 arealmost zero. In ●T and ●F, ‖𝜻𝒊 ‖𝑒 are almost
distributed in the range of 0.12 to 0.15 [m/s]. Thus,
dynamical friction 𝑭𝟏 and 𝑭𝟐 can exist moderately. In▲H and ▲N, ‖𝜻𝒊 ‖𝑒 are almost distributed in the rangeof 0.28 to 0.35 [m/s], dynamical friction 𝑭𝟏 and 𝑭𝟐 can’t
exist moderately. In fact, that roller surfaces have heatand be worn away by such dynamical friction. From 1),2), it is determined that ●T and ●F are adopted
optimum arrangement as |𝛼𝑖 | = 10°.
4. Conclusion
In this study, we focus the sphere slip velocity andderive relationship sphere slip speed and sphere mobile
speed. and evaluated the roller arrangement in theworld teams. As a result , Tech United Turtles, Falcons
have adopted optimum roller arrangement. As a futuretask, we will propose kinetic model allow for the friction
between sphere and rollers in detail.
Reference© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, Beppu, Oita, Japan

551

Evaluation for Ball Dribbling

[1]

[2]

[3]

[4]

[5]

[6]

H. Suzuki, T. Yoshida, K. Watanabe et al., RV-Infnity
Team Description Paper 2017, RoboCup 2017 Nagoya(Nagoya, 2017)
K. Kimura, K. Ishii, Y. Takemura, M. Yamamoto,“Mathematical Modeling and motion analysis of thewheel based ball retaining mechanism”, SCIS & ISIS
(2016), pp.4106-4111
B. Cunha, A. J. R. Neves, P. Dias et al.,
CAMBADA’2015: Team Description Paper, RoboCup
2015 Hefei (Hefei, 2015)
Rob.Hoogendijk: “Design of a Ball Handling Mechanism
for Robocup” Bachelor End Project, departement
Mechanical Engineering, Technische Universiteit
Eindhoven
K. Fujimoto, K. Ishii, Y. Kitazumi, Development of aball handling mechanism for a RoboCup Soccer Robot,
Robotics and Mechatronics Conference (Okayama,2011)
R. Junkai, X. Chenggang, X. Junhao et al., A Control
System for Active Ball Handling in the RoboCup Middle
Size League, 2016 Chinese Control and Decision
Conference (CCDC), (IEEE, Yinchuan, 2016), pp. 43964402© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan552

Robot-Control Method Based on Personal Space
Kota Kawamoto, Ryusuke Fujisawa, Eiji Hayashi

Department of Mechanical Information Science and Technology, Kyushu Institute of Technology
680-4, Kawazu, Iizuka-City, Fukuoka, 820-8502, Japan
E-mail: kawamoto@mmcs.mse.kyutech.ac.jp , fujisawa@.ces.kyutech.ac.jp, haya@.mse.kyutech.ac.jp

Abstract
In our daily life, we unconsciously adjust our personal space according to the intimacy with others. It is also said that
personal space is classified into non-verbal communication. In this research, we analyzed the distance that a person
can tolerate when the robot approaches to identify the minimum distance that the person is able to interact comfortably.
As a result, we could get parameters which can be index of initial distance between robot and users.
Keywords: personal robot, personal space, nonverbal communication, intimacy

determine the interactions they can have with other people. In a word, we all are creating our own portable bubble in public space to keep others far from each other.
Interpersonal distances are classified into non-verbal
communication. Furthermore, the interpersonal distance
between them is reflected the social relationship they
have. E.T. Hall has proposed a model to represent thesedifferent spaces; it divides the space around a person in
4 distinct zones (i), (ii), (iii), (iv), as shown in Figure 1.

1. Introduction
Nowadays, the rapid development of high technology
has produced robots not only for industrial factories (industrial robots) but also for museums, homes, healthcare
institutions and so on (non-industrial robots). Among
various types of non-industrial robots, recently, service
and personal robots have attracted global attention[1].
What makes them unique is that they are required tohave "user affinity" that gives the user a sense of affinity as well as functions such as intellectual behavior andcommunication ability. Generally, human communication can be divided into two simple categories verbal
communication and nonverbal communication. Moreover, it is said that almost 60 percent of the human communication consists of nonverbal communication
whether through facial expression, the tone of voice orbody movement[2]. All of these are essential to conveying intent or emotional state. Thus, robot control systemshould be considered the impression from the standpointof users.
E.T. Hall introduced for the first time the concept ofinterpersonal distances[3]. People create their own territories around them unconsciously, which define and(i) Intimate distance(0~0.46m)(ii) Personal distance(0.46~1.2m)(iii) Social distance(1.2~3.6m)(iv) Public distance(3.7m~)

[m]
0

1

2

3

4

5

Fig. 1. Definition of Hall’s personal space model.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

553

Kota Kawamoto, Ryusuke Fujisawa, Eiji Hayashi

He defined that the Intimate distance is for embracing,
touching or whispering, the Personal distance is for interactions among good friends or family, the Social distance is for interaction among acquaintances, the Publicdistances used for public speaking. Moreover, spacewithin (i) intimate distance and (ii) personal distance iscalled personal space.
In this research, we designed the robot system andconducted an experiment with considering the personal
space between human and robots.

Control Computer
Target vector(Vx,Vy,ω)ObjectKinect v2

Location

Microcontroller

BLDC Motor

Motor Driver

Rotation
Speed
[rpm]

Voltage
Output[V]

Fig. 3. Control diagram

2. Structure of the Robot

4.

The robot we developed is depicted in Figure 2. This
robot consists of 4 major parts: the head with 2
degrees of freedom, the Body with 3 degrees of
freedom, the Arm with 6 degrees of freedom and theBase with a 4-wheel-drive. The height of the robot isapproximately 1.5 m.

Motion Model of the Moving Device

The mecanum wheel adopted for the Base part iscovered with a barrel whose surface is inclined at 45
degree with respect to the axle. By controlling therotation direction of each wheel as shown in Figure 4,
omnidirectional movement is realized by a
combination of rotation of the wheel and movementof the barrel on the circumference.

Head

Arm

Base

Body

Fig. 2. Structure of the robot
3.

Fig. 4. Various combination of wheel rotation

System Configuration

Let V0 be the vector according to the angular velocity.
be a velocity vector in rectangular coordinates.
Let
In equation (3), "I" represents the distance from the center of gravity to the lateral edge, and "L" represents the
distance to the longitudinal edge.

The control diagram is depicted in Figure 3. In thissystem, each part of the robot constructs a control
system. The control flow of the Base part is as follows.
First, a user is detected by the Kinect V2 sensor, anda target vector command is sent from the PC to themicrocomputer according to the information received
from the sensor. The microcomputer calculates speed
and direction from each parameter and control the
motors via the servo controller.

=[
] ∈
=[
] ∈ ×
1 −1 −( + )
( + )
1 1
=
1 1 −( + )1 −1 ( + )×© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

554

(1)
(2)
(3)

Paper Title (First 4 Words)

Vw can be expressed by using the equation (2), (3).
=∙As the velocity increased, the standard deviation tended
to be higher. However, there was no significant relationship between the velocity and the distance. The average
distance was almost same as any velocity.

(4)

Here, the speed control of the moving device gives thevector ( x, , ω ) as input to the microcomputer andcontrols the speed Vw of each wheel by PWM output.

0.9
0.8
0.7
0.6
Distance[m]5. System Flow of the Movement Control
Figure 5 shows the flowchart of the movement controlof the Base part. In the movement control, base part iscontrolled so as to keep a specific position according to
the sensor value. Specifically, the distance and angle arecalculated, and the robot approaches the user until the
distance of the threshold.

Average value
vertical line :standard
deviation

0.5
0.4
0.3
0.2
0.1
0
0.1

0.2

0.3

0.4

0.5

0.6

0.7

Velocity[m/s]Fig. 6. Measurement result
7. Evaluation of the personal distanceBased on the measurement values obtained from theprevious experiment, the evaluation experiments wereconducted. The moving speed was controlled within therange of 0.0 ~ 0.3 [m/s] so that the boundary of the personal space was kept about 0.6 m, which was the average distance of the previous experiment.
The evaluation experiment was conducted as follows.

Fig. 5. Flow chart of location control(i) A subject sat on the chair on casters in front of therobot.
(ii) The subject moved freely in the specified range fora minute.
(iii) The robot kept the distance as shown in Figure 7.
(iv) After the experiment, the subject answered thequestionnaire of the impression about the robot’s
approach. Repeat for the 12 subjects.

6. Measurement of the personal space
As I said in chapter 2, People update, control, and adjust their personal spaces continuously. Therefore, psychological strains between human to robot could bemeasured by physical distances.
In this time, the personal space between robot andsubject were measured using the following procedures.
(i) The robot approached at each velocity (0.2, 0.3, 0.4,
0.5, 0.6[m/s]) to a subject who was standing(ii) The subject raised his hands as a signal if he felt
uncomfortable (too close to the robot).
(iii) The distance was measured by a sensor installed at
the front of the robot and these procedures wererepeated for the 10 subjects.

Uncomfortable

If anything,uncomfortable

Neither

If anything,comfortable

Comfortable

Test

Fig. 7. Evaluation questionnaire
The result of the evaluation experiment is depicted
in Figure8. Each result of the questionnaire was quantified as shown in Table 2.

The measurement result is depicted in Figure 6.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

555

Kota Kawamoto, Ryusuke Fujisawa, Eiji Hayashi

6

Evaluated Value

5
4
3
2
1
0
1

2

3

4

5

6

7

8

9

10

11

12

Subject number

Fig. 8. Evaluation result
Table. 2. Conversion table for the result

Evaluatedvalue

Uncomfortable
1

----2

Neither
3----4

Comfortable
5

Table.3.Average and standard deviation of the result
average
3.8Standard deviation

1.1

The average score is shown by a red horizontal line. The
average score was 3.8 and close to “comfortable”.
8. Conclusion and Future work
It is considered that the velocity range and the distancecould be adapted to realize the appropriate approach tomost of the users. It means that the distance can be index of initial distance between robot and users. However, to optimize the personal space for each users, weneed to create the database to store the information.
References
[1]
[2]
[3]
[4]

S.Yumitori, Overview of Robot R&D in NEDO
K.Buhardwaj, Professional Communication
E.T. Hall, The Hidden Dimension (Vol. 609. Garden City,
NY: Doubleday, 1966).
J.leroy, M.mancas and B.gossel, Personal Space Augmented Reality Tool

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan556

Recognition Method of Target Objects for Autonomous Tomato Harvesting Robot
Keiji Kamei*

Department of Production Systems, Graduate School of Engineering, Nishinippon Institute of Technology
1-11 Aratsu, Kanda, Miyakogun, Fukuoka 800-0394, Japan†
Hiroyuki Moriyama

Department of Information Systems, School of Engineering, Nishinippon Institute of Technology
1-11 Aratsu, Kanda, Miyakogun, Fukuoka 800-0394, Japan
E-mail: kamei@nishitech.ac.jp
www.nishitech.ac.jp/~kamei/

Abstract
Recently, the shortage of agricultural workers and depression of agricultural production efficiency have been
said in Japan. To solve those problems, the researches that apply information and robot technology are studied
actively. The aim of our research, we focus on the autonomous tomato harvesting robot, and develop the recognition
method for target objects. The color identification from a camera and distance measurement between target tomatoes
and a robot are key points. To solve those problems, we propose to identify color based on HSV values using LookUp-Table for acceleration of calculation and to measure distance for tomatoes using depth camera. From theexperiments, we succeeded in acquiring a tomato from a camera and in measuring distance.
Keywords: Computer Vision, Recognition of Target Objects, Autonomous Robot, Harvesting of Field Crops.

The common aims for the harvesting robots whichhave been researched and developed so far aredevelopment of the functions for full automation
harvesting of crops in order to increase production
efficiency. Generally, those robots use cameras forrecognition of crops color and shape, and use sensors,e.g., laser range finder, to measure distance between arobot and crops. However, those robots have problems of
economic feasibility. In addition, those take slowly
actions, so that the number of harvesting crops within
unit time is a small amount compared with agricultural
workers. For this, it can be said that the autonomous
harvesting robots have not yet put into use on actual farm.
In this research, our aim is development of therecognition system of target crops for the harvesting

1. Introduction
According to the report from Japanese Ministry ofAgriculture, Forestry and Fisheries(MAFF)1, theshortage of agricultural workers and depression ofagricultural production efficiency have been said inrecent Japan. From that report, the number of agriculturaland forestry workers decreases steadily with each passing
year; in addition, the number of that elderly workers who
are over 65 years old have been increasing. To solve
those problems, the researches that apply information androbot technology are studied actively, and those arecalled “Smart Agriculture.” In this research, we focus on
the automation system for harvesting, which is theharvesting robot, of field crops in Smart Agriculture.
*

In 2007 he obtained his PhD at the Department of Brain Science and Engineering, Kyushu Institute of Technology. From April, 2007 to March, 2014,
he was a lecturer. Since April, 2014, he has been an associate professor, Nishinippon Institute of Technology.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

557

Keiji Kamei and Hiroyuki Moriyama

robot, and target crop is tomatoes because our harvesting
robot participate in “Tomato Robot Competition2.” Our
harvesting robot is developed by joint team oflaboratories at Nishinippon Institute of Technology, andwe take charge of the recognition system on that robot.
The color identification from a camera and the distancemeasurement between target crops and a robot are key
points. To solve those problems, we propose to identify
the color based on “HSV values” using “Look-UpTable(LUT)” for acceleration of calculation and tomeasure distance for target crops using “Depth camera.”
From the experiments on Tomato Robot Competition, we
succeeded in recognizing a tomato from camera and inmeasuring distance.

Fig. 1. Overview of competition field. (Ref. The homepage of“5th Tomato Robot Competition2.”)2.1. Qualifier 1
Tomatoes are hung separately one by one in the tomatoes
area. A harvesting robot moves from the home position
toward front of a tomato, then harvests tomatoes. The
minimum requirement to pass the qualifier 1 stage is totouch a tomato. The teams that pass the qualifier 1 stage
are able to go on the qualifier 2 stage.

2. Tomato Robot Competition
In Tomato Robot Competition, there are two categories
for the harvesting robot style, i.e., “Rail Style” and “FreeStyle,” and there are two sections for the control of arobot in each category, i.e., “Remote Control” and“Autonomous Control.” We participate in “Rail Style”
and “Autonomous Control.” Rail style category is that a
harvesting robot moves on the rail which is laid in
advance. The competition is that a robot harvests as many
tomatoes as possible in the time limitation. In addition to
the number of tomatoes, the states of tomatoes, i.e., colorof tomatoes, bruises on skin of tomatoes, etc., areelements of deduction or addition to the score. The
competition consists of 3 stages, i.e., qualifier 1(Q1),
qualifier 2(Q2) and final stage. Table 1 shows the
specifications of field for all of stage of competition, andFig. 1 illustrates the overview of competition field.

2.2. Qualifier 2
The difference of the condition for the tomato area
between the qualifier 1 stage and the qualifier 2 stage isthat tomatoes are hung as bunches of tomatoes. The
minimum requirement to pass the qualifier 2 stage is toharvest at least one tomato. The teams that pass thequalifier 2 stage are able to go on the final stage.
2.3. Final stage
In the final stage, there are several plants of tomatoes inthe tomato area, and a robot harvests ripe tomatoes which
are selected by recognition system from plants. The scorewill be deducted if a robot harvests tomatoes are not ripe.
3. Tomato Harvesting Robot and Recognition
SystemTable 1. Specifications for competition field(RailStyle category). Unit is millimeter.
Stage
Distance
between rail andtomatoes rack
Distance
between rail andtomatoes
Distance
between floor
and tomatoes
Diameter ofpipe for rail
Rail gauge

Q1

Q2

Our harvesting robot is developed by joint team oflaboratories at Nishinippon Institute of Technology. We
explain our harvesting robot and recognition system fortarget tomatoes in this section.

Final
200
100 ~ 400

200

3.1. Hardware of harvesting robot
The hardware of our harvesting robot is developed by
Takemura Laboratory at Nishinippon Institute ofTechnology. Fig. 2 shows the overview of our harvesting
robot. This robot has a pair of harvesting scissors, andthat scissors are able to expand and contract by air

800 ~ 1200
∅50600(±5)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan558

Recognition Method of Target

cylinder. A sensor for recognition of tomatoes is set on
upper side of a harvesting arm.

image. Subsequently, that system attempts to sample
green area that is neighboring to red area in order to findstem. Then, it measures the distance to red and green areafrom depth sensor, and selects red and green objects
which have similar distance. Finally, recognition systemdetermines red as a tomato and green as stem.
Start

Robot
Controller

Getting an Image
Sampling of red area

Fig. 2. Overview of tomatoes harvesting robot.

Find / Center?
Y
Sampling of green area
neighboring red area

3.2. Recognition system of target crops
We take charge of the recognition system on our
harvesting robot. We have to recognize tomatoes and tomeasure the exact distance between a robot and tomatoes;
therefore, we propose to apply a sensor that is integrated
camera and depth sensor to recognition and measuring.
There are several types of such sensor, we choose “IntelRealSense R2003, 4” because that sensor connects via
USB3.0 and so does not need external power source.
Table 2 and Fig. 3 shows the specifications of IntelRealSense R200 and overview, respectively.

Resolution
Aspect ratio
Viewing angle
Frame rate

N

Measuring the distance to red andgreen area from depth sensor
Selection of red and green objects
in similar distance
Recognition
Red: Tomato, Green: Stem

Fig. 3. The flowchart of recognition of a tomato.

We apply conversion of color space of taken images,i.e., RGB to HSV, because HSV images have robustness
for alternation of luminance. In process of conversion, weapply Look-Up-Table(LUT) method to conversion ofcolor space. LUT method converts color space based onconversion table that is calculated in advance; therefore,a pixel value in RGB is converted by a corresponding
value in HSV on conversion table. As a consequence,
LUT method allows calculation for recognition of redarea(tomatoes) and stem to accelerate.

Table 2. Specifications for Intel RealSense R2003, 4
Camera sensor(Color image)
1920, 1080
16 : 9
H:70, V:43, D:77
30[fps]MovingsignalSelection of a red object thenearest to center of an image

Distance sensor(Depth image)640, 480
4:3
H:59, V:46, D:70
60[fps]

4. Experiments
We conduct 2 kinds of exmpriments, i.e., in laboratory
and in Tomato Robot Competition. This section shows
those results.

Fig. 2. Overview of Intel RealSense R200.

Fig. 3 illustrates the flowchart of a tomato recognition
and the process of distance measurement. Firstly, therecognition system attempts to sample red area fromtaken color image in order to find tomatoes. That systemsends moving signal to robot controller if it does not find
red area or red area is not center of a taken image. Robot
moving signal is sent until red area is centered in an

4.1. Experiment 1: In Laboratory
This experiment demonstrates capability of recognition
for a tomato and stem, and validates accuracy ofmeasurement distance between the sensor and a tomato
from depth image. We use an imitation tomato in thisexperiment. Fig. 4(a), (b) and (c) show the original© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan559

Keiji Kamei and Hiroyuki Moriyama

image(input), recognition of red area(tomato) andrecognition of green area(stem), respectively.

does not affect to harvest because harvesting scissors
moves to the farther than actual distance.
4.2. Experiment 2: Tomato Robot Competition(a)In the competition, our harvesting robot was not able totouch and to harvest a tomato in time limitation, so that
our robot failed to pass the qualifier 1. The reason for
failure is that our robot was not able to move until redarea(a tomato) is centered in a color image due toprogramming failure of robot controller; namely, ourrobot continued to move back and forth so that a tomato
is at the center of an image. We took part in to thequalifier 2 as an open participant. In the qualifier 2, we
modified the program of robot controller. The result ofqualifier 2 is that our robot was able to move toward frontof a tomato and to touch that stem; consequently, it canbe said that the proposal in this research succeeded inrecognizing of a tomato and in measuring of distance.

(b)

(c)
Fig. 4. Original and Processed images. (a)Original(input),(b)Recognition of red area, (c)Recognition of green area.

From the result of comparison of Fig. 4(a) and (b), it isable to recognize a tomato from red area, and so our
proposal succeeded in extracting the image of a tomato.
The center of visible region in Fig. 4(c) shows the stemat the joint to a tomato, so that succeeded in finding the
cutting point. Next, we validate accuracy of measurement
distance by 2 kinds of distance between cutting point ofstem and sensor; those are set to 200[mm] or 500[mm].
The number of trials for validation is 10 times for eachdistance. Table 3 shows the results of validation.

5.

The shortage of agricultural workers and depression ofagricultural production efficiency have been said. To
solve those problems, the researches that apply the
information and robot technology are studied actively.
The aim of in this research is development of tomato
harvesting robot. Our robot is developed by joint team,and we take charge of the recognition of tomato. From
the results in laboratory, our proposal succeeded inrecognizing a tomato and in finding the cutting point. In
addition, our proposal were able to measure the distancefor cutting point within allowable error. By contrast, ourrobot was not good results in Tomato Robot Competition;
however, our robot was able to touch the stem of a tomato
in the end, so that our proposal succeeded in recognizingof a tomato and in measuring of distance.

Table 3. The results of validation of measuring distance
Distance
Trial 1
Trial 2
Trial 3
Trial 4
Trial 5
Trial 6
Trial 7
Trial 8Trial 9Trial 10
AverageStandard deviation

200[mm]202.531203.188204.125
213.719214.031213.656214.188214.156213.938214.938210.847
5.246

Conclusions

500[mm]510.031517.313514.469515.938510.250514.188509.281517.031516.719512.125
513.735
3.100

References
1. Ministry of Agriculture, Forestry and Fisheries, Statistics
on agricultural labor force(in Japanese),
http://www.maff.go.jp/j/tokei/sihyo/data/08.html.
2. Center for Socio-Robotic Synthesis, Tomato Robot
Competition,
http://www.lsse.kyutech.ac.jp/~sociorobo/ja/tomatorobot2017.
3. Intel Corporation, RealSense, https://realsense.intel.com/.
4. R. Hatsune et. al., Intel RealSense SDK Sensor
Programming(in Japanese), 1st edn. (SE Book, 2015).

From the results in Table 3, the measurement error isabout 10[mm], and measurment distances tend to be long
distance compared with actual diatances. The reason for
those results is that there are occasional times that sensor
measures the distance between a leaf behind stem on hullof a tomato and sensor. However, this measurement error© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

560

Color Image Arrangement Based on Histogram Matching
Using Smoothed Brightness Histogram (I) --- Overall Smoothing --Yusuke Kawakami*, Tetsuo Hattori, Yoshiro Imai, Koji Kagawa, Yo Horikawa

Graduate School of Kagawa University, 2217-20 Hayashi
Takamatsu City, Kagawa 761-0396, Japan
R. P. C. Janaka Rajapakse

Tainan National University of the Arts, 66 Daci
Guantian District, Tainan 72045, Taiwan
E-mail: riverjp2002@gmail.com, {hattori imai, kagawa, horikawa}@eng.kagawa-u.ac.jp, janaka@mail.tnnua.edu.tw

Abstract
Although we have already reported that automated color image arrangement by Histogram Matching based on
Gaussian Distribution (HMGD) gives good feeling impression if the brightness histogram of the original image hasa single peak. However, if there are multiple peaks in the histogram, the HMGD processing does not always bringgood results. In this paper, we propose a Histogram Matching method using overall smoothing over the brightness
histogram of original image as the reference one instead of using Gaussian distribution. Also in this paper, we
present the experimental results.
Keywords: Image processing, Brightness Histogram, Histogram matching, HMGD

1. Introduction
D processing does not always give good feeling
impression. Through the experimental results, we haveconsidered if the reference histogram make change tosatisfy brings good results whether or not the number oforiginal image brightness histogram peaks7.
In this paper, we propose a Histogram Matching method
using overall smoothing over the brightness histogramof original image as the reference one instead of using
Gaussian distribution. Also in this paper we illustrate
the experimental results using proposed Histogram
Matching method.

These days, automated image processing forenhancement of color images has been more familiar tous, for example, Digital Signage, Smart Phone, etc1-3.
In the previous paper, we have already presented thatthe Histogram Matching based on Gaussian Distribution(HMGD) processing is one of the automated image
arrangement method using Elastic Transformation4-5
based on the brightness axis and gives good feeling
impression if the original images have single peak6.
However, we have understood that if the original
images have multiple-peak brightness histogram, HMG-

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

561

Yusuke Kawakami , Tetsuo Hattori, Yoshiro Imai, Koji Kagawa, Yo Horikawa, R. P. C. Janaka Rajapakse

2. Principle of Histogram Matching based on
Gaussian Distribution (HMGD)

And we can derive Eq. (7) from differential of Eq. (6).
xd
L f x dx
dx 0

In the section, we describe the principle of HMGD
processing.
Fig. 1 shows the conceptual image of HMGD. Let f(x)and h(y) be two probabilistic density functions (PDF) on
real variables x and y, respectively. The PDF iscorresponding to histogram of image brightness level
which is discretely defined.
In addition, let y= (x) be a continuous and monotonic
increase function corresponding to cumulative
histogram of image brightness level between variables x
and y 7-9. And let y= (x) be defined by Eq. (1).
x

y

x

L ' x

y

(8)

y

h(y)

y= (x)(L)(x0+dx)
y
(x)
L

(2)

h y Lf x .

dz. (7)2

L ' z

y= (z)

At first, we have to expand brightness level of original
image histogram and convert into uniform distribution
histogram, because we aim to match Gaussian
distribution. From Eq. (1) and Fig. 1, we can derive Eq.
(2) and (3).
h y ' x2

That is, we understand that f(x) becomes Gaussian
distribution Gauss(z) when we take the transformfunction as (1) and (5). Thus, HMGD processing is thefunction which defined by cumulative histogramtransformation the original histogram into Gaussian
histogram6,7.

0

f x

z
2

If we perform Eq. (7),(1)

L f x dx.

z

d
L
exp
dz 2 2 0

L
x

1
.
L

h y

(3)

z0

1
2

2

2

z
2

exp

.

2

Gauss(z)z

L Gauss z dz

(4)

z

1
2

2

In the previous section, we have described about the
principle for HMGD processing. In this section, we
propose the concrete process of Histogram Matching
using smoothed brightness histogram

2

z

exp

2

0

dz .

2

(i) Input original image and calculate moving averageof brightness histogram of the one.
(ii) If not need to calculate moving average of it further(if it satisfies number of times recursive), proceedto (iii). Otherwise, calculate moving average of thebrightness histogram recursively.

Here, Fig.1 shows the relationship between y= (x) andy= (z). So we can be obtained following Eq. (6).
x

L f x dx
0

z

L
2

2

exp
0

z
2

2
2

dz.

f(x)

3. Histogram Matching Using Smoothed
Brightness Histogram

(5)

0

x

Fig. 1. Conceptual image of HMGD6, 7.

z

y

x0+dx

1
L

We understand the histogram of original image f(x)becomes uniform distribution h(y) by Eq. (3). Thismeans that brightness level of original image f(x) is
expanded to h(y).
Then, let Gauss(z) and (z) be the function that isdefined by Eq. (4) and (5), respectively.
Gauss zz z0+dz

x0

z

(6)

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan562

Color Image Arrangement Based

1400

1400

1200

1200

1200

1000

1000

1000

1000

800
600

800
600

Frequency

1400

1200

Frequency

1400

Frequency

Frequeny(a) Image: Original6, 7 (far left), Histogram Matching using Moving Average (5 times: second from left, 10 times: second from right, 100 times: far right)

800
600

800
600

400

400

400

400

200

200

200

200

0

0

0

1

25

49

73

97 121 145 169 193 217 241

1

25

49

73

Bright enss

97 121 145 169 193 217 241

0
1

25

49

73

Bright ness

97 121 145 169 193 217 241

1

25

49

73

Bright ness

97 121 145 169 193 217 241
Bright ness

70000

70000

60000

60000

60000

50000

50000

50000

50000

40000
30000

40000
30000

Frequency

70000

60000

Frequency

70000

Frequency

Frequency(b) Histogram: Original (far left), Histogram Matching using Moving Average (5 times: second from left, 10 times: second from right, 100 times: far right)

40000
30000

40000
30000

20000

20000

20000

20000

10000

10000

10000

10000

0

0

0

1

25

49

73

97 121 145 169 193 217 241
Brightness

1

25

49

73

97 121 145 169 193 217 24101Brightness

25

49

73

97 121 145 169 193 217 241
Brightness

1

25

49

73

97 121 145 169 193 217 241
Brightness(c) Cumulative Histogram: Original (far left), Histogram Matching using Moving Average (5 times: second from left, 10 times: second from right, 100 times: far right)

Fig. 2. Example of results and the corresponding histogram.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

563

Yusuke Kawakami , Tetsuo Hattori, Yoshiro Imai, Koji Kagawa, Yo Horikawa, R. P. C. Janaka Rajapakse

2. B. Jahne, Digital Image Processing --Concepts,
Algorithms, and Scientific Applications-- 4th edition(Springer, 1995).
3. E. S. Umbaugh, Computer Vision and Image Processing:
A Practical Approach Using CVIP tools (Prentice Hall
PTR, 1998).
4. W. Burger and J. M. Burge, Principles of Digital Image
Processing: Fundamental Techniques (Springer 2009).
5. T. Izumi, T. Hattori, S. Sugimono, and T. Takashima,
Color Image Arrangement Using Elastic Transform on
Principal Component Axis (in Japanese), Journal ofJapan Society of Kansei Engineering 8(3) (2009) 667674.
6. Y. Kawakami, T. Hattori, D. Kutsuna, H. Matsushita,
Y. Imai, H. Kawano, R.P.C. Janaka Rajapakse,
Automated Color Image Arrangement Method Based on
Histogram Matching - Investigation of Kansei impression
between HE and HMGD -, International Journal of
Affective Engineering 14(2) (2015) ISSN 2187-5413, 8593.
7. Y. Kawakami, T. Hattori, Y. Imai, K. Ando, Y. Horikawa,
R. P. C. Janaka Rajapakse, Reference Histogram
Matching Based on Moving Averaged Brightness
Histogram, IEEJ Transactions on Electronics
Information and Systems 138(9) (2018) ISSN 1348-8155,
1131-1132.

(iii) Generate the reference histogram hr from the resultof (ii).
(iv) Perform Histogram Matching processing of usingreference histogram as hr.
(v) Output processed image.
4. Experimentation
In this experimentation, we perform moving average arange of ± 25 with respect to each brightness value oforiginal image histogram.
Fig. 2 shows the example of results and thecorresponding histogram. In this case, we understandthat Histogram Matching processing image using
moving average of brightness histogram is enhancing
contrast than original image.
From far right of Fig. 2(a) ~ (c), we notice that theseresults like Histogram Equalization (HE) or Histogram
Matching based on Gaussian Distribution (HMGD).
That is, we consider that this processing result will get
close to HE or HMGD if we increase in the number oftimes moving-average recursive. And we consider thatthe processing result does not always give us good
Kansei impression in such case.
5. Conclusion
In this paper we have proposed the color image
arrangement based on histogram matching using
moving average of original image brightness histogram.
As for the concrete processing method, we have usedmoving average recursively from the original image
brightness histogram to generate reference histogram.
Then we have taken a histogram-matching processed
image.
From the experimentation, histogram matchingprocessed image is enhancing contrast from original
image and this processing result will get close to HE or
HMGD if we increase in the number of times movingaverage recursive. That is, we consider that proposed
method is simple and convenient as Kansei impression
enhancement instead of HMGD.
References
1. R. C. Gonzalez and R. E. Woods, Digital Image
Processing (Addison-Wesley Publishing Company,
1993).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan564

Color Image Arrangement Based on Histogram Matching
Using Smoothed Brightness Histogram (II) --- Piecewise Smoothing --Yusuke Kawakami*, Tetsuo Hattori, Yoshiro Imai, Koji Kagawa, Yo Horikawa

Graduate School of Kagawa University, 2217-20 Hayashi
Takamatsu City, Kagawa 761-0396, Japan
R. P. C. Janaka Rajapakse

Tainan National University of the Arts, 66 Daci
Guantian District, Tainan 72045, Taiwan
E-mail: riverjp2002@gmail.com, {hattori imai, kagawa, horikawa}@eng.kagawa-u.ac.jp, janaka@mail.tnnua.edu.tw

Abstract
Although we have already reported that automated color image arrangement by Histogram Matching based on
Gaussian Distribution (HMGD) gives good feeling impression if the brightness histogram of the original image hasa single peak. However, if there are multiple peaks in the histogram, the HMGD processing does not always bringgood results. In this paper, we propose a Histogram Matching method using piecewise smoothing over thebrightness histogram of original image as the reference one, after Histogram Analysis by curvature computation.
Also in this paper, we present the experimental results.
Keywords: Image processing, Brightness histogram, Histogram matching, HMGD

1. Introduction
D processing does not always give good feeling
impression. Through the experimental results, we haveconsidered if the reference histogram make change tosatisfy brings good results whether or not the number oforiginal image brightness histogram peaks7-8.
In this paper, we propose a Histogram Matching method
using piecewise smoothing over the brightness
histogram of original image as the reference one, after
Histogram Analysis by curvature computation. Also inthis paper we illustrate the experimental results.

These days, automated image processing forenhancement of color images has been more familiar tous, for example, Digital Signage, Smart Phone, etc1-3.
In the previous paper, we have already reported that theHistogram Matching based on Gaussian Distribution(HMGD) processing is one of the automated image
arrangement methods4-5 on the brightness axis and gives
good feeling impression if the original images havesingle peak6.
However, we have understood that if the original
images have multiple-peak brightness histogram, HMG-*

1882-36, Yashima-nishi, Takamatsu, Kagawa 761-0113 Japan© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

565

Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai, Koji Kagawa, Yo Horikawa, R. P. C. Janaka Rajapakse

2. Principle

From Eq. (6), we understand that the curvature R(x)varies the sign according to the value of x. That is, if
x<a → R>0 (downward convex shape), and if x>a →R<0 (upward convex shape).
As for valley detection, we can approximate the shapeof the neighbor points at the valley point as 1.0 - g(x).
So, around the valley point, the curvature R(x) is shownas follows:

In another paper, we have described principle ofHistogram Matching based on Gaussian Distribution(See Ref. 8). In this section, we describe the principle ofpeaks and valleys detection method, and varianceestimation by regression analysis.
2.1. Peaks and Valleys Detection of Histogram9
Fig. 1 shows the conceptual image of the original image
histogram which is variance 2 and average a. And Fig.
2 shows its cumulative histogram.
Let y be a function with respect to x, the definition
curvature R(x) is given by Eq. (1)

d2y
1
dx 2

Rx

dy
dx

2

3
2

(a) x<0 → R<0 (upward convex shape)(b) x>a → R>0 (downward convex shape)
In any case, it holds that if R>0, the shape of theoriginal histogram curve is downward convex. In thisway, from the value of curvature over the cumulative
histogram, we can detect peaks and valleys in theoriginal histogram curve.

(1)

.

y

Let g(x) and K be a Gaussian distribution function
approximated at a peak and its coefficient that are
defined by the following equations.
K

g x

2

2
K

L
2

2

x a2
2 2

exp

g(a)

(2)

u a2
du 1
2 2

exp

0

0

Next, let y=f(x) be a function representing the
cumulative histogram which is represented Eq. (3).
That is, dy/dx and d2y/dx2 be described as Eq. (4) and (5),respectively. From Eq. (4) and (5), we obtain theapproximation of curvature R(x) as Eq. (6).
xf x

g u du

K

g x

d2y
dx 2

2

2

dg x
dx
a x

Rx

2

2

0

dy
dx

x

K

2

1 g x

exp
0

x a
2 2

exp

a x
2

g x
3
2 2

u a
2 2
2

.

g x

a x
2

g x.

2

a

a

2

a

2

L

x

Fig. 1. Conceptual image of the original image histogram9.

ydu. (3)

R a

2

a

2

R(a)

(4)

Ra

0

(5)

2

a

2

a

L

x

Fig. 2. Conceptual image of the original image cumulative
histogram9.

(6)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

566

Color Image Arrangement Based

1800

1800

1600

1600

1600

1400

1400

1400

1400

1200

1200

1200

1200

1000
800

1000
800

Frequency

1800

1600

Frequency

1800

Frequency

Frequeny(a) Image: Original6 (far left), Histogram Matching using Moving Average (5 times: second from left, 10 times: second from right, 100 times: far right)

1000
800

1000
800

600

600

600

600

400

400

400

400

200

200

200

200

0

0

0

1

25

49

73

97 121 145 169 193 217 241

1

25

49

73

Bright enss

97 121 145 169 193 217 241

0
1

25

49

73

Bright ness

97 121 145 169 193 217 241

1

25

49

73

Bright ness

97 121 145 169 193 217 241
Bright ness

70000

70000

60000

60000

60000

50000

50000

50000

50000

40000
30000

40000
30000

Frequency

70000

60000

Frequency

70000

Frequency

Frequency(b) Histogram: Original (far left), Histogram Matching using Moving Average (5 times: second from left, 10 times: second from right, 100 times: far right)

40000
30000

40000
30000

20000

20000

20000

20000

10000

10000

10000

10000

0

0

0

1

25

49

73

97 121 145 169 193 217 241
Brightness

1

25

49

73

97 121 145 169 193 217 24101Brightness

25

49

73

97 121 145 169 193 217 241
Brightness

1

25

49

73

97 121 145 169 193 217 241
Brightness(c) Cumulative Histogram: Original (far left), Histogram Matching using Moving Average (5 times: second from left, 10 times: second from right, 100 times: far right)

Fig. 3. Example of results and the corresponding histogram.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

567

Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai, Koji Kagawa, Yo Horikawa, R. P. C. Janaka Rajapakse

4. Experimentation
In this experimentation, we perform moving average arange of ± 10 with respect to each valley-to-valley
brightness value of original image histogram.
Fig. 3 shows the example of results and thecorresponding histogram. In this case, we understandthat histogram matching processing image using
piecewise moving average of brightness histogram isenhancing contrast than original image naturally.
From far right of Fig. 3(a) ~ (c), we notice that theseresults turn unnatural color from original image andeach peak shifted to higher brightness.
That is, we consider that this processing will getenhance brightness if we increase in the number oftimes piecewise-moving-average recursive. And if weincrease number of it too much, we consider that theprocessing result does not always give us good Kansei
impression in such case.

2.2. Variance Estimation by Regression Analysis9
From Eq. (6), the R(x) can be shown as the following Eq.
(7).

a x

Rx

1

g x

2

2

a x g x.

(7)

And, let C=1/ 2 and H(x) = (a-x)g(x) respectively, we
have Eq. (8).

Rx

CH x

(8)

Now, we can calculate a constant C by using leastsquare regression analysis as in the following Eq. (9).

Ri

CH i

i

~ N 0,

Thus, we can evaluate
n

C

i

2

,

2

i 1,

n

(14)

as the following Eq. (10).

H i Ri

i 1
n

,
Hi

2

2

1
C

5. Conclusion
In this paper we have proposed the color image
arrangement based on histogram matching using
piecewise moving average of original image brightness
histogram.
As for the concrete processing method, we havedetected each peak and valley, and estimated eachvariance of original image histogram by regression
analysis. Then, we have used piecewise moving average
recursively from the original image brightness
histogram to generate reference histogram. Finally wehave taken a histogram-matching processed image.
From the experimentation, we consider that histogram
matching processed image by proposed method isenhancing brightness from original image naturally,
unless we increase number of piecewise-movingaverage recursive times too much. That is, we consider
that proposed method is simple and convenient as
Kansei impression enhancement instead of HMGD.

(15)

i 1

3. Histogram Matching Using Smoothed
Brightness Histogram
In this section, we propose the concrete process ofHistogram Matching using piecewise smoothedbrightness histogram.
(i) Input original image and peaks and valleys
detection, and then perform variance estimation ofeach peak.
(ii) Calculate piecewise moving average of brightness
histogram.
(iii) If not need to calculate piecewise moving averageof it further (if it satisfies number of times
recursive), proceed to (iv). Otherwise, calculatethem recursively.
(iv) Generate the reference histogram hr from the resultof (iii).
(v) Perform histogram matching processing of usingreference histogram as hr.
(vi) Output processed image.

References
1. R. C. Gonzalez and R. E. Woods, Digital Image
Processing (Addison-Wesley Publishing Company,
1993).
2. B. Jahne, Digital Image Processing --Concepts,
Algorithms, and Scientific Applications-- 4th edition(Springer, 1995).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

568

Color Image Arrangement Based

3. E. S. Umbaugh, Computer Vision and Image Processing:
A Practical Approach Using CVIP tools (Prentice Hall
PTR, 1998).
4. W. Burger and J. M. Burge, Principles of Digital Image
Processing: Fundamental Techniques (Springer 2009).
5. T. Izumi, T. Hattori, S. Sugimono, and T. Takashima,
Color Image Arrangement Using Elastic Transform on
Principal Component Axis (in Japanese), Journal ofJapan Society of Kansei Engineering 8(3) (2009) 667674.
6. Y. Kawakami, T. Hattori, D. Kutsuna, H. Matsushita,
Y. Imai, H. Kawano, R.P.C. Janaka Rajapakse,
Automated Color Image Arrangement Method Based on
Histogram Matching - Investigation of Kansei impression
between HE and HMGD -, International Journal of
Affective Engineering 14(2) (2015) ISSN 2187-5413, 8593.
7. Y. Kawakami, T. Hattori, Y. Imai, K. Ando, Y. Horikawa,
R. P. C. Janaka Rajapakse, Reference Histogram
Matching Based on Moving Averaged Brightness
Histogram, IEEJ Transactions on Electronics
Information and Systems 138(9) (2018) ISSN 1348-8155,
1131-1132.
8. Y. Kawakami, T. Hattori, Y. Imai, K. Kagawa,Y. Horikawa, R. P. C. Janaka Rajapakse, Color Image
Arrangement Based on Histogram Matching Using
Smoothed Brightness Histogram (I) --- Overall
Smoothing ---, in Proceedings of International
Conference on Artificial Life and Robotics (ICAROB
2019) (Oita, Japan, 2019), To Be Published.
9. Y. Kawakami, T. Hattori, Y. Imai, K. Ando, Y. Horikawa,
R. P. C. Janaka Rajapakse, Histogram Analysis Method
Based on Gaussian Distribution and Curvature
Computation (I) ---- Peaks and Valleys Detection ----, inProceedings of International Conference on Artificial
Life and Robotics (ICAROB 2018) (Oita, Japan, 2018)
ISBN 978-4-9908350-3-3, 367-370.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

569

Parameter Estimation Method for Compartment Model---- PET Inspection ---Tetsuo Hattori*, Yoshiro Imai*, Koji Kagawa*, Yo Horikawa*, Yusuke Kawakami**

Kagawa University
2217-20 Hayashi Takamatsu City, Kagawa 761-0396, Japan
Hiromichi Kawano†

NTT Advanced Technology Company Ltd.
1-19-8 Nakamachi, Musashino City, Tokyo 180-0006, Japan*

E-mail: {hattori, imai, kagawa, horikawa}@eng.kagawa-u.ac.jp**E-mail: riverjp2002@gmail.com, †E-mail: hiromichi.kawano@ntt-at.co.jp

Abstract
This paper proposes a new general method for parameters estimation in compartment model using DOC(Differentiation of Convolution). In this paper, we apply the DOC method to the parameter estimation in PET(Positron Emission Tomography) inspection especially. Pet inspection is done for getting physiological andbiochemical information of organs. The PET camera acquires the image of radioactive distribution inside thehuman body. In this paper, we make a mathematical analysis of the PET compartment model and describe the DOC
method.
Keywords: Compartment model, PET Inspection, Parameters estimation, Difference of Convolution

1.

This paper newly proposes a method for such parameter
estimation using DOC (Differentiation of Convolution)
operation.

Introduction

Compartment model is used in a wide variety of fields
such as pharmacokinetics, chemical kinetics, modelingof neurons, epidemiology, and simulation of natural
phenomena, etc 1-6.
And in many cases, the parameter estimation of themodel is performed from actual measurement data,
based on the adopted compartment model.
Moreover, parameters to be estimated are often the
exponent of the exponential in the convolution integral
such as 3-component model ones of FDG (fluorodeoxyglucose) PET (Positron Emission Tomography)
inspection in the field of nuclear medicine.

2.

Compartment Model

2.1. Multi-compartment
The compartment model, especially, multi-compartment
model is a type of mathematical model used fordescribing the way that materials or energies aretransmitted among the compartments of a system (Fig.1).
Such compartment models are mostly described as in
the form of simultaneous differential equation systemwith respect to time t that contains some parameters of
the transmission flow.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

570

Tetsuo Hattori, Yoshiro Imai, Koji Kagawa, Yo Horikawa, Yusuke Kawakami, Hiromichi Kawano

And, in most cases, the parameters estimation of thetransmission flow is quite an important problem toknow the situation of the compartment modelled objects
such as PET inspection in medicine.

C1

C2

The solution including the flow parameters K1, k2, k3, k4,
is described in the following Eq.(1).

Ci ( t ) =1
( k + k −  ) e − t + (  − k3 − k4 ) e −  t  −  3 4
 K1C p ( t )C3 ,  = k + k + k( 2 3 4)

( k2 + k3 + k4 ) − 4k2k4  1 2 
2

where  means the convolution.

C5

C4

(1)

C6
3.

DOC Method

Fig.1. Multi-compartment model.

3.1. Fundamental Theorem and its Corollary
Our DOC (Differentiation of Convolution) method isbased on the following Theorem 1 and its Corollary 1.

2.2. Compartment Model in FDG PET Inspection
and Parameter Estimation

Theorem1:

The 3-compatment model used in FDG PET inspection
is shown in Fig.2. The simultaneous differential
equation system at time t [min] used in the FDG PET
model is also shown in Fig.2.(b). The flow parameters
K1, k2, k3, k4, ( [ml  g −1  min −1 ] or [min −1 ] ) show thetransport and binding rates of the tracer between two
compartments.

Let f ( x, t ) andand its partial derivative function that is also continuous
over the intersection a  x  b , respectively.
Then
b 
d b
=
f
(
x
,
t
)
dx
a t f ( x, t )dx
dt a

Ci (t )

K1

k3

k2

If the a and b are the functions a(t) and b(t) with respect
to t, respectively, thenCm (t )Ce (t )C p (t )f ( x, t ) be a continuous functiontb( t ) d b( t )
f
(
x
,
t
)
dx
=
a(t ) t f ( x, t )dx
dt a(t )db ( t )da ( t )+ f (b (t ) , t )− f (a (t ) , t )
dt
dt

k4
(a)

dCe ( t )= K1C p ( t ) − ( k2 + k3 ) Ce ( t ) + k4Cm ( t )dtdCm ( t )= k3Ce ( t ) − k4Cm ( t )dtFrom Theorem 1, we obtain the following Corollary 1.

Ci ( t ) = Ce ( t ) + Cm ( t )Corollary 1:(We omit the description of the proof of Theorem1.)

Let J (t ) be the convolution between eshow in the following.

(b)

− kt

and C (t ) ast

J (t ) = e − kt  C (t ) =  e − k ( t − )C ( )dFig.2. Compartment model in FDG PET inspection. (a) 3Compartment model of the FDG PET with four tracer’s flow
parameters. (b) Simultaneous differential equations on the
FDG PET compartment model.

0

Then

dJ (t )
 J (t ) = ( −k ) J (t ) + C (t )dt

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

571

Parameter Estimation Method for

Proof.
Let a (t ) = 0, b(t ) = t , f ( x, t ) = e
Then from Theorem1, we have−k (t−x )C ( x) .

r" (t ) 
db(t )da− f (a, t )f ( x, t )dx + f (b(t ), t )
t
dt
dt
t 
dt
d0
=
f ( x, t )dx + f (t , t ) − f (0, t )
0 t
dt
dt
b( t )=a= ( −k )  et− k (t − x )0

C ( x ) dx + e

d 2 r (t )dt 21
( − ) J'1 (t ) + ( −  ) J'2 (t )  + K1C' p (t ) − 

( J ( t ) +  J ( t ) )
=
2

2

1

C ( x ) − f ( 0, t )  0− k ( t −t )2 −+ ( k3 + k4 ) − (  +  ) K1C p ( t ) + K1C p ( t )= ( −k )  e − k (t − x )C ( x ) dx + C ( t )
t

0

(10)

= ( −k ) J ( t ) + C ( t )Since

Eq.(11) by substituting those representation into Eq.(10).

J (t ) = ( −k ) J (t ) + C (t )(2)
Q.E.D.

r ( t ) = ( k3 + k4 ) K1C p ( t ) + K1C p ( t )− ( k 2 + k3 + k 4 ) r  ( t ) − k 2 k 4 r ( t )3.2. Parameter Estimation Using DOC Method
Let r ( t ) be the measured value by PET camera at time

(11)
Then, integrating both sides twice with respect to time t,
we obtain the following Eq.(12).

t. Then r ( t ) is approximately represented as shown in
the following equation, where the parameters are thesame as shown in Eq.(1).

1J (t ) + J 2 (t )
 − 1r (t ) = AC pII (t ) + BC pI (t ) + Cr I (t ) + Dr II (t )
(12)

(3)
where

r I (t ) = t r ( )d0tr II (t ) =  r I ( )d0C I (t ) = t C ( )d
0 p
 p

t
C pII (t ) =  C pI ( )d0 A = K 1 (k 3 + k 4 )B = K
1C = −(k 2 + k 3 + k 4 ) D = −k 2 k 4

where

J 1 (t ) = ( k3 + k 4 −  ) e

J 1 (t ) and J 2 (t ) are represented by r (t ) ,r (t ) and C p (t ) from Eq.(3) and Eq.(9), we can obtain

Accordingly, we haver (t ) = Ci (t ) =(9)

Similarly,

J (t )=1
( − ) J 1 (t ) + ( −  ) J 2 (t )  + K1C p (t ) − 

r(t ) =− t

J 2 ( t ) = (  − k3 − k 4 ) e−t K1C p (t )(4)

 K1C p (t )(5)Applying the aforementioned Corollary 1 to the Eq.(3)(5), we have the following equations.

dr (t )1=
J' (t ) + J'2 (t )
 − 1dt
J'1 (t ) = ( − ) J 1 ( t ) + ( k3 + k4 −  ) K1C p ( t )(7)J'2 (t ) = ( −  ) J 2 (t ) + (  − k3 − k4 ) K1C p (t )(8)r' (t ) 

(6)

Then we have
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

572

(13)

(14)

(15)

Tetsuo Hattori, Yoshiro Imai, Koji Kagawa, Yo Horikawa, Yusuke Kawakami, Hiromichi Kawano

Table 1. Estimated value from r(t) and Cp(t) for 60 minutes’
measurement, and its accuracy.

Since the time series data r(t) and Cp(t) are given by
PET Camera and blood sampling respectively, we canobtain the values of parameters A, B, C, D in Eq.(15) byleast squares method based on the Eq.(12). So, we canestimate the parameters K1, k2, k3, k4 from the computed
A, B, C, D.
4.

r(t) and Cp(t): 60 minutes' time series data
true
value

parameter

Experimentation

We have experimented for parameters estimation using
the simulated time series data Cp(t) that means the tracer
plasma radioactivity concentration and r(t) that is thecomputed PET Camera’s time series data according toEq.(1) where the values of parameters K1, k2, k3, k4 are
set beforehand.
Cp(t)estimated
value

relative error[%]

K10.20000.20040.2024k2

0.13000.13040.3737k3

0.06000.0601

0.0952k4

0.00700.00700.0956

Relative error [%]

1600

0.4

1200

0.3

800

0.2

400

0.1

K1
k2
k3

t

0
0

900

1800

2700

k4

0
0

3600

Fig.3. Model function of Cp(t) (plasma)[Bq/ml] at time t [sec]for numerical experimentation.

600 1200 1800 2400 3000 3600
Measurement time [sec] used for estimation.

Fig.5. Accuracy of the parameters estimation for K1, k2, k3, k4
with respect to the measurement time in the case where
Gaussian noise are not added to r(t).

r(t)

References
1. Godfrey, K., Compartmental Models and Their
Application, (Academic Press, 1983).
2. R. P. Maguire and K. L. Leenders, ed., PET
Pharmacokinetic Course Manual, (International Societyof Cerebral Blood Flow and Metabolism, Summer
School 2007, May 2007).
3. https://en.wikipedia.org/wiki/Multi-compartment_model.
4. Hiroshi WATABE, Yoko IKOMA, Yuichi KIMURA,
Mika NAGANAWA, and Miho SHIDAHARA, PET
kinetic analysis—compartmental model, Annals of
Nuclear Medicine, Vol. 20, No. 9, 583–588, 2006.
5. Kudomi N, Choi E, Yamamoto S, Watabe H, Kim K,
Shidahara M, et al., Development of a GSO detector
assembly for a continuous blood sampling system, IEEE
Trans. Nuclear Science, Vol. 50, No.1, pp.70–73, 2003.
6. Naganawa M, Kimura Y, Ishii K, Oda K, Ishiwata K,
Matani A, Extraction of a plasma time-activity curvefrom dynamic brain PET images based on independent
component analysis, IEEE
Trans. Biomedical
Engineering, Vol. 52, No.2, pp.201–210, 2005.

t
Fig.4. Computed r(t) [Bq/ml] at time t [sec] from Cp(t) without
Gaussian noise addition.

The results are shown in Table 1. As a result, we havefound that the relative error rate of parameters
estimation using the proposed DOC method is less than
0.4 [%], if we use the time series data of r(t) and Cp(t)for more than 600 seconds (10 minutes).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

573

Application and Trial Evaluation of Document Writing Support System to Avoid Emotional
misUnderstanding
Shunsuke Doi, Yoshiro Imai*, Koji Kagawa, Tomohiko Takagi, Yusuke Kawakami, Tetsuo Hattori

Graduate School of Engineering, Kagawa University,
2217-20 Hayashi-cho, Takamatsu, Kagawa, 631-0396, Japan*

Corresponding author. E-mail: imai@eng.kagawa-u.ac.jp
Tel: +81-87-864-2244, Fax: +81-87-864-2244

Abstract
This paper describes a newly developed system of document designing, handling and self-checking in order for
writers to avoid their readers' unnecessary emotional misunderstanding and possible repulsion which may occur as aresult. It is realized as server-client computing system, namely its server is written in Perl and PHP just like LAMP
(Linux-Apache-MySQL-PHP/Perl) and its client is implemented in JavaScript executing on the major Browsers. The
server can obtain user document through the client, scan and separate the regarding document into word-level
expressions, check each word while matching it with our sentiment dictionary, calculate the relevant sentimental
values for the document and then generate the corresponding radar chart based on emotional axis such as delight,
anger, sorrow, pleasure etc. on the relevant user's Web. This is one of examples of his/her document visualization.
Document writers utilize such a system before transferring and/or archiving, they can check sentimental values fortheir documents and recognize how their ones have a lot of emotional feelings which would probably include nonsuitably emotional expressions. With this facility, the user can check his/her document and recognize whether the
document possibly generates readers' unnecessary emotional misunderstanding and moreover repulsion which may
occur as a result.
Keywords: Sentimental value, Sentiment analysis, Visualization, Japanese Language processing, Documenttransferring, Document archiving..

analysis seem to be excellent approaches to reveal hidden
relation behind phenomena people have never found
before and bring new recognition about such a relation to
our daily lives for possible decision-making.
Sentiment analysis1 can play another important role ofinformation processing for sentences, expressions anddocuments not only from SNS/Internet but also fromnormal, daily verbal communication. Due to preprocess
related with sentimental analysis, we can find whether
our writing document includes ''unnecessary or nonsuitable'' emotional expressions or not. So we would be
relieved to transfer our documents as well as archive ones
if we had employed any preprocess to detect suchemotional expressions and then had removed these
expressions.

1. Introduction
People used to be writing several kinds of documents
such as memoranda, message, e-mail etc. for the third
persons to read possibly with emotional feeling. Afterthey prepared such documents, sometimes some of their
documents might unintentionally hurt other's heart due totheir careless emotional expressions. If some kind ofchecking services were available, people could checktheir documents, recognize whether their documents
included careless emotional expressions, and avoid towrite documents which would unintentionally hurtother's heart.
Data mining becomes more and more popular and thenbig data analysis has been consistent with the trend of
information processing. Data mining and big data

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan574

Shunsuke Doi, Yoshiro Imai, Koji Kagawa, Tomohiko Takagi, Yusuke Kawakami, Tetsuo Hattori

Before describing our sentimental analysis forpreprocessing, this subsection explains how to definesentimental dictionary and how to calculate sentimental
value for each focused sentence. With Japanese language
processing, our system obtains some documents in thetarget domain, decomposes such a document into
sentences and decompose them into a series of words by
MeCab. Such words are classified into cluster andcomputed with sentimental values according tosentimental dictionary2.

This paper describes our Document Transferring andArchiving Service with Sentiment Analysis-based
Preprocessing Facility. Such service is designed andimplemented as server-client computing model and forusers to manipulate it through Web-based application.
The paper introduces a whole system and then explains apreprocess service to generate Sentimental Dictionary
and to perform Sentimental Analysis to detect emotional
expressions graphically. It illustrates some trial ofvisualization of documents through sentimental analysis
in the third section. It demonstrates documenttransferring and archiving by means of our services in the
fourth section. And finally it concludes our summaries in
the last section.

3. Visualization through Sentimental Analysis
3.1 Process flow for sentimental analysis
Process for Sentimental Analysis starts decomposing theobtained document into sentence, decomposing eachsentence into a series of words by MeCab, and thenchecks whether each word is registered in the sentimental
dictionary or not. If each word of the sentence isregistered, then sentence is calculated with sentimental
values according to the conditions including "Negation"
and/or "Conjunction". Because Negation and/or
Conjunction is included in the sentence, calculation ofwhole sentimental value for the relevant sentence must
be modified in the ways shown in Fig.2 and Fig.3.

2. Detail of System Configuration and Process
Our system has been design and implemented as a typicalserver-client computing model, where the server isrealized in the LAMP type and the client executes on the
major browsers written by JavaScript. Fig.1 shows amain flow of the system, the three major menus, namely
input menu, show graph one and e-mail transfer one, areprovided on the client and then the related three
procedures, namely sentimental analyzing, simple mailtransferring and PDF generating are invoked on theserver and performed as server tasks written in PHP and
Perl scripts. The system has two major databasemanaging modules which execute on the server together
with SQL database.

Fig.2. Example for Determination of Negation.

Fig.3. Example for Determination of Conjunction.

3.2 Computing and assigning of sentimental values
After picking up the relevant sentimental value for eachword from sentimental dictionary together with detectionof "Negation" and "Conjunction", calculation ofsentimental values for a whole sentence can be performedin the following way. Real example of calculation ofsentimental value for simple document is illustrated in
Fig.4. A lots of emotional feelings have been classified
and categorized into 10 clusters, for example, `delight',`anger', `sorrow' and `pleasure' are the four major axis,

Fig.1. Main flow of the system.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

575

Application and Trial Evaluationof calculating sentimental analysis. It is very useful andattractive for each user to understand what kind of
emotional expressions in his/her writing documents varyin chronological order and to obtain a meaningful hint tomodify his/her documents into more suitable not only forhimself/herself but also for those who shall probably readhis/her documents near future.

and then we have defined other four clusters, some ofthem seem to be combined and modified with the four
major axis. The bottom two lines of 5 clusters shown in
Fig.4, which indicate the sentimental values for 10
clusters, are the results of calculated sentimental values
for the relevant document described in the expression at
the top line of Fig.4.

Fig.4. Example of Calculation and Assignment of Sentimental
Values for Simple Document.

3.3 Visualization based on Sentimental Analysis
Users understand what kinds of emotional feelings areincluded in their writing document, so it is moreconvenient for them to understand emotional feelingwhether their writing document includes emotional
expressions or not through graphical expression rather
than through Fig. 4. Carefully using with those
information, writers, namely users of our system, would
be able to avoid such non-suitable emotional expressions
thank to visualization of emotional feeling in the relevant
document.
As you know, it is more useful for visualization to show
the results of sentimental analysis graphically than toshow them in the ways of calculated tabulation or
numerical expression just like Fig.4. So our system canobtain document by user through input mode. And thenit can calculate sentimental value and visualize the resultof sentimental analysis graphically just like Fig.5. Users
can modify their writing documents with emotional
feeling into more acceptable contents which includes
suitably emotional expressions and reduces non-suitableones according to visualized results of calculating
sentimental analysis for their previous written documents.
Each user can check how his/her writing document varies
in chronological order from the viewpoint of sequential
line graph for the results of calculating sentimental
analysis. Fig.6 shows sequential line graph for the results

Fig.5. Example for Visualization of Sentimental Values.

Fig.6. Another Visualization of Sentimental Values.

4. Application of System and its Evaluation
This section describes typical application of the systemwith other useful services for document transferring andarchiving after sentimental analysis and visualization of
its calculated results. And it reports the last but not least
description about trial evaluation of our system.
At first, document transferring service has been
implemented in the system through invocation of serverside e-mail handling facilities called "Postfix" as MTA© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

576

Shunsuke Doi, Yoshiro Imai, Koji Kagawa, Tomohiko Takagi, Yusuke Kawakami, Tetsuo Hattori(Mail Transfer Agent) and "Dovecot" as MUA (MailUser Agent). A user write his/her document onto the
Web-based client of the system, check what kinds ofemotional feelings are included in his/her writing
document, and then instruct to let the relevant document,
which has been already checked by Sentimental analysis,be transferred into someone’s e-mail address. Fig.7
shows e-mail transfer menu of the system. With oursystem, the document to be transferred shall be modified
without non-suitable emotional feelings for anyone.

(1) "In the case of e-mail writing, I used to be afraid my
expression would be probably harmful for the friends
who received my e-mail. Therefore this system will bevery useful for such my case to write e-mail very
comfortably and moreover it will be able avoid mycareless expression to hurt others’ feeling."(2) "The system would let us write such documents that
did not get offended against others and support us totransfer our documents into PDF styled articles
automatically through LaTeX typesetting tool."5.

Conclusion

Our system provides sentimental analysis to calculatesentimental values for each document in order tovisualize what kinds of emotional feelings are included
in such a document. And it also performs to transfer the
relevant documents without non-suitable emotional
expressions as well as to archive such documents as PDF
files, for example to form them through external LaTeX
facility. With our system, users, namely writers of
documents, can realize document management
effectively and efficiently.

Fig.7 Mail Transferring on Web application of the system.

Secondarily, document archiving service has been also
implemented in the system in order to generate a PDF
style, namely the system can support to transform the
documents into PDF. By means of usage of LaTeX
typesetting software, the relevant document can beformed into more easily readable together with dating,signature and other important information automatically.
Fig.8. shows PDF generating menu of our system.

References
1. T. Kumamoto, et al., "Proposal of Impression Mining from
News Articles", Lecture Notes in Computer Science
Volume 3681, pp 901–910, 2005.
2. M. Kyokane, et al., "A Study of Visualization for Hidden
Relation between Published Documents and Message fromTwitter," International Journal of E-Learning and
Educational Technologies in the Digital Media Vol.2, No.1,
pp.217– 224, 2016.
3. S. Doi, et al., "Calculation and Comparison of Sentimental
Values for Tweeting Message during Multiple Years," Proc.
of The Second International Conference on Electronics andSoftware Science(ICESS2016@ Takamatsu), pp. 165 – 169,
Nov.2016.
4. E. Cambria, et al., "Statistical approaches to concept-level
sentiment analysis," IEEE Intelligent Systems Volume 28,
No.3, pp.6 - 9, 2013.
5. P. Bo and L. Lee, "Opinion mining and sentiment analysis,"
Foundations and Trends in Information Retrieval,Volume.2, No.1–2, pp.1-135, 2008.

Fig.8. PDF generating by LaTeX.

As trial evaluation of our system, we have heard fromusers about the two types of comments for our system asfollows;© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan577

A Method of Haiku Generation Using Deep Learning for Advertising Generation
Jumpei Ono

Vocational School of Digital Arts Sendai
Sendai, Miyagi, 980-0014, Japan
Takashi Ogata

Iwate Prefectural University
Sugo, Iwate, 020-0693, Japan
E-mail: j.ono@sugawara.ac.jp, t-ogata@iwate-pu.ac.jp

Abstract
We tried various approaches to story generation. One of them was haiku generation using deep learning. The method
we tried had several tasks, one of which was to use only word time-series data as training data. Therefore, the resultof training can be a model similar to generation based on simple word-transition probability. Consequently, weorganized the current tasks, tried to study the features used for training, and generated by applying the learned results.
Furthermore, we consider how to use the method proposed in this paper in advertisement generation and in generating
different types of stories.
Keywords: Haiku, Haiku Generation, Deep Learning

including “kireji.” Semantic features include “kigo”(seasonal words), which directly or indirectly imply
scenes to some extent (There are also haiku which
deviates from the above characteristics). Fig.1 is a haiku
written by Matsuo Basho. The haiku reads as “A frog
jumped in an old pond. Sound of water echo.” (Becausea haiku can describe scenes in extremely short phrases, it
must be stated here that English translations can vary
greatly depending on the translator).

1. Introduction
In this study, we attempted to generate various types ofstories using both artificial intelligent techniques andnarratological models. One of the approaches proposed
story generation with a framework of mutual conversion
between a haiku and a narrative. We approached thisfrom two perspectives—a generation method based on
haiku practices and a generation method based on a deeplearning method. Here, we describe the generation of
haikus using deep learning. Based on an attempt togenerate haikus by Ito, Igarashi & Ogata1,2, we increased
training data and confirmed the change that occurred in
the generated result.

古 池 や
Furuike ya/

蛙 飛 び 込 む
kawazu tobikomu

/

水 の 音
mizu no oto

Fig.1 An example of a haiku(Matsho Basho, “Oku no hosomichi”)
Haiku generation using training is a method that uses
deep learning by Yoneda et al.3 and seqGAN by Hirota,
et al.4 among others. In addition, Rzepka & Araki
proposed a system to present not only artificial haiku but
also sounds and images related to the haiku5.

2. BackgroundA haiku is a poem composed of relatively short phrases.
As a formal feature, an ordinary haiku consists of threephrase, whereby, the first, second, and third phrases
contain five, seven, and five mora, often rhyming to someextent. Furthermore, the number of mora are filled by© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

578

Jumpei Ono, Takashi Ogata

Fig.3 shows the generation results of 100 iterations of
training, and Fig.4 shows the generation results of 900
iterations of training.

We used statistical methods based on deep training;
however, from the perspective of the entire framework ofhaiku generation, differentiation is also achieved byincorporating the approach from the narrative theory. The
aspect of that is discussed in Ito & Ogata6, Ito, Ono &Ogata7. Our proposed haiku generation mechanism
proposed is single and incomplete. In the haiku
generation mechanism, our developed system is linked tothe integrated narrative generation system8, and it has thepossibility of being linked to the generation of haikus,
stories, music, and video. Ultimately, we treat haiku as
one of the inputs to and outputs from story generation,and the haiku generation mechanism is integrated as amechanism for generating a more versatile story.
3.

(a). [洗い髪 のせ哭く後ろ 向き背中](b). [込むアパート ゆるめし眠れ アンデルセンやわらかい](c). [十一月投ず梅]Fig.3 A haiku based on words training (100 iterations)(d). [生涯に あざやか秋切る は哀しり](e). [夏野笑ぐ の峯日のコップ](f). [はじまりて 会うとや見える ]Fig.4 A haiku based on words training (900 iterations)
Subsequently, we attempted to generate a haiku
without controlling the number of mora, but as in (a) of
Fig.3, a haiku suitable for the number of mora withmeaning was generated. Furthermore, (b) in Fig.3, where
the number of mora is large, these can be used as a haiku
if you adjust them using any method. Conversely, theexamples in (c) in Fig.3, (e) in Fig.4, and (f) in Fig.4 havemeaning and a form that can be considered to havegenerated parts of haiku. Consequently, in the narrative
approach, it is considered to be a generation result usefulfor generating parts of haiku. Generation results of
Igarashi1,2 concluded that many data existed that were thesame as input data due to overtraining, and the overall
results thus resulted in potentially useful results for future
development.

Haiku Generation

As demonstrated in Igarashi1,2, this paper uses a methodthat uses “Chainer,” a deep training framework. Previous
research1,2 used 62 phrases in “Oku no Hosomichi” astraining data. learned pattern is the appearance pattern ofwords and characters.
In this paper, 2000 haikus from the “modern haiku
database” (http://www.haiku-data.jp/) were used astraining data. LSTM was used as the training method. By
dividing a haiku into word units using the morphologicalanalyzer “MeCab,” a time-series data of word sequenceis achieved.
We generated haikus based on data from 100 training
iterations. In addition, we generated haikus based on datafrom 900 training iterations. Therefore, word training of
900 training iterations is the intersection point of theapproximate straight line (inverse proportionality
relationship between the understandability of haiku andinteresting haiku). Fig.2 shows the result of previous
research2.

4. Conclusion
In this paper, we attempted to generate a haiku by training
using LSTM. This improved training data from previous
trials. Compared with previous attempts, the variance ofthe quality of the generated haiku increased, but wesucceeded because we aimed to generate a haiku
irrespective of the input. In the future, while trying tocontrol the formal aspect, we will try to create haikus
from the fused narrative approach.
References
1. K. Igarashi, T. Ito and T. Ogata, An attempt of Haiku
generation by deep learning. Proc. of Special Interest
Group on Language Sense Processing Engineering, 57(2018), 35-37.
2. J. Ono, T. Ito and T. Ogata, Considering a Haiku
Generation Process Using Deep Learning. Proc. of theFig.2 Deep training based on words2(X-axis is each Haiku/ Y-axis is averages of evaluations
each Haiku. The solid line: understandability / The
broken line: interest)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan579

A Method of Haiku

3.

4.

5.

6.

7.

8.

28th Annual Conference of the Japanese Neural Network
Society, (2018), 166-167.
K. Yoneda, S. Yokoyama, T. Yamashita and H.
Kawamura, 2018. Generated haiku distinctions by deeplearning. IPSJ SIG Technical Report, 2018-ICS-192 (8)(2018).
A. Hirota, N. Oka, M. Araki and K. Tanaka, Haiku
generation by seqGAN that divided learning data set. Proc.
of the Twenty-fourth Annual Meeting of the Association for
Natural Language Processing, (Okayama, Japan, March2018), 1292-1295.
R. Rzepka and K. Araki, Haiku Generator That Reads
Blogs and Illustrates Them with Sounds and Images, Proc.
of the Twenty-Fourth International Joint Conference on
Artificial Intelligence (IJCAI 2015)
T. Ito, and T. Ogata, On haiku/narrative generation. In
Proc. of the 56th Special Interest Group on Language
Sense Processing Engineering, (Morioka, Japan, Dec. 0106, 2017), 35-37.
T. Ito, J. Ono and T. Ogata, Haiku Generation Using Gap
Techniques, 2018 International Conference on Artificial
Intelligence and Virtual Reality, V-004.
T. Ogata, The Integrated Narrative Generation System From Mechanism to Contents-. In T. Ogata, Y. Kawamura& A. Kanai Information Narratology, (Tokyo, HAKUTOSHOBO publishing, 2018), 247-288.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

580

The Pragmatic Features of Onomatopoeias in the Recipe Corpus
Hiroki Fukushima*

University Department, Kyushu Women’s University,1-1, Jiyugaoka, Yahata Nishi Ward,
Kitakyushu City, 807-0000, Japan†
E-mail: fukushima@kwuc.ac.jp
www.hiroki.fukushima.jp

Abstract
In this study, we analyzed the pragmatic features of the onomatopoeias in the recipe corpus.As a clear result is thatthe onomatopoeias tend to co-occur with the words for taste or flavor (i.e. sweetness, acidity, umami, flavor, and soon.). This tendency suits to the function of the onomatopoeias in the tasting description corpus which the author haverevealed.

Keywords: Onomatopoeia, recipe, form-masking method

smaller than chop and slot. Thus, vowels have something
to do with sound symbolism.
In the previous studies[1], researchers focused oncollecting symbolic expressions and grouping them into
categories, identifying what sound corresponds to whatmeaning. Cross-linguistic studies have been conducted toshow if there are universal trends in sound symbolism.
However, little systematic research has been done toshow how symbolic words are used in a perceptual
domain.

1. Introduction
In a natural language, the connection between form andmeaning is said to be arbitrary. There is a rich vocabulary
whose form (i.e., sound) is suggestive of its meaning. In
English, for example, flare and glare are different in
intensity, and the semantic difference is due to the sounddifference between f- and g-. The same goes for thedifference between sarasara (i.e., smooth) and zarazara(i.e., rough) in Japanese. Thus, in the case of soundsymbolism, the meaning of a word is partially influenced
by the sound of the word. Sound symbolic expressions
are often used to describe the perception of taste andsmell in Japanese.
In this study, we focus on Japanse sound symbolic
words relating to tastes. Onomatopoeia is a typicalexemplar of sound symbolism, and yet, the sphere ofsound symbolism is much broader. Chip and slit sound

2. MethodIn other to identify the semantic functions of soundsymbolism in text, the author have developed a new
method or what we call “the form-masking method, ” bywhich we were able to reveal the co-occurrence© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

581

Hiroki Fukushima

3. Resultrelationships within corpora, without considering
specific meanings of symbolic words. [2]By this method, we can reveal the pragmatic features
such as “what do onomatopoeias modify in general?”, or“what is the pragmatic difference between the
onomatopoeia-containing sentences and the nononomatopoeia-containing sentences”, rather than theindividual usage of the onomatopoeias.

The result of Chi-square test (with Yate’s correction) is
shown in the Appendix A and B. Note that Bonferroni
correction for the multiple comparisons is adopted.
4. Conclusion
In this study, we analyzed the pragmatic features of theonomatopoeias in the recipe corpus.
As a clear result is that the onomatopoeias tend to cooccur with the words for taste or flavor (i.e. sweetness,
acidity, umami, flavor, and so on.). This tendency suits
to the function of the onomatopoeias in the tasting
description corpus which the author have revealed[2].

2.1. Procedure
0.
1.
2.
3.

Prepare and cleansing the corpus
Extract the onomatopoeias in the corpus
Replace each onomatopoeia with a masking
word
Extract only the “onomatopoeia(maskingword)-containing sentences” from the whole
corpus. Remains are the sub-corpus that is “nononomatopoeia-containing sentences”.

Reference

2.2. Data
In this study, we analyze the recipe database of Cook Pad,
the recipe-posting website in Japan.The “whole corpus”
consists of the title and abstract of the posted recipe.
Table 1 shows the detail of the corpora. Note that theOS (onomatopoeia-containing sentences corpus) and thenonOS (non-onomatopoeia-containing sentences corpus)
are the sub corpus of the Whole Corpus.

[1] L. Hinton, J. Nichols and J. J. Ohala, Sound symbolism.
Cambridge University Press, 2006.
[2] H. Fukushima, M. Imai and S. Tanaka, "The Usage
Mechanism of Japanese Ideophones in the Description of
Taste: Morphological and co-occurrence analysis of thedescription of wines and sakes", International Journal ofComputational Linguistics Research, vol. 8, no. 3, pp.
109–122, 2017.

Table 1

Whole
Corpus

OS

nonOS

token

9,928,126

1,310,839

8,617,286

type

83,86822,467

80756

total
sentences

2,726,551

304,145

2,422,405

2.3. Analyzed words

In order to reveal the pragmatic features of OSand nonOS (i.e. co-occurrence features of
onomatopoeia), following words are
examined:
Top 150 words of both OS and nonOS
Characteristic words of each corpus (75words for
each, by Jaccard Index)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

582

The Pragmatic Features ofAppendix A.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

word

χ2

ｐ

eat
vegetable
taste
sweet
summertexturegingerjuicy
vitaminnutrition
outsidefeelingfinished
umami
umami taste
hot
fiber
matchingmorningtasty
doughvolumebodyfinish(ranked)
smell
lemon
stew
effective
aromatic
vinegarsoftlyflavorput into
smell(V)appetite
acidityrich
fleshrich taste
soft
sweetnessaddcookieseasonfeeling

7477.34
18277.38
5322.535506.438365.99443.461409.64
2006.38
1456.316603.28594.9922571.46
1669.46
1923.03
3102.725006.834832.192389.915012.831380.171365.361326.29
1219.91174.17939.67927.63840.1
829.07799.22793.22791.57790.29734.93706.24705.59688.95667.59647.83611.91
548.25542.19529.02491.86458.61410.63402.08<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001

47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*

fresh
bread
ponzu sauce
hot(spicy)
a dishget coldgentlediffer
broil
soup
cake
labor
adult
skin
egg
bread
roll
go forward
Welsh onion
beer
leef
seasoningchill
cheesehealthy
breakfastume plum
salad
water
cabbagehamburger
salty-sweet
sake
Chinese cabbage
good
sauce
use
bit hot
time
deep-fry
chickenbutteryoghurtmayonnaise

392.69381.98377.3
364.74332.52321.55312.77233.14215.72204.96
199.74185.88165.19162.24156.1
140.59129.83115.69112.87110.02
107.27107.24104.72
80.56
74.53
67.6
66.5
61.35
61.17
53.73
44.1
23.83
19.97
17.55
17.3
12.3
11.72
11.05
10.08
9.72
8.78
6.13
5.93
5.79

<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.0001<.00010.00050.00060.00090.00150.00180.003
0.01330.01490.0161

df = 1, * p < .00026© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

583

*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*

Hiroki Fukushima

Appendix B.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

Words
easy
home
microwave ovencharacter Bento
toast
topics (fn.)
baconregular dishes
Natto
thank
de (fn.)left overcanned tuna
green pepper
dietsoya milk
frying pan
cut
avocadospinachgratinperfection
cookpot
soy sauce
carrotcorn
original
salmoneggplant
balsam pear
mayonnaise
think(fn.)
saucechickenfreezing
left overbaby food
cheep
many
boil
kimchiingredients
calorie
preferencecookstir-frylove(V)
recipeingredientsremakedress

χ213889.69
2616.292549.44
1902.71
1836.53
1825.70
1770.35
1761.51
1724.92
1711.191658.441638.78
1626.43
1612.86
1590.76
1589.39
1551.741545.641536.34
1429.801419.131412.121402.671398.101384.23
1381.03
1368.23
1342.171328.151319.16
1287.921255.01
1233.68
1231.091222.561205.04
1196.35
1172.891165.581162.38
1160.85
1151.25
1150.34
1149.27880.23811.39780.48673.15669.48607.09510.49ｐ< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001

53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95

*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*

OK
easy-madeBentorice cooker
broccoli
miso
mix together
home-like (fn.)
simpleone bowl meal
pasta
cook
boiled food
Kimpirausage (fn.)
jamhandmade
pizza
delitedseasoning
likeon the market
recommend
salted Koji
Kombu kelpcurrylol (fn.)
pork
Chinesefridgepudding
pork
potatomade (fn.)
childrenarrangetomatospringsalt
go wellboil
chicken467.08373.65363.52320.40303.19290.29259.54255.45253.49218.44215.26213.03204.61202.36201.42179.00175.67171.04151.40136.93134.07128.51
128.35
127.54118.29109.91
93.34
86.04
78.22
68.19
48.73
48.31
47.88
45.17
42.45
42.30
41.46
29.31
20.93
20.17
13.82
13.26
6.27

< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001< .0001
0.012© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

584

*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*

Storytelling in the Conversation of Aged People
Yuki Hayashi

Graduate School of Science and Engineering, Chiba University, 1-33 Yayoi-cho, Inage-ku, Chiba-shi
Chiba, 261-0004, Japan*
E-mail: yu@chiba-u.jp
Akinori Abe

Faculty of Letters, Chiba University, 1-33 Yayoi-cho, Inage-ku, Chiba-shi
Chiba, 261-0004, Japan / DWANGO Artificial Intelligence Laboratory, Tokyo, Japan †E-mail: ave@ultimaVI.arc.net.my

Abstract
For early detection of dementia, we regarded dementia as a type of communicative problem which was gradually
articulated through communication with people concerned. From the viewpoint of that, we examined narrative datastored in DIPEx-Japan and classified characteristic phrases into several categories.
Keywords: early detection of dementia, analysis of semi-structured interview data, illness narrative.

shown in Manual for preventing nursing4. According tothis list, if an elderly person is often pointed out thathe/she is forgetful, his/her cognitive functions will besuspected to decline. However, it is difficult to determine
whether an elderly person is dementia or not. Generally,
dementia is defined as ``the sustainable decline ofintelligent abilities and the state which interfere withdaily life and social life due to several types of cognitive
disorders''6. We think early detection of dementia isdifficult due to the ``sustainable'' decline of intelligent
abilities. In this paper, we will discuss the difficulty ofearly detection of dementia based on narrative data.

1. Introduction
It is said that early detection of dementia is important. For
example, Honma pointed out the importance of the early
detection as follows1.
In case of Alzheimer's dementia, its progress will bedelayed by Donepezil hydrochloride, which is a typeof medicine. Therefore, a period of in-home care will
be able to be extended.
The right of self-determination will be respected. For
example, the elderly people will be able tocommunicate with their family in regard to wealthmanagement or nursing before they will suffer fromseveral symptoms of dementia.
QOL (Quality of Life) will be preserved. If caregivers (including family members) recognize thestate of dementia in advance, they will be able tokeep communicating with a person with dementia.
To detect dementia, several tools have been
recommended. For example, fundamental check list is

2. Dementia problem
Alzheimer's disease, Lewy body disease or
cerebrovascular disease are known as the factor of
dementia. However, an appearance of these disease is not© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan585

Yuki Hayashi, Akinori Abe

3.1. Method
We have analyzed semi-structured interview data storedin DIPEx-Japan2. DIPEx is a database of personal
experiences of health and illness. It contains patientinterviews data and the database is opened to publicthrough the Internet. So patients, their family members
and medical professionals can access the narrative.
DIPEx-Japan has various theme of narrative shown in
Fig~\ref{dipex}. In each section, there are voice data(some interviewees hide their face) and text data like
Fig~\ref{dipex_sample}. In this chapter, we will focus
on narrative about ``beginning of dementia''.

Fig. 1. Fundamental Check List4

always equal to the beginning of dementia. Deguchiinterpreted dementia ``not just as physical and somatic
problem for the elderly people but as interactive problem
due to communicative disorder between the elderly
people and people concerned''5. So we regard dementia
as communicative problem which is gradually articulated
through communication with people concerned. Throughan interview data, Deguchi illustrated the phases ofdementia troubles as follows.
Zero point of an unspecialized trouble,
Variation of an interpretation or definition about thetrouble,Confusing how to deal with the trouble,
Regarding the trouble as a claim,
Discussing how to deal with the trouble amongpeople concerned,
Determining how to deal with the trouble amongpeople concerned
To discover the beginning point of dementia trouble(``Zero point of an unspecialized trouble''), Deguchiinterviewed the elderly people with dementia and theirfamily. However, there are not so many samples in
Deguchi (1999) so we'll analyze the other type ofinterview data in the next chapter.
Furthermore, in articulation and cognitive anomie
hypothesis, Nakagawa examined the process of dementia.
According to his hypothesis, ``a claim on unspecialized
trouble will refuse a process of normalization. Then theclaim will gradually be articulated and be classified into
deviation, social problem or the other type of problems.
Finally, the factor and solution for the claim will also be
articulated''3. In the following chapter, we will analyze
communicative aspect of dementia and examine thedifficulties of early detection of dementia.

Fig. 2. The construction of interview data. Upper side is voice
data and lower side is text data (in Japanese).

3.2. Results
We examined how caregivers (including family
members) noticed the beginning of dementia before some
type of troubles are clearly classified into dementia. In
the following analysis, characteristic phrases will beshown in several categories.
The first type of narrative is the narrative in which people
didn't regard as dementia. There are some examples asfollows (with Japanese translations).
I didn't imagine the possibility of disease. (病気の予想なんかしてなかった。)
I think our family had never considered thepossibility of dementia. (認知症かどうかっていう

3. Analysis© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

586

Storytelling in the Conversation

Fig. 3. The construction of narrative in DIPEx-Japan.

He made a phone call to a certain person and didsoon again to the same person. (さっき電話をかけたところにまたかけ直す、みたいなことがあって。)
These behaviours are nearly equal to the dementia
symptoms which are known as typical ones. In addition,some people focused on the other type of behaviour.
Her bahaviours, especially in the way of walking,looked wrong for me. (やっぱり挙動ですかねえ。
歩き方がおかしいとか。)
Her sleeping hours gradually increased. (睡眠時間がだんだん長くなってきたようなのを覚えています。)
As we mentioned, the beginning point of dementia is
usually not clear. In fact, the following examples shows
the aspect of dementia and a type of confusion for family
members.
I'm not sure when is the beginning of dementia. (いつごろ始まったかっていうのは、はっきりしない。)
In those days, I was not sure whether my husband isdementia or not. (そのときは、やっぱり、分からなかった。)

ような風にはみんな捉えてなかったんじゃな
いかと思う。)
In several cases, we observed the reason why people had
not regarded as dementia.
My mother had depression and saw her doctor
regularly. So I was not sure whether she wasdementia or not. (うつになって以来神経科とかにかかってて、それでずーっと来たので、その異変ていうのが、どこから異変ていうのがわからなくて。)
I thought she looked strange due to her age. (年をとって、そういうふうになっているみたいな。)
On that day, he might be not feeling very well. (その日は、体調が悪いのかなとか思って。)
Secondly, there are the narrative in which people pointed
out some specific events they felt wrong or strange asfollows (with Japanese translations).
I think she often left her belongings. (置き忘れとか
はね、結構あったと思う。)
It took so long time for my husband to search his
belongings. (あれどこいったんだろうって、探す時間が長くなって。)
He sometimes forgot a password for ATM. (たまに暗証番号忘れてお金が下ろせなかったりとか。)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

587

Yuki Hayashi, Akinori Abe

3. N. Nakagawa, Social problem game and researcher'sgame, the Bulletin of faculty of liberal arts, Toyama
University, vol. 25, pp. 57--81, 1995.
4. Revision committee on manual for preventing nursing, On
preventing nursing, in Manual for preventing nursing, pp.
1--37, March 2012.
5. Y. Deguchi, Micro politics of troubles about senile
dementia---retrospective narrative on family care-givers'
trouble experiences---, Sociologist, faculty of sociology,
Musashi University, vol. 1, pp. 39--75, 1999.
6. Y. Takebayashi, H. Ueno, Development of fundamental
technology for understanding emotions of people withdementia and its applications to communication support,
The 27th Annual Conference of the Japanese Society for
Artificial Intelligence, 3A1-NFC-03-2, 2013.

However, there are the narrative in which people pointed
out something wrong or strange as follows (withJapanese translations).
A few years ago before my husband was diagnosed
with dementia two years ago, I think something
strange happened around him. (その診断の 2, 3 年
前には、何かいろいろあったような気がする。)
My mother was mature for her age. However, there
were something strange with her. (しっかりした母親やったんですよ。ところが、何か様子がどうも。)
When I visited my mother, I felt something wrong
with her. (訪ねて行ったときに、どうもいつもと様子が違う感じを受けた。)
3.3. Discussion
Usually people didn't determine whether their family
member was dementia or not because they compared
their family with common people. In addition, it may bedifficult for them to detect the differences between
normal state and wrong state.
In the second type of narrative, people pointed out somespecific events they felt wrong or strange. However,these behaviours are nearly equal to the dementia
symptoms which are known as typical ones. So it is not
so suitable for early detection of dementia. On the other
hand, in the last type of narrative, people told they felt
something wrong or strange. These narrative seem not tohave significant meanings, but feelings which family
members as caregiver may feel are important. We thinkthese narrative are less influenced by cognitive bias.
4. Conclusion
For early detection of dementia, we regarded dementia asa type of communicative problem which was gradually
articulated through communication with people
concerned. From the viewpoint of that, we examined
narrative data stored in DIPEx-Japan and classified
characteristic phrases into several categories.
References
1. A. Honma, Early detection of senile dementia and future
prospect in local cities, Gerontology New Horizon, vol. 15,
pp. 20--23, 2003.
2. DIPEx-Japan, Narrative on health and illness, referred on
27th April 2017.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

588

Expression of the taste of Japanese sake and metaphor
Akinori Abe

Division of Behavioral Science, Faculty of Letters, Chiba University, 1-33 Yayoi-cho,Inage-ku, Chiba 263-8522, Japan
Dwango Artificial Intelligence Laboratory, Tokyo, Japan
E-mail: ave@ultimaVI.arc.net.my, ave@chiba-u.jp

Abstract
Seto explained ``tasting expression'' from the aspects of synesthetic expressions (both direction), which are metaphor,metonoymy, and synecdoche. When we express the taste of Japanese sake, we sometimes use the expression``Refreshing scent such as the scent of muskmelon and pear is elegant...'' It is rather easy to express the taste by using
the well-known taste. However, how we control the path of the metaphor flow? For instance, we frequently use theexpression ``cheek like apple.'' Why we assume the colour is red? Similarly, for the taste of the sake, we frequently
use the expression ``taste like apple.'' Which apple we assume? In this paper, I will discuss the flow of the metaphor
from the aspect of discussion above..
Keywords: expression of taste, metaphor, Japanese sake, affordance

I will discuss the taste of sake from the aspect of
metaphor and flow from the source to target.

1. Introduction
Seto explained ``tasting expression'' from the aspects ofsynesthetic expressions (both direction), which aremetaphor, metonoymy, and synecdoche [1]. When we
express the taste of Japanese sake, we sometimes use theexpression ``Refreshing scent such as the scent ofmuskmelon and pear is elegant...'' It is rather easy to
express the taste by using the well-known taste. For thedescription of the taste of Japanese sake, I previously
discussed in [2] and [3]. I discussed taste has a shape.
Several researchers have discussed the taste by very
simple shape such as round and rectangle (For instance,[4]). However, I thought more complex shape should be
dealt with for the taste. Because the taste is rather
complicated and it is not easy to express only by simple
shape such as round and rectangle. In addition, the shape
can be extended to metaphoric expression. In this paper,

2. About the expression of taste
Sommeliers have several words for the expression of thetaste of wines. These words can be regarded as poetic,
emotional or affective words. Accordingly they can
express the taste better but sometimes are not easy tounderstand. Seto explained ``tasting expression'' from the
aspects of synesthetic expressions (both direction), which
are metaphor, metonoymy, and synecdoche. It means thattaste can be expressed by the figurative expression, andthat taste is expressed by using touch, smell, sight, andhearing [1]. For instance, in the description ``[t]he tasteof chat masala is explained that the rounded spicy tasteof Aji Amarillo is added to the spicy taste of garam
masala,'' sight is used to explain the taste. The description
can be regarded as a metaphor. The shape is used as a

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan589

Akinori Abe

metaphor to express the non-sharp taste. Thus when we
express the taste, we frequently use a metaphor, because
it is rather difficult to find or create a nice word. A
metaphor is a good tool to express novel concept.

candy(https://www.delongwine.com/blogs/de-longwine-moment/14610411-tasting-terms-pear-drop). Thus
all are strictly defined.
4.

3. Expression of the taste of wine

Expression the taste of Japanese sake

For the taste of Japanese sake, according to Fukushima,
there exist only critical words such as rough taste (雑味(zatsumi)) which can express a bad taste. This is becausesuch words are usually used for (critically) checking thetaste of sake in sake factories. Thus words for a good taste
have not been established yet. In addition, there exists a
textbook for studying to pass the master of sakeexamination. It will not be used as a general dictionary.
Fukushima has built the ``Japanese Sake Tasting
Dictionary'' [5], but it is not large dictionary. Accordingly,
when we express the taste of Japanese sake, we will use
our own expression. For instance, in ``LOVE
Japanese sake!, 2014, 2 Gakken,'' the taste of Isojiman
Junmai ginjo ( 磯 自 慢 純 米 吟 醸 ) was described asfollows:
Refreshing scent such as the scent of muskmelon andpear is elegant...
(マスクメロンや洋ナシを思わせる爽快とした香りは気品すら感じさせるもの....)The phrase describes the taste by borrowing the taste offruits. Perhaps, the taste of muskmelon and pear will be
the same for everybody. However for a certain sake,ambiguous express can be used. Since we do not have a
dictionary, in such case, the impression may be different
according to the culture. As pointed above, a strict
definition may be required in such cases. However
sometimes without such a strict definition, we canunderstand the meaning of metaphor.

There exist several dictionaries for the wine tasting. All
definitions in the dictionary are clear. For instance, taste
is defined by using well known taste such as butter. When
we describe the taste of wine, we usually use the phrases
in the dictionary. For instance, in BC Liquor Stores, Vol.
19, Issue 2 (summer 2017), the following descriptions
can be found:
Quail's Gate Pinot Noir (2015)
Elegant and sophisticated, this Burgundian-style Pinot
Noir expresses bright strawberry, red cherry and spicewith hints of forest floor and earth. Medium-bodied
with fine tannins, this will pair well with salmon.
We can easily imagine the taste of wines. In fact, ``hints
of forest floor'' is frequently used phrase for nice wines.
In addition, ``forest floor'' is used for matured wine. In
general, the phrase is used for the (nice) wines produced
in Bourgogne. A forest floor is defined as the aboveground layer of a forest made up of tree roots, soil anddecaying matter (Collins English Dictionary). Thus thetaste of wine can be imagined as a soil and decaying(matured) taste and flavour. The richly organic layer ofsoil and debris is a characteristic of forested land. This
will also be a characteristic of the wine. This phrase canbecome a type of a good metaphor. Readers may
understand correctly the taste of wines by the above
descriptions. As for the metaphor of the wine taste, there
seems very few misunderstandings. For instance, thewine taste is strictly defined as follows:
Apple: A smell often found in young white wines. Unripe
apple is often a sign that a wine has not undergone
malolactic fermentation.
Apricot: Common in the white Rhone's of Condrieu andChateau Grillet and other examples of the Viognier grape,and in wine from botrytis-affected grapes.
In addition, the cultural difference is understood as
follows:
Pear drop is a term often used by wine writers to describe
the flavor of Beaujolais Nouveau. These writers would
all be British since pear drops are virtually unknown inNorth America. They're a traditional type of candy thatthe British call boiled sweets and Americans call hard

5. Metaphors are common?
In [6], I discussed the process of understanding metaphor.
When we use the phrase including metaphor ``a cheeklike an apple,’’ we assume that the cheek belongs to apretty young girl and its colour is red. Actually there areseveral colour of apples, for instance, red, yellow etc.
Why we can understand the colour is red? I discussed theprocess of understanding metaphor by using the conceptof metaphor. Jung et al. discussed the metaphor in aninterface (human computer interaction) design as shownin figure 1.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

590

Expression of the taste

concept of affordance. Affordance was introduced by
Gibson [7] [8]. Gibson ecologically introduced theconcept of affordance for perceptional phenomena. Itemphasizes the environmental information available inextended temporal pattern in optic arrays, for guiding the
behaviors of animals, and for specifying ecologicalevents. Thus he defined the affordance of something as``a specific combination of the properties of its substanceand its surfaces taken with reference to an animal.''I described the taste of Kagatobi just-squeezed raw nonadditional-water Ai (加賀鳶 しぼりたて生原酒 藍) as
follows (part of):The taste is nicely and strongly harmonized. At last, ataste of fruit (pear) repeats in the mouth. Before that apungent taste slightly comes on the top of tongue. (うまく、濃くまとまっている。最後に口の中に薄く果実(梨)の味が残る。その前に舌先に軽くピリ感。)
I used the word ``pear’’ in the description. We haveseveral types of pear in Japan. For instance, niju-ichiseiki (二十一世紀), kosui (幸水), and tyoujurou (長十郎). A niju-ichi-seiki is rather sour and others are rather
sweet. In the above description, for pear I imagined thetaste of niju-ichi-seiki, which is fresh and clear. Perhaps,
even if a person has not taste the sake, he/she will
imagines the taste of niju-ichi-seiki. Because the taste ofsuch a just-squeezed raw sake is usually fresh and clear.
As discussed in [6], if we mention a pear, there should beseveral possibilities in taste. However, why we candetermine the taste of a pear as one exact taste? For thissituation I think it is possible to consider the sameprocedure as in the determination of the colour of anapple. For the interpretation of the taste of a pear, we cansee several affordance to select a proper one by abduction.
If it is formalized by Theorist [9], the formalization is asfollows:
F ∪affordance ⊢ ideal feature.
F ∪ affordance ⊬ □.
In the case of the taste of pear, the following can beconsidered.
F ∪pear ∪{affordance= niju-ichi-seiki} ⊢ fresh andclear taste.
F ∪pear ∪{affordance= niju-ichi-seiki} ⊬ □.
Thus when tasting ` Kagatobi just-squeezed raw nonadditional-water Ai ‘ and image the taste of pear, we canselect an affordance that pear is niju-ichi-seiki. Then we
can determine the type of pear is niju-ichi-seiki.

Figure 1: User interaction mediated through different
formal aspects
The process is for an interface design, it will not possible
to apply this structure to the normal metaphor, but theconcept is interesting. From source to target, three links
are connected and the understanding is performed by thefollowing three aspects:--The surface aspect of interactive forms is about how adigital application is represented through its interface toindicate what a user can do with it.
--The behavioral aspect is about how an artifact responds
to interaction events by unfolding corresponding formalchanges that are coded behind the surface.
--The systemic aspect of an interactive form is about how
its multiple states are connected to each other in a
coherent structure.
Thus Jung et al. claimed that forms and meanings ofartifacts are connected across various material domains
and that metaphors implicitly or explicitly play a key rolein bringing a new design perspective from one domain toanother, sometimes reified as design conventions. Theirinvestigation extended the perspective on affordancesfrom perceivable action possibilities to invitations forinterpreting forms and meanings of an interactive artifact.
They also highlighted the role of metaphors as asystematic strategy for exploring materialities andaffordances of digital media.
This position is for the HCI design, so, as pointed out
above,
it is rather different from the metaphor
understanding in our domain. However, the concept ofaffordance can be applied to our domain and theapplication will be significant.
6. Metaphor understanding by affordance
selection
As discussed in the above, the function (selection) ofaffordance will be important in metaphor understanding.
However, how to select the proper affordance formetaphor understanding? As shown in the previous
section, the meaning of pear drop differs according to thecountry. The phenomenon can be considered by the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

591

Akinori Abe

7. Conclusions[9] D. Poole, R. Goebel and R. Aleliunas, "Theorist: A
Logical Reasoning System for Defaults and Diagnosis,"in The Knowledge Frontier: Essays in the
Representation of Knowledge, N. J. Cercone and G.
McCalla, Eds., Springer-Verlag, 1987, pp. 331--352.
[10] H. Jung, H. Wiltse, M. Wiberg and E. Stolterman,"Metaphors, materialities, and affordances: Hybridmorphologies in the design of interactive artifacts,"
Design Studies, vol. 53, no. C November, pp. 24--46,2017.

In this paper, I discussed the interpretation of metaphor.
In fact metaphor is used to express the taste of food. The
metaphor is used for instance based on shape and familiarthings. When we use a word ``rounded’’ for the spicy
taste, the metaphor is used to reduce the level of spicy.
That is, to remove the sharpness from the taste, rounded
is used as a metaphor. For the taste description, familiar
fruits are frequently used as a metaphor. For instance, forwine tastings, apple and apricot are used as a metaphor.
A phrase `` a smell often found in young white wines.’’
is used. In fact in Japan there are various types of apples.
Perhaps this description is made in Europe. Accordingly
the apple will be a Jonathan. However, in Japan, if weread this phrase, we will be wonder which apple theymention. If we properly select an affordance from apples,such a wondering will be solved.
The main theme of this paper is affordance and abduction
based metaphor understandings. Such ambiguous
metaphor can be determined by selecting a proper
affordance and the affordance is abductively selected.
This paper mainly deals with a metaphor in tasting. In
addition, a metaphor in poems will be dealt with in thesame or similar way as in tasting. In the next paper, I will
also deal with interpretations of metaphors in poems.
Furthermore it can be extended interpretations ofmetaphors in stories.
References[1] K. Seto, Ed., Language will be over the taste, A quest fora description of a good taste, Kaimeisha, 2003.
[2] A. Abe, "About the better expression of taste," Proc. ofthe 2nd. Int'l Workshop on Language Sense on Computerin IJCAI2017, pp. 28--34, 2017.
[3] A. Abe, "Is the expression of taste abstract orrepresentational?," Proc. of SMC2018 Workshop on
Narratology and Cognitive Content Generation, pp. 18-23, 2018.
[4] C. Velasco, A. T. Woods, O. Deroyc and C. Spence,"Hedonic mediation of the crossmodal correspondence
between taste and shape," Food Quality and Preference,
vol. 41, pp. 151--158, 2015.
[5] H. Fukushima, Japanese Sake Tasting Dictionary, Keio
Gijuku University, 2013.
[6] A. Abe, "Why a young girls cheek is red?," Technical
Report of JSAI, Vols. SIG-LSE-B803, pp. 1--8, 2018.
[7] J. J. Gibson, "The Theory of Affordances," in Perceiving,
Acting, and Knowing, R. Shaw and J. Bransford, Eds.,
Lawrence Erlbaum Association Publishers, 1977, pp. 67-82.
[8] J. J. Gibson, The Ecological Approach to Visual
Perception, Houghton Mifflin, 1979.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan592

An Analysis on Advertising Techniques of Beverages Using Positive Factors of Evaluation
Database System
Yoji Kawamura

Faculty of Business Administration, Kindai University, 3-4-1 Kowakae
Higashi-osaka, Osaka 577-8502, Japan
E-mail: kawamura@bus.kindai.ac.jp
www.kindai.ac.jp

Abstract
This study aims to systematically shed light on what kind of effects are brought about by the film techniques inindividual CMs. Based on audiovisual experiments, with positive factors as the evaluation index, we considered
what could be analyzed from film techniques and their effects. For this study, we conducted audiovisualexperiments of communication effects (positive factors, interest, and the willingness to buy) by using 100
commercials for beverages (tea, canned coffee, and beer), and analyzed the relationships between film techniqueand product and category characteristics, historical transitions, interest, and the willingness to buy. Subsequently,
we examined the results of the audiovisual experiments. According to the tabulation results, the tea Sokenbicha(Coca Cola) and Namacha (Kirin) tended to focus on “atmosphere 2 (sexy or cute)”; canned coffee Boss (Suntory)
tended to focus on the “story,” “mise-en-scène (person),” and “atmosphere 1 (humor)”; canned coffee Wonda(Asahi) tended to focus on “message” and “mise-en-scène (person)”; and Super Dry beer (Asahi) tended to focus on“message,” “mise-en-scène (background),” and “empathy.” These kinds of film techniques and effects structure a
product’s brand image or “identity.” According to our quantitative formulation, “mise-en-scène (person),” “miseen-scène (background),” “story,” and “sound” are important for increasing the level of interest. “Empathy,” “miseen-scène (background),” and “message” are important for increasing the level of willingness to buy.
Keywords: Commercial Film, Film Technique, Effect, Positive Factor, Beverage

positive factors. The positive factors of the CM Soken
Consulting have by no means one to one
correspondence with film techniques, but a number ofthem involve film techniques. Probably, the institutealso has a research approach that examines the effects
and transitions in film techniques based on acomprehensive evaluation database by using positive
factors as the evaluation index. However, the CM
likability surveys of the CM Soken Consulting dependon the ad placement frequency and social environment(such as trends and current affairs), and hence they do
not evaluate the film technique of an individual CM.

1. Introduction
Even though film techniques and effects of televisioncommercial message (CM) have been studied byresearchers such as Stewart and Furse1 and Kawamura,2,
3, 4, 5, 6
CMs are complex information media, and hence ittakes a significant amount of time and effort to evaluatea CM comprehensively. Therefore, attempts tosystematically study film techniques and their effects
have been largely unsuccessful. Meanwhile, in theadvertising business, the CM Soken Consulting built asystem for evaluating CM expressions comprehensively,
using CM likability studies within the framework of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

593

Yoji Kawamura

This study aims to systematically shed light on whatkind of effects are brought about by the film techniques
in individual CMs. Based on audiovisual experiments,
with positive factors as the evaluation index, we
considered what could be analyzed from film techniques
and their effects. For this study, we conductedaudiovisual experiments of communication effects(positive factors, interest, and the willingness to buy) byusing 100 commercials for beverages (tea, canned
coffee, and beer), and analyzed the relationships
between film technique and product and category
characteristics, historical transitions, interest, and thewillingness to buy. Subsequently, we examined theresults of the audiovisual experiments.

Table 1. The film techniques/effects and positive factors.
Film techniques/effects(9 categories)
MessageFilmtechStorynique
Mise-en-scène(person)Mise-en-scène(background)Sound
EffectEmphathy
Atmosphere1

Atmosphere2

2. Positive Factors and Film Techniques

Other

The CM Soken Consulting conducts monthly CM
likability surveys and provides a database service for
evaluating CM expressions. The CM likability surveys
study likability by gender and age groups, by using 15
types of positive factors.7 Our study evaluatedimpressions (feelings) after viewing the CMs, withitems relating positive factors to effects (impressions
and feelings from the commercial as a whole) and items
relating to film techniques (such as story, mise-en-scène,
and sound). The positive factors were set empirically,and hence a challenge was that the questions about
items related to film techniques did not cover film
techniques precisely or exhaustively. However, due tothe existence of questions relating to major film
techniques, we believe that the questionnaire provides avaluable database for the analysis of film techniques.
According to film theory,8, 9, 10, 11, 12 films are composedof stories (how the intended content is constructed bymeans of cinematic expression), mise-en-scène (howshots [information elements and compositions on thescreen] are structured), editing (how shots areconnected), and sound (what type of sound is added toshots and the edited video products). In this study, we
performed our analysis by taking into account these
considerations and mapping the positive factors to thefilm techniques/effects shown in Table 1.

Positive factors(15 indicators)“impressive catch phrases”“a good story”“talents or characters”“good visual images/videos”“impressive music/sound”“attracted by the product”“it is pleasantly persuasive”“it is humorous”“not fashionable but likeable”“reassuring”“because it is sexy”“cute”“it feels at the cutting-edge oftime”“good surrounding reputation”“the firm’s attitudes appear tobe sincere”

Table 2. The targets for analysis.
Category

Product (Company)

Tea

Oolongcha (Suntory), Namacha (Kirin),
Sokenbicha (Coca Cola)
Boss (Suntory), Fire (Kirin), Wonda(Asahi)Moltz (Suntory), Lager (Kirin), Super
Dry (Asahi), Kuro Label (Sapporo)

Canned coffee
Beer

3. Analysis Method
3.1. Analysis Targets
Beverages are low involvement products, and compared
to CMs for other products, CMs for beverages make fulluse of film techniques. Therefore, we chose 10 major
products (Table 2) within the beverage category (tea,canned coffee, and beer) as targets for analysis. To
investigate from a diachronic perspective, we surveyeda total of 100 CMs created during a span of 10 years
from 1997 to 2006 (number of samples: 10 CM’ foreach product).
3.2. Subjects
The subjects were students between the ages of 20 and
23 years. The number of samples was 100 humansubjects for each of these commercials.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan594

An Analysis on Advertising

3.3. Indicators and Evaluation
As positive factor indicators for the audiovisualexperiments, we used the CM Soken Consulting’s
indicators (15 indicators) (Table 1). We used a 2-point
scale for evaluating the positive factors (“left an
impression” and “did not leave an impression”). For
tabulation and analysis, we also analyzed indicatorsclassified from the standpoints of film techniques andeffects (film techniques/effects: 9 categories). To
measure the level of interest, we used a 5-point scale (1:Not interested, 2: Not particularly interested, 3: Fairly
interested, 4: Pretty interested, 5: Extremely interested).
To measure the level of willingness to buy, we used a 5point scale (1: Definitely would not buy, 2: Probably
would not buy, 3: Might buy, 4: Probably would buy, 5:
Definitely would buy).

Fig. 1. The totals of interest level and the willingness to buy.

3.4. Audiovisual Experimental Method
We showed the CM to 100 subjects and got them toevaluate their level of interest (5-point scale),willingness to buy (5-point scale), and positive factors(2-point scale, 15 types). For tea and canned coffee, we
aired six CMs (CMs for three products, created in two
different years) to the same subjects and gatheredevaluations. For beer, we aired and evaluated eightcommercials (commercials for four products, created in
two different years). We elicited responses fromsubjects concerning their attributes of gender andconsumption frequency (daily, two to three times per
week, once a week, once a month, and almost neverconsumed).

Fig. 2. The positive factors’ totals.

3.5. Tabulation and Analysis Method
Based on the collected evaluation data, we tabulated
film techniques and effect categories and analyzed
characteristics by product, characteristics by category,and historical transitions. We also performed a
regression analysis of the relationships between theinterest and willingness to buy and film techniques andeffect categories.
4. Results and Analysis
4.1. SummaryFig. 3. The results of tabulation of film techniques/effects.

A total of 10,000 samples were tabulated, comprising
50.5% and 49.5% of males and females, respectively.
The totals for the interest level and the willingness tobuy are shown in Fig. 1. The proportion of subjects
interested (3~5 scales) and those who expressed awillingness to buy (3~5 scales) accounted for 70.2% and58.7%, respectively. The positive factors’ totals areshown in Fig. 2. Overall, the highest-rated factor (left an
impression) by positive factors was “impressive
music/sound.” This was followed by “talents or
characters,” “good visual images/videos,” “it is© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

595

Yoji Kawamura

humorous,” and “impressive catch phrases.” Based on
the categories shown in Table 1, the results of tabulationof film techniques/effects are shown in Fig. 3. For film
techniques/effects, overall, the “atmosphere 1” received
a very high evaluation. In these categories, the next
highest were “ sound,” “mise-en-scène (person),”“mise-en-scène (background),” and “message.”
4.2. Relationship between the Interest Level andWillingness to Buy and the Positive Factors
We performed regression analysis, with the interest
level and the willingness to buy as explained variables(5-point scale) and positive factors as explanatory
variables. Fig. 4 shows the level of influence (theregression formula coefficient) of interest andwillingness to buy on the positive factors. “Talents or
characters” had the highest impact on interest, followed
in order by “attracted by the product,” “a good story,”and “good visual images/videos.” “Attracted by the
product” had the highest impact on the willingness tobuy, followed in order by “it is pleasantly persuasive,”“because it is sexy,” “good visual images/videos,” and“impressive catch phrases.” Overall, the “impressive
music/sound” was most highly evaluated, but it did nothave a very significant impact on the interest or
willingness to buy.

Fig. 4. The regression formula coefficient of interest andwillingness to buy on the positive factors.

Fig. 5. The totals by the product of film techniques/effects.

4.3. Characteristics of Film Techniques/effects
Categories by ProductFigure 5 shows the totals by the product of film
techniques/effects. Sokenbicha (Coca Cola) andNamacha (Kirin) had the highest evaluation among theteas for “atmosphere 2,” and the canned coffee Boss
(Suntory) had the highest evaluations for “story,” “miseen-scène (person),” and “atmosphere 1,” and canned
coffee Wonda (Asahi) had the highest evaluations for“message” and “mise-en-scène (person).” Super Drybeer (Asahi) had the highest evaluations for “message,”“mise-en-scène (background),” and “empathy.”

Fig. 6. The totals by category of film techniques/effects.

4.4. Characteristics by Film Techniques/effects by
Category

had the characteristic of being highly evaluated for“story,” “mise-en-scène (person),” and “atmosphere 1.”

Figure 6 shows the totals by category of film
techniques/effects. Tea had the characteristic of being
highly evaluated for “atmosphere 2,” and canned coffee© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan596

An Analysis on Advertising

4.5. The Historical Transition of Film
Techniques/effects
Fig. 7 shows the historical transition of film
techniques/effects. Compared to 1997, in 2006, there
was an increasing trend in the ratio of “mise-en-scène(person)” and “empathy” and a decreasing trend in theratio of “atmosphere 1.’
4.6. Relationships between the Interest and theWillingness to Buy and Film
Techniques/effects
Fig. 7. The historical transition of film techniques/effects.

Fig. 8 shows the relationships (cross tabulation)
between interest and willingness to buy and film
techniques/effects. Depending on the level of interest
and willingness to buy, the proportion of which film
technique or effect category being influenced underwent
a change. When the levels of interest and willingness tobuy become higher, the proportions of "atmosphere 1"
and "sound" become smaller. When the levels of interest
and willingness to buy become higher, the proportionsof “mise-en-scène (background)” and “empathy” tend to
become larger.
We performed a regression analysis, with the level ofinterest and willingness to buy as explained variables
and film techniques/effects as explanatory variables.
The regression formula was:(1) Interest= 2.30 + 0.24 x Message + 0.39 x Story+ 0.41 x mise-en-scène (person)+ 0.39 x mise-en-scène (background)+ 0.32 x Sound + 0.36 x Empathy+ 0.18 x Atmosphere 1 + 0.20 x Atmosphere 2(2) Willingness to buy= 2.09 + 0.31 x Message + 0.21 x Story+ 0.27 x mise-en-scène (person)+ 0.35 x mise-en-scène (background)+ 0.28 x Sound + 0.59 x Empathy+ 0.09 x Atmosphere 1 + 0.27 x Atmosphere 2
Fig. 9 shows the level of influence (regression
formula coefficient) of film techniques/effects on theinterest and willingness to buy. “Mise-en-scène (person)”had the greatest impact on interest, followed by “miseen-scène (background),” “story,” “empathy,” and“sound.” “Empathy” had the greatest impact on
willingness to buy, followed by “mise-en-scène(background)” and “message.” Mise-en-scène (person
and background) and story fulfill large roles in gaining

Fig. 8. The cross tabulation between interest and willingness tobuy and film techniques/effects.

Fig. 9. The regression formula coefficient of film
techniques/effects on the interest and willingness to buy.

interest. It is important to obtain empathy in order todrive the willingness to buy, and the message that forms
the context for the mise-en-scène fulfills an importantrole.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

597

Yoji Kawamura

5.3. Automation of Communication

5. Outcomes and Future Challenges

In other studies, through a variety of audiovisualexperiments, in addition to the formulation
demonstrated in 5.3., Kawamura13, 14 has attempted to
extract the film techniques that draw interest and thosethat drive the willingness to buy. If these techniques(rules) can be made specific and used with information
systems technologies, then they would significantly
contribute toward upgrading marketing automation.

5.1. Quantification of Brand Images
This study conducted audiovisual experiments ofcommunication effects (positive factors, interest, andthe willingness to buy) by using 100 CMs for beverages(tea, canned coffee, and beer) and analyzed therelationships between film technique and product andcategory characteristics, historical transitions, interest,
and the willingness to buy, based on the evaluation of
positive factors. When compared to a large number of
CMs created in a 10-year period (said to exceed 30,000),
100 CMs represent a small number, but they havesignificance in that their effects for advertising the samebrands over the period of 10 years were evaluated.
According to the tabulation results, the tea Sokenbicha(Coca Cola) and Namacha (Kirin) tended to focus on“atmosphere 2 (sexy or cute)”; canned coffee Boss
(Suntory) tended to focus on the “story,” “mise-enscène (person),” and “atmosphere 1 (humor)”; canned
coffee Wonda (Asahi) tended to focus on “message”
and “mise-en-scène (person)”; and Super Dry beer
(Asahi) tended to focus on “message,” “mise-en-scène(background),” and “empathy.” These kinds of film
techniques and effects structure a product’s brand image
or “identity.” Our study is significant because it wasable to elucidate this in a quantitative way. It is possible
to utilize these techniques and effects as quantitative
indicators for maintaining and innovating brand images.
This study does not specifically address this point, but it
is possible to use this methodology to analyze thehistorical transitions of brand images for each brand.

5.4. Analysis of Film Techniques and their Effects
Using a CM Evaluation Database Using
Positive Factors as an Evaluation Index
At present, the CM Soken Consulting do not cover any
survey items regarding interest or willingness to buy in
its CM likability surveys, but it is possible to calculate
the interest and willingness to buy based on theformulated functions. The formulation clearly depicts
relationships between levels of interest and “mise-enscène (person),” “mise-en-scène (background),” “story,”“empathy,” and “sound,” and there is a strong
relationship between the level of willingness to buy and“empathy.” Simply put, we believe that “film technique”(= “a good story” + “talents or characters” + “goodvisual images/videos” + “impressive music/sound”) is
an indicator for evaluating interest, and “empathy” (=“attracted by the product” + “it is pleasantly
persuasive”) is an indicator for evaluating thewillingness to buy. If we use these methods of analysis
as the basis for analyzing data in a comprehensive
system for evaluating CM expressions, then we would
be able to analyze the film techniques comprehensively
and quantitatively and analyze their effects for diverse
product categories and from a variety of time periods.

5.2. Quantitative Formulation of Relationships
between Interest and the Willingness to Buy
and Film Techniques/effects

5.5. Future Challenges

It is not easy to quantitatively evaluate the effects offilm techniques, but this study attempted a quantitative
formulation of the relationship between interest and thewillingness to buy and film techniques/effects. Our
study is significant because it could shed quantitative
light on the kinds of film techniques on which efforts
must be focused in order to trigger interest and thewillingness to buy. According to our quantitative
formulation, “mise-en-scène (person),” “mise-en-scène(background),” “story,” and “sound” are important forincreasing the level of interest. “Empathy,” “mise-enscène (background),” and “message” are important forincreasing the level of willingness to buy. A creator ofadvertising would probably see these results as obvious,
but the significance of the results is that they clearly
demonstrate the impact in a quantitative way.

This study targeted beverages for analysis, and hence itwill also be necessary to attempt to analyze otherproduct categories. Additionally, our subjects comprised
the younger generation (aged 20 to 23 years), and hence
it will also be necessary to attempt to analyze other age
groups. Based on these analysis, in the medium- andlong-term, we would like to conduct a comprehensive
analysis of film techniques and their effects using anadvertising evaluation database, and formulate a systemout of these.
Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number 22500102.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan598

An Analysis on Advertising

References
1. D. W. Stewart and D. H. Furse, Effective Television
Advertising: A Study of 1000 Commercials (Lexington
Books, Lexington, MA, 1986).
2. Y. Kawamura, An analysis of the rhetoric of commercial
film: Toward the building of a commercial film
production support system based on image rhetoric, in
Proc. 2003 IEEE Int. Conf. on Systems, Man andCybernetics, (USA, Washington DC., 2003), pp.9931000.
3. Y. Kawamura, An analysis of the rhetoric of commercial
film: Toward the building of a commercial film
production support information system, J. of Advertising
Sci., (45) (2004), pp.122-139. (In Japanese)
4. Y. Kawamura, An analysis on rhetoric of the commercial
film of beer and personal computer: A symbolic approachon rhetoric of commercial film, J. of Advertising Sci.,(47) (2006), pp.33-48. (In Japanese)
5. Y. Kawamura, A study on the technique-rhetorics andeffects of commercial film, Cognitive Studies, 14(3)(2007), pp.409-423. (In Japanese)
6. Y. Kawamura, An analysis on the story and editing
techniques of commercial film: Toward the building of acommercial film production support information system,
J. of Advertising Sci., (50) (2009), pp.16-32. (InJapanese)
7. CM Soken Consulting, Monthly top 10 CM of new works,
CM INDEX, 12(1)-21(12) (1997-2006). (In Japanese)
8. R. Whitaker, The Language of Film, (Prentice-Hall,Englewood Cliffs, NJ, 1970).
9. J. Monaco, How to Read a Film: the Art Technology,
Language, History and Theory of Film and Media,(Oxford University Press, New York, NY, 1977).
10. G. Deleuze, Cinéma 1: L' Image-mouvement, (LesEditions de Minuit, Paris, France, 1983).
11. D. Bordwell, Narration in the Fiction Film, (Universityof Wisconsin Press, WI, 1985).
12. K. Asanuma, Techniques for image text, For Movie II,(Suiseisha, Tokyo, Japan, 1990), pp.130-247. (InJapanese)
13. Y. Kawamura, A viewing experiment on the effects ofadvertising story, J. of Robotics, Networking andArtificial Life, 3(2) (2016), pp.92-95.
14. Y. Kawamura, Rhetoric of commercial film and the
response of viewers, J. of Robotics, Networking andArtificial Life, 4(2) (2017), pp.158-162.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan599

High-Performance Computing for Visual Simulations and Rendering
Jasmine Wu

National Center for High-performance Computing, National Applied
Research Laboratories, Hsinchu 300, Taiwan
Chia-Chen Kuo, Shu-Hsin Liu, Chuan-Lin Lai, Chiang-Hsiang Lien, Ming-Jen Wang, Chih-Wei Wang

National Center for High-performance Computing, National Applied
Research Laboratories, Hsinchu 300, Taiwan
1803040@narlabs.org.tw, 1803031@narlabs.org.tw
www.narlabs.org.tw

Abstract
National Center for High-performance Computing (NCHC) built a render farm that provides a platform for theindustry to render their work in a much more efficient timeframe. It allows not only CPU rendering, but also GPU
rendering. The throughput is also greatly improved to support complex and large scale simulations. The main goalof this session is to introduce scientific discoveries and technical innovation through prospective computing
technology and platform.
Keywords: render farm, visual simulation, virtual desktop infrastructure, high-performance computing

using our platform by animation students in
Taiwan, as well as companies from the industry.

Introduction
We have a reputation for global technology
manufacturing and professionals in Taiwan. But,
more importantly, we bring forward the culture
aspect of Taiwan with our developing technology.
The central thinking of National Center for
High-performance Computing (NCHC) is
whether technology may be closer to people, andhow it can act as a tool for preservation andcreativity in culture1.

Fig. 1. An animation contest conducted once a year by
NCHC since 2011, providing contestants services of
the render farm.

NCHC built a render farm, a cluster of
networked computers with a high performance
speed and enormous amount of data storage
devoted to rendering images, with hopes to help
improve the film industry in Taiwan. The renderfarm can be directly accessed through Virtual
Desktop Infrastructure (VDI), allowing users totake advantage of this platform with any device
available to them at any location. This wasintended for students who may not have accessto an equipped device when needed. It alsomeans that users can use the render farm and still
do work on their own device with nointerference, allowing companies to work more
efficiently.

The newest Render Farm was launched in Marchof 2018, which allowed not only CPU rendering,
but also GPU rendering. The throughput is also
greatly improved to support complex and large
scale simulations.
1. BackgroundA lot of movies and TV productions today
involve computer generated imagery (CGI),
because most of the time it is cheaper and easierto create virtual props in the computer than it is
to build them in real life. These visual effects
can be so realistic that the audiences are fooled
into thinking that it is an actual footage filmedwith real-life setups. Not only props andSo far, over 100 films in Taiwan were rendered600

video stream is used to make himself at home.

backgrounds can be made with CGI, but also
actors and creatures. For example, in a lot ofaction films, it is often too dangerous and risky
to ask the actor/actress to do certain moves. In
this case, CGI artists will build a 3D model ofthe actor/actress, and animate it to do the job.

Just like some traditional render farms, Deadline
scheduler manages batch rendering on the GPU
render farm in NCHC, and many authorizedsoftware programs here are ready-to-use,
including animation software (Maya, 3ds Max),ray-tracing renderers (Arnold, V-Ray), visual
effects simulators (RealFlow, Houdini), etc.

Any image generated by means of computer
programs, whether it’s photorealistic or
non-photorealistic, needs to go through theprocess of rendering in order for the details
created to be seen. To render a video means torender a sequence of images each second thevideo plays. The more details an image have,
whether it is geometry, texture, lighting or
shadows, the more render time will be requires.

3. Importance or Significance of Tutorial
Modern film and TV footage runs at 24 frames
per second, a single frame from Monsters
University can take up to 29 hours to render4.
This means that the film, which is 1h44m long,
would have taken about 10,000 years to finish if
it was rendered with one CPU. This explains
why render farms are needed in the film
industry.

Rendering is the way a computer draws animage by calculating the color of each pixel. The
process of rendering, for example, is when an
image contains a pixel that is red, and a filter isadded to make the red pixel blue, the software
will have to recreate this blue pixel when it
exports the finished image2.

Visual simulation, also known as 3D
visualization, are created through computer
computation by taking the input data and run it
through algorithms base on given variables.
Visual simulations are not only used for movies
or animations, but they also allow researchers tomake a more accurate prediction of anexperiment without having to run the test
physically. It can also simulate population,
traffic, or human behaviors5.

When creating CGI that involves visual
simulations, a powerful computing system isrequired to render the images. Simulation is an
imitation of a real-world process generated
through computer calculations. For example, thecomputer can simulate a water crown splash
through computing. Instead of animating fluidby hand, a simulation can be generated withinformation such as the size and the speed of thewater drop. Once all the information is gathered,
the computer can imitate the reaction of a dropof fluid hitting surface from some distanceabove.

Simulations can also be done in real-time. Whencreating any experience that involves interaction,
the rendering process will have to be in real-time,
because the video react base on the action
detected.

Fig. 3. Texture Distortion Method6.

The texture distortion method shown in figure 3
is an example of real-time rendered simulation.
The project was to create a shader for materials
that distorts its UV to create a water effect.

Fig. 2. The way the water drop is predicted throughtime. (a) t = 2.0; (b) t = 3.0; (c) t = 4.0; (d) t = 5.0.3

Thanks to remote virtual desktop, complete
production pipeline, and batch renderer cluster,frequently network transportation of high polymodels, which is one of the bottlenecks of
traditional render farms, is avoided on the GPU
render farm in NCHC. Cooperating is available
as well for a group of users in the cloud studio,with high throughput and IOPS of the storage
system supporting multi-user performance.

2. Concerned Contents
In order to achieve cloud studios on HPC, theGPU render farm in NCHC, mainly based on
VMware Horizon, comprises vSphere ESXi
hypervisor, vCenter server, View Connection
server, View Composer service, etc. Despite thecomplex structure, end users can access remote
desktop via VMware Horizon Client, simply
typing in the IP, username, and password. On
account of limited network discharge for someuser, H.264 encoding, accelerated by GPU, of

4.

601

Possible Outcome

NCHC is ready to inquire more advanced
software and plugin for the render farm.
Provided software/ plugin are used for creating amore realistic visual content. For example, agrooming plugin allows user to create more
realistic fur, and is able to simulate the way it
moves or reacts with objects.
With new tools at hand, users can create a moreadvanced visual for their content. The amount oftime render farm can save in terms of rendering,
provides the opportunity to create a better
product with the same amount of time spent.
5. Conclusion
Simulation is very important to many fields
today. Rendering is especially needed for visual
focused content such as animated films, movies,
or video games. Whether it is rendering forvideos or real-time rendering for interactable
experiences, a high-performance computing
system is required for an efficient workflow.
6.

Reference

1.

Maxim- Eden- RobynedTomaszMSwidzinski-Sareesh Sudhakaran –https://wolfcrow.com/blog/what-is-a-render-farm/
Lin, Shi-Min, Wang, Chih-Wei, and Kuo,
Chia-Chen. 2018. “HPC Rendering for Culture
Recreation – An Example of Koji Pottery in
Taiwan” In Proceedings of SEAIP 2018.
Chang, Chih-Wen, Chiu, Pao-Hsiung, and Kuo,
Chia-Chen. 2014. “Milkcrown Simulation using
OpenACC
Accelerated
Framework.”
In
Proceedings of SIGGRAPH 2014.
Takahashi, Dean. “How Pixar Made Monsters
University, Its Latest Technological Marvel.”
VentureBeat,
24
Apr.
2013,
venturebeat.com/2013/04/24/the-making-of-pixar
s-latest-technological-marvel-monsters-university/.
Lytle, Sam. “What Is a Visual Simulation?”
CivilFX, 26 Jan. 2015,
http://www.civilfx.com/visual-simulation/
Hsiao, Shao-Ting and Kuo, Chia-Chen. 2018.
“Real-time Rendering for New Media Art” In
Proceedings of SEAIP 2018.

2.

3.

4.

5.
6.

602

Virtual Reality as an Art Form
R.P.C. Janaka Rakapakse, Yi-Ping Hung

Tainan National University of the Arts, 66 Daci,
Guantian District, Tainan 72045, Taiwan
E-mail: janakaraja@gmail.com, hung@csie.ntu.edu.tw

Abstract
After introducing content creations in the immersive environments, the VR content creation process has been changed
dramatically. It has enhanced the direct engagements of the artists and reduces the time it takes to get a part from theidea stage to the production stage. Time consuming VR content creation process has been reduced to hours or days.
Not only is the time reduced, but the cost is also significantly reduced. The reduced time and cost allows artist to
work in near real-time, with the design and production happening at nearly the same time. This section of the session
is focused on VR content creation process and its artistic perspectives. Especially, not only concerning VR contents
creation in immersive environment is an alternative to find new form of art. And the last part of the speech concerns
about the role of VR in the live performance and new form of theater.
Keywords: virtual reality, art form, VR contents, immersive, interaction, content creation

Nowadays, VR-application2 are using in many fields.
VR can be used as a training application in the medical
field where doctors can experience surgery on a virtualhuman before performing the real cases. The militaryextensively use VR as a training tool. The VR can aid inphysical therapy for children as well as adults. Recently,
researchers in the U.S. have developed an ingenious
system that uses VR to help prevent falls by detecting andreversing balance impairments in elderly people. In
Entertainment, VR was being incorporated into music
videos for more than a passive experience. Also popular
TV-Star programs incorporated 360-degree VR with
Dancing Stars.
VR gaming is becoming moremainstream now as well. VR can be used as a design
support tool in the fields of architecture and interior
design. It can be used to evaluate the state of a new
design, or it can be used to modify a design with the
feedback of customer in a real-time virtual 3D space. VR
can also be used to sell a product or a building.
However, authors do not think VR should beconstruction software or all your modeling should be inthere. There are methodologies to working in VR related
to software, hardware, and techniques. Therefore, this

1. Introduction
According the Cambridge Advanced Learner’s
Dictionary, the abbreviation of VR for virtual reality: a
set of images and sounds, produced by a computer, thatseem to represent a place or a situation that a person cantake part in virtual experience. Generally, people may
think of VR as an experience on the computer using aheadset. But according to the literature, interactive virtual
experiences have been around for several decades. In
1962, the Sensorama1 introduced a 3D experience that
included motion, color, stereo-sound, aromas, wind,vibrations, etc. The ViewMaster allowed us to view animage in stereo which is the same principle as a today’s
VR headsets as shown Fig. 1.

Fig. 1. Evaluation of VR headsets.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan603

R.P.C. Janaka Rakapakse, Yi-Ping Hung

Fig. 2. VR workflows and optimizations

paper of the tutorial session is focused on VR content
creation process and its artistic perspectives.

•

2. Objectives
One lesson learned early on was that VR can feeloverwhelming. If anyone is going to drop someone in a
virtual experience, it is important to have clear goals andnot let them feel lost in the woods. There is no technologythat can compensate for lackluster content or the absenceof a clear idea or strategy. we think that there’s a lot of
people, across many industries that basically see VR as a
big fad that they need to get a piece of. It is a hard task,
because it only takes one bad experience for user toforever after feel weary of VR. So here, we are very
specifically concerning the VR workflow as shown in fig.
2. And there is a misconception that many have that doing
anything with VR requires a totally separate workflow
from what creators are used to do in content creation. But
the reality is, if creator’s workflow already includes 3Dmodeling, texturing of some kind, lighting of some kind,or rendering, maybe from commodity 3D applications, heor she is on the way. Actually, building for VR is anextension of what creator is already doing. In order toclear all misconceptions, this tutorial concerns following
listed objectives.
•••

affordances of VR: scale, presence, design options,experiencing decisions while creators are in process.
To study how to communicate design intent withrelatively simple VR preproduction, teachingcontents to consider a complex or photo-realistic
content experience.
To learn how to use VR at every stage of the design
process, tutorial contents focuses some open source
content creation applications too.
To understand how to cycle a VR methodology into
workflows, tutorial concerns two major workflows
such as static VR and immersive VR, probably
focuses already using software that has VR
capabilities, and even those that don’t can interface
well with more specialized VR software.

3. VR Workflows
There are different workflows for virtual reality currently
being utilized in the content creation industry. Thistutorial would share a few of them here and discuss how
an artist try to build and incorporate them into theirproduction process.
3.1. Static VR (Panorama or 360 Degree VR)
Panoramas and 360 images are most often displayed onphones, tablets, and untethered head mounted displays(HMD), similar to google cardboard and other Gear VR
as shown in Fig. 3. Based on the devices people choose

To understand the power of designing in VR overtraditional methods, tutorial contents considers the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

604

Virtual Reality as anconcern other options like the Oculus Rift. These HMD’s
offer a much more immersive experience, where almost
anything is possible. As shown in Fig. 4, the headset iscurrently wired to computer, although there are third
party add-ons being released soon allowing the Vive to
become wireless. We chose the Vive because of the room
scale mobility and stable VR headset that came withtracked controllers. Room scale allows the user to walkand explore in a real-time interaction.
To getting started start out the immersive VR project,
there are several application and authoring tools. If you
are creating a VR application, you should be considering
it from the beginning. This is to confirm that most of
creator’s time can be spent creating fun experiences andtroubleshooting, rather than re-modeling other creative
parts. Therefore, this tutorial will be explained following
workflows in step by step.
• Unity4 Workflow• Autodesk Stingray/ Max Interactive6 Workflow• Unreal Workflow7

they have the choice between a simple 2D sphericalimage or 3Dstereoscopic image.

Fig. 3. Static VR workflow

CG (Computer Graphics) generated or captured
panoramic or spherical renderers display cleanly on
mobile devices such as phones and tablets withaccelerometers. This functionality allows the user to holdthe device in the air and experience the space by moving
the device side to side and up and down. People also havethe option to control the display of the device by using
user’s finger on the face of the screen to rotate and scale
the image. As shown in Fig 3. once your render is created,upload it to an app that is built for viewing on yourchosen hardware. The extension of static VR with 3D
stereoscopic, new cinematography of VR movies,
strength and weakness will be described in this section.

Tutorial also focuses on the building contents in
Maya-2018 or Blender Application softwares. This is thebasic workflow for Unity VR builds that we have foundsuccessful. We recommend additional training if you
have never used Unity before. And we have found that
our VR builds do not need to be photo real. For most of
our VR needs a very simple ambient light and a sun setup
works best, we don’t fill our scenes with artificial lighting.
This keeps the project simple and optimized. The
contents don’t usually include reflection probes orcomplex effects. The simpler you keep your project themore geometry you can load into it. We will provide
some model data which can use for initial VR project.
Teleportation vs. Locomotion: Motion sickness is one of
the biggest problems developers have to contend with in
VR. Tutorial will cover teleporting and VR content
navigation techniques. The advantages of teleporting are
obvious, a disadvantage worth noting is the way
teleporting bypasses the passing of time. The participants
can experience our recent project works which useddifferent teleporting techniques with LeapMotion8 sensor
interaction for hand and figures.

Fig. 4. HTC VIVE equipped with VR-Ready Notebook PC(image courtesy: RoadToVR: https://goo.gl/c7zJLc )3.2. Immersive VR
Immersive VR provides fully immersive experiences.
User has ability to interact with the VR environment.
This part of tutorial uses the HTC Vive3, but learners can

4.

Possible Outcomes

The participants of this tutorial primarily get a training
and knowledge about the VR workflows as well as on-© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan605

R.P.C. Janaka Rakapakse, Yi-Ping Hung5. Conclusion

hand experience in the provided VR demonstrations.
Hopefully, shared knowledge and experience helps tochoose a path into VR that works for their future research
directions. After VR is proven valuable there is a richpotential to use VR as a creativity support tool in thecreative industry. By our experience (as shown in Fig.
5.), VR is a presentation tool, design tool, communicationand creativity support tool

The contents we have covered in this tutorial, haveemphasized the major workflows and processes you cantake to optimize the experience and create VR contents
for your research areas. This session encourages you tocontinue to explore new ways to adapt and work withreal-time contents creations for achieving endlesspossibilities. If you approach each aspect with the sameattention and consideration, from the learning of different
workflows and mastering them, what you can accomplishin VR has no limits.
Acknowledgements
The tutorial is supported by the project (E984704):
Research of Advanced Interaction Technologies for
VR/AR and Their Applications to Revitalizing Cultural
Heritage, MOST Grant- in-Aid, Ministry of Technology,
Taiwan.

Fig. 5. VR as a creativity support tool.

References

To set up the true learning experience in this tutorial, we
will go through a few our research examples9,10, and weare trusting that the audience of this tutorial will have theability to connect the dots and apply these methods tomore complex projects in their research carrier. Fig. 6
shows a snap of a group of people taking experiencing
VR content design in our earlier held tutorial session in
Taiwan 2018. Lastly, this tutorial will show you how toturn VR model into a Social VR model, and describe
some case studies11 which create a personal VR
experience in where multiple media integration with live
performances, review the recent projects virtually that
support recall memories and take some selfies all while
immersed in Virtual Reality.

1. Sensorama, https://en.wikipedia.org/wiki/Sensorama
2. A.B. Craig, W. R. Sherman and J. D. Will, Developing
Virtual Reality Applications: Foundations of Effective
Design, 1st edn. (Morgan Kaufmann, 2009)
3. HTC VIVE, https://www.vive.com/eu/
4. Unity, https://unity3d.com/
5. Maya-2018,https://www.autodesk.com/products/maya/overview
6. 3ds max interactive, https://area.autodesk.com/blogs/the3ds-max-blog/get-3ds-max-interactive/
7. Unrealengine,https://www.unrealengine.com/enUS/what-is-unreal-engine-4
8. LeapMotion interaction for the hands, http://leapmotiondeveloper.squarespace.com/
9. J. Lee, R. P. C. J. Rajapakse, Y. Hung and Y. Tokuyama,
The Development of a Virtual Doll Companion for Haptic
Interaction, in Proc. 2018-Nicograph International(NicoInt), (Tainan, Taiwan, 2018), pp. 9292. doi:
10.1109/NICOINT.2018.00038
10. J. Lee, R. P. C. J. Rajapakse, Y. Hung and Y. Tokuyama,
The Development of an Interactive Device for Tactile
Sensation with Haptic Companion Doll(poster), in Proc.
of the 16th ACM SIGGRAPH International Conference
on Virtual-Reality Continuum and its Applications in
Industry (VRCAI2018), (Tokyo, Japan, 2018) Tokyo,
Japan.
11. Draw Me Close: a VR Application for performancehttps://www.drawmeclo.se/

Fig. 6. VR as a creativity support tool.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

606

TS-3 Using Quill as a Tool for Real-Time Rendering
Jasmine Wu

National Center for High-performance Computing, National Applied
Research Laboratories, Hsinchu 300, Taiwan
Chia-Chen Kuo, Shu-Hsin Liu, Chuan-Lin Lai, Chiang-Hsiang Lien, Ming-Jen Wang, Chih-Wei Wang

National Center for High-performance Computing, National Applied
Research Laboratories, Hsinchu 300, Taiwan
1803040@narlabs.org.tw,
www.narlabs.org.tw

Abstract
Real-time rendering are performed through interactive videos such as in video games, Virtual Reality (VR), andAugmented Reality (AR). Oculus Quill can be used as a tool to create content for VR, which can then betransformed into AR using game engines such as Unity. The intuitiveness of Quill allows a greater possibility fordevelopers.
Keywords: render farm, visual simulation, virtual desktop infrastructure, high-performance computinginterference, allowing companies to work more
efficiently.

1. Introduction
We value art and culture as much as the effort
we put into our developments in technology, so
we built a render farm as a tool that allows theentertainment industry in Taiwan to prosper.
With the provided rendering service at National
Center for High-performance Computing(NCHC), related fields are presented with muchbigger possibilities.

So far, over 100 films in Taiwan were renderedusing our platform by students and companies
from the industry. The newest Render Farm waslaunched in March of 2018, which allowed not
only CPU rendering, but also GPU rendering.
The throughput is also greatly improved tosupport complex and large scale simulations.
For having a reputation for global technology
manufacturing and professionals in Taiwan, weconstantly update ourselves with the newest
technological trends. Virtual Reality (VR) is oneof the most talked about topics in thetechnological industry, and it has been utilized inmany fields, but most successfully in film,
animation, and video game.

Render farm is a cluster of networked computerswith a high performance speed and enormous
amount of data storage devoted to rendering
images, typically used for producing animatedfilms. We made render farm accessible throughVirtual Desktop Infrastructure (VDI), allowing
users to take advantage of this platform with any
device available to them at any location. Thiswas intended for students who may not haveaccess to an equipped device when needed. This
also means that users can use the render farmand still do work on their own device with no

2. Background
Rendering is the way a computer draws animage by calculating the color of each pixel. The
process of rendering, for example, is when an607

image contains a pixel that is red, and a filter isadded to make the red pixel blue, the software
will have to recreate this blue pixel when it
exports the finished image1. This means that any
image edited or generated through computer
programs, whether it’s photorealistic or
non-photorealistic, needs to go through theprocess of rendering.
Videos are a sequence of images played in order,
so to render a video means to render a sequenceof images. When it comes to movies oranimations that involves Computer Generated
Imagery (CGI), the more details an image have,
whether it is geometry, texture, lighting or
shadows, the more render time will be requires.

Fig. 1. The interface and produced content of Quill.

3. Concerned Contents
For interactive video such as video games andmost VR experiences, the rendering process
cannot be done ahead of time, because what theuser sees corresponds with the way the user
moves in the scene. Rendering process that isdone as the video plays are called real-timerendering, and this requires a high-performancecomputing system.

To put it in perspective, modern film and TV
footage runs at 24 frames per second, a single
frame from Monsters University can take up to
29 hours to render2. This means that the film,
which is 1h44m long, would have taken about
10,000 years to finish if it was rendered with one
CPU. This explains why render farms are neededin the film industry, because it can shorten thistime to couple of years.

Another example to demonstrate NCHC’s
real-time render farm was by performance art. It
has a high data throughput that can help the
performer to do a large scale VFX simulationand to create new kinds of cross-disciplinary
performances. A project that was done at NCHC
shows the performer dancing with projectedvisual effects, which reacts in response to theperformer. The rate of how the visual effects
plays and reacts shows the performance speed of
the real-time renderer.

VR is like a monitor, but instead of having a still
screen in front of the user, the screen is allaround. The VR devices today typically come inheadsets that cover the eyes to prevent peripheral
vision, cutting out any context that would
disconnect the user from the virtual world. The
user may spin their head, and the camera would
follow, pointing to the direction the user wants tosee in a scene, thus giving the user an immersive
experience.

The render farm computers at NCHC provides
real-time rendering supported by both Unity andUnreal, which are two of the most used gameengines by the public today3. Using the real-timerendering engines on the render farm computerspresent a faster frame rate.

Having a 360 degree video also means that therender needs to cover all around instead of one
direction. When VR content is created in 3D, theuser may not only see an environment in 360,
but also move around in the virtual space. Anyinteractive experience will require real-timerendering.

Render farm may help with the render time, as
well as the speed of software provided by astronger machine which improves the workflow.
However, the production time cannot bedecreased. The steps in the animation production
pipeline before the point of rendering from atypical animation studio includes but not limited
to the following4:
Modeling
SurfacingRigging
Layout/ Set dressing/ Anim Prep
Character Animation
CrowdsCharacter Effects/ Simulation
Effects
Technical Director
Matte Painting
Lighting

Quill is an example of a VR experience. It
provides a creative platform for users toillustrate and animate using the Oculus Rift. It
allows users to paint directly inside a 3
dimensional virtual space. Users can create
animated content by using the Oculus Touch topaint 3D materials, and move them frame byframe to create an animation.

608

necessarily require detailed models, since there
are still limitations to most AR devices available
that restricts files that are too big. Quill is a great
tool for such users to quickly create their own IP
for demonstration purposes.

For big animation studios such as Disney, there
are enough members in the team to distribute theworktime. But, for one creator or a small studio,
it is a lot harder to create a complete animationeven when they have created a great concept toanimate.
Today, there are a few illustration/3D modelingand animation tool for the VR, which canshorten the traditional 3D animation process
done in 3D software. It is a new way ofanimating that is more intuitive, which has aneasier transition to using our real-time renderer.
Quill is one of the more intuitive tools that will
be introduced.

Fig. 3. Quill is also a great tool for creating contents
for smartglasses.

5. Possible Outcome
One specialty about Quill is that there is no limit
to the canvas size; it can be zoomed in and out
endlessly. The only limit to Oculus Quill is thesystem that it is running on. A powerfulcomputer is required to run the Oculus Rift, andits processor and graphics card determines how
smoothly Quill runs on the Oculus Rift.

4. Importance or Significance of Tutorial
Quill is a great tool to create content for Unity or
Unreal, which can then be rendered in real-timeby the super computers at NCHC. Render isnormally the last step to the production pipeline,
and each step is important for the next. For
example, if a 3D model is not rigged properly, it
will not animate properly. This makes it harderfor one person to complete the job without any
help.

NCHC are finding ways to connect VR
equipment to the render farm through virtual
desktop. If succeeded, this opens up a whole
new way of using the render farm, and extends
the possibility of VR creations to the next level.

The traditional 3D animation pipeline would
most likely take years to master each step.
Oculus Quill can skip the steps of modeling,
rigging, texturing, and go directly into animating.
Even though it may not have a final product asrealistic or as detailed as an animation madewith 3D software, but it is a great option formaking quick and simple animations, which canthen be used for other purposes.

6. Conclusion
The central thinking of NCHC is to fill the gap
between technology and art. However, it has
been a hard task due to the limited people whohave knowledge to both sides. Thanks to thedevelopers of Quill, the bridge between
technology and art seem to be found.

At NCHC, animations created with Oculus Quill
are imported into game engines supported by therender farm, which can be built into VR or even
Augmented Reality (AR).

7.

Reference

1.

MaximEdenRobynedTomaszMSwidzinski-Sareesh Sudhakaran –https://wolfcrow.com/blog/what-is-a-render-farm/Takahashi, Dean. “How Pixar Made Monsters
University, Its Latest Technological Marvel.”
VentureBeat, 24 Apr. 2013,venturebeat.com/2013/04/24/the-making-of-pixar
s-latest-technological-marvel-monsters-university/.
Unreal Engine 4 Vs. Unity: Which Game Engine
Is Best For You?
https://www.pluralsight.com/blog/film-games/unr
eal-engine-4-vs-unity-game-engine-best
Willett, Nora, et al. “Animation Production
Pipeline.”
Physbam,
29
Mar.
2011,
physbam.stanford.edu/cs448x/old/Animation_Pr
oduction_Pipeline_(2d)_Nora_Willett.html.

2.

3.

4.

Fig. 2. An example of how content created by Quill
can be applied into VR/AR.

Although VR/AR is beginning to emerge withthe animation industry, most AR developers are
not animators who can create 3D models.
Moreover, a lot of applications of AR do not609

System to decide unit’s layout of cell assembly machine by GA(Big Mutation in mutation process)Yoshiki Ito

Department of Human Information Systems, Gifu University, Yanagido 1-1,
Gifu, Gifu 501-1194, JAPAN
Hidehiko Yamamoto

Department of Mechanical Engineering, Gifu University, Yanagido 1-1
Gifu, Gifu 501-1194, JAPANTakayoshi Yamada

Department of Mechanical Engineering, Gifu University, Yanagido 1-1
Gifu, Gifu 501-1194, JAPANE-mail: w4525008@edu.gifu-u.ac.jp, yam-h@gifu-u.ac.jp, yamat@gifu-u.ac.jp

Abstract
This study develops the system which assists the unit placement decision of the automatic assembling equipment toassemble efficiently. The unit is a part supply or a robot hand constituting assembling equipment. Conventional
research has used reinforcement learning to determine efficient unit placement but this research adopts Genetic
Algorithm (GA) instead of Reinforcement Learning. First, as input data, information such as an arrangeable region ofeach unit is input, and the input region is divided into fixed sizes. Next, the unit is placed in the divided region.
Prepare a group of individuals whose genes are the placement of each unit, and repeat selection, crossover, mutation,and evaluation to improve placement. This system introduces the concept of Big Mutation that changes the mutation
rate of GA every certain generation. As a result, the possibility of searching for an optimal unit placement increased,
and it is expected that the system simulation time is shortened.
Keywords: Unit placement decision, Big Mutation, Double arm robot, Cell assembly machine,Genetic
algorithm

system that supports unit layout determinations ofefficient cell type assembly machine is developed byusing genetic algorithm (GA).

1. Introduction
In recent years, it has been demanded to produce alarge quantity of products in a short time at the
production site. So, automation of the assembling workby the assembling apparatus has been carried out. Whendeveloping an automatic
assembling machine,
production efficiency changes depending on thearrangement of each unit. However, in many factories,
the placement of units is based on the experienced
engineers It remains questionable whether thedetermined placement by the experienced engineers isreally efficient. In this research, the layout decision

Fig. 1. Cell type assembly machine© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan610

2. Cell type assembly machine

3.1 Condition creation module
The procedure of the condition creation module is asfollows.
StepⅠ: Read work data of parts assembly, placement
possible area, parts arrangement data andcreate their database.
StepⅡ: Divide possible placement area into a lattice
shape to create possible placement place.
StepⅢ: Carry out genes coding.
StepⅣ: Define the fitness function.
StepⅤ: Determine various parameters.

As shown in Fig. 1, the cell type assembly machine to
be used in this study has a configuration in which an
assembly machine is arranged at the center, a work
stand at the front and assembly units on the periphery.
Therefore, after each arm chucks the needed unit, thearm carries it to the workbench to assemble the parts. In
this research, approach 1, approach 2 and the chuckposition of the robot arm movement route are defined as
the three points that exist on the arm moving route as
follows
・Chuck position ... position to position and grip unit
・Approach 1... position upward away from thechuck position
・Approach 2... position further upward than
Approach 1

3.2 GA module
The process flowchart of the GA module is shown in
Fig. 3. The process procedure is shown below.

As shown in Fig. 2, the arm passes from approach 2
to approach 1 and arrives at the chuck position.

Fig. 2. Movement of arm
Fig. 3. Flowchart of GA module

3. Layout determination system

Step 1: Generate an initial population.
Step 2: Calculate the placement coordinates of each unit.
Step 3: Calculate the time taken for each work.
Step 4: Calculate fitness.
Step 5: Apply genetic operations (selection, crossover,mutation).
Step 6: Judge whether the termination condition issatisfied. If it is, go to Step 8. If not ,go to step3.
Step 7: The unit arrangement of individuals with thesmallest operation time is adopted as the
optimum arrangement and the output as asolution.

The layout determination system we develop decides
the arrangement of each unit with the best operation
efficiency during designing a cell type part assembly
machine by using GA. The system consists of two
modules, the condition creation module and the GA
module. The condition creation module performs to readthe arrangement possible area of the unit and work
contents, to divide the arrangement possible area and to
decide various necessary GA parameters. The GA
module determines the arrangeable place of the units,
evaluates the working time and outputs coordinates of
the arrangement place of the unit and the layout imageof the unit.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

611

3.3 Big mutation in mutation process
In general, the phenomenon of local solution
convergence occurs when a considerable number ofgenerations in GA elapse. Therefore, we call Big
mutation that only some superior individuals are carriedover to the next generation as they are and a certain
generation is given a high mutation rate. Because of Big
mutation, it is expected to escape local solution
convergence and to have a high possibility easy toconverge to an excellent solution. Also, the simulation
time can be shortened. However, if the mutation rate isincreased too much, it is difficult to converge thesolution and it is not possible to expect the effect.
Therefore, it is necessary to carry out the trial and error
decisions of the Big mutation rate according to work
contents and GA parameters. The setting method of the
Big mutation rate is shown below.

・ The system that determines unit layout by
reinforcement learning
・ The system that determines unit layout by using GA
without Big mutation
The units that determine the placement has eight lefttray, right part tray, left parallel hand, left three nailhand, right parallel hand, right four nail hand, jig A, jig
B and also the assembly work. The layout map is shownin Fig. 5. Fig. 6 is the left and right part trays with 52
parts. Each number represents each part.

Step i :Set the mutation rate as k% which is generally
adoptedStep ii :Set the interval m which occurs Big mutation
between generation to generation
Step iii :Set the Big mutation rate as j% which occurs
Big mutation.
Step ⅳ :Carry out Big mutation
3.2 Step 5 of the GA module into the process ofmutation in genetic manipulation incorporate Big
mutation.
Fig. 5. Units layout

Fig. 4 shows the Big mutation outline. The horizontal
axis of the Fig. 4 is the number of generations, thevertical axis is the mutation rate

Big Mutation

Big Mutationm

Big Mutationmjk
Fig. 6. Parts trays

Fig.4 Situation of Big Mutation

4.

Next, Table 1 and Table 2 show the position data andplaceable area data of the dual arm robot.

Application examples of system andperformance comparisons

Table 1. Coordinates of double arm robot

The layout decision system developed in the research
was applied to the assembly machine and production
simulations were carried out. As a comparison,
simulations of the following two systems were carriedout.

Dual arm Robot

x[mm]

y[mm]

z[mm]

2,210

915

740

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

612

Table 2. Placement area of each unit
Unit

x1[mm]

y1[mm]

z1[mm]x2[mm]

y2[mm]

z2[mm]

1,811.5

1,760

740

1,918.5

1,804.5

740

2,522

1,806.5

740

2,628

1,807.5

740

1,280

1,480

629

2,320

1,480

629

1,280

1,480

629

2,320

1,480

629

2,100

1,480

629

3,140

1,480

629

2,100

1,480

629

3,140

1,480

629

Jig A

3,034

83

740

3,656

1,117

740

Jig B

3,034

83

740

3,656

1,117

740

Left
tray
Right
tray
Left
parallel
hand
Left
three
claw
hand
Right
parallel
hand
Right
four
claw
hand

4.1

Table.4 Comparisons with GA system not using Big

mutation
Two pointcrossoverTwo point crossover(Big Mutation)

237.251

237.147237.262237.168

31.08828.0720.015394

0.013559

Minimum
cycle time[s]Averagecycle time[s]Averagesimulationtime[s]Standard
deviation of
Cycle time

Fig. 7 shows the output diagram of the unit
arrangement with the highest efficiency in two-pointcrossover using Big mutation.

Simulation results

Simulations were carried out ten times in each of thethree systems. The results of the simulations aredescribed below.
・Minimum cycle time：
The smallest work time in each simulation
・Average cycle time：
Average work time of simulation for 10 times
・Average simulation time：
Average time taken for 10 simulations
・Standard deviation ：Indicator indicating variation of the data
From Table 3, in all simulation results, the values ofthe systems using GA were calculated systems using
GA beter than that of reinforcement learning. Table 4shows
reduced simulation time and stability ofsimulation results by using Big mutations.

Fig.7 Best Layout of Two point crossover(Big Mutation)

5.

The layout determination system developed in thisresearch is the system aiming at improvement of design
and production efficiency by automating the unit
arrangement determination of a cell type assembly
machine by GA.
As a result of comparing the performance of thesystem using reinforcement learning and the GA system
without Big mutation and the layout decision system,
the layout determination system was able to obtain amore efficient unit arrangement than the other two
systems. Therefore, it was found that the layout
determination system is useful for determining thearrangement of units in a cell type assembly machine.

Table.3 Comparisons with reinforcement learning

Minimum
cycle time[s]Averagecycle time[s]Averagesimulation time[s]Standard deviationof Cycle time

Reinforcement

Two point

learning

crossover237.287

237.147237.332

237.168495.32728.0720.027449

0.013559

Conclusions© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan613

Autonomous decentralized FMS that adopts
priority order structure based on AGV’s lie
Kento Uejo

Department of Mechanical Engineering Gifu University, Yanagido 1-1
1-1,Yanagido,Gifu, 501-1194, JAPAN
Hidehiko Yamamoto*1

Department of Mechanical Engineering, Gifu University, Yanagido 1-1
1－1，Yanagido, Gifu 501-1194, JAPANTakayoshi Yamada*2

Department of Mechanical Engineering, Gifu University, Yanagido 1-1
1－1，Yanagido, Gifu 501-1194, JAPANE-mail: *1; yam-h@gifu-u.ac.jp *2; yamat@gifu-u.ac.jp

Abstract
We introduce the model of AGV mind to eliminate path interferences between AGVs when running an autonomous
decentralized FMS and have carry out behavior controls. The conventional autonomous decentralized FMS believes
all information sent by agents as correct. However, the information sent by some agents may be incorrect due tomalfunctions. We define this incorrect information as a “lie” and carry out the AGVs action controls which doesn’t
decrease the outputs of autonomous decentralized FMS even if lies occur.
Keywords: Autonomous decentralized FMS, AGV, Priority order structureorder structure” and the AGV whose priority order
structure belongs to a higher position has the higher
possibility to be given a job.
We carry out the production simulations of theautonomous decentralized FMS with the proposed
method and get the productivity to prove the goodness ofthe proposed method. As a comparison, we also calculate
the productivity by using AGVs that adopted the“reliability structure” which is the past research that ourlab developed and verify the effectiveness of this
proposal method.

1. Introduction
We have researched an autonomous decentralized
flexible manufacturing systems(FMS) factory [1]. Here,when operating an autonomous decentralized FMS, we
researched avoiding route interferences between AGVs.
Especially, we controlled AGVs by inserting two modelsof mind, an arrogant mind and a modest mind, into
AGV[1].
In this research, all information transmitted by agentsis treated as true. In the actual operations of anautonomous decentralized FMS, it is the possible
situation that incorrect information is sent due to machine
breakdowns. In order to solve this problem, we define
this incorrect information as a “lie” of AGV. We study
how to control the AGVs not to decrease the productivity
in spite that AGVs sometimes lies
The control method is to give an order to each AGV
according to the number of lies. We propose the “priority

2. Mechanism of autonomous decentralized FMS
2.1. Description of autonomous decentralized
Fig.1 shows the example of an autonomous
decentralized FMS factory. The factory floor is divided
into a grid pattern where an AGV moves on the line. FMS© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

614

Kento Uejo, Hidehiko Yamamoto, Takayoshi Yamada

manufactures the products by carrying the parts from thewarehouse to machining centers (MCs).
An autonomous decentralized FMS does not have amanagement mechanism that controls the whole system
between agents that constitute the system to theknowledge exchange, to determine the behavior bycoordination.

We describe the internal functions of MUM. When thearrogant AGV has a path interference, A1 is increased by
1. Keeping the situation of the interference, the A1 value
repeats to be increased by 1. When the A1 value finallybecomes the threshold value, the A1 becomes satisfied
and the AGV is changed to a modest mind. As humble
AGVs give way each other and continue to give way, we
increase the A2 satisfaction by one. When modest AGVs
try to make way for each other and A2 becomes satisfied,
MUM sends a signal to the load. The load that received
the signal will change the units A1 and A2 from satisfied
to normal. As a result, AGV will become an arrogant
AGV from a modest AGV.
In this way, a mind switching between arrogant mind
and modest mind avoids path interferences between
AGVs of autonomous decentralized FMS.

Fig.1 Example of autonomous decentralized FMS

2.2. Model of mind and behavior of AGV
As multiple AGVs in the autonomous decentralized
FMS autonomously determine the behavior, this research
gives the AGV a mind and realize the AGVs moving
control by a mind. That is, some AGVs have arrogant
minds and others have modest minds to move.
AGV with an arrogant mind moves forcibly to thedestination. AGV with a modest mind gives way to other
AGVs.
We express the mind as the mind model of Fig.2,
Minimum Unit of Mind (MUM). A1 and A2 of Fig.2 are
the units, X is the load and the arrow is a stimulation
vector. If the internal value of the unit reaches thethreshold, we call it "satisfied". And if it does not reachthe threshold, we call it "normal".
When the signal is sent to the unit, MUM sends thesignal to the direction of the arrow if it is satisfied, and
MUM does not send if it is normal.
The load has the function to change the internal valueof the unit. When the signal is sent to the load, the valueof the unit is increased by the value of X.
The stimulation vector is a line connecting the load
and the unit and a signal is sent to the load or unit when
the signal comes. When A1 is kept satisfied, we call it"modest mind" and when normal, we call it “arrogantmind”. Also, by the amount of the signal input to the unit,
MUM changes the mind frequently, arrogant or modest.

3.

AGV’s lie and its reliability structure

We define the AGV as “liar” when the AGV sends anincorrect information and as “honest” when the AGV
sends a correct information. We also define the lies as thefollowing three kinds of lies that the AGV has
1. Sending incorrect location information(USO1).
2. Reading incorrect location information(USO2).
3. Do not send location information(USO3).
The reliability structure which is our past study[2]determines which AGV can be trusted by counting thenumber of lies and is the hierarchical structure including
the hierarchical relationship in the AGV group. Thisreliability structure gives the job to the AGVs with highhierarchies. The AGVs with lower hierarchies takeactions to give way to other AGVs.

Fig.3 Example of reliability structure AGV
4.

4.1

Priority order structure

Concept of priority order structure

When we carry out jobs at a company or a school, theefficiency of carrying out jobs is needed. In order to carryFig.2 Model of Mind

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan615

Autonomous decentralized FMS thatout the high efficiency jobs, assigning a priority to works
and choosing the high priority works are one of the ideas.
We propose the priority order structure(POS) to “prioritize
the AGVs based on the number of lies and construct POS”.
The POS gives a move to the AGV that does not lie
and AGV which often lies makes a movement not todisturb other AGVs actions. In spite there are AGVs
that lie, it is thought that a production can be smoothly
carried out without decreasing production efficiency.
4.2

knowledge of other AGVs that interfere with the myself
route are compared. If the order is higher, the AGV
becomes arrogant mind and if the order is lower, theinternal value of the unit of MUM is changed in order tobecome a modest mind. This is the mind change and thechange can shorten the time for avoiding route interference.
4.3.1

Case 1:Two AGVs interference

When the AGV1 and the AGV2 interfere with eachother as shown in Fig.6, the order in myself knowledgeof AGV1 and the order of AGV2 in myself knowledge ofother AGVs are compared. Finding AGV1 order is high,
AGV1 keeps the arrogance and does not change the mind.
As AGV2 compares the order in knowledge of myself
with the order of AGV1 in knowledge of other AGVs andfinds the order is lower than AGV1, AGV2 turns into amodest mind, gives way and avoids a route interference.

Formation of POS

In order to make POS in the AGVs, as shown in Fig.4,
every AGV adds the information on the number of times
that AGV lied and the order of AGV.

Fig.4 AGV Knowledge
The following algorithm shows how to make POS.
Step 1. At the end of production, information about the number of lies described in AGV knowledge is transmitted to all other AGVs.
Step 2. Save the number of lies received from other
AGVs in the AGV knowledge of other AGVs.
Step 3. Arrange from a small number of AGV lies ofmyself and other AGVs.
Step 4. Form POS according to the above arrangement.
Step 5. Check the order of each AGV from the formed
priority structure and save it as a knowledge of myself
and a knowledge of other AGVs.
In this way, if each AGV performs Step1 to Step5, theorder of each AGV is generated as the order shown in Fig.4.
As a result, the POS as shown in Fig.5 is constructed.

Fig.6 Example of path interference
4.3.2

Case 2:Interference of the same order AGV

When AGV1 and AGV2 that have the same orderinterfere each other as shown in Fig.7 below, their orderin the knowledge of myself of AGV1 is compared withthe order of AGV2 in the knowledge of other AGVs.
Because it is the same, the mid of arrogance is kept.
AGV2 also compares in the same way and the arrogant
mind of AGV2 is kept because the order is the same. In
this case, using MUM shown in Section 2.2, routeinterference by mind change is avoided.

Fig.5 Example of priority order structure AGV
4.3

Behavior control using POS

When the AGV group that formed the POS interfereswith the path, the order in the knowledge of myself and theFig.7 Example of the same order© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

616

Kento Uejo, Hidehiko Yamamoto, Takayoshi Yamada

4.3.3

Table 1 Probability of occurrence of lies

Case 3:Multiple AGVs interference

When multiple AGVs (n) interfere with each other asshown in Fig.8, first the AGV1 compares the order inknowledge of myself with the order of all AGVs thatinterfere. Finding that the order is the highest, thearrogant mind of AGV1 is kept. Secondly, the AGV2
compares the order in knowledge of myself with theorder of all AGVs that interfere. Finding that there is an
AGV with a higher order than my order, the modest mind
is changed. In this way, every AGV that interferes witheach other compares orders as described above. If there
is no higher AGVs, the arrogant mind is kept. If there isa higher AGV, the modest mind is changed. This action
is performed n times and only AGV of an arrogant mind
goes forward. Other AGVs of modest mind takes action
to give way. As AGV1 with the highest order passes, (n1)AGVs with modest minds will interfere, similarly, (n1)AGVs compare the order each other. The AGV mind
with the highest order among (n-1)AGVs is changed toarrogant. This mind change is repeated until aninterference is disappeared.

AGV1
AGV2
AGV3
AGV4
AGV5
AGV6
AGV7
AGV8
AGV9

USO1
0
0
0
1/20
1/2
0
0
1/200
1/2

USO2
1/2
0
1/20
0
0
0
1/200
0
0

USO3
1/2
0
1/20
0
1/200
0
1/200
0
0

The Table 2 shows the production simulation outputsof FMS with reliability structure and POS.
Table 2 Simulation results of production outputs

Production
outputs(20 days)

FMS
reliability
structure
4,397

FMS priority
order structure
4,440

From the simulation outputs results, it is found that theoutputs of POS is more than that of the reliability
structure. The difference was 43. It is ascertained that
POS is useful.
6. Conclusions
In order to prevent the production output decrease of
FMS which lies occurs. We proposed POS. After the
production simulations, it is found POS is useful.
Reference[1] Hidehiko YAMAMOTO, Takayoshi YAMADA and
Shinsuke KATO, AGV Mind Model and its Usage forAutonomous Decentralized FMS by Change of Mind,
Proceeding of Third KES International Symposium, KESAMSTA 2009, Agent and Multi- Agent Systems : Technologies
and Applications, Lecture Notes in Artificial Intelligence,
Edited by Anne Hakansson, Ngoc Thanh Nguyen etc. , ISSN
0302-9743, Uppsala, Sweden, pp.744-753, June(2009),
Springer.

Fig.8 Example of multiple AGVs
5. Simulation and results
We carried out production simulations of autonomous
decentralized FMS with POS and the reliability structure.
The reliability structure adopted the model in which thenumber of hierarchies is three, the number of hierarchies
1 is one, the number of hierarchies 2 is three, and thenumber of hierarchies 3 is five.
In this production simulation, the number of AGVs
was 9, the number of MCs was the 24 MCs have 3 typeof the same MC. The operating time was 8 hours a day,and carried out 20 days.
The probability that AGV lies shows in Table 1. Thisis the probability that each lie occurs when the AGV
passes through the intersection.

[2] Moving Robots Lies and Their Minds with Degree ofConfidence in a Decentralized Autonomous FMS, Shizuka
Tanaka, Hidehiko Yamamoto and Takayoshi Yamada,
Proceedings of The 2016 International Conference© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

617

CULET: Cubelet Lego Turing Machine
Ricardo Q. Figueroa

Artificial Life Robotics Lab, Escuela Superior de Cómputo, Instituto Politécnico Nacional,
MéxicoDaniel A. Zamorano

Artificial Life Robotics Lab, Escuela Superior de Cómputo, Instituto Politécnico Nacional,
MéxicoGenaro J. Martínez

Artificial Life Robotics Lab, Escuela Superior de Cómputo, Instituto Politécnico Nacional,
México. Unconventional Computing Laboratory, University of the West of England,
Bristol, United Kingdom
Andrew Adamatzky

Unconventional Computing Laboratory, University of the West of England,
Bristol, United Kingdom
E-mail: qf7.ricardo@gmail.com, daz23ayala@gmail.com, genaro.martinez@uwe.ac.uk, andrew.adamatzky@uwe.ac.uk

Abstract
In this paper, we show the construction of CULET: a robot that simulates the execution of any 2-symbols Turingmachine. CULET is constructed with Cubelets (small autonomous robot-cubes used for teaching basic robotics andprogramming for kids) and Lego bricks. Cubelets are a kind of high level programming where you do not use
traditional code but rather achieves different robots trough the concatenation of blocks, i.e. robots assembled. CULET
is designed to performance the read and write of a Turing machine.
Keywords: Turing machine, Robotics, Cubelets, Lego, Cellular Automata.

machine: machines simulating machines. Cubelets,
helped with pieces Lego, proof how a robotic machine isable to simulate an abstract computable machine. The
brain of this robot is concentrated in a cube which canread and other two cubes that move the head and the arm(write). Of course, these cubes need be re-programed foreach table of transitions.
For the implementations we programming two special
Turing machines. The first one is a universal Turingmachine able to simulate the behaviour of a complex
elementary universal cellular automaton known as Rule1103. The second Turing machine programmed in
CULET doubles the number of ones in the tape.

1. Introduction
In the history of Turing machines an ample number ofphysical mechanical devices have been developed.
Indeed, during the Turing's 100th birthday, Lego built anautonomous mechanical Turing machine adding binary
numbers in 2012 year1. So we, inspired in the idea ofmachines self-reproducing from cellular automata
theory2, construct the robot CULET, a robot based insmall robots-cube Cubelets and some pieces Lego. The
main feature of this proposal is that it is the first machine
constructed with robots and bricks. This way, this robotic
machine is able to simulate the function of other© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

618

Ricardo Q. Figueroa, Daniel A. Zamorano Genaro J. Martínez, Andrew Adamatzky

The paper has the next structure. In Section 2 weintroduce quickly the Cubelet robots and describe theTuring machines implemented. Section 3 describes thedesign (hardware and software) and finally in Section 4
we give some final remarks.

squares. The machine move in the tape and the head readthe value in each square sequentially7. Formally, theTuring machine is a set of quintuples of the form:

2. Preliminaries

Where q is the set of states, s is the set of symbols andd indicates the direction of movement of the head in thetape (right or left); i is the time step. So, the first twoelements of the quintuple indicate the current state andsymbol, the next two elements indicate the following
state and symbol, and the last one indicates the directionof movementa.
One of the greatest achievements of Turing’s work is
the conceptualization of the so called universal machines.
A universal machine U can do everything that any
machine can do. U receives as input two things, a
description of a machine A, and the input for that machine
M; and, as output U puts on the tape the same thing thatthe machine A would put when have as input M.

( ,2.1 The Cubelets
The Cubelets are small cubes designed to teachfundaments of robotics and programming to kids. Eachcube has a different functionality; the general idea is toconstruct robots only through the concatenation ofdifferent cubes, this way Cubelets can be represented asa formal language, where every valid construction is avalid expression in the language, thus we can see anunlimited number of possible robots. This can be seen asa high level of programming: the child does not modify
the original program of each cube, but instead, use thisprogram together with the interaction among cubes for
produce a new robot. The different cubes are classified inthree categories4:
Input sensors: cubes used for obtain data from theenvironment. The most common are the distance andlight sensors.
Actuators: cubes used for make movements, e.g.
drives or rotators.
Thinkers: cubes for make computations. Some ofthem are inverters, maximum, minimum, etc.
A single cube communicates with his neighborhoods
through a shared channel that is established in the
concatenation of blocks. The input sensors publish hisreading thought the channel; the thinkers obtain thesereadings, make his computation, and send theinformation to the actuators.
Another, less popular, approach is to reprogram the
cubes for obtain different behaviors. This reprogrammingis done in C or Scratch-like code. The CULET, which is
described in the next section, follows this approach. In
Ref 5 is described the API for programming the cubes.

,

,

, )

(1)

2.3 Elementary cellular automata
Cellular automata were invented by von Neumann in Ref.
2 as, ironically, a mean for study decentralized systems
and been able of design non-von-Neumann-stylecomputers. The cellular automata consist of a grid ofsquare cells, each of which is a finite-state automaton (thesame rules for all the cells simultaneously) and where thestate of each cell in the next generation depends on his
current close neighborhood. Von Neumann wasdeveloped a very complex automaton with 29 states
evolving in two dimensions.
Another important event in cellular automata history is
the popularization of Conway’s Game of Life in the 70’s9.
This is a cellular automaton with only two states and aneighborhood of eight cells (Moore neighbourhood). The
popularity of this automata comes given his very
complex and unpredictable behavior from simple rules.
Conway demonstrated the universality of this automata
constructing a register machine in Ref. 10.
In the 80's years, Wolfram invented the elementary
cellular automata as an effort for understanding the
dynamics of cellular automata11. This is an extremely
simplified version of the previous studies: the space isunidimensional (a row of cells) and the neighbourhood
consist exclusively on the two closest cells of each.

2.2 Universal Turing machines
The Turing machines was invented by Turing in 19366 tosolve problems about completeness and decidability. A
Turing machine is a finite-state machine associated withan external memory medium, an infinite tape divided inaSome other standard literature, like Ref. 8, defines the machine interms of quadruples instead of quintuples.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

619

CULET: Cubelet Lego

Wolfram hypothesis was that if we cannot understandsuch minimal models, then we could never understandbigger ones. The interesting part comes when even in
those small automatons are still, in some of them, very
complex behavior.

2.5 A duplicator Turing machine
A duplicator Turingmachine was used byRendell as part of theTuringmachineconstructed in theGame of Life cellular
automaton, for details
see Ref. 17. We
implement
thisFigure 1. Duplicator machine.
machine in CULET. In
Fig. 1, we show a 3-symbols version: it starts advancing
to the right, for each one scanned, it put another one in
the left until there is no more ones to the right. For the
implementation in the CULET we used a 2-symbols
version of this machine.

2.4 Universal ECA: rule 110
In Ref. 12 Cook proofed that the elementary cellular
automaton Rule 110 is universal. The proof consists inshowing how, with particle interactions13 typical of therule, emulate the operation of a cyclic tag system. This
one is the smallest cellular automaton proofed
computationally
universal
simulatingavery
sophisticated sequential machine by particle collisions in
one dimension. This way, a new machine was invented
to simulate a computable function known as cyclic tag
system in this automaton, for details see Ref. 12 and 14.
This way, Rule 110 can simulate a cyclic tag system12,
cyclic tag systems can simulate tag systems12,14, tagsystems can simulate Turing machines7, and a Turingmachine (developed by Eppstein in Ref. 12) can simulate
Rule 110 and therefore such a Turing machine isuniversal. Finally, CULET can simulate the universal
Turing machine which simulate the behavior of Rule 110,complex behavior, and collisions of particles15,16.

3. CULET Design
3.1 General description
In the Fig. 2.1 we show a snapshot of CULET. The tapeis represented as an array of Lego blocks over rectanglesof black paper; the position of the block (front or back)indicates the symbol on the cell (zero or one respectively).
The Cubelet car moves through the tape reading andwriting in the tape; the read process is achieved with adistance sensor that test the position of the Lego block
under scanning; the writing is done with a lever of Legoand a rotate cube: depending on the desired symbol, theleaver push or pull the block of the tape. The movement
across the tape is done with a couple of wheels that movein right or left direction. In the backwards of the car is
attached a rail that helps to maintain the alienation of the
car with respect to the tape.

Table 1. 7x2 Rule 110 Turing Machine.
Sx0
S01
S11
SL
Tx0
T01
T11

0
(0, Tx0, R)(1, Tx0, R)(1, Tx0, R)(0, Tx0, L)(1, Sx0, R)(1, S01, R)(1, S11, R)

1(1, T01, R)(1, T11, R)(0, T11, R)(1, Tx0, L)(0, SL, L)
X
X

3.2 Hardware

The Table 1 shows the transition table of the machine
implemented. Every generation in which the head of themachine reaches a right end, a new generation of the rule
110 is computed. There are some considerations withrespect to this machine: the initial condition and therepresentation of the cellular automata symbols. The
initial condition of the machine must be, from left to right,
an infinite string of zeros, the encoded version of theinitial that the machine is going to emulate, and an
infinite repetition of the string “10”. A “1” of the cellular
automata is represented like “11” in the machine, and a“0” like “00”; the symbol “10” is used like a blanksymbol.

In the Fig 2.2 we show a picture of the cubes needed by
the CULET: a couple of batteries, a couple of drives, two
distance sensors, one rotator, one passive and one light.
In figure 2.3 are shown the Lego components needed(from left to right): a rail of 4 blocks width; a stabilizercar (it must go over the rail); a leaver; the symbols of the

Figure 2. (1) The CULET; (2) and (3) Cubelets and Lego.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

620

Ricardo Q. Figueroa, Daniel A. Zamorano Genaro J. Martínez, Andrew Adamatzky

cells.

more than two states.
References

3.3 Software

1. Lego Turing Machine.
https://www.ecalpemos.nl/filmmaking/lego-turingmachine/, consulted in 04/10/2018.
2. J. von Neumann, Theory of Self-Reproducing Automata,edited and completed by A.W. Burks (University of
Illinois Press, London, 1966).
3. Rule 110 repository.
http://uncomp.uwe.ac.uk/genaro/Rule110.html,
Consulted in 05/10/2018.
4. The Cubelets. https://www.modrobotics.com/cubelets/,
Consulted in 3/10/2018.
5. Cubelets API. http://api.modrobotics.com/cubelets/,
Consulted in 3/10/2018.
6. A.M. Turing, On Computable Numbers with an
Application to the Entscheidungsproblem, in Proc. of the
London Mathematical Society 2(1) (1936), pp. 230-265.
7. M. Minsky, Computation: finite and infinite machines(Prentice-Hall, 1967).
8. M. Davis, Computability and Unsolvability (McGrawHill, 1958).
9. M. Gardner, Mathematical Games: The Fantastic
Combinations of John H. Conway’s New Solitaire Game
‘Life’, in Scientific American (224) (1970), pp. 120-123.
10.E.R. Berlekamp, J.H. Conway, R.K. Guy, Winning Waysfor Your Mathematical Plays, 2nd ed. (Academic Press,
London, 1982).
11.S. Wolfram, Cellular Automata and Complexity (AddisonWesley Publishing Company, 1994)
12.M. Cook, Universality in Elementary Cellular Automata
in Complex Systems 15(1) (2004), pp. 1-40.
13.A. Adamatzky, Collision-Based Computing (Springer,
2002).
14.S. Wolfram, A New Kind of Science (Wolfram Media,
Champaign Illinois, 2002).
15.G.J. Martínez, J.C.S.T. Mora, H. Zenil, Computation andUniversality: Class IV versus Class III Cellular Automata7(5-6) (2013) pp. 393-430.
16.G.J. Martínez, A. Adamatzky, H.V. McIntosh, A
Computation in a Cellular Automaton Collider Rule 110,In Advances in Unconventional Computing: Volume I
Theory, edited by A. Adamatzky (Springer, 2017), pp.
391-428.
17.P. Rendell, Turing Machine Universality of the Game ofLife (Springer, 2016).
18.CULET: Cubelets Lego Universal Turing Machine ECA
Rule 110, https://youtu.be/GIQDA5Gnxkc, Consulted in
05/10/2018.
19.CULET: Cubelets Lego Doubler Turing Machine,
https://youtu.be/CrUrKtIw34w, Consulted in 05/10/2018.
20.Code developed for CULET,
http://www.comunidad.escom.ipn.mx/ALIROB/CULET ,Consulted in 03/12/2018.

As we point out in the previous section, we reprogramsome of the cubes for obtain the desired behavior. The
central cube, the one that has encoded the production
rules of the Turing machine, is the rotator. In the state
diagram of the Fig. 3 we show the stages of the programof this cube. The scan operation is achieving reading thevalue of the sensor that is pointing to the cell; the othersensor, the one that is pointing up, control the movementof the leaver: it stablishes when stop moving the rotator
motor; both sensors have the original program. The othertwo cubes that have custom programs are the motors;
they receive instructions from the rotator: if they receivea 1, they move to the right; if they receive a 2, they moveto the left; a 0 indicates stop moving.
The car starts with a cell below the reading sensor; itscans the position of the block and determine the next
action: write a one, write a zero, or doesn’t write
anything; then, it moves left or right and start over again.
The decisions of this automaton (the symbols to write,
the directions to move) are determined accordingly to theautomaton of the machine that is being executed; thus, inany moment, there are two descriptions of the program
working at different levels: the low level one, that
determines the behavior of the hardware components(move left, move right, scan, write); and the high levelone, which is given by the rules of the Turing Machine.
4. Final Remarks
We develop a robotics Turing machine named CULET,
able to simulate any 2-symbols Turing machine. CULET
is constructed with autonomous cube robots Cubelets andLego bricks. A video showing the function of the Turingmachine simulating Rule 110 with CULET is available
in Ref. 18 and the duplicator Turing machine in Ref 19.
The full code to reprogram Cubelets is available from Ref.
20. In the future work, we will develop machines reading

Figure 3. State diagram of the main program.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan621

Automated Guided Vehicle System Analysis in Foam Manufacturing Plant Using Petri-net
Amornphun Phunopas*, Wisanu Jitviriya, Noppadol Pudchuen and, Sutee Kumjaikong

Faculty of Engineering, King Mongkut’s University of Technology North Bangkok, 1518 Pracharat 1 Road,
Bangsue, Bangkok 10800, Thailand†
Songklod Tunsiri

Urban Community Development College, Navamindradhiraj University, Wachira Phayaban,
Khao Road, Dusit, Bangkok 10300, Thailand
Eiji Hayashi

Computer Science and Systems Engineering, Kyushu Institute of Technology, 680-4 Kawazu
Iizuka, Fukuoka 820-0067, Japan
E-mail: amornphun.p@eng.kmutnb.ac.th, haya@mse.kyutech.ac.th
www.kmutnb.ac.th

Abstract
The logistic robot is very popular nowadays. The robots run in autonomous, reactive, and parallel situations. These
kinds of work are defined as a stochastic problem. We focus on task distribution, routing, scheduling, and queueing.
This paper presents a multi AGVs operational analysis in the foam manufacturing plant using the Petri-net. Eachrobot is designed a role-assignment supervisor with the role of a robot fixed in a zone. According to the work capacity,
the supervisor assigns the role of high capacity work to another robot to help in a new zone. To monitor the work
situation, the supervisor continually receives feedback information of the system. The analysis is necessary forpreventing any failure that can make the central software go down. The results show system management for robotic
assistants in the foam manufacturing process.
Keywords: AVG, EPS manufacturing, Central Software, Petri-net

obtain the highest productivity. Commonly, theoperational area is static; therefore, the path can bemarked with permanent methods such as magnetic guidetape, inductive loop optical marks, etc. The advantage ofthese methods is fast execution, but the disadvantage isthe fixed path, which is difficult to modify. Path planning
is a vital role for AGV. It is vital to find the optimal pathfrom the starting point to the goal point in the workspacearea based on optimal criteria such as minimum time,shortest distance, collision avoidance with other AGVs
(multi-robot) and obstacles, etc. [4]-[6]. In this paper, an

1. Introduction
Nowadays, the Automated Guide Vehicle (AGV) is
becoming popular and important in manufacturing
applications, and it is one of the key machines forautomated logistic transportation systems [1]-[2]. The
AGV operates without any human intervention [3].
AGVs are dynamic and synchronized to other systems
throughout the entire process. There is a central system
to link and control every module. Meanwhile, the central
system has to respond to the modules concurrently to*

Amornphun Phunopas, Department of Production Engineering, King Mongkut’s University of Technology North Bangkok, 1518 Pracharat 1 Road
Bangsue, Bangkok, 10800, Thailand. Email: amornphun.p@eng.kmutnb.ac.th© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

622

Amornphun Phunopas, Wisanu Jitviriya, Noppadol Pudchuen, Sutee Kumjaikong, Songklod Tunsiri, Eiji Hayashi

overview system is proposed in section 2. The main
controller of all AGVs is the central software, which is
described in section 3. In section 4, the Petri-net model isused to find the shortest path for AGVs and theimplementation in Zone1 and Zone2. Finally, theconclusion shows in section 5.

room to Zone5. The workers pack the products forsending to the customers.

Zone
Zone0

Zone1

2. Process Overview

Zone1

Zone2

Zone2

Zone3

Zone3

Zone4

Zone4
Zone5

Zone5
Zone1

Zone2

Zone5

Table 1. Defined zones and operations

This paper describes applying AGVs to the
manufacturing process of EPS (expandable polystyrene)
products. Foam is widely used for protecting fragile
products in packaging. It is cheap, light and simple toshape in molding. In Fig.1 the process starts to put EPS
raw material to the machine to pre-expand EPS by thestream. The EPS can be shaped in various products suchas protective packaging and home appliance components.
Next step, the products go to the drying room andpacking. All finished products will be stored in thewarehouse waiting for sending to the customers. AGV
operates after the products come out from the machine todeliver them to the drying room and packing zone bycontaining in a cart. AGV carry the cart to do the next
process.
MachineEPS Raw
Material

Operation
Zone0 is the rest area of the AGVs.
Zone1 is the empty cart area.
The AGV delivers the empty cart toZone2 where the products come out fromthe molding machine.
Zone3 is the waiting zone before entering
the drying room according to theschedule plan.
Zone4 is the drying room.
Zone5 is the packing area.
After packing, the worker delivers theempty carts to Zone1.
*Special0, the products do not need to go
to the drying room.

Product

EPS Beads
MOLDING
StreamStreamPRE-EXPANSION

Warehouse

STORAGE

Stream Stream Stream

Protective
PackagingHome Appliance
Components

PackingDrying room

Fig. 2. AGV operation zones in the plant.

Fig. 1. The manufacturing process of EPS products.

The manufacturing process of EPS is defined in the zones
in Table.1 and Fig.2. The AGV moves from zone to zone
using magnetic line detection. There are nine molding
machines to feed new products in Zone2. The workers
check the product quality and arrange the products into
the empty carts for each molding machine. When a cart
is full, the worker will push a button to tell the central
software to call an AGV to take the full cart to Zone3.
The drying process takes one to two hours as per theschedule plan to open and close the door of two drying
rooms in Zone4. When the products are done with thedrying process, the AGV takes the cart from the drying

Fig. 3. The central software and connected modules.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan623

Automated Guided Vehicle System

3. Central Software

button is pushed, it clears the state, and another AGV
carries the empty cart to replace the full cart. The queueof the push button is ordered by FIFO.

The computer server is the central control unit that
communicates to every module, including push buttons,
waiting carts, AGVs, and drying room, as shown in Fig.
3. This paper shows an implementation in Zone1 andZone2. Therefore, there are two related modules: pushbuttons and AGV.

3.2. AGV
There are four AGVs in the system that operate indifferent zones. An AGV is a differential wheel platformthat can take a payload of 100 kg. It takes a cart byhooking and locking under the cart. An AGV has two
cart-carrying configurations: along-lined lock and crosslined lock. An AGV can turn together with a carrying cart
in Fig. 5. To verify the AGV's position, there are RFIDs
under every junction to sense its current location. The
AGV receives all commands from the central software,such as go to a generated shortest path, go to hook, go tolock, and turn itself at its current position.

3.1. Push buttons
The initial condition is that the empty carts are in frontof every molding machine. The box of the push button is
installed in front of all molding machines. There are threebuttons for a worker to push to communicate to thecentral software in Fig. 4. When the yellow button ispushed, the AGV comes to take a full cart to the waitingzone. When the green button is pushed, the AGV comesto take a full cart to the packing zone. When the red

4.

Petri Net

The Petri Net is a tool that verifies that the system's
operation is a stochastic problem to ensure that adeadlock condition does not occur. We focus on task
distribution, routing, scheduling, and queueing.
4.1. Shortest path
The central software can generate the shortest path foreach AGV using the undirected graph in Fig. 6. After
pushing the button, the central software defines thedestination as the position in front of the molding
machine. Then, the central software selects theappropriate AGV for the task and sends the current
position of the AGV to generate the shortest path.
However, it is possible for paths to conflict when all of
the AGVs are operating. The central software will assign
a new path to the AGV or wait for another AGV to pass.

Fig. 4. (Left) The push buttons. (Right) The cart in front of
the molding machine with the products inside.

Turning

Cross-lined lock

Along-lined lockFig. 5. AGV movement on the magnetic line and the cartcarrying configuration.

Fig. 6. The shortest path is generated from the central
software.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

624

Amornphun Phunopas, Wisanu Jitviriya, Noppadol Pudchuen, Sutee Kumjaikong, Songklod Tunsiri, Eiji Hayashi

4.2. Petri-net Model of Zone1 and Zone2

implementation, this research uses Java language. Table
2 shows the relation of the Petri-net model to Java
language.

The model is triggered by pushing the buttons. There aretasks for two AGVs to deliver a full cart and take anempty cart. The following places and transitions aredefined for a Petri net modelof the supervisor in Fig.
7.

Table 2. Petri-net and Java programming

Structure := ( , , , ).
Places : = { , , , , , }.
Transitions : = { , , , , , }.
Input function : ( ) = { }, ( ) = { }, ( ) ={ , }, ( ) = { }, ( ) = { }, ( ) = { , }.
Output function : ( ) = { }, ( ) = { }, ( ) ={ , }, ( ) = { }, ( ) = { }, ( ) = { , }.

The transitions are defined as follows.

Token

Class Method
Block, semaphore, Blocking
Queue, Synchronized channel
Thread-safe data and resources

Deadlock-free PN

Deadlock-free program

Acknowledgements

T1 : AGV1 is in a good position to deliver a full cart,
T2 : AGV2 is in a good position to deliver a full cart,
T3 : AGV1 presently finishing tasks, it is a good position todelivering role,T4 : AGV1 presently finishing tasks, it is a good position to
taking role,
T5 : AGV2 presently finishing tasks, it is a good position to
taking role,
T6 : AGV2 presently finishing tasks, it is a good position todelivering role.

This work is supported by the Thai Research Fund (TRF)
and Polyfoam Manufacturing Co., LTD under thecontract no. RDG6050047.
References
1. Handke G, Production flexibility and modern AGV
system[J], Proc of 6th Int Conf on AGVS, IFSLtd.UK,
1988:pp344-347
2. T. L. Bui, P. T. Doan, H. K. Kim, V. G. Nguyen and S. B.
Kim, “Adaptive Motion Controller Design for an
Omnidirectional AGV Based on Laser Sensor”, Lecture
Notes in Electrical Engineering, Vol. 282, pp. 509-523,
2013.
3. Mantel. Ronald, Design and operational control of an
AGV system[J], International Journal of Production
Economics, 1995(41):pp257-256
4. P. Bhattacharjee, Multi-Robot Path-Planning Using
Artificial Bee Colony Optimization Algorithm, IEEE
conference, 2011.
5. M. Likhachev, S. Koenig, Lifelong Planning A* andDynamic A* lite: The proofs. Technical report, Atlanta,
2001.
6. M.S. Alam and M. U. Rafique, Mobile robot path planningin environment cluttered with non-convex obstacles using
particle swarm optimization, In Control, Automation andRobotics (ICCAR), 2015.

T1
P1

P2

P3

P5
T2

T4

Event-Driven transition
Place

This research successfully created an example case of a
Petri-net model in Zone1 and Zone2 to verify theoperation without deadlock. The tasks are complicatedand have more conditions to complete the whole system.
The next step is to convert the Petri-net model to Java
programming for real implementation for AGVs.

P1 : AGV1 delivering a full cart, AGV2 taking an empty cart,
P2 : AGV1 taking an empty cart, AGV2 delivering a full cart,
P3 : AGV1 taking an empty cart,
P4 : AGV1 delivering a full cart,
P5 : AGV2 delivering a full cart,
P6 : AGV2 taking an empty cart.

P4

Java
Runnable Class

5. Conclusion

The following places are defined to model the supervisor.

T3

Petri-net
Autonomous transition

T5

T6
P6

Fig. 7. A petri-net graph for role assignment supervision.

The Petri-nets model must be clear and verified in many
software tools, for example, CPN tools and PIPE. For real

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

625

Preliminary Comparative Experiments of Support Vector Machine and Neural Network
for EEG-based BCI Mobile Robot Control
Yasushi Bandou, Takuya Hayakawa, Jun Kobayashi

Department of Systems Design and Informatics, Kyushu Institute of Technology
Kawazu 680-4, Iizuka, 820-8502, Japan
E-mail: jkoba@ces.kyutech.ac.jp
lab.jkoba.net

Abstract
This study aims at achieving a practical EEG-based BCI for mobile robot control by means of machine learning. To
this end, the authors had conducted experiments for collecting EEGs as training/test dataset and built some neuralnetworks (NN) using the data; however, satisfactory performance has not been obtained yet. We have employed
Support Vector Machine (SVM) for further improvement and compared the performance with the NNs. As a result,
the SVMs outperformed the NNs in almost all cases.
Keywords: brain computer interface, electroencephalography, support vector machine, neural network, mobile robot
control

pattern classifiers used in BCI and evaluated them.
However, practical classification performance could not
be achieved in the NNs.
Support Vector Machine (SVM) is also a popularmachine learning method that has been applied to various
fields and accomplished tremendous results. In this study,
we employ SVM as classifiers for EEG patterns andcompare their performance with the multi-layer NN weused in the previous studies.

1. Introduction
Brain Computer Interface (BCI) provides you with ameans to convey your intention and volition to acomputer via electroencephalogram. The technology not
only makes up for impairments but also expands andenhances abilities of human beings. Electric wheelchair
control and robotic arm control are typical application
examples of BCI. R. Single et al. developed a SSVEPbased BCI for controlling a wheelchair using multi-class
SVM1,2. J. Meng et al. experimentally investigated anoninvasive BCI for reach and grasp task of robotic arm3.
Mobile robot control by EEG-based BCI is also achallenging and promising technology4.
The authors have been developing an EEG-based
BCI for mobile robot control by means of machinelearning techniques, especially multi-layered neuralnetworks (NNs)5,6. We experimented with anomnidirectional mobile robot to collect brain waves of
persons picturing a movement of the mobile robot in theirbrains. Then, we trained multilayered NNs with themeasured brain waves in order to obtain brain wave

2. Experiments for EEG Data Collection
The authors carried out EEG measurement experiments
in order to collect EEG signals for training NNs andSVMs. The same experiments had been conducted for 3
subjects in the previous study5. For this study, we have
newly obtained EEG signals from 3 other subjects, a 21year-old and two 22-year-old male college students; EEG
signals recorded from totally 6 subjects (named A, B, C,
D, E, F) were used in this study. As with the previous
EEG measurement experiments, the EEG headset shownin Fig. 1 were used for recording EEG signals, and the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

626

Yasushi Bandou, Takuya Hayakawa, Jun Kobayashi

forward, backward, to right, and to left for the mobile
robot, respectively.

omni-wheel mobile robot shown in Fig. 2 that moves
following predetermined commands was employed.
Commands for move forward, backward, to right, and toleft were given to the mobile robot in these experiments.

Fig. 3. Arrows indicating directions (up, down, right, and left)

Fig. 4 shows the procedure of the trials in theexperiments for OPEN-EYES task. At the beginning of
the procedure, a subject counts 5 sec silently while gazing
the mobile robot being at rest. During the task period, thesubject keeps gazing at the mobile robot moving to acertain direction and imagine the robot's motion for 5 sec.
As the task finish, the subject closes their eyes for rest
and the mobile robot automatically moves back to theinitial position. The subject can notice when they should
end the rest even with closed eyes because the mobile
robot emits a sound during the return movement.
The EEG signals recorded with 14 electrodes of the
EPOC for 5 sec were preprocessed according to thefollowing methods. First, EEG signal components in thefrequency range from 8 to 30 Hz were extracted by aband-pass filter. Second, power spectrum of the filtered
EEG signals was calculated, and then moving average
was applied to them. Finally, the power spectrum was
normalized between 0 and 1, and resampled with 1 Hz.
This preprocessing produces a 322-dimensional feature
vector from 5 sec EEG signals, which was used as an
input into the EEG pattern classifiers.

Fig. 1. EMOTIV EPOC (wireless portable EEG headset)

Fig. 2. Omni-wheel mobile robot

Each subject imagined 30 trials per direction, i.e. 120
trials in total, at each day for 5 different days; therefore,
we collected EEG data of 600 trials for each subject. One
trial consists of about 5 sec rest, 5 sec task (imagining),and 5 sec rest again. Brain waves of a subject wererecorded throughout the 15 sec trials. The subject silently
counted 5 sec during the first rest, and then pressed a keyof a keypad just before getting started the task so that a
PC for EEG recording could recognize the timing. Whena beep sounded after 5 sec, the subject stopped the task
and took a rest until the next trial for about 5 sec.
The subjects were given two different imagining
tasks, called CLOSED-EYES and OPEN-EYES.

3. EEG Signal Classification with Multi-layered
Neural Network and Support Vector Machine
Stacked Autoencoder (SAE) with a softmax layer at theoutput was adopted as multi-layered NN for EEG patternclassification. In the training process of the NNs, thehyperparameters were optimized by Bayesian
optimization7. Table 1 describes the optimized
hyperparameters and their options. The SAEs andBayesian Optimization were implemented using
Chainer14 (ver. 1.18.0)8 and GPyOpt13 (ver. 1.2.0)9.
As another EEG pattern classifier, nonlinear SVM
with RBF kernel function was employed. In the training
process of the SVMs, input feature vectors were scaled,
and parameters of RBF function and regularization were

CLOSED-EYES: close your eyes and imagine a
specified arrow
OPEN-EYES: watching the mobile robot moving toa certain direction, imagine the robot's motion
The CLOSED-EYES experiment was carried out first,
then the OPEN-EYES one. Fig. 3 shows arrows, one ofwhich the subjects imagined in a trial of the CLOSEEYES task in the EEG measurement experiment. Up,
down, right, and left arrows correspond to moving© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

627

Preliminary Comparative Experiments of

optimized using grid search. LIBSVM (ver. 3.22)10 wasused for implementing the SVMs.
The classification performance of the NNs andSVMs trained in this study was evaluated using 5-foldcross validation. Table 2 shows the computation
environment for development of the SAEs and SVMs.

4.

Results and Discussion

Table 3 and Table 4 shows results of the comparative
experiments of the NNs and SVMs trained with the EEG
signals recorded in CLOSED_EYES task. The values in
the rightmost columns are averaged classification rate
percentages of the 5 experiment days. The better
averaged classification rate between the NNs and SVMs
are shown in red. Comparative results in the case of
OPEN_EYES are shown in Table 5 and Table 6.
These results clearly show that the SVMs
outperformed the NNs regardless of the task types,
CLOSED_EYES and OPEN_EYES. The averaged
classification rate percentages of all subjects were
55.03% for the NNs and 63.49% for the SVM in
CLOSED_EYES; 50.61% for the NNs and 61.17% for
the SVM in OPEN_EYES.
Table 3. Classification rate percentages of NNs trained with
Bayesian optimization using 120 samples recorded in
CLOSED_EYES

Fig. 4. Procedure of trials in experiments for OPEN-EYES
Table 1. Hyperparameters and options
Hyperparameter
Number of hidden layers
Number of nodes in 1st hidden layer
Number of nodes in 2nd hidden layer
Number of nodes in 3rd hidden layer
Iteration of pretraining each AE (epoch)
Iteration of finetuning (epoch)
Dropout 50% in pretraining AEs
Dropout 50% in finetuning

Options
1~3
250 or 300
150 or 200
50 or 100
1000 or 2000
5000 or 10000
OFF or ON
OFF or ON

1st
day

2nd
day

3rd
day

4th
day

5th
day

Ave.

A

55.00

57.50

66.67

79.17

92.50

70.17

B

50.00

70.00

53.33

66.67

54.17

58.83

C

44.17

49.17

51.77

76.67

33.33

51.02

D

58.34

70.00

54.17

50.83

40.83

54.83

E

45.83

45.00

56.67

44.17

29.17

44.17

F

76.67

42.50

40.00

39.17

57.50

51.17

Table 4. Classification rate percentages of SVMs trained using
120 samples recorded in CLOSED_EYES

Table 2. Specifications of computation environment fordevelopment of SAEs and SVMs
CPU
MemoryStorage
GPU
OS

Subject

Intel® CoreTM i7-6800K CPU @ 3.40GHz
16GB (DDR4-2133 4GBx4)
SSD 240GB + HDD 1TB
NVIDIA GeForce GTX1060 6GB GDDR5Ubuntu 16.04 LTS

Subject

1st
day

2nd
day

3rd
day

4th
day

5th
day

Ave.

A

57.97

49.04

64.40

58.80

62.74

58.59

B

64.70

63.04

71.79

72.50

56.61

65.73

C

93.63

86.73

55.36

59.70

62.02

71.49

D

52.44

49.94

55.95

61.25

55.71

55.06

E

64.64

71.43

65.00

72.32

58.33

66.34

F

65.18

68.16

65.18

63.40

56.79

63.74

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

628

Yasushi Bandou, Takuya Hayakawa, Jun Kobayashi

5. Conclusion

Table 5. Classification rate percentages of NNs trained with
Bayesian optimization using 120 samples recorded in
OPEN_EYES
Subject

1st
day

2nd
day

3rd
day

4th
day

5th
day

Ave.

A

95.83

48.33

45.83

41.67

75.00

61.33

B

66.67

57.50

60.83

33.33

43.33

52.33

C

60.00

40.00

49.17

76.67

38.33

52.83

D

52.50

79.17

37.50

38.34

35.00

48.50

E

49.17

33.34

45.84

19.17

31.67

35.84

F

51.67

62.50

55.84

48.34

45.83

52.84

This paper presented the comparative experimental
results of the SVMs and NNs that were trained to be able
to work as an EEG-based BCI for mobile robot control.
Consequently, in almost all cases, the SVMs with RBF
kernel function have outperformed the NNs trained withhyperparameter optimization. The SVMs transformed
the input feature vectors into a higher dimensional feature
space using RBF kernel function in order to make thefeature vectors be linearly separable as far as possible.
The authors have assumed that the kernel trick, i.e.
transforming feature vectors into a higher dimensionalfeature space, was the key factor for the results of thisstudy. Therefore, from that point of view, the authors will
design an EEG pattern classification method including
feature extraction in order to achieve practical EEGbased BCI mobile robot control.

Table 6. Classification rate percentages of SVMs trained using
120 samples recorded in OPEN_EYES
Subject

1st
day

2nd
day

3rd
day

4th
day

5th
day

Ave.

A

63.04

56.61

62.68

66.31

65.83

62.89

B

68.75

73.57

66.90

63.75

77.02

70.00

References

C

65.00

58.57

67.20

60.24

69.40

64.08

D

69.52

49.88

55.06

52.86

60.00

57.46

E

62.62

67.50

63.39

54.94

72.20

64.13

F

54.64

56.67

45.24

43.21

42.62

48.48

1. R. Singla et al., Influence of stimuli color on steady-statevisual evoked potentials based BCI wheelchair control, J.
Biomedical Science and Engineering 6(11) (2013) 1050–
1055.
2. R. Singla et al., Influence of stimuli color in SSVEP-based
BCI wheelchair control using support vector machines, J.
Medical Engineering & Technology 38(3) (2014) 125–134.
3. J. Meng et al., Noninvasive Electroencephalogram BasedControl of a Robotic Arm for Reach and Grasp Tasks,
Scientific Reports 6(38565) (2016).
4. L. Bi et al., EEG-Based Brain-Controlled Mobile Robots:
A Survey, IEEE Trans. On Human-Machine Systems
43(2) (2013) 161–176.
5. T. Hayakawa and J. Kobayashi, Open Eyes Influence on
Electroencephalography Signals for Constructing Neural
Network Classifiers as Mobile Robot Control BrainComputer Interface, in Proc. of Int. Conf. on Advanced
Mechatronic Systems (Xiamen, China, 2017), ThuM01-03.
6. T. Hayakawa and J. Kobayashi, Improving EEG-based
BCI Neural Networks for Mobile Robot Control by
Bayesian Optimization, J. Robotics, Networking andArtificial Life 5(1) (2018) 41–44.
7. J. Snoek et al., Practical Bayesian Optimization of
Machine Learning Algorithms, in Proc. of Int. Conf. on
Neural Information Processing Systems (Lake Tahoe,
Nevada, 2012), pp. 2951–2959.
8. Chainer: A Powerful, Flexible, and Intuitive Framework
for Neural Networks, https://chainer.org/.
9. GPyOpt: A Bayesian Optimization framework in Python,
http://github.com/SheffieldML/GPyOpt.
10. LIBSVM: A Library for Support Vector Machines,
https://www.csie.ntu.edu.tw/~cjlin/libsvm/.

The input feature vectors to the classifiers wereextracted from the EEG signals obtained in theexperiments as mentioned in Section 2. The EEG signals
were recorded when the subjects were imagining one of
the 4 motions of the mobile robot. That means theextracted feature vectors belongs to one of the classes
representing mobile robot motions, such as move forward,
backward, to left, and to right. The NNs and SVMs weretrained to recognize the class of the input feature vectors.
Nonlinear SVM transforms input feature vectors into ahigher dimensional feature space so that a decisionboundary can linearly separate the feature vectors intoclasses more easily. The transformation is called kernel
trick. In this study, the nonlinear SVMs were constructedwith RBF kernel function. The kernel trick, transforming
the input feature vectors into a higher dimensional space,might have led to the results that the SVMs performed
the better classification performance.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

629

Position Tracking Error Constrained Super-Twisting Dynamic Surface Control with
Disturbance Observer for Robot Manipulators
Seong Ik Han

Mechanical System Engineering, Dongguk University Gyeongju, 123 Dongdae-ro
Gyeongju, Gyeongsangbuk-do 38066, Korea Republic
E-mail: skhan@dongguk.ac.kr

Abstract
A tracking error constrained dynamic surface control (DSC) with a super-twisting algorithm (STA) and a nonlinear
disturbance observer is proposed for robot manipulator systems, where the system dynamics are partially known. The
stabilizing controls of the DSC were designed by combining the STA with the virtual tracking errors defined fromthe DSC design concept. A nonlinear disturbance observer was designed to estimate the unknown nonlinear functionand external disturbance instead of using parameterization of the nonlinear function or adaptive law in theconventional STA and DSC schemes. Moreover, a simpler tracking error constraint method is also proposed. The
proposed tracking error constrained STA-DSC scheme combined with a nonlinear disturbance observer has better
tracking error performance and robustness to uncertainty of robot manipulator system than control systems with aconventional DSC and disturbance observer. The control scheme was evaluated through simulations with thearticulated manipulator system.
Keywords: Dynamic surface control, super-twisting algorithm, nonlinear disturbance observer, robot manipulator.

of both estimation and tracking errors and improves upon
the weakness of uncertainty problems with theconventional DSC system by exploiting the merits ofboth DSC and STA systems. Next, to avoid this complex
variable transformation, a simple constraining control
method is considered. Simulations for two-link
manipulator system present to show the efficacy of the
proposed control scheme.

1. Introduction
In recent years, backstepping control (BSC) has beendeveloped to control manipulator systems with matchedor unmatched uncertainties without neglecting or
linearizing nonlinear terms by using recursive design
procedures [1]. However, the repeated differentiation of
the virtual controls appeared in the BSC design steps
result in the “explosion of complexity”. In order toovercome this drawback, dynamic surface control (DSC)
was proposed by introducing a first-order filter in eachstep of BSC scheme [2]. A conventional DSC is also
designed using the ordinary Lyapunov stability theorem,
so the convergence for such systems is at best
exponential with infinite settling time. In this study, we
considered a DSC system combined with STC [3](SDSC) and propose a nonlinear disturbance observer(NDO). This approach achieves finite-time convergence

2. Dynamics of Manipulator and Actuator
The manipulator dynamics are described as follows:
M (q )q C (q, q )q G (q ) FddiL
Ri kb q V ,
dt

,

(1)
(2)

where q , q, q R n 1 are the joint position, velocity, andacceleration vectors, respectively; M (q) Rn n denotes© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

630

Seong Ik Han

where T pi (t ) is a smooth and strictly bounded function

a positive definite symmetric moment of the inertia
matrix; C (q, q) Rn n is the centripetal Coriolis matrix;
G(q) Rn 1 is the gravity vector; Fd R n 1 thedisturbance torque vector including external disturbanceRn 1 nkt i is the control torqueand joint friction;
vector obtained from the joint actuators; Lm and Rm are
the inductance and resistance of the motor drive,
respectively; kb is the back emf constant; i is the motor
current; and V is the voltage applied to the motor drive.
Property 1. The symmetric and positive definite inertia
matrix M (q ) is bounded by M ( q ) K M for all q ,and 0/ 2 is a constant gain.

i

3.2. Design of nonlinear disturbance observer
The dynamics of a nonlinear disturbance observer is
proposed as follows:ˆ( x)( x )[ p ( x )Property 2. M (q ) 2C (q, q ) is a skew- symmetricmatrix and sT [ M (q ) 2C (q, q )]s 0 .
Property 3. C (q, q ) and G (q) are bounded by some

( x)

KG .

ˆ and

ˆ , it follows that

0i ,

x1

(3)

where x1i

T pi (t )tanei (t )
, i 1,..., n ,ii (t )g ( xi )

z]

x
(7)

x2 ,

x2 f02 ( x) g02 x3
x3 f3 ( x) g3u ,y x1 ,

limt
i (t ) , and ai are appropriately
defined positive constants. The constant i confines thesize of the tracking errors under a steady state. The
decreasing rate ai of i (t ) regulates the required speed ofconvergence of the tracking errors. A error
transformation is defined aswheref ( x)

3.3 Design of super-twisting dynamic surface controller
The strict-feedback state space form of (1) and (2) isrepresented as

A smooth decreasing positive performance function
can be defined to satisfy the partial errori (t )
constraints as follows:i,( x)[ p ( x)z ),The result in (7) can remove the coupled residual
estimation error term that appears in the derivativeequation of the Lyapunov function candidate as shown
later.

3.1. Transformation of the tracking error

ai t

(6)

p( x)
x
x

( x)
(

3. Design of Nonlinear Disturbance Observerand Controller

i )e

z] ,

R n . By using

0 for all xˆThe objectives of the controller design are:(i) All signals of the closed-loop system converge to zeroin finite-time.
(ii) The position tracking errors are constrained within
the prescribed boundary without depending on a highcontrol gain

( 0i

f ( x) g ( x )p ( x) / x

positive constants, KC and KG , such that C (q, q )i (t )(5)

where p(x) are nonlinear functions to be designed and( x)
p ( x ) / x or p(x) are chosen such that

where K M is a diagonal positive constant matrix.

K C q and G (q ),n ,p ( x ), i 1,

q , x2M 0 1G0 (q ) ,

2

1

2,

(8)

q , x3i , u V , g3

M 0 1[

Mx2

Lm1 , f 02 ( x)G ( x1 ) Fd ] , g 02

Lm1 ( Rm x3

kb x2 ) , and u V . The
M 0 nkt , f3 ( x)command tracking error and new states are defined as
z1
i ,1

(4)

x1

1 , z2

ki ,1 zi

zi
t

i ,2

x2
1/ 2

ki ,2 [ki2,1 zi
0

2 , z3x3

3,sign( zi ) ,

(10)

3
1
1/2
ki ,1 zi sign( zi )sign( zi )]d ,2
2
i 1, 2,3 ,(11)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

631

(9)

Position Tracking Error Constrained

where

yd , ki ,1

1

0 and ki ,2Theorem 1. Consider the closed-loop states of a strict
feedback system (9) consisted by a plant (1), nonlinear
estimate law, virtual stabilizing functions, and control
law. The following conditions then holds: if the gains forguaranteeing the positive definiteness of Qi satisfy

0 are constants. The

stabilizing controls and final controller are selected as1

c1,1 1,1 c1,2T p 1,1

2

g02 [ c2,1 2,1

1,2 ,

1

1 1

ˆ2

f 02

(12)

( z2

y2 )

2

2,2 ],2 2

ug03 [ c3,1 3,1

2 2

f3

( z2

y2 )

3,2 ] ,

3

(13)

c1,1

(14)

k1,2

3 3

0 are constants,

where ci , j

ci ,1

i are passed through first-

yi
4.

i

i 1,

i

i 1 , and

i

i (0)

ki ,2

ki ,1

i

i 1 (0), i

1

i ,1

8 1

c1,2T p ,1

2 1 (c1,1 c1,2Tp ) ,

2 i3

i
i, i
8 i
2 i ci ,1 , i 2,3 ,

(20)

2,3,

i

order filters with a time constant i as
i

2 13

(21)

then, zi 0 and i 0 are equilibrium points that arestrongly and globally asymptotically stable, and the
Lyapunov function candidate in (19) is a global andstrong Lyapunov function. The convergence time is givenby (19).
Proof. See [3].

2,3 , (15)

1 1/2
z1
.
2

Stability Analysis

We define the Lyapunov function candidate asV

5. Simulation Results

3

Vi ,

(16)

T
i Pi i ,(17)

The manipulator parameters are presented in Table 1. The
constraint functions were selected as follow: 1 (t ) (0.5
0.008)e 5t 0.008 (rad ) and 2 (t )(0.5 0.008)e 5t
0.008(rad ) . The joint position commands are selected as
qd1 0.1sin(0.8 t ) (rad ) and qd 2 0.15sin(0.8 t )(rad ) . Among the four links of the robot manipulator, only
two links (upper arm = link1, forearm=link2) were selected
for experimental verification. Hence, the dynamic
parameters of the articulated manipulator are given as
follows:

i 1

Vi

where

[ i ,1i

1
2

Ti ,2 ] , Pi

4 i2

i

2 i
, and the1

2 iconstants i 0 and i 0 . A Lyapunov description offinite-time is given as the following inequality:
3

V( )

Vi

i 1
3

T
i i Qi i

i 1
1V (

2V

( ),

(18)

0 . V (t ) converges to an
equilibrium point in finite-time T f given by [4]

where

1

0 and

)

1/2

Tf

2
1

V
ln 1

M11 M12
M 21 M22

M 11

( m1

M 12

m 2 L22

L1 L2 m 2 cos ( q 2 ),M 21

m 2 L22

L1 L2 m 2 cos ( q 2 ), M 22

C (q, q )

2

1/ 2

M (q)

G (q)
( (0))

2.

(19)m 2 ) L12m 2 L222 m 2 L1 L2 cos ( q 2 ),m 2 L22 ,2m2 L1 L2 sin(q2 )q2

m2 L1 L2 sin(q2 )q2

m2 L1 L2 sin(q2 )q1

0

m2 L2 gcos (q1 q2 ) (m1 m2 ) L1 gcos (q1 )m2 L2 gcos (q1 q2 )2

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

632

Seong Ik Hanlink articulated manipulator. The SDSC system has more
improved performance than the DSC system. The
tracking errors of the PSDSC system and PDSC-NDO
systems remain within the prescribed error boundaries,
but the PSDSC-NDO system reveals more advanced
performance due to compensating effect on the nonlinear
disturbance.
(a)SymbolTable 1. Manipulator parameters
ParameterValue

m1 , m2

mass of link1 and 2

L1 , L2

mass of link1 and 2

0.3m, 0.41m
65.5

Rmi

gear ratio of reduction gearresistance of motor

kti

torque constant

0.0182 Nm/Akbiback emf constant

0.182

ni

12.1kg , 3.59kg

0.8294
V/rad/sec(b)

6. Conclusions
A hybrid DSC scheme was combined with the STA andthe nonlinear disturbance observer. The tracking andvirtual tracking errors in the recursive DSC design weretaken as the error component of the error state surfaces of
the STC. The virtual and final controllers were derived
without introducing differentiation of the filtered output
disturbance were estimated by the super-twisting
nonlinear disturbance observers with tracking error
constraint.

(c)

Acknowledgements
This work has supported by the National Research
Foundation of Korea (NRF) grant funded by the Korea
government (MSIT) (No. NRF-2018R1A2B6005128)
References(d)Fig. 1 Simulation results for two-link manipulator system. (a)Tracking output for link 1 of the PSDSC and PSDSC-NDOB
systems. (b) Tracking output for link 2 of the PSDSC and
PSDSC-NDOB systems. (c) Tracking errors for link1 of theDSC, SDSC, PSDSC, and PSDSC-NDO systems. (d) Tracking
errors for link1 of the DSC, SDSC, PSDSC, and PSDSC-NDO
systems.

1. M. Kristic, I. Kanellakopoulos, P. V. Kokotovic,
Nonlinear and adaptive control design (Wiley, NY, 1995).
2. D. Swaroop, J. K. Hedrick, P. P. Yip, and J. C. Gerdes,
Dynamic surface control of a class of nonlinear systems,
IEEE Trans. Automatic Control, 45(10) (2000) 1893-1899.
3. J. A. Moreno, and M. Osorio, Strict Lyapunov functions
for the super-twisting algorithm, IEEE Trans. on.
Automatic Control, 57(4) (2012) 1035-1039.
4. S. Yu, X. Yu, B. Shirinzadeh, and Z. Man, Continuous
finite-time control for robotic manipulators with terminal
sliding mode, Automatica, 41 (2005) 1957–1964.

Fig. 1 shows the joint tracking simulation results for two-

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

633

Image Processing for Picking Task of Random Ordered PET Drinking Bottles
Chen Zhu

Graduate School of Information, Production and Systems, Waseda University, 2-7 Hibikino,
Wakamatsu Kitakyushu 808-0135 Fukuoka, Japan
Takafumi Matsumaru

Graduate School of Information, Production and Systems, Waseda University, 2-7 Hibikino
Wakamatsu Kitakyushu 808-0135 Fukuoka, Japan
E-mail: zhuchen@toki.waseda.jp, matsumaru@waseda.jp
www.waseda.jp

Abstract
In this research, six brands of soft drinks are decided to be picked up by a robot with a monocular RGB camera. The
drinking bottles need to be located and classified with brands before being picked up. A Mask R-CNN is pretrainedwith COCO datasets to detect and generate the mask on the bottles in the image. The Inception v3 is selected for thebrand classification task. Around 200 images are taken, then, the images are augmented to 1500 images per brand byusing random cropping and perspective transform. The results show that the masked image can be labeled with its
brand name with at least 85% accuracy in the experiment.
Keywords: Image Processing; Robotics Picking; Deep Learning; COCO Dataset

using a 3D image sensor, the position and pose of theobject to be picked up are calculated (3).
On the other hand, for the industrial random picking
robots, FANUC, YASKAWA, etc. have developed thebin picking robot by using the structured light orbinocular camera.

1. Introduction
Under the lower birth rate and aging society, the cost of
human labor is becoming higher. In a warehouse, thepicking task for goods sorting takes more than half of thetotal cost (1). During the festival and special events, thedrinks are randomly put in a big box or a cooler box withwater and ice. The existing picking robot can hardly
process the overlapping of the objects without modelingor the same objects (2). In this paper, the image
processing for the robot picking task is discussed.

1.2. Contributions
In some special application such as bottles being put inthe ice water, a normal 3D sensor cannot get the correct
depth information. In this paper, a deep-learning-based
image processing method is purposed to detect andsegment the randomly ordered PET bottles by using a
monocular RGB camera instead of a depth sensor.
Additionally, this research also discusses the brands’
recognition under the overlapped conditions by using theInception v3 without the knowledge on the target.

1.1. Related Works
Random picking is a challenging problem in the robotics
and computer vision fields. The aim of this task is to pickup objects which are manipulated under structured layout
by using a robot arm’s end-tip effector. Bin picking wasstudied when Amazon started the picking challenge. By

2. Methodology© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan634

Chen Zhu, Takafumi Matsumaru

In this research, random piled up drinking bottles ofdifferent sizes and brands are required to be picked up.
The bottle is not limited to one type of bottle, so deeplearning-based detection method are used to solve thisproblem. The whole process is divided into two stages:
the training stage and the detection stage. The training
stage is to train the network in order to get thecorresponding kernel and bias value. The detection stage
is to detect and generate a mask on each bottle and findout the brands of the bottle.

The training of Mask R-CNN takes 160 epochs in total,
where 40 epochs for the classification head, and 120
epochs for the ResNet-101 backbone.
Around 80% of the images are randomly selected for the
training, and the remaining 20% of images are used forvalidation. The training is stopped when the validation
accuracy no longer rising along with the training
accuracy.
2.2. Detection Stage
The detection stage contains four steps as shown in Fig.
2. First, the ROI (Region of Interest) box and the mask
are needed to be generated by using Mask R-CNN. Next,
the mask is bitwise-AND with the original image. Then,by using the ROI generated in the first step, a bottle is cut
out from the image with a black background. Finally, foreach image with only one bottle visible are sent to the
Inception v3 network for brand recognition.

2.1. Network Training StageThe network training is divided into five steps as shownin Fig. 1. First, the Mask R-CNN (Regional
Convolutional Neural Network) (4) is pretrained by theMicrosoft COCO (Common Object in Contest) dataset.
The COCO dataset has a large number of images withlabels and segmentation lines. To prevent overfitting, theMask R-CNN is trained with all 80 classes of COCO
dataset. Second, around 200 photos are taken or found foreach brand of bottle. Next, the dataset of bottles is used
for fine tune the Mask R-CNN. Then, all the images areaugmented with random cutting and perspective
transform to increase the dataset size to 1500 brands perbrands. Finally, the augmented images are used fortraining for brand recognition.

Mask and ROI Generation using
Mask R-CNN
For each mask bitwise-AND on
image with the image

Cut out the image by using the ROI

Train the Mask R-CNN with COCO
datasetBrand Recognition using Inception
v3

Collect the photos of the bottle topick upFig. 2. Detection Process
The evaluation of the brand recognition is based on thecomparison between the human and the network output,
so that it can filter out the overlapped bottles that cannot
be recognized by a human. Assume the number of objects; the number of correct
detected from the image isbrands recalled by the Inception v3 is ; the number ofwhich brands cannot be recognized by a human is N in
which correctly recalled by Inception v3 is . Then theaccuracy P of the brand recognition is calculated by thefollowing formula.
−(1)
P=
× 100%−

Fine Tune the Mask R-CNN

Resize and perspective
transformation on collected image
Train the Inception v3 for brandrecognition

Fig. 1. The network training process© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

635

Image Processing for Picking

3. Result
Based on the method mentioned in the previous section,
the experiment is performed. To run the different
network on the same machine, a library called “Protocol
Buffer” is used as data exchange.
3.1. Training of Mask R-CNN
The bottles are the primary detection target in thisresearch. However, the number of images that can beused to retrain the whole network is limited. The COCO
dataset comes with 80 classes for object detection plus 1class for background. So, all 81 classes are used fortraining the whole network at first. Then the bottles takenfrom the test subject are labeled with a class name and amask as shown in Fig. 3.

Fig. 4. Image Augmentation
The training of the Inception v3 stops, based on theaccuracy that convergence to around 0.85 as shown in Fig.
5. The blue line shows the validation result of the 20% of
the images which can be used to indicate the best fitting
point.

Fig. 3. Image Segmentation for Training
The training rate in this step is set to 0.01 and only
training the mask and classification parts in the network.
3.2. Training for brand recognition
The brand recognition is implemented by the Inception
v3. Retraining the whole network will cause too muchtime and easy to get overfitted. So, the initial weight ofInception v3 is transferred from the object recognition
network. In this research, six kinds of drinking in theJapan market including Oiocha, Coca-Cola, Calpis,
Afternoon tea, Irohasu, and Nama cha are selected as test
subject. For each brand, around 150-200 images arecollected from the Internet or taken directly. Then, theimages are processed randomly with cropping,
perspective transform, rotation and zooming to increase
the number of images up to 1500 for each brand as shownin Fig. 4.

Fig. 5. Training accuracy and validation accuracy
Continuing the training steps will increase the training
accuracy but not validation accuracy, which will lead to
overfitting of the network. In this dataset, validation
accuracy reaches the top when the training steps are 4000
steps.
3.3. Evaluation of Mask and ROI Generation
Fig. 6 shows the result of the mask and ROI generation.
The evaluation is based on the real image taken from anormal monocular camera as shown in Fig. 6(a). By© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

636

Chen Zhu, Takafumi Matsumaru

Table 1 Labeled result and output scoreusing the mask and ROI generated from the network
shown in Fig. 6(b), the original image can be masked andcut off as shown in Fig. 6(c) and (d).

(a) Original Image

Human Labeled
Top Result
OiochaOiochaCalpisCalpisNamachaNamachaNamachaNamachaCoca cola
Coca Cola
Irohasu
Irohasu
Afternoon Tea
Coca Cola
NamachaNamachaCalpisCalpisNamachaNamachaCoca Cola
Coca Cola
Coca Cola
Coca Cola
Irohasu
Irohasu
Number of correct answers

Score
0.66
0.94
0.93
0.56
0.87
0.98
0.62
0.99
0.92
0.97
0.96
0.29
0.91
10

4. Conclusion & Discussion(b) Mask Generation

The combination of the Mask R-CNN and Inception V3
can detect and recognize the brand of the bottles withoverlapping in at least 80% accuracy. Based on the mask
center point and the area of the mask, the robot can be
guide for the picking task.
Acknowledgements
This research is supported by Japan Society for The
Promotionof
Science(KAKENHI-PROJECT17K06277), to which we would like to express our
sincere gratitude.

(c) Masked Image(d) Image Cut out by ROI
Fig. 6. Mask and ROI cutting on the original image
As shown in the result, bottles with color can be correctly
detected from the image. However, bottles with atransparent appearance have a lower detection rate.

References
1. JJ. Bartholdi and ST. Hackman, Warehouse &Distribution Science (The ISyE department Georgia
Institute of Technology, Atlanta, GA, 2014).
2. K. Kim, J. Kim, S. Kang, J. Kim and J. Lee, Vision-based
bin picking system for industrial robotics applications, 9thInternational Conference on Ubiquitous Robots andAmbient Intelligence. (Daejeon, South Korea, 2012).
3. N. Correll and K.E. Bekris et al, Analysis andObservations From the First Amazon Picking Challenge,
IEEE Transactions on Automation Science andEngineering. 15(1) (2016) 172-188.
4. K. He, G. Gkioxari, P. Dollár and R. Girshick, Mask RCNN, 2017 IEEE International Conference on Computer
Vision (ICCV), (Venice, Italy).
5. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens and Z.
Wojna, Rethinking the Inception Architecture for
Computer Vision, 2016 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), (Las Vegas, NV,
USA, 2016)

3.4. Evaluation of the brand recognition
The brand recognition is based on the image cut off fromthe Fig 6(d). These images are resized to 299x299 andsent to Inception V3 for the brand recognition one by one.
The output with the highest score is selected as the result.
Here, we select one more group of test data besides theimages in “Fig. 6. Mask and ROI cutting on the original
image”, and the result of the brand recognition is shownin Table 1.
As the result shows, although the network gives out all
the correct result, the score of output is not very satisfying
in some cases, because the score under 0.6 will be seen
as unacceptable result.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

637

The Use of Importance Ranks to Derive Suitable Timing of Visual Sensing
in Manipulation Task Containing Error Recovery
Akira Nakamura1, Kazuyuki Nagata1, Kensuke Harada2, Yukiyasu Domae11Intelligent Systems Research InstituteNational Institute of Advanced Industrial Science and Technology (AIST)
Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki, 305-8568 Japan
2

Robotic Manipulation Research Group, Systems Innovation Department
Graduate School of Engineering Science, Osaka University
1-3 Machikaneyama, Toyonaka 560-8531, Japan
E-mail: a-nakamura@aist.go.jp, k-nagata@aist.go.jp,
harada@sys.es.osaka-u.ac.jp, domae.yukiyasu@aist.go.jp
www.aist.go.jp

Abstract
In general, a manipulation task can be composed of many skill primitives. Therefore, it is desirable to perform
plural visual sensing in most skill primitives; however, performing sensing all the time is difficult. In this paper, we
propose the addition of importance ranks to the attribute of skill primitives to derive a suitable timing for
performing sensing. Furthermore, we show that the skill primitives distinguished by their high importance ranks
considerably correlates with error recovery.
Keywords: manipulation skill, task stratification, error recovery, error classification, physical distribution

Ideally, a robotic task must be successfullycompleted as planned. However, in actual tasks ofcomplicated plant maintenance, the execution of a task
may terminate before completion. Therefore, as error
recovery is an important research theme for robots thatmust perform actual tasks, various techniques have beenreported.4-7 However, most of the reports on error
recovery are related to orbit adjustment and only few
studies have reported on the application of complicatederror recovery in maintenance tasks.
We have researched error recovery technique inrobotic tasks so that robots can be used for suchcomplicated tasks.8, 9 In our method, error recovery
through a forward correction process is used by slightly
correcting the preplanned task. Also, error recovery

1. Introduction
In recent years, studies on robotic manipulation for
performing required tasks have been conducted invarious fields. The use of robotic manipulation to
perform plant-maintenance tasks and produce industrial
products (Fig. 1) has been researched upon
considerably.1-3 These manipulation tasks tend to becomplex, necessitating the formulation of composition
rules for the entire work process.
By analyzing the assembly and disassembly
sequences performed by humans, those tasks were
determined to be composed of several significant
motion primitives, each of which is called a “skill.”
Furthermore, most maintenance tasks were shown to becomposed of a number of skills.1-3© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

638

Akira Nakamura, Kazuyuki Nagata, Kensuke Harada, Yukiyasu Domae

variations of the number of contact points in the skill
primitives.2 Moreover, many skills can be defined basedon modified versions of these three fundamental skills.

task (max)……

task (2)task (1)task (0)Fig. 1 Maintenance robot

Task

2.2. Stratification of tasks
Skillprimitive

Figure 2 shows a hierarchy of manipulation tasks.8 If we
ignore the servo layer, the skill layer comprisingelements, such as the three skills mentioned earlier, islocated in the lowest layer called the task(0) layer.
Further, the task(1) layer is located one tier above thetask(0) layer. Similarly, task(i+1) is composed ofsequences of task(i) elements. The top layer, in which
the error recovery loop is closed, is called task(max).

Fig. 2 Manipulation hierarchy

through a backward correction process is used forsignificantly correcting the preplanned task.
A manipulation task consists of several skill
primitives, most of which require the use of visual
sensing. However, performing sensing all the time isdifficult because of some restrictions. In this paper, we
propose to distinguish skill primitives according to theirimportance ranks to select the primitives with suitabletiming of visual sensing. Furthermore, we show that theskill primitive distinguished according to a high rank is
deeply related to the skill primitive in which error
recovery is considered in advance. In addition, visual
sensing is assumed to be performed precisely to stopmovement before and after execution of skill primitive.
The next section explains manipulation skills andthe stratification of manipulation tasks. The procedureof marking manipulation skill according to theimportance ranks is proposed in Section 3, and aconcept of error recovery and a method of error
classification are described in Section 4. Section 5
presents the effectiveness of applying importance ranks
to skill primitives through examples of repacking tasks.

3. Addition of Importance Ranks in Skill Primitives
to Derive Suitable Timing for Visual Sensing
Ideally, visual sensing should be performed twice ineach skill primitive to derive the exact posture of theobjects before execution and confirm the last state ofobjects after execution. However, in actuality, it isdifficult to perform sensing at all sensing points becauseof the restrictions of time and hardware.
We researched the suitable timing of visual sensing
in skill sequences with error recovery in Ref. 10. It is
efficient that a visual sensing is performed byconsidering the difficulty in accomplishing a skill
primitive, as shown in Case 4 of Section 4 of Ref. 10.
The degree of necessity of a visual sensing is derived
based on the consideration of the following idea in thispaper. An importance rank derived from the degree ofthe necessity of sensing was added to the attribute ofeach skill primitive. Moreover, the priority of
performing visual sensing in each skill primitive can bederived by deciding the whole value of the importancerank in the total task. To decide the value of animportance rank, an operator’s teaching is allowed,
although it is desirable to decide the value automatically.
Let us consider, for example, importance ranks witha transfer task (Fig. 3) and grasping and gripper-open
tasks (Fig. 4). In the transfer task, a skill primitive inwhich strict visual sensing is needed is designated as
first priority and a skill primitive with the task ofgrasping one or more objects has second priority (Table1). In the grasp and gripper-open skills, the skill
primitive in which the contact state, with the grasped
object, changes has the top priority (Table 2). The
timing of efficient visual sensing in a total task can be

2. Task Stratification
Let us first explain our concept of skills andstratification of tasks; the details can be found in Ref. 1.
2.1. Concept of skills
We analyzed human motions in tasks, such as
disassembly and reassembly, and found that the
movements consisted of several significant motion
primitives. We call such motion primitives “skills”.1 We
considered three fundamental skills, namely move-totouch, rotate-to-level, and rotate-to-insert, all of whichplay an important part in the tasks. A specific task iscomposed of sequences of skill primitives. The skill
sequences can be decided through several methods. In
our previous study, we presented a method that uses

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

639

The Use of ImportanceStart
Sensing(All-over task (max) )Grasp

Fig. 3 Transfer task

Modeling(All-over task (max) )Gripper-open

Planning(All-over task (max) )Fig. 4 Grasping and Gripper-open task

Table 1 Importance rank of transfer skill

The hand is grasping one or more objects.
Other than the above.

HT (1) > HT (2) > LT (= HT (3))

Table 2 Importance rank of grasp and gripper-open skills
Importance rank
HG (1)LG ( = HG (2))(0)
task (1)

(1)

(max)

task ( imax )LT ( = HT (3))

Re-planning(task( i(1)
) ※3
1)

(0)
task (2)※1※1Classification of error

Conditions

task (0)( i0 )※1

The contact state with the grasped object changes.

task ((0)i0 )Other than the above.

Class T (max+1)
Stopping the process andexecuting all necessary changes

HT (2)※3
Modeling(task( (1)i1 ) )Class T (2) Executing thenecessary changes

The target position and orientation are
demanded strictly.

※3Class 2 Changing theplanning parameters
Class 3 Changing themodeling parameters
Class T (1) Changing thesensing parameters

HT (1)

Sensing(task( (1)i1 ) )Class 1

Conditions

task ( imax , …………… , i1 )Importance rank

Execution※1

Sensing※2

task (0)( i0 )

Modeling

HG (1) > LG (= HG (2))

Confirmation

error

correct

derived by considering an importance rank in each skill
primitive according to Tables 1 and 2.
End

4.

Accurate expression:

Error Recovery Through Classification

※2

※3
(0)

(0)

task ( i ) ＝ task ( iThe real performance may differ from the ideal
performance. In actual manipulation, errors often occur
from various causes. This section describes the
proposed concept of error classification and process
flow, with error recovery in the task hierarchy (for moredetails, please refer to Ref. 8).

0max, ……… , i ) ,
0

(1)

(1)

task ( i ) ＝ task ( i1max , …… , i1 )Fig. 5 Process flow with error recovery

result is judged as correct or incorrect by an automatic
process or a human operator. Error recovery is
performed using the following error classifications.
Class 1: The error is judged to be an execution error,
and task(1)(i1) is executed again without correcting theparameters.
Class 2: The error is judged to be a planning error,
and task(1)(i1) is executed again with different planningparameters.

4.1. Classification of errors
The causes of manipulation failures can be attributable
to several kinds of errors. We grouped the error states
into several classes, namely execution, planning, modeling, and sensing, according to the possible causes.8
Merely remedying the causes of these errors does
not always solve the problem. It may be necessary toreturn to a previous step when the working environment
greatly changes because of the error.

Class 3: The error is judged to be a modeling error,
and task(1)(i1) is executed again with different
modeling parameters.
Class T(1): The error is judged to be a sensing error,
and task(1)(i1) is executed again with different sensing
parameters.
Class T(2): task(2)(i2) is executed again after the
execution of the necessary changes and returns to theinitial position one tier above layer task(1)(i1).
::4.2. Error recovery based on classification
A generalized process flow of stratified tasks thatconsiders error recovery was proposed in Ref. 8. Figure
5 illustrates the central portion of Figure 10 in Ref. 8.
This process is performed based on error recovery
through a backward correction process. At theConfirmation step in each skill primitive task(0)(i0) , the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan640

Akira Nakamura, Kazuyuki Nagata, Kensuke Harada, Yukiyasu Domae

Home(Skill3) Approach: Moving to the grasping point at
low speed.
(Skill4) Grasp: Grasping the target object.
(Skill5) Lift-up: Lifting the grasped object.
(Skill6) Departure: Moving to a specific point in areference frame of a box from which theobject is taken.
(Skill7) Move-between-reference-frames: Moving toa specific point in a reference frame of a box,in which the object must be placed.
(Skill8) Move-to-destination: Moving to the destination point.
(Skill9) Lower: Bringing down the grasped object.
(Skill10) Hand-open: Opening to place the object.
(Skill11) Leave: Moving to the safe area.
(Skill12) Home: Returning to the starting point for the
next approach motion.

P10 (=P1 )P1
Home
Move-toapproach

P5

Move-betweenreference frames

P9
P6 Move-to-

destination

Pre-grasp

Approachdirection Departure

P4
P2

P7
Gripper-open Bring-

Lift

down
heightLower

Lift-up height

Leave

P8

Approach
ApproachlengthGra spP3

Small box
La rge box

Fig. 6 Picking and placing task using a gripper

Class T(max): task(max)(imax) is executed again after the
execution of the necessary changes and returns to theinitial position at (max – 1) tier above layer task(1)(i1).

B. Importance ranks of skills in repacking tasks

Class T(max + 1): When it is judged that too manychanges will be required, the process being executedis aborted.

Here, we consider the importance ranks of skills in
the repacking task. An importance rank of each skill
primitive is derived (Section 3) as follows:

5. Importance Ranks in Skill Primitives with
Illustration of Repacking Tasks

Skill1: LT,
Skill2: LG,
Skill3: HT (1),Skill4: HG (1), Skill5: HT (2), Skill6: HT (2),Skill7: HT (2), Skill8: HT (2), Skill9: HT (1),Skill10: HG (1), Skill11: LT,
Skill12: LT

Let us consider robots playing an active role in physicaldistribution. Then, consider tasks involved in repacking
objects from a large box into a small box at a
distribution center and the processes of error recovery
for the robot tasks. Here, a typical task of picking up an
indicated object by using the parallel jaw gripper on the
robot11 is selected, and an importance rank is considered
for each skill primitive according to Section 3.
This section describes the main errors in therepacking tasks. In addition, we explain the process ofrecovery with respect to each error. Moreover, weconsider the relation between typical errors of the task
and skill primitives with high importance degrees.

Let us suppose that the following relation exists
among the importance ranks.
HT (1) > HG (1) > HT (2) > L (= LT = LG)(1)

These skill primitives, in turn, can be arranged as follows based on the importance ranks with inequality (1).
First priority: Skill3, Skill9
Second priority: Skill4, Skill10
Third priority: Skill5, Skill6, Skill7, Skill8
Fourth priority: Skill1, Skill2, Skill11, Skill12
C. Candidate errors in repacking tasks

A. Sequence of repacking tasks

Next, we will consider the main types of errors in
the tasks.

Figure 6 shows the processes of picking up andplacing objects such as PET bottles by using amanipulator with a gripper. The flow of the skill
sequence if there is no error is shown in Fig. 7. These
primitive motions are as follows.

(1) Errors when grasping and lifting(1. a) error of height; the gripper does not reach a
PET bottle in Skill3 (Fig. 8 (a))(1. b) error of parallel movement to the bottom of
the box; the open gripper is not positioned
around the cap of a PET bottle in Skill3 (Fig. 8(b))(Skill1) Move-to-approach: Moving to the starting
point of the approach motion.
(Skill2) Pre-grasp: Opening to grasp the object.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

641

The Use of Importance

Skill1： Move-to-approach

Transfer

Skill2： Pre-grasp

Manipulation

Skill3： Approach

Transfer

Skill4： Grasp

Manipulation

Move-to-destination

Approach

Approach
Lift

ApproachlengthApproachlengthA shortage

A gap

But itdoesn’t
move.

Falling

Skill5： Lift
Skill6： Depart
Skill7： Move-betweenreference frames(a) Case of (1.a)
Transfer(c) Case of (1.c)

Fig. 8 Errors when grasping and lifting

Skill8： Move-to-destination
Skill9： Lower
Skill10： Gripper-open(b) Case of (1.b)

Manipulation

Lower

Lower

A gap

Shift

Plan

Practice

Fig. 9 Errors when carrying
Lower

Lower

Shift

Stop

A ga pSkill11： Leave
Skill12： Home

Transfer

Fig. 7 Task sequence of picking and placing a bottle(a) Case of (3.a)(b) Case of (3.b)(c) Case of (3.c)(d) Case of (3.d)

Fig. 10 Errors when packing(1. c) error in which a PET bottle cannot be
extracted owing to tight packing in Skill5 (Fig.
8 (c))(1) Cases (1. a), (1. b), and (3. a)(2) Errors when carrying an object

Class 1: Execution is simply repeated.

(2. a) error in which a PET bottle is dropped in Skilli(i = 6, 7, 8) (Fig. 9)Class 2: The process is re-executed from theplanning step after the planning parameters are
modified from the gap.

(3) Errors when packing

Class 3: The process is restarted from the modelingstep after modification of the geometric modelsof the bottles.

(3. a) error of parallel movement to the bottom of thebox; a gap exists between the target position in
Skill9. No movement of another bottle isperformed (Fig. 10 (a)).

Class T(1): The process is re-executed from thesensing step after modification of the coordinate
system.

(3. b) error in which the grasped PET bottle pushes
the surrounding PET bottles in the box in Skill9(Fig. 10 (b)).

(2) Cases (1. c) and (3. d)Class 1: Execution is simply repeated.

(3. c) error in which (3. a) and (3. b) occur
simultaneously (Fig. 10 (c)). Gaps occur both
between the positions of the grasped bottle andsurrounding bottles.

Class 2: The process is re-executed from theplanning step after the modification of theplanning techniques such as the motion withslightly shaking in lifting or lowering.

(3. d) error in which Skill9 is stopped before
completion (Fig. 10 (d)) because the available
space is too small.

Class 3: The process is restarted from the modelingstep after modification of the geometric modelsof the bottles or box.

D. Error-recovery processes

Class T(1): (The same as (1))

This section presents the explanation of the recovery
process from each error. The following numbers andletters in brackets coincide with those describing thecandidate errors in the previous section. The corrections
were performed based on definite or indefinite causes
derived from the error classification, and the processrestarts from the corresponding step.

(3) Case (2. a)In most cases, the task of picking up a bottle in thebox is changed to the task of lifting the bottle thathas dropped to the bottom of the path; Class T(2) ischosen. The skill sequence of this task is executedfrom the beginning.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

642

Akira Nakamura, Kazuyuki Nagata, Kensuke Harada, Yukiyasu Domae(4) Cases (3. b) and (3. c)selection of skill primitives in which visual sensing
should be performed. Furthermore, we showed that skill
primitives distinguished according to their highimportance ranks considerably correlate with those inwhich error recovery is considered in advance. This
characteristic can be used for selecting skill primitives
in which error recovery is planned in advance.
In the future, we will apply our proposed procedure
to sensing, modeling, planning, and execution tasks andattempt to apply our error-recovery method to actual
maintenance robots.

The original task is completed, and the processreturns to the start of the flow. Next, the task tomove the bottle to the correct position is executed.
When the movement of many bottles is demanded, it
is desirable to consider whether or not the paths
converge efficiently.
E. Sensing time in repacking tasks
We considered typical errors in C and recovery
processes of errors in D. Furthermore, we introduced theimportance ranks of skill primitives in B so that Skill3(Approach skill) and Skill9 (Lower skill) became highpriority tasks. Then, these are included in skill
primitives treated in C and D. That is, Skill3 is relevantto (1. a) and (1. b), and Skill9 is relevant to (3. a), (3. b),(3. c), and (3. d). Thus, the skill primitive in which error
recovery is almost considered has a deep relation withthe skill primitive in which visual sensing is needed.
Moreover, skill primitives that are important alongside
the above-mentioned skills are grasp and gripper-open
skills, i.e., Skill4 and Skill10, respectively, and transfer
skills: Skill5, Skill6, Skill7 and Skill8. Especially, Skill5
and Skill7 are skill primitives with high importanceranks, as in (1) and (2) in C, respectively.
When considering error recovery processes in thetask composed of several skill primitives, the total
number of skill primitives in which our error-recovery
procedure8, 9 can be applied increases considerably. It isdifficult to consider all the recovery processes as great
effort is necessary for planning them. When using theimportance ranks proposed in Section 3, the selection ofskill primitives, for which considering an error-recovery
procedure is suitable, becomes relatively easy.

References
1. T. Hasegawa, T. Suehiro and K. Takase, A model-based
manipulation system with skill-based execution, IEEE
Trans. Robot. Autom., 8(5) (1992) 535-544.
2. A. Nakamura, T. Ogasawara, T. Suehiro and H. Tsukune,
Skill-based backprojection for fine motion planning, in
Proc. IEEE/RSJ Int. Conf. on Intell. Robots Syst., (Osaka,
Japan, 1996), 2, 526-533.
3. A. Nakamura, T. Ogasawara, K. Kitagaki and T. Suehiro,
Using robust and simplified geometric models inskillbased manipulation, in Proc. IEEE/RSJ Int. Conf. onIntell. Robots Syst., (Hawaii, USA, 2001), 1, 138-145.
4. B. R. Donald, Planning multi-step error detection andrecovery strategies, Int. J. Robot. Res., 9(1) (1990) 3-60.
5. C. M. Baydar and K. Saitou, Off-Line error prediction,
diagnosis and error recovery using virtual assembly
systems, in Proc. IEEE Int. Conf. Robot. Autom., (Seoul,
Korea, 2001), 818-823.
6. M. Scheutz and J. Kramer, Reflection and reasoning
mechanisms for failure detection and recovery in a
distributed robotic architecture for complex robots, in
Proc. IEEE Int. Conf. Robot. Autom., (Roma, Italy, 2007),
3699-3704.
7. P. Pastor, M. Kalakrishnan, S. Chitta, E. Theodorou and
S. Schaal, Skill learning and task outcome prediction for
manipulation, in Proc. IEEE Int. Conf. Robot. Autom.,(Shanghai, China, 2011), 3828-3834.
8. A. Nakamura and T. Kotoku, Systematization of error
recovery in skill-based manipulation, Artificial Life andRobotics, Springer, 14(2) (2009) 203-208.
9. A. Nakamura, K. Nagata, K. Harada, N. Yamanobe, T.
Tsuji, T. Foissotte and Y. Kawai, Error recovery using
task stratification and error classification for manipulation robots in various fields, in Proc IEEE/RSJ Int. Conf.
on Intell. Robots Syst., (Tokyo, Japan, 2013), 3535-3542.
10. A. Nakamura, K. Nagata, K. Harada and N. Yamanobe,
The Suitable Timing of Visual Sensing in Error Recovery
Using Task Stratification and Error Classification, J.
Robotics, Networking and Artificial Life, 4(2), (2017)134-137.
11. A. Nakamura, K. Nagata, K. Harada and N. Yamanobe,
Error recovery using task stratification and error classification for manipulation robots in physical distribution,
Trans. Control and Mechanical Syst., 3 (2014) 122-131.

6. Conclusion
In general, a manipulation task can be composed ofmany skill primitives. It is desirable to perform pluralvisual sensing in most skill primitives as geometricmodeling and task planning are performed before
execution, and task achievement is confirmed after
execution. However, the performance of sensing at allsensing points is difficult because of the restrictions oftime and hardware. In this paper, we proposed the
addition of importance ranks to the attribute of skill
primitives to derive suitable timing to perform visual
sensing. The use of the proposed method eases the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan643

Electronic Measurement and Gamification of Balance Tests
Yan-Xin Liu, Massimiliano Leggieri, Henrik Hautop Lund

Center for Playware, Technical University of Denmark, Building 326,
2800 Kgs. Lyngby, DenmarkE-mail: s161642@student.dtu.dk, maxleggieri@gmail.com, hhl@playware.dtu.dk
www.dtu.dk

Abstract
We gamified three common balance tests for fall risk diagnosis, Timed Up & Go (TUG), Chair Stand (CS), and Four
Square Step Test (FSST), by using the Moto Tiles. The embedded pressure sensor of the Moto Tile was used to detect
the movement of subjects and to time the tests. The proposed Moto measurement was compared with traditional
stopwatch measurement. A total of 51 samples were analyzed. The intraclass correlation coefficients of the TUG and
CS tests were very high (0.98 and 0.94). The FSST was analyzed by Pearson’s correlation coefficient and the result
also indicated strong correlation (0.79). Based on the results, we concluded that the proposed Moto tests are as reliable
as traditional tests.
Keywords: Playware, gamification, balance test, fall risk.

tiles on ground and step on the tiles to play games. In thepast years, the Moto Tiles has been proved to be aneffective tool for physical and cognitive training.7-10 The
Moto Tile trainings produced significant improvementwith less training times than traditional training.9, 11 The
amazing effects were resulted from the essence of play,which allows subjects to forget that they are training andto be able to devote themselves into the play.
We selected three typical balance tests and gamified
them by the Moto Tiles. They are Timed Up & Go
(TUG)12, Chair Stand (CS)13, and Four Square Step Test(FSST)14. A total of 51 older subjects took the three tests
and their times were measured by both stopwatch and theMoto Tiles. The two time measurements were compared
to examine the validity and reliability of the Moto tests.

1. Introduction
Fall is one of the most common threaten of injures of theelderly. About 30% elderly aged over 65 fall at least oncea year.1, 2 Serious falls can even result in disability ordeath.3, 4 The causes of falls are often associated with
impaired balance ability.5 Hence, the assessment ofbalance is a critical part of fall prevention.
There are many broadly accepted standardizedbalance tests which have been proved to be accurate
predictions of fall risks. These balance tests usuallyrequire nurse practitioners or physiotherapists to instructsubjects to accomplish a physical movement task in a
specific test environment. Such test environment isnormally stressful and boring, which prevents the best
performance of subjects and may also make highfrequency regular test (e.g., daily) impractical.
In order to let subjects fully engage in balance tests
and reduce the workload of testers, we applied the Moto
Tiles6 to set the test sites and time the test. The Moto Tiles
is a modular distributed system, which consists of ten
tiles and a tablet. The tiles are controlled by the tablet
wirelessly. Typical usage of the Moto Tiles is to place the2. The Design of the Moto Balance Test
There are numerous balance tests developed over theyears. We selected balance testes for the gamification
based on the following criterion:
The critical steps of the test can be measured bypressure sensors (e.g., sit down to a chair or step to aparticular position).

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

644

Yan-Xin Liu, Massimiliano Leggieri, Henrik Hautop Lund

counterclockwise). The cut point of the FSST test for fallrisk is 15 seconds14. Goh et al.19 showed that the FSST
test has a high correlation with TUG scores (r = 0.59; P= 0.02), and a high intrarater reliability (ICC = 0.83).
Traditionally, the time of the above tests aremeasured by stopwatch by an operator manually pressing
start and stop on the stopwatch. By contrast, the Motobalance tests use the Moto Tiles to automatically measure
the timing of subjects’ physical movement. For the TUG
and CS tests, a Moto Tile is placed on the chair to detect
the rise and sit down movement of the subject (see Fig.
2). For the FSST test, four Moto Tiles are place on thefloor to form the four squares (see Fig. 3). One tile is lit
in green to indicate next target step, while other tiles are
lit in blue. The detail of the time measurements can befound in Table 1.

Fig. 1. Schematic diagram of the FSST test.

The test is relatively simple and can be done withinminutes
The validity and reliability of the test are widely
accepted.
Then the TUG, CS, and FSST tests were selected. The
TUG test measures the time that a subject takes to risefrom a chair, walk three meters, turn around, walk backto the chair, and sit down. Typical cut point of the TUG
test is 14 seconds12. The TUG test has been proved tohave a high inter-rater reliability (intraclass correlation
coefficient [ICC] = 0.99)15, and significant association
with falls.16 The CS test measures the time that a subject
takes to stand up from a chair and sit down five times.
The cut point of the CS test for fall risk is 15 seconds17.
The CS test has a high correlation with the TUG
test(r=0.64, p<0.001) and a strong reliability(ICC=0.95).18 In the FSST test, the floor is divided into
four squares as Fig. 1 shows. At the start of the test,
subjects stand with both beet on the Square 1 and face to
Square 2. Then, subjects are required to step as fast as
possible into each square with both feet in the following
sequence: Square 2, 3, 4, 1, 4, 3, 2, 1 (clockwise andTable 1. Comparison between stopwatch measurement and theMoto measurement.
Test name
TUG

Ordinary test
Start:
whentheinstructor says go.
End: when the subject
sit down to the chair

CS

Start:
whentheinstructor says go.
End: when the subject
sit down to the chair
5th time

FSST

Start: when the firstfoot reach the Square2.
End: when both feettouch the Square 1.

Moto Test
Start: when the Moto
tile on the chair isreleased.
End: when the Moto
tile on the chair ispressed.
Start: when the Moto
tile on the chair isreleased.
End: when the Moto
tile on the chair ispressed 5th times.
Start: when the Moto
tile No. 2 is pressed.
End: when the Moto
tile No. 1 is pressed.

3. Validity and Reliability of the Moto PhysicalTest
In order to examine the validity and reliability of the
proposed Moto balance tests, we presented the tests incooperation with a number of elderly care institutes
around Copenhagen area. Totally 51 older adults(average age 80.1 years) participated the tests. Two ofthem took only TUG test due to their impaired mobility
and endurance.
Since the proposed Moto TUG and CS tests do notchange any test standard, the subjects took only the TUG
and CS test once but they were timed by both stopwatch
and the Moto Tiles, and therefore it was the reliability of
the Moto measurements that were analyzed. However,Fig. 2. The Moto Tile for the TUG and CS test.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

645

Electronic Measurement and Gamification

stopwatch time (see Table 1). The correlation between
the two measurements were also strong (r = 0.79).
4. Discussion & Conclusion
The presented results indicate that the proposed Motobalance tests are very reliable and can be a good
replacement of traditional test method. The Moto balance
tests allow seniors to take the physical test along with
Fig. 3. The Moto Tiles for the FSST test.

Moto

Moto

Moto

the proposed Moto FSST test is not exactly the same asordinary FSST test, which allows participants to step onlarger squares than the Moto Tiles. Therefore, subjects
were required to take an ordinary FSST and a Moto FSST
separately. The sequence of the two FSST tests were
determined by coin flipping in order to compensate thetest-retest improvement. Due to the small difference ofthe two FSST tests, they were regarded as two different
tests and it was the validity of the Moto FSST in
association with the traditional FSST that were analyzed
Fig. 4 shows the relationship between the stopwatch
measurement and the Moto measurement of the threetests. The reliability of the TUG and CS tests were
examined by ICC, and the validity of the Moto FSST was
examined by Pearson’s Correlation Coefficient. The
differences between stopwatch and the Motomeasurement are evaluated by Wilcoxon Signed-Rank
Test.
For the TUG test, there was no significant difference
between stopwatch and the Moto measurement (p = 0.84).
A consistent relationship between the two times can beobserved (ICC = 0.98, 95% CI = [0.97, 0.99]).
For the CS test, the Moto measurement wassignificantly faster than stopwatch measurement (p =0.01). This is because a few seniors with poor limb
strength had unsuccessful risings, which were notcounted by instructor, but triggered the Moto tile.
However, the correlation between the two measurements
were also very strong (ICC = 0.94, 95% CI = [0.89, 0.96]).
For the FSST test, the difference between the
traditional and the Moto tests was marginallyinsignificant (p = 0.06). The mean value of the Moto time
was faster than that of the stopwatch time, which meetthe fact that the Moto time is one step less than theFig. 4. Relationship between the stopwatch and the Motomeasurement. Dotted line represents unity. (a) TUG (b) CS (c)FSST.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

646

Yan-Xin Liu, Massimiliano Leggieri, Henrik Hautop Lund

GAMES FOR HEALTH: Research, Development, and ClinicalApplications, 3 (2014) 277-283.
8. H.H. Lund, Modular interactive tiles for rehabilitation:evidence and effect, in Proceedings of the 10th WSEAS
international conference on Applied computer science (2010),
pp. 520-525.
9. Y. Liu, H.H. Lund, L. Wu, Playful Cognitive Training with
Physical Interactive Tiles for Elderly, in 2018 International
Conference on Information and Communication Technology
Robotics (ICT-ROBOT) (2018), pp. 1-4.
10. Y. Liu, H.H. Lund, Effect of Playful Physical Interactive
Tiles on Cognition, in 2018 International Conference on
Information and Communication Technology Robotics (ICTROBOT) (2018), pp. 1-4.
11. H.H. Lund, Play for the elderly-effect studies of playful
technology, in International Conference on Human Aspects of
IT for the Aged Population (2015), pp. 500-511.
12. A. Shumway-Cook, S. Brauer, M. Woollacott, Predicting
the probability for falls in community-dwelling older adults
using the Timed Up & Go Test, Physical therapy, 80 (2000)
896-903.
13. J.M. Guralnik, E.M. Simonsick, L. Ferrucci, R.J. Glynn, L.F.
Berkman, D.G. Blazer, P.A. Scherr, R.B. Wallace, A shortphysical performance battery assessing lower extremity
function: association with self-reported disability andprediction of mortality and nursing home admission, Journal ofgerontology, 49 (1994) M85-M94.
14. W. Dite, V.A. Temple, A clinical test of stepping andchange of direction to identify multiple falling older adults,
Archives of physical medicine and rehabilitation, 83 (2002)
1566-1571.
15. S.S. Ng, C.W. Hui-Chan, The timed up & go test: its
reliability and association with lower-limb impairments andlocomotor capacities in people with chronic stroke, Archives ofphysical medicine and rehabilitation, 86 (2005) 1641-1647.
16. T. Nakamura, K. Meguro, H. Sasaki, Relationship between
falls and stride length variability in senile dementia of the
Alzheimer type, Gerontology, 42 (1996) 108-113.
17. S.L. Whitney, D.M. Wrisley, G.F. Marchetti, M.A. Gee,
M.S. Redfern, J.M. Furman, Clinical measurement of sit-tostand performance in people with balance disorders: validity ofdata for the Five-Times-Sit-to-Stand Test, Physical therapy, 85(2005) 1034-1045.
18. A. Goldberg, M. Chavis, J. Watkins, T. Wilson, The fivetimes-sit-to-stand test: validity, reliability and detectablechange in older females, Aging clinical and experimental
research, 24 (2012) 339-344.
19. E.Y. Goh, S.Y. Chua, S.-J. Hong, S.S. Ng, Reliability andconcurrent validity of four square step test scores in subjects
with chronic stroke: a pilot study, Archives of physical medicine
and rehabilitation, 94 (2013) 1306-1311.
20. K. Berg, S. Wood-Dauphinee, J. Williams, The Balance
Scale: reliability assessment with elderly residents and patientswith an acute stroke, Scandinavian journal of rehabilitation
medicine, 27 (1995) 27-36.

regular Moto playful training. The digital solution canalso record and track the test performance automaticallyand thus can easily present the improvement in the tests
to users, which would motivate users to engage more in
the training.
Further possibility of the Moto balance tests can be
the development of more standardized test (e.g., 30
Second Chair Stand, Tandem walk, etc). All the Motobalance tests can be later integrated together as an overall
assessment of fall risk like the Berg Balance Scale,20
which evaluates both the static and dynamic balanceabilities.
Acknowledgements
The authors would like to thank Dagcentret PåTværbommen, Dagcenter Vennerslund, DaghjemmetBlåklokkevej, Betaniahjemmet, Hillerød Sundhedscentret, and Omsorgscentret Toftehøjen for the cooperationof the trial and also thank all participants of thisexperiment.
Part of the work was produced as part of the EU
Horizon 2020 project REACH under grant agreement no.
690425.
References
1. A.J. Milat, W.L. Watson, C. Monger, M. Barr, M. Giffin, M.
Reid, Prevalence, circumstances and consequences of falls
among community-dwelling older people: results of the 2009
NSW Falls Prevention Baseline Survey, New South Wales
public health bulletin, 22 (2011) 43-48.
2. M.G. Amin, Y.D. Zhang, F. Ahmad, K.D. Ho, Radar signal
processing for elderly fall detection: The future for in-homemonitoring, IEEE Signal Processing Magazine, 33 (2016) 7180.
3. R.W. Sattin, D.A. LAMBERT HUBER, C.A. DEVITO, J.G.
RODRIGUEZ, A. ROS, S. BACCHELLI, J.A. Stevens, R.J.
Waxweiler, The incidence of fall injury events among theelderly in a defined population, American journal of
epidemiology, 131 (1990) 1028-1037.
4. P.K. Canavan, L.P. Cahalin, S. Lowe, D. Fitzpatrick, M.
Harris, P. Plummer-D'Amato, Managing gait disorders in older
persons residing in nursing homes: a review of literature,
Journal of the American Medical Directors Association, 10(2009) 230-237.
5. K.B. Foreman, O. Addison, H.S. Kim, L.E. Dibble, Testingbalance and fall risk in persons with Parkinson disease, anargument for ecologically valid testing, Parkinsonism & relateddisorders, 17 (2011) 166-171.
6. http://www.moto-tiles.com/
7. H.H. Lund, J.D. Jessen, Effects of short-term training ofcommunity-dwelling elderly with modular interactive tiles,© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

647

Playful Body and Brain Test with the Moto Tiles
Yan-Xin Liu, Massimiliano Leggieri, Henrik Hautop Lund

Center for Playware, Technical University of Denmark, Building 326,
2800 Kgs. Lyngby, DenmarkE-mail: s161642@student.dtu.dk, maxleggieri@gmail.com, hhl@playware.dtu.dk
www.dtu.dk

Abstract
We propose novel playful body and brain test. The test includes a series of standardized Moto Tiles games, which
cover both physical and cognitive abilities. Reference scores of the tests were modeled by data collected from a large
number of participants of different ages. The reference scores can be applied to evaluate a user’s body and brain
abilities in comparison with the average performance at the same age. By inversing this model, a new model was
obtained. The input of the new model was game score and output was user’s body and brain age, which provides aninteresting and attractive manner to present the test results.
Keywords: Playware, cognition, aging, game.

presented a Moto training regime which could enhance
the reaction speed of the elderly. The versatility of theMoto Tiles can be attributed to its modular andminimalistic design which allows both developer andusers to create their games and usages with infinite
possibilities. Based on this feature, we propose a novel
Moto body & brain age (BBA) test, which is aimed atproviding an early detection solution in a user friendly,
enjoyable, and attractive environment.

1. Introduction
Aging is becoming a prevalent social issue in many
developed countries. There were 27.3% of Japanese aged
over 65 years old by 2016.1 The figure at the global level
was also predicted to reach 22% by 2050.2 A directconsequence of population aging is the increased public
expense of medical resources, which will become aserious social burden. There can be two possible
directions to reduce the stress from the technology
perspective. First is the development of economic
personal health facilities which can improve a person’shealth level and delay the aging. Second is thedevelopment of intelligent hardware and software which
can detect disease, impairment, and any characteristics ofaging at an early stage. Based upon the early detection, aprevention or treatment plan can be established in order
to avoid some of the aging-related health problems.
During the past decade, the Moto Tiles3 has been
proved to be an effective training tool which meets thefirst direction described above. Early investigation
showed that the Moto Tiles training significantly
improved the balance of older adults with less training
period than ordinary training.4-6 Later Liu et al.7

2. The Design of the Moto BBA Test
The Moto Tiles Bode & Brain Age test includes four
short Moto Tiles games which can evaluate different
aspects of body and brain abilities. Each game lasts for
only 30 seconds so that participants can complete the test
quickly within 2 minutes. The four games and theirdescriptions are listed below:
Color Race: One tile lights up in red. Player steps on
the lit tile. Each correct press is counted as one point.
Special One: One tile lights up in a special color(another color than the rest of the tiles). Player steps
on this special tile. Then all colors change and next
round begins. Each correct press is counted as one
point.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

648

Yan-Xin Liu, Massimiliano Leggieri, Henrik Hautop Lund

Final Countdown: All tiles light up in red. The tiles
count down by turning off the LEDs in sequence.
Player keeps light in all tiles by pressing on them
before any tile count down to zero. Each correctpress is counted as one point.
Remember: Four tiles light up in different colors.
Player remembers the color and position. After four
seconds, the tile are turned off. Player has to step on
the tiles according to a color sequence displayed on
the tablet. Each correct press is counted as one point.
Besides, wrong presses are also recorded and error
rate is calculated for analysis.

model, which is the inverse of the first model. That is, theinput is a user’s game performance and the output is theuser’s body & brain age. This model presents a user’s
performance in an interesting and fun manner, which canmotivate the user’s further training and obtain better
effects.
In order to construct an accurate model which canreflect the average performance of overall population, we
conducted a large-scale test with over 100 participants
enrolled. At the moment of handing this paper, totally
127 samples aged from 4 to 97 were collected.
The models are constructed by polynomial fitting.
The data was randomly divided into a training set, avalidation set, and a testing set, each of which accounts
for 70%, 20%, and 10% respectively. The training set
was used for calculating the polynomial coefficients. The
validation set was used for finding the best order of thepolynomial and avoiding overfitting. The testing set wasused for a final evaluation of the model.

3. The Moto BBA Model

Score

Score

Error rate

Score

We propose two models for the BBA evaluation. The firstmodel is an age-score model, where the input is age andthe output is a normative performance at the input age.
The model is able to evaluate if a user performs better or
below average and based on this recommend a training
program to the user. The second model is a score-age

Fig. 1. The age-score model.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

649

Playful Body and Brain

indicate that the participant may need some training to
improve corresponded abilities. For the senior users, a
low score may also indicate potential age-related diseases
or fall risk. The model of the Remember game provides
a reference of short-term memory, which can be used forthe early diagnosis and screen of certain types ofdementia.
Fig. 2 shows the score-age model of the four games.
The fitted formula of the Color Race game is:Fig. 1 shows the age-score model of the four games.
The fitted formula of the Color Race game is:−3.1 ∙ 10+ 4.7∙+ 8.5 ∙ 10∙

− 0.081 ∙+ 2.8 ∙
(1)

+ 0.45 ∙+ 21.56(2)

The formula of the Special One game is:−0.0069 ∙The formula of the Final Countdown game is:−1.39 ∙ 10 ∙1.89 ∙ + 26.92+ 4.63 ∙ 10∙

− 0.053 ∙+(3)+ 0.30(4)

And the formula of The Remember game is:
1.14 ∙ 10∙− 0.0065 ∙−1.454 ∙+ 91.257(5)−2.035 ∙+ 94.006(6)−1.198 ∙+ 90.794(7)74.627 ∙+ 26.581(8)

The formula of the Special One game is:

The formula of the Final Countdown game is:

In general the fitting results match the growth andaging process of human. Abilities grow as age increases
until maturity. Then after 30 years old the abilities
gradually decrease. This models of the first three games
can be used as a judge of physical abilities such as speed,
mobility, balance, etc. Scores below the model output

Age
Age

Age

Age

And the formula of The Remember game is:Fig. 2. The score-age model.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan650

Yan-Xin Liu, Massimiliano Leggieri, Henrik Hautop Lund

2-m in M oto Tiles
Body-Brain test

Perform ance
analysis

Risk analysis

Recom m endation

A utom atic
personalised
training protocol

Fig. 3. Flow diagram of the Moto BBA test and analysis.

Note that all the four score-age models are first orderpolynomials. This is because both children and seniors
performed at a relatively low level, which means certainscores are connected with both very low and high ages.
Thus, higher orders of polynomials would easily cause
overfitting. When more children and teenager samples
are available, the model can be further improved by
dividing it in two parts for fitting: a youth part wherescores have a positive correlation with age, and an adultpart where scores have negative correlation with age.

Fig. 4. The Moto BBA analysis and recommendation.

Acknowledgements
The authors would like to thank all participants of thisexperiment. Part of the work was produced as part of the
EU Horizon 2020 project REACH under grant agreement
no. 690425.
References
1. Japan Statistical Yearbook 2018, Statistics Bureau, Ministryof Internal Affairs and Communications, 2018.
2. D.E. Bloom, A. Boersch-Supan, P. McGee, A. Seike,
Population aging: facts, challenges, and responses, Benefits andcompensation International, 41 (2011) 22.
3. http://www.moto-tiles.com/
4. H.H. Lund, J.D. Jessen, Effects of short-term training ofcommunity-dwelling elderly with modular interactive tiles,
GAMES FOR HEALTH: Research, Development, and ClinicalApplications, 3 (2014) 277-283.
5. H.H. Lund, Play for the elderly-effect studies of playful
technology, in International Conference on Human Aspects of
IT for the Aged Population (2015), pp. 500-511.
6. H.H. Lund, J.D. Jessen, Effect of playful balancing traininga pilot randomized controlled trial, 2013 ISAROB, (2013).
7. Y. Liu, H.H. Lund, L. Wu, Playful Cognitive Training with
Physical Interactive Tiles for Elderly, in 2018 International
Conference on Information and Communication Technology
Robotics (ICT-ROBOT) (2018), pp. 1-4.

4. Discussion & Conclusion
The perspective of the tests when these correlations havebeen verified as above, is that we will be able to perform,
for instance, test, risk analysis and recommendation for
motivating treatment, as indicated in Fig. 3.
This may be incorporated in the Moto Tiles as a tests,
which provides the analysis and provides theautomatically generated training protocols indicating
which Moto Tiles games to play and for how long inorder to improve on the abilities that may berecommended to improve upon. An initial example ofsuch an integration into the Moto Tiles system is shown
Fig. 4.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

651

Extracting Co-occurrence Feature of Words for Mail filtering
Seiya Temma

Graduate School of Sciences and Technology for Innovation, Yamaguchi University, 1677-1 Yoshida
Yamaguchi, Yamaguchi 753-8512, JAPANManabu Sugii

Faculty of Global and Science Studies, Yamaguchi University,1677-1 Yoshida
Yamaguchi, Yamaguchi 753-8541, JAPAN
Hiroshi Matsuno

Graduate School of Sciences and Technology for Innovation, Yamaguchi University, 1677-1 Yoshida
Yamaguchi, Yamaguchi 753-8512, JAPANE-mail: p040de@yamaguchi-u.ac.jp, manabu@yamaguchi-u.ac.jp, hmatsuno@yamaguchi-u.ac.jp

Abstract
Spam mail filters often take advantage of appearance frequency of words in a text for mail classification. However
the appearance frequency is one of the most important attribute information with which the mail can be characterized,
not a few spam mails can not be distinguished with only the appearance frequency of words. In order to search new
attribute information to characterize and classify the mails, we analyzed relationship between words in a text of mails
by text data mining. Also we visualized the word network by the co-occurrence and multi-dimensional scaling
analysis with the jaccard coefficient in real mails. The co-occurrence network analysis showed important wordconnections with noun and verb in the same kinds of mails. Multi-dimensional scale analysis showed some wordclusters extracted from the same kind of mails.
Keywords: mail-filtering, attribute information, text mining, co-occurrence network.

1.

filtering4. The system uses appearance frequency of eachword and its sequence patterns in a mail body. Thesefeatures (the appearance frequency and its sequence
patterns) are attributes of words, and these attributes cancontribute to characterize and classify each mail.
We focused on three advantages for mail filtering
method based on the attribute information5 as follows;
1. reducing information
2. protection of private information and privacy
3. easy apply to multi-language
For example, amount of information of a mail body canbe reduced to about 1/4 by converting each word into asingle character as an attribute symbol represented its
appearance frequency. Then, it is not easy to recover the

Introduction

We are still getting many spam mails from the Internet,
nevertheless a lot of kinds of mail filtering system havebeen developed by using database or machine learning
based on Bayes’ theorem1-3. It seems kind of like “a catand-mouse game” because spam mail senders release
new spam mails by taking advantage of vulnerability ofa new filtering method based on features extracted frompast spam mails.
We investigated the method for characterizing mails
with machine learning, and we developed a new filteringsystem which could break out of the bad loop of “a catand-mouse game” by using features in ham mails for© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

652

Seiya Temma , Manabu Sugii, Hiroshi Matsuno

2.1. KH coder

original mail body from the converted attribute symbols.
Finally, it is not necessary to process natural language
analysis by this method because the attribute conversion
does not depend on specific language. As a result, it iseasy to apply the system to any languages.
Some attributes of words like appearance frequency,
a part of speech, tf-idf value, cosine similarity and theJaccard index are known as important factors tocharacterize words and documents, and they are applied
to mail filtering system to characterize spam mails. But it
is still difficult to classify some kinds of spam mails
similar to ham mails like unsolicited mails about “Online
dating service”.
In this study, we are trying to find better attributes forthe spam mail filtering by text mining. We consideredthat it is possible to extract the new attributes which cancluster the same kind of mails, and we focused on a cooccurrence network of words as the one of the attributes.

KH coder6 is a free software for analyzing text data
statistically. And it was produced to analyze various
social survey data such as free description ofquestionnaire, interview article, newspaper article etc.
KH coder has functions of aggregating and searchingwords in data, and multivariate analysis and visualizationby using R and morphological analysis module, Chasen
or MeCab. The Jaccard index was calculated by KH
coder, and a part of speech of each word was also
identified with MeCab in KH coder. We visualized cooccurrence networks and words distribution bymultidimensional scaling with the Jaccard index.
2.2. Co-occurrence network
Co-occurrence network is a network diagram composedof edges and nodes with the Jaccard index over a
threshold. Some clusters are usually formed in the cooccurrence network which is produced with the words
from mail bodies. The words in the cluster represent astrong relationship and similarity between these words inthe sample mails.

2. Method of Searching attributes
In order to find new attributes that can remove typicalspam mails or perform high accuracy mail filtering, itneeds to search not only characteristics of words in mails,
but also characteristics of groups based on the connectionof words in mails. Thus, we visualized a co-occurrence
network with the Jaccard index and analyzed clusters
extracted by multidimensional scaling with it. The
Jaccard index is represented by the equation (1) and it
indicates co-occurrence intensity between two words inthis study.
( , )=

| ∩ |
,
| ∪ |

2.3. Multidimensional scaling
Multidimensional scaling is a method to visualize
similarity of individual elements of dataset. Similarity
between each element is regarded as distance of them,and all elements are laid on the N-dimensional spacedepending on the distance without contradiction. As aresult, high similarity elements are laid from a shortdistance on the space. In this study, we applied theJaccard index to the similarity between words in sample
mails.

(1)for given sets A and B.
The Jaccard index between each two words arecalculated from a number of mails which contain thesewords according to equation (1). In both methods, cooccurrence network analysis and multidimensional
scaling, some clusters of words are visualized on two
dimensional space with similarity values between two
words. The result of the visualization is useful to grasp
the whole image of the relationship between each wordin sample mails.

2.4. Sample mailIn this study, we used the 2007 TREC Public Spam
Corpus7. It is composed of 30,338 mails (spam: 50,199
mails, ham: 25,220 mails) accepted from 2007/4/4 to
2007/7/6. Two thousand mails were extracted from eachspam and ham partial set of TREC07 as a sample mail set,
and input them into KH coder.
3. ResultFig. 1 shows a co-occurrence network with all words
from whole TREC07 mail samples. There are many

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan653

Extracting Co-occurrence Featureclusters composed of co-occurrence words (the Jaccard
index >= 0.3) in the same kind of sample mails. For
example, the words extracted from the mails about
software development or advertising mails or having thesame topic mails show a tendency to make one cluster.
But it is too complicated to see details of each clustercomponents in Fig. 1 because too many words are shownin the network, and the layout of words in each cluster is
too small.
Fig. 2 shows the re-construction of the co-occurrence
network with only nouns from ham mails. It has two
major clusters, A and B. The words in the cluster A and
B are extracted from the mails with a common mailsignature and system log announcement, respectively.
This result shows that each cluster in co-occurrence
network is composed of the words extracted from similarkinds of mails. In other words, a mail can becharacterized with the co-occurrence word set which
composes such clusters in the co-occurrence network.
Fig. 3 shows the re-construction of the co-occurrence
network with nouns and verbs from ham mails. It has still
two major clusters, A and B extracted from the same mailset. However these clusters have nouns and verbs (redunder line) as well, a lot of co-occurrence connections
between nouns and verbs are shown in the cluster A. But
there is only one verb in cluster B because the mails
which compose the cluster B contain few verbs in the firstplace. This result suggested that the co-occurrence
network between nouns and verbs might indicate animportant feature to characterize a certain mail.
Fig. 4 shows the result of multidimensional scaling.
Each word is distributed on the 2-dimensional space
which has unknown coordinates, and high similarity
words are located at the close region. We focused on
three regions and words in these regions, A, B and C. The
words in region A, B and C are extracted from the mails
about CNN news, the mails with a common mailsignature and the system log announcement, respectively.
When nouns and verbs are distributed on 2dimensional space by multidimensional scaling (Fig. 4b),
the verbs are located around the center region, and they
are close to the region B. Fig. 4c shows the result ofmultidimensional scaling without the words from thesystem log announcement. The distribution of the words
changed completely but verbs were still located aroundthe center region.

Fig. 1. Co-occurrence network of TREC07 mail sample

A

B

Fig. 2. Co-occurrence network with only nouns from ham
mails

B

A

Fig. 3. Co-occurrence network with nouns and verbs fromham mails© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan654

Seiya Temma , Manabu Sugii, Hiroshi Matsuno

A
A
C
C
B
B

a : only nouns.

b : nouns and verbs

c : nouns and verbs without
system log announce mails

Fig. 4. The distribution of words on 2-dimensional space by multidimensional scaling

4.

like “Online dating service” because these mails have
similar use of words in ham mails. We will plan how toutilize it for mail filtering as our future work.

Discussion

Most of all mail filtering system try to characterize mails
as only two categories, spam or ham. Of course, it is thevery simple method for the mail filtering but it might bealso difficult process to find features because there are
too many kinds of mails to characterize as two categories
in both spam and ham. In other words, it is very difficult
to find common features from all spam mails or ham
mails.
The result of this study shows that it is possible togroup mails into several types with the combination ofco-occurrence words because the words extracted fromthe same kind of mails cluster easily in the co-occurrence
network. Then, the words with the low level cooccurrence index can be visualized as a cluster in the cooccurrence network by removing the mails which containthe high level co-occurrence words8.
Some kinds of spam mails like the system log
announcement and advertising mails which contain only
URL can be removed by checking the co-occurrence
connection between nouns and verbs. They have the quite
few connections between them.
We do not have a concrete measure to use the cooccurrence connection set between nouns and verbs as
the attribute but we considered that the combination ofthem is very important to characterize the type of mails.
At least, it could be better attribute than an appearance
frequency of words for the filtering of unsolicited mails

5.

Acknowledgements

This work was supported by JSPS KAKENHI Grant
Number JP16K12438.
6.

References

1. P. Graham, “A Plan For Spam”, 2002:
http://www.paulgraham.com/spam.html
2. P. Graham, ”Better Bayesian Filtering”, 2003:
http://www.paulgraham.com/better.html
3. bsfilter-bayesian spam filter:
https://ja.osdn.net/projects/bsfilter/
4. M. Sugii, H. Matsuno, Decision Tree Representation of
Spam Mail Features by Machine Learning, 2007(16(2007DPS-130)), 183-188, 2007-03-01,2007.
5. N. Fujii, M. Sugii, and H. Matsuno, Finding the
Candidates of Attributes in Mail Words Filtered by
Bayesian Method, IEICE Technical Report, vol. 116, no.
525, MSS2016-89, pp. 43-48, 2017.
6. KHcoder：http://khc.sourceforge.net/
7. TREC 2007 Public Corpus：
http://plg.uwaterloo.ca/~gvcormac/treccorpus07/
8. S. Temma, M. Sugii, H. Matsuno, Searching Attribute
Information for Mail Filtering based on Text Mining, ITCCSCC2018 proceedings, 596-599, 2018.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

655

Learning Style Classification with Weighted Distance Grey Wolf Optimization
Duangjai Jitkongchuen

College of Innovative Technology and Engineering
Dhurakij Pundit University, Thailand
Piyalak PongtaweviratCollege of Innovative Technology and Engineering
Dhurakij Pundit University, Thailand
E-mail: duangjai.jit@dpu.ac.th, piyalak.pot@dpu.ac.th
http://cite.dpu.ac.th

Abstract
This research aims to improve the performance of multiclass classification by using grey wolf optimization
algorithm. The proposed algorithm presents a solution to improve the grey wolf optimization performance using
weighted distance and immigration operation. The weight distance is used for the omega wolves movement isdefined from fitness value of each leader. The proposed technique is based on learning style prediction which
addresses multiclass classification problem. The results showed that the proposed technique obtained the higher
accuracy rate than other classification techniques.
Keywords: Data mining, Classification, Grey wolf optimization algorithm, Learning styleclassification problems and predictions in various areas.
Examples of these techniques are decision trees (DT),naïve bayes (NB), artificial neural networks (ANN) andsupport vector machines (SVM)2. In addition, thealgorithm in the group of evolutionary algorithm has
been also introduced to solve the classification problem3.
However, in each classification techniques have many
advantages and disadvantages. As a result, the mostappropriate technique for each dataset and its data types
should be examined in order to obtain the appropriate
built classification model for specific problem.
The remainder of this paper is organized as follows.
In Section 2, learning domain, learning style and theoriginal grey wolf optimizer algorithm is provided. In
Section 3, the proposed algorithm weighted distancegrey wolf optimization is described. In Section 4, aperformance evaluation of our proposed methods isdone compared to other classification techniques (cAntMiner, PART, C4.5) on dataset of learners’ information

1. Introduction
The one of the most important areas in machinelearning is classification. The classification goal is tofind determinations of classes to un-classified andunseen instances (data samples) based on information
previously learned. Solving classification problems
intends to design good classifier(s) which make rightdeterminations as often as possible1.
Generally, the classification techniques can bedivided into two categories. The binary classification
identifies the output class from two possible classes.
The second category is the multiclass classification
which predicts the result from multiple classes. Hence,
the techniques to improve the classification performancein terms of accuracy for multiclass classification should
be investigated.
There are many commonly used classification
techniques that have been used to deal with© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan656

Duangjai Jitkongchuen, Piyalak Pongtawevirat

where t is the iteration number, X is the grey wolf
position and X p is the prey position. The vectors A
and C are coefficients are calculated as follows

related to classification model prediction. Finally, in
Section 5, the conclusions will be discussed.
2. Related workA

2.1. Learning Domain and Learning Style
The cognitive learning which refers to humanknowledge is one of the learning domains for human4.
The cognitive learning focuses on individual level ofknowledge. The individual learning is difference which
is based on personal learning style. As a result,everyone prefers different kind of learning content
based on their learning style. This conduces to moredeveloping learners’ abilities.
The learning style refers to an individual's uniqueapproach to learning based on strengths, weaknesses,
and preferences. Several researchers have been worked
to define the learning style for learner. Many learning
styles such as the Felder-Silverman Index of Learning
Styles have been proposed5. There are two approaches
to identify a learning style which are questionnaire andcomputer model prediction. Firstly, the questionnaire
approach needs the learner to fill in their information
respecting the available model6. Secondly, the learner's
learning style is determined using computational
techniques and the learning style prediction model iscreated based on available information.

(1)

D

C X p(t ) X(t )

(2)

C

(4)

2r2

X1

X ( t 1)

X2
3

X3

(5)

where X 1 , X 2 , X 3 are determined as in Eq. (6)-(8),respectively.

The grey wolf optimizer algorithm (GWO)7 is
intelligence algorithm which simulates the hierarchy
structure and hunting mechanism of wolf pack. The
social dominant hierarchy of the wolf pack is divided
into four levels. From top level to bottom grade arealpha (α) wolves, beta (β) wolves, delta (δ) wolves andomega (ɷ) wolves. An ability of grey wolves has torecognize the positions of prey and to encircle them.
The hunting (optimization) is guided by alpha, beta anddelta. The omega wolves are iteratively improved byfollowing the other dominant wolves.
Grey wolves encircle prey during the hunt. The
mathematically of model encircling behavior as follows
X p(t ) A D(3)

The value of a is decreased linearly from 2 to 0 over thecourse of iterations and r1 , r2 are random vectors in therange [0, 1]. The vector C is used to furnish a random
weight to mention attractiveness of prey is a random
value in the range [0, 2].
Hunting strategy of the grey wolves can bemathematically modeled by approximating the prey
position with the help of alpha, beta and delta solutions.
The first three best solutions get so far and force theother search agents (including the omegas) to update
their positions according to the position of the three best
search agents. Each wolf can update their positions byfollowing equations.

2.2. Grey wolf optimization algorithm

X ( t 1)

2a r1 a

X1

X

A1 D

(6)

X2

X

A2 D

(7)

X3

X

A3 D

(8)

where X , X , X are the first three best solutions at agiven iteration t, A1 , A2 , A3 are determined as in Eq. (3),and D , D , D are determined as in Eq. (9)-(11),respectively.

D

C1 X

X

(9)

D

C2 X

X

(10)

D

C3 X

X

(11)

where C1 , C2 , C3 are determined as in Eq. (4). The
parameter a that controls the tradeoff between
exploration and exploitation is linearly updated to range
from 2 to 0 in each iteration as shown in Eq. (12)© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

657

Learning Style Classification

a

2 t

2
MaxIterremoved by mean of weak member is member that havefitness value less than the average fitness value of pack.
The all process is repeated until the termination
criteria are convinced or a predefined number of
iterations are achieved.

(12)

where t is the iteration number and MaxIter is the total
number of iteration.
3. The proposed approach
The process of traditional grey wolf algorithm has
only one wolf pack and has opportunity of local
optimum problems. The proposed algorithm presents
grey wolf algorithm with more than one pack andintegrate the immigration operation between pack to
improve the exploration and exploitation.
The position of omega wolves in traditional grey
wolf algorithm are identified by the position of threeleaders, alpha, beta and delta wolf while the position ofomega wolves in proposed algorithm are identified
using weight value difference of three leaders. The
omega wolves positions are updates by following
equations.

X ( t 1)W1 X 1 W2 X 2 W3 X 3
3(13)

Fig. 1. The framework of learning style classification withweighted distance grey wolf optimization.

where W1 ,W2 ,W3 is the weight value of alpha, beta anddelta wolf, respectively. The weight value of each leader
are calculated as follows

W1
W2

W3

f

f
f

f

f
f

f

f

f

f
f

f

4.

Experimental results

The learning style can be classified into 4 class,
Active and Reflective, Sensing and Intuitive, Visual and
Verbal, and Sequential and Global. Nevertheless, somestudents can grouped as multiple classes so this class oflearning style is classified as others. The experimentused data source from published research work on
learning style classification in university students8. The
attributes of data consists of learners’ information
related to classification model prediction such as gender,
age, educational background, current grade point
average, major unit’s enrolment and major unit’s grade
point average. There are 400 records in this dataset andthe information related to this dataset can be seen from
Table 1.

(14)
(15)
(16)

where f , f , f is the fitness value of alpha, beta anddelta wolf, respectively.
The immigration operation is active in next process
to use to distribute value to expand the search space.
The omega wolves which low fitness value would be
immigrated to create new pack and generate new wolves
member to fill up the old pack. The center value of packis used to generate the new wolves members for sustain
the boundary of pack.
The immigration has an effect on continually
increasing pack amount in each iteration. Therefore,pack which have maximum weak member would be

Table 1. Datasets used in experiments.
Learning Style
Active and Reflective
Sensing and Intuitive
Visual and Verbal
Sequential and Global
Others
Total

Instances
66
81
122
67
64
400

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan658

Duangjai Jitkongchuen, Piyalak Pongtawevirat

The training data have three hundred randomly
selected records and the remains one hundred records
were used for testing data. The classification model isbuilt by using training data and evaluates performanceof model by using testing data.
The proposed algorithm was evaluated performanceswith cAnt-Miner, C4.5 and PART. The metric todetermine the performance of the classification is basedon accuracy rate. The experimental results are shown in
Table 2.
The experimental results shown that the proposed
algorithm can perform better in terms of accuracy ratecompared to other classification techniques. In addition,
it performed better than C4.5 and PART. The proposed
algorithm can improve classification performance byanalyzing and predicting the continuous attributes in thedataset so can used to predict the dataset, learning styleof learner, efficiently.

improve the performance of multiclass classification byusing weighted distance grey wolf optimization
algorithm and integrate immigration operation. This
which leads to generate new candidate individuals inorder to enhance the classification performance. The
results from the experiments based on learning styledatasets have shown that the proposed technique canperform better in terms of accuracy rate compared toother classification techniques.
References
1. C. M. Bishop, Pattern Recognition and Machine
Learning, (Springer-Verlag GmbH, Berlin, Germany,2006).
2. X. Wu, et al., Top 10 Algorithms in Data Mining,
Knowledge and Information Systems, vol. 14(1) (2008),
pp. 1-37.
3. D. Jitkongchuen, and A. Thammano, A Hybrid DE-ACO
Approach for Discovering Classification Rules (Proc.
International Technical Conference on Circuits/Systems,
Computers and Communications (ITC-CSCC), 2012).
4. B.S. Bloom, Taxonomy of educational objectives: theclassification of educational goals (David McKay, 1956).
5. R. M. Felder and L. K. Silverman, Learning andTeaching Styles in Engineering Education, Engr.
Education, vol. 78(7) (1988), pp. 674-681.
6. A. Klašnja-Milićević, B. Vesin, M. Ivanović, and Z.
Budimac, E-Learning personalization based on hybrid
recommendation strategy and learning style identification,
Computers & Education, vol. 56(3) (2011), pp. 885-899.
7. S. Mirjalili, S.M. Mirjalili and A. Lewis, Grey Wolf
Optimizer, Advances in Engineering Software, vol.69(2014), p.46-61.
8. W. Paireekreng and T. Prexawanprasut, An Integrated
Model for Learning Style Classification in University
Students Using Data Mining Techniques (Proc. 12th
International Conference on Electrical Engineering/Electronics, Computer, Telecommunications andInformation Technology (ECTI-CON), 2015).

Table 2. Comparison of accuracy rate.
DatasetA
B
C

cAntMiner
86.75
85.50
84.50

C4.5

PART

Proposed

82.00
71.70
76.92

73.44
70.59
73.56

92.15
91.45
91.75

Fig. 2. A comparison of accuracy between proposed algorithm
and other classification techniques.

5. Conclusions
The multiclass classification seems to be difficult toimprove the performance of the built classification
model. One of the novel approaches in recent years is
bio-inspired algorithm. Grey wolf optimization
algorithm is one of bio-inspired algorithm which good
capacity to distinguish between an exploration phaseand an exploitation phase. This research aims to© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

659

The Relevance Research among Destination Image, Brand Trust and Satisfaction: Tainan
Festival in Taiwan as Example
Shang-Hui Li

Department of Food and Beverage Management, Far East University, Taiwan (R.O.C.)
E-mail :a0956358700@gmail.com
Shu-Fang Hsu

Department of Food and Beverage Management, Far East University, Taiwan (R.O.C.)
Yi-Tai Shang

Department of Food and Beverage Management, Far East University, Taiwan (R.O.C.)

Abstract
In recent years, all local governments of Taiwan have held sightseeing festivals to enhance their popularity,
promote their economy, to establish their brand. Tainan, the second biggest city in southern Taiwan, also promotes
the various agricultural products and sightseeing to serve customers as the region's annual event. This study took
Tainan Festival as an example to explore the relevance of the destination image, brand trust, and satisfaction. The
participants for those who took part in 2017 Tainan Festivals, from May to July, were the main subjects on thisstudy. 500 questionnaires were issued, 465 valid questionnaires collected, 93%. The finding results showed that thedestination image of Tainan festival activities had a positively effects on satisfaction, brand trust had a positiveeffects on satisfaction, destination image had a positive effects on brand trust, brand trust had the intermediaryeffect to the satisfaction and destination image.

Keywords: Festival, Destination Image, Brand Trust, Satisfaction

1. Introduction

Yujing, Nanxi, Nanhua, Danei and Zuozhen from Juneto July and Baihe Lotus Festival from July to August etc.
It is hoped to promote the development of local andurban tourism culture through the integration of
resources in industry, tourism and culture. This study
took Tainan festivals as an example to explore
destination image, brand trust and satisfaction in orderto understand the consumer's satisfaction with thefestival activities. Did the first impression and the
feeling of participating in the event affect theirsatisfaction with the activity? Will it affect thesatisfaction of consumption on the festival brand if
consumers generate trust in the festival brand?

In recent years, the Taiwan government has actively
promoted the development of agricultural products invarious regions with local characteristics andencouraged to hold the festival with combining the local
industry to activate the economy and culture. The
festival also promoted food-related activities which
could increase the economic impact of tourism. In
addition to promote local specialty industries, local
festivals can also take advantage of opportunities toincrease industrial revenues and achieve marketing
goals. Festivals are held in every administrative districtof Tainan, Taiwan every year.
For example, the Dongshan District Farmers’ Day inMay, the Mango Festival was held in conjunction with

2. Literature Review© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

660

Shang-Hui Li, Shu-Fang Hsu, Yi-Tai Shang

Taiwan local governments will like to hold the uniquefolk culture, festivals and industrial cultural activities toattract tourists to consume. It is not only shaping the
local characteristics, but also activating the local
economy, it can also promote the development of thelocal tourism industry for local marketing purposes [1].
The destination image is an important indicator in thevisitors’ eyes [2]. It is also the emotional connection
between the tourist destination and the consumers’
psychology, with positive, neutral and negative feelings
respectively [3].
Destination image is a key factor in the tourism
decision-making process and destination selection
behavior [4] [5]. Destination image contains a
multidimensional oriented construct of cognition,
emotion, and intention [6] [7]. Qu, Kim, and Im argued
that destination brand imagery should contain uniqueimages, and unique images refer to images withimpressive attributes or traits [8]. In recent years, foodtourism has developed rapidly and diet has become amotivation for people to travel [9]. Therefore, thedestination food image and the destination tourist image
are parts of the destination image. Morgan and Huntdefined trust as the reliability and costiveness of trading
partners [10]. The transaction will take place under the
consumer clearly feels the sincerity of the other party
and has a sense of security for the brand. Arjun &Morris believed that consumers are willing to take theinitiative to trust and trust the products offered by thebrand [11]. The idea of customer satisfaction was first
proposed by [12]. Cardozo believed that customer
satisfaction was the degree of customer satisfaction withthe product's expectations and results [13]. Day
considered customer satisfaction as a perception inwhich customers evaluate the pre-purchase expectations
and post-purchase actual performance [14].

Destination
Image

H1

H3

Satisfaction

Brand Trust

H2

H4
Figure 1. Research Structure

Various hypotheses are proposed based on the purposeof this study, and the results of the data analysis willverify if the hypothesis was true. The research
hypothesis was as follows:
H1: destination image has a positive impact on
satisfaction.
H2: brand trust has a positive impact on satisfaction.
H3: destination image has a positive impact on brandtrust.
H4: The brand trust has a mediating effect between
satisfaction and destination image.
Destination image refers to the consumer's perception
and thoughts about the activity. The scale was based on
the questionnaires of G’omez, Lopez and Molina (2015),
with a total of 16 questions. Brand trust refers to theconsumer's trust to the brand. The scale was based on
the questionnaires of Chaudhuri and Holbrook (2001), a
total of 4 questions. Satisfaction refers to the consumers’satisfaction in participating this activity. The scale wasbased on the questionnaires of Lu, Chi and Liu (2015),
with a total of 4 questions. The study objects and scopewere mainly for the consumers who participated in the
Tainan festivals from May to July, 2017. 500
questionnaires were distributed under convenient
sampling method, and 465 valid questionnaires (93%)
were collected. The version 12.0 of SPSS (Statistical
Package for the Social Science) was used for empirical
data analysis.

3. Research Method
This study explored the relevance among destination
image, brand trust and satisfaction. The research
structure was constructed after basing on the researchbackground, research purposes and related literature, as
shown in Figure 1.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

661

The Relevance Research among

4. Running Heads

5. Conclusions and Recommendations

The study used regression analysis to analyze if
destination image has a significant correlation tosatisfaction. Destination image predicted the 42.7%
explanatory power of satisfaction, F=347.342, P=.000,
reaching a significant level. It was shown that the
explanatory power had the statistical significance, andthe result of the coefficient estimation indicated that thedestination image can effectively affect satisfaction. The
beta coefficient was 0.655 (t=18.637 p=0.000) whichindicated that the higher the consumer's destination
image, the higher the satisfaction. The result supported
hypothesis 1.
The study examined if the brand trust of Tainan’s
festival activities affected satisfaction (Table 2). Brand
trust predicted the explanatory power of 27.2% ofsatisfaction, F=177.727, P=.000, reached a significant
level which indicated that the explanatory power was
statistically significant. Therefore, the brand trust caneffectively affect the satisfaction. The Beta coefficientreached 0.527 (t=13.331 p=0.000), indicated that thehigher the consumer's brand trust, the higher thesatisfaction. The result supported hypothesis 2.
The study examined if the destination image of thefestival affected the brand trust, predicting the
explanatory power of the brand trust 30.4% with thedestination image, F=202.698, P=.000, reached asignificant level which was shown that explanatory
power had statistical significance. The results of thecoefficient estimation indicated that the destination
image can effectively affect the brand trust, and the betacoefficient was 0.552 (t=14.237 p=0.000), indicated thatthe higher the consumer's destination image, the higher
the brand trust. The result supported hypothesis 3.
Regression Analysis was used to verify if the brand trust
had a mediating effect on destination image andsatisfaction. Incorporating destination image, brand trustand satisfaction into regression mode, shown that
destination image and brand trust had no significant
effect on satisfaction, and then compare β value, thelatter β=0.527 is smaller than the former β=0.655.
Overall, the results supported hypothesis 4, whichindicated that the brand trust had a mediating effect ondestination image and satisfaction.

The higher on destination image, and the higher on the
consumer's satisfaction. The industry can strengthen thedestination image of festivals of Tainan, for example, alarge number promotion on media, online, Facebook, IG
and Line to enhance consumer impressions of festivals
which could help consumers recognize their destinations
and increase satisfaction.
The higher on destination image, the higher on the
consumer's brand trust, and the better regional image
and reputation could enhance the trust of visitors.
Some suggestions could be offered by this study such as
local government and the Council of Agriculture, COA,
could cooperate with public welfare organizations toenhance the trust of tourists in Tainan area.
The higher on the brand trust of consumers in Tainan’s
festivals, the higher on their satisfaction. It may be thatthe activities are mostly organized by government
agencies and have a certain sense of trust for consumers.
It could be believed that more consumers could be
attracted to the festivals if local farmers could be invited
to support and publicize their products in the future
activity. It is difficult to explore one by one item due tomanpower considerations. It is recommended that thefollow-up researchers can add more different variables
to conduct the further study, so that the research can be
more complete.
References
1. Chen, P. T., Chen, F. D. & Pei, L., Evaluation of PlaceMarketing Effects Induced by Festivals. Journal of Sport,
Leisure and Hospitality Research. 9(3), 2014, pp. 23-28.
2. Hunt, J. D., Image as a factor in Tourism Development.
Journal of Travel Research. 13, 1975, pp. 1-7.
3. Woodside, A. G. & Lysonski, S., A general model oftraveler destination choice. Journal of Travel Research.
17(4), 1989, pp. 8-14.
4. Echtner, C. M. & Ritchie, J. R. B., The meaning andmeasurement of destination and the influence ofgeographical location upon that image. Journal of Travel
Research. 14(4), 1991, pp. 18-23.
5. Baloglu, S. & Mangaloglu, M., Tourism destination
images of Turkey, Egypt, Greece and Italy as perceived
by US-based tour operators and travel agents. Tourism
Management. 22, 2001, pp.1-9.
6. Beerli, A. & Martin, J. D., Factors influencing destination
image. Annals of Tourism Research. 31(3), 2004, pp.
657-681.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan662

Shang-Hui Li, Shu-Fang Hsu, Yi-Tai Shang

7. Prayag, G. (2007). Exploring the relationship between
destination image and brand personality of a tourist
destination: An application of projective techniques.
Journal of Travel and Tourism Research. 7(2), 111-130.
8. Qu, H., Kim, L. H. & Im, H. (2011). A model ofdestination branding: Integrating the concepts of thebranding and destination image. Tourism Management.
32(3), 465-476.
9. Lin, Y. C., Pearson, T. E. & Cai, L. A. (2011). Food as aform of Destination Identity: A Tourism Destination
Brand Perspective. Tourism and Hospitality Research.
11(1), 30-48.
10. Morgan, R. M. & Hunt, S. D. (1994). The commitmenttrust theory of relationship marketing. Journal ofMarketing. 58, 20-38.
11. Arjun, C. & Morris, B. H. (2001). The Chain of Effects
from Brand Trust and Brand Affect to Brand
Performance: The Role of Brand Loyalty. Journal ofMarketing. 65, 81-93.
12. Drucker, P. F. (1950).The new society. William
Heinemann, Ltd.
13. Cardozo, R. N. (1965). An experimental study ofcustomer effort, expectation, and satisfaction. Journal ofMarketing Research. 24, 244-249.
14. Day, R. L. (1984). Modeling choices among alternative
responses to dissatisfaction. In Thomas C. Kinnear (ed.).
Advances in Consumer Research. 496-499.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

663

Third-party Logistics Service Supply: Catering Service Trade as Example
Shu-Fang Hsu

Department of Food and Beverage Management, Far East University, Taiwan (R.O.C.)
E-mail:h99582004@gmail.com
Shang-Hui Li

Department of Food and Beverage Management, Far East University, Taiwan (R.O.C.)

Abstract
With the development of third-party logistics, refrigeration technology, and transportation which made the food to
obtain easily, and improve the quality of food preservation. Now, the catering chefs just ordered his needed foodsby line, telephone, fax, or fill the order list etc., to inform the third-party logistics suppliers and there will besomeone assigned to the designated location which could save purchasing time and catering chefs don’t need torush around to procure foods. This study analyzed the collected information through literature, in-depth interviews,
and observation methods. Third-party logistics suppliers conducted the information, logistics, and cash flow by thedifferent deal due to the different nature of suppliers. Third-party logistics suppliers could be exhaustive on thequality management, logistics activities, interactive mode of trust mechanism, which could not only win the good
reputation by the catering chefs, but the professional core competence.
Keywords: Supply Chain, Third-party logistics Service Supplier, Cateringchanges in the dishes with the development of the thirdparty logistics supply. In this case study, Catering Chef
Lee (hereinafter refers to as Chef Lee), in PingtungCounty, Taiwan, has been engaged in catering service
for more than 40 years. And all his partners including
vegetables supplier, meat supplier, and seafood supplier
are referred to the third-party logistics service.

1. Introduction
Recently, the quality of food preservation has beenenhanced by the frozen technology and the developed
transportation. The demand for refrigeration facilities
are even more urgent due to the traditional catering
service is held outdoors. With the continuous
development of cryogenic equipment, from refrigerator
to frozen car, not only the preservation of food got moresecure, but the availability of fresh foods throughout thecountry [1]., and this was attributed to the developmentof low-temperature technology and transportation [2].
Comparing the past time, catering chefs needed to rusharound the procurement of foods, now, catering chefs
obtained simply the food by line, telephone, fax, fillorders, etc., telling third-party logistics suppliers and theneeded foods will be delivered to the designatedlocation which could save a lot of procurement time.
Fresh foods are made easier to obtain and there are more

2. Literature Review
Table-keeping activities have long existed in Taiwanesesociety. The general discussion will be explained by themeaning of "co-food": a group of people prepare dishes
together and enjoy dishes together, and enhance therelationship between family members and friends bysharing food, or increase popularity and popularity.
Actually, "Do table" was not only to serve the gooddishes for the guests, but contained many taboos andetiquette. Different dishes and different etiquette are for© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

664

Shu-Fang Hsu, Shang-Hui Li

different theme. Each catering service had its certain
rules including tableware, tables and chairs furnishingsetc. Even the purpose of catering service can be seenfrom the menu. And the owner did not require to pay therental space cost except the catering expense [3].
American Logistics Management Association (1998)
proposed that logistics was a part of the supply chain
process which focused on items, services, and therelated information. In other words, logistics refers to
the effective circulation, storage planning, execution,and management from starting to consumption to meetcustomer’s requirements. Taiwan Association of
Logistics Management (TALM, 1996) considered thatlogistics as an entity's physical circulation activities.
During the process of circulation, it could create value
to meet the needs of customers through the effective
combination of transportation, storage, packaging,
distribution, and information such as from supply to
production facility, then to distribution facilities, andfinally to the sales market. The different logistics
network was provided through the establishinginternational logistics service system which including
intra-regional and inter-regional maritime, land and airtransport and warehousing complex system. And
gradually complete international logistics network
provided a more diversified international logistics
service options [4]. According to [5]., the so-called
logistics outsourcing referred to the business owners
who organized the operation of logistics operations
within the organization and appointed to the third-party
logistics companies for planning and service which
encompassed the warehouses of the third-party logistics
supplier, used mass or contractual transport, andprovided the business owner with the best logistics
supplier to fulfill their needs. To effectively use thelimited resources was the main reason for logistics
outsourcing, the third-party logistics supplier would
take all the responsibility of transportation, warehousing,or other logistics activities of the company. In addition,
the third-party logistics supplier made their efficientmanagement via the professional logistics solutions, thedeveloped transport networks, as well as a variety of
information technology. [6].argued that third-party
logistics was mainly a collection of all important
logistics services including transportation, warehousing,packaging, material management, and various auxiliary
management [7]. Therefore, it can be argued that

advancements in the field would benefit from thedevelopment of a systematized conceptual baseconsisting of common definitions of important
constructs and variables.
3. Research Method
Triangulation test refers to the same conclusion, using
different methods, in different situations and time, totest different people in the sample, the purpose is tocarry out the conclusions that have been establishedthrough as many channels as possible. Test to obtain the
maximum degree of truth of the conclusion. In
qualitative research, the most typical way to conduct
correlation tests is to combine both interviews andobservations [8]. Observations allow us to see thebehavior of the researcher, and interviews can help us
understand the motivations of their behavior. By
comparing the results of the interviews, we can test therelationship between what the researcher said and what
he did. This study uses semi-structured interview studies
through in-depth interviews, case studies, observations,etc., plus the multiple views of the literature-supported
triangulation. The research framework was shown as
Figure 1. Based on logistics activities between catering
service and third-party logistics supply such as logistics,
money flow, and information flow as well. Meanwhile,
study the supply process between the two sides and thethird-party logistics supply concerning on warehousemanagement of goods （ such as inventory control,
product management, packaging, and inventory
management）, and quality management (good, saveetc.).

Figure 1. Research framework

4. Data Analysis
4.1 On-site Workflow of Catering Service© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

665

The Relevance Research among

This case study explored some basic information as
catering for thirty tables (and a table can be seated forten guests), will start the catering service at 12:30p.m.
there are six infield workers. Usually, the working
members will bring their own knives and wearing
kitchen suits to collect at the scene during 5-6a.m. andChef Lee will bring dry food to the scene. Then, all ofthem will begin their work such as inspect the cook
tools, washing pan and bowl, and other cleaning actions.
Then, wait for all the suppliers to deliver the goods tothe scene. All the delivered goods must be inspected by
Chef Lee personally, and any unqualified foods will be
returned directly to supplier. Supplier will then replace
the qualified food to the scene. Actually, it had a very
low rate of return from the scene due to the food will beinspected by the supplier from the wholesale. Chef Lee
always processed the vegetable first no matter which
one of three suppliers came the first because seafood
and meat took some time to de-ice, however, the live
seafood was left aside for processing. And fruit andsweet soup were cold in the fridge, waiting for dish.
Cleaning was the first step, and then drained. Cutting
was responsible by the six workers, they worked
together to deal with cleaning and cutting. Allvegetables are required to be blanched in a pot of boiled
water for a short time by Chef Lee personally, then,mushrooms, shark's fin, squid, meat, chicken, andfinally seafood in order. Cold dish was followed. As forchicken soup, lobster mushroom soup, and steamed
groupers were produced and insulated with the different
steamers to cook at the same time. The last one was
sweet soup.

customers of Chef Lee were from Linbian Township,
Pingtung, and all adjacent townships. The living
seafood supplier, therefore, could be supplied within 1
to 2 hours. The paper orders will be transformed into
electronic files by the seafood supplier after receiving
the order from Chef Lee. Vegetable order was
confirmed by paper list and telephone. Meat supplier
made a special notebook to confirm the order details
instantly. Usually, the seafood needed to be frozen
under -18ºC after seafood supplier purchased from thewholesale supplier and some are stored in 5-0ºC. Living
food must be delivered to catering scene within one
hour after purchasing from wholesale supplier. As formeat, all the meat must be frozen under at -18ºC
according to the various parts of meat. Fruit andvegetable supplier provided an on-site delivery of theiritems to the designated scene after purchasing fromwhole market at the same day.
4.3 Logistics Activities
Chef Lee needed to give the exact orders to all suppliers
five days in advance. This could give enough time forthe three suppliers to place orders to their upstream
suppliers. For example, seafood for 3 days, meat for 1day, however, vegetable and fruit are at the same day.
Under such arrangement, the error rate for the delivery
time of the above suppliers, therefore, was very low. As
for the inspection, seafood could be checked by its color,
and touched the body by hand if was still alive or not.
Meat could be viewed by its color, and touched by handto make sure there was no sticky feeling. Vegetable
could be checked by the fresh appearance or use handflick to the surface for testing solid sense. Seafood
supplier delivered the frozen and live food throughfreezer at below 5℃ to the catering scene. Meat supplierused refrigerated trucks delivered the meat to the scene.
Vegetable supplier trucked vegetable to the scene atroom temperature. As for the price, all the productprices were based on the price of middle-wholesale.
Chef Lee paid to all the suppliers by cash. The deliveryof third-party logistics service saved a lot of time forcatering service businessmen, and enhanced the qualityof food management.

4.2. Goods Quality Management and Warehousing
Management of the Third-party Logistics Supply
Food can be distinguished from the appearance, shape,
weight, color, smell, taste, and vacuum. Chef Lee
believed that seafood was good as long as it was theliving body while sent to the scene. All the fresh fish,shell, raw meat will be kept at 2-2ºC while they sent tothe catering scene regardless of circulation, trafficking,or vacuum packaging. The low temperature gas storage
method was the most commonly used for fresh fruits
and vegetables. The three suppliers had placed theirorders to their upstream suppliers after Chef Leeordered. Based on the geographical sake, most of

5. Conclusions and Recommendations
The finding results showed that both the third-party
logistics suppliers and Chef Lee can’t be sloppy© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

666

Shu-Fang Hsu, Shang-Hui Li

7. Marasco, A. (2008). Third-party logistics: A literature
review. International journal Production Economics. 113,
127–147.
8. Chen, H. M. (2011). Social science qualitative research.
Wunan Book Publishing Co., Ltd.

regarding the quality requirements and inspection
details of food. Only under this condition, then canmaintain the food safety for guests. The error rate wasvery low for all the suppliers, and they asked themselvesto keep independent quality management to the food
which, therefore, reduced the quality and quantity ofinspection, Chef Lee could focus on his cuisine of
cooking. The ingredients are shipped from the thirdparty logistics service to the catering scene. And all
kinds of food packaging was to protect the product toavoid being contaminated, and facilitate the handling of
warehousing to ensure the correctness of the quantity offood, the quality, the safety of work, and timely supply
to the designated place.
The results also found that the logistics center has thefunction to short the upstream and downstream industry
on circulation process in the vertical integration of
production and marketing, which has the intermediaryfunction could reduce the production and marketing gap,
but also for the horizontal relationship between the same
industry, and the different industry exchange andintegration support, it has become a pivotal player in themodernization path revolution. In addition, theresponsibility attribution would transfer from the
supplier to Chef Lee when all the logistics supplier andChef Lee had completed the inspection. Especially the
double quality requirements, from quality requirementof the third-party logistics supplier to quality
requirement of Chef Lee, were the long-term corecompetence for catering service career.
References
1. Kuo, J. C. & Chung, C. C. (2006). Cool storage type lowtemperature home delivery system. China Hydropower
Refrigeration Air Conditioning Monthly. April, 1-10.
2. Chen, K. H. & Huang, D. H. (2011). Eat, Eat Well, Eat
Cleverly, and Eat Healthfully-Study on the Developmentof Menu Items in Taiwanese Banto. Journal of
Hospitality and Truism. 8(2), 97-126.
3. Lin, X. Y. (2013). Taiwan's table culture. Ph.D. Thesis,
Department of History, College of Liberal Arts, China
University of Culture, Taipei.
4. Su, H. I (2012). Supply Chain Management. Taipei,
Taiwan: Zhisheng Culture Business Co., Ltd.
5. Yeh, C. C. & Lai, M. C. (2005). Logistics and supply
chain management. Taipei, Taiwan: San Min Book Co.,Ltd.
6. Liu, W. L. (2008). Supply Chain Management. Taipei,
Taiwan: Yufeng Information Co., Ltd.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

667

Pedal Scale Control Device for Weighing Food Ingredients or Chemical Materials
Shang-Hui Li, Shu-Fang Hsu, Yi-Tai Shang, Kuan-Ying Chen
Department of Food and Beverage Management, Far East University, Taiwan (R.O.C.)
E-mail:a0956358700@gmail.com

Abstract
Safety and hygiene is one of the most important elements when weighing ingredients or chemical materials. It may
cause cross-contamination when preparing them if the operators conduct the scale button and hold the foodingredients or chemical materials by using both hands. The invention relates to a pedal scale control device
comprising a scale body and a foot unit. The wireless signal on the foot unit can be connected to the scale body tozero and buckle the function without using of both hands. This invention has obtained Taiwan's invention patent,
which is more hygienic and safety for the users when weighing the food ingredients or chemical materials.
Keywords: Scale, nvention, Patent

announced as the statutory weighing instrument that

1. Introduction

should be verified [1]. There are many types of scales,

It is often necessary to use scales to weigh the raw
materials when making baked ingredients, preparing
pharmaceuticals or chemicals material etc., so that they
can be made into uniform specifications. However, it isnecessary to repeatedly use the finger to touch the
button when using the electronic scale which is not only
easy to have health concerns, but also may cause
pollution, especially in pharmaceutical or chemical raw
materials. And the study found a way to solve thisproblem. The study designed for using the pedal control
device to scale food ingredients, chemical, medical or
pharmaceutical materials which will be enable tomeasure the weight without the hands to maintaincleanliness, safety and hygiene. This device has
obtained the patent certificate No. I588458 from Taiwan,
and hopes it will give some contribution to the weighingof raw materials in various industries such as food,chemical and pharmaceutical fields.

and the electronic scales is the most commonly used one
in market, which measures the weight of the object by
the gravity of the object, saving manpower and time.
Furthermore, the common electronic scales are divided
into electronic counting scales, electronic weighing
scales, electronic pricing scales, electronic balances,electronic crane scales, electronic medical scales andelectronic small scales [2]. The weighing of different
electronic scales is different. For example, electronic
scales need precise weighing. Usually, the sensitivity is
1 to 5 digits (0.1g-0.01mg), and the maximum weighingis less than 60kg. However, the electronic platform
scales are used for weighing large goods, and range
from 500kg to 10 tons. In general, electronic scales areequipped with some functions such as returning to zero

2. Pre-case Discussion

which could measure the net weight of articles for the

Scales are closely related to the daily lives of consumers.

users. This function is important for referring to recipes

It will be used in various fields such as warehousing,or cooking etc.
In other word, the weight required for various materials
are generally marked whether it is in chemical,
pharmaceutical or recipe formula. It is often necessary

freight, chemical, pharmaceutical and food etc. Usually,
the scale is one of the most important elements as thebasis for pricing in market transaction, and it is also

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

668

Shang-Hui Li, Shu-Fang Hsu, Yi-Tai Shang, Kuan-Ying Chen

to continuously use the weight-reduction function in theprocess of blending materials. According to the users
experience, when the user prepares the materials withboth hands, the electronic scale must be touched by theuser’s hand, which will lead the operation interface ofthe electronic scale often contaminates the material.
Moreover, the operation interface of the electronic scaleoften has the pollution of greasy and fouling if thecleaning is not performed after repeated using. It is
inevitable that there is a concern about hygiene andsafety. One of similar patents No. 326771 "Electronic
Scales Improved Structure" announced by Intellectual
Property Office, Ministry of Economic Affairs, Taiwan,
2015[3] which could pull out the display platform
independently, and is convenient for transaction use.
However, the operation interface is easy to be dirty, andthe interface is damaged by long-term use when used
for the raw materials of the food. And sometimes it is
not operate for one person. When used repeatedly, it
may cause health and safety concerns. Therefore, apedal-type scale control device was proposed throughcontinuous experiment and research to improve theabove-mentioned defects.

FR

Figure 1 System architecture diagram

Placing the container

Make the scale return-to-zero via the pedal

Put the item to be scaled into the container

Read net weight

3. System Design

Is the same
container to berescaled?

The pedal scale control device comprises a scale body,which divided into a processing unit and a return-tozero module, and a pedal unit. The pedal unit isseparated from the scale body. The pedal unit isprovided with a tread portion, which could connect its
signal to the scale body to be zeroed. And a radio
frequency (RF) connects between the scale body and thepedalsection,which
has
high-frequency
electromagnetic waves with long-distance transmission
capability, and radio frequency technology is widelyused in the field of wireless communication (Fig.1).
In other words, the pedaling transmits a signal to thescale body, and a wireless transmission interface signal
A is set on the scale body, and the A is connected to thezero return module for zeroing or deduction. The pedal
section also has another wireless transmission interface
signal B, which connects the pedal section and A, totransmit the pedal signal to the zeroing module. The
Pedal Scale Flow Chart for the pedal scale control
device is shown as Figure 2.

Yes

No
Figure 2. Pedal Scale Flow Chart

The pedal scale control device was subjected to actual
simulation experiments. The actual finished product was
shown as Figure 3 and the actual operation was shownas Figure 4. The study showed that the expected results
can be achieved. Described as follows:(1).Obviously, the return-to-zero function could be
operated by the pedal scale control device, and theoperator can empty the hands to prepare the ingredients
without touching the scale body, which is more hygienic
and clean when cooking.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan669

Pedal Scale Control Device(2).The operation mode is more intuitive and simpler
than returning to zero or weighing by hands, and theprocess of deduction is smoother and faster.

(4). This design allows the operator to avoid touching
the scale body and reduce cross-contamination.
Obviously, it makes the scale body more hygienic, cleanand safe, and more secure for consumers.

(3).The scale body also has an operation button, and thepedal unit can be removed, and the scale body is used asa general scale.
(4).Connect the scale body and the pedal unit bywireless to avoid possible interference such as wireinterference and easy tripping during using.

Figure 4. Actual operation diagram ofthe pedal scale control unit
Figure 3. Actual operation diagram of the pedal
scale control unit

5. Reference[1]Bureau of Standards, Metrology & Inspection,
Ministry of Economic Affairs, Taiwan.(2018).
“Weights and Measures”, July 30, 2018 fromhttps://www.bsmi.gov.tw/bsmiGIP/wSite/ct?xItem=366
23&ctNode=4077&mp=1

4. Results and Discussion
The results of this study propose an isolation system
between the pedal control device and the scale, which
can effectively isolate the safety distance between thescale and the sanitary. This design has the following
features[2]T-Scale Company limited. (2018). “Electronic Scales”
July 25, 2018 from http://www.tscale.com.tw/index.
php?do=product_details&id=174 .

(1). The design utilizes the wireless method to improve
the traditional electronic scales operation which needs
to use both hands to repeatedly touch the screeninterface, and thus obtained the certificate of Taiwan
invention patent No. I588458.

[3]Intellectual Property Office, Ministry of Economic
Affairs, Taiwan. (2015). Taiwan Patent Search System.
May 18, 2015 from https://twpat7.tipo.gov.tw/tipotwoc/tipotwkm.

(2). The pedal scale control device has been turned intoa specific product from the concept, and is currently
seeking cooperation with manufacturers, and it isexpected to commercialize the product in the future.
(3). The pedal scale control device can be used inchemical, medical, food industry.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

670

Convolution Neural Network Based Fault Diagnosis of Induction Motor
Jong-Hyun Lee

School of Electronics Engineering Kyungpook National University
Daegu, Korea
In-Soo Lee

School of Electronics Engineering Kyungpook National University
Daegu, Korea
E-mail: whdugs8428@knu.ac.kr, insoolee@knu.ac.kr

Abstract
Induction motors are one of the most important components for machine and industrial equipment in modern
industrial applications. Therefore, it is necessary to develop a fault diagnosis system that detects the operating
condition and fault of the induction motor early. This paper presents a induction motor fault diagnosis system usinga CNN (Convolution Neural Network) model. In the proposed method, the fault diagnosis of the induction motor is
performed by using the vibration signal data obtained from the induction motor experiment environment as the input
value of the convolution neural network model. And then the fault diagnosis was made using convolution neuralnetwork. In this paper, fault diagnosis of normal state, rotor and bearing fault of induction motor is performed. Fromthe experimental results, it is confirmed that the proposed method is suitable for diagnosis of rotor and bearing fault
of induction motor.
Keywords: Induction motor, Convolution neural network, Rotor fault, Bearing fault, Fault diagnosis system

Non-model-based methods include fault diagnosis basedon measurements, experience, or physical reviews in thetarget system, and thresholding, expert systemtechniques, and neural networks.[2]In particular, neural networks have advantages such asparallel processing ability, nonlinear function description
and learning function, and there have been many studies
to use neural network for trouble diagnosis based on this.
However, the structure of the neural network model ofthe perfect connection layer can’t learn the invariant
property. This leads to a problem in that it can’t belearned by extracting effective features from periodic
vibration signals.[3]In this paper, CNN model with local connectivity isused for fault diagnosis. And the vibration signals of
three states normal, rotor fault and bearing fault are
obtained from the induction motor, and the vibration

1. Introduction
Induction motors are one of the most importantcomponents for driving DC motors, rectifiers and DC
motor systems and industrial equipment. However, theshutdown of the plant due to an unexpected induction
motor fault results in significant economic losses.
Therefore, it is necessary to develop a fault diagnosis
system that detects the operating condition and fault of
the induction motor early.[1]Fault diagnosis is to detect faults in the system and toclassify faults, which can be largely classified intomodel-based methods and non-model-based methods. In
the model-based method, the fault of the system isdiagnosed based on the mathematical model of thesystem, but it is not easy to obtain an accurate
mathematical model due to the nonlinearity of the system.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

671

Jong-Hyun Lee, In-Soo Lee

signals are used for fault classification. The performancesof the proposed fault diagnosis method are verified
through the experiment and simulation.

3. The proposed fault diagnosis method for
induction motor

2. Experimental environment for fault diagnosis
of induction motor

The proposed fault diagnosis system consists of avibration sensor part for measuring the vibration signal
generated by the induction motor as shown in Fig. 3, adata collection part for digitizing and storing thevibration signal obtained from the vibration sensor, And
a Convolution Neural Network (CNN) based fault
classification unit which classifies the types of faults
generated in the induction motor.

3.1. CNN based fault diagnosis system

Fig. 1. Experimental set up.

For the fault diagnosis of the induction motor, asimulator was constructed as three three-phase induction
motors that caused normal, rotor fault, and bearing fault
as shown in Fig. 1. The induction motor located at thecenter is in the normal state. The induction motor locatedon the left side of the induction motor is in the state ofrotor fault, and the induction motor located on the rightside is in the state of fault of the bearing. We acquired thevibration data from induction motor by using NI9234 andvibration sensor.

(a) Rotor fault

Fig. 3. Structure of the proposed fault diagnosis method.

3.2. Convolution neural network
Convolution neural network (CNN) is a model that
integrates feature extraction and classification, which areseparate from existing pattern recognition methods. CNN
is basically composed of a plurality of convolution layer
and a subsampling layer. [4] The structure of the CNN
used in the induction motor fault system consists of one
input layer, two convolution layers, two max polling
layers and one fully connected layer and three output
layers as shown in Fig. 4.

(b) Bearing fault

Fig. 2. Fault types of induction motor.

As shown in Fig. 2, there are two types of faults
occurring in induction motors: rotor fault and bearing
fault. As shown in Fig. 2 (a), the rotor bar of the induction
motor was punctured by a drill, and the bearing was wornby putting powder into the bearing as shown in Fig. 2 (b).

Fig. 4. Structure of the proposed CNN model.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan672

Convolution Neural Network Based Fault Diagnosis of Induction Motor

The convolution layer extracts features from the input
data. The convolution layer consists of a filter thatextracts features. The filter detects the characteristics ofthe data and maps the detected features to the feature map.
The extracted feature map applies the activation function.
In this paper, Relu function is used as an activation
function.[5]The subsampling layer performs the process of reducing
the dimension of the extracted feature map. There arevarious techniques for pooling. In this paper, It use themax pooling method which extracts the maximum value
from the feature map. Using the max pooling techniquereduces the size of the entire data, which reduces theamount of computation and extracts only the necessary
data. [6]In CNN, when the convolution layer and the max
pooling layer are repeated, only the main features are
extracted, and the extracted characteristics aretransmitted to the Fully connected Layer. Since theconvolution layer and the max pooling layer have twodimensional data, they are transformed into a onedimensional layer for transmission to the fully connected
layer and then transferred to the output layer.
Softmax function was used for the output layer. softmax
is used to transform the class classification problem, thatis, the output from the previous layer, into the probabilityof each state when solving the state classification
problem. Take the exponent on each output and divide by
the normalization constant so that the sum is 1.

(a) Vibration data for normal state(b) Vibration data for rotor fault(c) Vibration data for bearing fault

4. Simulation and Results

Fig. 5. experimental data.

The simulation environment was implemented using
Python 3.5 and Keras library in Windows 10 OS. The
CNN model used in the simulation consists of an input
layer with 1024 nodes, a first convolution layer and a max
pooling layer, a second convolution layer and a max
pooling layer, and finally, it consists of a fully connected
Layer with 256 nodes and an output layer with 3 nodes.
The two convolution layers and the fully connected layer
use the activation function ReLU and the output layer
uses the softmax function.
In order to test the performance of the proposed
induction motor fault system, vibration signal data wereacquired using a vibration sensor by artificiallygenerating normal, rotor fault, and bearing fault in theexperimental apparatus shown in Fig. 5. The vibration
signal data can be obtained with 1024 data per 0.1 second.
As a result, 1024 pieces of data are used as a model input.
The data of the three states obtained from theexperimental apparatus are shown in Fig. 5 (a), (b) and(c).

The data used for learning and verification data are dataof normal, rotor fault and bearing fault state acquired
from the experimental apparatus as shown in Table 1. For
learning, 200 pairs of each state, that is, a total of 614,400
data, are used. The vibration data used in the learning wasused without the frequency domain transformation. The
epoch is 30. In the case of the test, data of 100 pairs foreach state, that is, a total of 307,200 samples, were used.
The results of the test are shown in Fig. 6, which shows
two cases of fault diagnosis when the inverter changes
from the normal state to the rotor fault state and when the
inverter changes from the normal state to the bearing
fault state. In Fig. 6, the x-axis represents the number ofvibration data, and the y-axis represents the diagnostic
value of the model. If classification result is 1, it is in thenormal state. If 2 is output, it is the rotor fault state. If 3
is output, it is the bearing fault state.
Fig. 6 (a) shows the simulation result when the rotor ischanged from the normal state to the rotor fault state. It© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan673

Jong-Hyun Lee, In-Soo Lee

5. Acknowledgment

can be confirmed that the model correctly determines therotor fault state when the vibration signal inputted at the102,400 samples changes to the rotor fault state. Fig. 6(b) shows the simulation results when the normal statechanges to the bearing fault state. It can be confirmed thatthe model accurately determines the fault state of the
bearing when the vibration signal inputted at 102,400
samples changes into the bearing fault state.

This study was supported by the BK21 Plus project
funded by the Ministry of Education, Korea(21A20131600011).
6. Conclusion
In this paper, an induction motor fault diagnosis systemusing CNN was proposed. In the proposed method, thefault diagnosis of the induction motor was performed byusing the data obtained from the simulator as the input of
CNN. As a result of the simulation, it was confirmed thatthe state of the motor was completely diagnosed by the
motor data obtained from the simulator and the fault
diagnosis can be performed without frequency domaintransform process. The further study will need to apply
the proposed method to the industrial equipment.

Table 1. The data pares for simulation.

Induction
motor state

Number oftraining data

Number oftest data

Normal204,800

102,400

Rotor fault

204,800

102,400

Bearing fault

204,800

102,400

Total

614,000

307,200

References
1. In Soo Lee, “Fault Diagnosis System Development of
Induction Motors Using Discrete Wavelet Transform and
Neural Network”, Journal of Korean Institute ofInformation Technology, vol. 9, no. 8, pp. 53-61, 2011.
2. In Soo Lee, “Fault Diagnosis of Induction Motors Using
Discrete Wavelet Transform and Artificial Neural
Network”, HCI International 2011, Florida, USA, 9-14
July 2011.
3. In Soo Lee, "Diagnostic system development for statemonitoring of induction motor and oil level in press
process system", Journal of Fuzzy Logic and Intelligent
Systems, vol. 19, no. 5, 2009.
4. Abien Fred Agarap, “An Architecture Combining
Convolutional Neural Network (CNN) and Support Vector
Machine (SVM) for Image Classification”, arXiv preprint
arXiv:1712.03541, 2017
5. Wenjun Sun, Rui Zhao, Ruqiang Yan, Siyu Shao, Xuefeng
Chen, “Convolutional Discriminative Feature Learning forInduction Motor Fault Diagnosis”, IEEE Transactions on
Industrial Informatics, vol.13, no. 3, June 2017.
6. Rui Zha, Ruqiang Yan, Zhenghua Chen, Kezhi Mao, PengWang, Robert X.Gaoc, “Mechanical Systems and Signal
Processing”, Mechanical Systems and Signal Processing
vol. 115, pp. 213–237, January 2019.

(a) Result of rotor fault diagnosis(b) Result of bearing fault diagnosis
Fig. 6. Result of the Induction motor fault diagnosis.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan674

Analysis of Value Chain on Food and Beverage Micro-enterprises:
Mobile Diner as Example
Shu-Fang Hsu

Department of Food and Beverage Management, Far East University, Taiwan (R.O.C.)
E-mail:h99582004@gmail.com
Shang-Hui Li

Department of Food and Beverage Management, Far East University, Taiwan (R.O.C.)

Abstract
This study was based on the theory of value chain proposed by Porter, 1985, in conjunction with the mobile dinerindustry from the value activities created in the value chain. The main activity was to analyze the competitive
advantage of the mobile diner and the added value to meet customers’ need. In this study, the Delphi method wasused, and then conducted with interview and observation method on 10 homogeneous mobile diners to gain an indepth understanding of the development and current situation in southern Taiwan. Exploring their competitiveness,and discovering that the businessmen purchased the necessary food by themselves, which could ensure safety. In
terms of marketing and sales, due to the mobility and easy acceptance of customization, the relative value of products
and services are relatively improved. This study could provide a practical reference for micro-entrepreneurs who want
to invest in food and beverage in the future.
Keywords: Supply Chain, Third-party logistics Service Supplier, Catering

coffee mobile diners in Taiwan. Gradually, other types ofmobile diners have begun to appear, such as selling
coffee, sandwiches, Japanese ramen or American meals.
The overall value presented by the value chain is madeup of various value activities and margins. Value chainsare all kinds of material and technical specific activities
carried out by enterprises, and also the basis forenterprises to create valuable products for customers. The
purpose of this study was to find out the value in the main
activities of the mobile diners in value chain, and to (1)to explore the logistics and production operation
management of the mobile diner. (2) to explore theshipping logistics, marketing and service value of themobile diner.

1. Introduction
Most micro-entrepreneurs used vans to become as themobile diner when Taiwan began to introduce theconcept of mobile dining cars. However, it was illegal toconvert the regular car into a mobile diner and ran it on
the road. It would be taken as a general street vendor todrive away by police, unlike the restaurant which has ageneral location, if it parked for a long time. In recent
years, Taiwan’s traffic laws have been revised to allow
the mobile diner owners (hereinafter as “owner”), to havemore room to operate. Entrepreneurship itself requires a
lot of capital and must bear a lot of risks. Every
entrepreneur wants to create the greatest economic
benefits with the least amount of capital. The mobile
diner can move freely, and also drive to crowded places
to do business opportunities with high mobility, and savestore rent. In recent years, attention has been paid to the

2. Literature Review
In recent years, Taiwan has promoted the “Micro-Startup
Phoenix Project”. The Project was to improve the labor© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

675

Shu-Fang Hsu, Shang-Hui Li

3. Research Method

force participation rate of Taiwan female labors, middleaged nationals, outlying island residents, and to build afriendly environment for entrepreneurship. Moreover,assist in the development of micro-enterprises and create
employment opportunities. Provide entrepreneurial loaninterest subsidy, entrepreneurial companionship service
and financing credit guarantee project [1]. For this reason,
22 classes of "Entrepreneurship and Development
Classes" and 15 classes of "Entrepreneurship Advanced
Classes" had been launched throughout Taiwan in 2016.
The classes provided the necessary professional
knowledge of entrepreneurship, upgraded the credit rateof youth entrepreneurial loans, and implementing youth
entrepreneurship to enhance the policy goal of youth
entrepreneurship [2]. Many business newcomers were
eager to run their mobile diners, and regarded it as a way
to realize their own value, and then they were willing to
dedicate themselves to it which was able the mobile dinerto achieve their goal of self-realization.
The value chain was proposed by Michael Porter in 1985.
It needs to resolve the business process of the enterprise
into a series of value creation processes if a company
wants to develop its unique competitive advantage orcreate higher added value for shareholders, and the
conjunction of this value process is the value chain. If thevalue of the product or service that the company holds ishigher than all the costs it uses to create the product, thenthe company is a profitable company. In other words, all
the various "value activities" and "profits" constitute theoverall value of the value chain [3].
Enterprises can differentiate themselves from others
through each link in the value chain. The construction
and purchase as the underlying value chain model
combined with marketing effectiveness and demand
supply chain management, thereby enhancing the overall
competitive advantage [4]. It mentioned that the value
created by a company for its customers may come frommany aspects, such as manufacturing, research anddevelopment, logistics, branding, advertising or accessetc. The companies can attach all aspects of theiradvantages to products or services through a variety ofpotential sources of value, and differentiate themselvesfrom others to form a unique competitive advantage.

In this study, the Delphi Method was used to design andinterview the questionnaires, and the data was collected
and collated by the re-validation method. Triangulation
was used to strengthen the theoretical basis and improve
the intrinsic validity of the research in the process of datacollection and analysis. Triangulation test refers to thesame conclusion, using different methods, in different
situations and time, to test different people in the sample,
the purpose is to carry out the conclusions that have been
established through as many channels as possible. Test to
obtain the maximum degree of truth of the conclusion. In
qualitative research, the most typical way to conduct
correlation tests is to combine both interviews andobservations [5]. In detail, methods triangulation, data
triangulation, and analyst triangulation were used to
improve the reliability and objectivity of the researchdata in the process of collection and analysis. In themethod of triangulation, various data collection methods
are used included in-depth interviews with experts,
special lecture speeches, expert TV interview records,
written documents and other sources of information toenhance integrity [6]. There were 10 mobile diners forthis studies coded A, B, C, D, E, F, G, H, I, J respectively.
Because value creation is not a single activity that can beformed, it must be complemented by relevant
departments and closely related to each other. How to
pursue production innovation and marketing sales is thetarget of this study.
During the interview, there were 10 mobile dining cars.
Each interviewee had a different personality during theinterview. For example, one of the respondents was moreconservative, and such respondents were less willing toprovide information and more likely to refuse to accept
and answer. Part of the problem, when faced with private
issues, usually do not want to answer, even if the
respondent is willing to provide information, may not beable to provide the correct information The researchframework was shown as Figure 1.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

676

The Relevance Research amongindustries. . Due to the long working hours of thebusiness car, a total of 9 are full-time, only the I businessis semi-retired, and the work attitude is more casual.

Theoryof Value
Chain

Delphi
Method

4.2. Feed Logistics and Production Operations
6 of the 10 owners purchased the fresh ingredients bythemselves except A owner. The reason for A was thatthe mass fresh ingredients were not easy to preserve. In
addition, there was no bargaining space due to thefrequent purchase of small number ingredients. Most
owners purchased their raw ingredients from food storesor suppliers, which are more expensive than supermarket.
J owner bought his raw ingredients from traditional
market because he needed a variety of fresh food. In
terms of purchasing time, most owners chose to buy
before business, mostly in the morning. All owners
separated food ingredients between business andhousehold to ensure hygiene and quality. 7 owners
changed the new flavors within one month. A ownershowed his new taste in half a year, A considered that hewas no time to develop the new taste because of his long
business hours. As for B, E, F, J owners, they introduced
a new product at least one month. Of 6 of 10 diner cars
were equipped with refrigeration equipment, and the rest
were not refrigerated, but with the related equipment forstoring raw ingredients and semi-finished products. Mostof the owners prepared their related ingredients before
the business except F, and F owner did his work before
and after the business since F owner’s prepared materials
was close to the finished products which could be
prepared and stored after the business. Most of the mobile
diner sold the simple products with a short preparing time.
E, F, and I owners did their food with more cumbersome
process. Most of the owners ran their business within 6hours daily. A and D owners divided their business timeand locations into two sections, so their business hours
are longer than others.
The mobile diner has a low investment amount and iseasy to get started. Because of its small scale, it canmaintain flexible and adaptable environment and can bequickly reformed. The dining car needs to introduce newtastes to attract consumers from time to time.
Competitive, but there may be difficulties in being
imitated by the industry or not thinking about new ideas.

Analysisof DataFigure 1. Research framework

4. Data Analysis
4.1 General Review
The mobile diners need to overcome various tasks anddifficulties, such as weather factors, storage space, andthe single nature of the goods, which will affect thebusiness income. In this study, there were four mobile
diners sold chicken cakes, two offered sandwiches, andfour others provided simple meals. As for gender, 4 werefemale and 6 for male. Half a year was the shortest period
for running diner business, and the longest was 8 years.
Most owners graduated from high school and college,
and two of them graduated from food-related department.
Most of the mobile diners are modified their car body
based on Varica and Veryca. The high mobility were themainly considered by most of owners, which is lower
than the operating cost of the store. As for venture capital,
8 were below NT$500,000, and the other two were
between NT$ half million to 1 million. Among theseowners, 6 received the traffic ticket. And H owner rent
the space for his business, and B and E paid particularattention to the annual activities of department stores
because of being invited, there was no traffic ticket
problem.
Among these operators, a total of 6 received a fine, andthe remaining four did not receive a ticket; the H operator
chose to operate in the leased area, and the B and E
operators were mainly located in the department store
activities, due to the invitation, No ticket issue. Amongthe 10 mobile diners, only E and I operators have not
returned to the book, 8 operators have graduated from thecatering-related departments, and the other two operators
have not contacted the catering-related departments, andfour of them have not engaged in catering-related

4.3 Marketing, Sales and Service© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

677

Shu-Fang Hsu, Shang-Hui Li

finished product due to insufficient space for the mobile
car.
Mobile diner owners must face the pressure of real losses
while realizing self-worth. It’s impossible to go all theway on the road to entrepreneurship, and there will be an
off-season time in the face of this business. If there are
no regular customers, the owners must adjust their moodat any time, optimistically face and maintain the initialintention of starting-up a business in addition to the dailypressure. However, the mobile diners have a low amountof capital with high mobility and free business hours. The
owners can choose the most appropriate time and place
for sale according to their product category and customer
group. It is also relatively easy to change the businesslocation.
This study results showed that “chicken cake” is asuitable choice for micro-enterprise beginners because“chicken cake” has traditional characteristics, an easy
way to make, and with a wide range of consumer groups.
The cost waste will not be occurred due to the productcan be produced and sold out on the same day. Therefore,
the maximum benefit can be obtained in the shortest timewith the least cost.

9 mobile diners operated on social networking or
professionally marketed via fans except H. Because H
has done his business for 10 years with the simple
products and easy to produce, and he has the regularcustomers. The A, B, C and D owners were different fromthe traditional chicken cake. Their price was higher than
the traditional ones through diversified the appearance,
ingredients and stuffing. B was the most famous ones
with the high price because the ingredients he used in the
stuffing were special and innovative. Seven owners often
showed their new products, seasonal and daily limited
products, and used hunger marketing techniques to attract
customers' attention except H, I, and J, they offered theregular products. All owners accepted customized
services and were willing to accept the return andexchange products. However, G owner, without
customized services, can be presented to the customers
only after heating because the goods were already semifinished products.
After the modified diner has a unique style, it can quickly
attract the attention of consumers, so there is no need for
DM and advertising, and nowadays the network is moredeveloped to increase exposure. The mobile dining car
has a single product type and a fast meal speed, but thecustomer selection will be limited. The 10 mobile dinervisited by the group were all prepared after the materials
were prepared. All the operations were completed by theoperators themselves. The front-end operations of themobile dining cars were not shorter than the general
restaurant preparation time. As [7] puts forward the viewof Taiwanese service, the customization of Taiwaneseservice is not only for high-priced products, but also forthe top-level customer, but for ordinary products, it canbe tailored to customers. The dining car operator is alsousing this concept regardless of the price level. And theaction dining car is less rent-free, even if the ticket isreceived, it is cheaper than the store rent.

References
1. Micro-Startup Phoenix (2017). Introduction to theproject. 2017 Retrieved fromhttps://beboss.wda.gov.tw/cht/index.php?code=list&ids=
262
2. Implementation Notes for the 105-Year Government
Project Plan (2016). Taiwan Youth Entrepreneur
Association.
2017
Retrievedfrom
http://www.careernet.org.tw/modules.php?name=web&fi
le=project&op=105
3. Porter, M. E. (1995) The Competitive Advantage of the
Inner City. Harvard Business Review. May-June.
4. Ssu Tu, T. H. （ 2011
A new theory of strategic
management-conceptual architecture and analyticalmethods. Taipei, Taiwan: Zhisheng Culture Business Co.,Ltd.
5. Chen, H. M. (2011). Social science qualitative research.
Taipei, Taiwan: Wunan Book Publishing Co., Ltd.
6. Yen, C., & Lin, S. Y. (2010) A discussion of the impactof packaging design benefits on brand equity. Journal of
Design Research.10, 71-79.
7. Shen, F. C. (2016) Taiwan taste service: Shen Fang-Cheng
leads the way. Taipei, Taiwan: World Magazine.

5. Conclusions and Recommendations
It is known from the conclusion of this study that theweather factors are uncontrollable. When the rain is rainy,
the shelter can be sheltered from the rain, but theoperators can change their attitudes and find ways to deal
with them. Furthermore, if you want to move the system
flow of the storefront into the dining car, you will not beable to fully utilize the integrity of the process and© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

678

Suppression of Roll Oscillation in Turning of Quadruped Robot by Asymmetric Amplificationof Central Pattern Generator Output Waveform
Makoto Kitani, Ryo Asami, Noritaka Sato, and Yoshifumi Morita

Morita and Sato Laboratory, Electrical and Mechanical Engineering, Nagoya Institute of Technology,
Gokiso-cho, Showa-ku, Nagoya, Aichi, 466-8555 Japan
E-mail: m.kitani.810@nitech.jp
http://watt.web.nitech.ac.jp/index_2.html
Tomofumi Fujiwara, Takahiro Endo, and Fumitoshi Matsuno

Matsuno Laboratory, Kyoto university,
Kyoto daigaku-Katsura, Nishikyo-ku, Kyoto, 615-8530 Japan
E-mail: matsuno@me.kyoto-u.ac.jp
http://www.mechatronics.me.kyoto-u.ac.jp/index.php?ml_lang=en

Abstract
Quadruped robots experience excessive vibrations in the roll direction when turning by controlling their hip yaw joint.
To overcome this problem, we proposed the asymmetric amplification of the output waveforms of central patterngenerators. We implemented the proposed method on a robot by using a simulator and verified the effectivenessduring the turning operation.
Keywords: CPG, quadruped robot, suppression of roll oscillation

1. Introduction
Legged robots can move on various terrain because they
have several degrees of freedom. They are controlled byplanning their motion based on a strict model of a robot
and an environment. However, planning their motion forvarious environments is quite complex, and they are
inefficient at moving on various terrains at high speed.
To overcome this problem, central pattern generators(CPGs) can be used. It was observed that this mechanism
is related to rhythmical movements such as walking andswimming.1,2 The CPG, an animal motion generation
method, enables the motion of legged robots on various
terrain by using only one algorithm because of a pull-in
phenomenon.
Because of the above advantages, CPGs have beenused in robots in several studies. Kimura et al.
implemented the CPG on a real robot. The robot “Patrush”

Fig. 1 Tekken body angle in turning on flat land4 a

can walk3 and “Tekken4” can turn4 on rough terrain. The
turning motion is realized by controlling the hip yaw joint.
However, the vibrations of the robot body are large
because this method uses falling in the roll direction.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan679

Makoto Kitani, Ryo Asami, Noritaka Sato, Yoshifumi Morita, Tomofumi Fujiwara, Takahiro Endo, Fumitoshi Matsuno

In teleoperation, an operator controls a robot while
watching the images from a camera mounted on the robot.
Therefore, the image vibration should be small. In theexperiment conducted by Kimura et al.,4 the robot
experiences vibration in the roll direction. Fig. 1a shows
the graph of the posture of the main body and walking
speed when turning on a flat ground.4 The vertical axis of
the graph corresponds to the roll angle of the robot body,the pitch angle, and the walking speed. The horizontal
axis corresponds to time. According to the body rollangle (thick black line) in the graph, the robot vibrates,
and it is finally overturned.
In the method proposed by Kimura et al., the camera
image will vibrate making teleoperation difficult.
According to the study conducted by Tsubaki,5 if thefrequency of the image vibration is the same, a larger
amplitude makes the operator uncomfortable. Therefore,in teleoperation, the image vibration should be small. Fig.
2 b illustrates a graph showing the relation between thesubjective evaluation and the rotation angle for motion
sickness.6 Vibration in the roll direction is most likely tocause motion sickness at any speed of rotation.

2. Walking on flat ground by using CPG
2.1. Nonlinear first-order simultaneous
differential equations constituting CPG
The basic formula of the CPG is a nonlinear first-ordersimultaneous differential equation shown in Eq. (1)–
(3).3
̇ =−

−

+

+

+

(1)

̇ =− +
(2)
=
(0, )
(3)
, , and are the value, fatigue state, and output ofis the steady input thatthe th neuron, respectively,
generates oscillation,is the feedback such as thejoint angle, is the fatigue state factor, and are thetime constants of and , respectively, andis theconnection factor that determines the influence of theother neurons. A pair of CPGs was mounted on each joint,
and four equations were formed corresponding to eachleg. This method is based on the biped walking program7of the CPG where the connection factor is defined asfollows. The first character [R: Right, L: Left], the second
character [F: Fore, H: Hind], the third character [H: Hip,
K: Knee], the fourth character [E: Extensor, F: Flexor].
- Between the extensor and flexor of the same joint- Between the right and left of the same extensor andflexor- Between the knee and hip joints of the extensor- Between the knee and hip joints of the flexor- Others
Fig. 3 shows the proposed CPG configuration. We
generated a waveform for walking by multiplying thedifference in the output of the CPG of each extension
muscle flexor and the amplification gain. The walking
waveform is given by Eq. (4).

Fig. 2 Subjective evaluation value of motion sickness withrespect to speed of rotation when rotation relative to yaw,
pitch, and roll axis is given in video6 b

The vibration of the robot body causes discomfort tothe operator. Therefore, the purpose of this study is tosuppress roll vibration in legged robots. To suppress roll
vibration, we proposed a turning method using theasymmetric amplification of the CPG output waveform.
In this study, we verified the previous method4 that uses
the yaw hip joint and the proposed method by using asimulator.

=

(

−

)

(4)

is the target angle of the joint,is theamplification gain at the joint,andare the CPG
output values of the extensor and flexor, respectively.

ab
Adapted from “Realization of dynamic turning motion on irregular
Adapted from “Visually induced motion sickness” by Hiroyasu Ujike,terrain of a quadruped robot "TEKKEN4" with neural oscillators” by
2007, Information and Television Engineers, 61(8) p.1124, Copyright
Yasuhiro Fukuoka and Hiroshi Kimura, 2006, Journal of the Robotics
2007 by The Institute of Image Information and Television Engineers
Society of Japan, 72(724) p.3852, Copyright 2006 by The Japan Societyof Mechanical Engineers© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

680

Suppression of roll oscillation

Table 1
Parameter

CPG parameters
Value

Parameter

3.0

′

4.

2.5

-1.0

0.10

-0.50

0.080

-0.30

Amplification factor
Gain (straight)

Gain (turn)

Joint name

Right

Left

Right

Left

FH

2.2

2.2

4.2

1.2

FK

0.8

0.8

1.8

0.8

2.2. Robot for experiment
A verification robot implementing the CPG is shown in
Fig. 4. We developed the robot using a simulator called
Gazebo. The CPG parameters are listed in Table 1.

-2.0

0

Table 2

Fig. 3 Proposed CPG configuration

Value

HH

2.2

2.2

4.2

1.2

HK

0.8

0.8

1.8

0.8

Verification

4.1. Verification condition
To verify the proposed method, we used the Gazebo
simulator as described in Section 2. We made the robot
walk on a flat ground using the simulator and evaluatedits performance in the following sequence: adjusting thesteady input to have the same turning radius, measuring
the roll angle of the robot body when the robot walks on
the turning trajectory, and comparing the data. The
following conditions were applied so that the robot canwalk on the same turning radius.
Conventional method
Rotation angle of front leg - 0.15 [rad]Stationary input - 2.7
Proposed method
The amplification factor of the joint angle - The“gain(turn)” column of Table 2
Stationary input - 3.0

Fig. 4 A robot in simulator

3. Turning movement
3.1. Turning method with yaw joint (conventional
method)In the conventional method, the degree of freedom in theyaw direction is added to the hip joint. The turning radius
can be changed by the angle of the yaw axis of the leg.
However, in this method, the left front leg protrudes
outside the robot. This increases the moment around thefalling axis. We assumed that large vibrations would
occur because of the increase in the moment.
3.2. Turning method without yaw joint (proposed
method)In order to reduce the moment around the falling axis, wesuggest a change in the amplification factor of the CPG
output between the left and right legs. The robot is able
to turn when the amplification factor of the joint angle on
the outside of the turning circle is increased. The turning
radius can be changed by changing the amplification
factor. Table 2 lists the amplification factors at straightand turning.

4.2. Results and discussion
Figs. 5 and 6 show the graph of the robot posture whenturning in orbit shown as Fig. 7. The peak-to-peak valueof the roll angle in the conventional method is -0.2 to 0.15
[rad] and that in the proposed method is -0.1 to 0.1 [rad].

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

681

Makoto Kitani, Ryo Asami, Noritaka Sato, Yoshifumi Morita, Tomofumi Fujiwara, Takahiro Endo, Fumitoshi Matsuno

5. Conclusions and future work
The purpose of this study is to suppress roll vibration in
the turning motion of legged robots. We suggest a turning
method that changes the amplification factor of the joint
angle between the left and right legs to suppress the robot
body vibration. We verified the two methods using asimulator. As a result, we succeeded in suppressing
43.7% vibration in the roll direction and 7.4% in the pitch
direction when compared to the conventional method.
The proposed method will reduce the motion sickness
caused by the image vibration during remote operation.
In the future, we will realize walking of legged robots onrough terrain by installing CPG feedback.

Fig. 5 Robot body roll angle

References
1.

2.

3.

Fig. 6 Robot body pitch angle

4.

5.

Fig. 7 Orbit of robot turning

In this study, we compared the conventional andproposed methods. As shown in the graph, the vibrationin the roll direction could be suppressed. By comparing
the maximum peak-to-peak value of both methods, weobserved that the proposed method is able to suppress
43.7% vibration in the roll direction. Moreover, theproposed method suppressed 7.4% vibration in the pitch
direction compared to the conventional method becauseof the suppression of vibration around the falling axis.
Moreover, motion sickness can be reduced because thevibration in the roll direction can be suppressed.

6.

7.

G. S. Stent, W. B. Jr.Kristan, W. D. Friesen, C. A. A. Ort,
M. Poon and R. L. Carabrese, Neuronal generation of theleech swimming movement, Science, 200 (1978) 1348–
1357
J. T. Bachanam and S. Grillner, Newly identified
glutamate interneurons and their role in locomotion inlamprey spinal cord, Science, 236 (1987) 312–314
Hiroshi Kimura, Seiichi Akiyama and Kazuaki Sakurama,
Dynamic walking on irregular terrain and running on flat
terrain of the quadruped using neural oscillator, Journal of
the Robotics Society of Japan, 16(8) (1998) 1138–1145
Yasuhiro Fukuoka and Hiroshi Kimura, Realization of
dynamic turning motion on irregular terrain of aquadruped robot "TEKKEN4" with neural oscillators,
Journal of the Robotics Society of Japan, 72(724) (2006)
3848–3854
Ikuko Tsubaki, Toshiya Morita, Kiyoharu Aizawa and
Takahiro Saito, The analysis of oscillatory motion ofvideos affected by camera shaking for visually induced
motion sickness, The Journal of the Institute of Electronics,
Information and Communication Engineers, J89-A (3)(2006) 262–267
Hiroyasu Ujike, Visually induced motion sickness, Image
Information and Television Engineers, 61(8) (2007) 1122–
1124
Open
dynamics
enginehttp://www.kojm.sakura.ne.jp/ode/index.php?

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan682

Tumble avoidance system for rescue robot by estimating the contact points
using a 3D depth sensor
Noritaka Sato*, Kotaro Ohshima, Yoshifumi Morita

Department of Electrical and Mechanical Engineering, Nagoya Institute of Technology, Gokiso-cho, Syowa-ku
Nagoya, Aichi 466-8555, Japan
E-mail: sato.noritaka@nitech.ac.jp
http://hi.web.nitech.ac.jp†

Abstract
Rescue robots are expected to perform works in hazardous areas. However, when the robot runs on a rough terrain,
fatal rollover falling sometimes occurs. Therefore, we propose a tumble-risk-assessment system, which calculates anormalized energy stability margin by using a 3D depth sensor. Moreover, a control system to avoid the tumble
situation by moving the sub-crawlers automatically is proposed. We implemented the proposed systems to a realrobot, carried out experiments, and confirmed the effectiveness of the proposed method.
Keywords: Rescue robot, Tumble avoidance, Normalized energy stability margin, 3D depth sensor

1. Introduction
Recently, times of emergency have necessitated rapid
searches for victims in collapsed buildings. However,searches conducted by humans involve risks ofsecondary disasters. Therefore, it is expected that rescue
robots search for victims before entry by humans [1].
Most existing rescue robots are controlled byteleoperation, because fully autonomous control may be
dangerous in uncertain disaster environments. However,full teleoperation is difficult for a human operator,
because a rescue robot has multiple degrees of freedom
in general. Therefore, human support control system,“semi-autonomous control” in other words, is needed [2].
One of most critical incidents is falling down of therobot. Figure 1 shows a robot which falls down in the
RoboCup Rescue Robot League competition. As shownin Fig. 1, the fallen robot cannot run again without
someone’s supports.

Fig. 1. Fallen robot in RoboCup Rescue Competition.
Okuda et al. proposed a semi-autonomous control
system to avoid the falling-down situation [3] by using
the normalized energy stability margin (SNE) proposed by.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

683

Noritaka Sato, Kotaro Ohshima, Yoshifumi Morita

Fig. 2. Normalized energy stability margin.

Fig. 4. Estimation result of contact points and axes of fall.
(White points are environment points of 3D map, yellow
points are contact points between ground and robot, andred points are points which configure axes of fall.)
2. Calculation Method of Normalized Energy
Stability Margin
Fig. 3. Detection area for contact points.

2.1. Normalized Energy Stability Margin
The normalized energy stability margin proposed by
Hirose et al. [5] is one of the most popular indexes of riskof fall because the calculation is very simple. The
normalized energy stability margin SNE is obtained by
=ℎ
− ℎ,
(1)
where hmax is the highest height of the COG (center ofgravity) of the robot when the robot falls down and h is
the current height of the COG of the robot, as shown in
Fig. 2. As mentioned in Section 1, to calculate SNE, thepositions of the contact points are necessary because theaxis of the fall is obtained by the positions of the contact
points.

Hirose et al. [4]. However, the authors assume that thecircumference of the robot contacts with the ground.
Therefore, the system often cannot calculate the risk of
the fall precisely, when the robot runs on a rough terrain,
because the contact point between the robot and theground may locate on the inner part of the robot.
In the existing works, the assumption about thecontact points between the ground and the robot arenecessary because SNE is obtained from the position of
the contact points and it is difficult to get the precise
position of the contact point. However, the assumptionsof existing studied do not often satisfy when the robot
runs on a rough terrain.
In our previous work, we developed estimating
system of the contact points by using a reconstructed 3D
map from the point cloud data of a 3D depth sensor, such
as KINECT [5]. Therefore, the purpose of this study is todevelop a semi-autonomous system to avoid the fallingdown situation. In our system, we use partial
differentiation of the SNE. The contact points are
estimated by the system which is developed in ourprevious work.

2.2. Estimation of Contact Points
To estimate the contact points between the ground andthe robot, the 3D map is reconstructed by using a 3D
depth sensor. The SLAM (Simultaneous Localizationand Mapping) technique is used for the map building
with ROS (Robot Operation System) by Willow Garage.
By using ROS, a developer can easily use someone’s
open-source programs. In this study, we use RTAB-Map
developed by Labbe et al. [6]. And since we also use ROS,
other developers can easily use our system, only putting
a 3D depth sensor on the robot.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan684

Tumble avoidance system for=

Fig.5. Definition of

,

,

,

(Top view)

2.3. Estimation of Axis of fall
We set the detection area along with the shape of thebottom part of the robot as shown in Fig. 3. If a point in
the 3D map exists in the detection area, the point is set asa contact point as shown in Fig. 4. A convex polygon ismade from the contact points and the edges of the convex
polygon are set as the axes of fall. After obtaining theaxes of fall, the normalized energy stability margin SNE
is calculated by Eq. 1.

Fig. 6. Flow chart of the proposed control system.
satisfying Eq. (2), the target angle of the sub-crawlers
should be set as Eq. (4).
In the proposed control system, the robot is teleoperated by an operator. However, when the SNE
becomes almost zero, the target angles of the subcrawlers are used as Eq. (4) instead of the input by theoperator to avoid the tumble situation. Figure 6 shows thecontrol flow chart of the proposed control system.

3. Control Method of Sub-crawlers
To avoid the tumble situation, SNE should be increased bymoving the sub-crawlers, when the SNE becomes almost
zero. Therefore, the sub-crawlers should be moved so
that
> 0.
(2)

4. Experiment4.1. Experiment method

SNE is a function of the contact points and the angle of the
sub-crawlers. In this study, we assume that the contact
points are known by the proposed estimation systemdescribed in Section 2 and SNE is the function of the angleof the sub-crawlers only. Therefore,
d
=
+
dt
+

+

.

The purpose of the experiment is to verify theeffectiveness of the proposed method compared to theexisting method [3]. The experiment environment and the
motion of the robot is shown in Fig. 7. The left side of therobot is on the concrete block and the other side is on the
ground. The robot repeats stop and go for 2.5 seconds.
The evaluation index is SNE.

(3)

4.2. Result

The definition of=,,,is shown in
Fig. 5. The target angle of the sub-crawlers are set as Eq.
(4) in this study.
=

( :

)

The result of the experiment is shown in Fig. 8. The
horizontal axis is time and the vertical axis is SNE. The
green and orange line shows the SNE by the existing
method and the proposed method, respectively. As
shown in Fig.8, in the case of the proposed method, thedecrease of the SNE becomes gradual after 9.2 s and the
SNE increases after 10 s. On the other hand, in the case of
the existing method, the SNE continues to decrease after
9.2 s.

(4)

By applying Eq. (4) to Eq. (3), we obtain following
equation:
=

+

+

+

(5)

Therefore, in the case of > 0, the right side of Eq. (5)is always positive, i.e. to avoid the tumble situation by© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

685

Noritaka Sato, Kotaro Ohshima, Yoshifumi Morita

Fig. 7. The experiment environment and the motion of the robot (Proposed Method).

Fig. 8. The result of the experiment.

Fig. 9. Angle of the sub-crawlers (Existing method).

4.3. Consideration
As shown in Fig. 8, the large difference between the
proposed method and the existing method is observed
after 9 s. Therefore, Fig. 9 and Fig. 10 show the angle ofthe sub-crawlers after 9 s in the case of the existing
method and the proposed method, respectively.
As shown in Fig. 9, the sub-crawlers did not movebecause of the assumption that the circumference of therobot contacts with the block and the ground. On theother hand, as shown in Fig. 10, in the case of the
proposed method, the front-right and rear-right subcrawlers are moved after 9.2 s, and then the SNE increases,
because the contact points and the axis of the fall are
precisely estimated and the motions of the sub-crawlers
are correct.

Fig. 10. Angle of the sub-crawlers (Proposed method).
References
1. S. Tadokoro, Special Project on Development ofAdvanced Robots for Disaster Response (DDT Project),
Proc. of IEEE Workshop on Advance Robotics and its
Social Impacts (2005) 66–72.
2. K. Ohno et al., Semi-autonomous control system of rescue
crawler robot having flippers for getting over unknownsteps, Proc. of the 2007 IEEE/RSJ Int. Conf. on Intelligent
Robots and Systems (2007) 3012–3018.
3. Y. Okada et al., Semi-autonomous operation of tracked
vehicles on rough terrain using autonomous control ofactive flippers, Proc. of IEEE/RSJ Int. Conf. on Intelligent
Robots and Systems (2009) 2815–2820.
4. S. Hirose et al., Normalized energy stability margin and its
contour of walking vehicles on rough terrain, Proc. ofIEEE Int. Conference on Robotics and Automation (2001)
181–186.
5. N. Sato et al., Assessment System of Tumble Stability
Risk for Rescue Robot Using a 3D Depth Sensor, Proc. Of
Int. Conf. on Mechatronics and Information Technology(2015) 146–150.

5. Conclusion
In this study, we propose a tumble-risk-assessment
system, which calculates a normalized energy stability
margin by using a 3D depth sensor. Moreover, a control
system to avoid the tumble situation by moving the subcrawlers automatically is proposed. We implemented the
proposed systems to a real robot, carried out experiments,
and confirmed the effectiveness of the proposed method.
The future works are to carry out the experiment onrough terrains and to develop a system not to become adangerous situation by predicting the SNE.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

686

Rule based Intrusion Detection System by Using Statistical Flow Analysis Technique for Software
Defined Network
Mahnoor Ejaz, Osama Sohail, Talha Naqash, Zain ul Abideen, Sajjad Hussain Shah
Department of CS
Bahria University,Islamabad, Pakistan
mahnoorejaz96@gmail.com, sohailosama71@gmail.com, talhanaqashtn@gmail.com, Zain@bahria.edu.pk, sajjadshah@ieee.org

Abstract
Network Security is a vast field making progress around the globe very fast. In every progressing year, developers have
implemented different tools, which include Intrusion detection systems. Nowadays Intrusion Detection System (IDS) is
one of the popular tools, which are drawing the attention of many researchers. Applying it in Software Defined
Networking (SDN) facilitates network management and enables to enhance the productivity of network monitoring.
Classifying packets based on their statistics and separating the forward process of network packets from the routing
process is the main challenge. In this paper, rule-based classification is done in order to differentiate between viruses
and normal packets. Statistical analysis of different network traffic flows are done through which segregation is madeand intrusion is detected in Software Defined Networking. The proposed system is experimentally tested on UNB ISCX
datasets

Keywords__ Network Security, Software Defined Networking (SDN), Intrusion Detection System (IDS), Network Monitoring, Network traffic flows
1.

Introduction

being used in this context that are fuzzy logic, machine learning,
genetic algorithms etc. For the security hazards our focus is todesign an application for software defined network that will beefficient enough to detect the intruders in network thus making it
safe from malicious attacks. When it comes to SDN architecture
unlike traditional networks they don’t distribute the decision
making power among the routers and switches rather it is all
done by the centralized controller that uses Open Flow
protocol[4] which tells the switches to regulate the traffic to mostsuitable path in network. Most important aspect that comes tomind using any network is to have reliable and securecommunication. SDN provides programmable platform todevelop an application that can detect the malicious traffic. For
that purpose many work has been done at hardware level by the
researchers that is an efficient way to secure the networks from
intruders. Authorized Authentication Authority [8] is one ofthese devices that are used for intrusion detection in networks
but are very costly. To avoid this disadvantage many solutions
are provided, we are developing an application that will run in
SDN controller and doesn’t require a separate device and it is acost effective remedy to it.

The predominance of the Internet and nearby systems, intrusions
plays an important role in computer systems. Exposure of suchattacks is an important issue in computer network security.
Intrusion Detection Systems (IDS)[1] innovation is a powerful
methodology in managing the issues of system security. An
intrusion discovery framework scans for strange or malignant
practices in the examples of action in the review trail. Most
intrusion detection systems are programming projects.
Individuals use screen to monitor the happening in a computer
framework or system, examination the framework events,
recognize suspected interruption, and afterwards generates analarm. Intrusion Detection System (IDS) on the exactness of thisdepiction explains the framework execution in a huge extent, andthis portrayal is a trouble in research region of intrusion
identification. Unlike traditional networks, Software Defined
Networking (SDN) has introduced centralized controller that ismaking SDN beneficial in terms of simplified network hardware,
efficient infrastructure and programmable architecture [3] that
help in deployment of new applications due to which it is
becoming common in big companies. This evolution occur withthe upbringing Open Flow protocol that is used in SDN [2] tocontrol and regulate traffic in controller. Having a view on root
cause to write this research paper up brings the study of all work
done by researchers in this objective. Different techniques are2.

Software Defined Networking© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

687

Mahnoor Ejaz, Osama Sohail, Talha Naqash, Zain ul Abideen, Sajjad Hussain Shah

Software defined networking is regulating the traffic via
controller, instead of using large number of routing devices.
SDN constitutes three layers connected to north and south boundAPI’s to build a communication between the application anddevices. SDN controller is a software that is regulating the trafficin network. SDN programmable environment bring an ease toprogram a software like controller for reliability and security
issues. SDN is trending in present era whose many applications
are developed and deployed by many service providers likeAmazon, Facebook etc. thus replacing the traditional networks.
Controllers [9] are efficient enough to administrate the traffic
from single instance and can be deployed to various hardware
devices. SDN have the capability to support 5 different type ofcontrollers.
Open Daylight open-source SDN controller
Open Contrail SDN controller.
Floodlight open SDN controller
Ryu SDN controller
FlowVisor SDN controller

The one that is used in this research is V1.3.This version
provides many messages and structures that are utilized to get
the statistical information regarding traffic. We have used
OFPFlowStatsReply message to get the required statistics.

3.

Related WorkAs the study begins related to Intrusion Detection System wecame across many techniques that might have used by many
researchers and developers for its development. Typically, IDS[4] is required to this concern instead using a simple firewall tool.
Intrusion detection system that is responsible to take care of the
network from the unwanted data manipulation. IDS is
programmed using different knowledge based techniques likemachine learning. In order to detect the manipulated traffic thatis caused either by malicious attack by attacker or by anapplication that intrudes the network and get it damage IDS tools
are in recent market. IDS can be distributed IDS in which resultsfrom disseminate junctions are then governed by IDS administeror it can be parallel IDS that is monitoring the traffic on a single
point. Several anomaly based and signature based methodologies
are used and giving adequate results as per developers need.
Detection and prevention further based on protocols (TCP,
ICMP, UDP), attacks (DOS, DDOS, UDP flood attack)[10]. In
order to follow a signature or rule based methodology to develop
IDS require different tools like SNORT, BASE [13] etc. These
tools provide overall analysis of network and help detecting themalicious flow. Different rules according to understanding
regarding harmful traffic can be written and implemented in any
language. In result to these rules results are generated thus
detecting the intruders in network. SNORT tool is the mostcommon tool used in signature based technique that is being
used in implementing many IDS. It can also be used merged withother frameworks regarding intrusion detection. In Signature rule
based[12,16] model is developed using SNORT sensors thathave rules databases that are further updated keeping in view theresults of previous rules to bring efficiency and improvement todetect the malicious traffic. SNORT provides fast singlethreaded and multi-threaded versions.

Fig 1: Software Defined Networking Outline

A. Ryu SDN OpenFlow Controller.
Different categorized controllers are used by separate developers
in Software Defined network to build their componentialarchitecture for developing their applications for several areas. It
provides programmable environment in python for SDN
operating system. Ryu controller [6] provides a platform fordeveloping in application in SDN frameworks with many API’s
that are well-defined for understanding by the developers. The
conventions utilized by the controller are Open Flow protocolthat is responsible for tasking the SDN to regulate the traffic to
optimized path. The versions of Open Flow that are fully
supported by RYU are V1.0, V1.2, and V1.3.

IV. Proposed Framework to Detect Viruses using rules
We are going to develop an application in Ryu SDN controllerthat will be able to detect the malicious packets and play a roleof NIDS [7, 15] in any network system. When a traffic flow issend IP and MAC addresses are learned by the controller toregulate the next flows. Major technique observed by us inclassification of virus and normal traffic is stating the rules basedon five statistical information obtained from Open Flowmessage structure that are Pckt_Count, Byte_Count and© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

688

Rule based Intrusion Detection

Duration. Other two features are calculated using them. Passing
through the rules established using these features; traffic isclassified into virus and normal. A separate database ismaintained containing all the IP and MAC addresses that is usedby our application to separate the addresses of those that are
malicious to network. Which are then placed in grey list databaseand are temporarily blocked, if the same action repeated thanplaced to black list database result the permanent blockade.
Features that are obtained using OFP Flow Stats Request[5]message gives the following flow statistics. Table no.1 shows thefeatures.

Suppose x is the bytes from the traffic flow and t is the duration
in
which
it
is
communicating.
Then
x=1,2,3….n and t=1s,2s,3s,4s…then

Explanation

1

Flags

Kept zero

2

Match

Instance of OFP match

3

Duration_nsec

Duration of flow inseconds

4

Idle_timeout
Idle time out for a flow5

Hard_timeout

Hard time out for a flow

6

Pckt-Count

Pcaket count of a flow

7

Byte-Count

Byte count of a flow

8

CookieMatching entry values

9

Out_port

Output port for new
entries10Out_group

Output group for new
entries∑/(2)

Rules To Detect The Attacks/Viruses

Attacks are of different types with different vulnerabilities. Oneof the famous attacks is the Denial of Service (DoS) attack [11],which makes the system hang up as the number of pings is way
too much in less duration. By this general hypothesis we canassume that if the duration is less and pckt-count is high then it isa DoS attack [14]. To define the high and low, statistical analysis
is applied on the data to check the difference in behavior ofviruses and normal packets. During this analysis we realized that
pckt-count, byte-count are very important with respect toduration in which packets are send. A basic rule is as follows:

Table No.1: Traffic Flows features

FeatureName(1)

Can be used to derive the bit-rate from the traffic flow.
Similarly for the packet-rate

A.

S.NO

∗

∑

If duration isLOW && pckt-count isHIGH then attack==DoS
If pckt_count isLOW && byte-count isHIGH then attack ==DOS
B.
Flow of the Application
Any application has a flow in which it is executed. The flow of
this application is explained by this flow diagram

For our application these OFP Flow Stats Request and Reply
messages are used to parameters as per our requirements todevelop the rules are separated, observed, and stored in a
separate database. Some of the basic features, which are required
to build an IDS. Using some of the basic features we can derive
other features such as bit-rate and packet-rate. Applying basic
formulas to produce bit-rate can help us making decision about
the malicious packets. Formulas are used to calculate therequired features.
Fig 2: Flow Diagram of Proposed Solution© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

689

Mahnoor Ejaz, Osama Sohail, Talha Naqash, Zain ul Abideen, Sajjad Hussain ShahIn fig no.2 the basic flow is defined of the application wherecontroller is set on at the start of the procedure. After controller
is set on, now to make the network virtually, topology is made.
After topology is created now the application is ready to let
ongoing traffic pass through it. As soon as traffic is sent, if thefile is sent completely then it will apply rules on the ongoingtraffic otherwise it will be again directed towards controller.

Blackhole
Version 2-6
Cutwail

For the evaluation of the proposed solution, we utilized the
UNB-ISCX data set this dataset consist of various malware
traffic recorded in complex scenarios. The detail of the attack
available in the data set is given in table II also the flow
instances with malware launched by these attacks are given intable III.

Power
LoaderTrojanBlackhole
Version
2-6
Load
money
Salty
Version
1-4
Zeus

Cutwail
Version
1-3

Darkness
DDOS

Ponyzeus

Purplehaze

Spy Eye

Tbot
Version 15
Zero access
Version 1-2

DrowerWorm

Pony
Loader
Tiny
Zbot

Flows

Packets

Alien Spy Rat
Version 1-4

7111

9124

1801

35358

Kuluoz18913

178353Purplehaze

6872

294810

Pony Loader

289

3798

Power Loader

221

4740

SalityVersion 1-4
Tbot
Version 1-5
Zeus
Zero access
Version 1-2
Normal Traffic
Normal 1

9448

19881

474

9993

382
481

6892
5111

4982

25587

Normal 2

21842

1750963

Normal 3

23691

216332

Normal 4367211063445

We recoded the normal traffic (without virus traffic) in a secureenvironment and mix the files with malware to check theaccuracy of the algorithm. The proposed system successfully
detected the flows as shown in the figure 3. Both rules detected
the malware traffic by using their statistical values, which proves
the accuracy of the system.

Table 3 : Malware Quantities Used in the Experiment
Virus

7429

Version 1-3

Table 2 : Malware Traffic Used in the Experiment
Alien
Spy Rat
Version
1-4
Kuluoz482© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

690

Rule based Intrusion Detection

Figure 3: Accuracy of Rules against the Various Malwares of ISCX2012

4

Conclusion

Intrusion Detection systems (IDS) nowadays are the basic need forthe network security. Monitoring network traffic flows anddetecting the malicious packets was the main challenge. In thispaper, we devised an Intrusion Detection System (IDS) by
applying statistical analysis based on flow statistics extracted byusing OFPFlowStatsReply message. Our system incorporates theconcept of rule based classification of virus and normal traffic.
The developed application have defined rules when flow passes
through these rules will be detected if the flow is malicious and isblocked permanently on repetition. Two databases for grey andblack list flows are maintained. We use Low and high keywords
for defining rules for features that are packt_count,byte_count andduration_nsec which are first used to train the system and thenused to classify the test data. We have used UNB ISCX datasets
for training and evaluating our system. Results that are generated
after experimentation depicts the effectiveness of our system.
Detection rates of the developed application shows that it can be
widely applicable at larger data sets. In addition to detection, thissystem can further be used in order to develop a software toprevent the networks from intruders, which will enhance thereliability and demand of IDS systems.
.References

1.

M. Z. Masoud, Y. Jaradat, I. Jannoud, "On preventing ARP
poisoning attack utilizing Software Defined Network (SDN)
paradigm", Jordan Conference on Applied Electrical Engineeringand Computing Technologies (AEECT), pp. 1-5, 20152.

https://www.edgewaternetworks.com/blog/2014/07/benefits-sdnsolving-network-infrastructures-root-canal-problem/. Available atwww.edgewaternetworks.com Accessed on:[13-10-18]

3.

www.sdxcentral.com”what is SDN” Available
https://www.sdxcentral.com/sdn/definitions/sdncontrollers/openflow-controller/. Accessed on:[13-10-18]

4.

M.Hellman, “Fuzzy logic introduction”, Laboratoire Antennes
Radar Telecom, F.R.E CNRS 2272, Equipe Radar Polarimetrie.

5.

M. Erel; E.Teoman and Y. Özçevik etal “Scalability analysis andflow admission control in mininet-based SDN environment” . IEEE
Conference on Network Function Virtualization and Software
Defined Network. 2015 pp 18-19

6.

M. Enamul, K. Jiankun “A statistical Framework for Intrusion
Detection System” IEEE 11th International Conference on Fuzzy
Systems and Knowledge Discovery, 2014 , pp. 941-946

7.

www.sdxcentral.com,”Software Defined Networking”, 2017.
Available at: https://www.sdxcentral.com/sdn/definitions/what-thedefinition-of-software defined-networking-sdn/ Accessed:[10-0718].

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

691

Mahnoor Ejaz, Osama Sohail, Talha Naqash, Zain ul Abideen, Sajjad Hussain Shah8.

www.cisco.com,”What is a network switch or router”,Available:
https://www.cisco.com/c/en/us/solutions/small-business/resourcecenter/connect-employees-offices/network-switch-what.html
Accessed:[21-07-18

9.

M.Hellman, “Fuzzy logic introduction”, Laboratoire Antennes
Radar Telecom, F.R.E CNRS 2272, Equipe Radar Polarimetrie.

10. V.Ali,R.Maryam,S.Babak,”A fuzzy Intrusion Detection System
based on Categorized of Attacks”, 6th Conference on Information
and Knowledge Technology (lKT 2014), May 28-30, 2014,
Shahrood University of Technology, Tehran, Iran, pp. 50-55
11. Meng, Y., & Kwok, L. F. “Adaptive context-aware packet filter
scheme using statistic-based blacklist generation in network
intrusion detection.”, 7th International Conference on Information
Assurance and Security (IAS) 2011, pp. 74-79.
12. H.Yajam, K.Younes, A.Mohammadali , “Deniable Encryption
based on Standard RSA with OAEP”, 2016 8th International
Symposium on Telecommunications (IST’2016) pp. 84-88
13. H.Khalid, S.Jawad, V.Dillshad, M.Nafees, M.Awais, “An Adaptive
SYN Flooding attack Mitigation in DDOS Environment”, IJCSNSInternational Journal of Computer Science and Network Security,
VOL.16 No.7, July 2016.pp 27
14. Shanmugavadivu, R., & Nagarajan, N. “Network intrusion
detection system using fuzzy logic”. Indian Journal of Computer
Science and Engineering (IJCSE) 2011, 2(1),pp 101-111.
15. S.Kritsanapong , “The Improving Decryption Process of RSA by
Choosing New Private Key”, 8th International Conference on
Information Technology and Electrical Engineering (ICITEE),
Yogyakarta, Indonesia.2016, pp1-4© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan692

Development of Test-bed for SDN to know its feasibility of deployment on access layer
Rafay Shah, M Osama Shaikh, TalhaShaikh

BE Telecommunication, MUET (Mehran University of Engineering & technology),
Jamshoro, Pakistan 76090
Email: rafayshah26@gmail.com, osama.shaikh12369@gmail.com,talhaazmatshaikh21@gmail.com
https://www.muet.edu.pk

Abstract
Today, the network requirement of organizations includes enhancing performance, realizing broader connectivity
and security regulations. SDN technology makes easier for an IT Administrator to easily deploy and update
regulatory policies. We have developed a Test-bed kit using Zodiac-FX OpenFlow Switch for SDN and have tested
SDN feasibility on Access Layer. We have also simulated a network of Faculty of Electrical Electronics Computer
Engineering of MehranUET and have found that SDN performs better than traditional network architecture of
FEECE of MehranUET.
Keywords: SDN, Network, OpenFlow, SDN Controllers, Zodiac-FX

1. Introduction
Load Balancing
Firewall

Nowadays, organizations have been challenged by
the need of scalability in their network in order tomake growth of application. When OpenFlow
Protocols were implemented on the data centers,
since then service providers have been
implementing OpenFlow Protocols to ease theiroperations [6]. Today, the network requirement oforganizations includes:
Enhancing performance
Realizing broader connectivity
Security regulations
In order to comply with all of these criteria,
Networking Protocols have been evolving
significantly over the last few decades.

Implementing SDN on Northbound interface ofcontroller, it lets the installed applications tocommunicate with the controller. Northbound APIs
are used by different types of organizations including,Non-profit
Educational
IT Organization and others
In SDN, Southbound Interfaces are the OpenFlow
protocol which enables communication between
controllers, switches and other network nodes. Thisfurther lets the router to,
Identifies the topology of Network
Determine the Flows on the Networks

2. Software Defined Network Architecture
Yet, northbound interfaces allow devices tocommunicate among the higher level components.
While the traditional networks uses 2 mechanisms tocontrol packets on data plane.

Some of the popular southbound APIs are OpenFlow,
Cisco, and OpFlex and other vendors that support©The 2019International Conference on Artificial Life and Robotics (ICAROB2019),Jan. 10-13,B-ConPlaza, Beppu, Oita, Japan693

Rafay Shah, M Osama Shaikh, TalhaShaikh.

OpenFlow include IBM, Dell, Juniper, Arista andmore [7].
3.

OpenFlowswitch.It has various versions from version
1.0.0 to version 1.5.0 [6]. Thisprotocol is what mostuniquely identifies OpenFlow technology. At its
essence, the protocol consists of a set of messages thatare sent from the controller to the switch and acorresponding set of messages that are sent in the
opposite direction. The messages, collectively, allow
the controller to program the switch so as to allow finegrained control over the switching of user traffic [9].
The most basic programming defines, modifies anddeletes flows. The OpenFlow protocol has evolved
significantly with each version of OpenFlow, so we will
cover the detailed messages of the protocol in theversion-specific sections that follow. The specification
has evolved from development point release 0.2.0 on
March 28, 2008 through release V.1.5.0, released in
2014. Numerous point releases over the intervening
years have addressed problems with earlier releases andadded incremental functionality. OpenFlow was viewed
primarily as an experimental platform in its early years.
As such, there was little concern on the part of thedevelopment community advancing this standard toprovide for interoperability between releases. As
OpenFlow began to see more widespread commercial
deployment, backwards compatibility has become anincreasingly important issue. There are many features,
however, that were introduced in earlier versions of
OpenFlow that are no longer present in the current
version. Since the goal of this chapter is to provide aroadmap to understanding OpenFlow as it exists today,
we will take a hybrid approach of covering the major
releases that have occurred since V.1.0. We focus on
those key components of each release that became thebasis for the advances in subsequent releases and do not
focus on functionality in earlier releases that has been
subsumed by new features in subsequent releases [10].

Openflow Controllers

Controllers are the core of an SDN Network. In SDN,
Controller is an application that manages FlowControl
to enable networking. SDN Controllers are based on
protocols such as OpenFlow which allows server to tellswitches where to send packets [1]. It provides
communication between applications and devices. Any
communication between them goes through thecontroller as it chooses optimal path for traffic
applications. Controllers act as OS in networks.
Controller facilitates automated network management
and makes it easier to integrate and administrate.
Vendors of SDN Controller include:
Big Switch Networks
HPIBM
VM WareJuniper
RYU is a component based open source frame work
implemented via Python. It supports various component
including OpenFlow Protocols (version 1.1, 1.2, 1.3 and1.4), event management, applications management anddifferent services [4]. At API layer RYU has OpenStack Quantum Plugin which supports theconfigurations based on VLANs or GRE based overlay
and supports REST interfaces for its OpenFlow
operations.
POX controller is also referred as NOX and wasdeveloped by Nicira. It was the known as the first open
source OpenFlow Controller. It provides APIs tointeract with OpenFlow Switches including connection
handler. Additional components checks the availability
ofAPIs, provides Host Tracking, Routing, Creating andAnalyzing Topology. NOX/POX applications are

5.

Access layer provides a communication path andconnects hosts computer to the network. Access layer isreferred as third tier model of network. It uses TCP/IP
protocol hierarchy which defines network how totransmit IP datagram [11]. Typically a logical network
ID is created by associating physical port of a switch or
VLAN to a specific logical network ID with its own
routing protocol and forwarding tables. Wheneverpackets will be transmitted it will destined to that port
associated to VLAN ID on logical network. With SDN,
a controller can determine the logical network of every
flow and then tunnels the traffic to top the end of logicalnetworks.

SANE: SANE is an approach to represent
network as a file system.
Ethane: Ethane is a Stanford University
research application for centralized network security
[13].
4.

Implementation of SDN on Access Layer

Openflow Protocol

The OpenFlow protocol defines the communication
betweenan
OpenFlowcontroller
andan© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan694

Development of SDN Test-bed

Web Server
VPN Connector

To check the performance of SDN first of all we will
conFig. Zodiac FX switch and bind the controller withthe switch. We have used HPE Van SDN and RYUOpenFlow Controllers to check the connectivity
between controller and switch through ping commandfrom controller to switch. Once we get a stableconnection between switch and controller we added theflows by using GUI of controllers.
6.

SPI connector allows zodiac fx to connect otherdevices using the industry standard Serial Peripheral
Interface (SPI) and is a collection of 8 pins whereas
JTAG Debugger is used for developing zodiac
firmware and uploading using specially designed
cable. The zodiac fx provides serial connectivity
through
USB
port
for
performing
changes/configuration in switch. Zodiac FX also
provides web interface for the configurations in GUI.
Our SDN Test-bed Kit consists of Zodiac FX
OpenFlow SDN Switch and SDN Controllers. Firstly
we have conFig.d Zodiac FX IP Address, Subnet
Mask, OF-Controller IP Address and Gateway. Afterconfiguring Zodiac FX OpenFlow SDN Switch we
integrate it with different SDN Controllers i.e. HPE
Van SDN Controller, RYU SDN Controller and
POX SDN Controller. Once, successfully integrated
we have tested SDN Test-bed Kit and calculated
various results.

SDN Controllers and their integration

As shown in Fig.1, Zodiac FX switch has four ports inwhich port number 4 is native port and other ports areused to connect end devices. Connect port number 4
with controller using UTP cable as connection isestablished so port status will become up.

8. Results
Graph in Fig.2 shows the percentage of usage ofCPU by different controllers. This experiment was
performed on Zodiac-FX kit. Higher the percent of
CPU Usage the lower will be the performance.

Fig. 1: Zodiac FX Topology in HPE Van SDN
Controller

Zodiac-FX switch has four ports in which port number 4
is native port and other portsare used to connect end
devices. Connect port number 4 with controller using
UTP cable as connection is established so port statuswill become up.
Zodiac FX OpenFlow Switch has four ports in which
port number 4 is native port and other ports are used toconnect end devices. Connect port number 4 with
controller using UTP cable as connection is establishedso port status will become up.
7.

Fig. 2: CPU usage VS TimeThe values of graph in Fig. 2 is calculated using
SDN Controllers Usage Tool and shows that theperformance of RYU SDN Controller is better thanother SDN Controllers.

Zodiac FX Switch based Test-bed kit

Zodiac FX is a 4 port network development hardware
kit designed for network development and SDN
implementations. It is an open source firmware whichallows the user to modify or create Zodiacs firmware
according to their work of projects. For example

Throughput is calculated using IPERF tool. The
graph in Fig. 3 shows that the throughput of SDN isbetter than Traditional network.

Router and Bridge© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

695

Rafay Shah, M Osama Shaikh, TalhaShaikh.

Fig. 3: Throughput comparison

Fig. 5: Latency

Here it can be observed in Fig. 3that Traditional
Network, throughput, at 6 ms interval of time, was 91.5
Mbps whereas in SDN it was 94.5 Mbps. Hence, it canbe concluded that the SDN can perform much better
than Traditional Network.

To calculate latency we just use ping test. The graph inFig. 5 shows that the SDN Switches transfers 10 packets
within just 0.6 (ms) but the traditional switches took 0.8(ms) to 1.2 (ms) time to transfer 10 packets from source
to the destination.

The graph in Fig.4 shows the Throughput of SDN andtraditional network of FEECE Faculty.

9.

Conclusion and Future WorkIn our work, we have developed a Test-bed for SDN
andis based on Zodiac-FX OpenFlowswitch, on which
students can gain hands on experience on SDN and do
research on it. We have tested the feasibility of
deploying SDN on access layer. The performance isevaluated in terms of delay, throughput and scalability.
We also have simulated FEECE faculty of MUET on
SDN using mininet with different controllers. Threedifferent controllers have been used and these are HPE
VAN SDN Controller, RYU SDN Controller and POX
SDN Controller.

Fig. 4: Number of User versus Throughput
This graph show that the throughput of SDN is better
than Traditional Network. Here it can be observed thatin Traditional Network when the number of user was 40
throughput is 2 Mbps whereas in SDN throughput is 2.9
Mbps. Here it is also observed that SDN Switches havemaintained the throughput even the number of users
exceed up to 60 Users. Hence, it can be concluded that
the SDN Networking can perform much better than
Traditional Networking.

In this test-bed, we have used single Zodiac FX
OpenFlow switch and in future multiple switches canbe installed to test various/multiple topologies. We haveinstalled controller on a laptop but in future, controller
can be install on a dedicated Raspberry pi. We havesimulated FEECE faculty on mininet but in future,
deployment with hardware switches is also possible.
References

As shown in Fig. 4, the latency of the packets is less in
SDN than the latency of the packets in traditional
networks. This result is calculated on Zodiac-FX
especially for SDN and for traditional network, we havejust used simple common devices.

1.
2.

3.

H. Huang, S. Guo, P. Li, W. Liang, and A. Y. Zomaya.
Cost minimization for rule caching in software defined
K. Salah, K. Elbadawi, and R. Boutaba. Performancemodeling and analysis of network firewalls. IEEE
Transactions on Network and Service Management,
9(1):12–21, March 2012.
G. Hampel, M. Steiner, and Tian Bu. applying softwaredefined networking to the telecom domain. In 2013 IEEE
Conference on Computer Communications Workshops(INFOCOM WKSHPS), pages 133–138, April 2013.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan696

Development of SDN Test-bed

4.

S. Asadollahi, B. Goswami, and M. Sameer. Ryu
controller’s scalability experiment on software defined
networks. In 2018 IEEE International Conference on
Current Trends in Advanced Computing (ICCTAC),
pages 1–5, Feb 2018.
5. J. Alcorn, S. Melton, and C. E. Chow. Portable sdnTestbed prototype. In 2017 47th Annual IEEE/IFIP
International Conference on Dependable Systems andNetworks Workshops (DSN-W), pages 109–110, June
2017.
6. A. Bianco, R. Birke, L. Giraudo, and M. Palacin.
Openflow switching: Data plane Performance. In 2010
IEEE International Conference on Communications,pages 1–5, May 2010.
7. T. Huang, F. R. Yu, C. Zhang, J. Liu, J. Zhang, and Y.
Liu. A survey on large-scale software defined networking(sdn) Test-beds: Approaches and challenges. IEEE
Communications Surveys Tutorials, 19(2):891–917
Second-quarter 2017.
8. S. Khorsandroo and A. S. Tosun. An experimental
investigation of sdn controller live migration in virtual
data centers. In 2017 IEEE Conference on Network
Function Virtualization and Software Defined Networks(NFV-SDN), pages 309–314, Nov 2017.
9. V. Tanyingyong, M. Hidell, and P. Sjdin. Improving pcbased openflow switching performance. In 2010
ACM/IEEE Symposium on Architectures for Networking
and Communications Systems (ANCS), pages 1–2, Oct
2010.
10. R. Gopakumar, A. M. Unni, and V. P. Dhipin. An
adaptive algorithm for searching in flow tables of
openflow switches. In 2015 39th National Systems
Conference (NSC), pages 1–5, Dec 2015.
11. S. Zerrik, D. El Ouadghiri, and M. Bakhouya.
Performance evaluation of software defined networking
architectures using network calculus. In 2016 5thInternational Conference on Multimedia Computing and
Systems (ICMCS), pages 543–547, Sept 2016.
12. Y.Wang, Y.Dar, and G.Chang. Sdn-based dynamic
multipath forwarding for interdata center networking. In
2017 IEEE International Symposium on Local andMetropolitan Area Networks (LANMAN), pages 1–3,
June 2017.
13. R. Jain and S. Paul. Network virtualization and software
defined networking for cloud Computing: a survey. IEEE
Communications Magazine, 51(11):24–31, November
2013© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan697

Spectrum Sensing using Unsupervised Learning for Cognitive Radio
Asmara Shaukat, DanishKhan, TalalArshad, Haseeb Ahmad

College of Electrical Engineering, PAF Karachi Institute Of Economics And Technology,
Karachi, Pakistan
E-mail: asmarashaukat93@gmail.com,maddanish2050@gmail.com, Sam.samad22@gmail.com,
mianhaseebahmad149@gmail.com

Abstract
Cognitive Radio is the type of wireless communication which has the ability to learn from its surrounding and reconfigure its
operating parameters. It uses radio spectrum efficiently by detecting which communication channels are in use and which are not, andinstantly move into vacant channels while avoiding occupied ones. This optimizes the use of available radio-frequency (RF) spectrum
while minimizing interference to other users. Spectrum sensing is one of the most important tasks performed by the CR.Spectrum
sensing refers to the ability of a CR to measure the spectrum activities due to ongoing transmissions over different spectrum bands andto capture the related parameters, it tells us about spectrum usage and existence of primary users in a geographical area. In this work
we perform Spectrum sensing using k-mean clustering and find vacant spectrum.

Keywords: Cognitive Radio, Radio frequency spectrum, Primary user, Spectrum sensing, K mean clustering

the ability to observe and learn from its environment,
adapt to the environmentalconditions, and makedecisions to use the radio spectrum moreefficiently [2].
One of the most important task of CR is spectrum
sensing and detect Primary user activity. As soon as thePrimary user activity is detected, they have to vacate thechannel [3].

1. Introduction
Usable radio spectrum is a precious natural resource
whose allocation is governed by various government
bodies.This fixedassignmentof the frequencies hascaused spectrum scarcity [1].CR is an innovative
approach to wireless networking in which the radio
device is aware of its environment and has the ability toestablish and adjust its parameters autonomously. It has

2. Spectrum sensing
Spectrum sensing is one of the most important task
performed by the CR. Spectrum sensing refers to theability of a CR to measure the spectrum activities due toongoing transmissions over different spectrum bands
and to capture the related parameters. The ultimate
objective of the cognitive radio is to obtain the best
available spectrum through Cognitive Capability andRe-configurability as described above. Since there isFig.1.Cognitive Radio Working Principle.

©The 2019International Conference on Artificial Life and Robotics (ICAROB2019),Jan.10-13,B-ConPlaza, Beppu, Oita, Japan

698

Asmara Shaukat et al.
3.2. Algorithmic steps for k-means clustering

already a shortage of spectrum, the most important
challenge is to share the licensed spectrum without
interfering with the transmission of other licensed users.
The cognitive radio enables the usage of temporally
unused spectrum, which is referred to as spectrum holeor white space. If this band is further used by a licensed
user, the cognitive radio moves to another spectrum
hole or stays in the same band, altering its transmission
power level or modulation scheme to avoid
interference.Machine learning algorithms have been
widely used for the pattern classification problems,
where the feature vector extracted from the training datais fed into the classifier to categorize the pattern into acertain class. Spectrum sensing can be thought of as abinary-class classification problem. Weare using kmean clustering.
In general, two types of learning algorithms exist,
namely the “supervised” and the “unsupervised”. Incase of the supervised learning, the training feature
vectors are fed to the classifier with their actual labels;
while in case of the unsupervisedlearning, the same are
fed without any label [4].

Recalculate diatance between each data point andobtain new cluster center
Randomly select cluster centers
Assign data point to each cluster center whose
distance from the center is minimum
Recalculate the new cluster center
calculate distance beteen data point and cluster
centers
After we have these k new centroids, a new binding has
to be done between the same data set points and thenearest new center. A loop has been generated. As aresult of this loop we may notice that the k centerschange their location step by step until no morechanges are done or in other words centers do not move
any more. Finally, this algorithm aims at minimizing an
objective function knows as squared error function
given by:
4. Simulations and Results

3. K-mean clustering
K-means clustering is a type of unsupervised
learning, which is used to characterize data points ingroups. This characterization is done on the basis offeatures similarity. After every iteration, each data
points are assigned to a particular cluster. Two maintasks performed by K-mean clustering algorithmare:1.

To obtain the centroids of the K clusters, which
can be used to label new data

Fig.2: Signal values some are zero2.

Assign cluster to every data point.

3.1. K-mean clusteringAlgorithmk-means
is oneof thesimplest
unsupervised learning algorithms that solve thewellknown clustering problem. The procedure follows a
simple way to classify data in certain numberof clusters.

Fig. 3: Signal values© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan.10-13, B-Con Plaza, Beppu, Oita, Japan699

Spectrum Sensing using Unsupervised
3. K-mean clustering based cooperative spectrum
sensing in generalized к-μ fading channels,V.
Kumar, D. C. Kandpal, M. Jain, R. Gangopadhyay
and S. Debnath, 2016 Twenty Second National
Conference on Communication (NCC), Guwahati,
2016,pp.1-5.doi: 10.1109/NCC.2016.7561130

Fig. 4: Signal values more values are zeroFig. 5: Signal values no values are zeroFig.6: Signal values no values are zero

References
1. Techniques for Dealing with Uncertainty
in Cognitive Radio Networks Fatima Salahdine1,2, Naima
Kaabouch1, Hassan El Ghazi21Electrical Engineering
Department, University of North Dakota, Grand Forks,
USA2STRS Laboratory, National Institute of Posts andTelecommunication, Rabat, Morocco
2. Markov Model Based Jamming and Anti-Jamming
Performance Analysis for Cognitive Radio Networks
Wednel Cadeau, Xiaohua Li, Chengyu Xiong,Department of Electrical and Computer Engineering,
State University of New York at Binghamton,
Binghamton, USA.
© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan.10-13, B-Con Plaza, Beppu, Oita, Japan700

Tracking Secondary User in Cognitive radio for 5G Communication
Muhammad Adnan,Muhammad Umair,Sameed bin Junaid

College of Engineering PAF-KIET
Karachi ,Pakistan
E-mail: adnan13july1995@gmail.com, Engr.umairshahbaz@gmail.com

Abstract
Cognitive Radio (CR) is a promising technology which resolves the spectrum scarcity issues and allows
secondary user (SU) to use the vacant spectrum space without causing interference to the primary user (PU).
Many secondary users are allocated different spectrum bands to enhance spectrum utilization. SU co-exists with
the PU in the CR network and continuously monitors PU activities in order to use vacant space of PU in the
spectrum. The movement of SU in the CR network is random and therefore it is necessary to track the positionof SU and measure radio signal strength indicator (RSSI) in order to avoid interference with the PU. In this work,
we present the implementation and comparison of two algorithms KALMAN Filter and State Space Least Mean
Square (SSLMS) to track the position of SU in the CR network under log distance and log normal shadowing
path loss models. Both algorithms track the position of SU by measuring RSSI and Signal to Noise Ratio (SNR)
from the PU; however SSLMS gives better performance as compared to KALMAN Filter.
Keywords: Cognitive radio, Spectrum sensing, Energy detection, SSLMS

1. Introduction

secondary user as quick as possible and to track it
accurately, previously on different detection
techniques which are used to the sensing purpose of
the primary user such as cyclostationary detection,
wavelet detection , match filter energy detection [1, 2,
3].
Between the transmitter and receiver both have samefading effect which can create a problem of fluctuating
during sensing and by acknowledging these different
paths used for fading effect such as log distance pathloss and log normal shadowing ,these are the effected
propagation model between the transmitter andreceiver . In this paper we tried to approach theflexibility of Cognitive Radio in receive signal
strength based on the different position of moving
secondary user with respect to primary user receive
signal strength depends on the different factor andIn the most latest research paper on the behave ofattacker which create disturbance to the functionallyof Cognitive Radio network, because of several of user
at the same time detect the spectrum and utilize it fordifferent reasons and applications .cognitive Radio is
to solve the issues of the spectrum and design theprediction sense of availability of the channel inwireless communication.
However, Cognitive Radio potentially solve thespectrum utilization for the users which is directly
related to the secondary user and primaryuser .spectrum sensing is the major part of Cognitive
Radio and it helps to find the primary user whether itis present or not due to this approach most of theresearch work on detecting algorithm to track the© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan701

Muhammad Adnan,Muhammad Umair,Sameed bin Junaid

areas we are focusing on the distance between primaryuser and secondary user .
Most of the research is based on usage of KALMAN
filter for the detection of moving secondary user toestimate the position and velocity but KALMAN filter
have some covariance which is the main disadvantage
throughout the estimation because there are different
errors which are continuously effecting on the
behavior of estimation. KALMAN filter is used todetect the fault on the industrial fields and robotic
system, image and signal processing [4, 5] and in thefield of different criteria like navigation system,
automation, stock marketing.
Throughout the research we are interested to track the
S.U to estimate the secondary user with the help of
STATE SPACE LEAST MEAN SQUARE algorithm
is better than KALMAN filter like less computational
cost and in this algorithm there is no covariance whichmeans that there is no effect to estimate the position of
secondary user, after that we use energy detection
method to sense the primary user from probalistic
method using signal noise ratio (SNR).
The goal of this paper is to contribute the sensing
purpose to make the performance better in low cost ascompared to KALMAN filter of the real time detectionof primary user with respect to the moving mobility of
secondary user.

Fig. 1 Moving S.U around primary user
2.1 Propagation Model
Log-distance path loss is generic model which is used
to finding the path losses between the transmitter andreceiver. The Receive Signal Strength (RSS) isrepresented in dB between primary user and secondaryuser= ( ) -(P +10( )) (1)( ) =( )2. System Model

Log-distance model Eq. (1) we also represented by,(

We describe our scenario networks model in Fig. 1
which we point out our purpose in this paper, thismodel is solve by the mathematical equations. In ourstudy we have describe two basic approaches incognitive radio networks
The fixed primary user (base station).
Random movement mobility (secondaryuser) in different direction.

(

(

)

)

( )

(
)

(

)

)=

= -10
(

)

( )

-P

(2)
(3)

(4)
P = 20We assume the shadowing effect in log-distance pathloss, so we get the log normal shadowing due the one
additional part so the Eq. 2 is represented by,
(

(

)

( )

(
)

)

= - 10( )+(5)Where ( ) is the receive power, is the path loss
exponent in different areas,is the reference
distance so we set the ( = 1 ), is the distance
between transmitter and receiver, is the wavelengthof the signal, P reference path, these parameter are© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

702

Tracking Secondary User inused to determine the path losses propagation modelandis the Gaussian distribution with zero means
and standard deviation [6–7]. , we calculate the signal
to noise ratio (SNR) by using the restive signal
strength (RSS) between primary user and secondary
user is represented by
(

,

(

,

(
Eq. 7 .

) =
)(

(

)=

)

( )
(

performance at the secondary user the energy, we canwrite the Eq. 10 as follows,
∑ | |
E1 =
(11)
E=
∑ | +
|
E2 =
Eq.11 also expressed in as

(6)
)

( )(

)-

(

E =

) (7)

2.2. Spectrum Sensing ModelIn this model we represent local spectrum sensing inenergy detection, the sensing process or the detectionof the primary user divide into two parts first one iscooperative spectrum sensing (CSS) and the second
part is local spectrum sensing (LSS).for the local
spectrum we take the secondary user each has energy
detector which is used to sensing purpose for primaryuser and identifying the primary user is absent ( ) or
present ( ),•When the primary user is absent so the output signal
at the secondary user is defined as

Pd = P (E > δ
(13)
P

(12)

= P(E > δ)=)

( 2γ , √δ )

( , )
( )

(14)

is the Marcum Q function [9] which directly
determine the theoretical value, Γ is the gammafunction [8] and δ is the threshold decision to checkthe against energy of signal.
3. Results(8)

Scenario 1 (log distance path loss)•When the primary user is present so the output signal
at the secondary user is defined as
Y (t) = x (t) + w (t)→(2γ) →

x , x (2γ), these parameter to determine using thechi square distribution central and non-central with thedegree of freedom , (γ) is the real value of SNR
which are converted to from dB, to check the sensing
performance of the primary user, we use theprobability of detection against the false alarm
detection) is the noise power, so the SNR in dB is

Y (t) = w (t)E1 = xE2 = x(9)

Y (t) is the receive signal strength at the secondary
user under the AWGN channel which are use in bothcases log distance path loss and log normal shadowing,
after the receiving signal the secondary user decidedtoorand sense the primary user with respect tothe energy at the secondary user ,
∑ | |
E=
(10)
is the receive signal with the sample size , thesecondary user mobility continuously moving so thatthe effect on sensing which is bandwidth of signal, due
to this behavior we assume the two energy

Fig.2 SU movement© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

703

Muhammad Adnan,Muhammad Umair,Sameed bin Junaid

In Fig.4 the graph is in between signal to noise ratio
and iteration. In this figure we have assign a thresholdor a reference line that if the signal is above thereference we have high signal to noise ratio at different
position of secondary user.

In Fig.2 we are tracking secondary user throughprimary user for that we have to assign a coverage area.
If secondary user is in the coverage area the primaryuser will detect it and then we are predicting the future
value of secondary user through KALMAN filter and
SSLMS

Scenario 2 (log normal shadowing path loss)

Fig.5 SU movements
Fig.3 SU RSS measurements

In Fig.5 secondary user is out of the coverage area andthen we apply KALMAN filter and SSLMS to predict
the next value of secondary user through primary user
for that we have to assign a coverage area. If secondary
user is in the coverage area the primary user will detect
it and then we are predicting the future value of
secondary user through KALMAN filter and SSLMSIn Fig.3 the graph is in between receive signal strengthand iteration. In this figure we have assign a thresholdor a reference line that if the signal is above thereference we have high receive signal strength

Fig.4 SU SNR measurements
Fig.6 RSSI measurements© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan704

Tracking Secondary User inIn the above Fig. 5 we see that the secondary user isout of the coverage area then we calculate it’s receive
signal strength through KALMAN filter and SSLMS
in Fig. 6 and we see that its graph is decreasing tothreshold or reference line. and is also decreasing
calculate the receive signal strength then we see that if
it goes out of the coverage area the graph will go underthethresholdorreferencelin

Fig.8 Detection of SU
In Fig.8 this graph shows the probability detection vs.
false alarm at 10 dB, we observed that if the graph isaway from the diagonal the probability of detection isgreater and if it is near to diagonal the probability offalse alarm is greater.
Fig.7 SNR measurements
In Fig.7 the value of signal to noise ratio is decreasing
to threshold because the secondary user is out of thecoverage area that’s why its SNR is decreasing.
The graph is in between signal to noise ratio anditeration. In this figure we have assign a threshold or areference line that if the signal is above the reference
we have low signal to noise ratio at different positionof secondary user.

Fig.9 Detection of SU© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

705

Muhammad Adnan,Muhammad Umair,Sameed bin Junaid

8.

In Fig.9 this graph shows the probability detection vs.
false alarm at 30 dB, we compare both the above graphand observe that which have a greater probability ofdetection. If the value of signal to noise ratio is greater
the probability of detection is greater and vice versa

9.

And if this graph is away from the diagonal
the probability of detection is greater and if it
is near to diagonal the probability ofdetection is lower.

Gradshteyn, I. S., & Ryzhik, I. M. (1994).
Table of integrals, series, and products (5thed.). Waltham: Academic Press
Nuttall, A. H. (1975). Some integrals
involving the QM function. IEEE
Transactions on Information Theory, 21(1),95–96.

References
1.

2.

3.

4.

5.

6.
7.

Akyildz, I. F., Lo, B. F., & Balakrishnan, R.
(2011). Cooperative spectrum sensing in
cognitive radio networks: A survey. PhysicalCommunication, 4(1), 40–62.
Akyildz, I. F., Lee, W. Y., Vuran, M. C., &Mohanty, S. (2006). NeXt generation/dynamic spectrum access/cognitive
radio wireless networks: A survey. Computer
Networks, 50(13), 2127–2159.
Ben Letaief, K., & Zhang, W. (2007).
Chapter: Cooperative spectrum sensing.
Cognitive wireless communications networks(pp. 115–138). Berlin: Springer.
Michael, P., & Phillip A. L. (2003). Kalmanﬁlter recipes for real-time image processing.
Real-Time Imaging, 433–439.
Abhik, M., Partha, P. A., Nandi, P-K.,
Pinakpani, P., & Das, J. (2002). Estimation of
atmospheric boundary layer using Kalmanﬁlter technique. Signal Processing, 1763–
1771.
Andrea, G. (2005). Wireless communications.
Cambridge: Cambridge University Press.
Sohra, S., Mohammed, F., & Abdelmalek, A.
(2012). Seamless infrastructure independent
multihomed NEMO handoff using effective
and timely IEEE 802.21 MIH triggers.
International Journal of Wireless and Mobile
Networks (IJWMN), 4(3), 119–139.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

706

Wireless Power Transfer And Data Communication
For Biomedical Application
Hammad Saleem, Muhammad Awais, Siraj Din, Attequa, Asma Mahar

BEelectrical,PAF KIET(Karachi Institute of Economics and Technology),KorangiCreek
Karachi,75190, Pakistan
Email:hammadmughal265@gmail.com,awaischaudhry101@gmail.com,attequa.engineer@gmail.com,
qureshi8932@gmail.com,asmamahar@hotmail.com,www.pafkiet.edu.pk

Abstract
A technique of wireless power transfer(WPT) system for biomedical applications with backscatteringcommunication ,a rectifier use with active bias mechanism(dependent and independent stages) overcome thediode device losses has been presented in this paper .A part from conventional and static Vth cancellation
technique rectifiers, it achieve s more than double efficiency and achieves powerless reduction in both forward
and reverse biased condition s it decrease turn on voltage in forward bias condition along with decrease in reverse
leakage current during reverse bias condition. Under the input conditions (Vin=1Vp-p coupling coefficient’s=0.01
at 200MHzfrequency with transmitter and receiver inductanceof22nH),actively biased differential drive rectifier
sable to achieve DC voltage of 880mv for 50kΩ load resistance independent stage and achieves DC voltages of
803.2mv for 50kΩ load resistance in independent stage. Backscattering communication has been performed using
switch by changing the resonance frequency of the receiver.

Keywords: wireless power transfer (WPT), backscattering communication, inductor, rectifiers.

facilitated while using the same transmitting powerfrom the external device. To achieve the target, besides improving the performance of coil transfer, therectifier also plays a very important role in the total
efficiency of the WPT system.
The faraday’s law of induction was proposed byMichael Faraday in 1831, provided the basis foradvancements in wireless power transfer WPT systems.
The concept of WPT was not new, however this concept
was first found by Nikola Tesla centuries ago.In1891,
Nikola Tesla was performed resonant wireless powertransfer experiments and after four year she wasreceived patents in1894.In that patent she was claimed
the use of resonance to achieve high efficiency of WPT.
In 1960-70s wireless power transfer WPT was utilized
in many applications such as Bluetooth, mobile phone,
Biomedical and smart watches. In multiple biomedical

1. Introduction
Recently, wireless power transfer (WPT) technology
has become a very popular research topic,
particularly in applications related to portable device
chargers and biomedical devices. Since the WPTidea was proposed, many researchers have tested theprobability of applying this concept to commercial
applications. The WPT has been prevented fromwidespread used for a number of reasons, but the
major issues are the power transfer efficiency.
Therefore, the target of all research in the field of
WPT is to improve the efficiency of the WPTsystem so t hat less transmitting power is required
for a longer distance between the externaltransmitting device and the implant device can be©The 2019International Conference on Artificial Life and Robotics (ICAROB2019),Jan. 10-13,B-ConPlaza, Beppu, Oita, Japan

707

Hammad Saleem,Muhammad Awais,Siraj Din,Attequa,Asma Mahar

fig.1. It has an external device and an internal device.
The power is transferred through coupling among
primary and secondary coil. The primary coil whendriven by the power amplifier generates AC magnetic
fluxes. This variable flux couples to the secondary coiland produces an AC signal at the receiving end.
Received AC signal is rectified to power up device.
Most often this dc signal is also processed to send backdesired output voltage inside the chip. If the device has
sensor in it, and sensor data is required externally, thenit is sent instead of DC encoded signal over thebackscattering channel. Sensor data proper reception onreceiving end also ensures the required power
generation.
For backscattering communication, varactor or a switch
is connected in parallel to the receiver circuit to vary thepeak voltage at data frequency. This data is received
externally on a separate coil resonating at frequency ofbackscattered data. The frequency of data is kept below
the WPT frequency to keep the efficiency degradationof WPT minimum. This data is processed to extract
required information. For the available area of
710*710um2, highest possible Q, L and SRF is required
for a planar inductor design. The planar inductors have
many shapes such as square, hexagonal, octagonal or
circular. However circular planar inductor has highest Q
factor, while square planar inductor has the minimum.
But as the required inductor needs to couple withexternal coil, coupling coefficient ‘k’ is desired to be
maximized and the coupling coefficient ‘k’ is directly
dependent on the area of the coupling inductors. Where,
coupling coefficient ‘k’ is given by: = √ ∗ (1) If
the area of the coupling coils is not same, ‘k’ isdetermined by the area of the smaller coil whichsignifies the dependence of ‘k’ on the area of on-chip
inductor because off-chip coil could be kept larger.
Therefore, the square inductor is most suitable for suchapplications where device performance is dependent on
the coupling among coils. While square inductors Q
factor is also accounted for WPT performance, Lower Q
factor of the square inductor because of the sharp turns
at the edges which increases conduction losses of theinductor. Then the sharp edges are replaced by the
diagonally cutting edges which improves Q factor by
3.5%.
Custom inductor design with 25um width, spacing equal
to 6um, inner radius of 175um, outer radius of 355um
and number of turns equal to 6, Fig 2 shows was
simulated on EM Simulator for 150nm 6 metal A1
CMOS technology provided by L Foundry. Ports of
inductor are defined on same layer to maintainapplications using resonant coupling. With theadvancement in CMOS process, the integration of 2
dimensional and 3 dimensional inductors have opened
doors for wireless power transfer to these small size
devices.
In this work two different techniques are proposed toachieve regulating DC voltage along with enhancebackscattering communication. A high Q on-chipcustom inductor design is also presented to meet area
constraint of bio-medical implants. The on-chip receiver
inductor is used for both receiving power andcommunicating data back to the external device. Bothhighly efficient approaches are discussed with each ofthem having its own advantages in certain regard. While
striving for high WPT efficiency, receiving circuitry
efficiency is taken into account along with the rectifier
conversion efficiency. To ensure sufficient wirelesspower transfer through EM coupling, coupling
coefficient ‘k’ is estimated prior to the circuit design. ‘k’for a certain application is determined by the transmitted
power, inductor Q factor and distance between coils. To
maximize the signal reception at receiving end, high Q
inductor design has been proposed in [1]. [2] Suggests
an improved resonator with better transmission
efficiency in contrast to spiral and helical resonators
occupying same volume. Another design methodology
integrating rectifier with the antenna (rectenna) for
incident power as low as 5- 100uW/cm2 has beenpresented in [3].

Fig. 1. Inductive Wireless Power Transfer System block
diagram

In addition to the forward power link, most of theseapplications require a reverse data link to acquire
information on an external device for processing. The
data is sent back on a wireless communication channel
using the same on-chip receiver coil. Immense researchhas been carried out recently to avoid the degradation of
power efficiency due to conventional backscattering
Techniques [5]. This will be discussed later in section.
A complete pictorial representation of WPT systemalong with backscattering communication is shown in© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

708

Wireless Power Transfer For

proposed which automatically reduces reverse leakage
current by increasing Vth in reverse biased condition
while keeping Vth low during forward bias condition.
This active mechanism achieves double the efficiency
as compared to static Vth cancellation. The rectifier is across coupled configuration driven by a differential
input signal as shown in fig. 3. The WPT frequency is200MHz. When Vac+ has its positive cycle, Vac- is
negative hence reducing the turn on voltage. Similarly,
when Vac+ gets negative, Vac- is positive, hence
increasing Vth in the reverse biased condition.
Fig.

symmetry as different metal layers have different
resistivity and distance from substrate.

Fig. 2. Inductor Design

2. Major Headings
Fig. 3. Differential Drive Rectifier circuit (Dependent Stages)

Major headings should be typeset in boldface with thefirst letter of important words capitalized.
Sub-headings should be typeset in boldface italic and3. Differential Drive Rectifier with Active Bias
Mechanism
The power conversion efficiency of a rectifier circuit isdependent upon the forward and reverse biased power
loss of the diode connected devices. Where forward
power loss is due to the turn on voltage of the device
and reverse power loss is due to leakage current.
Power conversion efficiency (PCE) of rectifier isdefined as.
= −/ −P _ in = P _ out + P _ rectloss (3)Where, P_out and P_in represents output and input
power respectively show in eq 2 and P_rectloss
represents power loss across rectifier show in eq 3.

Fig. 4. Voltage at output of stage 1 and stage 2

The stage is copied twice with first stage providing bias
voltage to the second stage, shown in fig 4. This is
known as self Vth cancellation as the rectifier itself isproviding the required bias voltage.
Another interesting feature of this rectifier circuit is that
if the first and second stage N and PMOS devices areproperly matched, the intermediate stage capacitor canbe removed or a small capacitor could be sufficient to
reduce the voltage ripple. This is because the forward
current of NMOS can be sunk by the PMOS and viceversa.
Also, multiple stages of this rectifier can be used tofurther improve the output dc voltage according to therequirement. But as this circuit works on self-bias
mechanism, the input voltage should be sufficient4.1. Active Bias Mechanism (Dependent stages)
The static Vth cancellation techniques constantly
provide Vth at the gate and source of the transistor,
irrespective of the instantaneous input AC signal
Amplitude. This helps achieve very small forward bias
resistance but also increases the reverse leakage current.
In ref [4] and [5], an active bias mechanism has been

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan709

Hammad Saleem,Muhammad Awais,Siraj Din,Attequa,Asma Mahar

enough to provide Vth to the first stage devices to start
functioning.
4.2. Active Bias Mechanism (Independent stages)
Reconfigurable rectifier (dependent stages) was a good
rectifier in respect of voltage, but the problem is byincreasing its rectification stages the voltage are
increases but due to its dependent stage, the DC shifts
and connects to the second stage, By this reason thelower PMOS of second stage starts to lose the current
due to unavailability of negative peaks and by thereason the overall power efficiency started to decrease.
To avoid this problem, a new design is proposed that is
independent stages reconfigurable rectifier.
This rectifier has two stages that are doing individual
rectification. First stage generates positive DC andsecond stage generates negative DC show in fig 5, andwe drive a load by these two independent DC voltages.
As per its independent stage, voltage isn’t shifts to DC
at second stage and it’s both NMOS works accurately.
This circuit gives 80.3% efficiency on 50 K load, 82.2%
efficiency on 100 K load and 83% efficiency on 300 K
load show in fig 6.The difference voltage at load shownin fig 7.

Fig .6. Voltage at output of stage 1 and stage

Fig .7. Difference Voltage at Load

5. Backscattering Communication
Conventional backscattering employs a switch inparallel with the receiver coil. The amplitude of thereceived signal is varied by switching it ON for either ‘1’
or ‘0’ of the data, while keeping it OFF for the other.
The ON resistance is usually kept sufficient enough toobtain backscattering signal amplitude of 200-250mV.
This is shown in fig 8.The main concern of this paper isthat the efficiency of WPT should be least effected bybackscattering. Data rate is kept quite low as compared
to WPT frequency, but the dc for both ‘1’ and ‘0’ of thedata should be sufficient enough to meet therequirements of the device

Fig .5. Differential Drive Rectifier circuit (Independent
Stages)Fig .8. Conventional backscattering technique© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

710

Wireless Power Transfer For

6. Conclusion
A differential drive rectifier with active Vth cancellation
technique (Dependent and independent stages) has beenpresented. The output DC voltage generated is 0.835mV.
The efficiency improvement is considerable. Along withhigh efficiency, circuit shows a self- regulated output
DC voltage for different coupling conditions. Whichadds robustness to the design? But backscattering ispossible in dependent stage because of their dependency
backscattering not affect the input voltages but theirleakage current increase and if number of independent
states increased then it reduce their reverse leakage
current but backscattering was not efficient.
Backscattering using switch by changing the resonancefrequency of the receiver has been shown. For a 200mV
backscattered data, DC change of only 110mV wasobserved, effecting WPT efficiency by only 7%.
References[1] Chia-Chu Wu, Hung-Wei Chiu, Da-Sheng Lee, MinHsiang Chang, Chien-Chi Lu, and Chao-N ing Chang, “High
Q Inductor Design Using M odified Magnetic Substrate
Structure”, IEEE International Conference on Wireless Power
Transfer, pp 284-287, May 2014.
[2] WeiWeiYoshihiro Kawahara and Tohru As ami,
“Experimental Analysis of Double Spir al Resonator for
Wireless Power Transmissio n”, IEEE transaction, pp 9-12,
2013.
[3] Zoya Popovic, “Far-Field Wireless Power Deli very andPower Management for Low-Power Sensors”, IEEE
transaction, pp 1-4, 2013.
[4] K. Kotani and T. Ito, “Self-Vth-cancellation highefficiency CMOS rectifier circuit for UHF RFIDs,” IEICE
Trans. Electron., vol. E92-C no. 1, pp. 153–160, Jan. 2009[5] A.Sasaki, K.Kotani, and T.Ito, “Differential-drive CMOS
rectifier for UHF RFIDs with 66% PCE at -12 dBm input,” inProc. IEEE ASSCC, pp. 105–108, Nov. 2008.

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019),Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

711

Feasibility Study of UAV Implementation In Route Surveying
Hazry Desa

Centre of Excellence for Unmanned Aerial Systems (COEUAS), UniMAP, Kangar, Perlis
Kangar, Perlis, Malaysia
Mohd Azizi Azizan

Centre of Excellence for Unmanned Aerial Systems (COEUAS), UniMAP, Kangar, Perlis
Kangar, Perlis, Malaysia
Safwan Suhaimi

Centre of Excellence for Unmanned Aerial Systems (COEUAS), UniMAP, Kangar, Perlis
Kangar, Perlis, Malaysia
Zulaiha Ramli

Centre of Excellence for Unmanned Aerial Systems (COEUAS), UniMAP, Kangar, Perlis
Kangar, Perlis, Malaysia
Mohamad Syafiq Abdul KhadirCentre of Excellence for Unmanned Aerial Systems (COEUAS), UniMAP, Kangar, Perlis
Kangar, Perlis, Malaysia
Zainuddin Hat

IP Fokus Sdn Bhd
Bandar Baru Bangi, Selangor, Malaysia
E-mail: aziziazian@unimap.edu.my, msyafiq@unimap.edu.my
www.unimap.edu.my

Abstract
Unmanned Aerial Vehicle (UAV) are widely used in numerous field and the technology keeps growing. Generally,
conventional method use in data collection for engineering work is tedious and requires a lot of manpower. This
research focuses on the investigation of the suitability of UAV for route survey. Three stages involved: datacollection, data processing and data analysis. By using KAMA BETA, the data was collected. For data processing,
PIX4D MAPPEER used for point cloud and AutoCAD 2014 for analysis stage. The result is mainly focused on
extracting the road profile and test the point cloud data using RMSE. The result from both method, conventionaland UAV from the point cloud data using RMSE show only the small difference, with only 6.67% from total 60
points out of tolerance.
Keywords: UAV, engineering work, land survey, drone.

engineering works such as highways, pipelines, canals
or railroads. Survey operations in route survey include
topographic survey, cross-section and longitudinal
section [2].
Unmanned aerial vehicles (UAVs), also known as“drones”, are among the most important technologicaladvances that have been introduced to the land
surveying industry in quite some time. It can be piloted
by remote and can take detailed survey information
while simultaneously transmitting that data back to thehead office. UAVs create a highly accurate map andprovide valuable data to companies and individuals who

1. Introduction
Engineering survey is a survey undertaken for the
purpose of obtaining information essential to theplanning of an engineering project. Engineering survey
provides accurate and reliable dimension data throughthe construction process from the base topographicalsurvey, cross-section and ground modeling information
to cut and fill calculation, dimensional control tosetting-out on-site and finished as-built drawing [1].
Route surveying is comprised of all survey
operation required for designing and construction of© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan712

Hazry Desa, Mohd Azizi Azizan, Safwan Suhaimi, Zulaiha Ramli, Mohamad Syafiq Abdul Khadir, Zainuddin Hat

camera calibration matrix, including non-linear
distortion coefficient.
To check the accuracy of the camera calibration
results the total final error must be checked. In bothmethods the data sets are not the same. For lab
calibration the data set is the grid of the pattern and for
filed calibration are the 67 targets points. According to
PhotoModeler tutorial a value less than 1.0 pixel
indicates a good calibration and very good calibrations
can have a final total error smaller than 0.4 pixels(www.photomodeler.com). In this case, the lab
calibration has a final error of 1.940 pixels.

are considering major projects on large areas of land.
Research about the reliability of these techniques to be
used in topographic mapping was conducted in land
survey method [3].
2. Methodology
2.1. Planning and Preparation
Planning and preparation are needed before the datacollection. Planning and preparation cover the scope ofwork such as planning for the area of the survey and the
preparation for the instrument and software used in thisproject. A good planning and preparation enable toensure efficiency and maintaining a high quality of thedata collection in this study. Furthermore, proper
planning and preparation help in saving time and cost as
time increase, the cost will increase as well. The figure
below shows general planning and preparation for UAV
and TS.

Table 1. Total final error and residuals of the camera
calibration projects.
The total error a bit higher than the recommended. The
field calibration has a total error of 0.282 pixels, whichis assumed to be a very good calibration project. Alsochecking the marking residuals is a good way to test thecalibration quality [6]. PhotoModeler tutorialrecommended having a largest marking residual lessthan 1.0 pixel [3]. In both cases the largest marking
residuals are less than 1.0 pixel. The lab calibration has
0.723 pixels and the field calibration 0.700 pixels(Table 1).
The accuracy of field calibration was also checked
to compare the GPS coordinates of the targets with thecoordinates obtained with PhotoModeler. This software

2.2. Data Collection
Data collection for this study is focused into two
methods. The first method of data is used as aconventional method by total station. Besides that, thesecond method for this study used a new method by
Unmanned Aerial Vehicle (UAV).
2.3. Data Processing
Data processing for this study refers to processing forthe laser and the conventional method. Data processing
for a conventional method uses the Civil Design &Survey software. Besides that, for the UAV processing
used Pix4D Mapper. Both use AutoCAD as data
analysis processing.

3.

Final total error (pixel)
Largest marking residual(pixel)Overall RMS (pixel)Labcalibration
1.940
0.723

Field calibration(50m flight high)
0.282

0.245

0.341

0.700

only needs three control points to change from relative
to absolute coordinates. For this process, the targets
number one, two and four were used. The planimetricand the altimetric RMS were calculated. The
planimetric RMS was 0.028 m and the altimetric 0.026
m, which is really small and indicates the accuracy
level of the project.

Result and Analysis

3.1. Camera Calibration
The camera calibration is done before do the flightplanning. The purpose of camera calibration is todetermine the focal length, principal point, and lensdistortion from the camera. This process was carried out
to figure out the unstable element in the camera likeinterior orientation and lens distortion parameters [5].
In this project, the camera calibration is processedusing PhotoModeler Software. This software is an
automatic lens calibration, which uses A1 paper sheet
as a calibration target. It supports estimations of the full

3.2. Georeferencing
Ground control point (GCP) and verification point(VP) are very important data that are used to involve inaerial triangulation phase, where it can be identified on
the ground features and can be georeferenced with theaerial image used in geometric correction of the
distorted image [8]. Therefore, pre-marking for GCP’sand VP’s must be carried out before a flight mission.
This is because the processing of the tie point image© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

713

Feasibility Study of UAV

could be more efficient and accurate. The coordinatesand z-value that were observed using leveling method(land survey).

color is referring to the lowest elevation [8].Therefore,
it can clearly be seen that the UAV data gives a greaternumber of point compare to TS data. Besides that, theprofile generated by using UAV is denser than the TSdata due to the great number of point [9].

Fig. 3. DSM of the project area (Location: Chuping,
Perlis, Malaysia).

Fig. 2. Location of GCP in UAV images in Chuping,
Perlis, Malaysia (Orthophoto).

3.4. Statistical Analysis

Figure 2 shows the strategic location of placement six(6) GCP’s. The location of GCP was in Chuping, Perlis,
Malaysia. Each of GCP is observed by GPS equipment(Topcon – GR5) using Static Method. The reason ofthis method selection because this method is the mostwidely used differential technique for control andgeodetic surveying, involved 1 hour of observation inorder to resolve the integer ambiguities between thesatellite and the receiver. By using this method, theaccuracies in the sub-centimeter range can be obtained.

The road profiles are tested with Root Mean Square
Error (RMSE) which is done by analyzing comparisonof differences between and within the group. The roadprofile is statistically tested by used the RMSE test at
each chainage to ensure that the data of UAV and TS issignificant to each other.
Root Mean Square Error (RMSE) is the standard
deviation of the residuals (prediction errors). Residuals
are a measure of how far from the regression line data
points are; RMSE is a measure of how to spread out
these residuals are. In other words, it tells you how
concentrated the data is around the line of best fit. Rootmean square error is commonly used in climatology,
forecasting, and regression analysis to verify
experimental results [9].

3.3. Digital Surface Model (DSM)
Aerial image was taken from above with 14 minutes
flight duration. To achieve this project the overlap is80%, sidelap is 70% and the altitude is 110 meters. Thisoverlapping and flight altitude is required to avoid
potential missing areas of coverage and to ensure goodcoverage. Based on the data that has been processedusing Pix4D software produced an Orthophoto and
DSM in raster as shown in Figure 2 and Figure 3.
Figure 3 shows the sample result road profile of
UAV and TS data for CH 1 to CH 10, which is locatedin Chuping, Perlis, Malaysia. Based on Figure 3, the redcolor is referring to the highest elevation while soft blue

The formula is:(1)
Where:
Σ = summation (“add up”)
(zfi – Zoi)Sup>2 = differences, squared
N = sample size

© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan714

Hazry Desa, Mohd Azizi Azizan, Safwan Suhaimi, Zulaiha Ramli, Mohamad Syafiq Abdul Khadir, Zainuddin Hat

focusing on planning for a number of GCP and flightplanning. Number of GCP depend on a number of flightplanning where need minimum visible from the aerialview and location of GCP. After that, the procedure forroute survey by using UAV need to focus on processing
the point cloud data. Starting with data registration
point cloud data use minimum three images that visible
each other. Then, the next process is point clouddensification using PIX4D Mapper software. Then next
process produces DSM and Orthophoto. In this research,
two methods was applied, land survey and UAV survey.
A land survey is used as a benchmark for thisresearch. Point cloud data produce in 2D data with 3D
coordinate which can use to generate road profile forroute survey. The third objective for this study toanalyses the accuracy between land survey data and
UAV data. Series of test and analysis have been
conducted in term of the reliability of UAV data incompleting route survey. The reliability of UAV datashows in quantitative assessment by comparing resultroad profile from UAV and TS method.

There is three type of sample are tested using RMSE
formula, verification point, positioning and elevation.
Values from the real measurement are compared withthe value from the 3D modeling to get the error of every
sample.
Based on Figure 4, it can be seen that the graphvalues of RMSE Verification Points for minimum andmaximum errors. All the VPs are under tolerance which
is below ±0.10m.Based on Figure 5, it can be seen thatthe graph values of RMSE Cross Section for minimum
and maximum errors for Different Elevation (H). The
best result is ±0.03m (VP5). VP9, VP10, and VP18 areout of tolerance which is above ±0.10m.
The values of RMSE XY for minimum andmaximum errors can be referred to Figure 6. All the XY
are under tolerance which is below ±0.10m except at
VP19 with value ±0.125m. These results might beaffected by image matching algorithm that was used in
the same software during image processing. The error
was usually caused by flying height during image
acquisition, image matching during image processing
and motion movements such as omega, phi, and kappa[10].

Acknowledgements
The author acknowledge ICAROB for giving theopportunity to join this conference. Highly appreciation
to UAS research team for excellent contribution. A
special gratitude to IP Fokus Sdn Bhd for full support incompleting this project.

4. Conclusion
As a conclusion, three objectives of this study has
been achieved. The first objective to process image
capture from Unmanned Aerial Vehicle (UAV). The
most important procedure for route survey work is datacollection. Planning and preparation in this study
Fig.4. RMSE error verification points.

RESIDUAL E RROR VERIFICATION POINTS
0.100

Err or0.050
0.000-0.050-0.100
VP 1 VP 2 VP 3 VP 4 VP 5 VP 6 VP 7 VP 8 VP 9
Diff Easting (E)

VP
10

VP
11

VP
12

VP
13

VP
14

VP
15

VP
16

VP
17

VP
18

VP
19

VP
20

-0.08 -0.07 0.020 0.060 0.050 0.020 -0.09 0.040 0.050 0.060 0.040 -0.05 0.070 0.080 -0.06 0.050 0.050 0.070 0.060 0.020

Diff Northing (N) 0.090 0.020 0.070 0.070 0.060 0.030 -0.07 -0.05 0.040 0.070 0.060 0.050 -0.06 -0.08 0.080 0.070 -0.03 -0.01 0.090 -0.02
Diff Elevation (H) -0.04 -0.03 -0.05 0.060 -0.07 0.040 -0.07 0.040 -0.07 0.060 0.070 0.080 0.040 0.070 0.060 0.050 -0.04 0.090 0.050 0.080© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan

715

Feasibility Study of UAV

Fig. 5. RMSE error cross section.

Err orRESIDUAL ERROR CROSS SECTION
0.150
0.100
0.050
0.000
-0.050-0.100-0.150
VP 1 VP2 VP3 VP4 VP5 VP6 VP7 VP8 VP9Diff Easting ( E )VP1 VP1 VP1 VP1 VP1 VP1 VP1 VP1 VP1 VP1 VP2
0
1
2
3
4
5
6
7
8
9
0

0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

Diff Northing ( N ) 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
Diff Elevation ( H ) 0.070 0.073 -0.08 -0.09 -0.03 0.093 -0.06 0.032 -0.10 0.110 0.038 0.034 -0.05 -0.05 0.030 -0.09 0.008 0.100 0.041 -0.07

Fig. 6. Residual Error XY.
0.140
0.120

RESIDUAL ERROR XY

0.100

Er ror

0.080
0.060
0.040
0.020
0.000

VP 1 VP2 VP3 VP4 VP5 VP6 VP7 VP8 VP9 VP10 VP11 VP12 VP13 VP14 VP15 VP16 VP17 VP18 VP19 VP20

Diff Distance ( m ) 0.037 0.072 0.032 0.035 0.040 0.045 0.084 0.093 0.087 0.062 0.012 0.042 0.082 0.099 0.027 0.085 0.054 0.035 0.125 0.066
Diff Northing ( N ) 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
Diff Elevation ( H ) 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

References
6.
1. Ahmad, A., Hashim, K. A., & Samad, A. M.
(2010).Aerial mapping using high resolution digital
camera and unmanned aerial vehicle for Geographical
Information System. 2010 6th International Colloquiumon
Signal
Processing&
itsApplications.
doi:10.1109/cspa.2010.5545303
2. Eisenbeiss, H. (2004). A mini unmanned aerial vehicle(UAV): system overview andimage acquisition. 1.
3. Everaerts, J. (2008). The use of unmanned aerial vehicles(UAVs) for remote sensing and mapping. 1-2.
4. Peterman, V., & Mesaric, M. (2012). Land Survey from
Unmanned Aerial Vehicle. Int. Arch. Photogramm.
Remote Sens. Spatial Inf. Sci, XXXIX-B1, 447-451.
doi:10.5194/isprsarchives-xxxix-b1-447-2012
5. Pérez, M., Aguera, F., & Carvajal, F. (2012). Digital
Camera Calibration Using Images Taken From an

7.

8.
9.

10.

Unmanned Aerial Vehicle. Int. Arch. Photogramm.
Remote Sens. Spatial Inf. Sci, XXXVIII-1/C22, 167-171.
doi:10.5194/isprsarchives-xxxviii-1-c22-167-2011
Samad, A. M., Kamarulzaman, N., Hamdani, M. A.,
Mastor, T. A., & Hashim, K. A. (2013). The potential of
Unmanned Aerial Vehicle (UAV) for civilian andmapping application. 2013 IEEE 3rd International
Conference on System Engineering and Technology.
doi:10.1109/icsengt.2013.6650191
Zulasmin,W. and Ibrahim,W. “Towards a Sustainable
Quarry Industry in Malaysia
Limestone
andGranite Quarry Industry,” no. December, 2007.
Alias, R.B (2013). Statistic for science and engineering.
Kuala Lumpur, Malaysia : Universiti Technologi MARA
Bienert a. s. schiller. keana, e ., mohan f and nugent, c(2007). Tree detection and diameter estimation by
analysis forest
Diego Gonzalez-aguilera, angle munoz-nieto, Javier© The 2019 International Conference on Artificial Life and Robotics (ICAROB2019), Jan. 10-13, B-Con Plaza, Beppu, Oita, Japan716