１－５１6

ICAROB2018

PROCEEDINGS OF THE 2018
INTERNATIONAL CONFERENCE
ON ARTIFICIAL LIFE AND ROBOTICS
February 1- 4 2018
B-Con Plaza, Beppu, Oita, JAPANInternational Meeting Series
Editor-in-Chief
Masanori Sugisaka
Editors: Yingmin Jia, Takao Ito, Ju-Jang Lee
ISBN 978-4-9908350-3-3

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Proceedings of The 2018 International Conference on

ARTIFICIAL LIFE AND ROBOTICS(ICAROB2018)

February 1- 4, 2018
B-Con Plaza, Beppu, Oita, JAPANInternational Meeting Series
Editor-in-Chief
Masanori Sugisaka
Editors: Yingmin Jia, Takao Ito, Ju-Jang Lee
ISBN 978-4-9908350-3-3©ICAROB 2018 ALife Robotics Corp. Ltd.

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Contents
1

Organization, etc.

1

2

Messages

93Time Table

14

4

Opening Ceremony

17

5Technical paper index

18

6Abstracts
PS abstracts
IS abstracts
OS abstracts
GS abstracts

41
42
44
88

7

Authors index

1068

Conference room

117

6-1
6-2
6-3
6-4

©ICAROB 2018 ALife Robotics Corp. Ltd.

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

SPONSEREDICAROB society (ALife Robotics Corporation Ltd.)ORGANIZED BYInternational Steering Committee of International Conference on Artificial Life and Robotics(ICAROB)TECHNICAL CO-SPONSORED BYIEEE Fukuoka Section (Japan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-1-The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

CO-ORGANIZED BY
Chinese Association for Artificial Intelligence (CAAI, P. R. China)

SUPPORTRD BY
Oita Prefectural Government
Oita Prefectural Tourism association (Tourism Oita)
ADVISORY COMMITTEE CHAIR
Moshe Kam (New Jersy Institute of Technology, Former IEEE President, USA)ADVISORY COMMITTEE
Adam Grzech (Wroclaw University of Technology, Poland)
Bruce Eisenstein (Drexel University, Former IEEE President, USA)
Guang-Ren Duan (Harbin Institute of Technology, P.R.China)
Hidenori Kimura (JST, Japan)
Jeffrey Johnson (The Open University, UK)
Jerzy Świątek (Wroclaw University of Technology, Poland)
Joshua M. Epstein (The Johns Hopkins University, USA)
Kai-Tai Song (National Chiao Tung University, Taiwan)
Masayoshi Tomizuka (University of California Berkeley, USA)
Moshe Kam (New Jersey Institute of Technology, Former IEEE President, USA)Paul Kalata (Drexel University, USA)Paul Oh (Drexel University, USA)
Robert Fischl (Drexel University, USA)
Steen Rasmussen (University of Southern Denmark, Denmark)©ICAROB 2018 ALife Robotics Corp. Ltd.

-2-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GENERAL CHAIR
Masanori Sugisaka (Alife Robotics Corporation Ltd., Japan)(Visiting Professor, Open University, UK)
CO-GENERAL CHAIRS
Yingmin Jia (Beihang Univesty, P. R. China)
Takao Ito (Hiroshima University, Japan)
Ju-Jang Lee (KAIST, Korea)
VICE GENERAL CHAIR
Henrik. H. Lund (Technical University of Denmark, Denmark)John. L. Casti (International Institute for Applied Systems Analysis, Austria)
Jangmyung Lee (Pusan National University, Korea)
Luigi Pagliarini (Technical University of Denmark, Denmark)(Academy of Fine Arts of Macerata, Italy)
PROGRAM CHAIRMAN
Makoto Sakamoto (University of Miyazaki, Japan)
CO-PROGRAM CHAIR
Marion Oswald (Technische Universität Wien, Austria)
INTERNATIONAL ORGANIZING COMMITTEE
Akira Fukuda (Kyushu University, Japan)
Andrew Gadsden (University of Guelph, Canada)
Caoli Wang (University of Shanghai for Science and Technology, P. R. China)
Chan Gook Park (Seoul National University, Korea)
Eiji Hayashi (Kyushu Institute of Technology, Japan)
Evgeni Magid (Kazan Federal University, Russia)
Fuzhong Wang (Henan Polytechnic University, P. R. China)
Hazry Desa (University of Malaysia, Perlis, Malaysia)
Hidehiko Yamamoto (Gifu University, Japan)
Hiroki Tamura (University of Miyazaki, Japan)
Hiroshi Kage ((Mitsubishi Electric Corporation, Japan)
Hiroshi Matsuno (Yamaguchi University, Japan)
Jiwu Wang (Beijing Jiaotong University, P. R. China)©ICAROB 2018 ALife Robotics Corp. Ltd.

-3-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Jovana Jovic (CNRS-AIST JRL, Japan, France)
Junping Du (Beijing University of Posts and Telecommunications, P. R. China)
Katsunori Shimohara (Doshisha University, Japan)
Kazuo Ishii (Kyushu Institute of Technology, Japan)
Kevin Voges (Canterbury University, New Zealand)
Kuo-Hsien Hsia (Far East University, Taiwan)
Kuo-Lan Su (National Yunlin University of Science and Technology, Taiwan)
Kyungho Park (Micro-System Programs, RDECOM, ITC-PAC, U.S. ARMY, Japan and USA)
Masao Kubo (National Defense Academy of Japan, Japan)
Masanao Obayashi (Yamaguchi University, Japan)
Maxim Sokolov (Innopolis University, Russia)
Mehta Rajiv (New Jersey Institute of Technology, USA)Peter Sapaty (Ukrainian Academy of Science, Ukraine)
Pierre Parrend (University of Strasbourg, France)
Qiang Cai (Beijing Technology and Business University, P. R. China)
Takashi Kohno (LIMMS/CNRS-IIS, Institute of Industrial Science, The University of Tokyo, Japan)
Takashi Ogata (Iwate Prefectural Univerity, Japan)
Teruhisa Hochin (Kyoto Prefectural University, Japan)
Tetsuro Hattori (Kagawa University, Japan)
Thi Thi Zin (University of Miyazaki, Japan)
Toru Yamamoto (Hiroshima University, Japan)
Victor Berdonosov (Komsomolsk-on-Amur State University of Technology, Russia)
Yasunari Yoshitomi (Kyoto Prefectural University, Japan)
Yi Chai (Chongqing University, P. R. China)
Yo Horikawa (Kagawa University, Japan)
Yoshifumi Morita (Nagoya Institute of Technology, Japan)
Yoshiro Imai (Kagawa University, Japan)
Zengqiang Chen (Nankai University, P. R. China)
INTERNATIONAL PROGRAM COMMITTEE
Akira Nakamura (AIST, Japan)
Ali Selamat (University of Technology of Malaysia (UTM), Malaysia)
Andre Rosendo (ShanghaiTech University, P. R. China)
Bin Zhang (Beijing University of Posts and Telecommunications, P. R. China)
Arsit Boonyaprapasorn (Chulachomkloa Royal Military Academy, Thailand)
Dongmei Ai (University of Science and Technology Beijing, P. R. China)
Endra Joelianto (Bandung Institute of Technology, Indonesia)
Fengzhi Dai (Tianjin University of Science & Technology, P. R. China)
Haisheng Li (Beijing Technology and Business University, P. R. China)
Haruna Matsushita (Kagawa University, Japan)
Hidetsugu Suto (Muroran Institute of Technology, Japan)
Hiroyuki Iizuka (Osaka University, Japan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-4-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Huailin Zhao (Shanghai Institute of Technology, P. R. China)
Hussein Abbass (University of New South Wales and ADFA, Australia)
Istvan Harmati (Budapest Institute of Technology and Economics, Hungary)
Ivan Tanev (Doshisha University, Japan)
Jiandong Zhao (Beijing Jiaotong University, P. R. China)
Joono Cheong (Korea University, Korea)
Kathryn Elizabeth Merrick (University of New South Wales and ADFA, Australia)
Kunikazu Kobayashi (Aichi Prefectural University, Japan)
Lin Li (University of Shanghai for Science and Technology, P. R. China)
Lin Zhao (Qingdao University, P. R. China)
Manabu Yamada (Nagoya Institute of Technology, Japan)
Masahide Ito (Aichi Prefectural University, Japan)
Masayoshi Tabuse (Kyoto Prefectural University, Japan
Masaomi Hatakeyama (University of Zurich, Switzerland)
Norrima Mokhtar (University of Malaya, Malaysia)
Noritaka Sato (Nagoya Institute of Technology, Japan)
Palakorn Tantrakool (King Mongkut’s Institute of Technology Thonburi, Thailand)
Ping Wang (North China Electric Power University, P. R. China)
Qinglin Sun (Nankai University, P. R. China)
Satoshi Ikeda (The University of Miyazaki, Japan)
Sanjay S. Joshi (College of Engineering, University of California, USA)
Seong-Ik Han (Pusan National University, Korea)
Shingo Mabu (Yamaguchi University, Japan)
Shumin Fei (Southeast University, P. R. China)
Shyi-Ming Chen (National Taichung University of Education, Taiwan)
Takashi Kuremoto (Yamaguchi University, Japan)
Takayoshi Yamada (Gifu University, Japan)
Taishiro Kishimoto (Keio University, Japan)
Tetsuro Katayama (The University of Miyazaki, Japan)
Thunyaseth Sethaput (Thammasat University, Thailand)
Toshinori Nawata (Kumamoto National College of Technology, Japan)
Tsunehiro Yoshinaga (Tokuyama National College of Technology, Japan)
Weicun Zhang (University of Science and Technology Beijing, P. R. China)
Xiaoyan Fu (Capital Normal University, P. R. China)
Xuemei Ren (Beijing Institute of Technology, P. R. China)
Yunzhong Song (Henan Polytechnic University, P. R. China)
LOCAL ARRANGEMENT COMMITTEE
Makoto Sakamoto (University of Miyazaki, Japan)
Masanori Sugisaka (ALife Robotics Corporation Ltd., Japan)(Visiting Professor, Open University, UK)
Takao Ito (Hiroshima University, Japan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-5-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

HISTORY
The International Conference on Artificial Life and Robotics (ICAROB) resulted from the AROBsymposium (International Symposium on Artificial Life and Robotics) whose first edition was held in1996 and the eighteenth and last edition in 2013. The AROB symposium was annually organized by
Oita University, Nippon Bunri University (NBU), and ALife Robotics Corporation Ltd., under thesponsorship of the Science and Technology Policy Bureau, the Ministry of Education, Science, Sports,
and Culture (Monbusho), presently, the Ministry of Education, Culture, Sports, Science, andTechnology (Monkasho), Japanese Government, Japan Society for the Promotion of Science (JSPS),
the Commemorative Organization for the Japan World Exposition ('70), Air Force Office of Scientific
Research, Asian Office of Aerospace Research and Development (AFOSR/AOARD), USA. I would liketo express my sincere thanks to not only Monkasho (annually fund support from 1996 to 2013) but
also JSPS, the Commemorative Organization for the Japan World Exposition ('70), and various otherJapanese companies for their repeated support. The old symposium (this symposium has been held
every year at B-Con Plaza, Beppu, Oita, Japan except in Oita, Japan (AROB 5th ’00) and in Tokyo, Japan(AROB 6th ’01).) was organized by the International Organizing Committee of AROB and was cooperated by the Santa Fe Institute (USA), RSJ, IEEJ, ICASE (Now ICROS) (Korea), CAAI (P. R. China),
ISCIE, IEICE, IEEE (Japan Council), JARA, and SICE. The old AROB-symposium expanded much byabsorbing much new knowledge and technologies into it. This history and character of the former
AROB symposiums are passed on the current ICAROB conference and to this journal, International
Journal of Robotics, Networking and Artificial Life (JRNAL). From now on, ALife Robotics Corporation
Ltd. is in charge of management of both the conference and the journal. The future of the ICAROB isbrilliant from a point of view of yielding new technologies to human society in the 21st century. Thisconference invites you all.
AIMS AND SCOPE
The objective of this conference is the development of new technologies for artificial life and robotics
which have been recently born in Japan and are expected to be applied in various fields. Thisconference presents original technical papers and authoritative state-of-the-art reviews on thedevelopment of new technologies concerning robotics, networking and artificial life and, especiallycomputer-based simulation and hardware for the twenty-first century. This conference covers abroad multidisciplinary field, including areas such as:
Artificial intelligence & complexity
Artificial living
Artificial mind research
Artificial nervous systems for robots
Artificial sciences
Bipedal robot
Brain science and computing
Chaos
Cognitive science©ICAROB 2018 ALife Robotics Corp. Ltd.

-6-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Computational Molecular biology
Computer graphics
Data mining
Disasters robotics
DNA computing
Empirical research on network and MOT
Environment navigation and localization
Evolutionary computations
Facial expression analysis, music recommendation and augmented reality
Foundation of computation and its application
Fuzzy control
Genetic algorithms
Human-welfare robotics
Image processing
Insect-like aero vehicles
Intelligence in biological systems
Intelligent control
Management of technology
Medical surgical robot
Micro-machines
Multi-agent systems
Nano-biology
Nano-robotics
Networking
Neural circuits
Neuro-computer
Neuromorphic Systems
Neuroscience
Pattern recognition
Quantum computing
Reinforcement learning system & genetic programing
Robotics
Software development support method
System cybernetics
Unmanned underwater vehicles
Unmanned Aerial Systems Technologies
Unmanned Aerial Systems designing, controls and navigation
Unmanned Aero vehicles
Virtual reality
Visualization
Hardware-oriented submissions are particularly welcome. This conference will discuss new
results in the field of artificial life and robotics©ICAROB 2018 ALife Robotics Corp. Ltd.

-7-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

COPYRIGHTS
Accepted papers will be published in the proceeding of The 2018 International Conference on
Artificial Life and Robotics (ICAROB 2018) by ALife Robotics Corp. Ltd. Copyright belongs to ALife
Robotics Corp. Ltd. Some of high quality papers in the proceeding will be requested to re-submit theirpapers for the consideration of publication in an international journal ROBOTICS, NETWORKING AND
ARTIFICIAL LIFE under agreement of both Editor-in- Chief Dr. Masanori Sugisaka and 3 reviewers. All
correspondence related to the conference should be addressed to ICAROB Office.

ICAROB Office
ALife Robotics Corporation Ltd.
Hig Handadai, Oita 870-1112, JAPANTEL/FAX：+81-97-597-7760
E-MAIL：icarob@alife-robotics.co.jp
Home Page：http://alife-robotics.co.jp/©ICAROB 2018 ALife Robotics Corp. Ltd.

-8-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

MESSAGES
Masanori Sugisaka
General Chair of ICAROB

Masanori Sugisaka
General Char
(Visting Prof. of Open University(UK), President of ALife Robotics
Co., Ltd. (Japan))

It is my great honor to invite you all to The 2018 International Conferenceon Artificial Life and Robotics (ICAROB 2018).
This Conference is changed as the old symposium from the first (1996) tothe Eighteenth (2013) annually which were organized by Oita University,
Nippon Bunri University(NBU), and ALife Robotics Corporation Ltd. under
the sponsorship of the Science and Technology Policy Bureau, the Ministryof Education, Science, Sports, and Culture (Monbusho), presently, theMinistry of Education, Culture, Sports, Science, and Technology(Monkasho), Japanese Government, Japan Society for the Promotion of
Science (JSPS), The Commemorative Organization for the Japan World
Exposition (’70), Air Force Office of Scientific Research, Asian Office ofAerospace Research and Development (AFOSR/AOARD), USA. I would liketo express my sincere thanks to not only Monkasho (annually fund support
from 1996 to 2013) but also JSPS, the Commemorative Organization for theJapan World Exposition (’70), Japanese companies for their repeatedsupport.
The old symposium was organized by International Organizing Committeeof AROB and was co-operated by the Santa Fe Institute (USA), RSJ, IEEJ,ICASE (Now ICROS) (Korea), CAAI (P. R. China), ISCIE, IEICE, IEEE (Japan
Council), JARA, and SICE. The old AROB symposium was growing up by
absorbing many new knowledge and technologies into it.
This history and character was inherited also from ICAROB2014(The 2014
International Conference on Artificial Life and Robotics, included a series of
ICAROB proceedings in SCOPUS and CPCI now. From now on, ALife
Robotics Corporation Ltd. is in charge of management. This year we have
The 2018 International Conference on Artificial Life and Robotics(ICAROB2018) (23rd AROB Anniversary). The future of The ICAROB isbrilliant from a point of view of yielding new technologies to human society
in 21st century.
I hope that fruitful discussions and exchange of ideas between researchers
during Conference (ICAROB2018) will yield new merged technologies forhappiness of human beings and, hence, will facilitate the establishment of
an international joint research institute on Artificial Life and Robotics infuture.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-9-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Yingmin Jia
Co-General Chair of ICAROB

Yingmin Jia
Co-General Chair(Professor, Beihang University,
R .P. China)

It is my great pleasure to invite you to The 2018 International Conferenceon Artificial Life and Robotics (ICAROB 2018), in B-Con Plaza, Beppu, Oita,
Japan from February 1 to 4, 2018.
ICAROB develops from the AROB that was created in 1996 by Prof.
Masanori Sugisaka and will celebrate her 23rd birthday in 2018. Doubtless,new mission and big challenges in the field of artificial life and robotics will
promote ICAROB to start a new stage and attract wide interests amongscientist, researchers, and engineers around the world.
For a successful meeting, many people have contributed their great efforts
to ICAROB. Here, I would like to express my special thanks to all authors
and speakers, and the meeting organizing team for their excellent works.
Looking forward to meeting you at ICAROB in Beppu and wishing you enjoy
your stay in Japan.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-10-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Takao Ito
Co General Chair of ICAROB

Takao Ito
Co-General Chair(Professor Hiroshima
University, Japan)

It is my great honor to invite you all to The 2018 International Conferenceon Artificial Life and Robotics (ICAROB 2018). This Conference is changed
as the old symposium (ISAROB) from the first (1996) to the Eighteenth. I
am pleased to welcome you to The 2018 International Conference on
Artificial Life and Robotics in the wonderful city of Beppu, Oita city, Oita
Prefecture, Japan.
The ICAROB has its long history. The former organization of the ICAROB wasdeveloped under the strong leadership of the President, famous Professor
Masanori Sugisaka, the father of AROB. We gathered many researchers,
faculty members, graduate students from all over the world, and publishedmany high-quality proceedings and high-reputational journals every year.
Over the years, dramatic improvements have been made in the field ofartificial life and its applications. The ICAROB has becoming the unifying theexchange of scientific information on the study of man-made systems thatexhibit the behavioral characteristic of natural living systems including
software, hardware and wetware. Our conference shapes the developmentof artificial life, extending our empirical research beyond the territory
circumscribed by life-as-we-know-it and into the domain of life-as-it-could–
be. It will provide us a good place to present our new research results,
excellent ideas, and valuable information about artificial intelligence,complex systems theories, robotics, and management of technology.
The conference site is B-con Plaza, one of the most famous international
convention centers in Kyushu island, Japan. You can find many fantastic
scenic spots and splendid historical places in Beppu, Oita city. Enjoy yourstay and take your time to visit Beppu, Oita city.
I am looking forward to meeting you in Beppu, Oita city, during the ICAROB
2018 and to sharing the most pleasant, interesting and fruitful conferencewith you.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-11-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Ju-Jang Lee
Co-General Chair of ICAROB

Ju-Jang Lee
Co-General Chair(Professor, KAIST)The First International Conference on Artificial Life and Robotics (ICAROB)
was held in Oita City, Oita, Japan from Jan. 11th to 13th, 2014. This
year’s Conference will be held amidst the high expectation of theincreasingly important role of the new interdisciplinary paradigm ofscience and engineering represented by the field of artificial life androbotics that continuously attracts wide interests among scientist,
researchers, and engineers around the globe.
Distinguished researchers and technologists from around the world arelooking forward to attending and meeting at ICAROB. ICAROB is becoming
the annual excellent forum that represents a unique opportunity for theacademic and industrial communities to meet and assess the latest
developments in this fast growing artificial life and robotics field. ICAROB
enables them to address new challenges, share solutions, discussresearch directions for the future, exchange views and ideas, view theresults of applied research, present and discuss the latest developmentof new technologies and relevant applications.
In addition, ICAROB offers the opportunity of hearing the opinions ofwell-known leading experts in the field through the keynote sessions,provides the bases for regional and international collaborative research,and enables to foresee the future evolution of new scientific paradigms
and theories contributed by the field of artificial life and robotics andassociated research area. The twenty-first century will become thecentury of artificial life and intelligent machines in support of humankind
and ICAROB is contributing through wide technical topics of interest that
support this direction.
It is a great honor for me as a Co-General Chair of the 5th ICAROB 2018
to welcome everyone to this important event. Also, I would like to extend
my special thanks to all authors and speakers for contributing theirresearch works, the participants, and the organizing team of the 5th
ICAROB.
I’m looking forward to meeting you at the 5th ICAROB in Beppu, Oita
Prefecture and wishing you all the best.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-12-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GENERAL SESSION TOPICS
GS1 Robotics I (3)GS3 Complexity (5)GS5 Neuroscience (3)GS7 Intelligent Control (3)GS9 Poster (11)

GS2 Bipedal Robots & Human-Welfare Robotics (6)GS4 Pattern recobnition & image processing (4)GS6 Virtual reality (5)GS8 Robotics II (5)GS10 Others (4)ORGANIZED SESSION TOPICS
OS1 Computer Science and Information Processing (5)OS2 Intelligent Navigation (6)OS3 New Challenges to Adaptive & Learning Control(9)
OS5 Advanced Regional Engineering (4)OS7 Mobile Robotics (8)OS9 Theory and Implementation of Neuromimetic
Systems (6)OS11 Educational Application Making Control
Engineering Approach (5)OS13 Human Interface and Artificial Intelligence (5)OS4 Aspects of Natural Computing (3)OS15 System and Control (11)OS17 Automated content generation and cognitive
content generation (7)OS19 Advanced Control (5)OS21 Robot Competitions for Social Contribution (5)OS6 Kansei Engineering and Applications (4)OS8 Intelligence Control Systems and Applications (11)OS10 Intelligent Robotic Manufacturing (2)OS12 Software Development Support Method (4)OS14 Advanced Technology on Sensing Technology,
Devices, Application (6)OS16 Recognition and Control (9)OS18 Intelligent Control (6)OS20 Advances in Marine Robotics and It’s
Applications (5)OS22 Navigation and Control (3)©ICAROB 2018 ALife Robotics Corp. Ltd.

-13-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

2/1(Thu.) 17:30-19:30

Welcome Party (Hotel Shiragiku)

2/1(Thu.) - 2/4(Sun.)ICAROB Secretariat

2/4(Sun.) 15:10-15:40

Farewell Party (Conference Site: 3F, Meeting Room 32)

TIME TABLE (2/2)
2/2(Fri.)

Conference Room

Meeting Room 31

Meeting Room 32

8:40-

Meeting Room 33

OS8-1 Intelligence Control
Systems and Applications(5)
Chair: Kuo-Hsien Hsia

OS19 Advanced Control(5)
Chair: Yingmin Jia

10:15-10:30

Coffee break

10:30-11:00

Opening Ceremony (Conference Room)

11:10-12:10

Chair: Ju-jung Lee
Invited session IS-1, IS-2 (Conference Room)
Henrik Hautop Lund, Luigi Pagliarini
Lunch

12:10-13:10
13:10-14:10

Chair: Yingmin Jia
Plenary Speech PS-1 (Conference Room)
Jeffrey Johnson

14:10-14:30

Coffee break
OS21 Robot
Competitions forSocial Contribution(5)
Chair: Kazuo ISHII

OS8-2 Intelligence Control
Systems and Applications(6)
Chair: Kuo-Hsien Hsia

16:00-16:20
16:20-18:20

OS5 Advanced Regional
Engineering (4)Chair: Toru Hiraoka

OS4 Aspects of
Natural Computing (3)Chair: Marion Oswald

OS3-1 New Challenges
to Adaptive & Learning
Control (5)Chair: Shin Wakitani

OS9 Theory and
Implementation of
Neuromimetic Systems (6)Chair: Takashi Kohno

GS4 Patternrecognition & image
processing (4)Chair: Jiwu Wang

OS15 System and Control(10) ➡ (3+1)
OS16 Recognition andControl (9) ➡ (2)Chair: Fengzhi Dai

GS2 Bipedal robot &Human-walfare
Robotics (6)Chair: Ju-jung Lee

Coffee break
GS8 Robotics II (5)Chair: Hazry Desa

OS7 Mobile Robotics (8)Chair: Evgeni Magid

Meeting Room 31: Committee waiting room and Rest room
©ICAROB 2018 ALife Robotics Corp. Ltd.

Meeting Room 4

Registration (3F)

9:00-10:15

14:30-16:00

Meeting Room 1-14-

OS3-2 New Challenges
to Adaptive & Learning
Control (4)Chair:Takuya Kinosita

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

TIME TEBLE (2/3)
2/3(Sat.)Conference Room

Meeting Room 31

Meeting Room 32

8:409:00-9:45

Meeting Room 33

GS9 Poster (11)Chair Evgeni Magid

GS1 Robotics I (3)Chair: Hidehiko Yamamoto

OS22 Navigation
and Control (3)Chair: Chan Gook Park
Coffee break

10:00-11:00

Chair: Takao Ito
Plenary Speech PS-2 (Conference Room)
Masato Nakagawa

11:00-11:10

Coffee break

11:10-12:10

Chair: Jangmyung Lee
Invited session IS-4, IS-6 (Conference Room)
Pierre Parrend, Halimahtun M. Khalid
Lunch

12:10-13:10
13:10-14:10

Chair: Marion Oswald
Plenary Speech PS-3(Conference room)
Ken-ichi Tanaka

14:10-14:30

Coffee break
Poster session

OS18 Intelligent Control (5)Chair: Yingmin Jia

Poster session

18:30-20:30

OS12 Software
Development Support
Method (4)Chair: Tetsuro Katayama

OS2 Intelligent
Navigation (6)Chair: Jangmyung Lee

GS3 Complexity (5)Chair:

Banquet: Hotel Shiragiku

Meeting Room 31: Committee waiting room and Rest roomk©ICAROB 2018 ALife Robotics Corp. Ltd.

OS10 Intelligent Robotic
Manufacturing (2)Chair: Kensuke Harada

GS5 Neuroscience (3)Chair: Masao Kubo

OS14 Advanced
Technology on Sensing
Technology, Devices,
Application (6)Chair: Hiroki Tamura

OS20 Advances inMarine Robotics andIt’s Applications (5)Chair: Kazuo ISHII

Coffee break

16:00-16:20
16:20-17:20

Meeting Room 4

Registration

9:45-10:00

14:30-16:00

Meeting Room 1-15-

GS7 Intelligent Control (3)Chair: Kunikazu Kobayashi

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

TIME TABLE (2/4)
2/4(Sun.)Meeting Room 31

Meeting Room 32

Meeting Room 33

8:40-

Meeting Room 1

Registration
GS6 Virtual reality (5)Chair: Andre Rosendo

9:00-10:15

OS6 Kansei Engineering andApplications (4)Chair: Tetsuo Hattori

10:15-10:30

Coffee break

10:30-11:10

Chair: Makoto Sakamoto
Invited session IS-3 (Meeting Room 31)
Takashi Yokomori

11:10-11:30

Coffee break

11:30-12:10

Chair: Takao Ito
Invited session IS-5 (Meeting Room 31)
Yuji Shinano
Lunch

12:10-13:10
13:10-14:55

OS17 Automated content OS13 Human Interface and OS1 Computer Science andgeneration and cognitive Artificial Intelligence (5)Information Processing (5)content generation (7)Chair: Yasunari Yoshitomi
Chair: Makoto Sakamoto
Chair: Hiroki Fukushima
Farewell Party (15:10-15:40) Meeting Room 32

Meeting Room 31: Committee waiting room and Rest room
GS10 Others (4)no presentation, only papers©ICAROB 2018 ALife Robotics Corp. Ltd.

Meeting Room 4-16-

OS11 Educational Application
Making Control Engineering
Approach (5)Chair: Kazuo Kawada

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

The 2018 International Conference on
ARTIFICIAL LIFE AND ROBOTICS(ICAROB2018)

February 1 (Thursday)
17:30-19:30

Welcome Party (Hotel Shiragiku)

February 2 (Friday)
10:30-11:00

Opening Ceremony (Conference Room)
Chair: Marion Oswald (Technische Universität Wien, Austria)
Welcome Addresses
1. General Chairman of ICAROB
2. Co-General Chairman of ICAROB
3. Co-General Chairman of ICAROB
4. Co-General Chairman of ICAROB
5. Vice General Chair of ICAROB
6. Vice General Chair of ICAROB

Masanori Sugisaka (ALife Robotics Coｒporation, Ltd., Japan)
Yingmin Jia (Beihang University, P. R. China)
TaKao Ito (Hiroshima University, Japan)
Ju-Jang Lee (KAIST, Korea)
Henrik Hautop Lund (Technical University of Denmark,Denmark)Jangmyung Lee (Pusan National University, Korea)

February 3 (Saturday)
Banquet: Hotel Shiragiku
18:30-20:30
Chair: Takao Ito (Hiroshima University, Japan)
Welcome Addresses
Prof. Jeffrey Johnson (The Open University, UK)
Executive Fellow, Mr. Masato Nakagawa (DENSO CORPORATION, Hiroshima University, Japan)
Executive Fellow, Mr. Ken-ichi Tanaka (Mitsubishi Electric Corporation, Japan)
Prof. Yingmin Jia (Beihang University, P.R. China)
Prof. Henrik Hautop Lund (Technical University of Denmark, Denmark)©ICAROB 2018 ALife Robotics Corp. Ltd.

-17-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

TECHNICAL PAPER INDEX
February 2 (Friday)
8:40-Registration
Conference Room
10:30-11:00 Opening Ceremony
Chair: Marion Oswald (Technische Universität Wien, Austria)
11:10-12:10
Invited session IS-1, IS-2
Chair: Ju-Jang Lee (KAIST, Korea)
IS-1 Modular Playware and Personal Health Technology
Henrik Hautop Lund (Technical University of Denmark, Denmark)IS-2 How new technologies might lead to a paradigm shift in psychological and neuropsychologicalresearch
Luigi Pagliarini, Henrik Hautop Lund (Technical University of Denmark, Denmark)13:10-14:10
Plenary Speech PS-1
Chair: Yingmin Jia (Beihang University, P. R. China)
PS-1 Dynamic Structures for Evolving Tactics and Strategies in Team Robotics
Jeffrey Johnson and Ruggero Rossi (The Open University, UK)

Meeting Room 31
14:30-15:45 OS21 Robot Competitions for Social Contribution (5)Chair: Kazuo Ishii (Kyushu Institute of Technology, Japan)
Co-Chair: Yasunori Takemura (Nishinippon Institute of Technology, Japan)
OS21-1Analysis of Team Relationship using Self-Organizing Map for University Volleyball Players
Yasunori Takemura, Kazuya Oda, Michiyoshi Ono (Nishinippon Institute of Technology, Japan)

OS21-2

Optimization for Line of Cars Manufacturing Plant using Constrained Genetic Algorithm
Keiji Kamei, Takafumi Arai (NishiNippon Institute of Technology, Japan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-18-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS21-3

Slip model of roller driven ballKenji Kimura1, Shota Chikushi, Kazuo Ishii(1Nippon-Bunri University, Kyushu Institute of Technology, Japan)

OS21-4

Strategy Analysis of Multi-Agent Games Using Self-Organizing Map
Moeko Tominaga, Yasunori Takemura1, Kazuo Ishii(Kyushu Institute of Technology, 1NishiNippon Institute of Technology, Japan)

OS21-5

Analysis of Characteristics of Tomato Fruits in Infrared Images Toward Automatic Tomato
Harvesting System
Takuya Fujinaga, Shinsuke Yasukawa, Binghe Li, Kazuo Ishii(Kyushu Institute of Technology, Japan)

16:20-17:35 GS8 RoboticsII (5)Chair: Hazry Desa (Universiti Malaysia Perlis, Malaysia)
GS8-1

Design System of Cell Type Assembly Machine with Dual Arms Robot by GA
Keita Honda, Hidehiko Yamamoto and Takayoshi Yamada (Gifu University, Japan)

GS8-2

Spherical Mobile Robot Driven by Biorthogonal Omnidirectional Wheels
Liu Wei, Ma Shuanglong, Duan Lunqin, Yu Jiangtao (Beijing Jiaotong University, China)

GS8-3

Coordinated behaviour with a Pepper Humanoid robot to estimate the distance of other robot
using Inverse Perspective Mapping
M. Hassan Tanveer, Antonio Sgorbissa, Carmine T. Recchiuto (University of Genova, Italy)

GS8-4

Control Techniques of Quadrotor Uavs: A Concise Study
1
Syed Faiz Ahmed, 1Athat Ali, 1M. Kamran Joyo, 1Khusairy Abd Kader, 2Hazry Desa,
3
Sheroz Khan(1Universiti Kuala Lumpur, British Malaysian Institute, Malaysia,
2
Universiti Malaysia Perlis, Malaysia, 3International Islamic University Malaysia, Malaysia)

GS8-5

GIS Based Hydrological Model for River Water Level Detection & Flood Prediction featuringmorphological operations.
Sarmad Zafar, H.M.SohaibAzhar, Aqeel Tahir(Mohammad Ali Jinnah University, Pakistan)

Meeting Room 32
9:00-10:15 OS8-1 Intelligence Control Systems and Applications (5)Chair: Kuo-Hsien Hsia (Far East University, Taiwan)
Co-Chair: Chung-Wen Hung (National Yunlin University of Science & Technology, Taiwan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-19-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS8-7

Novel Detection Scheme for Stolen Password File
I-Hsien Liu, Chia-Hsiu Chen, Jung-Shian Li (National Cheng Kung University, Taiwan)

OS8-8

Honeypot System of SCADA Security Survey
Kuan-Chu Lu, I-Hsien Liu, Jung-Shian Li (National Cheng Kung University, Taiwan)

OS8-1

An Automated Optical Inspection system for a tube inner circumference state identification
Chung-Wen Hung, Jhen-Gu Jiang, Hsien-Huang P. Wu, Wei-Lung Mao(National Yunlin University of Science & Technology, Taiwan)

OS8-9

An EtherCAT Battery Test System
Chung-Wen Hung , Bo-Min Wang, Wen-Ting Hsu, Jhen-Gu Jiang(National Yunlin University of Science & Technology, Taiwan)

OS8-10Surface Defect Detection for Tube Object Based on Single Camera
Hsien-Huang Wu, Chang-Jhu He (National Yunlin University of Science & Technology, Taiwan)

14:30-16:00 OS8-2 Intelligence Control Systems and Applications (6)Chair: Kuo-Hsien Hsia (Far East University, Taiwan)
Co-Chair: Chung-Wen Hung (National Yunlin University of Science & Technology, Taiwan)
OS8-2

Implementation of the Mobile Based Robot Arm for Image Recognition
Ji-Hua Li, Jr-Hung Guo, Kuo-Lan Su (National Yunlin University of Science & Technology, Taiwan)

OS8-3

Reinforced Quantum-behaved Particle Swarm Optimization Based Neural Networks for Image
Inspection
Li-Chun Lai and Chia-Nan Ko (Nan Kai University of Technology, Taiwan)

OS8-4

Development of IoT Module with Backup and Data-security Functions
Jr-Hung Guo, Kuo-Hsien Hsia, Kuo-Lan Su(National Yunlin University of Science & Technology, Taiwan)

OS8-5

Development of Auto-Stacking Warehouse Truck
Kuo-Hsien Hsia, Ming-Guang Wu, Jun-Nong Lin, Hong-Jie Zhong, and Zh-Yao Zhuang(Far East University, Taiwan)

OS8-6

Development of Four-axis SCARA Robotic Arm Built on Automation Control System
Jr-Hung Guo, Kuang-Wei Chuang, Kuo-Lan Su(National Yunlin University of Science & Technology, Taiwan)

OS8-11

Client Searching Privacy Protection in Encrypted Database
I-Hsien Liu, Chuan-Gang Liu, Cheng-Jui Chang, Jung-Shian Li(National Cheng Kung University, Taiwan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-20-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

16:20-18:20 OS7 Mobile Robotics (8)Chair: Evgeni Magid (Kazan Federal University, Russia)
OS7-1

An Empirical Evaluation of Grid-based Path Planning Algorithms on Widely Used in Robotics
Raspberry Pi Platform
Anton Andreychuk, Andrey Bokovoy (Peoples’ Friendship University of Russia (RUDN University)
Russian Academy of Sciences, Russia)
Konstantin Yakovlev (Russian Academy of Sciences, Higher School of Economics, Russia)

OS7-2

Development of the insectoid walking robot with inertial navigation system
Vitaly Egunov, Andrey Kachalov, Michail Petrosyan, Pavel Tarasov, Elena Yankina(Volgograd State Technical University, Russia)

OS7-3

Enhancing semi-dense monocular vSLAM used for multi-rotor UAV navigation in indoor
environment by fusing IMU data
Andrey Bokovoy (Russian Academy of Sciences, Peoples’ Friendship University of Russia (RUDN
University), Russia)
Konstantin Yakovlev (Russian Academy of Sciences, Higher School of Economics, Russia)

OS7-4

Method of finding the android program motion for the ZMP trajectory of a certain type
Alexander Gorobtsov, Pavel Tarasov, Andrey Skorikov, Alexey Markov, Andrey Andreev(Volgograd State Technical University, Russia)

OS7-5

Path Planning for Indoor Partially Unknown Environment Exploration and Mapping
Aufar Zakiev¹, Roman Lavrenov¹, Vadim Indelman² and Evgeni Magid¹(¹Kazan Federal University, Russia ²Technion-Israel Institute of Technology, Israel )OS7-6

Simulation of service robot swarm behavior
Alexei Lushnikov¹, Vlada Kugurakova¹, Timur Satdarov², Arthur Nizamutdinov¹(¹Higher School of ITIS, Kazan Federal University, ²KUKA Robotics, Russia)

OS7-7

Smart Spline-Based Robot Navigation on Several Homotopies: Guaranteed Avoidance of Potential
Function Local Minima
Roman Lavrenov (Kazan Federal University, Russia)

OS7-8

Virtual Experimental Stand for Automated Fiducial Marker Comparison in Gazebo Environment
Ksenia Shabalina1, Artur Sagitov1, Hongbing Li2, Edgar A. Martinez-Garcia3, Evgeni Magid1(1Kazan Federal University, Russia)(2Shanghai Jiao Tong University, China)(3Universidad Autónoma de Ciudad Juárez, Mexico)

Meeting Room 33
9:00-10:15 OS19 Advanced Control (5)Chair: Yingmin Jia (Beihang University (BUAA), China)
Co-Chair: Fuzhong Wang (Henan Polytechnic University, P.R.China)©ICAROB 2018 ALife Robotics Corp. Ltd.

-21-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS19-1

Revisit Constrained Control of Chaos
Yunzhong Song, Ziyi Fu and Fuzhong Wang (Henan Polytechnic University, P.R.China)

OS19-2

Research on Filtering for Random Data Packet Dropouts and Delays in Wireless Sensor
Networks
Sumin HAN, Fuzhong WANG (Henan Polytechnic University, P.R.China)

OS19-3

Optimal Hohmann-Type Impulsive Ellipse-to-Ellipse Coplanar Rendezvous
Xiwen Tian, Yingmin Jia (Beihang University (BUAA), China)

OS19-4

Research on SVG Control Method Under Unbalanced Conditions
Zheng Zheng, Yousong Zhou (Henan Polytechnic University, China)

OS19-5

Dynamics Analysis of Payload On-orbit Catapult Separation Based on ADAMSYi Li, Yingmin Jia (Beihang University (BUAA), China)

14:30-15:45 OS3-1 New Challenges to Adaptive & Learning Control (5)Chair: Shin Wakitani (Hiroshima University, Japan)
OS3-1

A sound-based measurement of sway angle for anti-sway control of overhead crane
Miki Matsunaga, Masayoshi Nakamoto, and Toru Yamamoto (Hiroshima University, Japan)

OS3-2

Sampled-data PID control system with Sensitivity Function for a Second-order Plus Dead-time
SystemRyo Kurokawa, Takao Sato, and Yasuo Konishi (University of Hyogo, Japan)
Ramon Vilanova (Universitat Autonoma de Barcelona, Spain)

OS3-3

Experimental Evaluation of a Data-Driven Control System using an Electronic Thermal Regulator
Yuka Okubo, Yoichiro Ashida, Takuya Kinoshita, and Toru Yamamoto (Hiroshima University, Japan)

OS3-4

Self-repairing Adaptive PID Control for Plants with Sensor Failures
Masanori Takahashi (Tokai University, Japan)

OS3-5

Design and Development of a Constant Temperature Reservoir for a Database-Driven Smart
Cultivation System
Shin Wakitani, Sharma Sneha, and Toru Yamamoto (Hiroshima University, Japan)

16:20-17:20 OS3-2 New Challenges to Adaptive & Learning Control (4)Chair: Takuya Kinoshita (Hiroshima University, Japan)
OS3-6
OS3-7

Design of a Performance-Adaptive 1-Parameter Tuning PID Controller
Yoichiro Ashida, Shin Wakitani, and Toru Yamamoto (Hiroshima University, Japan)
Sticking Fault Detecting Method for CARIMA Model©ICAROB 2018 ALife Robotics Corp. Ltd.

-22-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Toyoaki Tanikawa, and Tomohiro Henmi(National Institute of Technology, Kagawa College, Japan)
OS3-8

Design of a Neural Network based on E-FRIT and Its Application
Kento Kinoshita, Shin Wakitani, and Shuichi Ohno (Hiroshima University, Japan)

OS3-9

Parameter Optimization with Input/Output data via DE for Adaptive Control System with Neural
Network
Taro Takagi (National Institute of Technology, Maizuru College, Japan)
Ikuro Mizumoto (Kumamoto University, Japan)

Meeting Room 1
9:00-10:00 OS5 Advanced Regional Engineering (4)Chair: Toru Hiraoka (University of Nagasaki, Japan)
Co-Chair: Minoru Kumano (University of Miyazaki, Japan)
OS5-1

Discovering Successful Determinants of Efficiency of MICHINOEKI in Chugoku Area
Minoru Kumano (University of Miyazaki, Japan)
Takao Ito (University of Hiroshima, Japan)
Toru Hiraoka (University of Nagasaki, Japan)
Hirofumi Nonaka (Nagaoka University of Technology, Japan)
Masaharu Hirota (Okayama University of Science, Japan)

OS5-2

Relationship Analysis on the Number of Customers of Michinoeki in Kyushu Region
Toru Hiraoka, Shiori Nishimura (University of Nagasaki, Japan)
Hirofumi Nonaka (Nagaoka University of Technology, Japan)
Minoru Kumano (University of Miyazaki, Japan)

OS5-3

Emotional Contribution Analysis of Online Reviews
Elisa Claire Alemán Carreón, Hirofumi Nonaka (Nagaoka University of Technology, Japan),
Toru Hiraoka (University of Nagasaki, Japan), Minoru Kumano (University of Miyazaki, Japan),
Takao Ito (Hiroshima University, Japan), Masaharu Hirota (Okayama University of Science, Japan)

OS5-4

An Approach to Visualize Place of Interest and Shooting Spot Using Geo-Tagged Photographs
Masaharu Hirota (Okayama University of Science, Japan), Masaki Endo (Polytechnic University,
Japan), Hiroshi Ishikawa (Tokyo Metropolitan University, Japan)

14:30-16:00 OS9 Theory and Implementation of Neuromimetic Systems (6)Chair: Takashi Kohno (The University of Tokyo, Japan)
Co-Chair: Timothée Levi (The University of Tokyo, Japan, University of Bordeaux, France)
OS9-1

Study of real-time biomimetic CPG on FPGA: behavior and evolution©ICAROB 2018 ALife Robotics Corp. Ltd.

-23-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Timothée Levi1,2, Kazuyuki. Aihara1, Takashi Kohno1(1The University of Tokyo, Japan, 2University of Bordeaux, France)
OS9-2

A Metaheuristic Approach for Parameter Fitting in Digital Spiking Silion Neuron Model
Takuya Nanami, Takashi Kohno (The University of Tokyo, Japan)

OS9-3

Real-time Digital Implementation of HH neural network on FPGA: cortical neuron simulation
Farad Khoyratee1, Sylvain Saïghi1, Timothée Levi1,2(1University of Bordeaux, France, 2The University of Tokyo, Japan)

OS9-4

Finding appropriate parameter voltages for driving a low-power analog silicon neuron circuit
Atsuya Tange, Takashi Kohno (The University of Tokyo, Japan)

OS9-5

A low-power silicon synapse circuit with tunable reversal potential
Ashish Gautam, Takashi Kohno (The University of Tokyo, Japan)

OS9-6

New methodology of neural network reconstruction for "in vitro" culture on Multi ElectrodeArray (MEA)
Timothée Leleu1, Timothée Levi1,2, Takashi Kohno1, Kazuyuki Aihara1(1The University of Tokyo, Japan, 2University of Bordeaux, France)

16:20-18:20 OS15 System and Control (11)Chair: Fengzhi Dai (Tianjin University of Science & Technology, China)
OS15-1

Analog circuit design of a novel 4D chaotic system
Hong Niu (Tianjin University of Science & Technology, China)

OS15-2A method of end-to-end self-understanding of Chinese paper-dictionaries
Zhijian Lv (Beijing Institute of Science and Technology Information (BISTI), P.R.China),
Yizhun Peng (Tianjin University of Science and Technology,P.R.China)

OS15-3A multi-robot rescuing system
Huailin Zhao, Zheng Wu, Xiaoxing Wang (Shanghai Institute of Technology, China)

OS15-4

Dynamic analysis and FPGA implementation of A novel hyper-chaotic system with one
equilibrium point
Shanfeng Wang, Hongyan Jia, Zhiqiang Guo (Tianjin University of Science and Technology, China)

OS15-5

Analysis and circuit implementation for a new fractional-order chaotic system
Zhiqiang Guo, Hongyan Jia, Shanfeng Wang (Tianjin University of Science and Technology, China)

OS15-6

Application of a conservative chaotic system in image encryption
Shilong Liu, Mei Zhang, Wei Xue (Tianjin University of Science and Technology, China)

OS15-7

Study of plant disease detection based on near-field acoustic holography©ICAROB 2018 ALife Robotics Corp. Ltd.

-24-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Jiangfan Wang, Xiuqing Wang(Tianjin University of Science & Technology, China)
OS15-8Simulation of cell dielectric properties based on COMSOL
Shudong Li, Xiaoyan Chen, Fengze Han(Tianjin University of Science & Technology, Chin)

OS15-9Research on the method of electrical impedance tomography based on conjugate gradient
iterative algorithm
Yuanli Yue1, Xiaoyan Chen2, Ze Liu1, Fengzhi Dai2
(1 Beijing Jiaotong University, China, 2 Tianjin University of Science and Technology, China)

OS15-10

Research on acoustic emission wireless detection system
Xiuqing Wang1, Yang Li2, Qing Liu1, Jiming Zhao1(1 Tianjin University of Science & Technology, China;2 TEXAS A and M University, U.S.A)

OS15-11

ORiN PAC (PC based Automation Controller)
Toshihiro Inukai (DENSO WAVE INC., Japan)

OS16 Recognition and Control (9)Chair: Fengzhi Dai (Tianjin University of Science & Technology, China)
OS16-1

Research on the synchronization and circuit realization of a four-wing chaotic system
Yiqiao Qin, Fengzhi Dai, Yuxing Ouyang, Qijia Kang, Ce Bian, Baochang Wei, Runhua Mao,
Shengbiao Chang (Tianjin University of Science & Technology, China)

OS16-2

Design of control system for reboiling part of distillation process
Lingran An, Fengzhi Dai, Yujie Yan, Yuxing Ouyang, Zhongyong Ye, Xia Jin, Baochang We(Tianjin University of Science and Technology, China)

OS16-3

Research on intelligent control system based on machine vision
Ce Bian1, Fengzhi Dai1, Yuxing Ouyang1, Yiqiao Qin1, Baochang Wei1,
Runhua Mao1, Meili Li2(1Tianjin University of Science and Technology, 2 China University of Petroleum - Beijing, China)

OS16-4Development of NC power based on Buck circuit
Yuxing Ouyang, Fengzhi Dai, Runhua Mao, Ce Bian, Baochang Wei, Yiqiao Qin, Shengbiao Chang,
Qijia Kang (Tianjin University of Science and Technology, China)

OS16-5

Simulation of PID temperature control system based on neural network
Yujie Yan, Fengzhi Dai, Lingran An, Yuxing Ouyang, Zhongyong Ye, Xia Jin, Ce Bian
(Tianjin University of Science and Technology, China)

OS16-6

Design and research of real-time material management system based on production process
Baochang Wei, Fengzhi Dai, Yuxing Ouyang, Haifang Man, Yiqiao Qin, Runhua Mao, Ce Bian
(Tianjin University of Science and Technology, China)

OS16-7

Research on the balance control of the inverted pendulum©ICAROB 2018 ALife Robotics Corp. Ltd.

-25-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Zhongyong Ye, Fengzhi Dai, Yuxing Ouyang, Yujie Yan, Xia Jin, Lingran An, Hongtao Zhang(Tianjin University of Science and Technology, China)
OS16-8Design of intelligent saving robot based on six-legged robot
Xinyu Zhang, Xiaokun Lin, Fengzhi Dai(Tianjin University of Science & Technology, China)

OS16-9Research on multi-object recognition algorithm based on video
Yong Hou, Runhua Mao,Yuxing Ouyang, Ce Bian, Binhu Song, Baochang Wei, Yiqiao Qin,
Shengbiao Chang, Fengzhi Dai ( Tianjin University of Science and Technology, China)

Meeting Room 4
9:00-9:45 OS4 Aspects of Natural Computing (3)Chair: Marion Oswald (Technische Universität Wien, Austria)
Co-Chair: Yasuhiro Suzuki (Nagoya University, Japan)
OS4-1

Mathematical Expression of Minakata Kumagusu’s Philosophy of
Natural Science
Yasuhiro Suzuki (Nagoya University, Japan)

OS4-2

Toward Artificial Intelligence by Using DNA Molecules
Yasuhiro Suzuki, Rie Taniguchi (Nagoya University, Japan)

OS4-3

Differentiation and Integration of Sensation and its Application
Yasuhiro Suzuki (Nagoya University, Japan)

14:30-15:30 GS4 Pattern recognition & image processing (4)Chair: Jiwu Wang (Beijing Jiaotong University, China)
GS4-1

Study on Detection of Nests on Pylon from Overhead View Based on Halcon
Jiwu Wang, Haibao Luo, Pengfei Yu (Beijing Jiaotong University, China)

GS4-2

Human gait recognition based on Caffe deep learning framework
Jiwu Wang, Feng Chen (Beijing Jiaotong University, China)

GS4-3

Unsupervised Image Classification Using Multi-Autoencoder and K-means++
Shingo Mabu, Kyoichiro Kobayashi, Masanao Obayashi, Takashi Kuremoto(Yamaguchi University, Japan)

GS4-4

Anomaly Detection of Disaster Areas from Satellite Images Using Convolutional Autoencoder
and One-class SVM
Kohki Fujita, Shingo Mabu, Takashi Kuremoto(Yamaguchi University, Japan)

16:20-17:50 GS2 Bipedal robot & Human-walfare Robotics (6)©ICAROB 2018 ALife Robotics Corp. Ltd.

-26-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Chair: Ju-Jang Lee (KAIST, Korea)
GS2-1

Design and Evaluation of Passively Powered Knee Exoskeleton (PPKE) for Squat Lifting
R.K.P.S. Ranaweera1, R.A.R.C. Gopura1, T.S.S. Jayawardena1, G.K.I. Mann2(1University of Moratuwa, Sri Lanka), (2Memorial University of Newfoundland, Canada)

GS2-2

Behavior design of robot arm imitating the consciousness mechanism of living organisms
Representation of facial expression in transition process of emotion
Ryohei Anshi, Eiji Hayashi (Kyushu Institute of Technology, Japan),
Wisanu Jitviriya (King Mongkut’s University of Technology, Thailand),
Sakmongkon Chumkamon (Rajamangala University of Technology, Thailand)

GS2-3

Study of Robot Navigation for Forest Management
Ayumu Tominaga, Eiji Hayashi (Kyushu Institute of Technology, Japan)
Abbe Mowshowits (The City College of New York, USA)

GS2-4

Development of the sense system that is combined force feedback and visual feedback-Deformable virtual objects simulation by using LEMKaito Nagano, Eiji Hayashi (Kyushu Institute of Technology, Japan)

GS2-5

A four-legged robot's soft feet structural design and walking gait generated fromInverse kinematics
Amornphun Phunopas1 and Eiji Hayashi2(1King Mongkut’s University of Technology North Bangkok, Thailand,
2
Kyushu Institute of Technology, Japan)

GS2-6

Powered Ankle Exoskeletons: Existent Designs and Control Systems
A.H. Weerasingha, W.P.K. Withanage, A.D.K.H. Pragnathilaka, R.K.P.S. Ranaweera,
R.A.R.C. Gopura(University of Moratuwa, Sri Lanka)©ICAROB 2018 ALife Robotics Corp. Ltd.

-27-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

February 3 (Saturday)
8:40-Registration
Conference Room
10:00-11:00
Plenary Speech PS-2
Chair: Takao Ito (Hiroshima University, Japan)
PS-2 EU-Way Development - Effective and Rational Development Way Masato (Max) NAKAGAWA (DENSO CORPORATION, Japan)
11:10-12:10
Invited session IS-4, IS-6
Chair: Jangmyung Lee (Pusan National University, Korea)
IS-4 Artificial Immune Ecosystems: challenges for a new generation of bio-inspired secure and resilientsystems
Pierre Parrend (ECAM Strasbourg-Europe, France)
IS-6 Trust of Virtual Agent in Multi Actor Interactions
H. M. Khalid, W. S. Liew, B.S. Voong, M.G. Helander (Damai Sciences, Malaysia)
13:10-14:10
Plenary Speech PS-3
Chair: Marion Oswald (Technische Universität Wien, Austria)
PS-3 Innovation on Manufacturing Generated by Intelligent Technologies
Ken-ichi Tanaka, Haruhisa Okuda (Mitsubishi Electric Corporation, Japan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-28-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Meeting Room 31
9:00-17:20 GS9 Poster (11)Chair: Evgeni Magid (Kazan Federal University, Russia)
GS9-1

Analysis of onboard sensor-based odometry for a quadrotor UAV in outdoor environment
Aidar Gabdullin, Grigory Shvedov, Mikhail Ivanou, Ilya Afanasyev (Innopolis University, Russia)
Konstantin Yakovlev (Russian Academy of Science, Russia)

GS9-2

Memristive neuron integration in digital robotic embodiment
Max Talanov, Evgenii Zykov, Yuriy Gerasimov, Evgeni Magid, Aleksander Elizarov( Kazan Federal University, Russia)
Victor Erokhin (Institute of Materials for Electronics and Magnetism, Italian National Council of
Research, Italy)

GS9-3

Russian mobile robot Servosila Engineer: designing an optimal integration of an extra
laser range finder for SLAM purposes
Neil Alishev, Yuriy Gerasimov, Roman Lavrenov (Kazan Federal University, Russia)

GS9-4

Toward automated open wound suturing using haptic feedback: detecting wounds andplanning the suture
Artur Sagitov¹, Hongbing Li², Evgeni Magid¹(¹Kazan Federal University, Kazan, Russia)(²Shanghai Jiao Tong University, Shanghai, China)

GS9-5

Establishing Effective Teaching for Robotics: a comparison study of Bachelor students in
Introduction to Robotics course
Tatyana Tsoy, Leysan Sabirova, Mikhail Abramskiy, Evgeni Magid(Kazan Federal University, Russia)

GS9-6

Development of a heterogeneous aerial swarm control framework for forest management
Yuriy Gerasimov, Artur Sagitov, Evgeni Magid (Kazan Federal University, Russia)

GS9-7

Implementation of ROS package for simultaneous video streaming from several different
camerasRamil Safin, Roman Lavrenov (Kazan Federal University, Russia)

GS9-8

Consumers Preferences and Purchase Intention on New Taste of Salted Duck Eggs
Shang-Hui Li (Far East University, Taiwan)

GS9-9

The Study of Green Food Image, Satisfaction and Loyalty through the Perspective of Elaboration
Likelihood Model
Shu-Fang Hsu (Far East University, Taiwan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-29-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS9-10Self-Generated Dataset for Category and Pose Estimation of Deformable Object for Manipulation
by RobotYew Cheong Hou, Khairul Salleh Mohamed Sahari (Universiti Tenaga Nasional, Malaysia)

GS9-11

Design and Development of Three Arms Transmission Line Inspection Robot
Muhammad Fairuz Abdul Jalal, Khairul Salleh Mohamed Sahari, Ho Ming Fei,
Justin Chan Tuck Leong (Universiti Tenaga Nasional, Malaysia)

Meeting Room 32
9:00-9:45 GS1 Robotics I (3)Chair: Hidehiko Yamamoto (Gifu University, Japan)
GS1-1

Production simulation of autonomous decentralized FMS including AGVs with different
personalities of mind
Ryuichi Tsujii, Hidehiko Yamamoto and Takayoshi Yamada (Gifu University, Japan)

GS1-2

Development of Automatic Recognition of Hazmat Marking Chart for Rescue Robot
Wisanu Jitviriya, Poommitol Chaicherdkiat, Noppadol Pudchuen(King Mongkut’s University of Technology North Bangkok, Thailand)
Eiji Hayashi (Kyushu Institute of Technology, Japan)

GS1-3

Development of Autonomous Robot for Laborsaving of the Forestry - Discrimination between
trees and weeds using RGB-D Shingo Yamaguchi, Eiji Hayashi, and Ayumu Tominaga (Kyushu Institute of Technology, Japan)

14:30-15:45 OS18 Intelligent Control (5)Chair: Yingmin Jia (Beihang University (BUAA), China))
Co-Chair: Weicun Zhang (University of Science and Technology Beijing, China)
OS18-1

Rolling Bearings Fault Diagnosis Method using EMD Decomposition and Probabilistic Neural
Network
Caixia Gao, Tong Wu, Ziyi Fu (Henan Polytechnic University, China)

OS18-2Detection of Dangerous Driving Behavior via Fuzzy Inference System
Shangzheng Liu1, Qinghui Zhu1, Fuzhong Wang 2(1 Nanyang Institute of Technology, 2 Henan Polytechnic University, P.R. China)

OS18-3

Feature Points Designing and Matching for the Target Spacecraft in the Final Approaching Phaseof Rendezvous and Docking
Wenjing Pei, Yingmin Jia (Beihang University (BUAA), China)

OS18-4

Construction and Visualization of Atmospheric Environment Data Map
Dongmei Fu, Gaoyuan Wang, Chao Wu, Mengchen Cui(University of Science and Technology Beijing, China)©ICAROB 2018 ALife Robotics Corp. Ltd.

-30-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS18-5Multiple-Model Adaptive Estimation with New Weighting Algorithm
Weicun Zhang (University of Science and Technology Beijing, China)

16:20-17:20 OS12 Software Development Support Method (4)Chair: Tetsuro Katayama (University of Miyazaki, Japan)
Co-Chair: Tomohiko Takagi (Kagawa University, Japan)
OS12-1

Negative Test Case Generation from an Extended Place/Transition Net-Based Mutants
Tomohiko Takagi1, Tetsuro Katayama2(1Kagawa University, 2University of Miyazaki, Japan)

OS12-2Development of a Mutant Generation Tool Using a Genetic Algorithm for Extended
Place/Transition Nets
Tomohiko Takagi, Shogo Morimoto (Kagawa University, Japan)

OS12-3

Implementation of RETUSS to Ensure Traceability between Class Diagram in UML and Java Source
Code in Real Time
Keisuke Mori*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, and Naonobu Okazaki*(*University of Miyazaki, †Tokyo University of Technology, Japan)

OS12-4

Prototype of a Tool to Detect Specific Comments
Satoshi Tanoue*, Tetsuro Katayama* Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, and Naonobu Okazaki*(*University of Miyazaki, †Tokyo University of Technology, Japan)

Meeting Room 33
9:00-9:45 OS22 Navigation and Control (3)Chair: Chan Gook Park (Seoul National University, Korea)
OS22-1Comparative Study of Sequential Processing TRN (Terrain Referenced Navigation)
Hyun Cheol Jeon and Chan Gook Park (Seoul National University, Korea)

OS22-2

Road Marking Map Matching for Road Vehicle Localization
Kyuwon Kim and Gyu-In Jee (Konkuk University, Korea)

OS22-3Control System Design of Directionally Maneuvering Multicopter with Independent Heading Rate
Byoungjin Lee, Jaehue Bae, Gwang Soo Park and Sangkyung Sung (Konkuk University, Korea)©ICAROB 2018 ALife Robotics Corp. Ltd.

-31-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

14:30-16:00 OS2 Intelligent Navigation (6)Chair: Jangmyung Lee (Pusan National University, Korea)
OS2-1

Robot Manipulator Arm Inverse Kinematics Analysis by Jacobian
Sun Oh Park, Min Gyu Jung , Jin Gon Yoon , Min Cheol Lee (Pusan National University, Korea)

OS2-2

Indoor Position recognition using the pseudo-range estimation
Jongwoo An, Hosun Kang, Jiwook Choi, Jangmyung Lee(Pusan National University, Korea)

OS2-3Distance measurement algorithm based on the object recognition
Hosun Kang, Nahyun Lee, Jangmyung Lee (Pusan National University, Korea)

OS2-4

An-ion air purifier system using IoT Technology
Heeje Kim, Dohyun Kim (Pusan University, Korea)

OS2-5

Solar Panel Temperature Control System using IoT
Min-soo Kim, Hee-je Kim (Pusan National University, Korea)

OS2-6

Enhancement of Mobile Robots Stability and Hardware Based on Reinforcement Learning
Ki-seo Kim, Jeong-hwan Moon, Jang-myung Lee (Pusan National University, Korea)

16:20-17:35 GS3 Complexity (5)Chair:
GS3-1

Management of digital records inspired by Complex Systems with RADAR
Anne Jeannin-Girardon1, Alexandre Bruyant1, Nicolas Toussaint1, Pierre Collet1 andPierre Parrend1,2(1University of Strasbourg, ; 2 ECAM Strasbourg-Europe, France)

GS3-2

Integrated optimization of differential evolution with grasshopper optimization algorithm
Duangjai Jitkongchuen, Udomlux Ampant (Dhurakij Pundit University, Thailand)

GS3-3

Efficient collective search by agents that remember failures
Masao Kubo, Nhuhai Phung, Hiroshi Sato (National Defense Academy of Japan, Japan)

GS3-4

The analysis of band structures of photonic crystals
Wang Chenxu (Muroran Institute of Technology, Japan)
Fu Ziyi (Henan Polytechnic University, China)

GS3-5

A Data Estimation Technique for Incomplete Telemetry Data Based on a Genetic Algorithm with
Data’ Statistical Properties
Masahiro Tokumitsu, Kaito Mikami (National Institute of Technology, Yonago College, Japan)
Fumio Asai (The Radio Amateur Satellite Corporation, USA)
Taku Takada (National Institute of Technology, Kochi College, Japan)
Makoto Wakabayashi (National Institute of Technology, Niihama College, Japan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-32-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Meeting room 1
9:00-9:30 OS10 Intelligent Robotic Manufacturing (2)Chair: Kensuke Harada (Osaka University, Japan)
Co-Chair: Akira Nakamura (AIST, Japan)
OS10-1

Technique of Recovery Process and Application of AI in Error Recovery Using Task Stratificationand Error Classification
Akira Nakamura*1, Kazuyuki Nagata*1, Kensuke Harada*2 and Natsuki Yamanobe*1(*1 National Institute of Advanced Industrial Science and Technology (AIST),*2 Osaka University, Japan)

OS10-2

Motion selection for 3D robotic snap assembly
Peihao Shi1, Kensuke Harada1, Weiwei Wan1, Ixchel G. Ramirez1, Juan Rojas2, Hiromu Onda3(1Osaka University, Japan, 2Guangdong University of Technology, China, 3AIST, Japan)

14:30-16:00 OS14 Advanced Technology on Sensing Technology, Devices, Application (6)Chair: Hiroki Tamura (University of Miyazaki, Japan)
Co-Chair: Koichi Tanno(University of Miyazaki, Japan)
Co-Chair: Y.W.R.Amarasinghe (University of Moratuwa, Sri Lanka)
OS14-1

A Study on the Lumbar Burden Evaluation of Work using One Smartphone
Mizuki Maiguma, Hiroki Tamura, Koichi Tanno (University of Miyazaki, Japan)

OS14-2

A Study on High Accuracy Stride Estimation on Smartphone Combining Acceleration Sensor andGyro Sensor
Shunta Nonaka, Hiroki Tamura, Koichi Tanno (University of Miyazaki, Japan)

OS14-3Development of Multi-Sensory Smart Objects Tracking Module for Mobile Robot Platforms
B. A. D. J. C. K. Basnayake, Y. W. R. Amarasinghe (University of Moratuwa, Sri Lanka)

OS14-4

Design and Development of a Conductive Polymer Based 3D - Printed Tactile Sensor with SquareType Spring Structure
W.H.P. Sampath, A.H.T.E. De Silva, Y.W.R. Amarasinghe (University of Moratuwa, Sri Lanka)

OS14-5

Design and Development of a Shape Memory Alloy Spring Actuated Gripper for Minimally
Invasive Surgeries
Roshan T.A.U, Amarasinghe Y.W.R, Dayananda N.W.N. (University of Moratuwa, Sri Lanka)

OS14-6

Design and Development of Quantum Tunneling Composite based Tactile Sensors
T.D.I. Udayanga, D.A.M.R. Fernando, H.L.P.L. Chaturanga, B.A.D.J.C.K. Basnayake
and Y.W.R. Amarasinghe (University of Moratuwa, Sri Lanka)©ICAROB 2018 ALife Robotics Corp. Ltd.

-33-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

16:20-17:05 GS7 Intelligent Control (3)Chair: Kunikazu Kobayashi (Aichi Prefectural University, Japan)
GS7-1

Fractional Order Sliding Mode Control Applying on the HIV Infection System
Thunyaseth Sethaput1, Arsit Boonyaprapasorn2(1Thammasat University, 2Chulachomklao Royal Military Academy, Thailand)

GS7-2

Skill-based Job Rotation Scheduling for Occupational Noise Exposure Control
Pavinee Rerkjirattikarn, Chatdanai Kaorapapaong, Sun Olapiriyakul(Thammasat University, Thailand)

GS7-3

A Training Method for the Speech Controlled Environmental Control System Based on Candidate
Word Discriminations
Taro Shibanoki, Masaki Watanabe (Ibaraki University, Japan), Go Nakamura, Takaki Chin
(Hyogo Rehabilitation Center, Japan), Toshio Tsuji (Hiroshima University)

Meeting room 4
9:00-9:30 GS5 Neuroscience (3)Chair:
GS5-1

Improving EEG-based BCI Neural Networks for Mobile Robot Control by Bayesian Optimization
Takuya Hayakawa, Jun Kobayashi (Kyushu Institute of Technology, Japan)

GS5-2

Selective synchronization of the coupled bifurcating neurons for phase shift of backgroundoscillation
Akihiro Yamaguchi1, Yutaka Yamaguti1, Masao Kubo2(1 Fukuoka Institute of Technology, 2 National Defense Academy of Japan, Japan)

GS5-3

Review on computational techniques in solving aircraft landing problem
Aminurafiuddin Zulkifli, Nor Azlina Ab. Aziz, Nor Hidayati Abdul Aziz
(Multimedia University, Malaysia)
Zuwairie Ibrahim(Universiti Malaysia, Malaysia)
Norrima Mokhtar(University of Malaya, Malaysia)

14:30-15:45 OS20 Advances in Marine Robotics and It’s Applications (5)Chair: Kazuo Ishii (Kyushu Institute of Technology, Japan)
Co-Chair: Keisuke Watanabe (Tokai University, Japan)
OS20-1A land testbed for experimental research on autonomous ship navigation
Keisuke Watanabe, Kazumasa Harada, Koshi Utsunomiya(Tokai University, Japan)

OS20-2

Seafloor Image Color Enhancement Method based on Retinex modeland Experiment Report in the undersea environment
Jonghyun Ahn1, Shinsuke Yasukawa2, Yuya Nishida1, Takashi Sonoda1, Keisuke Watanabe3
Kazuo Ishii1(1Kyushu Institute of Technology, 2University of Tokyo, 3Tokai University, Japan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-34-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS20-3

Automatic recognition of benthic species using image processing
Yuki Soejima, Yuya Nishida, Takashi Sonoda, Kazuo Ishii(Kyushu Institute of Technology, Japan)

OS20-4

AUV homing using acoustic chirp signal
Koji Masuda, Yuya Nishida, Takashi Sonoda, Kazuo Ishii(Kyushu Institute of Technology, Japan)

OS20-5

Simulation of horizontal vibration suppression of a suspended structure for seabed mining
Keisuke Watanabe (Tokai University, Japan)
Kazuo Ishii (Kyushu Institute of Technology, Japan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-35-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

February 4 (Sunday)
8:40-Registration
Meeting Room 31
10:30-11:10 Invited Session IS-3
Chair: Makoto Sakamoto (University of Miyazaki, Japan)
IS-3 Natural Computing Paradigm ─ A Concise Introduction
Takashi Yokomori (Waseda University, Japan)
11:30-12:10 Invited session IS-5
Chair: Takao Ito (Hiroshima University, Japan)
IS-5 Harnessing over a Million CPU Cores to Solve a Single Hard Mixed Integer Programming Problem
on a Supercomputer
Yuji Shinano (Zuse Institute Berlin, Germany)
Meeting Room 32
13:10-14:55 OS17 Automated content generation and cognitive content generation (7)Chair: Hiroki Fukushima (Kyushu Women's University, Japan)
Co-Chair: Jumpei Ono (Iwate Prefectural University, Japan)
Co-Chair: Takashi Ogata (Iwate Prefectural University, Japan)
Co-Chair: Akinori Abe (Chiba University, Japan)
OS17-1

Reinventing the Flavor Wheel
Hiroki Fukushima (Kyushu Women's University, Japan)

OS17-2Acquiring Short Scripts and Setting a Case Frame in each Acquired Script: Toward Random Story
Generation
Jumpei Ono, Takashi Ogata (Iwate Prefectural University, Japan)

OS17-3

A Method of Naimaze of Narratives Based on Kabuki Analyses and Propp’s Move Techniques for
an Automated Narrative Generation System
Takashi Ogata (Iwate Prefectural University, Japan)

OS17-4

Narratology goes to Creativity ---Cognitive Content Generation
Akinori Abe (Chiba University, Japan)

OS17-5

A Framework for Haiku Generation from a Narrative
Takuya Ito, Takashi Ogata (Iwate Prefectural University, Japan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-36-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS17-6

Cognitive content generation for healthy ageing
Yuki Hayashi (Chiba University, Japan)

OS17-7

Museum Visitors’ Behavioural Change Caused by Captions
Kotone Tadaki (Chiba University, Japan)

Meeting Room 33
9:00-10:15 GS6 Virtual reality (5)Chair:
GS6-1

Interactive musical editing system to support human errors and offer personal preferences for an
automatic piano -Inferring performance expression by considering change of pitchMasahiro Ushio, Eiji Hayashi (Kyushu Institute of Technology, Japan)

GS6-2

Analysis of Malaysian Facial Expressions for Designing Virtual Agents
Halimahtun Khalid (Damai Sciences, Malaysia), Liew Wei Shiung (University of Malaya, Malaysia)

GS6-3

Development of VR system to enhance understanding process of robot mechanisms
Alexei Lushnikov, Vlada Kugurakova, Timur Satdarov and Artur Nizamutdinov(Kazan Federal University, Russia)

GS6-4

Towards the immersive VR: measuring and assessing realism of user experience
Vlada Kugurakova, Murad Khafizov, Alexander Elizarov, Aleksei Lushnikov
and Artur Nizamutdinov (Kazan Federal University, Russia)

GS6-5

Lessons on the Reality-Gap: Iterations between Virtual and Real Robots
Andre Rosendo (ShanghaiTech University, China)
Charlie Houseago, Fumiya Iida (Cambridge University, UK)

13:10-14:25 Human Interface and Artificial Intelligence (5)Chair: Yasunari Yoshitomi (Kyoto Prefectural University, Japan)
Co-Chair: Masayoshi Tabuse (Kyoto Prefectural University, Japan)
OS13-1

A Recipe Decision Support System with Recognition Ability Recoding Function Using Knowledge
Information and Agent
Keita Saito, Taro Asada, Yasunari Yoshitomi, Ryota Kato, and Masayoshi Tabuse(Kyoto Prefectural University, Japan)

OS13-2

A System for Analyzing Facial Expression and Verbal Response of a Person while Answering
Interview Questions by AgentTaro Asada1, Daichi Kogi2, Ryouichi Shimada3, Yasunari Yoshitomi1, and Masayoshi Tabuse1(1 Kyoto Prefectural University, 2 S.Ten Nines Kyoto Co.,Ltd., 3 JFE Systems, Inc., Japan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-37-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS13-3

Facial Expression Analysis and its Visualization While Writing Messages
Yasunari Yoshitomi1, Taro Asada1, Kenta Mori2, Ryoichi Shimada3, Yuiko Yano1
and Masayoshi Tabuse1(1 Kyoto Prefectural University, 2 Neyagawa City Hall, 3 JFE Systems, Inc., Japan)

OS13-4

Recognition of Finger Spelling from Color Images Using Deep Learning
Yusuke Yamaguchi, Masayoshi Tabuse (Kyoto Prefectural University, Japan)

OS13-5

Recognition of Texting While Walking Using Convolutional Neural Networks
Junpei Miyachi, Masayoshi Tabuse (Kyoto Prefectural University, Japan)

Meeting Room 1
9:00-10:00 OS6 Kansei Engineering and Applications (4)Chair: Tetsuo Hattori (Kagawa University, Japan)
Co-Chairman: Yoshiro Imai (Kagawa University, Japan)
OS6-1

Histogram Analysis Method Based on Gaussian Distribution and Curvature Computation (I) ---Peaks and Valleys Detection ---Yusuke Kawakami*, Tetsuo Hattori**, Yoshiro Imai**, Kazuaki Ando**, Yo Horikawa**,
R. P. C. Janaka Rajapakse*** (*DynaxT Co., Ltd., Japan), (**Kagawa University, Japan)(***Tainan National University of the Arts, Taiwan)

OS6-2

Histogram Analysis Method Based on Gaussian Distribution and Curvature Computation (II) --- Experimentation ---Yusuke Kawakami*, Tetsuo Hattori**, Yoshiro Imai**, Kazuaki Ando**, Yo Horikawa**,
R. P. C. Janaka Rajapakse*** (*DynaxT Co., Ltd., Japan), (**Kagawa University, Japan)(***Tainan National University of the Arts, Taiwan)
Experimental Evaluation of Change Detection Ability in New Sequential Probability Ratio
Yoshihide Koyama*1, Tetsuo Hattori*1, Yoshiro Imai*1, Yo Horikawa*1,
Yusuke Kawakami*2, Hiromichi Kawano*3, Takeshi Tanaka*4(*1Kagawa University, *2DynaxT Co., Ltd.,*3NTT advanced technology Co., Ltd., *4Hiroshima
Institute of Technology, Japan)

OS6-3

OS6-4

Application Proposal of Sequential Probability Ratio to Dynamic System State Estimation
Tetsuo Hattori*1, Yusuke Kawakami*2, Yoshihide Koyama*1, Yoshiro Imai*1, Yo Horikawa*1,
Hiromichi Kawano*3, Takeshi Tanaka*4(*1Kagawa University, *2DynaxT Co., Ltd., *3NTT advanced technology Co., Ltd,
*4Hiroshima Institute of Technology, Japan)

13:10-14:25 OS1 Computer Science and Information Processing (5)Chair: Makoto Sakamoto (University of Miyazaki, Japan)
Co-Chair: Yu-an Zhang (Qinghai University, China)©ICAROB 2018 ALife Robotics Corp. Ltd.

-38-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS1-1

k-Neighborhood Template A-Type Two-Dimensional Bounded Cellular Acceptors
Makoto Sakamoto1, Yu-an Zhang2, Masamichi Hori1, Haruka Tsuboi1, Satoshi Ikeda1,
Kenji Aoki1, Tsutomu Ito3, Takao Ito3, Yasuo Uchida4, and Tsunehiro Yoshinaga5(1University of Miyazaki, Japan, 2Qinghai University, China, 3Hiroshima University, Japan,
4
Ube National College of Technology, Japan, 5Tokuyama College of Technology, Japan)

OS1-2

An efficient structure of organization with complete individual guidance
Mamoru Yoshimura, Tsutomu Ito1, Makoto Sakamoto, Takao Ito1, Yuji Shinano2, Satoshi Ikeda(University of Miyazaki, Japan) (1Hiroshima University, Japan) (2Zuse Institute Berlin, Germany)

OS1-3

Consideration for the Possibility to the Tourism Support Contents by the Markerless AR
Technology
Masamichi Hori1, Makoto Sakamoto1, Koshiro Mitsuhashi2, Yukari Kodama2, Takeshi Tanaka1
Mihoko Fukushima1, Chikashi Deguchi1, Masahiro Yokomichi1, Masayuki Mukunoki1,
Kunihito Yamamori1, Atsushi Iiboshi3(1University of Miyazaki, 2Miyazaki Multimedia Academy, 3Takachiho Muratabi Co., Ltd., Japan)

OS1-4

Fundamental study on tourism support using 3DCG
Haruka Tsuboi1, Makoto Sakamoto1, Masamichi Hori1, Yosuke Iriyama1, Yuki Kai1,
Hazuki Watanabe1, Yu-an Zhang2, Atsushi Iboshi3, Koshiro Mitsuhashi4, Yukari Kodama4(1University of Miyazaki, Japan), (2Qinghai University, China), (3Muratabi, Ltd, Japan)(4Miyazaki Multimedia Academy, Japan)

OS1-5

Clustering Analysis Based on Improved Fuzzy C - Means Algorithm
Qiongqiong Hu1, Yiyang li1,Yong Ge1,Yu-an Zhang 1, Qinglian Ma2,Makoto Sakamoto 2(1Qinghai University, China), (2 University of Miyazaki, Japan)

Meeting Room 4
9:00-10:15 OS11 Educational Application Making Control Engineering Approach (5)Chair: Kazuo Kawada (Hiroshima University, Japan)
Co-Chair: Yoshihiro Ohnishi (Ehime University, Japan)
OS11-1

Control Performance Assessment Method as Assessment of Programming Learning Achievement
Yoshihiro Ohnishi (Ehime University, Japan)

OS11-2

Practice of Control Education by Experiment using Robot
Shinichi Imai, Hideto Matsui and Akira Yamada (Tokyo Gakugei University, Japan)

OS11-3Development of Support Teaching Material for Nurturing Cooperativity through Playing
Kazuo Kawada (Hiroshima University, Japan)

OS11-4On Methods for Teaching in Training for Keeping Tempo Constant in Music
Hideyuki Tanaka and Keita Ueda (Hiroshima University, Japan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-39-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS11-5

Reinforcement Learning as a Theoretical Framework for Education
Masayasu Nagamatsu, Yuki Moriguchi (Hiroshima University, Japan)

Farewell Party
GS10 Others (4)GS10-1

A review on fundamental advancements of black hole algorithm
Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari(Universiti Malaysia Pahang, Malaysia)
Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz(Multimedia University, Malaysia)
Tasiransurini Ab Rahman(Universiti Tun Hussein Onn Malaysia, Malaysia)
Asrul Adam, Zulkifli Md Yusof(Universiti Malaysia Pahang, Malaysia)
Mohd Ibrahim Shapiai(Malaysia Japan International Institute of Technology, Universiti Teknologi Malaysia, Malaysia)
Norrima Mokhtar(University of Malaya, Malaysia)

GS10-2A survey on applications of black hole algorithm
Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari, Asrul Adam, Zulkifli Md Yusof(Universiti Malaysia Pahang, Malaysia)
Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz(Multimedia University, Malaysia)
Tasiransurini Ab Rahman(Universiti Tun Hussein Onn Malaysia, Malaysia)
Mohd Ibrahim Shapiai(Malaysia Japan International Institute of Technology, Universiti Teknologi Malaysia, Malaysia)
Norrima Mokhtar(University of Malaya, Malaysia)

GS10-3Black hole white hole algorithm with local search
Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari, Asrul Adam, Zulkifli Md Yusof(Universiti Malaysia Pahang, Malaysia)
Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz (Multimedia University, Malaysia)
Tasiransurini Ab Rahman (Universiti Tun Hussein Onn Malaysia, Malaysia)
Norrima Mokhtar (University of Malaya, Malaysia)

GS10-4Tele-Operation of a Legged Robot by a Virtual Marionette System- First report: The first prototype and the usefulness of the reaching taskNoritaka Sato, Yasuhiko Sawai, Ryo Asami, Makoto Kitani, Yoshifumi Morita(Nagoya Institute of Technology, Japan),
Tomofumi Fujiwara, Takahiro Endo, Fumitoshi Matsuno (Kyoto University, Japan)©ICAROB 2018 ALife Robotics Corp. Ltd.

-40-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

PS abstracts
PS-1 Dynamic Structures for Evolving Tactics and Strategies in Team Robotics
Jeffrey Johnson, Ruggero Rossi (The Open University, UK)
The autonomous robot systems of the future will be teams of robots withcomplementary specialisms. At any instant robot interactions determine relational
structures, and sequences of these structures describe the team dynamics as
trajectories through space and time. These structures can be represented inalgebraic forms that are realizable as dynamic multilevel data structures within
individual robots, as the basis of emergent team data structures. Such formalisms
are necessary for robots to learn new individual and collective behaviours. The
theory is illustrated by the example of robot soccer where robot interactions create
structures and trajectories essential to the evolution of new tactics and strategies
in a changing environment.

PS-2 EU-Way Development - Effective and Rational Development Way Masato (Max) Nakagawa (Executive Fellow, DENSO CORPORATION, Japan)
In this Plenary Speech will cover three elements describe as follows; Firstly,“Factory-IoT” technology which is DENSO company manufacturing strategy
globally. Secondly, collaboration between Germany and Japan in terms ofinnovation. Thirdly, EU-Way development which is effective and rationalmanner of development. From his working experience in Germany, United
Kingdom and the Netherlands in consecutive 14 years, EU-Way development
will be explained. One of the key points is that they make “Competition Field”
and “Non-Competition Field” in their technical domains. In the “NonCompetition Field” which is also called a “Cooperative Field”, they cooperate
together in the same industry sector to create common specification or toestablish the standardization so that they can concentrate on differentiatedtechnology as a “Competition Field” This approach is one of their driving forces
for development and innovation.

Fig. Development Style
Comparison between Japan andGermany

PS-3 Innovation on Manufacturing Generated by Intelligent Technologies
Ken-ichi Tanaka, Haruhisa Okuda (Mitsubishi Electric Corporation, Japan)
In order to cope with social problems such as labor population decrease, theautonomous cell production robot system had been developed to realize
automation in variable variety multi production. The intelligent technologies
such as random bin picking, force control, and error recovery were developed
to apply the robot from simple-transport-work to assemble-task. In addition to
the talk about such technologies, it will also be described that open innovation
method cooperating with universities to create research results which ourcompany would not achieve independently. The examples of the latest robot
technology development using artificial intelligence will also be described.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-41-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

IS abstracts
IS-1 Modular Playware and Personal Health Technology
Henrik Hautop Lund (Technical University of Denmark, Denmark)In this paper, we describe the development of personal health technology suchas wearable systems monitoring health conditions. It has been advocated thatsuch personal health technology through monitoring health status may motivate
people to perform health related actions and life-style changes. Here we
describe a methodology on how these personal health technologies may rather
be used as a tool for designing and adapting game activities, which motivate
people to perform the desired actions. Especially, we exemplify thismethodology with the use of FitBit monitoring of steps and heart rate to thedesign of appropriate, physically demanding games for the modular interactive
tiles, Moto Tiles, which are used by older adults for prevention andrehabilitation. Thereby, the motivation to perform the actions arrives from thefun play on the Moto Tiles, whereas the personal health technology is used as atool to monitor the effect and guide the game development.

IS-2 How new technologies might lead to a paradigm shift in psychological andneuropsychological research
Luigi Pagliarini, Henrik Hautop Lund (Technical University of Denmark, Denmark)
Recently, we started to apply old psychological and neuropsychologicalstandard tests and therapeutic exercises into a different domain of technology.
Therefore, while maintaining the whole conceptual and logical structure of the
traditional approach – i.e. tests mostly made by hand on paper or with the mouseand computers’ screens - intact, we moved them onto the physical interactive
and playful technology, Moto Tiles. Such a translation of input/output, likewise
involving dissimilar perceptions and different parts of our motor system (i.e.
legs, feet and balance system), might imply a change in subjects’ mental andbehavioural strategies, variations in efficacy and efficiency, as well as aredefinition of reaction times. In short, this shift, besides providing a significant
extra tool to older practices, might work as a crucial test for all of the existing
theoretical background, too. We describe our new method and report few of thefirst empirical applications.

IS-3 Natural Computing Paradigm ─ A Concise Introduction
Takashi Yokomori (Waseda University, Japan)
Natural computing (NC) is an emerging area of research that investigatescomputing techniques and models inspired by nature on one hand, and it also
investigates phenomena taking place in nature in terms of computational
methodologies on the other hand. Thus, research in NC congenitally hasinterdisciplinary flavor, which bridges between computer science and various
disciplines of natural science. Because of its interdisciplinary nature, NC
connects and covers a broad spectrum of fundamental research fields including
biology, chemistry, physics, medical science, and so forth. In this article, wegive a concise introduction to the new computing paradigm of NC. Specifically,
we give an overview of selected topics of the fields from theory to experiments,
where the stress is primarily put on theoretical achievements in computing
paradigms called molecular computing and chemical reaction computing.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-42-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

IS-4 Artificial Immune Ecosystems: challenges for a new generation of bio-inspired
secure and resilient systems
Pierre Parrend(ECAM Strasbourg-Europe, France)
The rapid evolution of IT ecosystems significantly challenges the security
models our infrastructures rely on. Beyond the old dichotomy between open
and closed systems, it is now necessary to handle securely the interaction
between heterogeneous devices building dynamic ecosystems. To this regard,
bio-inspired approaches provide a rich set of conceptual tools, but have failed
to lay the basis for robust and efficient solutions. Our research effort intends torevisit the contribution of artificial immune system research to bring immune
properties: security, resilience, distribution, memory, into IT infrastructures. We
introduce the concept of artificial immune ecosystems, which encompass acomprehensive immune protocol, libraries for detection and investigation of
anomalies, and an underlying middleware layer, for bringing immunity to IT
infrastructures, the Cloud, and IoT environment.

IS-5 Harnessing over a Million CPU Cores to Solve a Single Hard Mixed Integer
Programming Problem on a Supercomputer
Yuji Shinano (Zuse Institute Berlin, Germany)
The performance of mixed integer programming (MIP) solvers has improved
tremendously in the last two decades and these solvers have been used to sol ve
many real-word problems. ParaSCIP is the most successful parallel MIP solver
in terms of solving previously unsolvable instances from the well-known
benchmark instance set MIPLIB by using supercomputers. ParaSCIP has beendeveloped by using the Ubiquity Generator (UG) framework, which is a generalsoftware package to parallelize any state-of-the-art branch-and-bound based
solvers. ParaSCIP is a parallelized MIP solver of a single thread solver SCIP.
Since Xpress is a multi-threaded solver and ParaSCIP can run at least 80,000
processes in parallel for solving a single MIP, ParaXpress could handle over a
million CPU cores. In this talk, a ground design of the UG framework and its
latest extensions to harness over a million CPU cores will be presented andpreliminary computational results will be provided.

IS-6 Trust of Virtual Agent in Multi Actor Interactions
H. M. Khalid, W. S. Liew, B.S. Voong, M.G. Helander (Damai Sciences, Malaysia)
Trust is crucial when integrating virtual agents in human teams. Our study
investigated the combined use of subjective (general trust, psychological) andobjective (physiological) measures in predicting human trust of agents
undertaking social tasks. The subjective measures comprised twelve self-report
trust scores on ability, benevolence and integrity (ABI). The objective measures
included facial expressions, voiced speech, camera-based heart rate andgestural posture. Forty-eight subjects participated in a 2x2x2 within subject
design. They interacted with two virtual agents - a male Chinese and a female
Malay. Each experimental condition comprised three human subjects (ateleoperator and two clients) and one avatar. The interactive dialog involved abusiness set up. A neuro-fuzzy algorithm extracted rules from thepsychophysiological data to predict trust at low, medium and high trust levels.
The results revealed that trust can be predicted with an accuracy of 88%.
However, if gestures were excluded, the accuracy increased to 90%.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-43-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS abstracts
OS1 Computer Science and Information Processing (5)OS1-1 k-Neighborhood Template A-Type Two-Dimensional Bounded CellularAcceptors
Makoto Sakamoto1, Yu-an Zhang2, Masamichi Hori1, Haruka Tsuboi1, Satoshi Ikeda1,
Kenji Aoki1, Tsutomu Ito3, Takao Ito3, Yasuo Uchida4, and Tsunehiro Yoshinaga5
1( University of Miyazaki, Japan, 2 Qinghai University, China, 3Hiroshima University, Japan,
4
Ube National College of Technology, Japan, 5Tokuyama College of Technology, Japan)
In this paper, we deal with three-dimensional computational model, kneighborhood template A-type two-dimensional bounded cellular acceptor on
three-dimensional tapes, and discuss some properties. This model consists of apair of a converter and a configuration-reader. The former converts the given
three-dimensional tape to two-dimensional configuration. The latter determines
whether or not the derived two-dimensional configuration is accepted, andconcludes the acceptance or non-acceptance of given three-dimensional tape. We
mainly investigate some open problems about k-neighborhood template A-type
two-dimensional bounded cellular acceptor on three-dimensional tapes.

OS1-2 An efficient structure of organization with complete individual guidance
Mamoru Yoshimura, Tsutomu Ito1, Makoto Sakamoto, Takao Ito1, Yuji Shinano2, Satoshi Ikeda(University of Miyazaki, Japan) (1Hiroshima University, Japan) (2Zuse Institute Berlin, Germany)
This research clarifies the efficient organizational structure using mathematicalmodel. Previous studies have shown that the structure of an efficient organization
is limited under given settings when there is only one criterion to evaluate. In thisstudy, we show that an optimal organizational structure which has not been
known until now occurs under the setting different from the previous research.

OS1-3 Consideration for the Possibility to the Tourism Support Contents
by the Markerless AR Technology
Masamichi Hori, Makoto Sakamoto, Koshiro Mitsuhashi1, Yukari Kodama1, Takeshi Tanaka,
Mihoko Fukushima, Chikashi Deguchi, Masahiro Yokomichi, Masayuki Mukunoki,
Kunihito Yamamori, Atsushi Iiboshi2(University of Miyazaki,
1
Miyazaki Multimedia Academy, 2Takachiho Muratabi Co., Ltd., Japan)
Currently, the tourism industry in Miyazaki prefecture has various problems, andvarious measures are taken. On the other hand, in 2016 virtual technology
attracted much attention also on AR technology. We thought that through the AR
technology experience, we could aim for an increase in tourists. Therefore, wewill create new applications for smartphones using AR technology and aim fortourism support. First, prototype an application with markerless and features for
smartphones etc., and conduct experiments. As a result, there was improvement
and it was not completed. However, it seemed possible to become an applicationthat realizes tourism support.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-44-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS1-4 Fundamental study on tourism support using 3DCG
Haruka Tsuboi1, Makoto Sakamoto1, Masamichi Hori1, Yosuke Iriyama1, Yuki Kai1,
Hazuki Watanabe1, Yu-an Zhang2, Atsushi Iboshi3, Koshiro Mitsuhashi4, Yukari Kodama4(1University of Miyazaki, Japan 2Qinghai University, China 3Muratabi, Ltd, Japan,
4
Miyazaki Multimedia Academy, Japan)
The penetration rate of CG (computer graphics) in recent years has become
remarkable. Currently, there are various examples of utilization such as movies
and animation, various simulation and tourism projects, sales visuals and so on.
Also, as the Tokyo Olympic Games are held in 2020, the demand for tourism inJapan is getting bigger. Accordingly, we believe that efforts towards tourism
support are necessary even in local government units. Therefore, we considered
fundamental research of this moment, considering whether we can PR such as we
do not exist by using CG technology for the rich nature and sightseeing spot of
Miyazaki prefecture where we live. In this research, we focused on "Takachihocho", which is known as a land with a connection to "Japanese mythology" among
Miyazaki prefecture. As a goal, tourism support such as combining the created
3DCG model with real underwater images and aerial images, introducing richnature as the main and historic tourist attractions.

OS1-5 Clustering Analysis Based on Improved Fuzzy C - Means Algorithm
Qiongqiong Hu, Yiyang li, Yong Ge, Yu-an Zhang, Qinglian Ma1, Makoto Sakamoto1(Qinghai University, China, 1University of Miyazaki, Japan)
Cluster analysis is one of the most important technologies in the field of data
mining, and it is also a hot topic in academic research. So far, it has made great
achievements in theory and method, and plays an important role in data analysis invarious fields. The fuzzy C-means algorithm (FCM) maintains the simplicity of its
thinking, the time complexity is close to linear, and it is efficient and scalable forlarge-scale data mining. Particle Swarm Optimization (PSO) is an optimization toolwith global optimization ability, and the group intelligence is formed by theinteraction between particles in the group, and the optimal result is found by theintelligence. However, with the deepening of algorithm research, its researchprospects and application areas are increasingly bright and compelling.

OS2 Intelligent Navigation (6)OS2-1 Robot Manipulator Arm Inverse Kinematics Analysis by Jacobian
Sun Oh Park, Min Gyu Jung, Jin Gon Yoon, Min Cheol Lee (Pusan National University, Korea)
There are three methods for solving inverse kinematic that are algebraic, geometricand numerical. The advantage of algebraic and geometric method is that cananalyze exactly but these two methods are not only difficult to analysis but alsoneed to recalculate when DOF or DH-parameter are changed. That is why this paper
will present how to approach numerical method by MATLAB. Otherwise previous
two methods, the numerical method by using Jacobian can solve any DOF and DHparameter easily except singular point. Therefore, we are going to explain Jacobian
analysis and avoid singular point by using condition number. In the future, we will
adapt this result to 7DOF dual robot arm and then compare our numerical method
and real system©ICAROB 2018 ALife Robotics Corp. Ltd.

-45-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS2-2 Indoor Position recognition using the pseudo-range estimation
Jongwoo An, Hosun Kang, Jiwook Choi, Jangmyung Lee (Pusan National University, Korea)
In this paper, we propose a method of position recognition in indoor environment bypseudo range estimation of each satellite. Position recognition is a core technology
for operating autonomous intelligent robots such as Unmanned Aerial Vehicle
System(UAV), Automated Guided Vehicle(AGV). The Position recognition
technology is divided into outdoor position recognition and indoor position
recognition. The outdoor position is estimated using Global Positioning
System(GPS) and the indoor position is estimated using various sensors such as
Beacon and Inertial Measurement Unit(IMU). Due to the development of thetechnology, outdoor position recognition using GPS can be precise position
estimation, but indoor position recognition using various sensors has a cumulative
error caused by various factors and it is difficult to operate for a long time. In thispaper, to overcome this drawback, we analyze the relationship between the velocityof the moving object and pseudo-range shift each satellite, and estimate the indoor
position of the moving object by estimating the pseudo-range.

OS2-3 Distance measurement algorithm based on the object recognition
Hosun Kang, Nahyun Lee, Jangmyung Lee (Pusan National University, Korea)
This paper implements an algorithm that recognizes an object using a vision system
and measured the distance to the object. Recently, autonomous navigation systems
such as UAV and AUV are attracting attention. In order to operate these systems
safely, it is necessary to recognize obstacles on the moving path accurately andquickly. Generally, various sensors are used to recognize the surrounding
environment. These methods are affected by disturbance caused by the surrounding
environment. To overcome these drawbacks, this paper proposes an obstacle
recognition algorithm using a stereo camera. For obstacle recognition, disparity is
obtained by removing the remaining part of the object except the recognized object
from the image in order to improve the computation speed. Finally, after calculating
the distance, we confirm the algorithm by comparing with the actual distance.

OS2-4 An-ion air purifier system using IoT Technology
Heeje Kim, Dohyun Kim (Pusan National University, Korea)
We have developed the an-ion air purifier which can remove various fine dust very
efficiently. It was based on the principle of high voltage corona discharge that was
fabricated for air cleaning from various fine dust. That is a big issue for humansociety. The best switching frequency was 10kHz, and the output voltage wasapproximately 20kV. In order to remove the fine dust, the an-ion air purifier wascontrolled properly for removing the fine dust by using the Arduino and personalsmart-phone app. From these experiments, we could remove various fine dust much
more easily and effectively©ICAROB 2018 ALife Robotics Corp. Ltd.

-46-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS2-5 Solar Panel Temperature Control System using IoT
Min-soo Kim, Hee-je Kim (Pusan National University, Korea)
Solar photovoltaic systems are renewable energy sources that are used widely aroundthe world. On the other hand, the efficiency decreases as the temperature of the solar
panels increases. To prevent this phenomenon, a cooling fan can be installed on theback side of the solar panel to increase the efficiency. The solar system efficiency also
decreases due to weather conditions and unexpected situations. To overcome thisproblem, an IoT (Internet of Things) system was used to monitor the state of the solar
system and control the cooling fan. The core microprocessor used in IoT systems was
Arduino. Using Arduino, an IoT system can be implemented simply and inexpensively.
The entire system was designed and tested and the efficiency increased byapproximately 4.7%. Although it is a small 30W capacity photovoltaic system, its
efficiency is expected to be increased by applying it to a photovoltaic system of morethan 1kW in the near future.

OS2-6 Enhancement of Mobile Robots Stability and Hardware Based on
Reinforcement Learning
Ki-seo Kim, Jeong-hwan Moon, Jang-myung Lee (Pusan National University, Korea)
In this paper, we apply the obstacle avoidance run reinforcement learning algorithm
used in the mobile robot using the LiDAR sensor. Q-learning and Deep Learning wereapplied to reinforcement learning algorithms. We solve the unnatural movements inthe deceleration and acceleration of the robot using feedback learning. The results ofsimulation are applied to actual hardware to show how to increase the safety of driving.

OS3 New Challenges to Adaptive & Learning Control-1(5)OS3-1 A sound-based measurement of sway angle for anti-sway control of overhead
craneMiki Matsunaga, Masayoshi Nakamoto, Toru Yamamoto (Hiroshima University, Japan)
The cranes are well used in field such as transportation of load in construction field,
factory and dock. For anti-swing control of overhead crane, a deflection angles mustbe estimated. But it is difficult to estimate deflection angles directly. Therefore, weshow a new measurement method for the deflection angles by using two microphones
as angle sensors. The feature of our method is to achieve contact-less sensing, highflexibility of installation, low realization cost, easy to maintenance. The method
employs a time delay of arrival (TDOA) of acoustic signals which are picked up by thetwo microphones. Also, we show an algorithm to obtain the deflection angles from the
TDOA by using the Newton's method. Finally, we show experimental results of two
patterns (stationary and swaying) to demonstrate the effectiveness of the proposed
method.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-47-.
Fig.1. Overhead craneThe 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS3-2 Sampled-data PID control system with Sensitivity Function for a
Second-order Plus Dead-time System
Ryo Kurokawa, Takao Sato, and Yasuo Konishi (University of Hyogo, Japan)
Ramon Vilanova (Universitat Autonoma de Barcelona, Spain)
A sampled-data proportional-integral-derivative (PID) control system is designed for asecond-order plus dead-time system which includes an under-damping system, where
the controlled plant and the controller are continuous-time and discrete-time systems,
respectively. In order to guarantee the stability margin, the maximum value of thesensitivity function is used for designing the proposed control system. As a result, thecontrol system is optimized subject to the user-specified stability margin. Because therelationship between the stability margin with the tracking performance is trade-off, thestability margin is selected based on the modeling accuracy, Hence, high tracking
performance is selected when the modeling error is small. Furthermore, because theproposed PID control system is designed for a normalized system, the optimal PID
parameters are calculated without solving the optimization problem.

Please attach one typicalfigure in your paper
here.

OS3-3 Experimental Evaluation of a Data-Driven Control System using an
Electronic Thermal Regulator
Yuka Okubo, Yoichiro Ashida, Takuya Kinoshita, Toru Yamamoto(Hiroshima University, Japan)
Since majority of industrial processes are of nonlinearities, a data-driven PID
controller has been proposed to deal with such processes. According to that scheme,
the suitable set of PID parameters are automatically computed based on a set of I/Odata stored in the database. However, the data-driven technique is hardly implemented
in the electronic thermal regulator due to restricted memory capacity. Therefore, theplatform is required, in which the PID parameters are calculated by data-driven
technique and are sent to the electronic thermal regulator shown in Fig. 1. In this study,
the control results will be shown to demonstrate the effectiveness of the proposed
scheme by conducting the experiment.

Fig. 1. Proposed platform.

OS3-4 Self-repairing Adaptive PID Control for Plants with Sensor Failures
Masanori Takahashi (Tokai University, Japan)
In our previous works, several types of self-repairing controls have been developed
for plants with unknown sensor failures. The self-repairing control system (SRCS) can
automatically replace the failed sensor with the healthy backup to maintain its stability
if the failure occurs. However, in most existing SRCSs, roughly estimated parametersof plants are required. This paper presents a new design method for a self-repairing
adaptive PID control system. The control system has the adaptive adjustingmechanisms for the PID gains, and also can detect the sensor failure by self-test using
the integrator (I-controller). Hence, for plants with unknown parameters, SRC can be
successfully attained. Furthermore, in this paper, the control performances (stability,
failure detection and recovery) are theoretically analyzed.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-48-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS3-5 Design and Development of a Constant Temperature Reservoir
for a Database-Driven Smart Cultivation System
Shin Wakitani, Sharma Sneha, and Toru Yamamoto (Hiroshima University, Japan)
Decreasing the number of agricultural working population is serious problem.
Production of agricultural products in plant factories is as one of the solutions to theabove problems. However, it is not easy to set optimal environment conditions for manykinds of plants in food factories. This research aims for realization of a smart plantcultivation system that uses a database as a core technique. In this work, a small size
constant temperature reservoir that can control several environment conditions isdeveloped, and a simulator of the system is designed. Moreover a database-driven
nonlinear temperature control is performed.

Fig. 1 Cultivation System

OS3 New Challenges to Adaptive & Learning Control-2 (4)OS3-6 Design of a Performance-Adaptive 1-Parameter Tuning PID Controller
Yoichiro Ashida*, Shin Wakitani*, and Toru Yamamoto (Hiroshima University, Japan)
Many self-tuning PID controllers have been proposed. In these controllers, PID gains
are tuned adaptively and can maintain a good control performance to time-variant
systems. However, it is difficult to employ these methods to actual systems because of
low reliability of an on-line PID gain estimator. In addition, it is not preferred to tune
three PID gains simultaneously in terms of the safeness of a closed loop system. In thiswork, a performance adaptive controller that has twostage of PID parameters
tuning mechanisms is proposed. One of the tuners is a one-parameter tuner whichcalculates only proportional gain using the recursive least squares. The other is a PID
gain tuner using ordinary least squares. When the one-parameter tuner cannot maintaina good control performance, the PID gain tuner determine new PID gains to maintain
good control performance. The effectiveness of the proposed method is evaluated by asimulation example.

Fig. 1. Block diagram.

OS3-7 Sticking Fault Detecting Method for CARIMA Model
Toyoaki Tanikawa, Tomohiro Henmi(National Institute of Technology, Kagawa College, Japan)
To improve the safety of control systems, it is important to detect various system faults
automatically. This paper proposes a sticking fault detecting method on CARIMA
model which detect the sticking fault of both the input and the feedback value. It
consists of model estimation and a fault diagnosis. In the model estimation, the systemparameters are estimated from the input and output data using the recursive least square
method with the forgetting factor. On the other hand in the fault diagnosis, an evaluation
function derived from CARIMA model is introduced. It generates a fault signal fromthe input and output data of the detecting period, using the estimated systemparameters. Numerical simulations are performed and it is shown that this method can
detect the sticking fault of the input and the feedback value.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-49-

.

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS3-8 Design of a Neural Network based on E-FRIT and Its Application
Kento Kinoshita, Shin Wakitani, Shuichi Ohno (Hiroshima University, Japan)
PID controllers have been widely used for process systems. However, a good control
result is not always obtained with fixed PID gains when a controlled object has
nonlinearity. This paper proposes a design method for a nonlinear PID controller thatutilizes a neural network to overcome the problem. In the proposed controller, PID
gains are tuned online by a neural network and a controlled object is manipulated by
the PID controller with the tuned PID gains. The neural network is learned by an offline
learning algorithm based on the extended fictitious reference iterative tuning (E-FRIT)
and the backpropagation. E-FRIT is a method that tunes control parameters directly byusing operating data. Simulation examples are provided to show the effectiveness ofthe proposed method. Moreover, the experimental result of a level control of a tanksystem is also given to demonstrate the performance of the proposed method.

OS3-9 Parameter Optimization with Input/Output data via DE for Adaptive Control
System with Neural Network
Taro Takagi (National Institute of Technology, Maizuru College, Japan)
Ikuro Mizumoto (Kumamoto University, Japan)
During the last decades, several almost strictly positive real (ASPR) based adaptive
control systems have been proposed. ASPR based adaptive control systems have arobustness with respect to uncertainty and disturbance. However, ASPR conditions aresevere conditions for actual systems. Therefore, the introduction of parallel
feedforward compensator (PFC), which made it possible to construct ASPR based
adaptive control system, has been proposed. Also, feedforward input generated byneural network (NN), which can eliminate the tracking error caused by PFC, has been
recently proposed and ASPR based adaptive control system became more practicalmethod. Although the method became practical, it is still difficult to adopt because theparameters of adjust law for NN have to be decided by the designer and it will bedecided by trial and error. In this paper, the parameter optimization of PFC and NN will
be done by using one-shot experimental data via differential evolution.

.

OS4 Aspects of Natural Computing (3)OS4-1 Mathematical Expression of Minakata Kumagusu’s Philosophy of
Natural Science
Yasuhiro Suzuki (Nagoya University, Japan)
A mission of Natural Computing is understanding nature by algorithms; since analgorithm is just a sequence of computing steps, we have to extract an “algorithm”
from nature which requires “viewpoints” that indicate “how to observe and consider”.
Minakata Kumagusu [1867-1941], a Japanese naturalist, proposed a novel philosophyof science based on Buddhism and it affords the viewpoint. However, his philosophyof science has almost been ignored, hence we extract the principle of his philosophy
from the viewpoint of Natural Computing and transform it into a mathematical form
by using Category Theory.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-50-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS4-2 Toward Artificial Intelligence by Using DNA Molecules
Yasuhiro Suzuki, Rie Taniguchi (Nagoya University, Japan)
Molecular Robotics have been realized by using bio molecules such as DNA orproteins, for example in the Molecular Robotics Research Project [2012-17, JSPS].
Such bio molecules, like DNA or proteins, are highly structured. They already have akind of “intelligence” and adapt to environmental change. Hence, we have tried to
extract their “ability” in order to induce intelligence artificially. We have used wellknown DNA reactions, Seesaw gate reaction, and we found that DNA molecules cansense the concentration of a single strand input sequence or quasi-input (mismatch,point mutation, including sequence) and choose the one with higher concentration.
This result shows that DNA molecules (short DNA sequences) can adapt to theirenvironment.

OS4-3 Differentiation and Integration of Sensation and its Application
Yasuhiro Suzuki (Nagoya University, Japan)
Our senses can be seen as a dynamical system in a multi-dimensional sensory space,
where each dimension corresponds to a type of sense such as vision, hearing, touch,and so on. Our sensory systems are composed of differentiators and we perceive
sensory stimulations through differentiating sensory stimulations. In order to calculate
the differentiation of sense, we have to set a measurable quantity of the sense. We can
not only use quantity measures such as brightness of picture or loudness of sound, butalso quality measures like the value of Semantic Differential (SD) method formeasuring impressions or emotions, such as beauty, fear, happiness. We propose amethod of Differentiation and Integration of sensation and we show the application ofthis method by transforming a painting of Piet Mondrian’s “1921” into a music piece.

OS5 Advanced Regional Engineering (4)OS5-1 Discovering Successful Determinants of Efficiency of MICHINOEKI in
Chugoku Area
Minoru Kumano(University of Miyazaki, Japan),Takao Ito(University of Hiroshima, Japan),
Toru Hiraoka(University of Nagasaki, Japan), Hirofumi Nonaka(Nagaoka University ofTechnology, Japan), Masaharu HirotaOkayama University of Science, Japan)
Many theories and analyses of Michinoeki from the viewpoints of break function,
information dispatch function, and regional cooperation function have been publishedin the past decades. Based on our investigation, additional dimensions called disasterprevention function need to be added because a variety of activities have beendeveloped recently. Specific to the context of Michinoeki systems, this paper develops
and empirically tests a mathematical model of Michinoeki from the standpoint of how
those four basic functions significantly influence its performance, thus uniquely
contributing to extant knowledge. Using data drawn from Michinoeki in Chugoku area,
this research attempts to shed light on the relationship between four basic functions andits performance including sales revenue and number of customers who passed throughthe shop cashier, to confirm the validity of the new four function model. Based on thefindings, the managerial implications are discussed, the study limitations are identified
and directions for further research are suggested.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-51-

.

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS5-2 Relationship Analysis on the Number of Customers of Michinoeki in Kyushu
RegionToru Hiraoka, Shiori Nishimura (University of Nagasaki, Japan), Hirofumi Nonaka (Nagaoka
University of Technology, Japan), Minoru Kumano (University of Miyazaki, Japan)
We extract relevant factors or existence of facility / service to the number of customersof Michinoeki in Kyushu region, and estimate the number of customers from each factor
or existence of facility / service. We pick up facility area (whole), facility area (sales
office), number of parking units, traffic volume, number of items (sales office), overall
project cost, and municipal population. By correlating these factors and the number ofcustomers, we find factors that are related to the number of customers. And we pick up
the 44 facilities / services, and analyze the correlation between the 44 facilities / services
and the number of customers. In addition, we perform multiple regression analysis toestimate the number of customers from these factors or existence of facilities / services,and verify the performance of multiple regression equations for each factor or existenceof facility / service.

OS5-3 Emotional Contribution Analysis of Online Reviews
Elisa Claire Alemán Carreón, Hirofumi Nonaka (Nagaoka University of Technology, Japan),
Toru Hiraoka (University of Nagasaki, Japan), Minoru Kumano (University of Miyazaki,
Japan), Takao Ito (Hiroshima University, Japan), Masaharu Hirota (Okayama University of
Science, Japan)
In response to the constant increase in population and tourism worldwide, there is a needfor the development of cross-language market research tools that are more cost and timeeffective than surveys or interviews. In order to address the issue, we extracted the mostinfluential keywords in emotional judgement from Chinese online reviews. Using an
entropy based mathematical model and SVM, we determined the words that most closely
represent the demands and emotions of this customer base. Classifying these words
further as grammatical or subject words, we analyzed both the tendencies in writing
online reviews for this customer base and their specific topics of interest.

OS5-4 An Approach to Visualize Place of Interest and Shooting Spot Using
Geo-Tagged Photographs
Masaharu Hirota (Okayama University of Science, Japan), Masaki Endo (Polytechnic
University, Japan), Hiroshi Ishikawa (Tokyo Metropolitan University, Japan)
In this paper, we extract shooting spots and area of interests using geo-tagged
photographs from social media sites. Shooting spot is one of a type of hotspot in an area
where many photographs have been taken. Also, area of interest is usually attractive asa tourist spot for many people (e.g., Colosseum, Statue of Liberty). These area is usefulto analyze tourism industry. We demonstrate our approach by extracting areas of interestand shooting spot using photographs annotated with metadata from Flickr.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-52-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS6 Kansei Engineering and Applications (4)OS6-1 Histogram Analysis Method Based on Gaussian Distribution and Curvature
Computation (I) ---- Peaks and Valleys Detection ---Yusuke Kawakami*, Tetsuo Hattori**, Yoshiro Imai**, Kazuaki Ando**,Yo Horikawa**, R. P. C. Janaka Rajapakse***(*DynaxT Co., Ltd., Japan), (**Kagawa University, Japan)(***Tainan National University of the Arts, Taiwan)
This paper presents an analysis method of image histogram using curvature computation
based on Gaussian distribution. By the image histogram and curvature computing overthe cumulative histogram, we can not only detect the peaks and valleys in the image
histogram, but also obtain the accompanying information on the peaks and valleys suchas those shapes and sizes. In this paper, we propose the information extraction methodfrom image histogram using the curvature computation, based on the assumption thatthe peaks and valleys are approximated by the same number of Gauss density functions.

OS6-2 Histogram Analysis Method Based on Gaussian Distribution and Curvature
Computation (II) ---- Experimentation ---Yusuke Kawakami*, Tetsuo Hattori**, Yoshiro Imai**, Kazuaki Ando**,Yo Horikawa**, R. P. C. Janaka Rajapakse***(*DynaxT Co., Ltd., Japan), (**Kagawa University, Japan)(***Tainan National University of the Arts, Taiwan)
This paper presents an analysis method of image histogram using curvature computation
based on Gaussian distribution. By the image histogram and curvature computing overthe cumulative histogram, we can not only detect the peaks and valleys in the image
histogram, but also obtain the accompanying information on the peaks and valleys suchas those shapes and sizes. In this paper, we expose the experimental results and show theeffectiveness of the proposed method.

OS6-3 Experimental Evaluation of Change Detection Ability in New Sequential
Probability Ratio
Yoshihide Koyama*1, Tetsuo Hattori*1, Yoshiro Imai*1, Yo Horikawa*1, Yusuke Kawakami*2,
Hiromichi Kawano*3, Takeshi Tanaka*4 (*1Kagawa University, *2DynaxT Co., Ltd.,*3
NTT advanced technology Co., Ltd, *4Hiroshima Institute of Technology, Japan)
Previously, we have already proposed a novel method using New Sequential Probability
Ratio (NSPR) for the structural change detection problem of ongoing time series data
instead of using SPRT (Sequential Probability Ratio Test). And in the previous paper, wehave revealed the experimental results by applying the both methods, i.e., NSPR and
SPRT, to time series data that are generated by a multiple regression model. In this paper,
we present the evaluation results of NSPR more concretely in two cases.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-53-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS6-4 Application Proposal of Sequential Probability Ratio to Dynamic System
State Estimation
Tetsuo Hattori*1, Yusuke Kawakami*2, Yoshihide Koyama*1, Yoshiro Imai*1, Yo Horikawa*1,
Hiromichi Kawano*3, Takeshi Tanaka*4(*1Kagawa University, *2DynaxT Co., Ltd.,*3
NTT advanced technology Co., Ltd, 4Hiroshima Institute of Technology, Japan)
Previously, we have already presented a relation between the SPRT (Sequential
Probability Ratio Test) based on two hypotheses (i.e. Null hypothesis andAlternative hypothesis) and a binary channel in information theory. And we haveshown that the Bayes’ updating process for obtaining a posteriori probability by
the received signal through binary channel is very similar to the SPRT computing.
In this paper, extending the definition of SPRT to the multi hypotheses condition,
we propose to apply the extended SPRT to the internal state estimation of
dynamic system.

P( z1 | S i )Si

P( z1 | Si )P( z 2 | Si )Si

P ( z 2 | Si )

z
1

z
2

OS7 Mobile Robotics (8)OS7-1 An Empirical Evaluation of Grid-based Path Planning Algorithms on Widely
Used in Robotics Raspberry Pi Platform
Anton Andreychuk, Andrey Bokovoy (RUDN University, Russian Academy of Sciences, Russia)
Konstantin Yakovlev (Russian Academy of Sciences, Higher School of Economics, Russia)
We compare three widely used grid-based path planning algorithms, A*, Jump
Point Search, Theta*, in terms of runtime performance on a desktop personal
computer and on the Raspberry Pi 2 embedded computer widely used in modern
robotics. We also evaluate the performance of recently introduced angleconstrained path planning algorithm, LIAN, on both computers. Two principal
questions are targeted. First: to what extent modern embedded computers are
slower than conventional desktops when solving path finding problems on grids.
The answer is – one order of magnitude. Second: how well the former scale up tolarger problems. The answer is – in the same way as desktop PCs. To provide afair comparison all the algorithms are coded from scratch using the same data
structures and programming techniques.

110
100
90
80
70
60
50
40
30
20
10
0

OS7-2 Development of the insectoid walking robot with inertial navigation system
Vitaly Egunov, Andrey Kachalov, Michail Petrosyan, Pavel Tarasov, Elena Yankina(Volgograd State Technical University, Russia)
The article is devoted to the problem of managing the walking insectoid robot hexapod. The developed robot uses an inertial navigation system based on
accelerometer ADXL345 and gyroscope ITG-3200. Options for filtering data
received from the inertial sensors, the methods of combining accelerometer andgyroscope data for determining the roll and pitch are considered. To visualize theposition of the mobile robot the original software was designed. The program wasdeveloped for a microcomputer that transmits the calculated total variance (rolland pitch) over a wireless Wi-Fi network to the personal computer running
program that processes data values. The obtained data is processed and used
todisplay the object with the help of the OpenGL library.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-54-

.

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS7-3 Enhancing semi-dense monocular vSLAM used for multi-rotor UAV
navigation in indoor environment by fusing IMU data
Andrey Bokovoy (Russian Academy of Sciences, RUDN University, Russia)
Konstantin Yakovlev (Russian Academy of Sciences, Higher School of Economics, Russia)
We propose the enhancement of the modern vision-based monocularsimultaneous localization and mapping (vSLAM) method, e.g. LSD-SLAM, used
for compact multi-rotor UAV indoor navigation, by fusing inertial measurement
unit (IMU) data with camera images. We suggest removing the cost-expensive
loop-closure optimization algorithm from the vSLAM pipeline and replacing itwith the computationally efficient flow estimation procedure based purely on
IMU data. The input IMU flow is being processed by the Extended Kalman Filter
based techniques for localization purposes and used further in LSD-SLAM
algorithm for UAV pose estimation. We evaluate the proposed algorithm using
the modeled indoor environment originally used for RoboCup Rescue Simulation
League 2013 competition and “hector_quadrotor” – commonly used in modelling
simulated UAV model. Evidently, implementation of the suggested enhancement
procedure results in significant drop-down of the runtime and leads to obtaining
maps and trajectories of higher accuracy.

OS7-4 Method of finding the android program motion for the ZMP trajectory of acertain type
Alexander Gorobtsov, Pavel Tarasov, Andrey Skorikov, Alexey Markov, Andrey Andreev(Volgograd State Technical University, Russia)
The problem of finding the program motion of biped robot with a given ZMP
trajectory is considered. It is assumed that this provides a stable ZMP trajectoryof the robot motion. The equations of the robot motion consist of the equationsof the robot dynamic itself and the stability conditions equations, which include
the coordinates of the ZMP trajectory. It is proposed to seek the solution ofstability equations on a limited set of some concerted movements of the robot.
Several types of such concerted movements are considered. The application ofthis method for the android with a mass of 60 kg and height of 160 cm isdescribed. To implement the method the original multibody system dynamics
modeling software (MBS software) is used. The computational aspects of themethod and approaches to speed-up computations are considered.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-55-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS7-5 Path Planning for Indoor Partially Unknown Environment Exploration and
Mapping
Aufar Zakiev¹, Roman Lavrenov¹, Vadim Indelman², Evgeni Magid¹(¹Kazan Federal University, Russia)(²Technion-Israel Institute of Technology, Israel)This paper addresses a problem of partially unknown environment
exploration and mapping. The proposed path planning algorithm provides
global and local goals search taking into account limited sensing range andvisibility constraints that arise from obstacles. Looking for local goals near a
global path maximizes robot utility and helps avoiding returns to regions withlow potential gain. All stages were tested in ROS/Gazebo simulations andresults were compared with a naive algorithm that was proposed earlier.

OS7-6 Simulation of service robot swarm behavior
Alexei Lushnikov¹, Vlada Kugurakova¹, Timur Satdarov², Arthur Nizamutdinov¹(¹Higher School of ITIS, Kazan Federal University, Russia, ²KUKA Robotics, Russia)
In this paper, we present an implementation 3D simulation of a swarm ofintelligent service robots in Unity3D. Swarm’s task and purpose is clean-up of anassigned area. We compared and analyzed different possible algorithms of such
swarm’s behavior. Implementation includes prevention of collision between
robots themselves and humans that walk through their area of effect and can be
expanded to any other obstacle, moving or stationary. The implementation issuitable for serving as a basis for future real-life construction of a service robot
swarm.

OS7-7 Smart Spline-Based Robot Navigation on Several Homotopies: Guaranteed
Avoidance of Potential Function Local Minima
Roman Lavrenov (Kazan Federal University, Russia)
Potential function based methods provide powerful solutions in tasks of local andglobal path planning. They are characterized implementation simplicity, butsuffer from navigation function local minima. In this paper we propose amodification of our original spline-based planning algorithm. Voronoi-based
approach provides a good initial path as first iteration. A new safety criterion is
integrated into path planning to guarantee path safety. The modified algorithm
was implemented in Matlab environment and demonstrated significant
advantages over the original algorithm.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-56-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS7-8 Virtual Experimental Stand for Automated Fiducial Marker Comparison in
Gazebo Environment
Ksenia Shabalina1, Artur Sagitov1, Hongbing Li2, Edgar A. Martinez-Garcia3, Evgeni Magid1(1Kazan Federal University, Russia)
2( Shanghai Jiao Tong University, China)(3Universidad Autónoma de Ciudad Juárez, Mexico)
Fiducial marker systems are used by multiple visual applications, including
robotics, augmented reality, industrial production, good packaging anddispensing, and all these applications require certain quality assurance formarkers. In this paper, we present an experimental automated approach forfiducial marker systems comparison in virtual environment. Previously, in a setof pilot experiments, we had compared ARTag, AprilTag, CALTag marker
systems under three types of conditions: systematic occlusion, arbitrary overlap
with an object and marker rotation. In effort to statistically improve our previous
work, we faced a necessity to conduct over a thousand of additional experiments
and, as such amount of experiments is impossible to perform manually, weconstructed a virtual robot in Gazebo environment that performs all necessary
manipulations. This paper overviews our environment and automated comparison
algorithm. Further, we will introduce Gaussian noise in order to bring our virtual
experiments closer to real world conditions.

OS8 Intelligence Control Systems and Applications (11)OS8-1 An Automated Optical Inspection system for a tube inner circumference state
identification
Chung-Wen Hung, Jhen-Gu Jiang, Hsien-Huang P. Wu, Wei-Lung Mao(National Yunlin University of Science & Technology, Taiwan)
An Automated Optical Inspection, AOI, system for a tube inner circumference
state identification is proposed. In order to improve the efficiency of the
traditional chopsticks industry in the process, the AOI system is developed. In thechopsticks production line, this AOI system is installed after the material feed
equipment to complete the screening to reduce the unnecessary cost of
expenditure. Chopsticks tube states identification base on the machine vision
software –MVTec HALCON, and it will be implemented with EmguCV library.
The different algorithms of image processing are used to sort the material into
five groups: “Normal”, “Large”, “Small”, “deformation” and “empty”. The
algorithm will be detailed in this paper, and the experimental results will also be
shown in this paper to present the proposed AOI system is workable.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-57-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS8-2 Implementation of the Mobile Based Robot Arm for Image Recognition
Ji-Hua Li, Jr-Hung Guo, Kuo-Lan Su(National Yunlin University of Science & Technology, Taiwan)
The paper develops a mobile based robot arm using KNRm system. The mobile
based robot arm contains two parts： One is mobile platform: The other is a fourjoint robot arm. The core of the KNRm system uses the NI Single-Board RIO
9606 module that is manufactured by National Instruments (NI) Company. The
structure of the mobile based robot arm is built by Matrix element. The mobile
based robot arm integrates some sensors, three DC servomotor motors, four RC
servomotors, a controller, and an image processing module. Trapezoidal
acceleration and deceleration algorithm and Proportional-Integral-Derivative(PID) algorithm are used to control each DC servomotor. The mobile platform
embeds a robot arm with four degrees of freedom on the front side. The driverdevice of the robot arm is RC servomotor. In the experimental results, the mobile
based robot arm can search and recognize the assigned billiard ball using image
binarization method and Otsu algorithm by the image system. Finally, the robot
arm moves approach to the assigned color ball, and catches the ball moving to theassigned position, and puts down the ball.

OS8-3 Reinforced Quantum-behaved Particle Swarm Optimization Based Neural
Networks for Image Inspection
Li-Chun Lai, Chia-Nan Ko (Nan Kai University of Technology, Taiwan)
The paper combines the niche particle concept, quantum-behaved particle swarm
optimization (QPSO) method with chaotic mutation to train neural networks forimage inspection. When constructing the reinforced quantum-behaved particle
swarm (RQPSO) to train neural networks (RQPSONNs) for image inspection,
first, image clustering is adopted to capture feasible information. Then the
database of image can be built. In this research, the use of support vector
regression (SVR) method determines the initial architecture of the neuralnetworks. After initialization, the neural network architecture can be optimized
by RQPSO. Then the optimal neural networks can perform image inspection.
In this paper, the program of RQPSONNs for image inspection will be built. The
values of root mean square error (RMSE) and peak signal to noise ratio (PSNR)
are calculated to evaluate the efficiency of the RQPSONNs. Moreover, theexperiment results will verify the usability of the proposed RQPSONNs forinspecting image. This research can be used in industrial automation to improve
product quality and production efficiency.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-58-

Video Capture Card

Control PlatformC

CCD

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS8-4 Development of IoT Module with Backup and Data-security Functions
Jr-Hung Guo, Kuo-Hsien Hsia, Kuo-Lan Su(National Yunlin University of Science & Technology, Taiwan)Internet of Things (IoT) is one of the most popular research topics. There havebeen many studies and products about this topic. However, there is few researches
about the correctness of data and the mutual support of modules. In this paper,
we developed an IoT module with multi-sensor and communication interface. It
debugs and confirms the data from multiple modules and sensors using the fusion
and the redundant algorithms. It can also sustain or replace a possibly fail IoT
module via multiple communication interface. The dynamic identification
technology is used to ensure the security of data transmission. Hence the IoT
system can be more stable and more secure. In this paper, the program of
RQPSONNs for image inspection will be built. The values of root mean squareerror (RMSE) and peak signal to noise ratio (PSNR) are calculated to evaluate
the efficiency of the RQPSONNs. Moreover, the experiment results will verify
the usability of the proposed RQPSONNs for inspecting image. This research canbe used in industrial automation to improve product quality and production
efficiency.

OS8-5 Development of Auto-Stacking Warehouse Truck
Kuo-Hsien Hsia, Ming-Guang Wu, Jun-Nong Lin, Hong-Jie Zhong, and Zh-Yao Zhuang(Far East University, Taiwan)
Warehouse automation is a very important issue for the promotion of traditional
industries. For the production of larger and stackable products, it is usuallynecessary to operate the stacker for the stacking and storage of the products. The
general autonomous warehouse-truck does not have the ability of stacking
objects. In this paper, we develop a prototype of auto-stacking warehouse-truckthat can work without direct operation by a skill person. Just commanded with an
RFID card, the stacker truck can move under the prior-planned route, and pickand deliver the product from the designated storage area or deliver and put the
product to the designated storage area in the warehouse. It can significantly
reduce the manpower requirements of the skilled-person of fork lift technician
and improve the safety of the warehousing area.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-59-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS8-6 Development of Four-axis SCARA Robotic Arm Built on Automation Control
SystemJr-Hung Guo, Kuang-Wei Chuang, Kuo-Lan Su(National Yunlin University of Science & Technology, Taiwan)
The paper presents a integral design method to integrate “DFM” (Design for
Manufacturing) and “DFV” (Design for Verification), and completes the
production system from simulation to actual verification using the SCARA robot
arm. In the aspect of kinematics of horizontal joint, the 2D model is establishedby mathematical derivation and the analysis of the robot arm characteristics using“Jacobian” kinematic, and integrates into the verification design. The
electromechanical integration of the proposed system is combined SCARA’scharacteristics, automatic control components, sensors; electrical, pneumatic
circuit design and software programming for implement. Then user can lead therobotic arm to run the coordinates and movement path with proper accuracy,efficiency and reliability. The program language of the SCARA uses “DRL”(DELTA Robot Language) that is developed by DELTA Company. This researchalso proposes a design method using automatic palletizing simulate layout for the
SCARA robot arm, and uses the framework of “IoT” (Internet of Things) to build
the near-end and remove applications via LAN and WLAN (WiFi).

OS8-7 Novel Detection Scheme for Stolen Password File
I-Hsien Liu, Chia-Hsiu Chen, Jung-Shian Li (National Cheng Kung University, Taiwan)
Over the past decades, with the popularity of personal computers and mobile
devices, the convenience of the Internet, and the development of social networks,
more and more people rely on the Internet to deal with or share things on life,
such as online payment, online shopping, etc. All of the above are need to use theuser authentication to achieve identity confirmation in order to use theappropriate application. The evolution of the password so far has developed avariety of forms to achieve certification. For example, fingerprints, sound waves,
retina and so on. However, the traditional digital password authentication
credential is still widely accepted by the public, but in the past few decades, thecontent of this certification mechanism hasn’t been much change. The use of
traditional password is the most common and important authentication credential
for today's online society. However, with the fast development of network
technology and easily memorized and guessed password with low strength,
password cracking events frequently occur. Consequently, the password files areleaked. The events are not easily to be detected and it results in great businesslosses. How to avoid password cracking and detect the event of stolen password
file has become an important information security issues. Recently, Juels andRivest published a paper about Honeyword system using one account withmultiple passwords for detection of illegal intrusions. Our research explores therelevant password cracking technology and password-related policy.
Accordingly, our research improves the Honeyword system proposing a new
password storage mapping method. If the password file is stolen, an attacker
could not know any user’s correct password. This method can reduce additional
space of storing passwords with ability to detect security event for stolen
password files. We apply our method to the latest version of OpenLDAP Server
to prove its feasibility.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-60-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS8-8 Honeypot System of SCADA Security Survey
Kuan-Chu Lu, I-Hsien Liu, Jung-Shian Li (National Cheng Kung University, Taiwan)
Now the essential infrastructure is used SCADA system monitoring, like nuclear
plants, Water Conservancy Bureau, manufacturing, and logistics industry. With
the rapid development of the internet, now the IOT (internet of things) turn every
massive infrastructure from the closed network into the open way made the wholesystem became more convenient. by contrast, the system will face the threat neverbefore, that is why the whole system needs to build a safe and protectedmechanism through remote control for prevented the external attack, to avoid
SCADA system crashed lead to international security and economy issue. Forprotecting the important assets, we need to understand the tools can preserve
system safety to prevent the attackers invade the SCADA system. This study
investigating the defense mechanism of the SCADA system and proposed aneffective defensive structure to protected the safety of SCADA system.

OS8-9 An EtherCAT Battery Test System
Chung-Wen Hung , Bo-Min Wang, Wen-Ting Hsu, Jhen-Gu Jiang(National Yunlin University of Science & Technology, Taiwan)
A battery test system is a key equipment to ensure lifetime of batteries andportable device performance. On the other hand, because of the real timerequirement in industry 4.0, EtherCAT communication is considered a standard
interface of industry 4.0. It is based on the Ethernet hardware, and its advantages
are larger transmission bandwidth, shorter update time, and on minimum
synchronization time jitter. In order to support EtherCAT communication
function for battery test systems, this paper purposed an EtherCAT interface foran original battery test system. The TwinCAT environment developed by
Beckhoff is adopted to be the EtherCAT interface Master, and Renesas’s EC-1
evaluation board is used as the EtherCAT interface slave controller. The
commands are sent to the battery test system through the RS485 interface andModbus protocol from the EC-1 board. Finally, a C# software is proposed toprovide friendly GUI interface for user.

OS8-10 Surface Defect Detection for Tube Object Based on Single Camera
Hsien-Huang Wu, Chang-Jhu He (National Yunlin University of Science & Technology, Taiwan)
Extrusion tube is a very popular product in the plastic industry produced by a
extruder machine, for example, tubes used in washer machine or in reverse
osmosis (RO) water purification system. To guarantee the quality of the tube
produced for medical use, a surface defect detection system is needed. In thispaper, we use a single camera combined with a telecentric lens and properly
designed mirror set to achieve the complete coverage of the object surface for full
defect detection. Based on the properly designed image acquisition equipment
and image analysis techniques, the final system for extrusion tube defectdetection can successfully detect defects of black spots, bumps, injury, etc., witha defect detection rate of about 95%, and the detection speed is up to 1578 mm/s,
which can be deployed for the online usage.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-61-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS8-11 Client Searching Privacy Protection in Encrypted Database

Hsien Liu, Chuan-Gang Liu, Cheng-Jui Chang, Jung-Shian Li(National Cheng Kung University, Taiwan)

In recent years, Cloud Storage Providers (CSP) works hard to successfully
achieve cheap cloud storage. To prevent losing important data in their own device,some of the people backup their data to the cloud. It becomes much more
important for securing the data stored in the cloud. Although CSP claims thatstorage service is much more secure than before, malicious attackers invade theservers continuously. To prevent data in cloud stolen by the hackers, searchable
encryption can be an useful tool. The key point of searchable encryption can relax
the assumption that CSP is always truthful. Data owner can authorize users how
to search and extract his files accordingly. Our proposed scheme, called SESD
(Searchable Encryption Support Dynamic), provides multi-keyword based
searchable encryption method for the semi-trusted cloud simulations, weobserved the behaviors of the user searching in the encrypted cloud server.
Secondly, the scheme employs a more dynamic way on cloud storage. By
verification in the cloud data service, we found the proposed scheme effective insearchable encryption.

OS9 Theory and Implementation of Neuromimetic Systems
OS9-1 Study of real-time biomimetic CPG on FPGA: behavior and evolution
Timothée Levi1,2, Kazuyuki. Aihara1, Takashi Kohno1(1The University of Tokyo, Japan, 2University of Bordeaux, France)
Locomotion is one of the most basic abilities in animals. Neurobiologists haveestablished that locomotion results from the activity of half-center oscillators thatprovides alternation of bursts. Central Pattern Generators (CPGs) are neuralnetworks capable of producing rhythmic patterned outputs without rhythmic
sensory or central input. We propose a network of several biomimetic CPGs using
biomimetic neuron model and synaptic plasticity. This network is implemented
on a FPGA (Field Programmable Gate Array). The network implementationarchitecture operates on a single computation core and in real-time. The
implementation of this CPGs network is validated by comparing it with biological
data of leech heartbeat neural network. From these CPGs, we study the evolution
and the behavior of chains of CPGs to understand how CPG evolves over timeand how single broken CPG can be fixed.

OS9-2 A Metaheuristic Approach for Parameter Fitting in Digital Spiking Silion
Neuron Model
Takuya Nanami, Takashi Kohno (The University of Tokyo, Japan)
DSSN model is a qualitative neuronal model designed for efficient
implementation in digital arithmetic circuit. In our previous studies, we developed
automatic parameter fitting method using the differential evolution algorithm forregular and fast spiking neuron classes. In this work, we extended the method tocover low-threshold spiking and intrinsically bursting. Firstly, we optimized
parameters of the fast subsystem of the DSSN model in order to reproduce thatof the ionic-conductance model. Secondly, we optimized remaining slow
parameters of the DSSN model that control bursting dynamics in order toreproduce that of the ionic-conductance model.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-62-

.

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS9-3 Real-time Digital Implementation of HH neural network on FPGA: corticalneuron simulation
Farad Khoyratee1, Sylvain Saïghi1, Timothée Levi1,2( University of Bordeaux, France, 2The University of Tokyo, Japan)
1Millions of people worldwide are affected by neurological disorders. The longterm goal of replacing damaged brain areas with artificial devices also requires
the development of neuronal network models. The hardware set-up that will be
used to interface the biological component is a Spiking Neural Network (SNN)
system. This SNN implements biologically realistic neural network models,spanning from the electrophysiological properties of one single neuron up to
network plasticity rules. The most biologically relevant mathematical neuronmodel was proposed in 1952 by Hodgkin and Huxley (HH). Here, we propose areal-time digital implementation of hundreds biomimetic HH neurons designed
by FPGA as a central processing unit and DAC (Digital to Analog Converter) as
the spike generator. To validate our HH neuron, we simulate different neuron
family of cortical neurons. This research will be used for neurological diseasemodeling and bio-hybrid experiments.

OS9-4 Finding appropriate parameter voltages for driving a low-power analog
silicon neuron circuit
Atsuya Tange, Takashi Kohno (The University of Tokyo, Japan)
Silicon neurons are VLSI circuits thatwhich mimic the dynamical behaviors ofreal neurons with electronic circuits. They are expected to used as elements inof
large-scale networks of artificial biomimetic neuron circuits or bio-silico hybrid
systems. We focused on a silicon neuron circuits designed by with a qualitative
neuronal modeling approach for low power consumption. But tThis circuit can
mimic a variety of neuronal cell classes by selecting appropriate values for its
bias voltages. Its’s characteristics are influenced by some factors like temperatureor mismatch and secondary effects of transistors. These factors cause error
between numerical simulation results in the designing stage and thecharacteristics of implemented circuits. Therefore we have to tune parameterthebias voltages of individual neuron circuits to get desired dynamical behaviors
after circuit implementation. We constructed the algorithm to automatically findappropriate values ofr the parameterbias voltages.

OS9-5 A low-power silicon synapse circuit with tunable reversal potential
Ashish Gautam, Takashi Kohno (The University of Tokyo, Japan)
Synapses are the basic building blocks of signal processing and computation inboth real and artificialsimulated neuronal networks. We present the concept andsimulation results of a pseudo five bit, low power silicon synapse circuit thatemulates realistic postsynaptic current profiles by fitting them using the
exponential current-voltage relationship of MOSFET in the sub-threshold regimeof operation. Adjusting the parameters, our circuit is capable of generating both
excitatory (AMPA/NMDA type) and inhibitory (GABA type) postsynaptic
currents covering a wide range of time constants. Also emulating the real synapse,
the post-synaptic current is proportional to the difference between thepostsynaptic membrane potential and a tunable synaptic reversal potential.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-63-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS9-6 New methodology of neural network reconstruction for "in vitro" culture
on Multi Electrode Array (MEA)
Timothée Leleu1, Timothée Levi1,2, Takashi Kohno1, Kazuyuki Aihara1(1The University of Tokyo, Japan, 2University of Bordeaux, France)
The realization of neural prostheses, that could improve the quality of life formillions of people around the world affected by cognitive and/or motor disorders,requires understanding the details of the micro-connectivity between neurons in
the biological tissue. Techniques for infering the network structure are generally
based on cross-correlations and result in ambiguous reconstruction. Recently, we
have proposed a method for which there is one-to-one correspondence between
statistical properties of packets of spikes (or avalanches) and the network
structure. This method utilizes the higher order statistics of spike trains in order toreconstruct the network structure unambiguously. We show using numericalsimulations of various biological neuron models that the proposed method is
general, and is particularly well-fitted for the analysis of neural activity recorded
from cultured neuronal networks coupled to microelectrode arrays.

OS10 Intelligent Robotic Manufacturing (2)OS10-1Technique of Recovery Process and Application of AI in Error Recovery
Using Task Stratification and Error Classification
Akira Nakamura*1, Kazuyuki N agata*1, Kensuke Harada*2 and Natsuki Yamanobe*1(*1 National Institute of Advanced Industrial Science and Technology (AIST), Japan*2
Osaka University, Japan

task2

Class 4

error
error

Class 3

task1

Class 1

Process flow with error recovery

Class 2In manipulation tasks of industrial production, plant maintenance, andhousework, error recovery is an important research theme for robots. However,systematical methods of error recovery have not been appeared yet. We haveproposed error recovery using the concepts of both task stratification and error
classification. In the error recovery, the judgment of the error is performed inprocesses of the practice of the system. In this paper, the recovery process after
the judgment of error is described in detail. In particular, we will explain how tochange the parameters of planning, modeling and sensing when error recovery is
performed. This contributes to deal with error recovery systematically. On theother hand, to experience a lot of error recoveries can accumulate the data about
the recovery method. Furthermore, in this paper, we also refer to apply techniqueof artificial intelligence such as Deep Learning to error recovery.

Changeof several
parameters

Classification of error

taskk

error

Class 1 → Execution error
Class 2 → Planning error
Class 3 → Modeling error
Class 4 → Sensing error

OS10-2 Motion selection for 3D robotic snap assembly
Peihao Shi, Kensuke Harada, Weiwei Wan, Ixchel G. Ramirez(Osaka University, Japan)
Juan Rojas(Guangdong University of Technology, China), Hiromu Onda(AIST, Japan)
In this paper, we aim to provide an assembly method for a snap joints assemblytask. We create a 3D cellphone model and use ADAMS simulation environment
to analyze the relative motion between screen and backer part. We focus on two
points in this research. 1) Two kinds of relative motion between the screen andbacker parts, i.e., the rotation-based and the translation-based methods, arecompared, and 2) difference between assembly and disassembly is analyzed. By
using the maximum elastic energy in an assembly process, we show that 1) the
rotation-based assembly motion has better robustness than the translation-based
assembly motion in cellphone assembly tasks when we set the same initial
position error, and 2) The rotation-based assembly method is more effective forsnap joint disassembly.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-64-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS11 Educational Application Making Control Engineering Approach (5)OS11-1 Control Performance Assessment Method as Assessment of Programming
Learning Achievement
Yoshihiro Ohnishi (Ehime University, Japan)
It is an important problem how to estimate the learning achievement of the
programming learning. However, the quantitative evaluation method according
to the learning achievement is difficult. On the other hand, "work" of a control
program influences the control performance of the plant by petrochemicalindustry, and to have an influence on the operation cost, Improvement of aprogram and am planning for productivity improvement are suggested byevaluating the control performance quantitatively. This research considers theusing control performance assessment method for the achievement value of the
programming learning.

The example of thegood control result
and the bad control
result.

OS11-2 Practice of Control Education by Experiment using Robot

Shinichi Imai, Hideto Matsui and Akira Yamada (Tokyo Gakugei University, Japan)

For students who attend lectures in control engineering, there are not many
students w1-1e many formulas. For that reason, various studies have been
conducted on control education. Therefore, in this research, we propose acontrol education method using a robot so that interests and interests are given
to students, practice lessons and verify its effectiveness.

OS11-3 Development of Support Teaching Material for Nurturing Cooperativity
through Playing
Kazuo Kawada (Hiroshima University, Japan)

I have been conducted educational activities utilizing robots corresponding toeach stage from kindergartens to junior high schools. In this study, I propose
and verify an educational material that supports collaborative operating by
experiences related to people and things. Specifically, three children have a
joystick for operating the robot. The robot does not move unless you
manipulate the joystick at the same time. I developed the supporting teachingmaterial that nurture cooperativeness through play.

OS11-4 On Methods for Teaching in Training for Keeping Tempo Constant in Music
Hideyuki Tanaka and Keita Ueda (Hiroshima University, Japan)
Ability of keeping rhythm is one of important factor in playing music. In thispaper, the authors carry out experiments of training for keeping tempo
constant in music. Experiments are carried out for experts and non-experts ofmusic, and two methods for giving instructions are compared: The first
method gives feedbacks by graphs and the other one by language. In order toensure objectivity of the experiments, measurement data are collected byProcessing, and changes of tempo etc. are analyzed by defining evaluate
functions, where Scilab is used for computing the values of the evaluate
functions. The experiments reveal that there are differences between experts
and non-experts in giving instructions.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-65-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS11-5 Reinforcement Learning as a Theoretical Framework for Education
Masayasu Nagamatsu, Yuki Moriguchi (Hiroshima University, Japan)
Reinforcement Learning as a computational model of human learning, hasrecently
becomethesubjectofintensive
investigation(Barto,A.G.,2005;Vigortio,C.M., 2010). Recent findings include: neuralsignature of hierarchical reinforcement learning, and multiple roles of reward
signals in the human brain. These findings not only provide further theoreticalevidences that complementary for traditional statistical inference in educational
research, but also provide framework for model based design of educational
processes. Types of evidence for education, and its relation to recent findings ofhuman reinforcement learnings are discussed.

OS12 Software Development Support Method (4)OS12-1 Negative Test Case Generation from an Extended Place/Transition
Net-Based Mutants
Tomohiko Takagi1, Tetsuro Katayama2(1Kagawa University, 2University of Miyazaki, Japan)
EPN (Extended Place/transition Net)-based mutants are formal behavioral
models of software that contain intended failures. In negative testing, wellselected EPN-based mutants are used to generate negative test cases to confirm
that software does not include serious or possible failures. However, the large
state space and feasibility problem on EPNs cause the difficulty of generating thenegative test cases from EPN-based mutants. The aim of our research is toconstruct a technique in which effective negative test cases for detection of
intended failures are generated from EPN-based mutants. In this technique using
ant colony optimization, ants heuristically search better paths (i.e., negative test
cases) to find foods (i.e., intended failures) on a field (i.e., EPN-based mutant).

t1 (nat p) [] / v:=v+p;p1

t1 (nat p) [] / v:=v+p;t4 () [] /...

p2

p1

p3

failure appeared[0, 1, 0], v=0

t1 (1)[1, 0, 0], v=1

t2 ( )t2 ( )[0, 1, 0], v=3[0, 1, 0], v=1

t1 (1)

t1 (2)

t1 (2)

t1 (0)...

(2) Construction and selection of EPN-based mutants(1) Construction of an EPN

t1 (3)

p3

changedt2 () [] /... t3 (nat p) [v>=p] / v:=v-p;t2 () [] /... t3 (nat p) [v>=p] / v:=v-p;[1, 0, 0], v=3

t4 () [] /...

p2

t1 (2)

t3 (0)[0, 1, 1], v=0t1 (nat p) [] / v:=v+p;p1

t4 () [] /...

p2

p3

t2 () [] /... t3 (nat p) [v>=p] / v:=v-p;(3) Generation of negative test cases from selected EPN-based mutants

OS12-2 Development of a Mutant Generation Tool Using a Genetic Algorithm for
Extended Place/Transition Nets
Tomohiko Takagi, Shogo Morimoto (Kagawa University, Japan)
An EPN (Extended Place/transition Net) is used as a formal model that represents
the behavior of software. When mutation testing is performed based on the EPN,
failures are intentionally inserted into an original EPN (EPN that represents the
expected behavior of software) in order to create mutant EPNs. A large numberof higher-quality mutant EPNs are needed to expect the higher degree of accuracy
for a mutation score, but the techniques to generate them have not been
established. To address this problem, we construct an algorithm to generatemutant EPNs, and develop a tool to execute the algorithm. In this algorithm using
a genetic algorithm, a set of mutant EPNs corresponds to a chromosome, and thefitness of each chromosome is evaluated based on an original EPN weighted by
metrics.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-66-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS12-3 Implementation of RETUSS to Ensure Traceability between Class Diagramin UML and Java Source Code in Real Time
Keisuke Mori, Tetsuro Katayama, Yoshihiro Kita†, Hisaaki Yamaba, Kentaro Aburada,
Naonobu Okazaki(University of Miyazaki. †Tokyo University of Technology, Japan)
It’s increasing the importance of software in society, and it’s becoming more
important to secure the quality of software. Ensuring the traceability ofdeliverables is one of effective methods to secure the quality of software. It canverify that the requirements are reflected in the programs, and close the gap
between the documents and the source code. But it has two problems: taking laborand time, and causing mistakes by human handling. This paper has implemented
RETUSS (Real-time Ensure Traceability between UML and Source-code
System) in order to solve the above two problems. RETUSS can ensure thetraceability between Class diagram in UML and Java source code in real time.

OS12-4 Prototype of a Tool to Detect Specific Comments
Satoshi Tanoue, Tetsuro Katayama Yoshihiro Kita†, Hisaaki Yamaba, Kentaro Aburada,
Naonobu Okazaki(University of Miyazaki, †Tokyo University of Technology, Japan)
This paper has developed a prototype of a tool to detect specific comments. With
the prototype, you can use regular expressions, describe patterns of detected
strings in a configuration file, and then detect a specific comment. Even whenother comments to be newly detected appears, it can flexibly be coped with byadding the pattern of the detected strings to the configuration file. The prototype
was applied to projects with 5,000 LOC (Lines of Code) or more, and examined
the precision and recall rate. As a result, the recall rate achieved 100% and theprecision rate achieved 80%. By using the prototype, it is possible to detectcomments which express tasks remaining in the source code and improper
comments presented in the source code. Reducing their comments can improveunderstandability of the source code.

OS13 Human Interface and Artificial Intelligence (5)OS13-1 A Recipe Decision Support System with Recognition Ability Recoding
Function Using Knowledge Information and Agent
Keita Saito, Taro Asada, Yasunari Yoshitomi, Ryota Kato, and Masayoshi Tabuse(Kyoto Prefectural University, Japan)
We have developed a system for recipe recommendation with recognition ability
recoding function using collaborative filtering and impression words. As a humaninterface, we have adopted an agent named as MMDAgent. In the proposedsystem, the first half recommendation process using collaborative filtering andthe second half recommendation process using impression words of ourpreviously proposed system are modified for recording the recognition ability ofan elderly person as a user. The agent asks some questions to the user for findingout the initial decay on his/her recognition ability at the appropriate timings.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-67-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS13-2 A System for Analyzing Facial Expression and Verbal Response of a Person
while Answering Interview Questions by AgentTaro Asada1, Daichi Kogi2, Ryouichi Shimada3, Yasunari Yoshitomi1, and Masayoshi Tabuse1(1 Kyoto Prefectural University, 2 S.Ten Nines Kyoto Co.,Ltd., 3 JFE Systems, Inc., Japan)
We have developed a system for analyzing facial expressions of a person whileanswering interview questions by agent. The image signal input from web camera
while answering questions is analyzed by our real-time system. Moreover, thefundamental frequencies and the time to utterance of the answerer just after aninterview question is terminated are measured for estimating the mental state of theanswerer. In our previously developed system for analyzing facial expression of aperson while speaking with another person, facial expression intensity could be
affected by (1) a conversation topic, (2) a partner and (3) the facial expression of thepartner. In this research, all of (1), (2) and (3) are fixed by using an agent’s interview.
The experimental result suggests the usability of this system for an interview where
the facial expression and the verbal response are important factors.

Interview by Agent

Analyzing Facial Expression
and Verbal Response

OS13-3 Facial Expression Analysis and its Visualization While Writing Messages
Yasunari Yoshitomi1, Taro Asada1, Kenta Mori2, Ryoichi Shimada3, Yuiko Yano1, and Masayoshi Tabuse1(1 Kyoto Prefectural University, 2 Neyagawa City Hall, 3 JFE Systems, Inc., Japan)
We have developed a real-time system for analysis of facial expressions and its
visualization. The image signal input from web camera is analyzed by our realtime system using image processing software (OpenCV) and the previously
proposed feature parameter (facial expression intensity). Our real-time systemdraws the graph expressing the facial expression intensity change using OpenCV.
We applied the system for expressing emotion as a pictograph based on the facial
expression intensity in writing messages. The experimental result suggests that
our system can be useful for expressing emotion in writing messages.

Facial expression intensity
measurement while writing
message

OS13-4 Recognition of Finger Spelling from Color Images Using Deep Learning
Yusuke Yamaguchi, Masayoshi Tabuse (Kyoto Prefectural University, Japan)
We have developed a system to recognize finger spelling in Japanese sign
language using deep neural networks. As deep neural networks, we adopt Faster
R-CNN. By defining an output class for each finger letter and learning the object
detection network, it is possible to output where the finger letter exists in the input
image. This method does not require depth cameras, magnetic sensors, or otherspecial equipment when used. Furthermore, this does not require preprocessingthat extracts the hand region using human skin colored regions and color gloves
used in other methods using color images. We synthesized a training data set byprocessing images taken with Kinect. As the test data, we input images of
performing finger letters into the trained network and check the score of theoutput area and class.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-68-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS13-5 Recognition of Texting While Walking Using Convolutional Neural
Networks
Junpei Miyachi, Masayoshi Tabuse (Kyoto Prefectural University, Japan)
The number of people who operate a smartphone while walking increases with the spreadof smartphones. Pedestrians who are texting while walking decrease attention to theirsurrounding environment. This is the cause of falling from station’s platform or bumping
into someone in public areas such as railway stations. The purpose of this study is to detectpedestrians who operate a smartphone using surveillance camera for the support of accident
preventions in railways. The pedestrians have the same features of posture such as hand
position and facial direction. In order to recognize the posture, we employ convolutional
neural networks (CNN), which has improved results in various fields in recent years. In thispaper, we propose the method of the texting while walking recognition using CNN. We
show the classification results of a pedestrian who is texting while walking.

OS14 Advanced Technology on Sensing Technology, Devices, Application (6)OS14-1 A Study on the Lumbar Burden Evaluation of Work using One
Smartphone
Mizuki Maiguma, Hiroki Tamura, Koichi Tanno (University of Miyazaki, Japan)
In this paper, we propose the human lumbar burden evaluation method and state
estimation system using smartphone and it is application to agricultural work. The
proposed system consists of two functions, 1) "State estimation" has a function of
estimating posture (Stand up, Sit down, Crouch, Walk, etc.,) of the subject. 2)"Lumbar burden estimation" has a function to estimate the angle of waist from theangle of the subject's upper body and calculate the lumbar burden in combination
with other parameters. The aim of this paper is to get the data of the subject's work
status in the agricultural field in simple manner and quasi-real time and help improve
the agricultural work efficiency by constructing the agricultural work burden
evaluation system using smartphone. In this paper, we show on the experimental
results of two functions and the evaluation of the actual agricultural work.

OS14-2 A Study on High Accuracy Stride Estimation on Smartphone Combining
Acceleration Sensor and Gyro Sensor
Shunta Nonaka, Hiroki Tamura, Koichi Tanno (University of Miyazaki, Japan)
The stride is an important parameter in human motion analysis and has been
extensively studied by many researchers. This paper focuses on smartphones that are
popular in the world, regardless of the measurement environment. In previous
research, the stride estimation using the acceleration sensor or the gyro sensor
mounted on the smartphone had been studied. However, the estimation accuracy of
stride has an error rate of about over 10 %. In this paper, we propose a highly accurate
stride estimation method using stride correction parameter obtained from crosssection movement in addition to conventional sagittal surface stride using
acceleration sensor and gyro sensor mounted on smartphone. From the results, theerror rate of our proposed method with the stride estimated by kinect sensor (RGBD sensor) as the true value was about 6%, making it possible to estimate the stride
with high accuracy.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-69-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS14-3 Development of Multi-Sensory Smart Objects Tracking Module for Mobile
Robot Platforms
B. A. D. J. C. K. Basnayake, Y.W.R.Amarasinghe (University of Moratuwa, Sri Lanka)
In the mobile robot systems, the ability to navigate in its environment is important.
Often, the surrounding environment is dynamic and unknown. Object tracking is asensing system employed with most of the robotics navigation systems. Thesesensors can provide real-time information related to the obstacles and target
detections. Living object identification and positioning are difficult compared withstationary obstacles identification, because of their behaviors. But it is important for
robotic platforms which operate in living premises in order to perform collision-freenavigation and localization features. Therefore, we have developed infrared thermal
image based smart living object tracking system which capable of detecting andpositioning movable and stationary living object in a certain range. The system
mainly consists of none contact thermal array sensor and sonar sensor that mounted
in the 2-axes rotatable platform. These sensors capture the surface temperature,
directions to the heated objects and related distance and later the captured
information is analyzed by the filtering and detection algorithms. Based on the above
information, the system will generate real-time point cloud related to the living
objects and fix-obstacles placed in the environment. Developed system will use innavigation, human following and obstacles avoiding applications in mobile robot
systems.

OS14-4 Design and Development of a Conductive Polymer Based 3D – Printed
Tactile Sensor with Square Type Spring Structure
W.H.P. Sampath, A.H.T.E. De Silva, Y.W.R. Amarasinghe (University of Moratuwa, Sri Lanka)
Industrial Robot Manipulators used in the industry lacks force sensory feedback
relating to the force exerted on the grabbing object by the artificial manipulator. In
most of the manipulators, the degree of controlling and handling only done byidentifying the presence and position of objects using optical proximity sensors.
However, it is possible to enhance the controlling of the artificial arm by introducing
a force sensory feedback unit. In this research, design, development and testing of aforce feedback system were discussed using a novel conductive polymer based 2x2
sensing element array incorporated to a novel 3D printed square type spring structure
to provide the user with closed-loop controlling. Arch was featured by a combinationof two – two-point splines where the gradient of starting and end points are zero. The
incorporated conductive polymer is a silicone rubber based polymer which hasenhancements by silica and carbon black, with Silane as the coupling agent. The
spring structure stated in this paper has been designed for force scaling purpose andnumerically analyzed using COMSOL Multiphysics prior to the fabrication to avoid
mechanical failures. The inherent problems in conductive polymer piezo-resistive
characteristics such as operation range deviation w.r.t. chemical composition could
also be overcome by this design approach.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-70-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS14-5 Design and Development of a Shape Memory Alloy Spring Actuated
Gripper for Minimally Invasive Surgeries
Roshan T.A.U, Amarasinghe Y.W.R, Dayananda N.W.N. (University of Moratuwa, Sri Lanka)
MIS is a novel and emerging surgical procedure in the medical surgery field.
Miniaturization of the tools and the actuators used in MIS is important. The
properties such as high energy density, excellent biocompatibility, corrosion
resistance, miniaturization capability etc. highlights the selection of SMA as theactuation material. Currently there were several studies available about SMA
powered grippers, but in this study, the main intension is to preserve simplicity in thedesign and controlling stage. A kinematic and dynamic analysis was performed in thesoftware based environment using MATLAB and COMSOL Multiphysics
respectively. In the development stage the structural components of the gripper ismade by 3D printing. The SMA spring is made out of Nitinol (55% Ti and 45% Ni)
wire with 0.5 mm diameter, fixed as a spring (6 mm in Dia.) and aged in a Muffle
furnace at 400 C for 15 mins. For the heating of SMA material electric current wasused and joule heating was enabled the actuation. This SMA spring actuator is
combined with a biasing spring which is made out of normal steel in order to achieve
the required opening and closing action of the gripper jaws. Among the different
biasing methods, this was the suitable design as this gripper operates on the opening
mode which is the best way to avoid SMA overheating. Temperature measurement
and a Gripper jaw displacement measurement was taken in order to discuss thefactors influencing gripper performance. As well the tests were conducted to matchthe best compromise between influence of cooling method, SMA activation current,
and operation frequency. The controlling was done by Pulse Width Modulation(PWM) method as it significantly reduces the energy consumption and robust to theexternal disturbances.

OS14-6 Design and Development of Quantum Tunneling Composite based Tactile
Sensors
T.D.I. Udayanga, D.A.M.R. Fernando, H.L.P.L. Chaturanga, B.A.D.J.C.K. Basnayake, Y.W.R.
Amarasinghe (University of Moratuwa, Sri Lanka)
Tactile Sensing is measuring tactile parameters with the aid of physical touch.
Currently, specifically regarding the robotics and biomedical fields, research werelimited by having lack of tactile feedback systems. Hence, tactile sensing has cometo the spotlight of research and has developed so considerably when comparing withthe past decades. Also application areas of tactile sensing has been expanded as new
fields of applications has emerged recently regarding tactile sensing. In this study, aminiaturized novel enclosed tactile sensor was designed and developed using
Quantum Tunneling Composite (QTCTM), which is a conductive polymer composite,
as the sensing element. An enclosed novel structure was proposed so that the sensing
element and spring will be omitted from the environmental effects. Proposed sensor
structure was analysed and manufactured. Experiments were carried out and resultsshows that the sensitivity of this developed tactile sensor to be 0.02 V/N andrepeatability of ±3 N.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-71-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS15 System and Control (11)OS15-1 Analog circuit design of a novel 4D chaotic system
Hong Niu (Tianjin University of Science & Technology, China)
In this paper, a novel four-dimensional (4D) autonomous chaotic system is reviewed,
and its corresponding new analog circuit is presented based on the modified modulebased approach to chaotic circuit design. The chaotic phase portraits of the newcircuit are given to illustrate the good qualitative agreement between the numericalsimulation and the experimental realization.

OS15-2 A method of end-to-end self-understanding of Chinese paper-dictionaries
Zhijian Lyu(Beijing Institute of Science and Technology Information (BISTI), P.R.China),
Yizhun Peng (Tianjin University of Science and Technology, P.R.China)
This paper introduces a method of end-to-end self-understanding of Chinese paperdictionaries. In this method, a page of Chinese paper-dictionaries is scanned into anelectronic image. And then the electronic image is preprocessed, including undistortion, side scrapping, binarization, and so on. Finally, using an end-to end deeplearning method, the pre-processed electronic image is intelligently segmented, text
recognized, and context understood. Our method has been applied to selfunderstand a serial of Chinese paper-dictionaries, which have more than 13000 pages
and 2.1 millions of phrases. And its correct rate of self-understanding of Chinesephrases is more than 99.5%. Its performance has proved it’s availability.

OS15-3 A multi-robot rescuing system
Huailin Zhao, Zheng Wu, Xiaoxing Wang (Shanghai Institute of Technology, China)
The study subject is a multi-agent system consisting of a few mobile robots. In thesystem, each robot collaborates with the other ones, and they all work together tocomplete the disaster rescuing tasks such as the fire fighting. The multi-agent system
control theory is applied in the system, where each mobile robot is an independent
intelligent individual. The robots communicate wirelessly among th を m. Even
though there is trouble with the individual robots, the others will collaborate among
them in time and adjust their control policy to ensure the system itself working
normally and complete the disaster rescuing task. In the system, an upper computeris designed for supervising, human-machine interfacing and being the transferring
station of the data exchange. The experiment shows that the multi-robot distributed
control system based on WIFI, GPS and INS information is able to achieve theconsistent action such as rescue gathering.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-72-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS15-4 Dynamic analysis and FPGA implementation of A novel hyper-chaotic
system with one equilibrium point
Shanfeng Wang, Hongyan Jia, Zhiqiang Guo(Tianjin University of Science and Technology, China)
In recent decades, the study on the chaotic system and hyper-chaotic system attracts
more and more interest with developing of non-linear subjects. This paper firstly
presents dynamic numerical analysis of a novel hyper-chaotic system with one
equilibrium, which includes the stability analysis of equilibrium point, bifurcation
diagram, Lyapunov exponent and so on. Then, based on the method of converting
continuous-time differential equation to discrete-time differential equation,
FPGA(Field Programmable Gate Array) implementation for the novel hyper-chaotic
system is finished, and the results from FPGA implementation are consistent withthose from numerical analysis. The work in this paper may provide a method for theapplication of the hyper-chaotic system.

OS15-5 Analysis and circuit implementation for a new fractional-order chaotic
system
Zhiqiang Guo, Hongyan Jia, Shanfeng Wang
(Tianjin University of Science and Technology, China)
2.5

2

1.5

z

This paper firstly recommends a new fractional-order chaotic system, only including
six terms and three multipliers, which is different from the Lorenz and other existing
fractional-order systems. Then some numerical analyses including the phasetrajectory diagrams, Lyapunov exponents diagrams and bifurcation diagrams are
given to investigate the different dynamical characteristics of the chaotic system. At
last, an analog circuit is designed to implement the system, further verify theeffectiveness of the fractional-order chaotic system in practical application.

1

0.5

0
-5

-4

-3

-2

-1

0
y

1

2

3

4

5

OS15-6 Application of a conservative chaotic system in image encryption
Shilong Liu, Mei Zhang, Wei Xue(Tianjin University of Science and Technology, China)©ICAROB 2018 ALife Robotics Corp. Ltd.

-73-

2

1

z

The dynamic behaviors of a new conservative system are analyzed based on its phasetrajectory, Poincaré section, Lyapunov spectrum. On this basis, the conservative
system is applied to digital image encryption and the security performance of thealgorithm is analyzed and verified. The results show that compared with thedissipative systems, the chaotic attractor in conservative system is not apparent. The
conservative system also has the characteristics of pseudo randomness and similar
noise, whose safety is high, and is not easy to be cracked. Therefore, the conservative
system has a quite high research value and application prospect.

0

-1

-2
2
1

2
1

0

0

-1
y

-1
-2

-2

x

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS15-7 Study of plant disease detection based on near-field acoustic holography
Jiangfan Wang, Xiuqing Wang (Tianjin University of Science & Technology, China)
In the process of crop cultivation, pests and diseases on crop growth anddevelopment, yield and quality of the most obvious. To control pests and diseases
quickly and effectively, real-time understanding of the growth of crops to the disease
based on reasonable and accurate spraying of pesticides is particularly important. The
key to the implementation of precision spraying is the accurate positioning of crop
diseases. In this paper, the main stem of the plant was taken as the object of study.
Based on the near-field acoustic holography, the sound field model of the disease wasestablished and the three-dimensional spatial sound field was reconstructed by thealgorithm to analyze the sound field of the sound source. Identify the sound source
signal of the plant, locate the position of the sound source, make effective judgmenton the damage status of the plant, determine the disease condition, and establish the
optimal control strategy.

OS15-8 Simulation of Cell Dielectric Properties Based on COMSOL
Shudong Li, Xiaoyan Chen, Fengze Han (Tianjin University of Science & Technology, China)
The dielectric properties of single cell can be observed by injecting a low amplitude
current at different frequencies (1MHz~200MHz). The simulation work is taken on
the software platform named COMSOL Multiphysics. The electric field and the cell
model is created with prior conductivity, the extracellular and the intracellular fluidis 1 S/m and the membrane is 1*e-5 S/m separately. By simulation, it’s verified thatat low frequencies, the region of interest (ROI) behaves the conductivity
characteristic as the electrical signal cannot pass through the cell membrane due its
capacitor properties. With the excitation frequency increasing, the ROI behaves more
permittivity characteristic as the current flowing through the cell membrane more
and the current density increases. The research of the cell dielectric property provides
an auxiliary method to diagnose the status of the cell.

OS15-9 Research on the method of electrical impedance tomography based on
conjugate gradient iterative algorithm
Yuanli Yue1, Xiaoyan Chen2, Ze Liu1, Fengzhi Dai2
( Beijing Jiaotong University, China. 2 Tianjin University of Science and Technology, China)
1Electrical impedance tomography is a new technology of medical imaging ,that canbe used to detect the real time internal electrical impedance. This article is doing
research on the method of electrical impedance tomography by using conjugate
gradient iterative algorithm. Using the finite element analysis software Comsol tomake a two-dimensional modeling, and make the target circular field with 16
electrodes is established. And the empty field and full field stimulation of different
excitation current analysis, there are relatively adjacent to motivate andencourage .By using Matlab to make tomography reconstruction, using two different
reconstruction algorithms. After the reconstruction of the graphic ,using a variety ofevaluation of the graphic image reconstruction and numerical analysis, such as theuse of the correlation coefficient, and the similarity relative error evaluation.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-74-

.

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS15-10 Research on acoustic emission wireless detection system
Xiuqing Wang 1, Yang Li2, Qing Liu 1, Jiming Zhao1( Tianjin University of Science & Technology, China;2 TEXAS A and M University, U.S.A)
1

A Fiber Bragg grating (FBG) acoustic emission wireless sensing system waspresented in this paper. The acoustic emission signal source was simulated by themethod of breaking the automatic pencil lead core, the signals collected by the sensor
were processed by photoelectric conversion, amplification, filtering and so on. The
system adopted waveform feature extraction method, through wireless sensor
network, characteristic values uploaded to the host computer software. A comparative
experiment of piezoelectric ceramic acoustic emission sensing system and FBG
acoustic emission sensing system was carried out. It can be seen from the results thatthe acoustic emission signal waveforms and characteristic parameters collected by
the two sensing systems were different. These differences were related to the inherent
structural characteristics of the sensor itself. The upper and lower surfaces of
piezoelectric ceramic sensors can cause the superposition of acoustic emission
signals, however, FBG acoustic emission sensor does not exist this problem.
Therefore, the waveform collected by FBG sensor does not produce distortion, whichis the advantage of FBG acoustic emission sensing system than piezoelectric ceramic
acoustic emission sensing system in principle.

FBG
acoustic
emission
sensorLaser lightsourceOpticalfiber
connector

Signalamplifying andfiltering
circuitPhotoelectric
converter

Acquisition and processing module
Wireless transmitting module

Wireless receiving module
Upper computer

OS15-11 ORiN PAC (PC based Automation Controller)
Toshihiro Inukai (DENSO WAVE INC., Japan)
This paper introduces application examples of ORiN PAC (PC based control systemutilizing ORiN as middleware). The manufacturing system is composed of hugekinds of devices, and it is necessary to make device programs for controlling thesedevices in cooperation. The mfg. system becomes increasingly complicated, andalong with the increase in programming cost, the improvement of programing
efficiency became a big issue. In this paper, we propose ORiN PAC as a solution torealize "one software architecture" applicable from small scale to large scale mfg.
systems. This solution has been used in the real factories in the world (not just ademo), and it will be quite important also for realizing "Industrie 4.0" world. ORiN:
Open Resource interface for the Network

OS16 Recognition and Control (9)OS16-1 Research on the synchronization and circuit realization of a four-wingchaotic system
Yiqiao Qin, Fengzhi Dai, Yuxing Ouyang, Qijia Kang, Ce Bian, Baochang Wei, Runhua Mao,
Shengbiao Chang (Tianjin University of Science & Technology, China)©ICAROB 2018 ALife Robotics Corp. Ltd.

-75-

100

50

0

z

This paper introduces a three-dimensional four-wing chaotic system and the system
is analyzed and circuit verified. The paper adopts two synchronous control strategies
to control the three - dimensional four-wing chaotic system. One is the feedback
synchronization controller based on the observer, and the other is the synchronization
controller based on theory of stability. The feasibility of these two methods is provedby analyzing the numerical data and the circuit implementation. On this basis, we
conducted signal encryption research. The results verify the feasibility of thesynchronous design and encryption design.

-50

-100

-150

-200

0

1

2

3

4

5
a

6

7

8

9

10

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS16-2 Design of control system for reboiling part of distillation process
Lingran An, Fengzhi Dai, Yujie Yan, Yuxing Ouyang, Zhongyong Ye, Xia Jin, Baochang We(Tianjin University of Science and Technology, China)
This paper selects the control system for re-boiling part of distillation process for
major research. The control scheme is determined according to the influence of therelevant parameters on the distillation process. The hardware part uses the Siemens
S7-200 series PLC as the main control machine. The lower position machine uses
STEP 7-Micro / WIN programming software, and writes the control program throughthe ladder diagram. It combines the PID algorithm and single closed-loop control tocomplete the field data collection and processing and communication with the hostcomputer. The host computer uses the configuration king named kingview6.55
monitoring system software to complete the automatic control system monitoring.

PIC
101

VA101

PI
102

V104
真空缓冲罐
TI
102

VA102

E101
塔顶冷凝器

F101
冷却水流量计

FI
101

SIC
101
VA109
VA108

P101
真空泵

VA110

TIC
101

T101
板式精馏塔

VA111
VA107
LIC
101

T102
冷却塔

VA106
VA104

VA120
VA114

E102
再沸器

VA103

V101
取样罐

TI
103

VA117
LI
102

VA113

E103
塔釜冷却器

VA105

VA118

V102
塔釜出料罐

VA115

VA112

VA116

V103
冷却水箱
VA119

OS16-3 Research on intelligent control system based on machine vision
Ce Bian1, Fengzhi Dai1, Yuxing Ouyang1, Yiqiao Qin1, Baochang Wei1, Runhua Mao1, Meili Li2(1Tianjin University of Science and Technology, China; 2 China University of Petroleum, China)
Surface Mount System is a device which function is move attachment head to makesurface mount components accurately placed on the PCB pad. SMT has machine
vision system and intelligent control system compose. The machine vision systemthrough the camera to capture images of components. Then the software for image
processing and camera calibration problem is one of key issue of machine visionresearch. Ultimately determine the component coordinate axes. The mainly functionof intelligent control system is that controller drives X, Y, Z, R each axis motor tocarry out the intelligent logic movement and controls the digital signal to carry on
the logical response. Data exchange between Machine vision and the intelligent
control part base on upper computer. On this basis, the position of the component on
the PCB can be identified by the intelligent operation of machine vision.

OS16-4 Development of NC power based on Buck circuit
Yuxing Ouyang, Fengzhi Dai, Runhua Mao, Ce Bian, Baochang Wei, Yiqiao Qin, Shengbiao
Chang, Qijia Kang (Tianjin University of Science and Technology, China)
This paper develops a kind of power-supply module which uses digital control andbuck circuit. The module can convert 220v alternating current into 0-30v direct
current. In the aspect of circuit design, it is mainly AC- DC, DC-DC conversion
circuit and SCM minimum system circuit. In the aspect of control circuit, for
feedback signal of current sensor, the power-supply module can obtain average value
by multiple means and remove the mutation value.

OS16-5 Simulation of PID temperature control system based on neural network
Yujie Yan, Fengzhi Dai, Lingran An, Yuxing Ouyang, Zhongyong Ye, Xia Jin, Ce Bian
(Tianjin University of Science and Technology, China)
The system designed in this paper is a combination of neural network and PID
control. BP neural network has a great advantage in solving the control of nonlinear
and uncertain systems. It can use the steepest descent learning method to adjust thethreshold and weight values by back propagation. The ultimate goal is to adjust the
PID controller adjustable parameters by the neural network learning algorithm. Using
MATLAB software to simulate, the results show that the system has a strong
following performance and a high anti-interference ability. Experiments show that
the PID temperature control system based on neural network has some practicality.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-76-

P102
冷却水泵

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS16-6 Design and research of real-time material management system based on
production process
Baochang Wei, Fengzhi Dai, Yuxing Ouyang, Haifang Man,Yiqiao Qin, Runhua Mao, Ce Bian
(Tianjin University of Science and Technology, China)
The paper primarily introduces the real-time material management system in thecourse of machinery production. This system is developed and designed using C#
language and SQL2008 database, using C/S and B/S mixed mode, the C# languagerelated programming skill and database information together, through the C/S and
B/S architecture for network communication, the database Server-related production
material information sharing management system feedback to the specific user, thus
enhancing the production efficiency.

OS16-7 Research on the balance control of the inverted pendulum
Zhongyong Ye, Fengzhi Dai, Yuxin Ouyang,Yujie Yan, Xia Jin, Lingran An, Hongtao Zhang(Tianjin University of Science and Technology, China)
There are many strategies in the field of automatic control to achieve stable controlof the inverted pendulum. The typical control strategy is PID control algorithm. But
PID controller can only point to point control, so this paper proposes an improved
strategy. This paper adopts multi-loop PID strategy to realize the stability control of
inverted pendulum. Through simulation analysis, this strategy is more efficient than
PID control algorithm.

OS16-8 Design of intelligent saving robot based on six-legged robot
Xinyu Zhang, Xiaokun Lin, Fengzhi Dai (Tianjin University of Science & Technology, China)
In recent years, along with the humanity to have high mobility in the complex
environment, and high reliability, and easy to expand increasingly urgent demand formobile platform, plus multi legged robot research and algorithm improvement ofgait, scientists have considerable research to the application of biped multi legged
robot, biped robot is in the past wheel type or track system, the motion is confined tothe plane, unable to overcome many mountain disaster or rough terrain. Therefore, arescue robot based on six legged robots came into being. This text robot can use
mobile phone app to switch robot mode or remote control robot through Bluetoothor wireless mode. And the robot can switch the automatic mode and switch theautomatic mode, and the robot can realize obstacle jumping through sensors to
identify the types of obstacles. And the robot comes with an electronic compass, acamera and a gyroscope, which can intelligently record the path of travel andautomatically return to its starting position along the source path.

OS16-9 Research on multi-object recognition algorithm based on video
Yong Hou, Runhua Mao,Yuxing Ouyang, Ce Bian, Binhu Song, Baochang Wei, Yiqiao Qin,
Shengbiao Chang, Fengzhi Dai (Tianjin University of Science and Technology, China)
This paper mainly studies and improves the detection and tracking algorithm of therobot fish video and presents a background extraction method based on HSV color
space. This method is more efficient than the background extraction method of RGB
color space, which improves the recognition, anti - jamming ability and robustness.
Through the experiment, it has a strong adaptability to external influences such as
background water surface fluctuation, reflective, light change shadow.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-77-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS17 Automated content generation and cognitive content generation (7)OS17-1 Reinventing the Flavor Wheel
Hiroki Fukushima (Kyushu Women's University, Japan)
In this paper, the author proposes a new type of flavor wheels. Flavor wheel is madefor the tasting professional as like sommeliers, in order to visualize the tasting words
in a hierarchical way. An ordinary flavor wheel has two or three hierarchy; abstract
genres (e.g. “floral”, “fruit”) are set in the inner circle, and concrete words (e.g.
“jasmine”, “apple”) are set in the outer part. The hierarchical structure can be
rewritten into a tree structure; hence the wheel shape needs to have its advantage incomparison to the tree structure. The author points out mainly two drawbacks ofordinal flavor wheel: 1) the whole area of the circle is evenly split by the number of
the total words. And 2) the lack of the relationships among the words and genres.
This paper aims to overcome these two drawbacks and propose a new type of flavor
wheels.

OS17-2 Acquiring Short Scripts and Setting a Case Frame in each Acquired Script:
Toward Random Story Generation
Jumpei Ono (Graduate School of Iwate Prefectural University, Japan),
Takashi Ogata (Iwate Prefectural University, Japan)
The integrated narrative generation system, that the authors have developed,
generates a story to translate the story into the surface representation. In the story
generation process, the INGS uses narrative knowledge that was automaticallyacquired from existing narrative works. This paper presents a method to acquire shortscripts, which are a kind of narrative knowledge, from existing works in Aozora
Bunko for the story generation. This paper presents a mechanism to generate random
story-like event sequences by using 23,751,142 bi-gram scripts acquired based on
the method proposed below. The authors aim to use the scripts generated by themethod as a first set to be revised through the next learning process.

Verb concept “X”
Verb “X”

X1

X2

Verb “Y”

Y1

Y2

・・・

・・・
Verb concept “Y”

Xn

Yn

OS17-3 A Method of Naimaze of Narratives Based on Kabuki Analyses and Propp’s
Move Techniques for an Automated Narrative Generation System
Takashi Ogata (Iwate Prefectural University, Japan)
Naimaze in kabuki has been known as a narrative creation method for combining
components in existing kabuki works or the other genres’ narratives to make a newwork. Various components, such as story, plot, character, place, are used as thecomponents in naimaze. The author aims to use the naimaze method in an automatednarrative generation system that we have been developing, namely the Integrated
Narrative Generation System (INGS). In this paper, we present an approach to design
the narrative techniques of naimaze in the INGS by applying the way for combining“move” in V. Propp’s narratological study, “morphology of folktales.” A move by
Propp means a narrative macro level unit or a kind of sequence that is composed ofseveral “functions” and he showed various ways that combine several moves toconstruct an entire narrative structure. This paper shows ideas of a naimaze
techniques and the implementation of experimental programs based on various
kabuki analyses and the Propp’s move method.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-78-

Narrative “A”

Narrative “B”

Narrative “C”

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS17-4 Narratology goes to Creativity ---Cognitive Content Generation
Akinori Abe (Chiba University, Japan)
In the previous studies, I discussed the creativity in IMDJ (Innovators Marketplace
on Data Jacket). The Innovation Game seems a game where a new production will
be obtained during the combination of various techniques, materials and previous
products. Accordingly I discussed IMDJ performance from the perspective of
abduction. In addition, more important matter is the effect of communication among
participants which is plays a significant role in creativity. That is, for creativity,
important matters are "action of a novel relational product" and "the role ofinteraction and collaboration with other individuals." Then I discussed the role ofcomfortable communication (narratology) by robots in the creative situation as wellas usual situation. In this paper, I will discuss the role of narratology which is basedon the cognitive content generation in the creative task.

OS17-5 A Framework for Haiku Generation from a Narrative
Takuya Ito, Takashi Ogata (Iwate Prefectural University, Japan)
In this paper, the authors propose a framework for the mutual transformation between
a narrative and haiku-like sentences. In particular, this paper shows a method forgenerating haiku-like sentences from a narrative fragment. The authors have beendeveloping an integrated narrative generation system that automatically generates
narratives. This paper aims to combine to this mechanism relating to haiku with thesystem in the future. Basic elements for generating a narrative and a haiku-like
sentence are commonly the selection, arrangement, and modification of the elements.
In order to generate haiku-like sentences from a narrative, the proposed method
selects particular elements from a narrative and arranges them according to the formof a haiku-like sentence. Further it gives variations to the haiku-like sentence. In thispaper, the authors show several forms for haiku-like sentence generation toexperimentally generate haiku-like sentences from a narrative.

OS17-6 Cognitive content generation for healthy ageing
Yuki Hayashi (Chiba University, Japan)
In the previous studies, we discussed some problems in super-aging society and how
to support older people. Recently, healthy ageing has been an important concept todecrease risk of disease, care independence or any other troubles. For example, WHO
has recommended healthy lifestyle to younger and middle-aged people. In addition,a concept “ME-BYO” which was originally generated in oriental medicine has been
accepted around the world. Accordingly to figure out the best strategy for treatingolder people in Japan has been a challenging problem. However, there remains somematters, thus, how to evaluate the state of healthy and how to provide personalized
approaches for healthy ageing. In this paper, we’ll discuss the evaluation of healthy
state and content generation for personalized approaches for healthy ageing.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-79-

Narrative
父が家屋で狼に困る.
A father is embarrassed a wolf in his house.
息子のアランが家屋で父を励ます.
His son, Alain encourages his father in his house.
父が「アランが外出する」ことを禁止する.
The father forbids Alain from going out.
アランが家屋で泥を洗う.
Alain washes off mud in his house.
アランが牧場で衣服を乾かす.
Alain dries clothes in a stock farm.
狼が牧場で羊を襲う.
The wolf attacks sheep in a stock farm.
アランが狼を待ち構える.
Alain wait for the wolf.
アランが狼を襲う.
Alain attacks the wolf.
アランが狼を殴る.
Alain hits the wolf.
狼が逃げる.
The wolf escapes.
アランが狼を逃がす.
Alain misses the wolf.
父がアランを称える.
The father praises Alain.

Haiku
Agent-focusing Type狼やOkami (a wolf) ya
襲う羊をOsou (attacks) Hitsuzi (sheep) wo
逃げるかな
Nigeru (escapes) kana

Location-focusing Type
父困る
Titi (father) Komaru (is embarrassed)
アラン洗うやAran Arau (washes) ya
家屋かな
Kaoku (a house) kana

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS17-7 Museum Visitors’ Behavioural Change Caused by Captions
Kotone Tadaki (Chiba University, Japan)
In museums, especially in art museum, there are several visitors only reading
captions (short explanation displayed next to artwork inexhibition room) without
seeing any artworks. When we regard caption as a Shikake which is aiming to makevisitors see artwork, each caption already has a physical trigger. Since caption andartwork are usually displayed on the same wall so that visitors can easily see both
caption and artwork in the same time. Although there are physical trigger, captions
are not functioning as expected. In this paper, we try to add psychological triggers byadding some features to captions. The presence of change in how they see artwork
was measured by time spent to see artworks, movement from caption to artwork, andparticipants’ impressions to each artworks and each captions. The result of theexperiment shows museum visitor’s generation of story based on information
provided by captions and artworks.

OS18 Intelligent Control (5)OS18-1 Rolling Bearings Fault Diagnosis Method using EMD Decomposition and
Probabilistic Neural Network
Caixia Gao, Tong Wu, Ziyi Fu (Henan Polytechnic University, China)
Aiming at the problem that the vibration signal of the early fault is weak, the fault
information is difficult to be extracted, the accuracy of single fault feature
identification is low, and the fault feature is redundant. A fault diagnosis method ofrolling bearing combined with empirical mode decomposition (EMD), multi - featureparameter principal component analysis (PCA) and probabilistic neural network(PNN) is proposed. First, the EMD is used to decompose the vibration signal into thesum of several IMF components to achieve the smoothness of the non-stationary
signal. Since the bearing fault information is concentrated in the high frequency
band, the first five IMF vectors of high frequency are used to obtain their energy,
kurtosis and skewness characteristics to form the eigenvector. Finally, the feature
vector group is de-redundant by principal component analysis, and it is put into PNN
for fault recognition. The simulation results are compared with the intrinsic modeenergy method of EMD. It is concluded that EMD-based multi-feature parameter
principal component analysis has higher accuracy. At the same time, the relationshipbetween the number of principal components of the method and the correctness ofdiagnosis is analyzed.

Troubleshooting flowchart

OS18-2 Detection of Dangerous Driving Behavior via Fuzzy Inference System
Shangzheng Liu1, Qinghui Zhu1,Fuzhong Wang 2( Nanyang Institute of Technology, P.R. China, 2Henan Polytechnic University, P.R. China)
1

Aiming at identifying fatigue driving and drink driving, a new detection scheme forthe analysis of dangerous driving behavior was proposed in this paper. First, three
eye movement characteristics, average number of saccade, average fixation duration,and fixation counts. Second, membership functions and fuzzy inference system (FIS)
were established. Finally, fatigue driving and drink driving were identified by the
proposed FIS. In the Matlab/Simulink simulation environment, the simulation
system was built. The simulation is carried out to verify the effectiveness of the
proposed FIS.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-80-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS18-3 Feature Points Designing and Matching for the Target Spacecraft in the
Final Approaching Phase of Rendezvous and Docking
Wenjing Pei, Yingmin Jia (Beihang University (BUAA), China)
Motion information of the target spacecraft can be calculated according coordinatechange of feature points based on the motion control model. In order to extract
motion information including velocity, location, attitude of the target spacecraft in
the final approaching phase of rendezvous and docking, feature points marking andmatching for the target spacecraft is an essential step. In this paper, feature points aredesigned on two solar panels and the fore-end cabin in the target spacecraft by using
three baselines and the feature circle methods. Thus 51 feature points can beobtained. Feature points matches are obtain by applying an improved feature points
matching strategy to feature points which have been designed. Experimental resultsshows that running speed is improved greatly and high accuracy is maintain.

OS18-4 Construction and Visualization of Atmospheric Environment Data Map
Dongmei Fu, Gaoyuan Wang, Chao Wu, Mengchen Cui(University of Science and Technology Beijing, China)
This paper includes two points: 1, how build the China's map of the atmosphericcorrosion level, according to the ISO9223-2012 Two methods are used. One is
the inverse distance weighting method, and other is the improved inverse
distance weighting based on density. 2, how build the China's map of carbon steel
corrosion rate, according to sparse carbon steel corrosion data. Two methods areused. One is a multi-faceted function method, and other is an improved multifaceted function method.

OS18-5 Multiple-Model Adaptive Estimation with New Weighting Algorithm
Weicun Zhang (University of Science and Technology, China)
This paper presents a new scheme of weighted multiple-model adaptive estimation
for a discrete-time stochastic dynamic system with large parameters uncertainty, inwhich the classical weighting algorithm is replaced by a new weighting algorithm toreduce the calculation burden and to relax the convergence conditions. Simulation
results verified the effectiveness of the proposed scheme.

OS19 Advanced Control (5)OS19-1 Revisit Constrained Control of Chaos
Yunzhong Song, Ziyi Fu and Fuzhong Wang (Henan Polytechnic University, P.R.China)
Only two conditions were considered in paper titled as constrained control of chaosin [Y.-C. Tian, M. O. Tade,. and D. Levy, Physics Letters A296 (2002) 87], andthereafter a special problem was investigated and the corresponding solving strategy
was provided in paper titled as some comments on constrained control of chaos
appeared in [Y. Z. Song, G. Z. Zhao, and D. L. Qi, Physics Letters A359 (2006) 624],
in this Letter, a soft control scheme which is chattering free dominated just by single
flexible factor is suggested to improve the already existing conclusions, andsimulation results indicate the validity of our project.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-81-

Control results of u+ with M=3,
 =1/4

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS19-2 Research on Filtering for Random Data Packet Dropouts and Delays in
Wireless Sensor Networks
Sumin Han, Fuzhong Wang (Henan Polytechnic University, P.R.China)
A wireless transmission data model is proposed to solve the problem of packet loss
and delay in data transmission of wireless sensor network data transmission system.
Based on the minimum variance estimation, the optimal fusion filter and crosscovariance matrix are derived. The effectiveness of the system is verified by asimulation example.

OS19-3 Optimal Hohmann-Type Impulsive Ellipse-to-Ellipse Coplanar Rendezvous
Xiwen Tian, Yingmin Jia (Beihang University (BUAA), China)
This paper devotes to the problem of ellipse-to-ellipse coplanar rendezvous in low
eccentricities, where the solution and distribution of Hohmann-type optimal
impulsive rendezvous are investigated. The process of ellipse-to-ellipse coplanar
rendezvous is described in the reference frame built in a circular orbit near two
spacecrafts. Based on which, the analytical relation between the initial states andrendezvous time are derived for Hohmann-type, and the optimal impulse amplitudes
are given thereupon. By selecting rendezvous time as the X-coordinate and special
phase angle as Y-coordinate, the distribution boundary of Hohmann-type model isobtained according to the Hohmann transfer and Hohmann with coasts. Simulations
are demonstrated to analyze the influences of the solution and distribution.

The distribution of Hohmann-type

OS19-4 Research on SVG Control Method Under Unbalanced Conditions
Zheng Zheng, Yousong Zhou (Henan Polytechnic University, China)
SVG (static var generator) is an important equipment to improve the power quality.
Aiming at the complexity of the alternating current positive and negative sequence
component separation method under unbalanced operating conditions, the method
based on complex filter is used to reduce the control computation. Because thetraditional PI control can not eliminate the steady-state error of the grid current in the
nonlinear disturbance, the control method of the proportional complex integral isadopted, and the traditional proportional complex integral control based on the α βcoordinate system according to the equivalent transformation relation of the phasor.
The conversion to the three-phase coordinate system is based on the reduction of theamount of computation required for the traditional control to be improved by the
Clark transform and the Clark inverse transform, and then gives a complete set ofcontroller parameter tuning methods. Finally, a set of experimental platform basedon DSP28335 is built. The experiment and simulation results verify the fastness andaccuracy of the proposed positive and negative sequence component calculation
methods and the stability and feasibility of the proportional complex integral control
method.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-82-

ea
eb
ec

Load

ea
SignalDetection

SignalDetection

Control
Circuit

DriverCircuitMain circuit topology of
SVG

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS19-5 Dynamics Analysis of Payload On-orbit Catapult Separation Based on
ADAMSYi Li, Yingmin Jia (Beihang University (BUAA), China)
In this paper, a catapult separation device equipped with guide mechanism isdesigned firstly, and the corresponding dynamic model is built through multi-bodydynamics theory. Then, based on the dynamic simulation software ADAMS, thevirtual prototype model of the separation device is established and the whole processof payload on-orbit catapult separation is simulated. Finally, the structure design
parameters of the guide mechanism such as the number of roller group, distance
between rollers or the guiding length are studied and the influences of the parameterson the separation process are analyzed. The simulation results can be applied in thedesign and optimization of the separation device.

l angular velocity of payload under
different distances

OS20 Advances in Marine Robotics and It’s Applications (5)OS20-1 A land testbed for experimental research on autonomous ship navigation
Keisuke Watanabe, Kazumasa Harada, Koshi Utsunomiya (Tokai University, Japan)
In the development of automated ship technology, experimental system is essential.
The main navigation system is supposed to consist of GPS, AIS (Automatic
Identification System), RADAR and cameras. In our land testbed, 2 ultrasounddistance sensors are used as a model of GPS, which determine the coordinate of the
testbed measuring the distances from 2 walls. A swinging ultrasound ranging sensor
with a servo motor is used to simulate the RADAR. A xbee wireless communication
module simulates the AIS. The camera is used for image processing to determine thecontrol algorithm to avoid the collision from other moving obstacles. In this paper,
we introduce our testbed’s system assembly and an experiment to confirm functionsof our concepts for our future experimental research on autonomous ship navigation.

OS20-2 Seafloor Image Color Enhancement Method based on Retinex modeland Experiment Report in the undersea environment
Jonghyun Ahn1, Shinsuke Yasukawa2, Yuya Nishida1, Takashi Sonoda1, Keisuke Watanabe3
Kazuo Ishii1 (1Kyushu Institute of Technology, 2University of Tokyo, 3Tokai University, Japan)
Underwater robot is one of the important research tools to explore deep-sea wherehigh pressure, darkness, radio attenuation prevent humans from direct access.
Especially, Autonomous Underwater Vehicles (AUVs) attract attentions because they
don't have tethered cables and can move freely. One of the important tasks of AUV
is to take the seafloor images, however, the sea-floor images should be enhanced

as red color attenuates rapidly and images become bluish, and the small
differences of AUV altitude to the sea-floor affect the brightness of images
because of light attenuation. In this paper, automatic camera parameters
adjustment method is proposed to enhance seafloor images and evaluatedthrough experiments in the Suruga-bay.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-83-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS20-3 Automatic recognition of benthic species using image processing
Yuki Soejima, Yuya Nishida, Takashi Sonoda, Kazuo Ishii(Kyushu Institute of Technology, Japan)
For sustainable use of fisheries resources, it is important to estimate the amount of
resources such as TAC (Total Allowable Catch), TAE (Total Allowable Effort) andtheir investigations have been conducted by bottomed net fishing, however, themethod suffers environmental damages. The survey method for wider area that does
not affect the ecosystem is needed. Recently, research using an autonomous
underwater robot (AUV) is introduced as a new survey method as the solution. We
also introduced an AUV for the survey of fish and crabs in the Sea of Okhotsk. The
number of marine lives is measured manually, and the technique of automatic
measurement of the number is needed for the observation. In this research, we
propose an automatic marine life recognition method from seafloor images.

OS20-4 AUV homing using acoustic chirp signal
Koji Masuda, Yuya Nishida, Takashi Sonoda, Kazuo Ishii(Kyushu Institute of Technology, Japan)
For autonomous underwater vehicle (AUV), high autonomy is required in order toaccomplish mission such as inspection, observation, manipulation under extreme
environments, deep-ocean. One of necessary function for AUV is acoustic
navigation. In this paper, we conducted homing experiments using a small AUV andacoustic pinger with chirp signal. The chirp signal has the feature that the frequencyof signal change gradually, so that the signal is suitable to calculate the correlation
between the time difference between hydrophones. The homing experiments arecarried out and evaluated through tank tests.

OS20-5 Simulation of horizontal vibration suppression of a suspended structure forseabed mining
Keisuke Watanabe1, Kazuo Ishii (1Tokai University, Kyushu Institute of Technology, Japan)
In a project of subsea mining in the near future, many work class robots such as
crawlers, drilling machines, or ore collectors will be installed on the seabed. These
robots are heavy but they must be landed on the seabed very softly to avoid impact
damage from the seabed. So the installation method is very important and effective
installation method should be investigated. In this paper, we examined vibration
motion of a suspended heavy machine by a crane vessel through dynamics modelingand simulation in the horizontal plane. Descending speed of the machine affects theamplitude of the vibration, that is, if the descending speed is higher, larger amplitude
will be observed. To suppress this vibration, we examined a velocity feedback
control by attaching a thruster to the machine. From simulation results, we foundhorizontal vibration will be effectively suppressed by using simple feedback control.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-84-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS21 Robot Competitions for Social Contribution (5)OS21-1 Analysis of Team Relationship using Self-Organizing Map for University
Volleyball Players
Yasunori Takemura, Kazuya Oda, Michiyoshi Ono (Nishinippon Institute of Technology, Japan)
In Japan, sports efforts are actively being carried out to host the 2020 Olympic games.
Especially in the field of sports science, researches on ergonomics, development ofsports equipment and pattern recognition technology using artificial intelligence are
actively researched. In previous research, we developed a clustering algorithm for
positioning adaptation and relation ships in team sports using Self-Organizing Mapsin university rugby players. However, I have not yet confirmed whether thedeveloped algorithm can be applied to other team sports. For this reason, we applied
the same algorithm to an university volleyball player. Then, as an algorithm, we
verify whether it can be generally used for team sports

OS21-2 Optimization for Line of Cars Manufacturing Plant using Constrained
Genetic Algorithm
Keiji Kamei, Takafumi Arai (NishiNippon Institute of Technology, Japan)
Recently, improvement of production efficiency on cars manufacturers is required
owing to increasing of demands in developing countries. However, that
improvements under existing circumstances are depending on experience andintuition by workers. For this, we propose to objectively and efficiently improve a
production line based on a GA. The production line of candidate for optimization is“Picking up assemblies” area, so that our proposal optimizes the positions of racks
and boxes for assemblies by a GA. The difficulty of applying a GA is that the numberof racks and boxes is defined in advance. For this, we apply constrained GA to
optimization for those positions. The results of simulation for virtual production line
show that our proposal succeeded in reducing about 10 seconds per car comparedwith random positioning.

1st Gen(Random)： Maximum about 70.9[sec]Minimum about 65.3[sec]The last gen： Minimum about 61.7[sec]Reduction time: about 10[sec] per car

OS21-3 Slip model of roller driven ballKenji Kimura1, Chota Chikushi, Kazuo Ishii( Nippon-Bunri University, Kyushu Institute of Technology, Japan)
1

Spherical wheel motion driven by rollers attract attention due to the possibility ofuneven surface. RoboCup is a platform designed to promote the research fields suchas Artificial Intelligent (AI) and robotics, and one of examples is the ball dribblingmechanism of RoboCup Middle size league robot, and this mechanism controls theball with two driving rollers. As the result of the survey at the 2017 World Cup, allteams in the world have determined the roller arrangement heuristicaclly, and nomathematical consideration has been made. In this study, we propose a model
considering slip, discuss the relationship between slip ratio and robot speed using
evaluation function, and verify the model by experiment.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-85-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS21-4 Strategy Analysis of Multi-Agent Games Using Self-Organizing Map
Moeko Tominaga, Yasunori Takemura1, Kazuo Ishii(Kyushu Institute of Technology, 1NishiNippon Institute of Technology, Japan)
It is one of important subjects to know how humans and robots share the same space
to realize a symbiotic society, for examples, strategies such as their positions anddefense methods in multi-agent game, soccer. In the soccer games, the player’s
behavior changes depending on the game situation such as wining or losing, score
gap, remaining time. The players act more offensive when their team is losing, ormore defensive when their team is winning with minimum score difference. In thispaper, we observe and analyze human soccer game, and estimate how the state vectorof the player affects team behavior. The team strategy is analyzed based onparameters such as the positions of robots, the number of robots in play, scores, timeand actions of robots using Self-Organizing Map (SOM).

OS21-5 Analysis of Characteristics of Tomato Fruits in Infrared Images Toward
Automatic Tomato Harvesting System
Takuya Fujinaga, Shinsuke Yasukawa, Binghe Li, Kazuo Ishii(Kyushu Institute of Technology, Japan)
Labor shortages and aging are big problems in Japanese agriculture field and farmers
need systems for automation, efficiency and optimization. We focus on tomato fruits
harvesting, which is one of the major fruit vegetables. Tomato harvesting needs
longer working time than other fruits harvesting. We aim at the realize of a tomato
automatic harvesting system, and have been researching and developing tomato
harvesting robots that can harvest fruits and monitor the growth situation of tomato.
It is necessary to detect the position and maturity of tomato fruit. In this paper, weanalyze the plant characteristics of tomato using infrared images and propose tomato
fruit recognition system. Images acquired at the tomato greenhouse of large scale
facility were used. The analysis results of tomato fruits features in infrared images
are reported.

OS22 Navigation and Control (3)OS22-1 Comparative Study of Sequential Processing TRN (Terrain Referenced
Navigation)
Hyun Cheol Jeon and Chan Gook Park (Seoul National University, Korea)
This study performs comparative analysis on the performance of sequential
processing terrain referenced navigation (TRN) by applying various filtering
techniques. The TRN is a navigation system that prevents divergence of inertial
navigation system using terrain information. In this paper, we consider extended
Kalman filter (EKF) and point mass filter (PMF) which are widely used in the TRN.
The EKF estimates state variables by linearizing a nonlinear models but it has risk ofdivergence due to linearization errors. On the other hands, the PMF estimates thestate variables robustly but it has a disadvantage of high computational load. Throughthis study, the advantages and disadvantages of the TRN when applying the EKF and
PMF are compared and analyzed.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-86-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS22-2 Road Marking Map Matching for Road Vehicle Localization
Kyuwon Kim and Gyu-In Jee (Konkuk University, Korea)
In this paper, we propose map matching algorithm using road marking intensity map
made of 3D LIDAR. The road marking is very useful in terms of availability because
it is existed anywhere on the vehicle. The map matching algorithm is used phasecorrelation using FFT to enable very fast matching. In addition, the road marking
map is generated by binarizing the intensity. As a result, a large-scale global map can
be stored with a small capacity. In addition, LIDAR scan data was accumulated forgenerating a high precise density local map. Accumulated local map can be supported
sufficient information, especially longitudinal information. Therefore, it can be morerobust and precise vehicle localization.

OS22-3 Control System Design of Directionally Maneuvering Multicopter with
Independent Heading Rate
Byoungjin Lee, Jaehue Bae, Gwang Soo Park and Sangkyung Sung (Konkuk University, Korea)
In this paper, we present a comprehensive control system design of directionally
maneuvering multicopter platform with independent heading rate. A directionally
maneuvering multicopter system is capable of generating an arbitrary vehicle flighttrajectory while it maintains an independent rotation with constant heading rate. For
this, a quaternion based attitude controller is adopted in the control algorithm
construction. Torque difference between clockwise and counterclockwise rotors isbasically derived for yaw moment generation, while their thrust difference isharmonically configured for the desired waypoint flight via guidance logic.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-87-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS abstracts
GS1 RoboticsI (3)GS1-1 Production simulation of autonomous decentralized FMS including AGVs
with different personalities of mind
Ryuichi Tsujii, Hidehiko Yamamoto and Takayoshi Yamada (Gifu University, Japan)
This study researches Automated Guided Vehicles (AGVs) moving control byusing the model of a mind in order to avoid AGVs' interferences in autonomous
decentralized Flexible Manufacturing Systems (FMS). The model of a mind can
avoid the interference by repeating the two types of mind changes, the arrogant
mind and the modest mind. Further, we give some personalities to the mind. The
personalities have different time to change from the arrogant mind and the modest
mind. By carrying out the production simulations using the AGV group with thedifferent personalities, we acquire the production outputs. As a result, theoutputs are improved compared with the traditional AGV group with a single
personality. It is ascertained that the models of minds expressing different
personalities are useful.

GS1-2 Development of Automatic Recognition of Hazmat Marking Chart for
Rescue Robot
Wisanu Jitviriya1, Poommitol Chaicherdkiat, Noppadol Pudchuen(King Mongkut’s University of Technology North Bangkok, Thailand)
Eiji Hayashi (Kyushu Institute of Technology, Japan)
A long history in the 1st place awards of World RoboCup Rescue Robotcompetitions are Invigorating Robot Activity Project (iRAP) such as iRAP_PRO,
iRAP_FURIOUS, iRAP_JUNIOR and iRAP_ROBOT teams. In this paper, we
would like to introduce and explain an autonomous system of our rescues robot for
detection and recognition of Hazardous Material (Hazmat) marking chart. All
Hazmat tags are considered and computed by using Speeded-Up Robust Feature(SURF) combined with Fast Library for Approximate Nearest Neighbors (FLANN)
in order to match with the templates. Finally, the paper presents experimental
results based on real situations in order to confirm an effectiveness of the patternrecognition of robotics system.

GS1-3 Development of Autonomous Robot for Laborsaving of the Forestry- Discrimination between trees and weeds using RGB-D Shingo Yamaguchi, Eiji Hayashi, Ayumu Tominaga (Kyushu Institute of Technology, Japan)
In traditional forestry, the weeding task places a large burden on workers.
Therefore, it has been expected that such the problem will be solved by a robot. We
have been developing an autonomous robot that eliminates the weed plants without
damaging trees (these are resources). The robot should recognize trees exists in theenvironment. In this research, we have been focused on distinguishing between
trees and other them. The system computes the RGB histogram of pixels
corresponding point-cloud data of the ground extracted by pane detection. The
threshold values are determined to discriminate between trees and weeds by thishistogram automatically. We show the effectiveness of the developed system.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-88-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS2 Bipedal robot & Human-welfare robotics (6)GS2-1 Design and Evaluation of Passively Powered Knee Exoskeleton (PPKE) for
Squat Lifting
R.K.P.S. Ranaweera1, R.A.R.C. Gopura1, T.S.S. Jayawardena1, G.K.I. Mann2(1University of Moratuwa, Sri Lanka), (2Memorial University of Newfoundland, Canada)This paper proposes a passively-powered knee exoskeleton (PPKE) to provide
power assistance during squat lifting of objects from the ground. The passive
powering mechanism is designed to capture 20% of the waste mechanical energy
available at the biological knee joint during decent phase and to release the stored
energy in ascending phase in a complete squatting cycle. The novel passive locking
mechanism enables the wearer to ambulate freely posing minimum restrictions inbetween squat lifting tasks. The anthropomorphic exoskeleton structure wasfabricated using acrylic glass and powering system include elastic helical springs,
where the entire working model only weighs at 900 g per unit. The effectiveness ofproposed system was verified by experiments using surface electromyography(sEMG) signals under different test conditions. The exoskeleton is capable ofreducing peak RMS averages of sEMG signals of knee extensor muscles by 15%.
As such, preliminary results proved PPKE’s potential to reduce human effort.

Thigh
Support
Passive
Lo c k i n g
Mechanism

CablesHelicalSprings

PassiveP o we r i n g
S ys t e m

ShankSupportGS2-2 Behavior design of robot arm imitating the consciousness mechanism of livingorganisms Representation of facial expression in transition process of emotion
Ryohei Anshi, Eiji Hayashi (Kyushu Institute of Technology, Japan)
Wisanu Jitviriya (King Mongkut’s University of Technology, Thailand)
Sakmongkon Chumkamon (Rajamangala University of Technology,Thailand)
Although various service robots are being developed, social robots are required tohave "user affinity", which gives the user a sense of affinity, in particular. In thisresearch, we aim to realize user affinity by paying attention to living organisms thatembrace healing and affinity, giving the robot consciousness, behavior, andemotion of living organisms. In the transmission of emotions, the eyes areimportant parts that express various psychological states due to changes in pupilsize, eyelid opening / closing degree, shape, and the like. Therefore, we give twodegrees of freedom to the eyeball and the eyelid, we can also change the pupil
diameter and blink cycle, and constructed the system so that the expression alsochanges finely according to the change of the ratio of the eight emotions using theexpression.

GS2-3 Study of Robot Navigation for Forest Management
Ayumu Tominaga, Eiji Hayashi (Kyushu Institute of Technology, Japan)
Abbe Mowshowits (The City College of New York, USA)We have been developing a robotic system for labor-saving of forestry that moves
between trees and cut weeds plants without damaging the trees. The trajectorycomputation of the forest is carried out based on a weighted graph with trees asvertices. The line graph L (G) calculated from first graph represents a graphconnecting the midpoints of each tree pairs that the robot can safely pass through,and the Hamiltonian circuit calculated therefrom is give a path plan for the robot topass through all the tree pairs and return to its starting position. In order to follow
the trajectory path, we combined the obstacle avoidance behavior and the visionsystem which has actuator for pan angle rotate.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-89-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS2-4 Development of the sense system that is combined force feedback and visualfeedback -Deformable virtual objects simulation by using LEMKaito Nagano, Eiji Hayashi (Kyushu Institute of Technology, Japan)
In the medical treatment and bio-technology field, doctors and researchers needtechnologies that can accurately perform minute work. Such minute work is
improved by using micromanipulators, but their operation is difficult because theoperator has no sense of force. As a result, a person who has the skill to do minutework is needed for all minute works. The purpose of this research is to develop a
combined sense system that uses both force feedback and visual feedback on a
deformable virtual objects simulation. Especially, focus on a way to produce a force
that is calculated in simulation of deformation using a haptic device. As the firststage, our simulation of deformation was improved by using LEM (Long Element
Method) and evaluated accuracy of a virtual object’s deformation.

GS2-5 A four-legged robot's soft feet structural design and walking gait generated
from inverse kinematics
Amornphun Phunopas1, Eiji Hayashi2( King Mongkut’s University of Technology North Bangkok, Thailand,
2
Kyushu Institute of Technology, Japan)
1

The conventional wheel is famous in the industrial mobile robot because it is simple
to build and easy to control and to maneuver. On the other hand, the legged robot
is complicated to control but has a high performance in locomotion like highly
evolved legged-creatures. This research describes the four-legged robot design
platform with the soft feet, which are made by silicone-material forming. The
robot’s feet are implanted by the strain gauge sensor. The robot can percept theexternal force sensitively when its feet touch something. The research demonstrates
the forward kinematics and inverse kinematics of a four-legged robot with three
joints for each. The result has validated the kinematic equations byimplementing them on the real dog-like pet robot. The robot walked with a purewalking gait and showed a sensing signal from the soft foot and ground contact.

GS2-6 Powered Ankle Exoskeletons: Existent Designs and Control Systems
A.H. Weerasingha, W.P.K. Withanage, A.D.K.H. Pragnathilaka, R.K.P.S. Ranaweera, R.A.R.C.
Gopura(University of Moratuwa, Sri Lanka)
Powered ankle exoskeleton is a wearable device fitted to the ankle joint for
purposes such as rehabilitation, power augmentation and locomotion assistance.
Although variety of technologies were developed, commercial application was
found to be minimal owing to various limitations with respect to both function andperformance. Notably, conservation of metabolic energy, anatomical conformance,
agility during operation, etc. are lacking factors in contemporary exoskeletonsrobots. In order to realize rational solutions, an improved understanding of theexistent designs and control systems is paramount. In that context, this paper
performs a comprehensive review on active ankle exoskeletons powered byexternal means. Initially, anatomy of the ankle joint complex is highlighted. Next,
associated research challenges and design difficulties are recognized for thedevelopment of robotic hardware systems. Then, mechanisms, actuation methods,
and control methods are systematically presented. Here, exoskeletons designs from2005 to 2017 were studied to recognize the latest paradigm shifts. As such, thispaper provides a classification, a comparison and an overview of related
technologies. Finally, design recommendations are made on observed trends andpatterns for the development of powered ankle exoskeletons.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-90-

Ankle Exoskeleton Robots
Externally Powered / Active
Actuation
MethodElectrical

Unpowered / Passive

Control
MethodInputbased

Outputbased

HydraulicEventbased

Modelbased

Pneumatic

EMGbased

Impedancebased

Other

Other

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS3 Complexity (5)GS3-1 Management of digital records inspired by Complex Systems with RADAR
Anne Jeannin-Girardon, Alexandre Bruyant, Nicolas Toussaint, Pierre Collet,
Pierre Parrend (University of Strasbourg, ECAM Strasbourg-Europe, France)(Big)Data storage poses two major challenges: security (data can be very
sensitive, e.g. medical or financial records) and robustness (IT infrastructure may
not be reliable due to faulty hardware, bad internet access or even power outage,
especially in developing countries). Centralization in data-centres can therefore
lead to bottlenecks and significant losses, in case of failure or intermittent
connexions. Sensitive data must also be reliably backed-up. However, manual or
automatic backups are prone to errors or failures. Consequently, we propose to
apply the principles of Complex Systems (sets of autonomous interacting entities
exhibiting emergent behaviour) to the problematics of storing and backing-up
sensitive data in a platform called RADAR (Robust Anonymous DAta Records).
Storage is no longer centralized: RADAR instances automatically fragment,
replicate and distribute the data randomly over a cooperative network foranonymized and encrypted robust and secure backups and storage even over nonreliable IT infrastructure.

GS3-2 Integrated optimization of differential evolution with grasshopper
optimization algorithm
Duangjai Jitkongchuen, Udomlux Ampant (Dhurakij Pundit University, Thailand)This paper proposes a scheme to improve the differential evolution algorithm (DE)performance with integrated the grasshopper optimization algorithm (GOA). The
grasshopper optimization algorithm which mimics the behavior of grasshopper.
The characteristic of grasshoppers is slow movement in the larval stage but sudden
movement in the adulthood which seem as exploration and exploitation. The
grasshopper optimization algorithm concept is added to DE to guide the search
process for potential solutions. The efficiency of the DE/GOA is validated by
testing on unimodal and multimodal benchmarks optimization problems. The
results prove that the DE/GOA algorithm is competitive compared to the othermeta-heuristic algorithms.

GS3-3 Efficient collective search by agents that remember failures
Masao Kubo, Nhuhai Phung, Hiroshi Sato(National Defense Academy of Japan, Japan)
The BRT is an algorithm that the agents can find appropriate collective actions bychanging their agreement in a trial-and-error manner. When the solution is the only
one, computer simulations shows the number of times to find the solution is
proportional to the square of the number of their collective choices on average. In
this paper we propose an agent that remember failures behaviors to improve thesearching time. Since the agent does not propose wrong behavior, it is trivial thatthe whole search time will be shortened. In this research, we exchanged the BRT
agent and this "never suggest again" agent and investigated the search time and its
ratio. As a result, we found that even a small number of the proposed agents canimprove search time.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-91-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS3-4 The analysis of band structures of photonic crystals
Wang Chenxu (Muroran Institute of Technology, Japan),Fu Ziyi (Henan Polytechnic University, China)
In this paper, a finite element is developed to calculate the band structures of onedimensional photonic crystals. The Maxwell equation governing the propagationof electromagnetic waves, which combine with Bloch theory, the FEM transforms
the complex band diagram problems into a simple eigenvalue problem by solving
the eigenvalue with respective to wave vector K by frequency

GS3-5 A Data Estimation Technique for Incomplete Telemetry Data Based on a
Genetic Algorithm with Data’ Statistical Properties
Masahiro Tokumitsu, Kaito Mikami (National Institute of Technology, Yonago College, Japan)
Fumio Asai (The Radio Amateur Satellite Corporation, USA)
Taku Takada (National Institute of Technology, Kochi College, Japan)
Makoto Wakabayashi (National Institute of Technology, Niihama College, Japan)
Satellites transmit telemetry data to ground stations in order to provide the data on
their missions. The telemetry data contain a sort of data such as observations,experiments, and satellites’ health status. However, the telemetry data received
by the ground stations may contain errors by effects through the transmissions. This
paper proposes a data estimation technique for incomplete telemetry data to attaina high availability of the received telemetry data. The proposed technique is basedon a genetic algorithm with data’ statistical properties. This paper demonstrates
the proposed technique with simple examples. In simulations, this paper examines
the proposed technique for applying the actual telemetry data obtained by theparticular satellite.

GS4 Pattern recognition & Image processing (5)GS4-1 Study on Detection of Nests on Pylon from Overhead View Based on Halcon
Jiwu Wang, Haibao Luo, Pengfei Yu (Beijing Jiaotong University, China)
There are many difficulties when researchers want to dectect the nests on pylons in
the electricity transmission network with image processing methods. One majordifficulty is to position the pylon precisely in the image because of the variance ofshooting angle and shooting distance.Plus the unconspicuous features of nest's
edge, color and shape in the background of the image make the single-feature-based
detection methods fail to realize precise positioning. Thus, this paper addresses aneffective method based on Halcon, which could detect the nest in the images fromunmanned aerial vehicles(UAVs).This method eliminates most of the backgroundinterference by recognizing the pylon area in the image using the Blob algorithm.
Within this area, the fusion of texture and gray-scale features of nest are employed
to realize the detection. An experiment result is presented to validate that thismethod can actually realize the precise detection of nests on pylon with images shotby the UAVs.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-92-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS4-2 Human gait recognition based on Caffe deep learning framework
Jiwu Wang, Feng Chen (Beijing Jiaotong University, China)
Human gait recognition as an emerging biometrics technology has important
theoretical significance and practical value. At present, the research on human gait
recognition is still in the stage of theoretical exploration, and most of the methods
to recognize the targets are realized by the traditional pattern recognition methods,
which have no many advantages in detection accuracy and speed of detection. With
the development of deep learning theory and technology, this paper will achieve
human gait recognition of specific targets based on Caffe deep learning framework
through the faster-rcnn algorithm. The main contents of the paper include thesamples processing, model training and result testing.

GS4-3 Unsupervised Image Classification Using Multi-Autoencoder and Kmeans++
Shingo Mabu, Kyoichiro Kobayashi, Masanao Obayashi, Takashi Kuremoto(Yamaguchi University, Japan)
Image classification using machine learning has been actively studied. However,machine learning, especially deep learning, needs a large number of training data.
It is tough and impractical task for users to give correct class labels to the data.
Therefore, an unsupervised image classification system is proposed in this paper.
The proposed method consists of multi-autoencoder and K-means++. Since it isdifficult to realize unsupervised classification system due to the lack of class
information, the proposed method applies some image processing techniques to theimages, and the feature values are extracted from the original images and theprocessed images using multi-autoencoder. The images with the extracted features
are clustered by K-means++ algorithm, which can make groups of similar images.

.
Ground piano cluster

GS4-4 Anomaly Detection of Disaster Areas from Satellite Images Using
Convolutional Autoencoder and One-class SVM
Kohki Fujita, Shingo Mabu, Takashi Kuremoto (Yamaguchi University, Japan)
In recent years, research on detecting disaster areas from satellite images have been
conducted. When machine learning is used for disaster area detection, a large
number of training data is required, however, we cannot obtain so much training
data with correct class labels. Therefore, in this research, we propose an anomaly
detection system that finds abnormal areas that deviate from normal ones. The
proposed method uses a convolutional autoencoder (CAE) and One-class SVM(OCSVM). First, CAE executes feature extraction from satellite images. Second,
OCSVM is trained using images of normal areas only. Finally, anomaly detection
for the testing images is carried out by the trained OCSVM. In the performance
evaluation, the proposed method shows better performance than the method with
OCSVM only.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-93-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS5 Neuroscience (3)GS5-1 Improving EEG-based BCI Neural Networks for Mobile Robot Control by
Bayesian Optimization
Takuya Hayakawa, Jun Kobayashi (Kyushu Institute of Technology, Japan)
The aim of this study is to improve classification performance of neural networks
as an EEG-based BCI for mobile robot control by means of Bayesian Optimizationof learning parameters and hyperparameters in training the neural networks. The
parameters were intuitively decided in our preceding study. However, theperformance will be improved if the parameters are determined in an appropriate
way. There are several methods for parameter optimization. Grid Search draws agrid on a parameter search space, examining all intersection representing a
combination of the parameters. The method can find the best parameters with ahigh probability, but it takes a huge amount of time. The authors have used
Bayesian Optimization to optimize the parameters in a shorter time and achieved
the better performance.

GS5-2 Selective synchronization of the coupled bifurcating neurons for phase shiftof background oscillation
Akihiro Yamaguchi1, Yutaka Yamaguti1, Masao Kubo2 (1 Fukuoka Institute of Technology,2
National Defense Academy of Japan, Japan)
Synchronization in the coupled bifurcating neurons was studied in the view pointof the selective formation of cell assembly. The bifurcating neuron is a simple
chaotic neuron that exhibits chaotic inter-spike interval dynamic by adding thesinusoidal background oscillation. In this research, we introduce the phase shift ofthe background oscillation and the several types of phase response to the input spikesequence. The coupled 16 bifurcating neurons with complete bidirectional coupling
were numerically simulated and their synchronized behaviors were observed,
where each 4 neurons have the same phase shift value that is different with the otherneurons. Fig. 1 shows the synchronization ratio of each pair of neurons. This result
indicates the formation of chaotically synchronized cell assembly for the same
phase shift value.

Fig. 1 Synchronization ratio of
neurons

GS5-3 Review on computational techniques in solving aircraft landing problem
Aminurafiuddin Zulkifli, Nor Azlina Ab. Aziz, Nor Hidayati Abdul Aziz
(Multimedia University, Malaysia)
Zuwairie Ibrahim(Universiti Malaysia, Malaysia)
Norrima Mokhtar(University of Malaya, Malaysia)
The problem of sequencing and scheduling arriving aircraft landing is commonly
known as aircraft landing problem (ALP). This problem, due to various constraints
such as the number of arriving aircrafts, the number of runways, the mode of
runway operation, the type of arriving aircrafts, the minimum separation between
each arriving aircraft, and the weather condition, is considered to be a NP-hardproblem. Therefore, it is almost impossible to compute every possible solution andcomputational intelligence methods had been adopted to solve ALP. In this paper,
we review the computational intelligence techniques used in ALP. The maintechniques include the evolutionary algorithms namely; genetic algorithm, genetic
programming, scatter search and bionomic algorithm, the swarm intelligencealgorithms like particle swarm optimization and ant colony optimization and also
other methods such as the constrained position shifting and dynamic programming.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-94-

Figure 1. Cost variationin time window during
flightThe 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS6 Virtual reality (5)GS6-1 Interactive musical editing system to support human errors and offer
personal preferences for an automatic piano -Inferring performance expression byconsidering change of pitchMasahiro Ushio, Eiji Hayashi (Kyushu Institute of Technology, Japan)
Electronic musical instruments’ sound quality and ambience are inferior to realmusical instruments. Therefore, we developed automatic piano by using grandpiano. Pre-edit is needed to play music in the manner of a live pianist. In the caseof piano music, there are often 1000 or more notes in the score of even a short pieceof music, requiring that an editor spend a huge amount of time to accurately
simulate the emotionally expressive performance of a highly skilled pianist.
Therefore, we have developed an interactive musical editing system that utilizes adatabase to edit music more efficiently. This paper, we made a hypothesis that the
pianist's performance is characterized by the pitch change, and we introduced the
pianist features which considered the pitch change to further improve the efficiency.

GS6-2 Analysis of Malaysian Facial Expressions for Designing Virtual Agents
Halimahtun Khalid (Damai Sciences, Malaysia), Liew Wei Shiung(University of Malaya, Malaysia)
The perception of emotion based on facial expressions is culturally mediated. East
Asians tend to focus on the central region around the nose and eyes, while Western
Caucasians observe the eyebrows and mouth regions. Therefore, designing a virtualavatar with realistic facial expressions to convey emotions should consider thedemographics of the users who would be interacting with the avatar. We conducted
an online study to determine the facial measurements for specific emotions asperceived by Malaysians. Twenty-four facial expressions were shown to 37
subjects and they were instructed to select which emotion label matched each
expression. The expressions were then decomposed into facial action unit
measurements using OpenFace. Regression analysis was used to determine the
combination of measurements corresponding to each emotion label. The facial
expressions were then used to design the virtual agents in an experiment on humanavatar interaction where 48 subjects evaluated the virtual agents on trust.

GS6-3 Development of VR system to enhance understanding process of robot
mechanisms
Alexei Lushnikov, Vlada Kugurakova, Timur Satdarov, Artur Nizamutdinov(Kazan Federal University, Russia)
For engineers and technicians in training honing one's skills in working withmachines are essential. Very unfortunately, access to facilities required for this ispretty much may not always be provided: such facilities might be in short supply,or the time one is allowed to use them might be restricted, or they might be simply
not present in the vicinity. The solution to this problem that we propose is the useof VR simulation of tasks that would be normally carried out at the specialized
facilities, such as assembling-disassembling, maintenance, and use. To provide themost realistic and immersive experience, the simulation would be fitted with haptic
feedback. This study describes the ongoing project on design and implementationof such VR system.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-95-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS6-4 Towards the immersive VR: measuring and assessing realism of user
experience
Vlada Kugurakova, Murad Khafizov, Alexander Elizarov, Aleksei Lushnikov,
Artur Nizamutdinov (Kazan Federal University, Russia)
The range of uses for Virtual Reality Continuum grows more diverse, with potential
fields of application spanning from research and education to leisure and work,
which increases demand for a method which can allow to precisely and reliably
measure VR effectiveness. How can we determine the quality of VR? How dodifferent people respond to its varying implementations? And could it be true thatdifferent social groups interact with VR differently? In this paper, we propose a
new approach to this emerging problem, which involves applying user interface(UI) metrics to VR experience. We review existing ways to increase VR immersion
and outline a series of possible experiments to study and, hopefully, find acorrelation between immersiveness and VR user interface effectiveness.

GS6-5 Lessons on the Reality-Gap: Iterations between Virtual and Real Robots
Andre Rosendo (ShanghaiTech University, China),
Charlie Houseago, Fumiya Iida (Cambridge University, UK)
Due to abstractions and approximations between the virtual and real world, thetransfer of knowledge from virtual (simulations) to real (robots) world is
problematic. As physical conditions are prone to unknown and stochastic noise
sources, the predictability reduces. In here we use data experiments from 100 realrobots to tune the parameters of a simulation, and later used this tuned simulator to
improve the design of the previous robots and find the optimum robot. We compare
the simulated and real world behavior of this robot, and discuss our results. In thispaper we address the Reality Gap by tuning a Bullet physics simulator with results
extracted from real experiments with robots. Parameters defining the interaction
between the robot and the floor (friction and restitution) are tuned to minimize theerror between the simulated and actual behavior, and the simulated trajectories of
100 agents are compared to the trajectories of the real-world counterpart. With the
newly found set of physical parameters, the simulation is then used to predict,through a Random Search approach, a robot capable of outperforming the previous
ones. This new robot is built and we observe that although differences exist between
the simulated and actual behavior, the final robot’s trajectory strongly resembles
the one predicted by the simulation. Next steps should consider stochastic
differences between experiments to improve the predictability of the system.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-96-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS7 Intelligent control (3)GS7-1 Fractional Order Sliding Mode Control Applying on the HIV Infection
SystemThunyaseth Sethaput1, Arsit Boonyaprapasorn2( Thammasat University, 2Chulachomklao Royal Military Academy, Thailand)
1

The mathematical model of HIV dynamical system explains the interaction
between immune system and virus. It represents the relationship among
population size of uninfected CD4+ cells, infected CD4+ cell, and virus. The aimof the treatment is to drive the amount of uninfected CD4+ cells to the desired
level and the amount of both infected CD4+ cells and virus particles approach tozero as time increased. The main objective of this study is to apply the fractional
order sliding mode control (FOSMC) method to regulate HIV infection. The
performance of the control method was investigated via simulation. According to
the simulation of the controlled HIV system, the state variables approach to thedesired values. Thus, it is feasible to apply the FOSMC method to define thetreatment of the HIV infection system.

.

GS7-2 Skill-based Job Rotation Scheduling for Occupational Noise Exposure
Control
Pavinee Rerkjirattikarn, Chatdanai Kaorapapaong, Sun Olapiriyakul(Thammasat University, Thailand)
Job rotation is an inexpensive and flexible administrative control measure, used toprevent workers from being exposed to harmful levels of occupational hazards.
Despite its advantages, job rotation can affect process continuity and workers’
control over the timing of their work hours, resulting in reduced productivity andjob satisfaction.
This paper proposes a job rotation methodology for
manufacturing systems with independent workstations. An integer-programming
model is formulated, to determine workforce schedules capable of meetingcustomer demands and skill requirements for tasks, while simultaneously
minimizing operational cost and reducing noise exposure levels of workers to a safe
level. The overtime assignment is considered to ensure the fairness andappropriateness of the work schedules. The cost effectiveness of workforce
schedules is evaluated under different overtime policies and production capacity
levels. The results of this study will help decision makers to select the best
workforce schedule, based on the desired production levels and overtime policies.

.

GS7-3 A Training Method for the Speech Controlled Environmental Control System
Based on Candidate Word Discriminations
Taro Shibanoki, Masaki Watanabe (Ibaraki University, Japan), Go Nakamura, Takaki Chin
(Hyogo Rehabilitation Center, Japan), Toshio Tsuji (Hiroshima University)©ICAROB 2018 ALife Robotics Corp. Ltd.

-97-

Commandgeneration…

Intention
estimation

Voice SpeechControl
Infraredsignal recognition command signals
results
Featureextraction

This paper proposes a concept of a training system for the speech controlledenvironmental control system: Bio-Remote based on candidate word
discriminations. The proposed system can provide three-types of voice signal
training: (1) volume, (2) timing and (3) candidate word which are important for
accurate speech recognition based on false recognition results. During the training,such three kinds of features are extracted from measured voice signals and visually
and auditory fed back to the user in real time. This allows the user to train speech
abilities even if false recognition results are extracted because of slurred speech
The efficacy of the proposed system is demonstrated through training experiments
for slurred speech conducted with healthy participants. The results show that the
proposed training system is capable for train the speech abilities.

Domestic
appliances

Operator
VolumeTimingCandidate word

(L)

Microphone

(L)

M11
M21

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS8 RoboticsII (5)GS8-1 Design System of Cell Type Assembly Machine with Dual Arms Robot by GA
Keita Honda, Hidehiko Yamamoto and Takayoshi Yamada (Gifu University, Japan)

The purpose of this research is to develop a system named DELUGA which
automatically decides to place a lot of assembled parts, jigs and robot hands of acell type Assembly Machine by genetic algorithm (GA). The robot of our research
adopts a dual arms robot. DELUGA consists of two modules, the conditions
module and GA module. The conditions module reads work data for robot, andsets various parameters required for GA.GA module decides the efficientarrangement place of parts, for example jigs and robot hands, by GA, and outputs
the acquired arrangement image. Because the left and right robot arms
simultaneously carry out assembly works, DELUGA adopts the concept of
waiting time of each robot arm to prevent arms interference. We applied
DELUGA to assembly machine of an in-wheel motor for electric vehicles. The
application resulted that the production efficiency increased. It is ascertained that
DELUGA is useful.

GS8-2 Spherical Mobile Robot Driven by Biorthogonal Omnidirectional Wheels
Liu Wei, Ma Shuanglong, Duan Lunqin, Yu Jiangtao (Beijing Jiaotong University, China)
This paper presents a new spherical mobile robot driven by omnidirectional wheels.
The structure is that there are two independent driving omnidirectional wheels
located at the ends of the inside vertical diameter of the sphere shell, and the twowheel axes are in orthogonal position. Its principle is, that the omnidirectional
wheels can roll on the inner surface of the spherical shell by the friction force, thendrive the spherical shell rolling. The lower end omnidirectional wheel is
responsible for walking drive, and the upper end one for steering drive.

.

GS8-3 Coordinated behaviour with a Pepper Humanoid robot to estimate thedistance of other robot using Inverse Perspective Mapping
M. Hassan Tanveer, Antonio Sgorbissa, Carmine T. Recchiuto(University of Genova, Italy)
This article discusses a strategy for real time distance determination applied to ahumanoid robot (Pepper, Softbank Robotics) that is requested to collaborate withother robots. However, using the monocular camera embedded on the robot head,
it is difficult to determine distance to objects in front of the robot with any degreeof certainty. The reason is non linear relationship between the height allocation ofcamera and its distance to an object in a camera frame. So, the proposed method isbased on the extraction of accurate bird’s-eye view images, by applying Surface
Substraction and Inverse Perspective Mapping (IPM) to the images captured with
the Pepper head camera. The algorithm is implemented in the Python language
using the OpenCV libraries©ICAROB 2018 ALife Robotics Corp. Ltd.

-98-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS8-4 Control Techniques of Quadrotor Uavs: A Concise Study
1

Syed Faiz Ahmed, 1Athat Ali, 1M. Kamran Joyo, 1Khusairy Abd Kader, 2Hazry Desa,
Sheroz Khan(1Universiti Kuala Lumpur, British Malaysian Institute, Malaysia, 2Universiti
Malaysia Perlis, Malaysia, 3International Islamic University Malaysia, Malaysia)

3

Quadrotors, the four wing UAV has drawn prominent attention amongst the
researchers. The simple nonlinear structure design requires a robust control toensure the stability during the flight. The noisy and environment with unexpecteddisturbances may cause issues such as drift in position. In order to tackle the severe
conditions a robust control is required. This paper provides an overview of theavailable quadrotor models and the control techniques with their application inquadrotor control issues.

GS8-5 GIS Based Hydrological Model for River Water Level Detection & Flood
Prediction featuring morphological operations.
Sarmad Zafar, H.M.SohaibAzhar, Aqeel Tahir(Mohammad Ali Jinnah University, Pakistan)
Mostly river water level and flood forecasting methods are based on
gauging stations measurements at discrete locations, which limits theircapability to provide accurate and timely data over large extent, also limited or nodata available on remote locations. So here we present an idea to use high resolution
satellite images for real time mapping of river water level. In this project, wedeveloped a Web based GIS system for mapping river water level, early warningand mapping for flood disasters. To improve flood forecasting/warning, wedeveloped a decision support system (DSS) for flood monitoring and predictionthat integrates GIS, satellite image processing and hydrological modeling. We
present the methodology for data integration, floodplain delineation, and online
map interfaces. Our Web based GIS system can dynamically display observed andpredicted water levels for decision makers and the general public. The users canaccess a Web-based GIS system which models current flood events and displays
satellite imagery and 3D visualization integrated with the flood plain area. The
output from the hydrological modeling will be used for flooding prediction for the
next 1 day to 2 days (24 and 48 hours) along the lower Indus River. In this stage
river water level analysis has been achieved, work on the hydrological modeling isin progress to acquire river water stage & flood level and the prediction.

GS9 Poster (11)GS9-1 Analysis of onboard sensor-based odometry for a quadrotor UAV in outdoor
environment
Aidar Gabdullin, Grigory Shvedov, Mikhail Ivanou, Ilya Afanasyev(Innopolis University, Russia)
Konstantin Yakovlev (Russian Academy of Science, Russia)
This article introduces a comparative analysis of a quadrotor UAV trajectories
evaluated by onboard sensors (camera, IMU and GPS) and ROS-based monocularvisual odometry. Parrot Bebop drone was launched outdoor in autonomous mode
according to teleoperated closed-loop trajectory along an a-priori known
perimeter of a square work area (measured by a tape measure as a ground truth)
and recorded sensors telemetry data for offline processing. To verify UAV
odometry trajectory, we used Bebop’s firmware with integrated visual-inertial
velocity estimation, the ground truth and an external observer.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-99-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS9-2 Memristive neuron integration in digital robotic embodiment
Max Talanov, Evgenii Zykov, Yuriy Gerasimov, Evgeni Magid, Aleksander Elizarov(Kazan Federal University, Russia)
Victor Erokhin(IItalian National Council of Research, Parma, Italy)
In the current paper, we introduce a high-level approach for an integration of aneuromorphic meistive neuron in a real-time operating robotic system. The
memristive neuron schematic, which we had presented in our earlier works, iscapable of inhibitory and excitatory learning (eSTDP, iSTDP) as well as
modulation via dopamine input. We discuss a possibility of integration of theanalog memristive neuron into a digital robotic embodiment and present block
diagram of an adapter that includes pseudo-neuronal encoder and decoder.

GS9-3 Russian mobile robot Servosila Engineer: designing an optimal integration ofan extra laser range finder for SLAM purposes
Neil Alishev, Yuriy Gerasimov, Roman Lavrenov (Kazan Federal University, Russia)
In our current research we determine an optimal design for integration of Hokuyo
UTM-30LX-EW LRF into control system of Russian mobile robot Servosila
Engineer. We designed and constructed a special static stand with an option to
select an inclination of a scanning beam toward the surface of an environment.
RBPF SLAM algorithm was tested to perform localization and mapping.
Experiments were conducted in order to determine the best configuration of the
LRF position and RBPF SLAM algorithm parameters.

GS9-4 Toward automated open wound suturing using haptic feedback: detecting
wounds and planning the suture
Artur Sagitov¹, Hongbing Li², Evgeni Magid¹(¹Kazan Federal University, Russia)(²Shanghai Jiao Tong University, China)
One of the key disadvantages of robotic surgery as of today is the lack of haptic
feedback. While in traditional surgery surgeons using their haptic senses in alltasks most medical robot lack this ability. Using robots with haptic feedback havethe potential of reducing the time spend by surgeon on such tedious subtasks assuturing helps reducing surgeon fatigue. Also decreasing the manual input of thesurgeon enables remote surgery even using long communication links with big
latency. In this paper we present a framework of wound detection and sutureplanning. We plan to implement and test our algorithms using KUKA iiwa
manipulator.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-100-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS9-5 Establishing Effective Teaching for Robotics: a comparison study of Bachelor
students in Introduction to Robotics course
Tatyana Tsoy, Leysan Sabirova, Mikhail Abramskiy, Evgeni Magid(Kazan Federal University, Russia)
Global demand in robotics specialists arose necessity of establishing educational
programs in robotics, which sets new challenges for modern educational system.
In 2017 Kazan Federal University launched master’s program in Intelligent
Robotics to produce specialists considering core engineering competences andsuch important skills as self-motivation, critical thinking etc. This paper presents
results of continuous survey among undergraduate students of Introduction to
Robotics course that helps understanding dynamics of student’s motivation to
study robotics, attitude toward education methods to improve teaching strategies

GS9-6 Development of a heterogeneous aerial swarm control framework for forest
management
Yuriy Gerasimov, Artur Sagitov, Evgeni Magid (Kazan Federal University, Russia)
As the prevalence of UAVs is increasing, they are becoming more accessible forwider applications. Our interest is in application of UAVs for forest management
challenges including health and safety monitoring and commercial exploitation
in a sustainable manner. We propose a swarm control framework for managing agroup of UAVs for aforementioned tasks, including survey of tree health with
infrared cameras and chemical sensors, detecting potential risky situations ofillegal logging, smoke and fires, and estimating potential volume measurements.
The proposed framework manages planning flight trajectories, sensor fusion andcollaborative mapping. On the next stage, we plan to simulate the framework in
ROS/Gazebo environment, and further to implement a pilot project with a groupof DJI Phantom quadrotors and a large-size fixed-wing UAV.

GS9-7 Implementation of ROS package for simultaneous video streaming fromseveral different cameras
Ramil Safin, Roman Lavrenov (Kazan Federal University, Russia)
Real time video stream capturing and processing is important for a variety of tasks
in robotics. We created ROS package that captures concurrent video stream from
4 different cameras of Russian mobile robot “Servosila” Engineer. V4L2 API wasused to configure video devices and to retrieve raw data from cameras. Memory
mapping approach of mapping device buffers increased overall performance byeliminating redundant memory copies. We demonstrate the comparison of our
new package and OpenCV based package.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-101-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS9-8 Consumers Preferences and Purchase Intention on New Taste of Salted Duck
Eggs
Shang-Hui Li (Far East University, Taiwan)
Egg is an indispensable food in people's daily life. Egg is rich in protein, easily
digested by human body and metabolism. In addition to direct consumption,
people created many different processed egg products by the characteristics of
eggs. The traditional method of making salted duck eggs is to place the eggs insalt water or coated in laterite without any additional chemical composition.
Except the eggs, this study joined other natural ingredients To produce eight
different flavors of salted duck eggs. In the formal test, 70 convenience sampling
were taken from Tainan area for sensory evaluation, 63 valid questionnaires,
SPSS18.0 was used as the statistical analysis. Sensory evaluation included thecolor, oil, smell, salty and taste of egg yolk and the salty, taste of protein.
Moreover, with the overall acceptance, purchase intention and willingness to buy
again etc. The study results showed that cinnamon red salted duck eggs and fourherb salted duck eggs are the highest acceptance to consumers, followed by thesalted citronella duck eggs. However, four-herb salted duck eggs dipped by thesalted water are not favored by consumers.

GS9-9 The Study of Green Food Image, Satisfaction and Loyalty through the
Perspective of Elaboration Likelihood Model
Shu-Fang Hsu (Far East University, Taiwan)

This study aims to explore the relevance of consumer satisfaction and loyalty togreen food image through the perspective of Elaboration Likelihood Model. The
results show that the positive impact of green food image; and will indirectly
affect the satisfaction of re-purchase intention; satisfaction positive impact
loyalty. We hope that the results of this study can provide relevant business
management strategies and suggestions, the green food consumption theory andpractice can be dedicated.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-102-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS9-10 Self-Generated Dataset for Category and Pose Estimation of Deformable
Object for Manipulation by Robot
Yew Cheong Hou, Khairul Salleh Mohamed Sahari (Universiti Tenaga Nasional, Malaysia)
This work considers the problem of garment handling by a general householdrobot that focuses on the task of classification and pose estimation of a hanging
garment in unfolding procedure. Classification and pose estimation of deformableobjects such as garment are considered a challenging problem in autonomous
robotic manipulation because these objects are in different sizes and can be
deformed into different poses when manipulating them. Hence, we propose a selfgenerated synthetic dataset for classifying the category and estimating the poseof garment using a single manipulator. We present an approach to this problem
by first constructing a garment mesh model into a piece of garment that crudely
spread-out on the flat platform using particle based modeling and then theparameters such as landmarks and robotic grasping points can be estimated fromthe garment mesh model. Later, the spread-out garment is picked up by a single
robotic manipulator and the 2D garment mesh model is simulated in 3D virtualenvironment. A dataset of hanging garment can be generated by capturing thedepth images of real garment at the robotic platform and also the images ofgarment mesh model from offline simulation respectively. The synthetic dataset
collected from simulation shown the approach performed well and applicable ona different of similar garment. Thus, the category and pose recognition of thegarment can be further developed.

GS9-11 Design and Development of Three Arms Transmission Line Inspection
Robot
Muhammad Fairuz Abdul Jalal, Khairul Salleh Mohamed Sahari, Ho Ming Fei, Justin Chan
Tuck Leong (Universiti Tenaga Nasional, Malaysia)
The high-voltage transmission line had been used primarily for power distribution
from power plant or power station to the end users. However, the transmission
line is highly prone to damage due to exposure to various thermal - mechanicalloading and material degradation. Therefore, periodical inspection ontransmission line after prolonged service is needed to prevent any failure before
it happens. In this paper, we present a new design of three arms inspection robot
for transmission lines. The robot is able to transverse along the line and bypass
the in-line obstacles namely the anti-vibration hammers, spacer, strain clamps andothers. The design of the inspection robot in term of the robot design andconfiguration with slotted cam at each arm is presented. The detailed analysis via
simulation with respect to the robot stability; kinematic and movement analysis;
and power consumption during operation is executed to make sure the proposed
design able to do the inspection without any unexpected difficulties. Later, thelab testing on the developed prototype is done for feasibility study and validation.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-103-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS10 Others (4)GS10-1 A review on fundamental advancements of black hole algorithm
Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari(Universiti Malaysia Pahang, Malaysia)
Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz(Multimedia University, Malaysia)
Tasiransurini Ab Rahman(Universiti Tun Hussein Onn Malaysia, Malaysia)
Asrul Adam, Zulkifli Md Yusof(Universiti Malaysia Pahang, Malaysia)
Mohd Ibrahim Shapiai(Malaysia Japan International Institute of Technology, Universiti Teknologi Malaysia, Malaysia)
Norrima Mokhtar(University of Malaya, Malaysia)
In recent years, there is a growing interest in the design and development of natureinspired optimization algorithms. One of the algorithms is black hole algorithm(BHA), which is inspired by the black hole in general relativity and cosmology.
This paper reports a review of the BHA. This review emphasizes on thefundamental advancements of BHA. A brief research gap is presented at the end of
this paper.

Event horizon
for the blackholeBlack hole
Agent iRBH
Disappear into theblack hole.
(by re-initialization)

Fig. 1. Illustration of a black holewith its event horizon.

GS10-2 A survey on applications of black hole algorithm
Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari, Asrul Adam, Zulkifli Md Yusof(Universiti Malaysia Pahang, Malaysia)
Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz(Multimedia University, Malaysia)
Tasiransurini Ab Rahman(Universiti Tun Hussein Onn Malaysia, Malaysia)
Mohd Ibrahim Shapiai(Malaysia Japan International Institute of Technology, Universiti Teknologi Malaysia, Malaysia)
Norrima Mokhtar(University of Malaya, Malaysia)
P

In recent years, there is a growing interest in the design and development of natureinspired optimization algorithms. One of the algorithms is black hole algorithm(BHA), which is inspired by the black hole in general relativity and cosmology.
This paper reports a review of the black hole algorithm (BHA). This survey
emphasizes on the applications of BHA.

Initialization

Fitness evaluation

Update black hole, calculate
event horizon

Update position based on
pseudo code 1

No
No
Stopping condition

Yes
Yes
Return best solution, XBH

Fig. 1. Flowchart of BHA.

GS10-3 Black hole white hole algorithm with local search
Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari, Asrul Adam, Zulkifli Md Yusof(Universiti Malaysia Pahang, Malaysia)
Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz(Multimedia University, Malaysia)
Tasiransurini Ab Rahman(Universiti Tun Hussein Onn Malaysia, Malaysia)
Norrima Mokhtar(University of Malaya, Malaysia)
PBlack hole algorithm (BHA) is an optimization algorithm inspired by the black holediscovery in relativity theory. Recently, white hole operator, which is based on the
opposite of black hole, has been introduced in BHA. In this paper, a local is addedin the BHA with white hole operator.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-104-

Initialization

Fitness evaluation

Update black hole, calculate
event horizon

Update position based on
pseudo code 1

No
No
Stopping condition

Yes
Yes
Return best solution, XBH

Fig. 1. Flowchart of BHA.

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS10-4 Tele-Operation of a Legged Robot by a Virtual Marionette System- First report: The first prototype and the usefulness of the reaching taskNoritaka Sato, Yasuhiko Sawai, Ryo Asami, Makoto Kitani, Yoshifumi Morita(Nagoya Institute of Technology, Japan),
Tomofumi Fujiwara, Takahiro Endo, Fumitoshi Matsuno (Kyoto University, Japan)
I8n general, a gamepad and images from a camera mounted on a robot are used forthe teleoperation of a rescue robot. However, teleoperation is difficult for
inexperienced users because the relationship between the user input and robot
motion is not always intuitive. Therefore, we have developed an interface in which
the operator operates a robot as if touching it directly in a virtual 3D space. In ourprevious research, we developed a virtual direct teleoperation system for a tracked
rescue robot. In this study, we adapt our system to inputting target positions of ahand or a foot of a legged rescue robot. We verify the effectiveness of the proposedsystem by comparison with the conventional method which is used by the teamparticipating in the DARPA Robotics Challenge.

©ICAROB 2018 ALife Robotics Corp. Ltd.

-105-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

AUTHORS INDEX
Notation of session name
PS: Plenary Session IS: Invited Session, OS: Organized Session, GS: General
Session,Note: 33/90 = (page no. in Technical Paper Index) / (page no. in Abstracts)[A]
Ab. Aziz

Ab Rahman

Alishev
Nor Azlina

Tasiransurini

Neil

Amarasinghe

GS9-3

29/100OS14-3

33/70

GS5-3

34/94

GS10-140/104OS14-433/70

GS10-2

40/104OS14-5

33/71

GS10-3

40/104OS14-6

33/71

GS10-140/104

AmpantUdomlux

GS3-2

32/91

GS10-2

40/104An

Jongwoo

OS2-2

32/46

GS10-3

40/104

An

LingranOS16-225/76

Abd Kader

Khusairy

GS8-4

19/99

OS16-5

25/76

Abdul Aziz

Nor Hidayati

GS5-3

34/94

OS16-7

26/77

GS10-140/104

OS6-1

38/53

GS10-2

40/104

OS6-2

38/53

GS10-3

40/104

Andreev

Andrey

OS7-4

21/55

GS9-1130/103

Andreychuk

Anton

OS7-1

21/54

Anshi

Ryohei

GS2-2

27/89

Abdul Jalal

MuhammadAndo

FairuzKazuakiAbe

Akinori

OS17-4

36/79

Aoki

Kenji

OS1-1

39/44

Abramskiy

Mikhail

GS9-5

29/101Arai

Takafumi

OS21-2

18/85

AburadaKentaroOS12-3

31/67

Asada

Taro

OS13-1

37/67

OS12-431/67

OS13-2

37/68

GS10-140/104OS13-3

38/68

GS10-2

40/104

Asai

Fumio

GS3-5

32/92

GS10-3

40/104Asami

Ryo

GS10-440/105AshidaYoichiro

OS3-3

22/48

OS3-6

22/49

H.M.Sohaib

GS8-5

19/99

JaehueOS22-331/87

OS14-3

33/70

Adam

Asrul

Afanasyev

Ilya

GS9-1

29/99

Ahn Ahmed

Syed Faiz

GS8-4

19/99

Ahn

JonghyunOS20-2

34/83

AiharaKazuyuki

OS9-1

24/62

OS9-6

24/64

Azhar

[B]

Alemán Carreón

Elisa Claire

OS5-3

23/52

Bae

Ali

Athat

GS8-4

19/99

Basnayake©ICAROB 2018 ALife Robotics Corp. Ltd.

-106-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

OS14-6

33/71

OS16-1

25/75

Bian

Ce

OS16-125/75

OS16-2

25/76

Bian

Ce

OS16-3

25/76

OS16-3

25/76

OS16-425/76

OS16-425/76

OS16-5

25/76

OS16-5

25/76

OS16-6

25/77

OS16-6

25/77

OS16-926/77

OS16-7

26/77

OS7-1

21/54

OS16-8

26/77

OS7-3

21/55

OS16-926/77

OS14-5

33/71

Bokovoy

AndreyBoonyaprapasorn

Arsit

GS7-1

34/97

Dayananda

Bruyant

Alexandre

GS3-1

32/91

DeguchiChikashi

OS1-3

39/44

Desa

Hazry

GS8-4

19/99

OS14-433/70

Lunqin

GS8-2

19/98

[C]

De Silva

Chaicherdkiat

Poommitol

GS1-2

30/88

Chan

Tuck Leong,GS9-1130/103

Duan

Justin[E]

Chang

Cheng-Jui

OS8-11

20/62

Egunov

Vitaly

OS7-2

21/54

Chang

Shengbiao

OS16-1

25/75

Elizarov

Alexander

GS6-4

37/96

OS16-4

25/76

GS9-2

29/100OS16-9

26/77

Endo

Masaki

OS5-4

23/52

OS14-6

33/71

Endo

Takahiro

GS10-440/105

Erokhin

Victor

GS9-2

29/100

Ho Ming

GS9-1130/103OS14-6

33/71

Chaturanga
Chen

Chia-Hsiu

OS8-7

20/60

Chen

Feng

GS4-2

26/93

Chen

Xiaoyan

OS15-825/74

OS15-9

25/74

Fei

[F]

ChikushiShota

OS21-3

19/85

Fernando

Chin

Takaki

GS7-3

34/97

Fu

Dongmei

OS18-4

30/81

Choi

Jiwook

OS2-2

32/46

Fu

Ziyi

OS18-130/80

ChuangKuang-Wei

OS8-6

20/60

OS19-1

22/81

Chumkamon

Sakmongkon

GS2-2

27/89

GS3-4

32/92

ColletPierre

GS3-1

32/91

Fujinaga

TakuyaOS21-519/86

CuiMengchenOS18-430/81

Fujita

Kohki

GS4-4

26/93

Fujiwara

Tomofumi

GS10-440/105

Fukushima

HirokiOS17-1

36/78

Fukushima

Mihoko

OS1-3

39/44

[D]
Dai

FengzhiOS15-925/74©ICAROB 2018 ALife Robotics Corp. Ltd.

-107-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

[G]

GS2-4

27/90

GS2-5

27/90

GS6-1

37/95

G. Helander

M.

IS-6

28/43

Hayashi

Yuki

OS17-637/79

Gabdullin

Aidar

GS9-1

29/99

He

Chang-Jhu

OS8-1020/61

Gao

Caixia

OS18-1

30/80

Henmi

Tomohiro

OS3-7

23/49

GautamAshish

OS9-5

24/63

Hiraoka

Toru

OS5-1

23/51

Ge

Yong

OS1-5

39/45

OS5-2

23/52

Gerasimov

Yuriy

GS9-2

29/100

OS5-3

23/52

GS9-3

29/100

OS5-1

23/51

GS9-6

29/101

OS5-3

23/52

GS2-1

27/89

OS5-4

23/52

GS2-6

27/90

Honda

Keita

GS8-1

19/98

Hori

Masamichi

OS1-1

39/44

GopuraHirotaMasaharu

Gorobtsov

Alexander

OS7-4

21/55

Guo

Jr Hung

OS8-2

20/58

OS1-3

39/44

OS8-4

20/59

OS1-4

39/45

OS8-6

20/60

OS6-1

38/53

OS15-4

24/73

OS6-2

38/53

OS15-5

24/73

OS6-3

38/53

OS6-4

38/54

Guo

Zhiqiang

Horikawa

[H]

Yo

Hou

Yew Cheong

GS9-1030/103Han

FengzeOS15-8

25/74

Hou

Yong

OS16-9

26/77

Han

Sumin

OS19-2

22/82

Houseago

Charlie

GS6-5

37/96

HaradaKazumasa

OS20-1

34/83

Hsia

Kuo-Hsien

OS8-4

20/59

HaradaKensuke

OS10-1

33/64

OS8-5

20/59

OS10-2

33/64

Hsu

Shu-Fang

GS9-9

29/102

OS6-1

38/53

Hsu

Wen-Ting

OS8-9

20/61

OS6-2

38/53

Hu

Qiongqiong

OS1-5

39/45

OS6-3

38/53

Hung

Chung-Wen

OS8-1

20/57

OS6-4

38/54

OS8-9

20/61

GS5-3

34/94

Hattori

TetsuoHayakawa

Takuya

GS5-1

34/94

Hayashi

Eiji

GS1-2

30/88

GS1-3

30/88

GS2-2

27/89

GS10-140/104

GS2-3

27/89

GS10-2

40/104[I]
Ibrahim©ICAROB 2018 ALife Robotics Corp. Ltd.

-108-

Zuwairie

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018
GS10-3

40/104Jeon

Hyun Cheol

OS22-1

31/86

Jia

Hongyan

OS15-424/73

OS15-5

24/73

OS18-3

30/81

Iida

Fumiya

GS6-5

37/96

Ikeda

Satoshi

OS1-1

39/44

OS1-2

39/44

OS1-3

39/44

OS19-3

22/82

OS1-4

39/45

OS19-5

22/83

OS8-1

20/57

OS8-9

20/61

OS16-2

25/76

IiboshiAtsushiJia

Yingmin

Imai

ShinichiOS11-2

39/65

Jiang

Jhen-Gu

Imai

Yoshiro

OS6-1

38/53

OS6-2

38/53

OS6-3

38/53

OS16-5

25/76

OS6-4

38/54

OS16-7

26/77

Jin

Xia

Indelman

Vadim

OS7-5

21/56

Jitkongchuen

Duangjai

GS3-2

32/91

InukaiToshihiro

OS15-1125/75

Jitviriya

Wisanu

GS1-2

30/88

Iriyama

Yosuke

OS1-4

39/45

GS2-2

27/89

Ishii

Kazuo

OS20-2

34/83

JohnsonJeffrey

PS-1

18/41

OS20-3

35/84

Joyo

M. Kamran

GS8-4

19/99

OS20-4

35/84

Jung

Min Gyu

OS2-1

32/45

OS20-5

35/84

OS21-319/85[K]

OS21-4

19/86

Kachalov

Andrey

OS7-2

21/54

OS21-5

19/86

Kai

Yuki

OS1-4

39/45

Ishikawa

Hiroshi

OS5-4

23/52

Kamei

Keiji

OS21-2

18/85

Ito

Takao

OS1-1

39/44

Kang

Hosun

OS2-2

32/46

OS1-2

39/44

OS2-3

32/46

OS5-1

23/51

OS16-1

25/75

OS5-3

23/52

OS16-4

25/76

Kang

Qijia

Ito

TakuyaOS17-5

36/79

Kaorapapaong

Chatdanai

GS7-2

34/97

Ito

Tsutomu

OS1-1

39/44

Katayama

TetsuroOS12-1

31/66

OS1-2

39/44

OS12-3

31/67

GS9-1

29/99

OS12-4

31/67

IvanouMikhail[J]
Jayawardena

GS2-1

27/89

Kato

Ryota

OS13-1

37/67

KawadaKazuo

OS11-3

39/65

Kawakami

Yusuke

OS6-1

38/53

Jeannin-Girardon

Anne

GS3-1

32/91

OS6-2

38/53

Jee

Gyu-inOS22-2

31/87

OS6-3

38/53

©ICAROB 2018 ALife Robotics Corp. Ltd.

-109-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

KawanoHiromichi

OS6-4

38/54

GS5-2

34/94

OS6-3

38/53

OS7-6

21/56

OS6-4

38/54

GS6-3

37/95

GS6-4

37/96

OS5-1

23/51

OS5-2

23/52

OS5-3

23/52

GS4-3

26/93

GS4-4

26/93

Ryo

OS3-2

22/48

Khafizov

Murad

GS6-4

37/96

KhalidHalimahtun

IS-6

28/43

GS6-2

37/95

Khan

Sheroz

GS8-4

19/99

Khoyratee

Farad

OS9-3

24/63

Kim

Ki-seo

OS2-6

32/47

Kim

Dohyun

OS2-4

32/46

Kim

Heeje

OS2-4

32/46

OS2-5

32/47

Kugurakova

KumanoKuremoto

Kurokawa

Vlada

MinoruTakashi[L]

Kim

KyuwonOS22-2

31/87

Kim

Min-soo

OS2-5

32/47

Lai

Li-Chun

OS8-3

20/58

KimuraKenji

OS21-319/85

Lavrenov

Roman

OS7-5

21/56

Kinoshita

Kento

OS3-8

23/50

OS7-7

21/56

Kinoshita

Takuya

OS3-3

22/48

GS9-3

29/100Kita

YoshihiroOS12-3

31/67

GS9-7

29/101OS12-431/67

Lee

ByoungjinOS22-331/87

LeeJangmyung

OS2-2

32/46

Kitani

MakotoGS10-440/105Ko

Chia-Nan

OS8-3

20/58

OS2-3

32/46

Kobayashi

Jun

GS5-1

34/94

OS2-6

32/47

Kobayashi

Kyoichiro

GS4-3

26/93

Lee

Min Cheol

OS2-1

32/45

KodamaYukari

OS1-3

39/44

Lee

Nahyun

OS2-3

32/46

OS1-4

39/45

Leleu

Timothée

OS9-6

24/64

Levi

Timothée

OS9-1

24/62

Kogi

DaichiOS13-2

37/68

Kohno

Takashi

OS9-1

24/62

OS9-3

24/63

OS9-2

24/62

OS9-6

24/64

OS9-4

24/63

Li

BingheOS21-5

19/86

OS9-5

24/63

Li

Hongbing

OS7-8

21/57

OS9-6

24/64

GS9-4

29/100

Konishi

Yasuo

OS3-2

22/48

Li

Ji-Hua

OS8-2

20/58

KoyamaYoshihide

OS6-3

38/53

Li

Jung-Shian

OS8-7

20/60

OS6-4

38/54

OS8-8

20/61

GS3-3

32/91

OS8-11

20/62

Kubo

Masao

©ICAROB 2018 ALife Robotics Corp. Ltd.

-110-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Li

Meili

OS16-325/76

LiShang-Hui

GS9-8

Li

Shudong

Li

Magid

Evgeni

OS7-5

21/56

29/102

OS7-8

21/57

OS15-8

25/74

GS9-2

29/100Yang

OS15-10

25/75

GS9-4

29/100Li

Yi

OS19-5

22/83

GS9-5

29/101

Li

Yiyang

OS1-5

39/45

GS9-6

29/101

Lin

Jun-Nong

OS8-5

20/59

MaigumaMizukiOS14-1

33/69

Lin

XiaokunOS16-826/77

MaoRunhuaOS16-125/75

LiuChuan-Gang

OS8-11

20/62

OS16-325/76

LiuShangzheng

OS18-230/80

OS16-425/76

Liu

Shilong

OS15-6

24/73

OS16-6

25/77

Liu

I-Hsien

OS8-7

20/60

OS16-9

26/77

OS8-8

20/61

Mao

Wei-Lung

OS8-1

20/57

OS8-11

20/62

Man

Haifang

OS16-6

25/77

GS2-1

27/89

Liu

Qing

OS15-10

25/75

Mann

Liu

Wei

GS8-2

19/98

Markov

Alexey

OS7-4

21/55

Liu

Ze

OS15-925/74

Martinez-Garcia

Edgar A.

OS7-8

21/57

Lu

Kuan-Chu

OS8-8

20/61

MasudaKoji

OS20-4

35/84

Lund

Henrik

IS-1

18/42

Matsui

HidetoOS11-2

39/65

Matsunaga

Miki

OS3-1

22/47

HautopIS-2

18/42

MatsunoFumitoshi

GS10-440/105

Luo

Haibao

GS4-1

26/92

Mikami

Kaito

GS3-5

32/92

Lushnikov

Alexei

OS7-6

21/56

Mitsuhashi

Koshiro

OS1-3

39/44

GS6-3

37/95

OS1-4

39/45

GS6-4

37/96

MiyachiJunpeiOS13-5

38/69

OS15-2

24/72

Mizumoto

Ikuro

OS3-9

23/50

Mohammed

Suad Khairi

GS10-140/104

GS10-2

40/104

GS10-3

40/104

GS9-1030/103

GS9-1130/103

GS5-3

34/94

Lv

Zhijian[M]
Ma

Qinglian

OS1-5

39/45

Ma

Shuanglong

GS8-2

19/98

Mabu

Shingo

GS4-3

26/93

GS4-4

26/93

GS10-140/104

GS10-2

40/104

GS10-140/104

GS10-3

40/104

GS10-2

40/104

Md Yusof

Zulkifli

Mohamed Sahari

KhairulSallehMokhtar©ICAROB 2018 ALife Robotics Corp. Ltd.

-111-

NorrimaThe 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

GS10-3

40/104

Ogata

TakashiOS17-2

36/78

Moon

Jeong-hwan

OS2-6

32/47

OS17-3

36/78

Mori

KeisukeOS12-3

31/67

OS17-5

36/79

Mori

Kenta

OS13-3

38/68

Ohnishi

Yoshihiro

OS11-1

39/65

Moriguchi

Yuki

OS11-5

40/66

Ohno

Shuichi

OS3-8

23/50

Morimoto

Shogo

OS12-2

31/66

OkazakiNaonobu

OS12-3

31/67

MoritaYoshifumi

GS10-440/105OS12-4

31/67

Mowshowits

Abbe

GS2-3

27/89

Okubo

Yuka

OS3-3

22/48

Mukunoki

Masayuki

OS1-3

39/44

Okuda

Haruhisa

PS-3

28/41

Olapiriyakul

Sun

GS7-2

34/97

Onda

HiromuOS10-2

33/64[N]
Nagamatsu

Masayasu

OS11-540/66

OnoJumpeiOS17-2

36/78

Nagano

Kaito

GS2-4

27/90

Ono

Michiyoshi

OS21-1

18/85

NagataKazuyukiOS10-1

33/64

OuyangYuxingOS16-1

25/75

Nakagawa

Masato

PS-2

28/41

OS16-2

25/76

Nakamoto

Masayoshi

OS3-1

22/47

OS16-3

25/76

Nakamura

Akira

OS10-1

33/64

OS16-425/76

Nakamura,

Go

GS7-3

34/97

OS16-5

25/76

NanamiTakuya

OS9-2

24/62

OS16-6

25/77

NishidaYuya

OS20-2

34/83

OS16-7

26/77

OS20-335/84

OS16-926/77

OS20-435/84

Nishimura

Shiori

OS5-2

23/52

[P]

Niu

Hong

OS15-1

24/72

Pagliarini

Luigi

IS-2

18/42

Nizamutdinov

Artur

OS7-6

21/56

Park

Chan Gook

OS22-1

31/86

GS6-3

37/95

Park

Gwang Soo

OS22-3

31/87

GS6-4

37/95

Park

Sun Oh

OS2-1

32/45

OS5-1

23/51

Parrend

Pierre

IS-4

28/43

OS5-2

23/52

GS3-1

32/91

OS5-3

23/52

Pei

Wenjing

OS18-330/81

OS14-2

33/69

Peng

YizhunOS15-2

24/72

Petrosyan

Michail

OS7-2

21/54

Phung

Nhuhai

GS3-3

32/91

Amornphun

GS2-5

27/90

GS2-6

27/90

NonakaNonakaHirofumi

Shunta[O]
Obayashi

Masanao

GS4-3

26/93

Phunopas

Oda

KazuyaOS21-1

18/85

Pragnathilaka©ICAROB 2018 ALife Robotics Corp. Ltd.

-112-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Pudchuen

Noppadol

GS1-2

30/88

[Q]
Qin

YiqiaoR. P. C.

Ixchel G.

Ranaweera

OS1-4

39/45

OS1-5

39/45

OS14-4

33/70

OS7-6

21/56

GS6-3

37/95

25/75

SampathOS16-325/76

Satdarov

OS16-425/76

OS16-6

25/77

Sato

Hiroshi

GS3-3

32/91

OS16-9

26/77

Sato

Noritaka

GS10-440/105

Sato

Takao

OS3-2

22/48

Sawai

Yasuhiko

GS10-440/105SethaputThunyaseth

GS7-1

34/97

Sgorbissa

Antonio

GS8-3

19/98

OS6-1

38/53

JanakaRamirez

39/44

OS16-1[R]
Rajapakse

OS1-3

Timur

OS6-2

38/53

Shabalina

Ksenia

OS7-8

21/57

OS10-2

33/64

Shapiai

Mohd

GS10-140/104

GS2-1

27/89

GS2-6

27/90

GS10-2

40/104

IbrahimRecchiuto

Carmine T.

GS8-3

19/98

Shi

PeihaoOS10-2

33/64

Rerkjirattikarn

Pavinee

GS7-2

34/97

Shibanoki

Taro

GS7-3

34/97

Rojas

Juan

OS10-2

33/64

ShimadaRyouichiOS13-2

37/68

Rosendo

Andre

GS6-5

37/96

OS13-3

38/68

OS14-5

33/71

IS-5

36/43

PS-1

18/41

OS1-2

39/44

RoshanRossi

RuggeroShinano[S]

Yuji

ShiungLiew Wei

GS6-2

37/95

Shvedov

Grigory

GS9-1

29/99

S. Liew

W.

IS-6

28/43

SkorikovAndrey

OS7-4

21/55

S. Voong

B.

IS-6

28/43

Sneha

Sharma

OS3-5

22/49

Sabirova

LeysanGS9-5

29/101

Soejima

Yuki

OS20-3

35/84

Safin

Ramil

GS9-7

29/101

Song

Binhu

OS16-926/77

Sagitov

Artur

OS7-8

21/57

Song

Yunzhong

OS19-1

22/81

GS9-4

29/100

SonodaTakashi

OS20-2

34/83

GS9-6

29/101

OS20-335/84

OS20-4

35/84

OS8-2

20/58

Saïghi

Sylvain

OS9-3

24/63

Saito

Keita

OS13-1

37/67

Sakamoto

Makoto

OS1-1

39/44

OS8-4

20/59

OS1-2

39/44

OS8-6

20/60

Su

©ICAROB 2018 ALife Robotics Corp. Ltd.

-113-

Kuo-LanThe 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Subari

Norazian

GS10-140/104OS14-2

33/69

GS10-2

40/104

TanoueSatoshiOS12-4

31/67

GS10-3

40/104

Tanveer

M. Hassan

GS8-3

19/98

Tarasov

Pavel

OS7-2

21/54

OS7-4

21/55

Sung

SangkyungOS22-3

31/87

Suzuki

Yasuhiro

OS4-1

26/50

OS4-2

26/51

Tian

Xiwen

OS19-3

22/82

OS4-3

26/51

Tokumitsu

Masahiro

GS3-5

32/92

Tominaga

Ayumu

GS1-3

30/88

GS2-3

27/89

[T]
TabuseMasayoshi

OS13-1

37/67

Tominaga

Moeko

OS21-4

19/86

OS13-2

37/68

Toussaint

Nicolas

GS3-1

32/91

OS13-3

38/68

Tsoy

Tatyana

GS9-5

29/101OS13-438/68

TsuboiHaruka

OS1-1

39/44

OS13-5

38/69

OS1-4

39/45

Tadaki

KotoneOS17-7

37/80

Tsuji

Toshio

GS7-3

34/97

Tahir

Aqeel

GS8-5

24/99

Tsujii

Ryuichi

GS1-1

30/88

Takada

Taku

GS3-5

32/92

Takagi

Taro

OS3-9

23/50

TakagiTomohikoOS12-1

31/66

Uchida

Yasuo

OS1-1

39/44

OS12-2

31/66

Udayanga

OS14-6

33/71[U]Takahashi

Masanori

OS3-4

22/48

Ueda

Keita

OS11-4

39/65

Takemura

Yasunori

OS21-1

18/85

Ushio

Masahiro

GS6-1

37/95

OS21-4

19/86

Utsunomiya

Koshi

OS20-1

34/83

Ramon

OS3-2

22/48

Talanov

Max

GS9-2

29/100

TamuraHirokiOS14-1

33/69[V]

OS14-2

33/69

Vilanova

TanakaHideyukiOS11-4

39/65

TanakaTakeshi

OS1-3

39/44

TanakaTakeshi

OS6-3

38/53

Wakabayashi

Makoto

GS3-5

32/92

OS6-4

38/54

Wakitani

Shin

OS3-5

22/49

[W]

TanakaKen-ichi

PS-3

28/41

OS3-6

22/49

Tange

Atsuya

OS9-4

24/63

OS3-8

23/50

Taniguchi

Rie

OS4-2

26/51

Wan

WeiweiOS10-2

33/64

Tanikawa

Toyoaki

OS3-7

23/49

Wang

Bo-Min

OS8-9

20/61

Tanno

KoichiOS14-1

33/69

Wang

Chenxu

GS3-4

32/92

©ICAROB 2018 ALife Robotics Corp. Ltd.

-114-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Wang

Fuzhong

OS18-2

30/80[Y]OS19-1

22/81

Yakovlev

OS19-2Konstantin

OS7-1

21/54

22/82

OS7-3

21/55

GS9-1

29/99

OS12-3

31/67

OS12-4

31/67

Wang

GaoyuanOS18-4

30/81

Wang

JiangfanOS15-7

25/74

Wang

Jiwu

GS4-1

26/92

GS4-2

26/93

YamadaAkira

OS11-2

39/65

OS15-4

24/73

YamadaTakayoshi

GS1-1

30/88

OS15-5

24/73

GS8-1

19/98

Wang

Shanfeng

YamabaHisaakiWang

Xiaoxing

OS15-3

24/72

Yamaguchi

Akihiro

GS5-2

34/94

Wang

Xiuqing

OS15-7

25/74

Yamaguchi

Shingo

GS1-3

30/88

OS15-10

25/75

Yamaguchi

YusukeOS13-4

38/68

Watanabe

Hazuki

OS1-4

39/45

Yamaguti

Yutaka

GS5-2

34/94

Watanabe

Keisuke

OS20-1

34/83

Yamamori

Kunihito

OS1-3

39/44

OS20-2

34/83

Yamamoto

Hidehiko

GS1-1

30/88

OS20-5

35/84

GS8-1

19/98

GS7-3

34/97

OS3-1

22/47

GS2-6

27/90

OS3-3

22/48

OS16-1

25/75

OS3-5

22/49

OS16-2

25/76

OS3-6

22/49

OS16-3

25/76

Yamanobe

Natsuki

OS10-1

33/64

OS16-4

25/76

Yankina

Elena

OS7-2

21/54

OS16-6

25/77

Yan

Yujie

OS16-2

25/76

OS16-926/77

OS16-5

25/76

GS2-6

27/90

OS16-7

26/77

Watanabe

Masaki

Weerasingha
Wei

Baochang

Withanage

Yamamoto

Toru

Wu

Chao

OS18-4

30/81

Yano

Yuiko

OS13-3

38/68

Wu

Hsien-Huang

OS8-1

20/57

Yasukawa

ShinsukeOS20-2

34/83

OS8-1020/61

OS21-5

19/86

OS16-2

25/76

Wu

Ming-Guang

OS8-5

20/59

Ye

Wu

Tong

OS18-130/80

OS16-5

25/76

Wu

Zheng

OS15-324/72

OS16-7

26/77

[X]
Xue

Wei

OS15-6

24/73©ICAROB 2018 ALife Robotics Corp. Ltd.

-115-

Zhongyong

Yokomichi

Masahiro

OS1-3

39/44

Yokomori

Takashi

IS-3

36/42

Yoon

Jin Gon

OS2-1

32/45

Yoshimura

Mamoru

OS1-2

39/44

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

Yoshinaga

Tsunehiro

OS1-1

39/44

Zheng

Zheng

OS19-4

22/82

Yoshitomi

YasunariOS13-1

37/67

Zhong

Hong-Jie

OS8-5

20/59

OS13-2

37/68

Zhou

Yousong

OS19-422/82

OS13-3

38/68

Zhu

Qinghui

OS18-2

30/80

Zh-Yao

OS8-5

20/59

GS5-3

34/94

GS9-2

29/100Yu

Jiangtao

GS8-2

19/98

ZhuangYu

Pengfei

GS4-1

26/92

Zulkifli

Yue

YuanliOS15-9

25/74

Zykov

Zafar

Sarmad

GS8-5

19/99

Zakiev

Aufar

OS7-5

21/56

Zhang

HongtaoOS16-7

26/77

Zhang

Mei

OS15-6

24/73

Zhang

WeicunOS18-5

31/81

Zhang

Xinyu

OS16-8

26/77

Zhang

Yu-an

OS1-1

39/44

OS1-4

39/45

OS1-5

39/45

[Z]

Zhao

HuailinOS15-3

24/72

Zhao

Jiming

OS15-1025/75©ICAROB 2018 ALife Robotics Corp. Ltd.

-116-

Evgenii

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

©ICAROB 2018 ALife Robotics Corp. Ltd.

-117-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

©ICAROB 2018 ALife Robotics Corp. Ltd.

-118-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

©ICAROB 2018 ALife Robotics Corp. Ltd.

-119-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

©ICAROB 2018 ALife Robotics Corp. Ltd.

-120-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

©ICAROB 2018 ALife Robotics Corp. Ltd.

-121-

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), B-Con Plaza, Feb. 1- 4, Beppu, Oita, Japan, 2018

©ICAROB 2018 ALife Robotics Corp. Ltd.

-122-

Dynamic Structures for Evolving Tactics and Strategies in Team Robotics
Jeffrey Johnson

The Open University, Faculty of Science, Technology, Engineering and Mathematics
Milton Keynes, Buckinghamshire, MK7 6AA, UK.
Ruggero Rossi

The Open University, Faculty of Science, Technology, Engineering and Mathematics
Milton Keynes, Buckinghamshire, MK7 6AA, UK.
jeff.johnson@open.ac.uk, ruggero.rossi@open.ac.uk
www.complexityanddesign.org

Abstract
The autonomous robot systems of the future will be teams of robots with complementary specialisms. At any
instant robot interactions determine relational structures, and sequences of these structures describe the teamdynamics as trajectories through space and time. These structures can be represented in algebraic forms that are
realizable as dynamic multilevel data structures within individual robots, as the basis of emergent team data
structures. Such formalisms are necessary for robots to learn new individual and collective behaviours. The theory
is illustrated by the example of robot soccer where robot interactions create structures and trajectories essential tothe evolution of new tactics and strategies in a changing environment.
Keywords: multi-robot systems, robot soccer, hypernetworks, multilevel systems, multilevel dynamics

1. Introduction

rugby. Thus team robotics provides a class of complex
socio-technical systems that is more complex than many
physical systems, but less complex than most social
systems. The methodological framework developed here
for team robots is intended to be extended and applied
to social systems.

We use team robotics as a model for the planning,
design, management and control of multilevel complex
socio-technical systems including organisations suchhospitals, airlines, and banks, and socio-economic
policy at national and international levels. We chooserobot soccer as the example for team robotics because it
has well-established rules, there are many games tostudy, and it provides a platform for experimentationand communication.

The challenge of robot soccer goes back more thantwenty years (Kitano et al 1997)1, and can be simply
stated as “By the middle of the 21st century, a team offully autonomous humanoid robot soccer players shall
win a soccer game, complying with the official rules ofFIFA, against the winner of the most recent World Cup.”
Of many varieties of robot soccer we choose to study
the RoboCup simulation league. This enables us tofocus on the spatio-temporal structures of the teaminteractions without being distracted by the considerable
engineering challenges of building physical robots.

The great advantage of studying team robotics as an
exemplar of complex systems is being able to focus onteam interactions without being distracted by the greater
complexities of human agents. For example, unless such
behaviour is programmed into them, robots will not pickup the ball and run with as William Webb-Ellis is said
to have done in 1823, thereby inventing the game of© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan1

Jeffrey Johnson, Ruggero Rossi

Fig 2(b) shows the goalkeeper’s dilemma as a 4-ary
relation <r2, b0, Gb, ball> which, again, is not just a
combination of 2-ary relations (Gb is the blue goal).
Apart from the need to generalise networks to higher
dimensional edges (which are called simplices) anotheressential generalisation of network theory is to include
the relation symbol in the notation. Thus the defenders
dilemma can be better represented by the notation

2. Multidimensional Structure in Robot Soccer
In the 1970s Atkin proposed analysing the relational
structure of chess with his methodology of Q-analysis2.
Atkin studied the relations between the pieces and thesquares of the board. E.g., in Fig. 1(a) the black rook,
black king and white knight form a structure called aknight fork while in Fig. 1(b) they do not. However theconfiguration in Fig. 1(b) is a precursor to the knight
fork, and this plays a part on the evolving structure of
the game. Although not a knight fork itself, unless black
moves to prevent it, the white knight can move to form
the knight fork configuration of Fig. 1(a).

(a) a knight fork in chess<r1, r2, b1, ball ; Rdefenders_dilemma>
which allows the relational structure to be discriminatedfrom other 4-ary relations on these elements – of which
there may be many in a soccer match. A set of nelements with and n-ary relation between them is calleda hypersimplex. A set of hypersimplices is called a
hypernetwork 6.

(b) a knight fork precursor

Fig. 1. Structure in the game of chess1-dimensional 2-dimensional
binary3-aryedge
triangle

In 1979 Gould and Gatrell4 suggested that Atkin’s
method could be applied to soccer analysing the 1977
Football Association Cup Final in the UK. In 1999
Johnson suggested applying the method to robot soccer5
and it was further developed by Johnson and Iravani6
using Johnson’s theory of hypernetworks7. Recently
Ramos et al showed how the hypernetwork approach
can be applied in sports science8.

3-dimensional
4-ary
tetrahedron

4-dimension
5-ary
5-hedronFig. 3. Hypersimplices as multidimensional polyhedra

Fig. 3(a). shows how the binary (2-ary) relations of
networks can be generalised. A 3-ary relation between
three players can be represented by a triangle with threevertices; a 4-ary relation between four players can be
represented by a tetrahedron with four vertices; a 5-ary
relation between five players can be represented by amultidimensional polyhedron with five vertices, and so.

At the heart of the hypernetwork methodology is thefact that in most complex systems the relationships ofinterest are not just between two elements but can be
between n > 2 elements. This is illustrated in Fig. 2.
The ‘defender’s dilemma’ (Fig. 2(a)) involves threeplayers – two red team players and a blue team player.
This can be written as <r1, r2, b1> which generalises thenotation of network edge. In fact this structure could be
written as three binary relations <r1, r2 > + <r1, r2 > + <r1,
r2 > but this would be a poor representation because thedilemma involves all the players at once, not just a
combination of binary relations.

Fig. 4 shows that hypersimplices can have higher
dimensional connectivities, e.g. in Fig 4(a) the
tetrahedra and triangles share edges, or 1-dimensional
connections. Figure 4(b) shows how two tetrahedra canbe connected through a 2-dimensional triangular face. In
general simplices can be connected through multidimensional faces, generalising network connectivity
through single vertices. This provides new kind ofconnectivity structure with potential for exploitation in
robot soccer.

(a) The defender’s dilemma(a)(b) The goalkeeper’s dilemma

Fig. 2. n-ary relation structures in soccer (from 6)(b)Fig. 4 Multidimensional connectivity between hypersimplices© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan2

Dynamic Structures for Evolving

The RoboCup simulation league soccer games that will
be used for illustration last for 6,000 ticks of the clock(10 minutes if ten ticks represent one second). Although
it would be possible to provide information about thewhole game, e.g. the percentage time each team haspossession, chunks of time naturally occur overintervals as short as a hundred ticks. Fig. 5 illustrates
how the 6,000 ticks of a soccer game can be structured
by possession between the red Gliders’ team and theblue CYRUS team.

In soccer three kinds of neighbourhood structures canbe identified:(i) a player has possession of the ball(ii) a player dribbles the ball(iii) a player tackles an opponent
The first of these is characterised by a relation between
the player and the ball, <player, ball; Rpossession; z, t >.
The symbol t denotes time. In the simulation version ofrobot soccer in this paper the game lasts for 6,000 ticksof the clock, notionally 10 minutes at 10 ticks per
second. Let <player, ball; Rpossession; z, t > be called apossession point. The symbol z represents the positionof the possession point. At its simplest we could write z= (xplayer, yplayer, xball, yball). Thus the symbols z and t
represent the space-time structure of the robot soccer
dynamics.

The red rectangles represent periods when the red
players are closest to the ball. For example, the period 0– 170 is controlled by the red team at the kick-off andthe period 1204 – 1346 is controlled by the blue team.
The pink (cyan) line at the beginning indicates that theball is not moving, i.e. the red (blue) team is taking along time to initiate play. The solid red (blue) rectangles
are periods when red (blue) is in direct possession. The
white gaps at the bottom are periods when neither red
nor blue players are close enough to the ball to be inpossession but, in this case, the red (blue) team controls
the ball. The black lines show that the ball is out of play.

The second structure, dribbling the ball, can be denoted
as a sequence of possession points involving just one
player < <player, ball; Rpossession; z, t >, <player, ball;
Rpossession; z; t’ >; <player, ball; Rpossession; z; t” >; … ;Rdribble > >.
The third structure is a relation between members of
opposite teams, r in the ‘red’ team and b in the ‘blue’
team, denoted <<r, ball; Rpossession; z; t >, <b, ball;
Rpossession; z; t’ >; Rtackle >.
Another fundamental structure in robot soccer is(iv) a player passes the ball to a team-mateThis structure can be characterised as <<player, ball;
Rpossession; z; t >, <player’, ball; Rpossession; z; t’ >; Rpass>.
Let this be called a pass structure.

3. Multilevel Systems
This structures (iii) and (iv) in the previous section areinteresting because they are built on relations between
two possession points, i.e. they are structures formed
from more elementary structures. If the players and ballare said to be elements at Level N, a structure such as apossession point exists at Level N+1, and a pass or atackle exists at Level N+2. A multilevel representation
is essential for soccer because, as will be illustrated, thegame is one of players trying to create ‘good’ multilevelstructures and trying to destroy ‘bad’ multilevelstructures.

Fig. 5. The temporal possession structure in robot soccer© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan3

Jeffrey Johnson, Ruggero Rossi

4. An illustrative Example

Fig. 6. A soccer ball trajectory beginning at time 4131,

Fig. 7. The ball position at time 4145.

Fig. 6 shows a green-coloured trajectory for the ballbetween a throw-in at A and a goal being scored at I.
The red team has players denoted r1, r2, r3, … , andattacks the goal on the right. The blue team has players
denoted b1, b2, b3, … , and attacks the goal on the left.
The trajectory has discernable parts beginning with AB
where r5 takes the throw-in to r8.

From Figures 6 and 7 it can be seen that the structuralchange between t = 4131 and t = 4145 can besummarised as follows.
Time t = 4145 components
C1
C2
C3
C4

Let Rpass be the passing relation where < r5, r8; Rpass>means that r5 passes to r8, and <r5, r8; z; t = 4131; Rp>
means that r5 passes to r8 at time t = 4132.

<r1, r2, r4, r7, b11; Rc><r3, r6, b10; Rc> + < r11, b1, b2, b3, b6; Rc><r5, r8, b9; Rc> + <r10, b4, b7; Rc><r9, b8; Rmc>

As can be seen two of the components, C1 and C4,
remain unchanged while the two closest to the ball, C2
and C3, break into substructures.

The closest opponent relation, Rclosest, between theplayers at time 4131 is shown by thin grey lines. A thickgrey line between players shows that they are mutuallyclosest to each other under the relation Rmutually_closest.
The closest opponent relation here results in fourconnected components:
Rc components at time t = 4132
C1
C2
C3
C4

<r1, r2, r4, r7, b11; z; t; Rclosest><r3, r6, r11, b1, b2, b3, b6, b10; z; t; Rclosest><r5, r8, r10, b4, b7, b9; z; t; Rclosest><r9, b5, b8; Rclosest>

The fundamental hypothesis is that structures such asthese may be predisposed to good or bad outcomes forone or other team. In soccer the ultimate objective is toscore goals, but usually goals only emerge at the end ofa sequence of ‘good’ discrete structures such as that
illustrated by the ball trajectory in Figure 6. The
challenge is to be able to recognise structures as being‘good’ or ‘bad’ and to move so that ‘good’ structures
emerge.

Fig. 8. The pitch possession structure at t = 4145

Figure 8 shows the ‘closest to’ relation between theplayers and small areas of the pitch. At first sight thisstructure suggests that the red and blue teams havecomparable domination of the pitch with the blue teamcontrolling more of it.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan4

Dynamic Structures for Evolving

However, as Fig. 9 shows player r6 has enabled new
structures to emerge by dribbling the ball from C to D.
During this time the configuration <r1, r2, r4, r7, b11; Rc>
has changed, with b11 moving to the left and r7 moving
to the right. This structural change is bad for blue sincer7 now controls more space (Fig. 10), enabling r6 to pass
the ball to it at E.

Fig. 12. The pitch-possession structure at t = 4188

By moving in this way the blue team has allowed a‘corridor’ to emerge, shown as a yellow triangle in Fig.
12. This provides a space for the ball to be passed into
an area controlled by r9. As shown in Fig. 13, r7 takes
this opportunity and passes the ball to r9 at G.
Fig. 9. The position at t = 4165

Fig. 13. The position at t = 4217

Fig. 10. The pitch possession structure at t = 4165

Following this pass, r7 dribbles the ball to F (Fig. 11).
During this time the structure <r9, b5, b8; Rc> haschanged with r9 becoming closest to its target goal andb5 moving upwards – a very bad structure for blue!

This involves r9 moving a considerable distance, butnote too that r11 has broken free from its component at
the centre of the pitch and has created space in front of
the goal, connected to the space occupied by r9 and theball.

Fig. 11. The position at t = 4188

Fig. 14. The pitch possession structure at t = 4217

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan5

Jeffrey Johnson, Ruggero Rossi

In this case a set of n players is said to be R-related ifthey form a polygon containing no opponents. For
example, <r7, r9, r10, r11; R> is shown as a pink polygon
at time = 3727. These structures are remarkably
enduring in time, sometimes lasting many ticks of theclock. However, inevitably the relations change andnew structure emerge. Structures of particularimportance are the ‘tight’ configurations of opponent
players, as illustrated by the triangle <r6, b6, b10; R>.
Such configurations occur frequently in the goal area
and are often precursors to scoring goals.
Fig. 15. The position at t = 4230

6. Conclusions

The ball then passed from r9 to r11, who scores the goal,
as shown in Fig. 15.

It has been shown that configurations of soccer players
in the context of areas of the pitch and it goals have amultidimensional representation as hypersimplices. This
leads to three major research questions:

5. Multidimensional hypersimplices

Research Question 1: Are some hypersimplices
particularly disposed to scoring goals?

Although the discussion in the previous section was
based on the binary relations ‘r is closest to b’ and ‘b isclosest to r’, the small networks induce n-ary relations
on the players. Of course, n-ary relation can be definedthat cannot be resolved into binary relations, as
illustrated in Figure 16.

Research Question 2: Can the relevant n-ary relationsof robot soccer be learned from historical games, ormust they be formulated by humans?
Research Question 3: how can the multidimensional
connectivity of hypersimplices be exploited to plan andcontrol the evolutionary dynamics of robot soccer?

References
1. Kitano H, Asada M, Kuniyoshi Y, Noda I, Osawa E(1997) Robocup: The robot world cup initiative. In:
Johnson L (ed) Proceedings of the first international
conference on autonomous agents (Agents-97). ACM
Press, New York, pp 340–347
2. Atkin, R. H., Mathematical Structures in Human Affairs,
Heinemann (London), 1974.
3. Gould, P., Gatrell, A., ‘A structural analysis of a game:
The Liverpool v Manchester united cup final of 1977’,
Social Networks, 2(3), 253-273, 1979-1980.
4. Johnson, J., ‘Robot football, artificial life & complexity’,
Artificial Life and Robotics, 3, 230-235, 1999.
5. Johnson, J., Iravani, P., ‘The multilevel hypernetwork
dynamics of complex systems of robot soccer agents’.
ACM Transactions on Autonomous and Adaptive Systems,
2007
6. Johnson, J., Hypernetworks in the Science of Complex
Systems, Imperial College Press, (London), 2014
7. Ramos, J., Lopes, R. J., Marques, P., Araújo, D.,‘Hypernetworks Reveal Compound Variables That
Capture Cooperative and Competitive Interactions in a
Soccer Match’, Frontiers in Psychology, 28 Aug 2017(a) time = 3727(b) time = 3735
Fig 16. Configurations as multidimensional hypersimplices© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

6

EU-Way vs JP-Way Development – Efficient & Effective Development Approach Masato (Max) Nakagawa

DENSO CORPORATION , Global Technical Affairs
Tokyo Office, 2-7-1 Nihonbash, Chuo-ku, Tokyo, 103-6015, Japan*
Hiroshima University, Guest Professor
1-3-2 Kagamiyama, Higashi-Hiroshima 739-8511, Japan

Abstract
This Plenary Speech will cover three elements described as follows; Firstly, “Factory-IoT” technology which is
DENSO’s globally strategy manufacturing. Secondly, collaboration between Germany and Japan in terms ofinnovation. Thirdly, “EU-Way” development which is an efficient and effective manner of development. From 14
consecutive years experiences working in Germany, The United Kingdom and the Netherlands, “EU-Way”
development will be explained. One of the key points is that in EU they differentiate between “Competition Field”
and “Non-Competition Field” in each technical domains. In the “Non-Competition Field”, also called a “Cooperative
Field”, Europeans cooperate together in the same industry sector to create common specifications or to establish
standardization so that they can concentrate on differentiated technology in the “Competition Field”. This approachis one of the driving forces for development and innovation within EU.

1. DENSO’s “Factory-IoT”
1.1 Roadmap of Production System
DENSO is a manufacturing company mainly for
automotive components and systems. DENSO
started using the single automated manufacturing
process like a spot machine in the 1950’s. After that
DENSO expanded automation to the production line
unit and then further expanded to factory unit and
now to global unit like a global network.
One of the features of DENSO manufacturing is thein-house development and fabrication of the robot
machines since the early 1970’s. Figure-1 represents
the roadmap of the DENSO production system.
DENSO pursuits “DANTOTSU” factory.

Fig.-1 Roadmap of Production System

1.2 DANTOTSU Factory“DANTOTSU” originates from DANZEN + TOP
in Japanese, meaning Outstanding. DENSO’s targetof manufacturing is the status of “DANTOTSU
factory”. By pursuit of “DANTOTSU factory”,
DENSO respects human development which is acompany legacy and in the company DNA.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan7

Masato Nakagawa

Japanese car manufacture MAZDA developed the Rotary
Engine and put into the market.

In addition, DENSO pursues Concurrent
Engineering as a foundation of their manufacturingconcept.
With respect to Automation and Robotics, DENSO
uses automation technology, mainly in Assembly
and, Visual inspection processes and for in-plant
logistics by using in-house machines and robots.
There is however still human-based manufacturing
process left for these three fields. DENSO has beenworking on these fields by using the intelligent
technology of robots. One of the features of thistechnology is “collaborative robots” which meansthat two-robots collaborate and co-operate eachother. It contributes to working efficiency andquality in the manufacturing plant.

2.2 Example of Diesel Engine collaboration
The Diesel Engine was developed by Dr. Rudolf Diesel
in 1892. His motivation was to develop an Internal
Combustion Engine with better fuel consumption
compared to the Gasoline Engine. From this viewpoint,
his achievement was a big contribution to our society.
Especially, in Europe, Diesel Engines account for
approximately 50% of Internal Combustion Engines forpassenger vehicles. On the other hand, heavy-duty andbus/truck applications are almost 100% powered by
Diesel Engines.

1.3 Principle of “Factory-IoT”
It is said that IoT technology is a tool for keeping
maintenance of stable production in the field of
manufacturing. DENSO’s unique point of IoT is thathumans are involved in this process. By utilizing
human skill and knowledge, sustainable growth andcontinuous evolution can be achieved. This meansthat both human and machine can provide theoptimal solutions thanks to co-creation by humanand machine. DENSO pursuits the “Factory-IoT”
technology for all of plants globally by connecting
each plant.

Fig.-2 Diesel Engine Development History

When Dr. Rudolf Diesel invented the diesel engine, theengine size was too big and heavy for small vehicle
applications. Mr. Magokichi Yamaoka who is a founderof Yanmar wanted to utilize a diesel power unit for
agriculture applications.

2. Collaboration between Japan and Germany
2.1 Example of collaborations
There are several examples of collaboration between
Japan and Germany in terms of innovation. One example
is that a Germany engineer invented the Facsimile
machine in 1929. However it was not popular in themarket at that time due to the large size of the machine
and high price. TELEX was a tool for communication
device until the early 1990’s. In Japan, the Japanese
language characteristics are so complicated such as Kanji,
Hiragana and Katakana with many different characters.
Therefore, TELEX is not a suitable tool for the Japanesemarket and society. This is the driving force for Japaneseengineers to further develop the Facsimile machine to
make it compact with reasonable price.
Another example is the Rotary Engine for use inautomobiles. The Rotary Engine was invented by aGerman engineer, however it was not mass production.

Fig.-3 Dr. Rudolf Diesel Profile

He was born into a farming family in Japan. At that time,machines were not available for agriculture works. All of
the jobs were done by human power. He saw very heavy
load and work every time and he wanted reduce thehuman work load by machine. He decided to go toGermany to study the diesel engine to make it morecompact and light weight for agriculture applications like

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

8

EU-Way vs JP-Way Development

EU, the German development way is used as a typicalexample. There are significant and remarkabledifferences in the EU development way compared to thatof Japan-Way. In Germany, there is a clearly two fields
of development style. One is “Competition Field” and theother is “Non-competition Field” for development. They
co-operation and collaborate among the same industry
domains in the base technology field. They establish thecommon technology field like standardization and/or
regulations. Thanks to this system, they can concentrateon their resources on development within the unique
technology field. Then, they could create thedifferentiated technology. On the other hand, in Japan,
OEMs develop almost all technologies by themselves. In
this approach, they could create unique technology,
however, recently there are many new technologies to be
developed like Electrification, AI and IoT connect toautomobile.
Therefore, there are concerns regarding the lack ofresources for the development of their unique
technologies.

small tractors and combine machines. He developed theworld-first small diesel engine for mass-production
applications based on Dr. Rudolf Diesel’s engine
technology. Both engineers contributed to the Internal
Combustion Engine for us. Mr. Magokichi Yamaoka
received the Diesel Gold Prize by the Germany Patent
Agency thanks to his contribution. He donated theJapanese garden in the “Rudolf Diesel Anniversary Park”
in the city of Augsburg where Dr. Rudolf Diesel spent
most of his life.

Fig.-4 Mr. Mogokichi Yamaoka Profile

Later, the city of Augsburg decided to name streets inthe city from Mr. Magokichi Yamaoka’s home town inJapan. Nagahama-Alle and Amagasaki-Alle. This is oneof the successful and beautiful collaboration stories
between Japan and Germany. We should recognize theformer engineer innovation spirit.

Fig.-6 Comparison of Europe and Japanese Characteristics

3.2 Collaboration Concept between Germany & Japan
There are good points to be learnt from German-Way
development. For example, efficient and rational
development by using industry standard tool-chains,model base development, cutting-edge simulation
automated calibration etc. In addition Germany OEMs
cooperate with R&D Engineering companies for theirvehicle and engine development in not only the researchphase but also for application engineering. There areremarkable R&D Engineering companies in Europe whohave sophisticated development tools, testing facilities
and high-talented engineering resources. In Japan also,
there are good points for their development way like
teamwork and harmonized development approach with
dignity.

Fig.-5 Nagahama-Alle & Amagasaki-Alle in Augsburg

3. EU-Way vs JP Way Development
3.1 Comparison between EU-Way and JP-Way
Figure 6 represents the comparison between EU and JP
development ways.
This chart summarizes the main features development
ways, working styles and business models based on the
Automotive sector of Japan and Europe. In this chart, for© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan9

Masato Nakagawa

Fig.-7 Collaboration Concept between Europe and JapanThe key point is to respect each other of theirdevelopment ways, then reflect and accept the good
points into their development ways. Figure 7 represents
the concept of collaboration between Europe and Japan.
The important point is that both good points to be wellarranged like a fusion. In this case, a new development
way which is a fusion between Europe and Japan could
become a global competitive engineering approach.
Both parties should respect their development ways andinspire each other in research and development fields.

References
1. M. Nakagawa, 26th International AVL Conference Engine
2020: Sparks versus compression ignition in a newenvironment (Graz, Austria, September 2014)
2. M. Nakagawa, Dr.-Ing. O. E. Hermann and Dipl.Ing.
Sebastian Visser, Diesel Powertrain Energy Management
via Thermal Management and Electrification (Detroit SAE
April 2017)
3. M. Nakagawa, DENSO’s Contribution for Future
Predictive Powertrain Control including Electrification(Korea, FISITA 2016)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan10

Innovation on Manufacturing Generated by Intelligent Technologies
Ken-ichi Tanaka

Corporate Research and Development Group, Mitsubishi Electric Corporation,
8-1-1, Tsukaguchi-Honmachi, Amagasaki City, Hyogo 661-8661, Japan*
Haruhisa Okuda

Autonomous Control System Development Project Group, Advanced Technology R&D Center,
8-1-1, Tsukaguchi-Honmachi, Amagasaki City, Hyogo 661-8661, Japan
E-mail: Tanaka.Kenichi@ah.MitsubishiElectric.co.jp, Okuda.Haruhisa@ct.MitsubishiElectric.co.jp
http://www.mitsubishielectric.com

Abstract

In order to cope with social problems such as labor population decrease, the autonomous cell production
robot system had been developed to realize automation in variable variety multi production. The
intelligent technologies such as random bin picking, force control, and error recovery were developed toapply the robot from simple-transport-work to assemble-task. In addition to the talk about suchtechnologies, it will also be described that open innovation method cooperating with universities to create
research results which our company would not achieve independently. The examples of the latest robot
technology development using artificial intelligence will also be described.
Keywords: cell production robot, intelligent technologies, open innovation, artificial intelligence

1. Main Slides

Fig. 1. Changes in Production Systems

Fig. 2. Artificial Intelligence in Robotics© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan11

Ken-ichi Tanaka, Haruhisa Okuda

Fig. 3. Academia-Industry Collaboration

Fig. 6. Prototype of a Robot Cell System

Fig. 4. Information Processing System of a Robot

Fig. 7 Bin Picking Robot

Fig. 5. History of Robot Project

Fig. 8. Proposed Method© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan12

Innovation on Manufacturing Generated

Fig. 9. Fast Graspability Evaluation

Fig. 12. Prototype of a Parts Feeding Robot

Fig. 10. Regrasp Motion Planning

Fig.13. Environment
Industry

Fig. 11. Examples of Adoptable Parts Shapes

Fig. 14. Further Transition of Production Systems

surrounding

Manufacturing© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan13

Ken-ichi Tanaka, Haruhisa Okuda

Fig. 15. AI technologies for Smart Factory

Fig. 18. AI Application in FA field -3

References
1.
2.

3.

4.

Fig. 16. AI Application in FA field -1
5.

6.

K. Tanaka, T. Sawaragi, Prospect of intelligent
technologies realizing robotic cell production,
SICE SI2010, 1J2-2, pp.657-659, (2010). (in Japanese)
A. Noda, K.Tanaka, H. Okuda, T. Nagatani, Y. Kitaaki
and Y. Domae, Intelligent Robot Technologies for Cell
Production System, ISFA 2010 (2010 International
Symposium on Flexible Automation), JPS-2562, (2010).
H. Okuda, R. Haraguchi, Y. Domae and K. Shiratsuchi,
Novel Intelligent Technologies for Industrial Robot in
Manufacturing Architectures and Applications, 2016
International Symposium on Robotics (ISR2016), pp. P3.8,(2016).
Y. Domae, H. Okuda, Y. Taguchi, K. Sumi and T. Hirai,
Fast Graspability Evaluation on Single Depth Maps, 2014
IEEE International Conference on Robotics andAutomation (ICRA), pp. 1997-2004, (2014).
R. Haraguchi, Y. Domae, K. Shiratsuchi,Y. Kitaaki, H.
Okuda, A. Noda, K. Sumi, T. Matsuno, S. Kaneko and T.
Fukuda, Development of Production Robot System thatcan Assemble Products with Cable and Connector,
Journal of Robotics and Mechatronics, Vol.23 No.6,
pp.939-950, (2011).
S. Ryu, K. Shiratsuchi, R. Haraguchi and H, Okuda,
Development of industrial robot controller with force
control functions, SICE SI2011(The 12th SICE System
Integlation division annual conference), 1E2-4, (2011).
(in Japanese)

Fig. 17. AI Application in FA field -2© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

14

Modular Playware and Personal Health Technology
Henrik Hautop Lund

Center for Playware, Technical University of Denmark, Building 326, 2800 Kgs. Lyngby, DenmarkE-mail: hhl@playware.dtu.dk
www.playware.elektro.dtu.dk

Abstract
In this paper, we describe the development of personal health technology such as wearable systems monitoring healthconditions. It has been advocated that such personal health technology through monitoring health status may motivate
people to perform health related actions and life-style changes. Here we describe a methodology on how these
personal health technologies may rather be used as a tool for designing and adapting game activities, which motivate
people to perform the desired actions. Especially, we exemplify this methodology with the use of FitBit monitoringof steps and heart rate to the design of appropriate, physically demanding games for the modular interactive tiles,
Moto Tiles, which are used by older adults for prevention and rehabilitation. Thereby, the motivation to perform theactions arrives from the fun play on the Moto Tiles, whereas the personal health technology is used as a tool tomonitor the effect and guide the game development.
Keywords: Playful technology, Playware, Personal Health Technology, FitBit, Adaptive Games.

With this view of personal health technology as apersuasive technology, researchers and practitioners
have investigated nudging through the new opportunities
arising with the technology. It is well-known that
information, priming, peer pressure, etc. may guidepeople to certain behavioral actions (e.g. choosing acertain behavior) – this has nowadays been termed
nudging. Nudging with personal health technology
includes the simple nudge that an arm wrist wearable acts
as a physical reminder to be active, and the interface on
the physical device or the app acts as a positive or
negative reinforcement by presenting e.g. statistics,
target amount, and time since last action.

1. Introduction
Personal health technology such as different wearable
systems monitoring health conditions have beendeveloped over the last decade and finds its way to thegeneral public. These personal health technology devices
are often described as a means to engage and motivate
the public to perform health related actions, and e.g.
change behavior to become healthier. A typical example
is that wearing a step counter is believed to engage andmotivate people to perform more steps (walk more, run
more, bike more, etc.) due to goal setting (e.g. 10,000
steps per day) and insight into their own performance(how many steps did I take today?). As such, the personal
health technology is viewed as a persuasive technology[1]. Persuasive technology is technology designed tochange people's beliefs and behaviors, e.g. creating
certain health habits amongst the users of the technology
product.

There are numerous health nudging apps and devices forhydrating, moving, sleeping, etc. It appears that the mainidea of nudging in these kind of personal healthpersuasive technologies is based on presenting a
numerical indicator, a graph and a numerical goal, whichtranslates to nudging/persuading the user by providing© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 2-4, B-Con Plaza, Beppu, Japan15

Henrik Hautop Lundthe user with an insight into the personal health status.
Such an insight is believed to be a strong enoughmotivation for people to perform more of the desired
healthy behavior.

Indeed, it is important to distinguish between extrinsic
and intrinsic motivation [3], as the Big Five personality
model may predict motivation in e.g. education, sports,
and exercise, where the Big Five traits are associated withdifferent levels of self-determined motivation. Hence,people may be motivated to different degrees and not all
are given to respond positively to extrinsic motivationsuch as the nudging suggested by the personal healthtechnologies. For instance, Hart et al. [4] found that
“Conscientiousness, openness, and extraversion werepositively associated with intrinsic achievement
motivation, whereas extraversion, conscientiousness, andneuroticism were positively related to extrinsic
achievement motivation. Agreeableness was also foundto be negatively associated with extrinsic achievement
motivation. Conscientiousness was anomalous in that it
was positively related to both intrinsic and extrinsic
motivation.”In this context, the connotation of nudging is that such an
insight into personal health measurements is such astrong motivation that people will perform more of thedesired healthy behavior. Accordingly, the numericalinsight into one’s own personal health should act as astrong extrinsic motivation.
However, it is questionable if everybody will respondwell to such a nudging based on numerical insight into
one’s own personal health acting as a strong extrinsic
motivation. Will everybody act equally well to suchinformation? For instance, it is known from message
tailoring that adapting persuasive messages to recipients’characteristics is more effective than providing the samemessage to all recipients, since recipients with different
characteristics will respond more or less to the samemessage. For instance, Hirsch et al. [2] demonstrated
effective message tailoring using a comprehensive modelof recipients’ personality traits. “Respondents judged anadvertisement emphasizing a particular motivational
concern as more effective when that concern was
congruent with their own personality characteristics.
Message-person congruence effects were thus observed
by manipulating the framing of an appeal to target abroad variety of motives, including desires for
excitement and social rewards (extraversion), connection
with family and community (agreeableness), efficiency
and goal pursuit (conscientiousness), safety and security(neuroticism), and creativity and intellectual stimulation(openness/intellect).” [2]. These are termed the Big Five
personality dimensions:• Openness to experience: (inventive/curious vs.
consistent/cautious).
• Conscientiousness: (efficient/organized vs.
easy-going/careless).
• Extraversion: (outgoing/energetic vs.
solitary/reserved).
• Agreeableness: (friendly/compassionate vs.
challenging/detached).
• Neuroticism: (sensitive/nervous vs.
secure/confident).

To investigate the effect of intrinsic motivation inpersonal health technology, we suggest to put focus onplay. Play is often defined as a free and voluntary activity
that you do with no other purpose than play itself and forthe enjoyment that it brings [5]. There is no product oroutcome as such of play, but people play for their own
enjoyment and fulfillment, i.e. there seems to be an
intrinsic motivation to play, when people are in a play
dynamics. Hence, it becomes interesting to understandhow people enter into such a play dynamics, in which
they fully engage in play and seem to forget about timeand place. It has been suggested that there exists playforces, which drive people into a play dynamics, and that
playware can act as such a play force [6]. The hardware
and software augmentation of objects to become
playware [7, 8] may for instance draw people into
moving, touching, turning, etc. due to emission of sound,light, odor, etc. and immediate explicit feedback from the
playware drive people into further actions, acting as aplay force pushing people into a play dynamics ofcontinuous actions and feedback.
Even if there is no product or outcome of such play basedon the intrinsic motivation for own enjoyment, there may
be collateral effects of play [9]. For instance, thecollateral effects of play can be motor skill enhancement,
cognitive and physical rehabilitation, etc. These© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 2-4, B-Con Plaza, Beppu, Japan16

Modular Playware and Personalcollateral effects of play, e.g. on the health, can besignificant and important [10].

Here we propose that the short-term effect measurements
can be used for designing for increasing the long-term
effect (the collateral effect of play).

Therefore, in the present work, we put focus on thecreation of playful experiences and intrinsic motivation.
So instead of viewing the personal health technology as
a motivational tool with the health status as the primaryeffect, our methodology involves the personal healthtechnology as a tool for designing and adapting gameactivities, which are the ones to intrinsically motivate
people and with which the health status is a collateraleffect.

This demands that there is a relationship between the
short-term (run-time) effect measurement and the longterm effect, and that we can find such relationship(s).
This clearly depends on the selection of which short- andlong-term health parameters we are looking at, i.e. which
physiological states are measured and with whichmeasuring tools at short-term and at long-term. It will bea whole field in itself (e.g. within AI and Big Data) tofind this coupling and relationship for the range ofmeasuring tools, measurements and intervention areas.
Here, we are therefore limited to provide only a first idea.
Nevertheless, to exemplify this approach, in this study,
we use a personal health technology, Fitbit, to measurerun-time effects and their correlations with scores fromplaying on the Moto tiles

2. Method
We exemplify the methodology with the use of FitBit
monitoring of steps and heart rate to the design ofappropriate, physically demanding games for themodular interactive tiles, Moto Tiles [11], which are usedby older adults for prevention and rehabilitation. Thereby,
the motivation to perform the actions arrives from the funplay on the Moto Tiles, whereas the personal healthtechnology is used as a tool to monitor the effect andguide the game development.

Fitbit is an activity tracker, which is wireless-enabled
wearable technology that measure number of steps
walked, heart rate, sleep pattern, and steps climbed. It isworn as a wristband, and connects to a mobile app
synched via Bluetooth. It this study, we use the Fitbit tomeasure heart rate and steps.

There is an important distinction to be made between thegeneral long-term effect and the short-term effect. It is
possible to measure both the short-term effect and thelong-term effect, and it is clearly interesting to obtainknowledge of the potential relationship between the
short-term effect and the long-term effect.

Moto tiles are modular interactive tiles used for playful
prevention and rehabilitation by allowing users to playdifferent games on the tiles that light up in different
colours [11]. The Moto tiles can be put together likepuzzle pieces in different patterns, and different games
can be selected from a tablet to be played on the Mototiles. In a game, the user typically will have to movearound to hit different tiles according to the game pattern.
In the present study, we use three games called Color
Race, Final Countdown, and Reach (see videos of thegames [12]).

The general effect of an intervention can typically bemeasured in pretest and posttest in a randomized
controlled trial (RCT) with comparison between theintervention group and the control group. In our case here,
we view this as a long-term effect compared to the shortterm effect (or we can even call this the run-time effect),which is typically provided with the personal healthtechnology such as the FitBit. The short-term effect or
run-time effect may be measured as physical activity
level (e.g. steps, heart-rate, etc.) or mental activity level(e.g. blood flow in prefrontal cortex to indicate
concentration).

3. Experiments and Results
In the first experiment, we used the Fitbit and Moto tiles
data collection to investigate potential relationships
between physiological measurements and game score indifferent games. The hypothesis was that there exist such
relationships, and that these may guide future game© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 2-4, B-Con Plaza, Beppu, Japan17

Henrik Hautop Lund

design to create games, which may result in collateral
health effects of play.

In order to investigate the relationship between the runtime effect and the long-term effect, we conducted
another pilot study. In this case, 11 older adults
performed Moto tiles training twice a week for 10 weeks
in a senior activity center in Lyngby municipality,
Denmark, as part of the EU project REACH. The older
adults were pre- and posttested with standard tests suchas timed-up-and-go, chair-to-stand, 6 minutes walking
test, and Berg Balance Scale. Here we will look only at
the data from the Berg Balance Scale, which wascompletely performed by 7 out of the 11 participants.

A university student would wear the Fitbit while playing
each game in different tiles configurations each 10 times.
The first configuration were 2*3 tiles connected together
in a rectangle with the games Color Race, Final
Countdown and Reach. The second configuration was 2,
2, and 2 tiles put in a triangle with 1.5m in between eachof the 2 tiles islands with the games Color Race and Final
Countdown. The third configuration was 2, 2, and 2 tiles
put in a triangle with 2.5m in between each of the 2 tiles
islands, and with the games Color Race and Final
Countdown.

We compared the maximum score on games played on
the Moto tiles of each individual with the score of theposttest (Berg Balance Scale).

Table I shows the Pearson correlation coefficients
between the Moto game score of each game and thedifferent activity data collected from the Fitbit (the endheart rate, the maximum heart rate during a game, the risein heart rate during a game, and the number of steps
measured during a game).

Figure 1 shows the game score (each column
accumulated maximum game score of the three games)
and the score of the Berg Balance posttest on the x-axis.
Even if the data here is very small from such an initial
pilot study, it is interesting to speculate whether there canbe a correlation between the game score and the healthscore (such as represented with e.g. Berg Balance testscore). For instance, the cut-point for low risk of fallingand medium risk of falling in Berg Balance test is 40, andcut-off point 45 has been suggested for multiple fall risk
and assistive need, so with a much larger data set it would
be interesting to see if some game score could correlate
to a Berg score above or below such cut-points.

It can be seen that both the end HR data and max HR data
are able to show positive correlation in 6 out of the 7cases, which is apparently more reliable than percentagechange in HR. The average correlation coefficients of the
6 successful cases of end HR and max HR are 0.6819 and0.7044 respectively, which implies that max HR can be abetter indicator of the game intensity. In addition, theresult of steps shows fairly strong correlation with Moto
game score in all games.
The general indication of this small pilot study istherefore, for these six cases, a higher Moto game scoresignifies that the user has taken more steps and havereached a higher heart rate. It is therefore indicated, in
this case, that the run-time effect is predictable from theMoto game score.

Fig 1. Relationship between Berg score (posttest) andmaximum Moto game score for 7 older adults.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 2-4, B-Con Plaza, Beppu, Japan18

Modular Playware and Personal

4. Discussion and Conclusion

References

It is important to notice that there may be habituation to
the games, so one has to be careful in using game scoreas a predictor for the general health status (though one
has to distinguish between habituation to the tool (i.e.
Moto tiles) and habituation to the activity (i.e. game) that
may change), and much more work is needed.
Nevertheless, it is interesting with the small indicators ofrelationship between game score and measurements fromthe personal health technology (heart rate and steps)
representing the short-term effect, and the standard
balancing test representing the long-term effect. Thispilot study should therefore open up for much morethorough studies on such potential relationships, theirnature and their relevance.

1. B. J. Fogg "Persuasive technology: using computers tochange what we think and do." Ubiquity 2002. December(2002): 5.
2. J. B. Hirsh, S. K. Kang, & G. V. Bodenhausen“Personalized persuasion: Tailoring persuasive appeals torecipients’ personality traits.” Psychological science,23(6), 578-581, 2012.
3. R. M. Ryan & E. L. Deci. Intrinsic and extrinsic
motivations: Classic definitions and new directions.
Contemporary educational psychology, 25(1), 54-67,
2000.
4. J. Hart, M. Stasson, J. M. Mahoney, and P. Story. The big
five and achievement motivation: Exploring therelationship between personality and a two-factor modelof motivation. Individual Differences Research. 5. 267274, 2007
5. J. Huizinga. Homo Ludens Ils 86 (Vol. 3). Routledge.
2014
6. C. Jessen, & H. H. Lund, H. H. On play forces, play
dynamics and playware. Unpublished manuscript, 2009.
7. H. H. Lund, and C. Jessen, “Playware - Intelligent
technology for children’s play,” Technical Report TR2005-1, June, Maersk Institute, University of Southern
Denmark, 2005.
8. H.H. Lund, T. Klitbo, and C. Jessen, “Playware
Technology for Physically Activating Play”, Artificial Lifeand Robotics Journal, 9:4, 165-174, 2005
9. H. H. Lund, “Play for the Elderly - Effect Studies ofPlayful Technology,” in Human Aspects of IT for the
Aged Population. Design for Everyday Life. (LNCS Vol.
9194, pp 500-511, Springer-Verlag, 2015)
10. H. H. Lund, and J. D. Jessen, “Effects of short-term
training of community-dwelling elderly with modular
interactive tiles,” GAMES FOR HEALTH: Research,
Development, and Clinical Applications, 3(5), 277-283,
2014.
11. http://www.moto-tiles.com (checked: 28/11/2017)12. http://www.moto-tiles.com/video.php?L=en&s=games(checked 28/11/2017)
13. G.N. Yannakakis, J. Hallam, H.H. Lund “Capturing
entertainment through heart-rate dynamics in the playware
playground” Proceedings of the 5th International
Conference on Entertainment Computing Lecture Notes in
Computer Science, vol. 4161, Springer, Cambridge, UK,
pp. 314-317, 2006.

The relationship shown between game design and set-up,
heart rate and steps suggest that such physiologicalmeasurements can be used in the design process to design
physical interactive games with different qualities. Some
designs may be more effective than others, and they may
be qualified, tested and verified for their effect as has
been indicated in the work presented here.
Previously, it has also been shown that children’s heart
rate can be captured on a prototype playware playground,and that children's notion of entertainment correlates
highly with their average heart rate during the game [13].
Importantly, the interaction was mediated by the Mototiles acting as a playware, which motivated theparticipants to play for their own enjoyment. In this study,
the personal health technology in the form of Fitbit acted
as a measuring tool, and it was the playware (Moto tiles),which acted as the persuasive technology.
Acknowledgement
The experiments with Fitbit measurements of Moto tiles
games were performed by Yancy Liu. The pilot study
with older adults in Lyngby municipality was performed
as part of the EU project REACH: Responsive
Engagement of the Elderly Promoting Activity andCustomized Healthcare (thanks to J. Jessen, DTU
Management (Prof. Andersen), Lyngby Municipality,and project partners).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 2-4, B-Con Plaza, Beppu, Japan

19

How the new technologies might lead to a paradigm shift in psychological andneuropsychological research.
Luigi Pagliarini 1,21Henrik Hautop Lund 1

Centre for Playware, Technical University of Denmark, 2800 Kgs. Lyngby, Denmark2
Academy of Fine Arts of Macerata, Via Berardi, 6, 405111 Macerata, Italy
luigipagliarini@gmail.com
Abstract

Recently, we started to apply old psychological and neuropsychological standard tests and therapeutic exercises intoa different domain of technology. Therefore, while maintaining the whole conceptual and logical structure of the
traditional approach – i.e. tests mostly made by hand on paper or with the mouse and computers’ screens - intact, we
moved them onto the physical interactive and playful technology, Moto Tiles. Such a translation of input/output,
likewise involving dissimilar perceptions and different parts of our motor system (i.e. legs, feet and balance system),might imply a change in subjects’ mental and behavioural strategies, variations in efficacy and efficiency, as well asa redefinition of reaction times. In short, this shift, besides providing a significant extra tool to older practices, mightwork as a crucial test for all of the existing theoretical background, too. We describe our new method and report few
of the first empirical applications.
Keywords: Playful robotics, Psychology, Neuropsychology, Therapy, Clinical Tests, New Paradigm.

expertise that were originally implemented and validatedwith traditional tools (i.e. paper-and-pencil, mouse andcomputers’ screen, and etc.) to a new and possiblycommon input/output flow, the Moto Tiles technology[2]. We look at this “translation” as a great opportunity
for providing:A significant extra tool to older practices;A crucial test for all of the existing practice andtheoretical background.
A common platform (and consequently set ofdata) to all of the tests;

1. Introduction
In both psychology and neuropsychology there areseveral tests and exercises that enable professionals toevaluate a large set of humans psychological responses tospecific stimuli and try to measure or improve theirabilities. This is true for both normal and clinical patients.
These psychological tests have the great advantage ofbeing properly validated and, as a consequence, they
produce "an objective and standardized measure of asample of behaviour” [1] with results that are very solidand steady. There is a large set of tests with different
options and methods that differs between each other forat least three characteristics:a) Trials used to measure abilities;b) Trials used to improve abilities;c) Domain of application.
Of course, a single test might cover up both (a) and (b)simultaneously while quite rarely a single tool can range
from one domain to another (c).
In this pilot study we will only be considering
psychological tests that focus on basic cognitive features
and abilities domain, so excluding those that try to reflect
or measure psychological entities such as aptitudes,
emotions, personality, interests, IQ, etc.
Moving from this valuable tradition, we considered totransfer such consistent set of data and long lasting

Sure enough, our transfer of input/output is to be verified
and weighted, since besides requiring different
perceptions, it implies the participation of different motor
planning/acting systems (i.e. legs, feet and balancesystem). Therefore, this change in behavioural demandmight imply a change in the subjects’ mental andbehavioural strategies, variations in efficacy andefficiency, as well as a redefinition of reaction times andsome of the behavioural parameters.
Anyway, our effort should be rewarded by the advantage
of moving on a common digital platform that can collectdata from different experiments in a much more
standardized way.
In this paper, we present a couple attempts to transfer old
and consolidated tests from the psychological literature© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 2-4, B-Con Plaza, Beppu, Japan

20

Luigi Pagliarini, Henrik Hautop Lund

regarding the attentional system in order to examine thefeasibility of the above described methodology.
2. Inspiring Psychological Tests
The first set of psychological tests we moved from are
two traditional tools used in clinical environments toevaluate subjects’ attentional system efficiency. They areLine Bisection tests
Visual Search tests

Fig. 2. Visual search tests.

This simple test is very predictive of subjects’ efficiency
on visual attention tasks, and, originally, it was a tool
used by Treisman in defining is Feature Integration
Theory (FIT) on Selective Spatial Attention [7].

2a. Line Bisection tests
The Line Bisection [3] is used in neuropsychology toquickly assess both the type and the severity of Unilateral
Spatial Neglect (USN). Probably it is the most-used andquickest test for such a diagnosis; therefore it is popular,solid and efficient. It consists of a set of paper-and-pencil
tests where a line (or more) a few centimetres long is
drawn and the patient is then asked to dissect the line at
the midpoint (see Fig. 1). Patients exhibiting, for example,
left-sided neglect will respond with a rightward deviationof the line's true midpoint. The same test can also be used
for normal adults [4, 5], so it might be seen as a moregeneric valuation of spatial attention efficiency in any
subject.

3. Implementation of the tests on Moto Tiles
The two psychological tests described above were usedas starting point and inspiration material for
implementing respectively two new Moto Tiles
games/exercises. Despite of the fact that, changing
physical platform implies a crucial change on tests
significance, we tried to keep as much faithful as possible
to the original tests conception.
3a. Neglect.
The Neglect game/exercise is an attempt to transfer thepsychological Line Bisection tests knowledge to theMoto Tiles environment. Neglect can be a single player
task or a challenge between two players. The gameindirectly measures the users’ RTs (Reaction Times) andthe general state of vigilance, and therefore might also be
used as a therapeutic exercise for elderly and, inparticular, for all of those patients with inattentive
problems subsequent to a stroke, an ictus or degenerative
neurological syndromes.
Tiles Configuration.
One tile is positioned in the (virtual) centre of the roomand four in each side of it (left and right).

Fig. 1. Line Bisection tests. Up a single line test example. Down
a multiple line test example.

2b. Visual Search tests.
The well-known visual search paradigm, designed toassess the function of visual attention, has had anenormous impact on many aspects of science. The vast
majority of visual search studies performance is assessedwith response times. Typically, the task is to determine
whether a target is present or absent [6]. Search slopesthat measure how response times change as more
distractors are added to the display are presumed toassess the speed of the search and whether attention is
involved or, clinically speaking, the efficiency level of agiven subject attentional mechanisms.

Fig. 3. Testing the Neglect game/exercise on Moto Tiles.

Lateral tiles can be positioned both in a squared shape(2*2) and in a linear shape (4*1, as shown in Fig. 3),
accordingly to the kind data we want to collect or thegame/exercise dynamic we want to impress.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

21

How the new technologies

1. Number of players, one (for therapeutic reasons) or
two (gaming);
2. Colour for each player;
3. Number of reiteration of the exercise/game;
4. Timing parameters (will influence the game
difficulty);
5. Pattern discrepancy (level of difficulty in terms ofphysical/visual distance between different patterns);
6. Target Feature. The target tile lacks or has a special
feature (will influence the game difficulty and target
analysis);
7. Game start.

Game Options.
The game options are to be selected on the tablet’s
software interface, and they are:
1. Number of players, one (for therapeutic reasons) or
two (gaming);
2. Colour for each player;
3. Number of reiteration of the exercise/game;
4. Timing parameters (will influence the game
difficulty);
5. Game start.
Game Dynamics.
Once the game is started from the tablet, the central tile
is lit up with one of the two colours (in both one or two
players sessions). To start the real section the player
identified with that very colour has to smash it to start thereal sub-section. At that point, one of the 8 lateral tiles(pseudo-randomly chosen) will be lit up for a short time
(to be defined accordingly to gaming and therapeutic
sections) and the goal of each individual (either in asingle player or in a one-to-one section) is to squeeze it
as fast as possible (using one of its feet). Timing is
calculated and a threshold applied so that after a given
number of milliseconds the lit tile, if not squeezed, isturned off and at that point, whatever case occurs, anothertile is lit up. Such an operation is reiterated for 8 to 16
times (also accordingly with section’s purposes). After
that, in the case of one-2-one competition, the central tile
is lit up again, with the other colour, and waits for the
new sub-section to be started. At the end of the second
section the tablet calculates the shortest timing andprovides a visual feedback that underlines the winner. In
the single player case the game/exercise simply comes toan end.

The patterns correspond to different configuration of theactivation of the LEDs within one tile (see Fig. 4 for someexamples).

Fig. 4. Designing the Special Pattern game/exercise.

Game Dynamics.
When the game is started all the tiles are lit up. But threeout of four tile will show the same pattern, while thefourth one will show a different pattern. The user(s) haveto react as fast as possible to try to crush on the “different”
tile’s pattern amongst the four (neglecting the threewhich show the very same patterns). The game/exercise
keeps on rolling like that up to the last stimuli set. At theend it provides a visual feedback that shows the winnerof the one-2-one competition or, for the single player
option, provides a visual feedback about the performance.

The one-2-one game/exercise is pretty dynamic, very
easy to understand and quite funny. It is to be highlighted
that, in both cases, the level of difficulty can also be tuned
by shaping the morphology of the playground (lateral
tiles are positioned far).
3a. Special Pattern or Special One.
The Special Pattern game/exercise is an attempt totransfer the psychological knowledge on visual searchtests in our Moto Tiles environment. In short, Special
Pattern can be used as a single player task or a challenge
between two players. In the one player version four tiles
are positioned in a linear (4*1 tiles) or squared shape (2*2tiles), while in the one-2-one version everything isdoubled.
Gam
e Options.
The game options are to be selected on the tablet’s
software interface, and they are:Both versions of the game/exercise, also accordingly tothe options tuning made at the start when the play isconfigured, result in a quite effort-demanding increase of
attention and/or visual search. The one-2-one version, inparticular, when timing gets crucial, has been reported
and turns out to be mentally speaking deeply challenging.
4. Knowledge transfer
We tried to move a couple of tests regarding Spatial
Attention and Visual Search from the old paper-andpencil school, normally used in psychological© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

22

Luigi Pagliarini, Henrik Hautop Lund

The actual state of the art is that we have built two games
inspired to two corresponding traditional psychologicaltests, whose methodology has been shown here, and weare in the process of validating them with an empiricalcase study.
Of course, if such a pilot experiments will result positive,
we intend to move on to a larger panorama on cognition(e.g. perception, memory, sound, and etc.) and, further,on different psychological domains (e.g. aptitudes,
emotions, personality, interests, IQ, etc.).

experiments, to a new realm of technologies. Since these
new technologies (i.e. the Moto Tiles) are made of digitaland electronic devices and as input/output demand for a
pretty different set of perceptions and action, our biggest
effort is to try being as faithful and coherent as possible
with the former experience and knowledge. It is too early
to think of a significant outcome, though we believe weare going in the right direction. This is because the Moto
Tiles technology has shown to be very adaptive todifferent tasks’ specifications and demonstrated to bepractically very close, as behavioural practice, to theoriginal methods used.
In particular, we believe the Neglect task we have built,
while not superimposable to the Line Bisection task, itshould require a similar amount of spatial attention,
specially if compared to Multiple Line Bisection, and acomparable in terms of precision/effort of action inphysical space.
The Special Pattern task seems to be even closer to thecorresponding Treisman’s figure integration test, since it
seems to require both a similar demand of selective
spatial attention and the same precision/effort of action inphysical space.
Of course, the two games/exercises might need to be
perfected, integrated with additional features or finetuned, and that’s why at the very moment we are in theprocess of building a number validation tests in which wewill compare data from traditional tests with those from
Moto Tiles in a group of human subjects

References
1. Urbina, S. Anastasi, A. Psychological testing (7th ed.).
Upper Saddle River, NJ: Prentice Hall. p. 4. 1997.
2. H. H. Lund, “Play for the Elderly - Effect Studies ofPlayful Technology,” in Human Aspects of IT for the
Aged Population. Design for Everyday Life. (LNCS Vol.
9194, pp 500-511, Springer-Verlag, 2015)
3. S. Chokronab, P. Bartolomeo, M.T. Perenine, G. Helftf, M.
Imbertg. Scanning direction and line bisection: a study ofnormal subjects and unilateral neglect patients with
opposite reading habits. Cognitive Brain Research.
Volume 7, Issue 2, October 1998, Pages 173-178
4. A. Varnava, M. McCarthy, J.G. Beaumont. Line bisection
in normal adults: direction of attentional bias for near andfar space. Neuropsychologia. Volume 40, Issue 8, 2002,
Pages 1372-1378
5. A.D. Milner, M. Brechmann, L. Pagliarini. To halve andto halve not: an analysis of line bisection judgements innormal subjects. Neuropsychologia. 1992 Jun; 30(6):51526.
6. Neisser, U. (1963) Decision-time without reaction-time:
Experiments in visual scanning. American Journal of
Psychology 76: 376–385.
7. A. M., Treisman & G. Gelade, (1980). A featureintegration theory of attention. Cognitive Psychology,
12(1), 97-136.
8. H. H. Lund, and J. D. Jessen, “Effects of short-term
training of community-dwelling elderly with modular
interactive tiles,” GAMES FOR HEALTH: Research,
Development, and Clinical Applications, 3(5), 277-283,
2014.
9. BrainBaseLine. https://www.brainbaseline.com/

5. Discussion
In this paper we depicted one method to move old
psychological and neuropsychological knowledge in to a
new technological tool. This is not a fully new experiencesince in the past the old pencil-and-paper psychologicaltests have been brought into software-based
environments, see [8] for an example. Therefore, ourattempt is just one more step towards digitally based tests,
but with a little stronger stress on the standardization of
the tests data and physical representation of the test tool.
Our main goal is to bring old findings regarding brain
oriented clinical diagnosis and therapy to a new domainof technology, the Moto Tiles, which would allow us tocollect data, as well as run trials and therapy exercises, inan environment that is basically standardizable, highly
stimulating and playful, as we know from our former
experiences [2, 8]. In doing this we are limiting our
experience to a specific subset of possibilities, visuoattentional tests in the cognitive domain of Psychology.
This is because we believe that the Moto Tiles
game/exercise can be, in general, very predictive of thegeneral state of sensory and motor alertness of anybody[2] and, in the particular case of neurological patients,
might measure the state of the attentional system quite
efficiently.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan23

Natural Computing Paradigm ─ A Concise Introduction
Takashi Yokomori
Department of Mathematics, Faculty of Education and Integrated Arts and Sciences,
Waseda University, Tokyo 169-8050, Japan
E-mail: yokomori@waseda.jp

Abstract
Natural computing (NC) is an emerging area of research that investigates computing techniques and models inspiredby nature on one hand, and it also investigates phenomena taking place in nature in terms of computational
methodologies on the other hand. Thus, research in NC congenitally has interdisciplinary flavor, which bridges
between computer science and various disciplines of natural science. Because of its interdisciplinary nature, NC
connects and covers a broad spectrum of fundamental research fields including biology, chemistry, physics, medical
science, and so forth. In this article, we give a concise introduction to the new computing paradigm of NC.
Specifically, we give an overview of selected topics of the fields from theory to experiments, where the stress is
primarily put on theoretical achievements in computing paradigms called molecular computing and chemical reaction
computing.
Keywords: Natural computing, molecular computing, self-assembly, chemical reaction computing.

has edited a special issue in a journal entitled MolecularComputing Paradigms. His initial efforts in the early daysof molecular computing have been succeeded by Head's
original work on Splicing Systems and Languages
whose theoretical results on splicing phenomena should
be highly appreciated, in that it was the first achievement
by means of mathematical analysis
onbio-chemical
operations of DNA
recombination. It
ispossibletodiscuss about newcomputingparadigms inspired bynaturewithageneral schema of“ Computing with
X ” , where X
rangesfromvarious sorts ofmaterials to those

1. Introduction
Natural computing (NC) is a novel computing paradigm
inspired by nature in which NC concerns a variety ofcomputational aspects and potential features found innatural phenomena. Also, NC has close relations tocomputational process inspired by those phenomena.
The goal of NC includes the following research themes.
(i) From nature to information processing which means
investigating computational mechanism and its
implementation based on lessons from nature, and(ii) From information processing to nature which meanscomputational analysis of nature in terms of designing
new computing models and computer experiments(simulations). Further, NC also intends to exploit new
applications in the fields of various disciplines involved.
It is rather surprising that the conceptual source of theidea of NC has already appeared in an article of Biofizika
in 1973 where an idea of cell molecular computers was
discussed. Since 1974 Conrad has been conducting
consistent research on the information processing
capability using macro-molecules (such as proteins), and© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan24

Takashi Yokomoriof principles. Figure 1 illustrates a part of natural
computing paradigms proposed so far within the“Computing with X ” schema.
In this short article, among various computing
paradigms we take up two computing paradigms“Molecular computing” (from materials), and“Chemical reaction computing” (from principles), andgive an outline of fundamental results on computing
capability of these new challenging ingenuity.
2. Molecular Computing Paradigm
In the early days, the goal of molecular computing
seemed to replace silicon-based hardware with biomolecular one. Unlike this intention of the early schema,
however, by now it is commonly recognized thatmolecular computers should explore challenging
applications unsuitable for existing computers.
In the theory of molecular computing, a variety ofcomputation models have been proposed and investigated.
Among others the models of computation based on selfassembly are well recognized to be of great importance inthat they can provide one of the most basic
frameworks for molecular computing paradigms. In fact,
Adleman's groundbreaking wet lab experiment [1](solving a small instance of the Hamiltonian Path
Problem) is a typical example of molecular computation
based on the self-assembly principle.
When we make a brief revisit to the Adleman's
original work in 1994, it turns out that his algorithm may
be formulated into the schema where component
structures are sophisticatedly encoded into linear
molecules. From these observations, a general schemaof “one pot” self-assembly computation model is outlined
as follows:
1. design a finite set of basic units for assemblycomputation (one may take this finite set as amolecular program),
2. put all those basic units with sufficiently highconcentration into one pot, to create a random poolof a single pot (that is, leave the pot for a certain
time period, resulting in producing all possible
assembly of basic units),
3. (if necessary) perform screening operations to
extract only necessary (or unnecessary) assemblyof basic units,
4. detect whether or not there is an assembly withdesired constraints, then answer yes if there is, andno otherwise.

are translated(encoded) into specific forms of twodimensional structural molecules. These basic blocks(units) of structural molecules with sticky ends aredesigned so that they may form a DNA complex linearlygrowing by self-assembly property. This means that YAC
performs a simulation of computation (generation) by G,
where the input string is accepted by YAC if and only if
the final DNA complex forms a completely hybridized
double-stranded (that is a desired constraint to be checkedby screening mechanism).
It is shown that any recursively enumerable language canbe recognized by YAC model. Thus, the following holds.
Proposition 1. ([2]) There effectively exists a computing
model based on self-assembly principle whose
computational power is Turing universal.
Other molecular computing models using highdimensional structures have been proposed to solve NPcomplete problems such as the satisfiability (SAT)
problem by Jonoska et al., where 3-dimensional graphstructures are used and the algorithm for SAT runs in time
proportional to the number variables involved in a given
formula. On the other hand, Yokomori investigated aself-assembly computing model in an abstract level,
called Computing by Conformational Change (CCC),
and proposes the following general schema that(Computation)=(Self-Assembly)+(Screening Mechanism),

where “self-assembly” is due to hybridization of either
uncoded or coded molecular components, while“screening mechanism” is regulated by either natural or
artificial constraint ([2]).
3. Chemical Reaction Paradigm
Inspired by the work of reaction systems (initiated by[3]), the notion of reaction automata has been introduced
in [4] by extending sets in each reaction (of a reaction
system) to multisets. We start this section by reviewing
basic notions concerning reaction automata.
For a finite set S, a reaction in S is a 3-tuple a = (Ra,Ia, Pa) of finite multisets, such that Ra, Pa ∈ S#, Ia ⊆ S
and Ra ∩ Ia = , where S# denotes the set of all finite
multisets over S. The multisets Ra and Pa are called thereactant of a and the product of a, respectively, while the

Figure 2 illustrates the overview of computation schema
based on self-assembly.
In this section, we will briefly review one of the typicalmodels based on self-assembly computation, called YAC
model. Based on a unique normal form grammar G(proposed by Geffert) that can generate any recursively
enumerable language, a computing model YAC has been
proposed, in which an input string and every rule in G

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan25

Natural Computing Paradigm ─ A

set Ia is called the inhibitor of a. These notations areextended to a multiset of reactions. A reaction a = (Ra,Ia, Pa) is applicable to a multiset D if Ra⊆D and Ia∩ D= .
As a result, D' (=(D-Ra)∪Pa) is derived.
With a simple example, we introduce the notion ofreaction automata. Let us consider a reaction automaton(RA) = (S, , A, D0, f) defined as follows:
S = { p0, p1,a, b, a', f } (symbols of objects),={a, b} (input symbols),
A = {a0, a1, a2, a3, a4 } (reactions)}, wherea0 = ( p0, aba', f ), a1 = ( p0a, b, p0a' ), a2 = ( p0 a'b, , p1 ),a3 = ( p1, a'b, a, p1 ), a4 = ( p1, aba', f),
D0 = p0 (an initial multiset), and f is the final symbol.

Figure 3. A graphic illustration of interactive processesn nfor accepting strings in L={a b | n ≧ 0}.

Proposition 2. ([4,5]) The computing power of reaction
automata in maximally parallel manner coincides withthat of reaction automata in sequential manner with input mode. Further, both of those computing powers are
Turing universal.
On the other hand, the equivalence may not hold forreaction automata in sequential manner with ordinary
input mode (i.e., without the use of -input mode).

In order to study the role of inhibitors of reactions in thecomputing power of RAs, we introduced and studied a
restricted RA called Chemical Reaction Automaton(CRA) in which no inhibitor is allowed ([6]).
Further, the space complexity issues of reaction
automata (RAs) have been considered. By restricting thevolume (i.e., the state of an RA) used in reaction process
for a successful computation, some subclasses of RAs
were introduced and investigated on relations between
classes of languages accepted by those subclasses of RAs
and language classes in the Chomsky hierarchy.
Let f(x) be a function defined on the set of natural
numbers and X in {sq, mp}. The workspace offor w
is intuitively defined as the mini-max size of multisets(appearing in all reaction sequences) necessary for
accepting w. Then, an RA is said to be f(n)-bounded
if for any w in LX( ) with n=|w|, the workspace of for
w is bounded by f(n). Then, four classes of RA languages
have been investigated.
Suppose that an RA
is f(n)-bounded. If a function
f(n) is a constant k (linear, exponential), then is termed
constant-(resp. linear-, exponential-) bounded. The classof languages accepted by constant-bounded RAs (linearbounded RAs, exponential-bounded RAs, RAs, CRAs) in
X manner is denoted by(resp.,,). The class of languages accepted byconstant-boundedRAs(linear-bounded
RAs,
exponential-bounded RAs, RAs, CRAs) in X mannerwith -input mode is denoted by(resp.,,,).
The results on whole view of language class relations
are summarized in Figure 4, where
denotes the class of regular (resp. context-free, contextsensitive, recursively enumerable) languages in Chomsky
hierarchy. Further,denotes the class of languages
generated by Petri net systems. Thus, except for theclass, each class in Chomsky hierarchy is exactly
characterized by a subclass of RA languages. Further, thefollowing result has been obtained.

Proposition 3. ([5]) There exists a recursively enumerable language which cannot be accepted by any reaction
automaton in sequential manner.

Proposition 4. ([6,7]) A language L is in
if and only if L is a homomorphic
image of some language in.

Let w=aabb be a given input string. Since
has the initial
multiset D0 (=p0), there is no reaction in A applicable to D0.
When receiving the 1st a of w, the multiset D0 of becomes
p0a to which only a1 is applicable and, as a result, D1 (=p0a') is
derived. On receiving the 2nd a, the multiset of becomes
p0a'a from which D2 (=p0a'a') is derived by applying a1. Then,receiving b and applying a2 lead D2 to D3 (=p1a'), and onreceiving the final b, makes D3 into D4 (=p1) by a3. After
applying a4, D4 eventually leads to the final multiset f. Thus, an
input string aabb is accepted by . This example shows asuccessful reaction process of
performed in sequential
manner and the set of strings thus accepted by is denoted by
Lsq( ).

Figure 3 illustrates an overall view of possible reaction
processes in with inputs anbn for n ≧ 0, and we seethat Lsq( ) = { anbn | n ≧ 0 } which is a context-freelanguage. We remark that this interactive process can bealso performed by in maximally parallel manner, i.e.
in the manner that every applicable reactions are
performed exhaustively. That is, it holds that
Lmp( )=Lsq( ).

Besides both manners of applications in RAs, we often
consider an extension of the sequential manner where RA
allows a reaction without receiving any input symbol ateach step, which is called sequential manner with -input
mode.
We investigated the accepting powers of reaction
automata with these manners of applying reactions, andobtained the following.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

26

Takashi Yokomori

4. Broader Perspective

References

Natural computing,(NC) comprises a very broad range ofcomputational principles that varies from wellestablished classical research areas to newly emerging
dynamical research areas with great potentiality of many
promising applications to interdisciplinary fields.
The former family may include Cellular computation,
one of the oldest computing models studied since late
1940s. In Neural computation, artificial neural networks
are proposed as computing systems inspired by the
biological neural networks mimicking brain functions.
Evolutionary computation provides a bunch ofalgorithms for global optimization strategy inspired bybiological evolution. Further, Artificial life is a longlasting research theme from which artificial immune
algorithms have been lately developed. As for the latterfamily of research areas in NC, there are quite a few
topics to be mentioned, while due to the space limit wecan regretfully name only a part of those emergence offascinating thoughts here: Membrane computation,
Reaction-diffusion computation, Optical computation,
Quantum computation, Swarm intelligence, etc.
Further, their applications to practical problems in thereal world such as drug discovery in “nano-levelengineering” and “medical/life science” are highly
recommended to be tackled. For one example, one may
refer to a medical application, lately reported, of the useof C.elegans for detecting cancer patients.
Finally, the reader interested in more details about
topics discussed here and other many subjects in NC iscordially advised to consult reference papers orappropriate bibliographic sources (such as [8,9,10]).

1.
2.

3.
4.
5.

6.
7.

8.
9.

L. Adleman, Molecular Computation of Solutions to
Combinatorial Problems, Science, vol.266, pp.1021-1024,
1994.
T. Yokomori, Computation = Self-assembly + Conformational Change: Toward New Computing Paradigms,
Proc. of 4th International Conference on Developments in
Language Theory DLT'99, Aachen, July, pp.21-30, 1999.
A. Ehrenfeucht and G. Rozenberg, Reaction systems,
Fundamenta Informaticae vol.75, pp.263-280, 2007.
F. Okubo, S. Kobayashi and T. Yokomori, Reaction
Automata, Theore. Comput. Sci., 429, pp.247-257, 2012.
F. Okubo, On the Computational Power of Reaction
Automata Working in Sequential Manner, 4th Workshop
on Non-Classical Models for Automata and Applications,
book@ocg.at series 290, pp.149-164, Osterreichische
Computer Gesellschaft, 2012.
F. Okubo
and T. Yokomori, The Computational
Capability of Chemical Reaction Automata, Natural
Computing, 15, pp.215-224, 2016.
F. Okubo, S. Kobayashi and T. Yokomori, On theProperties of Language Classes Defined by Bounded
Reaction Automata, Theore. Comput. Sci., 454, pp.206221, 2012.
Gh. Pӑun, G. Rozenberg, A. Salomaa (Eds.), Handbook of
Membrane Computing, Oxford Univ. Press, 2010.
G. Rozenberg, T. Bäck and J.N. Kok (eds.), Handbookof Natural Computing, 4 Volumes, Springer, 2012.

10. 萩谷昌己, 横森貴（編）, ナチュラルコンピューティング・
シリーズ第０巻～第７巻：近代科学社，2011 年．

Figure 4. The diagram of the relations among RA
language classes and Chomsky hierarchy. (A solid
arrow denotes a proper inclusion.)

Acknowledgements
The work partly supported by Partly supported by JSPS
KAKENHI, Grant-in-Aid for Scientific Research (C)17K00021 of The Ministry of Education, Culture, Sports,
Science, and Technology, Japan, and by Waseda University
grant for Special Research Project: 2017K-121.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan27

Artificial Immune Ecosystems: the role of expert-based learning in artificial cognition
Pierre Parrend*

ICube laboratory, University of Strasbourg, UMR CNRS 7357, Strasbourg, France, ECAM Strasbourg-Europe,
Schiltigheim France
Fabio Guigou, Julio Navarro, Aline Deruyver, Pierre Collet

ICube laboratory, University of Strasbourg, UMR CNRS 7357, Strasbourg, France

Abstract
The rapid evolution of IT ecosystems significantly challenges the security models our infrastructures rely on. Beyond
the old dichotomy between open and closed systems, it is now necessary to handle securely the interaction between
heterogeneous devices building dynamic ecosystems. To this regard, bio-inspired approaches provide a rich set ofconceptual tools, but they have failed to lay the basis for robust and efficient solutions. Our research effort intends torevisit the contribution of artificial immune system research to bring immune properties: security, resilience,
distribution, memory, into IT infrastructures. Artificial immune ecosystems support a comprehensive model for
anomaly detection and characterization, but their cognitive capacity are limited by the state of the art in machinelearning and the rapid evolution of cybersecurity threats so far. We therefore propose to enrich the cognitive processwith expert-based learning for reinforcement, classification and investigation. Application to system supervisionusing system logs and supervision time series confirms the relevance and performance of this model.
Keywords: Artificial Immune Systems, Cybersecurity, Immunity, Computational Ecosystem, Anomaly detection

knowledge management layer [3]. The AIE shall embed
the properties of natural immune systems: security,
resilience, distribution, memory. To this aim, their
cognition capacity for anomaly identification andcharacterization in a context of dynamic systems andrapidly evolving threats is a key requirement. We areconvinced that cognition, i.e. the support of search,
adaptability, memory and learning [4] in such
heterogeneous environments will not be achieved in ashort time span through fully automated solution, andthat the contribution of human creativity in the analysis
process is a major enabler for adaptation and learning.
We therefore propose in this paper to extend the AIE withexpert-based learning for reinforcement, classificationand investigation. We apply this model to systemsupervision using system logs and supervision time series
to confirm its relevance and performance.
Section 2 introduces related works. Section 3 defines therole of cognition in the Artificial Immune Ecosystem(AIE). Section 4 formalizes the expert-aided learning in

1. Introduction
Old closed IT infrastructures increasingly turn into open
interconnected ecosystems of heterogeneous devices:
this shift is put to light in particular by the numerous
challenges posed by IoT Security [1]. In this context, thedetection of anomalies caused by cyberattacks becomesan ever more tedious task to identify known anomalies,
new ‘zero-day’ anomalies or signs of system instability
following these anomalies. Bio-inspired architectures
and algorithms provide to this regard a promising set ofconcepts and solutions for multi-scale dynamic
ecosystems. However, no robust and efficient solution
exists to solve the issue of generic anomaly detection sofar. We therefore work on a new generation Artificial
Immune Systems (AIS) called Artificial Immune
Ecosystem (AIE): it encompasses not only the analysis
layer as classical AIS do, but also a basis layer for robust
and resilient data management, an immune
communication protocol [2] as well as a supervision andE-mail: pierre.parrend@unistra.fr, http://ecam-strasbourg.eu/, https://www.unistra.fr/*© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

28

Pierre Parrend, Fabio Guigou, Julio Navaro, Aline Deruyver, Pierre Collet

the AIE. Section 5 presents the experiments and theevaluation of this model. Section 6 concludes this work.

satisfy a viability constraint” [8], i.e. if it is capable ofreaction in a context where its viability is at risk.
Although these properties have been devised in atheoretical context, they prove to be key to theconstruction of an efficient Artificial Immune Ecosystem
for multi-scale IT infrastructures.

2. Related Work2.1. Mechanisms of natural immunity
It is possible to summarize natural immunity as theperception of three types of signals by the immune
system [5]:1-signal: the affinity between antigens and antibodies
enable these latter to identify known pathogens. This
process typically occurs through B-cells and T-cells inacquired immunity.
2-signals: the affinity between immune receptors and
PAMPs (Pathogen-Associated Molecular Patterns, suchas endotoxin from certain bacteria) triggers infectious
inflammation and multiplies the immune reaction. This
process typically occurs through Pattern Recognition
Receptors (PRR) in innate immunity.
3-signals: the affinity between immune receptors and
DAMPs (Danger Associated Molecular Patterns, such as
signs of necrosis or intracellular proteins like DNA whenreleased in the intercellular milieu) triggers noninfectious inflammation. This process typically occurs
through TLR (Toll-Like Receptors, a specific kind of
PRR) in innate immunity.

3. Cognition and the Artificial Immune
Ecosystem
3.1. The artificial immune process
A major difference exists between natural immunity andartificial immune processes: In natural systems, theperception of the three (3) signal types occurs through thevarious immune cell types in parallel. In artificial
systems, detection mechanisms are best defined as aprocess involving several analysis approaches so far, as
defined in the artificial immune detection process inFigure 1. 1) The first step for anomaly detection isdeterministic memory-based recognition of pathogens. It
typically uses explicit rules with a tightly define scope,such as in antiviruses. 2) The second step consists instochastic immune detection: 2.1) detection based on
known antigens, for instance using fuzzy detectors,
similar to the 1-signal model of natural immunity, 2.2)
recognition of a generic model similar to the 2-signals
model and 2.3) detection of danger signal similar to the
3-signals models. 3) The third step consists in thecharacterization of the anomaly. It can be automated or
assisted by the expert as specified in Section 4. 4) Next,
the system memory is updated to leverage the knowledge
gained through the characterization step. 5) A reaction is
performed – which is beyond the scope of thiscontribution. One should note the difference between
step 1) which is deterministic and well understood andstep 2.1) which leverages stochastic technologies and isan active research field.

2.2. Cognition and immunity
Natural immunity is a reference model of learning andcognition where a fully decentralized and autonomous
system supports a comprehensive loop of information
input, processing and subsequent reaction [6]. This loop
is known since Varella as the enactment principle [7].
The cognitive domain covers the range of search,
adaptability, memory and learning [4] capabilities.
Search is the capability of processing input information
for partially hidden signals. Adaptability is the capabilityof letting information processing evolve over time tocope with environment evolution. Memory enables toretain raw or processed information about passed events.
Learning occurs when memory content trigger theadaptation of search capability. It can concern both facts
and behaviours.
Bourgine and Steward propose a more global definitionof cognition: “A system is cognitive if and only if sensory
inputs serve to trigger actions in a specific way, so as to© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

29

Artificial Immune Ecosystems:

Figure 2 : Anomaly characterisation
The output of steps 3.1, 3.2.1 and 3.2.2 enables thecontinuous adaptation of the learning artificial immune
process. Step 3.2.3 requires that the process can embedcomplex knowledge in latter analyses such as new
detectors, new algorithms or new algorithm parameters.

Figure 1 : The artificial immune detection process
On can note several radical differences between thenatural immune system (IS) and the artificial immune
system process:1) Artificial immunity greatly benefits of integrating
deterministic detectors since such technologies are
already available and mature, whereas natural systems
rely on probabilistic affinities only.
2) In natural IS, no notion of process flow exists, immune
mechanisms occur in parallel, whereas an artificial
process embeds expert knowledge in its own design.

4. The role of the Expert in the Artificial
Immune Ecosystem
The expert intervenes in anomaly characterization forreinforcement, classification of investigation.
4.1. Expert-based reinforcement (step 3.2.1)
In expert-based reinforcement, the expert approves orinvalidates the classification into a given category.

3.2. Anomaly characterization
The step 2.2) of anomaly characterization entails an
automated step and a manual-driven step, as shown in
Figure 2. We share the vision of D. Engelbart that in
highly evolutive environments, interactive systems
which “increas[e] the capability of a man to approach acomplex problem situation, to gain comprehension to suithis particular needs, and to derive solutions to problems”[9] are a necessity to overcome the intrinsic limitationsof bottom-up fully automated systems. Step 3.1) consists
in automated detection, typically using statistical ormachine learning approaches. If the analysis processprovides a satisfactory output here, anomaly
characterization is completed. Otherwise, the output of
this automated step is presented to the expert for further
analysis. If analysis requires a manual confirmation of
automated classification for reinforcement, the expert
validates or invalidates the system proposal (step 3.2.1).
If the analysis requires a manual classification, the expert
allocates the anomaly to a category (1 dimension),
potentially creating a new one (step 3.2.2). If thecharacterization is more complex (2 or more dimensions),
the expert performs a deeper investigation (step 3.2.3).

As stated in Equation 1, a classification proposal propthat a signal SU pertains to a given category Ci is made by
the system. If the expert labels this classification ascorrect, the membership of SU to Ci is confirmed, as given
in Equation 2. If the expert labels this classification as
incorrect, the membership of SU to Ci is invalidated, asgiven in Equation 3.
4.2. Expert-aided classification (step 3.2.2)
In expert-aided classification, the expert provides thecategory of unknown items, when the system does not
manage to find a satisfactory category itself.

Equation 5 states that system knowledge K is compoundof a set of known anomaly categories { Ci }. Equation 6
states that this set of known anomaly categories entails aset of n traces of Anomalies {Ai} [2], a set of m traces of© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

30

Pierre Parrend, Fabio Guigou, Julio Navaro, Aline Deruyver, Pierre Collet

Anomaly Accompanying Pattern {AAPi} [3], a set of otraces of danger {Di} [2]. Equation 7 states that when thissignal SU is affected by the expert ( ) to an existing
anomaly category Ci which can be either a known
anomalous event Ai, a known pattern AAPi or a known
danger signal Di symptom of abnormal system use, thisexisting category Ci is extended with anomaly U. This
process occurs when Ci does not contain U yet. Equation
8 states that when this signal SU is affected by the expert
to a new category Cn, which is either an anomalous event
A, a pattern AAP or a danger signal D, the set C of
anomaly categories is extended with Cn, that entails asingle anomaly U at this point.

based recognition of pathogens and 2) stochastic
recognition of pathogens embed the search facility. Step
3) Anomaly characterization embeds adaptability. Step
4) Memory update ensures the memory property together
with its exploitation in steps 1), 2), 3). The combinationof search, adaptability and memory ensures the capabilityof learning facts; step 3.2.3) investigation ensures thecapability of learning behaviours. In our current model,
the human expert supports the reaction process, theviability constraints of IT systems equipped with an AIE
and does most of the adaptability capability and behavior
learning. The expert-augmented AIE is thus a cognitive
system in the sense of Bourgine’s definition [8], butimmense research questions are open before thecommunity can transform the proposed process into afully autonomous one.

4.3. Expert investigation (step 3.2.3)
In expert investigation, the experts performs thecharacterization of the properties of an unknown item.
This characterization can typically lead to the definitionof new detectors.

5.2. Expert-based reinforcement
We studied the efficiency of expert-based reinforcementwith the Morwilog tool [10], which exploits expert
feedback to validate the identification of sequences ofsystem logs as occurrences of multi-step attacks.
Although these results are theoretical, they provide
highly promising insights wrt. the capability of handling
rapidly evolving complex signals in the context ofcybersecurity.

As stated in Equation 9, expert investigation EI leads tothe analysis of an unknown signal SU as a set of anomaly
properties {pi}, which then becomes a known signal Sk.
As stated in Equation 10, the intervention of expert
design process ED can lead to the creation of a detector
DK for the signal Sk. As stated in Equation 11, theknowledge base K is then extended with signal Sk, and thedetector base D with detector DK. Note that in the case ofinvestigation, the knowledge is compound of thesubproperties Sk = {pi} of each anomaly, rather than outof anomaly categories as in the previous cases ofclassification. Expert-aided classification is thus a special
case of the investigation where |Sk = 1| and Sk {Ci}, i.e.
the properties can be assigned to an explicit, well-knowncategory.
5. Evaluation

Figure 3 : Evaluation of expert-based reinforcement
in Morwilog

5.1. A cognitive Artificial Immune Ecosystem

5.3. Expert-aided classification

In the frame of Bersini’s definition of cognition domains[4], the proposed artificial immune process is actually a
cognitive process: it supports search, adaptability,
memory and learning, and does so for ensuring theviability of the system. Steps 1) deterministic memory-

We studied the efficiency of expert-aided classificationin the application of the Artificial Immune Ecosystem to
network supervision [10], in the particular case of theanalysis of time series information. Again, the theoreticalresults show that the involvement of an expert in the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan31

Artificial Immune Ecosystems:

learning process, here for classification, greatly quickens
the analysis capability. On a dataset of 10.000 entries, 30
requests on expert feedback, i.e. 0,3%, enables to achieve
89,7% of precision as given by classification F-score. 500
expert feedbacks, i.e. 5%, provides 99% precision.
Maximum
expertfeedback
F-score

20

30

50

100

500

Unlimited

.36

.70

.72

.72

.78

.78

Vaslin, Véronique Legrand. This work was founded by
ICube Laboratory (SENSAI project), French BanquePublique d’Investissement (BPI) under AAP-19 HuMa
project, Agence Nationale de la Recherche
Technologique (ANRT) and the enterprise IPLine.
References
1. S. M. Hashemi and J. He. La-based approach for iot
security. Journal of Robotics Networking and Artificial
Life, 3(4):240–248, 2017.
2. P. Parrend, P. David, F. Guigou, C. Pupka, and P. Collet.
The AWA artificial emergent awareness architecture
model for artificial immune ecosystems. In IEEE
Congress on Evolutionary Computation 2017, Special
Session on Artificial Immune Systems: Algorithms,
Simulation, Modelling & Theory, June 2017.
3. F. Guigou, P. Parrend, and P. Collet. An artificial immune
ecosystem model for hybrid cloud supervision. In
Complex System Digital Campus’15, Sep 2015.
4. H. Bersini and F. J. Varela. Hints for adaptive problem
solving gleaned from immune networks. In International
Conference on Parallel Problem Solving from Nature,pages 343–354. Springer, 1990.
5. J. Greensmith, U. Aickelin, and S. Cayzer. Detecting
danger: The dendritic cell algorithm. In Robust Intelligent
Systems, pages 89–112. Springer, 2008.
6. J. E. Hunt and D. E. Cooke. Learning using an artificial
immune system. Journal of network and computer
applications, 19(2):189–212, 1996.
7. F. Varela. Invitation aux sciences cognitives, 1988.
8. P. Bourgine and J. Stewart. Autopoiesis and cognition.
Artificial life, 10(3):327–345, 2004.
9. D. C. Engelbart. Augmenting human intellect: A
conceptual framework. Summary report afosr-3223 undercontract af 49 (638)-1024, SRI project 3578 for air force
office of scientific research. Stanford Research Institute.
Retrieved March, 1:2007, 1962.
10. J. Navarro Lara, A. Deruyver, and P. Parrend. Morwilog:
an ACO-based system for outlining multi-step attacks. In
IEEE Symposium Series on Computational Intelligence(IEEE SSCI 2016). IEE, Dec 2016.
11. F. Guigou, P. Collet, and P. Parrend. The artificial immune
ecosystem: a bioinspired meta-algorithm for boosting timeseries anomaly detection with expert input. In
EvoApplications, 20th European Conference on the
Applications of Evolutionary Computation, Apr 2017.

Table 1: Impact of expert-based classification
5.4. Expert investigation
We studied the efficiency of expert investigation in thecontext of Morwilog tool: before the reinforcement phasecan start, the AIE provides suspicious traces as input tothe expert to enable him to characterize prototypicalmulti-step attacks. Based on a list of suspicious traces,
the expert thus identifies the actual threat and confirmsthat a given event sequence builds a single attack.

Figure 4 : Attack steps identified through Morwilog
investigation
6. Conclusions and perspectives
In this contribution, we propose a process for enforcing
artificial immunity, i.e. a bio-inspired model supporting
the search for three immune signals: pathogens,
pathogen-associated patterns and danger. The artificial
immune process entails rich detection capabilities, aswell as anomaly characterization, which is key to ensure
adaptability and memory, and thus learning. In thecontext of a rapidly evolving environment, we show thatinvolving the expert in this process for reinforcement,
classification or investigation provides significant
improvements in the learning process, which can then beconsidered a cognitive one.
Acknowledgements
We express our thanks to the colleagues who havecontributed to projects in which we developed theArtificial Immune Ecosystem: Véronique Thomas-© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan32

Harnessing over a Million CPU Cores to Solvea Single Hard Mixed Integer Programming Problem on a Supercomputer
Yuji Shinano

Mathematical Optimization and Scientific Information, Zuse Institute Berlin,
Takustrasse 7, 14195 Berlin, Germany*
E-mail: shinano@zib.de

Abstract
The performance of mixed integer programming (MIP) solvers has improved tremendously in the last two decades
and these solvers have been used to sol ve many real-word problems. ParaSCIP is the most successful parallel MIP
solver in terms of solving previously unsolvable instances from the well-known benchmark instance set MIPLIB byusing supercomputers. ParaSCIP has been developed by using the Ubiquity Generator (UG) framework, which is a
general software package to parallelize any state-of-the-art branch-and-bound based solvers. ParaSCIP is aparallelized MIP solver of a single thread solver SCIP. Since Xpress is a multi-threaded solver and ParaSCIP can run
at least 80,000 processes in parallel for solving a single MIP, ParaXpress could handle over a million CPU cores. In
this talk, a ground design of the UG framework and its latest extensions to harness over a million CPU cores will bepresented and preliminary computational results will be provided.
Keywords: mixed integer programming problem, massively parallel computing, SCIP, ParaSCIP, ParaXpress, UG

SlidesOutlineØ Background and Purpose

Harnessing over a Million CPU Cores
to Solve a Single Hard Mixed Integer
Programming Problem
on a Supercomputeru State-of-the-art Mixed Integer Programming (MIP) solversu Parallelization of MIP solvers

Ø Ubiquity Generator (UG) framework and ParaSCIP
Ø Computational results for solving previously unsolved MIP instanceson supercomputers
Ø How to harness over a million CPU cores
Ø Concluding remarks

Yuji Shinano
Zuse Institute Berlinug[SCIP, *]: joint work with Tobias Achterberg, Timo Berthold, Gerald Gamrath, Stefan Heinz,
Thorsten Koch, Stephen J. Maher, Daniel Rehfeldt, Stefan Vigerske, Michael Winkler
ug[Xpress,*]: joint work with Timo Berthold, Stefan Heinz
ug[PIPS-SBB,MPI]: joint work with Lluis Miquel Munguia, Geoffrey Oxberry, Deepak Rajan
UGS: joint work with Lluis Miquel Munguia

The 2018 International Conference on Artificial Life and Robotics

04/02/2018

The 2018 International Conference on Artificial Life and Robotics

1© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

33

2

Yuji Shinano

Background and Purpose

Progress in a state-of-the-art MIP solver

MIP (Mixed Integer Linear Programming)Ø minimizes or maximizes a linear function
Ø is subject to linear constraints
Ø has integer and continuous variables

Time limit:
10000 sec.
Test set:
3741 modelmin{ c€ x : Ax Æb, l Æx Æu, x j œZ, for all j œI }A œ R m ◊ n , b œ R m , c, l, u œ R n , I ™ { 1, . . . , n}

- 235 discarded
due to inconsistentanswers- 934 discarded
none of theversion can solve- speed-up
measured on>100s bracket;
1205 models

The most general form of combinatorial optimization problems
Many applications

The 2018 International Conference on Artificial Life and Robotics

The 2018 International Conference on Artificial Life and Robotics

3

6

Background and Purpose
MIP (Mixed Integer Linear Programming)Ø minimizes or maximizes a linear function
Ø is subject to linear constraints
Ø has integer and continuous variablesmin{ c€ x : Ax Æb, l Æx Æu, x j œZ, for all j œI }A œ R m ◊ n , b œ R m , c, l, u œ R n , I ™ { 1, . . . , n}
The most general form of combinatorial optimization problems
Many applications
MIP solvability has been improving

Development of a massively parallel MIP solver- can solve instances that cannot be solved by state-of-the-art MIP solvers- keeps catching up performance improvements of state-of-the-art MIP solvers
The 2018 International Conference on Artificial Life and Robotics
The 2018 International Conference on Artificial Life and Robotics

7

4

Parallelization of MIP solvers

Background and Purpose
MIP (Mixed Integer Linear Programming)

Branch-and-bound looks suitable for parallelization

Ø minimizes or maximizes a linear function
Ø is subject to linear constraints
Ø has integer and continuous variables

Ø MIP solvers: LP based Branch-and-cut algorithm

x1 £ 0x2 £ 0min{ c€ x : Ax Æb, l Æx Æu, x j œZ, for all j œI }A œ R m ◊ n , b œ R m , c, l, u œ R n , I ™ { 1, . . . , n}x1 ³1

x 2 ³1 x £ 0
5Obj.

x 5 ³1gap

The most general form of combinatorial optimization problems
Many applications
MIP solvability has been improving

Subproblem(sub-MIP)€

min{ c x : Ax Æb, l i Æx Æui , x j œZ, for all j œI }time

Subproblems (sub-MIPs) can be processed independently
Utilize the large number of processors
for solving extremely hard MIP instances(previously unsolved problem instances from MIPLIB)

The 2018 International Conference on Artificial Life and Robotics

The 2018 International Conference on Artificial Life and Robotics

5© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

34

8

Harnessing over a Million

Performance of state-of-the-art MIP Solvers

Dynamic load balancing is neededp Highly unbalanced tree is generated

Huge
performancedifference!

p Two types of irregularity can be handled well
§ Irregular # of nodes are generated by a sub-MIP

MIP solver benchmark (1 thread): Shifted geometric mean of results taken fromthe homepage of Hans Mittelmann (23/Mar/2014).
Unsolved or failed instances are accounted for with the time limit of 1 hour.

11,297,605

As of
14/April/2017§ Irregular computing time for a node solving

0.001sec
The 2018 International Conference on Artificial Life and Robotics

9

1.5h

The 2018 International Conference on Artificial Life and Robotics

12

GAMS and Condor:
M.R.Bussieck and M.C.Ferris (2006)

Solving techniques involved in SCIP

The 2018 International Conference on Artificial Life and Robotics

Real observation
for solving ds in parallelwith 4095 solvers

10

The 2018 International Conference on Artificial Life and Robotics© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan35

13

Yuji Shinano

Why can it handle large scale?
LoadCoordinatorOpen nodes:p

Solver 1

Solver 3

Solver 2

Solver 4

1240

Solver n"Solver26""Solver27""Solver28""Solver29""Solver3""Solver30""Solver31""Solver32""Solver33""Solver34""Solver35""Solver36""Solver37""Solver38""Solver39""Solver4""Solver5""Solver6""Solver7""Solver8""Solver9"Collecting mode Solver

1220

Objective Function Value

Solver 5"Incumbents""Optimal""GlobalLBs""Solver1""Solver10""Solver11""Solver12""Solver13""Solver14""Solver15""Solver16""Solver17""Solver18""Solver19""Solver2""Solver20""Solver21""Solver22""Solver23""Solver24""Solver25"

1200

The # of solvers at a time is restricted- Starts from 1- Dynamically switching

1180

1160

1140

The number is increased by at most
250 even if run with 80,000 Solvers

1120

Solver which has best dual bound node

1100

1080

0

50

100

150

200

250

Computing Time (sec.)

300

350

The 2018 International Conference on Artificial Life and Robotics

Dynamic load balancing
LoadCoordinatorOpen nodes:p

Solver 1

Solver 3

Solver 2

400

18

Layered presolvingmin{ c’€ x : Ax’ Æb,’ l ’ Æx Æu,’ x j œZ, for all j œI ’}

Solver 4

Solver 5

A: Original(sub-)problem

A’: Presolved(sub-)problem

A’: Original(sub-)problem

A’’: Presolved(sub-)problem

Solver n

Try to keep p open nodes in LoadCoordinator
Notification message: best dual bound, # nodes remain, # nodes solved- Send periodically and asynchronously- Interval is specified by a parameter
LoadCoordinator makes
selected Solvers in collecting mode
Expected to haveheavy nodes:
large subtree underneath

Global view of tree search

Global view of tree search

The 2018 International Conference on Artificial Life and Robotics

16

The 2018 International Conference on Artificial Life and Robotics

Dynamic load balancing
LoadCoordinatorOpen nodes:

p p*mp

Solver 1

Solver 3

Solver 2

Solver 4

1240

Objective Function Value

1220

Solver 5

Solver n"Incumbents""Optimal""GlobalLBs""Solver1""Solver10""Solver11""Solver12""Solver13""Solver14""Solver15""Solver16""Solver17""Solver18""Solver19""Solver2""Solver20""Solver21""Solver22""Solver23""Solver24""Solver25""Solver26""Solver27""Solver28""Solver29""Solver3""Solver30""Solver31""Solver32""Solver33""Solver34""Solver35""Solver36""Solver37""Solver38""Solver39""Solver4""Solver5""Solver6""Solver7""Solver8""Solver9"Collecting mode Solver

1200

-

1180

-

1160

Changes search strategy tobest dual bound firstSends requested number of nodes

1140

1120

Solver which has best dual bound node

1100

1080

0

50

100

150

200

250

Computing Time (sec.)

The 2018 International Conference on Artificial Life and Robotics

300

350

400

17

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

36

19

Harnessing over a Million

Check pointing

The biggest and the longest computation
Solving rmine10: 48 restarted runs with 6,144 to 80,000 cores
Incumbents
Global LBs

Titan with 80,000 cores
The others: HLRN III-1913-1913.5-1914-1914.5-1915-1915.5
-1916

0

1x10

6

2x10

6

3x10

6

4x10

6

5x10

6

6x10

6

7x10

It took about 75 daysand 6,405 years of
CPU core hours!

6

Computing Time (sec.)

How upper and lower bounds evolved

Only the essential nodes are saved
depending on run-time situation

Number of Nodes

1x10

90000

# nodes left
# active solvers# nodes in check-point file

10

80000
70000

1x10 8
1x10

60000

6

50000
40000

10000

30000
20000

100
1

10000
1x106

0

2x10 63x106

4x10 65x106

Number of Active Solvers + 1

Objective Function Value-1912-1912.5

UG canhandle up
to 80,000
MPI
process

0
7x106

6x10 6

Computing Time (sec.)

How open nodes and active solvers evolved
The 2018 International Conference on Artificial Life and Robotics

The 2018 International Conference on Artificial Life and Robotics

21

Restarting24

Combining with internal parallelization
ug[Xpress,MPI] : ParaXpressu A powerful massively parallel MIP solveru Can handle, hopefully efficiently, up to
80,000 (MPI processes ) x 24 (threads) = 1,920,000 (cores)

Connection network

ParaXpress
UG
Solverprocess
XpressUG
Solverprocess
XpressUG
Solverprocess
XpressUG
Solverprocess
XpressthreadthreadthreadthreadthreadthreadthreadthreadthreadthreadthreadthreadthreadthreadthreadthreadUG
Solverprocess
XpressthreadthreadthreadthreadHuge trees might be thrown away,
but the saved nodes’ dual boundvalues are calculated more precisely.

UG
Solverprocess
XpressthreadthreadthreadthreadthreadthreadthreadthreadOnly the essential nodes are saved
depending on run-time situation

UG
Solverprocess
XpressthreadthreadthreadthreadLoadCoordinator
Process
XpressShared Memory

Shared Memory

Processing Element or Compute node

Processing Element or Compute node

: CPU coreThe 2018 International Conference on Artificial Life and Robotics

22

The 2018 International Conference on Artificial Life and Robotics

Solving open instances of MIPLIB2010
Number of open instances solved

Main results for MIP solving by ParaSCIP

November 2015 triptim3 was solved with 864 cores in 9.5 hours
December 2015 rmine10 was solved with 48 restarted jobs
Titan: Cray XK7, Opteron 6274 16C 2.2GHz, Cray Gemini interconnect
ISM: Fujitsu PRIMERGY RX200S5HLRN II: SGI Altix ICE 8200EX (Xeon QC E5472 3.0 GHz/X5570 2.93 GHz)HLRN III: Cray XC30 (Intel Xeon E5-2695v2 12C 2.400GHz, Aries interconnect)
The 2018 International Conference on Artificial Life and Robotics

25

35

http://miplib.zib.de

30

3

25
20
15
10
5

MIPLIB2010
waspublished

75 open instances left

28

HLRN II -> HLRN III
7,000 cores -> 43,000 cores

4

1

6

7

ParaXpress
started

1

4

5

4

0
3

2014

2015

2016

2017

Not UG (commercial solvers and SCIP)

UG

0
2011

2012

2013

Year

New ParaXpress

2
0

The 2018 International Conference on Artificial Life and Robotics

23© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

37

26

Yuji Shinano

References
1. Y.Shinano: The Ubiquity Generator Framework: 7 Years ofProgress in Parallelizing Branch-and-Bound, ZIB-Report (1760), 2017.
2. Y.Shinano, S.Heinz, S.Vigerske,M.Winker: FiberSCIP - Ashared memory parallelization of SCIP, INFORMS Journal on
Computing, 30(1):11–30, 2018
3.Y.Shinano, T.Achterberg, T.Berthold, S.Heinz, T.Koch,
M.Winker: Solving Hard MIPLIB2003 Problems withParaSCIP on Supercomputers: An Update, Parallel &Distributed Processing Symposium Workshops (IPDPSW),
2014 IEEE International, pp.1552 – 1561, 2014
4. Y.Shinano, T.Achterberg, T.Berthold, S.Heinz, T.Koch,
M.Winker: Solving Open MIP Instances with ParaSCIP on
Supercomputers using up to 80,000 Cores, Proc. of 30th IEEE
International Parallel & Distributed Processing Symposium,
pp.770-779, 2016
5. Y.Shinano, T.Berthold, S.Heinz, A First Implementation of
ParaXpress: Combining Internal and External Parallelization
to Solve MIPs onSupercomputers, to appear ICMS 2016
proceedings.
6. Lluis-Miquel Munguia, Geoffrey Oxberry, Deepak Rajan,
Yuji Shinano: Parallel PIPS-SBB: Multi-Level Parallelism
For Stochastic Mixed-Integer Programs, ZIB-Report (17-58),
2017.
7. Gerald Gamrath, Thorsten Koch, Stephen Maher, Daniel
Rehfeldt, Yuji Shinano: SCIP-Jack – A solver for STP andvariants with parallelization extensions, Mathematical
Programming Computation, 9(2), pp. 231-296, 2017

Run different solvers with different
configurations in parallel
UGS: UG synthesizer• Runs many different solvers in parallelas MPMD(Multiple Program, Multiple Data) model MPI program
UGS
Mediatesolutionsharing

ugs
ugs
ugs
ugs
ugs
ugs
Xpress Xpress Gurobi CPLEX ParaXpress PAC_Xpress
config1
config1 config2 config1 config1 config1: B&B solvers: Heuristic solvers…

can be distributed
memory solvers

The first job:
Multiple ParaXpresses run with different configurations
with completely different solver implementations
sharing incumbent solutions
The following jobs:
Restarted from the most promising check pointing fileThe 2018 International Conference on Artificial Life and Robotics

28

Conclusions
UG is a general framework to parallelize any kind ofstate-of-the art branch-and-bound based solver.
ug[SCIP,* is tool to develop parallel general branchand-cut solvers. Customized SCIP solver can beparallelized with least effort.
ug[SCIP-Jack,*] is solver for Steiner Tree Problems
and its variants, namely only the solver which can run
on a distributed memory computing environment(solved three open benchmark instances.
ug[Xpress, MPI](=ParaXpress) and ug[PIPS-SBB,MPI] is ready to run on over a million CPU cores.
UGS is another general framework to configure aparallel solver that can realize any combination ofalgorithm portfolio and racing.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan38

Trust of Virtual Agent in Multi Actor Interactions
Halimahtun M. Khalid

Damai Sciences, A-31-3 Suasana Sentral, Jalan Stesen Sentral 5,
504700 Kuala Lumpur, Malaysia*
Liew Wei Shiung

Faculty of Computer Science and Information Technology, University of Malaya,50603 Kuala Lumpur, Malaysia
Voong Bin Sheng

Damai Sciences, Suite R26-11 Dua Sentral, No. 8 Jalan Tun Sambanthan,50470 Kuala Lumpur, Malaysia
Martin G. Helander

Damai Sciences, A-31-3 Suasana Sentral, Jalan Stesen Sentral 5,
50470 Kuala Lumpur, Malaysia
E-mail: mahtunkhalid@gmail.com, liew.wei.shiung@gmail.com, sh3ng527@gmail.com, mahelander@gmail.com

Abstract
Trust is crucial when integrating virtual agents in human teams. Our study investigated the combined use ofpsychological and physiological measures in predicting human trust of agents undertaking social tasks. The
psychological measures comprised trust scores on ability, benevolence and integrity. The physiological measures
included facial expressions, voice, heart rate and gestural postures. Subjects interacted with two avatars. A neurofuzzy algorithm extracted rules from the psychophysiological data to predict trust. Results revealed that trust can bepredicted with 88% accuracy.
Keywords: Trust, virtual agent, measurement, human-agent interaction.

measures were mapped to predict human trust of
humanoid agents in performing social tasks in a multiactor and bilingual contexts. In this instance, the role ofthe humanoid agent or avatar was to mediate between ateleoperator and clients using communication dialog ineither English or Bahasa Melayu (Malay language).
Therefore, the main objective of our study was toestimate human trust of the avatar in a business mediation
context. We hypothesized that trustor’s (evaluator) trustof the trustee (agent) is influenced by factors of gender,
ethnicity and trust components.

1. Introduction
Trust is a key element in the development of effective
human-agent-human relationships, as trust affects systemeffectiveness of safety, performance, and usability. With
the development and integration of virtual agents inhuman teams, the issue of predicting trust has become afocal concern1. One of the gaps in research is the lack ofa reliable measure of human-robotic trust. Past studies
have emphasized subjective measurements only2. We
present a method where subjective (general trust,
psychological) measures and objective (physiological)

2. Method© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

39

Halimahtun M. Khalid, Liew Wei Shiung, Voong Bin Sheng, Martin G. Helander

In this section, we describe the study as follow.

B. The experimenters and the server for recording data
were positioned in an observation room.

2.1. Subjects and location

2.3. Measures and ROS integration

Forty-eight subjects (25 males and 23 females) aged
between 21 to 46 years (mean age 26 ±5 years)
participated in the study. Twenty-two of the subjects
were Malays and 26 were Chinese. To achieve ecologicalvalidity the experiment was conducted in an office settingof a R&D studio.

We measured trust subjectively using criteria of: (a)General Trust and (b) Psychological Trust. General Trustis made up of three factors: Ability, Benevolence andIntegrity (ABI)3. For each factor there were four
attributes presented as semantic differential word pairs.
Altogether there were 12 attributes in the General Trust
scale as summarized in Table 1. These attributes werereplicated from previous studies4,52.2. Avatars and equipment
Two avatars were designed for the experiment. One
represented a male Chinese and the other a female Malay
using humanoid characteristics derived from an earlier
study. Figure 1 illustrates the avatars.

Table 1. General Trust as defined by three main categories,
each consisting of four polar attributes
Trust Category
Positive
Negative
Competent
Incompetent
Knowledgeable Not Knowledgeable
ABILITY
QualifiedUnqualified
SkilledUnskilled
Cheerful
Not Cheerful
FriendlyUnfriendly
BENEVOLENCE
Kind
UnkindPleasant
Unpleasant
Dependable
Undependable
EthicalUnethical
INTEGRITY
HonestDishonest
Reliable
Unreliable

Fig. 1 Male and female avatars

Microsoft Kinect V2 technology was used to record theaudiovisual data, and a computerized scale was used torecord subjective measures. Data was logged directly into
a server via a gigabit router and Ethernet cables, whichconnected the server to the laptops used by the subjects.
Headset microphones were used to record voices.

The above attributes in Table 1 were embedded in theinteractive dialog.

Fig. 2. Arrangement of experimental session

Figure 2 shows the layout of the experimental session,
which comprised of 3 rooms. Trustor A (Teleoperator)
was in Room A and Trustors B and C (Clients) in Room

Data collection and analysis were made possible via the
Robot Operating System (ROS) framework, which
integrated the agents, Graphical User Interface (GUI),© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan40

Trust of Virtual Agentsand hardware. The audiovisual and evaluation data wererecorded on the server computer acting as ROS master.
The avatar was controlled by a trigger signal sent bysubjects through the GUI.

The experimental design was a within-subjects design
with manipulations of gender, ethnicity, language andcompliance to instructions by teleoperator, clients andavatar.
The dialog comprised a communication script of aBusiness scenario that concerns establishing a factory.
This scenario was selected, as it was proven from theresults of previous studies to be the most effective than aDisaster or Healthcare scenarios1,2.

The objective data, which was obtained from the Kinectand headsets, were analyzed separately. The video, audio,
and evaluation recordings for each subject weresegmented based on the ROS timestamps corresponding
to the time when subjects evaluated each trust attribute.
The segments were then processed as follow: Voicerecordings were processed using PRAAT6 and Audio
Analysis Library7. Facial expressions were processedusing OpenFace8. Heart rate signals were estimated fromthe subjects’ faces using independent component
analysis9. Gestural postures were estimated from
Kinect’s joint coordinates. All extracted objective
features were then appended with contextual andsubjective information such as the trust evaluation score,dialog condition, and the avatar type.

2.5. Procedure

2.7. Trust classification

2.4. Experimental design and dialog

To predict trust, we mapped the objective features fromthe facial expressions, heart rate variability, voice, andgestures10 of subjects to their subjective scores, to create
a dataset. A neurofuzzy classifier ensemble method wasthen used to learn and predict subjects’ trust of the virtual
agents.

First, subjects were briefed on the experiment and theirtasks. Second, they signed an informed consent form andcompleted a subject profile sheet. Third, they weretrained on the GUI to familiarize them with the online
scoring method and dialog box.
Testing was done in a group of three subjects. The
subjects were randomly assigned to the role of Operator
or Clients based on a draw lot technique. Typically, theteleoperator in Room A initiated the interaction, followedby the robot confirming its task, and interacting with
Trustor B and Trustor C in Room B, who then responded
to the avatar, which in turn communicated back to theteleoperator. Subjects were instructed to act out the scriptin their respective roles. To test for systems operability
and usability, a pilot study with two groups of subjects
was carried out prior to the actual experiment.
Experimenters were positioned in a separate observation
room and monitored the experiment via a screen and aheadset. The computer server was placed in theobservation room to record data in real time from theKinects, headsets, and subject evaluation forms.
After completing each application scenario, subjects
were given a 10-minutes break to complete a post-hoc
questionnaire, which assessed their opinions andpreferences regarding the avatar and its features.

3. Results and Discussion
The results from MANOVA and correlation analyses
revealed the following.
3.1. Gender and ethnicity
The gender and ethnicity of the trustee (virtual agent) had
no significant effect on trust. But the interaction effects
of gender and ethnicity of the trustors (human subjects)
influenced their evaluations of the trustees, with F (1,2304) =18.02, p<.001. Gender-wise, female subjects
were more trusting, giving higher trust scores than malesubjects, F (1,2304)=24.69, p<.001. This finding
confirmed our hypothesis on the influence of gender andethnicity in trust evaluations.
However, ethnicity alone had no significant effect. Itsinteraction with trust components produced a significant
effect, F (2, 2304)=5.01, p=.007. On average, Chinese
subjects scored the agents highly on ‘Ability’ and‘Integrity’ attributes, while Malay subjects scored theagents highly on the ‘Ability’ attributes only.
In addition, there were significant effects of intraethnicity on trust, with F (2, 2304)=4.17, p=.016. When

2.6. Data analysis
We performed factor analysis, MANOVA and Pearson
Correlations on the subjective data to test the hypotheses.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

41

Halimahtun M. Khalid, Liew Wei Shiung, Voong Bin Sheng, Martin G. Helander

the groups consisted of just Chinese subjects, theevaluation of trust was higher than when the groups
consisted of Malays only, or when it represented a mixedethnic composition. However, the gender composition of
the group had no significant effect on trust evaluations.
This finding suggests the importance of cultural andethnic composition of teams in trust-related tasks.

Acknowledgment

3.2. Subject preferences

1. P. Robinette, A. M. Howard, and A. R. Wagner, Effect ofrobot performance on human–robot trust in time-criticalsituations, in IEEE Trans. Hum. Mach. Syst. 47(4) (2017)425–436.
2. A. Steinfeld, T. Fong, D. Kaber, M. Lewis, J. Scholtz, A.
Schultz, and M. Goodrich, Common metrics for humanrobot interaction, in Proc 1st ACM SIGCHI/SIGART Conf.
Hum Rob Interact (Salt Lake City, Utah, USA, 2006), pp.
33–40.
3. R. C. Mayer, J. H. Davis, and F. D. Schoorman, An
integrative model of organizational trust, in Acad. Manage.
Rev. 20(3) (1995) 709–734.
4. H. M. Khalid, W. S. Liew, P. Nooralishahi, Z. Rasool, C.
K. Loo, and M. G. Helander, Predicting trust in social
communication: Implications for human-robot interaction,
in Proc. HFES 2016 Int. Annu. Meeting (Washington,
USA, 2016), pp. 19–23.
5. H. M. Khalid, W. S. Liew, P. Nooralishahi, M. G.
Helander, and Z. Rasool, Toward a theory ofpsychological trust for human-robot-human interaction:
Effects of scenarios, gender, and ethnicity, in Proc.
SEANES Int. Conference (Bandung, Indonesia, 2016), pp.
321–330.
6. P. Boersma, Praat: doing phonetics by computer, in
http://www. praat. org (2006).
7. T. Giannakopoulos and A. Pikrakis, Introduction to Audio
Analysis: A MATLAB Approach, in Academic Press(2014).
8. T. Baltrušaitis, P. Robinson and L. P. Morency, Openface:an open source facial behavior analysis toolkit, in 2016
IEEE Winter Conf Appl Comput Vis (Lake Placid, New
York, USA, 2016), pp. 1–10.
9. M. Lewandowska, J. Rumiński, T. Kocejko, and J. Nowak,
Measuring pulse rate with a webcam—a non-contact
method for evaluating cardiac activity, in 2011 Federated
Conference on Computer Science and Information
Systems (Szczecin, Poland, 2011) pp. 405–410.
10. H. M. Khalid, W. S. Liew, B. S. Voong, M. G. Helander
and C. K. Loo, Technology for estimating trust in humanrobot interaction, in 4th Asian Conf. Def. Tech. (Tokyo,
Japan, 2017).

We gratefully acknowledge the financial support by the
US Aerospace Research and Development office(AOARD), Japan and the US Air Force Office of
Scientific Research (AFOSR), Washington D.C. underGrant No. FA2386-14-1-0016.
References

An analysis of the post-experiment questionnaire
revealed that subjects’ trust evaluation scores correlated
significantly with their explicit Trust, r=0.09, p<.001,
and Liking preferences for the agents, r=0.06, p=.006.
Subjects who trusted one or more agents explicitly scoredhigher than subjects who explicitly trusted neither agent,
F (3,2304)=7.81, p<.001.
In addition, subjects who explicitly liked the avatars
scored highest; subjects who explicitly disliked theavatars scored second highest, and subjects who were
undecided scored lowest, F(2,2304)=3.681, p=.025.
3.3. Trust prediction
Using the neurofuzzy classifier ensemble method10, wewere able to predict subject’s trust in humanoid agents,
with an accuracy of 88.3% + 0.2% (F1-score 0.8401),
when all physiological data was used. Greater accuracy
at 90% was achieved when gestural posture data was
excluded, whether for Low, Medium, or High Trust.
Details of the analysis are described elsewhere10.
4. Conclusion
This study has shown that virtual agents can be trusted toperform social tasks using communication dialogs in a
multi-actor and bilingual contexts. The trust by humans
varies with the gender and ethnicity of the evaluator. Also,
trust can be easily evaluated for attributes of ‘Ability’ and‘Integrity’ using dialogs than for ‘Benevolence’. Thisimplies that our design and modeling of the humanoid
agents requires further anthropomorphism to appeal tohumans, persona-wise.
Integration of the various data via ROS allows for realtime annotation of the data recordings. This enables
segmentation of the relevant objective data to each trustscore evaluation. As such, the accuracy of the neurofuzzyclassifier was enhanced to predict trust.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan42

Production simulation of autonomous decentralized FMS
including AGVs with different personalities of mind
Ryuichi Tsujii

Department of Human Information Systems, Gifu University, Yanagido 1-1,
Gifu, Gifu 501-1194, JAPAN
Hidehiko Yamamoto

Department of Mechanical Engineering, Gifu University, Yanagido 1-1
Gifu, Gifu 501-1194, JAPANTakayoshi Yamada

Department of Mechanical Engineering, Gifu University, Yanagido 1-1
Gifu, Gifu 501-1194, JAPANE-mail: v3128018@edu.gifu-u.ac.jp, yam-h@gifu-u.ac.jp, yamat@gifu-u.ac.jp

Abstract
This study controle Automated Guided Vehicle(AGV) moving by using mind model in order to avoid AGVs’interference. The mind model can avoid the interference by repeating the two types of mind changes, the arrogant
mind and the modest mind. In this study, further, provide multiple personality in the mind of the model. By themind of the model of this multiple personality, we improve the increase of the production out-puts.
Keywords: Autonomous decentralized FMS, AGV with a mind, Different personality

1. Introduction

2. Structure of Autonomous Decentralized FMS

We’ve developed the computer control systems to realize
a decentralized autonomous flexible manufacturingsystem (FMS). To do this, it is necessary to avoid the pathinterference of automated guided vehicles (AGVs) in adecentralized autonomous FMS. We controlled AGV's
behavior by using if-then rules. However, as the numberof AGVs increases, it is difficult to use the rules foravoiding route interferences in advance. Therefore, we
install the model of mind to AGVs and research the
behavior control of AGVs. In this research, we givedifferent personalities to this model of mind, and perform
production simulation using AGV group of different
personalities, and investigate the influence on different
personalities of mind to production efficiency.

2.1. Outline of Autonomous Decentralized FMS
An example of an autonomous decentralized FMS
factory is shown in Fig.1. The factory floor is divided intoa grid pattern where AGV moves on the line. The system
of automated factory performs the production by carrying
the parts to the warehouse or machining centers (MCs).
The decentralized autonomous FMS does not have amanagement mechanism that controls the whole system.
The agents (MCs, AGVs and Warehouses) which are thecomponents of the FMSs exchange knowledge anddetermine the behavior of each agent.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

43

Ryuichi Tsujii, Hidehiko Yamamoto, Takayoshi Yamada●When AGVs with an arrogant mind enter the sameintersection, one of the AGVs changes its mind to amodest heart and gives way.
●When AGVs with a modest mind enter the sameintersection, one of the AGVs changes its mind to anarrogant mind and the other AGV gives way.
In order to express the mind, we use the model ofmind shown in Fig.3, Mini-mum Unit of Mind (MUM)
Fig. 1. Example of autonomous decentralized FMS

2.2. Model of Mind and Behavior of AGV
In the autonomous decentralized FMS, AGVs
autonomously determine the behavior. In order to avoid
path interferences between AGVs, the conventional
research used the if-then rules. However, it was notpossible flexibly to cope in the case of increasing thenumber of AGVs and changing the layouts of the factory.
Therefore, in this research, in order to solve this, we
introduce a model of mind to AGV and realize
autonomous movement without AGVs’ interferences.
The mind of AGV expresses two kinds of anarrogant mind and a modest mind as shown in Fig.2.
Specifically, AGVs with an arrogant mind takes actions
forcibly to approach the destinations and AGV with amodest mind takes action to give way to other AGVs.

Fig. 3. MUM. A1 and A2 called as unit, X is load, and an arrow is a
stimulation vector. The threshold is determined by theunit. If the internal value reaches the threshold, we call it"excited". And if it does not reach the threshold, we callit "normal”. When a signal is sent to the unit, it sends asignal to the direction of the arrow if it is excited, and it
is not sent if it is normal. Load has the function to change
the internal value of the unit. When the signal is sent to
the load, the value of the unit is increased by the value of
X. Stimulation vector is a line connecting the load andthe unit, and it gives a signal to the load or unit if thesignal comes. When A1 is kept excited, we call it "modest
mind", and when normal, we call it “an arrogant mind”.
By the amount of the signal input to the unit, MUM
changes the mind to arrogant or modest frequently. Next,
we describe the internal functions of MUM. When thearrogant AGV has path interference, A1 is increased by
1. Keeping the situation of the interference and being
increased by 1, the A1 value becomes the threshold, and
the AGV is changed to a modest one. When AGVs with[1]

Fig. 2. Example of changing mind

By switching minds between these two kinds ofmind, so-called, changing mind, AGVs perform threeactions as follows.

a modest mind keep giving a way, the value of A2 isincreased by 1. When the situation is repeated at
optional times, A2 becomes excited and a signal is sent
to a load. The received load decreases the values ofunits A1 and A2 by optional values. Owing to this, A1
and A2 are returned to normal and a modest mind is● When the AGV with an arrogant mind and the AGV
with a modest mind enters the same intersection, theAGV with a modest mind gives way and the AGV withan arrogant mind is given the priority to enter theintersection.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

44

Production simulation of autonomouschanged to an arrogant mind. In this way, AGV avoids
the path interference by the change of heart.

that of A. Therefore, it frequently changes mind. C:
Personality hard to change mind is given a large of both
unit A1 and unit A2. This personality does not change themind much. D: Personality easy to maintain an arrogant
mind is given the conditions unit A1 is large and unit A2
is small. This makes it easy to maintain an arrogant mind.
E: Personality easy to maintain honest mind is given theconditions unit A1 is small and unit is A2 large. This
makes it easy to maintain honest mind.

3. Personalities of AGVs
In this study, Personalities of AGVs are defined by thedifference of the time from a modest mind to an arrogant
one or from an arrogant mind to a modest one. In actual
human society, there are people with various personality
and they are doing cooperative actions. By giving a big
difference of personalities to AGVs like human beings,
we try to improve the efficiency of production. This is
like bringing the behavior of AGVs closer to humansociety.
In order to achieve this, by assigning a large or small
difference to the threshold of the MUM unit, we make thedifference of personality, that is the changing time
becomes different. For example, as the value of the unit
threshold increases, the number of path interference
increases. As a result, the time to start changing time
becomes long. In the contrary, as the value of the unit
threshold decreases, the number of path interference
decreases. As a result, the time to start changing time
becomes short.
The followings are five personalities with a standard
personality and big different personalities. The previous
personality corresponding to the small, changing mind
time is also described.
A: Standard personality
B: Personality easy to change mind
C: Personality hard to change mind
D: Personality easy to maintain an arrogant mind
E: Personality easy to maintain honest mind

4. Simulations and results
The simulation of FMS operation is carried out by AGV
group of single personality adopting only one personalityof A-E and AGV group of different personalities
combining personality of A-E. Then, we examine theinfluence on production efficiency by comparing the caseof AGV group with single personality and different
personalities.
4.1. Simulation conditions
Apply personality to model of mind of 9 AGVs in FMS
and run 8 hours simulations’ 10 times. The following
combination of personality is applied to nine AGVs.
●AGV group of single personality· Combination ① A×9 units· Combination ② B×9 units· Combination ③ C×9 units· Combination ④ D×9 units· Combination ⑤ E×9 units● AGV group of different personalities· Combination ⑥ A×3 units, B×3 units, C×3 units· Combination ⑦ A×5 units, B×2 units, C×2 units· Combination ⑧ A×2 units, B×5 units, C×2 units· Combination ⑨ A×2 units, B×2 units, C×5 units· Combination ⑩ A×3 units, D×3 units, E×3 units· Combination ⑪ A×5 units, D×2 units, E×2 units· Combination ⑫ A×2 units, D×5 units, E×2 units· Combination ⑬ A×2 units, D×2 units, E×5 units

Table 1 shows the values of thresholds for unit A1
and unit A2 of five personalities.
Table.1. Threshold values of MUM personality

A
B
C
D
E

Unit A1
10～14
5～9
20～24
20～24
5～9

Unit A2
30～34
15～19
60～64
15～19
60～64

4.2. Simulation results
As a result of the production simulation, Table 2
compares the production numbers and Table 3 shows thecomparison of the average occupancy rates of the MCs.
Compared with the combination ① which is the standard,
the number of production decreased in the single
personality AGV group. In the AGV group of different
personality, the combination of A, B and C increased the

A: Standard personality is the personality used in theprevious research. B: Personality easy to change mind isgiven a smaller value of both unit A1 and unit A2 than

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

45

Ryuichi Tsujii, Hidehiko Yamamoto, Takayoshi Yamada

number of production outputs, the combination of A, D,
E greatly decreased.

production efficiency declined. In the AGV group(combination ⑥ - ⑨ ) which combined the standard
personality, personality easy to change mind andpersonality hard to change mind, the number of
production slightly increased. In the case of combination⑦ , the production number was the highest, and thenumber of production increased by an average of 5.7
compared with the combination ① which is the standard.
This is because by combining AGVs with different
threshold. It is thought that the timing of change of mind
shifted in each AGVs and it was possible to effectively
eliminate the path interference. However, in the AGV
group (combination ⑩ ~ ⑬) which combined standard
personality, personality easy to maintain an arrogant
mind, and personality easy to maintain honest mind, thenumber of production dropped sharply. As the number of
AGVs with personality easy to maintain an arrogant mind
increased, the number of production declined further.
Just like the combination ④, this personality does notgive way to the concession of the route, so it seems thatthe route interference could not be resolved

Table.2. Simulation results of production outputs

Combination of Production Rate of
AGV group
outputs increasecombination1 A9
274.5
1.000
combination2 B9
273.9
0.998
combination3 C9
271
0.987
combination4 D9
90
0.328
combination5 E9
268.9
0.980
combination6 A3B3C3
280.2
1.021
combination7 A5B2C2
280.7
1.023
combination8 A2B5C2
278.9
1.016
combination9 A2B2C5
279.1
1.017
combination10 A3D3E3
193.9
0.706
combination11 A5D2E2
249.4
0.909
combination12 A2D5E2
94.1
0.343
combination13 A2D2E5
260.5
0.949
Table.3. Simulation results of MC efficiency

Combination ofRate ofMC efficiency
AGV group
increasecombination1 A9
0.367
1.000
combination2 B9
0.365
0.994
combination3 C9
0.361
0.984
combination4 D9
0.126
0.344
combination5 E9
0.358
0.976
combination6 A3B3C3
0.371
1.011
combination7 A5B2C2
0.370
1.007
combination8 A2B5C2
0.370
1.007
combination9 A2B2C5
0.367
0.999
combination10 A3D3E3
0.264
0.718
combination11 A5D2E2
0.334
0.909
combination12 A2D5E2
0.132
0.361
combination13 A2D2E5
0.348
0.947

6. Conclusions
In autonomous decentralized FMS which adopted AGVs
with different personalities, production number slightly
increased. This is because the AGV group with different
personalities performed cooperative action more
efficiently than the AGV group with single personality.
From this result, it was found that it is effective to use
models of minds of different personalities in autonomous
decentralized FMS. In this research, we performed
simulations with limited combinations of personalities,
but there are infinite number of other combinations. How
to obtain the optimum combination of personalities is our
future task.

5. Considerations

References

Compared combination②-⑤ with combination① (single
personality AGV group using standard personality), theproduction number declined in any other single
personality AGV group. Especially, in combination ④(Personality easy to maintain an arrogant mind), thenumber of production dropped sharply. As thispersonality tends to maintain an arrogant mind, it isthought that AGVs did not give way to the route so much,the route interference could not be resolved, and the

1. Hidehiko YAMAMOTO, Takayoshi YAMADA and
Shinsuke KATO, AGV Mind Model and its Usage for
Decentralized Autonomous FMS by Change of Mind,
Proceeding of Third KES International Symposium, KESAMSTA 2009, Agent and Multi- Agent Systems : Technologies
and Applications, Lecture Notes in Artificial Intelligence,
Edited by Anne Hakansson, Ngoc Thanh Nguyen etc. , ISSN
0302-9743, Uppsala, Sweden, pp.744-753, June(2009),
Springer.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan46

Development of Automatic Recognition of Hazmat Marking Chartfor Rescue Robot
Wisanu Jitviriya, Poommitol Chaicherdkiat, and Noppadol Pudchuen

Faculty of Engineering, King Mongkut’s University of Technology North Bangkok.
1518 Pracharat 1, Wongsawang Rd., Bangsue, Bangkok, Thailand 10800
wisanu.j@eng.kmutnb.ac.th, poommitol.c@gmail.com, and noppadol.pudchuen@gmail.com
Eiji Hayashi

Mechanical Information Science and Technology, Kyushu Institute of Technology.
680-4, Kawazu, Iizuka, Fukuoka, 820-8502, Japan.
haya@mse.kyutech.ac.jp

Abstract
A long history of first place awards in World RoboCup Rescue Robot competitions is Invigorating Robot Activity
Project (iRAP) such as iRAP_PRO, iRAP_FURIOUS, iRAP_JUNIOR and iRAP_ROBOT. In this paper, we would
like to introduce and explain an autonomous system of our rescue robots for detection and recognition of hazardous
material (hazmat) marking charts. All hazmat tags are considered and computed by using Speeded-Up Robust Feature(SURF) combined with Fast Library for Approximate Nearest Neighbors (FLANN) to match with the templates.
Finally, the paper presents experimental results based on real situations to confirm an effect of the pattern recognitionof robotics.
Keywords: Hazmat marking chart, SURF, FLANN and Rescue robot.

1. Introduction
Invigorating Robot Activity Project (iRAP) is therobotics club of students from King Mongkut’s
University of Technology North Bangkok, Thailand.
Every year, we participate in the World RoboCup Rescue
Robot competitions, and our teams have won first place
many times in the competitions [1]. Our rescue robot canbe divided into three major parts such as the mechanicalrobot part, the electrical system part and the softwaresystem part. The unique point of the iRAP robot is thehuman-robot interface (the operator console), which canshow and present extensive sensor values and visual data
(CO2 sensor, visual temperature sensor, laser-scanner,
IMU sensor and four cameras) in real time as shown in
Fig.1. Moreover, our rescue robot is designed based on
proficiency by the agility and performance tests.

Fig.1 Operator console of iRAP ROBOTTherefore, the robots can move over rough terrain byusing a caterpillar module combined with four flipper
modules. The advantage point of flipper parts is that they
can help the robot to balance and go through all kinds of© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan47

Wisanu Jitviriya, Poommitol Chaicherdkiat, Noppadol Pudchuen, Eiji Hayashi

Fig.2. iRAP Rescue Robot

obstacles. In this paper, we would like to develop andimprove the software system by adding an automatic
recognition of Hazmat tags. The recognition model isproduced and computed with Speeded-Up Robust
Feature (SURF) [2], [3] combined with Fast Library for
Approximate Nearest Neighbors (FLANN) [4], [5]. The
results of detection and recognition will be presented andverified in a real situation.

Fig.3. Initialization phase of the proposed method

2. Structure of iRAP Robot
The structure of the iRAP rescue robot is an original
platform for the rescue robot which has two pairs offlippers (in the front and in the rear) as shown in Fig. 2.
In addition, the 6-DOF robotic arm (1 prismatic joint and5 revolute joints) is set up on a body of the robot, tosearch and observe the victim and situation in thesurrounding area. The multi-sensors are installed in theend-effector of robotic arm, and also, the camera on the
robotic arm is used for recognizing the Hazmat chartautonomously.
3. Automatic Recognition of Hazmat Chart
The proposed paper that presents the SURF method
for extraction the information from the image and thenusing the FLANN model, which is used to match the
input image with Hazmat chart templates. The overview
system can be divided in two phases: the initialization
phase (Fig.3) and real-time processing phase.

Fig.4. Hazmat chart templates

Pre-processing image
In the pre-processing image, there is common task to
improve or adjust the feature of image (scale, color, edge,
area etc.).
Feature extraction
The SURF algorithm is applied to the feature
extraction. It consists of detector (Hessian matrix), as
expressed and descriptor schemes (Haar wavelet). The
determinant of a hessian matrix in Equation (1) shows theexpression of the local change around the area, where

3.1. Initialization Phase
Load all template images
The first part of initialization phase is to input the all
the Hazmat chart templates. There are twelve templates
for training the proposed methods as displayed in Fig. 4.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan48

Development of Automatic Recognition

H ( x, ) is the hessian matrix, as illustrated in Equation(2), and Dxx ( x, ), Dxy ( x, ) in Equations (3) and (4) are
the convolution of the integral image and the second
derivative of the Gaussian function.

det( H approx )H ( x, )Dxx Dyy (wDxy )2 d

Dxx ( x, ) Dxy ( x, )Dxy ( x, ) Dyy ( x, )Dxx ( x, )I ( x) *Dxy ( x, )I ( x)*

2

x2
2

xy

(1)

Fig.5. The wavelets response. Black and white areas show the weight-1 and 1 for the Haar kernels.

(2)

g( )

(3)

g( )

(4)

After having create the response map, the next step is toconsider a non-maximum suppression based on thedeterments of the hessian matrix that is the objective of
the SURF detection.

Fig.6. Real-time processing phase

Compute descriptors
At each interest point, we have to indicate the uniquedescription of a feature that do not depend on the size androtation. The SURF descriptor is based on the Haar
wavelet model that can be calculated efficiently withintegral images. The wavelets response in the x and y
direction is defined as dx and dy respectively as shown in
Fig. 5. For each sub-area of a vector, descriptors arecalculated based the Equation (5)

vsub

dx,

dy,

dx ,

dy

In matching process, FLANN is used to classify the
input image by comparing it with 12 sample feature
images. FLANN is open source of a library model for
performing fast approximate nearest neighbors. It is now
one of the most popular model for nearest neighbormatching.
3.2. Real-time Processing Phase
The real-time processing phase (Fig.6) is similar to
an initialization phase. First, the acquired image is input
into the main process. The SURF method is used to
extract the interest points and compute the descriptors.
And then, the descriptors of the input image arecompared with all sample images. The similarity score of
the keypoint is calculated in each class, Finally, the input
image is specified in the winner class, which is the
maximum similarity score. Then, the position of thehazmat image is in the frame that will be considered to
calculate the centroid of an interest area.

(5)

Concatenate descriptors
In this sub-process, it is created by the new platformof descriptors to the sample feature. Then, we have 12
sample feature descriptors to match with the input image.
Build FLANN method from concatenate
descriptors© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

49

Wisanu Jitviriya, Poommitol Chaicherdkiat, Noppadol Pudchuen, Eiji Hayashi

4.

Experimental results

The experimental results were verified with the
Visual Studio C# program and an open source librarysuch as Emgu CV library. In case there is only one
hazmat image in the frame, the results of the proposed
method could recognition and detection autonomously, it
did not depend on the size of input image and rotation asshown in Fig.7. In addition, we had more testing of the
proposed method by adding 4 hazmat charts in the input
image as displayed in Fig.8. The results that confirm the
proposed method can classify and identify the hazmat
chart accurately and correctly.
5. Conclusion
This paper presents the development of the automatic
recognition of a hazmat marking chart for rescue robots.
The results that affirm the proposed method can be usedin detection and recognition in the real situation. In the
future work, we will improve not only the speed of therecognition system but also the accuracy of method.

Fig.7. One hazmat chart matching

6. References
1. A. Phunopas, A. Blattler and N. Pudchuen, “iRAP Robot:
World RoboCup Rescue Championship 2016 and Best inClass Mobility Award,” the Lecture Notes on Artificial
Intelligence, Springer book on RoboCup 2016, volume9776.
2. S. U. R. F. (SURF), “Herbet bay, andreas ess, tinne
tuytelaars, luc van gool,” Elsevier preprint, 2008.
3. Wikipedia, “Blob detection — Wikipedia, the freeencyclopedia,” 2011, [Online; accessed 14-July-2010].
[Online],Available:https://secure.wikimedia.org/wikipedi
a/en/wiki/Blob detection
4. Marius Muja and David G. Lowe: "Fast Matching ofBinary Features," Conference on Computer and Robot
Vision (CRV) 2012.
5. Marius Muja and David G. Lowe, "Fast Approximate
Nearest Neighbors with Automatic Algorithm
Configuration", in International Conference on Computer
Vision Theory and Applications (VISAPP'09), 2009.

Fig.8. Four hazmat charts matching© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan50

Development of Autonomous Robot for Laborsaving of the Forestry
Discrimination between trees and weeds using RGB-D
Shingo Yamaguchi

Department of Mechanical Information Science and Technology, Kyushu Institute of Technology
680-4, Kawazu, Iizuka-City, Fukuoka, 820-8502, Japan
Email: yamaguchi@mmcs.mse.kyutech.ac.jp
Eiji Hayashi

Department of Mechanical Information Science and Technology, Kyushu Institute of Technology
680-4, Kawazu, Iizuka-City, Fukuoka, 820-8502, Japan
Email: haya@.mse.kyutech.ac.jp
Ayumu Tominaga

Department of Mechanical Information Science and Technology, Kyushu Institute of Technology
680-4, Kawazu, Iizuka-City, Fukuoka, 820-8502, Japan
Email: tominaga@.mmcs.mse.kyutech.ac.jp

Abstract
In traditional forestry, the weeding task places a large burden on workers. Therefore, it has been expected that suchthe problem will be solved by a robot. We have been developing an autonomous robot that eliminates the weed
plants without damaging trees (these are resources). The robot should recognize trees exists in the environment. In
this research, we have been focused on distinguishing between trees and other them. The system computes the RGB
histogram of pixels corresponding point-cloud data of the ground extracted by plane detection. The threshold values
are determined to discriminate between trees and weeds by this histogram automatically.
Keywords: Autonomous Self-Moving, Depth Sensor, Forest, Image processing

true for Japanese forestry, which has a remarkable delay
in mechanization compared to other countries. In theNordic region, which includes countries with a very
developed forestry industry, harvesting using heavymachinery has improved not only the production
efﬁciency but also worker safety. However, in Japan’s
forestry, most deployed areas are mountainous forest
areas that are steep with a rough terrain. In addition, theplanting density can be as high as 3000 per hector (theworld standard is 0), and the standard planting interval
between two trees must be 2.5 m. For this reason, it is
difﬁcult for large-scale heavy machinery introduced inother countries, such as harvesters, to enter the forest

1. Introduction
In primary industries such as agriculture, forestry, andfishery, labor shortages caused by the aging worker
population and the declining employment rate has been aproblem. To solve this issue, interest in and expectations
for ﬁeld robots working in the natural environment has
been increasing, and it is expected that robotics will be
introduced to the agricultural industry in the future.
Although many investigations and experiments in realﬁelds have already been accomplished for robots inagriculture, the mechanization and modernization of theforestry industry has been stagnant. This is especially© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan51

Shingo Yamaguchi, Eiji Hayashi, Ayumu Tominaga

backland; thus, most of the work must be performed
manually by workers.
In this research, we focus on the introduction of ﬁeld
robots to the forestry industry.
The robot we have developed is a compact andautonomous robot that can enter the forest hinterland.
The main task of this robot is weeding with autonomous
motion. In traditional forestry, weeds are primarily
eliminated by workers, and the workers must movearound all areas of a vast plantation. Grass weeding is an
essential task for promoting the growth of trees, which
are essential resources, and for improving underfoot
visibility. Ideally, weeding must be done more than twice
in a summer term, but it is a severe task that is performed
under the blazing sun. Thus, automating weeding is anurgent task for improving safety and productivity inforestry.
Our research has focused on developing a robot
platform, weeding mechanism, and autonomous moving
system. The system is based on an all-terrain vehicle(ATV), and obstacle avoidance in the forest is realized bydetecting the external environment with a depth sensor.
In this study, we describe the way of discriminating
between trees and weeds in a real environment.
Discrimination between trees and weeds is an importantfactor for an autonomous mobile robot to eliminate
weeds without damaging trees. Therefore, we havedeveloped the system that discriminate between trees andweeds using RGB-D data.

is 15 m. In addition, the robot has an inertial
measurement unit (IMU) for pose detection and
RTKGPS for global localization. The power source is agasoline engine. The brake lever, accelerator lever, andsteering are all controlled by DC motors. A weeding task
is performed by a weeding mechanism that comprises amulti blade structure attached to the front of the ATV.

Fig.1. Robot appearance
3. Autonomous moving system
One of the required functions for the robot is to avoid
collisions with the trees (especially young trees, which
are the objects to be raised) while eliminating weeds.
Collision avoidance is realized by the 3D point cloudobtained from the depth sensor. The process ﬂow for thetree-avoidance behavior is shown in Fig.2. Here, thedepth image is a gray-scale image expressing the depth
information as shades of color, and the top-view image
represents the result of projecting the 3D point-cloud dataon a two-dimensional (2D) plane. The location
information of objects is obtained by processing labels of
the top-view image, after which the spacing between two
objects that the robot can pass through is computed.
Object avoidance is realized by ensuring that the robot
constantly aims at the middle point of this space between
objects.
Once the robot reaches the target point between theobjects, the process returns to the observation processusing the depth sensor. The judgment of when the target
point is reached is performed using the traveling distancemeasured by a rotary encoder attached to the rear-wheel
drive shaft.

2. The robot platform
Fig.1 shows the robot developed in this study. The
actual mountain hillside where we will operate the robot
is a steep slope, and the road is rough. Therefore, therobot is based on an ATV (Kawasaki, Inc.) to handle therough terrain. The robot measures an area that is 1400mm long, 900-mm wide, and 1200-mm high. In theJapanese forest industry, the planting interval isapproximately 2.5 m, which is narrow compared toforestry regulations in other countries. As shown in Fig.1,
the size of our developed robot is sufﬁciently small
compared to the planting interval that it can travel into
the hinterland of the mountain forest. The robot has four
wheels, of which the front two are turning wheels and therear two are driving wheels. The external environment isrecognized by a depth sensor and a charge-coupleddevice camera equipped on the ATV. The depth sensor
can acquire an object’s position as three-dimensional(3D) point-cloud data via an infrared laser measurement.
The maximum measurable depth direction of the sensor© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan52

Development of Autonomous robot)

Fig.3. Flow-chart of discrimination system
A. Change luminanceIn many case, both weeds and trees are green. Therefore,
the values of Red, Green, and Blue are similar, whichmakes it difficult to discriminate. Therefore, we defined
the luminance rate as P which is calculated from acquired
data. We solve the problem by using this value. The
luminance level P is calculated by using luminance value
Y that is expressed eq.3 and expressed eq.4.
= 0.30 + 0.59 + 0.11
(3)P=
∗ 100 (4)Fig.2. Flow-chart of autonomous moving system
When the target position is determined, the list of thecentroid coordinates of the observed objects, expressedas shown in eq.1, and the control input for the robot,
denoted by eq.2, are recorded for the localization system.
Here, λ represents the steering angle andis thedistance that the robot should travel to reach the target
position at each discrete time t. The steering angle is notchanged until the traveling distance measured by therotary encoder satisﬁes.
= ̅ , , , = 0, 1, …
(1)
[λ
]
=
(2)

B. Divide the data of the tree from ground
Many weeds are present on the ground. Therefore, inthis system, we first divide the data into the ground andthe other. we divide the acquired data into the ground andother parts by using plane detection with Point Cloud
Library, and we acquire the ground and other RGB data.

C. Compute the tree region in RGB histogram
In this system, we compute the tree region in eachhistogram of Red, Green and Blue by using data that isbisected by plane detection. It is only necessary for the
robot to discriminate whether the object in front of robot
is trees or not. Therefore, we use Otsu’s discriminant
analysis method for computing the tree region. Thatmethod computes threshold value for bisecting thehistogram[1]. Fig.4 is an example using this method. In
Fig.4, shown the histogram of red, green and blue fromleft to right. The black line in the figure is the thresholdvalues that is computed. In each histogram, it is definedthat density value is from 0 to threshold value is class A
and larger than threshold is class B. As in the Fig.4, using
only this method, we do not know which of class A or

4. Discrimination between trees and weeds
The method for discrimination between trees and weeds
is necessary so that the robot will not damage the trees
while perform a weeding task. Therefore, we propose
method using RGB-D data to differentiate trees fromweeds. The process flow for the discrimination is shownin Fig.3.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan53

Shingo Yamaguchi, Eiji Hayashi, Ayumu Tominaga

class B is the tree region in each histograms. Therefore,
we use the relation between Red, Green and Blue and therelation between Green and luminance to solve thisproblem.

A
Fig.4. Discriminant analysis method applied example
C.1. selection of class in Green histogram
As can be seen from eq.1, green is the brightest color
among red, blue and green. For this reason, it can be seenthat there is a positive correlation between green value
and luminance. Therefore, we compare the luminancelevels of the planes and the other to calculate the tree
region in the green histogram. Specifically, class A isregarded as the tree region in the green histogram in caseof the luminance level of other than ground is 80% ormore of the luminance level of the ground, but in othercases class B is regarded as the tree region.

A

C.2. selection of class in Red histogram
From the relation between Red, Green and Blue, thecolor is brownish when the density value of R is large.
When the weeds die, they become brownish colored.
Therefore, in the Red histogram, class A is always
regarded as the tree color region so that we can also
discriminate dead weeds and trees.

A

Fig.5. case1 result

Fig.6. case2 result

Fig.7. case3 result

B

B

B

Conclusion & Future work

C.3. selection of class in Blue histogram
As in the case of red, from the relation of each color,when the density value of blue is larger than that of green,
the color is bluish. Therefore, in tree color, green density
value is considered to be lower than blue density value.
For this reason, class A is regarded as tree region in theblue histogram in case of that the tree region is class A in
the green histogram, but in other cases both class A andclass B are regarded as the tree region.

We can said that it is possible to compute the tree region
in Red, Green and Blue histograms by using this method
except the case shown Fig7. However, using only thismethod, it is impossible to discriminate between trees
and weeds completely when some weeds is in tree color
region shown Fig.6.
In future work, we will perform shape learning on the
result image in order to discriminate between trees andweeds. In addition, improve a system to consider
sunshine and shade.

5. ResultFig.5, Fig.6 and Fig.7 show the result of experimentusing this method. Fig.5 shows the case1 that luminancelevel of ground is larger than that of the other. Fig.6
shows the case2 that luminance level of ground is lower
than that of the other. Fig.7 shows the case3 of that there
is sunshine and shade. In each figure, the image named A
is original image and image B is the result using thismethod.

References[1]

Nobuyuki, Otsu (1980) “An Automatic Threshold Selection
Method Based on Discriminant and Least Squares Criteria,“IEICE, Vol.J63-D(4), pp.349-356© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan54

Design and Evaluation of Passively Powered Knee Exoskeleton (PPKE) for Squat Lifting
R.K.P.S. Ranaweera1, R.A.R.C. Gopura1, T.S.S. Jayawardena2

Bionics Laboratory, Department of Mechanical Engineering1, Department of Textile and Clothing Technology2,
University of Moratuwa, Katubedda, Moratuwa 10400, Sri Lanka
G.K.I. Mann

Faculty of Engineering and Applied Science, Memorial University of Newfoundland,
230 Elizabeth Ave, St. John's, NL A1B 3X9, Canada
E-mail: pubudur@uom.lk, gopurar@uom.lk, jaya@uom.lk, gmann@mun.ca
www.mrt.ac.lk

Abstract
This paper proposes a passively-powered knee exoskeleton (PPKE) to provide power assistance during squat liftingof objects from the ground. The passive powering mechanism is designed to capture 20% of the waste mechanicalenergy available at the biological knee joint during decent phase and to release the stored energy in ascending phasein a complete squatting cycle. The effectiveness of proposed system was verified by experiments using surface
electromyography (sEMG) signals under different test conditions. PPKE reduced the peak root-mean-square averagesof sEMG signals of knee extensor muscles by 15% during squatting.
Keywords: knee exoskeleton, energy harvesting, passive mechanisms, squatting.

resulted in formulation of bulky and heavy
electromechanical systems that is slow in operation.2
Alternatively, recent research studies indicate that
passive-dynamic powering systems are effective at
metabolic cost reduction and enhancing overall humanperformance.3 The body-powered exoskeletons areenergetically autonomous and soft lightweighttransmissions
providesuperior
biomechanical3
compliance. However, performance and endurance
augmentation through biomechanical energy harvesting
still remains an open challenge. A wearable energy
harnessing and releasing system that is capable ofrecycling the waste mechanical energy dissipated in thehuman body can help reduce metabolic energy
consumption during activities of daily living.3
Complementing limb muscles energetically will augment
the joint strength and moreover enhances endurance
limits of muscle fatigue. As a result, exoskeleton
wearer’s productivity will improve over a period of time.

1. Introduction
Exoskeletons systems are mainly developed focusing on
two different user groups, namely physicallyincapacitated population and able-bodied population.1
According to literature, limited number of full lowerextremity exoskeletons or joint-level exoskeletons havebeen developed to carry out work related tasks byproviding power-assistance.1 The application areas aremostly limited to either locomotion assistance, tool
holding or lift assistance.1 Although functionality of
autonomous externally powered robotic exoskeletons are
satisfactory, wearer’s performance is notablycompromised.2 In particular, such devices struggle toreduce the human effort. The main cause is the inabilityof the system to match the complex kinematic and kinetic
requirements demanded by human biological joints. Here,limitations of the complementing technologies such as
gear drives, actuators, sensors, and power sources© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan55

R.K.P.S. Ranaweera, R.A.R.C. Gopura, T.S.S. Jayawardena, G.K.I. Mann

2.1. Functional
Among different lifting techniques the most commonlyused approaches are squat lifting and stoop lifting.5 The
knee exoskeleton is envisioned to assist wearer’s knee
joint to lift moderately heavy loads. Hence squat or semisquat lifting is selected as the technique in focus fordeveloping the proposed device, as it also promotes useof biomechanically more favorable approach.5
The device should essentially allow the wearer toambulate freely in-between lifting tasks and pose norestriction for knee flexion during swing phase of human
gait. The knee joint’s limits of ranges of motion over thesagittal plane is measured to be -100 of hyper-extension
and 1400 of flexion.4 According to gait data, 6 maximum
flexion of knee joint during swing phase is 600.
2.2. BiomechanicalFig. 1. Technique (A) and biomechanical data (B) of squatting
adapted from Ref. 4 and 5.

Muscle activity of thigh muscles are of interest when
performing a squatting cycle.7 Notably, the quadriceps
show greater muscle activity than the hamstrings during
the complete ascent phase.4 The focus is to relieve muscle
groups of the burden of having to concentrically andeccentrically work to control motion.
Biomechanical data during squatting (see Fig. 1(B))
were analyzed to identify the role of lower limb joints togenerate the mechanical power for lifting.4 The notion isto recover the potential energy dissipated as a result of
lowering the center of gravity of human body which
henceforth referred to as passive energy. Here, hip andknee joint dissipate energy through eccentric contractionof the major muscle groups on lower extremity during
entire decent phase (2 seconds period).4 Similarly, during
the entire ascending phase (2 seconds period), energy is
generated while undergoing concentric contraction.4 In
contrast, during initial stages (0-15% cycle) of the decent
phase, the knee flexor muscles act concentricallyfollowed by extensor muscles acting eccentrically for theremainder of the descent (15-53% cycle).4 The positive
power is assumed to unlock the knee to initialize thedescent. Throughout the start of the ascending phase (5385% cycle) power is generated by knee extensor muscles
by acting concentrically.4 However, towards end of thephase (85-100%) knee flexor muscles started to dissipate
energy gathered using knee flexors acting eccentrically.4
Negative power is assumed to help prevent undesirable
levels of hyperextension of knee joint.

In this context, a design of a passively powered knee
exoskeleton is proposed to provide power assistanceduring squat lifting of objects from the ground (see Fig.
1(A)). The bio-inspired powering system is designed torecycle the mechanical energy at the knee joint during
descending and ascending phases in sequence to help lift
wearer’s own mass during a squatting cycle. The knee
exoskeleton shall not pose any restrictions for walking
and passively controlled locking mechanism only
engages during a lifting task to capture and releasemechanical energy. The design considerations for theknee exoskeleton targeting squat lifting and walking aredescribed in section 2. The design details of theexoskeleton and the working principle of the passive
powering mechanism is presented in section 3. The
device was evaluated using surface muscle signals while
performing semi-squats under different test conditions
and results are compared in section 4.
2. Design Considerations
Load lifting is a typical industrial activity that is energy
intensive.4 It involves moving an object from one position
to another while traversing in vertical and/or horizontal
directions. Considering the requirements for squatting aswell as for walking, design considerations in relation to
the knee joint were identified under functional andbiomechanical points of view.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

56

Design and Evaluation ofFig. 2. PPKE on human model (A), PPKE with passive-powering mechanism (B), and exploded-view of PPKE (C).
and sagittal planes (see Fig. 2(A)). In order to avoid
collision of thigh rest and shank rest at full squat position,
shank connectors were designed with longer legs (see Fig.
2(B)). Thigh connectors were shorter in length tominimize interference at the medial side. The structure isheld firmly on the limb segments using four Velcro straps
placed through the linear slots on the connectors. Resting
links can be easily replaced to accommodate varying
limb cross-sections, if needed. The curved slots at thehinged end of the rest links provide the ability to adjust
relative angular position of rests with the connectors. The
slender acrylic structure assume the body shape over thefrontal plane if contact forces are significantly high. The
compliant structure hence reduce the possibility ofinternal joint injuries due to misalignments.

3. Design of Passively Powered Knee Exoskeleton
Design objectives were to develop a simple, lightweight
and affordable knee exoskeleton with ergonomic
conformance and provide power assistance during squat
lifting while posing least resistance for walking. Design
specifications were derived for 50th per. male population.
3.1. Material Selection
Acrylic glass is considered as material of choice fordeveloping the PPKE. Acrylic glass sheets can be easily
cut in to various contours using computer numericallycontrolled laser cutting machines and formed in tocomplex shapes using thermoforming methods at anaffordable cost. Acrylic is also considered to be non-toxic
to human skin and environmental friendly. Thus making
it a suitable option that has not been previously exploredin exoskeleton development. In order to counter the
lower tensile strength of acrylic material (approx. 70MPa), the design of the exoskeleton should besatisfactory. The flow of forces through the acrylic
structure should be entirely compressive in nature.
Bending or shear forces have been eliminated byeffectively utilizing the symmetricity of the design.

3.3. Passive Powering Mechanism
It is assumed that industrial workers are more likely toassume a semi-squat position during lifting tasks. Hence,the design has been planned for parallel-squat operation,
where the knee bends approx. 700 to 1000. In addition,
wearer need to have freedom to ambulate at normal
walking speed6 (1.3 m/s). Consequently, the knee flexion
should be left unconstrained up to 600. The springs
should engage to capture the negative energy when theknee flexes beyond this limit. Assuming the maximum
knee angle happens up to 900 in a regular squat liftingtask, the effective angular displacement of 300 can beutilized to energize a collection of linear elastic elements.

3.2. Structural System
A computer-aided design model of a human subject was
prepared using scanned data. The sizing and shaping of
the structural system was then performed over the frontal© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan57

R.K.P.S. Ranaweera, R.A.R.C. Gopura, T.S.S. Jayawardena, G.K.I. Mann.

number-four (600). As a result, during rest of the decent
phase up to 900, available passive energy is harnessed.
The spring selection should be done to maximize theenergy retention. Considering the limited tensile strengthof acrylic glass and the overall size of the device, themaximum force applied on a single spring is limited to
150 N. The limits were determined by analyzing thestress distribution over the critically loaded acrylic
components of the exoskeleton. In particular, the shankconnectors and pulley disks were subjected tocompression loadings and torsional loadings respectively.
Finite element simulation provided an insight on thesafety limits for the same. To accommodate
manufacturing and material defects, the factor of safety
were maintained at 1.5 and above for all acrylic parts.
The structure and powering mechanisms are capable ofwithstanding such heavy loads as a consequence ofsymmetrical positioning of passive actuators.
The energetic response of the proposed passive
energy recycling system can be mathematically modeled
as follows. If a bilateral system with elastic helicalsprings having 5875 N/m stiffness coefficient is used, thetotal strain energy stored in the system over a 300 knee
angle can be determined using,Fig. 3. Working model of PPKE on male subject (A)and close-up view of passive powering mechanism (B).
The proposed mechanism to achieve this function iscomprised of energy capturing springs held parallel toshank, wire cables, pulley disks, smaller return springsand pins (see Fig. 2(C)). The pulley disks are fitted on
bearings and are free to rotate about the axis of the shaft.
However, by form the mechanical limits are in place toprevent rotation of the pulley in opposite or counterclockwise direction. The clockwise rotation will be
prevented by the smaller return springs that is held
between the retainer disk and the pulley disk using set ofpins. The lock pin 1 can be inserted in to a pulley disk
locking hole that are spaced 200 apart over a range of 00
to 1200. If the lock pin is removed the knee joint can beeasily flexed without notable restriction from theexoskeleton. Here the stiffer springs act as a rigid bodycausing the flexible return springs to extend. As a result
the pulley disk will rotate in a clockwise sense tomaintain wire cable length unchanged. When the knee
traverse back to the rest position, the return spring gives
way and the pulley disk returns to its initial position. By
placing the lock pin 1 in position number-one (00) the
pulley disk gets immediately locked with retainer disk.
Consequently, when the knee gets flexed the stiffer
springs starts to load gradually capturing energy ofdecent. If the lock pin is inserted in to position numbertwo (200), the knee is free to flex up to the same limit andthe pulley disk will come to a stop as the lock pin 1
traverse and reach the mechanical limit over the curved
slot. If PPKE is supposed to allow free flexion during
walking, the lock pin 1 should be inserted in position

E=

1
2

kx

2

(1)

Where, k – stiffness coefficient (N/m), x – spring
extension (m) and E – accumulated energy (J or Nm).
The total energy dissipated and generated at the knee
joint during decent and ascent phases frombiomechanical studies were found to be 45 J and 50 J
respectively. The selected springs thus have the capacity
to collectively (eight springs on two PPKE units) capture
and return approximately over 20% of total mechanicalenergy required considering loses due friction. Here, themaximum spring force is evaluated to be approx. 133 N.
4.

Evaluation of PPKE

Overall mass of PPKE (see Fig. 3) is approx. 900 g
inclusive of helical springs made of spring steel. In orderto evaluate the effectiveness of the proposed system, apreliminary study was set up and conducted with ahealthy average male subject (Age: 37 years, Weight: 72
kg, Height: 177.8 cm). Test protocols were developed tomeasure relative changes of muscle activity for squatting© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan58

Design and Evaluation of

without the exoskeleton (WO),
with exoskeleton minus passive powering (UPO),
with exoskeleton having passive powering (PO).
4.1. Experimental Set-Up
Squatting exercises were performed to monitor possible
benefits of using the passive energy recycling system toenhance strength and improve endurance limits of thighmuscles. Each of the tests were conducted for a period of
100 seconds with an ascent-descent cycle of four seconds
that ranged over 00 to 1000 knee angle. Wired surface
electromyography (sEMG) electrodes connected to
Bagnoli Desktop EMG system were placed on muscle
bellies of rectus femoris (RF) and bicep femoris (BF)
muscles on both limbs to measure the electrical signals.

Fig. 4. RMS values of (A) RF and (B) BF muscles onright lower limb during first five squatting cycles.

4.2. Results and Discussion
The root mean square (RMS) average of the electricalpotential of RF and BF muscles were post-processedusing raw sEMG signals. When considering the first ten
squatting cycles (see Fig. 4(A)), the RF muscles show adecrease in peak RMS values when wearing exoskeleton
with the passive powering system by 15%. Alternatively,
during the last ten cycles (see Fig. 5(A)), the peak RMS
values are greater when wearing the exoskeleton with orwithout the powering system. Interestingly, BF muscles
respond in an opposite manner as the cycles elapsed.
During the first ten cycles of squatting (see Fig. 4(B)),
exoskeleton tend to increase the peak RMS values than
the norm, which is especially evident when it is worn
without the assistance of the powering system. However,towards the latter end, in particular during last ten
squatting cycles, the peak RMS values when wearing theexoskeleton remain consistent while for the case of no
exoskeleton a notable increase in noted (see Fig. 5(B)).
Therefore, the preliminary study has revealed that PPKE
has both positive and negative impact. Considering therelative significance of quadriceps muscles to perform
squat lifting, device has successfully satisfied its purpose.
5.

Fig. 5. RMS values of (A) RF and (B) BF muscles onright lower limb during last five squatting cycles.
Acknowledgements
Authors acknowledge Senate Research Council of
University of Moratuwa (Grant No. SRC/CAP/2014/08).
References
1. B. Chen et al., “Recent developments and challenges oflower extremity exoskeletons,” J. Orthop. Transl., vol. 5,
pp. 26–37, 2016.
2. A. J. Young and D. P. Ferris, “State of the art and future
directions for lower limb robotic exoskeletons,” Trans.
Neur. Syst. Rehab. Eng., vol. 25, no. 2, pp. 171–182, 2017.
3. S. H. Collins et al., “Reducing the energy cost of humanwalking using an unpowered exoskeleton,” Nature, vol.
522, no. 7555, pp. 212–215, 2015.
4. D. G. E. Robertson et al., “Lower extremity muscle
functions during full squats,” J. Appl. Biomech., vol. 24,
no. 4, pp. 333–339, 2008.
5. S. Hwang et al., “Lower extremity joint kinetics andlumbar curvature during squat and stoop lifting,” BMC
Musculoskelet. Disord., vol. 10, no. 1, p. 15, 2009.
6. B. R. Umberger and P. E. Martin, “Mechanical power andefficiency of level walking with different stride rates,” J.
Exp. Biol., vol. 210, no. 18, pp. 3255–3265, 2007.
7. J. P. Pollard et al., “Forces and moments on the knee
during kneeling and squatting,” J. Appl. Biomech., vol. 27,
no. 3, pp. 233–241, 2011.

Conclusion

PPKE is energetically autonomous, adaptable and nimble
thus making it ideal for industrial use. Working model
served as a proof of concept, but clinical testing withmetabolic cost estimation over wide range of participants
is essential to assess the design for further improvements.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan59

Behavior design of robot arm imitating the consciousness mechanism of living organisms
Representation of facial expression in transition process of emotion
Ryohei Anashi

Mechanical Information Science and Technology, Kyushu Institute of Technology.
680-4, Kawazu, Iizuka, Fukuoka, 820-8502, Japan.
anshi@mmcs.mse.kyutech.ac.jp
Wisanu Jitviriya

Faculty of Engineering, King Mongkut’s University of Technology North Bangkok.
1518 Pracharat 1, Wongsawang Rd., Bangsue, Bangkok, Thailand 108000
wisanu.j@eng.kmutnb.ac.th
Sakmongkon Chumkamon

Faculty of Engineering, Rajamangala University of Technology Thanyaburi.
39 Moo 1 Rangsit-Nakhonnayok Rd Thanyaburi Pathum Thani 12110, Thailand.
s.chumkamon.jp@ieee.org
Eiji Hayashi

Mechanical Information Science and Technology, Kyushu Institute of Technology.
680-4, Kawazu, Iizuka, Fukuoka, 820-8502, Japan.
haya@mse.kyutec.ac.jp

Abstract
In this paper, we present a system that facial expression expressions in the case of embracing multiple emotions
occurring in the transition of emotions of robots having eight basic emotions. In addition to changes in eyeballs andeyelids, changes in pupil diameter and changes in blink cycle were incorporated. The facial expression is made inreal time by changing these according to the always changing emotion ratio. As a result of verification experiments
comparing the conventional system and this system, it was confirmed that facial expressions of the emotionaltransition process, which could not be expressed in the conventional system, could be expressed.
Keywords: facial expression, pupil diameter, blink cycle, emotion ratio.

behavior and communication ability. In this research,
we aim to realize user affinity by paying attention toliving organisms that embrace healing and affinity,
giving the robot consciousness, behavior, and emotionof living organisms. In this fiscal year, we developed asystem such as change in pupil diameter and blink cycle

1. Introduction
Various service robots such as home and nursing careare being developed. In particular, social robots are
required to have "user affinity" that gives the user asense of affinity as well as functions such as intellectual© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan60

Ryohei Anshi, Wisanu Jitviriya, Sakmongkon Chumkamon, Eiji Hayashi

so that smooth and detailed facial expressions can bemade with emotional change.

3.1. External situation recognition by camera
Images acquired from the small camera are simplified
into five colors of green, blue, red, skin color, and other,and labeling processing is performed. Of these, we set
green / red objects as favorite objects and blue objects
as dislikes. The recognition result of what the robot
likes or dislikes is reflected in the calculation of
motivation. Also, recognize the facial expression of theuser shot by the CCD camera. We acquire the featurepoints of each part of the face learned beforehand, andcreate the shape model. Identify the region of the face inthe captured image from the image around the featurepoint and classify the emotion from the user's facial
expression based on the difference in shape vector.

2. System configuration
Fig.1 shows the robot developed in this study. Represent
the robot's consciousness, behavior and emotion by
arm's behavior and face expression. The arm has 7
degrees of freedom and a small camera inside the hand.
Face has a CCD camera and a small display on its neckwith 2 degrees of freedom, and this small display
projects the created eyeball.

3.2. Calculation of motivation according tosituation and condition
As motivation to cause behavior of living things, thereis the occurrence of intracerebral neurotransmitter
dopamine. In this system, positive dopamine isrecognized when a favorite red or green object isrecognized, and negative dopamine is generated when adislike blue object is recognized. The generation of thisdopamine was expressed using a control model. Then
we defined the second order lag response with the sum
total of positive and negative dopamine as input
motivation which is the motivation of the robot's
behavior. The robot determines the behavior according
to this motivation. The occurrence of dopamine andmotivation is shown in Fig.3.

Fig.1 Overview of the Arm Robot

3. System flow
The robot recognizes the external situation by thecamera. Motivation is calculated as an internal factor
from the recognition result of the object. Also recognize
the facial expression of the user and classify emotions.
Then, based on the motivation and the emotion
classification result of the user, the behavior and facial
expression of the robot is selected. The flow of thissystem is shown in Fig. 2.

Fig.3 Relation between dopamine and motivation

3.3. TopoART(Topological Adaptive Resonance
Theory)ART (Adaptive Resonance Theory) is a theory
developed based on the relationship between the livingorganism's brain and consciousness. As a flow of thelearning algorithm, category data having information on

Fig.2 Flowchart for Autonomous Emotion-Behavior© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan61

Behavior design of robot

the center position, class, and radius in the output layer,
and alert parameters for narrowing the search range ofthe input are defined in advance. A category with acenter in the input warning parameter is set as a searchtarget, and a vision test is performed. If category andinput class match, expand the category to include theinput and output the result. In case of inconsistency,
decrease the warning parameter and re-search thecategory.
Two ARTs are prepared, each of which has roles ofshort-term memory and long-term memory, and layered
to constitute a TopoART system. Fig.4 shows
TopoART algorithm.
In this system, motivation of the robot and informationof the user's emotion are input, and the ratio of eachemotion is output. As a result, one facial expression
including multiple emotions is selected.

4.1. Creating facial expressions by changing
rotation angle
As shown in Fig. 6, the eyelids and the eyeball are given
2 degrees of freedom so that expression can be
expressed. The angle of these two degrees of freedom is
set for each of the facial expressions of the eight basic
emotions. By updating these angles using the output of
TopoART, continuous facial expression creation wasmade possible.

Fig.6 Axis of rotation

4.2. Change in pupil diameter
The pupil diameter is scaled down due to external
factors and internal factors. We focused on internal
factors in this study. Pupils are scattered for visual
stimuli that are of interest and are reduced for stimuli
that evoke discomfort. Based on this feature, the pupil
diameter in each basic emotion is set, and the pupil
diameter is determined by Equation (1) using theemotion ratio [%] which is the output of TopoART. The
emotion ratio satisfies equation (2). R is the pupil
diameter [mm], and ρ is the emotion ratio.

Fig.4 TopoART algorithm=++

4. Representation of expression in transition
process of emotion
It is said that neutral, hope, happiness, surprise, fear,sadness, disgust, anger are the basic feelings. In thissystem, expressions for expressing basic emotions werecreated. It is shown in Fig.5.

+

× 100.0 +× 68.4 +× 42.1 ++++× 78.9 +× 73.7 +× 47.4++× 89.5
× 73.7
+
=1

(1)
(2)

4.3. Change in blink period(a)Neutral(b)Hope(c)happiness(d)Surprise(e)Fear(f)Sadness(g)Disgust
Fig.5 Facial Expression(h)Anger

Types of blinks are reflective blinks, voluntary blinks,
and spontaneous blinks. In this study, we focused onvoluntary blink and spontaneous blink. voluntary
blinking is a blink of consciousness. Spontaneous
blinking is a blink of eye under unconsciousness, and© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan62

Ryohei Anshi, Wisanu Jitviriya, Sakmongkon Chumkamon, Eiji Hayashi

expression expressed by the previous system is (B). In
case of 1, the emotion ratio is outputting joy andsurprise. (1 - A) expresses the expression in which thetwo emotions are compounded, but in (1 - B) the
expression reflecting strongly the pleasure of theemotion ratio is expressed. In case of 2, the emotion
ratio is sadness, hatred, fear and extremely few
expectations are outputted. (2 - A) expresses theexpression in which each emotion is compounded, but
in (2 - B) the expression of sadness of basic emotion
and the difference are not seen and other emotions are
not reflected.
Based on the experimental results, in the conventional
system, the expression reflecting strongly the emotion
with the greatest emotion ratio was expressed in bothcases 1 and 2, and other emotions were not reflected. On
the other hand, in this system, expressions reflecting
multiple emotions were both expressed. Also, in theconventional system, the facial expression transition isan intermittent change and a problem that it is an
unnatural expression was seen. However, since thissystem creates facial expressions according to the
output of TopoART, facial expression transitions
become continuous, smooth facial expressions become
possible, and unnatural emotional expressions until lastyear have been solved.

the blink cycle of adults is 3000 to 3750 [ms]. The blink
cycle becomes longer at rest and concentration, andbecomes shorter when there is no tension, excitement orinterest. Based on these features, we set the blink period
in each basic emotion, and calculate the blink period byequation (3) according to emotion ratio.
=++× 4300 +× 3300 +× 2700 +× 6000 +× 2150 +× 5000× 1500× 3500(3)

5. Inspection result
We verified experimentally whether the expression ofrobot's expression changes more smoothly and finely by
the developed system. The target of comparison is thesystem until the last year. The results are shown in Fig.7.

(a)

6. Conclusion
We introduced expression of pupil diameter and blink
cycle corresponding to change of rotation angle change
and transition of emotional ratio, and it became possible
to express more diversified facial expressions.
Currently the behavior of the robot is determined only
by motivation. Therefore, as a future task, we aim toconstruct actions accompanied by smooth facial
expressions including detailed changes of emotions
made possible this time.

(b)

(c)

(1-A)

(d)

(1-B)
(2-A)
Fig.7 Result(2-B)

References
1. S. Chumkamon, “Study on Emotion and Behavior
Generating System based on Facial Expression
Recognition”, Kyushu Institute of Technology, 2017,
Ph.D. thesis.

(a) shows the trend of motivation, and (b) shows thetransition of emotion ratio. 1 recognizes a favorite green
object and when motivation rises, 2 recognizes a dislike
blue object and is when motivation falls. The emotion
ratio of each state is shown in (c) (d), the facial
expression expressed by this system is (A), and the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

63

Study of Robot Navigation for Forest Management

Ayumu Tominaga

Graduate School of Computer Science and System Engineering, Kyushu Institute of Technology,
680-4 Iizuka, Fukuoka, Japan*
E-mail: tominaga@mmcs.mse.kyutech.ac.jp
Eiji Hayashi

Graduate School of Computer Science and System Engineering, Kyushu Institute of Technology
680-4 Iizuka, Fukuoka, Japan
E-mail: haya@mse.kyutech.ac.jp
Abbe Mowshowits
Department of Computer Schience, The City College of New York,
Convent Avenue at 138th Street, NY 10031, U.S.A
E-mail: abbe@cs.ccny.cuny.edu

Abstract
We have constructed a robotic system enabling an autonomous robot to move between the trees without damaging
them and to cut the weeds as it traverses the forest. Computation of a trajectory in a forest is facilitated by treating
the trees as vertices in a graph. This paper proposes how recognition the unreached area in the field to navigate inorder to make the robot move whole of the forest. Finally, the proposed method was experimented in the experimental
forestKeywords: Autonomous Robot, Navigation, Forest Management, Graph theory

We have been developing the robot achieved byautomatically elimination weed plants in the forest. As
mention above, the trees in the environment can be used
as a landmark for navigating of the robot in the forest.
When patrolling in the forest, it is necessary for the robot
to traverse whole areas of the forest as possible as.
Because, the weeding work is important for promoting
the growth of the trees, and ideally all weed plants should
be rejected from the forest. In this paper proposes amethod based on graph theory to realize it. First, thegraph structure which has Hamiltonian cycle isconstructed trees as vertices. Then the destination
location will be determined to traverse of its. When the
robot moves to the destination, the obstacle avoidance

1. Introduction
The application of robotics in forestry is wellestablished [1]. These applications have been introduced
to increase productivity, to enhance safety, and tocompensate for the workforce which trends toward adecrease. Particularly, by improvement Simultaneous
Localization and Mapping (SLAM) techniques, themonitoring and management of a forest resource could be
automatically [2, 3]. Such systems and related techniques
allow for utilizing the autonomous robot for support offorestry industry by identifying individual tree location.
In this research, we have been focused a navigation
assuming using of them for an autonomous moving robot.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan64

Ayumu Tominaga, Eiji Hayashi, Abbe Mowshowits

behavior is performed without using precision control
system. The Hamiltonian cycle will be separated to twocycles least, that can be represented by the unreachedarea for the robot.
This paper reports the proposed system in theexperimental forest.

robot’s heading is obtained from the Inertial
Measurement Unit (IMU). In this research, externalsituation recognition is achieved mainly with the aid of adepth sensor. This sensor can provide 3-dimensional
point cloud data obtained by scanning with an infraredlaser.

2. Forestry Robotic System “SOMA”

2.2. Obstacle Avoidance
Obstacle avoidance is an essential function for anautonomous mobile robot. The trees in a forestry
application can be viewed as obstacles to be avoided bya robot so as not to damage them. Until recently, such
systems were designed for the use of image processing
with color information obtained from monocular cameras[4,5]. Improvements in the technology of measurement
using infrared lasers, such systems were possible toconstruct using 3-dimensional point cloud data that canobtain in real time in outdoor. The depth sensor (R200,
Intel Inc.) used on SOMA.
Fig.2 shows a flowchart of the obstacle avoidance
procedure. After getting point cloud data from the depth
sensor, a top view 2-dimensional image is produced.
Next, labeling processing is carried out on the top view
image to enable recognition of the boundary point and thelocation of the objects in the robot’s field of vision. Then,the length of line segment between the boundary pointsof each object is computed, and longest line segment isdetermined. The midpoint of the longest line segment isdefined as the robot’s target point. By performing theprocedure recursively, the robot can avoid the tree andmove forward between two trees at the next location in
front of the robot.

2.1. System description

Fig. 1. Appearance of forestry robot “SOMA”

2.3. Following a destination position methods
This subsection describes the method of following ofdestination position. This is accomplished by means of
the pan rotate actuator on the vision system that can rotate
up to 30 degrees on one side. The program works asfollows.
(i) The current position of the robot is represented .
is a destination position that given to the robot.
(ii) Calculate the angle , direction angle fromto
. The robot heading
is obtained from IMU.
(iii) Turn the pan rotate actuator to angle− .
Following its obstacle avoidance procedure the robot
approaches.

Fig.2. Flow of obstacle avoidance behavior

The autonomous robot “SOMA” shown in Fig. 1 isbuilt on the platform of an All-Terrain Vehicle (ATV).
An ATV was chosen for its robust design for outdoor use
in rough terrain. The power source is a gasoline engine,
and the throttle, brake lever, and steering are controlled
by DC motors. Weeding operation is facilitated by aspecialized device attached to the front of the ATV. Also
important is the ATVs ability to climb steep hills, up to
an incline of 26 degrees.
SOMA is equipped with a Real Time Kinematic GPS(RTK-GPS) for determining position. The angle θ of the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

65

Study of Robot Navigation(iv) When the robot reaches the circle of radius 1.0
meters centered on, the program returns to (i)with update a new destination point.

(iii) Computing the vertex within the spanning tree thatis farthest from the current position. This vertex isfor thedetermined as the destination point
robot (shown in Fig.6).

Fig.2. Destination position following method
Fig. 4. Position of trees in experimental forest

3. Navigation Method in the Forest
Fig. 4 shows the relative positions of the trees
determined by measurements of RTK-GPS in anexperimental forest. The positive directions of the y-axis
and x-axis are oriented true north and true east
respectively. The size of the field was defined 35 metersquare.
In this research, the navigation system for the robot iscarried out using two types set of coordinate points. One
is a set of the coordinate point of individual trees, otherone is a set of midpoint of every pair of tree. These tree’s
pairs in order to compute midpoints was decided under
the condition that each tree of the pair exists on light-ofsight with direct distance less than 10 meters. In addition,these sets can be represented a graph = ( , ) by
associating each coordinate points as a vertex.
First, generate a Hamiltonian graphto define theinitial enclosed space. The Hamiltonian cycle represents
an enclosed space in 2-dimensional space (shown in
Fig.5). Then, determine the destination point in order tomake the robot traverses such enclosed space. When the
H was traversed accompany intersected any 2 edges by
the autonomous robot, it can be separated 2 enclosed
space. These enclosed spaces will be consideredsuggesting unreached spaces in environmental map. How
determine the destination point is performed by follows.
(i) From the set of midpoint coordinates, a set of interior
points within the constructed enclosed space isextracted.
(ii) Generate a minimum spanning tree using theseinterior points as vertices after the complete graph
was constructed using these.

Fig. 5. Initial Hamilton cycle (enclosed space) in theexperimental forest

Fig.6. Minimum spanning tree using midpoint coordinatesthat interior points within initial enclosed space© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan66

Ayumu Tominaga, Eiji Hayashi, Abbe Mowshowits

The robot performs obstacle avoidance mentioned
above until the robot reached. When the robot
traversed 2 edges that constructing the closed space, thenthe Hamiltonian cycle is updated. Let the edges whichintersected by the robot be = { , }= { , },
update of Hamiltonian cycle will be completed by simply
swapping the vertices such that={ , }={ , } (see Fig.7).

(a)Fig.7 Separation of the enclosed space

4. Experimental Result
The experimental forest on the campus of the Kyushu
Institute of Technology consists of 39 trees. There are no
object, other than the trees, in this area that might impede
the movement of a robot. This trees is small enough toallow for determining the positions of the trees andmeasuring.
The Hamiltonian cycle and the minimum spanning tree
used the experiment was shown in Fig.5 described in thepreceding section. Fig.8(a) is shown the initial position, Fig.8(b) isof the robot and the destination point
the trajectory history that the robot performed
autonomous moving. As the result in this experiment, therobot could traverse the enclosed space, the Hamiltonian
cycle could be separated two cycle. These enclosed
spaces are recognized as the unreached area for the robot.

(b)
Fig.8. Experimental result

References
1. R. Parker, K. Bayne, and P.W. Clinton, "Robotics inforestry," New Zealand Journal of Forestry, 60, 2016, pp.
8-14.
2. G. Brolly, and G. Király,. "Algorithms for stem mappingby means of terrestrial laser scanning," Acta Silv. Lign.
Hung. 5, 2009, pp. 119-130.
3. T. Ritter, M. Schwarz, A. Tockner, F. Leisch, and A.
Nothdurft, "Automatic mapping of forest stands based on
three-dimensional point clouds derived from terrestrial
laser-scanning," Forests, 8(8), 2017, pp. 1–19.
4. K.H. Chen and W.H. Tsai, "Vision-based obstacle
detection and avoidance for autonomous land vehicle
navigation in outdoor roads," Automation in Construction,10(1), pp.1–25, 2000.
5. A. Ohya, A. Kosaka, and A. Kak, "Vision-basednavigation of mobile robot with obstacle avoidance using
single-camera vision and ultrasonic sensing," Proceedingsof the 1997 IEEE/RSJ International Conference onIntelligent Robot and Systems. Innovative Robotics for
Real-World Applications. IROS ’97, 2, 1997, pp. 704–711.

5. Conclusion
In this paper, we proposed a navigation method of
autonomous forestry support robot. We are aiming to
guide in order to make the robot traverses all area of aforest, using the coordinate of trees which unchanged in
the long term. In order to achieve an patrol of the wholeforest, the system decides a destination point based on aminimum spanning tree constructed by the midpoints oftree pairs within enclosed space expressed by
Hamiltonian cycles. In the experiment, the robot could
separate the spaces and succeeded in breaking the
Hamiltonian cycle into two. In the future works, we will
experiment the robot operate of autonomous moving forlong time to verification for effectiveness of the system
proposed.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan67

Development of the sense system that is combined force feedback and visual feedback-Deformable virtual objects simulation by using LEMKaito Nagano

Graduate School of Computer Scienc and System Engineering
Kyushu Institute of Technology, 680-4, Iizuka City, Fukuoka, Japan, 820-8502
nagano@mmcs.mse.kyutech.ac.jp
Eiji Hayashi

Department of Mechanical Information Science and Technology
Kyushu Institute of Technology, 680-4, Iizuka City, Fukuoka, Japan, 820-8502
haya@mmcs.mse.kyutech.ac.jp

Abstract
The purpose of this research is to develop a combined senses system that uses both force feedback and visual feedback
on a deformation simulation. Especially, focus on a way to produce a force that is calculated in simulation of
deformation using a haptic device. As the first stage, our simulation of deformation was improved by using LEM(Long Element Method) and evaluated accuracy of a virtual object’s deformation.
Keywords: force feedback, haptic interface, simulation, LEM

object. For this research, focus on a way to produce aforce calculated in simulation of deformation using a
haptic device. As the first stage, our simulation of
deformation was improved by using LEM (Long Element
Method) and evaluated accuracy of a virtual object’s
deformation.

1. Introduction
In the medical treatment and bio-technology field,
doctors and researchers need technologies that canaccurately perform minute work. Such minute work is
improved by using micromanipulators, but theiroperation is difficult because the operator has no sense offorce. The operator relies only on sight through a
microscope. As a result, a person who has the skill to do
minute work is needed for all minute work. The
efficiency of minute work would be improved if theoperator is able to feel a sense of force while using amanipulator.
Here we describe the development of a more efficientsystem for minute operations. Our aim is to develop the
combined system that that uses both force feedback fromthe manipulator and visual feedback on a deformation
simulation. By reproducing force sense and visual senseof a sample, an operator can feel these sense of a virtual

2. System Structure
The system structure is shown in Fig. 1. This system
consists of a force feedback part and a visual feedback
part. At a force feedback part, this system is equippedwith the haptic device and a microcomputer forcontrolling the haptic device. At a visual feedback part,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

68

Kaito Nagano, Eiji Hayashi

Fig.2. The structure of the haptic device.

It is equipped with a simulation video in a main computer.
The main computer also sends parameters to the haptic
device. The parameters are obtained by measuring
dynamic characteristics of a sample using microscope.

Fig.3. The connection between the haptic device
and a microcomputer.
4. Simulation interface
In this study, the working system using a microscope,
a haptic device and a simulation is built. A fundamental
element was simulating the deformation of a minuteobject. Fig. 4 shows the graphical user interface (GUI) of
the simulator. A graphic tool was created using OpenGL
to draw the object.

Fig.1. The system structure.

3. Haptic device
Fig. 2 provides a diagram of the haptic device. It
consists primarily of a rotor, a laser, and a positionsensitive detector (PSD). The angle of the rotor can bemeasured by the laser and the PSD. The haptic device hasa coil on the rotor with a polarity magnet. When current
flows through the coil, the coil experiences a force(Lorentz force) perpendicular both external magnetic
field and to the direction of the current flow. The rotor is
moved depend on an input waveform that has thefrequency less than 36Hz. An operator holds a moving
part like a pen and receives the thrust which is generated
in the rotor. Hereinafter, the moving part is called a pen
slide.
Fig. 3 shows the connection between the haptic
device and a microcomputer. The haptic device iscontrolled by command signals from the microcomputer
process.

Fig.4. Simulator

5.

Modeling of an object using by LEM

In the previous deformable virtual objects simulation,model of an object and reproduction of the deformation
were performed by the spring · mass · damper model.
However, It is difficult to model an object withcomplicated shape and measure spring-damper
coefficient between mass points. Therefore, the modelingof an object is performed by LEM. The deformation of
the object is also reproduced using LEM.
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan69

Development of the sense( =1

LEM (Long Element Method) is suitable for the
deformation reproduction of living tissue like surgicalsimulation. As a feature of LEM, The pressure, densityand volume required for displacement calculation can bemeasured relatively easily. Moreover, It is said that theaccuracy of deformation reproduction using LEM ishigher than that of spring mass point model. Fig. 5 showsa method of reproducing the deformation. The deformed
object is divided into LE(Long Element), and the surface
shape is reproduced by the set of deformation of each LE.

7. Deformation simulation
In the modeling using the mounted LEM, a part ofsilicone gel(αGEL, Taica) and salmon roe was modeledby LEM. Fig. 6 shows states of deformation simulation.
Fig. 7 shows a state of the deformable virtual objects
simulation combined with the haptic device.
A part of silicone gel spreading infinitely is modeled
for the silicone gel. Hemisphere of the salmon roe ismodeled for the salmon roe. As reference values formodeling the silicone gel, the size is 20 mm × 20 mm ×
10 mm and the Young's modulus is 28.9 kPa. As
reference values for modeling the salmon roe, thediameter is 6.2 mm and the Young's modulus is 5.58 MPa.
The cross-sectional area and the surface tension dependon the number of partitions.

6. Implementation of LEM
In modeling by LEM, mainly three equations aretreated. The first equation is given by equation (1) that
describes preservation of volume. Here, A is the crosssectional area, ΔL is the amount of displacement of LE,
and N is the number of LE.
=0(1)

The second equation is given by equation (2) which
explains LE in contact with the pen slide. Here, α
represents the LE in contact with the pen slide, and yrepresents the displacement amount of the pen slide.
∆=(a) silicone gel

+

∈

∆

(b) salmon roe

Fig.6. The simulation of deformation.

(2)

The third equation is given by equation (3) about the
LE not touching pen slide. Where E is the Young's
modulus, k is the surface tension, ΔP is the pressure
difference inside and outside the object, d is the density,
g is the gravitational acceleration, h is the distance fromthe surface Nbr indicates the i-th adjacent LE .
∆≠ )In this study, the deformation of a part of thedeformed object, not the whole, is reproduced. For thisreason, the displacement (ΔL) of each LE was obtained
from the equations (2) and (3) without using equation (1).

Fig.5. LEM model.

∆

;

−∆

−∆ =

ℎ

Fig.7. The simulation system combined with thehaptic device.

(3)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan70

Kaito Nagano, Eiji Hayashi

8. Conclusion
In this study, modeling an object and reproducing thedeformation of a sample is performed using by LEM. For
reproducing objects, the deformation of a part of silicone
gel and salmon roe is reproduced. By using LEM, thereproducibility is improved as compared with theconventional simulation.
Future research should focus on calculating force
from equations of LEM and generating calculated force
with the haptic device. Such a system would make itpossible to test smaller samples easily.

9. Acknowledgement
This research was partially supported by the Ministryof Education, Science, Sports and Culture, Grant-in-Aid
for Scientific Research, 2012.
10. References
1. T.OKADA(2017) Re-creation of a membrane puncture’s
sense of an object constituted of liquid and an outermembrane by a haptic device and a deformation
simulation of the virtual objects. International conference
on artificial life and robotics (ICAROB2017), Jan 19-22,
Miyazaki, Japan, 2017
2. Costa IF, Balaniuk R: LEM - An approach for real timephysically based soft tissue simulation. Proc of IEEEICRA 2001, Korea, 2001, pp. 2337-2343.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan71

A four-legged robot’s soft feet structural design and walking gait generated from inverse
kinematics
Amornphun Phunopas

Department of Production Engineering, King Mongkut’s University of Technology North Bangkok, 1518 Pracharat 1
Road Bangsue, Bangkok, 10800, Thailand
Eiji Hayashi

Computer Science and Systems Engineering, Kyushu Institute of Technology, 680-4 Kawazu, Iizuka
Fukuoka, 820-0067, Japan
E-mail: amornphun.p@eng.kmutnb.ac.th, haya@mse.kyutech.ac.th
www.kmutnb.ac.th

Abstract
The conventional wheel is famous in the industrial mobile robot because it is simple to build and easy to control andto maneuver. On the other hand, the legged robot is complicated to control but has a high performance in locomotion
like highly evolved legged-creatures. This research describes the four-legged robot design platform with the soft feet,
which are made by silicone-material forming. The robot’s feet are implanted by the strain gauge sensor. The robot
can percept the external force sensitively when its feet touch something. The research demonstrates the forward
kinematics and inverse kinematics of a four-legged robot with three joints for each. The result has validated thekinematic equations by implementing them on the real dog-like pet robot. The robot walked with a pure walking gait
and showed a sensing signal from the soft foot and ground contact.
Keywords: Soft material robot, Robot’s foot sensing, statics balance condition

is walking without paying attention to everyday walking
path. However, it is possible to make an accident. For
those views basically, the robot has to be able to sensewhen the feet contact the ground in every direction. For
the animal, the body is covered by skin which has a nerve
cell to sense the impact force. The reviewed works use
various methods regarding the sensors of the robot’s
perception. The humanoid robot foot has an effective
contact area (ECA) of perception. Thus, the sensor isa flexible force sensor array (FFSA) with the spatial
rectangle in Ref. 1. The robot transfers weight to its footwhile it is walking under a dynamic equilibrium criterion
or Zero Moment Point (ZMP). Next, a curvature sensor
module can be embedded into the soft-bodied robots in
Ref. 2. The sensor type is an electro-mechanical sensor
that is fabricated and merged into a soft robot. For an

1. Introduction
The legged robot consumes energy more than wheel typerobot. The equation of motion of the legged robot is also
complicated. However, the legged robot has a goodmaneuver and mobility like the revolute vertebrate
animal, which has legs. The robot can perform
mimicking from animal’s movement using amathematical model or a trained behavior. The smooth
floor is a primary test that the robot can repetitive it's
movement pattern when it walks with stable balance
control condition. Moreover, the robot has to adapt its
movement simultaneously on the stepping terrain or
multi-levels. In the high-level cognition in fast movement,
the human has the experiences to plan its motion before
moving for example in a hurdle race. Elsewhere, thehuman can sometimes focus on the smartphone while he© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

72

Amornphun Phunopas, Eiji Hayashi

application, a soft robotic grove has a soft fiberreinforced actuator to sense the human hand movement
as a human interface device in Ref. 3. Another sensor isa strain gauge, which is a skin-like, ultra-thin sensor. It
can be bent and used for prosthetic devices in Ref. 4.
There is research about soft robotic fingertips that is asimilar idea to its use as a robot foot soft tip. The soft
robotic fingertip has a metal bar as a bone, a body, and askin layer. The receptors are strain gauges and PVDF
films. The body is made of liquid silicone rubber in Ref.
5. The soft robot is a series elastic actuator (SEA) thatinterfaces with an environment. To control the soft robot
is designed with impedance-controlled compliant
actuators in Ref. 6. For previous work of this research inRef. 7, the robot has a plastics material for robot foot.
This paper presents to test the robot walking pattern witha defined static balanced condition and the sensing
signals from the robot foot.

However, it does not sense the torsional and bending
load. A strain gauge sensor has an amplifier circuit which
sends the analog signal to the analog to digital converter.

Fig. 2. The robot foot material is the end-effector of the robot
leg or the soft tip.

3. Kinematics
The summary of forward and inverse kinematics is used
to control the robot posture. The equation is individuallyapplied to each leg. First is the forward kinematics in Eq.
(1-3)
=−
+
−
−
(1)
=
+
+
(2)
=−
−
−
−
(3)
Where a1, a2, a3 and d0 are link’s length parameters.
Second is inverse kinematics in Eq. (4-6) that is thesolved equations from the Eq. (1-3) using the Harmonic
Addition Theorem. The singularity condition is that θ2
equals to zero degrees in case of dividing by zero in Eq.
(6).

2. The robot structures
The robot has four legs that have the same module foreach leg. It has three joints for shoulder, wing and, knee.
It has 12 joints. The robot size is approximately a small
pet in 1 cubic-feet volume. The robot weight is about two
kilogram including mini pc and battery. The robot body
frame is made from ABS plastic as in Fig. 1.
d0

1

2

a2

= acosa1
3

a3= acos= acos(x,y,z)

(

) (

(

(

)

)
(

(
(

)
)

)(

−

(4)
)

)

(5)
(6)

Where A, B, and C are derived coefficient.
The sequence of the robot walking gait and static balance
configuration is generated in the walking gait pattern.

Fig. 1. (Left) A robot leg configuration. (Right) The robot
structure with four legs.

2.1. Robot’s soft foot design
The robot feet are made from soft material which is
implanted a strain gauge sensor inside to sense the impact
force as in Fig. 2. From the sensor placement, it can senseon the vertical axial load because of its weight pressing
and distribute in various directions along the axial load.

4.

Walking gait pattern© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan73

A four-legged robot’s soft

The normal vector is perpendicular to the robot's bodyplane. The point of the normal vector represents the tilt
plane and the transfer CG position. The robot walks bylifting its legs one by one. Therefore, when the other threelegs are on the floor, the CG will transfer the ZMP area
to the floor area contacted by the three feet. The normal
vector will move along the diagonal axis 1 from point P2
to point P4 and along diagonal axis 2 from point P1 topoint P3, as shown in Fig. 5.

The walking gait possibly has many patterns under thecondition of a smooth floor with the Zero Momentum
Point (ZMP) area as a stability criterion. Commonly,
ZMP is used for dynamical walking, but in this case, it is
to initiate some conditions to make the robot walk
continuously with no falling head over heels or stumbling
as in Fig. 3. The robot configurations are posed bykinematic equations to keep all feet’s tip or the endeffector of the robot’s legs on the same planar floor. The
center of gravity is approximately in the ZMP area on thefloor plane. The robot's body plane orientates itself
around the XYZ axes, as shown in Fig. 4.

Fig. 5. From the top view, the point of the normal vector moves
in the unit circle of the normal vector area.
Fig. 3. The robot's body plane can change its planar slope whenchanging each leg's height on the smooth floor.

4.1. Control system
Respectively to the walking-gait pattern, the robot walks
in a sequence, sensing the floor contact simultaneously.
Fig. 6 shows the hardware and the software that support
the robot system. The computer sends a command to therobot by writing the motor's trajectory. The strain gauge
sensor amplifier is the differential bridge operational
amplifier circuit with gain adjustment. The output analog

Fig. 4. The defined normal vector of the robot's body plane has
three orientations: roll, pitch, and yaw.

The normal vector is created to define the robot's bodyplane in Eq. (7-8). In Eq. (9), there are three rotations,roll, pitch, and yaw, for the X, Y, and Z axes respectively.
= ̂+ ̂+(7)=√ + +
(8)
(
,
ℎ,
)= (
) (
ℎ) (
) (9)
Where a is the position of the normal vector on the Xaxis, b is the position of the normal vector on the Y-axis,
and c is the position of the normal vector on the Z-axis.

voltage is 0-3.5 volts, which is compatible to the analog
input of the general microcontroller.
Fig. 6. The soft tip foot of each leg can sense the floor contact
signal. Moreover, the computer reads the signals of the floorcontact detection.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

74

Amornphun Phunopas, Eiji Hayashi

immoderately deformed its shape. The strain gauge
sensor is very fragile and easy to be broken. Due to therobot moves dynamically, CG, inertia and external force
affect the system. The walking stability constraint is very
complicated to maintain its balance when the robot walks.

5. Result
The first experiment showed the signal from a robot footin Fig. 7 when it contacted the floor in the axial load.
Similarly, the four legs have the same result because it
has the same structure.

7. Conclusion
The robot can walk and sense the floor contact from thesoft foot using the implanted strain gauge sensor. The
foot tip position and trajectory can independently bedesigned depending on the solved kinematic equations
coherently. However, the walking gait in the fixed patternis very difficult to define the stability state. It had better
apply the control law or soft computation for robot
control.
Fig. 7. A soft robot foot contacted the floor. The analog voltage
signal is measured by oscilloscope.

Acknowledgements
King Mongkut’s University of Technology North
Bangkok and Kyushu Institute of Technology havesupported this research.

In the second experiment, the robot walks applying thedesigned walking gait. The walking gait is generated
from kinematic equations. For every walking step, one
leg paces and the remaining three legs are on the floor, as
in Table 1 and Table 2.

References
1. Wu, B., Wang, Z., Luo, J., & Wu, Z. (2013). Perception ofeffective contact area distribution for humanoid robot
foot. Measurement, 46(7), 2093-2098.
2. Ozel, S., Keskin, N. A., Khea, D., & Onal, C. D. (2015).
A precise embedded curvature sensor module for softbodied robots. Sensors and Actuators A: Physical, 236,
349-356.
3. Polygerinos, P., Wang, Z., Galloway, K. C., Wood, R. J.,& Walsh, C. J. (2015). Soft robotic glove for combined
assistance and at-home rehabilitation. Robotics andAutonomous Systems, 73, 135-143.
4. Sencadas, V., Mutlu, R., & Alici, G. (2017). Large area
and ultra-thin compliant strain sensors for prosthetic
devices. Sensors and Actuators A: Physical, 266, 56-64.
5. Hosoda, K., Tada, Y., & Asada, M. (2006).
Anthropomorphic robotic soft fingertip with randomly
distributed
receptors. Robotics
andAutonomous
Systems, 54(2), 104-109.
6. Calanca, A., & Fiorini, P. (2017). Impedance control ofseries elastic actuators based on well-defined force
dynamics. Robotics and Autonomous Systems, 96, 81-92.
7. Phunopas, A., Suwichien, N. (2017). Kinematic analysis
and trajectory of a dog-like robot. The 8th TSMEInternational Conference on Mechanical Engineering, 8,
101.

Table 1. One pace of the front left leg and the relative joint
angles (degree) of all legs.
Front Left Leg

Rear Right Leg

Front Right Leg

Rear Left Leg

t

θ1

θ2

θ3

θ1

θ2

θ3

θ1

θ2

θ3

θ1

θ2

θ3

0

20

-5

55

-20

-5

-55

-20

5

-55

20

5

55

1

20

-5

55

-19

-5

-64

-20

5

-55

20

5

55

2

18

-5

70

-19

-5

-64

-20

5

-55

20

5

55

3

18

-6

79

-20

-5

-55

-20

5

-55

20

5

55

Table 2. One pace of the front left leg and the relative foot tip
XZ-positions (mm) of all legs.
Front Left Leg

Rear Right Leg

Front Right Leg

Rear Left Leg

x

z

x

x

x

z

0

-103.57-103.07-103.57-103.07-103.57-103.07-103.57-103.071-103.57-103.07-103.57-93.07-103.57-103.07-103.57-103.072-87.86-102.65-103.57-93.07-103.57-103.07-103.57-103.073-76.01-103.07-103.57-103.07-103.57-103.07-103.57-103.07tzz

6. Discussion
The soft material is made of silicone. A hardness level isa maximum number, but it is still too soft to take the robot
weight. Consequently, the soft robot foot is© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan75

Powered Ankle Exoskeletons: Existent Designs and Control Systems
A.H. Weerasingha, W.P.K. Withanage, A.D.K.H. Pragnathilaka, R.K.P.S. Ranaweera, R.A.R.C. Gopura

Bionics Laboratory, Department of Mechanical Engineering,
University of Moratuwa, Katubedda, Moratuwa 10400, Sri Lanka
E-mail: amashiweerasingha@gmail.com, kesarawithanage5@gmail.com, kasun.adkh@gmail.com,
pubudur@uom.lk, gopurar@uom.lk
www.mrt.ac.lk

Abstract
This paper performs a comprehensive review on active ankle exoskeletons powered by external means. Initially,
anatomy of the ankle joint complex is highlighted. Next, associated research challenges and design difficulties arerecognized for the development of robotic hardware systems. Then, mechanisms, actuation methods, and control
methods are systematically presented. Here, exoskeletons designs from 2005 to 2017 were studied to recognize thelatest paradigm shifts. As such, this paper provides a classification, a comparison and an overview of related
technologies. Finally, design recommendations are made for the development of powered ankle exoskeletons.
Keywords: powered ankle exoskeleton, mechanisms, actuation methods, control methods.

achieve proper functionality while ensuring safety of thewearer. However, their performance in terms of
metabolic energy saving, ergonomic conformity, agility,
etc. need further improvements. In order to eliminate
such limitations, an improved understanding of theirfeatures and characteristics is paramount. In that context,
this paper focus on reviewing both mechanical andcontrol systems of recently developed ankle exoskeletonsat research-level. In order to limit the scope of the study
and to provide a broad understanding of the knowledge
base, powered ankle exoskeletons using different stateof-the-art technologies are selected for this review. In
particular, exoskeletons designs from 2005 to 2017 is
studied to recognize the latest paradigm shifts. In
summary, section 2 summarizes the design difficulties
arising from anatomical and/or biomechanicalcomplexity of ankle joint. Under section 3 and 4, existent
powered ankle exoskeletons which support locomotion
are reviewed based on actuation and control methods
respectively. The discussion in section 5 explains latest
trends, patterns and possible future directions fordeveloping assistive powered ankle exoskeletons.

1. Introduction
Robotic exoskeletons are classified as upper-extremity,
lower-extremity, and other systems.1 Lower extremity
exoskeletons (LEE) are predominantly used for gait
rehabilitation, power augmentation and motion
assistance.2 According to statistics around the globe,
elderly population exceeds 10% of the total population.
Conversely, people suffering with neurological or
muscular disorders are on the rise.3 As such, medical
industry has shown a growing interest to use LEE forrestoring patients’ mobility and to alleviate health careissues arising from locomotion difficulties.3
LEE is designed to support anatomical functions of
the six different human joints of the lower extremity. In
general, hip, knee and ankle have seven degrees of
freedom (DOF) per limb. Here, the ankle joint complex,
having tri-planar motions, plays a major role inperforming locomotion. Over the years, powered ankle
exoskeletons have been rapidly advancing with theintroduction of novel materials, actuators, and sensors.
Contemporary ankle exoskeletons have been designed to© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan76

A.H. Weerasingha, W.P.K. Withanage, A.D.K.H. Pragnathilaka, R.K.P.S. Ranaweera, R.A.R.C. Gopura

7

Fig. 1 Anatomy of ankle joint complex: (A) bone structure and DOF, (B) oblique axis of talocrural joint , (C) oblique axis of subtalar
7

4

joint , (D) muscles responsible for triplanar ankle motions .

2. Design Considerations for Ankle Exoskeletons

2.2. Design Difficulties of Ankle Exoskeletons

Anatomy of ankle joint complex and design difficulties
of ankle exoskeletons is summarized in this section.

According to biomechanics studies, synovial hinge joints
at ankle plays a crucial role in supporting motions ofhuman lower extremity8. The fitting of the exoskeleton at
the ankle is more difficult than hip and knee, due to strict
requirements of smaller space as well as anatomicalcomplexity introduced by talocrural and subtalar joints.7
The anatomical axes (see Fig. 1(B and C)) of talocruraland subtalar joints are placed in an oblique sense thatcause foot to move across all three planes allowing
pronation and supination to occur during walking.7
Hence, constraining motions to a single plane can lead toabnormal joint movements, poor muscle recruitment andincreasing overall energetic cost.8
Ground irregularities, body weight and walking speed
directly affect the mechanical stiffness assumed by ankle
and foot.9 Hence, the exoskeleton should adjust thesystem resistance to account for the variable stiffness ofthe biological counterpart during walking, running, etc.
If stiffness not properly adjusted, it may even lead to lossof stability or even injury of the joint.10
If a person weighing 75 kg walks at an average speedof 1.3 m/s, maximum torque and power generated at theankle joint according to kinetic analysis is 150 Nm and
200 W respectively.8 This is considerably greater than thecalculated values at hip and knee joints. A powerfulelectromechanical, pneumatic or hydraulic actuation
system can be introduced to satisfy this requirement,11
however such a system will ultimately increase size andweight of the exoskeleton. Here, bulky designs tend to
disrupt the normative motions of the lower extremity andadded inertia may burden the actuators energetically.12
Although, tethered systems are acceptable forrehabilitation applications, portability is an indispensable
trait for power-assist ankle exoskeletons.13

2.1. Anatomy of Ankle Joint Complex
The three significant DOF of ankle joint complex areplantarflexion-dorsiflexion, eversion-inversion andabduction-adduction (see Fig. 1(A)). Plantarflexion
during stance phase supplies majority of the energy
required for propulsion while walking.4 Range of
motions (ROM) for each DOF are plantarflexion 500 dorsiflexion 200, inversion 350 - eversion 250, andabduction 200 - adduction 250.5 However, studies shownthat ROM may vary with age and gender.6
Ankle movements mainly happens about talocruraland subtalar joints (see Fig. 1(A)). Talocrural joint isformed by talus, tibia and fibula bones.4 Top of the talus
fits inside the socket formed by tibia and fibula. This joint
is mechanically similar to a mortise and tenon joint
providing plantarflexion-dorsiflexion. Bottom of thetalus locates on the calcaneus forming subtalar joint thatallows eversion-inversion and abduction-adduction of
ankle.7 Thus joint complex supports triplanar motions.
Collection of muscles are responsible for generating
the motions of foot segment during ambulation (see Fig.
1(D)). The tibialis anterior, extensor digitorum longus,
extensor hallucis longus and fibularis tertius muscles
with tendons dorsiflexes the foot. The gastrocnemius,
soleus, plantaris, fibularis longus, fibularis brevis, tibialis
posterior, flexor digitorum longus and flexor hallucis
longus muscles with tendons plantarflexes the foot.
Fibularis longus, Fibularis brevis, Fibularis tertius
muscles and their tendons performs the eversion. Tibialis
anterior and tibialis posterior muscles along with related
tendons performs the inversion of ankle.4© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan77

Powered Ankle Exoskeletons:

Fig. 2. Ankle exoskeletons: (A) PAFO14 by Beijing Institute of Technology 2015, (B) autonomous exoskeleton15 by MIT 2014, (C)ankle exoskeleton16 by Carnegie Mellon University 2014, (D) soft exosuit17 by Harvard University 2013, (E) knee-ankle-foot robot18 by
NUS 2013, (F) achilles ankle exoskeleton19 by Delft University of Technology 2015 (G) AFO20 by Case Western Res. University 2012,(H) PPAFO21 by University of Illinois 2011, (I) soft orthotic device22 by Harvard University 2011.

walking during load carriage in 2014 (See Fig. 2(B)). A
device is a self-contained and has all necessarycomponents on board such as power supply, actuator andcontrollers. The bilateral exoskeleton was comprised ofthree main sub-assemblies such as a pair of fiberglass
struts attached to each foot, unidirectional actuators
mounted on the anterior of shank segments, and control
package including battery worn on the waist. Power istransmitted to the ankle joint by pulling on the struts
during propulsion phase. Features such as autonomous
operation, lightweight architecture and ergonomic
conformity have contributed to its successful
implementation.15
An exoskeleton that provide greater peak torque, peakvelocity and range of motion, was developed by Carnegie
Mellon University in 2014. It was actuated by a powerful
off-board motor and real-time controller with mechanicalpower transmission through a flexible and tethered
Bowden cable.16 The exoskeleton frame includes
rotational joints on either side of the ankle with axes ofrotation approximately co-linear with the human joint.
Each of the frames are attached to the foot and shanksegments (See Fig.2(C)). The foot mount has a lever arm
posterior to the ankle that wraps around the heel. A
Bowden cable tethered to the lever transmit the required
joint power to assist propulsion. Proposed system mainly
served as a universal platform to test influence ofpowered assistance.
The Soft Exosuit developed by Harvard University in2013 is a battery-powered portable device actuated by acable-driven system.17 It is essentially worn like a
garment with a backpack unit (See Fig. 2(D)). Hence, theeffective dynamic stiffness of the flexible suit and its
interaction with the lower extremity are important design
considerations. The sheath of the Bowden cable isstrapped to the suit, where one terminal is located at theFig. 3. Classification of powered ankle exoskeletons

3. Mechanisms and Actuation Methods
In this paper both actuation methods and mechanisms areclassified as in Fig. 3. Details are then presented underelectric, pneumatic, hydraulic, and other categories.
3.1. Electric Actuators
Beijing Institute of Technology in 2015 developed aportable powered ankle foot orthosis (PAFO), which
employs electric motor drive to provide plantarflexion
and dorsiflexion assistance (see Fig. 2(A)) for treatmentof ankle injuries.14 The powertrain is composed of anelectric actuator, which is the elementary unit to provide
energy for ankle movement, and a transmission system,that deliver power to the ankle joint at desirable speeds.
The transmission is composed of a harmonic drive, abevel gear drive, and a synchronous belt drive. Although
the device is capable of delivering higher torques, thesystem is bulky and heavy to warrant everyday use.
An autonomous battery powered exoskeleton15 wasdeveloped by MIT to reduce metabolic cost of human© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan78

A.H. Weerasingha, W.P.K. Withanage, A.D.K.H. Pragnathilaka, R.K.P.S. Ranaweera, R.A.R.C. Gopura

backside of the calf muscle and other connected to themotor drive unit carried on the back. A moment isgenerated directly about the biological ankle joint bymeans of pulling the heel upward using Bowden cable.
The cable acts in a similar way to the biological muscle
that stretches pre-stressed tendons. The Exosuit isdesigned to support power absorption through passively
extending as well as assisting the muscle contractions.
Series elastic actuator (SEA) is a type of electricallypowered actuation method with elastic spring elements
connected in series. National University of Singapore in2013 developed a portable knee-ankle-foot orthosis using
modified SEA which eliminate conventional limitationsof earlier SEA models.18 It (See Fig. 2(E)) consists of a
DC motor, a ball screw, a pair of spur gears and a pair ofspring assemblies.
Similarly, Achilles Ankle Exoskeleton developed by
Delft University in 2015 uses a SEA to provide push-off
assistance for healthy subjects during walking.19 Achilles
exoskeleton is comprised of three units. The two bootunits include SEAs, sensors and mechanisms to transmit
power to the human leg and, and a backpack unit, to carrybatteries and power management system (See Fig. 2(F)).
It is the most commonly used actuation method in recent
powered ankle exoskeletons. The active power submitted
to the ankle joint by the electric motor is supplemented
by the passive energy stored in the springs. Hence, anincrease of the overall energetic input to the joint results
in better performance of the actuator.
An Ankle Foot Orthosis (AFO) was developed byCase Western Reserve University in 2012 to assist
plantar-dorsiflexion.20 It is comprised of a series elastic
actuator, a four-bar linkage and a crank-slider mechanism(See Fig. 2(G)). Actuation force is derived from a pretensioned spring and torque is delivered to the ankle via
four-bar linkage. The drive link which the actuator acts
on is attached to the slider mechanism that support theinsertion point of the four-bar linkage mechanism.
Thereby the moment arm of the spring is dynamicallyadjusted. Yet, low speed of operation of this powered
device is a notable drawback.

through rotary pneumatic actuator mounted at the ankle(See Fig. 2(H)). The recycling scheme is used to capture
the exhaust gas from plantarflexion cycle and to drive thefollowing dorsiflexion cycle. The normal circuit consistsof two solenoid valves which can be used to control
rotary actuator in both directions. An additional solenoid
valve and custom accumulator were added into thepneumatic system, to implement the recycling scheme.
Bio inspired active soft orthotic device, developed by
Harvard University in 2011, has multiple pneumatic
artificial muscles (PAM) that imitate morphology andfunctionality of the actual biological muscles to control
plantarflexion-dorsiflexion
andinversion-eversion
movements.22 Flexible reinforced closed membrane, thecore element of PAM is attached between limb segments
to transmit mechanical power (See Fig. 2(I)). The
behavior of PAM closely represents the biologicalmuscles in generating forces and motions in addition toabsorbing shock loads. Bulkiness of the pneumatic circuit
limit its use for stationary rehabilitation robots andlaboratory experimentations.
3.3. Hydraulic Actuators
Ankle exoskeleton powered by hydraulic actuators aremost rare for locomotion assistance. BLEEX developed
by University of California in 2005 include an ankle joint
having three DOF.23 BLEEX uses double acting linear
hydraulic actuators, which have two opposite facing
pistons to achieve force control in both directions.
Although hydraulic actuators have the highest power-toweight ratio, the complexities in attaching a hydrauliccircuit on body is notably high.
3.4. Other Actuators
AFO developed by Osaka University in 2007 has been
actuated using a computer controlled Magnetorheological (MR) type damper, which uses viscousdamping to achieve resistive plantar-flexor torque during
selected phases of gait.24 Passive AFO was developed by
University of Toledo in 2014, which consists of a hinged
brace and series of SMA wires.25 The super-elastic wires
can store mechanical energy during plantarflexion andrelease it during dorsiflexion to compensate the inability.
These types of actuators are compact and lightweight
than conventional ones and behaves similarly to thenatural change in stiffness of ankle joint in normal gait.

3.2. Pneumatic Actuators
A Portable Powered Ankle-Foot Orthosis (PPAFO) was
designed by University of Illinois in 2012.21 Pneumatics
was used to supply plantarflexion and dorsiflexion torque© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan79

Powered Ankle Exoskeletons:

(A)

Valve

Pressure

Assistance
Plantar
Dorsi
None
Event trigger

Heel sensor
Toe sensor

Heel
Toe
TorqueGait event sensor sensor
assistOn Off On Off
Heel strike
Dorsi
None
Foot flat
Heel off
PlantarToe off
Dorsi

Fig. 4.
2015,

Gravity andfrictioncompensation

(B)
PPAFO
system- Δq Variable τ
impedance++
controller

Inner PD
torque+controller-

Fuzzy regulator Fuzzy regulator
swing phasestance phase
q̇

q̇

q

Contact 1

Contact 2q̇
GRF < On groundGRF < On groundor heel switch onq
HumanThresholdFoot force sensor

q̇

Swing

Ankle angle or
GRF out of range

Control algorithms in powered ankle exoskeletons: (A) event-based control method21 2011, (B) impedance control method29(C) finite-state control algorithm31 2004.

linguistic variables, the EMG signal and the acceleration
angle with one output linguistic variable, the motor status.
EMG signals contain important details useful todetermining motion intention. However, raw EMG
signals are random in shape and readings can be different
from user to user. Strength of EMG signal can be affectedby different thicknesses of muscle tissues and noisy
electrical environments. Further it cannot be used forpatients who have lost their muscle control.

In this paper, control methods of ankle exoskeletons areclassified (see Fig. 3) and presented under input-based,
output-based and other control methods.
4.1. Input-Based Control Methods
Event-based Controlling Method

In event-based control method, current gait phase isdetected by means of monitoring sensor inputs.
Accordingly, assistive torque is provided while speed andposition control is achieved using controller modules.
PPAFO developed by University of Illinois in 2012 usedan event-based control algorithm (see Fig. 4(A)). It
describes the gait phase detection and torque assistingalgorithm according to the force sensing input. Eventswere detected when sensor magnitudes exceeded thetuned user-specific thresholds for the heel and metatarsal
sensors.21 Similarly, Achilles ankle exoskeleton in 2015
was controlled with force and encoder feedback using an
event-based control scheme. A feed-forward signal was
taken as a function of gait phase by using a non-linear
filter that fits a number of primitives such as Gaussianfunctions to the signal to be extracted.19
4.1.2.

Lower
exoskeleton

GRF > On ground AND
ankle angle < Neutral
GRF > On ground AND
ankle angle > Neutral

Safe state

4. Control Methods

4.1.1.

τ

Custom-built
force sensor
×

I,D,K
Fuzzy impedance
regulator

q

+

(C)

4.2. Output-Based Control Methods
4.2.1.

Model-based Controlling Method

In a model-based control structure, a desired robotic
action is computed based on a human–exoskeleton model,
relying on the accuracy of the model which requires
series of sensors to recognize kinematics and dynamics
variables. Generally, this control method can be divided
into two types, based on the model used: the dynamic
model and the muscle model.27 The dynamic model canbe achieved by modeling the human body as rigid links
joined together, using mathematical method. And themuscle model predicts the muscle forces by taking EMG
signal inputs and gives an output of force estimation. A
Neuromuscular model (NMM) based controlled ankle
exoskeleton by NINRA in 2017 used a reflex-based force
feedback controller, to explore the adaptability tovariable speeds of the wearer.28

EMG-based Controlling Method

In this controlling method electromyography (EMG)
signals are taken from limb muscles to determine motion
intention of the wearer. EMG sensors produces a signal
which varies in amplitude proportional to the magnitudeof muscle recruitment. The AFO, developed by
University of Michigan in 2010, was controlled using
EMG-based control system.26 It consists of two input

4.2.2.

Impedance Controlling Method

The notion of impedance control method in rehabilitation
robots is to regulate the dynamic relation between theassistive device and the wearer by relating the position
error (joint angles) to interaction force/torque via© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan80

A.H. Weerasingha, W.P.K. Withanage, A.D.K.H. Pragnathilaka, R.K.P.S. Ranaweera, R.A.R.C. Gopura

mechanical impedance (output impedance of theexoskeleton, which is usually modelled as a mass-damper
spring system). High output impedance will increase theassistance from the exoskeleton to guide the patient’s
limb onto the reference trajectory. If the patient shows
greater control in the training, the desired output
impedance of the exoskeleton will similarly be low toallow the patient to deviate more from reference
trajectory. An AAFO (see Fig. 4(B)) designed by Natural
Science Foundation of China (NSFC) in 2015, wascontrolled using a fuzzy based impedance control
architecture.29

this paper. As such, Table 1 include a comparison of thetechnical characteristics of contemporary ankle
exoskeletons based on method of actuation. The mostcommon and recent actuation method used for powered
ankle exoskeleton is electrical actuators with different
types of transmission systems such as belt drives,
Bowden cable drives, harmonic drives and others
transmission methods. Even though SEA has a limited
bandwidth and performance issues, there is a notable
trend of their usage in powered ankle exoskeletons since
its inception. SEA is capable of storing passive energy inthe spring element and has low output impedance whichmakes it a suitable actuation method for ankle
exoskeletons. However, to overcome performancedrawbacks and sensor integration issues, continuous
improvements to SEA technology is required from afuturistic point-of-view. On the other hand, pneumatic
actuators have a notable advantage over electric actuators
as they are more comparable to biological muscles. The
artificial pneumatic muscles can remarkably mimic
human kinematics and dynamic behavior by closely
reproducing joint stiffness properties of the lower limbs.
A better actuation system with efficient powertransmission method can be developed by combining
different technologies reviewed in this paper. Even
though there is a trend of combining different actuation
methods in modern LEEs or prostheses, it has not yet
been realized with ankle exoskeletons. In general,
powered ankle exoskeletons designed for locomotion
assistance only focus on actively supporting one DOF:
namely plantarflexion and dorsiflexion, as evident form
Table 1. Instead, a hybrid actuation method can be
introduced to support multi DOF to overcome the key
design difficulties described under section 2.2. On theother hand, a modern ankle exoskeleton design shouldnecessarily focus in achieving ergonomic conformity,
metabolic energy saving and improving agility.
In order to support the complex kinematic anddynamic behavior of human motion, a combination ofcontrol methods can be implemented. However, no
existent exoskeletons have attempted to integrate control
strategies as a whole. An exoskeleton controller has tomeet several requirements. One of the key requirements
can be identified as controllability as per the user’s
intention. This requirement can be implemented using
EMG-based control method. But EMG signal generation
is inconsistent among user population.

4.3. Other Control Methods
Adaptive control method is a special type of nonlinear
control system which can alter its parameters to adapt toa changing environment. This can cover a set oftechniques which provide a systematic approach forautomatic adjustment of controllers, to achieve or tomaintain a desired level of control system performance,
when the parameters of the plant dynamic model areunknown and/or change in time. An AEO has been
proposed in 2016, which the human muscle activity is
feedback to adapt the assistive joint torque behavior.30
The triggers or transitional parameters for the finite statealgorithm are based on gait events (see Fig. 4(C)). The
MIT AAFO has been designed to assist ankle movementsof patients having drop-foot gait.31 Ground reaction force(GRF) sensors are used to determine the gait phase andankle angle sensor is used to determine the angle between
the shank and the foot. In a Contact-1 state (from heel
strike to mid-stance), the objective of the controller was
to prevent foot slap. During a Contact-2 state, (from midstance to toe-off) the controller minimized the impedanceof the brace so as not to impede plantarflexion
movements. Finally, during swing state, the user’s footwas lifted to prevent toe drag. Safe state is created to shut
down the device if any unexpected situation occurs.
5. Discussion and Future Directions
Powered ankle exoskeletons are commonly designed toperform selected task(s) of the biological ankle joint.
However, mechanisms, powering systems andcontrolling methods employed in modern robotic devices
are diverse. In that context, several research-level
powered ankle exoskeletons representing various
technological domains have been presented throughout© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan81

Powered Ankle Exoskeletons:

Table 1 Technical characteristics of modern ankle exoskeletons
Actuation
MethodName /
Institute / Year
PAFO14 /Beijing Inst. /2015
Auto. Exo. 15 /
MIT /
2014

Electric
Actuators

Pneumatic
Actuators

HydraulicActuators

Ankle Exo. 16 /Carnegie Mel.
Uni. / 2014
Exosuit17 /Harvard Uni. /2013
Achilles AE19 /Delft Uni. /2016
PKAFO18 /NUS/
2013Soft Orth. Dev. 22 /Harvard Uni. /2011
PPAFO21 /Uni. of Illinois /2011
BLEEX23 /Uni. of California/2005Powering
MethodHarmonic,
bevel and belt
drive, Brushless
DC motor
Belt drive,
Brushless DC
motor

Control
MethodEvent-based
control with
PID

Wt
(kg)
N/A

Sensory Information

Advantages

Limitations

Encoders, Groundreaction force sensors

Portable,High power andtorqueBulky, heavy

Event-based
control

Incremental opticalencoders

Low metabolic cost

Limited rangeof control

4.0

Bowden cabledrive, Offboard AC
servomotor
Bowden cabledrive, Brushless
DC motor
SEA, Brushless
DC motor

Adaptive
control with
iterative
learn.
Event-based
control

Optical encoders, Foot
contact switches,
Torque sensors

High peak torqueand bandwidth,
Lightweight

Tethered,
Bulky

0.87

Foot switches,
Force sensors

No rigid external
structure, Flexible

12.15

Low metabolic cost

SEA, Brushless
DC motor

Sensorintegrationand control
Needcompressed
air supply

0.85
(Ac.)

Pneumaticrotary actuator

EMG-based
control withPD
Event- andModel-based
hybridcontrol
Event-based
control

Incremental andabsolute encoders,
Pressure sensors
sEMG sensors, Linerpotentiometers, Rotary
encoders, IMUs
Strain sensors,
Pressure sensors,
IMUs

Heavy
backpack andsuit
Limited speedof operation

Potentiometers,
Force sensors

Event-based
control

Limitedduration ofoperation
Bulky, NeedhydraulicsupplyN/AHydraulicpistons

High systemefficiency,
Low fuel cons.
Active control ofmultiple DOF

PAM

Event-based
control

Encoders, Linear
Accelerometers,
Force sensors

Modular design,
High bandwidth
Lightweight,
Greater flexibility,
Higher Stability

N/A

0.95

N/A

N/A - Not Available

frequency and high-amplitude external perturbationinduced by robot-human interaction. This can cause
vibrations which may reduce the exoskeleton
performance.
In summary, the functional performance of the robot
device can be further enhanced with improvements toboth actuation method and control system. It is suggested
to perform experiments on hybrid powered ankle
exoskeleton which consist of combined actuation
methods, efficient power transmissions and modular
control systems to achieve multi DOF motions while
offering greater degree of agility and support to thewearer.

A patient with muscular disorder will fail to use an EMG
controlled devices effectively. To overcome this situation,
a separate (pressure/ force) sensor based control strategy
can be implemented concurrently with EMG.
Importantly, with the EMG controller the user can
instruct the device in rapid succession using electricalsignals of the muscles. However, it may be difficult tofind a stable correlation between the EMG signals and theoutput of the actuators.
Ability to assist user movements according to varying
conditions is also another consideration when developinga robust controller for ankle exoskeletons. Here, theadaptive controller architecture have been successful inachieving this goal. When multiple DOF are controlled,
the centralized single controller won’t be much effective,
as the single controller will be overburdened to control
multiple DOF. Thus, modular controllers would besuitable choice to assist the multiple DOF separately forincreasing system performance. Stability is the next issue
in designing a control system, which occurs due to high-References
1. M. Cenciarini and A. M. Dollar, “Biomechanicalconsiderations in the design of lower limb exoskeletons,”in IEEE Int. Conf. Rehab. Robot. (ICORR), 2011, pp. 1–6.
2. B. Chen et al., “Recent developments and challenges oflower extremity exoskeletons,” J. Orthop. Transl., vol. 5,
pp. 26–37, 2016.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan82

A.H. Weerasingha, W.P.K. Withanage, A.D.K.H. Pragnathilaka, R.K.P.S. Ranaweera, R.A.R.C. Gopura

18. H. Yu et al., “Mechanical design of a portable knee-anklefoot robot,” in IEEE Int. Conf. Robot. Autom., pp. 2183–
2188, 2013.
19. W. Van Dijk, C. Meijneke, and H. Van Der Kooij,“Evaluation of the achilles ankle exoskeleton,” IEEE
Trans. Neural Syst. Rehabil. Eng., vol. 25, no. 2, pp. 151–
160, 2017.
20. A. Polinkovsky, R. J. Bachmann, N. I. Kern, and R. D.
Quinn, “An Ankle Foot Orthosis with Insertion Point
Eccentricity Control,” in IEEE/RSJ Int. Conf. Int. Robot.
and Syst., 2011, pp. 1603–1608.
21. K. A. Shorter, G. F. Kogler, E. Loth, W. K. Durfee, and E.
T. Hsiao-Wecksler, “A portable powered ankle-footorthosis for rehabilitation,” J. Rehabil. Res. Dev., vol. 48,
no. 4, 2011, pp. 4594–472.
22. Y. L. Park et al., “Bio-inspired active soft orthotic device
for ankle foot pathologies,” in IEEE Int. Conf. Intell. Robot.
Syst., pp. 4488–4495, 2011.
23. A. B. Zoss, H. Kazerooni, and A. Chu, “Biomechanicaldesign of the Berkeley lower extremity exoskeleton(BLEEX),” IEEE/ASME Trans. Mechatronics, vol. 11, no.
2, pp. 128–138, 2006.
24. K. I. J. Furusho et al., “Development of Shear Type
Compact MR Brake for the Intelligent Ankle-Foot
Orthosis and Its Control,” in IEEE Int. Conf. Rehab. Robot.
(ICORR), 2007 , pp. 89–94.
25. L. Deberg and M. T. Andani, “An SMA Passive Ankle
Foot Orthosis : Design , Modeling , and Experimental
Evaluation,” Smart Mater. Res., vol. 2014, 2014.
26. P. C. Kao, C. L. Lewis, and D. P. Ferris, “Invariant ankle
moment patterns when walking with and without a robotic
ankle exoskeleton,” J. Biomech., vol. 43, no. 2, pp. 203–
209, 2010.
27. H. S. Lo and S. Q. Xie, “Medical Engineering & Physics
Exoskeleton robots for upper-limb rehabilitation : State of
the art and future prospects,” Med. Eng. Phys., vol. 34, no.
3, pp. 261–268, 2012.
28. S. M .T. Reza, N. Ahmad, I. A. Choudhury, and R. A. R.
Ghazilla, “A fuzzy controller for lower limb exoskeletons
during sit-to-stand and stand-to-sit movement using
wearable sensors,” Sensors (Basel)., vol. 14, no. 3, pp.
4342–4363, 2014.
29. J. V Mccall, S. A. Philius, R. W. Nuckols, and G. S.
Sawicki. Performance of a powered ankle exoskeleton
using neuromuscular model-based control over a range ofwalking
speeds[Online].
Available:
http://hpl.bme.unc.edu/McCall_ASB_2017.pdf.
30. L. Peternel, T. Noda, T. Petri, A. Ude, J. Morimoto, and J.
Babic, “Adaptive control of exoskeleton robots forperiodic assistive behaviours based on EMG feedback
minimisation,” PLoS One, vol. 11, no. 2, 2016.
31. J. A. Blaya and H. Herr, “Adaptive Control of a VariableImpedance Ankle-Foot Orthosis to Assist Drop-Foot Gait,”
IEEE trans. Neur. sys. Rehab. Eng., vol. 12, no. 1, pp. 24–
31, 2004.

3. N. Li, L. Yan, H. Qian, H. Wu, J. Wu, and S. Men,
“Review on Lower Extremity Exoskeleton Robot,” OpenAutom. Control Syst. J., vol. 7, pp. 441–453, 2015.
4. J.E. Muscholino, L. Whitney, Kinesiology – The skeletal
system and muscle funtion, 2nd edition, Mosby, 2011, pp.
124 -316.
5. T. R. D. J. Hamill and K. M. Knutzen, Biomechanicalbasis of human movement, 4th edition, Philadelphia :Wolters Kluwer Health, 2015.
6. B. M. Nigg, V. Fisher, T. L. Allinger, J. R. Ronsky, and J.
R. Engsberg, “Range of motion of the foot as a function ofage,” Foot Ankle, vol. 13, no. 6, pp. 336–343, 1992.
7. D. A. Neumann, “Ankle and Foot,” Kinesiology of themusculoskeletal system, 2nd edition, Mosby, 2010, pp. 1–
8.
8. Y. Fan and Y. Yin, “Mechanism design and motion control
of a parallel ankle joint for rehabilitation robotic
exoskeleton,” in IEEE Int. Conf. on Robot. Biomim.
(ROBIO), 2009, pp. 2527–2532.
9. J. L. McKay and L. H. Ting, “Optimization of muscle
activity for task-level goals predicts complex changes inlimb forces across biomechanical contexts,” PLoS Comput.
Biol., vol. 8, no. 4, pp. 1–17, 2012.
10. M. Cestari, D. Sanz-Merodio, J. C. Arevalo, and E. Garcia,“An adjustable compliant joint for lower-limb
exoskeletons,” in IEEE/ASME Trans. Mechatronics, vol.
20, no. 2, pp. 889–898, 2015.
11. M. Alam, I. A. Choudhury, and A. Bin Mamat,“Mechanism and Design Analysis of Articulated Ankle
Foot Orthoses for Drop-Foot,” Sci. World J., vol. 2014, pp.
1–14, 2014.
12. S. Superiore, S. Anna, N. Vitiello, S. Superiore, and S.
Anna, “Robotics and Autonomous Systems Review ofassistive strategies in powered lower-limb orthoses andexoskeletons," J. Rob. Aut. Syst., vol. 64, pp. 120–136,
2015.
13. K. Z. Takahashi, M. D. Lewek, and G. S. Sawicki, “Aneuromechanics-based powered ankle exoskeleton toassist walking post-stroke: a feasibility study,” J.
Neuroeng. Rehabil., vol. 12, no. 1, pp. 12–23, 2015.
14. Y. Bai, X. Gao, J. Zhao, F. Jin, F. Dai, and Y. Lv, “APortable Ankle-Foot Rehabilitation Orthosis Powered byElectric Motor,” Open Mech. Eng. J., vol. 9, no. 1, pp.
982–991, 2015.
15. L. M. Mooney, E. J. Rouse, and H. M. Herr, “Autonomous
exoskeleton reduces metabolic cost of human walking,” J.
Neuroeng. Rehabil., vol. 11, no. 1, pp. 1–5, 2014.
16. K. A. Witte, J. Zhang, R. W. Jackson, and S. H. Collins,“Design of two lightweight, high-bandwidth torquecontrolled ankle exoskeletons,” in IEEE Int. Conf. Robot.
Autom., pp. 1223–1228, 2015.
17. A. T. Asbeck, S. M. M. De Rossi, K. G. Holt, and C. J.
Walsh, “A biologically inspired soft exosuit for walking
assistance,” Int. J. Rob. Res., vol. 34, no. 6, pp. 744–762,
2015.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan83

Management of digital records inspired by Complex Systems with RADAR
Anne Jeannin-Girardon, Alexandre Bruyant

ICube laboratory, University of Strasbourg, UMR CNRS 7357, Strasbourg, France
Nicolas Toussaint, Ismaila Diouf

Complex System Digital Campus UNESCO Unitwin
Pierre Collet

ICube laboratory, University of Strasbourg, UMR CNRS 7357, Strasbourg, France
Pierre Parrend

ICube laboratory, University of Strasbourg, UMR CNRS 7357, Strasbourg, France, ECAM Strasbourg-Europe,
Schiltigheim France

Abstract
Storing sensitive data in a centralized way can lead to significant loss, should the central node or networks links be
defective as is the case in numerous countries, especially developing countries. Consequently, we propose aplatform called RADAR (Robust Anonymous DAta Records) that automatically fragments, replicates anddistributes data all over the network for anonymized, robust, encrypted storage and backup. RADAR gets nodes ofa network cooperate for storage and backups of sensitive data, even in case of non-reliable IT infrastructure.
Keywords: Epidemic replication, distributed storage, complex systems, anonymity

1.

Most approaches focus on improving access time,
bandwidth consumption while guaranteeing thatabsolutely no piece of data can be lost. This is very
costly and very difficult to achieve. Paper [1] proposes ahierarchical approach to ensure replicas are consistentover the network. The cluster of databases used to
implement this approach stores newest data at the
topmost level, while the lowest levels propagate replicas
from one level to the next in an epidemic way. This
allows them to define a range of Quality of Service thatis related to the consistency level that is required by theclient application. Epidemic replication protocol is
widely used to replicate data consistently in distributed
databases. This approach is evaluated in [2]: in practice,
it is very difficult to propose an analytic methodology toevaluate the variables associated with epidemic
replication (such as bandwidth consumption, replicas
consistency, ...) so the authors propose a simulator
allowing one to optimize the parameters of theirreplication algorithms that scale with their network
configuration. Across large scale networks, such as data
grids, hybrid approaches are used, such as in [3]. In

Introduction

In some of the most important countries around theworld such as India, China but also in Africa, Internet
access is still precarious and therefore not reliable. Thismeans that it is difficult for such nations to setup anduse digital platforms that would help them to bridge thegap with highly connected countries which, on theirside, are plagued with the opposite problem: having avery reliable network infrastructure makes them want todevelop deterministic and perfect data management
systems that can simply not be implemented in practice.
The RADAR system presented in this paper proposes amore pragmatic approach, that can overcome theproblems inherent to an unreliable network while at thesame time, providing a reliable enough Quality Of
Service that may satisfy the demands of high-tech
countries.
2.

Related work

Distributed databases all use data replication to improverobustness.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan84

Anne Jeannin-Girardon, Alexandre Bruyant, Nicolas Toussaint, Ismaila Diouf, Pierre Collet, Pierre Parrend

order to ensure efficient access to the distributed data of
the grid, a scalable replica management system (using
both hierarchical and flat topologies) and a dynamic
data distribution algorithm are used jointly. The
hierarchical topology contains nodes propagating
replicas to direct ancestors/children while the flat
topology propagates replicas using a Peer-to-Peer
protocol. Such protocols are also widely used in
distributed databases systems. Data in Peer-to-Peersystems are often assumed to be static and not subject tomany updates. But, for many applications, data arefrequently added, deleted or updated. In paper [4], theauthors propose an epidemic replication method
dedicated to Peer-to-Peer systems. One main
characteristic of Peer-to-Peer systems is that noassumptions can be made on whether the machines of
the network are online or not. Using rumor spreading
algorithms [5] and push/pull methods so that peers canrequest updates or send updates to peers that sent arequest update, the authors ensure some level ofconsistency for the data and achieve low latencies.

Modus operandi
The whole RADAR system is composed of many(ideally more than 50) loosely interconnected machines.
Because the network is deemed unreliable, machines
never “expect” anything from other machines unlessthey need to recover from a complete data loss.
Therefore, under normal operation mode, machines:● backup their own data by sending it to n othermachines they know about (using a nonconnected protocol such as UDP),● accept data from other machines for backup.
Whenever a machine needs to store a new local piece of
data, it is timestamped, encrypted using the local
cryptographic key of the machine, replicated anddepending on the required level of reliability, sent to nother machines randomly chosen in the network (themore required reliability, the greater n) for backup.
If the piece of data is of a sensitive nature (such as
medical records) it is furthermore fragmented into
several pieces before each piece is sent to n othermachines. For instance, medical records are fragmented
into:
1. patient personal data (name, address, telephone
number, e-mail address, ...),
2. medical textual data (sex, date of birth, weight,
blood pressure, a.s.o. along with symptoms,
diagnostics, medication, ...),
3. digital data (medical imaging, audiogram,
electrocardiogram, ...),
4. private data not related to medical condition(death of brother, ...) for a personal follow-up
of the patient.
For each patient, each fragment is encrypted and sent
separately to n machines chosen at random among the
known machines. This means that if data is stolen on agiven machine, patient data is not compromised because
for a particular patient, each machine will keep only oneof the 4 (encrypted) fragments. For instance, if medical
data are stolen, the attacker has no information about thepersonal data of the patient such as his/her name.

In this paper, we propose the RADAR model : it relies
on Complex Systems principles and does not need todefine complicated hierarchical architectures or
algorithms in order to ensure data consistency. We do
not make assumptions about the level of reliability of
the network on which RADAR is deployed, allowing
unreliable infrastructures to use RADAR for distributed
data storage. Moreover, our model ensures security andcan thus be used for sensitive data storage such as
medical records.
3.

Proposed RADAR model

Complex Systems
A Complex System can be defined as a set of
autonomous entities in interaction exhibiting multi-levelemergent behaviour where - as Aristotle said some 2300
years ago - the whole is more than the sum of the parts.
The typical example is turbulence. The local behaviourof particles results at a higher scale in the apparition ofswirls and vortices that can be described using NavierStokes equations. However these equations do notappear at all at the particle level.
RADAR implements storage entities that have no global
picture of the network, but that as a whole, behave like areliable and secure data storage system: machines relyon each other to automatically backup their data.

Security●

If a hacker gets hold of the cryptographic key
of one machine and gets access to thismachine, only the local data is compromised.
All other data stored as a backup for othermachines remains protected because it is© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan85

Management of digital records

encrypted with the cryptographic keys of theother machines.
● If a hacker gets hold of all the cryptographic
keys of all the machines in the network, andphysically gets access to one machine, the dataof the machine is compromised (as above) andthe hacker will be able to read all the fragments
stored by the machine as a backup for othermachines. However, because data is
fragmented, the hacker will either get thepersonal data of a patient, but not his medical
data or digital data or private data because
when data are backed up on the network ofmachines, they are fragmented and all thefragments are sent to different machines of the
network chosen at random.
To sum up, with RADAR, the only way to have accessto a particular patient’s record that is not locally stored
on one machine is to have full access (with passwords)
to enough machines in the network to hope to gain
access to all the fragments belonging to a particularpatient, which in practice is not feasible if the network
is made of hundreds of machines situated in different
places (hospitals, medical centers spread across thecountry).

This makes it possible to verify the origin and theintegrity of the data. A hash code is generated using thedata to be signed. The code is encrypted thanks to theprivate key of the sender. The receiver can verify theauthenticity of the signature by deciphering the hashcode and by comparing it with a new code generated on
the data.
Each node receives an RFC 5280 signed certificatedelivered by the administrator of the network, thatguarantees that the node is indeed part of the network.
Data loss and recovery
RADAR comes in when a local database is completely
lost and not recoverable. What happens next is that theperson in charge with the database that has disappearedcontacts the administrator of the network, asking for a
database rebuild. When the new machine is ready, it isadded to the network again, but in recovery mode. The
rebuild starts with one machine of the network passing
over the message to the other machines it knows(epidemic message transmission) that all files that havebeen received in the past from the failed machine must
be sent back to it. The machine in recovery mode thenstarts to receive from the whole network all the data it
previously sent. It uses the timestamps of the data toreconstruct its database in a consistent way. During therecovery, the machine keeps an account on the numberof replicas it has received for all its data. After recovery,
the machine sends again the missing number of replicas
of a given data to random machines, so that it sums backto n.

Hybrid encryption
RADAR uses the RSA [6] asymmetric encryption
algorithm with 2048 bits keys: it is an asymmetricalgorithm that uses a public key to encrypt the data, anda private key for decryption. Unfortunately, thisalgorithm is computationally intensive and cannot
natively encrypt data that are longer than its key (2KB
in our case). In order to circumvent these limitations, we
use RSA in conjunction with AES [7], a much faster but
symmetric algorithm (the same key must be used forencryption and decryption).

Data loss probability
It is very difficult to estimate the probability that when amachine has lost all its data, all n copies of a given
patient record are also lost at the same time. Computing
this probability is so complex that most similar data
replication systems use network simulation tools such as
Chive [2] and Network Simulator (NS) [8] to run
empirical tests to evaluate the robustness of theirapproach. Using the same methods shows that for most
realistic RADAR configuration and failure probabilities,
it is very difficult to effectively lose data.

Fig. 1. Hybrid encryption used by RADAR

Expected disk consumption

The data are encrypted thanks to AES using a randomly
generated key, that is itself encrypted with RSA (cf. Fig.
1).

Depending on n, the number of replications that ischosen, for the reliability of a particular network,
expected disk consumption is in average n+1 times the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan86

Anne Jeannin-Girardon, Alexandre Bruyant, Nicolas Toussaint, Ismaila Diouf, Pierre Collet, Pierre Parrend

3. H. Lamehamedi, B. Szymanski, Z. Shentu and E. Deelman,disk consumption of a single machine. Indeed, if one
machine sends n copies of its data to other machines in
the network, this means that it will receive data from (inaverage) n other machines in the network. Scaled to the
medical information system of Senegal, (for which
RADAR was initially designed) the system could use
around 1000 medical centers (with at least one machine
per center) to deal with a population of 15M inhabitants.
With a uniform distribution of patients per center(which is of course not the case), each machine would
store the medical records of 15k local patients. Withn=3, this means that each machine would need to store
the equivalent of 45k medical records, which is areasonable amount by 2018 disk storage space. This is
to be compared with the cost needed to create, maintainand operate a data center capable of dealing with all 15
million medical records of the country in a reliable andsecure way over an unreliable internet network.
4.

"Data replication strategies in grid environments," Fifth
International Conference on Algorithms and Architectures for
Parallel Processing, 2002. Proceedings., Beijing, China, 2002,
pp. 378-383.
4. A. Datta, M. Hauswirth and K. Aberer, "Updates in highly
unreliable, replicated peer-to-peer systems," 23rd International
Conference on Distributed Computing Systems, 2003.
Proceedings., 2003, pp. 76-85.
5. Laijun Zhao, Hongxin Cui, Xiaoyan Qiu, Xiaoli Wang, Jiajia
Wang, SIR rumor spreading model in the new media age, In
Physica A: Statistical Mechanics and its Applications, Volume392, Issue 4, 2013, Pages 995-1003
6. Xin Zhou and Xiaofei Tang, "Research and implementation of
RSA algorithm for encryption and decryption," Proceedings of2011 6th International Forum on Strategic Technology, Harbin,
Heilongjiang, 2011, pp. 1118-1121.
7. D. M. Alghazzawi, S. H. Hasan and M. S. Trigui, "Advanced
Encryption Standard - Cryptanalysis research," 2014
International Conference on Computing for Sustainable Global
Development (INDIACom), New Delhi, 2014, pp. 660-667.
8. C. Leng, M. Lehn, R. Rehner, and A. Buchmann, “Designing a
testbed for large-scale distributed systems,” in Proceedings of
the ACM SIGCOMM 2011 conference on SIGCOMM, New
York, NY, USA, 2011, pp. 400-401.

Conclusion

Where most other systems focus on perfect consistency
and reliability resulting in over-complex time
consuming algorithms which, eventually, are not 100%bullet-proof, RADAR accepts that data may be lost, but
focusses on reducing such occurrence to very low
probability. It offers a system that is simple, whilerobust enough to work over an unreliable network.
RADAR is currently tested on a huge patient cohort ofmore than 300 000 patients over 60 nodes in India.
Following work will concentrate on reliability thanks tolarge scale testing using network simulation tools.

Acknowledgements
The authors thank the N’Light foundation that funds thisproject, as well as the JIVA Ayurveda Indian company,on which RADAR is currently tested.
References
1. I. Arrieta-Salinas, J. E. Armend'riz-Iñigo and J. Navarro,"Classic Replication Techniques on the Cloud," 2012 SeventhInternational Conference on Availability, Reliability andSecurity, Prague, 2012, pp. 268-273.
2. A. Jiménez-Yáñez, J. Navarro, F. D. Muñoz-Escoí, I. ArrietaSalinas and J. E. Armendáriz-Iñigo, "Chive: A simulation toolfor epidemic data replication protocols benchmarking," 2014 9thInternational Conference on Software Engineering andApplications (ICSOFT-EA), Vienna, Austria, 2014, pp. 428-436.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

87

Integrated Optimization of Differential Evolution with Grasshopper Optimization Algorithm
Duangjai Jitkongchuen

College of Innovative Technology and Engineering
Dhurakij Pundit University, Thailand
Udomlux Ampant

College of Innovative Technology and Engineering
Dhurakij Pundit University, Thailand
E-mail: duangjai.jit@dpu.ac.th, udomlux.amp@dpu.ac.th
http://cite.dpu.ac.th

Abstract
This paper proposes a scheme to improve the differential evolution algorithm (DE) performance with integrated thegrasshopper optimization algorithm (GOA). The grasshopper optimization algorithm mimics the behavior ofgrasshopper. The characteristic of grasshoppers is slow movement in the larval stage but sudden movement in theadulthood which seem as exploration and exploitation. The grasshopper optimization algorithm concept is added to
DE to guide the search process for potential solutions. The efficiency of the DE/GOA is validated by testing on
unimodal and multimodal benchmarks optimization problems. The results prove that the DE/GOA algorithm iscompetitive compared to the other meta-heuristic algorithms.
Keywords: Meta-heuristic, Differential Evolution Algorithm, Grasshopper Optimization Algorithm, Optimization

algorithms are Differential Evolution Algorithms (DE)[2], and Biogeography-Based Optimizer (BBO) [3].

1. Introduction
Meta-heuristic algorithm techniques constitute range
from simple local search procedures to complex learning
processes. The basic concepts of meta-heuristics permit
an abstract level description. Nature-inspired metaheuristic can be grouped in four main classes: evolutionbased, physics-based and swarm-based.

The second class of meta-heuristic is physics-basedalgorithms that imitate the physical rules. The mostpopular algorithms are Gravitational Search Algorithm(GSA) [4], Galaxy-based Search Algorithm (GbSA) [5],and Curved Space Optimization (CSO) [6].

Evolution-based algorithms are inspired by the conceptsof nature evolution. The population in search process are
randomly generated which are developed oversubsequent generation. The next generation is created by
combination of the individuals in the previous generation.
These provide the population to be optimized over thecourse of generations. The most popular evolution-basedalgorithm is Genetic Algorithms (GA) [1]. Other popular

The third class of meta-heuristic is swarm-basedalgorithms that imitate the social behavior of swarms,
herds, flocks, or schools of animals. The most popularalgorithms are Particle Swarm Optimization (PSO) [7].
Other popular swarm-based algorithms are Bat
Algorithms (BA) [8], and Firefly Algorithm (FA) [9].

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan88

Duangjai Jitkongchuen, Udomlux Ampant

where j = 1, 2, ..., N. rand is a uniform random number
chosen in the range [0,1]. CR [0, 1] is the crossoverparameter. jrand is an index randomly chosen in therange [0, N].

The new meta-heuristic has been focused on this paper isdifferential evolution algorithm (DE) and to improve
performance of it, grasshopper optimization algorithm isadded to DE to improve performance in crossover
operation.

The last step is the selection operation to chosen a better
vector for next generation. The new generations are
selected by comparing the fitness value between thetarget vector and the trial vector.

The rest of the paper is structured as follows. Section 2
describes some backgrounds of the differential evolution
algorithm. In Section 3, the proposed algorithm ispresented. Section 4 shows the experimental results andthe conclusions will be discussed in Section 5.

X iG 1

2. The Differential Evolution Algorithm

3. The Proposed Algorithm
The schema of DE/current-to-best/1 in mutation
operation is updated by splitting F mutation parameter to
λ and F. The new mutation schema is showed in Eq. (4).

The procedure of differential evolution algorithm starts
to initialize target vectors X iG ( x1,i , x2 ,i ,...,x j ,i ,...,x N ,i )where i = 1, 2, ..., NP. NP is the population size. N is thedimension of the population. The superscript G identifies
the Gth generation.

ViGu j ,iv j ,ix j ,i,otherwise

jrand )

G
( X best- X iG ) F ( X rG1 - X rG2 )

G 1

(4)

G

- ( G c1 )(5)

While the mutation parameter F is used to perform theamplification of the difference between two random
population members. When the procedure is trapped in
the local optimum, the value of F is increased.

(1)

FG 1 FG

where r1 and r2 are integer number chosen from the set {1,G
2, ..., NP} and must be different from index i. X bestisthe best solution in the generation G. F is used to control
amplification of the differential evolution. The value of F
is set in the range [0, 2]. The target vectors and mutant
vectors are used to generate the trial vectors
U iG ( u1,i ,u2 ,i ,...,u j ,i , ...,u N ,i ) in crossover operation toincrease the diversity of population.
,if ( rand CR ) or ( jX iG

where λ is used to balance the difference vectors between
the best vectors and the target vectors. The value of λ isdecreased in each generation.

The target vectors are used to generate the mutant vectors
ViG ( v1,i , v2 ,i ,...,v j ,i , ...,vN ,i ) in next steps. The
DE/current-to-best/1 mutation scheme is showed in Eq.
(1).
G
X iG F ( X best
- X iG ) F ( X rG1 - X rG2 )(3)

The whole process is repeated until the termination
criteria are satisfied or a predefined number of iterations
are reached.

The differential evolution algorithm (DE) optimizes a
problem by sustaining a population of candidate
solutions and creating new candidate solutions by
combining existing ones according to its simple
procedure, and then keeping whichever candidate
solution has the best score or fitness on the optimization
problem at hand.

ViG

G
G
U iG ,if ( f ( U i ) f ( X i ))X iG ,otherwise( F G c2 )(6)

The parameters c1 and c2 are uniform random numbers.
c1 [0, 0.1] is a random value and c2 [0, 1] is a random
value. And the crossover parameter (CR) is regenerated
at the end of each generation to find an optimized
parameter. The new crossover parameter is generated by
Eq. (7).

CRG 1 CRG ( meanA ( SCR ) c3 )(7)

where meana is the usual arithmetic mean. SCR is theset of crossover parameter values. C3 [0, 1] is a uniform
random number.

(2)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan89

Integrated Optimization of Differential

The proposed algorithm has been evaluated performancewith thirteen benchmark functions [11]. The test
functions are unimodal f1 - f7 and multimodal
f8 - f 13 . The details of benchmark functions are shownin Table 1. The proposed algorithm was run 30 times oneach benchmark functions. The mutation parameters set
λ, F = 0.5, the crossover parameter set CR = 0.9 andpopulation size set NP = 100. Table 2 reported thestatistical. The performance of the proposed algorithm iscompared with two EAs: DE and Fast Evolutionary
Programming (FEP) and compared with swarm-basedalgorithm: PSO. The results of DE, FEP and PSO weretaken from the results reported in [12].

And the crossover operation is rearranged that starts withthe vectors that created from the target vectors and themutant vector is called new name is the prime vectors( U i ) . The prime vectors is used to generate the trial
vectors by comparing the fitness value between the prime
vectors and the mutant vectors, otherwise the trial vectors
is generated by grasshopper optimization algorithm.
The grasshopper optimization algorithm was proposed in[10] mimics the behavior of grasshopper. The
characteristic of the swarm in the larval stage is slow
movement and small steps but is long range and suddenly
movement in adulthood. These two characters seem as
exploration and exploitation. The search agents areencouraged to move suddenly in exploration and conduce
to move locally during exploitation. These two functions
as well as target seeking are performed by grasshopper
naturally.

According to the results of Table 2, DE/GOA is able toprovide very competitive results. This algorithm
outperforms all others in f1 , f 2 , f 3 and f 7 . Both
DE/GOA and DE were successful to find the optimal
solution in f 4 and f 6 . In the function f 5 , only DE
could solve the optimum. Therefore, these results show
the performance of DE/GOA in terms of exploiting the
optimum.

The proposed algorithm takes the feature of movement of
the grasshopper to generate a new value for the trial
vectors is showed in Eq. (8).

U id

x j - xi
ub - lbd
s x dj - xid
c
c d
di, j
2
j 1
NP

j i

T̂d

The multimodal functions have many local optima with
the number increasing exponentially with dimension.
This makes them suitable for benchmarking the
exploration ability of an algorithm. The experimental
showed that DE/GOA is able to provide very competitive
results on the multimodal benchmark functions as well.

(8)

where ubd and lbd is the upper bound and the lower boundin the Dth dimension. s( r ) fe-r l - e - r . d i , j is the
distance between the ith and the jth grasshopper. T̂d is thevalue of the Dth dimension in the target. The parameter c
is decreased equivalent to the number of iteration tobalancing exploration and exploitation is calculated as Eq.
(9).

c c max - l

cmax - cminL

5. Conclusions
This work proposed to improve the differential evolution
algorithm (DE) performance with integrated thegrasshopper optimization algorithm (GOA). Thirteen
benchmark functions were employed the performance interms of exploitation and exploration. The results showedthat DE/GOA was able to provide competitive resultscompared to well known heuristics such as DE, FEP and
PSO.

(9)

where cmax and cmin is the maximum value and theminimum value, respectively. l is the current iterationand L is the maximum number of iterations.
4.

The Experimental Results

Table 1. The Benchmark Functions© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan90

Duangjai Jitkongchuen, Udomlux Ampant
Functions

Range
N 2
i 1 xiNN
i 1 xii 1 xi2
N ( ii 1
j 1x j )f1( x )
f2( x )

f3( x )
f 4 ( x ) maxi

xi , 1 i[-100, 100][-100, 100][-10, 10]Nf9 ( x )

2 2
N 1
( xi
i 1 100 ( xi 1 xi )
2
N
0
5
x
.
i
i 1
4
N
random 0, 1i 1 ixi
N
x
sin(
xi )
i 1 i
N x 2 10 cos ( 2 x ) 10i
i 1 i

f10 ( x )20 expf11( x )1
4000

f5 ( x )f6( x )
f7 ( x )f8 ( x )f12 ( x )nu xi , a , k , m

0 .2
N 2
i 1 xi

10 sin y1

1
N[-100, 100][-30, 30][-100, 100][-1.28, 1.28][-500, 500]

N 2
i 1 xiexp

xiN
i 1 cos

1

i

1
N

N
i 1 cos 2 xi

k - xi - a m , xiyiyn - 1 2

yi 1
1

-a

2
n
i 1 x1 - 1 1[-5.12, 5.12]

20 e[-32, 32][-600, 600]

2
2
n -1
i 1 yi - 1 1 10 sin

k xi - a m , xi a
0,- a xi af13 ( x ) 0.1 sin 2 3 x1[-100, 100]1 )2ni 1 u xi ,10 ,100 ,4[-50, 50]xi 1
4

sin 2 3 xi 1

xn - 1 2 1 sin 2 2 xn

DE/GOA

DE

FEP

PSO

f1

2.35E-220

8.20E-14

5.70E-04

1.36E-04

f2

4.28E-11

1.50E-09

8.10E-03

4.21E-02

f3

3.50E-226

6.80E-11

1.60E-02

70.12562

f4

0.00E+00

0

3.00E-01

1.086481

f5

28.98017

0

5.06

96.71832

f6

0

0

0

1.02E-04

f7

2.11E-05

4.63E-03

1.42E-01

1.23E-01

f8-8831.53-11080.1-12554.5-4841.29

f9

6.76E-10

69.20

4.60E-02

46.70423

f10

4.44E-16

9.70E-08

1.80E-02

2.76E-02

f11

0

0

1.60E-02

9.22E-03

f12

2.87E-09

7.90E-15

9.20E-06

6.92E-03

f13

3.01E-08

5.10E-14

1.60E-04

6.68E-03[-50, 50]

4. E. Rashedi, H. Nezamabadi-Pour and S. Saryazdi, GSA: agravitational search algorithm, Information Science (2009),
pp. 2232–2248.
5. H. Shah-Hosseini, Principal components analysis by thegalaxy-based search algorithm: a novel metaheuristic forcontinuous optimization, Int. J. Computer Science andEngineering (2011), pp. 132–140.
6. FF. Moghaddam, RF. Moghaddam and M. Cheriet,
Curved space optimization: a random search based on
general relativity theory, (2012), arXiv:1208.2214.
7. J. Kennedy and R. Eberhart, Particle swarm optimization,
in Proc. IEEE Int. Conf. Neural Networks (1995), pp.
1942–1948.
8. XS. Yang, A new metaheuristic bat-inspired algorithm,
Springer Berlin (2010), pp. 65–74.
9. I. Fister, I. Fister Jr., XS. Yang and J. Brest, A
comprehensive review of firefly algorithms, Information
Science (2013), pp. 34–46.
10. S. Saremi, S. Mirjalili and A. Lewis, Grasshopper
optimization algorithm: theory and application, Advancesin Engineering Software (2017), pp. 30–47.
11. X. Yao, Y. Liu and L. Gao, Evolutionary programming
made faster, IEEE Transactions on Evolutionary
Computation (1999), pp. 82–102.
12. S. Saremi, SM. Mirjalili and A. Lewis, Grey wolf
optimizer, Advances in Engineering Software (2014), pp.
46–61.

Table 2. The Experimental Results
Functionsn
i 1u x1 ,5,100 ,4References
1. JH. Holland, Genetic algorithms, Scientific American(1992), pp. 66–72.
2. R. Storn and K. Price, Differential evolution – a simple
and efficient heuristic for global optimization overcontinuous spaces, Journal of Global Optimization (1997),
pp. 341–359.
3. D. Simon, Biogeography-based optimization, IEEE
Transactions on Evolutionary Computation (2008), pp.
702–713.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan91

Efficient collective search by agents that remember failures
Masao Kubo, Nhuhai Phung, Hiroshi Sato

Department of Computer Science, National Defense Academy of Japan,
1-10-20 Hashirimizu, Yokosuka, Kanagawa, 239-8686, Japan
E-mail: {masaok, ed17006, hsato}@nda.ac.jp

Abstract
The BRT agent is an algorithm that can find appropriate collective behaviors by changing the agreement contents ina trial and error manner. Computer experiments show that it is necessary to change the agreement contents thenumber of times that is proportional to the square of the number of choices on average. In this paper, we attempted
to shorten this search time by introducing an agent that memorizes behaviors that were not able to achieve theexpected effect of what we executed. As a result, we found that search time can be improved by just mixing a few
this taboo list agents.
Keywords: swarm intelligence, machine learning, complex systems

and notifies all of them. The agent changes its action inview of the number of supporters of each action. They
keep voting until actions of the all members are same.
This is a model that expands the 2-option consensus
building model against the agreement of Namatame3
and others so that it can be used even in the case ofmultiple options and incorporates the passage of time.
By appropriately designing the bias value θ which
expresses the degree of preference of the behavior
selected by itself, this algorithm has the feature ofreaching consensus in almost constant time regardlessof the number of agents in the group. A series ofcomputer experiments shows that it is possible to findsuitable consensus actions in a time proportional toapproximately M2 for the number of actions M. This isinefficient in cases other than dynamic environments
where the environment and tasks frequently change.
Here, we introduce a derivation agent of BRT thatmemorizes a set of behavior that were once executed
but not appropriate, and removes them from their own
action selection. In the following, this function is called
UEM (unsuitable experience memory), and an agent
having this function is called an UEM agent. The
memory used here is a short term, and it forgets

1. Introduction
Roughly speaking, if a set of required roles (for example,
collecting foods and resting) and the relationshipbetween these elements are embedded in individual
behavior beforehand, for example they can keep energyof them constant.
On the other hand, it is anticipated that it will not beable to lead to some reasonable macro states of theswarm by the above division of labor methods such as
when a new environment is encountered. They needmethods how to emerge the swarm in a better macrostate against such a situation.
To solve this issue we have to discuss individual's
learning but also learning capability as swarm and no
general models as to whether to discover, memorize,
and learn new micro interactions leading have not been
proposed. It has become a bottleneck in the constructionof a more advanced self-organizing swarm system4.
To this problem, a voting model (BRT model, Bias andRising Threshold model) has been proposed thatemerges a trial and error process at the macro level by
momentary change of agreement of an agent group6.
Every agent chooses one action from among the actions© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan92
1

Masao Kubo, Nhuhai Phung, Hiroshi Satoprobabilistically. As will be described later, if all agents
have UEM they can expect to discover a good
macroscopic behavior in proportion to M.
This paper is organized as follows. First, we will explain
the BRT and its characteristics. Next we propose an
UEM agent and estimate its characteristics. Finally, we
verify the characteristics of the proposed method bycomputer experiment. As a result, we found that it ispossible to reduce the time required for solution searcheven if all agents do not have UEM function.

2.2. Behavior example of BRT model and its
search time characteristics
Here we illustrate the behavior of the BRT model. We
will introduce its characteristics.
The fig.1 is rendition of what was described in our pastwork5. This figure shows the transition process from anagreed state. In the initial state, all of agents have agreed
to a0, but everyone agreed to another action (a1) in about
65 steps. Again, 1000 agents agreed to a0 action again in
125 steps. From the above it can be seen that agreed
behavior can be switched from either state.

2. Related works
2.1. BRT modelNow, we assumed that there are $n$ agents P1,…,Pn.
A={a1,..., aj,…, aM} is the set of agent’s behavior
candidates with M≧2. Aj(t)∈A is the behavior of agent
Pi at time t. The agent Pi has a bias θi (0<θi<1). n(aj) is
number of agents selecting behavior aj.
As a result of everyone performing the same action, amacro state of the group occurs, and if this is suitable
for the environment and tasks, this agreement behavior
is assumed to be "suitable". agoal is set of desirable
behavior candidates (agoal ⊂A). In advance, the agent
does not know agoal, only when all have agreed, it isunderstood that $a_{goal}$ includes that agreed
behavior.
At this time, the agent Pi decides the behavior Aj(t+1) at
the time t as follows. Ifn(A j (t))/Nic (t t i,last )Fig.2 Characteristics of search time cost by BRT

Here, we show that it is possible to agree with thedesired behavior by computer experiment. The
experimental results are shown in Fig.2. The + mark is
the average time required for discovery, and the ○ markis the value obtained by dividing it by M2. Since the ○mark is almost parallel to the x axis, it is mentioned thatthe average of the time required to discover the solution
is proportional to M2.

3. The proposed Agent(UEM agent)(1)

The BRT agent remembers the last action that wasagreed but did not suit, and select other behaviors.
Namely this short-term memory is limited to the mostrecent behavior. Then they repeatedly agree on
behaviors that were notsuitable a while ago. If the environment has not changed
since the last execution, this is not an efficient search.
Therefore, we will introduce an agent that memorizes
behavior that agreed but did not suit for a long time, andtried to make solution searching more efficient.
The agent has a list Li(t) that remembers actions thatwere agreed but not suitable, and adds it to this list withthe probability pin at the time when it is determined that
it is inappropriate while agreeing.

is satisfied, Aj(t+1)=Aj (t). Otherwise, behavior otherthan Aj (t) is stochastically chosen and becomes Aj(t+1)∈A\Aj. Here,τis a constant representing the incrementof proponents number. ti,last is the clock time at which
the agent Pi last changed its behavior, and (t-ti,last) is thetime, which the same action continues to be selected.

Fig.1 Transition example (Starting from agreed state)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan93

Efficient collective search by

Li (t 1)Li (t ) A i (t )Li (t )

probability p inotherwise

where Ai(t)={∀j,Ai(t)=Aj(t),Aj(t)≠agoal}

number of attempts to reach an agreement progresses, it
indicates that less agreement on already selected actions
will be reduced. Therefore, even if the population of the
UEM agent is small, the effect of shortening the searchtime can be expected.

(2)
(3)

When selecting a new action, select actions that are
not included in Li(t) stochastically. Namely,
Ai(t+1)⊂A \ Li(t)(4)
The agent stores it with Eq.4 and select a candidate with
Eq.2. This BRT agent will be referred to as an UEM(unstable experience memory) agent below.
Table 2. Probability to support one unselected action# trial BRT agent UEM agent
1
1
1st
M
M
1
1
2 nd
M 1
M 1
1
1
3rd
M 1
M 2
1
1
x th
M (x 1)
M 1

Fig.3 When changing the ratio of UEM agent

� + � ��( ) = 0� <3.1.��
Approximate
�( )� −characteristics∆� �� + �of+the proposed
methodFig.4 Relation between ratio of UEM agent and search time

Consider the ratio of UEM agents. The table 1 indicatesthe number of attempts to agree and the probability of
selecting an arbitrary unselected action. When trying anagreement for the first time, the probability of selecting
one action is 1/M with BRT, UEM agent. Both agentsknow that one action was inappropriate at the second
time, so both are 1/(M-1). However, in the third andsubsequent times, the behavior selection probabilities ofboth agents will differ. The BRT agent remains 1/(M-1),
but the UEM agent does not select actions that are
already selected as inappropriate, so candidates are
reduced and x(≦M). The selection probability of x trial
is 1/(M- (x-1)). In order to shorten the search time it isnecessary to reduce the number of times you agree withthe action you tried already. This corresponds to thevote of UEM agent.
M

u
(x 1)

4. The experimental result
This chapter shows the behavior of the UEM agent andconfirms that it has the expected characteristics. The
figure 3 represents the search time as a cumulative
probability density distribution, and the horizontal axis
is the time required. The vertical axis represents theprobability of finding a solution by this time. The solidline represents the case of pUEM= 0.1. The search time
has been found by 99% trial by 1000 steps. As estimatedby the formula 5, even a small number of UEM agentscan greatly improve search time.
The relationship between the ratio of UEM agent andthe average search time is shown in Figure 4. The
horizontal axis shows the ratio of the UEM agent, andthe vertical axis shows the average search time.
pUEM=0.0 requires an average of 834 steps, and pUEM =1.0 took an average of 462 steps. With pUEM = 0.001, it
is 810 steps, and search time can be improved just by
mixing minute UEM agents. As we increased the
proportion of UEM agents, dramatic improvements
were seen up to pUEM = 0.1 (meaning that pUEM = 0.1
required an average of 486 steps).

(5)

where u(<n) is the number of UEM agents. Thisequation shows that if the population of UEM agents islarge, it is less likely to choose actions that have alreadybeen tried. Also, as the denominator decreases as the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan94
1

Masao Kubo, Nhuhai Phung, Hiroshi Sato

From these two experiments, we found that the
proposed UEM agent can improve the search time of
BRT agent.
5. Conclusion
In this paper, in order to improve the efficiency of thetrial and error search behavior that the BRT agent group
emerges, we proposed a derived type of BRT agent fromthe candidates by recording the failed experience called
UEM agent. If all flocks are composed of UEM agents,
we can expect that the search time increases linearlywith M. However, even in the case of mixing with the
BRT agent, we estimate that even a small number of
UEM agents are effective, and verified the trend incomputer experiments. As a result, it turned out to show
the behavior as expected.
References
1. Eric Bonabeau, Guy Theraulaz and Jean-Louis

2.

3.
4.

5.

6.
7.

Deneubourg, “Fixed Response Thresholds and theRegulation of Division of Labor in Insect Societies”,
Bulletin of Mathematical Biology, Vol. 60, pp. 753–
807, 1998.
E. Castello, T. Yamamoto, F.D. Libera, W. Liu, F. Alan,
T.Wineld, Y.Nakamura, H.Ishiguro, “Adaptive
Foraging for Simulated and Real Robotic Swarms:
The dynamical response threshold approach”, SwarmIntelligence DOI, Vol. 10, pp. 1–31, 2016.
Akira Namatame (In Japanese), “Strategic Decision
Making”, Asakura Publisher, pp. 119–149, 2001.
Masao Kubo, Nhu Hai Phung, Hiroshi Sato, Akira Namatame, “Direction switch behavior to enclose a packof targets based on phase transition”, Swarm2015,
2015.
Nhu Hai Phung, Masao Kubo, Hiroshi Sato, Saori
Iwanaga, Akira Namatame, “Agreement algorithmwith trial and error method at macro level”,
Swarm2017, 2017.
Nhu Hai Phung (In Japanese), “A Study on Extensionof Enclosing Algorithm for Multiple Mobile Agents”,
Master’s thesis at National Defense Academy, 2017.
Wenguo Liu, Alan Winfield, Jin Sa, Jie Chen, and Lihua Dou, “A Strategies for Energy Optimization in a
Swarm of Foraging Robots”, Swarm Robotics, pp.14–
26, 2006.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

95

The Analysis of Band Structure of Photonic Crystals
Wang Chenxu

The school of Electrical Engineering and Automation, Henan Polytechnic University,
Jiaozuo, 454000/Henan, China *Fu ZiyiJiaozuo, 454000/Henan, China
next219@outlook.com, Fuzy@hpu.edu.cn
www.hpu.edu.cn

Abstract
In this paper, a finite element is developed to calculate the band structures of one dimensional photonic crystals. The
Maxwell’s equation governing the propagation of electromagnetic waves, which combine with Bloch theory, theFEM transform the complex band diagram problems into a simple eigenvalue by solving the eigenvalue withrespective to wave vector K by frequency. And then using the COMSOL for simulation.
Keywords: band structure, finite element method, Maxwell equation, eigenvalue, COMSOL.

There are many ways to calculate the energy band
structure, commonly used are the plane wave expansion
method, the finite difference time domain method, thetransfer matrix method, etc., but these methods have theirown drawbacks and limitations. For example, the use ofplane wave expansion method can only be limited tosimple geometry. Time-domain finite-difference
methods may not be useful for some practicalengineering problems in some cases. It is therefore
important to develop a solver that is fast and accurate incalculating the band structure.
In this paper, starting from Maxwell’s equation [4], toderive the scalar equation of wave equation, and using
the finite element method to get the eigenvalue matrix
equation, finally, using the COMSOL [6] for simulation,
COMSOL specify the scalar function and translate thescalar equation into weak form.

1. Introduction
Photonic crystals [1] are a kind of artificial microstructurematerial formed by periodic arrangement of the medium.
According to the periodic structural features in different
directions of space, photonic crystals are divided into
one-dimensional,
two-dimensionalandthreedimensional photonic crystals. When electromagnetic
waves propagate in photonic crystal, due to the periodic
arrangement of materials, Bragg diffraction [2] occurs andthe electromagnetic wave is modulated to form an energy
band structure. This energy band structure is called aphotonic band structure. If the ratio of the dielectric
constants of the constituent materials is large enough, abandgap may exist in the photonic band structure, whichis called Photonic Band Gap [3]. Ellipsoid, Brillouin, andBloch state in solid state physics can all be used todescribe photonic crystals.
As a starting point for the study of photonic crystal
theory, band structure is the object of all calculations. It
is one of the research focuses in the field of photonic
crystals to study the band structure of photonic crystals.

2. Numerical derivation of band structure ofphotonic crystals© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

96

Wang Chenxu, Fu Ziyi

The propagation of an electromagnetic wave through aperiodic medium (such as the photonic crystals) is
described by Maxwell’s equations.
∇× = +
(1)
∇×

∇∙

⃗∙ ⃗=

( ⃗) =

(2)

=−

∇∙ =0
(4)
where E and H are the electric and magnetic fields, isrelative permittivity, J and are the free charge andcurrent densities. With this type of medium in mind, we
can set =0 and J=0.
For mathematical convenience, this allows us to write aharmonic mode as a spatial pattern times a complex
exponential:(5)( , )= ( )( , )= ( )(6)put (5) and (6) insert into Maxwell’s equation, thewave equation can be expressed by the form ofelectric field:( )[∇ × ∇ × ( )] =

( )

1
( )

∅

−2

1
( )

∅

−

1
( )

∅=

∅

(11)

3. FEM solution
The finite element method is a numerical analysis methodthat is very important for solving the photonic crystals
problem. In FEM, it is first step to get the Weak form:∫ [ ()∅∅−2 ( ):
division points:(7)

In TE case, there is no electric field component in the
propagation.In this paper, assuming that Electromagnetic
waves propagate along the z axis, as shown in Figure 1.

∫{ ( )

L

x

∅

− ()

∅

−2 ( )∅−−[ ( )

∅+

=0

∅]

(12)

=0

∅]}

(13)

3.1. Discrete the area and constructs interpolation
functionIn this part, we will choose the second order element.
This can achieve higher accuracy without increasing thenumber of nodes. Every node has three nodes, in every
element, unknown function can be approximated as
second order function:∅ ( )=
+
+
(14)

z

y

Fig.1.The Multilayer film,1D photonic crystals. Where

,

(10)

( )

Setting Φ replace ( ), Φ = ( ), and put (10)insert into (2), we can get the scalar function:

(3)

=

̂∙⃗=

are different dielectric constants, L is spatial period.

In the TE case, E( ⃗)=(0, E, 0), (7) can be derived:− ( )∇( ⃗) =( ⃗)According to the Bloch State:( ⃗) =

( )

⃗∙ ⃗

(8)

solve the

∅
∅
∅

=
=
=

+
+
+

+
+
+

(
(
(

)
)
)

, , , put them insert into (14), we can get:
( )∅
(15)
⟹∅ ( )=∑

e: element i, j: local node
e:1,2,3,4…m
i, j: 1,2,3(9)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

97

The Analysis of Band

The interpolation function is given by:
( −
−
( −
=
( −
( −
=
( −
=

(

put (15) insert into (13):∑∅ ∫ { ()[()+)( − ))( − ))( − ))( − ))( − ))( − )
−2

]

}

( )

=0

−
(16)

Fig. 2. The band structure of a multilayer film for E-field formode.

sort out above equation, we can get eigenvalue matrix

for complex photonic crystals. weak form equations are
the usual approaches to general finite element
calculations

equation
∑

4.

∅

∫

=∑

Simulation by COMSOL

2∅ ∫ [ ()
( )−5. Conclusion
Discussed the basic characteristics of photonic crystals,
and introduced weak form and eigenvalue matrix
equation of one-dimensional photonic crystals fromMaxwell’s equations. Using COMSOL Multiphysics tocompute band structure of 1D photonic crystals.

(17)

COMSOL is a finite element solution software forsolving multiphysics. It can be used to numerically solve
the complex physical problems that are mutually coupled
by incorporating a variety of physics equations andcorresponding solvers. It is very important tool in physics
research. Itself is a finite element solver that can be set tosolve the equations needed for the weak form, and thensolved. However, not all problems can be solved by thebuilt-in weak form module. At this time, it is helpful toknow the weak form equation and its finite element
algorithm to solve practical physical problems. By setting
the material factor, selected Floquet periodic condition,using study to compute the frequency of any wave vectork.
Dielectric materials are alternately arranged, dielectric
constant =13, dielectric constant = 1.
When solving eigenvalues, we give the dependent
variable as frequency. Within the frequency range ofinterest, the eigenvector K, and the value of wave vector
K is limited to the simplest Brillouin zone. using weakform of COMSOL, we can carry out the band calculation

References
1. Sajeev John. Strong localization of photons in certain
disordered dielectric superlattices[J]. Physical Review
Latters, 1987,58(23): 2486-2489.
2. Eli Yablonovitch. Inhibited Spontaneous Emission in
Solid-State Physics and Electronics[J]. Physical Review
Latters, 1987, 58(20): 2059-2062.
3. J. D. Joannopoulos, R. D. Meade, and J. N.Winn. Photonic
Crystals: Molding the Flow of Light (Princeton University
Press, Princeton, 2008).
4. Hiroshi Isakari, Toru Takahashi, Toshiro Matsumoto.
periodic band structure calculation by the Sakurai-Sugiura
method with a fast direct solver for the boundary element
method with the fast multipole representation[J].
Engineering Analysis with Boundary Elements 68(2016)42-53.
5. Gagandeep Kaur, Amandeep Singh, Sappal Harjinder
Singh, “Band Gap Computation of 1-Dimentional Silicon
Photonic Crystal for High Contrast Grating Application”,
International Journal of Computer Applications (09758887) Volume 94-No.7, May 2014.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan98

Wang Chenxu, Fu Ziyi

6. J. Zhao, X. Li, L. Zhong, G. Chen, “Calculation ofphotonic band-gap of one dimensional photonic crystal.
Journal of Physics: Conference Series (Dielectrics 2009:
Measurement Analysis and Applications, 40th
Anniversary Meeting)”, 183 012–018. 2009.
7. Gagandeep Kaur, Amandeep Singh, Sappal Harjinder
Singh, “Band Gap Computation of 1-Dimentional Silicon
Photonic Crystal for High Contrast Grating Application”,
International Journal of Computer Applications (09758887) Volume 94-No.7, May 2014.
8. Y. Zhang, S. Bai, P. Chen, and S. Qu, “A novel secondorder DFP-based Volterra filter and its applications tochaotic time series prediction,” Proc. 31st Chinese Control
Conf., pp.1036-1039,2012.
9. Ayyub O B, Ibrahim M B, Briber R M, Kofinas P. SelfAssembled Block Copolymer Photonic Crystal for
Selective Fructose Detection[J]. Biosensors andBioelectronics, 2013, 46: 124-129.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan99

A Data Estimation Technique for Incomplete Telemetry Data based on a Genetic Algorithmwith Data’ Statistical Properties
Masahiro Tokumitsu and Kaito Mikami

Department of Control Engineering, National Institute of Technology, Yonago College, 4448, Hikonacho
Yonago, Tottori 683-0854, Japan
Fumio Asai

Mmeber, The Radio Amateur Satellite Corporation, 10605 Concord St, #304
Kensington, MD 20895-2526 USA
Taku Takada

Department of Social Design, National Institute of Technology, Kochi, 200-1, Monobe Otsu
Nankoku, Kochi, 783-0093, Japan
Wakabayashi Makoto

Department of Electrical and Computer Engineering, National Institute of Technology, Niihama College,
7-1, Yakumocho, Niihama, Ehime, 792-8580, Japan
Yoshiteru Ishida

Department of Computer Science and Engineering, Toyohashi University of Technology, 1-1, Hibarigaoka
Toyohashi, Aichi 441-8580, Japan
E-mail: tokumitsu@yonago-k.ac.jp
www.yonago-k.ac.jp

Abstract
Satellite in the space transmits their own mission data (so-called “telemetry data”) to ground stations. The telemetry
data contain a sort of data such as observations, experiments, and satellites’ health status. However, the data received
by the ground stations may contain errors by effects through the transmissions. This paper proposes a data estimation
technique for incomplete telemetry data to attain a high availability of the received telemetry data. The proposed
technique is based on a genetic algorithm with data’ statistical properties. This paper firstly demonstrates the proposed
technique with simple examples, and then it simulates our technique with actual telemetry data obtained by thesatellite.
Keywords: social diversity, intelligent information processing, telemetry data, data estimation, genetic algorithm.

scientific or technological missions. The CubeSats
usually communicate with ground stations in amateur
radio bands to transmit their data with a weak signal. The
satellites in the space transmit their own mission data (socalled “telemetry data”) to the ground stations. The
telemetry data contain a sort of data such as observations,

1. Introduction
Universities and Technical Colleges have developed
small satellites called “CubeSat” all over the world. Theydevelop the CubeSats to provide an opportunity forsatellite development. They also provide to achieve their© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan100

Masahiro Tokumitsu, Mikami Kaito, Fumio Asai, Taku Takada, Makoto Wakabayashi, Yoshiteru Ishidaexperiments, and satellites’ health status. However, thedata received by the ground stations may contain errors
by effects through the transmissions. The telemetry data,containing the errors, would reduce an availability of thedata. The major technical issues are as follows: (1) A
Doppler shift between a satellite and a ground station, (2)jamming by unlicensed radio stations, and (3) a signal
attenuation by an atmospheric attenuation.
In operations of the CubeSats, the transmitted data
usually contain additional redundant information to
detect or to correct the corrupted parts. The
implementations of the error correction depend on thepolicies of the satellite development projects. The error
correction is not implemented if the projects determine touse only error detection.
The errors contained in the telemetry data is anunavoidable issue in satellite communications. Some
organizations run interesting projects to support satellite
operational teams by gathering the telemetry data fromthe Internet. Volunteers submit the received telemetry
data to the servers to support operations of the satellite
projects. The telemetry data are published on thewebsites of the projects. Further, SatNOGS1 projectprovides information on the ground stations and thepositions of the satellites in maps.
The DESPATCH2 project by Tama Art University
showed an impact of aggregated support by thevolunteers. The DESPATCH project designed asculpture as the satellite, and then it launched it into deep
space. The sculpture equipped with the communication
devices. In this project, the volunteers supported themission by contributing the telemetry data. Finally, thecommunications with 4.7 million km between the
sculpture and the ground station were successful. The
communication of this immense distance was establishedby collecting the telemetry data provided by thevolunteers.
The SatNOGS and the Online Telemetry Forwarder3of the present active projects mainly focus collecting andpublishing the gather data on their website. The error
detection and correction would be feasible by intelligent
information processing with huge collected data. The
authors develop and operate “social diversity4”. The main
purpose of the project is to support satellite operational
teams by providing the processed data with intelligent
information processing. This paper proposes anintelligent information processing technique for data

estimation to recovery the incomplete data received by
the ground stations. The proposed technique is based ona genetic algorithm5, and then the technique estimates thedata with statistical properties of the data. This paper
evaluates the performance how many the proposed
technique estimate the original data. The remainder ofthis paper as follows. In Section 2, this paper describes
the proposed technique in detail and demonstrates two
illustrative examples. In Section 3, the simple experimentis presented. In Section 4, this paper discusses the issues
in the proposed technique and future work. In Section 5,
this paper summarizes this work.
2. Proposed TechniqueThis section describes the proposed technique indetail for incomplete telemetry data. The telemetry data
is expressed as multiple binary bits. The entire telemetry
data usually consists of multiple parts such as a header
information, satellites’ health status, and observation data.
The length of the telemetry data depends on the format ofeach data. Further, the telemetry data usually contains
redundant information for error detection or correction.
Some CubeSats implement only error detection totransmit the telemetry data. This paper assumes that thetelemetry data consists of the message body and a frame
check sequence (FCS) (Fig. 1).
This paper estimates the incomplete telemetry data byextending technique of the genetic algorithms. The
telemetry data in a packet level consists of multiple
binary bits. The genetic algorithm handles multiple
symbols to present individuals. The genetic algorithm issuitable for our purpose to present the digital data as theindividuals.
The proposed technique searches the best telemetry
data that matches the FCS if the FCS of the received data
is incorrect. However, a lot of the solutions can beconsidered as the candidate solutions in the estimation
process. The proposed technique uses the mean values as
the statistical property for searching the best solution.
Obviously, other knowledge and information of thetelemetry data are would be options for searching theoriginal telemetry data.
Fig. 1 shows an overview of the proposed technique.
The telemetry data consists of three data fields and the
FCS. The value of each data field is expressed 16 bits.
The estimated data is obtained if the proposed techniquefinds the data that matches the FCS. The error parts in the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan101

A Data Estimation TechniqueFig. 2 Test input data and its estimation result for theillustrative example 1. The symbol X in the frame
indicates unknown.

Fig. 1 Overview of the proposed technique by thegenetic algorithm with data’ statistical properties.
obtained telemetry data are corrected by the proposed
technique. The FCS with a short length detects the errorof the received data incorrectly. For the data estimation,
the probability of incorrect detection can be low bychoosing the FCS with the long length such as 32 bits.
The related technique for our data estimation is arainbow table attack6 that reverses input from a givenhash and a precomputed table. This attack is proposed forcracking a password. The proposed technique relates tothe rainbow table attack because our technique uses thepredefined mean values and the FCS.
This paper demonstrates simple two examples to
examine a significance of the proposed technique. Bothexamples use CRC-16 (CRC-CCITT) as the FCS. The
CRC-16 is one of the major FCSs and used in a sort ofapplications. The digits of the data are arranged in a bigendian style in the both cases. The number of individuals
in a population is 30. The two-point method is used for acrossover in the proposed technique. The fitness function
is defined to minimize an error between the values
presented by the individuals and it’s the mean value. The
individuals of the next generation are chosen by ageneration model. This paper implements a program forthe proposed method with Go language and a gago7. The
gago is an integrated library for implementing genetic
algorithms with supporting parallel processing and avariety of functions.
In the first example, the single data with 16 bits is
estimated by the proposed technique. The single data
presents an unsigned integer with 16 bits. The mean valueof the received data is only given to the proposed
technique. The mean value is 1000 and it is given to the
proposed technique as an input. The FCS is (347 ) .
The expected output is (1000) . In this example, theexpected value corresponds to the mean value of the best

Fig. 3 Test input data and its estimation result for theillustrative example 2. The symbol X in the frame
indicates unknown.
solution. In other words, the proposed techniqueoptimizes the solution to minimize the error between thecandidate solutions and the given mean value. This paper
carried out this example with the proposed techniquewith the above parameters. Eventually, the proposed
technique was successful for estimating the expected
solution. Fig. 2 shows the estimation sequence for the
first example.
In the second example, the triple data with 16 bits are
estimated by the proposed method. Each data alsopresents an unsigned integer with 16 bits. The meanvalues of the received data are also given only to the
proposed technique. The mean values of Data 1, Data 2,
and Data 3 are respectively(30022) , (19857) , and(10050) .
The FCS of the received data is (6459) . The
1,
2,
3) =
expected output values ((30000, 20000, 100025) . In this example, theexpected value does not correspond to the mean valuesof the best solution. This paper carried out this example© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan102

Masahiro Tokumitsu, Mikami Kaito, Fumio Asai, Taku Takada, Makoto Wakabayashi, Yoshiteru Ishida

Table 1. Numbers and names of the data fields

No.
1
2
3
4
5
6

Name of data nameBlack Chassis deg. C
Silver Chassis deg. C
Black Panel deg. C
Silver Panel deg. C
Solar Panel +X deg. C
Solar Panel -X deg. C

Fig. 4 The example of test input and estimated databy the proposed technique. The underlines of thedigits indicate errors.

Table 2. Estimated results by the proposed technique.

Item\ Number of data fields
3456Used data No.
1, 2, 3 1, 2, 3, 4 1, 2, 3, 4, 5 1, 2, 3, 4, 5, 6
Total length of frame
48
64
80
96
Number of total test input
500
500
500
500
Number of successful detection 346
283
237
202
Estimated success rate
0.692
0.556
0.474
0.404
with the proposed technique with the above parameters.
The proposed technique was successful for estimating thebest solution. Fig. 3 shows the estimation sequence forthe second example.

temperature. This paper chooses some kinds of the data
field from the telemetry data format. Table 1 shows thedata fields used in the simulation. Although the projectprovides sixteen kinds of the data in total, this paperchooses six kinds of the data fields. In the test dataset, all
the data fields represent half-precision floating-point
numbers that are presented by IEEE 754-2008 format
with 16 bits.
This paper evaluates the proposed technique bychanging the total length of the test telemetry data fromthree to six. The mean values of the data fields arecalculated by the data for 21 days (About three months).
The total number of the telemetry dataset is 24501. This
paper randomly chooses the test data from 10% of thedataset. The total number of the test dataset is 2451. The
test data contains randomized errors with two bits. The
FCS of each test data is calculated from the original data.
This section uses CRC-32 as the FCS instead of CRC-16
used in the previous examples because the authors
revealed that the proposed technique with CRC-16

3. Simulations
This section demonstrates the significance of the
proposed technique for the actual data transmitted by the
CubeSat. However, the telemetry data as the packet data
is not provided. The project only provides the numerical
data that present the satellite’s health status. This paper
manually makes the telemetry data from the publisheddata.
The simulation uses the dataset of FUNcube-18 (AO73) operated by AMSAT-UK. The dataset is provided by
the FUNcube-1 operational team in a website9. The
project website publishes weekly the telemetry data thatis recorded at every minute. The telemetry data of
FUNcube-1 contains its working status such as a solar
panel voltage, a battery voltage, and the battery© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan103

A Data Estimation Techniqueincorrectly estimated the original data. The proposed
technique with CRC-16 finds the incorrect solution thatmatches the given FCS where some parts of the incorrect
solution differ from the original data.
Table 2 shows the estimated results by the proposed
technique. This paper prepares 500 test inputs andexamines them by the proposed technique. The estimatedsuccess rate decreases as the total length of the data
increases. The short length of the frame is easier to findthe best solutions than using the longer-length frames
because the combinatorial patterns with the short-length
frames are small. The proposed technique obtained theestimated success rate 0.404 for the longest frame in thesimulation.

properties. This paper firstly demonstrated the proposed
technique with simple two examples. This paper also
demonstrated that the proposed technique estimated theoriginal data successfully.
Acknowledgements
The authors acknowledge useful comments andcontinuous encouragement in a collaborative education
forum for a design of networked satellites by thecollaborative project between Toyohashi University ofTechnology and Colleges of Technology in 2017. This
paper was supported by Grants-in-Aid for Scientific
Research in 2016 by the Telecommunications
Advancement Foundation. This paper also was supported
by Grants-in-Aid Program in 2017 by National Instituteof Technology, Yonago College. The KOSEN spacetechnology education project was supported by the
Human Resource Development program of the MEXT,
Japan, in 2017.

4. Discussion
The proposed technique is difficult to find the bestsolution where the values of the data fields deviate theirmean values. Because the fitness function in the proposed
technique optimizes the errors between the values
presented by individuals and their mean values. The
mutation models and crossover models produce morevariety of individuals than the present model. The
mutation models and crossover models would improve
the probability of finding the best solution.
The maximum length of the data estimation is 96 bits.
This length is not enough to apply to actual data such as
the telemetry data of the FUNcube. The length of the
FUNcube’s telemetry data is 2048 bits. The proposed
method needs to extend the limits of the maximum data
length. The long frame data would guild the problem
more difficult to find the best solution that matches thegiven FCS. However, the genetic algorithm by using
multiprocessing, GPGPU, and supercomputers would
relax this limitation.

References
1. Libre
SpaceFoundation,
SatNOGS
Network,https://network-dev.satnogs.org/ (2017). (Accessed on
20/12/2017)
2. ARTSAT, DESPATCH (FO-81) –ARTSAT,
http://artsat.jp/en/project/despatch, (2014). (Accessed on
20/12/2017)
3. M. Rupprecht (DK3WN), Online Telemetry Forwarder,
http://www.dk3wn.info/p/?p=69954, (2016). (Accessedon 20/12/2017)
4. Social diversity project, Social diversity: a global andintegrated telemetry receiving system with volunteers andan intelligent information processing, https://socialdiversity-devel.appspot.com/, (2017). (Accessed on
20/12/2017)
5. J. H. Holland, Adaptation in Natural and Artificial Systems,(1975, MIT Press).
6. P. Oechslin, Making a Faster Cryptanalytical TimeMemory Trade-Off, in Advances in Cryptology - CRYPTO
2003. CRYPTO 2003, Lecture Notes in Computer Science2729 (Springer, Berlin, Heidelberg, 2003).
7. M. Halford, GitHub - MaxHalford/gago: Golang genetic
algorithm library, https://github.com/MaxHalford/gago,(2017). (Accessed on 20/12/2017)
8. The FUNCube project, Welcome to the FUNcube Web
Site, https://funcube.org.uk/. (Accessed on 20/12/2017)
9. D. Johnson, G4DPZ, The Data Warehouse,
http://warehouse.funcube.org.uk/realtime.html?satelliteId=2, (2013). (Accessed on 20/12/2017)

5. Conclusion
Satellite in the space transmits their own telemetry
data to ground stations. The telemetry data may containsome errors by the effects through the transmissions. The
telemetry data generally have not only the message body
but also the redundant information to error the detection
or correction. This paper focused on the redundant data
for the error correction to estimate the original telemetry
data by the genetic algorithms. The proposed technique
is based on a genetic algorithm with data’ statistical© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan104

Study on Detection of Nests on Pylon from Overhead View Based on Halcon
Jiwu Wang1

School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University, China
Haibao Luo1

School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University, China
Pengfei Yu1

School of Mechanical, Electronic and Control Engineering, Beijing Jiaotong University, China
E-Mail: jwwang@bjtu.edu.cn,16121251@bjtu.edu.cn,16121256@bjtu.edu.cn

Abstract
Aiming at the disadvantages of low accuracy and efficiency of the existing methods of nest
detection, this paper proposes an effective method based on Halcon, which could detect the nestin the images from unmanned aerial vehicles(UAVs). This method eliminates most of the
background interference by Extracting XLD contour algorithm to identify the pylon area in theimage. Within this area, the fusion of texture and gray-scale features of nest are employed torealize the detection. An experiment result is presented to validate that this method can actuallyrealize the precise detection of nests on pylon with images shoot by the UAVs.
Keywords: XLD contour, pylon, nest detection, fusion of texture and gray-scale features

1. Introduction
detection method. This method is only better forthe bird's nest on the railway network Xu Jing etal[4] proposed a bird's nest detection method
based on the combination of color and texture
features. This method not only complicates thecalculation process when extracting the pylonarea, but also obtains the pylon area is still large
and does not eliminate complex backgroundinterference, which also poses great difficulties
for the final nest identification. Zhang Yilian etal[5] proposed a nest detection method based on
Coarse-to-fine detection on pylon. The method of
positioning the bird's nest is not accurate enoughbecause of the surrounding background offlowers and trees, mountains and mud whosecolor are very similar to nest, which will result ina large degree of false positives happen.
Therefore, how to accurately detect the pylon in acomplicated background and further search forthe nest's candidate position is the key todetecting the nest. Considering the shortcomings
of existing methods, this paper proposes an

In recent years, according to the statistics of theState Grid[1], the occurrence of transmission line
tripping caused by bird activities frequently.
Therefore, the detection of the bird's nest on thepylon has become a hot spot of current research.
in the existing detection algorithm for powertransmission system is mainly aimed at detecting
an object with special shapes and colors that are
obviously different from the background.
However, it's difficult to accurately detect andlocate the nests with the small size and no specificshape and color features. In recent years, somescholars have made a preliminary study on thedetection of the bird's nest on pylon. In recent
years, some scholars have made a preliminary
study on the detection of the bird's nest on pylon.
Sun F J et al[2] proposed a new kind ofrecognition scheme basing on digital image
processing. this method can only achieve goodresults if the image background is very simple.
Wu Xiao et al[3] proposed an image-based
railway contact net bird-nest abnormal condition© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

105

Jiwu Wang, Haibao Luo, Pengfei Yuof the object with obvious edge features such as wires
in the background of the image is also extracted. In thispaper, all the edge XLDs are screened by taking
advantage of the two important conditions of the outeredge contour of the pylon within a certain length andangular interval, and ultimately exclude the influenceof the image background object on the contour
extraction of the pylon. Converted to image format asshown in Figure 3:

effective method based on Halcon, which could
detect the nest in the images from unmanned
aerial vehicles(UAVs). This method eliminates
most of the background interference byExtracting XLD contour algorithm to identify thepylon area in the image. Within this area, thefusion of texture and gray-scale features of nest
are employed to realize the detection. Thereby
narrowing the detection range and improving thedetection efficiency and accuracy.
2. Pylon detection algorithm
In order to make the area of pylon in the captured
image appear to be compact, this paper adopts thec

UAV to take a picture from overhead view andthen extracts and filters the pylon sub-pixel edgedFig. 3. (c) is screening result image, (d) is final result
image

XLD contour algorithm to detect and extract thepylon area, that can greatly reduce the detected

3. Nest detection algorithm

range, effectively eliminate a large number of

Due to the material of the nest area itself and thecomplex background interference. Finally, based

influence of different illumination factors, the grayon this, a nest detection method based on texture

value of each pixel in the nest area is greatly different.

and gray value fusion is proposed. Algorithm ofTherefore, it is very difficult to separate the nest area

the overall process shown in Figure 1:

by the traditional threshold algorithm. The entropy of
the image is the judgment of how much the image
contains the information. It is of great benefit tocalculate the entropy of the image to judge the effect of
the image. The image with higher entropy has many
details and changes in brightness, which characterizes
theFig. 1. The overall process of the nest detection

clusteringcharacteristics

of

the

grayscale

distribution of the image. Therefore, this paper uses theentropy threshold segmentation technique to separate

Considering that the backgrounds in different images
are different, and it is difficult to separate the pylon
area from the complex background by using the
traditional threshold algorithm. Therefore, this paper
proposes a direct method for obtaining the sub-pixeledge XLD contour from the whole image. The effect isshown in Figure 2 (b):nest area from the image background the separated
effect shown in Figure 4 (b)By observing the result of threshold, it is found thatsome background pixels which the gray value is similar
to the nest area will not be eliminate. In this paper,Morphological opening operation is used to performthe pre-screening of nesting candidate regions. As
shown in Figure 4 (c), By using the area of the bird's
nest in a certain range, thus the other areas which areout of the range can all be deleted. The effect after theascreening is shown in Figure 4 (d).

bFig. 2. (a) is original image, (b) is XLD contour
When sub-pixel edge extraction is performed on thecontour of the pylon in the original image, the contour© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

106

Study on nest detectionabFig. . The curves of texture characteristics of the nest
and non-nest areas

4. Experiments
Using the images actually collected by UAV as acdata source, and it is a high-resolution colordimage of 6000 pixels multiply 4000 pixels.

Fig. . (a) is entropy image, (b) is entropy thresholdimage, (c) is opening operator image, (d) is
preliminarily detected result of the nest

As shown in Fig. 6, we can use the algorithm
proposed in this paper to test the images of the
bird's nest under two different background

After dividing the bird's nest area by the above-

conditions randomly, all of which can achievementioned method, although the large number ofaccurate detection.

background areas is removed, the final nest candidate
area may still contain non-nest areas. Thus, this paper
uses the fusion of the bird's nest texture and gray-scalefeatures to achieve accurate detection of the nest.
Considering the high roughness of the nest area,
resulting in a greater variance in the gray value of the
bird's nest area, so it is possible to eliminate thecandidate area of the nest whose gray value is not muchadifferent.

bFig. 6. The detection results of the bird's nest on thepylon under different backgrounds

The Grayscale Co-occurrence Matrix (GLCM) is acommonly used method for describing textures bystudying the spatial correlation of grayscale. So in the

5. Summary

paper, by selecting the area of the experimental test

According to the method proposed by theprevious researchers, the area of the pylon
detected is too large, which can’t well remove theinterference of the large number of complex
backgrounds to the final detection of the bird's
nest. By using the method of this paper, most of
the background interference can be eliminated,
thereby reducing the detection range andimproving the detection efficiency and accuracyof the bird's nest on pylon.

image that contains the bird's nest and the typicalbackground area, and calculate the texture feature
respectively to obtain the comparison curve of the
texture features in Figure 5. Through multiple
experiments, we found that the entropy of the bird's
nest area is generally higher than that of the non-nestarea, Entropy characteristics of the bird's nest area as
the final judge nest area conditions, to achieve thedetection of the nest on pylon.

References
1. Wang S H, Ye Z Q. Analysis of bird damage
accidents on overhead transmission lines andprevention
techniques[J].
High
Voltage
Apparatus, 2011, 47(2):61-67.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan107

Jiwu Wang, Haibao Luo, Pengfei Yu

2. Sun F J, Wang Z, Fan J Q. The Recognition
Research of Birds' Nest on Transmission Line
Towers[J]. Advanced Materials Research, 2013,
651:820-824.
3. Wu Xiao, Yuan Ping, Peng Qiang,et al. Imagebased railway contact net bird-nest abnormal
condition detection method:, CN 103745224 A[P].
2014.
4. XU Jing, HAN Jun, TONG Zhigang, et al. Methodfor detecting bird’s nest on tower based on UAV
image. Computer Engineering and Applications,2017, 53 (6):231-235.
5. Zhang Yilian, Chen Yong et al. Coarse-to-fine
detection for nests on pylon[J]. based on
information technology, 2017 (3): 104-109.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan108

Human gait recognition based on Caffe deep learning framework
Jiwu Wang1

School of Mechanical Electronic and Control Engineering of Beijing Jiaotong University of China
Feng Chen1

School of Mechanical Electronic and Control Engineering of Beijing Jiaotong University of China
E-Mail: jwwang@bjtu.edu.cn, 16125922@bjtu.edu.cn
www.bjtu.edu.cn

Abstract
Human gait recognition as an emerging biometrics technology has important theoretical significance and practicalvalue. At present, the research on human gait recognition is still in the stage of theoretical exploration. With thedevelopment of deep learning theory and technology, this paper will achieve human gait recognition of specifictargets based on Caffe deep learning framework through the faster-rcnn algorithm. The main contents of the paper
include the samples processing, model training and result testing.
Keywords: deep learning; faster rcnn; gait recognition; contour recognition; feature extraction.

1.

not well. Puspitasari proposes a method for gait
recognition based on Hough Transform4, he converts thehuman body information into the line information andgets the angle between the legs for further calculation.
Because of the occlusion and filming angle, therobustness of the method is poor.
In order to fulfil the accuracy and real-time
requirements of the recognition, and reduce theinterference that caused by some factors, such as thedifferent apparels and accessories, backgroundcomplexity and so on. As for the shortcomings ofexisting methods, this paper proposes a faster-rcnn
algorithm based on deep learning. We can use theconvolution neural network to extract and fit thefeatures of the body contour and the gait features fromthe prepared samples. Comparing the similarity with thefeatures of the experimental pictures. If the similarity

Introduction

The gait recognition is one of the main methods ofbiometrics, which verifies targets’ identity mainly
through their contours and behavior characteristics.
Compared with other biomeltrics methods, the gait
recognition has many advantages, just like long-distancerecognition, low-pixel recognition and non-contact
recognition. At present, there are more and moremethods of the gait recognition as the research develop
constantly. Among these methods, Bae et al. analyze
and research the stage of movement by Hidden Markov
Model1. Begg detects changes of the gait automaticallyby machine learning method2, which is limited to small
sample to recognize. Heng Zhang describes the contour
features of the human body by wavelet descriptor toachieve real-time recognition3, however, the accuracy is© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan109

Jiwu Wang, Feng Chen

exceeds the set threshold, the experimental pictures canbe regarded as the target.
2.

Sample Selection and Processing

The samples of this experiment is taken from the gait
database B5 that collected by the CASIA (Institute ofAutomation, Chinese Academy of Science). Amongthese samples, we choose 1503 pieces of pictures thattaked under 90° viewpoints include 1 to 4 characters,
and calibrate the target’s truth candidate region for theselected sample, as shown in fig.1 below:
Fig.2.The final feature map

Next, sending the feature map to the RPN (Region
Proposal Network) network. The network will put the
proposals map on the final feature map, and generate afixed-size feature map for each Rol through the Rolpooling layer. The proposal feature maps as the fig.3
follows:

Fig.1. Annotation example diagram

At the same time generate the corresponding file todescribe the picture. In practical samples preparationand target detection, there are some methods likebackground subtraction based on Gaussian of Mixture
Models can help process samples and experimental
pictures.
3.

Feature Extraction and Recognition

Fig.3. Roi_pool_conv5

The gait recognition of the target person mainlycompletes the feature extraction of the body contour andgait information of the target person. Completing theperson recognition main by comparising the features.
Each sample with the scale of 320x240x2 will be input
to the neural network layer to train. First, convoluting
and rectifing the samples twice and conducting max
pooling operation to get the initial characteristic data.
Next, putting the part of the feature data into the nextlayer of neural network to continue feature extraction,
so repeat 5 times until getting the final feature map. The
final feature map as the fig.2 follows:

Then, the combined training of classification probability
and bounding box regression is trained by Softmax Lossand Smooth L1 Loss. At last, drawing the loss andaccuracy curves of the training process and giving
backpropaga-tion data to make the proposals and thetarget frame position correct more precisely, which canmake the target feature extraction and description moreaccurate. The recognition will be better.
The experiment’s process is carried out on the basis ofdeep training. The experimental pictures are completed
feature extraction, feature comparison, classificationand marking sequentially. The result of the recognition
is displayed through the experimental pictures and theterminal.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan110

Human gait recognition based

convolutionalparameters.
4.

neuralnetwork

andadjustingExperiment
References

For the above deep learning method, we take somepictures that are not trained in the train process as the
samples of the experiment. Part of the pictures to bemodified accordingly in order to vertify theeffectiveness of the recognition.
In this experiment we selecte a total of 100 pieces ofpictures, include 70 pieces of normal pictures and 30
pieces of abnormal pictures. About the abnomal
pictures, there are 10 pieces of pictures that part of bodyof the target is blocked by accessories and other things,
there are 10 pieces of pictures that the person is similar
to the target but not the target, other pictures with thecomplicated background. The threshold of similarity issetted as: 0.85.
One of the experiment’s results as the fig.5 follows:

1. Bae J, Tomizuka M. Gait phase analysis based on a Hidden Markov Model [J].Mechatronics, 2011, 21(6):961–
970.
2. Begg R, Kamruzzaman J. A machine learning approach
for automated recognition of movement patterns using
basic, kinetic and kinematic gait data [J]. Journal of
Biomechanics, 2005, 38(3):401-408.
3. Heng Zhang. Gait Recognition based on the contour [D].
Nanjing University of Information Science & Technoogy,
2011.
4. Puspitasari C.IMPLEMENTASI HOUGH RANSFORMDALAM EKSTRAKSI FITUR GAIT MANUSIA.[J].
Fti/ia/skripsi/00398, 2011.
5. CASIA Gait Database [EB/OL].
http://www.cbsr.ia.ac.cn/english/Gait%20Database.asp

Fig.5. Sample annotation example diagram

The precision: 95%.
The spend time: 0.050s/piece of pictures.
5.

theSummary

There is the greate advantage on the accuracy and timeof recognition through the faster-rcnn algorithm basedon deep learning.Meanwhile, the method has the higher
requirements about the computer configuration,
especially the GPU. The samples selected and thenumber of the samples also have the significant effecton the recognition. As the amounts of samples is less inthe experiment, next experiment will be improved in thesample selection, amounts and other parameters, so as
to better enhance the recognition. In additon, we will
have more try on changing the layers of the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan111

Unsupervised Image Classification Using Multi-Autoencoder and K-means++
Shingo Mabu

Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai2-16-1
Ube, Yamaguchi 755-8611, Japan
Kyoichiro Kobayashi

Department of Information Science and Engineering, Faculty of Engineering, Yamaguchi University, Tokiwadai2-16-1
Ube, Yamaguchi 755-8611, Japan
Masanao Obayashi

Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai2-16-1
Ube, Yamaguchi 755-8611, Japan
Takashi Kuremoto

Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai2-16-1
Ube, Yamaguchi 755-8611, Japan
E-mail: mabu@yamaguchi-u.ac.jp, m.obayas@yamaguchi-u.ac.jp, wu@yamaguchi-u.ac.jp

Abstract
Supervised learning algorithms such as deep neural networks have been actively applied to various problems. However, inimage classification problem, for example, supervised learning needs a large number of data with correct labels. In fact, thecost of giving correct labels to the training data is large; therefore, this paper proposes an unsupervised image classification
system with Multi-Autoencoder and K-means++ and evaluates its performance using benchmark image datasets.
Keywords: neural network, deep autoencoder, K-means++, clustering

number of training data with correct labels are necessary
for the learning. Since it is a tough task to give correctlabels to many (hundreds ~thousands) images, this paper
proposes an unsupervised image classification(clustering) algorithm that does not use data with correctlabels to reduce the cost of preparing training data.
However, when we do not use correct labels, theclassifier cannot know the explicit information on theclasses. Therefore, sufficient information to distinguishthe differences between the images is necessary for
accurate classification. To obtain sufficient information,
we apply some image processing techniques to increase

1. Introduction
Recently, deep learning1 has been actively applied tovarious problems and shown distinguished performance.
Especially, in image classification problems, since deepneural networks can automatically extract featurescontained in each image2, it becomes easier to apply deepneural networks to image classification comparing to theconventional methods that extract features using
manually created feature extraction method.
However, classification systems using deep learning
are generally based on supervised learning. Thus, a large© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan112

Shingo Mabu, Kyoichiro Kobayashi, Masanao Obayashi, Takashi Kuremoto

the variation of the images. Then, the increased images
are inputted to Multi-autoencoder to extract feature
information. In summary, the proposed method has two
features; 1) increasing the variation of original images
using some image processing techniques, and 2)
extracting features using Multi-autoencoder that uses theoriginal and generated images. Autoencoder-based dataclustering has been already proposed3. However, the aimof the conventional method is to enhance Singleautoencoder, while the aim of the proposed method is toevaluate the effectiveness of Multi-autoencoder.
The rest of this paper is organized as follows. In
section 2, the proposed unsupervised image classification
algorithm is explained. In section 3, a benchmark image
dataset (caltech101) and experimental settings are
explained, then experimental results are shown. Section
4 is devoted to conclusions.

Start
Make variations of images
by image processing
Feature extraction by MultiAutoencoder
Image clustering by
K-means++
Stop

Fig. 1. Flowchart of the proposed method

2. Unsupervised image classification algorithm
The flow of the proposed method is shown in Fig. 1. First,
some image processing techniques (gray-scaletransformation and edge detection by Sobel filter4) areapplied to the original images to generate variations.
Second, several independent autoencoders (Multiautoencoder) are prepared (the number of autoencoders
is three corresponding to the number of datasets, i.e., theoriginal dataset and two generated datasets). Then,
feature values are obtained by the Multi-autoencoder.
Finally, the image clustering is carried out by K-means++algorithm.

Fig. 2. Feature extraction using image processing and Multiautoencoder

2.1. Feature extraction using Image processingand Multi-autoencoder
The image dataset used in this paper is caltech1015 thatis a set of color natural images (32 × 32 pixel) such as
watch, motorbike, airplane, grand piano, etc. We apply
gray-scale transformation and Sobel filter independently
to the original dataset, and two new datasets are
generated (Fig. 2). Then, the original and generated
images are encoded by Multi-autoencoder. Fig. 3 shows
9-layered autoencoder used for the feature extraction.
The original or generated images inputted to theautoencoder are encoded by second–fifth layers, decodedby sixth–ninth layers and finally reconstructed images
can be obtained. The autoencoder is trained so that themean squared error between the input and output values
is minimized. After finishing the training, the feature
values of the input image are obtained by the fourth layer.

Fig. 3. Structure of autoencoder

Here, let the original image data number be d, the feature
values of data d extracted by the autoencoder for the color( ), those for the gray-scale image be
image be( ) , and those for the Sobel filter image be( ) . Then, a feature vector combining
( ),
( ),
( )},
( ),
( ) ,
{
{
( ),
( )} or( ),( )isdefined by ( ) which is used in the next K-means++clustering6.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan113

Unsupervised Image Classification Using

2.2. Clustering by K-means++
Table 1. Parameters of Multi-autoencoder

K-means7 is a clustering algorithm that makes groups(clusters) of data where each data belongs to one of thegroups whose centroid is the nearest. K-means++ is anextension of K-means that improves the initialization ofcluster centroids. The centroid initialization method of Kmeans++ is summarized as follows.
1. Randomly select one data d out of N data as a cluster
centroid. Here, let m be 1.
2. Repeat the following until m=K (K is the total
number of clusters).
( ) which is the Euclidiani.
Calculate
and the nearest
distance between data
centroid.
ii.
Select one data ( = 1,2, … , − ) withprobability ϕ( ) as a new centroid.
( )ϕ( ) =
∑
( )
iii.
← +1

# of nodes(1st layer–5th layer)

3072 or 4096, 2048, 1024,
512, 256 or 171

Activation function
Pre-training epochFine-tuning epoch
Learning algorithm

Sigmoid
300
200
Adam

for the original images without feature extraction, 3)
clustering for the Histogram of oriented gradients (HOG)features8 of the original images. The parameters of Multiautoencoder are shown in Table 1. The number of layers
is nine, and the number of input nodes is determinedaccording to the sizes of input images. The size of theoriginal color images is 3072 ( 32 pixel × 32 pixel ×
3 channels ) and that of gray-scale images and Sobel
filter images is 4096 (64 pixel × 64 pixel × 1 channel).
Since gray-scale and Sobel filter images have one
channel, the size is enlarged to increase the number ofpixel information. The number of nodes in the 5th layer is
256 or 171 depending on the combination of
( ),
( ),
( ) . When two kinds offeature values are combined to make ( ), 256 nodes
are used and obtain ( ) with 512 (= 2 × 256) values.
When all the three kinds of feature values are combined,
171 nodes are used to obtain ( ) with 513 (= 3 × 171)values. This is for making the dimensional size of feature
vectors be almost the same between the different settings.
Multi-autoencoder is learned by stochastic gradientdescent method with Adam (adaptive moment
estimation)9 which is an online estimation method of
appropriate learning rate.
Table 2 shows the clustering performance of the
proposed method and some conventional methods
averaged over 10 independent trials. The proposed

By applying K-means++ to all the feature vectors ( ),similar images are put into the same cluster.
3. Experiment
The image data set is caltech101 which contains picturesof objects belonging to 101 classes. In this paper, imagesof 10 classes are selected and used for the experiment.
The 10 classes are airplanes, bonsai, faces_easy,
hawksbill, ketch, leopards, motorbikes, watch,
grand_piano, chandelier, and each class contains 90
images (totally 900 images). The clustering performanceof the proposed method is evaluated by F-measurecomparing to the conventional methods. The
conventional methods include 1) clustering for thefeatures extracted by single-autoencoder, 2) clustering
Table 2. Clustering performance
MethodImage set

Proposed methodwith Multi-autoencoder

Color+Sobel
Color+Gray
Gray+Sobel
Color+Gray+Sobel
Color
Gray
Sobel
Color
HOG feature

Conventional methodwith Single-autoencoder
Conventional method
without autoencoder

Precision [%]Recall [%]F-measure [%]

73.6
66.6
69.6
68.2
69.4
62.9
68.5
60.8
69.8

69.1
62.4
67.7
65.4
64.9
61.8
65.6
57.6
62.7

71.3
64.4
68.6
66.7
67.1
62.6
67.0
59.1
66.0

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan114

Shingo Mabu, Kyoichiro Kobayashi, Masanao Obayashi, Takashi Kuremoto

Table 3. Results of t-test
Methods
Color+Sobel vs. Color
Color+Sobel vs. Sobel

p-value
1.23 × 10
7.62 × 10

Color+Sobel: Proposed method with color and Sobel filter images
Color: Conventional method (Single-autoencoder) with color images
Sobel: Conventional method (Single-autoencoder) with Sobel filter
imagesFig. 4. Example of the generated cluster

extract features by Multi-autoencoders. From theexperimental results, it was clarified that the proposed
method showed better clustering accuracy than theconventional methods with significant differences. In the
future, we will make an application of the proposed
method, for example, medical image analysis where thenumber of data with correct labels is limited.

method with color and Sobel filter images shows the best
F-measure of 71.3%. The proposed method with gray andSobel filter images shows the second best F-measure of
68.6%. From these results, we can see that the Sobel filter
images can give additional useful information for theclustering. The proposed method with all the three kindsof images does not show the best result because too much(redundant) information would even cause bad effect on
the learning efficiency. Among the three settings ofSingle-autoencoder, the method with color images shows
the best F-measure; thus, we can say that the color
original image contains the useful information comparing
to gray and Sobel filter images. However, the differenceof F-measure between color and Sobel filter is small; thus,
we can see the Sobel filter still contains useful
information. As for the method without autoencoder,
HOG feature shows better F-measure than the method
using raw color images.
Table 3 shows the results of t-test for F-measure
between the proposed method with color and Sobel filter
images and the conventional methods with color or Sobel
filter images. We selected the methods for the t-test thatshow better F-measure among all the settings of the
proposed method or conventional methods. From Table
3, we can see that there are significant differences
between the proposed method and conventional methods.
Fig. 4 shows an example of the generated cluster.
This cluster mainly contains grand_piano images;
however, some other classes of images whose featuresare very similar are also contained. The squares in Fig. 4
show the images that are not grand_piano. The proposed
method is based completely on unsupervised learning,
thus it is difficult to realize perfect clustering due to thelack of class information. Nevertheless, the proposed
method can generally grasp the essential features forselecting similar images.

References
1. Y. LeCun, Y. Bengio and G. Hinton, Deep learning,
Nature. 521 (2015) 436–444.
2. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio and P. A.
Manzagol, Stacked Denoising Autoencoders: Learning
Useful Representations in a Deep Network with a Local
Denoising Criterion, Journal of Machine Learning
Research, 11 (2010) 3371–3408.
3. C. Song, F. Liu, Y. Huang, L. Wang and T. Tan, Autoencoder Based Data Clustering, CIARP2013, Part I, LNCS
8258 (2013) pp. 117–124.
4. N. Kanopoulos, N. Vasanthavada and R. L. Baker, Designof an image edge detection filter using the Sobel operator.
IEEE Journal of solid-state circuits, 23(2) (1988) 358-367.
5. L. Fei-Fei, R. Fergus and P. Perona, One-shot learning ofobject categories. IEEE transactions on pattern analysis
and machine intelligence, 28(4) (2006) 594–611.
6. D. Arthur and Dergei Vassilvitskii, k-means++: The
Advantages of Careful Seeding, Proc. of the 18th annual
ACM-SIAM symposium on Discrete algorithm (2007), pp.
1027–1035
7. J. MacQueen, Some methods for classification andanalysis of multivariate observations, Proc. of the Fifth
Berkeley Symposium on Mathematical Statistics andProbability, Volume 1: Statistics, (1967), pp. 281–297.
8. N. Dalal and B. Triggs, Histograms of oriented gradientsfor human detection, In IEEE Computer Society
Conference on Computer Vision and Pattern Recognition,
2005, 1 (2005) pp. 886–893.
9. D. Kingma and J. Ba, Adam: A method for stochastic
optimization, arXiv preprint arXiv:1412.6980 (2014)

4. Conclusions
This paper proposed an unsupervised image
classification algorithm using Multi-autoencoder and Kmeans++, which can increase the useful information by
applying some image processing and automatically© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan115

Anomaly Detection of Disaster Areas from Satellite Images Using Convolutional Autoencoder
and One-class SVM
Kohki Fujita
Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai 2-16-1, Ube, Yamaguchi
755-8611, JAPAN
Shingo Mabu
Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai 2-16-1, Ube, Yamaguchi
755-8611, JAPANTakashi Kuremoto
Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai 2-16-1, Ube, Yamaguchi
755-8611, JAPANE-mail: {w084vg, mabu, wu}@yamaguchi-u.ac.jp

Abstract

In recent years, research on detecting disaster areas from satellite images has been conducted. Whenmachine learning is used for disaster area detection, a large number of training data are required; however,
we cannot obtain so much training data with correct class labels. Therefore, in this research, we propose
an anomaly detection system that finds abnormal areas that deviate from normal ones. The proposed
method uses a convolutional autoencoder (CAE) and One-class SVM (OCSVM).
disaster images. Therefore, in this research, we propose
an anomaly detection system that finds abnormal areasthat deviate from normal ones. The proposed method
does not use abnormal (disaster) images for the training,but only uses normal images. The proposed method
consists of a convolutional autoencoder (CAE) and Oneclass Support Vector Machine (OCSVM)4. First, CAE
executes feature extraction from satellite images with
unsupervised learning method. Second, OCSVM istrained using the extracted features of normal areas only.
Finally, anomaly detection for the testing images iscarried out by the trained OCSVM. The proposed method
does not need training data with correct labels ofabnormal areas, thus the cost of collecting training data
can be reduced.
This paper is organized as follows. Section 2
describes the proposed method including CAE and OC
SVM. Section 3 introduces a satellite image data and the

1. Introduction
Japan is a country with many natural disasters such as
sediment-related disasters, earthquakes, volcanic
eruptions, typhoon, etc. compared to foreign countries. In
order to make provisions against such natural disasters, it
is important to quickly find abnormalities. Therefore,
there is a demand for a system to find the anomaly assoon as possible. In recent years, research on detecting
disaster areas from satellite images has been conducted1,2.
In addition, since machine learning, especially, deeplearning3 has shown distinguished abilities in image
classification problems, it is expected to apply machinelearning to disaster detection problems. When machinelearning is used for disaster area detection, a large
number of training data is required, however, we cannot
obtain so much training data with correct class labels, andespecially, it is quite difficult to obtain a large number of© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan116

Kohki Fujita, Shingo Mabu, Takashi Kuremoto

when applying CNN to the recognition of printed
characters, CNN can automatically extract importantshapes contained in the images such as vertical,
horizontal and diagonal lines, etc. in the convolutional
layers. By stacking the convolution layers, it becomes
possible to identify features in complex pictures andphotographs and classify each image. Convolutional
autoencoder (CAE) (Fig. 2) is a kind of CNN that is basedon unsupervised learning and executes feature extraction
from input images. Fig. 2 shows a CAE structure used inthis paper that consists of convolution, pooling andunpooling layers. The aim of the learning is the same as
autoencoder, that is, the input images are reconstructed in
the output layer. In fact, CAE can encode the input
images using the layers of Convolution 1 and Pooling 1,
and then decode using the layers of Convolution 2,
Unpooling 1 and Convolution 3. Therefore, the values
obtained at the layer of Pooing 1 can be used as feature
values of the input images. The activation function is
Rectified Linear Unit (ReLU)6.

Fig. 1. Flowchart of the proposed method

2.2. One-class SVM (OCSVM)
Fig. 2. Convolutional autoencoder (CAE)

SVM is originally a classification algorithm for twoclasses using supervised learning. OCSVM (Fig. 3) is anunsupervised learning algorithm that applies SVM toregion discrimination problems to estimate high-density
region of data. In other words, OCSVM is anunsupervised outlier detection method. OCSVM istrained using images of normal areas only. Anomaly
detection for the testing images is carried out by thetrained OCSVM and abnormal images can be detected
when they are out of the normal area. OCSVM has aparameter ν that determines the proportion of normal andabnormal areas. For example, when ν is set at a large
value as shown in Fig. 4, the normal area becomes small.
In this case, only the data that is very near to the centerof the normal area is regarded as normal. Conversely,
when ν is small, the normal area becomes large and it iseasier for the data to belong to the normal area as shownin Fig. 5. Because there is a trade-off between the false
positive (abnormal) and false negative (normal), thevalue of ν should be determined carefully.

simulation conditions and shows the simulation results.
Finally, Section 4 is devoted to conclusions.
2. Proposed method
A flowchart of the proposed method is shown in Fig. 1.
First, a satellite image is given as an input, convolutional
autoencoder (CAE) is trained and feature values are
extracted from the input image. Then, One-class SVM(OCSVM) is trained using the feature values of normal
areas. Finally, normal and abnormal areas of the testing(unknown) images are detected by the trained OCSVM.
2.1. Convolutional Autoencoder (CAE)
Autoencoder5 trains a neural network so that the input
and output become the same. In the same way as thegeneral neural network, the weights of the network arelearned by stochastic gradient descent. Autoencoders
have been used in dimensional reduction and expression
learning. Convolutional neural network (CNN)3 is known
to have higher performance than the conventional neuralnetworks such as multilayer perceptron (MLP),
especially in image and speech recognition. For example,

3. Simulations
3.1. Simulation conditions
In this paper, a satellite image data of Yagi–Midorii area
in Hiroshima, Japan (Fig. 6) are used for the evaluation© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan117

Anomaly Detection of DisasterFig. 3. One-class SVM (OCSVM)

Fig. 6. Satellite image and the area of landslide (Shading area
is excluded from the original image)Fig. 4. OCSVM (ν = Large)(a) Normal(b) Abnormal

Fig. 7. Examples of patch images (32 × 32 pixel)

Fig. 5. OCSVM (ν = Small)of anomaly detection. In 2014, large-scale landslides
occurred in this area due to the torrential rain. The
satellite image was taken by Advanced Land Observing
Satellite-2 (ALOS-2) that can observe the surface of theearth anytime without being influenced by day/night andweather due to its Synthetic Aperture Rader (SAR)7. Fig.
6 shows the preprocessed satellite image (1327 × 900)
and the lines in Fig. 6 are manually traced landslide areas.
The upper left shading area is excluded from the image
for the simulations because that area is located on theother side of the mountain ridge from ALOS-2 and theobtained signal is weak. This image is divided into
32 × 32 pixel patch images as shown in Fig. 7 and eachpatch is regarded as a region of interest (ROI) that isclassified by the proposed method. Then, the patch
images are divided into training data (Training), testingdata 1 (Test 1) and testing data 2 (Test 2). As shown in
Fig. 8 and Table 1, “Training” contains 589 images ofnormal areas, “Test 1” contains 70 images of normal
areas (that are not overlapped with Training), and “Test2” contains 70 images of abnormal (landslide) areas. In
the next subsection, the detection ability of the proposed
method is evaluated and compared with the method using
OCSVM only.

Fig. 8. Division of data for making training and testing data
Table 1. Number of patch images
Training
Test1
Test2

589
70
70

3.2. Simulation results
Table 2 shows the precision of the detection obtained by
the proposed method (CAE + OCSVM) where ν is set atvarious values in the range between 0.2 and 0.7. Table 3
shows the precision obtained by OCSVM with the samesettings of ν. From these tables, we can see that the
precision of the proposed method for Test 2 is better than
OCSVM only for all the settings of ν. When we pay
attention to the precision of Test 1 and Test 2 in Table 2,
the precision of Test 1 is better when ν is smaller, whilethat of Test 2 is better when ν is larger. If more than 80%© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan118

Kohki Fujita, Shingo Mabu, Takashi Kuremoto

Table 2. Precision of CAE + OCSVM (proposed method)

Precision [%]Training (normal)
Test1 (normal)
Test2 (abnormal)

0.2
80.1
78.7
65.9

0.3
70.1
69.7
80.3

0.4
59.9
60.3
85.3

ν

0.5
50.0
45.4
89.4

0.6
40.1
39.6
91.4

0.7
30.0
31.7
93.0

Table 3. Precision of OCSVM

Precision [%]Training (normal)
Test1 (normal)
Test2 (abnormal)

0.2
80.0
77.7
2.9

0.3
70.0
70.3
11.4

detection rate of abnormal areas is required, ν should be
set at 0.3 or more, and if more than 90% is required, νshould be set at 0.6 or more. However, the larger ν causes
larger false positives (over detection); thus, the value ofν has to be determined considering the balance between
the false positive and false negative rates. Generally, thefalse negatives should be minimized not to miss thedisaster areas; in this sense, larger ν is desired.

0.4
60.0
60.4
15.7

ν

0.5
50.0
50.6
21.6

0.6
39.9
35.6
38.0

0.7
30.0
30.3
56.0

4. L. M. Manevitz and M. Yousef, One-Class SVMs forDocument Classification, Journal of Machine Learning
Research, 2 (2001), pp. 139–154.
5. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio and P. A.
Manzagol, Stacked Denoising Autoencoders: Learning
Useful representations in a Deep Network with a Local
Denoising Criterion, Journal of Machine Learning
Research, 11 (2010) pp. 3371–3408.
6. V. Nair and G. E. Hinton, Rectified linear units improve
restricted boltzmann machines, Proc. of the 27th
international conference on machine learning, (2010) pp.
807-814.
7. K. Ouchi, Principles of Synthetic Aperture Rader for
Remote Sensing, 2nd edn. (Tokyo Denki University Press,2009).
8. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S.
Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A.
C. Berg and L. Fei-Fei, ImageNet Large Scale Visual
Recognition Challenge, International Journal ofComputer Vision, 115(3) (2015) pp. 211–252.

4. Conclusions
In this study, an anomaly detection algorithm using CAE
and OCSVM for satellite images was proposed. CAE
automatically extracts features from image data, andOCSVM detects abnormal areas. From the simulation
results, the proposed method showed better detection
ability of the abnormal areas comparing to the methodwith OCSVM only. Therefore, the effectiveness of thefeature extraction by CAE was clarified. As future work,
we will apply trained neural networks using ImageNet8
such as VGG and GoogleNet for enhancing the feature
extraction ability.
References
1. A. Bormudoi and M. Nagai, A remote-sensing-based
vegetative technique for flood hazard mitigation of Jiadhalbasin, India, Natural Hazards, 83(1) (2016) pp. 411–423.
2. N. Tamkuan and M. Nagai, Fusion of Multi-Temporal
Interferometric Coherence and Optical Image Data for the2016 Kumamoto Earthquake Damage Assessment, ISPRSInternational Journal of Geo-Information 2017, 6(7), 188(2017).
3. Y. LeCun, Y. Bengio and G. Hinton, Deep learning,
Nature, 521 (2015) pp. 436–444.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

119

Improving EEG-based BCI Neural Networks for Mobile Robot Control
by Bayesian Optimization
Takuya Hayakawa and Jun Kobayashi

Department of Systems Design and Informatics, Kyushu Institute of Technology,
Kawazu 680-4, Iizuka, 820-8502, Japan
E-mail: jkoba@ces.kyutech.ac.jp
lab.jkoba.net

Abstract
The aim of this study is to improve classification performance of neural networks as an EEG-based BCI for mobile
robot control by means of hyperparameter optimization in training the neural networks. The hyperparameters wereintuitively decided in our preceding study. It is expected that the classification performance will improve if you
determine the hyperparameters in a more appropriate way. Therefore, the authors have applied Bayesian optimization
to training the EEG-based BCI neural networks and achieved the performance improvement.
Keywords: brain computer interface, electroencephalography, neural network, hyperparameters, Bayesian
optimization, mobile robot control

studies, the subjects did not directly imagine desired
behavior of a controlled object.
As an EEG-based BCI for mobile robot control, theauthors are developing a neural network (NN) for EEG
signal classification. In our earlier studies6–8, we
instructed a subject to imagine an arrow representing adesired behavior of a controlled object with closed eyes
for removing visual stimuli unrelated to experiments. In
addition to the experiments with the subjects closed theireyes, we conducted an experiment to confirm open eyes
influence on constructing an EEG-based BCI NN9.
Considering practical use of BCI, actually imagining adesired motion of a controlled object with open eyes
would be a more appropriate way for controlling a mobile
robot. However, none of the constructed NNs achieved
practical performance.
The authors conjectured that one of the reasons forthe insufficient results was due to hyperparametersettings in training the NNs. The hyperparameters wereintuitively set in the preceding studies. It is expected thatthe classification performance will improve if you

1. Introduction
Brain Computer Interface (BCI) is a promisingtechnology that provides means of direct communication
through your brain. Since brain activities bring about
perception, recognition, and sensory-motor functions inhuman beings, BCI based on a state of your brain has
potential to be applicable to support many kinds ofhuman activities. A lot of studies on BCI have used noninvasive brain activity measurement methods, especiallyElectroencephalography (EEG) because of its superior
time resolution and ease of use. Furthermore, portable
and low-cost EEG measurement devices have become
readily accessible lately.
EEG-based BCI have already applied for assistinghandicapped people and augmenting human capability1,2.
R. Single et al. developed a SSVEP-based BCI for
controlling a wheelchair using multi-class SVM3,4. J.
Meng et al. experimentally investigated a noninvasive
BCI for reach and grasp task of robotic arm5. In these© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan120

Takuya Hayakawa, Jun Kobayashi

determine the hyperparameters in a more appropriate
way. Therefore, the study described in this paper aims at
improving the performance of the EEG-based BCI NNs
for mobile robot control by hyperparameter optimization.
For optimization, the authors adopted Bayesian approachthat optimizes hyperparameters for NN training bydetermining the hyperparameters to be next verified on
the basis of results already checked. In consequence, themethod can find the optimal hyperparameters efficiently.
In this paper, the authors present experiments
introducing Bayesian optimization using the EEG signals
recorded in our previous study. The experimental results
demonstrated
that
Bayesian
optimizationofhyperparameters improved the classification rate of theEEG-based BCI NNs for mobile robot control.

dimensionality reduction. G. E. Hinton and R. R.
Salakhutdinov revealed that AE surpasses Principal
Component Analysis (PCA) for dimensionality
reduction10.
SAE is a way to construct a multilayered NN
avoiding vanishing gradient in backpropagation. As
stated above, an initial multilayered NN is prepared bystacking encoders of pretrained AEs in cascade, and thewhole NN is trained again using the same input signals
and target ones. The last training is called finetuning.
3. Bayesian Optimization of Hyperparameters
Ahead of training a NN, you must determine someparameters, such as number of layers, number of nodes,
learning rate, and so on, which are called hyperparameter.
The preceding study has not achieved practical
performance of the SAEs for EEG-based BCI probably
because the hyperparameters were intuitively determined.
Optimization of hyperparameters is an essential process
to improve performance of NNs.
One of the methods for hyperparameter optimization
is grid search, in which combinations of hyperparameters
are uniformly arranged in the form of a grid in a
hyperparameter space, all of the parameter combinations
are verified. Therefore, the optimal hyperparameters arefound with a high probability if the grid arrangement in
the hyperparameter space is sufficiently dense. In thecase of high dimensional hyperparameter space, however,it becomes difficult to conduct verification in the dense
grid within a realistic time due to huge number ofhyperparameter combinations. In addition, if some of thehyperparameters have little influence on the performanceof the NN, calculation time for grid search is wasted on
searching in worthless regions of the hyperparameter
space.
J. Bergstra and Y. Bengio have shown empiricallyand theoretically that random search can find optimal
hyperparameters more efficiently than grid search11.
Furthermore, J. Snoek et al. have demonstrated that
Bayesian optimization for hyperparameter selection ofmachinelearning
algorithms
foundoptimalhyperparameters
faster,andoutperformed
hyperparameter selection by a human expert12. The
Bayesian approach determines the hyperparameters to be
next examined on the basis of results already verified.
First, Bayesian optimization randomly selects
hyperparameters and verifies them. Following that, a

2. EEG Signal Classification Using Stacked
Autoencoder
This section describes the structure of the multilayered
NNs used in the preceding study9. Stacked Autoencoder(SAE) was employed as an EEG-based BCI NN for
mobile robot control in the study. SAE is a multilayered
NN initialized by stacking encoder layers of pretrained
Autoencoder (AE), which is a three layered NN shown in
Fig. 1.

Fig. 1. Autoencoder (AE)A typical AE has equal-sized input and output layers,
and a less-sized hidden layer. The whole of AE is trainedin order that it can yield an output signal equal to an input
one. Since the hidden layer has less nodes than the input
layer, significant information is extracted from the input
signal through the encoder part, which is between the
input and hidden layers, for restoring the signal throughthe hidden and output layers; namely AE is a NN for© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

121

Improving EEG-based BCI Neuralregion where the optimal hyperparameters are likely toexist are estimated from the previous results andpreferentially verified hyperparameter in the region. The
estimation of the region to be examined is made by
maximizing an acquisition function that includes
probability and expected values under the assumptionthat a function to be optimized follows Gaussian process.
In this research, the authors attempted to optimize thehyperparameter of the SAEs for EEG-based BCI for
mobile robot control using GPyOpt13, which is a Python
library for Bayesian optimization.

options; the settings enclosed in parentheses are theoptions chosen in the preceding study9. GPyOpt
optimized the hyperparameters so that false recognition
rate is minimized. The false recognition rate of each SAE
was calculated using 5-fold cross validation.
Table 1. Specifications of computation environment fordevelopment and execution of stacked autoencoders
CPU
MemoryStorageGPU
OS

4. Results and Discussion
The authors empirically confirmed effects of Bayesian
optimization for tuning hyperparameters on improving
the classification performance of the SAEs for EEGbased BCI mobile robot control using the EEG signal
dataset obtained in the preceding study. Fig. 2 is aschematic diagram of the apparatus for EEG
measurement experiments conducted in the preceding
study9. In the EEG measurement experiments, twodifferent imagining tasks, named "CLOSED-EYES" and"OPEN-EYES", were given to three subjects as follows.

Intel® CoreTM i7-6800K CPU @ 3.40GHz
16GB (DDR4-2133 4GBx4)
SSD 240GB + HDD 1TB
NVIDIA GeForce GTX1060 6GB GDDR5Ubuntu 16.04 LTS
Table 2. Hyperparameters and options

Hyperparameter
Number of hidden
layersNumber of nodes in 1st
hidden layer
Number of nodes in 2nd
hidden layer
Number of nodes in 3rd
hidden layer
Iteration of pretraining
each AE (epoch)
Iteration of finetuning(epoch)Dropout 50% in
pretraining AEs
Dropout 50% in
finetuning

CLOSED-EYES: close your eyes and imagine a
specified arrow
OPEN-EYES: watching the mobile robot moving toa certain direction, imagine the robot's motion
Refer to Ref. 9 for the other experimental conditions of
the EEG measurement, such as experimental procedure,
information of the subjects, and so on.

Options (setting value in Ref. 9)
1 ~ 3 (3)250 or 300 (250)
150 or 200 (150)
50 or 100 (100)1000 or 2000 (1000)
5000 or 10000 (10000)
OFF or ON (ON)OFF or ON (ON)

Table 3 and Table 4 show the results obtained in thepreceding study9. Table 5 and Table 6 report theclassification rate percentages of the SAEs calculated inthis study with Bayesian optimization. Compared the
updated results with the previous ones, it is indicated that
we could improve the performance of the SAEs trainedwith the hyperparameters optimized by Bayesian
approach. However, the options of the hyperparameters
given to the optimization algorithm were very restricted.
Therefore, we would consider further improvement to bepossible by revising the hyperparameter options more
carefully.

Fig. 2. Experimental apparatus

Chainer14 (ver. 1.18.0) for implementing the SAEs
and GPyOpt13 (ver. 1.2.0) for Bayesian Optimization
were used in the confirmation experiments. Table 1
shows the computation environment for development
and execution of the SAEs. Table 2 describes thehyperparameters optimized in the experiments and their

5. Conclusion
This paper presented the performance improvement of
the EEG-based BCI neural networks for mobile robot
control proposed in the preceding study. The authors© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan122

Takuya Hayakawa, Jun Kobayashi

introduced Bayesian optimization in order to find thebetter hyperparameters for training the neural networks
and then experimentally confirmed the expected effect ofthe method. The authors will design the appropriate
options of hyperparameters that considerably ameliorate
the classification performance of the EEG-based BCI
neural networks for mobile robot control.

References
1. A. Kübler et al., Patients with ALS can use sensorimotor
rhythms to operate a brain-computer interface, Neurology
64(10) (2005) 1775–1777.
2. T. P. Luu et al., Gait adaptation to visual kinematic
perturbations using a real-time closed-loop brain–computer interface to a virtual reality avatar, J. NeuralEngineering 13(3) (2016).
3. R. Singla et al., Influence of stimuli color on steady-statevisual evoked potentials based BCI wheelchair control, J.
Biomedical Science and Engineering 6(11) (2013) 1050–
1055.
4. R. Singla et al., Influence of stimuli color in SSVEP-based
BCI wheelchair control using support vector machines, J.
Medical Engineering & Technology 38(3) (2014) 125–134.
5. J. Meng et al., Noninvasive Electroencephalogram BasedControl of a Robotic Arm for Reach and Grasp Tasks,
Scientific Reports 6(38565) (2016).
6. K. Tomonaga et al., Experiments on classification ofelectroencephalography (EEG) signals in imagination of
direction using a wireless portable EEG headset, in Proc.
of Int. Conf. on Control, Automation and Systems (Busan,
South Korea, 2015), pp. 1805–1810.
7. S. Wakamizu et al., Experiments on Neural Networks withDifferent Configurations for Electroencephalography(EEG) Signal Pattern Classifications in Imagination of
Direction, in Proc. of IEEE Int. Conf. on Control System,
Computing and Engineering (Penang, Malaysia, 2015), pp.
477–481.
8. K. Tomonaga et al., Experiments on Classification ofElectroencephalography (EEG) Signals in Imagination of
Direction using Stacked Autoencoder, J. Robotics,
Networking and Artificial Life 4(2) (2017) 124–128.
9. T. Hayakawa and J. Kobayashi, Open Eyes Influence on
Electroencephalography Signals for Constructing Neural
Network Classifiers as Mobile Robot Control BrainComputer Interface, in Proc. of Int. Conf. on Advanced
Mechatronic Systems (Xiamen, China, 2017), ThuM01-03.
10. G. E. Hinton and R. R. Salakhutdinov, Reducing theDimensionality of Data with Neural Networks, Science
313 (2006), pp. 504–507.
11. J. Bergstra and Y. Bengio, Random Search for HyperParameter Optimization, J. Machine Learning Research
13 (2012) 281–305.
12. J. Snoek et al., Practical Bayesian Optimization of
Machine Learning Algorithms, in Proc. of Int. Conf. on
Neural Information Processing Systems (Lake Tahoe,
Nevada, 2012), pp. 2951–2959.
13. GPyOpt: A Bayesian Optimization framework in Python,
http://github.com/SheffieldML/GPyOpt, 2016.
14. Chainer: A Powerful, Flexible, and Intuitive Framework
for Neural Networks, https://chainer.org/.

Table 3. Classification rate percentages of SAEs trained using
120 samples recorded in CLOSED_EYES without Bayesian
optimization9
Subject

1st
day

2nd
day

3rd
day

4th
day

5th
day

Ave.

A

50.00

54.17

58.33

73.33

85.83

64.33

B

47.50

60.83

61.67

55.83

48.33

54.83

C

31.67

40.00

45.00

66.67

31.67

43.00

Table 4. Classification rate percentages of SAEs trained using
120 samples recorded in OPEN_EYES without Bayesian
optimization9
Subject

1st
day

2nd
day

3rd
day

4th
day

5th
day

Ave.

A

95.00

45.00

52.50

31.67

67.50

58.33

B

60.00

49.17

53.33

25.83

43.33

46.33

C

54.17

30.00

38.33

59.17

25.00

41.33

Table 5. Classification rate percentages of SAEs trained using
120 samples recorded in CLOSED_EYES with Bayesian
optimization
Subject

1st
day

2nd
day

3rd
day

4th
day

5th
day

Ave.

A

55.00

57.50

66.67

79.17

92.50

70.17

B

50.00

70.00

53.33

66.67

54.17

58.83

C

44.17

49.17

51.77

76.67

33.33

51.02

Table 6. Classification rate percentages of SAEs trained using
120 samples recorded in OPEN_EYES with Bayesian
optimization
Subject

1st
day

2nd
day

3rd
day

4th
day

5th
day

Ave.

A

95.83

48.33

45.83

41.67

75.00

61.33

B

66.67

57.50

60.83

33.33

43.33

52.33

C

60.00

40.00

49.17

76.67

38.33

52.83

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

123

Selective synchronization of the coupled bifurcating neurons
for phase shift of background oscillation
Akihiro Yamaguchi*, Yutaka Yamaguti

Department of Information and Systems Engineering, Fukuoka Institute of Technology,
3-30-1 Wajiro-higashi, Higashi-ku, Fukuoka, 811-0116, JAPANMasao Kubo

Department of Computer Science, National Defense Academy of Japan,
1-10-20 Hashirimizu, Yokosuka, Kanagawa, 239-8686, JAPANE-mail: *aki@fit.ac.jp

Abstract
Synchronization in the coupled bifurcating neurons was studied in the view point of the selective formation of cell
assembly. The bifurcating neuron is a simple chaotic neuron that exhibits chaotic inter-spike interval dynamic byadding the sinusoidal background oscillation. In this research, we introduce the phase shift of the backgroundoscillation and the several types of phase response to the input spike sequence. The coupled 16 bifurcating neurons
with complete bidirectional coupling was numerically simulated and their synchronized behaviors were analyzed. As
a result, the formation of chaotically synchronized cell assemblies for the phase shift value were observed.
Keywords: chaotic synchronization, neural coding, bifurcating neuron, binding information

Sec. 2 and 3, we explain the network of the bifurcatingneuron and the coupling model. In Sec. 4, the numericalsimulation is performed to examine the selective
synchronization for the phase shift of the backgroundoscillation. We summarize results in Sec. 5.

1. Introduction
The synchronized neural activity is regarded to have an
important role in the neural information coding and thebrain information processing1. For The correlated firing
among neurons was observed in the visual information
processing in the brain2,3. In the view point of neuralcoding, authors have studied the chaotic synchronizationof neural spike response and its application to thesegmentation of input image4, the feature linking ofvisual image of moving objects5, and the decompositionof superimposed chaotic spike sequences6. In order torepresent multiple information by the chaotic
synchronized cell assemblies, coexistence of suchsynchronized assemblies is necessary.
In this study, we construct the network of thebifurcating neuron7 with complete bidirectional couplingand design the coupling model to achieve selective
chaotic synchronization based on the phase response. In

2. Network model of the bifurcating neuron
The bifurcating neuron was introduced by Lee andFarhat7. In our previous research6, we defined thebifurcating neuron as a form of the spike response model8
to extend the coupling term based on the phase response.
In the following sections, we explain the extended
bifurcating neuron model and its network. Let the
network of bifurcating neurons consist of neurons, and()be the -th neuron. The dynamics of the -th
bifurcating neuron is defined as follows:
()

( )=

+

( )( )()( )++ ( ),()( )+© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

124

(1)

Akihiro Yamaguchi, Yutaka Yamaguti, Masao Kubo
()

, (),
where ( ) ,, ( ) ,and ( ) denotes the()internal potential of, the resting potential, the kernel
function of internal dynamics, the positive coupling term,
the negative coupling term, and the external noise,respectively. The coupling terms are described in thenext section.
The kernel function ( ) is defined as
()

where

( )=

−

( , )=

()

+

(2

()

,

+ ),

()

;

without coupling exhibits various chaotic dynamics of
the inter-spike interval and the phase of the next firing
time is determined by the chaotic one dimensional mapof the phase of the last firing time6,7. The shape of thischaotic map is determined by the phase shift value ( ) .
Examples of this chaotic map is shown in Fig. 1.
Furthermore, the bifurcation diagram and the Lyapunov
exponents of this map are also shown in Fig. 2.

(2)

3. Coupling model with phase response

(3)

( )

( )

( )

Let the set ( ) = {,,, ⋯ } be a set of firing( )( )time of the neuron, whereis the firing time of-th spike of ( ) . The set of firing time of the input
spikes to ( ) is denoted as ( ) . For the bidirectional
complete coupling, Γ ( ) is defined as()isis the time constant of the kernel, and(). The internal potential ( ) is
linearly increasing by the kernel ( ) . The neuron ( ) isfiring when ( ) exceeds the threshold , and then ( ) is
reset to the initial potential given by the back groundoscillation ( , ). The constants , , and ( ) is theamplitude, the frequency, and the phase shift value of
background oscillation, respectively. The dynamics ofthe single neuron without coupling term is the same withthe original bifurcating neuron. The single neuron
the last firing time of

()

()

=

( )

()

.

(4)

and ( ) in Eq. (1) are the
The coupling term()
and thesummation of the positive phase response
negative ( ) such that()
± ( ) =

∈ ( ),()± ( ).

()(5)

Simple phase response is constant value such that()
± ( ) = ± ±,

(6)

where
and
are non-negative coupling constant.
The next firing time is hastened by the positive responseand delayed by the negative response. We call the
coupling with these simple responses the constantpositive coupling and the constant negative coupling,respectively.
We designed the phase response in order to strengthen
the chaotic synchronization between coupled neurons
with the same phase shift value6. The positive phase
response based on the predicted next firing time isdefined as

(a)
(b )
Fig. 1. Examples of the chaotic one dimensional map of thephase of the single bifurcating neuron with the phase shift (a)= /2 and (b) = 3 /2.

()( )= +

where ̂that

Fig. 2. The bifurcation diagram and the Lyapunov exponents ofthe single bifurcating neuron without coupling.

0

()

0

()
< ̂
−∆

̂( ) − ∆ ≤
̂

()

≤

< ̂

()

,

is the predicted next firing time of© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

125

(7)
()

such

Selective synchronization of the− ( )̂( ) = +(8).

We call the coupling with this response the adaptive
positive coupling. The negative phase response based on
the last firing time is defined as( )( )

⎧
⎪

= −
⎨
⎪
⎩

0
0

∆

()

()

<

()

≤

≤

()

()

+∆ <

+∆ .

(9)

We call the coupling with this response the adaptive
negative coupling.
If the time is within the range ∆ from the(), then the phase response
predicted next firing time ̂is positive to hasten the next firing time. Otherwise, if thetime is within the range ∆ from the last firing time()
, then the phase response is negative to delay the next
firing time. By using these phase response (Eq. (7) andEq. (9)), the decomposition of the superimposed chaotic
spike sequences with the different phase shift values ispossible6.

Fig. 3. An example of the generated spike sequences for the
network with the adaptive positive and negative coupling.

where #(⋅) denotes the number of elements and ∆ is thetime resolution of the firing time. Furthermore, in orderto characterize the assembly formation of thesynchronized cluster of neurons, we define the averagesynchronization ratio for the neurons with the same phaseshift value such as

4. Numerical experiments
4.1. Experimental setting
In this article, we numerically simulate the network of the
16 bifurcating neurons to analyze the selective
synchronization to the same phase shift value forcomplete bidirectional coupling. The different 4 phaseshift values are assigned as follows:()= ∆ ⌊ /4⌋ ( = 0, ⋯ ,15),

()

(10)
()

()

( )

,

()

;

( )

/48,

(12)

( )

=

()

,

( )

()

;

( )

/192.

(13)

4.3. Experimental results
As a result of the numerical simulation, an example of thegenerated spike sequences are shown in Fig. 3, where thephase response coupling is the adaptive positive andnegative one. The generated spikes were synchronizedfor each 4 neurons with the same phase shift value, andthe selective synchronization for the same phase shift
value was observed. In order to analyze the effect ofcoupling, we numerically simulated the network andevaluated the synchronization ratio and the averaged one
for the 5 types of the coupling defined in Sec.3. Results
are shown in Fig. 4 and Table 1. These results indicate

4.2. Characteristics of synchronization

As an index of the synchronization between two spikesequences, the synchronization ratio is defined as; ( ) = #{
−
≤∆ ,
()
( )
( )
∈
, ∈
}/#
,

=

and the average synchronization ratio for the neurons
with the different phase shift value such aswhere 0 < ∆ ≤ is the difference of the assigned phaseshift value. In this setting, each 4 neurons have the same
phase shift value. We examine the selective
synchronization among each 4 neurons with the same
phase shift value. Numerical simulation is performed
with the following setting: = 100, = −30,=
−70,
= 21.5 , = 1 ,== 2.1 , ∆ = 0.05 ,and ∆ = /2.

()

( )

(11)

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

126

Akihiro Yamaguchi, Yutaka Yamaguti, Masao Kubo

that the selective synchronization was observed for the
coupling types (c) and (e), and the synchronization ratio
is higher than 90% for the neurons with the same phaseshift value and lower for different ones.

1

1
0.8

0.8

1

0.6

1

0.6

0.8

0.8

0.4

0.4

0.6

0.2
0.4

0

0.6

0.2
0.4

0

0.2

0.2

0

0
14
12
10

14
12
10

5. Conclusion8

neuron ID

6
4

4
2
0 0

The network of the bifurcating neuron was constructed
and the formation of the chaotically synchronized cell
assemblies was examined for the several types ofcoupling model. As a result, the chaotically synchronizedcell assemblies was observed for the adaptive positive
coupling of the phase response, where the neurons whichhave the same phase shift value were selectively
synchronized.
This result might be indicate thepossibility of binding information based on the phaseshift value of the background oscillation. To realize suchinformation coding is one of our future works.

8

neuron ID

6

2

8

6

4

10

14

12

2
0 0

(a) Constant Positive

12

10

8

6

4

2

14

neuron ID

neuron ID(b) Constant Negative
1

1
0.8

0.8

1

0.6

1

0.6

0.8

0.8

0.4

0.4

0.6
0.4

0

0.6

0.2

0.2

0.4

0

0.2

0.2

0

0
14
12
10

14
12
10
neuron ID

8neuron ID

6

8
6
4

4
2
0 0

2

8

6

4

10

12

14

2
0 0

4

2

(c) Adaptive Positive

8

6

10

12

14

neuron ID

neuron ID(d) Adaptive Negative

1
0.8

1

0.6

0.8

0.4

0.6

0.2

References

0.4

0

0.2

1. Fuji, H., Ito, H., Aihara, K., Ichinose, N., and Tsukada, M.,
Dynamical cell assembly hypothesis – Theoreticalpossibility of spatio-temporal coding in the cortex, Neural
Networks, Vol.9, No.8, pp.1303-1350, 1996．
2. Gray, C.M., Koenig, P., Engel, A. K., and Singer, W.,
Oscillatory responses in cat visual cortex exhibit intercolumnar synchronization which reflects global stimulus
properties, Nature, 338, pp.334-337, 1989.
3. Eckhorn, R., Reitboeck, H.J., Arndt, M., Dicke, P., Featurelinking via stimulus-evoked oscillations: experimental
results from cat visual cortex and functional implications
from a network model, International Joint Conference on
Neural Networks, pp. 723-730, 1989.
4. M. Fujiwara, A. Yamaguchi, and M. Kubo, Synchronized
Response to Grayscale Image Inputs in Chaotic Cellular
Neural Network, J. Robotics, Networking and Artificial
Life, Vol.2, No.1, pp. 26-29, 2016.
5. A. Yamaguchi, S. Arakane, and M. Kubo, Feature Linking
using Synchronized Responses in Chaotic Cellar Neural
Networks for Visual Stimulus of Moving Objects,
J. Robotics, Networking and Artificial Life, Vol.2, No.4,
pp. 230-233, 2016.
6. A. Yamaguchi, Y. Yamaguti, and M. Kubo,
Decomposition of superimposed chaotic spike sequencesby using the bifurcating neuron, to appear in Advances incognitive neurodynamics (VI), Springer, 2018.
7. Lee G., and Farhat, N.H., The Bifurcating Neuron
Network 1, Neural Networks, Vol. 14, pp. 115-131, 2001.
8. Gerstner, W., and Kistler, W., Spiking Neuron Models:Single Neurons Populations Plasticity, Cambridge
University Press, 2002.

0
14
12
10
neuron ID

8
6
4
2
00

2

4

6

12

10

8

14

neuron ID(e) Adaptive Positive and Negative (both)Fig. 4. The synchronization ratio between two neurons for the
coupling type (a) constant positive, (b) constant negative, (c)adaptive positive, (d) adaptive negative, and (e) both ofadaptive positive and negative, respectively.
Table 1. Average Synchronization Ratio
Coupling type

()

( )

()

(a) Constant Positive

0.7861(b) Constant Negative

0.18190.4766
0.0638(c) Adaptive Positive

0.94140.3694(d) Adaptive Negative(e) Adaptive Positive
and Negative

0.65200.20570.97230.3519© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

127

( )

Review on computational techniques in solving aircraft landing problem
Aminurafiuddin Zulkifli, Nor Azlina Ab. Aziz, Nor Hidayati Abdul Aziz

Faculty of Engineering and Technology, Multimedia University, 75450 Melaka, Malaysia
Zuwairie Ibrahim

Faculty of Electrical and Electronics Engineering, Universiti Malaysia Pahang, Pekan, 26600 Pahang, Malaysia
Norrima Mokhtar

Faculty of Engineering, University of Malaya, 50603 Kuala Lumpur, Malaysia
Email: aminurafiuddin@gmail.com, azlina.aziz@mmu.edu.my, hidayati.aziz@mmu.edu.my, zuwairie@ump.edu.my,norrimamokhtar@um.edu.my

Abstract
The problem of sequencing and scheduling arriving aircraft landing is commonly known as aircraft landing problem (ALP).
This problem, due to various constraints such as the number of arriving aircrafts, the number of runways, the mode of
runway operation, the type of arriving aircrafts, the minimum separation between each arriving aircraft, and the weather
condition, is considered to be a NP-hard problem. Therefore, it is almost impossible to compute every possible solution andcomputational intelligence methods had been adopted to solve ALP. In this paper, we review the computational intelligence
techniques used in ALP. The main techniques include the evolutionary algorithms namely; genetic algorithm, genetic
programming, scatter search and bionomic algorithm, the swarm intelligence algorithms like particle swarm optimization
and ant colony optimization and also other methods such as the constrained position shifting and dynamic programming.
Keywords— aircraft landing problem; computational intelligence; evolutionary algorithms; swarm intelligence; scheduling;
runway operation

1.

Introduction

Over the years, many studies had been conducted in thefield of air traffic control. Since the aviation industry is
expanding, a need for closer attention to this field has arisen.
The complexity of air traffic operation is motivated by theincrease in air traffic volume every year. In Europe and US,
increase in air traffic demand is expected to double in the next
15 years [1]. In Malaysia, according to Malaysian Ministry ofTransport an increase of 2.7% was reported in term of total
commercial aircraft movements handled by Malaysian airport
in 2015 compared to 2014 [2].
One of the important task in air traffic control is managing
the air traffic operations for aircraft take-off and landing. Thistask is handled by air traffic controllers. The controllers
determine which aircraft is taking off from, or landing on,
available runway at airports, subject to operational constraints.
To handle the landing and taking-off of an aircraft, is a very
challenging process. It is highly related to safety, efficiency,
robustness, and competitiveness issues. The common practiceis to tackle the task using first come first serve (FCFS) basis.
This method may not be efficient but it is certainly thesimplest way to manage the operation.

However, with rapid development in computing,researchers have been developing multiple ways to aid theprocess of scheduling and sequencing aircrafts. This include
the adaptation of computational intelligence. In this paper, we
focus on the application of computational intelligence insolving the problem of sequencing and scheduling landing
aircrafts, which is commonly known as aircraft landing
problem (ALP).
The remainder of this paper is organized as follows. Basicconcept of aircraft landing problem is briefly explained in
section II. Section III describes the computational techniques
used in order to optimize ALP. Section IV concludes this
work.
2.

Aircraft Landing Problem

ALP can be defined as a process of sequencing andscheduling the arriving aircrafts. Nowadays with airports
operating with more than one runways, ALP can be viewed as
two objectives problem, which are sequencing the optimallanding order of the arriving flights and scheduling a runway
for each arriving aircraft [3]. In [1], the objectives are listed as;a) maximizing runway throughput, b) minimizing theapproach time of aircraft before landing, c) minimizing the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

128

Aminurafiuddin Zulkifli, Nor Azlina Ab. Aziz, Nor Hidayati Abdul Aziz, Zuwairie Ibrahim, Norrima Mokhtar

arrival delay, d) minimizing air traffic controller’s workload.
Thus, one of the challenges for the researchers in this field isto develop a solution capable of dealing with a variety ofobjectives. This raise the issue of which objective function to
adopt. By far ALP’s objective function causes the most
discussion among researchers. Arguments can convincingly bemade for many different objective functions. Different users
will, for perfectly legitimate reasons, use different objectivefunctions [4].
ALP is a dynamic problem as the controller need to operate
in real-time. The controller must generate an updated schedule
for the set of arriving aircraft to be sequenced and scheduledboth periodically and in response to aperiodic events, while
the length of the periodic cycle is related to the basic radar
update time interval, which is 4–12 seconds long [5].
In most studies, the common structure of ALP (static case)has n number of landing aircrafts, which the aircrafts denoted
as i = 1, 2, …, n, and the estimated earliest landing time is E,
and the latest landing time is L. Therefore, the time window
for an aircraft can is bounded within [E(i),L(i)]. Controller
will assign a target landing time for every landing aircraft, T,
and it must be in between E and L, hence, E(i) ≤ T(i) ≤ L(i).
The controller must also obey the minimum separation time
constraint between leading aircraft and trailing aircraft. This isa mandatory safety measure to avoid complications like thewake turbulence and aircraft collision.

the possible realistic level. The same formulation is also
adopted in various work [10],[11] ,[12] ,[13].
Another study regarding to the static case of ALP was
carried by Krishnamoorthy et al. [14] and then improved by
Beasley et al. [4]. The study comprises both single-runway
and multiple-runway cases. The objective is to minimize thetotal cost, where the cost is linearly related to deviation fromtarget landing time for landing planes. Figure 1 shows how thecost of aircraft landing is introduced in the paper. Although
the cost function shown is nonlinear, it can be linearized bydecomposing the two linear portions and formulate theproblem with a linear objective function. The paper includes
various mathematical constraints to manifest every possible
condition of ALP. This study has been cited by many ALP
researchers as it presented a general description of the model,
goals, and mathematical formulation of the landing planningfor one or more runways [15], [16], [17], [18],[19],[20].

2.1 Problem modellingPsaraftis [6] models ALP into a simple version. He sums up
the landing times of the arriving aircraft as cost/delay, which
is described as Total Passenger Delay (TPD) while thethroughput is measured based on the landing time of the lastaircraft in the sequence, described as Last Landing Time
(LLT). The objective is to minimize both TPD and LLT.
Psaraftis also stated that aircraft sequencing problem is thesame, as the NP-hard Travelling Salesman Problem (TSP),
where, the cost of the graph represents the landing cost andthe route represents the aircraft landing sequence. However,Psaraftis did not consider the aircraft latest landing, L, whichallows the aircraft to land at any given time.
On the other hand, Bayen et al. [7] focuses on formulatingholding time using two approaches. The scheduling process is
optimized when the holding time is minimized and, therefore,
minimize the sum of arrival time. They formulated ALP as asingle machine job scheduling problem. However, they
consider that all arriving flights in one large class. Thus, therequired separation between landings is independent of theaircraft type.
Cheng et al. [8] emphasis on runway assignment for ALP.
They discuss sequencing and scheduling a number of arrivals
to a number of runways. The paper focuses on static case of
ALP. The data set used contains the estimated time of arrivals(ETA). The aircraft performances and the flight pattern are also
taken into consideration. The main objective is to minimize thedelay, which is the difference between the scheduled time ofarrival (STA) and the earliest ETA in all the runways for anaircraft. Hansen [9] improves the implementation made by
Cheng with an experiment on larger set of aircrafts made up toThis section reviews the literature related to ALP. The
subsections are organized according to the computational
intelligence method used.

Figure 1. Cost variation in time window during flight3.

Computational Intelligence on ALP

3.1 Evolutionary algorithms
Genetic algorithm (GA) is a famous evolutionary algorithm.
It has been used in various problems including ALP. In
previous work of Cheng et al. [8] the runway assignment iscarried using genetic search algorithm. Four schemes ofgenetic search were used to solve a simple scenario involving
12 flights in 3 runways. The result shows application ofgenetic search gives an impressive result. The method is later
improved by Hansen [9] with a larger data set, which is nearer
to the realistic case. Hansen proposed the improved version ofgenetic algorithm called Genetic Programming (GP). GP
shows a better performance than GA when dealing with larger
set of arriving aircrafts. Liu [11] compared GA with newly
developed Genetic Local Search (GLS), which is an extensionof GA in solving ALP. Liu uses Hansen’s case as well toexperiment with GLS and it is proven to be more efficient interm of both optimality and time cost than GA.
Wang et al [3] adopted GA to solve ALP on parallel
runways, which include independent approach and relevant
approach. It is tested on 40 arriving flights.
Abela et al. [14] discussed two different solution for ALP
which are using GA and branch-and-bound. Both approaches
are tested with a data set and GA is proven to be effective insmall problem. Hu and Di Paolo [13] have integrated GA withreceding horizon (RH) strategy to tackle ALP. The
chromosomes in GA are constructed using the arriving order
and/or arriving time of each aircraft.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

129

Review on computational techniques

Pinol and Beasley [15] introduce a hybrid method using
another evolutionary algorithms, scatter search and bionomic
algorithm for ALP. Both methods are combined and used forproblem instances up to 500 aircrafts and 5 runways. In linear
ALP cases, scatter search performs better than bionomic
algorithm while in non-linear cases, bionomic algorithm
performs better than scatter search. These methods, areflexible to be adapted and able to change objective between
linear and non-linear ALP cases.
3.2 Swarm Intelligences
Other than evolutionary algorithms, swarm intelligencealgorithms are also used in solving ALP. Among the Swarm
intelligence algorithms used are particle swarm optimization(PSO), ant colony optimization (ACO) and gravitational
search algorithm (GSA).
A PSO hybridized with local search (LS) on rolling horizon(RH) is proposed by Girish [21] to minimize the deviation
cost of aircraft landings by using Beasley’s formulations [4].
It is then compared with another two PSO variants. The RH
framework with hybrid PSO-LS is proven to be more efficientand has shorter computational time.
Benceikh [19] uses ACO to solve ALP and the Beasley’s
formulation is improved by incorporating dynamic cases, forexample, flight cancellation and runway closing. They also
improved ACO with more robust heuristic named Improved
ACO (IACO) to reduce the penalty cost.
Kazem et al [10] introduce GSA in their paper for the
purpose of solving ALP. GSA is a swarm intelligencealgorithm which was inspired by Newton’s law of gravity [22].
Firstly, random sequence of aircrafts with their allocatedrunway are generated as initial solutions and each solution’s
fitness is computed. Using the Newton’s law of gravity, eachsolution is improved on the next iteration. The study uses
Hansen’s data set and compared with genetic search algorithm,
GA, scatter search and bionomic algorithm, and GSA is foundto provide better solution in shorter amount of time.
3.3 Other techniques
Dear [23] describes ALP as a dynamic problem andintroduced the constrained position shifting (CPS) method.
CPS works as the sequencing mechanism where the order oflanding aircraft is only shifted by limited k positions from its
FCFS position. Psaraftis [6] further improved Dear’s study
using backward dynamic programming algorithms that use thenumber of aircraft from each classes that has not yet beenscheduled to land and the class of the last aircraft to land as
the state variables. His method is also adaptable using CPS
without increasing the time complexity. However, Psaraftis
method assumes there is no landing restriction which means,the aircraft can land at any time. Balakrishnan et al [24]develop the work of Psaraftis’s CPS by simultaneously
handling precedence constraints, landing restriction and CPS
operational constraints, which reduces the computational timeand make it easy to be adopted in real-time for both static anddynamic cases. The updated work on CPS by Rodriguez-Diazet al [25] show how CPS is adapted on single mixed-mode
runway operation. They use simulated annealing (SA) toexperiment with CPS on a very large data set. The dataconsidered is up to 200 flights and 2000 instances. The study
emphasizes on how the focus of research should include
mixed-mode operation in their problem modelling.
Beasley et al. [4] on the other hand approach the problem
from a different perspective using mixed-integer zero-one
formulation which is adopted using a basic tree searchstrengthen with linear programing (LP) relaxation. A scenario
of 50 aircraft and four runways is tested and the result showed
the LP-based tree search are able to work with various
constraints that commonly encountered in practice. However,since the problem is NP-hard, it is likely for the computation
time to grow exponentially with the number of flights.
Bayen et al. [7] use dynamic programing (DP) and linear
programming relaxation (LP) with rounding approach. DP
approaches have the performance ratio of 5 for the sum ofarrival times of all aircraft and LP relaxation with rounding
has the performance ratio of 3 for the landing time of the lastaircraft.
Lieder et al. [26] introduce DP in solving ALP withdifferent aircraft classes on multiple runways with positive
target landing times and limited time windows. A new
dominance criterion is developed order to improve theperformance of DP approach. The criterion based on theformulation by Briskorn et al [27].
Table 1 comprises all the techniques discussed on thissection. The techniques are tabulated according to themethodology, the formulation and the instances used in theirrespective studies.
4.

Conclusions

This paper provides an overview on how computational
intelligence is used in ALP. From the review, it can be seenthat computational intelligence techniques are popular insolving ALP. The problem modelling of ALP is also observed
not to be uniform across the literature. For example, in someliteratures, ALP is viewed as static case while other view it as
dynamic case. For our future work, we will investigate on how
to solve ALP using computational intelligence approach thatare more robust and dynamic similar to the real problems
faced by air traffic controllers.
Acknowledgement
This research is funded by the Fundamental Research Grant
Scheme (FRGS/1/2015/ICT02/MMU/03/1), which is awarded
by Ministry of Higher Education Malaysia to Multimedia
University.
References[1] J. A. Bennell, M. Mesgarpour, and C. N. Potts, “Airport runway
scheduling,” Ann. Oper. Res., vol. 204, no. 1, pp. 249–270, 2013.
[2] M. of Transport, “Transport Statistics Malaysia 2015,” 2015.
[3] L. L. Wang and Q. L. Gu, “Optimization model for sequencing
arrival flights on parallel runways,” Proc. - 2014 Int. Conf. Inf.
Sci. Electron. Electr. Eng. ISEEE 2014, vol. 1, no. 61179042, pp.
489–492, 2014.
[4] J. E. Beasley, M. Krishnamoorthy, Y. M. Sharaiha, and D.
Abramson, “Scheduling Aircraft Landings — The Static Case,”© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

130

Aminurafiuddin Zulkifli, Nor Azlina Ab. Aziz, Nor Hidayati Abdul Aziz, Zuwairie Ibrahim, Norrima Mokhtar
Transp. Sci., vol. 34, no. 2, pp. 180–197, 2000.
[5] H. Erzberger and E. Itoh, “Design Principles and Algorithms for
Air Traffic Arrival Scheduling,” no. May, pp. NASA/TM—
2014–218302 Design, 2014.
[6] H. N. Psaraftis, “A Dynamic Programming Approach to The
Aircraft Sequencing Proble,” 1978.
[7] A. M. Bayen and C. J. Tomlin, “An Approximation Algorithm
for Scheduling Aircraft with Holding Time †,” pp. 2760–2767,
2004.
[8] V. H. L. Cheng, L. S. Crawford, and P. K. Menon, “Air traffic
control using genetic search techniques,” Proc. 1999 IEEE Int.
Conf. Control Appl. (Cat. No.99CH36328), vol. 1, pp. 249–254,
1999.
[9] J. V. Hansen, “Genetic search methods in air traffic control,”Comput. Oper. Res., vol. 31, no. 3, pp. 445–459, 2004.
[10] K. Dastgerdi, N. Mehrshad, and M. Farshad, “A new intelligent
approach for air traffic control using gravitational searchalgorithm,” Sadhana, vol. 41, no. 2, pp. 183–191, 2016.
[11] Y. H. Liu, “A genetic local search algorithm with a threshold
accepting mechanism for solving the runway dependent aircraft
landing problem,” Optim. Lett., vol. 5, no. 2, pp. 229–245, 2011.
[12] A. Salehipour, L. M. Naeni, and H. Kazemipoor, “Schedulingaircraft landings by applying a variable neighborhood descentalgorithm : Runway-dependent landing time case,” J. Appl.
Oper. Res., vol. 1, pp. 39–49, 2009.
[13] X. B. Hu and E. Di Paolo, “An efficient genetic algorithm withuniform crossover for air traffic control,” Comput. Oper. Res.,
vol. 36, no. 1, pp. 245–259, 2009.
[14] G. M. M. Krishnamoorthy, J. Abela, D. Abramson, A. De Silva,“Computing Optimal Schedules for Landing Aircraft,” in The
12th National Conference of the Australian Society forOperations Research, 1993, pp. 1–15.
[15] H. Pinol and J. E. Beasley, “Scatter Search and Bionomic
Algorithms for the aircraft landing problem,” Eur. J. Oper. Res.,
vol. 171, no. 2, pp. 439–462, 2006.
[16] J. E. Beasley, M. Krishnamoorthy, Y. M. Sharaiha, and D.
Abramson, “Displacement problem and dynamically schedulingaircraft landings,” pp. 54–64, 2004.
[17] F. Farhadi, A. Ghoniem, and M. Al-Salem, “Runway capacity
management - An empirical study with application to Doha
International Airport,” Transp. Res. Part E Logist. Transp. Rev.,
vol. 68, pp. 53–63, 2014.
[18] K. Dastgerdi, N. Mehrshad, and M. Farshad, “Journal of Soft
Computing and Decision Support Systems A New Intelligent
Approach to Aircrafts Take-off / Landing Planning at Congested
Single Runway Airports,” vol. 2, no. 2, pp. 17–25, 2015.
[19] G. Bencheikh, J. Boukachour, and A. Alaoui, “Improved Ant
Colony Algorithm to solve the aircraft landing problem,” Int. J.
Comput. Theory Eng., vol. 3, no. 2, pp. 224–233, 2011.
[20] I. Moser and T. Hendtlass, “Solving dynamic single-runway
aircraft landing problems with extremal optimisation,” Proc.
2007 IEEE Symp. Comput. Intell. Sched. CI-Sched 2007, pp.
206–211, 2007.
[21] G. B.S., “An efficient hybrid particle swarm optimization
algorithm in a rolling horizon framework for the aircraft landing
problem,” Appl. Soft Comput., vol. 44, pp. 200–221, 2016.
[22] E. Rashedi, H. Nezamabadi-pour, and S. Saryazdi, “GSA: A
Gravitational Search Algorithm,” Inf. Sci. (Ny)., vol. 179, pp.
2232–2248, Jun. 2009.
[23] Roger G. Dear, “The dynamic scheduling of aircrafting in thenear terminal area,” pp. 1–325, 1976.
[24] H. Balakrishnan and B. G. Chandran, “Algorithms for
Scheduling Runway Operations Under Constrained Position
Shifting,” Oper. Res., vol. 58, no. 6, pp. 1650–1665, 2010.

[25] A. Rodriguez-Diaz, B. Adenso-Diaz, and P. L. Gonzalez-Torre,“Minimizing deviation from scheduled times in a single mixedoperation runway,” Comput. Oper. Res., vol. 78, pp. 193–202,
2017.
[26] A. Lieder and R. Stolletz, “Scheduling aircraft take-offs andlandings on interdependent and heterogeneous runways,”
Transp. Res. Part E Logist. Transp. Rev., vol. 88, pp. 167–188,
2016.
[27] A. Lieder, D. Briskorn, and R. Stolletz, “A dynamic
programming approach for the aircraft landing problem withaircraft classes,” Eur. J. Oper. Res., vol. 243, no. 1, pp. 61–69,
2015.
Table 1. Overview of related literatures
SourceMethodology

Dear (1978)

Constrained
Position
Shifting
Constrained
Position
Shifting
Constrained
Position
Shifting
Constrained
Position
Shifting

Psaraftis(1978)
Balakrishnan
et al. (2006)
RodriguezDiaz (2017)

Beasley et al(2000)
Bayen(2004)
Cheng et al(1999)
Hansen(2004)
Wang (2014)
Liu (2011)
Abela et al(1993)
Hu and Di
Paolo (2009)
Pinol
and
Beasley(2006)
Girish(2016)

Benceikh et
al (2011)
Kazem et al(2016)

Type ofcase
Static/d
ynamicStaticObjective
Max throughput
Min Σ totaldelay
Min Σ costs/min
makespan

StaticMin Σ costs/min
makespan

StaticMin Σ delay

Linearprogramming
Linearprogrammingand dynamic
programming
Genetic
algorithm
Genetic
Algorithm,
Genetic
programming
Genetic
Algorithm
Genetic local
searchGenetic
algorithm,
Branch andbound
Genetic
algorithm

StaticStaticMin Σ penalty
cost
Min Σ arrivaltime
Min last aircraft
landing time
Min Σ delay

StaticMin Σ delay

Cheng et al(1999)

StaticMin Σ delay

Sampling data

StaticMin Σ penalty
cost
Min Σ deviation
cost

Beasley(2000)
Randomly
generated test
data
Hansen(2004)

Scatter searchand Bionomic
algorithm
Particleswarm
optimization
with
local
search
Ant Colony
Optimization

StaticMin Σ delayMax
arrivalqueuesMin Σ penalty
cost

StaticMin Σ penalty
cost

Beasley(2000)

Dynamic

Beasley(2000)

Gravitational
searchalgorithm

StaticMin Σ penalty
costanddisplacement
functionMin Σ totaldelay

StaticStaticStatic© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan131

Probleminstances
Randomgeneratedinstances
Randomgeneratedinstances
Denver airport
real data
Randomgeneratedinstances,
Beasley(2000)
andGatwick
airportBeasley(1990)
No instances(theoreticalanalysis)
Sampling data

Beasley(2000)

Hansen(2004)

Interactive musical editing system to support human errors and offer personal preferences foran automatic piano -Inferring performance expression by considering change of pitchMasahiro Ushio

Department of Mechanical Information Science and Technology, Kyushu Institute of Technology
680-4, Kawazu, Iizuka-City, Fukuoka, 820-8502, Japan
Email: ushio@mmcs.mse.kyutech.ac.jp
Eiji Hayashi
Department of Mechanical Information Science and Technology, Kyushu Institute of Technology
680-4, Kawazu, Iizuka-City, Fukuoka, 820-8502, Japan
E-mail: haya@ mmcs.mse.kyutech.ac.jp

Abstract
We have developed a system that allows a piano to perform automatically. In order to play music in the manner of alive pianist, we must add expression to the piano’s performance. In the case of music, there are often 1,000 or morenotes in the score, requiring that an editor spend a huge amount of time to edit. Therefore, we have developed aninteractive musical editing system that utilizes a database to edit music more efficiently.
Keywords: Automatic Piano, Knowledge Database, Computer Music, Music Interface

and thus they cannot simulate a human pianist’s musicalexpression. Therefore, we have developed an interactive
musical editing system that utilizes a database in order toedit music more efficiently 1.
In a current research, MIDI data regarding theperformances of highly skilled pianists has been analyzed
in order to observe the stylistic tendencies of theirperformances. The results showed that phrases having
similar patterns in the same composition were performedin the similar style.
We developed a system that searches for similarphrases throughout a musical score and infers the style ofthe performance. Here, we proposed a method using
Dynamic Programming (DP) matching as a way to searchfor similar phrases. In our interactive musical editing
system, we have created the Score Database whichcontains information regarding a musical score. This
database contains a field called “Note Value,” in whichdata indicates the type of note--e.g., a quarter note, a
triplet, and so on. This system converts notes into
character strings using the “Note Value” data. (Fig.3)
In addition, the system computes DP matching using
the character strings and calculates the degree of

1. Introduction
We have developed a system that allows a piano toperform automatically. In this system, 90 actuators havebeen installed on the keys and pedals of a grand piano.
These actuators execute key strokes and pedal
movements to govern the piano’s performance. (SeeFig.1)In order to develop an automatic piano that plays
music in the manner of a live pianist, we have to add
expression to the piano’s performance. Essentially,variations in tempo, dynamics, and so on are needed inorder to arrange the respective tones in a desired way.
Moreover, in the case of piano music, there will often be1000 or more notes in the score of even a short piece ofmusic, and the editor must spend a huge amount of timeto accurately simulate an actual emotionally expressive
performance that a highly skilled pianist could give.
Moreover, a highly skilled pianist is able to play anunfamiliar piece of music by sight, even if theperformance is not completely in accord with an intended
specific musical interpretation. Current computing
systems cannot perform a new piece of music by sight,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan132

Masahiro Ushio, Eiji Hayashi

disagreement between these strings. It uses an index tojudge the resemblance between the strings. For its
method of inferring performance expression, it uses thebest alignment for DP matching, which enables it toexpress the best correspondence between notes. In orderto edit music more efficiently, we must consider dynamic
marking, beat and so on and we created database contains
them. We developed an inferring process with regards tosimilar phrases using the best alignment and database.
In this paper, we describe the results of searching forsimilar phrases using DP matching and inferring for themusing DP matching and database. By the research until
last year, an editing system for notes with performancesymbols has been constructed. However, there is little
inference about notes without performance symbols, andthere is a problem that inference cannot be performed
sufficiently. In this fiscal year, we focused on the change
in pitch to improve the accuracy of the phrase without theperformance symbol. We aimed at developing a systemthat can get closer to actual performance by finding
characteristics of performance due to change in pitch andediting it based on its characteristics.

value of 1-127. “Gate” is the duration of the note inmilliseconds. “Step” is the interval of time between notes,
and it also exhibits tempo. “Bar” is the vertical line
placed on the staff to divide the music into measures.
The pedal information is comprised of fourparameters: “Key” (indicating the kind of pedal:“Damper” or “Shifting”), “Velo” (the pedaling quantity),“Time” (the duration for which the pedal is applied)”, and“Bar”.
2.2. System Architecture
As shown in Fig.2, the editing support system includes
a database and a user interface. The database is divided
into one storing the characteristics and one storing themusic information of the music, and refer to each otherwith the user interface.
In addition to a system that manually edits music data,
the user interface incorporates a system that can infer the
performance expression desired by the user based on theinformation of the database storing the characteristics.
Also, it consists of a search system and inference system.
Search system searches or phrases (similar phrases)
similar in sequence of phrases and notes input by the user.
Inference system edits the similar phrases using thedatabase.

Fig.2 The structure of the edit system
Fig.1. The automatic piano.

2.3. Search system
As a result of the analysis, it was found that phrases ofthe same pattern existing in the same tune are performed
in a similar expression. This time, we used DP matching
to search for similar phrases.
DP matching is a technique used widely in the field ofspeech recognition, bioinformatics and so on. It has afeature that can calculate the similarity between two
words that are different in a number of characters fromeach other.
In Fig.4, the route of minimum cost in each point istaken, and the route with the lowest cost is assumed
finally to be the optimal path. The cost at that time isdefined as the distance between patterns. In this system,

2. Editing support system
2.1. Performance information
The automatic piano that we have developed uses a
music data structure that is similar to MIDI. We defined
performance information, dividing it into two categories:
the notes and the pedals. The note information iscomprised of the six parameters involved in producing atone: “Key” (note), “Velo” (velocity), “Gate”, “Step”,“Bar” and “Time”. “Velo” is the dynamics, given by the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

133

Interactive Musical Editing Systemaveraging the slope values of the first order approximate
expression obtained from the pitch of each dynamics
marks and the distribution of Velo. Fig.5 shows the
distribution of Fortissimo's strength of sound. The “Gate/ Step” of the sound of repetition and non-repetition of
the same key was obtained by classifying all the sounds
in the song and averaging them.

this distance is handled as a threshold to judge whether
the phrases are similar to each other.
For example, if the cost moves up or to the right, thenit is increased by 1. If it moves to the upper right, then it
does not increase. Also, if the characters do notcorrespond in each point, then the cost is increased by 5.

Table.1 Search result of varying pitch
Performance trend
Gate/Step
Velo/Key
Repetition of the same Key
0.2950―
Not repetition of the same Key
0.8092―
Slope between Velo and Key―
0.2346
Fig.3 patterning of notes

Fig.4 DP MatchingFig.5 Fortissimo's strength distribution in the main melody

Inference was made using the values in Table.1. Velo's
inference method was based on the approximate linear
equation shown in Fig.5. Gate inference was made basedon the average value obtained in the survey.
The inference formula of Velo is shown in Expression(1), and the inference formula of Gate from nonrepetitive key (or repetitive keystroke) to repetitive
keystroke (or non-repetitive keystroke) is shown in
Expression (2). Inference is made for each note,
NewVelo and NewGate indicate the values after
inference, respectively, and Velo and Gate indicate
values before inference, respectively. Key 1 is Key ofsearch phrase, Key 2 is Key of similar phrase.

2.4. Performance information
2.4.1. System overview
Inference of similar phrases is performed based oncorrespondence by DP matching. Inference is done using
the value stored in the database, and the user determines
whether or not to use items such as dynamics marks andrendition style symbols and expresses similar phrases.
This year we added inference based on change in pitch tothis inference system. Hereinafter, this inference system
is called a new inference system.
2.4.2. Inference by pitch change

NewVelo = Velo + (Key1 − Key2) × 0.2346
0.2950( 0.8092)
NewGate = Gate ×0.8092( 0.2950)

In order to understand the characteristics of theperformance expressions due to the change in pitch, thetrend was investigated from the performance data of the
pianist. The studied songs are the first movement of "Appassionata ", performed by Gerhard Opitz.
As a result of the survey, it turned out that there are thefollowing two features.
(i) In the main melody, Velo also increases as the pitch
increases(ii) Gate / Step decreases in case of repetitive keystrokesof the same key
The obtained results are shown in Table.1. The
relationship between the pitch and Velo was obtained by

3. Inference experiment

(1)

(2)

3.1. Experiment procedure
We conducted experiments comparing the newinference system and the existing inference system. The
target song is the first movement of " Appassionata ".
First, similar phrases were inferred by a new inference
system and an existing inference system. Next, in order© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan134

Masahiro Ushio, Eiji Hayashi

to obtain the similarity with actual performance for eachinference result, DP matching was used. Specifically, in
DP matching, VeloRate and Gate / Step are patterned toobtain the distance. Finally, comparing each distance, we
made judgment of improvement / deterioration.
In this experiment, the phrase in Fjg.6 was set as asearch phrase. We searched from the search phrase in
Fig.6 and inferred all similar phrases found. The
inference result was compared with the existing inference
result.

Fig.8 Phrases with improved Gate

3.3. Consideration

3.2. Result

From Velo's improved phrase and Fig.7, it was
confirmed that the sound size (Velo) approached to thereal performance. This is because Velo is made larger
considering that the pitch of similar phrases is generally
higher than the search phrase.
From Gate's improved phrase and Fig.8, it was
confirmed that the value of Gate / Step became smaller
and it became close to real performance. This is because
Gate was reduced in consideration of the similar phrase
being repetitive keystrokes.

Table.2 shows the results of inference experiment for
all similar phrases found by the search system between
the new inference system and the existing inference
system.
From Table.2, we found that the number of phrases
from the new inference is higher than the existing one.
From this fact, it was possible to confirm theeffectiveness of the inference system which added thepitch change in consideration.

4.

Conclusion

We investigated the tendency of the pianist's
performance due to pitch change. The sound was inferredby adding two features to the system that are “soundmagnitude” and “sound length” that occur due to pitch ofchange. By inference experiments, it was confirmed thatthe accuracy was improved from the existing inference’s
result.
However, there remain problems that there are a large
number of phrases that could not be inferred and phrases
that did not change. In order to improve the accuracy,
improvement of the inference method corresponding tothose phrases is required. Future prospects include
inference methods considering fingering that focuses on
the movement of fingers.

Fig.6 Search phrase
Table.2 Comparison result with existing system
Quantity
Similar phrase
2264
Improved result
449
Deterioration result
51
Unchanged result
1764

References1.
Fig.7 Phrases with improved Velo

E. Hayashi et al, Behavior of piano-action in a grandpiano. I, Journal of acoustical Society of America,
Vol.105, pp.3534-3544, 1999.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

135

Analysis of Malaysian Facial Expressions for Designing Virtual Agents
Halimahtun Mohd Khalid

Damai Sciences, A-31-3 Suasana Sentral, Jalan Stesen Sentral 5,
50470 Kuala Lumpur, Malaysia
E-mail: mahtunkhalid@gmail.com
Wei Shiung Liew

Faculty of Computer Science and Information Technology, University of Malaya
50603 Kuala Lumpur, Malaysia

Abstract
The perception of emotion from facial expressions is culturally mediated. Designing facial expressions of avatars
should consider the demographics of users who would be interacting with them. An online survey was conducted todetermine the facial measurements for emotions. Sixteen facial expressions of Malaysians were presented to subjects.
They were instructed to select an emotion label that matched each expression. The expressions were decomposed
into action units using OpenFace. Regression analysis determined the combination of facial measurements for eachemotion.
Keywords: facial expressions; emotion; facial action units; avatar design.

differences between Western and Eastern cultures5
particularly for negative facial expressions6.
When designing realistic artificial faces for virtualavatars or humanoid robots, appearance matters as users
prefer robots that appear friendly and polite, thereby
encouraging emotional connection7. The robot’s ability
to perceive and express emotions via facial expressions
gives an illusion of empathy, which positively influences
human-robot interactions8. Cross-cultural variances infacial expressions would mean that the robots’expressions would have to be tailored specifically for thedemography of persons who would be interacting withthe robot.
In this study, we present our methodology and findings
for surveying and constructing a facial expression
emotion model for an East Asian demographic,
represented by two Malaysian ethnic groups: Chineseand Malays. The objective of the study was to determine

1. Introduction
The universality hypothesis states that certain facial
expressions represent six basic emotions: anger, disgust,fear, happiness, sadness, and surprise; and they are easily
recognizable by people regardless of culture, language or
ethnicity. Several studies have supported and/or refuted
the universality hypothesis. Ekman et al.1 showed
consistent agreement across cultures in theirinterpretation of emotions in facial expressions.
Arguments against the hypothesis typically cited several
factors such as the use of forced choice method, whereby
participants were given only a small list of possible
emotion labels to match with the facial expressions2, andthe use of posed as opposed to spontaneous facial
expressions3. A meta-analysis of emotion recognition4
indicated that emotions were recognized more accurately
when expressed by members of the same national, ethnicor regional group. More recent studies displayed© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan136

Halimahtun Mohd Khalid, Wei Shiung Liew

Low-Arousal,
Positive ValenceHigh-Arousal,
Negative
Valence
High-Arousal,
Positive Valence

how they express and perceive specific emotions basedon facial expressions, measured in terms of facial action
units. Emotion definitions can then be used in designing
realistic facial expressions of virtual avatars that mimic
Chinese and Malay faces.
Figure 1 presents the Arousal-Valence circumplex9
model for analysis of facial expressions. The Arousal
axis (vertical arrow) represents the strength of theemotion, with emotions such as anger and joy having
high arousal, while sad and passive have low arousal. The
Valence axis (horizontal arrow) is an indicator of the
intrinsic attractiveness/averseness of the emotion label.
For example, fear and disgust have negative valence,
while happiness and relaxed have positive valence.
Figure 1 also shows an arrangement of the test form,
comprising of a test facial image, instructions, response
format and emotion wheel.

Comfortable,
Happy,
Relaxed,SatisfiedAngry, Disgusted, Fearful, Sad

Aroused, Excited, Pleased, Surprised

The facial expressions were then processed using

2. Method for Facial Expression Survey
2.1. Study design, material and sample
The study was conducted using an online survey. The
material comprised video recordings of eight test
subjects; two subjects each represented a Malaysian Male
Malay, Male Chinese, Female Malay, and Female
Chinese. Sixteen facial expressions were selected fromeach subject corresponding to the emotion labels
presented in the Arousal-Valence circumplex model.
The sample comprised thirty-seven subjects. They wererecruited from the same four demographics and wereshown the selected facial expressions along with thecorresponding emotion label. Subjects were instructed toconfirm whether the facial expressions match the label.
If not, they were asked to select the most appropriate
emotion label from a given emotion wheel.
Table I shows sixteen emotion labels selected for the
study.

OpenFace10 to extract action unit (AU) measurements
according to the Facial Action Coding System.
2.2. Data analysis and results
The AU measurements for each facial expression werecombined with the emotion labels selected by the
subjects and compiled into a dataset. Each entry in thedataset represented a single facial expression evaluation
by one subject. The dataset was then pruned to exclude
emotion labels that were selected infrequently comparedto other emotion labels, namely: Delighted (1.7% of all
emotion labels), Depressed (2.2%), Enthusiastic (1.0%),
Fatigued (0.5%), and Sluggish (1.7%). The dataset wasthen processed using PART classifier11, which achieved
an accuracy of 51% in selecting the correct emotion label
given the AU measurements.

Table I. Sixteen emotion labels divided into four quadrants by
Arousal-Valence axes
Quadrants
Low-Arousal,
Negative
Valence

Emotion labels
Anxious, Bored, Miserable, Passive

2.2.1. Arousal-Valence classification
Classification of the dataset was repeated but the emotion
labels were replaced with the Arousal-Valence quadrant,
where the emotion labels were located. For example,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, Beppu International Convention Center, Oita, Japan

137

Analysis of Malaysian Facial

Jaw dropped, inner browslightly raised, outer brow notraised, lips slightly tightened,cheeks raised, upper lip
raised, lips parted
Upper lip raised, lips slightly
parted, eyelids not tightened,
lips not tightened

Angry, Fearful, and Delighted were replaced by HighArousal Negatives. Classification using PART achieved
72% accuracy in selecting the correct quadrant given the
AU measurements. Table III is a partial list of the 62 rules
derived from the PART decision list, and Table IV shows
the partial list of rules generated for each emotion
quadrant.
Table III. Action unit rules for each Arousal-Valence quadrant

Action unit rules
Lip corner depressed, lipcorner pulled, upper eyelid
raised, mouth opened
Lip corner depressed, lips
parted, upper eyelid raised,
lip corner not pulled, lips
not stretched, nose notwrinkled, upper lip slightly
raisedLip corner not pulled, lips
not stretched, nose notwrinkled, lip corner
depressed, upper eyelid
raised, lips parted, innerbrow slightly raised,
Lip corner not pulled, lips
not stretched, nose notwrinkled, upper eyelid
raised, lip corner depressed,
lips slightly parted, browslowered, upper lip slightly
raisedQuadrant
HighArousal
Positive
LowArousal
Positive

HighArousal
Negative

Support
3.98%(21/527)

Mouth opened, cheek raised,
upper lip raised, lips slightly
stretched
Upper lip not raised, jaw
dropped
Lips slightly parted, browslightly lowered, slight
dimple, lip corner not pulled,
nose not wrinkled
Lip corner pulled, cheeksslightly raised, outer brow notraised31.65%(44/139)

HighArousal
Negative:
Fearful

7.46%(10/134)

In Table IV, the dataset was segmented into four datasubsets, each containing only the emotion labels from one
quadrant. Classification of emotion labels was thenperformed individually for each quadrant. The PARTclassifier achieved 72% accuracy in selecting the correct
emotion label for facial expressions within the HighArousal Positive emotion labels, 64% accuracy for the
Low-Arousal Positive emotions, 63% for the LowArousal Negative emotions, and 35% accuracy for theHigh-Arousal Negative emotions.

2.27%(12/527)

2.27%(12/527)

3. Designing Virtual Agents
LowArousal
Negative

Using facial expressions attributes from Table III, we
modeled faces of virtual avatars with iClone software.
Facial features of the avatars were manipulated following
the action unit rules. The avatars were then used in ahuman-agent interaction experiment12 to emote various
facial expressions during dialog interactions withhumans. Figure 2 illustrates two avatars designed using
these rules.

4.93%(26/527)

Table IV. Action unit rules for individual emotion labels

Action unit rules

LowArousal
Negative:
Bored

Quadrant/
Emotion
HighArousal
Positive:
Happy
HighArousal
Positive:
Surprised
LowArousal
Positive:
RelaxedLowArousal
Positive:
Happy

Support21.48%(29/135)
16.29%(22/135)
14.38%(20/139)
13.66%(19/139)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, Beppu International Convention Center, Oita, Japan

138

Halimahtun Mohd Khalid, Wei Shiung Liew

9. J. A. Russell, A circumplex model of affect, in J. Pers. Soc.
Psychol. 39(6) (1980) 1161–1178.
10. T. Baltrušaitis, P. Robinson and L. P. Morency, Openface:an open source facial behavior analysis toolkit, in 2016
IEEE Winter Conf Appl Comput Vis (Lake Placid, New
York, USA, 2016), pp. 1–10.
11. E. Frank, E. and I. H. Witten, Generating accurate rule sets
without global optimization, in Proc Int Conf Mach Learn,
ed. J. W. Shavlik (Madison, Wisconsin, USA, 1998), pp.
144–151.
12. H. M. Khalid, W. S. Liew, B. S. Voong and M. G.
Helander, Trust of virtual agent in multi-actor interactions,in Proc 2018 Int. Conf. on Artificial Life and robotics (Oita,
Japan, 2018), in press.

Fig. 2. Male and female avatars exhibiting anger (top),sad (middle), and happy (bottom).
4. Conclusion
Clearly, the perception of emotion from facial
expressions is culturally mediated. Design of avatars’
facial expressions should consider the demographics ofusers who would be interacting with them.
Acknowledgements
This material is based on work partially supported by the
US Aerospace Research and Development office(AOARD), Japan and the US Air Force Office of
Scientific Research (AFOSR), Washington D.C. underGrant No. FA2386-14-1-0016.
References
1. P. Ekman and W. V. Friesen, Constants across cultures in
the face and emotion, in J. Pers. Soc. Psychol. 17(2)(1971) 124–129.
2. M. G. Frank and J. Stennett, The forced-choice paradigm
and the perception of facial expressions of emotion, J. Pers.
Soc. Psychol. 80(1) (2001) 75–85.
3. P. J. Naab and J. A. Russell, Judgments of emotion fromspontaneous facial expressions of New Guineans, in
Emotion 7(4) (2007) 736–744.
4. H. A. Elfenbein and N. Ambady, On the universality andcultural specificity of emotion recognition: a metaanalysis, in Psychol Bull, 128(2) (2002) 203–235.
5. R. E. Jack, O. G. Garrod, H. Yu, R. Caldara, R. and P. G.
Schyns, Facial expressions of emotion are not culturally
universal, in Proc Natl Acad Sci 109(19) (2012) 7241–
7244.
6. R. E. Jack, C. Blais, C. Scheepers, P. G. Schyns, P.G. and
R. Caldara, Cultural confusions show that facial
expressions are not universal, in Curr Biol 19(18) (2009)
1543–1548.
7. R. Parasuraman and C. A. Miller, Trust and etiquette inhigh-criticality automated systems, in Commun ACM,
47(4) (2004) 51–55.
8. B. Gonsior, S. Sosnowski, C. Mayer, J. Blume, B. Radig,
D. Wollherr, D. and K. Kühnlenz, Improving aspects of
empathy and subjective performance for HRI throughmirroring facial expressions, in RO-MAN 2011, eds. M.
Scheutz and H. I. Christensen (Atlanta, Georgia, USA,2011), pp. 350–356.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, Beppu International Convention Center, Oita, Japan139

Development of VR system to enhance
understanding process of robot mechanisms
Alexei Lushnikov

Higher School of ITIS, Kazan Federal University, Kremlyevskaya str., 18,
Kazan, Russia
Vlada Kugurakova

Higher School of ITIS, Kazan Federal University, Kremlyevskaya str., 18,
Kazan, Russia
Arthur NizamutdinovHigher School of ITIS, Kazan Federal University, Kremlyevskaya str., 18,
Kazan, Russia
Timur Satdarov

KUKA Robotics
E-mail: alexkennyy@gmail.com, vlada.kugurakova@gmail.com,
yalopata@gmail.com, Timur.Satdarov@kuka.com
www.kpfu.ru

Abstract
Training for assembly, disassembly, and maintenance skills for engineers or technicians is very important for various
industries. However, preparing these users for the effective use of new skills remains a challenge. Virtual Reality(VR) system integrated with haptic feedback as device interaction is seen as a powerful tool for development andimplementation of a more natural and intuitive user interface. Such user interface can greatly improve the training
experience required for successful execution of the mentioned tasks for complex machinery and at production
facilities. This study describes the ongoing project on the development of a VR training system for an assembly
process of robots or other machinery. A natural interface is proposed that increases the immensity of the virtual
learning environment. The combination of VR-enhanced training should be a great help for users in mastering theassembly procedure and maximize their skill performance as well.
Keywords: virtual reality, VR, immersive, training method, virtual training system, device interaction, NUI, naturaluser interface.

to perform certain tasks or works. Currently, various
industries and education sectors are looking foralternative training method in order to support their

1. Introduction
Training and learning are crucial processes fordeveloping skills and knowledge for new users in order© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Beppu, Oita, Japan

140

Alexei Lushnikov, Vlada Kugurakova, Arthur Nizamutdinov, Timur Satdarov

2. Related Workdemand in producing new technicians with high skill
performance. However, the process of learning, training
and acquiring new skills for assembly or disassembly forthese new users is quite challenging. There is a need tofind an alternative method to help these sectors in
producing effective user and employee with high skill
performance and able to perform the required tasks assoon as the training is completed. Surely Virtual Reality(VR) system is a powerful tool in developing andimplementing a more natural and intuitive interface as
training systems that have been developed in theassembly or disassembly training context. VR
environment offers an intuitive and immersive humancomputer interface, which can be an effective tool for
assembly or maintenance training because users canexperience and familiarise with virtual parts or 3Dmodels in real-time simulation in a virtual environment
system. Additional special equipment integrated with VR
environments such as haptic feedback technologies
which allow for interaction between humans and virtualparts will be beneficial and effective for the learning andtraining of assembly sequences [1].
Since 1990, the industry has seen the interest of VR. By
creating interactive and versatile environments andmock-ups, VR allows, inter alia, to evaluate theaccessibility, feasibility, and ergonomics problematic ofassembly and maintenance tasks, in early stages of thedesign process. It could have a very positive impact, in
terms of cost and time, by reducing the need for physicalmock-ups and validating (or not) early designs. In thecontext of fusion plants, the integration constraints,
maintenance, and safety requirement are key parameters
for the design. VR technologies are very efficient to
optimize such integration, integration within parallel
designing processes while taking into account assembly,
maintenance and safety issues.

Researchers are talking about learning with using VR,
but are not used to it yet. Nevertheless, the article [2]offers a simple assessment of the specific method of
instruction and the method of training for two
experiments with subsequent analysis of the results.
Article [3] describes the logical use of device haptic
devices to increase the immersion of such VR systems.
Actually, this approach is still brand new.
The researches that we've met describe a natural userinterface (NUI) as the main approach to successful
learning in VR.
Natural user interfaces (NUls) [6] aim to read andperceive user inputs without using a fixed position orlimited input devices such as a keyboard. As a rule, NUI
uses computer visualization to identify and track
movements of the whole body, hands or fingers. These
devices are widely being used in gaming applications
now. Devices, such as GloveOne™, Avatar™, or in caseof AR it can be limited by using Microsoft Kinect ™,
Leap Motion ™, allow to capture complicated bodygestures with high accuracy. Integration of NUI with
virtual reality (VR) [7] is a natural progression. Thus,
users have no need to rely on usual input methods forinteracting, instead they can interact directly with objects
in VR.
An interesting example is represented in [9] as the unionof Remote Access Laboratories (RAL), AR и NUI,
allowing users to interact with virtual objects in their
physical work area leading to the creation of interactiveengineering and scientific experiments.
Natural user interfaces (NUI) rely on specialized devices
for obtaining information about the physical position ofa user [10, 11]. Usually this is a real-time data stream in
the form of an array containing the Cartesian coordinatesof the X, Y, Z parts of user’s body, joins and bonesegments.. NUls are usually oriented on “gesture”
control. A gesture is a fixed set of known movements,
representing the specific transition, orientation and© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Beppu, Oita, Japan141

Development of VR system

position of users. NUI system identifies the set ofgestures, that are recognized by an appropriate devices.
A video output is identified on the basis of these gestures.
Recently, physical engines have been included in NUI
applications [12]. It helps to avoid the need for the
gestures based solely on parts of the user’s body. Thesephysical engines can allow the direct interaction between
virtual object, representing a user’s body in virtual threedimensional space, and each interaction between themand experimental objects..
The researchers in [14] investigate the effectiveness andefficiency of the first prototype of the virtual training
system developed within Virtual Simulation andTraining of Assembly and Service Processes in Digital
Factories, and results demonstrate, that virtual training
effectively reduces an error probability during thecompleting the task in comparison with traditional
training.

located in what level, the degree of their nesting and orderof disassembly. The convention in naming is necessary
to understand the ways of fastening elements, which are
finite.
Also the name contains parameters of connection or
fastening, for example, for a hinge – degrees of freedomof links, axes and degrees, which allow control the hinge;
for a rail type of fastening – the axis on which can be
moved, the percentage or a specific value, on which thenode can be moved.
Using the correctly constructed hierarchy of model,
describing the mechanism it can be implemented in-situ
restriction to get an opportunity of automatic movementof individual parts, but it is not developed by us as anincidental possibility of a hierarchical convention of thename. A full description of convention requires serious
consideration.

3. Our idea
It is necessary to be able to know each time how
components of each three-dimensional model of realmechanistic devices are attached and connected.
Our idea is to combine several positions: (1) ahierarchical description of the model elements; (2) aformation of a list of connection types (hinges in different
degrees of freedom, rail, swivel lock, complex composite
compounds, for example, grenade, etc.); (3) a formationof a list of fastener types characteristic of mechanicalobjects (bolt, clip, latch, clamp, yoke, pin, etc.); (4) typeof object from the list; (5) inclusion in a name of theparameters describing it; (6) a meaningful name for theused materials.
Our idea is to create models with the developed
convention of individual model elements (key nodes)
using hierarchical nesting and name, which allows in-situ
to disassemble such model automatically in VR, nomatter what it is - car, robot or another mechanism.
Hierarchy is necessary to understand what details are

Fig.1. Hierarchical nesting of sub-elements by the example of aturbine impeller

Moreover, in order to use NUI it is necessary to take inhands (in VR) or to exchange a type of controller on anecessary tool (spanner, hexahedron, torex, other
different types of screwdrivers (cruciform, slotted, etc.),
pliers, etc.) to disassemble an appropriate connection.
This appropriateness must be specified in a list ofconnection types.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Beppu, Oita, Japan142

Alexei Lushnikov, Vlada Kugurakova, Arthur Nizamutdinov, Timur Satdarov

A model of an abstract jet engine is taken for realization(Fig.2 – Assembled
jet engine model; Fig.3 –procedurally disassembled).

4. Our method
Our development is based on Unity. The parser, created
for realtime upload of models, using structure and model
name, adds necessary components to the prefab of theobject on the stage.
Such system allows to process each model, in which thenames of objects follow the chosen convention. All
uploaded models after the processing by the program are
structured so they can be assembled and disassembled incorrect order. For example, an object with a nameobj[3.3] can be disassembled only after an object with aname obj[2.3] is disassembled and so on. The essential
feature is independence of the program on a specificimplementation, i.e. on a specific model. A universal
constructor of virtual environment is developed.

Fig.3. Disassembled jet engine model

A similar solution has been developed for the popular
Ubot robot (Fig.4) by KUKA Robotics corporation todemonstrate our ideas of using combination of the special
convention for creating 3d models, natural user interface
to learn the structure of mechanisms, using disassembly
based on a approach of a changeable toolkit (Fig.5).

5. Concrete realization
The format of the training complex proposed by us
allows create product placement for each firm, thatspecializes in unique technique that needs not only forusing by individual purchasers, but for training, forexample, for training employees of repair services.

Fig.2. Assembled jet engine modelFig.4. Assembled KUKA Ubot robot model© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Beppu, Oita, Japan143

Development of VR system

6. Conclusion
References

These systems are great alternatives and are also effectivein learning and training in order to prevent the excessive
cost of using the real machine or components which
could cause damages; or create disruptions to the actual
working process.
This VR-solution helps to explore quickly the device of
unfamiliar technology, "in the field", to understand thestructure of mechanisms and their components, to study
the necessary tools for those or other connections.

1. T. Gutiérrez, J. Rodríguez, Y. Vélaz, S.C.A. Suescun, andE.J. Sánchez, IMA-VR: A Multimodal Virtual Training
System for Skills Transfer in Industrial Maintenance andAssembly Tasks (Journal of Mechanical Engineering,Vol. ME 40, No.1, 2009), pp.68-71.
2. M.W. Rohidatun, A.A. Faieza, M.Y. Rosnah, S. Nor
Hayati and I. Rahinah, Development of Virtual Reality(VR) System with Haptic Controller and Augmented
Reality (AR) System to Enhance Learning and Training
Experience, International Journal of Applied Engineering
Research, vol.11, no.6, 2016), pp.8806-8809.
3. F.A. Aziz and M. Mousavi, “A Review of Haptic
Feedback in Virtual Reality for Manufacturing Industry”,
in J. Mech. Eng., vol.40, no.1 (2009), pp.68–71.
4. V. Abramov, V. Kugurakova, A. Rizvanov, Abramskiy,
M., N. Manakhov, M. Evstafiev, M. and D. Ivanov, Virtual
Biotechnological Lab Development (BioNanoScience,
vol.7, no 2, 2017), pp.363-365.
5. V. Kugurakova, M. Khafizov, R. Akhmetsharipov, A.
Lushnikov, D. Galimova, V. Abramov and O. Correa
Madrigal, Virtual surgery system with realistic visual
effects and haptic interaction (ICAROB 2017: Proceedingsof the 2017 International Conference on Artificial Life andRobotics, 2017), pp.P86-P89.
6. W. Liu, Natural user interface-next mainstream productuser interface (Computer-Aided Industrial Design &Conceptual Design – CAIDCD), 2010 IEEE 11th
International Conference on, vol. 1, pp. 203-205, 2010.
7. D. Müller, A. Chilliischi, S. Langer, Integrating immersive
3D worlds and real lab equipment for teachingmechatronics (Remote Engineering and Virtual
Instrumentation – REV) 2012 9th International
Conference on 2012), pp. 1-6.
8. Y. Ohta, H. Tamura, Mixed reality: merging real andvirtual worlds (Incorporated: Springer Publishing
Company, 2014).
9. Maiti, A., Maxwell, A.D. and Kist, A.A., Using marker
based augmented reality and natural user interface forinteractive remote experiments (Proceedings of 2017 4thExperiment at International Conference: Online
Experimentation, exp.at 2017), pp. 159.
10. L. Garber, Gestural Technology: Moving Interfaces in a
New Direction [Technology News], Computer, vol. 46,
pp. 22-25, 2013.
11. M. N. Kamel Boulos, B. J. Blanchard, C. Walker, J.
Montero, A. Tripathy, R. Gutierrez-Osuna, Web GIS inpractice X: a Microsoft Kinect natural user interface forFig.5. Disassembled KUKA Ubot robot model

Main application of our idea is an educational system of
each mechanisms as a product placement for big
corporations. Also this method can be applied to otherapplications, for example, to study the composition ofchemical substances or for virtual prototyping of
nanostructures.
Acknowledgements
This work was funded by the subsidy allocated to the
Kazan Federal University for the state assignment in thesphere of scientific activities.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Beppu, Oita, Japan

144

Alexei Lushnikov, Vlada Kugurakova, Arthur Nizamutdinov, Timur Satdarov

Google Earth navigation (International Journal of Health
Geographics, 2011), vol. 10, pp. 45.
12. C. Donalek, S. G. Djorgovski, A. Cioc, A. Wang, J.
Zhang, E. Lawler et al., Immersive and collaborative datavisualization using virtual reality platforms (2014 IEEE
International Conference on Big Data), pp. 609-614.
13. H.G. Hoffmann, Physically touching virtual objects usingtactile augmentation enhances the realism of virtualenvironments (Virtual Real. Annu. Int. Symp. 1998, Proc.
IEEE, IEEE, 1998), pp. 59-63
14. Langley, A., Lawson, G., Hermawati, S., D'Cruz, M.,
Apold, J., Arlt, F. & Mura, K., Establishing the Usabilityof a Virtual Training System for Assembly Operations
within the Automotive Industry (Human Factors and
Ergonomics in Manufacturing, 2016), vol. 26, no. 6, pp.
667-679.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Beppu, Oita, Japan145

Towards the immersive VR:
measuring and assessing realism of user experience
Vlada Kugurakova

Higher School of ITIS, Kazan Federal University, Kremlyevskaya str. 18,
Kazan, Russia
Alexander ElizarovHigher School of ITIS, Kazan Federal University, Kremlyevskaya str. 18,
Kazan, Russia
Murad KhafizovHigher School of ITIS, Kazan Federal University, Kremlyevskaya str. 18,
Kazan, Russia
Alexei Lushnikov

Higher School of ITIS, Kazan Federal University, Kremlyevskaya str. 18,
Arthur NizamutdinovHigher School of ITIS, Kazan Federal University, Kremlyevskaya str. 18,
Kazan, Russia
E-mail: vlada.kugurakova@gmail.com, amelizarov@gmail.com,
murkorp@gmail.com, alexkennyy@gmail.com, yalopata@gmail.com
www.kpfu.ru

Abstract
The use of Virtual Reality Continuum grows more diverse with possible applications varying from research andeducation to leisure and work, so the demand rises to for a method for a precise and reliable way of measuring Virtual
Reality (VR) effectiveness. How can we tell apart "badly made" VR from the "good" one? How do different people
respond to different implementation and aspects of it? And could it possibly be true that different social groups handle
VR in a different way? In this paper, we propose a new approach to this emerging problem, which involves applying
user interface (UI) metrics to VR experience. We review existing ways to increase VR immersion and outline a seriesof possible experiments to study and hopefully find a correlation between immersiveness and VR user interface
effectiveness.
Keywords: Virtual Reality, Virtual environments, immersion, user interface, natural user interface.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

146

147

presence is considered the "psychological, perceptual andcognitive consequence of immersion" [2]. The origin of
two different names describing almost the same orsimilar phenomenons is due to the areas those terms arecommonly used in or come from. While presence is aterm originating from the area of teleoperation,
immersion is commonly used in areas like games ormovies [3].
There exist various conceptions of what immersion is,but an exact definition is hard to formulate. Among themost recognized definitions of immersion is the one by
Murray: Murray describes immersion as "the experienceof being transported to an elaborately simulated place".
He details that immersion is "the sensation of being
surrounded by a completely other reality, [...] that takes
over all of our attention, our whole perceptual apparatus"[4]. Lombard and Ditton analyzed several definitions andconcepts of presence and they identified a "central idea"
shared by all conceptualizations that are very close to
Murray’s definition of immersion: Presence means "theperceptual illusion of nonmediation" [4]. In other words,
when immersed, the user behaves as if the perceived
world would not be mediated by human-madetechnology but instead as if he would really be inside thevirtual environment. The user "fails to perceive or
acknowledge the existence of a medium in his/hercommunication environment and responds as he/shewould if the medium were not there" [5]. In the context
of VR systems, this can be interpreted as a mental andphysical state in which the user is not directly aware of
the VR system. Instead, the VE has taken over most (ifnot all) of his attention. As a consequence, the user
unconsciously behaves as if he would really be present in
the computer generated world and not as if he would
control a computer [6]. When being immersed, there is
no longer a border between the user and the virtual world
with its virtual objects because the technology behind theillusion becomes perceptually invisible [5].
As immersion is a perceptual phenomenon, immersion is
not a property of the system but instead a property of theuser. It is something individual and dynamic that isdifferent across users and also time-varying for eachindividual user [5]. A user’s degree of immersion
changes as he interacts with the VR system. In somemoments in time he may be more immersed than in others.
Research on immersion in games showed that once acertain degree of immersion is established, it can makethe user fail to notice usability issues and behavioral
discrepancies [7].

1. Introduction“Virtual reality is a high-end user-computer interface thatinvolves realtime simulation and interactions throughmultiple sensorial channels. These sensorial modalities
are visual, auditory, tactile, smell and taste" [1].
Well, now, when VR is emerging from the zone of purescience-fiction to become a ubiquitous technology, newproblems arise. The fundamental research is necessary
for the first place to find solutions to basic challenges
encountered in all application areas.
Virtual Reality (VR) allows people to experience
immersive virtual worlds in a way that makes them feel
as if they are in them. During such an experience,
multiple channels of the user’s perceptual system such as
the visual, auditory and haptic channel are stimulated by
the virtual world. These stimuli are triggered bycomputer systems simulating the Virtual Environment(VE).
Immersiveness is a degree of user involvement in VE. It
characterizes, how a user feels in it and how quickly hecan learn to manage it.
Thus, immersion is a user property to perceive virtual
space as an existing reality. By definition, the higher
value of immersion, the more comfortable and productive
a user interacts with VR.
As the problem of seeing and hearing the virtual world issolved by modern VR goggles, the next modality to be
included in VR systems is touch. As passive observation
will not be enough in the long term, solutions that let theuser interact with the virtual world in a natural way
become crucial. One possible way to make VR tangible
is to use real-world physical props, also called proxies,
that are coupled to virtual objects. The user can theninteract with virtual objects by interacting with thecorresponding representative prop.
To develop predictable systems it is necessary to know
about the influence of this mismatch on the user’s
behavior in the VR and his perception of the virtual
object.
To measure the immersiveness it is enough to measure
the effectiveness of interaction with VR. However, there
are other approaches to measure immersion.
2. Related Work
The term immersion is frequently mentioned in the same
breath with the term presence in VR literature. In fact,depending on which definition of immersion or presence
is considered, the difference between immersion andpresence can be so small that both terms are frequently
used as synonyms. However, subtle differences exist as
immersion is a more "technology-related" term while

2.1 Influential Factors and Layers of Immersion
As immersion originates from the interaction of the user
with the VR system, the VR system is not solely© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

148

Vlada Kugurakova, Alexander Elizarov, Murad Khafizov, Alexei Lushnikov, Arthur Nizamutdinov

prerequisite for each of those levels is that the barriers tothem (on user-side and system-side) are removed and thatall previous levels are already established. Fulfillment of
this prerequisite does not, however, guarantee the next
highest level of immersion to ensue but rather allows forit. The lowest level in this model is engagement, followedby an increased immersion in the level engrossment. The
ultimate level of immersion is called total immersion andrefers to the extreme case as defined above by Lombard
and Murray. Concerning interaction, Brown and Cairns
mention that "an invisibility of controls" is a fundamental
prerequisite for total immersion to accrue [8].

responsible for the experienced immersion. It is rather aproperty of the user. But some aspects of the VR system
can facilitate immersion while others can be obstructive.
VR systems aim to be media that foster long-lasting andstrong immersion. Concerning the audiovisual dimension
in VR systems, the usage of e.g. VR-HMDs andheadphones blends out the stimuli of reality andsubstitutes it with stimuli of the VE. By this, some factorsthat may inhibit immersion are effectively removed
which, in principle, opens the way to greater degrees ofimmersion. But it is important to note, that this is not anecessity for immersion to ensue. Immersion occurs also
during desktop gameplay at a PC with a normal sized
computer monitor that does not completely blend out thesurrounding [6]. The audiovisual implementation ispresumably an important factor influencing immersion
but certainly not the only one nor the most important.
While quality improvements in all three modalities(visual, auditory and haptic) of multimodal VRs does
influence immersion, structural aspects concerning thevirtual world seem to be at least as crucial [3, 6].
McMahan states that for an immersive VR experience,the conventions of the VE should match the
expectations of the user,
the user should be able to do non-trivial andmeaningful actions that influence the VE andthe virtual world must be consistent.
These structural factors promote interactivity which isencouraged by challenges and tasks. There even exists
evidence, that the difficulty of the challenges in thevirtual world can affect immersion [6]. This againemphasizes the importance of taking the step from pure
passive VR applications to interactive applications asinteractivity unlocks the potential of greater immersivity.
Ermi and Mäyrä developed a gameplay experience modelthat describes three kinds of immersion that occur during
gameplay [6]. These levels of immersion exist in VR and
non-VR gameplay. The first level is sensory immersion,
which describes the immersion coming purely from thesensation of audio/visual and haptic stimuli. The
virtuality-based sensations overpower those from reality
and thus immerse the player on a sensory level. The
second level is challenge-based immersion which is theimmersion coming from balanced challenges in the world.
The user’s tasks are demanding but match his abilities
and through interaction he enjoys to fulfill and masterthem. The third level of immersion is the imaginative
immersion that accrues when the user is absorbed in thestories of the game, uses his imagination and starts to feelfor or identify with a game character.
Brown and Cairns introduced another model ofimmersion which is divided into three levels representing
increased immersion [8]. According to the authors, the2.2 Virtual reality and natural user interfaces
A user interface is an interlayer, providing the sharing
information between a machine and a person. VR
technology allows a person to interact with a machinewith new, previously unused methods. Therefore, VR has
a variety of user interface, or its possible characteristic,
or a range of technologies, which can be used in theinterface.
It is necessary to define a new type of user interfaces:
natural user interface (NUI).
Currently, we are moving into the era of the NUI, whichuses a user’s natural gestures and voice, through the eraof the most widely - used GUI. Examples of NUIs
include a user interface based on gestures using fiber,magnetic sensors, and a data glove [17]. A UI using
physical devices will accurately recognize a user’s
gestures, but the specific physical device must be
attached to the user. Another example is a UI based onvoice, which is the most universal and easiest method ofcommunication between humans.
Characteristics of NUI are following:
Using for communication and interaction with amachine the same resources as for communicationand interaction with people and real world.
Intuitiveness, that is not easy to evaluate objectively,
due to its abstractness. In order to evaluate theintuitiveness of the proposed system, in [18] notprovide subjects with instructions on the proposed
hand-mouse interface, but measured the time taken
to learn how to use it. Most of the subjects learned
how to use the hand-mouse interface within 1 min.
The previous examples of using NUI: Anthropomorphic
Artificial Social Agent with Simulated Emotions forfree communication [13], Virtual Biotechnological Lab[14], Virtual surgery system with realistic visual effects
and haptic interaction [15], VR system to enhance
understanding process of robot mechanisms.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan149

Towards the immersive VR

With the development of new haptic devices NUI
application spheres are increased. The aim behind thenatural paradigm is to remove barriers and offer a styleof interaction more similar to the utilized in the realworld; conveniently, the use of gestures can provide an
attractive and natural alternative [19]. To our opinion, tomeasure the effectiveness of immersion it is sufficient tomeasure the effectiveness of NUI.
Haptic feedback can increase the realism wheninteracting with virtual objects in a virtual environment.
User test [20] shows that users can perceive the simulated
surface texture with at least 70% of correct rate.

Fig.1. View of the virtual environment, representing thebiotechnology laboratory

2.3 Immersiveness measurement
Different methodologies exist to measure immersion orpresence. One can try to capture the degree of immersion
by observing the user’s behavior or by monitoring
physiological signs [9, 10]. Moreover, a huge body ofliterature exists on presence measurement questionnairesthat try to deduce the degree of immersion from answers
and ratings of the user [11].
A common questionnaire used in the related literature is
the so called SUS-Presence-Questionnaire, named after
the authors Slater, Usoh and Steed [9-12]. It aims to map
the user’s degree of immersion to a number between 0(not immersed at all) and 6 (totally immersed). It consistsof six questions rated on a scale from 1 to 7 and the finalscore is the amount of high ratings (6 or 7) per person.
The SUS questionnaire primarily considers the "sense ofbeing there", the "extent to which the VE becomes
more ’real or present’ than reality" and the "extent towhich the VE is thought of as a place visited" [11].
Interesting results are represented in the article [16],where is defined the concept of Dynamic Passive Haptic
Feedback (DPHF) for VR by introducing the weightshifting physical DPHF proxy object Shifty, combines
actuators known from active haptics and physical proxies
known from passive haptics to construct proxies that
automatically adapt their passive haptic feedback.
The results show that such object enhances the perceptionof weight and thus the perceived realism by adapting its
kinesthetic feedback to the picked-up virtual object.
Такие подходы shown to increase the user's fun andperceived realism significantly, compared to an
equivalent passive haptic proxy. Adaptations in all
dimensions of proxy properties like for example shape,size, temperature, texture, absolute weight, andcombinations thereof, can be imagined.

3.

Technical implementation

Our aim is to develop an algorithm with different waysof increasing of the immersion, which could be typicalfor VR and frequently used.

Fig.2. Pouring liquid is accompanied with visual effects and thesound of falling water from a certain height

We chose (Fig.1) the development of virtual
biotechnology laboratory as the implementation [13].
This project was implemented in VR using Unity andaccompanied by additional visual effects such as pouringliquid (Fig.2), bouncing back of virtual objects fromvirtual surfaces, sound effects of broken glass, opened
doors (Fig.3), working engines and other effects.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

150

Vlada Kugurakova, Alexander Elizarov, Murad Khafizov, Alexei Lushnikov, Arthur Nizamutdinov

3.
4.
5.
6.

7.
Fig.3. Mechanisms are accompanied with sounds, fading withuser’s moving away; Opening doors are accompanied withsounds, that vary depending on door’s weight4.

8.

Results

Within the full synchronization of movement withaccording visual and audible feedback a user's perceptionof the virtual environment changes. Moreover, we claim,that users perceive a virtual object as heavier, more solidor softer depending on visual and audible feedback withan increasing number of according effects (sound of fall,
amplitude of rebound, etc.) in a real laboratory. On thecontrary, being decreased a virtual object is perceived as,
for example, more fragile or lightweight.
Realism, perceived by a user can be improved by greatly
synchronizing properties of virtual and real objects.
These statements were reflected in some ourexperimental estimates.
5.

9.
10.

11.

Future work

12.

Future work could integrate weight-shifting feedback
into vibrotactile VR controllers. it would be interesting toinvestigate all types of object changes that can beenhanced by weight shifts. Considering new haptic
devices as an input device for native user interfaces inVR, we can investigate how it changing interactions of auser with a virtual environment. we can be imagined avariation of adaptations in all dimensions of proxy
properties like for example shape, size, temperature,
texture, absolute weight, and combinations thereof.

13.

14.

15.

References
1.
2.

G. C. Burdea and P. Coiffet, Virtual Reality
Technology. 2 (New York, NY, USA: John Wiley &Sons, Inc., 2003).
D. Mestre, P. Fuchs, A. Berthoz and J. L. Vercher,
Immersion et présence. In: Le traité de la réalité
virtuelle. (Paris: Ecole des Mines de Paris, 2006), pp.
309–38.

16.

A. Mcmanhan, Immersion, engagement and presence(The video game theory reader 67, 2003), pp. 86.
J. H. Murray, Hamlet on the Holodeck: The Future of
Narrative in Cyberspace (Free Press, 1997).
M. Lombard and T. Ditton, At the Heart of It All: The
Concept of Presence (Journal of Computer-MediatedCommunication 3, 1997), N. 2.
L. Eemi and F. Mäyrä, Fundamental components ofthe gameplay experience: Analysing immersion(Worlds in play: International perspectives on digital
games research 37, 2005).
K. Cheng and P. A. Cairns, Behaviour, Realism and
Immersion in Games (CHI ’05 Extended Abstracts on
Human Factors in Computing Systems. New York,NY, USA: ACM, 2005). – ISBN 1–59593–002–7, pp.
1272–1275.
E. Brown and P. A. Cairns, A Grounded Investigationof Game Immersion (CHI ’04 Extended Abstracts on
Human Factors in Computing Systems. New York,NY, USA : ACM, 2004). – ISBN 1–58113–703–6, pp.
1297–1300.
B. E. Insko, Passive haptics significantly enhances
virtual environments (University of North Carolina at
Chapel Hill, Diss., 2001).
M. Usoh, K. Arthur, M. C. Whitton, R. Bastos, A.
Steed, M. Slater, Jr F. P. Brooks, Walking> walkingin- place> flying, in virtual environments(Proceedings of the 26th annual conference on
Computer graphics and interactive techniques ACM
Press/Addison- Wesley Publishing Co., 1999), pp.
359–364.
J. Van Baren and W. Ijsselsteijn, Measuring presence:
A guide to current measurement approaches(Deliverable of the OmniPres project IST-2001-39237,
2004).
M. Slater, M. Usoh and A. Steed, Depth of presence invirtual environments (Presence 3, 1994), Nr. 2, pp.
130–144.
V. Abramov, V. Kugurakova, A. Rizvanov,Abramskiy, M., N. Manakhov, M. Evstafiev, M. andD. Ivanov, Virtual Biotechnological Lab Development,
in BioNanoScience, vol. 7, no. 2 (2017), pp. 363-365.
V. Kugurakova, M. Talanov, N. Manakhov, N. and D.
Ivanov, Anthropomorphic Artificial Social Agent with
Simulated Emotions and its Implementation, inProcedia Computer Science (2015), pp. 112.
V. Kugurakova, M. Khafizov, R. Akhmetsharipov, A.
Lushnikov, D. Galimova, V. Abramov and O. Correa
Madrigal, Virtual surgery system with realistic visual
effects and haptic interaction, in Icarob 2017:
Proceedings of the 2017 International Conference on
Artificial Life and Robotics (2017), pp. P86-P89.
A. Zenner and A. Kruger, Shifty: A Weight-Shifting
Dynamic Passive Haptic Proxy to Enhance Object
Perception in Virtual Reality (IEEE Transactions on
Visualization and Computer Graphics 23 (4), 2007),
pp. 1312-1321.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan151

Towards the immersive VR

17. D. J. Sturman and D. Zeltzer, “A survey of glovebased input,” IEEE Comput. Graph. Appl., vol. 14, no.
1, pp. 30–39, Jan. 1994.
18. S. Kim, S. Yeom, O. Kwon, D. Shin, Hand-Mouse
Interface using Virtual Monitor Concept for Natural
Interaction(IEEE,
Access.
doi:10.1109/ACCESS.2017.2768405, 2017).
19. D. M. Kaushik and R. Jain, Gesture based interaction
NUI: an overview (Int. J. Eng. Trends Technol. 9(12),2014), pp. 633–636.
20. Z. Lin and S. Smith, A natural user interface forrealistic tactile perception of object surface texture(Proceedings - 9th International Conference onIntelligent Human-Machine Systems and Cybernetics,
IHMSC 2017), pp. 370.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan152

Lessons on the Reality-Gap: Iterations between Virtual and Real Robots
Andre Rosendo

Living Machine Laboratory, ShanghaiTech University
Shanghai,201210, China
Charlie Houseago, Fumiya Iida

Bio-inspired Robotics Laboratory, University of Cambridge
Cambridge, CB21PZ, United Kingdom
E-mail: arosendo@shanghaitech.edu.cn,charliehouseago@gmail.com, fi224@cam.ac.uk
www.shanghaitech.edu.cn

Abstract
Due to approximations between the virtual and real world, the knowledge transfer from simulations to robots is
problematic. As physical conditions are prone to unknown and stochastic noise sources, the predictability reduces.
We use data experiments from 100 real robots to tune the parameters of a simulation, and later used this tuned
simulator to improve the design of the previous robots and find the optimum robot. We compare the simulated andobserved behavior of this robot, and discuss our results.
Keywords: Reality gap, evolutionary robotics, virtual robots, design optimization, body-mind co-optimization.

1.

insure robust, transferable behavior, but is therefore
limiting in terms of broad applicability.
One successful approach to handling the reality-gap
problem for control policy evolution is the transferability
function, most notably employed in [4]. Under thisframework, an additional measure aside from theobjective function is employed which assigns atransferability to each potential control policy based on asmall number of experiments in reality. They were able
to demonstrate success in producing a transferability
function using only 10 evaluations from the hardware in
reality. Alternatively, rather than attempting to bridge thereality-gap for co-evolution of structure and control
policy, the work conducted in [5] evolved 200 robots in
the real world using an automated assembly andevaluation process. This consisted of a UR5 robotic arm,and a number of cubic modules, which were attached
together using hot-melt adhesive. This experimentdemonstrated the success of a real-world evolutionary

Introduction

The major obstacle of researchers creating a bridge
between computer simulations of robots and the behavior
seen in the real world is termed the ‘reality-gap’ problem[1]. Although simulations can be helpful to predict
behaviors, a few researchers in robotic simulation
community have repeatedly warned of the dangers ofusing over-simplified tools to simulate the behavior ofreal-world robotic agents [2]. A simulation by its very
nature involves a number of levels of abstraction fromtrue physical behavior, and thus fundamentally differ
from the real robots they are attempting to emulate.
One common way of overcoming the reality gap fordesign of controllers is to use the notion of transferability,
whereby it is hypothesized that different controllers willcorrespond better across the reality gap than others. The
work of Jakobi [3] stressed the use of minimal
simulations, which targeted areas of reality that would© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

153

Andre Rosendo, Charlie Houseago, Fumiya Iida

process which co-evolved both the physical structure andthe control policy of a population of robotic agents.
In here we use the training data of experiments
performed in the aforementioned experimental setting [6]to tune the parameters governing physical interaction insimulation. In this data-driven approach, we design our
simulation environment to create a stronger correlation
with the real-world and eventually achieve better
correlation between reality and simulation. In addition,
we wish to demonstrate that the relationship between
simulation and reality is dependent both on themorphology and the control, and thus there is nouniversal transferability function that is independent of
the morphology.

experiments and the simulated structure can be seen in
Fig.1.

2. Methods

Fig. 1. Comparison of the real-world construction (above) andthe simulation created from these constructions (below).

In the first stage of work, a simulation environmentis designed to conduct virtual experiments similar to theones performed at [6], and the output of which can bechanged with different floor parameters which describe
the response of the environment. Then we perform a
Principal Component Analysis (PCA) of the results fromreality with those of simulation to evaluate how close thebehaviors are. A training process is then applied tochoose the next set of parameters, and the processrepeated. The output of the process is the choice ofparameters which gives the smallest reality-gap, i.e. thosefor which the results from reality and from simulation areclosest together.

2.2. PCA of simulator with reality
In addition to comparing agents between reality andsimulation based on fitness, we can use a more advanced
method of matching agents based on the entire trajectory
travelled. A number of different methods of performing
2D trajectory comparison exist, and there is an existing
body of research into their applications in video motion
tracking. As an example, in [8], the Hausdorff distance isused to express the spatial similarity between two
trajectories.
In here, we employ an approach developed in [9],based on representing a trajectory as a cluster of data
points using it’s PCA coefficients. Given the relatively
short length of trajectories in our experiment, and thelarge number of evaluations, we will work on the entire
trajectory as one data cluster. The procedure while
applying to a simulated trajectory consists of:
1. Subsample simulated trajectory to 182 data points
2. Translate both trajectories to start at the origin
3. Rotate trajectories such that their vector is aligned
4. Compute principal components and compare
As it can be seen in Fig. 2 and 3, the main trajectoryof the real-world experiment depicted in Fig. 2 iscompared with nine simulated trajectories from Fig. 3.
The simulated behaviors from Fig. 3 have different floor
parameters, simulating different friction and restitution
coefficients. The number above each of these nine
trajectories indicate the PCA discrepancy to the realworld result. The simulations from the middle row of

2.1. Simulator and virtual model
A plentitude of physics simulators exists, and for thiswork we oriented our choices with the references [7], andwe eventually chose Bullet to recreate virtuallyexperiments that we performed a few years ago [6].
Bullet handles rigid collisions with greater accuracy, andwe avoid the main limitation of the environment – poor
damping behavior - due to the simplistic nature of theagents involved in this experiment.
The process was performed in a similar way to theaforementioned model-free morphology experiment. The
genome structure comprises of 10 numeric values per
cube (control and morphology), with one additionalparameter per robot which defines the total number ofcubes. Specifics of these parameters can be found at [6].
The comparison between the baseline real-world© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan154

Lessons on the Reality-gap

Fig.3 better approximate the real-world behavior, as not
only the first principal component (fitness) is closer, but
the shape of the trajectory curve is closer (better matchwith second and subsequent components.

agents body spends in contact with the terrain, and thus
directly impact the response due to friction.
In Fig. 4 we show a heat map of thefriction/restitution difference between the simulated
fitness and the real world fitness for the robot
morphology used to create the trajectory from Fig. 2.
This data was collected at 441 evenly spaced points
across the specified parameter ranges. The plot displays
much steeper gradients in regions of high friction andrestitution, as the response to changing either of theparameters becomes more severe. The shape of this plotis dependent on the robot morphology, in this case, theadopted robot moves with three active faces in contact
with the ground surface. Depending on the physicalvalues, different servos can be in contact with the groundat different parts of the motion cycle.

Fig. 2. Trajectory observed at the real-world experiment. The
initial position is at (0,0), and distances are in centimeters.
14.1044

0.01

0

0

-0.01

-0.01

-0.01

-0.02

-0.02

0

0.05

0.1

0.15

0.2

0.13942

0.3

-0.02

-0.03
-0.05

0

0.05

0.1

0.15

0.2

0.13755

0.4

0.2

-0.03
-0.05

0

0.2

0.05

0.1

0.15

0.2

2

2.5

0.15082

0.2

0.1

0.1
0
-0.1

0

0.5

1

1.5

2

2.5

0.41211

0.4

0

0

-0.2

-0.1

0

0.5

1

1.5

2

2.5

1.7392

0.2

0.5

1

1.5

2.3927

Fig. 4. Heat map of the difference between Simulated and Realbehavior for changes in friction and restitution. The yellow areashows a promising region, where those parametric choices
translate into a faithful reproduction in reality of the simulated
trajectory.

0.4

0

0.2

0

0.6

0.3
0.5

3.7152

0.01

0

-0.03
-0.05

Friction

8.7337

0.01

0

0.2
0.1

-0.2

0

0
-0.1
-0.5

0

0.5

1

1.5

2

-0.4
-0.5

0

0.4

0.5

1

Restitution

1.5

2

-0.2
-0.5

0

0.5

1

1.5

2

0.6

Fig. 3. Simulated trajectories with different friction andrestitution coefficients. The number above each of these ninegraphs stands for the difference between this particular case andthe real-world behavior.

3.

Results and Discussion

With an optimized simulator in our hands we decided
to conduct a Grid Search through the possible
morphological and control combinations to create thebest performing robot. The robot depicted in Fig. 52.3. Optimizing parameters
The interaction between the agent and the ground isvery complicated, and increasing the friction not only
increases the impulse given to the agent by collisions of
the rotating face with the terrain, but also increases thedrag on the body as the agent moves itself along the
ground. Changes in the restitution impact the time that an© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

155

Andre Rosendo, Charlie Houseago, Fumiya Iida

better solution for this problem. Although a strong
methodology was adopted, in this work we failed to
predict the behavior of our proposed robot, and webelieve that this work will guide others in search of asimilar answer.

showed the best behavior, and it was build and tested in
the real world.

4.

References

1. N. Jakobi, P. Husbands and Harvey I, “Noise and theReality Gap: The use of simulation in Evolutionary
Robotics,” in Advances in Artificial Life: Proceedings ofthe 3rd European Conference on Artificial Life, 1995.
2. R. A. Brooks, “Artificial Life and Real Robots,” in Towarda Practice of Autonomous Systems: Proceedings of theFirst European Conference on Artificial Life, 1992.
3. N. Jakobi, “Half-Baked, Ad-hoc and Noisy: Minimal
Simulations for Evolutionary Robotics,” in Fourth
European Conference on Artificial Life, 1997.
4. S. Koos, J. B. Mouret and S. Doncieux, “TheTransferability Approach: Crossing the Reality Gap in
Evolutionary Robotics,” IEEE Transactions on
Evolutionary Computation., vol. 17, pp. 122-145, 2013.
5. V. Vujovic, A. Rosendo, L. Brodbeck, F. Iida,“Evolutionary developmental robotics: Improving
locomotion of physical robots.” Artificial Life 23, 2 (2017).
6. L. Brodbeck, S. Hauser and F. Iida, “Morphological
Evolution of Physical Robots through Model-Free
Phenotype Development,” PLOS One, 2015.
7. A. Boeing and T. Braunl, “Evaluation of real-time physics
simulation systems,” Graphite '07, pp. 281-288, 2007.
8. I. Junejo, O. Javed and M. Shah, “Multi-Feature Path
Modelling for Video Surveillance,” Proceedings 17th
international Conference on Pattern Recognition, 2004.
9. F. I. Bashir, A. A. Khokhar and D. Schonfeld, “Segmented
trajectory based indexing and retrieval of video data,”
Proceedings International conference on Image
Processing, 2003.

Fig. 5. Picture of the best predicted morphology, to which the
behavior is finally compared with the simulated behavior.

In Fig. 6 we present the final outcome of this work.
The comparison between the simulated trajectory and thereal-world trajectory can be seen. Although bothtrajectories present a strong resemblance in trajectory
shape, the simulated behavior was apparently a scaled
down version of the real world output. One additional,
and extremely important caveat: This resemblance intrajectory shape was only possible in 15% of the trials, as
the robot followed different trajectories in other 60% of
the cases, and in the last 25% didn’t translate at thetesting platform (solely rotation).
The previous result brings in question thedeterministic approach to the Reality Gap: if thestochastic influence is such that a predicted behavior israrely occurring, probabilistic approaches, with means
and standard deviations of fitness values, might be a

Real World Trajectory2

Fig. 6.
The
difference
betweensimulated
andrealworld
trajectories,
and the low
repeatabilityof the results
puts
inquestion theefficacy ofdeterministic
approaches.

1

0

-1

-2

-3

-4

-5

-2

0

2

4

6

8

10

12

14

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan156

Fractional Order Sliding Mode Control Applying on the HIV Infection System

Thunyaseth Sethaput

Dept. of Mechanical Engineering, Sirindhorn International Institute of Technology, Thammasat University,
Pathum Thani, Thailand
Arsit Boonyaprapasorn

Dept. of Mechanical Engineering, Chulachomklao Royal Military Academy,
Nakhon-Nayok, Thailand
E-mail: thunyaseth@siit.tu.ac.th urarl.urarl9@gmail.com,
www.siit.tu.ac.th

Abstract
The mathematical model of HIV dynamical system explains the interaction between the immune system and virus. It
represents the relationship among population size of uninfected CD4+ cells, infected CD4+ cell, and virus. The aimof the treatment is to drive the amount of uninfected CD4+ cells to the desired level and the amount of both infected
CD4+ cells and virus particles approach to zero as time increased. The main objective of this study is to apply thefractional order sliding mode control (FOSMC) method to regulate HIV infection. The performance of the control
method was investigated via simulation. According to the simulation of the controlled HIV system, the state variables
approach to the desired values. Thus, it is feasible to apply the FOSMC method to define the treatment of the HIV
infection system.
Keywords: Fractional order sliding mode control, Nonlinear control, Mathematical model, HIV.

1. Introduction
HIV infection damages the body's immune system bydestroying CD4+ helper cells which are the defense cell.
The amount number of circulating CD4+ helper cells iscommonly used as an indicator of immune competence[1]. In other words, the HIV infection system describes
the dynamic between virus infection and the immune
system. To understand and analyze the dynamical HIV
infection system, many mathematical models of HIV
infection system have been developed [2, 3]. Based on
the basic HIV mathematical model, the previous studies
proposed both linear and nonlinear control method toassign the optimal treatment strategy or drug dosage [48].

Fig. 1. Schematic illustration of the basic HIV infection model(modified from [2]). The healthy CD4+ cells grow at a rateand die at a rate . The free viruses penetrate into the cells at a
rate and die at a rate . The infected CD4+ cells produce freeviruses at a rate and die at a rate .

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

157

Thunyaseth Sethaput, Arsit Boonyaprapasorn,) as in (3)corresponding to desired values ( ,[9])+ ( −)+ ( −= ( −)(3)In any previous studies, to the best of our knowledge, thefractional order sliding mode control (FOSMC) method
has not been developed for the HIV infection model. In
this study, we proposed the feasibility of the fractional
order sliding mode control method to regulate HIV
infection and provide the treatment strategy.

The differentiation of can be obtained as in (4)
̇ =
̇ +
̇ +
̇
[
(1 − )]
̇=
−
−
]+ [
+ [
−
−
−

2. Mathematical Model of HIV Infection System
The basic HIV mathematical model or the relationshipbetween virus infection and the immune system throughthe number of CD4+ cells was proposed by Wodarz and
Nowak [2]. As illustrated in Fig. 1, the model iscomposed of the interaction of three state variables: thehealthy CD4+ cells ( ), the infected CD4+ cells ( ),and the free viruses ( ).
̇ =
̇ =
̇ =

−

−

−

−

−

(1 − )

2) According to the fractional exponential approach asin (5) [9]( )− ( )=(5)
The first derivative can be determined as in (6)

(1)

−

)

∫ (

( )
)

[

Then,

( )]

( )] =
−

[ −
−

−
]+

( ) + ( )] −
]− [
−
−
)

[ −
]}/

( )+
[
(1 − )] + [−]( ) = {−
[
]− [
(−
−

(6)

(7)

−

(8)

See [9] for the proof of stability of the control system.
4. Simulation
To examine the ability of fractional order sliding mode
controller on HIV model, the simulations of the control
system are performed in MATLAB. The numericalintegration method is the Runge-Kutta method withsampling time of 0.01 days from 129 to 1000 days.

A definition of the fractional order derivative wasproposed by Caputo [9-11] as(( )+3) The control input can be solved from (7) given byequating (4) and (6) as follows

3. Design of Fractional Order Sliding ModeController

( )=

[

̇=−

Generally, to boost the immune system, a treatment of
HIV infection is to minimize the growth of new free virus
particles ( ) or to drive virus particles approach to zero.
As presented in (1), the control input ( ) is introduced astreatment effectiveness or drug dosage. The range ofcontrol input is between = 0 (non-effective treatment)
and = 1 (most effective treatment). represents the
production rate of healthy CD4+ cells. , andrepresent the death rate of healthy CD4+ cells, infected
CD4+ cells and the free viruses respectively.
represents the production rate of free virus particles
which depends on the population size of infected CD4+
cells. represents the intensity of HIV infection process.

( )=

] (4)

4.1. Simulation Example(2)

The parameter values and the initial conditions are
obtained from [5, 6, 12] as follows: = 7, d 0.007 ,0.0999 , c 0.2 , k 90.67 ,
00000042163 andthe initial condition at= 129 days isTx(129) 363, 57, 28860 . For the highest HIV
treatment effectiveness, the system requires the drive thewhere 0 < < 1 and is the order of the differentiation.
Controller design procedure for the considered HIVmodel in (1) can be summarized as follows:
1) The sliding surface is selected in term of a linear
combination of state variable ( , , ) and© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

158

Fractional Order Sliding Mode

population size of healthy CD4+ cells, infected CD4+
cells and the free viruses to reach an equilibrium level at[ , , ] =[ / , 0,0] .
The design control parameters are selected as follows:=800, =160, = 200, =0.1 and =0.01.
4.2. Simulation Results and Discussions
The simulation results show the ability of fractional ordersliding mode controller (FOSMC) and sliding mode
controller (SMC) on HIV model. As presented in Fig. 2,
after treatment, the population size of healthy CD4+ cells
increases as time increase and reach the assigned
equilibrium level. Both control methods provide almost
identical results. However, when the difference (error)
between the amount of healthy CD4+ cells and thedesired level is thoroughly observed, the FOSMC
provides a faster rate of convergence compared to SMC
as shown in Fig. 3. Fig. 4 shows the amount of infected
CD4+ cells after treatment. The amount of infected
CD4+ cells responses abruptly to effective therapy bydecreasing rapidly and approaching zero. Obviously,
FOSMC is able to drive the amount of infected CD4+
cells to desired level at a faster rate compared to SMC.
Similarly, as shown in Fig. 5, the population of free virus
particles drastically converges to equilibrium level atzero after effective therapy. The reasonable strategy oftreatment provided by FOSMC is shown in Fig. 6. To
prevent the production of new infected CD4+ cells andfree viruses, the system requires the highest treatment
effectiveness at early infection stage (loading dose). Then,the treatment effectiveness drops down and converges tomaintenance dose to ensure that all of state variables (theamount of healthy CD4+ cells, infected CD4+ cells andfree viruses) maintain at desired level.

1000
SMC: x1
FOSMC: x1 Alpha=0.5

900
800

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

x3

700
600
500

159

400
300
100

200

300

400

500

600

700

800

900

1000

Thunyaseth Sethaput, Arsit Boonyaprapasorn

2.
3.
4.
5.
6.
7.

8.

9.
10.
11.
12.

5. Conclusion
In this paper, the mathematical model of HIV infection
system with the fractional order sliding mode control
method has been proposed. The numerical simulation
results show the ability of fractional order sliding mode
controller to regulate the amount of healthy CD4+ cells,
infected CD4+ cells and free viruses. The results also
provide reasonable treatment strategy or drug dosage.
Based on the mathematical model, hence, the fractional
order sliding mode controller is feasible to control HIV
infection and direct the optimal guidelines for HIV
treatment.

virus (HIV) infection, Ann Intern Med. 111(3) (1989) 22331.
D. Wodarz and M. A. Nowak, Mathematical models of
HIV pathogenesis and treatment, BioEssays 24(12) (2002)
1178-1187.
A. S. Perelson and P. W. Nelson, Mathematical analysis of
HIV-1 dynamics in vivo, SIAM Review 41(1) (1999) 3-44.
I. K. Craig and X. Xia, Can HIV/AIDS be controlled?
Applying control engineering concepts outside traditional
fields, IEEE Control Systems 25(1) (2005) 80-83.
M. J. Mhawej, C. H. Moog, F. Biafore, and C. B. François,Control of the HIV infection and drug dosage, Biomedical
Signal Processing and Control 5(1) (2010) 45-52.
M. Shirazian and M. Farahi, Optimal control strategy for a
fully determined HIV model, Intelligent Control andAutomation, 1 (1) (2010) 15-19.
M. R. Zarrabi, M. H. Farahi, S. Effati, and A. J. Koshkouei,
Using sliding mode control in stability treatment of HIV
disease Advanced Modeling and Optimization, 14(1)(2012) 165-173.
A. Boonyaprapasorn, P. SaNgiamsunthorn, and T.
Sethaput, Synergetic control for HIV infection system of
CD4+T cells,
In Proceedings of the 2016 16th
International Conference on Control, Automation and
Systems (ICCAS) (2016) 484-488.
M. O. Efe, Fractional order sliding mode control withreaching law approach, Turk J. Electr. Eng. Comput. Sci.
18(5) (2010) 731–747.
C. A. Monje, Y. Q. Chen, B. M. Vinagre, D. Xue and V.
Feliu-Batlle, Fractional-order Systems and Controls, 1stedn. (Springer-Verlag London, 2010)
I. Petras, Fractional-Order Nonlinear Systems, 1st edn.
(Springer-Verlag Berlin Heidelberg, 2011)
X. Xia, Estimation of HIV/AIDS parameters, Automatica
39(11) (2003) 1983-1988.

References
1. H. Masur, F. P. Ognibene, R. Yarchoan, J. H. Shelhamer,
B. F. Baird, W. Travis, A. F. Suffredini, L. Deyton, J. A.
Kovacs, J.Falloon, CD4 counts as predictors ofopportunistic pneumonias in human immunodeficiency© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan160

Skill-based Job Rotation Scheduling for Occupational Noise Exposure Control
Pavinee Rerkjirattikarn, Chatdanai Kaorapapaong, Sun Olapiriyakul*

School of Manufacturing Systems and Mechanical Engineering, Thammasat University,
Pathum Thani, 12121, Thailand
pavinee.rerk@gmail.com, chatdanaik@gmail.com, suno@siit.tu.ac.th*
https://www.siit.tu.ac.th/

Abstract
This paper proposes a job rotation methodology for noise exposure control in manufacturing systems, with task skill
requirements. A situation where management needs workers to work overtime to meet demand is considered. An
integer programming is formulated, to simultaneously minimize the labor cost and to reduce the noise exposurelevels of workers to a safe level. The use of the model is illustrated by applying the model to prepare job rotation
plan over a 10-day period. The labor cost associated with different overtime policies is investigated.
Keywords: noise exposure, hearing loss, job rotation, scheduling, optimization, overtime

models for noise exposure control are developed, to copewith the assignment of workers with varying levels ofskill, experience, and limitations5-9. These previous
studies help improve the practical application of noisesafe job rotation measures. Nonetheless, crucial researchgaps remain in this field: 1) insufficient evaluation of theability of the workforce to fulfill demand requirements;
2) lack of quantitative assessment of productivity loss,
due to the inability of workers to fulfill skill requirementsof tasks; 3) insufficient understanding of the effects ofjob rotation on job satisfaction of workers. This study
aims to design a skill-based job rotation model, under a10-day planning horizon, with demand fulfillment
requirements. The objective is to simultaneously
minimize the labor cost and reduce noise exposure levelsof workers to a safe level. The effects of job rotation on
the productivity of the workforce are considered. This
paper also shows that job rotation plans, with anappropriate workforce size and overtime usage, can beused to meet demands, while maintaining the fairness ofovertime assignments.

1. Introduction
In most cases, a complete protection of workers against
excessive occupational noise exposure can be quite
challenging for complex and large manufacturingsystems. Noise mitigation strategies involving parts orequipment modification, usually require an extended
period of time for implementation. Job rotation is shown
to be an effective temporary noise exposure control
measure. As demonstrated in previous research, job
rotation can be used to control the noise exposure
duration, and the maximum daily noise exposure level ofworkers1. Job rotation can also be used to limit the
number of workers exposed to excessive noise levels2.
Recently, research on noise-safe job rotation has
contributed to an understanding of the effects of job
rotation on productivity. For systems with extended
process setup time, job rotation can cause significant
productivity loss, due to the excessive setup timerequired for workers to adjust to new working conditions3.
For systems comprising tasks with high skill
requirements, it is difficult to assign workers to the full
extent of their capacity. In noise-safe job rotation models
developed by Nanthavanij et al. (2010), productivity ismeasured in terms of competency-based scores anddefined as an optimization objective, ensuring that thehighest possible level of competency in performing tasksof the workforce is achieved4. Skill-based job rotation

2. Mathematical Model
An integer programming model is formulated to find an
optimal job rotation plan, to minimize labor cost and to
reduce the noise exposure levels of workers to a safe level.
The daily noise exposure limitations and demandrequirements of all independent workstations are© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

161

Pavinee Rerkjirattikarn, Chatdanai Kaorapapaong, Sun Olapiriyakul

formulated as constraints. In this study, the workforce
comprises skilled- and unskilled-workers. Skilledworkers perform tasks with higher productivity levels.
The assignment of unskilled workers is restricted tounskilled tasks only. In a workday, there are two regular
4-hour shifts, and a 4-hour overtime shift. Workers canbe rotated between tasks at the end of shifts. Other
important working conditions are summarized as follows.

2.1.3 Decision Variables

Yi,s,n,tZi,s,n,t
As,n,tWhen performing tasks at the same workstations formore than one shift consecutively, workers performtasks at higher levels of productivity, specified as an
extra production rate.
The number of workers assigned to tasks varies,
according to the production capacities needed tomeet demands over the planning horizon.
Labor cost can be classified into direct and indirect
costs. The direct cost includes regular- and overtimeshift wage costs, whereas the indirect cost isoverhead cost per worker.
All costs related to skilled workers are higher thanthat of unskilled-workers.
Overtime costs 1.5 times the regular shift wage.
A worker can perform only one task at a time.

Cs,n2.1.4 Mathematical Model
Minimize
∑
∑
∑

∑

∑

2∙∑

2.1.1 Indices

s
n
t

∑

,

, , ,

Subject to

2.1 Mathematical model formulationij= 1 if worker n, skill s, works at workstation i,
shift j, on day t; 0 otherwise.
= 1 if worker n, skill s, works at workstation i ofboth morning and afternoon shift; 0 otherwise.
= 1 if worker n, skill s, works at station i of both
afternoon and overtime shift; 0 otherwise.
= 1 if worker n, skill, s is scheduled to work on
day t; 0 otherwise.
= 1 if worker n, skill s, is selected under the job
rotation plan; 0 otherwise.

Xi,j,s,n,t

Number of workstations (i = 1,..., I)
Number of shifts (j=1,…, J; morning,
afternoon, and overtime shifts are denoted as
MO, AFT ,and OT, respectively)
Skill levels of workers (s = 1,…, S)
Number of workers (n = 1,..., N)
Number of workdays (t = 1,..., T)

∑

,

,

2.1.2 Parameters
Noise dose of workstation i
Demand requirement of workstation i
Production rate of workers with skill level s, atworkstation i
Ei,sExtra productivity gained by skill level s
workers for performing task at the workstation iover consecutive shifts.
LRs
Regular-shift wage of worker skill s
LOs
Overtime wage of worker skill s
OHs
Overhead cost of worker skill s
Max dose Maximum allowable noise exposure level
Max OT Maximum allowable overtime shifts
Di
Wi
Pi,s

,

∑

≤

≥

, , ,

≤

, , ,

∑
∑

≥

∑

, ,

∑

∑

,

−∑

,

∑

,

∙

, , , ,

,
,

,

≤
,

, , ,

,

, , ,

∙

, , ,

, , ,

+

, , ,

,

, , ,

+

,

∙

, , ,

,

, , ,

≤

,

∑

∙

, , ,

+∑

(2)

∀ , ,

(4)

,

, , ,

∙

, , ,

+
∀ , ,
∑

∙

,

≥

∙

(1)

∀ , , ,

(3)

∀ , ,

, , ,

∑

∑

, ,

≤

, , , ,

0

, , ,

, , ,

1)

∙

+∑ ∑
∙ ,

, ,

≤3∙

, , , ,

+ ∑

1)

∙
∑

≤1

, , , ,

∑

, , ,

∑
+∑

∑

(5)

∀

, , ,

∀, , ,

∙

,

∙

(6)

(7)

− (2 −∀, , ,(8)− (2 −∀, , ,(10)(11)∀, , ,
∀ ,
∀ ,

(9)

(12)

The objective function (1) minimizes the total labor cost.
Constraint (2) ensures that each worker can perform only
one task at a time. Constraint (3) records the working
days of workers. Constraint (4) ensures that the
maximum allowable noise exposure limit is maintained
for all workers. In a working day, constraint (5) specifies
that a worker, who works for both morning and afternoon© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

162

Skill-based Job Rotation Schedulingshifts, is eligible to work overtime. Constraint (6) ensuresthat the demand requirements of all workstations are
fulfilled. Constraints (7) and (8) record the continuity ofmorning and afternoon shift assignments. Constraints (9)and (10) record the continuity of afternoon and overtime
shifts. Constraint (11) counts the number of workers to
be included in a job rotation plan. Constraint (12) ensuresthat the number of overtime shifts, assigned to workers
over the planning horizon, does not exceed the maximum
allowable number of overtime shifts.

Table 3. Direct and indirect labor costs.
Unskilled
SkilledLabor Cost
Workers
Workers
Regular (THB per shift)
300
150
Overtime (THB per shift)
450
225
Overhead (THB per worker)
5000
3000

The use of overtime in job rotation, to meet demand, isconsidered. There are 3 overtime policies considered: 1)
no overtime allowed; 2) unlimited overtime; 3) limited
overtime (at most 5 times per worker).

3. Numerical Example

4.

A manufacturing system with 5 independentworkstations is considered. The workforce consists of 5
skilled workers and 5 unskilled workers. Demandrequirements and noise levels of workstations are shownin Table 1. The noise levels are recorded in decibel A
units (dBA), which are the relative loudness perceived byhuman ears. As recommended by OSHA, the safe limitof Daily Noise Dose (DND) is 1, which is equivalent to
90 dBA.

Result and Discussion

Opensolver version 2.8.6 is used as a problem solving
tool. Under the cost minimization objective, the laborcosts of the optimal rotation plans associated withdifferent OT policies are shown in Table 4.
Table 4. Total labor cost of each overtime policy.
Unlimited LimitedNo
overtime
overtime
overtime
Total Cost (THB)
70,70062,63567,875
No. of Workers
97
8
Average noise dose
0.549
0.783
0.661

Table 1. Demand requirements and noise levels ofworkstations.
Demand
Noise
Noise dose
Level
Workstation Requirement
per shift(Units)
(dBA)
W1
3,450
90
0.50
W2
1,265
93
0.76
W3
1,495
70
0.03
W4
2,300
85
0.25
W5
2,530
80
0.13

In this paper, the use of overtime to meet demand appears
to be cost-effective, by avoiding the cost of hiring
additional workers. The unlimited overtime plan has thelowest labor cost. However, this plan requires someworkers to work more than 9 overtime shifts during the
10-day period. This may lead to adverse effects, related
to safety, health, and job satisfaction. The limited
overtime plan, as shown in Table 5, enables a morebalanced distribution of regular and overtime shifts,
among each group of workers. When there is no
overtime, a larger number of workers is required to meetdemands, resulting in higher labor cost. However, the useof a large workforce size results in a reduced averagenoise dose level.

The regular and extra production rates of skilled andunskilled workers are shown in Table 2. The tasks atworkstation W2 and W3 are complex, and can only be
performed by skilled workers. For these workstations, theproduction rate of unskilled workers is specified as zero.
The wages of regular and overtime shifts, as well as theoverhead cost per worker, are shown in Table 3.
Table 2. Regular and extra production rate.
Regular Production
Extra Production
Rate (Units)
Rate (Percentage)
Workstation
Skilled Unskilled Skilled Unskilled
70
40%
25%
W1
100
0
W2
80
0
25%
0
35%
0
W3
60
15%
W4
60
35
20%
80
50
15%
35%
W5

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

163

164

Pavinee Rerkjirattikarn, Chatdanai Kaorapapaong, Sun Olapiriyakul

Table 5. Job rotation plan under the limited overtime policy
Daily noise dose
Workforce

SkilledworkerUnskilledworker

1
2
3
4
5
1
2
3
4
5

Day
1

Day
2

Day
3

Day
4

Day
5

Day
6

Day
7

Day
8

Day
9

Day
10

0.758
0.789
0.000
0.000
0.758
0.375
1.000
0.125
0.375
1.000

0.313
0.758
0.000
0.000
0.000
0.750
1.000
1.000
0.375
0.375

0.500
0.750
0.000
0.000
0.000
0.750
1.000
0.750
1.000
0.750

0.531
0.820
0.000
0.000
0.000
1.000
0.375
1.000
0.750
0.750

0.789
0.820
0.000
0.000
0.789
1.000
0.375
0.625
1.000
0.375

0.031
0.750
0.000
0.000
0.000
0.750
0.750
0.375
0.500
1.000

0.789
0.758
0.000
0.000
0.758
1.000
0.375
1.000
1.000
0.375

0.820
0.820
0.000
0.000
0.000
1.000
0.750
0.750
0.750
1.000

0.820
0.031
0.000
0.000
0.758
1.000
1.000
0.750
1.000
0.125

0.820
0.063
0.000
0.000
1.000
0.375
0.375
1.000
0.375
1.000

Cost (THB)*overhead cost3.

5. Conclusion
This study proposes a noise-safe job rotation
methodology that can be used in manufacturing systems
with skill and demand requirements. In the proposedmodel, job rotation is used at the cost of losing
productivity by performing the same tasks repetitively.
The validity of the proposed model is investigated, using
a numerical example where workers are assigned to workovertime to meet demands. The ability of the workforce
to meet demand, usually neglected from previous job
rotation studies, is included as one of the essential
components of our job rotation model. The developmentof noise-safe job rotation plans with overtime shifts is
also introduced. The result shows a case where the totallabor cost is minimized by employing a small-sized
workforce and assigning workers to work overtime. It isshown that, the limited overtime results in a more evenallocation of overtime shifts among each group of
workers. Our future study aims to focus more on theeffects of job rotation on job satisfaction. The fairness ofworkload distribution and injury risk perception ofworkers are important elements to be further explored in
the context of job rotation.

4.

5.

6.

7.

References
1.

2.

W. Thammaphornphilas, B. Green, B. J. Carnahan, B.
A. Norman, Applying mathematical modeling tocreate job rotation schedules for minimizing
occupational noise exposure, J. American Industrial
Hygiene Association. 64(3) (2010) 401-405.
S. Yaoyuenyong and S. Nanthavanij, Hybrid
procedure to determine optimal workforce without
noise hazard exposure, Computers & Industrial
Engineering. 51 (2006) 743-764.

8.

9.

Number of Shifts
Regular
18
17
0
0
7
20
20
19
20
19
27,300Overtime
5
5
0
0
1
5
5
5
5
5
10,575

10
10
0
0
8
10
10
10
10
10
30,000*

P. Rerkjirattikarn, S. Satitanekchai, S. Olapiriyakul,
Designing safe job rotation schedules with minimum
productivity loss, J. Supply Chain and Operations
Management. 14(2) (2016) 48-60.
S. Nanthavanij, S. Yaoyuenyong, C. Jeenanunta,
Heuristic approach to workforce scheduling withcombined safety and productivity objective, IJIE.
17(4) (2010) 319-333.
B. A. Norman, W. Thammaphornphilas, K. L. Needy,
B. Bidanda, R. C. Warner, Worker assignment incellular manufacturing considering technical andhuman skill, Int. J. Production Research. 40(6) (2002)
1479-1492.
V. Delijoo, S.M.J. Mirzapour Al-e-hashem, H.
Malekly, A. Bozorgi-Amiri, M.B. Aryanejad,
Applying multi objective modeling to create safe job
rotation schedules based upon workers' skills andidleness, Int. Conf. Computers and Industrial
Engineering, (Troyes, France, 2009), pp. 262-267.
T. Wongwien and S. Nanthavanij, Ergonomic
workforce scheduling under complex worker
limitation and task requirements: mathematical modeland approximation procedure, SJST. 34(5) (2012)
541-549.
F. Niakan, A. Baboli, T. Moyaux, V. Botta-Genoulaz,
A bi-objective model in sustainable dynamic cell
formation problem with skill-based worker
assignment, J. Manufacturing Systems. 38 (2016) 4662.
T. Wongwien and S. Nanthavanij, Priority-based
ergonomic workforce scheduling for industrial
workers performing hazardous jobs, J. Industrial and
Production Engineering. 34(1) (2017) 52-60.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

165

WorkDays

A Training Method for the Speech Controlled Environmental Control System
Based on Candidate Word Discriminations*
Taro Shibanoki†, Masaki Watanabe†, Go Nakamura‡, Takaaki Chin,‡ and Toshio Tsuji§†

Ibaraki University, ‡Hyogo Rehabilitation Center, §Hiroshima University

Abstract
This paper proposes a concept of a training system for the speech controlled environmental control system: BioRemote based on candidate word discriminations. The proposed system can provide three-types of voice signal
training: (1) volume, (2) tempo/timing and (3) candidate word which are important for accurate speech recognition
based on false recognition results. During the training, such three kinds of features are extracted from measured voice
signals and visually and auditory fed back to the user in real time. This allows the user to train speech abilities evenif false recognition results are extracted because of slurred speech.
The efficacy of the proposed system was demonstrated through training experiments for slurred speech conductedwith healthy participants. The results showed that the proposed system was capable for train the speech abilities.
Keywords: speech training, environment control system (ECS), speech recognition, candidate word, learning-type
look-up table.

The speech training system can support the recovery ofas much of a user’s speech as possible, and it has been
widely discussed as motivation and long-term
experiences for users [6], [7]. However, it can be difficult
to fully recover verbal functioning because of individual
differences and degrees of disability.
This paper proposed a training system for a speechcontrolled environmental control system based on
candidate word discrimination that can acquire the skillof fixed speech. After the training, the user’s intention
can be accurately discriminated, even if the user cannot
fully recover his/her verbal functioning.

1. Introduction
The number of disabled people in Japan continues toincrease annually and stands at 1.76 million. The
population of severely disabled people in particular wasaround 760,000, and such patients associated with aspeech disability reached 81,000 [1].
Against such a background, many speech-controlledenvironmental control systems (ECSs) have beendeveloped [2], [3]. However, it is difficult for patientswith dysarthria to use such systems, since the models
used in these systems considers standard adults’ speech.
Although some studies have investigated the use of
speaker-dependent models to support the learning ofindividual users’ voices [4], and the authors’ researchgroup also proposed the voice signal-based manipulation
method for ECS based on candidate word discrimination[5], fluctuating speech makes discrimination difficult.

2. Speech Training System Based on Candidate
Word Discrimination
Figure 1 shows the structure of the proposed speechtraining system based on candidate word discrimination.

This work was partially supported by JSPS/MEXT KAKENHI Grant Numbers 17K12723 and 26330226.
4-12-1, Nakanarusawacho, Hitachi, 316-8511 JAPAN (e-mail: taro.shibanoki.ts@vc.ibaraki.ac.jp, 14t4070a@vc.ibaraki.ac.jp)
http://bs.cis.ibaraki.ac.jp‡1070 Akebonocho, Nishi-Ku, Kobe,651-2181 JAPAN (e-mail: g_nakamura@assistech.hwc.or.jp, chin@assistech.hwc.or.jp)§
1-4-1, Kagamiyama, Higashi-Hiroshima,739-8527 JAPAN (e-mail: tsuji@bsys.hiroshima-u.ac.jp)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan
*

†

166

Taro Shibanoki, Maaski Watanabe, Go Nakamura, Takaaki Chin, Toshio Tsuji

Commandgeneration…

Featureextraction

Intention
estimation

Voice SpeechControl
Infraredsignal recognition command signals
results

The proposed system provides training for patients withdysarthria to speak the same way every time, even withslurred speech. This training can be applied to control thetraining of a voice-controlled environmental control
system [5] with slurred speech.
The proposed speech training system consists of a PC
with a feedback display, audio processor, andmicrophone. During the speech training, the display
provides the extracted features of voice signals andcurrent status of the users’ abilities to improve theirspeech skills. The details of the system are described in
the following subsections.

Domestic
appliances

Operator
VolumeTimingCandidate word

(L)
(L)

Microphone

Fig. 1. Structure of the proposed training system

M11
M21

2.1. Voice signal processing
The structure of the voice signal processing is shown in
Fig. 1. First, the amplitude and timing information ofvoice signals are extracted, and discrimination results arethen obtained using the candidate words/phenomes
Wh/Mh and the log-likelihoods T (Wh) with the candidate
word discrimination method [5].

(a) Volume training(b) Tempo/timing
training
Fig. 2. Scenes from the training(c) Candidate word
speech training

utter C words (corresponding to the control commands ofdomestic appliances) multiple times, and top V words Wcv
with their maximum log-likelihood and their phonemes
Mcv and log-likelihood T(Wcv) in H extracted words arecorresponded to each discrimination class (c = 1, 2, …,
C; v = 1, 2, …, V; V < H) in the learning stage. In thediscrimination stage, a new set of H words are extracted,
and the phenomes (D)Mu (u = 1, 2, …, U; U < H) of top U
words with their maximum log-likelihood are comparedto phoneme (L)Mci (i = 1, 2, …, Ic; Ic: number of learning
data for class c) of each discrimination class memorized
in the learning-type LUT. The coincidence , between(D)
Mu and (L)Mci is then calculated, and a class with amaximum value ofrepresenting the average of allvalues(thenumberofcoincident phonemes) is then,taken as the discrimination result. To disambiguate
discrimination, the difference between log-likelihoods
T((D)Wu) and T((L)Wci) is used to determine the result when
the same values for some classes are obtained.

2.1.1. Extraction of voice signal features
Voice signals were recorded using a microphone andsampled at 16 [kHz]. The amplitude information v(t) of
the measured voice signals with full-wave rectificationand low-pass filtering (cut-off frequency: 1 [Hz]) is
obtained based on the gains of the amplifier andmicrophone input levels.
Feature vector X used for speech recognition is thendefined as the low-frequency components of Melfrequency cepstrum coefficients (MFCCs) for each frame,
and the output probabilities P(X|W) of a feature vector X
from word W = {w1, w2, …, wK} (wk: word, K: number ofwords) are calculated using an N-gram model andphoneme-hidden Markov model (phoneme HMM)
dividing the words W into phonemes m = {m1, m2, …,
mJ} (mj: phoneme, J: number of phonemes) and matchingphoneme HMM to X. Subsequently, the top H words Wh(h = 1, 2, …, H) with the maximum log-likelihood, theirphonemes Mh, and log-likelihoods T(Wh) are extractedusing Julius [8].

2.2. Speech training for candidate word
discrimination method
For speech therapy, the treatment of articulation,
prosody and pitch range, speech rate, vocal volume, or
resonance is important [6]. The proposed speech trainingis therefore composed of three stages (see Fig. 2), as
explained in the following subsections.
Before the training, to make the learning data sets used
Microphone
in each training,
the trainee is instructed to utter C words

2.1.2. Intention estimation using candidate word
discrimination
The user’s intention is discriminated using the learningtype look-up table (LUT) [5]. The user is instructed to© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan167

A Training System for

T times, and the maximum/minimum and averagevolumes of each word are determined. The durations of
C× T words [startDct, endDct] used in the timing/tempo
control training are also determined using Julius [8].

⎧(100 − )/= (100 − )/
⎨
0
⎩

3.

In this training, the trainee practices controlling the
vocal volume level. During the training, the amplitude
information extracted from a measured voice signal ispresented on the display with the desired values(predetermined maximum/minimum and average). The
trainee controls his/her voice signal so the extracted
amplitude information follows the average value andfalls within the min./max. range. The training result isevaluated using the following equation.
100 − (

0

− (t)) ((≤ (t) ≤)
> (t), (t) >( )(0 < (

(

,

( )

< (

( )

) = 0),,

where (∙) represents Levenshtein distance.

2.2.1. Volume control training

=

( (

)≤
)

), (2)

Training Experiments

3.1. Method
To verify the efficacy of the proposed training system,
training experiments were performed with three healthy
males (subjects A–C, 22.3 ± 1.15 [year]). In theexperiments, participants were instructed to speak with
their tongue touching the maxillary central to simulateslurred speech. The parameters used in the experiments
were set as C = 7, T = 10, H = 10, V = 10, U = 5, D = 10,
= 1,
= 10,
= 500, and=50. The otherparameters, K, J, and Ic, were adjusted based on the input
voice signal durations and learning procedure results.
Ten sessions of each training stage were performed in the
training experiments, and discrimination experiments
were also performed before and after training to verify
the effectiveness of the proposed training method. In thediscrimination experiments, participants were asked toutter each word three times without feedback.

. (1)
)

The closer the amplitude information is to the average,
the higher the score, and if it exceeds the min./max.
values, the score becomes zero. The average of thevalue of speech duration is output at the end of each trial.
2.2.2. Tempo/timing control training
For accurate discrimination using the candidate word
discrimination method, it is also important to control thetiming and time from the start to the end of speech. In thistraining, the trainee controls the tempo/timing of his/herspeech. During the training, the stored voice signals ofeach word are randomly shown in the display. The trainee
is instructed to regulate his/her speech duration andtiming according to the pre-specified timing shown in
waveforms. The system evaluates the ratio of extracted
speech duration to pre-specified duration in this training.

3.2. Results and discussion
Figure 3 shows examples of experimental results. From
this figure, the training scores were stable as the numberof sessions increased, and the average discrimination
rates before and after training have relatively highaccuracy. Although participants simulated slurred speech,
they can utter each word the same way from the
beginning of the training.
Therefore, other training experiments were performed so
the participants could mimic other participants’ speech.
In the experiments performed, Sub. A trained using Subs.
B and C’s learning data sets. These experimental results
are shown in Fig. 4, and it is confirmed that Sub. A cannot
follow other participants’ speech during 10 training
sessions. An additional 10 training sessions were
therefore performed with words, the score of which was
below 40. In the latter 10 sessions, each score was
gradually increased as the number of training sessions
increased. The average discrimination rates before andafter 20 training sessions were 61.90 ± 48.56 [%] and80.95 ± 39.27 [%], respectively, and the significant

2.2.3. Candidate word speech training
Candidate word speech training is conducted so thetrainee speaks approximately the same way every time.
The candidate phonemes for each discrimination word
and extracted trainee’s phonemes are shown in thedisplay during this training. The trainee practices tocontrol his/her speech so similar candidate phonemes are
extracted. The score of this training is defined as a ratioof the number of complete/ambiguous coincidence incandidate and extracted phonemes:© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

168

Taro Shibanoki, Maaski Watanabe, Go Nakamura, Takaaki Chin, Toshio Tsujidifference before and after training was confirmed at alevel of 1 [%]. These outcomes indicated that the
proposed training system was capable of speech training.
4. Conclusion
This paper proposed a speech training system for thevoice signal controlled ECS based on candidate word
discriminations. The proposed training system provides
three types of speech training that are important to speakin the same way every time. In the training experiments,
it could be confirmed that the trainees’ speech skills were
gradually improved through training using the proposedsystem.
In future work, the authors plan to perform training
experiments for patients with dysarthria and establish an
online tuning method of training levels for each stage.

(a) Volume training(b) Tempo/Timing training(c) Candidate word speechtraining
Fig. 4. Experimental results(d) Discrimination rates

References
1. Ministry of Health, Labour and Welfare, “Ministry of
Health, Labour and Welfare Fact-Finding Investigation of
Fisically Disabled,” http://www8.cao.go.jp/shougai/
whitepaper/h25hakusho/zenbun/furoku_08.html (accessed
December 2017).
2. Glamo Inc., iRemocon Wi-Fi, http://i-remocon.com/
aboutiremoconwifi/ (accessed December 2017).
Nature Inc., Nature Remo, http://nature.global/ (accessed
December 2017).
3. M. S. Hawley, P. Enderby, P. Green, S. Cunningham, S.
Brownsell, J. Carmichael, M. Parker, A. Hatzis, P. O. Neill
and R. Palmer, A Speech-controlled Environmental
Control System for People with Severe Dysarthria,
Medical Engineering & Physics, 29 (5) (2007) 586-593.
4. T. Shibanoki, G. Nakamura, K. Shima, T. Chin and T.
Tsuji, Operation Assistance for the Bio-Remote
Environmental Control System Using a Bayesian
Network-based Prediction Model, Proceedings of 37th
Annual International Conference of the IEEE Engineeringin Medicine and Biology Society (Milan, Italy, 2015), pp.
1160-1163.
5. J. Mühlhaus, H. Frieg, K. Bilda,and U. Ritterfeld, GameBased Speech Rehabilitation for People with Parkinson’s
Disease, UAHCI 2017, Part III, LNCS10279 (M. Antona
and C. Stephanidis Eds) (2017) 76-85.
6. J. Tamplin, A Pilot Study into the Effect of Vocal
Exercises and Singing on Dysarthric Speech,
Neurorehabilitation, 23 (2008) 207-216.
7. Large vocabulary Continuous Speech Recognition Engine,
Julius, http://julius.sourceforge.jp/index.php (accessed
December 2017).

(a) Volume training(b) Tempo/Timing training(c) Candidate word speech training(d) Discrimination rates
Fig. 5. Experimental results using the other subjects’learning data sets (Sub. A).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

169

Design System of Cell Type Assembly Machinewith Dual Arms Robot by GA
Keita Honda

Department of Human Information Systems, Gifu University, Yanagido 1-1,
Gifu, Gifu 501-1194, JAPAN
Hidehiko Yamamoto

Department of Mechanical Engineering, Gifu University, Yanagido 1-1
Gifu, Gifu 501-1194, JAPANTakayoshi Yamada

Department of Mechanical Engineering, Gifu University, Yanagido 1-1
Gifu, Gifu 501-1194, JAPANE-mail: v3128027@edu.gifu-u.ac.jp, yam-h@gifu-u.ac.jp, yamat@gifu-u.ac.jp

Abstract
The purpose of this research is to develop a system named DELUGA which automatically decides by genetic algorithm (GA) where toplace a lot of assembled parts, jigs and robot hands in the workstation when designing a cell type assembly machine with a dual arms
robot. In DELUGA, since the left and right robot arms simultaneously perform assembly work, adopt the concept of waiting time toprevent interference between the left and right arms. The cycle time from assembly to completion is used as fitness, GA operation is
performed, and better arrangement is decided.
Keywords: genetic algorithm, Unit arrangement decision, Dual arm robot, Assembly machine
workbench in front of the robot and the units located aroundthe robot.

1. Introduction
In this research, we develop the system called DEciding
Locations of Units by Genetic Algorithm (DELUGA) for
determining the units arrangement of cell type assembly
machines with a dual arms robot by Genetic algorithm(GA). The unit means dual arm robot’s hands, parts trays
and jigs. Generally, the units arrangement which is oneof the components of the workstation was decided by the
production engineer. We aim to improve the design andproduction efficiency by automating the units
arrangement determination.

Fig. 1. Cell type assembly machine

2.1. Assembly process
The assembly machine of this study uses a dual arm robot,
that is, each arm chucks a unit necessary for an assembly
work, carries it to the assembly workbench and assembles
parts. In this research, we define three positions existingon the movement path of the robot arm, approach 1,

2. Cell type assembly machine
As shown in Fig. 1, the cell type assembly machine of thisresearch consists of an assembly dual arms robot which islocated in the center of the assembly machine, an assembly© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

170

Keita Honda, Hidehiko Yamamoto, Takayoshi Yamadasapproach 2 and chuck position. Each definitions is asfollows.
Chuck position: position to put or grasp the unit
Approach 1: position upward away from the chuckposition
Approach 2: position further upward than Approach 1
The arm passes from approach 2 through approach 1
and arrives at the chuck position. When the arm leaves
the unit, it performs the reverse operation (see Fig.
2).With this definition, the assembly time calculations
can be formulated.

3.1. Condition creation module
The conditions module is as follows.
Step 1: Read work data of parts assembly, placement
possible area, parts arrangement data and create their
database
Step 2: Divide placement possible area into a lattice
shape to create placement possible place
Step 3: Carry out genes coding
Step 4: Define the fitness function
Step 5: Determine various parameters3.2. GA module
The process flowchart of the GA module is shown in
Fig. 4.

Fig. 2. Movement of arm

3. DELUGA
DELUGA is the system that determines the arrangementof each unit with a high efficiency by using GA when acell type part assembly machine is designing. DELUGA
consists of two modules, the conditions module and GA
module (Fig. 3). The conditions module reads theplacement possible area of the unit and work data, divides
the placement possible area, and sets various parameters
required for GA. GA module decides the arrangement
place of the unit by GA, determines the arrangement ofparts in the parts tray which is one of the units by a fullsearch method, evaluates the total working time, andshows the coordinates of the arrangement place of each
unit and the output of the acquired arrangement image.

Fig. 4. Flowchart of GA moduleThis detailed process is described below.
Step 1: Generate an initial population
Step 2: Calculate the arrangement coordinates of each
unit
Step 3: Determine part placement inside a parts tray byfull search method
Step 4: Calculate the time taken for each work
Step 5: Calculate the cycle time of each individual inorder not to collide with both arms
Step 6: Calculate fitness
Step 7: Give selection, crossover and mutation
Step 8: Judge whether the termination condition issatisfied or not. If it is satisfied, go to Step 10
Step 9: Give Step 4 to new individuals
Step 10: The unit arrangement of individuals with thesmallest cycle time is adopted as the optimal arrangement
and output as a solution

Fig. 3. System construction of DELUGA© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

171

Design System of Cell

4. Application example of DELUGAThe example cell type assembly machine with dual arm
robot consists of the units, the left parts tray, the rightparts tray, the left parallel hand, the left three nail hand,
the right parallel hand, the right four nail hand, the jig A,
the jig B. The goal of DELUGA is to decide the betterlayout of the units. The layout image of the machine isshown in Fig. 5.

Fig. 7. Arrangement pattern of parts

Fig. 5. Units layout

4.1. System comparison
We compared DELUGA with the following two systems.
· System that determines unit arrangement only by
reinforcement learning·System that determines unit arrangement only by GA
The above two systems cannot find the parts layout inthe tray. On the contrary, DELUGA can determines thearrangement of parts in the tray.
Fig. 6 is the left and right parts tray. Each number
means each part. DELUGA decides the part arrangementin the tray by full search as follows.

Fig. 8. Arrangement of group A
Table 1. Number of patterns
Group A

6

Group B

6

Sorting A and B

2Group C

5Group D6Group E

6

Sorting C, D and E
All patterns

Left parts tray

Fig. 6. Parts trays

677,760

Table 1 shows 6 × 6 × 2 = 72 patterns in the left tray
and 5 × 6 × 6 × 6 = 1080 patterns in the right tray.
That means 72 × 1,080 = 77,760 patterns exist.

Right parts tray

4.2. Arrangement determination method
DELUGA calculated 77,760 pattern cycle times of parts
arrangement in the tray for each individual during GA
operations and the parts arrangement in the tray with thesmallest cycle time among them is adopted.

As shown in Fig. 7, we divide the parts in left and righttrays into two groups and three groups. Left tray hasGroup A and Group B, and right tray has Group C, GroupD and Group E. Group A has 6 arrangement patterns.
Group B to E also have arrangement patterns. Also, we
consider arrangements in which Group A and Group B
are rearranged, and Group C, Group D and Group E arerearranged. The number of each pattern is shown in Table3.

4.3. Details of placement possible area
Table 2 shows the detailed data for the possible
placement areas of each unit.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

172

Keita Honda, Hidehiko Yamamoto, Takayoshi Yamadas

Table 2. Placement area of each unit
Unit

x1

y1

z1

x2

y2

z2

1,811.5

1,760

740

1,918.5

1,804.5

740

2,522

1,806.5

740

2,628

1,807.5

740

1,280

1,480

629

2,320

1,480

629

1,280

1,480

629

2,320

1,480

629

2,100

1,480

629

3,140

1,480

629

2,100

1,480

629

3,140

1,480

629

Jig A

3,034

83

740

3,656

1,117

740

Jig B

3,034

83

740

3,656

1,117

740

Left
tray
Right
tray
Left
parallel
hand
Left
three
claw
hand
Right
parallel
hand
Right
four
claw
hand

Fig. 9. Layout of DELUGA

Table 3. Coordinates of dual arm robotxyz
Dual arm Robot
2,210
915
740
Fig.10. Layout of GA

Table 3 shows the coordinates of the dual arm robot.
All units of the label values are mm.
5. Simulation results
We acquired the simulation results as follows.
・ Minimum working time: Smallest working time ineach simulation
・ Average working time: average of 100 hours
simulation work time・ Plane drawing of unit arrangement: plan view ofarrangement of each unit whose working time is thesmallest
The minimum cycle time and the average cycle timeof 100 simulations are summarized in Table 4. Acquired
unit layouts are shown in Fig. 9, Fig. 10 and Fig. 11. From
the Table 4, DELUGA has the shortest cycle timecompared with other two systems.

Fig. 11. Layout of Reinforcement learning

6. Conclusions

Table. 4 Simulation results comparison (seconds)
ReinforcementGA
DELUGA
learning
Minimum time
237.41
237.25234.38Average time
237.68237.26234.89DELUGA is the system that aims the improvement ofdesign and production efficiency by automating the unit
arrangement determinations of cell type assembly
machine by GA. Compared DELUGA with a dual arm
robot system with using reinforcement learning and thesystem using only GA, we were able to obtain the unit
arrangement more efficient than the two systems. It is
ascertained that DELUGA is effective for unit
arrangement determinations in cell type assembly
machine.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan173

Spherical Mobile Robot Driven by Biorthogonal Omnidirectional Wheels
Liu Wei, Ma Shuanglong, Duan Lunqin, Yu Jiangtao
Mechanical Engineering Department, School of Mechanical, Electronic and Control Engineering,Beijing Jiaotong University, Beijing, 100044, P.R.China
E-mail: liuw@bjtu.edu.cn
www.bjtu.edu.cn

Abstract
This paper presents a new spherical mobile robot with two independent driving omnidirectional wheels locating at
the ends of the inside vertical diameter of the sphere shell, and the two wheel axes being in orthogonal position. The
omnidirectional wheels can roll along the inner surface of the spherical shell, and then drive the spherical shell rolling.
The lower omnidirectional wheel is responsible for walking drive, and the upper one for steering drive. The prototype
shows the spherical robot moving well.
Keywords: Spherical mobile robot, Omnidirectional wheel, Biorthogonal position, Eccentric gravitational moment.

eccentric torque drives the spherical robot rolling, asshown in Fig.1, M=mg·r·sin(θ). Here the influence ofrolling resistance is ignored, and only the torque M isdriving or steering the robot rolling. θ is the attitude
angle. r is the distance from the sphere center to the masscentroid m directly determines the quantity of the driving
moment M. There are lots of inventions using theeccentric moment mechanism to drive the sphericalrobots.3, ~, 8 One of the earliest and simplest sphericalrobots was invented by Halme etc.(1996).3 Its driving
mechanism is made up of a driving wheel, a steering shaft
and a balance wheel.
This paper proposes a new and simple spherical robot
mechanism, which uses the omnidirectional wheel torealize the driving principle of eccentric torque. The
omnidirectional wheel is arranged by a series of smallrollers around the circumference. The rollers’ axes areperpendicular to the wheel’s turning direction. The
mechanism is shown in Fig. 2. The effect is that the wheel
can be driven with full force, but will also slide laterally
on the wheel periphery. The lateral rolling of small rollers
does not interfere its circumferential driving function.
The new spherical robot driven by the biorthogonal
omnidirectional wheels will be analyzed in following

1. Introduction
The spheroid has simple shape, the smallest surface area,
the largest volume, and it can roll in all directions. It isan excellent rolling body. Tumbleweed is a typicalexample in the nature. The typical characteristics of thespheroid is: when the mass centroid is far away from thesphere center, the overturning spheroid has a strong
tendency to stabilize itself, otherwise, it is easy to losestability. The characteristics of spheroid inspire theenthusiasm of scholars to create the spherical robots. The
first appearance of spherical robot concept wasconsidered a new technology not only to lead theindustrial revolution in the implementation in planetmoon exploration, military purposes, but also raise thedemand in the toy entertainment industry.1
Based on the literature and patents of representative
spherical robots, the existing spherical robots withomnidirectional motion performance could be divided
into two categories according to their physical driving
principles: 1) Inertia moment driving type2 relies on therotating inertia moment to drive the omnidirectionalrolling of a spherical robot. 2) Eccentric torque drive
type relies on the motion of eccentric mass centroid to
produce the eccentric torque on the fulcrum, and the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan174

Liu Wei, Ma Shuanglong, Duan Lunqin, Yu Jiangtao

aspects, including mechanism composition, motion anddynamics analysis etc.

direction) freely, and it does not hinder the driving
function, so that the generated eccentric torque drives thespherical shell to roll backwards or forwards. Similarly,
when steering the upper omnidirectional wheel, thegenerated friction is transmitted to the frame, causing theframe to swing towards left or right. At this time, thedriving omnidirectional wheel rolls sideways withouthindering the steering drive, thereby an eccentric momentis generated to steer the sphere shell move leftwards orrightwards.
3. Mechanics Analysis

Fig. 1. Eccentric gravitational
moment drive

Before carrying out the mechanism mechanics analysis,
we assume that: 1) the rolling contact of the whole system
is pure rolling (i.e. temporarily excluding the influence ofsliding); 2) all the components are rigid bodies, including
omnidirectional wheels. In this way, a series of simplified
kinematic and dynamic analysis can be carried out.

Fig. 2. Omnidirectional
wheel

2. Mechanic Structure
The physical principle of biorthogonal omnidirectional
wheel drive is that the two omnidirectional wheels whoseaxes are perpendicular to each other are arranged on theupper and lower ends of the vertical diameter (frame) in
the spherical shell, as shown in Fig. 3. The omnidirectional wheel at lower end is for driving, and it

3.1. Kinematics analysis
The coordinate system G-XYZ is established on the
ground, and the inertial coordinate system O-xyz isestablished on the spherical center. As shown in Fig.4,
since the spherical shell always rotates with respect to theframe, the O-xyz coordinate system is fixed to the middle
point of the frame. To facilitate the description of thespherical robot position-orientation, Euler angle
coordinate system O-xeyeze is fixed to the center of theframe. In order to facilitate the kinematic analysis, theparameters are set as following: m is the mass of all theparts inside the spherical shell, r is the distance between
this mass centroid and the spherical center, ms is the massof the spherical shell, t is the wall thickness of thespherical shell, v is the velocity of the sphere center, ⍵ isthe angular velocity of the sphere rolling, ro is the radiusof omnidirectional wheel. ⍵o is the angular velocity of thedriving omnidirectional wheel, and vA is theinstantaneous velocity at point A outside the shell.

Fig. 3. The spherical robot mechanism with biorthogonal
omnidirectional wheels

To demonstrate the theoretical and mechanicalcharacteristics of this spherical robot, we decompose themotion into several phases: start-up, uniform motion,
obstacle crossing, taking off and ground landing etc.

produces friction at the inner surface of the sphericalshell under the co-function of two forces of gravity andspring pre-pressure. The omnidirectional wheel at upper
end is for steering, and it produces friction at the innersurface of the spherical shell with the function of thespring pre-pressure. The centroid within the sphere shell
deviates from the center of the sphere (distance r). Whendriving the omnidirectional wheel, the generated friction
is transmitted to the frame, which causes the frame swing
forwards or backwards. At this moment the steeringomnidirectional wheel rolls sideway (left or rightAt the phases of start-up and uniform motion, the velocity
and acceleration at point A can be analyzed. As shown in
Fig. 4, the absolute velocity of point A can be obtained
from G-XYZ:© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

175

Spherical Mobile Robot Driven

resistance couple of spherical shell rolling friction, Far is
air resistance, Mwl is the rolling friction resistance coupleof the walking omnidirectional wheel, Mwu is the rolling
friction resistance couple of the steering omnidirectional
wheel, Fs is the spring force on the frame, Fwl is thefriction force of the walking omnidirectional wheel, Fwu
is the friction force of the steering wheel, NA is thepressure to the omnidirectional wheel inside the sphericalshell, and Mwd is the driving torque of the omnidirectional
wheel.
The rolling of the spherical shell is due to the principle
moment of the mass eccentricity. Here 𝑱𝒔𝑮 𝝎 is themomentum moment of the spherical shell to the G-Y axis.

Fig. 4. Start-up and uniform motion𝑣𝐴 = 2𝑣 ∙ sin

𝜃

(1)

2

In addition, the absolute velocity at point A can beobtained by setting up a convected coordinate based on
the sphere center:𝑣𝐴 = 𝑣 + 𝑅⃑ × ⃑⃑⃑⃑⃑⃑⃑⃑
𝜔𝑓 − ⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑(𝑟𝑜 + 𝑡) × ⃑⃑⃑⃑⃑
𝜔𝑜(2)where 𝜔𝑓 is the convected angular velocity of the frame,
𝑣 + 𝑅⃑ × ⃑⃑⃑⃑
𝜔𝑓 is the instantaneous convected velocity atpoint A, ⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑(𝑟𝑜 + 𝑡) × ⃑⃑⃑⃑⃑
𝜔𝑜 is the relative velocity at point A.
When 𝜔𝑓 ＝0, the robot is in a state of uniform movement.

Fig. 5. The eccentric momentof couple

Fig. 6. The forces of theframe system

Put the Eq. (2) into Eq. (1), the relationship between the
angular velocity of the omnidirectional wheel ⍵o and thevelocity of the spherical center v can be acquired.

According to the angular momentum moment, there is:

The acceleration of point A is as follows:

where, the moment of inertia of the spherical shell is

𝑎𝐴 = 𝑎 + 𝑅⃑ × ⃑⃑⃑⃑⃑⃑⃑⃑
𝛼𝑓 + ⃑⃑⃑⃑
𝜔𝑓 × ⃑⃑⃑⃑⃑⃑⃑⃑
𝜔𝑓 𝑅 + ⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑(𝑟𝑜 + 𝑡) × ⃑⃑⃑⃑⃑
𝛼𝑤

𝑑
𝑑𝑡

𝐽𝑠𝐺 = 𝐽𝑠𝑂 + 𝑚𝑠 𝑅2 =

(3)

𝜃

5

( 3

𝑅 −(𝑅−𝑡)3) + 𝑚𝑠 𝑅 2

,

the

dt

In the spherical shell coordinate system Os-xsyszs, when
the frame is in the attitude angle 𝜃 , the moment of couple
reaches balance:(4)2

2𝑚𝑠 𝑅 5 −(𝑅−𝑡)5(6)

moment of inertia of the frame is 𝐽𝑓𝐺 = 𝐽𝑓 + 𝑚((𝑅 −𝑟 𝑐𝑜𝑠 𝜃)2 + (𝑟 𝑠𝑖𝑛 𝜃)2 ) , the resistance couple ofspherical shell rolling friction is 𝑀𝑓 = 𝛿(𝑚 + 𝑚𝑠 )𝑔 .
The Eq. (6) can determine the angular acceleration of thedspherical center as α＝ (ω).

where 𝑎 + 𝑅⃑ × ⃑⃑⃑⃑
𝛼𝑓 is the instantaneous convected
acceleration of the frame at point A, 𝛼𝑓 is the relative
acceleration of the frame; ⃑⃑⃑⃑
𝜔𝑓 × ⃑⃑⃑⃑⃑⃑⃑⃑
𝜔𝑓 𝑅 is the centripetal⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑⃑
acceleration of the frame; (𝑟𝛼𝑤 is the relative𝑜 + 𝑡) × ⃑⃑⃑⃑⃑
acceleration at point A, 𝛼𝑤 is the angular acceleration of
the omnidirectional wheel, and the Coriolis accelerationin Eq. (3) is 0. In addition, the acceleration at point A canbe obtained from G-XYZ:
𝑎𝐴 = ⃑⃑⃑⃑⃑⃑⃑
𝑎𝑡 + ⃑⃑⃑⃑𝑎𝑛(𝐽𝑠𝐺 𝜔 + 𝐽𝑓𝐺 𝜔)＝ 𝑚𝑔 ∙ 𝑟 ∙ 𝑠𝑖𝑛𝜃 − 𝑀𝑓 − 𝐹𝑎𝑟 ∙ 𝑅

𝑀𝑤𝑑 − 𝑀𝑤𝑢 − 𝑀𝑤𝑙 − 𝑚𝑔 ∙ 𝑟 ∙ 𝑠𝑖𝑛 𝜃 = 0

𝜃

(7)Where 𝑎𝑡 ＝2𝑎 𝑠𝑖𝑛 , 𝑎𝑛 ＝2𝑅𝜔 𝑠𝑖𝑛 .

The equation can deduce the position angle 𝜃 .

The relationship between the angular acceleration at thespherical center and its acceleration is:

When considering the effect of the rolling friction
resistance couple, the start-up phase can be divided into
the following sub-phases: 1) When the driving torque ofthe omnidirectional wheel 𝑀𝑤𝑑 > 𝑀𝑤𝑢 + 𝑀𝑤𝑙 , theomnidirectional wheel starts rolling inside the shell, i.e.
the attitude angle 𝜃 > 0; 2) When the eccentric moment
accumulates to 𝑚𝑔 ∙ 𝑟 ∙ 𝑠𝑖𝑛𝜃 > 𝑀𝑓 , the robot shell starts
rolling on the ground.

2𝛼𝑂 = 𝑎/𝑅

2(5)

3.2. Kinetic analysis
In order to facilitate kinetic analysis, the parameters are
set as follows, as shown in Fig. 5 and Fig. 6: Mf is the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan176

Liu Wei, Ma Shuanglong, Duan Lunqin, Yu Jiangtao

4. Prototype and Experiment

not only can obtain the necessary non-sliding friction
required by the omnidirectional wheel, but also reduces
the rolling friction resistance couple of the omnidirectional wheels. 2) to explore kinematic and dynamic
characteristics of this spherical robot, such as the
behaviors of obstacle crossing, and taking off, etc.

First, the mechanism of the omnidirectional wheel
spherical robot was simulated by using the 3D solid
modeling and multi-body dynamics software. Second, a
prototype was made after simulating and analyzing therobot's detailed properties and its feasibility, as shown in
Fig. 7. The experimental prototype uses two motors todrive the upper and lower omnidirectional wheels
respectively. In order to lower the height of centroid, thecontrol system, the power supply and the drive motor of
the upper omnidirectional wheel are all arranged away

Acknowledgements
This research was funded by China National HighTechnology Research and Development Plan (863 Plan)
2013AA041302.
References
1. Rhodri H A, Julian F v., Rolling in nature and robotics: a
Review, Journal of Bionic Engineering, 3(4) (2006) 195208．
2. Li Tuanjie, Zhu Chao, Design and Analysis of a sphericalomnidirectional rolling robot with a stable platform,
Journal of Xidian University, 33(1) (2006) 53-56.
3. Halme A, Schonberg T, Wang Y., Motion control of aspherical mobile robot, 4th IEEE International Workshop
on Advanced Motion Control, (Mie, Japan, Jan,1996) 259–264.
4. Bhattacharya S, Agrawa S K, Spherical rolling robot: adesign and motion planning studies, Proceedings of theIEEE International conference on Robotics andAutomation, (San Francisco, CA, April, 2000) pp.835-839
5. Joshi V A, Banavar R N, Hippalgaonkar R, Design andanalysis of a spherical mobile robot, Mechanism andMachine Theory, 45(2) (2010) 130-136.
6. Zhao Kailiang, Sun Hanxu, Jia Qingxuan, Liu Daliang, Shi
Chengkun,Analysis on Acceleration Characteristics ofSpherical Robot Based on ADAMS, Journal of Machine
Design, 27(6) (2009) 24-27.
7. Javadi A H A, Mojabi P., Introducing august: A novel
strategy for an omnidirectional spherical rolling robot,
IEEE International Conference on Robotics andAutomation, Washington, (DC, USA, 2002) 3527–3533.
8. Sun Hanxu, Zhao Wei, Zhang Yanheng, Mechanicalanalysis about a new kind of variable structure sphericalmobile robot, Journal of Mechanical Engineering, 49(19)(2013) 40-46.

Fig. 7. Physical prototype

from and below the center of the sphere. In order toensure that the upper and lower omnidirectional wheels
are always on the vertical diameter, an important task static balance needs to be completed before the framecomponents are installed into the spherical shell. The
walking and steering of the prototype have achieved the
expected effect.
5. Conclusion
The spherical robot's driving scheme makes use of thecharacteristics of the omnidirectional wheel to simplify
the driving mechanic structure, and it is better to increase
the distance from the mass centroid to sphere center so as
to generate a maximized eccentric driving torque. Thisorthogonal arrangement makes the walking omnidirectional wheel and the steering one independent atforce directions, so its motion directions are independent
too. This can facilitate the motion towards any direction
successfully, and is very conducive to the control system.
However, as the contact area of the sphere with theground is small, the stability of the sphere positionorientation is poor. It is necessary to use control method
to improve the motion stability. In addition, further
research is needed in the following aspects: 1) todetermine the proper pre pressure of the spring so that it© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

177

Coordinated behavior with a Pepper Humanoid robot to estimate the distance of other robot
using Inverse Perspective Mapping

M. Hassan Tanveer, Antonio Sgorbissa, Carmine T. Recchiuto

Department of Bio-Robotics and Intelligent systems (DIBRIS)
University of Genova 16145 Genova, Italy
Email: muhammadhassan.tanveer@dibris.unige.it, antonio.sgorbissa@unige.it, carmine.recchiuto@dibris.unige.it

Abstract
This article discusses a strategy for real time distance determination applied to a humanoid robot (Pepper, Softbank
Robotics) that is requested to collaborate with other robots. However, using the monocular camera embedded on the
robot head, it is difficult to determine distance to objects in front of the robot with any degree of certainty. The reason
is non linear relationship between the height allocation of camera and its distance to an object in a camera frame. So,
the proposed method is based on the extraction of accurate bird’s-eye view images, by applying Surface Subtraction
and Inverse Perspective Mapping (IPM) to the images captured with the Pepper head camera. The algorithm isimplemented in the Python language using the OpenCV libraries.
Keywords: Pepper Humanoid Robot, Monocular Camera, Inverse Perspective Mapping, OpenCV.

or LIDAR. They are active systems, potentially leading
to higher power requirements and interference issues, incomparison with optical camera sensors Ref. 7. It works
on relatively simple concept: signals transmitted from anantenna reflect back to the transmitter from the target
object. The distance can be calculated based on the lengthof time taken for the signal to travel from the target.
A robot should detect the relative position of anotherrobot, while the other robot is in the floor. A large amountof research has been done for distance determination toan object using forward facing cameras Ref. 8, SONAR,LASER Scanner that could not be specifically used if theheight of other robot is too low. A multi-camera setup as
employed by Ref. 9, provides depth information by
establishing feature correspondence and performing
triangulation. However, it also carries severe processing
and configuration overheads, which are undesirable inthe cost and power sensitive robots.
In this article, two robots has been adopted Pepper
Humanoid robot and iRobot Roomba. The Pepper robot
has capabilities for vision which include facial and shape

1. Introduction
One of the major aspects of research in robotic is toimplement social robots algorithms that help robots toanticipate the desired task which results the ease forhumans. During 20th century, many steps forward havebeen taken for the realization of fully autonomous
humanoid Ref. 1. In robots, the problem of position
estimation and distance determination has been one of theconcerned domain due to destruction in robots collisionRef. 2. Collision detection systems can provide robots
with a warning prior to a potential collision allowing
them to take preventative action Ref. 3. They can also be
incorporated into safety systems, as seen in Ref. 4 which
immediately stops the robot processes if an imminent
object is detected.
Distance determination in robots could allow us toprovide feedback and alerts to the user Ref. 5. This canpromote the commands to take preventative action, or
prepare the safety systems for an imminent collision [6].
Most of the research work for distance determination has
been conducted by using active systems such as RADAR© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan178

M. Hassan Tanveer, Antonio Sgorbissa, Carmine T. Recchiuto.

recognition (in-built cameras). Secondly, the iRobot
Roomba is a unicycle mobile robot that performs a wide
range of task from autonomous cleaning and surveillance,
to transportation and assistance to the elderly. By
combining both of these robots, a solution that consistsin the composition of a IPM view (bird’s eye view) of
Pepper head camera has been proposed. This view willserve as a virtual map for the iRobot Roomba in the
Pepper Frame of view (FOV) to help the user toadequately specify the commands to be sent to the other
Roomba under its control, as in Fig. 1 that is referred to
Pepper Frame of view. The proposed system consists ofa single forward facing Pepper camera, capturing video
images at 30 frames per second (fps).
Section II of this paper discusses distance theMethodology for distance determination that includes
Inverse Perspective Mapping which allows the image
perspective to be linearized, the surface subtraction
requirement for object detection and the calibration
issues. The technological platform with discussion of
results are described in section III. Future work
concludes the paper in section IV.

Fig. 1. Pepper FOV.

Using IPM, the aim is to map pixel points (u,v) to world
coordinate points (Xw,Yw,Zw), as in Eq. 1. The
requirement of a rotation about q, a translation along thecameras optical axis, and a scaling by the camera
parameter matrix Ref. 13, can be expressed as:=

2. Methodology

1
0
0 cos
0 sin
0
0
1
0
=
0
0

2.1. IPM Model
A transformation of coordinates system can be done
mathematically by using Inverse Perspective Mapping(IPM). It converts the coordinates of image from one
perspective to another Ref. 10. The final homogeneously
distribution of a remapped two dimensional acquired
image (input data image) form in pixels are firsttransformed by using IPM technique in order to remove
the perspective effect.
Obviously, the application of the IPM transform requires
the knowledge of the specific acquisition conditions(camera position, orientation, optics, etc.) Ref. 11 someassumption on the scene represented in the image (heredefined as a priori knowledge). Thus, the IPM transform
can be of use in structured environments, where, forexample, the camera is mounted in a fixed position or insituations where the calibration of the system and thesurrounding environment can be sensed via other kind of
sensor Refs. 12.
In this case we use IPM to attain a top-down view from
Pepper robot camera. The transformation in this manner
removes the non linearity of distances between world
frame and object.

(1)( , , , 1)

( , , 1) =

0
1
0
0

0
− sin
cos
0

0
0
0
0
1 −
0
1

0
0
0
1

(2)

(3)

The intrinsic parameters of the camera are represented by
the following matrix:
=

∗
0
0

∗
0

1

0
0
0

(4)

where h is the height of camera, f correspond to the focal
length measured in horizontal and vertical pixel units ku
and kv respectively. The positions u0;v0 are the principal
points where the optical axis fixes the image plane. The
camera performs a perspective projection of a 3D world
point onto an image point located on a retinal plane. By
using homogenous coordinate, the projective relation
between 3D world point and its image points can be
rewritten as in [5], It relates world points to image points
according to the camera location with respect to thereference frame, represented by a camera matrix K Eq. 4,
rotation matrix R Eq. 2 and a translation vector T Eq. 3.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan179

Coordinated behavior with a

( )=(
1

(5)

)

1

The 3 * 4 matrix M(i,j) is the transformation matrix in Eq.
5. When the structure under observation is a plane, asimpler formulation becomes available. Since the world
coordinates system can be set anywhere, it can beconveniently positioned on the plane, such that this latterhas zero Z coordinate. This choice reduces the projection
matrix to:
( )=(
1

)

(

1

)

(6)

The updated transformation that maps a world plane toan image points as per the given source image can be
rewritten as:( , )=(,

Fig. 2. Image coordinate system in relation to world
coordinate system

(7)
+
+

+
+

)

The sampling rate in X direction at this point is simply
the number of source image pixels between (X,Y,1)T andthe point (X+(dX,Y,1)T) divided by the distance between
these two points.
Fig. 3 is the frame captured by static camera and its IPM
view. The picture transformation can be seen in Fig. 4, as
the src coordinates in image changes as per perspective

Fig. 3. Static camera Image and its IPM view

2.2. Detection Phase
Normally, the available conventional techniques which
requires the prior knowledge of the object for detecting itwith a camera Ref. 14. In spite of it, a more efficientmethod of doing this is used by adopting a surface
subtraction (SS) algorithm. The wheeled robot Roomba
to be detected has a small green object on top, and thus
the surface subtraction algorithm starts from thedetection of green object. Considering the scene in Figs.
8 and 11, the object we must detect is deemed to be thenearest part of the image that is not surface directly in
From the above steps, the adaptation becomes easy indifferent light and surface conditions. By usingmorphology tool in OpenCV, we get the green object
detected in different light conditions as the range of BGR
channels can be manually fed. So, in conditions of light
impact the result will remains same in detection of green
color because of the defined constraints of upper andlower limit of BGR morphological tool. It do helps to

Fig. 4. Static camera IPM view with src and dst points

view. The transformation depends upon the Eq. 6 andcan be warped to its dst coordinates as mentioned in Eq.
7 front of the Pepper. Therefore the easiest way to fromthe image, leaving
only the required object, as mentioned in steps:
Green channel is separated to its constituent in the
BGR image.
Taking Green channel into account, the algorithm
calculates the average pixel values in x and y axesin the captured Pepper image.
The algorithm loops though every rows andcolumns in the image, creating a new binary image
where 0 signifies a plane pixel, and a 1 is a green
color pixel.
The algorithm repeats steps 2 and 3.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan180

M. Hassan Tanveer, Antonio Sgorbissa, Carmine T. Recchiuto.

detect the potential object by detecting its color ofdifferent ranges as well, so there is no distortions indifferent light conditions.
Before we can calculate the distance of the Roomba fromthe Pepper, we need to calculate the height of the object
in pixels from the bottom of the image as shown in Fig.
8. Beginning at the bottom of the binary image in each
RGB color space, we calculate the average pixel intensity
for each row of pixels in the image. From this we canascertain how high in the image the nearest object lies.
Only the area directly in front of the Pepper is analyzed.
This was to minimize the rate of false positive object
detections caused by other objects in Pepper captured
frame.
With the previous steps, the pixel points of potential
object measured in image is known but to calculate it as
per world frame requires a calibration tool in terms toconvert the available pixel value in understandable
measuring unit (i.e.: meters). There are several method of
calibration as in Ref. 15. The methods like chessboard
calibration and marking points calibration discussed inRef. 16 can be used as well but in this case, the proposedsystem would be implemented with a Pepper camera in a
fixed position. The placement of a green color box withknown distance (i.e.: known x and y axes distance) in the
FOV of the camera, along a flat surface, would allow for
the measurement of the number of pixels equivalent todistance. Using this value, the number of pixels in theimage could be converted to a real world value in meters.

Fig. 5. Pepper RedBall API response at static points.

3. Results and Discussion
The algorithm presented here was first investigated witha static camera and then implemented to Pepper in realtime. For this purpose, the Python language,
complemented by the OpenCV library of functions waschosen for implementation that allows the rapid
development of image processing algorithms without theneed to develop complex algorithms by using its open
source library of functions Ref. 17.
With Pepper robot, a database of video samples wascaptured at 30 frames per second (fps), at a resolution of
320 X 240 pixels. Considering that the change in distanceof potential objects is relatively slow compared to the full
frame rate of the system, a slower frame rate wasconsidered for the purposes of calculating distance andcomputing position. Using a sampling rate of every 10
frames produced smooth and reliable results. A rate of 10
frames per second was chosen as it provides a good trade-Fig. 6. Pepper Red ball API response while tracking a circularresponse.

off between computation time and number ofcalculations per second.
A comparative analysis has been shown in this section.
The first experiment is being conducted with Pepper Red
Ball API. The Red ball API works by considering the sizein pixels of the red ball, and comparing it to the known
size of the red ball to be detected. Obviously this works
well only if the red ball is directly in front of the camera.
A Red ball is being placed at a static point in the FOV of© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

181

Coordinated behavior with a

Pepper Head camera. First, it is placed at 0.7m on X-axis
and 0.22m on Y-axis, then the data have been recordedfor 300 samples. The same acquisition has been
performed in other three different points, as shown in Fig.
5.
It can be seen, how due to a continuous transient a second
test has been conducted: in this case the red ball has beenplaced on board the wheeled robot, while it is performinga circular path as done in Ref. 18, shown in Fig. 6. The
average error in this case is av|e| = 0.43m. It is clear withthe analysis that the API are not enough to determine thedistance of objects in Pepper camera view. However, inorder to determine the distance between the Pepper andRoomba in its FOV (as shown in Fig. 1), the first step isto determine where the Roomba lies in the plane withrespect to Pepper IPM view. So, Fig. 7 shows the Pepper
camera view and IPM view respectively. It can be seenfrom the green arrow in rectified bird’s-view image, Fig.
7, that the relationship between the potential object andits distance from the camera is linear in nature. However,the change in the length of the green arrow willproportionally reflect this difference in distance. In orderto determine the position of potential object in bird eye
view, the calibration with different distance on pixels is
considered and sum up in a formula as in Eq. 8. It goessomething like: we first calibrate the world points in Bird
eye view frame with a known distance of object. Then theobject is being placed at some distance from Pepper
camera and by measuring at the same time the pixel (inbird’s-view) corresponding to the object detection. The
apparent pixels is measured corresponding to thedetected object.
=

+

+

+

Fig. 7: Distant object as shown by Pepper Top-Down view,
Top Down view, distance in this image may be measureable

Fig. 8: IPM view pixel values in different distances.

The same test have been performed, in this case a green
box has been placed on top of the wheeled robot Roomba
as a potential object. The green box has the same size as
Red ball used in Red Ball API experiment. As per
discussed in methodology section, the use of a Pepper
Head camera does not directly provide depth information
in a scene so surface subtraction and IPM for getting abird’s-view image has been applied Fig.7.

(8)where i is frame point, o is the offset, p is the pixel. The
calibration result captured in Pepper FOV at different
distances is shown in Fig. 8.The static point comparison
between Pepper Red ball API and IPM view is shown in
Fig. 9.
Fig. 10, shows the Pepper camera and IPM view to clarify
the robot is moving in a circular path, the Roomba
position is being calculated depending upon the potential
objected tracked. So, it implies that the smaller the valueof pixel the farther the distance. The odometry of
Roomba while following a circular path and Pepper IPM
view response is presented in Fig. 10 to determine thedistance and position, with an average error

Fig. 9: comparison of IPM view and Redball API at static
point© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan182

M. Hassan Tanveer, Antonio Sgorbissa, Carmine T. Recchiuto.

av|e|=0:086m. It validates the IPM method (for static
camera and moving objects on the floor). Successfulcomparison between Red Ball and IPM view is shownand obtained from a video samples of Pepper camera inreal time operation. The experiments performed confirmsthat the implementation of an IPM approach with surface
subtraction gives better results in comparison with theusage of standard Red Ball API.
4.

Conclusion

This paper presented a IPM approach with surface
subtraction, implemented with a humanoid robot (Pepper,
Softbank Robotics) in charge of detecting the position ofa wheeled robot (Roomba, iRobot) by using the headmounted camera. The proposed approach has been tested
in different configurations, by keeping the wheeled robot
in static position, or by letting it move in circular path.
The comparison of the method with the API for distancedetection embedded in the humanoid robot shows thatusing IPM reduces the average error in distanceestimation. The limitation of the current approach is that
the IPM transform can be of use in structured
environments, where, for example, the camera ismounted in a fixed position. Future works will be aimed
at overcoming the current limitation, by including
Yaw/Pitch parameters with an adaptive IPM model.

Fig. 10. Response of IPM view in a circular path.
8. S. Tuohy, D. O’cualain, E. Jones, and M. Glavin,“Distance determination for an automobile environment
using inverse perspective mapping in opencv,” 2010.
9. L. Ma, X. Yang, and D. Tao, “Person re-identification overcamera networks using multi-task distance metriclearning,” IEEE Transactions on Image Processing, vol.
23, no. 8, pp. 3656–3670, 2014.
10. J. Jeong and A. Kim, “Adaptive inverse perspective
mapping for lane map generation with slam,” inUbiquitous Robots and Ambient Intelligence (URAI), 2016
13th International Conference on. IEEE, 2016, pp. 38–41.
11. R. Laganiere, “Compositing a bird’s eye view mosaic,”
image, vol. 10, p. 3, 2000.
12. J. Civera, A. J. Davison, and J. M. Montiel, “Inverse depth
parametrization for monocular slam,” IEEE transactions
on robotics, vol. 24, no. 5, pp. 932–945, 2008.
13. M. Oliveira, V. Santos, and A. D. Sappa, “Multimodal
inverse perspective mapping,” Information Fusion, vol. 24,
pp. 108–121, 2015.
14. X. Wang, “Intelligent multi-camera video surveillance: A
review,” Pattern recognition letters, vol. 34, no. 1, pp. 3–
19, 2013.
15. D. P. Boyle, H. V. Gupta, and S. Sorooshian, “Toward
improved calibration of hydrologic models: Combining
the strengths of manual and automatic methods,” Water
Resources Research, vol. 36, no. 12, pp. 3663–3674, 2000.
16. P. Miraldo and H. Araujo, “Calibration of smooth camera
models,” IEEE transactions on pattern analysis andmachine intelligence, vol. 35, no. 9, pp. 2091–2103, 2013.
17. A. Morro, A. Sgorbissa, and R. Zaccaria, “Path following
for unicycle robots with an arbitrary path curvature,” IEEE
Transactions on Robotics, vol. 27, no. 5, pp. 1016–1023,
2011.

References
1. B. Siciliano and O. Khatib, Springer handbook of robotics.
Springer, 2016.
2. S. Lemaignan, M. Warnier, E. A. Sisbot, A. Clodic, and R.
Alami, “Artificial cognition for social human–robot
interaction: An implementation,” Artificial Intelligence,
vol. 247, pp. 45–69, 2017.
3. G. Stein, E. Dagan, O. Mano, and A. Shashua, “Collision
warning system,” May 23 2017, uS Patent 9,656,607.
4. D. E. Maurino, J. Reason, N. Johnston, and R. B. Lee,
Beyond aviation human factors: Safety in high technology
systems. Routledge, 2017.
5. M. Rubenstein, A. Cornejo, and R. Nagpal,“Programmable selfassembly in a thousand-robot swarm,”
Science, vol. 345, no. 6198, pp. 795–799, 2014.
6. A. Mukhtar, L. Xia, and T. B. Tang, “Vehicle detection
techniques for collision avoidance systems: A review,”
IEEE Transactions on Intelligent Transportation Systems,
vol. 16, no. 5, pp. 2318–2338, 2015.
7. S. A. Hiremath, G. W. Van Der Heijden, F. K. Van Evert,
A. Stein, and C. J. Ter Braak, “Laser range finder model
for autonomous navigation of a robot in a maize field using
a particle filter,” Computers and Electronics inAgriculture, vol. 100, pp. 41–50, 2014.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan183

Syed Faiz Ahmed, Athat Ali, M. Kamran Joyo, Khusairy Abd Kader, Izhar A. Bakar, Hazry Desa, Sheroz Khan

Control Techniques of Quadrotor UAVs: a Concise Analysis
1Syed Faiz Ahmed, 1Athat Ali, 1M. Kamran Joyo, 1Khusairy Abd Kader, 1Izhar A. Bakar, 2Hazry Desa, 3Sheroz Khan

1

Universiti Kuala Lumpur, British Malaysian Institute, Malaysia
2
Universiti Malaysia Perlis, Malaysia
3International Islamic University Malaysia, Malaysia

E-mail: syedfaiz@unikl.edu.my, athar.ali@s.unikl.edu.my, m.kamran@s.unikl.edu.my, hazry@unmap.edu.my

Abstract
UAVs have become popular in the recent years. Quadrotors, the four wing UAV has drawn
prominent attention amongst the researchers. The simple nonlinear structure design requires arobust control to ensure the stability during the flight. The noisy and environment with
unexpected disturbances may cause issues such as drift in position. In order to tackle the severe
conditions a robust control is required. This paper provides an overview of the available
quadrotor models and the control techniques with their application in quadrotor control issues.
Keywords: Quadrotor UAVs; Control algorithm; flight stability.

These devices are useful in several kinds of hazardous
environments where reach of humans is not possible.*
The above-mentioned applications influence therequirement of such systems which have ability tooperate in severe conditions and perform complex anddifficult missions. In the unmanned aviation industry,helicopters and rotorcrafts have proved to be one of thefinest solutions due to their capabilities of hovering,accurate positioning, vertically takeoff and landing. 4
There exist numerous types of UAVs shown in Fig.1,
one of them is a quadrotor UAV consisting of four rotors
mounted at the four ends of the cross frame 5-6 which isprominent among the researchers. The cause of its
prominence is due to its light weight, simple mechanicalstructure that makes its assembly trouble-free.
In this paper, various quadrotor models are reviewed,
and a comparative analysis of these quadrotors ispresented in TABLE 1. The comparison is based on theparameters of control technique used for the position,
altitude or attitude stabilization and DOF. Furthermore,
paper provides the survey of different available control
algorithms. The perception of this research work is toanalyze the best available configuration and control
technique of the quadrotor UAV.

1. Introduction
Due to rapid advancement in technology and demand theunmanned aerial vehicles have become the focus of study
among the researcher. 1-2 UAVs are remote controlled
flight vehicles that works without any direct humanintervention.

Fig.1. Classification of UAVs.3

Several applications such as search and rescue missions,wild life surveillance and monitoring, inspection of
power plants and nuclear reactors, agricultural mapping
and photographing, marine operations, battle damage
assessment and law enforcement use such type of devices.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

184

185

Syed Faiz Ahmed, Athat Ali, M. Kamran Joyo, Khusairy Abd Kader, Izhar A. Bakar, Hazry Desa, Sheroz Khan

Table 1: Comparative Analysis of The Developed Quadrotor UAVs
Sr.No

DeveloperYear

Description

DOF
Control Technique1

Sujit Rajappa.8

2016

2

M. ZakiMustapa9

2015

3

Emran 1020144

M. Hassan
Tanveer.11

2014

5

M. Hassan
Tanveer.12

2014

6

Vijaykumar13

2014

7

Wei Wang.14

2013

8

2013

9

Shayegan
Omidshafiei15
Ankit Patel16

10

Elruby.17

2012

11

KonradRudin18

2011

12

Z. Fang19

2011

13

Alpen202009

14

Holger Voos21

2009

15

Patrick
Adigbli22

2007

2012

Super Twisting controller was developed and implemented
for the UAV, using the adaptive laws for tracking theposition of the UAV in the presence of uncertainties andexternal disturbances.
Model was derived using Newton-Euler equations for
attitude control of the UAV. PID controller was developed
for altitude control of quadrotor. The model is implemented
on a real-time system in MATLAB Simulink.
A model reference adaptive control method was used for anunder-actuated quadrotor assumed to be suffering fromparametric uncertainty and noise signals. A robust controller
was developed to overcome the effect of noise on thequadrotor.
A model was derived using PID control for stabilizing theattitude of quadrotor under external disturbances. The
system was derived using Newton Euler equations.
A system was designed with NMPC-PID controller for theattitude and altitude stabilization of UAV. The problems ofnoise and external disturbances were solved. The robustnessof the developed system was proved by the simulation.
Quadrotor UAV model was developed, simulated andimplemented using Fuzzy logic control. Quadrotor wasdeveloped using '+' configuration. Six FLC's were designed
to control all the six states of the quadrotor.
This model was developed for the attitude and altitudecontrol of the UAV quadrotor. For attitude control Model
Reference Sliding Mode control technique was used and for
altitude control Linear Quadratic Gaussian control technique
was applied. The MRSMC technique provides robustness tothe model.
A linear quadrotor model was developed, providing
waypoint- based trajectory control.
A model for fully actuated and stable quadrotor wasdeveloped. Control techniques are used to obtain thedesirable altitude and yaw angle.
CAD model was developed using Solid Edge ST to
determine the parameters for simulation. PID controller wasused to provide attitude control and Euler theory is used toobtain the desired position of the UAV.
The UAV model was developed with the attitude stability
and a robust control design that can overcome theuncertainties and external disturbances. The controller was
designed using Lyapunov Methodology.
Adaptive controller was designed for the UAV forcompensating the system uncertainties. The system wasdivided into two loops: Outer loop for horizontal position
controller and the inner loop for the attitude controller.
A non-linear dynamic model of quadrotor was developedwith attitude control. The model was suitable for higher
velocities.
The developed UAV model resolves the problems ofaccurate stabilization and attitude control using feedback
linearization method. Eight parameters were defined for themodel. The model comprises of two loops: The underlying
loop was meant for attitude control to stabilize the quadrotor
and the outer loop was for velocity control to determine thedesired trajectory.
A simple UAV model was developed with the attitude andposition controllers.

6

Adaptive Super Twisting
Controller

6

PID control

6Composite adaptive controller(Sliding mode and Adaptive
control)

3

PID control

4

NMPC-PID control

6Intelligent Fuzzy Control

4

1.
2.

Model Reference Sliding
Mode Control
Linear Quadratic
Gaussian controller

6

Reinforcement Learning

2

Sliding Mode control and PD
control

6

PID control

3

Non-linear hierarchical controller

6

Adaptive integral backstepping4

PD control

3

Feedback Linearization

5

1. Sliding Mode
2. Backstepping© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan186

Control Techniques of Quadrotor

Table 2. (Continued)
16

B. Erginer23

2007

17

Bouabdallah et
al.24

2006

18

Bouabdallah25

2006

19

Bouabdallah26

2005

20

Bouabdallah 27

2004

The UAV model with gyroscopic effects was derived. PD
control technique was used to control the altitude, pitch androll angles and independently the yaw angle of thequadrotor.
The model was to rectify the control parameters used in thecontrol technique and test obstacle avoidance techniques.
The attitude and altitude controllers for this UAV weredeveloped.
Fully autonomous UAV was developed using VTOL
methodology and consists of the avionics and energy devices
necessary for the flight. A dynamic simulation model wasderived.
The model was developed using the latest sensors, actuatorsand energy storage devices. Simulations were performed inopen and closed loops for stabilization and formation andvalidation of control laws.
Development of this UAV was initiated at the Autonomous
System Laboratory and is focused on the autonomous VTOLin indoor environment. It's equipped with the sensors that
could measure position, altitude and orientation.

4

PD control

6

1. PID & LQR
2. Backstepping

6

PD control

6

1. Backstepping
2. Sliding mode

6

PID and LQ

Table 2: Effects of PID Parameters

2. Survey of Former Designed Quadrotor
UAV(s)
Since last few decades, UAVs are gaining attention andare continuously developing at a phenomenal rate. At
present, there are more than 1000 UAVs developed bymore than 50 countries as a major contribution for the
military and civil assistance.17 Table 1 presents thecomparative analysis of the developed quadrotor UAVsso far, based on the control algorithm used in developing
the model and the DOF.
Different control techniques have been applied to thequadrotor. The available control techniques are mainly
divided into linear or non-linear in nature. Here in thecomparative analysis of few control techniques that havebeen implemented in recent years.

Controller

Rise
Time

OvershootSettling
Time

Kp

Decrease

Increase

Ki

Decrease

Increase

Small
ChangeIncrease

Steadystate
error
Decrease
Eliminate

The transfer function of PID controller is shown inequation 1:
Ki
s

KP

Kd s

(1)
MeasuredDesiredDisturbance

6
5

Amplitude

4

3. Discussion and Analysis

3
2
1

PID is widely utilized to control the industrial
applications.28,5 PID control technique can also be
applied in the quadrotors to address the issues related tovertical takeoff and landing, position and angularorientation. With the appropriate values of control
variables minimum overshoot and steady error
minimized to zero were achieved.29 TABLE 2 describes
the effects of PID parameters on the system.

00Offset=0 (secs)

1

2

3

4

5
Time (secs)

6

7

8

9

10

Fig.2. PID response along X-axis under air turbulence andsensor noisy conditions.35

A research was carried out on quad-rotor position
control. The analysis of the model was done on MATLAB.
PID control was implemented and compiled better
results.35 Fig. 2 shows that PID control is able toovercome disturbance while stability is highly effected
due to sensor noises.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan187

Syed Faiz Ahmed, Athat Ali, M. Kamran Joyo, Khusairy Abd Kader, Izhar A. Bakar, Hazry Desa, Sheroz KhanPID control may be applied to resolve above
mentioned issues of position and orientation of quadrotors.
The response of the PID control shows the stability withalmost zero steady state error.30

In,35 The nonlinear dynamic model of the quadcopter
is formulated using the Newton-Euler method andbackstepping control technique was applied forstabilizing the quadrotor by using the Lyapunov stability
analysis.
Adaptive control algorithms adapt the changing
parameters like uncertainties or change in time that
occurred in the system. In 2011 a technique called
Adaptive Integral Backstepping was developed for
position control of quad-rotor. Initially it was observedthat only integral backstepping was not enough so
adaptive technique was introduced to gain betterrobustness [19]. The performance of controller was betterbut it did not provide any solution to noisy measurement
issues.
To control the altitude of quadrotor adaptive sliding
backstepping control is an important field of research.
Adaptive elements in the sliding mode control
formulation the proposed control law avoids a priori
knowledge of the upper bounds on the uncertainty.
In 2011, a researcher developed robust adaptive
control for a quadrotor helicopter. The researcher applieda step function as disturbance to its system. The
developed controller could regain the equilibrium state.
To track a reference trajectory in the presence ofdisturbances and uncertainties a robust quadrotor
controller known as Super Twisting controller isintroduced in 2016. A Super Twisting controller isimplemented using the gain adaptation law, which has
the advantage of not requiring the knowledge of theupper bound of the lumped uncertainties. The design ofthe controller is developed without separation in two
nested control loops for altitude and position and design
is on regular form of quadrotor dynamics.8
In 2013, for controlling quadcopters reinforcement
learning based control technique was developed which
was applied to black box quadcopter system along withthe learning control. This technique treats system as a
black box and stores the results in a transition matrix,
hence preferred for nonlinear systems.15

A control technique used to achieve desired
performance optimally at minimal cost. LQR control
technique may also be used for quadrotor position,
altitude and attitude controlling. Results shown in Fig. 3
represent that LQR performs well under noisy conditions
but doesn’t performs well under disturbance conditions.
LQR response under heavy disturbance and noisy
conditions shown in Fig.3.36 In 2009, another researcher
used LQR for position controlling of a quadrotor. The
results were satisfactory but in his research work author
did not consider the effect of disturbance on a quadrotor
system.14
SMC is a non-linear control whose control output moves
in a prescribed path. The control is applied to stabilize thesystem effectively. Sometimes SMC is referred to as gain
scheduling

16

MeasuredDesiredDisturbance

14
12

Amplitude

10
8
6
4
2
0
0
Offset=0 (secs)

2

4

6

8

10
Time (secs)

12

14

16

18

20

Fig.3. LQR response under heavy disturbance and noisy
conditions [35].

control as its gain is actively changing from positive tonegative directions.
A researcher used an impulse function as a
disturbance to the quadrotor system while using SMC toovercome the applied disturbance.33 In 2007, anotherresearcher used sliding mode to observe the disturbances.
In the simulation tests step function were applied on thevehicle as a wind gust and external disturbance.34

A common network can be trained to directly map
state to actuator command with reinforcement learning,making any predefined control structure obsolete fortraining. The application of a learning algorithms thatallows the training of suitable control actions relaxes thedifficult task of nonlinear control design. Reinforcement
learning is applied as one form of unsupervised learning.

Quadrotors are highly nonlinear and under actuatedsystems and that what makes them a great platform forcontrol system research. Back stepping control (BSC) is
modular in nature and a recursive control algorithm
which divides system into subsystems to achieve stability.
Quadrotor motion can be divided into two subsystems; atranslational subsystem (altitude, x and y motion) and arotational subsystem (attitude and heading).

Besides these control algorithms other control
algorithms are also developed and implemented like:
Robust Control Algorithms with deals with the all typesof uncertainties and disturbances in the system andensures performances within acceptable ranges, Optimal© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

188

Control Techniques of Quadrotor[13] Vijaykumar Sureshkumar, K.C., 2014. Intelligent Fuzzy
Flight Control of an Autonomous. AIAA SciTech,(January), pp.1–10.
[14] Singh, A.M., Lee, D.J. & Chong, K.T., 2013. Attitude andaltitude controllers for quadrotor. 대한전자공학회
학술대회, 2013, pp.1436–1437. Available at:
http://www.dbpia.co.kr/Article/NODE02275371.
[15] Omidshafiei, S., 2013. Reinforcement Learning-based
Quadcopter Control. , (1).
[16] Patel, A.R., Patel, M. a & Vyas, D.R., 2012. Modeling andanalysis of quadrotor using sliding mode control.
Proceedings of the 2012 44th Southeastern Symposium on
System Theory (SSST), pp.111–114.
[17] Elruby, a Y., El-khatib, M.M. & Hashad, N.H.E.-A. a I.,
2012. Dynamics Modeling and Control of Quadrotor
Vehicle. Amme, pp.280–286.
[18] Rudin, K. et al., 2011. A robust attitude controller and its
application to quadrotor helicopters. IFAC Proceedings
Volumes (IFAC-PapersOnline), 18(PART 1), pp.10379–
10384.
[19] Fang, Z. & Gao, W., 2011. Adaptive integral backstepping
control of a Micro-Quadrotor. Proceedings of the 2nd
International Conference on Intelligent Control andInformation Processing, ICICIP 2011, (PART 2), pp.910–
915.
[20] Alpen, M., Frick, K. & Horn, J., 2009. Nonlinear modelingand position control of an industrial quadrotor with onboard attitude control. 2009 IEEE International
Conference on Control and Automation, ICCA 2009,
pp.2329–2334.
[21] Voos, H., 2009. Nonlinear Control of a Quadrotor MicroUAV using. , (April), pp.4–9.
[22] Adigbli, P. et al., 2007. Nonlinear Attitude and Position
Control of a Micro Quadrotor using Sliding Mode andBackstepping Techniques. European Micro Air Vehicle
Conference and Flight Competition, (September), pp.17–
21.
[23] Erginer, B. & Altug, E., 2007. Modeling and PD Control
of a Quadrotor VTOL Vehicle. 2007 IEEE Intelligent
Vehicles Symposium, pp.894–899.
[24] Bouabdallah, S., Becker, M. & Siegwart, R., 2007.
Autonomous miniature flying robots: Coming soon! Research, development, and results. IEEE Robotics andAutomation Magazine, 14(3), pp.88–98.
[25] Bouabdallah, S. & Siegwart, R., 2006, Towards Intelligent
Miniature Flying Robots. , pp.1–12.
[26] Bouabdallah, S. & Siegwart, R., 2005. Backstepping andsliding-mode techniques applied to an indoor micro
Quadrotor. Proceedings - IEEE International Conferenceon Robotics and Automation, 2005(April), pp.2247–2252.
[27] Bouabdallah, S. et al., 2004. PID vs LQ Control
Techniques Applied to an Indoor Micro Quadrotor.
IEEE/RSJ International Conference on Intelligent Robots
and Systems, 3, pp.2451–2456.
[28] John, S. (2013) Artificial Intelligent-Based Feedforward
Optimized PID Wheel Slip Controller. AFRICON, 12
September 2013, Pointe-Aux-Piments, 1-6.
[29] Li, J., & Li, Y. (2011). Dynamic analysis and PID control
for a quadrotor. Paper presented at the Mechatronics andAutomation (ICMA), 2011 International Conference.
[30] Salih, A. L., Moghavvemi, M., Mohamed, H. A., & Gaeid,
K. S. (2010). Flight PID controller design for a UAV

Control Algorithms which includes LQR, L1, H∞ andKalman filter, Feedback Linearization which converts
the non-linear system into its equivalent linear systemand hybrid control algorithms which involves more than
one control techniques for controlling thequadrotors.31,32,36
4. Conclusion
Several quadrotor models and their control algorithms
have been introduced in the recent years. This purpose ofthis document is to provide the review of available
quadrotors models and the control schemes applied forstabilizing the control of attitude, altitude and position toprovide a generic reference about existing solutions andassist the development of improved devices.
5. References[1] Muhammad Hassan, T., Hazry, D., Faiz, S., Muhammad
Kamran, J., Faizan, A. W., Zuradzman, M. R., ... &Abadal-Salam, T. H. (2014). PID based controller design
for attitude stabilization of Quad-rotor.
[2] Joyo, M.K. et al., 2013. Altitude and horizontal motion
control of quadrotor UAV in the presence of air turbulence.
Proceedings - 2013 IEEE Conference on Systems, Process
and Control, ICSPC 2013, (December 2013), pp.16–20.
[3] Máthé, K. & Buşoniu, L., 2015. Vision and Control for
UAVs: A survey of general methods and of inexpensive
platforms for infrastructure inspection. Sensors (Basel,
Switzerland), 15(7), pp.14887–916.
[4] Norouzi Ghazbi, S. et al., 2016. Quadrotors unmanned
aerial vehicles: A review. International Journal on Smart
Sensing and Intelligent Systems, 9(1), pp.309–333.
[5] Zulu, A. & John, S., 2014. A review of control algorithms
for autonomous quadrotors. Open Journal of Applied
Sciences, 4(December), pp.547–556.
[6] Huo, X., Huo, M. and Karimi, H.R. (2014) Attitude
stabilization control of a quadrotor uav by usingbackstepping approach. Mathematical Problems in
Engineering, 2014, 1-9.
[7] Cai, Guowei, Jorge Dias, and Lakmal Seneviratne. "Asurvey of small-scale unmanned aerial vehicles: Recent
advances and future development trends." Unmanned
Systems 2.02 (2014): 175-199.
[8] Rajappa, S. et al., 2016. Adaptive Super Twisting
Controller for a Quadrotor UAV. , (1), pp.2971–2977.
[9] Mustapa, M.Z., 2015. Altitude controller design for
quadcopter UAV. Jurnal Teknologi, 74(1), pp.187–194.
[10] Emran, B.J. & Yesildirek, A., 2014. Robust Nonlinear
Composite Adaptive Control of Quadrotor. International
Journal of Digital Information and WirelessCommunications, 4(2), pp.213–225.
[11] Hassan Tanveer, M. et al., 2013. Stabilized controller
design for attitude and altitude controlling of quad-rotor
under disturbance and noisy conditions. American Journalof Applied Sciences, 10(8), pp.819–831.
[12] Tanveer, H. & Hazry, D., 2014. Model Predictive Control
based reference point tracking of quad-rotor UAV in
prevalence of disturbance 1. , 8(4), pp.428–431.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan189

Syed Faiz Ahmed, Athat Ali, M. Kamran Joyo, Khusairy Abd Kader, Izhar A. Bakar, Hazry Desa, Sheroz Khan

quadrotor. Scientific Research and Essays, 5(23), 36603667.
[31] Yibo, L. (2012, 18-20 Oct. 2012). A survey of control
algorithms for Quadrotor Unmanned Helicopter. Paper
presented at the Advanced Computational Intelligence(ICACI), 2012 IEEE Fifth International Conference.
[32] Rinaldi, F., Chiesa, S., & Quagliotti, F. (2013). Linear
quadratic control for quadrotors UAVs dynamics andformation flight. Journal of Intelligent & Robotic Systems,
70(1-4), 203-220.
[33] Benallegue, A., Mokhtari, A., & Fridman, L. (2006).
Feedback linearization and high order sliding mode
observer for a quadrotor UAV. Paper presented at the
Variable Structure Systems, 2006. VSS'06.
[34] Besnard, L., Shtessel, Y. B., & Landrum, B. (2007, 9-13
July 2007). Control of a Quadrotor Vehicle Using Sliding
Mode Disturbance Observer. Paper presented at theAmerican Control Conference, 2007. ACC '07.
[35] Kamran Joyo M., Ahmed S.F., Bakar M.I.A., Ali A.
(2018) Horizontal Motion Control of UnderactuatedQuadrotor Under Disturbed and Noisy Circumstances. In:
Mishra D., Azar A., Joshi A. (eds) Information andCommunication Technology. Advances in Intelligent
Systems and Computing, vol 625. Springer, Singapore.
[36] S. Faiz Ahmed, Kushsairy Kadir and M. Kamran Joyo,
2016. LQR based controller design for altitude andlongitudinal movement of quad-rotor. J. Applied Sci., 16:588-593.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan190

A GIS Based Hydrological Model for River Water Level Detection & Flood Prediction
featuring morphological operations.
Sarmad Zafar

Mohammad Ali Jinnah University, Pakistan
Sarmadzafar10@gmail.com
H.M.Sohaib Azhar

Mohammad Ali Jinnah University, Pakistan
Engr.sohaib.azhar@hotmail.com
Aqeel Tahir

Mohammad Ali Jinnah University, Pakistan
Engr.aqtahir84@hotmail.com

Abstract

Mostly river water level and flood forecasting methods are based on gauging stations measurements
at discrete locations, which limits their capability to provide accurate and timely data over large extent,also limited or no data available on remote locations. So here we present an idea to use high resolution
satellite images for real time mapping of river water level. In this project, we developed a Web based GIS
system for mapping river water level, early warning and mapping for flood disasters. To improve flood
forecasting/warning, we developed a decision support system (DSS) for flood monitoring and predictionthat integrates GIS, satellite image processing and hydrological modelling. We present the methodology fordata integration, floodplain delineation, and online map interfaces. Our Web based GIS system candynamically display observed and predicted water levels for decision makers and the general public. The
users can access a Web-based GIS system which models current flood events and displays satellite imagery
and 3D visualization integrated with the flood plain area. The output from the hydrological modeling will
be used for flooding prediction for the next 1 day to 2 days (24 and 48 hours) along the lower Indus River.
In this stage river water level analysis has been achieved, work on the hydrological modelling is in progressto acquire river water stage & flood level and the prediction.
Keywords: Hydrological modelling, GIS based system, morphology, satellite imagery, 3D visualization.

of large regions or countries. If we observe the recent data
large-scale severe droughts have been observed indifferent areas of the world encompassing all continents
leading to economic and natural resources loss, thisdestruction lead to food shortages and starvation ofmasses. On the other hand floods are the most devastating
natural hazards in the world. Floods were most balefulthan any other natural disaster both in claiming more
lives and causing more property damage.
For effective management of disaster users like top level
policy makers at the national and internationalorganizations, middle level policy makers at local levels
consultants, researchers, relief agencies and local

1. Introduction
Droughts and floods are both water-related natural
disasters which have a widespread effect onenvironmental factors and activities like human life,
agriculture, vegetation, local economies and wild life.
These both are natural disasters which are thought to be
beyond human control, but drought is the one of the mostimportant weather-related natural disaster which is often
aggravated by human action. Drought affects a very large
area for months and even for years. So it has a severe
impact on regional food production, life expectancy ofthe entire populations and overall economic performance© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan191

Sarmad Zafar, H.M.Sohaib Azhar, Aqeel Tahirproducers which includes farmers, water managers
suppliers and traders are interested in reliable, accurate
and timely information of drought and flood. The disastermanagement activities can be grouped into three major
phases: The Preparedness phase in which prediction andrisk zone are identified, identification is done long before
the actual disaster event occur; in Prevention phasedifferent activities are carried out like, monitoring, early
warning & Forecasting, and preparation of contingency
plans that should be taken up just before or during theevent; and the Response/Mitigation phase includes theactivities which are done damage assessment and relief
management.
Though flood cannot be stop but their effect can be
minimized if we have proper data, so in this project ourobjective is to map the river flow data on geographicalinformation system. As rivers cover a large geographicalarea so we can obtain data using remote sensing
techniques and distribute information to control stations
rapidly over large areas by means of satellites ortransponders mounted on drones or aircraft. A satellite
orbits the Earth, can explore the whole surface in a few
days and repeat its survey of the same area at regularintervals. For these satellite images we will use publicallyavailable high resolution images like NASA MODIS,SPOT etc.

2.1. Satellite Image Processing ResearchSatellite image processing techniques will be used for
effective management of water bodies on land and used
for decision making for disaster management. An
extensive research has been done for identifying anddesigning the best suited algorithm for analyzing andprocessing the high resolution satellite images. The
process of identifying the stage of a river using satellite
images is a difficult process, because the images captured
from satellite are remotely sensed at a very long distance.
We utilize the Support Vector Machine algorithm for thispurpose. The work is divided into three phases thetraining phase, analysis phase and the testing phase. In
the training phase, water region is extracted from theimage and the stage of river is identified both by training
two different ANNs. In the testing process, the input raw
river image was passed through different process firstimage filtration to de-noised and morphologicaloperation was carried out on the de-noised image. Afterthat the river image was segmented into water regionswith the aid of the ANN. Finally in the analysis phase,the stage of the river water was categorized on scale ofthree whether river is in Normal, drought or flood
condition.

2. Methodology
The proposed project models a particular area on the GIS
and collect relevant data like satellite images of highresolution at different time period. The backend
processing application will take these images as input;
image will go through multiple stages of processing andoutput of this will be quantifiable data along will the geo
location. The front end GIS based web or mobile
application will take date from the data base and model iton the geographical information system.

2.2. Satellite Image Data Gathering
One of the most important thing in our project is the highresolution satellite image data, which is publicallyavailable on different platforms some are:NASA MODIS Imagery
SPOT Imagery
JAXA’s Global ALOS 3D WorldNOAA Data Access Viewer – Discover
Authoritative Datasets
Global Land Cover Facility – Derived Satellite
Data.

Fig. 1. GIS based Prediction system block diagram© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan192

A GIS Based Hydrological2.3. Layout Designing

2.4. Image Processing

Low fidelity prototypes have been designed, some basic
prototypes are shown below:

The developed algorithm is able to identify the water
bodies in the high resolution satellite images. In order tounderstand the flood dynamics and hydrologicalexchange between river and related floodplains, it relieson the data gathered of water levels at discrete locations
on main channel. But due to vast expansion of the river
basins limited gauging stations and river passes throughlocations selected where it’s difficult to have thesestations, so some other techniques must be implied inorder to improve the spatial sampling, like radar remotesensing using interferometric phase measurements thiscan improve the spatial sampling, but the phase istemporally incoherent over open water so this technique
is used to determine water levels. Satellite images areprocessed and passed through these different stages toextract the river stage.
Complete flow is shown with Image Processing steps in
Fig. 5:

Fig. 2. River Geographical Map View

Fig. 3. High resolution Satellite Image
Fig. 5. Steps involve in the process

2.4.1. Image Acquisition
Acquired an image from open access NASA database

Fig. 4. GIS Application Mockup
Fig. 6. This is a moderate resolution satellite image© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan193

Sarmad Zafar, H.M.Sohaib Azhar, Aqeel Tahir

The image is passed through different steps to extract theriver stage.

2.4.4. Anisotropic diffusion segmentation:
In image processing and computer vision: anisotropic
diffusion, is a technique used to reduce the noise level
without affecting substantial image contents parts. The
contents, usually edges, lines or other details that are
significant for the analysis of the image. In Anisotropic
diffusion a scale space has created, in which an image
generates a parameterized family of consecutively more
and more blurred images based on the result of diffusion
process. Each of the resulting image in this family are
given as a convolution between the image and a 2D
isotropic. The following equation details thesegmentation process.

2.4.2. Image Filtration:
Filtering is used to modify and enhance the image byremoving the unwanted pixel values in the image. For
example, you can filter out an image to emphasize certain
features or remove other features. Image processing
operations implemented with filtering include
smoothing, sharpening, and edge enhancement.
In image processing filtering is a neighborhood
operation, in which the value of each pixel in the output
image is calculated by applying some algorithm to thevalues of the pixels in the neighborhood of thecorresponding input pixel. A pixel's neighborhood is thesurrounding pixels, defined by their locations relative tothat pixel.

Equation 2. Delta Function applied to Image.

2.4.3. Lab color space Conversion

Where div is the representation for the divergence
operator, ∆ stands for the Laplacian, ▼ for the gradientand c(x,y,t) is the diffusion coefficient. c(x,y,t) is used tocontrol the rate of diffusion and it is selected as a functionof the image gradient in order to conserve edges in theimage.

The Lab color space mathematically describes allperceivable colors in the 3 dimensional space. L used forlightness, ‘a’ and ‘b’ for the color components, green–red
and blue–yellow. The Lab color space has moreenhanced than the RGB and CMYK color models. For
the conversion of RGB image to the Lab color format
below giver equations are used.

2.4.5. Image Binarization
The segmented image is transformed into binary image
which is then used for further processing. Binarization
is a process of converting an image by extracting
lightness (brightness & density) intensities as a feature
amount from the image. In this process pixel image isconverted into binary image. When a pixel area ispicked of an image, sensitivity value is added to or
subtracted, concerning the Y values of the selected pixel
to describe a threshold range value. Further, whenanother pixel is chosen, the same process is applied to
set new threshold value. The pixel with the value
concerning the Y value of any pixel in the image within
the threshold value range is extracted as the samebrightness as the selected pixel and the extraction result
is displayed. The output image after binarization contain
only black and white image.

After conversion to CIE Lab color image look like:

2.4.6. Morphological operation:
The morphological operation is applied in order toachieve the smoothness, so that intended information be
extracted. The impact of the morphological operation
depends on the application for example in machinelearning & computer vision, object detection in which
we may apply it to classify the structure of an image.
Morphological operation is useful to differentiate
boundaries of objects or to identify objects present in an

Fig. 7. CIE color converted Image.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan194

A GIS Based Hydrologicalimage. In this, we apply the “imclose” function as
morphological operation closes the image. The
following equation details the process.

∘

= (

⨁

3.

Conclusion
This paper presents the idea for the integration ofsatellite imagery, hydrological modeling andgeographical information system to come up with adecision support system which will be used forcontinuously monitoring the river water level. By
processing the satellite imagery we can extract theriver stage over the entire spatial range from where
the river passes.
The system for flood events prediction andmonitoring integrated with the web map interface
will facilitates in monitoring, prediction and decision
making of the flooding events. It works as an early
warning and mapping of flood disasters.

) ⊖Equation. 3. Image segmentation expression Here st is
structuring element which is required by which themorphological operation is applied.
Image after morphological operation looks like:Fig. 8. Morphological Image

4.

References

1.

Laurence C. Smith, "Satellite Remote Sensing ofRiver Inundation Area, Stage, And Discharge: A
Review", Hydrological Processes, Vol. 11,
pp.1427-1439, 1997
Flood Forecasting GIS Water-Flow Visualization
Enhancement (WaVE): A Case Study Timothy R.
Petty1, Nawajish Noman2, Deng Ding2, John B.
Gongwer3
DROUGHTS & FLOODS ASSESSMENT AND
MONITORING USING REMOTE SENSINGAND GIS by A.T. Jeyaseelan
Nipuni Hettiarachchi “Water Level Forecasting andFlood Warning System A Neuro-Fuzzy Approach”,
International Conference in Video Image andSignal Processing, At India
Michael T. Coe, "Calculation of river discharge andprediction of lake height from satellite radar
altimetry: Example for the Lake Chad basin", Water
Resources Research, Vol. 40, 2004
D. Mioc “EARLY WARNING AND MAPPINGFOR FLOOD DISASTERS”, The International
Archives of the Photogrammetry, Remote Sensing
and Spatial Information Sciences. Vol. XXXVII.
Part B4. Beijing 2008
HJ Barneveld “APPLICATION OF SATELLITEDATAFORIMPROVEDFLOODFORECASTINGAND
MAPPING”,
4th
International Symposium on Flood Defence:Managing Flood Risk, Reliability and Vulnerability
Toronto, Ontario, Canada, May 6-8, 2008
R. Loren and D. B. Benson (eds.), Introduction toString Field Theory, 2nd edn. (Springer-Verlag,
New York, 1999).
R. M. Karp, Reducibility among combinatorial
problems,
inComplexityofComputerComputations (Plenum, New York, 1972), pp. 85–
104.

2.

2.5. Training Data
So as to identify the stage of Support Vector Machine(SVM) technique is employed, SVM is trained by a setof water identifies images. Support Vector Machine is amachine learning method that classifies binary classes byfinding and using a class boundary the hyper planemaximizing the margin in the given training data. The
training data samples along the hyper planes close to theclass boundary are called support vectors, and the marginis the distance between the class boundary, support
vectors and hyper-planes. The SVM work on the basis of
the decision boundaries defined by the decision planes. A
decision plane separates between assets of objects having
different class memberships. SVM is a one of the very
useful technique for data classification. For classification
task training and testing date is utilized which consists ofsome data instances. Each instance in the training set
contains one “target value” (class labels) and several“attributes” (features).

3.
4.

5.

6.

7.

8.
9.

Fig. 9. River water levels© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

195

Analysis of onboard sensor-based odometry for a quadrotor UAV in outdoor environment
Aidar Gabdullin, Grigory Shvedov, Mikhail Ivanou, Ilya Afanasyev

Robotics Institute, Innopolis University
1 Universitetskaya street, 420500 Innopolis, Russia
E-mail: {a.gabdullin, g.shvedov, m.ivanov, i.afanasyev}@innopolis.ru
http://university.innopolis.ru

Abstract
This article introduces a comparative analysis of a quadrotor UAV trajectories evaluated by processing onboard
sensors (camera and IMU) with ROS-based monocular visual odometry software. Parrot Bebop drone was launchedoutdoor within teleoperated closed-loop trajectory along a known perimeter of a square work area, recording sensor’s
telemetry data for offline processing. We compared UAV monocular SLAM trajectories with Bebop visual-inertial
odometry and verified them with the ground truth estimated by an external observer (a hovering quadcopter DJI
Mavic, flying at the height of 40m with a camera pointing downwards and tracking near-surface Bebop’s flight).
Keywords: monocular visual odometry, GPS-denied environment, quadrotor UAV, SLAM, ROS

shakes and high-frequency vibrations from motors andpropellers. To solve this issue, it is used a camera with agood image quality and robust digital stabilization. In our
research, we employ a Parrot Bebop drone outdoor inteleoperated mode, making a closed-loop trajectory along
a known perimeter of a work area at a height of 1.5-2m,
acquiring sensors telemetry data for offline processing
with Robot Operating System (ROS) packages.
The rest of this paper is organized as follows. Section
2 describes system setup. Section 3 presents ROS-based
computations of visual odometry trajectories and theirverification by an external observer. Section 4 provides
outdoor experiment results with comparative accuracy
analysis. Section 5 concludes this paper.

1. Introduction
At present, the tasks of autonomous quadrotor UAV
localization, mapping and navigation in GPS-denied
environment are solved mainly by vision-based sensing
and inertial navigation systems (INS), combining visualfeature data from a monocular camera with inertial sensor
measurements [1-4]. During the last years many studies
of GPS-denied indoor navigation have focused on the
monocular visual localization applications [5-8] andcomparative analysis of different vision-basedalgorithms [9-11], since visual localization is frequently
a natural substitution of a GPS approach. However, there
is still a lack of investigations dealt with efficiency androbustness of monocular visual odometry methods for
UAV quadrotors in GPS-denied outdoor environment.
When hovering outdoors, especially in a windy
condition, a conventional quadrotor UAV withuncalibrated fixed camera actively changes pitch and rollangles, recording a video with a poor quality due to

2. System setup
In our outdoor experiments we used Parrot Bebop drone(Fig. 1), as a robust quadrotor UAV, which has a digitallystabilized camera and special flight modes with drone© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

196

Aidar Gabdullin, Grigory Shvedov, Mikhail Ivanou, Ilya Afanasyev

landing in case of collisions a . Bebop controls stability
and maneuverability of its flight by automatic processing
sensors data from 3D inertial measurement unit (IMU),
an ultrasound sensor of an 8m range, a pressure sensor
and a vertical camera (to track the speed). Bebop
quadrotor can record video in a 180° panoramic view by
the 14mpx "fisheye" camera with digital image
stabilization, taking stable aerial footage independently
on UAV maneuvers. Parrot Bebop drone has a firmware
with integrated visual-inertial velocity estimation(without GPS), and an open source ROS wrapperb to get
and process telemetry data.
For a top-view Bebop’s trajectory validation (as theground truth) we used a hovering DJI Mavic
quadrocopter c as an external observer. It was also
teleoperated to hold a position at the height of 40m witha camera pointing downwards and tracking near-surface
Bebop’s flight. Its camera is actively gyro-stabilized witha brushless gimbal, recording a video with the resolutionof 1920*1080 at 60 fps. For observer sensor’s dataprocessing we utilized OpenCV 3.0d and ROS Kinetice.

provide relevant results in steady-weather conditions, but
in strong wind and rapid changes in accelerations, weobserved a displacement at ~10m per cycle at a wind of
6m/s (Fig. 2) Therefore, we employed DJI Mavic drone
as an external observer for a ground truth.

(a) Parrot Bebop drone.

(b) Experimental work area

Fig. 1. Outdoor experiments with Parrot Bebop drone

3.2. Odometry verification by external observer
As a ground truth (reference trajectory) we used an
external observer data taken from a hovering DJI Mavic
quadcopter, flying at an altitude of 40m with a camera
pointed down and tracking near-surface Bebop’s flight.
We processed the image footages offline, tracking Bebop
with following steps: (1) Computing a dense optical flow
with color coding for each video frame [12]; (2)Converting images to grayscale and thresholding; (3)Detection of Bebop contour and its center position (x,y).
As additional DJI Mavic camera calibration the landmark
size in work area were measured in advance, thus wetransformed image pixels to geocoordinates in meters.

3. Bebop drone odometry and its verification
3.1. Bebop visual-inertial odometry
Parrot Bebop drone has a default option of odometry
calculation by a bebop_autonomy driverf, which allows
reading from UAV camera, odometry, GPS and drone
states (i.e. on-board data also known as Navdata). The
odometry integrates visual-inertial velocity over time(dead reckoning), using video stream from Bebop’s front
camera with a bebop_driver and Bebop’s position &velocity in an ENU aligned odometry (which is based on
REP-103 specifies an ENU (East, North, Up) coordinate
frame for outdoor navigation g ). Since odometery is
calculated from Bebop States (Navdata), the update rate
is limited to 5 Hz that can be a reason for growth of
trajectory deviation error during Bebop flight (see loop
closure displacements in trajectories in Fig. 2a during
three cycles of UAV flight along the same work area
perimeter in Fig. 1). Visual-inertial odometry seems to(a) Bebop visual-inertial odometry.

(b) DSO results

Fig. 2. Difficulties with UAV odometry estimations: (a) Threeloop closure displacements in Bebop visual-inertial odometry
trajectories (in meters) for UAV flight under the same work
area in windy conditions; (b) Divergence of UAV trajectory
received with Direct Sparse Odometry (DSO).

Parrot Bebop Drone description: http://global.parrot.com/
ROS driver for Parrot Bebop drone, based on Parrot’s official
ARDroneSDK3: http://wiki.ros.org/bebop_autonomyc
DJI Mavic drone: https://www.dji.com/mavicd
OpenCV 3.0 library: https://opencv.org/opencv-3-0.html

ROS Kinetic Kame middleware : http://wiki.ros.org/kinetic
ROS Driver for Parrot Bebop Drone (quadrocopter) 1.0 & 2.0:
http://bebop-autonomy.readthedocs.io/en/latest/index.htmlgREP-103 provides a reference for the units and coordinate
conventions used within ROS: http://www.ros.org/reps/rep-0103.html

a

e

b

f

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

197

Analysis of onboard sensor-based

3.3. ROS-based Visual SLAM
ORB-SLAM: Oriented FAST and Rotated BRIEFh [13]in most cases is one of the most accurate and robust
SLAM solution that has ROS-based package with bundle
adjustment, monocular feature-based observations andcamera trajectory estimation. Thus, ORB SLAM
includes features from different images in 3D space andvisual odometry tracks for unmapped regions (Fig. 3).
Finally, we built the UAV trajectory, but while testing we
detected some ORB-SLAM difficulties with: (1) randomcharacter of initialization (mostly, due to lack offeatures); (2) adding new points to a map (since ORBSLAM needs some time for new keyframes); (3) camera
rotations (then SLAM tracking tends to get lost features
and trajectory); (4) dependence trajectory deviation
accuracy on camera calibration and scaling to metricscale. To overcome some problems in our experiment weteleoperated Bebop drone along closed-loop trajectorywith slow rotations in the work area corners.

(a) Extracted features.

Fig. 4. LSD-SLAM featured depth map and UAV trajectory.

DSO: Direct Sparse Odometryj [15] combines sparse anddirect visual odometry methods, minizing a photometricerror evaluated directly from images. DSO is sensitive tocamera calibration and suffers from Bebop camera
distortions, therefore in our case instead of loop closure
we got divergence of UAV trajectories (Fig. 2b).
4. Outdoor Experiment and Results
Experimental dataset was recorded in bag-file during
Parrot Bebop drone flight and sensor’s telemetry data
was processed offline with visual SLAM algorithms. Fig.
5 illustrates the comparison of all UAV odometry
trajectories evaluated with ROS-based monocular visual
SLAM methods, Bebop visual-inertial odometry and thereference trajectory (ground truth) from the external
observer (DJI Mavic drone). For monocular SLAM
methods (ORB and LSD) the computed trajectories were
properly scaled. For error estimation we computed
maximum and average deviations with the formulae:(b) Point cloud and trajectory

Fig. 3. ORB SLAM point clouds and trajectory visualization

LSD-SLAM: Large-Scale Direct Monocular SLAMi [14]performs image-to-image alignment with simultaneous
tracking, depth map estimation and optimization (Fig. 4).
However, this method is highly demanding of computing
resources and sensitive to camera calibration. Despite
map optimization, which includes loop closure detection,
the resulting trajectory has lower precision than ORBSLAM one (although LSD-SLAM point cloud is muchdenser than for ORB-SLAM). In our case, Bebop’s
camera has a rolling shutter, thus the results were not soaccurate as they could be with global shutter camera.
Nevertheless, LSD-SLAM demonstrates stable work,
excepting the cases of fast camera rotation, and has noproblems with initialization.
hi(b) Point cloud and trajectory(a) Depth map with features.

.

. = max

.=

1

−
=1

2

−

+
2

−

+

2

−

,

2

.

(1)
(2)

where x,y are coordinates of odometry estimation andground truth correspondingly, and N is a number ofsamples (points) in path. Table 1 illustrates the error
estimations (deviations) for UAV odometry trajectories.
Table 1. Estimated max and mean errors of UAV trajectory
deviations for UAV sensor-based odometry
MethodBebop visualinertial odometry
ORB-SLAM
LSD-SLAM

ORB-SLAM2 package, github.com/raulmur/ORB SLAM2LSD-SLAM package, https://github.com/tum-vision/lsd slamj

Averagedeviation, m
2.53

Maximum
deviation, m
9.85

2.56
4.45

8.52
14.91

DSO package, https://github.com/JakobEngel/dso© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan198

Aidar Gabdullin, Grigory Shvedov, Mikhail Ivanou, Ilya Afanasyev

outdoor environment than Bebop visual-inertial
odometry and LSD-SLAM, although sometimes it has
difficulties with initialization and camera rotations.
Acknowledgements
The research was partially supported by RussianFoundation for Basic Research (RFBR), project ID 1507-07483, Russian Ministry of Education and Science
within the Federal Target Program (research grant ID
RFMEFI60917X0100) together with KAMAZ company
as the industrial partner, and Innopolis University.
References
1. M.J. Milford et al, Aerial SLAM with a single camera
using visual expectation, in IEEE ICRA (2011), 2506-2512
2. D. Scaramuzza et al., Vision-controlled micro flying
robots, Robotics & Automation Magazine 21 (2014)
3. P. Li, M. Garratt, A. Lambert, Monocular Snapshot-based
Sensing and Control of Hover, Takeoff, and Landing for a
Low-cost Quadrotor, Field Robotics, 32 (2015), 984-1003
4. M. Faessler et al., Autonomous, Vision-based Flight andLive Dense 3D Mapping with a Quadrotor Micro Aerial
Vehicle. Journal of Field Robotics 33.4 (2016), 431-450
5. A. Buyval, M. Gavrilenkov, Vision-based pose estimation
for indoor navigation of unmanned micro aerial vehicle
based on the 3D model of environment, in MEACS (2015)
6. K. Yakovlev et al., Distributed control and navigation
system for quadrotor UAVs in GPS-denied environments.
In Intelligent Systems, Springer (2015), 49-56
7. A. Buyval, M. Gavrilenkov, E. Magid, A multithreaded
algorithm of UAV visual localization based on a 3D modelof environment, in ICAROB (2017)
8. A. Bokovoy, K. Yakovlev, Original Loop-closure
Detection Algorithm for Monocular vSLAM, ArXiv
preprint arXiv:1707.04771 (2017)
9. A. Buyval, I. Afanasyev, E. Magid, Comparative analysis
of ROS-based Monocular SLAM methods for indoor
navigation, in Machine Vision, SPIE (2017), 103411K
10. M. Sokolov, et al., Analysis of ROS-based Visual andLidar Odometry for a Teleoperated Crawler-type Robot in
Indoor Environment, in ICINCO (2017)
11. I. Ibragimov, I. Afanasyev, Comparison of ROS-based
Visual SLAM methods in homogeneous indoor
environment, in IEEE WPNC (2017)
12. G. Farnebäck, Two-frame motion estimation based onpolynomial expansion, in Image analysis (2003), 363-370
13. R. Mur-Artal, and J.D. Tardós, ORB-SLAM2: an OpenSource SLAM System for Monocular, Stereo and RGB-D
Cameras, ArXiv preprint arXiv:1610.06475 (2016)
14. J. Engel, T. Schöps, and D. Cremers, LSD-SLAM: Largescale direct monocular SLAM. In European Conference
on Computer Vision. Springer, Cham (2014), 834-849
15. J. Engel, V. Koltun, and Daniel Cremers, Direct sparse
odometry, IEEE Trans. PAMI (2017)

Fig. 5. Comparison of Bebop drone trajectories computed bymonocular ORB-SLAM (blue curve), monocular LSD-SLAM(red curve), Bebop Visual-Inertial Odometry (Green curve).
Black curve is the ground truth (top view from the external
observer). The units for the XY axes are meters.

5. Conclusions
In this paper we studied the experimental results forapplication of some monocular visual odometry methods
to Parrot Bebop drone outdoor closed-loop flight andanalyzed UAV trajectories evaluated by offline
processing of onboard sensors (camera and IMU) with
ROS-based software. We compared UAV monocular
SLAM trajectories with Bebop visual-inertial odometry
and verified them with the ground truth estimated by an
external observer (a hovering quadcopter DJI Mavic,
flying at the height of 40m with a camera pointed down
and tracking near-surface Bebop’s flight).
The main conclusions from our experiment are: (1)There are difficulties with Parrot Bebop drone trajectory
recovering for all considered monocular SLAM
algorithms (due to possible tracking failure in case UAV
rotations, and SLAM sensitivity to Bebop camera
geometric distortions and calibration). (2) All monocular
SLAM methods need rescaling to metric scale for proper
UAV trajectory evaluation (that requires a reliableground truth). (3) In our experiments, visual odometry
based on ORB-SLAM is more accurate and robust to© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

199

Memristive neuron integration in digital robotic embodiment
Max Talanov, Evgenii Zykov, Yuriy Gerasimov, Evgeni Magid, Aleksander Elizarov

Kazan Federal University, Kazan, Russia
E-mail: max.talanov@gmail.com
Victor Erokhin

Institute of Materials for Electronics and Magnetism,
Italian National Council of Research, Parma, Italy
E-mail: victor.erokhin@fis.unipr.it

Abstract
In the current paper, we introduce a high-level approach for an integration of a neuromorphic memristive neuron ina real-time operating robotic system. The memristive neuron schematic, which we had presented in our earlier
works, is capable of inhibitory and excitatory learning (eSTDP, iSTDP) as well as modulation via dopamine input.
We discuss a possibility of integration of the analog memristive neuron into a digital robotic embodiment andpresent block diagram of an adapter that includes pseudo-neuronal encoder and decoder.
Keywords: artificial intelligence, robotics, spiking neural network, memristors.

embodiment system we have selected a memristive
approach that was introduced during several decades invarious optional implementations including organicpolyaniline3,4 and silicon memristive devices5 etc. Oneof promising features of memristive devices is spiketiming dependent plasticity (STDP) characteristic oflearning or self-learning of a memristive devices5,6.
Using this feature we have implemented excitatory andinhibitory STDP and dopamine modulation of amemristive artificial neuron7.

1. Introduction
Previously we have demonstrated successive attempts toimplement in bio-plausible manner four of eight basic
emotions or affects in a computational system1,2. Theseemotional drives could be useful for the self-learning
autonomous robotic systems to be used as reward-andpunishment systems. This development of our research
put us in a position to answer the question of a robotic
embodiment therefore we have started the project“Robot dream” 1.
The idea of proposed project is to discriminate twophases of real-time (a so-called “wake”) phase and not
real-time bio-inspired calculations (“dream”) phase. The“wake” phase is performed via a robotic system that issynchronized periodically with the “dream” phase,
which is implemented at a cluster or a supercomputer
due to their high performance impact. NeuCogAr
project2 describes in details the bio-plausible
neuromodulating cognitive architecture. For the purposeof implementation of the real-time operating robotic

2. Problem
The idea to use memristive device as synapse is not new
but still we have failed to find a successful
implementation of bio-plausible neural network capableof neuromodulation, excitatory (eSTDP) and inhibitory(iSTDP) STDP. The listed above basic neurobiologicalmechanisms are crucial for mammalian emotional
regulation, decision-making and behavioral strategies
generation and implementation8,9,10,11. Even if we could
create a bio-inspired memristive brain there is still a© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan

200

Max Talanov, Evgenii Zykov, Victor Erokhin, Yuriy Gerasimov, Evgeni Magid, Aleksander Elizarov

fundamental problem of its embodiment. We believethat the most promising approach for the embodimentproblem solution is through a use of a robotic body.
How do we integrate a spiking neural network with adigital robotic environment?
This position paper describes a high levelarchitecture for memristive neuromorphic devices
integration with a realtime or semi-realtime robotic
embodiment system and is based on our previous papers
on memristive neuromorphic computing solution forneuromodulating neuron with eSTDP and iSTDP11.

Figure 1 (in the middle) the lilac graph represents theresult of the memristive device learning the overall
conductivity. It is set by modulated learning impulsesthat are formed as Hebbian learning. For thesimplification of the simulation purposes we used two
different generators with phase shift to simulatedifferent ∆t. This way we could depict the entire
Hebbian learning in one graph. Simulation methods
were used to calculate required nominal values of theelectronic components and to validate the quality of the

3. Proposed Solution
3.1. Memristive neuron
Previously we have proposed an overall block diagram
and the implementation of memristive neuromodulatoryneuron11. We have demonstrated the bio-inspired
learning via STDP and modulation of learning functions
via dopamine input. The proposed schematic has three
types of input: excitatory, inhibitory and modulatory.
Excitatory and inhibitory inputs have their memristive
devices that implement the learning functions of eSTDP
and iSTDP. The modulating input alters the eSTDP
functions increasing and decreasing its amplitude in bioinspired manner12. Currently we carry on with making
the schematic more bio-plausible to get easierintegration option into biological environment. We havesuccessfully demonstrated the simulation results of the
Hebbian learning via levels of memristive device
conductivity that was determined by series of learning
impulses ∆w = 1/∆t where ∆t is the time lag between
pre-synaptic spike and post-synaptic spike or inboundand outbound impulses of the memristive neuron, andtheir modulation by the dopamine level. The dopaminelevel is identified via the setup of dopamine
potentiometer that influences the learning impulses
amplitude and in its turn influences the memristive
device conductivity.
The simulation results are presented in Figure 1. The
top graph depicts the dopamine (DA) level andidentifies the level of modulation of learning impulsesthat is visible as the increment of green graph amplitude(the bottom graph), that in its turn influences thememristive device conductivity, described below. In

Fig. 1. The simulation results of learning STDP: (top) level of
DA influence or setup of DA potentiometer, (middle) graph ofmemristive device conductivity, (bottom) learning impulses

proposed model. We used integrated schematic editor
and mixed analog/digital simulator LTspice. Figure 2
represents the wiring schematic, where excitatory andinhibitory learning impulses are transmitted tomemristive elements. Two different sources generatesignals with different phase to gain the effect of viable∆w. Temporal and amplitude characteristics of impulses
have been simulated. Impulses have been investigated in
the time range from 1 to 800 milliseconds. The
excitatory and inhibitory impulses replicate the Hebbianand ”sombrero'' learning functions according to thetheoretically predicated forms.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan201

Memristive neuron integration in

Fig. 2. Wiring schematic of modulatory, excitatory and inhibitory memristive neuron device.

3.2. Integration
The other interesting goal is the integration into digital
robotic embodiment system, which could be theanthropomorphic13 or non-anthropomorphic12 system.
The presented in the Figure 3 high-level architecture
includes: memristive neuron presented above. The
neuron has three types of inputs: sensory, inhibitory andmodulatory.
Sensory is usually excitatory and is the input fromdifferent sensors that are connected via controllers anddigital-to-neuronal converters (DNC) or encoders thattranslate the digital input in form of pseudo-neuronal
activity via, for example inverse rate encoding. The
pseudo-neuronal activity is similar to neuronal but isgenerated by artificial for example memristive neuronal
devices. The inhibitory input is usually the input thatcomes from different neurons; for example, forreciprocally inhibition, this is frequent pattern in spinal
cord. This could be considered as a blocking system forunwanted behavior.

Fig. 3. The basic integration high-level architecture tointegrate spiking memristive neuron with digital environment.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan202

Max Talanov, Evgenii Zykov, Victor Erokhin, Yuriy Gerasimov, Evgeni Magid, Aleksander Elizarov

2. M. Talanov, J. Vallverdú, B. Hu, P. Moore, A. Toschev,
D. Shatunova and A. Leukhin, Emotional simulations anddepression diagnostics. Biologically Inspired Cognitive
Architectures (2016).
3. A. V. Emelyanov, D. A. Lapkin, V. A. Demin, V. V.
Erokhin, S. Battistoni, G. Baldi, A. Dimonte, A. N.
Korovin, S. Iannotta, P. K. Kashkarov and M. V.
Kovalchuk, First steps towards the realization of a double
layer perceptron based on organic memristive devices.
AIP Advances (2016), 6 (11), 111301.
4. V. A. Demin, V. V Erokhin, A. V. Emelyanov, S.
Battistoni, G. Baldi, S. Iannotta, P.K. Kashkarov and M.
V. Kovalchuk, Hardware elementary perceptron based onpolyaniline memristive devices, Organic Electronics(2015), 25, pp. 16–20.
5. A. Serb, J. Bill, A. Khiat, R. Berdan, R. Legenstein, andT. Prodromakis, Unsupervised learning in probabilisticneural networks with multi-state metal-oxide memristive
synapses, Nature Communications (2016), 7, 12611.
6. M. Prezioso, F. Merrikh Bayat, B. Hoskins, K. Likharev,
and D. Strukov, Self-Adaptive Spike-Time-Dependent
Plasticity of Metal-Oxide Memristors, Scientific Reports(2016), 6 (1).
7. M. Talanov, E. Zykov, V. Erokhin, E. Magid, S.
Distefano, Y. Gerasimov and J. Vallverdú, Modelinginhibitory and excitatory synapse learning in thememristive neuron model, Proc. 14th Int.Conf. on
Informatics in Control, Automation and Robotics (2017),
V.2, pp. 514-521.
8. R. W. Picard, Affective computing, Cambridge (The MIT
Press 1997).
9. R. W. Picard, What does it mean for a computer to “have”
emotions, Emotions in Humans and Artifacts, MIT press(2002), pp. 213–235.
10. M. A. Arbib and J.-M. Fellous, Emotions: from brain to
robot, Trends in Cognitive Sciences (2004), 8(12), pp.
554–561.
11. K. N. Gurney, M. D. Humphries and P. Redgrave, A New
Framework for Cortico-Striatal Plasticity: Behavioural
Theory Meets In Vitro Data at the Reinforcement-Action
Interface, PLoS Biology (2015), 13(1), e1002034.
12. E. Magid, T. Tsubouchi, E. Koyanagi, and T. Yoshida,
Building a search tree for a pilot system of a rescue
search robot in a discretized random step environment, J.
of Robotics and Mechatronics (2011), 23(4), p. 567.
13. E. Magid, and A. Sagitov, Towards Robot Fall Detection
and Management for Russian Humanoid AR-601, KES
Int. Symposium on Agent and Multi-Agent Systems:
Technologies and Applications (Springer, Cham, 2017),
pp. 200-209.

The modulatory input balances the neuron towards
spiking or generating output impulses. It could represent
a reward subsystem of a robotic system embodiment.
This way we could “naturally implement the rewarding
associations with, e.g., power supply.
The output of the memristive neuron schematic is
processed via NDC (neuronal activity to digitalconverter) that converts pseudo neuronal activity into
digital output activating more intensively robot
actuators with higher firing rate of memristive artificial
neuron. This way in the boundaries of one neuron andseveral sensors, we could associate modulating rewardwith power supply and then train a robotic embodiment
system to look for the energy supply autonomously.
4. Discussion
In this position paper we proposed the overallarchitecture for the integration of memristive
neuromodulating neuron introduced earlier.
To the best of our knowledge, there is no existing
memristive schematic that successfully implements
neuromodulation, though it plays important role inemotional regulation of mammals. Therefore, it seems
fruitful to implement these mechanisms in a robotic
environment in order to solve the integration problem.
We have proposed the overall integration approachof memristive analog devices with digital robotic
environment. This approach could be used to implement"natural" reward system. The reward system could use
the dopamine modulating input of proposed earlier
memristive modulating device. Hopefully using
pain/pleasure stimulus we could train proper
associations in a real-time robotic embodiment similar
to mammalian training.
Acknowledgements
Part of the work was performed according to theRussian Government Program of Competitive Growthof Kazan Federal University.
References
1. A. Tchitchigin, M. Talanov, L. Safina and M. Mazzara,
Robot Dream, Smart Innovation, Systems andTechnologies (2016), V.58, pp. 291–298.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan203

Russian mobile robot Servosila Engineer: designing an optimal integration of an extra laser
range finder for SLAM purposes
Neil Alishev, Roman Lavrenov, Yuriy GerasimovIntelligent Robotic Systems Laboratory, Higher Institute for Information Technology and Information Systems,
Kazan Federal University, 35 Kremlyovskaya street,
Kazan, 420008, Russian Federation
E-mail: alishev.neil@gmail.com. lavrenov@it.kfu.ru, yurger2009@gmail.com
http://kpfu.ru/robolab.html

Abstract
In our current research we determine an optimal design for integration of Hokuyo UTM-30LX-EW LRF into control
system of Russian mobile robot Servosila Engineer. We designed and constructed a special static stand with an option
to select an inclination of a scanning beam toward the surface of an environment. RBPF SLAM algorithm was tested
to perform localization and mapping. Experiments were conducted in order to determine the best configuration of the
LRF position and RBPF SLAM algorithm parameters.
Keywords: Robotics, algorithm, ground mobile robot, SLAM, LRF, ROS, RViz, experiments.

special expensive equipment and can be performed using
standard monocular or stereo cameras but sometimes
lack accuracy, especially in indoor environments with
low number of distinctive features2. Laser-based SLAMon the other hand is known for its’ accuracy and stability
but requires special equipment. Sonar-based SLAM canbe beneficial in some situations where both previous
SLAM approaches fail (e.g. detecting doors and walls
made of glass).
In this research LRF-SLAM approach was selected
for building a map of the environment and localizing
robot within this map. As Russian crawler-type mobile
robot Servosila Engineer is not equipped with LRF in its
original configuration, we added a special platform on
the top of the robot head with Hokuyo UTM-30LX-EW
LRF, integrated this LRF into robot control system andverified its performance within LRF-SLAM.
The rest of this paper is structured as follows: Section
2 describes system setup. Section 3 describes the processof laser integration and SLAM. Section 4 summarizes
everything and hints on a future work.

1. Introduction
Sensory information is crucial for operation of any
mobile robot. In mobile robotics, sensors are typicallyused for such tasks as localization, mapping, obstacle
avoiding and more. In this research, we tackle thechallenge of simultaneous localization and mapping(SLAM) of a Russian crawler-type mobile robot
Servosila Engineer1. SLAM could be performed for
UGVs2 and UAVs3 in many ways using different sensors:- Using single or multiple cameras (visual based SLAM).
This type of SLAM algorithm uses monocular or stereo
camera as a main source of information about
environment.
- Using laser range finder (LRF-based SLAM). LRF
SLAM algorithms use reflected laser beam to get the
information about environment.
- Using sonar. These SLAM algorithms use reflectedsound to get information about environment.
Different SLAM approaches have their own benefits
and limitations. Visual-based SLAM does not require© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan204

Neil Alishev, Roman Lavrenov, Yuriy Gerasimov

Ubuntu to the robot. Being restricted with the OS version,
we decided that the best option of ROS would be Indigo.

2. Robot system setup
Servosila Engineer4 (Fig. 1) is a Russian crawler-type
mobile robot designed for various areas of use including
search and rescue operations5, operations within
dangerous or inaccessible for a human environment andmany other areas. The robot in its original configuration

3. Laser range finder integration and LRFbased SLAMFirst step in establishing SLAM for the robot was to getdata stream from the laser range finder. For theseFig. 1. Servosila Engineer crawler-type robot with a Hokuyo
UTM-30LX-EW LRF mounted on its top.

is equipped with an optical zoom camera and a pair ofstereo vision cameras. The robot is operated with theoriginal interface in a teleoperation mode only and an
operator controls speeds of the servos and positions of therobot’s parts.
For SLAM purposes, Hokuyo UTM-30LX-EW range
finder was used. We designed and constructed with a 3D
printer a special static stand for the LRF with an option
to select an inclination of a scanning beam toward thesurface of an environment. Due to mounting withadjustable angle it is possible to use the LIDAR in
overcoming obstacles and solving the problem of robot
balance while moving through 3D debris6. The stand was
attached to the top of the robot head (Fig. 2).
In addition to the original server and software, whichwere provided by the maker, we installed Robot
Operating System (ROS) Indigo in order to run ROS
nodes, which are required for the laser range finder
output data streaming and LRF SLAM algorithms. Even
though modern versions of ROS, e.g., Kinetic Kame, arealready available, we were restricted to use ROS Indigo
version because Sevosila Engineer comes with a special
version of Ubuntu 14.04, which is tailored to the robot’sinternals so we can’t install another version of Linux

Fig. 2. Hokuyo laser on a stand. Laser is strictly parallel to thesurface of the floor.

purposes urg_node ROS package was used. This package
allows to run ROS-node, which reads data from the laser
range finder and publishes LaserScan messages to the/scan topic7. Next, laser_scan_matcher ROS package wasused to get odometry from the laser scans8. This package
compares consecutive LaserScan messages to estimate
position of the LRF in space. Laser-based odometry
turned out to be quite accurate but in Section 4 we
describe our future plans in establishing odometry fromthe crawler encoders in order to further improveodometry accuracy through the use of multiple data
sources and combining this information together. After
getting odometry, we aimed to perform mapping andlocalization for our robot.
Prior to that, experiments were conducted in asimulation in order to identify the most suitable for our
purposes LRF-SLAM algorithm. We had tested
gmapping8,9, Google Cartographer10 and Hector SLAM11© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan205

Russian mobile robot Servosila

algorithms on prerecorded rosbag files with laser range
data and concluded that gmapping together withlaser_scan_matcher significantly outperformed other
SLAM algorithms. Thus, we opted for gmapping as the
package for SLAM8.
Gmapping SLAM package is implemented using Rao- Blackwellized particle filter. Each particle in thislocalization. During the experiment, the robot was
teleoperated from the start point to the target point while
performing LRF-SLAM and the generated map wasstored. Figure 3 demonstrates that the obtained map is notdetailed enough. Thus, further experiments wereconducted in order to improve mapping results.
Figure 4 demonstrates the results of the same
experiment, but this time the map, which provides
significantly more details about the environment, wasobtainedbymanipulatingthe
RBPF-SLAM
configuration parameters. For example, linear andangular update parameters were both set to 0.1. These
parameters depend on laser frequency and range,odometry information source and level of sensor shaking
when the robot moves. Therefore, these parameters mustbe finely tuned for each particular case in order to obtain
the best mapping and localization quality.
After fine-tuning of RBPF-SLAM parameters further

Fig. 3. RBPF-SLAM with default parameters results: virtual
experiment in RViz.

algorithm is a separate Dynamic Bayesian Network that
stores its own version of a map. Rao-Blackwellized
particle filter is applied to these particles in order to pickout the most plausible information about theenvironment12. Also, the process called marginalization
is used to reduce the number of particles. This process
solves the main challenge when using the particle filter.
It groups similar particles into one particle in some area
Fig. 5. RBPF-SLAM experiment in the corridor (left) andvirtual experiment in RViz (right).

experiments were conducted in another location. A longcorridor was selected as the next experiment location
because such locations are the most challenging ones
when the odometry is coming from the laser range finder.
In the experiment, the robot was teleoperated along thecorridor from the start point to the target point, and thegenerated map was stored. Figure 5 (on the right)
demonstrates that although the corridor itself does notcontain a large number of feature points,
laser_scan_matcher algorithm succeeded to provide quite
accurate odometry information, which in turn provided
the map without any significant errors.

Fig. 4. The same experiment with the new RBPF-SLAM
parameters: virtual experiment in RViz.

R. This reduces the number of particles and allows faster
execution of the algorithm and lower memory costs13.
After establishing gmapping for our robot with default
parameters, real-world experiments were conducted inorder to determine the quality of mapping and© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan206

Neil Alishev, Roman Lavrenov, Yuriy Gerasimov

3. V. Khithov, A. Petrov, I. Tishchenko and K. Yakovlev,
Toward autonomous UAV landing based on infraredbeacons and particle filtering, Proc. 4th Int. Conf. on
Robot Intelligence Technology and Applications,(Springer International Publishing 2016), pp. 529-537.
4. M. Sokolov, R. Lavrenov, A. Gabdullin, I. Afanasyev andE. Magid, 3D modelling and simulation of a crawler robot
in ROS/Gazebo, 4th Int. Conf. on Control, Mechatronics
and Automation (2016), pp. 61-65.
5. E. Magid and T. Tsubouchi, Static Balance for Rescue
Robot Navigation-Translation Motion Discretization Issue
within Random Step Environment, Proc. Int. Conf. on
Informatics in Control, Automation and Robotics (2010),
pp. 415-422.
6. E. Magid and T. Tsubouchi, Static balance for rescue robot
navigation: Discretizing rotational motion within random
step environment, Int. Conf. on Simulation, Modeling, andProgramming for Autonomous Robots (Springer, Berlin,
Heidelberg, 2010), pp. 423-435.
7. Laserscan
matcher,ROS
package:
http://wiki.ros.org/laser_scan_matcher
8. Gmapping, ROS package: http://wiki.ros.org/gmapping
9. M. Quigley, D. Stavens, A. Coates and S. Thrun. Submeter indoor localization in unmodified environmentswith inexpensive sensors, IEEE/RSJ Int. Conf. onIntelligent Robots and Systems, (2010), pp. 2039-2046.
10. Google
Cartographer,
repository:
https://github.com/googlecartographer
11. Hector
SLAM,
ROS
package:
http://wiki.ros.org/hector_slam
12. M. He, E. Takeuchi, Y. Ninomiya and S. Kat, Precise andefficient model-based vehicle tracking method using RaoBlackwellized and scaling series particle filters, IEEE/RSJ
Int. Conf. on Intelligent Robots and Systems (2016),
pp.117-124.
13. A. Doucet, N. De Freitas, K. Murphy and S. Russell, RaoBlackwellised particle filtering for dynamic Bayesian
networks, Proc. 16th Conf. on Uncertainty in artificial
intelligence (Morgan Kaufmann Publishers Inc. 2000,
June), pp. 176-183.
14. N. Tsuda, S. Harimoto, T. Saitoh and R. Konishi, Mobile
robot with following and returning mode, 18th IEEE Int.
Symposium on Robot and Human Interactive
Communication (2009), pp. 933-938.
15. M. Awai, T. Shimizu, T. Kaneko, A. Yamashita and H.
Asama, HOG-based person following and autonomous
returning using generated map by mobile robot equippedwith camera and laser range finder, Intelligent
Autonomous Systems 12 (2013), pp. 51-60.

4. Conclusions and future work
Accurate mapping and localization is important for the
majority of tasks in the mobile robotics. In our research
we used laser range finder to perform simultaneous
localization and mapping. We used laser range finder
both for SLAM and for getting odometry information.
We demonstrated that this configuration is sufficient forgetting quite accurate mapping and localization results.
Fine tuning is required and it strongly depends on aparticular laser range finder and mobile basecharacteristics.
Unfortunately,undercertaincircumstances (e.g., in feature-poor environments)
laser_scan_matcher may fail and start to accumulate
odometry error. Therefore, as a part of our future work
we plan to obtain odometry information from crawler
encoders of the robot and to perform sensor fusion oflaser range finder odometry and encoders odometry. Thisshould result in better odometry accuracy, which is
crucial for mapping and localization of a robot.
After completing SLAM part, we plan to apply it to
the problem of robot autonomous return to the starting
point. This means that on the outward way the robot willrecord its’ position and the map of the environment, andnext, on the return way, it will autonomously navigate
itself to the starting point similarly to14,15.
Acknowledgements
This work was partially supported by the RussianFoundation for Basic Research (RFBR) and Ministry of
Science Technology & Space State of Israel (joint project
ID 15-57-06010). Part of the work was performed
according to the Russian Government Program ofCompetitive Growth of Kazan Federal University.

References
1. I. Afanasyev, A. Sagitov and E. Magid, ROS-based SLAM
for a Gazebo-simulated mobile robot in image-based 3Dmodel of indoor environment, Int. Conf. on Advanced
Concepts for Intelligent Vision Systems. Springer, Cham(2015), pp. 273-283.
2. A. Buyval, I. Afanasyev and E. Magid, Comparative
analysis of ROS-based Monocular SLAM methods forindoor navigation, 9th Int. Conf. on Machine Vision. –International Society for Optics and Photonics (2017), pp.
103411K-103411K-6.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan207

Toward automated open wound suturing using haptic feedback: detecting wounds andplanning the suture
Artur Sagitov1, Hongbing Li2, Evgeni Magid1
1Intelligent Robotics Department, Higher School of Information Technology and Information Systems, Kazan Federal
University, Kremlyovskaya str. 35, Kazan, Russian Federation2
Department of Instrument Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
E-mail: sagitov@it.kfu.ru, lihongbing@sjtu.edu.cn, dr.e.magid@ieee.org
http://kpfu.ru/robolab.html

Abstract
One of the key disadvantages of robotic surgery as of today is the lack of haptic feedback. While in traditional surgery
surgeons using their haptic senses in all tasks most medical robot lack this ability. Using robots with haptic feedback
have the potential of reducing the time spend by surgeon on such tedious subtasks as suturing helps reducing surgeon
fatigue. Also decreasing the manual input of the surgeon enables remote surgery even using long communication
links with big latency. In this paper we present a framework of wound detection and suture planning. We plan toimplement and test our algorithms using KUKA iiwa manipulator.
Keywords: Robotics, robot assisted surgery, medical robotics, manipulator, algorithm, modelling, trajectory.

being planned and executed under surgeon expert
supervision.
We target to implement procedures of automatic
robot-assisted surgical suture using KUKA iiwa robot,
which includes haptic feedback sensors that could
provide significant contribution to interactions with soft
tissue. We plan a line of sutures that holds a wound edges
in approximation during a healing procedure, consistingof a continuous strand of material or a series ofinterrupted suture strands. The goal of this paper is toestablish a procedure for automatic wound detection andintervention planning that is a first step toward the suture
automation.
The rest of our position paper is organized as follows.
We overview several existing techniques in Section 2.
Section 3 presents suture factors and Section 4 describes
suture models. Suturing procedure and parameters are
discussed in Section 5. Wound detection and constructing

1. Introduction
Injury to tissues of a body, which are generally caused byphysical means with tissue continuity destruction, isdefined as a wound. In cases when a tissue has beendamaged severely up to the level that it cannot be healed
without intervention (i.e., without complications), a
complicated treatment procedure is applied. During theprocedure dead tissue (and possibly foreign objects)
should be removed, infection should be treated, and thetissue should be maintained in apposition until thehealing processes provides the wound with sufficientstrength to withstand mechanical stress without
additional support (i.e. sutures, staples, strips, or topicaladhesives). Intervention allowing the wound to go
through a repair is referred as wound suturing. Suturing
is a challenging and time-consuming task. Existing
manual suture techniques could be used for automatic
robot-assisted surgical suture, with all steps of suture© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan208

Artur Sagitov, Hongbing Li, Evgeni Magid

multiple types of needles, the algorithm should integrate
needle parameters into a preplanning procedure.
Suture. The suture is a strand of material being used toapproximate tissue. There are several parameters for
sutures, such as a suture size, a tensile strength, and anumber of strands. The suture size is stated numerically;
as a number of zeros in the suture size increases, thediameter of the strand decreases. With the size decrease
suture the tensile strength decreases. Sutures areclassified according to a number of strands of which they
consist. Monofilament sutures are made of a single strand
of material; they encounter less resistance as they passthrough tissue than multifilament suture material.
Multifilament sutures consist of several strands, twisted
or braided together to increase tensile strength, pliability,and flexibility.
Knot tying. Each knot that is placed forapproximation of tissues or ligation of vessels should betied with a high precision and each should hold with aproper tension. A particular knot type to be appliedlargely depends upon the used suture, the depth, and theamount of stress that will be placed postoperatively.
When the knot has been securely tied, the ends must be
cut in a way that they are left relatively long,
approximately 6mm from the knot.

suture exit/entry points algorithm is described in Section
6. Section 7 briefly explains suture plan supervision andfinally we conclude in Section 8.
2. Related Work
Autonomous suturing using various surgical robots is a
thriving research field. Previous studies examineddifferent separate parts of suture, e.g., such as knot
tying1,2,3, thread tracking4,5 and others, or described an
entire suturing procedure. R. C. Jackson and M. C.
Çavuşoğlu6 described development of an autonomous
needle path planning during a simple interrupted suture.
Using kinematic model of needle-tissue interaction andsurgeon specified entry/exit points, S. A. Pedram et.al.
developed an algorithm for optimal selection of a needle
diameter, shape, and path7. Using a mechanical needle
guide S. Sen et.al. developed a framework for optimizing
a needle size, trajectory, and control parameters4. All
related work required surgeon input in selecting a suture
path with a wound width and depth. T. Osa et.al.
proposed an online trajectory planning and a force
control system for surgery task automation5 that could beapplicable in real time suturing.
3. Suturing factors
Our research focus is on skin wounds. Skin is composedof an epithelium and an underlying dermis, which require
sharp needle for every stitch in order to minimize tissue
trauma. Sutures need only to be strong enough for natural
skin tension withstand while holding wound edges inapposition. A suturing technique for skin closure may beeither continuous or interrupted, but the interrupted
technique is usually preferred8. Each individual suture
strand is passed through the skin only once, thus reducing
a chance of a cross-contamination across the entire sutureline. We emphasize several factors that affect successful
suturing intervention:
Closing tension. Tension in the suture must be
sufficient to approximate tissue and eliminate dead space.
At the same time, the tension should be loose enough toprevent patient discomfort, tissue ischemia and necrosis.
Thus, there should be a proper control for closing tension.
Needle. During a placement of sutures in tissue procedure,
surgical needle is attached to the suture strand in order topermit repeated passes through the tissue. As there are

4. Sutures model
There are several types of suture techniques. With regard
to continuity we distinguish continuous and interrupted
sutures.
Continuous sutures are made with one strand that isconnected to itself at each end, or with both ends being
connected together. The mechanical strength comes fromthe tension distributed over all strand length. Thus, weshould produce enough tension, while avoiding excess
leading to tissue strangulation. Excessive tension can also
lead to full suture failure.
Interrupted sutures combine multiple strands to close
a wound with each stich being tied separately after theprocedure completion. This provides a more secure
closure, as while individual stitch breaks, the rest of thestitches would hold the wound edges close to each other.
Other types of primary sutures, such as deep sutures,buried sutures, purse-string sutures, and subcuticularsutures, are used for specific purposes.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan209

Toward automated open wound

Deep sutures are placed completely under an
epidermal skin layer. They may be placed as continuous
or interrupted sutures and are not further removed
postoperatively.
Buried sutures are sutures that are placed with a knot
protruding to the inside, under a layer, which is to beclosed.
Purse-string sutures are continuous sutures placedaround a lumen and tightened like a drawstring to invert
the opening.
The described above continuous and interrupted
sutures are in the focus of our long term research workand modelling.

intervention analysis. After completion of the plan suture
undergoes quality control by the surgeon.
6. Wound detection and constructing suture
exit/entry points
General recommendation for most areas of the body,
except a human face, is that sutures should be placed inskin 3–4 mm from a wound edge and 5–10 mm apart9.
As the wound is an irregularity of the skin, clustering willallow us to detect possible wound locations. We use
following successive image transformation to generate
wound borders and placing exit/entry points (Fig.2):
1. Apply low-pass filtering;
2. Extract cluster components using mean shift
clustering method;
3. From a wound cluster image extract points using askeleton transform;
4. Construct a wound centerline approximation spline;
5. Construct the upper and the lower wound suture
splines by placing points tangentially to local
vicinity by stich width;
6. Place exit/entry points on border splines;
7. Assign point order according to selected suturing
techniques;

5. Suturing procedure and parameters
Outline of suturing procedure is shown in Figure 1. After
capturing an image from a robot camera, we estimate
wound dimensional parameters. A supervising surgeon
then specifies a suture technique and material to be
applied. The robot proceeds with the generation of initialplan of an intervention, which consists of an exit/entry
and a knot point locations. After the surgeon finalizes theplan, the robot proceeds with individual point planningand procedure execution under the surgeon supervision.
Full sensor data of the process is recorded for further post

Fig. 2. Wound borders and exit/entry point evaluated step bystep for continuous suture

7. Suture plan supervision
After determining the entry/exit points we add knotting
points at the ends of each continuous stitch. The
combination of ordered list of entry/exit points and knots
comprise our initial suture plan.

Fig. 1. Automatic suture procedure© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan210

Artur Sagitov, Hongbing Li, Evgeni Magid

2. J. Van Den Berg, S. Miller, D. Duckworth, H. Hu, A. Wan,
X.-Y. Fu, K. Goldberg, and P. Abbeel. Superhumanperformance of surgical tasks by robots using iterative
learning from human-guided demonstrations, Proc. ofIEEE Int. Conf. on Robotics and Automation (2010), pp.
2074–2081.
3. H. Kang and J. T. Wen, Autonomous suturing using
minimally invasive surgical robots, Proc. of theInternational Conference on Control Applications (2000),
pp. 742–747.
4. S. Sen, A. Garg, D. V. Gealy, S. McKinley, Y. Jen and K.
Goldberg, Automating multi-throw multilateral surgicalsuturing with a mechanical needle guide and sequential
convex optimization, IEEE Int. Conf. on Robotics andAutomation (2016), pp. 4178-4185.
5. T. Osa, N. Sugita and M. Mitsuishi, Online TrajectoryPlanning and Force Control for Automation of Surgical
Tasks, IEEE Transactions on Automation Science andEngineering (2017), vol. PP, no. 99, pp. 1-17.
6. R. C. Jackson and M. C. Çavuşoğlu. Needle path planning
for autonomous robotic surgical suturing, in IEEE Int.
Conf. on Robotics and Automation (2013), pp. 1669-1675.
7. S. A. Pedram, P. Ferguson, J. Ma, E. Dutson and J. Rosen,
Autonomous suturing via surgical robot: An algorithm for
optimal selection of needle diameter, shape, and path, inIEEE Int. Conf. on Robotics and Automation (2017), pp.
2391-2398.
8. Ethicon Wound Closing Manual, Somerville, N.J.,
Ethicon, Inc. (1994).
9. N. B. Semer. Practical plastic surgery for nonsurgeons,
iUniverse (2007), ISBN 0595461891.

The initial suture plan that is generated by ouralgorithm is presented to a supervising surgeon. The
surgeonthen
canmodifytheplanbymoving/adding/deleting points/knots and changing theorder of stitching. It is possible to use such corrections toprogressively learn suturing procedure of a robot under
direct supervision. Thus, using artificial machine
intelligence, we can improve outcomes of surgicalprocedures and significantly reduce surgeon fatigue.
After a surgeon approval the assisting robot canproceed to plan and execute individual point operations.
The supervising surgeon continuously observes a current
action and a plan of next actions of the robot as well as
suture tension. Moreover, the surgeon is authorized tointerrupt and/or correct the robot at any moment. All
actions of the assisting robot and sensory data arecarefully recorded for future analysis.
8. Conclusions and Future Work
This position paper presented initial stages ofautonomous supervised suturing including a woundborder detection and exit/entry points generation, that are
followed by knot points allocation, being specific toparticular selected technique and parameters. Our future
work is to adopt the existing methods in needle path
planning and integrate them into our framework toevaluate performance on KUKA iiwa robot using haptic
feedback. Suture thread tension management using
haptic feedback and tracking a position or a deformationof the suture thread are also under consideration.
We plan to use ZED camera stereo vision toreconstruct a wound profile and to extend the planning
into a 3D space of tissue.
Acknowledgements
Part of the work was performed according to the Russian
Government Program of Competitive Growth of Kazan
Federal University.

References
1. A. Knoll, H. Mayer, C. Staub, and R. Bauernschmitt,Selective automation and skill transfer in medical robotics:
a demonstration on surgical knot-tying, The Int. J. of
Medical Robotics and Computer Assisted Surgery (2012),
vol. 8, no. 4, pp. 384–397.
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

211

Establishing Effective Teaching for Robotics: a comparison study of Bachelor students
participated in Introduction to Robotics course
Tatyana Tsoy, Leysan Sabirova, Mikhail Abramsky, Evgeni Magid

Higher Institute of Information Technology and Information Systems, Kazan Federal University,
35 Kremlyovskaya str., Kazan, Russia, 420008,
E-mail: {tt, sabirova, ma, magid}@it.kfu.ru
http://kpfu.ru/eng/itis

Abstract
Global demand in robotics specialists arose necessity of establishing educational programs in robotics, which sets
new challenges for modern educational system. In 2017 Kazan Federal University launched master’s program inIntelligent Robotics to produce specialists considering core engineering competences and such important skills as
self-motivation, critical thinking etc. This paper presents results of continuous survey among undergraduate studentsof Introduction to Robotics course that helps understanding dynamics of student’s motivation to study robotics,
attitude toward education methods to improve teaching strategies.
Keywords: Robotics, education, engineering education, master program, motivation.

highlighted history and current teaching environment inrobotics field in Russia to understand major problems
after considering which we developed robotics master’s
program at Higher Institute of Information Technology
and Information Systems (ITIS) of Kazan Federal
University (KFU)1. In September 2017 our team ofIntelligent Robotics Department (IRD) launched a new
track in Intelligent Robotics within Software Engineering
master’s degree program. All robotics courses are taught
in English, and the first semester includes such courses
as Introduction to Robotics, Introduction to Robotics
Operational System and Computer Vision.
In spring 2017, before opening the program, we had
conducted several surveys among bachelor students who
took Introduction to Robotics course in English to
capture drawbacks and receive feedback from students inorder to improve teaching methods and course modules1.
This paper presents results of our continuing research andthe rest of it is structured as follows: Section 2 describes
research method, Section 3 is focused on analysis and in
Section 4 we observe conclusions and future work.

1. Introduction
Global demand in robotics specialists increases every
year as robotics flourishes and integrates into more andmore fields of modern economy. Currently higher
educational institutions of Russia are facing problems inrobotics education: shortage of experts with special
education in robotics field, shortage of up-to-date
educational materials in Russian, shortage of highly
priced equipment etc.1,2,3 According to study carried out
by the HSE Institute for Statistical Studies andEconomics of Knowledge (ISSEK), over 60% of Russian
employers’ organizations in the field of robotics note thatthey are experiencing a lack of specialists, particularly
highly qualified researchers and engineers4. Those
contradictions between employers’ requirements andemployees’ expertise raise new challenges to prepare
effective and highly motivated engineers5, 6, 7.
Considering the fact that robotics is aninterdisciplinary field8,9 it is our task to develop amultifaceted educational program that would allow
students to diversify their educational activity anddetermine their area of interest. In our previous work we© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan212

Tatyana Tsoy, Leysan Sabirova, Mikhail Abramsky, Evgeni Magidsame time provide them with robotics terms anddefinitions in English.
The results demonstrate positive dynamics instudents’ comprehension (Fig.1a) of the material, though
we could observe subtle negative dynamics in confidence
when students talked in English. There were insignificant
fluctuations on the item “I do not want to participate inclass because it is conducted in English” (Fig.1b), which
demonstrates that for some students in the class English
was still a barrier to learn robotics through foreign
language. However, studying robotics increased students’motivation to improve their English skills by the end of
the course (Fig.1c)

2. Research Method
We implemented the same research methodology that we
had used in our previous work1 and provided students
with statements and questions on English language
comprehension, self-efficiency, active learning strategies,
significance of studying robotics, stimulation of learning
environment. Еeach item had 5-point scale with optional
answers - (1) SD, Strongly Disagree; (2) D, Disagree; (3)NO, No opinion; (4) A, Agree; (5) SA, Strongly Agree –which appear along X-axis in Fig.1-4. Y-axis of Fig.1-4
indicate percentage of respondents that selected thecorresponding options. Two surveys were conductedwithin the same semester: one after the first class (B1)and another after the end of the course (B2). Both surveys
were conducted in Russian language to guarantee that all
respondents understand the questions. Responses thatwere received in first survey depict students’ attitude
towards robotics and expectations from themselves andthe course. Results obtained after the course was finishedrepresent the student experience gained during the course.
Total number of respondents in the first survey was 37
and in second was 15. To follow the dynamics weselected responds of 9 students, which participated inboth surveys, and analyzed the results.
3. Analysis
The comparative analysis covers two check-points during
the same semester: after the first class (B1) and after thecourse was finished (B2). We compared results of survey
B1 with results of survey B2.
3.1. Student English language comprehension
Considering that Introduction to Robotics class was thefirst course taught in English for bachelors of ITIS andthe fact that 44,4% of students never took robotics related
courses in the past, there was positive dynamics inmaterial comprehension in English language in the class.
During the course Prof. Magid lectured in English
language, however students were allowed to talk, answer
questions and ask the lecturer to explain unclear terms
and material in Russian. Additionally, no strict English
grammar requirements were applied for students whentalking or making presentations. Such concessions
targeted to minimize limitations for students in course
participation, ease material understanding and at theFig.1 English language comprehension

3.2. Self-efficiency in studying robotics
By the end of the course there was no negative change inself-estimation on confidence of studying robotics, yet anumber of respondents thought that they could notunderstand difficult robotics definitions during the classincreased (Fig.2a). These results reinforce another graph(Fig.2b), which demonstrates that less students agreed on© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

213

Establishing Effective Teaching forthe statement that even though a home assignment wasdifficult, they did not do it; moreover, when homeassignments were too difficult for them, they did only
easy ones (Fig.2c). The survey showed that by the end of
the course the students’ applied more efforts in
implementing home assignments and in-class tasks.

Fig. 3 Active learning strategies

3.4. Significance of studying robotics
Unfortunately, there was negative dynamics in thinking
on significance of robotics in context of their futureoccupation (Fig.4a). Additionally, an equal number ofstudents thought that they could and couldn’t
independently program or model a robot, while by theend of the course the number of those who thought they
couldn’t - increased by 11,1%. This demonstrate that thestudents realized the complexity of robotics, which was apositive result of gaining knowledge, and more students
confirmed that robotics stimulated their thinking (Fig.4b).
Also there was slight positive shift in students thinking
that robotics is a team work item “4 – Agree” of 5-point
scale increased by 11,1%, and more students by the endof the course started considering robotics to be amultidisciplinary field.

Fig. 2 Student self-efficiency

3.3. Active learning strategies in studying robotics
4.

There was more motivation among students to learnmaterial and think on mistakes they had made in homeassignments or in-class tasks (Fig.3a), and there was asignificant increase in studying complicated concepts
when students searched sources by themselves to obtain
additional explanations (Fig.3b).

Conclusions and future work

The results of comparative analysis showed that bygiving students freedom to express their opinion either inEnglish or in Russian, we observed positive dynamics inclass content comprehension provided in English by theteacher, although students still felt less confident while
speaking English. For a small number of students theclass content understanding in English was still a barrier
to actively participate in the class. However, for otherstudents the class activities increased motivation (by the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan214

Tatyana Tsoy, Leysan Sabirova, Mikhail Abramsky, Evgeni Magidglobal market. In addition, we have conducted apreliminary research among robotics related companies
in Russia on required skills for a young roboticist. Next,
to research this topic in depth we plan to develop andlaunch a large survey among industry and academia
representatives in order to analyze and meet labor market
requirements by our robotics master program graduates,
both in Russia and abroad.
Acknowledgements
Part of the work was performed according to the Russian
Government Program of Competitive Growth of Kazan
Federal University.
References
1. T. Tsoy, L. Sabirova and E. Magid, Towards Effective
Interactive Teaching and Learning Strategies in Robotics
Education, IEEE 10th Int. Conf. on Developments ineSystems Engineering (2017), in press.
2. M. B. Dias, G.A. Mills-Tettey and T. Nanayakkara,
Robotics, education, and sustainable development, IEEE
Int. Conf. on Robotics and Automation (2005), pp. 42484253.
3. Z. Shiller, A bottom-up approach to teaching robotics andmechatronics to mechanical engineers, IEEE Transactions
on Education (2013), 56(1), pp. 103-109.
4. N.A. Shmatko and G.L. Volkova, Robotics: the need fororganizations in scientific personnel and competencies,
Science, Technology and Innovations series, Institute for
Statistical Studies and Economics of Knowledge, National
Research University – Higher School of Economics (2017),
URL: https://issek.hse.ru/data/2017/05/29/
1172143877/NTI_N_53_19052017.pdf
5. Kh.G. Tkhagapsoyev and M.M. Yakhutlov, To the
problems of engineering education in modern Russia,
Quality Management, IEEE Conf. on Transport andInformation Security, Information Technologies (2016),
pp. 224-227.
6. L. M. Ogorodova, V. M. Kress and Y. P. Pokholkov,
Engineering education and engineering in Russia:
problems and solutions, Engineering education (2012),(11), pp. 18-23.
7. I. A. Gurban and Jr, A. A. Tarasyev, Global trends ineducation: Russia case study, IFAC-PapersOnLine (2016),
vol. 49(6), pp. 186-193.
8. B. A. Maxwell and L. A. Meeden, Integrating robotics
research with undergraduate education, IEEE Intelligent
systems and their applications (2000), 15(6), pp. 22-27.
9. J. Han & R. Gao, Motivation and training program on
robotics education, Int. Conf. on Computational
Intelligence and Software Engineering (2009), pp. 1-4.

Fig.4 Significance of studying robotics

end of the course) to learn English language further.
Considering this, we included English language test for
master program entrance examinations.
Further self-efficiency in studying robotics overall
depicted no major change at the end of the course;
however, few issues were discovered that could be
related to difficult content of the classes for students.
Nonetheless, respondents used more active learning
strategies in studying robotics and spent extra time to findadditional information that could assist understanding thematerial. In cooperation with teaching methods andteaching environment (e.g., project work and problembased learning) robotics stimulated students to apply
more efforts in implementing home assignments and inclass tasks during the after-the-class time. As robotics
industry is yet not massively developed in Russia
comparatively to IT-leading countries and robotics field
is not widely highlighted among other engineering fields,
most of software engineering bachelor students are still
uncertain about robotics bright future and thus did notlink their future employment with robotics.
In our future work we plan to pursue our continuous
research and observations of student needs to improveteaching methods in order to make it easier for them tounderstand robotics. Our emphasis is to motivate students
in attaining skills required for robotics specialists in© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan215

Development of a heterogeneous aerial swarm control framework for forest management
Yuriy Gerasimov, Artur Sagitov, Evgeni MagidIntelligent Robotic Systems Laboratory, Higher Institute for Information Technology and Information Systems (ITIS),
Kazan Federal University, 35 Kremlyovskaya street, Kazan, 420008, Russian Federation
E-mail: yurger2009@gmail.com, sagitov@it.kfu.ru, dr.e.magid@ieee.org
http://kpfu.ru/robolab.html

Abstract
As the prevalence of UAVs is increasing, they are becoming more accessible for wider applications. Our interest is
in application of UAVs for forest management challenges including monitoring health and safety and commercial
exploitation in a sustainable manner. We propose a swarm control framework for managing a group of UAVs for
aforementioned tasks, including survey of tree health with infrared cameras and chemical sensors, detecting potential
risky situations of illegal logging, smoke and fires, and estimating potential volume measurements. The proposed
framework manages planning flight trajectories, sensor fusion and collaborative mapping. On the next stage, we plan
to simulate the framework in ROS/Gazebo environment, and further to implement a pilot project with a group of DJI
Phantom quadrotors and a large-size fixed-wing UAV.
Keywords: robotics, algorithms, swarm control, robotic framework, unmanned aerial vehicle (UAV)

In the literature, rotary-wing and fixed-wing UAVs
are both equally distributed among various case studies.
In the vast majority, off-the-shelf solutions are preferredover self-designed and self-constructed UAVs. Different
types of sensors are used in experiments: visible-red,green, and blue, multispectral invisible and near-infrared,
middle-infrared, thermal infrared imagery, and LRF4.
Forest applications span over different types of missions,
including available resources estimation, speciesclassification, spread of diseases mapping, fire and its
effects monitoring, spatial gaps quantification etc.
As UAVs applications continue to spread acrossdifferent domains, there is increasing need for control
framework that incorporates forestry specific mission
planning, sensor fusion and control framework. Using
heterogeneous aerial robotic swarm that could
successfully combine long endurance, range, andprocessing capabilities of mothership vehicle with low
cost, flexibility and maneuverability of UAVs swarms
was shown to be applicable for cooperative search,

1. Introduction
Over the last decade unmanned aerial vehicles (UAVs)
becoming increasingly used all over the world to collectdata for various applications including urban search andrescue1 and forest monitoring2, as a result of
miniaturization and cost reduction of GPS receivers,
inertial navigation system, computers, motors, batteries
and sensors for remote sensing.
UAVs are competing with aircrafts and satellites,
bringing the advantages, such as low material andoperational costs and flexibility in different sensors
loadout, i.e. choosing sensors according to the mission
objectives3. Moreover, data acquired with UAVs usually
have higher spatial resolution, which is essential forassess local-scale variation in forest stand and species
measures2. Disadvantages are limited flight time (current
systems typically have limited operation maximum of 12 hours) and payload capacity, limiting simultaneous
acquisition of the entire area of interest.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan216

Yuriy Gerasimov, Artur Sagitov, Evgeni Magid

acquisition, and tracking missions in urban environment5.
Therefore we recognize significant potential advantagesof such architecture in forestry applications, which could
reduce mission time and increase operational coverage.
The rest of this position paper is organized as follows.
We review common missions that influenced ourframework composition in Section 2. Section 3 outlines
applicable hardware architecture and in Section 4 we
focus on framework’s individual components, e.g.
assignment and tracking task status for individual UAVs.

necessary to predict fire spread direction and help plan
firefighting processes.
Forest health monitoring goal is to produce
distribution maps of dead trees and infestation levels to
support intervention by authorities.
3. Heterogeneous swarm structure
We investigate robot swarm that is composed of one
main vehicle (mothership) and multiple simple UAVs
(drones) being connected through wireless network. The
motherships act as data storage, recharging and sensor
fusion stations, assist negotiation between agents, andsupervise a mission while drone control is achieved using
cooperative mapping. Each set of drones is assigned to aparticular mothership. In the role of the mothership anaircraft, a ship or a ground vehicle could be used.
Drones are small light UAVs used for the datacollection and equipped with communication devices tocoordinate their operations locally. Commercial off-theshelf UAVs are usually not intended for group operation,and thus require hardware upgrades. We also assume thatan individual UAV is equipped with a low-level control
system that provides stability and altitude-hold capability.
The choice of deployed sensory loadout on UAVs
depends on specific mission planned and on required
spatial and temporal scales of the analysis.
The framework is designed in a way that each drone
could operate autonomously and recognize otherneighboring drones within the limited operational area inorder to form cooperative groups.

2. Typical UAV-assisted forest management
missions
Information about vegetation conditions of a forest isimportant both for monitoring protected areas and for
estimating potential economic value of the forest.
Sensors being used in majority of case studies are nearinfrared spectrometers and hyperspectral cameras6,7.
Estimation of dendrometric parameters missions
involve acquiring imagery from UAVs in order toestimate Lorey’s mean height, dominant height, stem
number, basal area, and stem volume estimation of stands
using various statistical models. This mission type is used
to estimate available resources, identify potential areas oflogging, and provide information on plant growth
dynamics and biomass allocation. In addition, it makes
possible to identify and quantify spatial gaps in foreststhat could indicate illegal logging activities.
Species classification goal is to differentiate forest
species and provide species distribution maps.
Knowledge of such distributions is an important
information, as it plays important role in sustainable
forest management, as some tree species provide higher
market value timber, influencing both economic value
and potential illegal logging. This information also
allows identification of invasion species, allowing earlierinterventions.
Post-fire recovery monitoring involves analyzing
burned areas and capturing ongoing processes of postfire recovery: agamic regeneration, deadwood dynamics,
and logging activities with or without log extraction8.
Forest fire measuring during an ongoing forest fire
UAVs could measure location and shape of the fire front,
its rate of spread, and the fire flame height using visualand infrared cameras4 and provide information that is

4. Framework structure
One common task for all above mentioned forestry
missions is mapping. Using collective sensory data, it ispossible to overcome the individual drone limitations andaddress the issue with environmental conditions
variability. SAGE experiment addresses this issue,
demonstrating the usage of a small UAVs group tocollectively monitor an agricultural field with weed
mapping10,11. Network composition is based uponheterogeneous aerial robot network architecture forsearch and tracking developed by Elston and Frew12.
Figure 1 depicts a schematic representation of aheterogeneous swarm control architecture. The highest
level is mothership-drone coordination level that is used
to allocate specific tasks through negotiation between© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan217

Development of a heterogeneous

infrared beacons, e.g.13,14. Localization during landing
could be assisted using a variation of the method by
Yakovlev et.al.15. The same landing procedure is
triggered by a low level of drone battery charge.
We also use data preprocessing that keeps tracking
drone’s data quality rating. As each data chunk contains
drone signature, if during data preprocessing we
identifying problem data (e.g., corrupted data orcontradictory data about a same particular map cell thatcomes from two different drones ) we consequently marka map cell as unexplored and add it to a task list withlowering drone data quality rating.
4.2. Mission planning
Mapping is achieved using task assignment based on adecomposed coverage map. The map is decomposed into
cells, which are defined by its boundary coordinates anddata acquisition parameters (e.g., velocity, flying altitudeetc.). The division of the mission area into cells simplifies
motion planning of each agent, and it insures that no two
agents inspect the same cell at the same time.
Mothership uses a network to assign drones withresponsibilities for free cells by transmitting cell
information. We treat drones that inspect cells as a simple
collision-free model in C-space while agents are treatedas points simplifying collision prevention strategy.
Covering mission area effectively with our drones
involves two stages - area decomposition into cells andpath planning inside the cells. Our framework is being
developed with modularity in mind so it supports
different methods of decomposition and path planning inindividual cells.

Figure 1. Framework structure

mothership and individual drones. When the mission type
and parameters are specified mothership is issuingsubtasks that are derived from the planning layer on themothership vehicle to the trajectory generation layer oneach drone. Here, each task is converted into a trajectory,
which is passed to the lowest Flight Control Layer that
performs low-level drone control. This decomposition isminimizing the computations performed by drones. The
mothership has a GPS-receiver and can plan its activities
on the global scale, including cooperation with othermotherships of a large fleet.
While
payloadcapabilities
ofUAVs
fundamentally limit amount of sensory information,communication, and processing that can be acquired andcarried out, a size of the mothership vehicles allows themto carry much complex systems that can perform
communications over larger distances, store and process
more information than individual drones. Consequently,
swarm architecture utilizes technical advantages of amothership to minimize the amount of communication,planning, and coordination performed by drones.

4.3. Collective Adaptive Mapping
Some of mission objectives require a precise mapping
only for particular affected areas, so a uniform coverageof the environment becomes ineffective. Therefore we
add adaptive strategies to increase efficiency of a swarm
by allocating more drones to such areas, while other areas
could be only monitored superficially or even entirely
skipped. At first mission, the exploration area is divided
into multiple cells and each cell is coarsely inspectedwith a fast flyby by a drone, with interesting areas being
added to the map and communicated to the mothershipand other drones. The mothership issues special orders to

4.1. Mission control
A mothership keeps tracking drones health (flight hours,
battery status, etc.). When amount of data acquired by adrone is reaching drone memory capacity limit, the drone
returns to the mothership to perform fast data transfer forstorage and processing. Drone is landed on themothership using methods of autonomous landing with© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan218

Yuriy Gerasimov, Artur Sagitov, Evgeni Magid

5. D. Gross, S. Rasmussen, P. Chandler, and G. Feitshans,
Cooperative operations in urban terrain (counter), Defense
Transformation and Network-Centric Systems 6249 (Proc.
SPIE, 2006).
6. J. R. K. Lehmann, F. Nieberding, T. Prinz, and C. Knoth,Analysis of Unmanned Aerial System-Based CIR Images
in Forestry - A New Perspective to Monitor Pest
Infestation Levels, Forests 6(3) (2015) 594–612.
7. R. Näsi, E. Honkavaara, P. Lyytikäinen-Saarenmaa, M.
Blomqvist, P. Litkey, T. Hakala, N. Viljanen, T. Kantola,
T. Tanhuanpää and M. Holopainen, Using UAV-Based
Photogrammetry and Hyperspectral Imaging for Mapping
Bark Beetle Damage at Tree-Level, Remote Sensing 7(11)(2015) 15467–15493.
8. I. Aicardi, M. Garbarino, A. Lingua, E. Lingua, R.
Marzano, and M. Piras, Monitoring Post-Fire Forest
Recovery Using Multitemporal Digital Surface Models
Generated from Different Platforms, Earsel Eproceedings
15(1) (2016) 1–8.
9. J.R. Martínez-de Dios, L. Merino, F. Caballero and A.
Ollero, Automatic Forest-Fire Measuring Using Ground
Stations and Unmanned Aerial Systems, Sensors (Basel)
11 (2011) 6328-6353.
10. D. Albani, D. Nardi and V. Trianni, Field Coverage andWeed Mapping by UAV Swarms, IEEE/RSJ Int. Conf. onIntelligent Robots and Systems (2017), pp. 1–7.
11. D. Albani, J. IJsselmuiden, R. Haken and V. Trianni,
Monitoring and Mapping with Robot Swarms for
Agricultural Applications, Intelligent Technologies forEnvironmental Monitoring Workshop, IEEE AVSS
Conference (2017), pp. 1–6.
12. J. Elston and E. W. Frew, Hierarchical distributed control
for search and tracking by heterogeneous aerial robot
networks, IEEE Int. Conf. on Robotics and Automation(2008), 170 - 175.
13. V. Khithov, A. Petrov, I. Tishchenko and K. Yakovlev,
Toward autonomous UAV landing based on infraredbeacons and particle filtering, Proc. 4th Int. Conf. on
Robot Intelligence Technology and Applications (Springer
International Publishing 2016), pp. 529-537.
14. W. Kong, D. Zhang, X. Wang, Z. Xian and J. Zhang,
Autonomous landing of an UAV with a ground-based
actuated infrared stereo vision system, IEEE/RSJ Int. Conf.
on Intelligent Robots and Systems (2013), pp. 2963-2970.
15. K. Yakovlev, V. Khithov, M. Loginov and A. Petrov,
Distributed Control and Navigation System for Quadrotor
UAVs in GPS-Denied Environments, Proc. 7th IEEE Int.
Conf. on Intelligent Systems, (Springer International
Publishing 2015), pp. 49-56.
16. M. Sokolov, R. Lavrenov, A. Gabdullin, I. Afanasyev andE. Magid, 3D modelling and simulation of a crawler robot
in ROS/Gazebo, Proc. 4th Int. Conf. on Control,
Mechatronics and Automation (2016), pp. 61-65.

inspect all interesting areas with more precision by
splitting interesting area into smaller cells. Drones thatare not performing high precision data acquisition could
be assigned to search the rest of the environment in orderto mark new potentially interesting areas.
5.

Conclusions

The designed framework of coordinated control for amothership and drones in heterogeneous swarm
architecture fully utilizes individual advantages of eachrobot type improving mission performance. This is aposition paper that overviews our framework, while
implementation of modules is an ongoing work. Our goal
in developing this framework is to demonstrateapplicability of aerial swarm robotics approach toforestry using heterogeneous robots, as forestry missions
are complex problems with large economic impact. We
intend to implement aforementioned missions in ROS
Gazebo simulator16 and use the simulated experiments
results to evaluate economic advantages and drawbacks
of a heterogeneous swarm robotics approach within theforestry industry.
Acknowledgements
This work was partially supported by the RussianFoundation for Basic Research (project ID 18-58-50008).
Part of the work was performed according to the Russian
Government Program of Competitive Growth of Kazan
Federal University.
References
1. E. Magid, T.Tsubouchi, E. Koyanagi and T. Yoshida,
Static balance for rescue robot navigation: Losing balanceon purpose within random step environment, IEEE Int.
Conf. on Intelligent Robots and Systems (2010), pp. 349356.
2. J. Zhang, J. Hu, J. Lian, Z. Fan, X. Ouyang and W. Ye,
Seeing the forest from drones: Testing the potential oflightweight drones as a tool for long-term forest
monitoring, Biological Conservation, 198 (2016) 60-69.
3. L.Tang, and G. Shao, Drone Remote Sensing for Forestry
Research and Practices, J. of Forestry Research 26(4)(2015) 791–797.
4. Y. Ban, S. Marullo and L. Eklundh, European Remote
Sensing: progress, challenges, and opportunities, Int. J. of
Remote Sensing 38(7) (2017).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan219

Implementation of ROS package for simultaneous video streamingfrom several different cameras
Ramil Safin, Roman LavrenovIntelligent Robotic Systems Laboratory, Higher Institute for Information Technology and Information Systems (ITIS),
Kazan Federal University, 35 Kremlyovskaya street
Kazan, 420008, Russian Federation
E-mail: safin.ramil@it.kfu.ru, lavrenov@it.kfu.ru
http://kpfu.ru/robolab.html

Abstract
Real time video stream capturing and processing is important for a variety of tasks in robotics. We created ROS
package that captures concurrent video stream from 4 different cameras of Russian mobile robot “Servosila” Engineer.
V4L2 API was used to configure video devices and to retrieve raw data from cameras. Memory mapping approachof mapping device buffers increased overall performance by eliminating redundant memory copies. We demonstrate
the comparison of our new package and OpenCV based package.
Keywords: Video Streaming, V4L2, ROS, C++11, OpenCV, mobile ground robot, experiments.

cannot stream video data from all cameras at one time.
Therefore, we created robot operating system (ROS)
package that enables streaming video from the fourcameras simultaneously. Our next goal is to stream realtime video data via a wireless connection and to develop
a server based on a real-time transport protocol (RTP)
that will enable clients to receive video data from therobot and improve teleoperation process as an operator
can receive more information about environment.

1. Introduction
Capturing video stream from a robot is an important task
for multiple purposes, e.g., visual simultaneous
localization and mapping (SLAM)1 in path planning2,
human-robot interaction3, teleoperation in urban searchand rescue4, etc. In some cases, it is necessary to retrieve
and process video data in a real-time mode. Moreover,while single cameras are used for monocular SLAM
algorithms5 for UAVs and simple robots, for moresophisticated robots multiple cameras could be used, forexample, as a stereo pair in order to implement SLAM.
Thus, as these algorithms may request large computing
powers a mobile robot may require transferring sensory
data to a more powerful computing device for
information processing and analysis.
In our ROS package development process, we used
Russian crawler robot Servosila Engineer6. It has fourcameras and client–server application programming
interface (API). Even though it is possible to switch
between the cameras using original GUI of the robot, it

2. Vision-related features of the robot
A crawler-type mobile robot Engineer (Fig.1) is equippedwith four cameras that provide good situation awareness.
Three of the four cameras in the robot head are locatedon the front side and one is a rear view camera:
frontal optical zoom camera
frontal wide-angle cameras pair (stereo vision)wide-angle rear view camera© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan220

Ramil Safin, Roman Lavrenov

The installed operating system (OS) is Ubuntu 14.04 LTS(Trusty Tahr). The CPU is Intel® Core™ i7-3517UE(1.70GHz) with 2 physical cores.

Fig. 2. Common V4L2 video capturing application workflow

Secondly, device capabilities are queried. The
purpose of this step is to ensure whether device is able tostream and capture video. It is necessary to negotiate
exchanged data (image) format in order to avoid
ambiguity. In our case, Engineer’s three cameras (stereo
pair and rare view) support only Bayer pattern format8 asa raw data from the device’s driver. The optical zoom
camera’s raw data presents an image encoded in YUV
color space.

Fig. 1. Servosila Engineer crawler-type UGV

3. Video streaming ROS package
3.1. Streaming with V4L2 API
To capture video from the cameras, we use the second
version of Video4Linux API (V4L2 API), which is preinstalled within the OS by the maker. Programming a
V4L2 video device consists of the following steps7:
Opening the device
Changing device properties
Selecting a video standard
Negotiating a data format
Negotiating an input/output method
The actual input/output loop
Closing the device
Video device parameters are adjusted by input/output
control (ioctl) requests. It is a system call for
manipulating underlying device parameters (i.e. video
format, frames per second, etc.).
The workflow of typical V4L2 application (Fig. 2)
may vary depending on use case. For example, ourimplementation does not render captured frames. The
initial step is to open the device as it is needed in order tobe able to adjust device for streaming. In order to invoke
ioctl requests, device’s file descriptor should be opened
for reading and writing operations.

Fig. 3. The process of retrieving frames from V4L2 device
using special V4L2 API calls (placing buffer in the incoming
queue and getting raw data from outgoing queue)

After negotiating video streaming data format, device
buffers are allocated. A buffer contains raw image dataexchanged by the application and device’s driver using
streaming input/output (I/O) methods. In order to handle
captured video data, memory mapping (mmap) I/O
approach is used. It increases performance by eliminating
redundant memory copies from user to kernel space. The
application and the driver exchange pointers to buffers.
Hence, we can access captured video data directly inapplication memory.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

221

Implementation of ROS package

ROS image format (i.e. “bayer_grbg8”).
Device path (i.e. “/dev/video0”).

Then, we need to query buffers in order to obtain
information about allocated buffers, such as memory
location (pointers) and buffer length (size).
The process of getting recent frames consists of thequeuing and dequeuing parts (Fig. 3). Special API calls
are invoked (VIDIOC_QBUF and VIDIOC_DQBUF).
Queue operation puts the buffer in the driver’s incoming
queue. The buffer will be waiting for the driver to fill itwith data. Dequeue operation is used to retrieve
processed buffer (with video data) from the outgoing
queue9.

Fig. 5. Our ROS video streaming package’s CPU usage
statistics (measured by the pidstat utility in the idle and nonidle modes)

3.2. ROS package implementation
Originally, no ROS distribution was installed on the
robot’s OS. Therefore, we installed Ubuntu 14.04
compatible ROS Indigo distribution. The architecture of
the package consists of the following main parts (Fig. 4):
ROS
publisher
node.
Each
cameraindependently publishes its raw image data.
V4L2 API layer. All low-level interactions withvideo devices are handled by V4L2 API.

Captured image is distorted, therefore, camera
calibration is an essential part of our future work plans.
At the development stage, several issues were faced.
The most important one consisted in the inability todetermine video frame’s captured time, which areimportant for stereo SLAM algorithms implementation.
V4L2 API gives the ability to determine frame’s captured
time by means of captured buffer structure’s parameter.
Originally, this parameter had an invalid state, and aproper solution was obtained from the OpenCV GitHub
repository10. The timestamp data of the buffer structure
should be extracted after dequeuing and before the next
queueing stage.

Fig. 4. Our ROS video streaming package architecture (ROS
publisher node, V4L2 API layer, raw image and camera
information topics)

Fig. 6. Average CPU usage of our ROS video streaming
package (in the idle and non-idle modes)

The ROS publisher node streams video data from aselected camera to the assigned topic. A subscribed node
could be a server or some image processing node. Also,
the following node’s streaming parameters are used:
Frames per second (fps). Video frames
publishing rate.
Image resolution.

All further experiments conducted on Servosila
Engineer’s OS.
The created ROS video streaming package shows
encouraging results (Fig. 5, 6). Note, that if there are no
subscribers to the video stream package’s topic CPU
usage tends to zero. Otherwise, if there is at least one© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

222

Ramil Safin, Roman Lavrenov

subscriber, then our package consumes approximately
4% of CPU resources. It should be mentioned that noimage data conversion is performed, and video data ispublished as a raw data by means of creating ROS image
message. Hence, we can decrease precious CPU usage
on a robot’s system, not wasting resource on image
conversion. Our package does not depend on the
OpenCV.

wireless connection. Our source code is available forpublic access on the GitHub version control system12.
Acknowledgements
This work was partially supported by the RussianFoundation for Basic Research (RFBR) and Ministry of
Science Technology & Space State of Israel (joint project
ID 15-57-06010). Part of the work was performed
according to the Russian Government Program ofCompetitive Growth of Kazan Federal University.
References
1. J. Fuentes-Pacheco, J. Ruiz-Ascencio and J. M. RendónMancha, Visual simultaneous localization and mapping: asurvey, Artificial Intelligence Review (2015), v 43(1), pp.
55-81.
2. E. Magid, R. Lavrenov and A. Khasianov, Modified
spline-based path planning for autonomous ground vehicle,
Int. Conf. on Informatics in Control, Automation andRobotics (2017), pp.132-141.
3. A. Karpov, S. Carbini, A. Ronzhin and J. Viallet, Twosimilar different speech and gestures multimodal
interfaces, Multimodal User Interfaces (2008), pp. 155184.
4. E. Magid and T. Tsubouchi, Static balance for rescue robot
navigation: Discretizing rotational motion within random
step environment, Int. Conf. on Simulation, Modeling, andProgramming for Autonomous Robots (Springer, Berlin,
Heidelberg, 2010), pp. 423-435.
5. A. Buyval, I. Afanasyev and E. Magid, Comparative
analysis of ROS-based Monocular SLAM methods forindoor navigation, 9th Int. Conf. on Machine Vision (2017),
pp. 103411K-103411K-6.
6. M. Sokolov, R. Lavrenov, A. Gabdullin, I. Afanasyev andE. Magid, 3D modelling and simulation of a crawler robot
in ROS/Gazebo, 4th Int. Conf. on Control, Mechatronics
and Automation (2016), pp. 61-65.
7. L. Yinli, Y. Hongli, and Zh. Pengpeng. The
implementation of embedded image acquisition based on
V4L2, IEEE Int. Conf. on Electronics, Communications
and Control (2011), pp. 549-552.
8. A. Lukin and D. Kubasov, High-quality algorithm forBayer pattern interpolation. Programming and Computer
Software 30.6 (2004), pp. 347-358.
9. V4L2 manual; https://linuxtv.org/downloads/v4l-dvbapis/uapi/v4l/v4l2.html
10. OpenCV
GitHubIssues,timestamp
issue;
https://github.com/opencv/opencv/issues/8763
11. GitHub
video_stream_opencv
ROS
package:https://github.com/ros-drivers/video_stream_opencv
12. LIRS video streaming ROS package; https://github.com/
chupakabra1996/lirs_ros_video_streaming

Fig. 7. Average CPU usage of OpenCV based ROS package incomparison with our implementation (in the idle and non-idle
modes) (using pidstat utility)

We reviewed other ROS video streaming packages.
The most interesting one seemed to be an OpenCV based
package11. Comparing CPU usage of this package to ourimplementation we concluded that OpenCV based one
uses considerably more computational resources (Fig. 7).
If we stream from more cameras, then CPU usage will
increase and computational power will not be enough forother applications. Also, it is important to note that in idle
mode, when there are no subscribers to the streaming
node, OpenCV based package continues to consume a lotof CPU resources.
4.

Conclusions and future plans

Streaming video from multiple cameras is an importanttask. The key point is not only to get frames from camera
but also to decrease the usage of system resources. In thispaper, we introduced video streaming ROS package
implementation. This package demonstrated relatively
low consumption of system resources in either idle or
non-idle modes. As our long-term goal, we plan todevelop an RTP server, which will enable to stream video
data from several cameras in a real-time mode via© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan223

Consumers Preferences and Purchase Intention on New Taste of Salted Duck Eggs
Shang-Hui Li

Department of Food and Beverage Management,
Far East University, Taiwan, R.O.C.
E-mail:a0956358700@gmail.com
Abstract
Egg is an indispensable food in people's daily life. Egg is rich in protein, easily digested by human body andmetabolism. In addition to direct consumption, people created many different processed egg products by thecharacteristics of eggs. The traditional method of making salted duck eggs is to place the eggs in salt water or coatedin laterite without any additional chemical composition. Except the eggs, this study joined other natural ingredients
To produce eight different flavors of salted duck eggs. In the formal test, 70 convenience sampling were taken from
Tainan area for sensory evaluation, 63 valid questionnaires, SPSS18.0 was used as the statistical analysis. Sensory
evaluation included the color, oil, smell, salty and taste of egg yolk and the salty, taste of protein. Moreover, with theoverall acceptance, purchase intention and willingness to buy again etc. The study results showed that cinnamon red
salted duck eggs and four-herb salted duck eggs are the highest acceptance to consumers, followed by the salted
citronella duck eggs. However, four-herb salted duck eggs dipped by the salted water are not favored by consumers.
Keywords: Duck Eggs, Laterite, Purchase Intention, Sensory Evaluation

1. Introduction

Salted Duck Eggs

Duck eggs are not less nutritious than eggs. Both arefilled with high-density cholesterol, which easily
digested by our body and metabolism. There are many
processing methods about duck eggs in China, the mostcommon style was being produced as the salted duckeggs. The traditional salted duck eggs are being coatedwith laterite and rice bran. The research team used thelaterite of Tainan General Hill to make the different
formula of salted duck eggs. There are two purposes forthis study, one is to understand the consumer preferences
for the new taste of salted duck eggs through sensory
evaluation, then to explore the consumers' purchaseintension if they buy the new taste of salted duck eggs.
In this study, duck eggs were added to the different
formulations, either being immersed by salt water orbeing coated with laterite for 20 days before theconsumer sensory evaluation. The research flowchart isshown as in Chart 1.

Different Formulas of Salted Duck
Eggs

Preferences Sensory Evaluation

Data Analysis

Results and Discussion

Chart 1. Salted Duck Eggs Test
Flowchart

2. Literature Review© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan224

Shang-Hui Li

medicine of traditional Chinese prescription, it is also
widely used in the modulation of folk food or drinks [5].

2.1 Duck Eggs
Usually, colored eggs (cyan or brown) have better priceand competitive advantages than white shell eggs in theegg market. In Taiwan, most of eggs laid by the brown
duck were the major resource of the salted duck eggs
rather than as the fresh eggs. Many duck eggs processing
businessmen expressed that it would have a high rate ofproduction if century eggs, preserved duck egg, made bycyan eggs, so they have a high preference on cyan shell
eggs [1].

2.4 Sensory Evaluation
Please According to the definition of Institute of Food
Technologists, IFT, the so-called sensory evaluation is touse of scientific and objective methods by human eyes,
nose, mouth, hands and ears to measure and analysis offood. In other words, sensory evaluation means to
combine five sensory systems, including sight, smell,taste, touch, and feel, with psychology, physiology,
physics, chemistry and statistics and other disciplines tomeasure and analysis of food in order to understand thehuman feelings or likes of these products and test thequality of the product itself [6]. For human, food is nolonger just for survival, it is a more importantconsideration to meet the human satisfaction [7].
Basically, sensory evaluation methods could be divided
into two major categories [8]. The first category is a typeof consumer-oriented evaluation either for the preferenceor for acceptance. The second category is for theexperimental analysis including the sensitivity test andanalysis of the differences and description. Hedonic test
refers to the consumer sensory evaluation is sensual, it
depends upon the consumer’s feeling to determine if
he/she likes it or not. Scoring method, simple comparison
method, ranking method, multiple comparative methods
are the common methods to evaluate whether the productis liked by consumers [9]. In this study, preference tests
were used for sensory evaluation.

2.2 Laterite
Laterite is generally applicable to many construction
projects due to it is sticky and tightly hold water, fertilizer,
good ventilation and will form a group easily. Clay mine
was composed of rocks and minerals in nature after longterm weathering formed by the relatively fine particles. It
can be called aluminum clay ore if the aluminum content
is higher, and called magnesia clay ore when high inmagnesium. It was laterite while in high iron manganese[2].
2.3 Seasoning
Kanehira, Taiwan endemic species, belongs to lauraceae.
Due to kanehira’s root flavor is similar with both Ceylon
and China cinnamon, it was often used as a substitute forcinnamon in Taiwan folk. Its roots, bark, and leaves havea special cinnamon and essential oil ingredients. It is oneof the most edible and medicinal species in Taiwan's
native plants [3].
Lemongrass, cymbopogon nardus rendlel, is a perennial
evergreen herb. Lemongrass was extensively cultivatedin mass tropical and subtropical regions. The whole grassof lemongrass can be used, which has a rich lemon flavor,
including fresh grass, dried leaves, and stems plant.
Lemongrass is applied widely as seasoning on soup andmeat in India, Vietnam, Thailand and other Asia
countries [4].
As for the ingredients of Chinese four-herbs, or ”Siwu”
includes prepared rehmannia root, ligusticum wallichii,debark peony root, and angelica sinensis. "Siwu" wasfirst recorded in the official medical book, "Taiping
Huimin Heji Fang" in Song Dynasty, China. For modern
clinical, "Siwu" often used to deal with blood deficiency,
women's irregular menstruation and dysmenorrhea andother issues etc. At present, "Siwu" is used as the basic

2.5 Purchase Intension
Purchase intension means that consumers are willing toconsider the possibility of buying products. The
consumer have the higher perceived value to the product,
the more intention to purchase the product [10]. The
intention to purchase is the most accurate predictor of
predictive buying behavior [11]. If consumers who holda positive purchase intention will form a positive
purchase promise [12]. [13] pointed out that willingness
to buy was the judgment to buy the agreed product again
for the current or possible demand. [14] proposed the useof perceived value to measure the purchase intention
measured by questions such as might purchase, want tobuy, consider buying. This study referred to the"purchase intention" of a total of 3 questions, the scale
was largely modified from the questionnaire by [15] to© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

225

Consumers Preferences and Purchase

test consumer intention to buy a new taste of salted duckeggs. Based on personal self-perception, questions were
evaluated on a scale of Likert five-point scale.
3. Research Method
The test is divided into preliminary tests and formal tests,
2 preparatory tests were immersed for both 16 days and
20 days in different taste of salted duck eggs. There were
24 kinds of samples for each preliminary test which areconducted with 20 persons who had evaluation
experience before. However, the formal test was basedon the Tainan participants, Taiwan, conducted thesensory evaluation by sampling distributed 70
questionnaires, and 63 valid questionnaires werecollected. The results of the evaluation were analyzed by
SPSS18.0 statistical software. And the most 8
appropriate products from the two preliminary test results
were conducted as the formal test sample. The evaluation
items were divided into appearance, color, egg yolk oil,smell, protein salty taste, egg yolk salty taste, protein
taste degree, protein tenderness, egg yolk softness, egg
yolk softness, egg yolk flavor, egg yolk flavor, the overall
acceptance, and purchase intention as well. Likert 5-point
scale was used for questionnaire scoring. 5 for the mostsalty, most concentrated or most favorite, 1 was the least
salty, the lightest or the least like. Demographic variables
subjected to gender, age, occupation, education level.

duck eggs (#514) wasin the lowest level of overall
acceptance.
Table 1: Overall Preferences of the Difference Analysis
Item

SampleNumberAverage Standard
deviation

F-testDistinctivenessOverall
Acceptance

348
514
872
165
122
951
423
946

63
63
63
63
63
63
63
63

3.21±1.180c
2.44±1.202a
3.24±1.254c
2.70±1.265ab
3.32±1.229c
2.92±1.182bc
3.21±1.109c
3.11±1.166bc

4.243

0.000***

1)*P<0.05 significant , **P<0.01 very significant, ***P<0.001unusual significant
2) 348(original salted duck egg),514(lemongrass salted duck eggs),872(cinnamon salted duck eggs), 165(Siwu salted duck egg),
122(original cyan salted duck egg), 951(lemongrass cyan salted duck eggs) ,423(cinnamon cyan salted duck).

5.Conclusion and Suggestion
References In this study, sensory evaluation preference
test was conducted to develop a new taste salted duck egg.
The conclusions are as follows:
1. Consumers’ evaluation on the siwu salted duck（#165）
was the lowest rating which showed that consumers did
not like siwu style. However, the siwu cyan salted duck（ #946 ） on the average number are more than 3,it
maybe the cooked siwu’s juice inserted into the duckeggs to stimulate the different flavor. Perhaps, theamount of siwu could be increased in the future or replace
other Chinese medicine, such as the “four gods” salted
duck eggs, Angelica salted duck eggs.

4. Data Analysis
Data sample structure included 31 males (48.4%) and 32
females (50%). In terms of age, most participants were
under age of 20, a total of 32 (50%), followed by 41 to
50 years old, a total of 13 people (20.3%). As foroccupation, to the largest number of students, a total of
32 people (50%), followed by the service industry, a totalof 11 people (17.2%). 29 participants graduated fromcollege (45.3%), followed by high school, a total of 19
people (29.7%). The overall preferences of thedifferences on innovative taste of salted duck eggs wereshown in Table 1. In the overall acceptance of thedifference analysis, the participants considered thelemongrass salted duck eggs (#514, mean 2.44) had thevery significant differences with the original salted duckegg (#348, mean 3.21), cinnamon salted duck eggs (#872,
mean 3.24),the original cyan salted duck egg (#122,
mean 3.32),and cinnamon cyan salted duck (#423, mean
3.32). Compared with the five, the lemongrass salted

2. Consumers are less favored on lemongrass salted duck,average number was less than 3（#514）, it probably
because of the lack of lemongrass flavor,
3. Suggest that it may increase the proportion oflemongrass into the salted duck in the future or replacedby other herbs such as vanilla grass.
4. The acceptance of cinnamon cyan salted duck（#423）
was also higher. it maybe the cooked cinnamon’s aroma
inserted into the duck eggs to stimulate the different
flavor.
5. Although the production process of laterite-coatedsalted duck egg was more complicated than without
laterite-coated. But, it is worthy to promote because it didenhance the flavor of salted duck eggs.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

226

Shang-Hui Li

15. Grewal, D., Krishnan, R., Baker, J. & Borin, N.
(1998). The effects of store name, brand name and pricediscounts on consumers' evaluations and purchaseintentions. Journal of Retailing, 74(3), 331-352. doi:
10.1016/S0022-4359(99)80099-2.

References
1. Lin, R. X., Du, S. H., Huang, Z. F., Liu, X. Z., Hu, Y.
H. & Chen, T. F. (2007). Study on consumers' cognitionand purchase tendency of general duck eggs and greenshell duck eggs. Livestock Research, 39(3), 149-163.
2. Huang, J. Y. (2005).Clay mine and life. Land
Resources Guide, 2, 58-59.
3. Yang, Z. C. (2010). Study on germination and storageof Kanehira seeds. Agricultural policy and agriculturalsituation.220,2017/6/15from
https://www.coa.gov.tw/ws.php?id=22227.
4. Zhang, L. R. (2011). Gramineous herb plantLemongrass. Taichung District Agricultural News,73,1316.
5. Qiu, X. L. (2010). Chinese medicine school-Siwu
Tang introduction. Pharmacist Weekly, 1665, 2017/6/30
from http://tcpa.taiwan-pharma.org.tw/node/5168.
6. Jellinek, G. (1985).Sensory evaluation of food. VCH
Publishers .Florida. U.S.A.
7. Yao, N. Z. (2002). Sensory evaluation introduction,
application and future development. Taiwan Food News,
192, 44.
8. Ou, A. S. M. (2003). Sensory Evaluation and Practice.
Wagner Publishing. Taipei.
9. Lin Q. S. & Ye R. Y. (2004). Application and practiceof sensory evaluation. Ruiyu Publishing. Pingtung.
10. Dodds, W. B., Monroe K. B. & D. Grewal. (1991).
The effects of price, brand and store information onbuyer’s product evaluations. Journal of Marketing
Research, 28(3), 307-319.
11. Morwitz, V. G. & Schmittlein, D. (1992). Using
segmentation to improve sales forecasts based onpurchase intent: Which "intenders" actually buy? Journalof Marketing Research. 29(4). 391-405.
12. Moorman, C., Zaltman, G. & Deshpande, R. (1992).
Relationships between providers and users of market
research: The dynamics of trust within and between
organizations. Journal of Marketing Research. 29(3).
314-328.
13. Hellier, P. K., Geursen, G. M., Carr, R. A. & Rickard,
J. A. (2003). Customer Repurchase Intention: A General
Structural Equation Model. European Journal ofMarketing, 37(11/12), 1762-1800.
14. Zeithaml, V. A. (1998). Consumer Perceptions ofprice, quality, and value: A meansend model andsynthesize of evidence. Journal of Marketing, 52, 2-22.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

227

The Study of Green Food Image, Satisfaction and Loyalty through the Perspective of
Elaboration Likelihood Model
Shu-Fang Hsu

Department of Food and Beverage Management,
Far East University, Taiwan, R.O.C.
E-mail:h99582004@gmail.com
Abstract
This study aims to explore the relevance of consumer satisfaction and loyalty to green food image through theperspective of Elaboration Likelihood Model The results show that the positive impact of green food image; and will
indirectly affect the satisfaction of re-purchase intention; satisfaction positive impact loyalty. We hope that the resultsof this study can provide relevant business management strategies and suggestions, the green food consumption
theory and practice can be dedicated.
Keywords: Green Food, Satisfaction, Loyalty, Elaboration Likelihood Modelconsumers considered that it is harmless, pollution-free,fresh, safe and nutritious in the food production,
manufacturing and sales process, and minimizes thedamage to the planet. Therefore, this study explored therelevance of consumer perceptions of green food tosatisfaction and loyalty under the perspective of
Elaboration Likelihood Model, meanwhile, explored the
mediating effect of satisfaction on green food image andloyalty.

1. Introduction
Many companies actively developed and manufacturedof green-related foods and delivered green food messages
to consumers via products, service packaging or
advertising. [1] studied food quality considered theintention of customer behavior was caused by customer
interaction. The purchase behavior of consumers was,therefore, gradually generated through many reactions.
[2] studied the concept of green practices arguing thatgreen affected customer attitudes and consuming
attitudes toward restaurants. And Elaboration Likelihood
Model (ELM) of [3] was used as the theoretical basis forthe study. ELM refers to the think way in our humanbrain which can be divided into the central path and thesurrounding path. Central path emphasizes the relevant
information processing, peripheral path refers to any
factors that can convince the central path. Satisfaction
and loyalty are the two most important variables ofconsumer behavior. As a result, many businesses arethinking about how to enhance consumer satisfaction andthereby increase their loyalty.
[4] pointed out that image refers to the internal beliefs
and impressions formed on the basis of individual
message processing. Image is an indirect experience,
however, satisfaction is a direct experience. And green
food was produced on the base of environmental
protection. This study defined the green food as

2. Literature Review
2.1 Elaboration Likelihood Model[5] put forward an "Elaboration Likelihood Model" forconsideration to know more about the formation of
consumption patterns, they tried to understand how todeal with the communicate of information and toconvince the effect of the consumers. Depend on the"Elaboration Likelihood Model", when consumers
received the various kinds of information and messages
from the outside world. The consumers would process
and evaluate this information in the form of central routeand peripheral route, then to make the final attitude or
opinion about a product or service. The degree which
consumers think about the message will influence how
they choose which message to evaluate. When consumers
think hard, it means that consumers invest a higher© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan228

Shu-Fang Hsu

cognitive mind. Consumers will carefully evaluate fromthe information related to the message, thus forming a
product attitude or view. The "Elaboration Likelihood
Model" has been widely used to apply the effects ofcommunication [6]; [7], such as the formation of wishes
and attitudes [8]; [9] and other issues.

according to the literature review, and in accordance withthe point of view of "Elaboration Likelihood Model". The
research framework was shown in Figure 1.

2.2 Satisfaction
The task of business is to create the satisfied customers.
Peter Drucker was the first to put forward the concept ofcustomer satisfaction. Satisfaction refers to the overall
assessment of the comprehensiveness of a service or
product generated by a customer after using the productor service.
[10] pointed out that consumer satisfaction came from theexpectations before the product was purchased and theactual perceived to the product features andcharacteristics after purchase and the happiness or
disappointment was formed after comparing the two. The
consumers felt dissatisfied if the result was not as goodas the expectations and consumers felt satisfied if theresult exceeded expectations. [11] studied the causal
relationship among destination image, attributesatisfaction, overall satisfaction and loyalty. The results
showed that destination image was divided into positive
impact on overall satisfaction and attribute satisfaction,
attribute satisfaction positively affected overallsatisfaction and loyalty, and overall satisfaction
positively affected loyalty.

H
H
1
2
gre
lo
H
en
ya
Figure 1. Research
3 FrameworkfoodltThis study,ima
therefore, proposed the following
researchyhypotheses:
ge
H1: Green food image positively affected satisfaction.

sati
sfac
tion

H2: Green food satisfaction positively affected loyalty.
H3: Green food image positively affected loyalty.
This study mainly explored consumer's issue of green
food image. Therefore, the consumers who had
purchased green food would be considered as the major
participants. Questionnaire was issued through internet
and distributed at the green food store. A total of 258
internet questionnaires were collected and 222 were
useful after excluding the invalid ones. As for the fieldsurvey, a total of 242 questionnaires were collected and
210 were the effective ones excluding the blank, chicken
writing and incompletely answers. In total, 432 valid
questionnaires were collected and the effective recovery
rate was 86.4%.

2.3 Loyalty
Loyalty is the preferred attitude and behavior of
consumers to purchased items. Consumers will collectand evaluate a variety of related product information andadvantages and disadvantages of product, then give thefinal decision to buy the product. How to form a purchaseintension, therefore, will be an important key. [12] foundthat loyalty was the customer purchased repeatedly,promised to introduce a friend to the company's products
or services, and is willing to persuade others to use thecompany's products or services, or voluntarily
recommend the company's products or are willing to help
improve their services [13].

4. Data Analysis
The obtained information was analyzed and stated on
statistical data after the formal questionnaires were
issued, then to test the structural equation modelaccording to research hypothesis and discuss theinfluence coefficient and the adaptability of the overall
mode among each variable. After the model analysis, theparameters of the overall model of this study can belearned from Table 1. The standardized coefficients ofthe potential variables of the overall model range wasfrom 0.74 to 0.87. None of them was greater or nearly 1
and standard errors were all between 0.010 and 0.027,
there were no negatives and all were significant.
Obviously, the study model did not produce an offending
estimate.

3. Research Method
The study explored the relationship between consumer
satisfaction and the loyalty of the image of green food

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

229

The Study of Green

Table 1: Standardized Parameter Estimates of Potential
Variables to Observed Variables
StandardizedParameter
Estima
tes
0.83*

T
Val
ues

Error
Vari
ation
Num
ber

T
Val
ues

Stan
dard
Erro
r

16.
49

0.67
*

5.4
6

0.02
7

Environ
0.87*
17.
0.75
mental12*
Safety
Meets
0.76*
17.
0.58
Require
56
*
ments
Wise
0.85*
20.
0.72
Satisfa
Choice
16
*
ction
Feel
0.85*
20.
0.73
Good
30
*
Feel
0.81*
18.
0.65
Happy
96
*
0.70
Repurch
0.84*
18.
*
ase
42
Intentio
n
Price
0.84*
18.
0.71
Toleran10*ceLoyaltyInitiativ
0.84*
17.
0.71
e to
89
*
Recom
mend
Cross
0.74*
16.
0.60
Purchas46*e* indicated the significant level at p = 0.05.

4.2
0

0.02
5

12.
26

0.01
7

10.
29
10.
13
11.
34
10.
85

0.01
3
0.01
3
0.02
0
0.01
0

Potential
Variab
les

Green
Food
Image

Observed
VariablesFresh &
Nutritio
n

10.
76

0.01
8

10.
80

0.01
3

12.
34

0.01
5

satisfaction. Namely, if the consumer's awareness ofgreen food was higher, the degree of satisfaction wasstronger. Therefore, H1 was established. The influence
coefficient of satisfaction on loyalty was 0.57 and t was
10.326, which also reached to a significant level. It was
clear that satisfaction was positive and significantly
affected loyalty. Namely, consumers were more satisfied
with green food image, and their loyalty is stronger.
Therefore, H2 was established. The coefficient of loyalty
for green food was 0.51 and t value was 8.709, reached asignificant level. Obviously, green food had a positiveand significant impact on loyalty. That was, the higher
consumers perceived the green food image, the stronger
their loyalty. Therefore, H3 was established. The direct
effects, indirect results, and hypothesis test of researchvariables results were shown in Table 2.
Table 2: The Direct Effects, Indirect Effects, andHypothesis Test of Research Variables
Indirect
Effects

Test
Results

0.57***－

0.51***－

H1
EstablishedH2
EstablishedH3
Established

Path

DirectEffects

Green Food
Image Satisfaction

0.41***

Satisfaction Loyalty
Green Food
Image Loyalty

5. Conclusion and Suggestion
The results showed that green food image positively
affects satisfaction and will indirectly affect the
repurchase intention through satisfaction. Satisfaction
positively affected loyalty. As about the impact ofsatisfaction on loyalty under the “Elaboration Likelihood
Model” showed that if the price of green food was higher
and the efficacy of the product could not be immediately
effective, then the consumers will perceive a higher risk.
Under such condition, the consumers will pay attention
to the product information or seek advice from others inattempt to reduce the purchase risk. At this point,
consumers mainly relied on central clues to evaluate
information, to think about the content of the productinformation in a rational way. If consumers perceived thereceived content as fair and persuasive, they would havemore positive thoughts about the product, and change
your attitude towards the product in the direction
described by product information. When consumers weremore satisfied with the green food image, their loyalty is

In this study, green food image was as an independent
variable, loyalty as a variable, the satisfaction was theintermediary variables of green food image and loyalty.
Based on this, this study used the structural equation
model to verify the path assumptions, to verify thecausality of satisfaction between green food image andloyalty. Satisfaction coefficient of green food was 0.41, t
value was 6.809, a significant level. Obviously, the green
food image had a positive and significant impact on the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan230

Shu-Fang Hsu

higher. Green food industries should commit to the
pursuit of a higher level of satisfaction in order toenhance consumers' positive consumption of green food.
Since satisfaction could affect loyalty for both directly
and indirectly. For the industry, therefore, themeasurement and promotion of satisfaction are important.
As [14] argued that the quality of the environment, food,and service are important determinants of restaurant
imagery. Food safety in Taiwan should promote "clean
green" instead of changing the public's response to theenvironment, food safety and quality. Hence, [15]suggested that industries should understand their market
segmentation with the right strategy and direction.
Finally, the industries need to define effectively theimage of green product in the minds of consumers.

8. Bhattacherjee, A., & Sanford, C. (2006). Influence Processes
for Information Technology Acceptance: An Elaboration
Likelihood Model. MIS Quarterly, 30(4), 805-825.
9. Agarwala, R., Angstb, C. M., & Magni, M. (2009). The
performance effects of coaching: a multilevel analysis using
hierarchical linear modeling. The International Journal of
Human Resource Management, 20(10), 2110–2134.
10. Kotler, P. (2000). Marketing Management: The Millennium
Edition. (10th ed.). NJ:Prentice Hall.
11. Chi, C. G. Q., & Qu, H. (2008). Examining the structural
relationships of destination image, tourist satisfaction anddestination loyalty: An integrated approach. Tourism
Management, 29(4), 624-636.
12. Lai, F., Griffin, M., & Babin, B. J. (2009). How quality,
value, image, & satisfaction create loyalty at a Chinese telecom.
Journal of Business Research, 62, 980-986.
13. Heskett, J. L. (2002). Beyond Customer Loyalty. Managing
Service Quality, 12(6), 355-357.
14. Ryu, K., Lee, H. R., & Kim, W. G. (2012). The influence ofthe quality of the physical environment, food, and service onrestaurant image, customer perceived value, customer
satisfaction, and behavioral intentions. International Journal ofContemporary Hospitality Management, 24(2), 200-223.
15. Noor, M. N. M., Masuod, M. S., Said, A. A., Kamaruzaman,
I. F. & Mustafa, M. A. (2016). Understanding Consumers andGreen Product Purchase Decision in Malaysia: A Structural
Equation Modeling - Partial Least Square (SEM-PLS)Approach. Asian Social Science, 12(9), 51-64.

References
1. Cheng, F. C., Cheng, F.Y. & Fan, W. S. (2013). Factors
Affecting Customer Behavior Intentions Toward Taiwan
Restaurant Industry Using Response Hierarchy ModelDiscussion on Brand Image as Moderating Variable. Journalof Sport, Leisure and Hospitality Research, 8(4), 4976.
2. Jeong, E., Jang, S. C., Day, J., & Ha, S. (2014). The impactof eco-friendly practices on green image and customer
attitudes: An investigation in a café setting. International
Journal of Hospitality Management, 41, 10-20.
3. Petty, R. E., & Cacioppo, J. T. (1981). Attitudes andPersuasion: Classic and Contemporary Approaches. Dubuque,
IA: William C. Brown.
4. Suh, J. C., & Yi, Y. (2006). When brand attitudes affect thecustomer satisfaction-loyalty relation: The moderating role ofproduct involvement. Journal of Consumer Psychology, 16(2),145-155.
5. Petty, R. E., & Cacioppo, J. T. (1996). Attitudes andPersuasion: Classic and Contemporary Approaches. Boulder,
CO: Westview Press.
6. Rucker, D. D., & Petty, R. E. (2006), Increasing theeffectivenessofcommunications
toconsumers:
Recommendations based on elaboration likelihood and attitude
certainty perspectives. Journal of Public Policy & Marketing,25 (1), 39-52.
7. Tormala, Z. L., & Petty, R. E. (2004). Source Credibility andAttitude Certainty: A Metacognitive Analysis of Resistance to
Persuasion. Journal of Consumer Psychology, 14(4), 427-442.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan231

Self-Generated Dataset for Category and Pose Estimation of Deformable Object
Yew Cheong Hou, Khairul Salleh Mohamed Sahari*

Department of Mechanical Engineering, Universiti Tenaga Nasional
Selangor, Malaysia
E-mail: ychou_my@hotmail.com,:*khairuls@uniten.edu.my

Abstract
This work considers the problem of garment handling by a general household robot that focuses on the task ofclassification and pose estimation of a hanging garment in unfolding procedure. Classification and pose estimationof deformable objects such as garment are considered a challenging problem in autonomous robotic manipulation
because these objects are in different sizes and can be deformed into different poses when manipulating them. Hence,
we propose a self-generated synthetic dataset for classifying the category and estimating the pose of garment using asingle manipulator. We present an approach to this problem by first constructing a garment mesh model into a pieceof garment that crudely spread-out on the flat platform using particle based modeling and then the parameters suchas landmarks and robotic grasping points can be estimated from the garment mesh model. Later, the spread-out
garment is picked up by a single robotic manipulator and the 2D garment mesh model is simulated in 3D virtualenvironment. A dataset of hanging garment can be generated by capturing the depth images of real garment at the
robotic platform and also the images of garment mesh model from offline simulation respectively. The synthetic
dataset collected from simulation shown the approach performed well and applicable on a different of similar garment.
Thus, the category and pose recognition of the garment can be further developed.
Keywords: Deformable object, robotic manipulation, computer vision, particle based model.

category and recognizing the pose of hanging garment
grasped by single robotic manipulator. The main idea of
this work is to generate a dataset of hanging garment thatcombine the real and synthetic images from the real
garment and deformable model respectively. The
deformation poses of the garment can be reduced byhanging the garment under gravity. A dataset of hanging
real garment by robotic manipulator can be obtained bycapturing the depth images. For deformable object
modeling, the particle based model is constructed to thegarment in an image and then used into the simulation toacquire the synthetic dataset. This work is extended toinclude the 2D garment mesh model that able to represent
the real garment that not found in the database. The
dataset will take advantages of this algorithm to generatelarge number of data for learning purposes without using

1. Introduction
Handling deformable object by robot is a challenging
problem in robotic research. To teach a generalhousehold robot to handle these deformable objects isdifficult compared with rigid object due to the variety ofsizes, textures and poses for these objects. Figure 1 shows
a pipeline for the autonomous garment handling by arobot. In order to manipulate it from initial state to final
desired state successfully, the category and pose of thegarment should be recognized by the robotic system. The
computer vision technique integrated with the robust
machine learning algorithm through garment perception
should be acquired by a robot to manipulate these objects
dexterously by recognized their deformed configuration.
In this research, we are focusing the task of classify the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan232

Yew Cheong Hou, Khairul Salleh Mohamed Sahari

has gained attention and increase significantly. The endto-end task of garment folding had been decomposed into
several sub-tasks, the existing approaches had been
proposed from different aspects to solve the particular
problem in order to achieve its accuracy and robustness.
Generally, majority contributions in the literature ofgarment handling mainly focus on the grasping skills,
recognition, and motion planning strategy of unfolding
and folding.
There has been previous work on the recognition andmanipulation of deformable objects. Doumanoglou et al.
proposed a dual-arm industrial robot to unfold different
garments such as sweater and trouser1. They focused onclassification and pose recognition of hanging garment
and then proceeded to unfold it into desired configuration.
Mariolis et al. proposed an approach of recognize thepose for garment using convolutional neural network.
Their model is trained by the combination of real andsynthetic dataset in 2D garment model2. Li et al.
presented a series of laundry works in recognition andmanipulation of garment. Initially, the model-driven
approach based on precomputed deformable model ofhung garments has been proposed. A set of deformablemodels had been trained using the method of Support
Vector Machine (SVM) to recognize the category andpose of garment3. They improve their recognition
approach using Kinect Fusion technique to reconstruct a
deformable 3D model4.

Fig. 1. The pipeline for garment handling in autonomous
robotic manipulation.

professional graphic software. The main contribution inthese works are:
The simple 2D garment mesh model is extracted
from the real garment that crudely spread-out on theflat platform. The multiple poses of hanging garment
model under gravity are simulated, and different
viewpoints of the model are captured as an image.
Different types and sizes of garment are simulated
and the dataset is generated from the synthetic dataof hanging garment mesh model integrated with thereal data.

3. Particle Based Polygonal Model
We suppose a piece of garment that is crudely spread-out
and uniformly placed on a flat platform. We perform acomputer vision analysis to a single color image that
captured from the top view from the garment. By
applying the dynamic programming algorithm, the real
garment is matched by the particle based polygonalmodel and its parameters are extracted. The flowchart forthe model generation is described as shown in Figure 2.

2. Related Work

3.1. Simulate the Hanging Garment

Modeling and simulation are commonly used to computethe deformation of the deformable objects and becomes
essential in future autonomous robotic manipulation
since the result produced from simulation is seeming real
and applicable. In robotic manipulation, the research on
autonomous laundry service for general household robot

The particle based modeling is applied into the dynamic
programming and used to simulate the movement anddeformation of the garment mesh model. The approachin the previous work is adopted and the mass-springmodel is utilized in this work5. The hanging process ofgarment mesh model generated from a real garment is© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan233

Self-Generated Dataset for Category

simulated as shown in Figure 1, where a piece of crudely
spread-out garment mesh model is placed flat on thesimulated environment. For each step, a specified particle
from the garment mesh model will be picked up by the
robot as a grasping point and then moved to the hanging
position following the simulated trajectory.

of images are captured for each pose of garment, themodel is rotated vertically with 360 degrees and thevirtual camera captures the image of the rotated garment
by increasing one rotating degree until full rotation isachieved.
4. Preliminary Results

3.2 Generate Synthetic Dataset

4.1 Robotic Manipulation Setup

One of the hardest problems need to be solved in deeplearning is to collect the large scale data into the desired
format. Collecting the data for different categories of real
garment are very time consuming particularly
encompassing the different types, sizes and material
properties with its deformation. Hence, a training set thatgather relevant information of garment which can be usedin visual recognition purposes is required. The features
such as colors and textures of the garments are lesssignificant in this work. The dynamic programming
modeling is applied to the garment mesh model, theRobotic manipulations have been conducted using a
Baxter robot with two Kinect depth sensor. The depth
images of the hanging garments have been acquired by a
Kinect depth sensor placed in front of the Baxter robot at
a fixed height. The second Kinect sensor is placed at thehead of Baxter robot which used to capture the images
for a piece of crudely spread out garment on the flat
platform.
4.2 Real and Synthetic Dataset
To collect the real garment dataset, a piece of the garment
placed on the flat platform is hung by a Baxter robot anda Kinect depth sensor located in front of the Baxter robot.
The hanging garment is then rotated 360 degree verticallyand the color and depth images are captured. As shownin Figure 3, the threshold of the depth images are
determined in order to distinguish the garment from thebackground.
The 200×340 pixels depth images that only consists of apiece of hanging garment are extracted from 640×480
original images. For category classification, three
categories which are towels, shirts and trousers are
selected in this current work. A total of 9 different real
garments have been used as the samples, 3 real garments
for each category. Each garment have been hung 2 or 3
times and about 360 depth images have been acquired foreach grasping point. For pose recognition, 21 poses for
shirts, 15 pose for trousers and 9 poses for towels are
selected by defined by the markers. Total around 48600
depth images including towels, shirts and trousers arecollected, the procedure of image acquisition isconsidered slow and time consuming.
For synthetic dataset, a large dataset of synthetic depth
images are constructed by using computer graphic
software and dynamic programming algorithm
respectively. Using Blender software, the total 9 garment
models which 3 models of towels, 3 models of trousers
and 3 models shirts are constructed with different in size

Input Image
Segmentation and Contour Finding

Category Classification

Garment Discretization and Particle Mesh Fitting

Mode Matching and Combination

Simulate the Hanging Garment
Generate Synthetic Dataset
Fig. 2. Flowchart for the particle based polygonal model
generation for synthetic dataset.

deformation poses of the model hanging by robotic
manipulator is simulated. By applying different physicalproperties of the garment mesh model, the model lookvisually similar with depth images can be produced fromthe virtual simulation. An amount of images that simulate
the deformation of real garment can be acquired. A series© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

234

Yew Cheong Hou, Khairul Salleh Mohamed Sahari

and shape. To simplify task in designing the model insimulation, the garment models are constructed in 2D byassuming the front and back sides of shirt and trouser are
not separated during grasped by robotic arm. In addition,
the 2D garment model also can be generated by using the
proposed dynamic programming algorithm. The particle
based polygonal model can be generated from a crudely
spread out garment placed on the flat platform.
The towel models consists of 81 particles, the trouser
models consists of 121 particles and the shirt model
consists of 209 particles are constructed. Due to thesymmetry axes of the garment and reduce the dense of
the particles in the model, only selected particles in thegarment model that matching with the markers in the real
garment are simulated. The hanging garment models canbe simulated by each of their particles. Different views of
the hanging garment model can be acquired by rotating
the model 360 degree in front of virtual camera thatplaced at fixed height as similar as possible with the setup
of Kinect sensor for image acquisition in robotic platform.
The approximately 97200 virtual depth images have been
generated. For synthetic dataset, the condition in thevirtual environment is set as close as possible to the depth
images in real dataset. Hence, the images acquired fromthe virtual simulation are comparable with real depth
images, then allow to use as training dataset in thelearning model.

polygonal model, then the different poses of hanging
garment can be simulated in the virtual environment.
Experiment results demonstrate the proposed approach isapplicable in dataset collection that need a large volumeof data to train the learning model. By using this
proposed approach, different types and sizes of garment
mesh model can be extracted and simulated rather thandesign them in the graphics software.
Acknowledgements
This work is supported by the Ministry of Higher
Education, Malaysia through research grant:
20140127FRGS.
References
1. A. Doumanoglou, A. Kargakos, T.-K. Kim, and S.
Malassiotis, Autonomous active recognition and unfoldingof clothes using random decision forests and probabilisticplanning, in Proc. ICRA, pp. 987–993, 2014.
2. I. Mariolis, G. Peleka, A. Kargakos, and S. Malassiotis,
Pose and category recognition of highly deformableobjects using deep learning, in International Conference
Advanced Robotics (ICAR), pp. 655–662, IEEE, 2015.
3. Y. Li, C-F Chen, and P.K. Allen, Recognition of
deformable object category and pose, In Proc. ICRA, 2014
4. Y. Li, C-F. Chen, and P.K. Allen, Real-time pose
estimation of deformable objects using a volumetricapproach. In Proc. IROS, 2014.
5. K.S.M Sahari, and Y.C. Hou. "3D Elastic Deformable
Object Model for Robot Manipulation Purposes." Journalof Advanced Computational Intelligence and Intelligent
Informatics, vol. 18, no.3, pp. 375-382, 2014.

5. Conclusion
In this work, an approach to generate the dataset of
hanging garment using both real and synthetic images is
proposed. The garment mesh model is extracted from thecrudely spread-out garment using particle based
Color

Depth

Threshold 1

Threshold 2Synthetic

Particle Based Model

Shirt

Fig. 3. Example images of color, depth, threshold 1 (white), threshold 2 (black), synthetic and particle based model of hanging garment
grasped by specified point.
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan235

Design and Development of Three Arms Transmission Line Inspection Robot
Muhammad Fairuz bin Abdul Jalal*, Khairul Salleh Mohamed Sahari, Ho Ming Fei, Justin Chan Tuck Leong

Department of Mechanical Engineering, College of Engineering, Universiti Tenaga Nasionaly, Jalan IKRAM-UNITEN,
Kajang, Selangor 43000, Malaysia
E-mail: mfairuz@uniten.edu.my, khairuls@uniten.edu.my
www.uniten.edu.my

Abstract
The high-voltage transmission line had been used primarily for power distribution from power plant or power station
to the end users. However, the transmission line is highly prone to damage due to exposure to various thermal mechanical loadings and material degradation. Therefore, periodical inspection on transmission line after prolonged
service is needed to prevent any failure before it happens. In this paper, we present a new design of three arms
inspection robot for transmission lines. The robot is able to transverse along the line and bypass the in-line obstacles
namely the anti-vibration hammers, spacer, strain clamps and others. The design of the inspection robot in term of
the robot design and configuration with slotted cam at each arm is presented. The detailed analysis via simulation
with respect to the robot stability; kinematic and movement analysis; and power consumption during operation is
executed to make sure the proposed design able to do the inspection without any unexpected difficulties. Later, thelab testing on the developed prototype is done for feasibility study and validation.
Keywords: inspection robot, transmission line, power line, obstacle avoidance, robot design, mobile robot

The process of inspections of transmission line involved
activities such as changing the ceramic insulators andswitching on or off the circuit in between poles2.
However, the manual inspection process of transmission
line exposed the workers to various unfavorable
circumstance of dangerous working environment that istaxing with long duration3. Therefore, it is crucial
demand to develop a mobile inspection robot to replace
manual transmission line inspection job. The
development of the mobile inspection robot will not only
create a safer work environment in the transmission line
inspection job, but also improve the efficiency of theinspection process3. A two arms transmission line robot
is developed by Wang et al.4 utilize a line walking

Introduction
High-voltage transmission lines is one of the mainelements in power distribution from the power plant or
power station to the customer. However, thetransmissions is exposed the harsh environment
conditions that leads to damages by various
circumstances namely effect of mechanical tension,material degradation and flashover. The transmissionslines undergoing such circumstances eventually lead tomany problems such electrical breakdown or even major
accident if transmissions lines were not being inspect,
fixed and replaced in appropriate time1.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

236

Muhammad Fairuz bin Abdul Jalal, Khairul Salleh Mohamed Sahari, Ho Ming Fei, Justin Chan Tuck Leongmechanism that based on biped structure either having
both feet placed on the line or each feet is placed
alternatively on and off the line. Another dual arm robot
self-balancing was developed by Songyi et al. The twoarm of the robot hold onto the transmission line and movewith the wheel installed on the arm. The robot is
integrated with counter weight to improve the robot
stability while travelling on the transmission line5.

Transmission Line Inspection Robot
The robot travel on the transmission line with the help ofrollers that installed in each arm and cross obstacles bymoving its arm up by rotating the power screw installedin each arm. The robot is installed with six motors; three
motors run the roller at each arm for traversing on thetransmission line and the others three motors attached tothe power screw for obstacles crossing process. The
conceptual design and the prototype are shown in Fig. 2.

Xu et al.6 developed a 3-arms transmission line robot thatfeatures 2 supporting arms and 1 assistant arm placed in
between both supporting arms. This robot was designed
to overcome obstacles by evading them with the liftingmechanism on each arm. Rui et al.7 developed 3-arm
transmission line inspection robot. The robot arms
consist of rubber wheel and the both outer arm installedwith gripper to hold unto the line. The rubber wheel
enables the robot to pass over minor obstacles like thesplicer and damper without avoiding them.

Fig. 2. The conceptual design (left) and prototype of the threearms transmission line inspection robot (right).

The three arms robot is hanged to the transmission line
by means of roller and lower jaw as a griping element.
The movement is propelled by the roller at each arm.
Each roller is powered by a motor as shown in Fig. 3.

Our research group already took initiatives to propose
various conceptual design and selection based onrequirement set by Tenaga Nasional Berhad (TNB), themain utility supplier to peninsular Malaysia8.
Transmission Line inspection Robot Design
General Design Requirement
There are many components, which are installed on thetransmission line for various purposes namely tension
insulator chain, suspension clamp, damper, tension
clamp, jumper, splicing sleeve and others depending ontype of transmission line as shown in Fig. 1. It is deemednecessary for robot to bypass these in-line components sothat the inspection process can be executed effectively9.

Fig. 3. The front view (left) and side view (right) of the gripper.

The mechanism to bypass obstacle is illustrated in Fig. 4.

Fig. 4. Mechanism to bypass inline obstacles.

The gripping position and bypassing obstacle movement
are realized by the rotational direction of the motor linked
to power screw. To bypass the in-line obstacle, the motor
is then actuated. The power screw pushes the arm
upwards. As the arm moves up, it follows the curve of theFig. 1. Transmission line environment9.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

237

Design and Development ofslotted profile. The bottom of the arm will have a pin joint
with a support to enable it to rotate as the arm follows thecurve of the slot. The arm will move back to its original
position by reversing the motor direction and the wheel
on arm attached back to the line. The same step applied
to the following next two arm. The mechanism to bypass
the obstacles is shown in Fig. 5.
Fig. 7. The arm gripper during loosening (left) and tightening(right) its grip to the line.

Stability analysis
The significant changes of robot center of mass during
the process to bypass the obstacle provide instability andtendency for the robot to swing. The analysis of thedesign center of gravity is done by means of center ofmass analysis of the following arm configurations:
Configuration 1: original configuration; all the arms
rest on the transmission line
Configuration 2: the first arm move up to avoid
obstacles while the other two arm rest on the line
Configuration 3: the second arm move up to avoid
obstacles while the other two arm rest on the line
Configuration 4: the third arm move up to avoid
obstacles while the other two arm rest on the line

Fig. 5. The arm movement to bypass obstacles.

The stability of the transmission line inspection robot is
an important aspect to be considered as it influences theoverall quality of the image taken during the inspection
process.
Result and Discussion
Capability to bypass obstacles
The robot must be able to avoid obstacles along thetransmission line or else the design will be deemed as a
failure. Fig. 6 shows that the roller and lower jaw do notinterfere with the line as the arm gripper loosening andtightening its grip.

The robot xyz axis and value of center of mass for eachconfiguration is extracted from the CAD software as isshown in Fig. 8. The coordinate center of mass of thedifferent arm configuration is tabulated in the Table 1.

Fig. 6. The arm gripper during loosening (left) and tightening(right) its grip to the line.

The clearance between the arm and the obstacles isranged from 90mm to 130mm with the maximum angleof arm rotation 21.6° as shown in Fig. 7.
Fig. 8. The arm gripper during loosening (left) and tightening(right) its grip to the line.
Table 1. The coordinate of the center of mass by various
arm configuration.
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

238
1

Muhammad Fairuz bin Abdul Jalal, Khairul Salleh Mohamed Sahari, Ho Ming Fei, Justin Chan Tuck Leong

Arm Configuration
Coordinate
x
y
z

1
88.45
250.65207.582
98.78
258.67207.583
98.80
258.83207.584
98.80
258.83207.58There is a small change of the coordinate of the center ofmass in the x-axis and y-axis as the robot change it
configurations from the original configuration while thez-axis remained constant throughout the operation. The
average change of x-axis is 10.33 mm and y-axis is 8.07
mm. The average distance between center of mass invarious arm configuration is 13.1 mm. The results areplotted in in x-y plane as shown in Fig. 9.
Fig. 10. Graph of motor torque (top) and power consumption(bottom) against time.

Conclusion
In this paper, the three arms transmission line inspection
robot is presented. The mechanism to bypass the
obstacles is achieved by using power screw and slotted
profile. The robot is considered stabile during operation
due to minimal changes of its centre of mass. The testing
and analysis in term of robot capability to bypass the
obstacles as well as analysis of motor torque and power
consumption.

Fig. 9. The coordinate of robot center of mass in x-y plane fordifferent arm configuration

Motor Torque and Power Consumption
The value of the motor torque required to rotate thepower screw to move the arm up for bypassing the
obstacle is analyzed in Solidworks Motion. The motor
speed is set to reach 225 RPM based on the motor
specification. The time needed for the arm to move up to
bypass the obstacle is 20 seconds. The average motor
torque and power consumption are 13Nmm and 0.254W.
The graph of the motor torque and power consumption
against time is shown in Fig 10.

Acknowledgements
This work was supported by Universiti Tenaga National
BOLD Research Grant 10289176/B/9/2017/63. Authors
would like to thank TNB for the access of informationand data for formulating the robot design specification.
References
Y.C. Zhang, Z.Z. Liang and M. Tan, Mobile Robot for
Overhead Powerline Inspection-a Review, in Robot 9(5)(2009), pp. 467-473.
L.E. Parker and J.V. Draper (1998). Robotics applications inmaintenance and repair, in Handbook of Industrial
Robotics (1998) pp. 1023-1036.
F.Y. Zhou, J.D. Wang, Y.B. Li, and H.R. Xiao, Control of
an inspection robot for 110KV power transmission lines
based on expert system design methods, in Proc. 2005
IEEE Conf. on Control Applications (2005), pp. 15631568.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

239

Design and Development of

L. Wang, F. Liu, Z. Wang, S. Xu, S. Cheng and J. Zhang,
Development of a practical power transmission line
inspection robot based on a novel line walking mechanism,
in IEEE/RSJ Int. Conf. on Intelligent Robots and Systems(2010), pp. 222-227.
D. Songyi, W. Xuefeng, D. Hang, and W. Tao, Developmentof a self-balance dual-arm robot for inspection of highvoltage power transmission lines, in IEEE Int. Conf. on
Mechatronics and Automation (2012), pp. 2482-2487.
B. Xu, X. Wang, Y. Zhu and H. Chen, Design of obstacle
crossing mechanism of high-voltage transmission line
inspection robot, in IEEE Int. Conf. on Robotics andBiomimetics (2015), pp. 2539-2544.
G. Rui, Z. Feng, C. Lei and Y. Jun, A mobile robot for
inspection of overhead transmission lines, in Proc. of the
3rd Int. Conf. on Applied Robotics for the Power Industry(2014), pp. 1-3.
M.F. Abdul Jalal, K.S. Mohamed Sahari, A. Anuar, A.D.
Mohd Arshad, M.S. Idris, Conceptual design fortransmission line inspection robot, in IOP Conf. Ser.:
Earth and Environmental Science, (2013).
G. Wu, T. Zheng, H. Xiao and C. Li, Navigation, locationand non-collision obstacles overcoming for high-voltage
power transmission-line inspection robot, in Int. Conf. on
Mechatronics and Automation (2009), pp. 2014-2020.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan240
1

A review on fundamental advancements of black hole algorithm
Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari

Faculty of Electrical and Electronics Engineering, Universiti Malaysia Pahang, Pahang, Malaysia
Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz

Faculty of Engineering and Technology, Multimedia University, Melaka, Malaysia
Tasiransurini Ab Rahman

Faculty of Electrical and Electronic Engineering, Universiti Tun Hussein Onn Malaysia, Johor, Malaysia
Asrul Adam, Zulkifli Md Yusof

Faculty of Manufacturing, Universiti Malaysia Pahang, Pahang, Malaysia
Mohd Ibrahim Shapiai

Malaysia Japan International Institute of Technology, Universiti Teknologi Malaysia, Malaysia
Norrima Mokhtar

Faculty of Engineering, University of Malaya, Kuala Lumpur, Malaysia
Email: zuwairie@ump.edu.my, suad.khairim@gmail.com, aziansubari@ump.edu.my,
azlina.aziz@mmu.edu.my, hidayati.aziz@mmu.edu.my, surini@uthm.edu.my,asrul@ump.edu.my, zmdyusof@ump.edu.my, md_ibrahim83@utm.my, norrimamokhtar@um.edu.my

Abstract
In recent years, there is a growing interest in the design and development of nature-inspired optimization algorithms. Oneof the algorithms is black hole algorithm (BHA), which is inspired by the black hole in general relativity and cosmology.
This paper reports a review of the BHA. This review emphasizes on the fundamental advancements of BHA. A brief
research gap is presented at the end of this paper.
Keywords— black hole algorithm, fundamental, optimization

1.

Introduction

A black hole is a region of space packed with so muchmatter that its own gravity prevents anything from escaping –even a ray of light. Black holes can form when massive stars
run out of fuel and collapse under their own weight, creatingstrong gravity. The idea of the black hole is shown in Figure 1.
The RBH determines the event horizon for the black hole. If a
star appears near the black hole (within the event horizon), thestar is pulled towards the black hole due to the massive gravityof the black hole.
The BH algorithm (BHA) [1] is a population-based metaheuristic algorithm inspired by the physical phenomenon of
black hole. In BHA, the agent with the best solution mimics
the black hole. The event horizon is calculated and any agent
within the event horizon vanishes and re-initialized in the
search space.

The BHA is shown in Figure 2. Since BHA is a populationbased algorithm, N number of agents are needed. Let d as thenumber of dimension for an optimization problem, a solution,
X, in a search space is kept by an agent i at iteration t as
follows:(1)
The BHA begins with initialization where a randomly
generated population of candidate solutions are placed in the
search space. For each agent i, the initial solution can be
represented as:© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

241

(2)

Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari, Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz, Tasiransurini Ab Rahman, Asrul Adam, Zulkifli Md
Yusof, Mohd Ibrahim Shapiai, Norrima Mokhtar

far solution is the best solution found from the initial iteration,
t = 0, until current iteration, t. Hence, for t ≠ 0, an agent i isselected as the black hole agent if the fitness value of thatagent, fi, is better than the fitness value of the black hole agent,
fBH. Specifically, for the case of function minimization, fi < fBH.
Once the black hole agent and normal agents are identified,
the radius of the event horizon, RBH, is formulated as follows:(4)
where fBH is the fitness value of the black hole agent, N is thenumber of agents, and fi is the fitness value of the ith star.
The next step is solution update, which is applied to allagents except the black hole agent. Other than black holeagent, the agents can be categorized into two groups. The firstgroup of agents is the agents located within the event horizon.
This agent will be swallowed by the black hole agent. Then, a
new agent following the swallowed one is generated anddistributed randomly in the search space. This generation is tokeep the number of agent constant. The second group ofagents are agents located far from the black hole agent. In
other words, these agents are not within the event horizon.
These agents move towards the black hole agent and the
updated solution can be computed as follows:

Fig. 1. Illustration of a black hole with its event horizon.

(5)
where Xi(t+1) and Xi(t) are the locations of the ith agent at
iterations t+1 and t, respectively. The rand is a random
number belonging to [0,1] and XBH is the location of the blackhole agent. This solution update can be summarized in the
Pseudocode 1.
PSEUDOCODE 1: Solution update in BHA
if agent ith position is within the event horizon
then
do re-initialization
if agent ith position is not within the event horizon
thenupdate the position based on Eq. (5)
else
end

Fig. 2. The black hole algorithm.
After the initialization, the fitness values of the population
are evaluated. The best agent, which has the best fitness value,
is chosen as the black hole while other agents are selected as
normal agents. For the case of function minimization
problems, during initialization, the black hole agent isdetermined as follows:(3)
In this study, the black hole agent keeps the best-so-far
solution, XBH. The best-so-far solution is different than thebest solution. The best solution is defined as the best solution
obtained at specific iteration, t. On the other hand, the best-so-

After all the agents have updated their position, the next
iteration begins if the termination criteria is not met.
Otherwise, the best-so-far, XBH, solution is reported.
2.

Fundamental advancements of black hole algorithm

As shown in Figure 3, the fundamental advancement of
BHA can be grouped into five categories: (1) accelerate thecomputation in software and hardware, (2) implementation inmembrane system, (3) extension of BHA for combination
optimization, (4) hybridization with others meta-heuristics,and (5) diversity improvement for premature convergence
avoidance.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan242

A review on fundamental

operator and hybridized with GSA in order to prevent facing
the premature convergence and to improve the abilities of
GSA in exploration and exploitation [7-8]. In other study, the
BHA is hybridized with a heuristic search algorithm toovercome the problem of BHA. The BHA is used to produce
an initial solution to the clustering problem and then aheuristic search algorithm is applied to improve the quality ofthis solution [9]. In addition, a bisecting k-means algorithm ishybridized with BHA [10] to improve the performance ofbisecting k-means algorithm. Moreover, Black Hole Artificial
Bee Colony (BHABC) algorithm has been introduced in 2016[11]. In this new algorithm, the BHA gives a high exploration
ability while maintaining the original exploitation ability of
the ABC algorithm.
To improve the diversity of BHA, genetic algorithm
operators have been used [12]. Other study has shown that the
chaotic characteristic also can be used to enhance theperformance of BHA. This algorithm is called Chaotic Inertia
Weight Black Hole (CIWBH) algorithm [13].
Recently, a white hole operator has been introduced in
BHA to avoid premature convergence in BHA [14]. In generalrelativity, theoretically, if black holes exist, then it should bepossible to reverse the equations governing them to get the
opposite of black hole, which is the white hole [15]. As
oppose to black hole agent in the BH algorithm, the white hole
was assigned to the worst agent in the population.

Fig. 3. Classification of fundamental improvement of blackhole algorithm.
In 2015, Tsai has introduced the BHA to accelerate theclustering speed by both software and hardware [2]. The
authors introduced a solution for the input data which exceeds
the size of storage or memory of such a system. To ease thisproblem, an efficient clustering algorithm, called MapReduce
Black Hole (MRBH), is presented to leverage the strength of
the black hole algorithm and the MapReduce programming
model. By using MapReduce, MRBH will then divide a large
dataset into several small data sets and cluster these smaller
data sets in parallel.
The BHA can be implemented based on membrane system[3]. In this study, the authors inserted the black hole clustering
algorithm in the membrane structure, provide the three partsof the membrane system to perform black hole clustering
algorithm, which are the membrane structure based onmembrane computing, the objects in elementary membrane
and the evolution rules in the elementary membrane, andanalyse the performance of the black hole clustering algorithm
based on membrane computing.
The original BHA need to be extended for solving
combinatorial optimization problems. One of the extension of
BH algorithm is called Binary Black Holes (BBH) and it isused to solve multiple instances of known benchmarks
obtained from the OR-library [4]. Another binary version of
BHA called BBHA was introduced by Pashaei, which is used
to solve feature selection problem in biological data [5].
The BHA has been actively hybridized with otheralgorithms. For example, the BHA is combined with the stars
gravities information and applied to unmanned combat aerial
vehicle (UCAV) planning [6]. The BHA is also used as an

3.

Concluding remark

All the meta-heuristic algorithms perform differently when
they are used to solve different optimization problems. One
algorithm may perform better than the other in solving one
problem and it may perform unsatisfactorily in other set ofproblems. Based on the review in this paper, BHA did not
performed well in solving numerical optimization problems.
In most studies, the BHA is weak to perform global searchcompletely especially in the big problem spaces, due to its
limited exploration and exploitation capability.
Using the progress of other optimization algorithms as
reference, the premature convergence of the BHA could be
avoided by applying local and global neighbourhood, or by
introducing more randomness in the search process using
chaotic methods, for example. Also, hybridization withanother optimization algorithms could be investigated, such aswith differential evolution [16], biogeography-based
optimization [17], simulated annealing [18], harmony search[19], and simulated Kalman filter optimization [20-21]. Since
all the publications on BHA deal with single-objective
optimization problem, multi-objective optimization algorithm
based on BHA could also be established.
It is also worth to mention a paper published in Information
Sciences journal which explained that the BHA is in fact a
simplified version of particle swarm optimization (PSO) withinertia weights. Hence, the novelty of BH algorithm isquestionable [22]. Based on the popular CEC2005 benchmark
set [23], they proved that the BHA is inferior to theperformance of PSO with inertia weights for most of the test
functions. This finding could motivate the researchers in this

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan243

Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari, Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz, Tasiransurini Ab Rahman, Asrul Adam, Zulkifli Md
Yusof, Mohd Ibrahim Shapiai, Norrima Mokhtar

optimizer: a nature-inspired algorithm for global optimization",field to search for a novel approach to establish a new version
Neural Computing and Applications, 27(2), 495-513, 2016.
of BHA that is different than the other optimization algorithms.
Nevertheless, modifications to the original BHA are still [16] R. Storn and K.V. Price, "Differential evolution – a simple andefficient heuristic for global optimization over continuous
needed to be done in order to continuously improve thespaces", J. Global Optim., 11(4), 341-359, 1997.
performance of this algorithm. Although some studies have [17] D. Simon, "Biogeography-based optimization", IEEE Trans.
been done on improving the BHA, the algorithm still has a lot
Evol. Comput., 12(6), 702-713, 2008.
of possible improvements.
[18] S. Kirkpatrick, C.D. Gelatt, and M.P. Vecchi, "Optimization by

Acknowledgement
This research is funded by the Universiti Malaysia Pahang(Research Grant RDU1703224).
References[1] A. Hatamlou, "Black hole: A new heuristic optimization
approach for data clustering", Information Sciences, 222, 175184, 2012.
[2] C.W. Tsai, C. H. Hsieh, and M.C. Chiang, "Parallel Black Hole
Clustering Based on MapReduce", IEEE International
Conference on Systems, Man, and Cybernetics (SMC), 2015.
[3] Q. Li and Z. Pei, "The Black Hole Clustering Algorithm Basedon Membrane Computing", 2015.
[4] A. G. Rubio, B. Crawford, R. Soto, A. Jaramillo, S. M.
Villablanca, J. Salas, and E. Olguín, "A Binary Black Hole
Algorithm to Solve Set Covering Problem", International
Conference on Industrial, Engineering and Other Applicationsof Applied Intelligent Systems, 2016.
[5] E. Pashaei and N. Alydin, "Binary black hole algorithm forfeature selection and classification on biological data", Applied
Soft Computing, 56, 94-106, 2017.
[6] A. Heidari and R. Abbaspour, "A gravitational black holealgorithm for autonomous UCAV mission planning in 3D
realistic environments", International Journal of Computer
Applications, 95(9), 2014[7] M. Doraghinejad and H. Nezamabadi-pour, "Black hole: A new
operator for gravitational search algorithm", International
Journal of Computational Intelligence Systems, 7(5), 809-826,
2014.
[8] S.K. Mohammed, Z. Ibrahim, H. Daniyal, and N.A. Ab. Aziz,"A New Hybrid Gravitational Search - Black Hole Algorithm",
The National Conference for Postgraduate Research, 834-842,
2016.
[9] P. Chandrasekar and M. Krishnamoorthi, "BHOHS: A two stage
novel algorithm for data clustering", International Conferenceon Intelligent Computing Applications (ICICA), 2014.
[10] M. Eskandarzadehalamdary, B. Masoumi, and O. Sojodishijani,"A new hybrid algorithm based on black hole optimization andbisecting k-means for cluster analysis", 22nd Iranian
Conference on Electrical Engineering (ICEE), 2014.
[11] J. C. Bansal, "Black Hole Artificial Bee Colony Algorithm", 6thInternational Conference Swarm, Evolutionary, and Memetic
Computing (SEMCCO 2015), 2015.
[12] S. Yaghoobi and H. Mojallali, "Modified Black Hole Algorithmwith Genetic Operators", International Journal ofComputational Intelligence Systems, 9(4), 652-665, 2015.
[13] H. Aslani, M. Yaghoobi, and M. R. Akbarzadeh-T, "Chaotic
inertia weight in black hole algorithm for function optimization",International Congress on Technology, Communication andKnowledge (ICTCK), 2015.
[14] S.K. Mohammed, Z. Ibrahim, H. Daniyal, and N. A. Ab. Aziz,"White Hole-Black Hole Algorithm", The National Conference
for Postgraduate Research, 824-833, 2016.
[15] S. Mirjalili, S.M. Mirjalili, and A Hatamlou, "Multi-verse

simulated annealing", Science, 220, 671-680, 1983.
[19] Z.W. Geem, J.H. Kim, and G.V. Loganathan, "A new heuristic
optimization algorithm: harmony search", Simulation, 76(2), 6068, 2001.
[20] Z. Ibrahim, N.H. Abdul Aziz, N.A. Ab. Aziz, S. Razali, and M.S.
Mohamad, "Simulated Kalman Filter: a novel estimation-based
metaheuristic optimization algorithm", Advanced Science
Letters, 22, 2941-2946, 2016.
[21] Z. Ibrahim, N.H. Abdul Aziz, N.A. Ab. Aziz, S. Razali, M.I.
Shapiai, S.W. Nawawi, and M.S. Mohamad, "A Kalman filter
approach for solving unimodal optimization problems", ICIC
Express Letters, 9(12), 3415-3422, 2015.
[22] A.P. Piotrowski, J.J. Napiorkowski, and P.M. Rowinski, "Hownovel is the “novel” black hole optimization approach?",Information Sciences, 267, 191–200, 2014.
[23] P.N. Suganthan, N. Hansen, J.J. Liang, K. Deb, Y.P. Chen, A.
Auger, and S. Tiwari, "Problem Definitions and Evaluation
Criteria for the CEC 2005 Special Session on Real-Parameter
Optimization", Nanyang Technol. Univ., Singapore, Tech. Rep.
KanGAL #2005005, IIT Kanpur, India, 2005.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan244

A survey on applications of black hole algorithm
Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari, Asrul Adam, Zulkifli Md Yusof

Universiti Malaysia Pahang, Pahang, Malaysia
Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz

Faculty of Engineering and Technology, Multimedia University, Melaka, Malaysia
Tasiransurini Ab Rahman

Faculty of Electrical and Electronic Engineering, Universiti Tun Hussein Onn Malaysia, Johor, Malaysia
Mohd Ibrahim Shapiai

Malaysia Japan International Institute of Technology, Universiti Teknologi Malaysia, Malaysia
Norrima Mokhtar

Faculty of Engineering, University of Malaya, Kuala Lumpur, Malaysia
Email: zuwairie@ump.edu.my, suad.khairim@gmail.com, aziansubari@ump.edu.my, asrul@ump.edu.my,
zmdyusof@ump.edu.my, azlina.aziz@mmu.edu.my, hidayati.aziz@mmu.edu.my, surini@uthm.edu.my,md_ibrahim83@utm.my, norrimamokhtar@um.edu.my

Abstract
In recent years, there is a growing interest in the design and development of nature-inspired optimization algorithms. Oneof the algorithms is black hole algorithm (BHA), which is inspired by the black hole in general relativity and cosmology.
This paper reports a review of the black hole algorithm (BHA). This survey emphasizes on the applications of BHA.
Keywords— application, black hole algorithm, optimization

1.

Introduction

A black hole is a region of space packed with so muchmatter that its own gravity prevents anything from escaping –even a ray of light. Black holes can form when massive stars
run out of fuel and collapse under their own weight, creatingstrong gravity.
The black hole algorithm (BHA) [1] is a population-based
meta-heuristic algorithm inspired by the physical phenomenonof black hole. In BHA, the agent with the best solution mimics
the black hole. The event horizon is calculated and any agent
within the event horizon vanishes and re-initialized in the
search space.
The BHA is shown in Figure 1. Since BHA is a populationbased algorithm, N number of agents are needed. Let d as thenumber of dimension for an optimization problem, a solution,
X, in a search space is kept by an agent i at iteration t as
follows:(1)

The BHA begins with initialization where a randomly
generated population of candidate solutions are placed in the
search space. For each agent i, the initial solution can be
represented as:(2)
After the initialization, the fitness values of the population
are evaluated. The best agent, which has the best fitness value,
is chosen as the black hole while other agents are selected as
normal agents. For the case of function minimization
problems, during initialization, the black hole agent isdetermined as follows:(3)
In this study, the black hole agent keeps the best-so-far
solution, XBH. The best-so-far solution is different than thebest solution. The best solution is defined as the best solution
obtained at specific iteration, t. On the other hand, the best-so-© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

245

Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari, Asrul Adam, Zulkifli Md Yusof, Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz, Tasiransurini Ab
Rahman, Mohd Ibrahim Shapiai, Norrima Mokhtar

far solution is the best solution found from the initial iteration,
t = 0, until current iteration, t. Hence, for t ≠ 0, an agent i isselected as the black hole agent if the fitness value of thatagent, fi, is better than the fitness value of the black hole agent,
fBH. Specifically, for the case of function minimization, fi < fBH.
Once the black hole agent and normal agents are identified,
the radius of the event horizon, RBH, is formulated as follows:(4)
where fBH is the fitness value of the black hole agent, N is thenumber of agents, and fi is the fitness value of the ith star.
The next step is solution update, which is applied to allagents except the black hole agent. Other than black holeagent, the agents can be categorized into two groups. The firstgroup of agents is the agents located within the event horizon.
This agent will be swallowed by the black hole agent. Then, a
new agent following the swallowed one is generated anddistributed randomly in the search space. This generation is tokeep the number of agent constant. The second group ofagents are agents located far from the black hole agent. In
other words, these agents are not within the event horizon.
These agents move towards the black hole agent and the
updated solution can be computed as follows:(5)
where Xi(t+1) and Xi(t) are the locations of the ith agent at
iterations t+1 and t, respectively. The rand is a random
number belonging to [0,1] and XBH is the location of the blackhole agent. This solution update can be summarized in the
Pseudocode 1.
PSEUDOCODE 1: Solution update in BHA
if agent ith position is within the event horizon
then
do re-initialization
if agent ith position is not within the event horizon
thenupdate the position based on Eq. (5)else
end
After all the agents have updated their position, the next
iteration begins if the termination criteria is not met.
Otherwise, the best-so-far, XBH, solution is reported.
2.

Applications of black hole algorithm

The BHA has been used to solve many problems andapplied in many fields. The BHA has been used to train aneural network, which is used as workload prediction model[2]. The BHA has been also used with higher order neuralnetwork for solving nonlinear classification problems of data
mining [3].

Fig. 1. Flowchart of BHA.
The BHA is used in engineering design of electromagnetic
devices [4]. The new optimization technique based on theblack hole phenomenon is called the black-hole-based
optimization technique. To show the effectiveness of the
proposed technique, it has been demonstrated on a magnetizer
by optimizing its pole face to obtain a desired magnetic flux
density distribution.
In the field of software engineering, Jeet and Dhir
combined a nature-inspired black-hole algorithm and a genetic
algorithm to propose a search-based technique for software
clustering [5]. This technique is used to manage a softwaresystem by partitioning it into smaller subsystems containing
highly related modules. Search-based software clustering
techniques are found to be beneficial in effective partitioningof software systems.
Moreover, the BHA has been used to solve the
manufacturing cell design problem, in which different
machines are grouped into sets or cells with the objective tominimize the movement of materials. In order to increase theefficiency and productivity, machines that create productswith similar requirements must be close to each other. Thismachine- grouping, known as cell, allow the production to be
dedicated and organized [6].
In civil engineering, the BHA has been introduced in thegeneralized constitutive law. The problem of model
identification is transformed to a problem of parameter backanalysis, which is a typical and complicated optimization. To
improve the efficiency of the traditional optimization method,a black hole algorithm is applied in this study, by combining
the generalized constitutive law for an elastic–plastic
constitutive model and black hole algorithm. A new back© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan246

A survey on applications

analysis method for model identification of rocks surrounding
underground roadways in coal mine is proposed to solve aparameter back analysis for elastic-plastic constitutive model[7]. While the slope instability of embankment is very
complex, which may develop locally, near the facing, within
the embankment, or through the foundation soil as local,
surficial, general, or deep-seated failure. The BHA is used toanalyze the stability of one high embankment slope for one
airport in the plateau loess area [8].
In the field of operations research, the BHA has been
employed by Soto et al., in which, they presented two new
systems for online control of enumeration strategies based on
bat algorithm and BHA [9]. The BHA is used to improve
performance in the exploration of search tree and updating theenumeration strategy online [10].
The BHA has been actively applied in the field of powersystem. The BHA has been combined with GSA to solve dayahead reactive power optimization problem in distribution
network [11]. Also, BHA has been used in non-technicallosses characterization [12] and optimal reactive power
dispatch problem [13]. The BHA has been combined with
BBO algorithm [14] and DE algorithm [15] to solve theeconomic dispatch problem incorporating the wind energy(EDIW) [16]. Bouchekara also used BHA to solve the optimal
power flow problem in a power system [17]. Finally, in thefield of power system, a gradient-based modified teaching–learning-based optimization combined with BHA to seek the
optimum operational cost [18].
Image enhancement is considered as an optimization
problem and a meta-heuristic optimization algorithm is used
to solve this problem. Image enhancement is a nonlinear
optimization problem with its constraints and the enhancement
process will be done by intensifying each pixel’s content.
BHA is employed to find the image’s optimum parameters of
the transfer function in order to get the best results [19].
In the medical field, the BHA has been integrated withdecision tree [20] in medical data classification. In addition,
Binary Black Hole Algorithm (BBHA) and Random Forest
Ranking (RFR) have been used for gene selection andclassification of microarray data [21]. While in the same field,
Pashaei and Aydin presented a binary version of BHA called
BBHA for solving feature selection problem in biological data
[22].
The original BHA was proposed for clustering problem.
Many variants of BHA have been introduced in clustering. As
an example, BHA has been used to accelerate the clustering
speed by both software and hardware [23]. Also, for clustering,the BHA has been combined with bisecting k-means
algorithm, this algorithm is called BH–BKmeans algorithm[24]. Jeet and Dhir proposed to use the hybrid of BHA,
auxiliary archive, and GA to optimize multiple objectives forclustering of android mobile applications [25].
The BHA has been used as multi-objective schedulingmethod for workflow application [26]. Also in scheduling, themulti-objective BHA has been used to optimize scheduling ofjobs on two machines taking into consideration the fuzzinessin the environment involved [27].

In the field of industries and induction motors, the design,
performance evaluation and control of induction motors are
based on circuit parameters. Though conventional techniques
produce good results for accurate measurement of electricalparameters, like resistance (or reactance), the swarm
intelligence-motivated techniques still better results for realworld optimization problems. The disrupted Black Hole
Artificial Bee Colony (DBHABC) algorithm is proposed andapplied for optimizing induction motor parameter estimation[28].
The set covering problem (SCP) is one of the mostrepresentative optimization problems. The BHA is used tosolve multiple instances of the problem with known
benchmarks obtained from the OR-library [29]. Set covering
problems were also solved in 2016 by employing a recent
nature-inspired metaheuristic based on the black holephenomena [30]. A multi-dynamic binary black holealgorithm for resolving the set covering problem has been
introduced [31]. The same problems were also solved bycuckoo search and BHA [32].
Finally, an application of BHA in mission planning has also
been reported. Unmanned combat aerial vehicle (UCAV)
mission planning in realistic test fields is a well-known
optimization problem, so many algorithms have been designed
to solve this multi-constrained problem. The BHA is one ofthese swarm intelligence approaches which applied to the
UCAV mission planning scheme [33].
3.

Concluding remark

This paper reports a survey on the applications of BHA. At
the moment, the BHA has been applied in artificial
intelligence, engineering design, software engineering,
manufacturing, civil engineering, operations research, image
processing, wireless sensor network, scheduling, parameter
estimation, power system, medical analysis, clustering, setcovering problem, and mission planning. Due to the
popularity of BHA, we expect the BHA will be applied inanother area of applications such as routing in VLSI [34],DNA sequence design [35], and adaptive beamforming forantenna engineering [36].
Acknowledgement
This research is funded by the Universiti Malaysia Pahang(RDU1703224).
References[1] A. Hatamlou, "Black hole: A new heuristic optimization
approach for data clustering", Information Sciences, 222, 175184, 2012.
[2] J. Kumar and A.K. Singh, "Dynamic resource scaling in cloudusing neural network and black hole algorithm," Fifth
International Conference on the Eco-friendly Computing andCommunication Systems (ICECCS), 2016.
[3] J. Nayak, B. Naik, and H.S. Behera, "Solving nonlinear
classification problems with black hole optimisation and higher
order Jordan Pi-sigma neural network: a novel approach,"International Journal of Computational Systems Engineering,2(4), 236-251, 2016.
[4] H.R. Bouchekara, "Optimal design of electromagnetic devices© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

247

Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari, Asrul Adam, Zulkifli Md Yusof, Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz, Tasiransurini Ab
Rahman, Mohd Ibrahim Shapiai, Norrima Mokhtar

using a black-hole-based optimization technique," IEEE
Transactions on Magnetics, 49(12), 5709-5714, 2013.
[5] K. Jeet and R. Dhir, "Software architecture recovery using
genetic black hole algorithm," ACM SIGSOFT Software
Engineering Notes, 40(1), 1-5, 2015.
[6] R. Soto, B. Crawford, N. Fernandez, V. Reyes, S. Niklander,and I. Araya, “Solving Manufacturing Cell Design Problems
Using the Black Hole Algorithm,” The Mexican International
Conference on Artificial Intelligence, 2016.
[7] W. Gao, M. Ge, D. Chen, and X. Wang, “Back analysis for rockmodel surrounding underground roadways in coal mine basedon black hole algorithm,” Engineering with computers, 32(4),675-689, 2016.
[8] W. Gao, X. Wang, S. Dai, and D. Chen, “Study on stability ofhigh embankment slope based on black hole algorithm,”
Environmental Earth Sciences, 75(20), 1381, 2016.
[9] R. Soto, B. Crawford, R. Olivares, S. Niklander, F. Johnson, F.
Paredes, and E. Olguín, “Online control of enumeration
strategies via bat algorithm and black hole optimization”,
Natural Computing, 16(2), 241-257, 2017.
[10] R. Olivares, R. Soto, B. Crawford, M. Barría, and S. Niklander,“Evaluation of choice functions to self-adaptive on constraint
programming via the black hole algorithm”, The Computing
Conference (CLEI), Latin American, 2016.
[11] H. Pei, Y. Li, and K. Liu, “A multi-object black holegravitational search algorithm for day-ahead reactive
optimization in distribution network,” The Guidance,
Navigation and Control Conference (CGNCC), Chinese, 2016.
[12] D. Rodrigues, C.C.O. Ramos, and J.P. Papa, “Black holealgorithm for non-technical losses characterization,” 6th Latin
American Symposium on the Circuits & Systems (LASCAS), 1-4,
2015.
[13] K. Lenin, B.R. Reddy, and M.S. Kalavathi, “Black holealgorithm for solving optimal reactive power dispatch problem,”International Journal of Research in Management, Science &Technology, 2(1), 2321-3264, 2014.
[14] D. Simon, “Biogeography-based optimization,” IEEE
transactions on Evolutionary Computation, 12(6), 702-713,
2008.
[15] R. Storn, and K. Price, “Differential evolution–a simple andefficient heuristic for global optimization over continuous
spaces,” Journal of global optimization, 11(4), 341-359, 1997.
[16] Z. Hasan and M. El-Hawary, “Economic dispatch incorporating
wind energy by BH, BBO and DE,” IEEE Canadian Conference
on the Electrical and Computer Engineering (CCECE), 2016.
[17] H. Bouchekara, “Optimal power flow using black-hole-based
optimization approach,” Applied Soft Computing, 24, 879-888,
2014.
[18] R. Azizipanah-Abarghooee, T. Niknam, F. Bavafa, and M. Zare,“Short-term scheduling of thermal power systems using hybrid
gradient based modified teaching–learning optimizer with blackhole algorithm,” Electric Power Systems Research, 108, 16-34,
2014.
[19] S. Yaghoobi, S. Hemayat, and H. Mojallali, “Image gray-level
enhancement using Black Hole algorithm,” 2nd International
Conference on the Pattern Recognition and Image Analysis(IPRIA), 2015.
[20] E. Pashaei, M. Ozen, and N. Aydin, “An application of blackhole algorithm and decision tree for medical problem,” IEEE
15th International Conference on the Bioinformatics andBioengineering (BIBE), 2015.
[21] E. Pashaei, M. Ozen, and N. Aydin, “Gene selection andclassification approach for microarray data based on Random
Forest Ranking and BBHA,” IEEE-EMBS International

Conference on the Biomedical and Health Informatics (BHI),
2016.
[22] E. Pashaei and N. Aydin, “Binary black hole algorithm forfeature selection and classification on biological data,” Applied
Soft Computing, 56, 94-106, 2017.
[23] C.-W. Tsai, C.-H. Hsieh, and M.-C. Chiang, “Parallel black holeclustering based on MapReduce,” IEEE International
Conference on the Systems, Man, and Cybernetics (SMC), 2015.
[24] M. Eskandarzadehalamdary, B. Masoumi, and O. Sojodishijani,“A new hybrid algorithm based on black hole optimization andbisecting k-means for cluster analysis,” 22nd Iranian
Conference on the Electrical Engineering (ICEE), 2014.
[25] K. Jeet and R. Dhir, “Software clustering using hybrid multiobjective black hole algorithm,” Paper presented at the SEKE,
2016.
[26] F. Ebadifard and S.M. Babamir, “Optimizing multi objective
based workflow scheduling in cloud computing using black holealgorithm,” 3th International Conference on the Web Research(ICWR), 2017.
[27] K. Jeet, S. Sharma and K.K. Nailwal, “Two-Machine Fuzzy
Flow Shop Scheduling Using Black Hole Algorithm,” Global
Journal of Pure and Applied Mathematics, 13(6), 1935-1946,2017.
[28] F.B. Sharma and S.R. Kapoor, “Induction motor parameter
estimation using disrupted black hole artificial bee colonyalgorithm,” International Journal of Metaheuristics, 6(12), 85106, 2017.
[29] Á.G. Rubio, B. Crawford, R. Soto, A. Jaramillo, S.M.
Villablanca, J. Salas, and Olguín, E., “A binary black holealgorithm to solve set covering problem,” The International
Conference on Industrial, Engineering and Other Applicationsof Applied Intelligent Systems, 2016.
[30] R. Soto, B. Crawford, I. Figueroa, S. Niklander and E. Olguín,“A black hole algorithm for solving the set covering problem,”
The International Conference on Industrial, Engineering andOther Applications of Applied Intelligent Systems, 2016.
[31] J. García, B. Crawford, R. Soto and P. García, “A multi
dynamic binary black hole algorithm applied to set covering
problem,” The International Conference on Harmony Search
Algorithm, 2017.
[32] R. Soto, B. Crawford, R. Olivares, J. Barraza, I. Figueroa, F.
Johnson, and E. Olguín, “Solving the non-unicost set covering
problem by using cuckoo search and black hole optimization,”
Natural Computing, 16(2), 213-229, 2017.
[33] A. Heidari and R. Abbaspour, “A gravitational black holealgorithm for autonomous UCAV mission planning in 3D
realistic environments,” International Journal of Computer
Applications, 95(9), 2014.
[34] M.N. Ayob, Z. Md Yusof, A. Adam, A.F. Zainal Abidin, I.
Ibrahim, Z. Ibrahim, S. Sudin, N. Shaikh-Husin, M.K. Hani, “A
particle swarm optimization approach for routing in VLSI,”2010 Second International Conference on Computational
Intelligence, Communication Systems and Networks (CICSyN),
49-53, 2010.
[35] S.M. Mustaza, A.F.Z. Abidin, Z. Ibrahim, M.A. Shamsudin,
A.R. Husain and J.A.A. Mukred, “A Modified Computational
Model of Ant Colony System in DNA Sequence Design,” IEEE
Student Conf. on Research and Development (SCOReD), 169173, 2011.
[36] K. Lazarus, N.H. Noordin, Z. Ibrahim, M.F. Mat Jusof, A.A.
Mohd Faudzi, N. Subari, and K.Z. Mohd Azmi, “AnOpposition-based Simulated Kalman Filter algorithm foradaptive beamforming,” International Conference on Applied
System Innovation (ICASI), 91-94, 2017.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

248

Black hole white hole algorithm with local search
Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari, Asrul Adam, Zulkifli Md Yusof

Universiti Malaysia Pahang, Pahang, Malaysia
Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz

Faculty of Engineering and Technology, Multimedia University, Melaka, Malaysia
Tasiransurini Ab Rahman

Faculty of Electrical and Electronic Engineering, Universiti Tun Hussein Onn Malaysia, Johor, Malaysia
Norrima Mokhtar

Faculty of Engineering, University of Malaya, Kuala Lumpur, Malaysia
Email: zuwairie@ump.edu.my, suad.khairim@gmail.com, aziansubari@ump.edu.my, asrul@ump.edu.my,
zmdyusof@ump.edu.my, azlina.aziz@mmu.edu.my, hidayati.aziz@mmu.edu.my, surini@uthm.edu.my,norrimamokhtar@um.edu.my

Abstract
Black hole algorithm (BHA) is an optimization algorithm inspired by the black hole discovery in relativity theory. Recently,
white hole operator, which is based on the opposite of black hole, has been introduced in BHA. In this paper, a local isadded in the BHA with white hole operator.
Keywords— black hole algorithm, fundamental, optimization

1.

Introduction

A black hole is a region of space packed with so muchmatter that its own gravity prevents anything from escaping –even a ray of light. Black holes can form when massive stars
run out of fuel and collapse under their own weight, creatingstrong gravity.
The BHA algorithm (BHA) [1] is a population-based metaheuristic algorithm inspired by the physical phenomenon of
black hole. In BHA, the agent with the best solution mimics
the black hole. The event horizon is calculated and any agent
within the event horizon vanishes and re-initialized in the
search space.
The BHA is shown in Figure 1. Let N is the number of
agents and d is the number of dimension for an optimization
problem, a solution, X, in a search space is kept by an agent i
at iteration t as follows:(1)
The BHA begins with initialization where a randomly
generated population of candidate solutions are placed in the
search space. For each agent i, the initial solution can be
represented as:(2)
After the initialization, the fitness values of the population
are evaluated. The best agent, which has the best fitness value,
is chosen as the black hole while other agents are selected as
normal agents. For the case of function minimization
problems, during initialization, the black hole agent isdetermined as follows:(3)
In this study, the black hole agent keeps the best-so-far
solution, XBH. The best-so-far solution is different than thebest solution. The best solution is defined as the best solution
obtained at specific iteration, t. On the other hand, the best-sofar solution is the best solution found from the initial iteration,
t = 0, until current iteration, t. Hence, for t ≠ 0, an agent i isselected as the black hole agent if the fitness value of thatagent, fi, is better than the fitness value of the black hole agent,
fBH. Specifically, for the case of function minimization, fi < fBH.
Once the black hole agent and normal agents are identified,
the radius of the event horizon, RBH, is formulated as follows:© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

249

Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari, Asrul Adam, Zulkifli Md Yusof, Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz, Tasiransurini Ab
Rahman, Norrima Mokhtar(4)where fBH is the fitness value of the black hole agent, N is thenumber of agents, and fi is the fitness value of the ith star.
The next step is solution update, which is applied to allagents except the black hole agent. Other than black holeagent, the agents can be categorized into two groups. The firstgroup of agents is the agents located within the event horizon.
This agent will be swallowed by the black hole agent. Then, a
new agent following the swallowed one is generated anddistributed randomly in the search space. This generation is tokeep the number of agent constant. The second group ofagents are agents located far from the black hole agent. In
other words, these agents are not within the event horizon.
These agents move towards the black hole agent and the
updated solution can be computed as follows:(5)
where Xi(t+1) and Xi(t) are the locations of the ith agent at
iterations t+1 and t, respectively. The rand is a random
number belonging to [0,1] and XBH is the location of the blackhole agent. This solution update can be summarized in the
Pseudocode 1.

An arbitrary agent i could be updated to a position in the
search space within the event horizon of the white hole. In thiscase, the agent is pushed by the white hole. Due to this, theposition of the agent i is updated as follows:

PSEUDOCODE 1: Solution update in BHA
if agent ith position is within the event horizon
then
do re-initialization
if agent ith position is not within the event horizon
thenupdate the position based on Eq. (5)
else
end

(8)

After all the agents have updated their position, the next
iteration begins if the termination criteria is not met.
Otherwise, the best-so-far, XBH, solution is reported.
2.

Fig. 1. Flowchart of BHA.

The black hole white hole (BHWH) algorithm

As oppose to black hole agent in the BHA, the white holecan be assigned to the worst agent in the population. Hence,
the white hole is updated as follows:(6)
Also, similar to the black hole, the white hole has its own
event horizon and the radius of the event horizon, RWH, canbe calculated based on the following equation:(7)
where fitWH is the fitness value of the white hole, N is thenumber of agents, and fiti is the fitness value of the ith star.

where Xi(t+1) and Xi(t) are the locations of the arbitrary agent
i at iterations t+1 and t, respectively. The rand is a random
number belonging to [0,1] and XWH is the location of the whitehole agent.
Figure 2 shows the flowchart of the black hole algorithmwith white hole operator, which is also termed as BHWH
algorithm [2]. The difference between the proposed BHWH
algorithm with the BHA shown in Figure 1 is the inclusion ofa white hole agent as the worst solution. Note that there are
two kind of event horizons. These are the event horizon forblack hole and white hole agents and should be calculated atevery iteration. Also, due to the inclusion of white hole agent
which pushes any nearby agent, the position update in BHWH
algorithm can be explained in Pseudocode 2.
3.

The proposed BHWH algorithm with local search

The basic idea of the local search is to find neighbourhood
solution around the best solution. In this study, not all theagents are subjected to local search. The white hole agent thatkeep the worst solution at the iteration, t, is selected and the
local search is applied to the white hole agent.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan250

Black hole white hole

The local search is applied to the worst agent in the
population. It is possible to select any worst solution at
iteration t during an implementation of an optimization
algorithm and apply Equation (9) to the worst agent. In thisstudy, the local search is incorporated in the BHWH
algorithms. As a result, a variant of BH algorithm can be
developed, which is called black hole white hole (BHWH)
algorithm with local search (BHWHLS). The flowchart of
BHWHLS algorithm is similar to the flowchart of BHWH
algorithm in Figure 2. However, considering the local search,
the solution for BHWHLS algorithm is updated according toPseudocode 3, which replaces the Pseudocode 2.
PSEUDOCODE 3: Solution update in BHWHLS algorithm
if agent ith is black hole agent thenposition not updated
if agent ith position is within the event horizon of black holethendo re-initialization
if agent ith is the white hole thenapply local search based on Equation (9)if agent ith position is within the event horizon of white holethenupdate the position based on Equation (8)else
update the position based on Equation (5)endFig. 2. Flowchart of BHWH algorithm.
Table 1. Experimental setting parameter
Iteration
Runs
AgentsDimension
Search Space

10000
51
100
50
[-100 100]

4.

PSEUDOCODE 2: Solution update in BHWH algorithm
if agent ith is black hole agent thenposition not updated
if agent ith position is within the event horizon of black holethendo re-initialization
if agent ith position is within the event horizon of white holethenupdate the position based on Equation (8)else
update the position based on Equation (5)endLet, the local search is applied to every
dimension, d, based on Equation (9).
(9)
where
is the solution after the local search is applied,
XBH is the location of the black hole agent, t is the iteration
number, Tmax is the maximum number of iteration, and randd∈ [0,1] is a random number, which is generated at every
dimension.

Experiment, result, and discussion

The experiments in this study were implemented based on
the parameter setting tabulated in Table 1. The performance of
the BHWHLS algorithm and the original BHA were studied
by solving CEC2014 benchmark functions [3]. The results aretabulated in Table 2. The values in bold indicate the smaller or
better result. Table 2 shows that the BHWHLS algorithm
outperforms BHA for all the unimodal functions, while forsimple multimodal functions, the accuracy of BHWHLS
algorithm is very good, except for Function 14. For hybrid
functions, the BHWHLS algorithm reaches to optimum
solution for all functions except for Function 18 and Function
19. Finally, even though the BHWHLS algorithm performs
BHA in most of the composition functions, the accuracy arequite unsatisfactorily for the case of Function 24 and Function
25. An example of convergence curve is shown in Figure 3.
Acknowledgement
This research is funded by the Universiti Malaysia Pahang(RDU1703224).
References[1] A. Hatamlou, "Black hole: A new heuristic optimization
approach for data clustering", Information Sciences, 222, 175184, 2012.
[2] S.K. Mohammed, Z. Ibrahim, H. Daniyal, and N.A. Ab. Aziz,"White Hole-Black Hole Algorithm", The National Conference
for Postgraduate Research, 824-833, 2016.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan251

Zuwairie Ibrahim, Suad Khairi Mohammed, Norazian Subari, Asrul Adam, Zulkifli Md Yusof, Nor Azlina Ab Aziz, Nor Hidayati Abdul Aziz, Tasiransurini Ab
Rahman, Norrima Mokhtar

Table 2. Performance comparison between the BHA and BHWHLS

Type of
Function
Unimodal

Simplemultimodal

HybridComposition

Function
ID
F1
F2
F3
F4
F5
F6
F7
F8
F9
F10
F11
F12
F13
F14
F15
F16
F17
F18
F19
F20
F21
F22
F23
F24
F25
F26
F27
F28
F29
F30

The ideal
value
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1400
1500
1600
1700
1800
1900
2000
2100
2200
2300
2400
2500
2600
2700
2800
2900
3000

BHA
(Mean value)
5611014.427
4997329.195
14041.12327
609.8508916
520.0161923
658.7739009
701.1662029
953.499765
1249.316564
3816.305871
8308.348714
1200.797984
1300.56279
1400.261316
1810.089933
1621.682464
639170.063
2476.577727
1960.011302
9023.306146
429192.2267
3786.065395
2652.810511
2665.506218
2749.939581
2796.226359
4729.276014
11732.29457
10839.35982
69850.72279

BHWHLS(Mean value)
2395827.785
8365.062195
6624.351763
549.1541895
520.0000065
657.1073197
700.0187492
928.9684462
1221.594196
3214.544468
7890.732838
1200.757164
1300.492432
1400.308738
1722.79273
1621.5487
235277.876
3470.864961
1965.653521
4135.276728
187917.412
3758.516987
2645.194641
2668.43428
2753.661216
2702.390476
4655.848694
10898.23985
8746.81227
37593.04308

Function 16
1624
BHA
BHWHLS

Mean Value

1623.51623

1622.5[3] J. J. Liang, B-Y. Qu, and P. N. Suganthan, "Problem Definitions
and Evaluation Criteria for the CEC 2014 Special Session andCompetition on Single Objective Real-Parameter Numerical
Optimization", Technical Report 201311, Computational
Intelligence Laboratory, Zhengzhou University, Zhengzhou,
China and Technical Report, Nanyang Technological University,
Singapore, December 2013.

1622

1621.5

0

1000

2000

3000

4000

5000
Iterations

6000

7000

8000

9000

10000

Fig. 3. Example of convergence curve (Function 16).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

252

Tele-Operation of a Legged Robot by a Virtual Marionette System- First report: The first prototype and the usefulness of the reaching taskNoritaka Sato, Yasuhiko Sawai, Ryo Asami, Makoto Kitani, Yoshifumi Morita

Department of Electrical and Mechanical Engineering, Nagoya Institute of Technology, Gokiso-cho, Syowa-ku,
Nagoya, Aichi 466-8555, Japan
Tomofumi Fujiwara, Takahiro Endo, Fumitoshi Matsuno

Department of Mechanical Engineering and Science, Kyoto University, Katsura Campus, Nishikyo-ku,
Kyoto, Kyoto 615-8530,Japan
E-mail: sato.noritaka@nitech.ac.jp
www.nitech.ac.jp/

Abstract
Generally, the target position of a hand or a foot is input into a virtual space by using a mouse as a user interface ofa legged robot. However, its operation time is too long. Further it is difficult for the operators to recognize the spatial
relationship between a robot and its environment. To solve these problems, we have proposed an interface that mixes
and presents both visual and haptic senses. In this study, we have adapted the system to the operation of a legged
robot.
Keywords: Rescue robot, User interface, Virtual Reality, Tele-operation

1. Introduction
Many rescue robots have been developed to gather
information instead of humans at disaster sites, such as
the sites of the Great East Japan Earthquake, the terrorist
attacks against the US on September 11, and otherdisasters [1][2]. Generally, these robots have been
crawler-type because they can move on the rough terrains
by using their tracks. However, a crawler-type rescue
robot cannot move up and down a ladder and is weak atmoving on circular stairs. Therefore, legged robots haverecently received a lot of attention. Currently, it isdifficult for an operator to tele-operate a legged robot,
because a legged robot has many degrees of freedom.
The Defense Advanced Research Projects Agency(DARPA) Robotics Challenge (DRC), which is aninternational competition for legged robots staged by
DARPA, was participated in December 2013. Figure 1
shows the user interface of DRC-HUBO for tele-

Fig. 1 Interface of DRC-HUBO [3]operation of the robot [3]. This is a typical user interface
which was found in the DRC. The operator inputs a target
position for the hand or the foot of the robot in a 3D
virtual space. The environment around the robot wasreconstructed in this virtual space. However, because theteam used a plane display to display the virtual space, it
is difficult for the operator to recognize the positional
relationship between the robot and its environment.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

253

Noritaka Sato, Yasuhiko Sawai, Ryo Asami, Makoto Kitani, Yoshifumi Morita, Tomofumi Fujiwara, Takahiro Endo, Fumitoshi Matsuno

Moreover, the team used a normal mouse to input thetarget position, and it took a relatively long time to input.
On the other hand, Arai et al. have developed the
Marionette System [4], which allows the operator to
input the posture of the robot directly by using a small
robot which has the same configuration as the controlled
robot. However, we think that it took time and money toapply their system to various robots, and the usability
may not be good because the environment cannot be
displayed near the small robot.
We have developed a system which can display visualand haptic senses simultaneously to the operator so thatthe operator can tele-operate a crawler-type rescue robot
intuitively and to reduce the operational load [5]. Figure
2 shows our developed system. The system shows theimages shown in Fig. 3 to the operator by using astereoscopic head mounted display (HMD). The HMD
shows computer-generated (CG) images of a model ofthe robot, the environment around the robot, and the handof the operator. Moreover, a haptic device is used toprovide a reaction force against the motion of pinching,at the thumb and the index finger of the operator, when apart of the CG model of the robot is located between the
CG model of the thumb and the first finger in the virtual
space.
In this paper, our developed system will be applied to thetele-operation of a legged robot.
As mentioned previously, we believe that theprevious user interfaces, [3] and [4], have the following
issues:- It took a long time to input the target position in the 3D
virtual space.
- It is difficult to recognize the positional relationshipbetween the robot and its environment.
- It is necessary to develop a real small “Marionette”
robot when the controlled robot is changed.
Therefore, in this study, a new user interface for the teleoperation of a legged robot, which overcomes the above
problems, was developed based on our proposed system[5] for a crawler-type rescue robot.
We called this new user interface the “Virtual
Marionette System (VMS)”. The purpose of this study
was to verify the effectiveness of the VMS by carryingout experiments, in which the operator changed theposture (configuration) of the robot by inputting a target
position for its hand.

HMD

Haptic device
Fig. 2 Robot operation device with haptic device [4]Fig. 3 Image showed by HMD[4]Fig. 4 Overview of the system in this research

2. Proposed system
2.1. Overview of the proposed system
Figure 4 shows an overview of the system. The left-upper
image is the controlled robot. The left-bottom image is aview which the operator sees through the HMD. In thisimage, CG image consists of a model of the robot, theenvironment around the robot, and the operator’s fingers.
The right image of Fig. 4 shows the situation of theoperator. When the operator moves his/her fingers or
hand, the CG models of the fingers in the virtual space
are moved to the positions corresponding to the real
fingers. The operator can control the robot in the virtual
space by using the CG models of the fingers. When the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan254

Tele-operation of a Legged

operator pinches the CG models of the hand or the footof the robot he/she feels the reaction force from the haptic
device. Therefore, the operator feels that he/she directly
pinches the hand or foot of the real robot in the remote
site.
The operator can change the posture of the robot while
he/she pinches the CG model of the hand or foot of therobot as he/she wishes. After he/she releases the CG
model, the system calculates the joint angles of the robot
to achieve the target position of the hand or foot andsends a command to the robot. In this study, we used thesimulator named “Gazebo” instead of a real robot for theexperiment.

Fig. 5 System configuration

2.2. System configuration
Figure 5 shows a system configuration of the proposedsystem. To measure the position and posture (x, y, z, roll,
pitch, and yaw) of the thumb and the index finger of theoperator, a Touch 3D stylus, developed by 3D Systems,
Inc., was used. Note that the function of generation of thereaction force by the Touch 3D stylus was not used in thissystem.
To apply a reaction force to the thumb and index
finger of the operator when a CG model exists between
the CG models of the thumb and index finger, a servo
motor (Dynamixel AX-18 developed by ROBOTIS, Inc.)was used. The servo motor was installed at the tip of thepen of the Touch 3D stylus, and parts similar to thehandle of a pair of scissors were made by using a 3D
printer and were attached to the joint of the servo motor.
Because the servo motor can measure the angle of thejoint, the system could calculate the distance between the
thumb and index finger. The system could then represent
this in the virtual space with the CG model of the fingers.
An Oculus Rift DK2 was used for the HMD of our
proposed system. As mentioned previously, the screen of
the HMD shows CG images of a model of the robot, theenvironment around the robot, and the fingers of theoperator in a 3D virtual space.

Fig. 6 The flowchart of operation process

Fig. 7 Operation process

situation. When the operator wants to move the robot
manually, he/she would stop the automatic control of therobot, and then he/she would tele-operate the robot. After
tele-operation, he/she could change the control mode of
the robot from tele-operation to automatic control.
In the case of the tele-operation mode, the operator
can move the CG model of the hand, the foot, or the bodyof the robot by using the CG models of the fingers of theoperator, which are synchronized with the operator’s
actual fingers, in the virtual space.

2.3. Operation procedure
Figure 6 shows a flow chart of the tele-operation of a
legged robot with our proposed system. Our system wasdesigned to be used when the operator wants to move
manually, such as in an emergency situation. We
assumed that the robot moves automatically in a normal© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

255

Noritaka Sato, Yasuhiko Sawai, Ryo Asami, Makoto Kitani, Yoshifumi Morita, Tomofumi Fujiwara, Takahiro Endo, Fumitoshi Matsuno

Figure 7 shows the operation process for the foot of
the robot. First, the operator grasps the body with his/herright or left hand and the foot (Leg Tip) with his/her other
hand by using the CG models of the fingers. This is asimilar interaction to the changing of the configuration ofa doll. The color of the CG models of the fingers is a flesh
color and the color of the CG model of the grasped partsof the robot is orange. After grasping, the operator could
move the foot of the robot, and the target joint angles ofthe leg of the CG model of the robot were calculated by
the inverse kinematics, which are explained in the next
section. The joint would then automatically move toachieve the target foot position. When the operator
grasped and moved the CG model, the target posture of
the leg was displayed as a translucent CG model. Afterthe operator moved the foot to the target position andreleased the CG model of the foot, the leg of the real robot
was controlled to achieve the posture of the translucent
CG model. By this procedure, the operator could control
the robot as if he/she were grasping a miniature of therobot on the diorama existing in front of him/her or as if
he/she were becoming a giant and grasping the robot withhis/her hands.

Fig.8 The weigh value of the leg angle

Fig.9 Experiment task

Therefore, the solutions obtained by inverse kinematics
were evaluated by
=∑
−
,
(1)
where

2.4. Inverse kinematics

⎧
⎪
⎪
⎪

In our proposed system, IKFast, which is open source
software, was used for calculation of the inverse
kinematics. IKFast is a module for inverse kinematics in
OpenRAVE and by using IKFast the calculation could be
performed quickly. To use IKFast, the developer sets up
a model file of a controlled robot. When a single target
position of the end effector is input, IKFast outputs thejoint angles to move the end effector on a linear trajectoryfrom the current position to the target position withconstant velocity.
When multiple target positions are input, the final one isset as the final target position and the others are set as
way points.
When multiple solutions were obtained by inverse
kinematics, the solution which satisfies the following
conditions was used in this study:- The minimum sum of the differences between the target
joint angles obtained by the inverse kinematics and thecurrent joint angles.
- Each joint angle obtained by the inverse kinematics isnear the center of the range of the movement

⎨
⎪
⎪
⎪
⎩

= 10 − 10 ∗(>= 10 − 10 ∗

,

−
−
−
−

>

( ≤
, ≤
=1
(otherwise)

−1
−1

)

.

(2)

)

In Eq. (2),,, andare maximum,
minimum and center angle of the range of the movementof i-th joint, respectively. In Eq. (1) and Eq. (2), is theangle of the i-th joint of the k-th solution obtained byinverse kinematics. Figure 8 shows a graph of the weight
coefficient . The solution which can minimize theevaluation function , was used to control the robot.
3. Experiment
The purpose of the experiment in this study was to verify
the effectiveness of the proposed system compared withthe conventional tele-operation system [2].

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

256

Tele-operation of a Legged

Table 1 Questionnaires
Number
Q1
Q2
Q3

Question
Were you able to recognize depth of thedisplayed environment?
Is it eagy for you to decide the leg position inthree dimensions?
Were you able to control the robot as you think?

3.1. Experimental method
Figure 9 shows the task of the robot in this experiment.
The robot is the WAREC-1 developed by Takanishi and
Hashimoto et al. [6]. We used a simulator built in Gazebo
for this experiment. One white ball was displayed near
the robot in the 3D virtual space. The operator grasped
and moved the CG model of the hand or foot of the robot,
by using the proposed system or the conventional system,to the ball. After touching the ball, a new ball was
displayed at a different position. The operator repeatedthis reaching task 10 times. The position and the order ofthe balls were the same during all the trials (One trial
consisted of 10 ball-reaching tasks.)
The total time was recorded for comparison. The
number of the subjects was 10. Five subjects used the
proposed system first and the other five used theconventional system first. After the experiment, thesubjects answered a questionnaire about the usability ofthe tele-operation system. The score 5 meant “I think sovery much,” 4 meant “I think so,” 3 meant “I cannot agree
with,” 2 meant “I don’t think so,” and 1 meant “I don’t
think so very much.”

Fig.10 The time of achieving the task

Fig.11 The questionnaire answer

position was input directly into the virtual space with theproposed system.
The answers to Q1 show that the operator recognized
the depth of the environment more easily with theproposed system. Moreover, the answers to Q2 show thatthe operator could input the target position of the hand or
foot of the robot more easily with the proposed system.
We confirmed that inputting the target position andrecognition of the environment was easier when using the
proposed system compared to the conventional system.

3.2. Experimental results
Figure 10 shows the average of the total time of the trial
with each tele-operation system. A paired t-test showsthat the total time was significantly shorter using the
proposed system than using the conventional system(p<0.05). Figure 11 shows the average score of thequestionnaires. The answers of Q1 and Q2 aresignificantly better when using the proposed systemcompared to using the conventional system (p<0.05).

4. Conclusion
In this study, a novel tele-operation system for a legged
robot was developed to solve the issues of existing
systems. We carried out an experiment in which theoperator moved the hand or the foot of a legged robot bytele-operation. As a result, we confirmed that inputting
the target position and recognition of the environment

3.3. Discussion
The total time was shorter when using the proposedsystem. The reason for this seems to be that the operator
can more easily feel the depth distance because astereoscopic HMD was used and the three dimensional© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

257

Noritaka Sato, Yasuhiko Sawai, Ryo Asami, Makoto Kitani, Yoshifumi Morita, Tomofumi Fujiwara, Takahiro Endo, Fumitoshi Matsuno

was easier when using the proposed system compared tothe conventional system.
In the future, we will carry out the experiment with areal robot and will apply the proposed system to the teleoperation of a legged robot in other tasks, such as stair
climbing, ladder climbing and the motions of recoveringfrom falling down.
Acknowledgements
This research was funded by ImPACT Program ofCouncil for Science, Technology and Innovation(Cabinet Office, Government of Japan).
References
1. J. Casper, R. Murphy, Human-Robot Interactions during
the Robot-Assisted Urban Search and Rescue Response at
the World Trade Center, IEEE Transactions on Systems,
man, and cybernetics society, vol. 33, No.3, pp. 367-385,
2003.
2. R. R. Murphy, S. Tadokoro, D. Nardi, A. Jacoff, P. Fiorini,
H. Choset, A.M. Erkmen, Search and Rescue Robotics,
Springer Handbook of Robotics, pp. 1151-1173, 2008.
3. M. Zucker, S. Joo, M. X. Grey, C. Rasmussen, E. Huang,
M. Stilman and A. Bobick, A general-purpose system forteleoperation of the DRC-HUBO humanoid robot, Journalof Field Robotics, Vol. 32, Issue 3, pp.336-351, 2015.
4. T. Takubo, K. Nishii, K. Inoue, T. Arai, Wholebody Teleoperation for Humanoid Robot by Marionette System,
Journal of Robotics Society of Japan, Vol. 25, No. 3,
pp.457-465, 2007. (In Japanese)
5. K. Kato, N. Sato, Y. Morita, Development of Direct
Operation System for Mobile Robot by Using 3D CG
Diorama, Proc. 12th International Conference on Control,
Automation and Systems, pp. 1486-1490, 2012.
6. T. Matsuzawa, K. Hashimoto, X. Sun, K. Uryu, A.
Koizumi, S. Hamamoto, T. Teramachi, A. Takanishi,
Development of Disaster Response Robot with Commonly
Structured Limbs and Experiment in Climbing Vertical
Ladder, Proc. International Conference on Advanced
Mechanics, pp.142-143, 2015.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan258

k-Neighborhood Template A-Type Two-Dimensional Bounded Cellular Acceptors
Makoto Sakamoto, Yu-an Zhang, Masamichi Hori, Haruka Tsuboi, Satoshi Ikeda, Kenji Aoki

Faculty of Engineering, University of Miyazaki, 1-1 Gakuen Kibanadai Nishi
Miyazaki, Miyazaki 889-2192, Japan
E-mail: sakamoto@cs.miyazaki-u.ac.jp
Tsutomu Ito, Takao Ito

Institute of Engineering, Hiroshima University, 4-1, Kagamiyama 1-chome
Higashi-Hiroshima, Hiroshima 739-8527, Japan
E-mail: itotakao@hiroshima-u.ac.jp
Yasuo Uchida

Department of Business Administration, Ube National College of Technplogy, Tokiwadai
Ube, Yamaguchi 755-8555, Japan
E-mail: uchida@ube-k.ac.jp
Tsunehiro Yoshinaga

Department of Computer Science & Electronic Engineering,
National Institute of Technology, Tokuyama College, Gakuendai
Shunan, Yamaguchi 745-8585, Japan
E-mail: yosimaga@tokuyama.ac.jp

Abstract
In this paper, we deal with three-dimensional computational model, k-neighborhood template A-type twodimensional bounded cellular acceptor on three-dimensional tapes, and discuss some basic properties. This model
consists of a pair of a converter and a configuration-reader. The former converts the given three-dimensional tape totwo-dimensional configuration. The latter determines whether or not the derived two-dimensional configuration is
accepted, and concludes the acceptance or non-acceptance of given three-dimensional tape. We mainly investigate
some open problems about k-neighborhood template A-type two-dimensional bounded cellular acceptor on threedimensional tapes whose configuration-readers are L(m) space-bounded deterministic (nondeterministic) twodimensional Turing machines.
Keywords: acceptor, configuration-reader, converter, neighbor, space-bounded, three-dimension, Turing machine.

automata as the computational models of threedimensional pattern processing has been meaningful.
From this point of view, we proposed several threedimensional automata as computational models of threedimensional pattern processing, and investigated their

1. Introduction
Due to the advances in many application areas such as
computed tomography, robotics, and so on, the study ofthree-dimensional pattern processing has been of crucial
importance. Thus, the study of three-dimensional© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

259

Makoto Sakamoto, Yu-an Zhang, Masamichi Hori, Haruka Tsuboi, Satoshi Ikeda, Kenji Aoki, Tsutomu Ito, Takao Ito, Yasuo Uchida, Tsunehiro
Yoshinaga

several accepting powers [2]. By the way, in the multidimensional pattern processing, designers often use astrategy whereby features are extracted by projecting
high-dimensional space on low-dimensional space. So,from this viewpoint, a new computational model, kneighborhood template A-type two-dimensional boundedcellular acceptor (abbreviated as A-2BCA(k) (k ∈{1,5,9})) on three-dimensional tapes was introduced, anddiscussed some basic properties[3]. An A-2BCA(k) is athree-dimensional automaton which consists of a pair ofa converter and a configuration-reader. The former
converts the given three-dimensional tape to the twodimensional configuration and the latter determines theacceptance or non-acceptance of given three-dimensional
tape whether or not the derived two-dimensional
configuration is accepted. This paper mainly investigatessome open problems about accepting powers of A2BCA(k)’s whose configuration-readers are L(m) spacebounded
deterministic(nondeterministic)
twodimensional Turing machines [1]. We first show that a
relationship between the accepting powers ofdeterministic A-2BCA(1) and nondeterministic A2BCA(1). We then show that there exists a language
accepted by any deterministic A-2BCA(9).

Fig.1. Three-dimensional tape.
Furthermore, we definex[(i1,i2,i3),(i’1,i’2,i’3)],
when 1≤ij≤i’j≤lj(x) for each integer j(1≤j≤3), as the threedimensional tape y satisfying the following (i) and (ii):(i) for each j(1≤j≤3), lj(y)=i’j-ij+1;(ii) for each r1, r2, r3(1≤r1≤l1(y), 1≤r2≤l2(y), 1≤r3≤l3(y)),
y(r1, r2, r3) = x(r1+i1-1, r2+i2-1, r3+i3-1). (We call x[(i1, i2,i3), (i’1, i’2, i’3)] the x[(i1, i2, i3), (i’1, i’2, i’3)]-segment ofx.).
We now introduce a k-neighborhood template A-type
two-dimensional bounded cellular acceptor (A-2BCA(k)),
which is a main object of discussion in this paper.
Definition 2.2. Let A be the class of an automaton
moving on a two-dimensional configuration. Then, an A2BCA(k) M is defined by the 2-tuple M=(R,B). R and B
are said to be a converter and a configuration-reader inview of its property, respectively.
(1) R is a two-dimensional infinite array consists of thesame finite state machines and is defined by the 6-tulpe
M = (Z2, N2, K, Σ, σ, q0), where① Z is the set of all integer, and the finite state machines
are assigned to each point of Z2(=Z×Z). The finite statemachine situated at coordinates (i,j)∈Z2 is called the(i,j)-th cell and denoted by A(i,j),② N2 (⊆Z2) represents the neighborhood template ofeach cell and N2 = {(i,j) |－1≤i,j≤1},③ K is a finite set of states of each cell and contains q#(the boundary state) and q0 (the initial state),④ Σ is a finite set of input symbols (#∉Σ is the boundary
symbol),⑤ σ:K9 × (Σ ∪ {#}) → 2K is the cell state transition
function. Let qi,j(t) be the state of the A(i,j) at time t. Thenqi,j(t+1) ∈ σ(qi-1,j-1(t), qi-1,j(t), qi-1,j+1(t), qi,j-1(t), qi,j(t),qi,j+1(t), qi+1,j-1(t), qi+1,j(t), qi+1,j+1(t), a), where a is the
symbol on the A(i,j) at time t. If qi,j(t) = q#, however,qi,j(t+1) = {q#} for each (i,j) ∈ Z2 and each t≥0.
(2) A set of input symbols of B is K－{q#} (where B ∈A). Intuitively, M=(R,B) moves as follows, given a threedimensional input tape x∈Σ(m1,m2,m3) (m1, m2,m3≥1) (x issurrounded by the boundary symbol #).
First, each cell A(i,j) of R(1≤i≤m1,1≤j≤m2,) reads eachsymbol on the first plane x(i,j,1) in the initial state q0, andall of the other cells read the boundary symbols #’s in theboundary state q#’s at time t=0. Starting from this

2. Preliminaries
Definition 2.1. Let Σ be a finite set of symbols. A threedimensional tape over Σ is a three-dimensional
rectangular array of elements ofΣ. The set of all threedimensional tapes over Σ is denoted byΣ (3). Given atape x∈Σ(3), for each integer j(1≤j ≤3), we let mj(x) be
the length of x along the jth axis. The set of all x∈Σ(3)with l1(x)=m1, l2(x)=m2, and l3(x)=m3 is denoted by ∑(m1,m2,m3). If 1≤ij≤lj(x) for each j(1≤j≤3), let x(i1,i2,i3) denotethe symbol in x with coordinates (i1,i2,i3), as shown in
Fig.1.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan260
1

k-Neighborhood Template A-Type Two-Dimensional

condition, R keeps reading x according to the cell statetransition function, and moving down the cell array by
one plane, every time R reads one plane all.
Next, B starts to move regarding a two-dimensional
configuration of R just after R finished reading x as a twodimensional input tape and determines whether or not can
accept the configuration. If B accepts it, x is said to be
accepted by M. Let T(M) be the set of all accepted threedimensional tape by M (see Fig.2).

(b) 5-neighbor.

(c) 9-neighbor
Fig.3. Illustration of neighborhood templates.
Definition 2.4. If the image generated by σ in Definitions
2.2 and 2.3 is a singleton, the converter is said to be
deterministic, and if not, it is said to be nondeterministic.
An A-2BCA(k) (k ∈ {1,5,9}), which converter isdeterministic (nondeterministic), is said to be adeterministic (nondeterministic). A-2BCA(k) and denotedby A-2DBCA(k) (A-2NBCA(k)).
We now consider the class of two-dimensional
automata described by the following abbreviations as theclass of the configuration-reader of A-2BCA(k) A.
In this paper, we assume that the reader is familiar withthe definition of these automata. If necessary, see [1].
・2-DTMS(L(m))…The class of L(m) space-bounded
deterministic two-dimensional Turing machine・2-NTMS(L(m))…The class of L(m) space-bounded
nondeterministic two-dimensional Turing machine・ DOS … The class of deterministic two-dimensional
on-line tessellation acceptor
・ DBS … The class of deterministic one-dimensionalbounded cellular acceptor
For example 2-DTMS(L(m))-2DBCA(9) represents suchthe class as its converter is deterministic and 9neighborhood, and its configuration-reader is an L(m)space-bounded deterministic two-dimensional Turingmachine. Moreover, for any A ∈ {2-DTMS(L(m)),
2NTMS(L(m))}, for any X∈{D, N} and for any k∈{1,5,9}, the class of set of all three-dimensional tapes acceptedby A-2XBCA(k) is denoted by L[A-2XBCA(k)].
Especially, the class of all three-dimensional tapes
accepted by A-2XBCA(k) whose input tapes are restricted
to cubic ones is denoted by LC[A-2XBCA(k)], and theclass of set of all square tapes accepted by A is denoted

Fig.2. A-2BCA(k).
Definition 2.3. An A-2BCA in Dfinition 2.2 is called a 9neighborhood template A-2BCA. If ② and ⑤ of (1) in
Definition 2.2 in replaced with ②’ and ⑤’ in Table 1,each A-2BCA is called a 1-neighborhood template A2BCA and a 5-neighborhood template A-2BCA,
respectively. From now on, we denote k-neighborhood
template A-2BCA by A-2BCA(k) (k∈{1,5,9}) (see Fig.3).

Table 1. 1- or 5-neighborhood template A-2BCA.

(a) 1-neighbor.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

261

Makoto Sakamoto, Yu-an Zhang, Masamichi Hori, Haruka Tsuboi, Satoshi Ikeda, Kenji Aoki, Tsutomu Ito, Takao Ito, Yasuo Uchida, Tsunehiro
Yoshinaga

by LS[A]. In this paper, we discuss regarding the threedimensional input tapes as cubic ones all.

It is clear that T(M’)=T(M) for M’=(R’,B’) constructedin this manner.
□

3. Main Results

Next, we show that there exists a language accepted by
a 2-DTMS(0)-2NBCA(1), but not accepted by any 2NTMS(L(m))-2DBCA(9) for any L(m)) = o(ℓog m).
Theorem 3.2. For any function L(m)=o(ℓog m),
LC[2DTMS(0)-2NBCA(1)]-LC[2-NTMS(L(m))-2DBCA(9)] ≠ φ.
Proof: Let C={w02w12…2wk | k≥1 & ∀i(0≤i≤k) [w1∈{0,1}+] & ∃j(0≤j≤k) [w0=wj r]}(where, for any onedimensional tape w, wr denotes the reversal of w), andT1={x∈{0,1,2}(3)+ |∃m≥3[l1(x) = l2(x) = l3(x) = m &x[(1,1,m), (1, m, m)] ∈C]}. Then, by using a techniquesimilar to that in the proof of Lemma 2(1) in [3], weshow that T1 is accepted by 2DTMS(0)-2NBCA(1), but
not accepted by any 2-NTMS(L(m))-2DBCA(9) for any
L(m)=o(ℓog m).
□

In this section, we discuss some properties of A2BCA(k)’s whose configuration-readers are L(m) spacebounded
deterministic(nondeterministic)
twodimensional Turing machines. First, we show that a
relationship between determinism and nondeterminism.
Theorem 3.1. For any X ∈ {D,N}, LC[2-XTMS(m2)2DBCA(1)]= LC[2-XTMS(m2)-2NBCA(1)].
Proof: From above definition, it is obvious that LC[2XTMS(m2)-2DBCA(1)⊆LC[2-XTMS(m2)-2NBCA(1)]. We
below show that LC[2-XTMS(m2)-2NBCA(1)] ⊆ LC[2XTMS(m2)-2DBCA(1)]. We now show only that LC[2DTMS(m2)-2NBCA(1)] ⊆ LC[2-DTMS(m2)-2DBCA(1)](We can prove another case (i.e., X=N) in the same way).
Now, let M=(R,B) (R=(Z2, N2, K, Σ, σ, q0), B=(K, Q,
Σ, Γ,δ, p0, F)) be some 2-DTMS(m2)-2NBCA(1) (see[1-5,8], if you would like to know about the 7-tuple
constructs B). Then, we consider an M’=(R’,B’) (R’=(Z2,N2, K’, Σ , σ ’, q0’), B’=(K’, Q, Σ , Γ , δ ’, p0’, F’))constructed as follows.
(1) Construction of R’① K’=2(K-{q#}) ⋃ {q#}, q0’={q0}.
② For any a∈Σ and any K’’∈K’-{q#},σ’(K’’, a) = ⋃q∈K”σ(q,a).
(2) Construction of B’
For any p∈Q and any K’’∈K’-{q#},δ’(p, K’’) = ⋃r∈K”δ(p,r).
Intuitively, we explain the movement of M’=(R’, B’)constructed in this way. Let us suppose that a threedimensional tape x ∈ Σ(3)+ is given to M’. R’ is oneneighborhood, so we can consider each cell of R’ as usual
one-dimensional finite automata. Then, each cell of R’
moves to store all states that each corresponding cell of
R can enter in each state at each time by using the wellknown subset construction method (see (1)).
B’ nondeterministically chooses only one state fromeach state of each cell of R’, and simulates the movementof B regarding the selected states as the input symbol. If
B’ can not accept the input, B’ selects the next input andsimulates the movement of B. From the way such as theabove manner, B’ checks the all input patterns, and if B’
can accept one input, B’ can accept the configuration ofR’ (see (2)).

From Theorem 3.2, we get the following.
Corollary 3.1. For any L(m) = o(log m) and any X ∈ {D,N}, LC[2-XTMS(L(m))-2DBCA(1)] ⊊ LC[2-XTMS(L(m))2NBCA(1)].
Remark 3.1. In Theorem 3 in [3], it has already shownthat for any function L(m) and any k ∈ {5,9}, LC[2XTMS(L(m))-2DBCA(k)]⊊LC[2-XTMS(L(m))-2NBCA(k)].
Finally, by using the well-known technique, we canshow that there exists a language accepted by a DOS2NBCA(1) and a DBS-2NBCA(1), but not accepted by any
2-DTMS(L(m)) – 2DBCA(9) for any function L (m) =o(log m).
Theorem 3.3. For any function L(m) = o(log m), (LC[DOS-2NBCA(1)] ⋂ LC[DBS-2NBCA(1)])-LC[2-DTMS(L(m))-2DBCA(9)] ≠φ.
4.

Conclusion

In this paper, we showed some properties of A2BCA(k)’s. We conclude this paper by giving thefollowing problem. For any X∈{D,N} and any L(m) (logm≤ L(m) and L(m) =o(m2)), LC[XTMS(L(m))–2DBCA(1)]⊊ LC [XTMS(L(m))–2NBCA(1)] ?
References

1. K. Inoue and I. Takanami, A Survey of Two-Dimensional
Automata Theory, Information Sciences, 55 (Elsevier Inc.,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan262

k-Neighborhood Template A-Type Two-Dimensional

Amsterdam, Netherlands, 1991), pp.99-121.
2. M. Sakamoto, Three-Dimensional Alternating Turing
Machines, Ph.D. Thesis (Yamaguchi University,
Yamaguchi, Japan, 1999).
3. H. Taniguchi, K.Inoue, and I.Takanami, k-Neighborhood
Template A-Type 2-Dimensional Bounded CellularAcceptor, IECE of Japan Transactions D, 69(3) (IECE,
Tokyo, Japan, 1986), pp.291-301.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan263

An efficient structure of organization with complete individual guidance
Mamoru Yoshimura*

Graduate School of Engineering, University of Miyazaki
E-mail: hm13056@student.miyazaki-u.ac.jp
Tsutomu Ito†

Graduate School of Engineering, Hiroshima University
E-mail: 0va71-2538f211n@ezweb.ne.jp
Makoto Sakamoto*Department of Computer Science and System Engineering, Miyazaki University
E-mail: sakamoto@cs.miyazaki-u.ac.jp
Takao Ito†

Graduate School of Engineering, Hiroshima University
E-mail: itotakao@hiroshima-u.ac.jp
Yuji Shinano‡

Zuse Institute Berlin

E-mail: shinano@zib.de
Satoshi Ikeda*

Department of Computer Science and System Engineering, Miyazaki University
E-mail: bisu@cs.miyazaki-u.ac.jp

Abstract
In the previous paper, we reported on the efficient organizational shape under a setting called complete group
guidance. In this paper, we investigate the shape of an efficient organization in the case of organizational
communication in general, and report the existence of shapes that have not seen so far.
Keywords: Combinatorial Optimization, Efficient structure of organization, Weight policy, Complete guidance3) The output of each member is classified as internal
contribution which is the effort to maintain theorganization and external contribution which is theeffort spent on achieving the purpose of theorganization.
4) The evaluation value of the organization structure is
the sum of external contributions of each member inthe organizational structure.
In the paper [1,2], if the organization has only one
evaluation measure, the hierarchical organization
structure that maximizes the evaluation value is classified

1. Introduction
Recently, a mathematical model that can quantitatively
evaluate the structure of the organization was proposed[1, 2, 3]. The basic idea of this model is as follows.
1) Represent a hierarchical organization as a rooted treewith its members as their nodes.
2) The effort of each member of the organization isrepresented as input / output working in the directionof the leaf node.

*†‡

1-1 Gakuen Kibanadai, Miyazaki, 889-2192, Japan
1-4-1 kagamiyama, Higashi-Hiroshima, 739-8527, Japan
Takustraße 7 14195 Berlin-Dahlem, Germany© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan264

Mamoru Yoshimura, Tsutomu Ito, Makoto Sakamoto, Takao Ito, Yuji Shinano, Satoshi Ikeda

into the three types in Fig. 1 according to the capacity
value of the member.

Fig. 1.

(| | ≥ 1) are finite sets.
Suppose that
andThroughout this paper,is interpreted as the set ofmembers of a given organization, which consists of, , … , . And
is the set of the evaluation measures.
We denote the set of rooted tree graphs with asvertex set as ( ) , and let each tree graph in ( )correspond as an organization tree.
For a given , we call ( , { } ∈ ) an evaluation
systemif: → ℝ ≡ { ∈ ℝ| > 0}
∈ .
We call ( ) the personal ability of ∈ with respect
to an evaluation measure ∈ .
In order for an organization to achieve its purpose toaim at, it is also necessary that appropriate instructions
are transmitted to subordinates from superiors. Thus, fora fixed organization treewith ( ) = , weconsidered that the output of is determined as theinteraction of ”ability value of subordinate ( )” and“accuracy of instruction from superior”.
For the subordinate ∈ who received instructions
from his superior, it is necessary to transmit appropriate
instructions to his own subordinates as superior, while
as subordinate carries out the instructions. For a given
organizational structure tree , we assume that the valueof the input for subordinate ≻ with ∈ ( ) is
to appropriate
obtained by multiplying its weight
instructions to his/her own subordinates as superior.
Weight range is0≤≤ 1,

3 types of efficient tree

Any of these three have an undifferentiatedorganizational structure. This result means that the reason
why modern organization are divided into multiple
divisions is that each division shares a task consisting ofdifferent values.
Then in the paper [3], we modified the mathematicalmodel of the paper [1, 2]. Major improvements arefollows.
1) Generalization of organization’s communicationstyle.
2) Categorize the contribution of each member tointernal and external.
The setting of the paper [1, 2] corresponds to the casewhere the form of communication is a complete group
guidance and the external contribution other than the leaf
node is set to 0.
In this paper, we report on the most efficientorganizational shape of generalized communication style(external contribution is 0).
2.

Let us set

1≤≻ , ∈ ( )≻ , ∈ ( )Mathematical Model≤ deg ∗ ( ).
= 1.

This case corresponds to the organization model which∈
as superior instruct his/her subordinates
individually. The style of communication at this time iscalled complete individual guidance.
Also,

Suppose that = ( ), ( ) is a graph. Throughout
this paper, a graph is a finite undirected simple graph withorder = | ( )|( > 2) and size as= | ( )| .
For ∈ ( ), by ( ) = { |{ , } ∈ ( )}, we denotethe set of vertices adjacent to , and call deg( ) =| ( )| the degree of ∈ ( ).For an arbitrary fixedrooted tree with its root and leaves ( ), we denotedeg( )∈ { } ∪ ( ),deg ∗ ( ) =deg( ) − 1
otherwise.
When… is the path on the tree graph , wedenote( , )≡… .
Writing ≺ for ∈ ( , ), we then define a partial
ordering on ( ), the tree-order associated with .

≻ , ∈ ( )= deg ∗ ( ) ⟺= 1.

This case corresponds to the organization model which∈ as superior complete his/her indication to all
subordinates with only one instruction.
Throughout this paper, we assume that{} ≻ , ∈ ( ) for ∈ \ ( ) is a sequence depends
only on deg ∗ ( ) and ∈ .the selection that we can dois which weight to assign whom. We call the way ofdetermination of a weights’ policy. For any weights’© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan265

An efficient structure ofpolicy, we assume that if deg ∗ ( ) ≥ deg ∗ ( ′) for ,
,
≥
≥⋯≥
∗( )

∈

1
2

and

then {≥≥⋯≥} ≻ , ∈ ( ) and {≤} ≻,= 1,2, … , deg

We will evaluate the rooted treemodel by
( )=

∈

∈

∈ (

) ∈

( ,)

∗( )

∗(

3

,

satisfy

6

) .

( ).

3.2.

Here, ( ) represents the parent node of ∈ , and for( ) the ability
convenience,( ) = 1 . We callvalue of with respect to ( , { } ∈ ) for {}, .
3.

Fig. 2. One of the efficient tree in complete individual
guidance in | | = 1.

Setting 2

The setting is| | = 1,} ≻ , ∈ ( ) = {1.0, 1.0⁄2.0 , … , 1.0⁄deg ∗ ( )},(1) = (2) = 2.0,
(3) = (4) = (5) = 1.2,
(6) = (7) = 0.1.
This case is a general weight policy setting. If deg ∗ ( ) =4 , { } ≻ , ∈ ( ) is {1.0, 1.0⁄2.0 , 1.0/3.0, 1.0⁄4.0} .
An efficient tree is Fig.3.
{

Results

Let us set = {1,2,3,4,5,6,7}, and the interpretation of
the members' ability values is shown in Table 1.
Table 1. Interpretation of the members’ ability value

ability value
Higher than 2
1 or more and 2 or less
Low than 1

47

as organization( )5Interpretation.
High
MiddleLow

1
2

We considered three different settings of Setting 1 andSetting 2 and Setting 3.

3

6

4

5

7

Fig. 3. One of the efficient tree in general communicationstyle in | | = 1.

3.1. Setting 1
The setting is

Evaluation value is 9.36, the weight is= 1.0,= 1.0,= 0.5,= 1.0,= 0.5,= 0.33.
Fig.3. is efficient shapes not found in Fig. 1.

| | = 1,1=,
deg ∗ ( )(1) = (2) = (3) = 2.0,
(4) = (5) = 1.2,
(6) = (7) = 0.1.

3.3. Setting 3
The setting is{This case is a complete individual guidance. A superior
instruct his/her subordinates individually. An efficienttree is Fig.2. Evaluation value is 6.42. Fig.2 is efficientshapes not found in Fig. 1.

= {a, b, c},∗ ( )},} ≻ , ∈ ( ) = {1.0, 1.0⁄2.0 , … , 1.0⁄
(1) = 2.0, (1) = 2.0, (1) = 2.0,
(2) = 2.0, (2) = 0.2, (2) = 1.2,
(3) = 1.2, (3) = 2.0, (3) = 0.2,
(4) = 1.2, (4) = 2.0, (4) = 0.2,
(5) = 1.2, (5) = 2.0, (5) = 0.2,
(6) = 0.2, (6) = 0.2, (6) = 2.0,

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

266
1

Mamoru Yoshimura, Tsutomu Ito, Makoto Sakamoto, Takao Ito, Yuji Shinano, Satoshi Ikeda

multiple evaluation criteria will be as shown in Fig. 6.

(7) = 0.2, (7) = 0.2, (7) = 2.0.
This case is a general weight policy setting in the same
Setting 2. An efficient tree is Fig. 4.
1

2

3

6

4

7
5

Fig. 4. One of the efficient tree in general communicationstyle in | | = 3.

Figure. 6. General form of an efficient tree in the evaluation
system with multiple evaluation criteria.

Evaluation value is 27.28, the weight is= 1.0,= 0.5,= 1.0,= 0.5,= 1.0,= 1.0.
Fig. 4. has a differentiated organizational structure whichis two divisions.

References
1. S.Ikeda, T.Ito and M.Sakamoto,”Discovering the2.

4.

Conclusion

Based on section 3, we found that the general shape ofthe efficient organization tree when the evaluation
measure is only one is Fig. 5.

3.

4.
5.

efficient organization structure: horizontal versus
vertical”, Artificial Life and Robotics, vol.15, 4,
pp.478-481, 2011.
S.Ikeda, “An Efficient Structure of Organization”,
International Interdisciplinary Workshop on
Robotics, Ecosystem, and Management, pp.72-78,
2010.
S.Ikeda, T.Ito, M.Sakamoto, T.Ito,”An Evaluation
Model of Internal and External Contributions in
Hierarchical Organizations”, International Journalof Mathematical Model and Methods in Applied
Sciences, Vol.9, pp.636-645,2015.
Reinhard Diestel, “Graph Theory(Third Edition)”,Springer, 2000.
Alvesson Mats, “A Flat Pyramid: ASymbolic
Processing of Organizational Structure “,International Studies of Management andOrganization, 19-4,5-23,1989/1990.

Figure. 5. General shape of an efficient tree.

This means that it is generally shown that there is anefficient organization tree different from the efficientorganization tree shown in the paper [1, 2].
Furthermore, from this result, it is inferred that thegeneral efficient tree shape in the evaluation system with© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

267

Consideration for the Possibility to the Tourism Support Contents
by the Markerless AR Technology
1

Masamichi Hori1

Graduate School of Engineering, University of Miyazaki,
1-1 Gakuenkibanadai-nishi Miyazaki-shi, Miyazaki, 889-2192, Japan
Makoto Sakamoto2＊, Haruka Tsuboi2, Koshiro Mitsuhashi3, Yukari Kodama3,
Takeshi Tanaka4, Mihoko Fukushima4, Chikashi Deguchi4, Masahiro Yokomichi2,
Masayuki Mukunoki2, Kunihito Yamamori2, Atsushi Iiboshi5
2Faculty of Engineering, University of Miyazaki,
1-1 Gakuenkibanadai-nishi Miyazaki-shi, Miyazaki, 889-2192, Japan

3

Miyazaki Multimedia Academy, 2-4-37 takachihodori Miyazaki-shi, Miyazaki, 880-0812, Japan
4

Faculty of Regional Innovation, University of Miyazaki,
1-1 Gakuenkibanadai-nishi Miyazaki-shi, Miyazaki, Miyazaki, 889-2192, Japan
5

Takachiho Muratabi Co., Ltd., 6604 ohazamukouyama takachiho-cho nishiusuki-gun,Miyazaki, 882-1103, Japan

1hm13061@student.miyazaki-u.ac.jp, 2sakamoto@cs.miyazaki-u.ac.jp, 3koshiro@multi-m.jp, 4deguchi@cc.miyazaki-u.ac.jp,
5muratabi@bz04.plala.or.jp (* Corresponding author)

Abstract
Currently, the tourism industry in Miyazaki prefecture has various problems, and various measures are taken. On theother hand, in 2016 virtual technology attracted much attention also on AR technology. We thought that through the
AR technology experience, we could aim for an increase in tourists. Therefore, we will create new applications for
smartphones using AR technology and aim for tourism support. First, prototype an application with markerless typeand features for smartphones etc., and conduct experiments. As a result, there was improvement and it was notcompleted. However, it seemed possible to become an application that realizes tourism support.
Keywords: Promotion, Augmented reality, Sightseeing, Traditional culture.

various problems and countermeasures have been taken[1]. Especially, it is urgent to introduce new technology.
In 2016 and 2017, the virtual technology called AR, VR,
MR showed great excitement and it was the year when it
became commonly recognized. Among them, AR ispremised on the real world and expands it. It can be seen
in a wide range of fields such as advertisement, tourism,
education, entertainment [2].

1. Introduction
Due to the popularization of mobile terminals and theemergence of interactive technology, the tourism
industry is changing drastically. In order to be a tourist
destination that is attractive and win the competition, it isessential to improve unique experiences and value-added
services. Miyazaki Prefecture · Takachiho Town is oneof the most famous sightseeing places in Japan, but it has© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

268
1

M. Hori, M. Sakamoto, H. Tsuboi, K. Mitsuhashi, Y. Kodama, T. Tanaka, M. Fukushima, C.Deguchi, M. Yokomichi, M. Mukunoki, K. Yamamori, A.
IiboshiIn this study, we will link AR with the tourism industry
and aim to solve problems related to sightseeing of
Takachiho Town. Specifically, we aim to producesightseeing applications that have map functions, AR
functions linked with places, and so on. We urge thecreation and re-visit of tourists by giving consumers who
visit sightseeing spots an AR technology experience andoffering "Omochikaeri (it means takeaways)" of tourist
spots that reproduce and share their experiences in any
environment. For that purpose, we will prototype
contents for sightseeing support by using markerless type
AR technology as a fundamental study, and evaluate andconsider it.

Fig.1：Takachiho Gorge(left) and Akimoto
shrine(right).
3. Literature review
According to Kangdon Lee's study [6], education andtraining is one of the areas where AR is mainly used.The
breakdown is a wide range of area such as astronomy,
chemistry, biology, mathematics and geometry. Anotheris business. Also application examples are wide, tourism,museums, games, industries, etc. What we can see fromthis is that the characteristics and maximum benefits ofAR are to extend the information based on reality. It isuseful to be able to visualize what is difficult to recognize
by 3DCG and to approach it as a new cognitive method.
AR is often combined with situation training." A
Situational Training System for Developmentally
Disabled People Based on Augmented Reality" [7] and "Augmented Reality Integrated Simulation Education in
Health Care" [8] is a situation training system build up areality-based level scenario and can repeat. The reality of
the AR is linked with simulation education, andsignificant results are obtained by improving the
cognitive ability.
"ToARist: An Augmented Reality Tourism App createdthrough User-Centred Design" [9] is an AR application
design with an emphasis on discussion with users. This
study was revealed what visitors really wanted in the AR
display, and how the AR navigation should be displayed.
"Augmented Reality and Gamification in Heritage
museums " [10] conducted a definition and analysis of
AR communication model and noise at the museum, andapproached better experience. Furthermore, adding thegame elements revealed that the depth of the AR system
will increase.
As a study of the meta-viewpoint of introducing AR,
there is " How can Tourist Attractions profit from
Augmented Reality?" [11]. This study revealed the AR
profit model through collaboration with stakeholders in
the field and showed the possibility to introduce AR withminimal risk.
The impact on sightseeing in the AR application"Pokèmon GO" which became one of the keywords in2016 is great. According to the official report, the

2. Background
The tourist site that is the target of this study is Takachiho
Town, Miyazaki Prefecture. Takachiho Town is locatedat the northernmost tip of Miyazaki Prefecture and thecenter of Kyushu. It has a beautiful natural treasure
Takachiho Gorge, a great change due to the four seasons,and it is blessed with wonderful nature. Takachiho Gorge
and Akimoto shrine are shown in Figure 1. The historyof the town is old and there are reports that there werevillages since about 4000 BC [3].
It is a place where God descended in Japanese mythology
and is regarded as a land with considerably deep
connection with the birth of Japan.
There are Kagura, born from the Amanoiwato legend ofJapanese mythology, and land that is related to myths. In
recent years as a power spot attracts attention from a wide
range of age groups.
Especially the Akimoto area called Okutakachiho
focuses on people living in the village. And they are
doing have tourists experience village-specificexperiences activities. Tourists can eat cuisine using
regional ingredients, and can visit shrines and sacredtrees standing in majestic nature. In 2016, it was selected
as a good example of the "treasure of Discover rural
mountain fishing village" designated by the Ministry ofAgriculture, Forestry and Fisheries, and it is thought that
tourists will further increase in the future [4].
However, as a change in the environment surrounding
tourism, Miyazaki prefecture states that the population
declines, the declining birthrate and aging, intensifiedcompetition, and the development of information andcommunication technology are left behind.
The number of sightseeing tourists is at a lower level [5]than at peak hours. Creating attractive sightseeing sitesthat you want to visit many times, and sending out
information using media, ICT, and SNS are issues.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

269

Consideration for the Possibility

collaboration event with Miyagi Prefecture in November
2016 mobilized about 100,000 people in 11 days, tourism
consumption amount of 2 billion yen [12], thecollaboration event with Tottori Prefecture in November
2017 mobilized about 89,000 people in 3 days, tourism
consumption amount of 1.3 billion yen and PR effect 500
million yen [13]. Players visit sightseeing spots triggeredby content capabilities, leading to attraction discovery
and economic activities. It is a characteristic result of AR
based on reality.

environment compatible with many platforms, and avariety of plug-ins accompanying it are also present.
5.3. Marker type AR and markerless type AR
The marker type is a technique for specifying thepresentation position of the 3D CG model with a fixedpattern as an input. The markerless one is inputting anatural image (an image having a use other than amarker). An example is shown in figure 3. We'd like to
run it as an application alone, and implement it on amobile terminal, especially on a smartphone.

4. Fundamental study
In this study is fundamental one, a markerless type AR
application is prototyped and compared with the marker
type AR application, characteristics and behavior arecompared. After that, we examine which one is actuallysuitable for applying to tourist spots. Throughconsideration, we show the definition of 'Omochikaeri'function which is unique point of application.

Fig3：AR marker(left) and Natural image as alogomark(right).

5. Methods
I will explain the method used to create the application.

6. Experiment and Result

5.1. 3DCG modeling

We look at the characteristics and operation of marker
type AR application that was used in previous study andmarkerless type AR application and compare them.

About 3DCG model production cooperate withvocational schools in Miyazaki prefecture. The 3DCG
model to be produced is shown in Figure 2. The left is
one of the gods in Japanese myths appearing in
Amanoiwato legend. In addition, the 3DCG model ofMiyazaki University's mascot character of the 3DCG
model used this time is shown on the right side of figure.

6.1. Development environment
We created the entire application using Unity 3 d and theBlender in 3D CG model. The mobile terminal used is an
Android smartphone.
6.2. Comparative experiment
Figure 4 shows the operation results. When we read thetarget with both marker type and markerless type,applications presented the 3DCG model. When thenatural image of the target is placed in the markerlesscamera, it keeps track of the model at high speed even if
the terminal is moved intensely up and down, left andright, and continues to display the model.

Fig.2：Bird’s eye view of Tadikarao(left) and
Miyadaimoukun(right).
5.2. Unity3D
We chose Unity 3d as a software that can produce
applications for mobile from the viewpoint of future
development and application creation that is not boundby OS. Unity 3 d is an integrated game development

Fig.4：markerless type(left) and marker type(right).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan270

M. Hori, M. Sakamoto, H. Tsuboi, K. Mitsuhashi, Y. Kodama, T. Tanaka, M. Fukushima, C.Deguchi, M. Yokomichi, M. Mukunoki, K. Yamamori, A.
Iiboshi

7. Conclusion
In this study, we tried to make an application using Unity
3d. We confirmed the operation which is excellent ininitial recognition and high follow-up performance in themarkerless type. Since natural images are input, we thinkthat it can fit in the landscape without incongruity by
inputting pamphlets etc. in the tourist spots. Also, the costof preparing markers by sightseeing spots is not required,
so I think that it is suitable for introduction. In addition,
through case studies of previous study and experiments,
I felt the lack of many functions until actually applied totourist spots.

2.

3.

4.

8. Future plans
We plan to develop an integrated Takachiho tourism
support application centering on markerless type AR
technology. We plan about the 3DCD model display
function of the attraction by loading the map and thefunction that can take virtual character andcommemorative shooting when reading buildings andsignboards. We also plan basic map functions and shop
profit information that are considered convenient for
visitors.
We plan "Omochikaeri" as one of the unique functions.
This is a function that will become possible when visiting
tourist resources, shrines and spots, it is an idea to unlock
special AR content functions as benefits. One of them
reproduces Kagura on hand. The aim is to have theopportunity to visit by telling Kagura that tourists
actually saw in a user-friendly way of understanding
easily.
However, in implementing these functions, it isimportant to closely discuss with stakeholders of tourist
spots and questionnaire to visitors. Because AR
technology extends the reality.

5.

6.

7.

8.

9.

10.

Acknowledgements
Miyazaki multimedia vocational school people are very
busy, thank you hearty thank you for your cooperation in
3DCG modeling and advice etc. Also, the Miyazaki
Tourism Convention Association, Inc., abc Corporation,Inc., Phoenix Seagaia Resort Co., Ltd., people in the
Akimoto district who took care of me in the interview,other prefectures within the prefecture I would like toexpress my gratitude to all concerned parties in eachmunicipality.
References

11.

12.

13.

1. Heisei27 July Miyazaki Prefecture Tourism Promotion
Plan ~ Toward revival of tourism miyazaki ~. [Online]
Official website of Miyazaki Prefecture
http://www.pref.miyazaki.lg.jp/kanko-suishin/kanko/miryoku/documents/21689_20160301163637-1.pdf (accessed
2017/12 / 1).
Takashi Kurida, Kiyoshi Kiyokawa, Takashi Okuma,"Fundamentals, Development and Practice of AR
Technology", Science Information Publishing, 2015, pp
377-385.
Overview of Takachiho | Introduction / History of the town
| administrative information | Takachiho cho [Online]
Takachiho cho information official website.
http://www.town-takachiho.jp/administration/history/outl
ine.html (accessed 2017/12/1).
Takachiho Mura Council "Treasure of Discover Rural
Farm Village (Mura)" was selected. | What's New |Takachiho-cho Tourist Association [Online]
Takachiho cho information official website.
http://takachiho-kanko.info/news/detail.php?log=147968
8433 (accessed 2017/12/1).
Miyazaki Prefecture tourist statistics survey result inHeisei27. [Online]
Official website of Miyazaki Prefecture
https://www.pref.miyazaki.lg.jp/kanko-suishin/kanko/mir
yoku/documents/26823_20161226163703-1.pdf(accessed 2017/12/1).
Kangdon Lee, “ Augmented Reality in Education andTraining” ，Tech Trends Vol56, No. 2, pp 13-21, 2012.
Tae-Young KIM, A Situational Training System forDevelopmentally Disabled People Based on Augmented
Reality” ， IEICE Transactions on Information and
Systems Vol.E96.D, No. 7, pp 1561-1564, 2013.
Kasey J.CarlsonRN, MSN, MEd , David J.Gagnon MS.
Kangdon Lee, “ Augmented Reality Integrated
Simulation Education in Health Care ” , Clinical
Simulation in Nursing, Vol.12, No.4, pp 123-127, 2016.
Williams, M, Yao, KKK and Nurse, JRC et al.,
“ ToARist: An augmented reality tourism app createdthrough user-centred design. ” , 31st British Human
Computer Interaction Conference, 2017.
Ramy Hammady, Minhua Ma, Nicholas Temple,“ Augmented Reality and Gamification in Heritage
Museums. ” , Serious Games, Vol. 9894, pp 181-187,
Springer, Cham, 2017.
Eleanor E Cranmer, M. Claudia tom Dieck, Timothy Jung,“ How can Tourist Attractions profit from Augmented
Reality? ” , Augmented Reality and Virtual Reality Empowering Human, Place and Business, pp 21-32,
Springer, Forthcoming, 2017.
About the number of tourists etc. of Pokèmon GO related
events - Official website of Miyagi Prefecture. [Online]
Miyagi Prefecture official website.
https://www.pref.miyagi.jp/release/ho20161219-6.html(accessed 2017/12/1).
Number of participants in "Pokémon GO Safari Zone in
Tottori Sand Dunes" and Economic Effect. [Online]
Tottori prefecture official website.
http://www.pref.tottori.lg.jp/271365.htm(accessed 2017/12/1).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

271

Fundamental Study on Tourism Support Using 3DCG
Haruka Tsuboi, Makoto Sakamoto＊, Masamichi Hori, Yosuke Iriyama, Yuki Kai, Hazuki Watanabe
Faculty of Engineering, University of Miyazaki, 1-1 Gakuen Kibanadai Hishi
Miyazaki, Miyazaki 889-2192, Japan
E-mail:sakamoto@cs.miyazaki-u.ac.jp
Yu-an Zhang
Department of Computer Technology and Applications,
Qinghai University 810016 China
E-mail: zhangyuan198101092000@yahoo.co.jp
Atsushi Iboshi
Takachiho Muratabi Co., Ltd.,6604 ohazamukouyama takachiho-cho nishiusuki-gun,Miyazaki, 882-1103, Japan
E-mail: muratabi@bz04.plala.or.jp
Koshiro Mitsuhashi, Yukari Kodama
Miyazaki Multimedia Academy,
2-4-37 takachihodori Miyazaki-shi, Miyazaki, 880-0812, Japan
E-mail: koshiro@multi-m.jp(＊Corresponding author, E-mail:sakamoto@cs.miyazaki-u.ac.jp)

Abstract
Three-dimensional computer graphics (3DCG) is a technique of generating an image having a stereoscopic effectby converting an object in a virtual three-dimensional space into two-dimensional information. It is also possible togenerate images that can not be distinguished from genuine objects using 3DCG. 3DCG is expected to be applied tovisual expression very much. Currently it is expected to be applied in tourism. While being a photorealist, we areaiming to make people interested by images from viewpoints that we can not normally see. In this thesis we willuse 3DCG for sightseeing
Keywords: 3DCG, Blender, Unity

Therefore, we considered fundamental research of thismoment, considering whether we can PR such as we do
not exist by using CG technology for the rich nature andsightseeing spot of Miyazaki prefecture where we live.
In this research, we focused on "Takachiho-cho", which
is known as a land with a connection to "Japanese
mythology" among Miyazaki prefecture(Fig 1, 2).
As a goal, tourism support such as combining thecreated 3DCG model with real underwater images andaerial images, introducing rich nature as the main andhistoric tourist attractions.

1. Introduction
The penetration rate of CG (computer graphics) inrecent years has become remarkable. Currently, there
are various examples of utilization such as movies andanimation, various simulation and tourism projects,
sales visuals and so on.
Also, as the Tokyo Olympic Games are held in 2020,
the demand for tourism in Japan is getting bigger.
Accordingly, we believe that efforts towards tourism
support are necessary even in local government units.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

272
1

Haruka Tsuboi, Makoto Sakamoto, Masamichi Hori, Yosuke Iriyama, Yuki Kai, Hazuki Watanabe, Yu-an Zhang, Atsushi Iboshi, Koshiro Mitsuhashi,
Yukari Kodama

Takachiho-cho, however, has a problem of population
decline due to the declining birthrate and aging
population.
This line chart is the future population estimate of
Takachiho-cho by the survey of the National Institute ofPopulation and Social Security Research(Fig.3).
As you can see, the population tends to decline to theright every year.
Due to this population decline, it is presumed that there
is great influence on the succession of traditional culture
and tourism resources.

Fig.3: Population estimate
2.

Principle

In recent years, CG has become more popular as a tool
for revitalizing tourism projects. It has been making it
possible to provide visitors with the experience gainedby using CG, such as the power and atmosphere that can
only be experienced locally, and the reproduction ofenvironment and buildings lost in the past.
In this research, I am considering attracting nature as themain among them. From a person's point of view only
the scenery lower than the mountain can be seen, butbirds and insects can look down Takachiho from theviewpoint higher than that. Likewise the same thing canbe said for aquatic creatures. Things that can not be seenfrom the usual point of view can be noticed by the good
point that we have not noticed by looking from adifferent point of view. By thinking as a hawk's point ofview, we could imagine what kind of living things areusually living, and thinking that Takachiho could bepromoted from a different viewpoint than before, we
tried out.[1][2]Fig.1: Amanoyasukawara

3.

Creation

This time I used Blender and Unity.
Blender creates animation models and animations.
Unity synthesized with the actual image using the modeland animation created.
Both are software that can be operated sensuously, so
we adopted it because it can be used easily and has highquality.
3.1. ModelingFirst I will create a model of Hawk and a river fish at
Blender. Figures 4 and 5 are created.

Fig.2: Takachihokyo© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan273

Fundamental Study on Tourism

Fig.4: River fish model

Fig.7: Hawk combined with video
Fig.5: Hawk's model
3.2. Material setting
The material setting is done by mapping. This time UV
mapping is executed. UV mapping is a part of texture
mapping and texturing is stretched to a stereoscopicallywrapped part without being stretched by applying a
texture to the two-dimensional expanded view of thesurface of the object. The result is shown in Figure 6.

Fig.8: Fish combined with video
Fig.6: Fabrication of map by UV development

5. Discussion
In this research, we made a blogger and Unity basedon the intention of composing CG of the falcon dancing
in the sky of Takachiho in the sky photographed in drone,and creating a CG animation to PR for Takachiho .
Although river fish could not prepare actual
underwater video for synthesis, I think that we could
expand the range of video production with the functionthat we can actually move on ourselves.
Ultimately, since the video shot in advance is used forthe composition of the video, it is necessary toincorporate the movement according to the video to theright place in the right place. Also, delicate light

4. Result
The function implemented this time can move 3DCG (3dimensional CG) model by user's keyboard operation.
Although it is a function unsuitable for compositing
with the picture taken beforehand, I thought that it
would be interesting to have game properties instead ofjust flowing video, and I implemented it.
A part of the finally formed model and action is shown
below (Fig. 7, 7).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita , Japan274

Haruka Tsuboi, Makoto Sakamoto, Masamichi Hori, Yosuke Iriyama, Yuki Kai, Hazuki Watanabe, Yu-an Zhang, Atsushi Iboshi, Koshiro Mitsuhashi,
Yukari Kodama

adjustment must be considered. In addition, it isnecessary to actually underwater and aerial photographs
diversely, and investigate what type of photographingwill be effective images.

6. Conclusion
In this research, from the viewpoint which is slightly
different from the past, we have created images
introducing the rich nature of Takachiho Town, historicsightseeing spots and people living there, and areinvestigating to link tourism support. In future, I thinkthat it is possible to add "fun" to this tourism support byusing Unity and incorporating game elements as one of
the contents.
References[1] Kai Yuki, Fundamental study on the description ofriver fish using CG, graduation thesis of Information
Systems Engineering, Faculty of Engineering, Miyazaki
University, 2017[2] Yosuke Iriyama, Approach on Tourism Support by
Aerial Photography Using CG, graduation thesis ofInformation Systems Engineering, Faculty ofEngineering, Miyazaki University, 2017

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan275

Clustering Analysis Based on Improved Fuzzy C - Means Algorithm
Qiongqiong Hu,

Department of Computer Technology and Applications,
Qinghai University 810016 China
Yiyang Li,

Department of Computer Technology and Applications,
Qinghai University, Xining 810016 China
Yong Ge,

School of Mechanical Engineering,
Qinghai University, Xining 810016 China
Yu-an Zhang*,

Department of Computer Technology and Applications,
Qinghai University 810016 China
Qinglian Ma,

Interdisciplinary Graduate School of Agriculture and Engineering,
University of Miyazaki, Miyazaki City 889-2192 Japan
Makoto Sakamoto,

Faculty of Engineering,
University of Miyazaki, Miyazaki City 810016 Japan

Abstract
Cluster analysis is one of the most important technologies in the field of data mining, and it is also a hot topic in academic
research. So far, it has made great achievements in theory and method, and plays an important role in data analysis in various
fields . The fuzzy C-means algorithm (FCM) maintains the simplicity of its thinking, the time complexity is close to linear,and it is efficient and scalable for large-scale data mining. Particle Swarm Optimization (PSO) is an optimization tool with
global optimization ability, and the group intelligence is formed by the interaction between particles in the group, and the
optimal result is found by the intelligence . However, with the deepening of algorithm research, its research prospects andapplication areas are increasingly bright and compelling .

Keywords—particle swarm optimization algorithm, clustering analysis, fuzzy C - means algorithm.

Among them, the fuzzy C-means algorithm (FCM) keeps its
thinking simple and easy. The time complexity is close tolinear, and it has high efficiency and scalability for large-scaledata mining [2].After years of development of particle swarm
optimization algorithm [3],its optimization speed, quality andalgorithmic robustness has been greatly improved. However,most of the current research focused on the realization,
improvement and application of algorithms [6]. At present,
there are still many unsolved problems in the study of particle
swarm algorithm:(1) stochastic convergence analysis; (2)determination of algorithm parameters; (3) discrete / binary
particle swarm optimization; (4) the use of different problems
to design a corresponding effective algorithm;(5) Should becommitted to complement and extend the algorithm with otheralgorithms or techniques to solve the problem of PSO easy tofall into the local optimum [4].

I. INTRODUCTION
Clustering analysis is one of the most important technologies
in the field of data mining and plays an important role in data
analysis in all fields [1]. Clustering is an unsupervised andefficient learning method, and its data is similar and unique toeach molecule, so it can be classified according to its different
characteristics. At present, clustering analysis has been widelyused in many aspects, and with the development of science
technology has been more and more attention. In essence,clustering belongs to the local search algorithm, which uses aniterative climbing technique to find the optimal solution [5].
Therefore, the traditional clustering algorithm has two fatal
problems: First, with large-scale data processing, the originalalgorithm fails or takes a lot of time, the second is easy to fall
into the local minimum.
*Corresponding author: Yu-an Zhang, e-mail:2011990029@qhu.edu.cn

The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4,B-Con Plaza, Beppu, Oita, Japan

276

Qiongqiong Hu,Yiyang Li,Yong Ge,Yu-an Zhang,Qinglian ,MaMakoto Sakamoto

gbest and pbest, that is, the individual optimal value and the
population optimal value, in each iteration process, the two
optimal values will be repeated according to the Formula 1
and Formula 2. The advantage of PSO is that the concept isrelatively simple, relatively easy to achieve, while the need toadjust the parameters less, and the convergence rate. The basic
features of the PSO algorithm can be summarized as follows:Intelligent search; progressive optimization; black box
structure; implicit parallelism; flexibility; global optimization;robustness; self-organization.

II. THE PSO ALGORITHM
The basic idea of the particle swarm algorithm is that the firststep is to initialize a group of random solutions to obtain theparticles in the particle swarm. The second step is to find the
optimal value in the algorithm by repeating the result of
repeated feedback. At each repetition the two optimal valuesof pbest and gbest are obtained according to Equation 1 andEquation 2 in the feedback. Fig.1 shows the PSO algorithm
flow chart.

Vi = v i * w

rand （1）* c1 *( rand （1）* c 2(1)
Present[i] = present[i] + v[i](2)
Vi is the velocity value of particle i, present [i] is the position*(gbest[i]+ pbest[i]-present[i]))of the current particle i, w is the inertia weight, c1 , c 2 is thelearning factor, rand (1) is between (0,1) random number.
The execution flow of the PSO can be described as follows:
1.Determine the population size of the particle N, determine
the maximum number of iterations, randomize the velocity
and position of the particles;
2.Calculate the fitness value of each particle according to thefitness function;
3.The current fitness value of each particle is taken as theinitial optimal value of the particle, and the whole optimalvalue is obtained by comparing all the initial optimal values.
4.Update the particle’s position and speed according to the
update formula;
5.Calculate the fitness value of the particles in the new
population;
6.Compare the new fitness value of each particle with theprevious optimal fitness value of the particle. If it is good,
update the individual optimal value of the particle, otherwise
it will not be updated.
7.The new fitness value of each particle and the overall fitness
value of the comparison, if the good, then update the overall
fitness value, or not updated;
8.Repeat steps 5 through 7 until a certain number of iterations
is satisfied.

Fig. 1. PSO optimization flow chart

III. CLUSTER ANALYSIS
A. The basic concept of cluster analysis© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

277

Clustering Analysis Based on

Clustering is a process of sorting according to data characteristics. If a class as a group, you can cluster as a datacompression form. Clustering can be more specific image isdefined as follows: Data space in the data set D in the numberof data points is n, each data points have d attributes, then each

c
i 1

Where

1,2, ...., k). In the process of clustering, it is possible that somedata points do not belong to any clusters. These data points arecalled outliers or noise points.

fuzzy groupi 1, d ij

c

n

i 1

j

m
ij

d ij2

(5)

ci is the clustering center ofx j is the Euclidean distanceci

between the i th cluster center and the jth data point; Is aweighted index. By constructing the Formula 6, we obtain thenecessary conditions for the Formula 5 to reach the minimum
value:

B. FCM (Fuzzy C-means Algorithm) clustering
algorithm
The fuzzy C-means algorithm (FCM) is an improvement of thegeneral C-means algorithm. The ordinary C-means algorithm
is hard for the data partitioning, and FCM is a flexible fuzzy
partition.
First, the concept of membership function is described. The
membership function is a function that represents the degreeof an object x belonging to the set A, usually denoted by A (x),whose range of arguments is all the objects that may belong to
the set A (ie, all the points in the space in which the set A isA( x ) 1 .
located) The value range is [0,1], that is, 0

J (U, c 1，...， c c ， λ 1，...， λ n )
c

n

i 1

j

here,

μ ijm d ij

2

n
j 1

λ j（

J(U, c 1，...， c c )
c
i 1

μ ij

n
j 1

λ j（

c
i 1

μ ij

1）

（6）

1）

j , j = 1 to n, is the Formula 4 of the n constraints of the

Lagrangian multiplier. Generate guidance for all input
parameters. It can be seen that the necessary condition for the
Formula 5 to be minimized is:
n
j 1
n

ci

1 means that x is completely subordinate to set A,

which is equivalent to x A on the traditional set concept. A
membership function defined on space X = {x} defines a fuzzy
set A, or a fuzzy subset defined on the universe X = {x}. For
X , X , X ,......, X n , the fuzzy set can be
the finite objects 1 2 3
expressed as:

x i ，x i x i

Ji

ij is between 0 and 1;

clustering the data set, we can get k different clusters Ci (i =

j 1

ij

n
ij

xj
（7）

m
ij

1
d ij 2（（ ）/ m 1）
k 1 d kjc

（8）that the fuzzy C-means clustering algorithm is essentially arepetitive feedback process. When running in a batch mode,
the FCM algorithm uses the following steps to determine theX(3)Through the concept of fuzzy set, in the problem of cluster
analysis, we can treat clusters clustered by clustering as a
fuzzy set, and all membership points belong to the group [0,1].
In the fuzzy C-means clustering algorithm, it divides n data
into c data sets and finds the center on the basis of this, so thatthe value function of the non-similarity index is minimized.
However, for each data set membership, its sum is equal to 1:Ac...，c c )J (U , c1，

entire data set can be seen as a matrix of ND dimensions. By

A(4)then, at this time FCM value function (or objective function)is:data point can be expressed as X i = ( x i1 , x i 2 ,..., x id ), theA( x )

1， j 1，...，nijclustering center ci and the membership matrix U[1]:Step 1: Initialize the membership matrix U This value requires
a random number in the range 0 to 1 to satisfy the constraint
in Equation 4.
Step 2: Calculate c clustering centers ci, i = 1, ..., c using
Formula 7© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4,B-Con Plaza, Beppu, Oita, Japan

278

Qiongqiong Hu,Yiyang Li,Yong Ge,Yu-an Zhang,Qinglian ,MaMakoto Sakamoto

Step 3: Calculate the value function according to Formula 5.
According to the value of the function to determine whether
the algorithm to stop.
Step 4: Use the Formula 8 to compute the new U matrix.
Return to step 2. The algorithm can also initialize the cluster
center and then repeat it.

implementation process will not record each of the optimal
solution steps will be achieved on the line Error stops, andoutputs the objective function. Based on the limitations of
FCM algorithm, it is easy to fall into the local optimal solution.
Therefore, it is considered that PSO will replace the gradientdescent method in FCM and change it to iterative calculation.
This advantage is that the whole algorithm is advancing in the
optimal solution, which is more conducive to finding the
optimal solution. Although a large number of calculations areadded, the similarity in the group is improved, the nonsimilarity is reduced, and the jb value is optimized.

1) The Achievement of FCM Algorithm
Where x is the 400 points from 0 to 1, that is, 400 sets ofdata, using the Rand () function in excel. By running thealgorithm, the data of the four types of data and the five typesof data are realized, and the value function (objective function)
Jb of the corresponding non-similarity index is obtained. The
following related images are generated using MATLAB 7.0,
and the different shapes in the picture represent different
clusters. Each population center has a triangle that is used torepresent the cluster center. In Fig.2, the four small circles It
is shown that the number of groups is 4 clusters, and the fivesmall circles in Fig.3 show that the number of groups is 5
clusters, and the Jb values are compared to prove theenhancement of the improved algorithm.

Fig. 3. When cn = 5, Jb = 2.0944 (a random result)Fig. 2. When cn = 4, Jb = 3.3191 (result of a random experiment)

C. Apply the particle swarm algorithm to fuzzy Cmeans clustering
Through the above we can see that the FCM clustering in theprocessing of 400 sets of data, the implementation process isin the gradient method of decline, very rigid, step by step theFig. 4. Under the condition of CN = 4, the JB value curve under
different strategies ((FCB represents the fuzzy C-means algorithm,© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan279

Clustering Analysis Based on

PSO represents the idea of integrating the particle swarm algorithm
in FCM, and the Inertia Weight represents the linearly decreasinginertia weight))

Cycle-index

Before improvement

First improvement

Secondary improvement22.116946479
2.127751484
2.114908503
2.123793236
2.118510332
2.116134805
2.108827784
2.107951404
2.110614058
2.11809571
2.127541954
2.115151641
2.113399393
2.109794086
2.109062738
2.129499201
2.120138722
2.110598647
2.115272508

2.091145987
2.091163385
2.091140444
2.091163526
2.091142244
2.091163586
2.091180206
2.091192388
2.091165532
2.091161191
2.091169919
2.091148045
2.091161057
2.09116062
2.09116664
2.091152719
2.091167363
2.091155037
2.091170088

2.081188193
2.081261095
2.081247198
2.081226471
2.0812091
2.081188193
2.081193212
2.081185936
2.081183326
2.081178895
2.081189653
2.08119233
2.081160813
2.081184913
2.081197084
2.081190877
2.081191144
2.081188943
2.081193307

3

D. Apply linear inertia weighting strategy

45

Taking into account the optimization in this hope that theearly have a higher global search capabilities, and later have ahigher open capacity to speed up the convergence rate, so thevalue of inertia weight should be diminishing, and therefore
on the basis of the initial improvement plus the decreasing
linear weight, the inertia weight value is no longer used with afixed value of 1, and it is set to a weight drop between 0.9 and
0.4.

6
7
8
9
10
11
12
13
14
15
16
17
18

Note: In order to describe the convenience in the form, theimprovement refers to: only the use of FCM, the initial improvement
refers to: FCM in the integration of the idea of particle swarm
algorithm; re-improvement refers to: the initial improvement on thebasis of the use of Linearly decreasing inertia weight values. The
following description has the same meaning.

19
20

The above two graphs show that the improved Jb value isstable and the value is small, which converges to 3.2934 and2.0811, respectively, when the program is run for 2 to 20 times
and the average of the Jb values is obtained.
TABLE ICONVERGENCE CURVE CYCLE DATA(CN=4)
Cycle-index

Before improvement

First improvement

Secondary improvement

2
3
4
5
6

3.319930289
3.355932514
3.42322518
3.426067281
3.400340325
3.364774648
3.406441959
3.363434596
3.425287692
3.352790831
3.358763164
3.426013215
3.33766005
3.388138626
3.315347435
3.370346834
3.376711892
3.350576588
3.3567382

3.30345825
3.303458248
3.303458172
3.303458416
3.303458182
3.303458318
3.30345819
3.303458244
3.303458303
3.303458203
3.303458242
3.303458326
3.303458286
3.303458241
3.303458217
3.30345827
3.303458259
3.303458256
3.303458276

3.293458078
3.293458177
3.293458117
3.293458145
3.293459015
3.293459013
3.293458177
3.293458196
3.293458745
3.293458676
3.29345866
3.29345836
3.293458688
3.293458702
3.293458521
3.293458323
3.293458509
3.293458746
3.293458488

7
8
9
10
11
12
13
14
15
16
17
18
19
20

Fig. 5. Under the condition of CN = 5, the JB value curve under
different strategies ((FCB represents the fuzzy C-means algorithm,
PSO represents the idea of integrating the particle swarm algorithm
in FCM, and the Inertia Weight represents the linearly decreasinginertia weight))

IV. CONCLUSION AND PROSPECTIn this paper, the main goal is to achieve the combination ofparticle swarm optimization and clustering analysis, so as tooptimize the clustering analysis algorithm and optimize it byintroducing PSO algorithm. Finally, the improved algorithm isproved on the convergence curve is superior to the originalalgorithm. With the advent of large data age, large data

TABLE IICONVERGENCE CURVE CYCLE DATA(CN=5)© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4,B-Con Plaza, Beppu, Oita, Japan280

Qiongqiong Hu,Yiyang Li,Yong Ge,Yu-an Zhang,Qinglian ,MaMakoto Sakamoto

analysis in the future will be more and more people’s attention.
The intelligent, diversity, good self-service checkability andadaptability of the group intelligence theory will bring great
prospects for the common development of particle swarm
optimization and clustering algorithm.

1. Lect.Shital P.Bora.Data Mining and Ware Housing[C]//3rd
International Conference on Electronics Computer Technology(ICECT),Kanyakumari,2011,1:15.
2. M.Kholghi, H.Hassanzadeh, M.R.Keyvanpour, Classificationand Evaluation of Data Mining Techniques for Data Stream
Requirements[C]//International Symposium on Computer
Communication Control and Automation (3CA), Tainan, 2010,
Vol.1: 474-478.
3. Song Z, Chen J, Zheng B. A Multi-Objective Optimization
Evolutionary Algorithm with Multiple Indicators Enhanced[J].
Journal of South-Central University for Nationalities (Natural
Science Edition), 201130389-93.
4. Jianguo ChenZhongshan Song Bojin Zhenget al. A Multiobjective Optimization Evolutionary Algorithm With Better
Performances on Multiple Indicators International Symposiumon Intelligent Computation and Application ISICA 201020104756.
5. Vanden Bergh F. An analysis of particle swarm optimizer[C].
Proceedings of the 1998 conference of Evolutionary
computation Piscataway IEEE Press199867-63.
6. Luo Huilan, WEI Hui. Clustering Algorithm for Mixed Data
Based on Clustering Ensemble Technique[J]. Computer Science,
2010(11):234-238.

Acknowledgements
This work is supported by Natural Science Foundation ofChina(No.61640206); Chunhui Research Program from
Ministry of Education (Z2016082,Z2015050) ; China Scholarship Council (CSC:201608635010);Youth Foundation of
Qinghai University(No.2014-QGY-30).

References© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan281

Robot Manipulator Arm Inverse Kinematics Analysis by Jacobian *Sun Oh Park, †Department of Mechanical Engineering, Pusan National University 2, Busandaehak-ro 63beon-gil, Geumjeong-gu,
Busan, 46241, Rep. of KOREA
Jin Gon Yoon

Department of Mechanical Engineering, Pusan National University 2, Busandaehak-ro 63beon-gil, Geumjeong-gu,
Busan, 46241, Rep. of KOREAMin Gyu Jung

Department of Mechanical Engineering, Pusan National University 2, Busandaehak-ro 63beon-gil, Geumjeong-gu,
Busan, 46241, Rep. of KOREAMin Cheol Lee‡

Department of Mechanical Engineering, Pusan National University 2, Busandaehak-ro 63beon-gil, Geumjeong-gu,
Busan, 46241, Rep. of KOREA
E-mail: sunoh425@pusan.ac.kr , y1k1000@pusan.ac.kr, aha0075@naver.com, mclee@pusan.ac.kr
Abstract
There are three methods for solving inverse kinematic that are algebraic, geometric and numerical. The advantage ofalgebraic and geometric method is that can analyze exactly but these two methods are not only difficult to analysis
but also need to recalculate when DOFs of a robot arm or DH-parameters are changed and increased. That is why
this paper will present how to approach numerical method by Jacobian in place of previous two methods. The
Jacobian based numerical method can easily solve any DOF and DH-parameter except singular point problem.
Therefore, Jacobian based solution for solving inverse kinematics of 7-DOFs robot arm is introduced. A method to
avoid singular point by using condition number is introduced. In the future, we will adapt this result to 7DOF dual
robot arm and then compare our numerical method and real system
Keywords: Inverse Kinematics, Jacobian, Singular Point, 7-DOF Robot.

geometric method and algebraic method[1][2]. A
numerical method is iterative method. Otherwise, if
Jacobian based on differential motion is obtained, inverse
kinematics can be solved by using differential motion
relation between robot frame and hand one. The
advantage of analytical methods quickly finds a correct
solution, but it is difficult to find a solution according toincreasing of DOF. On the other hands, the Jacobian

1. Introduction
For controlling robot manipulator, each joint angle at
a desired position and orientation of the end-effector
should be obtained when the manipulator moves tofollow a desired trajectory, which is inverse kinematics
problem. The inverse kinematics can be solved
analytically and numerically. Analytical methods include© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

282

Sun Oh Park, Jin Gon Yoon, Min Gyu Jung, Min Cheol Lee

based solution inverse needs time for calculating, but thismethod is adapted not only high degree of freedom(DOF)
but also DH-parameter change. In this research, the robot
manipulator has redundancy with 7-DOFs that is why
Jacobian based inverse kinematic solution is introduced.

So, each angle
position is decided.

can be computed when desired

3. Robot model and D-H parameter2. Concept of using JacobianFig. 2. Robot model
Table 1. The D-H parameter table
Joint
1

Fig. 1. This is 2-DOFs model to approach concept of Jacobian

=
=

=

=

−

−
−
−
( + 1) = ( ) +
=

n.

2

(1)
=
−

= ( )+=

1det( ) −+=( + 1) = ( ) +

−

−

−

= ( )+*d (mm)a (mm)

α (°)

425

0

90

278

0

0

0

90

-90

3

*

4

*

0

0

-90

5

*

326

0

90

6

*

0

0

-90

7

*

171

0

0

θ is rotation of z-axis, d is distance of z-axis, a is distanceof x-axis and α is rotation of x-axis. The following
equation can be derived using DH parameter.

(4)

4. Numerical approach about 7-DOFs modelNow, considering the 7-DOFs case. When a coordinate
system is defined, coordinator can be represented by 3
positions and 3 rotation angles(orientation).

The Jacobian[3] matrix is the matrix of all first orderpartial derivatives. If each angle is decided, the object
angles can be reached to goal position P by using
Jacobian method. Like this method, each angle iscalculated when initial position and object position are
decided.
=**: variable rotation angle

(2)
(3)

Where n is divided number and is each step from 1 to

=

θ (°)

⎡
⎢
⎢
⎢
⎢
⎣

(5)

⎡
⎤
⎢
⎥
⎢
⎥ = [ ]⎢
⎥
⎢
⎥
⎢
⎦
⎣

⎤
⎥
⎥
⎥
⎥
⎥
⎦

(8)

The 6-coordinator representation are composedand finite difference angle. So,
Jacobianconsidering 7-DOFs case, Jacobian has 6 × 7 matrix.
Angles are desired to get solution to reach object

(6)
(7)

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan283

Robot Manipulator Arm Inverse

Eq.(13). Eq.(14) is derived from Eq.(12) when only -th
joint is operating. is rotated aboutaxis so Eq.(15)
can be expressed. Therefore, use Eq.(15), Eq.(14) andEq.(13) Eq.(16) and Eq.(17) are defined. About angular
Jacobian, it is assumed that 1st joint only depend z-axis=
that is why 1st angular Jacobian component is[0 0 1] , and 2nd Jacobian component is depend on firstjoint and second joint and each joint only rotate on
relative coordinate z-axis, so we can obtain =.
From Eq.(17) and angular velocity, Jacobian matrix write
Eq.(19).
Now, all of parameter are defined so that next step isto approach the sequence of Jacobian method.
For using numerical method, first, sequence of solving
must be decided initial position and orientation.
The initial condition is considered like below
326⎡−425 ⎤
0 −1 0326⎢⎥
0 0 −425 → ⎢ 449 ⎥
= 1
0
0 1
449
⎢ 0° ⎥⎢ 0° ⎥
00 0
1⎣ 90° ⎦

position and orientation. Therefore, the equation is
writing on the below it.
⎡
⎢
⎢
⎢
⎢
⎢
⎣

⎤
⎥
⎥
⎥=[
⎥
⎥
⎦

⎡
⎢
⎢
× ]
⎢
⎢
⎣

)(

)

⎤
⎥
⎥
⎥
⎥
⎦

(9)

is must
Before calculating this equation,
calculated from , but is not square matrix so it is notpossible to compute inverse matrix operation. Therefore,
Pseudo concept is needed to solve inverse Jacobianmatrix

× [

(

(

×

×

(20)

=I

∗
×

]=

× )

=I

(31)

Where ∗× is new inverse matrix.
∗× is obtained from Pseudo concept. The concept ofPseudo is to compute the inverse of a square matrix usingtranspose matrices.
Of course, Jacobian matrix is necessary when calculate∗
× matrix, so that the Jacobian is considered how tomake correct form. The Jacobian matrix can be madenext concept.[4][5][6]
=

+

−

=

̇

= ̇

̇ =

=

×(

0
= 0 ,
1

̇

= ̇

−

)

=

=

(13)

,⋯,

(15)

×

)

=

−

−

⋯
⋯

)

×(

=

)

(16)
(17)
−

⋯

)

300
300
300
⎤
⎡200⎤
⎡200⎤
⎡200⎤
⎥(
) ⎢⎢100⎥⎥ (ℎ) ⎢⎢100⎥⎥ () ⎢⎢100⎥⎥⎥⎥ → ⎢ 0° ⎥ → ⎢ 0° ⎥ → ⎢ 80° ⎥⎥⎢ 0° ⎥⎢ 80° ⎥
⎢ 80° ⎥⎦
⎣ 80° ⎦
⎣ 80° ⎦
⎣ 80° ⎦The result of rotating between the z -axis by 90° andthen rotating the x-axis by 90° and the result of rotating
the 90° on the x-axis by 90° on the z -axis are totallydifferent result. That is why the rotations are separately
rotate to desire position.
Ex) n = 10, (1) = initial position [326 −425 449]
Object = final position [300 200 100]

(14)

×(

−

326
⎡−425
⎢ 449
⎢
⎢ 0°
⎢ 0°
⎣ 90°

(12)

×(

×(

And final positions are[300 200 100 80° 80° 80°]
But rotation must be keep rotate order, so rotation will
be rotated in order Roll-Pitch-Yaw.

=

(18)

[

]

[

]

=

[

]

(2) = (1) +, ⋯ , ( + 1) = (1) +,Of course, when calculating, the angle must be
calculated in radian despite of that it is shown in 60degree-form.
In summary, the Jacobian process is as follows, initialand object condition decided so that it can obtain dP, n(1)and (1) → (1) →

(19)

Where
is the coordinates of n in 0,
is distance
between origin and point n.
If each joint are independent, end-effect position canusually be written Eq.(12), Eq.(12) also can rewrite© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

284

Sun Oh Park, Jin Gon Yoon, Min Gyu Jung, Min Cheol Lee

(1) =
(1)
(2) = (1) + (1)(2) → (2) →(2) → (3) → ⋯ → ( + 1)
eventually We can get final angle ( + 1).

cylinders are each joint and blue cylinder are connector.
End-effect-coordinator is marked over 7-th joint.

Fig. 3. 7-DOF initial position

Acknowledgements

6.

5. Simulation Result [7][8]Conclusion

Jacobian based inverse kinematics solution was applied
to 7-DOFs robot arm. The result of the Jacobian
approaching is well matched to the desired position andorientation of end-effector. Therefore, it was confirmedthat the Jacobian approach method can adapt to easily
solve inverse kinematics problem of 7-DOFs system.
In the future work, the method to avoid a pose out ofworkspace due to rotation limitation of joint motor will
be studied.

Fig. 4. Define rotation

The Technology Innovation Program(10073147,
Development of Robot Manipulation Technology by
Using Artificial Intelligence) funded By the Ministry ofTrade, Industry & Energy(MOTIE, Korea).
References
1. K.S. Fu, R.C Gonzalez, C.S.G.Lee, ROBOTICS : Control,
Sensing, Vision, and Intelligence, McGraw-Hill, 1987
2. Saeed B Niku, Introduction to Robotics Analysis, Systems,
Application, PrenticeHall, 2001
3. Mark W. Spong, Seth Hutchinson, and M. Vidyasagar(2004), Robot Dynamics and Control pp 33-130.
4. Jeong-Seul, Robotics Application of Matlab &Simulink(2015) pp1 130-150
5. William A. Wolvich, “ROBOTICS: Basic Analysis andDesign”, CBS College Publishing, 1987
6. K. S. FU, R. R. C. Gonzales, C.S.G.Lee, “ROBOTICS”,
McGraw-Hill, 1987
7. John J. Craig, “Introduction to Robotics”, AddisonWesley Publishing, 1989
8. Peter Corke, “Robotics, vision and Control”, pp 137-160,
Springer Publishing, 2011

Fig. 5. Object point result image
⎡
⎢
⎢
⎢
⎢
⎢
⎣

⎤ ⎡126.49°⎤
300
⎡200⎤
⎥ ⎢201.32°⎥
0.0295 − 0.1717
0.9847 300
82.41°
⎢
⎥ ⎢
⎥
100⎥⎥
0.3353 − 0.9264 − 0.1716 200→⎢⎥ = ⎢131.45°⎥ → 0.9417
0.33520.0303 100⎢ 80° ⎥
⎥ ⎢ 103.25 ⎥⎢ 80° ⎥
00
01
⎥ ⎢ 50.25° ⎥⎣ 80° ⎦
⎦ ⎣ 14.90° ⎦The initial line is black and the circle over the line is
end-effect-point. The line color indicates the sequence ofthe robot movement from the initial value. The robot arm
moves in order of black, purple, green and finally blue.
After moving, we got final angle θ for reaching target
and target angle
position [326 −425 449]
[80° 80° 80°] . Robot shape are displayed that red© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

285

Indoor Position recognition using the pseudo-range estimation
Jongwoo An, Hosun Kang, Jiwook Choi, Jangmyung Lee

Dep. of Electronic Eng., Pusan National University, Jangjeon-dong, Geumjeong-gu, Busan, 609-735, Korea Republic
E-mail: jongwoo7379@pusan.ac.kr, hosun7379@pusan.ac.kr, jiwook7379@pusan.ac.kr,
jmlee@pusan.ac.kr
www.pusan.ac.kr

Abstract
In this paper, we propose a method of position recognition in indoor environment by pseudo range estimation of eachsatellite. Position recognition is a core technology for operating autonomous intelligent robots such as Unmanned
Aerial Vehicle System(UAV), Automated Guided Vehicle(AGV). The Position recognition technology is divided
into outdoor position recognition and indoor position recognition. The outdoor position is estimated using Global
Positioning System(GPS) and the indoor position is estimated using various sensors such as Beacon and Inertial
Measurement Unit(IMU). Due to the development of the technology, outdoor position recognition using GPS can be
precise position estimation, but indoor position recognition using various sensors has a cumulative error caused byvarious factors and it is difficult to operate for a long time. In this paper, to overcome this drawback, we analyze therelationship between the velocity of the moving object and pseudo-range shift each satellite, and estimate the indoor
position of the moving object by estimating the pseudo-range
Keywords: GPS, Pseudo-range, Navigation, Satellites

In the case of Beacon, a relatively precise position
estimation is possible by installing a separate device in a
specific area.
However, it can be used only in the space where thedevice is installed, and maintenance is required
separately. In the case of indoor location recognition
using the IMU sensor, there is no spatial limitation
because the acceleration, gyro, and compass information
measured by the sensor are fused with various filters, but
it is disadvantageous in that it is difficult to use for a longtime due to accumulated error.

1. Introduction
Position estimation is a core technology for airplanes,
ships, and Automated Ground Vehicles(AGV). The
Position recognition technology is divided into outdoor
position recognition and indoor position recognition. The
outdoor position is estimated using Global Positioning
System(GPS) and the indoor position is estimated using
various sensors such as Beacon and Inertial Measurement
Unit(IMU)[1,2].
In the case of outdoor location recognition, it is possible
to estimate the position with an error of cm unit due tothe development of GPS, but indoor location recognition
is not.
Since the satellite signals are blocked by the buildings in
the indoor area, it is difficult to use the GPS, and thelocation is estimated by using various sensors.
Beacon and IMU are used for indoor location estimation.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

286

Jongwoo An, Hosun Kang, Jiwook Choi, Jangmyung Lee
In this paper, proposed a method of independent
positioning using code pseudo-range. The code pseudorange at time t is defined as follows.
Ri j (t )ji (t )cji (t )(1)

In the above Eq. (1), Ri j (t ) is the pseudo-range
measured between the observation point i and thesatellite j , i j (t ) is the geometric distance between thereceiver and the phase, and c is the speed of light.
jis the amount of time bias and is thei (t )
combination of the receiver and the satellite clock
deviation with respect to the GPS reference time. In thisway, pseudo-ranges can be represented by a single spherecentered on the coordinates of each satellite, and theintersection of the spheres becomes the position of thereceiver. In this way, it is possible to use only threesatellites for the latitude, longitude and altitude
determination of the receiver's three-dimensional
coordinates, but only when there is no time error because
the receiver's clock and satellite clock have the same
performance. Generally, a clock used in a receiver has apseudo-range error due to a clock error when a low-cost
crystal having a low precision is used. In order tocompensate for this error, four unknowns, including thepositional components of the receiver and the clock bias,
are required, and pseudo-ranges measured from four ormore satellites are required to obtain these four
unknowns[6,7].

Fig. 1. Navigation System.

To overcome these drawbacks, in this paper, we proposeda method of indoor position estimation without additional
system through pseudo range estimation with GPS
satellites. The composition of the paper is as follows. In
the chapter 2, introduces the Global Positioning System,
chapter 3 Proposed Algorithm, In the chapter 4,
Experiments & result and finally conclusion with thechapter 5.
2.
GPS satellites are configured to determine the threedimensional location of the receiver, expressed aslatitude, longitude, and height. This point-by-point
receiver positioning method is called stand-alone
positioning. The stand - alone positioning determines the
position of the receiver using the measured distance foreach satellite based on the receiver present on the ground.

3. Proposed Algorithm
Since the GPS signal is blocked by the building whenentering the building, there is a blank space in the pseudorange information as shown in the figure below.

Fig. 2. Global Positioning System© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan287

Indoor Position recognition using

Fig. 3. Pseudo-range(outdoor)

Fig. 5. Proposed Algorithm

When the signal is blocked, a method of estimating thepseudo-range with the satellite uses a characteristic thatthe pseudo-range changes linearly as shown in Fig. 3 atconstant speed of object. Using this characteristic, thepseudo distance is estimated by adding the average value
to the previous information when entering the room byusing the average of the measured data for a certain
period of time.

Fig. 4. Pseudo-range(indoor)

As shown in Fig. 4, it is difficult to estimated the positionof the receiver because the satellite signal is blocked by
the buildings. In this paper, proposed a position
estimation method of GPS receiver by Pseudo-range
estimation while satellite signal is blocked.
The following figure is show the proposed algorithm.

4.

Experiment & Result

Fig. 6 shows the experimental environment. The
experiment was conducted at Pusan National University.
As shown in the figure below, the experiment was
conducted through the inside of the building whilemoving in a straight line.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

288

Jongwoo An, Hosun Kang, Jiwook Choi, Jangmyung Lee
atmospheric error. In this paper, we have experimented
with the assumption that these errors are the same as
those measured outdoors, and confirmed that it ispossible to estimate the position without additional
system through pseudo range estimation. In the future,
we intend to improve the accuracy of position estimationby considering various parameters.
Acknowledgements
This research was supported by the MOTIE (Ministry ofTrade, Industry & Energy), Korea, under the Industry
Convergence Liaison Robotics Creative Graduates
Education Program supervised by the KIAT (N0001126).
References
References are to be listed in the order cited in the text.
Use the style shown in the following examples. For
journal names, use the standard abbreviations. Typeset
references in 9 pt Times Roman.

Fig. 6. Experiment environment

Figure 7 shows the experimental results.

1. Y. Yuan, X. Huo, J. Ou, K. Zhang, and Y. Chai, Refining
the Klobuchar ionospheric coefficients based on GPS
observations, Aerospace and Electronic Systems, IEEE
Transactions on, vol. 44, no. 4, pp. 1498-1510, Feb. 2008.
2. Q. Zhu, Z. Zhao, and L. Lin, Real time estimation of slant
path tropospheric delay at very low elevation based on
singular ground-based global positioning system station,
Institution of Engineering and Technology(IET) Radar,
Sonar & Navigation, vol. 7, no. 7, pp. 808-814, Aug. 2013.
3. B. W. Parkinson and J. J. Spilker. Jr, Global Positioning
System: Theory and Applications, Volume I, (American
Institute of Aeronautics and Astronautics, Washington,
DC, 1996).
4. S. Y. Kim, C. H. Kang, and C. G. Park, Station based
detection algorithm using an adaptive fading Kalman filter
for ramp type GNSS spoofing, Journal of Institute ofControl, Robotics and Systems (in Korean), vol. 21, no. 3,
pp. 283-289, Mar. 2015.

Fig. 7. Experiment result

Experimental results show that although the position of
the moving object can be estimated through the pseudo
range estimation in the room, the error of about 10m
occurs5.
In this paper, we proposed a method to estimate theposition in the building by Pseudo-Range estimation.
The position of the user is estimated by taking into
account various parameters such as clock bias error and© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

289

Distance measurement algorithm based on the object recognition
Ho-sun Kang*

Dep. of Electronic Eng., Pusan National University, 2,Busandaehak-ro 63beon-gil Geumjeong-gu
Busan, 46241, Republic of Korea†
Na-hyun Lee‡

Dep. of Electronic Eng., Pusan National University, 2,Busandaehak-ro 63beon-gil Geumjeong-gu
Busan, 46241, Republic of Korea§
Jang-myoung Lee

Dep. of Electronic Eng., Intelligent Robot Laboratory, 2,Busandaehak-ro 63beon-gil Geumjeong-gu
Busan, 46241, Republic of Korea
E-mail: hosun7379@pusan.ac.kr, nahyun7379@pusan.ac.kr, jmlee@pusan.ac.kr
www.pusan.ac.kr

Abstract
This paper implements an algorithm that recognized an object using a vision system and measured the distance to theobject. Autonomous navigation systems such as UAV (Unmanned Aerial Vehicle) and AUV (Autonomous
Underwater Vehicle) have been a world-wide trend in recent year. To operate these systems safely, it is necessary torecognize obstacles on the moving path accurately and quickly. Generally, various sensors are used to recognize thesurrounding environment. These methods are affected by disturbance caused by the surrounding environment. To
overcome these drawbacks, this paper proposes an obstacle recognition algorithm using a stereo camera. For obstacle
recognition, disparity is obtained by removing the remaining part of the object except the recognized object from theimage to improve the computation speed. Finally, after calculating the distance, we confirm the algorithm bycomparing with the actual distance.
Keywords: Autonomous navigation system, key-point matching algorithm.

obstacles on the moving route and the securing of safety
through avoidance. In general, SLAM (Simultaneous
Localization and Mapping) is performed using various
sensors such as an infrared sensor and a laser sensor torecognize the surrounding environment. However, thesemethods are disturbed by the surrounding environment,
and the amount of information is enormous. Therefore,

1. Introduction
Recently, the development of various sensors and thedevelopment of technology are rapidly expanding themarket for autonomous navigation systems such as UAV
and AUV. The most important consideration in studying
the autonomous navigation system is the recognition of© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan290

Ho-sun Kang, Na-hyun Lee, Jang-myoung Lee

in recent years, the vision system is mainly used toovercome these shortcomings. A stereo camera canacquire distance information that is difficult to obtainwith a single camera through its role as a human eye.
Therefore, this paper also proposes an obstacle
recognition algorithm using a stereo camera [1].

the focal length, and z is the distance from the object tothe origin. When the equation Eq. (1) is transformed, thedistance can be obtained through disparity, baseline, andfocal length as shown in the following Eq. (2) [4].
==2. Used algorithm
2.1. ASIFT algorithm

−

=

=

×

×

(1)
(2)

The ASIFT (Affine Scale-Invariant Feature Transform)
is an algorithm that performs key-point matching using
SIFT key-point extraction algorithm and then matches
the images using affine transformation as shown in thefollowing fig.1 below. The SIFT algorithm is robust
against image size change and has excellent performance.
However, it has a drawback that it takes a long time to
extract feature points because of a large amount ofcomputation [2][3].
Fig. 2. Relationship between disparity and distance

3. Implementation
3.1. Basic environment
The basic environment for implementation is Visual
Studio 2017 and OpenCV3.2 version. And the stereo
camera was modeled by ZED. The ZED stereo camera
has a baseline of 120mm and a focal length of 350pixels.
Also, the image size is 640x480 pixels as shown fig.3
below.
Fig. 1. ASIFT algorithm overview

2.2. Distance measurement
With a stereo camera, you can get two images for thesame object. As shown in Fig.2 below, the two images
are separated by a base line. As a result, even if the two
images correspond to each other, a difference incoordinate value of the image occurs. This is calleddisparity. In addition, disparity must be grasped to obtain
distance information about an object. In Fig.2, thedisparity is shown as the difference between and ,which decreases as the distance between the object andthe camera increases. The formula for calculating the
disparity is shown in Eq. (1) below. Where B is the
distance between two cameras in the stereo camera, f is

Fig. 3. Stereo camera images

3.2. Key-point matching
Using ASIFT key-point matching algorithm, two imagesof a given object image and stereo camera are displayed© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

291

Distance measurement algorithm based

as rectangles as shown in Fig.4 and Fig.5 below. This isa called the Region of Interest(ROI).

computation speed will be reduced by a large amount ofdata. Therefore, in this paper, we use only the coordinatesof the four corners of the ROI drawn by the key-point
matching as shown in Fig.6 below. The disparity is
obtained by using the coordinate value from 1 to 4
coordinates, and the disparity is averaged by assumingthat the object is one-pixel point [5].

Fig. 6. Region of Interest

The result of applying it to the image taken at 50cm isshown in the following fig.7. In this figure, the horizontal
lines of green, emerald, white, and yellow are the lines
connecting each vertex. The more perfect horizontal line
is, the more accurate disparity can be obtained.
The disparity calculated by the above method isFig. 4. Key-point matching result(Left)

Fig. 7. Disparity

Fig. 5. Key-point matching result(Right)

substituted into the Eq.2, and the distance from the actual
distance of 400mm to the 1200mm is estimated from thefollowing Table.1.

3.3. Distance measurement
To obtain the distance from the camera to the object, thedisparity for the object should be calculated. If all the
pixels within the region of interest are calculated, the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan292

Ho-sun Kang, Na-hyun Lee, Jang-myoung Lee

object. As a result, it was confirmed that the maximum
error rate of the distance was calculated to be a very
accurate value of about -4.14582. However, when the
distance between the camera and the object is larger than
1100mm, the region extraction using the ASIFT keypoint matching is not performed properly. So, Result hasvery inaccurate distance values. Therefore, in future
research, we will study algorithms for correcting the keypoint matching when the distance between the camera
and the object is getting longer to solve this problem. In
addition, although the ASIFT algorithm has different
calculation speeds depending on computer performance,
it takes a very long time to obtain the results. Therefore,
it is necessary to develop a key-point matching algorithm
that increases the computation speed while maintaining
the performance of ASIFT.

Table.1. Result of distance measurement

Real distance
400
450
500
550
600
650
700
750
800
850
900
950
1000
1050
1100

Measured distance413.696465.5048
511.998565.576615.997664.248709.612759.599811.2512871.8664
904.3482
964.214979.3174
1061.651145.604

Error rate
-3.424-3.44551-2.3996-2.832-2.66617-2.192-1.37314-1.27987-1.4064-2.57252-0.48313-1.49621
2.06826-1.10952-4.14582

Acknowledgements
This research was supported by the MOTIE (Ministry ofTrade, Industry & Energy), Korea, under the Industry
Convergence Liaison Robotics Creative Graduates
Education Program supervised by the KIAT (N0001126).
References

1150
384.594
66.55704
1200
562.9423
53.08814
Expect for cases where the actual distance is larger than
1100mm, very accurate calculation results can beobtained from the maximum error rate of -4.14582 to theminimum error rate of 0.48313. Also, when the actual
distance is larger than 1100mm, as shown in Fig.8, thekey-point matching using the ASIFT algorithm is not
performed properly, resulting in a large error.

1. Jawon Seo, Changik Kim “Stereo camera image

processing technology and motion”, Vol.38 No.2, 2011.2,
31-36 (6 pages).
2. GUOSHEN YU, AND JEAN-MICHEL MOREL, ASIFT: An
Algorithm for Fully Affine Invariant Comparison, Image
Processing On Line,1 (2011), pp. 11–38.
3. Donggeun Kim, C++API OpenCV Programming, Game
publisher, 2016.
4. Yoonhee Lee, Dongsuk Kim, Hogi Jung, Falju Yoon,“Stereo Vision based front obstacle distance monitoring
system”, Korea Automootive Enginerrs Conferenc,
pp1181~1187.
5. Robert ragani, Computer vision programming using
OpenCV from basic image processing to advanced
computer vision, PACKT, 2015.

Fig. 8. Result of key-point matching(case : 1150mm)

4.

Conclusion

In this paper, we use ASIFT key-point matchingalgorithm to recognize objects by using stereo vision
camera. To improve the computation speed, distance iscalculated by solving only the disparity of the four
corners rather than the disparity of the whole recognized© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan293

An-ion air purifier system using IoT Technology
Heeje Kim

Department of Electrical Engineering, Pusan University, Busan, Republic of Korea
Dohyun Kim

Department of Electrical Engineering, Pusan University, Busan, Republic of Korea
E-mail: heeje@pusan.ac.kr, kdh8486@naver.com,

Abstract
We have developed the an-ion air purifier which can remove various fine dust very efficiently. It was based on theprinciple of high voltage corona discharge that was fabricated for air cleaning from various fine dust. That is a big
issue for human society. The best switching frequency was 10kHz, and the output voltage was approximately 20kV.
In order to remove the fine dust, the an-ion air purifier was controlled properly for removing the fine dust by using
the Arduino and personal smart-phone app. From these experiments, we could remove various fine dust much moreeasily and effectively.
Keywords: High voltage corona discharge, Fine dust, Arduino, Smart-phone app

1. Introduction
Recently, environmental pollution has increased theamount of fine dust, and air cleaning technology forremoving fine dust is increasing day by day.
Fine dust refers to particles less than 10 micrometers
in diameter and larger than 2.5 micrometers, and it isdifficult to remove because it is impossible to identify
with the naked eye. Therefore, filter method was mainly
used to remove fine dust. However, such methods has acertain disadvantages for example it is not economicallyefficient because it generates noises and the filter must be
periodically exchanged. If the filter is not replaced,bacteria will multiply in the filter, and the bacteria will
be discharged from the air cleaner during use, which will
adversely affect the human body.
In this paper, we used the high voltage corona method
differentiated from the conventional method and applied
the IoT(Internet of things) technology to design anefficient method to simultaneously perform fine dust
measurement and removal. The combination of IoTtechnology makes it possible to control fine dust and air
cleaning system at the same time, and it can solve
economical and environmental part at same time.

2. Design of air cleaner
Table 1. Design Considerations
Design Considerations
Frequency
10khz
Duty Ratio
50%
Input Voltage(DC)30~70V
Output Voltage(DC)20~25kv
Transformer ratio
5 : 380 (primary side : secondary side)
The air purifier system designed in this paper is anelectric dust collection system using corona discharge.
Basically, it is necessary to generate a high voltage of
20kV or more in order to discharge the corona. Designed
air cleaners consist of half bridge inverters, step-up
transformers, and Cockcroft Walton circuits.
When DC is input, the DC voltage is converted to AC
voltage through the half bridge inverter and then the AC© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan294

All Author’s Full Names

2.3 Cockcroft Walton circuit

voltage is boosted to AC high voltage first by using thestep-up transformer. The second boosted AC voltage isconverted to a DC high voltage through a Cockcroft
Walton circuit to cause a corona discharge.

The Cockcroft Walton circuit, when stacking n numberof rectifiers and capacitors (Fig. 3 N=8), If no load isapplied, a dc voltage of n times the secondary voltage
peak value of the transformer can be generated.
Using this principle, DC high voltage such as 20 ~ 25kV
can be obtained. Corona discharge occurs by using DC
high voltage. The generated corona discharge can remove
various fine dust quickly. [2]2.1. The half bridge inverter
Half bridge inverters divide the input voltage in halfand connect alternately to the load via two switches.
Therefore, the half bridge inverter uses two P-MOSFETs(IRF250). To divide the input voltage in half, twocapacitors of the same capacity are used, and the input
voltage is half the time. The switches S1 and S2 of thesingle-phase half-bridge inverters are turned onalternately and the output voltage is determined by theswitch state. When S1 is ON, the output voltage is -Vdc
/ 2. When S2 is ON, the output voltage is -Vdc / 2.
If the switches S1 and S2 are all ON, the input poweris shorted by the switches S1 and S2, so the switches S1and S2 should not be all ON. Therefore, since theswitches S1 and S2 are alternately turned on, the output
voltage alternates between Vdc / 2 and -Vdc / 2.[1]In this paper, PWM signal applied to S1 and S2 iscontrolled by DSP chip F28335.

Fig. 2. Cockcroft Walton circuit
2.4 Transformers &Cockcroft Walton circuitmolding
The design of the air cleaner proposed in this paper isbased on high voltage corona discharge. Therefore, thecircuit and the step-up transformer generate aconsiderable high voltage and the dielectric breakdown
easily occurs.
If isolation is not done properly, the transformer or
Cockcroft Walton circuit will be easily damaged due toinsulation breakdown. The insulation design is essential.
In this paper, dielectric breakdown is solved by molding
epoxy into transformers and Cockcroft Walton circuit.

Fig. 1. The half bridge inverter

2.2. Step-up transformers
The transformer in this paper uses EI core. Ferrites areused as high permeability materials in the frequency
range from low frequency to hundreds of MHZ.
Therefore, it is most suitable for economical part andefficiency in the step-up transformers design.
Table 2. Transformers design
Transformers designCore
FerriteTurns Ratio
76
Primary Voltage
15~35V
Secondary Voltage
1140~2660V
Frequency
10~20kHz

Fig. 3. Transformers & Cockcroft Walton circuitmolding© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

295

An-ion air purifier system

3. IoT Technology

3.3 The Air cleaner App

Iot technology refers to intelligent technologies andservices that connect all things based on the Internet andcommunicate information between people, between
smart devices plus inter device communications.
Recently, due to the development of internet and smart
phone, the utilization of Iot technology has been
diversified and much research has been done.
Iot technology is remotely controllable. Data can bereceived and transmitted remotely. In addition, thedesired data can be used in real time and the management
is very easy.
In this paper, Iot technology is combined with air
purifying technology using corona discharge. [3]There are many smartphone users in modern society inrecent years. And the utilization of smartphone issignificant. Iot technology is also growing rapidly in
conjunction with smartphone apps.
In this paper, we developed and implemented asmartphone app based on App Inventor 2.
The main function of the fine dust app is the fine dust
measurement function and ON / OFF control of the aircleaning system. Application is used to check thecurrent fine dust condition and the user controls the air
purifier

3.1 Arduino & Wi-Fi Shield
Arduino is an open source computing platform andsoftware development environment based on simple
microcontroller boards. Arduino can be used as smart
receiver as well, it can deal with real time data transmittersuch as sensors, switches
However, Arduino is not advance enough yet to
implement Iot technology. Get access on data frominternet cloud is still not approachable by Arduino
In this paper, Iot technology is implemented using
Arduino and Wi-Fi Shield. The combination of Arduino
and Wi-Fi Shield makes it possible to transmit andreceive information over the internet cloud. [4]Fig. 5. Fine dust App

4. Experiment
The experiment was conducted by combining IoTtechnology with the air cleaner using the designed highvoltage corona discharge.
1) Acrylic box dimensions[50cmx50cmx50cm] , finedust is generated into the box by using a mosquito net.
2) Fine dust application is used to measure the finedust concentration in the acrylic box.
3) Run the air purifier using the App to remove thethird fine dust.
4) Confirm the reduction of fine dust visually andthrough App, and finish.

Fig. 4. Arduino & Wi-Fi Shield
3.2. Fine dust sensor
Sharp optical dust sensors have internal infrared diodes
and diagonally arranged phototransistors. Light emitting
diode projects and phototransistors detect dark areas dueto the passage of fine particles. In addition to this function,
it is possible to check whether there is smoke or dust by
the output pulse pattern. Fine dust sensors can measure
up to 2.5 μm particles.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan296

All Author’s Full Names

Fig. 6. Before and After (air cleaner & App)
5. CONCLUSION
In this paper, we have developed a high voltage aircleaning system. It is operated by smart phone
application, with the help of Transformers & Cockcroft
Walton circuit Corona discharge is generated, and smoke
free acrylic box confirmed the fine dust removal. All thedata are supported by laboratory experiments. And this
proposed method can be used to keep the environment
clean.by removing the dust from air.
Future experiments will improve the removal level andremovable range of fine dust in the air cleaner.
References
1. Won Chungyeon “Power of Electronics” pp.268-273,
2. LEE Deokchul “High-Voltage Plasma Engineering”
pp.152-154.
3. W. Y. Lee, H.M. Ko, J. H. Yu, and K. B. Sim, “AnImplementation of Smart Dormitory System Based on
Internet of Things,” Journal of Korean institute ofIntelligent System, vol.26,no.4,pp.295-300,Aug.2016
4. S. H. Seo, and S. W. Jang, “Design and Implementationof a smart shoes module based on Arduino,” Journal of
the Korea Institute of Information and Communication
Engineering, vol.19, no. 11, pp. 2697-2702, Nov. 2015.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

297

Solar panel temperature control system using IoT
Min-soo Kim

Department of Electrical Engineering, Pusan University, Busan, Republic of Korea
Hee-je Kim

Department of Electrical Engineering, Pusan University, Busan, Republic of Korea
E-mail: rlaalstn5122@naver.com, heeje@pusan.ac.kr
www.pusan.ac.Kr
Abstract
Solar photovoltaic systems are renewable energy sources that are used widely around the world. On the other hand,the efficiency decreases as the temperature of the solar panels increases. To prevent this phenomenon, a cooling fan
can be installed on the back side of the solar panel to increase the efficiency. The solar system efficiency also decreases
due to weather conditions and unexpected situations. To overcome this problem, an IoT (Internet of Things) system
was used to monitor the state of the solar system and control the cooling fan. The core microprocessor used in IoT
systems was Arduino. Using Arduino, an IoT system can be implemented simply and inexpensively. The entire system
was designed and tested and the efficiency increased by approximately 4.7%. Although it is a small 30W capacity
photovoltaic system, its efficiency is expected to be increased by applying it to a photovoltaic system of more than
1kW in the near future.
Keywords: Arduino, Cooling fan, IoT, Smart phone, Solar panel, Wi-Fi

communicate over distance. So it controls the cooling fan
through Wi-Fi communication between Smartphone andArduino. Arduino is combined with a Wi-Fi shield, and there
is a server in the Wi-Fi shield. Through this server, various
sensor signals and switch signals can be sent to smart phone
and Arduino. This makes it possible to turn on and off theswitch directly after checking the solar panel temperature.
After making the actual model, we confirmed the normal
operation between Arduino and smart phone using Wi-Fi
communication and also confirmed that the temperature ofsolar panel is lowered. Through the monitoring and control
as above, the solar power generation system can make more
efficient electricity generation.

1 Introduction
Nowadays, IoT (Internet of Things) is getting popular inhome and industry [1]. IoT is a future network technologythat can share information and control things between people,
things, things and things [2]. This allows remote control andmonitoring from a distance. Based on this IoT, it can beapplied to solar power systems currently used worldwide.
Solar power generation system uses solar energy to
produce electricity, so if the temperature of the solar panel
rises, the power generation efficiency becomes low [3]. In
particular, at noon time, the surface temperature of the solar
panel reaches from 60 to 70 degrees. There are several ways
to prevent such degradation of efficiency. First, coolingwater is sprayed on the solar panel surface to lower thetemperature. Second, lower the temperature by attaching aheat sink. Third, use a cooling fan to lower the temperature.
Experimental results show that the last of these threemethods is optimal for lowering the temperature of the solar
panel.
However, this cooling fan must be activated when thetemperature of the solar panel rises, and turned off when thetemperature is low. Although the temperature can becontrolled automatically, it is impossible to control the entire
system when the temperature sensor fails or is not ideal. For
this reason, the system is manually controlled throughmonitoring.
The microcontroller used in this paper is Arduino.
Arduino is popular among engineers because it is easy to use
and inexpensive [4]. However, Arduino alone can’t

2 Major System Components
The system consists of several important parts. Arduino
which acts as a brain, Wi-Fi shield for Wi-Fi communication,
infrared temperature sensor for receiving infrared signals,
and relays that activate the cooling fan are importantcomponents. Sub-components include voltage, current
sensor, ambient temperature sensor, and LCD. The
temperature of the solar panel can be appropriately
maintained by the organic operation of these parts.
2.1 Arduino
Fig. 1 shows the Arduino Mega used in this system. The
main chip is atmega2560, which has 54 digital input / output
pins, 16 analog input pins, 256kb flash memory and 8KB
SRAM.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

298

Min-soo Kim,. Hee-je Kim

2.4 Relay
Relay receives signal from Arduino and turns cooling fan
on and off. The maximum input voltage is 250V for AC and
30V for DC.

Fig. 1.

2.5 Other sensors and devices
There are other sensors except for the main components.
For example, voltage and current sensor, ambient
temperature sensor, cooling fan, LCD. Other devices include
AC-DC converter, 30W solar panel, cradle, and electric box.

Arduino MEGA 2560

And Arduino is programmed in assembly language like clanguage and Java.

3 Fabrication
3.1. Design
The total system was designed using the parts describedabove. Various sensors and relays are attached to the centerof Arduino, and finally one system is completed. In addition,a smart phone application for operating the Arduino is also
designed. The app was designed with easy-to-use App
Inventor 2 interface. Fig. 3 and Fig. 4 show the system
schematic and App Inventor 2 interface.

2.2 Wi-Fi shield
The Wi-Fi shield is used in conjunction with Arduino tocommunicate with Arduino and the smart phone. The jsn270
model was used and the specifications are shown in Table 1
below.
Table 1. JSN270 Specification
Wireless Standard
IEEE 802.11 b/g/n
Frequency Range
2.412~2.477 GHz
Data Rates
Max 72Mbps
Modulation Type
OFDM, DSSSWEP 64/128bit, WPA/WPA2
Wi-Fi Security
personal
2.3 Infrared sensor
The infrared sensor is a device that measures thetemperature by detecting the infrared rays coming from theobject. Remote measurement is possible and failure does notoccur. In this system, non-contact infrared sensor(MLX90614) is used and can be measured from -70.01℃ to+382.19℃. It has resolution of 0.01℃ and communication
method is I2C. It is shown in Fig. 2.

Fig. 3. The Schematic of the System

Fig. 4. App Inventor 2 interface

Fig. 2.

3.2 Program CodeFinally, Arduino requires source code for device
operation. The source code is divided into four parts: Wi-Fi
connection, LCD display, temperature and voltage andcurrent sensors, and switch operation. The code below shows
only some important parts. The first is a header part where
you set the header file to use in your code. The second part

MLX90614© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan299

Solar panel temperature control

is for Wi-Fi connection, where you can set the SSID,
Password and encryption method.
/* Wi-Fi connect part */#define SSID
"air"// your wifi network SSID
#define KEY""// your wifi network password#define AUTH"NONE"// your wifi network security (NONE,
WEP, WPA, WPA2)
Fig. 6. Smart phone application/*Server part*/// HTTP headers always start with a
response code (e.g. HTTP/1.1 200 OK)// and a content-type so the client
emknows what's coming, then a blankline:
JSN270.println("HTTP/1.1 200 OK");
JSN270.println("Contenttype :text/html");
JSN270.println("Refresh: 30");
JSN270.println();

4 System Test
We confirmed that the manufactured system works
normally. The On / Off of the cooling fan was controlled bya smart phone and Heater was used to simulate sunlight. And
heat the solar panel using a heater as shown in Fig.7.

3.3 Final model
A final model was created as shown in Fig. 5. Six coolingfans were attached to the solar panel and a converter wasadded to change the input voltage to 12V. An LCD was
attached to the main box to verify the data directly. Finally,
the completed application is shown in Fig. 6. There is acooling fan switch button and added function to checkvarious data in real time.

Fig. 7. Testing set-up
The temperature of the solar panel was maintained atover 57°C and the ambient temperature was maintained at
28°C.
After running the cooling fan through the smart phone,
the temperature dropped by 40°C. At this time, it takes about
20 minutes. The temperature change of the solar panel isshown in Fig. 8. The temperature of the solar panel isgenerally lowered.

Fig. 5. Final model© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan300

Min-soo Kim,. Hee-je Kim

Fig. 8. Temperature change after cooling fan operation
5 Conlusion
In this paper, we designed and manufactured a solar panel
temperature control system. With Arduino and Wi-Fi shield,
it is now possible to control the temperature of the solar panel
at anywhere and anytime. By lowering the temperature of thesolar panel, the power generation efficiency can be increased.
In addition, this system control via smart phone is very
simple. In the near future, we need to research more about
technologies that can significantly lower the temperature ofsolar panels while reducing costs. It’s possible to maximize
power generation efficiency by applying this system togeneral household and large-scale photovoltaic system.
References[1] K. J. Park (2017), “Arduino makes your imagination a reality,”
Edu-I, pp.198[2] O. Vermesan et al.,“Internet of things strategic researchroadmap,” Internet of Things-Global Technological and Societal
Trends, vol. 1, pp.9-52, 2011.
[3] S. Dubey, J. N. Sarvaiya, B. Seshadri, “Temperature
Dependent Photovoltaic(PV) Efficiency and Its Effect on PV
Production in the World – A Review,” Energy Procedia, vol. 33,
2013, pp.311- 321.
[4] Adriansyah A, Dani, A.w., “Design of Small Smart Home
system based on Arduino,” in Electrical Power, Electronics,
Communications, Controls and Informatics Seminar (EECCIS),
2014, pp.121-125,27-28 Aug. 2014

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

301

Enhancement of Mobile Robot Stability and Hardware Based on Reinforcement Learning
Ki-Seo Kim*

Electrical and Computer Engineering, Pusan University, Pusan, 46241, Republic of Korea†
Jeong-Hwan Moon‡

Electrical and Computer Engineering, Pusan University, Pusan, 46241, Republic of Korea
Jang-Myung Lee§

Electric Engineering, Pusan University, Pusan, 46241, Republic of Korea

Abstract
In this paper, we apply the obstacle avoidance run reinforcement learning algorithm used in the mobile robot using
the LiDAR sensor. Q-learning and Deep Learning were applied to reinforcement learning algorithms. We solve theunnatural movements in the deceleration and acceleration of the robot using feedback learning. The results ofsimulation are applied to actual hardware to show how to increase the safety of driving.
Keywords: List four to six keywords which characterize the article.

considering the acceleration of the runway.
In this paper, we studied the algorithms to study natural
driving without solving the current state of the robot, as
well as solving these acceleration and deceleration
problems, as well as learning the current behaviors of therobot.

1. Introduction
Past robots have evolved in the field of industrial andaerospace industries. Past robots, however, have done
nothing but act solely on programmed behavior andbehavior and have not acted on their own. But the
hardware evolution and AI research changed the oldrobot industry. Examples include Google's AlphaGo, theworld champion of the world's champion, and IBM's
Watson, who is currently active in the medical industry.
However, these robots are not yet completed, but they are
becoming more and more evolvable. You know, youknow, rather than just repeating the changing behavior,
you know, you know, you know, you know, you know,
you know, you know, you know, we're going to have tolearn a better thing. It also led to the development of new,emerging LiDAR sensors and simulators as well as
autonomous driving of automobiles and mobile robots. In
reality, while driving is accelerating and slowing downwhen driving obstacles, the past studies the drag anddeceleration of the runway, avoiding obstacles, without

2. System Configuration
Robots that perform self-driving learning robots aremodeled on two-wheeled robotic drives. The ambientenvironment and obstacles surrounding the surroundings
were sensed by RPLIDAR-2 sensors. The two-wheel
drive can be rotated, and the sensor used is recognized as
360 degrees and the resolution is 1 degree. The sensor ismounted on the center of the robot, which can measure
the distance of the robot. Maximum detection distance is
6 m.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan302

Ki-Seo Kim, Jeong-Hwan Moon, Jang-Myung Lee

This gives a Q table for each state and updates the Q
value given the current states, the behavior to be taken,
the corresponding compensation r, and the next state s [s,a, r, s`].

Q * (s, a)( maxa 'Q(s ', a ') Q(s, a)) (1)Q(s, a)is the discount rate and a 'In the above equation,
represents the action to be taken by the next robot (input
1 to 5 in this system).
The action and reward applied to this learning areshown in the following table

Fig. 1. Mobile Robot

TABLE 1POLICY VALUE

Training Policy
ACTION

Fig. 2. LIDAR Sensor and Mobile Robot Simulation
Environment

rewardvelocity0

Go Straight1

Turn left in place2

Turn right in place3

Go straight and turn left

4

Go straight and turn right
Collisionx : 0.8 m/s
z : 0 r/s
x : 0 m/sz : 1.5 r/s
x : 0 m/sz : 1.5 r/s
x : 0.5 m/s
z : 0.7 r/s
x : 0.5 m/s
z : 0.7 r/s
-

5
-5
-5
-1
-1
-100

If the episode is repeated in the above procedure, the Q
value gradually converges to the optimal Q value (Q *).

3. Training Algorithm

3.2. Deep Q Network

Google's deep-minded team has trained Atari games inreinforcement learning, bringing AI's capabilities closer
to humans. In this paper, neural network is applied toaction-value estimation by applying idea of Deep
Learning algorithm to existing reinforcement learning,
and the learning performance is greatly improved as aresult. In this paper, Deep Q Network, which is a fusionof Q-learning and Deep learning, is applied.

After each learning of each episode in the general Qlearning table is completed, the Q value for each state isstored in the register. However, as the complexity of the
problem becomes more and more confusing to thestorage space, the application of Deep learning, which isa field of active research recently, is used to approximate
the optimal Q (Q *) Thereby enabling efficient learning
as well as complex problems. Network is that The Q
function is learned by using the loss function as thefollowing equation

3.1. Q-Learning
For an agent learning, defining a state is a very
important task. This is because the state provides thecriterion information for the agent to select the action thatthe agent will take by the policy. LiDAR sensor data is
input to the state used in the experiment, and the state
angles used for the learning input of the q function are 11,
and the forward direction is 15 degrees and the forward
direction is 180 degrees [105, 120, 135, 150, 165, 180,
195, 210, 225, 240 and 255].
Q-learning is one of the popular reinforcement learning
methods and the learning of Q value is the most important.

Li ( i ) E( s,a,r,s') U ( D)[(rmaxa 'Q(s ', a '; i ) Q(s, a; i ))2 ](2)

3.3. Experience Replay
In the existing DQN (Deep Q Network), similar data arecontinuously entered and learned during reinforcement
learning, which causes unbalanced learning that is biased
toward one side. This is called data correlation, and© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan303

Enhancement of Mobile Robot

Google's deep-mindedness has found an experience
replay approach to solve this problem. This trick isapplied to the autonomous driving learning method as
follows. First, the state, action, reward, and next state [s,a, r, s] obtained during obstacle avoidance driving
learning are all stored in the external memory. Next,
instead of learning the most recently obtained transitionat the start of learning as a Q-value, we randomly selectas many minibatch samples as possible from the storedtransition memory and input them into the network. Thisis because it uses the transition obtained from various
situations as a sample and thus obtains a Q-value that
does not deviate to a specific situation due to this method.
This plays a role similar to network generalization andhelps to keep the network from falling to the local
minimum.

Fig. 3. DQN Model

3.6. Training Network
The neural network has two hidden layers, and the ReLU
activator is set up for the neural network excluding theoutput layer. The learning was optimized using the Adam
optimizer and initialized before learning according to
Xavier initializer. The parameters used in the learning are
as follows.
The reinforcement learning of the proposed DQN
consists of the following steps.
1. Random environmental exploration andlearning with random action according to Egreedy law
2. robot will get minus score when robot hit an
obstacle or rotate in place and immediately reset
the learning environment
3. Save the previous action records of the robot in
the buffer [s, a, r, s`, m]
4. Adding action-memory and learning to replay
buffer
5. Target Q update and weight value optimizationaccording to defined target-Q update cycle

3.4. Target Q network and Q network
The target Q method is a method of learning the Q
network with the target value obtained by obtaining thetarget network separately from the target network and the
Q network when obtaining the target. Then, update target
network to Q network for every certain time step. In theexisting DQN method, the value of the target constantly
changes during a certain episode is fixed, and then therate of change of the target is applied to reduce thecorrelation between the action value and the target,
thereby contributing to the improvement of the learning
success rate.

3.5. Action-Memory

4.

In the conventional learning method, a simple laser
distance value is inputted into the network (eg. 0, 0, 4, 4,1) as a state to obtain the action value of the motor output,
and the robot is moved. The speed of the robot isdetermined by the action value determined for each frame.
Due to the rapid change in the action value due to thesurrounding environment, the speed change rate alsochanges sharply. As a result, the mobile robot can’tsmoothly travel.
In order to solve the abrupt speed change problem, PID
controller was used for speed control. In order to preventthe action which causes rapid change in speed during
acceleration and deceleration, enter it with state.
Therefore, the total neural network is as shown in thefollowing figure.

Experiment

Experiments were carried out with ROS and GAZEBO
simulator, and reinforcement learning experiments wereconducted. The robot scored according to the determinedpolicy, and the robot was free to run without any special
destination and ends the episode when it hit the obstacle.
The study was considered to be successful if it exceeded
score 1000 for the last 30 averages.

Fig. 4. Simulation Environment© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan304

Ki-Seo Kim, Jeong-Hwan Moon, Jang-Myung Lee

sudden action change of the agent, not only theapplication of the existing DQN research but also theinformation of the current speed, attitude, etc. of the robot
in this paper. We propose an algorithm to determine thenext behavior by feedback. This proved its validity
through simulation and actual experiments. Furthermore,
it is expected that the system will have naturalacceleration / deceleration without any input andmodeling of additional sensors, which will be effective
for reinforcement learning application of relatively small
scale system.
Fig. 5. Learning result when Action-Memory Buffer is notapplied6. Acknowledgment
This research was supported by Basic Science Research
Program through the National Research Foundation of
Korea(NRF) funded by the Ministry of Education( NRF2010-0024129 )This research was supported by the MOTIE (Ministry ofTrade, Industry & Energy), Korea, under the Industry
Convergence Liaison Robotics Creative Graduates
Education Program supervised by the KIAT (N0001126)

7. Reference[1][2]

Fig. 6. Learning result when Action-Memory Buffer is applied[3]

In the graph above, the blue line represents theacquisition score of each episode and the red line
represents the average score of the last 30 episodes. As
shown in the graph above, when the agent accelerates anddecelerates smoothly, the agent frequently hits the
obstacle due to sudden action change. On the other hand,when the action-memory is learned together with the
input, the agent learns the state change itself. It can be
confirmed that it has gotten high score by avoiding it well.

[4]
[5]
[6]

Yann LeCun, Yoshua Bengio & Geoffrey Hinton “Deep learning”
Nature 521, 436-444, 28 May 2015
David Silver, Aja Huang, Chris J et al “Mastering the game of Gowith deep neural networks and tree search” Nature 529, 484-489
28 January 2016
Iker Zamora, Nestor Gonzalez Lopez et al “Extending the OpenAI
Gym for robotics: a toolkit for reinforcement learning using ROS
and Gazebo” arXiv:1608.05742v2 [cs.RO] 7 Feb 2017
Jesse Levinson, Jake Askeland et al “Towards Fully Autonomous
Driving: Systems and Algorithms” 2011 IEEE Intelligent
Vehicles Symposium (IV) Baden-Baden, Germany, June 5-9, 2011
Volodymyr Mnih, Koray Kavukcuoglu et al “Playing Atari with
Deep Reinforcement Learning” NIPS Deep Learning Workshop
2013, USA, December 2013
Diederik P. Kingma, Jimmy Ba “Adam: A Method for
Stochastic Optimization” 3rd International Conference forLearning epresentations, San Diego, 2015

5. Conclusion
This study applied obstacle avoidance driving
experiment to a mobile robot by applying reinforcement
earning, which is popular among many kinds of deeplearning algorithms applied in many fields in recent years.
In order to prevent the unnaturalness caused by the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan305

A sound-based measurement of sway angle for anti-sway control of overhead crane
Miki Matsunaga

Graduate School of Engineering, Hiroshima University, 1-4-1, Kagamiyama,
Higashi-Hiroshima city, Hiroshima, Japan
Masayoshi Nakamoto

Graduate School of Engineering, Hiroshima University, 1-4-1, Kagamiyama,
Higashi-Hiroshima city, Hiroshima, Japan
Toru Yamamoto

Graduate School of Engineering, Hiroshima University, 1-4-1, Kagamiyama,
Higashi-Hiroshima city, Hiroshima, Japan
E-mail: matsunaga-miki@hiroshima-u.ac.jp, msy@hiroshima-u.ac.jp,
yama@hiroshima-u.ac.jp

Abstract
For anti-swing control of overhead crane, a deflection angle must be estimated. However, it is difficult to estimate
deflection angles with a contact sensor such as rotary encoder. Therefore, we show a non-contact measurement
method for the deflection angles by using two microphones. The method employs a time delay of arrival (TDOA) of
acoustic signals which are picked up by the two microphones. Also, we show an algorithm to obtain the angle fromthe TDOA by using the Newton's method. Finally, we show experimental results to demonstrate the effectiveness ofthe proposed method.
Keywords: Overhead crane, microphones, correlation function, Newton’s method

attached angle sensor. Here, we assume the payload
includes a sound source such as friction sound, squeaknoise or active noise. With two microphones, we detect
the sound (acoustic) signal generated in the payload. Thatis, we use the microphones for contact-less sensing of theangle. Next, we estimate the time delay of arrival(TDOA) between the acoustic signals obtained from thetwo microphones where the TDOA can be computedfrom the correlation function. Also, we transform the
TDOA to the sway angle by using the Newton’s method.
In the experiment, we treat the case when the payload
is swaying and the position of trolley is changed. Fromexperimental results, we show the proposed method canfollow the sway angle even when the payload is moving.

1. Introduction
Overhead cranes are well-used for transportation of loadin factory and dock. For safety and efficiency
transportation with the overhead cranes, the anti-sway
control1-5 has been received remarkable attention.
Since the dynamics of the sway angle of the crane canbe described by the state-space equations, the sway canbe controlled. However, the sway angle should beobtained for anti-sway control. It is known that themeasurement of the sway angle is very difficult withcontact sensor due to the hardware constraint.
From the background above, we focus on the soundsignal6 for the anti-sway control of overhead cranes. Our
measurement scheme is to obtain the sway angle without© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

306

Miki Matsunaga, Masayoshi Nakamoto, Toru Yamamoto

Table 1. Parameters of the overhead crane

2. Anti-sway Control of Overhead Crane
In the crane, a road is moved by motor driven trolley.
Table I shows the parameters of the overhead crane.
Let be the distance from fulcrum of the trolley to the
payload (sound source). x (0≦x≦X) is the position of thetrolley where x=0 is the left end and x=X is the right end.
That is, the distance between the Microphone 1 and thetrolley is x (variable). Also, θ is the angle of the line
(sway angle). Assume T is kinetic energy and U ispotential energy. Then, we have

T
U

1
1 2
Mx 2
mv
2
2
mg l w 1 cos

Also, define L=T-U is the Lagrangian. Now, let the statevector bex (t )
where
be

x xd

,

x

Fig. 1. Overhead crane with two microphones(Microphone 1 and Microphone 2) and sound sourceis the desired position, and its time derivation

x(t )

x

x
3. Sound-based Measurement of the Sway Angle

Based on the Euler-Lagrange equation, the dynamics ofthe crane can be written in a state space from:

x(t )

0

0

1 0

0

0
mg
M
M mg
Ml w

0 1

0
0

3.1. Estimate of the TDOA
Fig. 1 shows the overhead crane system with two
microphones (Microphone 1 and Microphone 2). With
two microphones, we observe the sound signal
originated from the sound source in the payload. Since
the microphones have a directivity, the sensitivity of
microphone is very low to the sound signal which iscoming behind the microphones. Now, let ( ) and( ) be the sound signal observed in the Microphone 1
and the Microphone 2, respectively. The distance
between the two microphones is X since the microphone
is located on the end of trolley rail. Further, let the
distance between the Microphone 1 and the soundsource be+ and the distance between theMicrophone 2 and the sound source be . Thus, isthe difference of the distances. Assume that x, X, , are
known. Now, the purpose is to estimate based on the
TDOA.
Using the two acoustic signals, we compute a
correlation coefficients of ( + ) ( ) as follows.

0 0 x(t )
0 0

0
0
1
u (t )
M
1
ml w

y(t )

1 0 0 00 1 0 0x(t )where u(t) is an input signal, y(t) is an output signal.
Based on the state-space model above, x and θ can becontrolled using a state feedback. To achieve the control,
the angle θ must be estimated.

T0

t 0

where

s1 t

s2 t

the number of sampled point.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

307

(1)

A sound-based measurement of

Let τ′ be the sampled point which maximizes (1).
Now, the TDOA between the Microphone 1 and theMicrophone 2 is τ
where is the sampling period.
If follows thatld

4.

Fig. 2 shows the experimental equipment. We use a
Gaussian random signal with 0～1000 [Hz] as the soundsource. The signal is generated by an electronic device.
The sampling frequency is 14700 [Hz]. The correlation
is calculated with data of samples corresponding to theposition x. We consider the case when the payload isswaying (i.e. the sway angle is moving). Fig. 3 shows
the result with x=20 where the vertical axis is the swayangle [deg.]. From Fig. 3, we can see that the payload isswaying around 0 [deg.]. In the same way, we show theresults the position x is changed as x=30, x=40, and
x=50. Similarly, Figs. 4-6 show the result with x=30,
x=40, and x=50, respectively.

(2)

Ts v ais the velocity of sound.

where

3.2. Conversion to angle θ
We introduce how to obtain θ with . Using the lawof cosines in Fig. 1, we have simultaneous equations as:

x2 l w

lm

lm ld

X

2

x

X

x

lw

2

2

2X

x lw cos

2

lw

2

2

2 xlw cos
2X

1
2

2

x lw cos

2

, ld

0,

,

2 2

ld
1
2

2
(5)

by computing

Hence, we can obtain θ corresponds
the solution which satisfiesf(3)

in (3) and (4), we have a function

x 2 lw, ld

2 xl w cos(4)

Now, eliminating
below.

f

2

Experiment.

Fig. 2. Experimental equipment(6)

It should be noted that since
can also be obtained
from (3) or (4), sensor delay can also be estimated by
the velocity of sound .
Here, let us show how to compute the solution of (6).
Now, we employ the Newton’s method to solve thesolution of (6). The Newton’s method uses the iterativeequation as
n 1

where

n

f
f

, ld
n , ldn

is the angle at n-th iteration.

(7)

Fig. 3. Experimental result with

= 20

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

308
1

Miki Matsunaga, Masayoshi Nakamoto, Toru Yamamoto

5. Conclusion
We have proposed the measurement method ofsway angle for anti-sway control of overhead cranewith two microphones. The method uses a timedelay of arrival (TDOA) of acoustic signals
measured from the microphones. Also, we havepresented the algorithm to obtain the sway angle
from the TDOA with the Newton’s method. In
experimental results, we demonstrate that oursound-based measurement method can follow thesway angle when the payload is moving.

Fig. 4. Experimental result with

References

= 30

1.

2.

3.

4.
Fig. 5. Experimental result with

= 40
5.

6.

Fig. 6. Experimental result with

C.-Y Chang and H. W. Lie: Real-Time Visual
Tracking and Measurement to Control Fast
Dynamics of Overhead Cranes, IEEE Trans. Ind.
Electron., vol.59, no. 3, pp.1640-1649, March 2012.
N. Sun and Y. Fang: New Energy Analytical Results
for the Regulation of Underactuated Overhead
Cranes: An End-Effector Motion-Based Approach,IEEE Trans. Ind. Electron., vol59, no.12, pp.47234734, Dec. 2012
L.-H. Lee. C.-H. Huang, S.-C. Ku, Z.-H. Yang, C.Y. Chang: Efficient Visual Feedback Method toControl a Three-Dimensional Overhead Crane,
IEEE Trans. Ind. Electron., vol61, no.8, pp.40734083, Aug.2014.
K. Zavari, G. Pipeleers and J. Swevers: GainScheduled Controller Design: Illustration on an
Overhead Crane, IEEE Trans. Ind. Electron., vol.61,
no.7, pp.3713-3718, July 2014.
N. Sun, Y. Fang and H. Chen: A New Antiswing
Control Method for Underactuated Cranes With
Unmodeled Uncertainties: Theoretical Design andHardware Experiments, IEEE Trans. Ind. Electron.,
vol. 59, no.3, pp.453-465, Jan. 2015.
M. Matsunaga, M. Nakamoto and T. Yamamoto: An
acoustic signal processing for anti-sway control ofoverhead crane system, Proc. of 2017 International
Workshop on Smart Info-Media Systems in Asia
(SISA 2017), pp.124-127, Sept. 2017.

= 50© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan309

Sampled-data PID Control System with a Sensitivity Function
for a Second-order Plus Dead-time System*
Ryo Kurokawa and Takao Sato

Department of Mechanical Engineering, University of Hyogo
2167 Shosha, Himeji, Hyogo 671-2280 Japan
E-mail: tsato@eng.u-hyogo.ac.jp
Ramon Vilanova

Department of Telecommunications and Systems Engineering, Universitat Aut ̀ nomade Barcelona
Edifici Q-Campus de la UAB, 08193 Bellaterra, Barcelona, Spain
E-mail: ramon.vilanova@uab.cat
Yasuo Konishi

Department of Mechanical Engineering, University of Hyogo
2167 Shosha, Himeji, Hyogo 671-2280 Japan
E-mail: konishi@eng.u-hyogo.ac.jp

Abstract
In this paper, we design a second-order plus dead-time (SOPDT) sampled-data Proportional-Integral-Derivative(PID) control system, where the continuous-time plant is controlled using the discrete-time controller. The proposed
control system is designed so that the tracking performance is optimized subject to the stability margin constraint. In
the present study, the servo and regulation optimal controllers are designed. Finally, the effectiveness of the proposed
method is demonstrated through numerical examples.
Keywords: PID control, Sampled-data system, SOPDT system, Sensitivity function, Robust

trade-off relationship between stability and tracking
performance, they cannot be optimized simultaneously.
Arrieta and Vilanova3,4 proposed a simple PID tuning
method that optimizes the tracking performance subject
to a prescribed robust stability. In this method, theoptimal PID parameters are decided based on a first-orderplus dead-time (FOPDT) continuous-time system. In
order to design a discrete-time control system, Tajika et

1. Introduction
Proportional-Integral-Derivative (PID)1,2 control has
been widely used in industry. Since the performance of
PID control depends on the tuning parameters, additionaltuning methods have been studied recently. Although thestability of a control system is critical, its tracking
performance is also important. However, because of the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan310

Ryo Kurokawa, Takao Sato, Ramon Vilanova, Yasuo Konishi( ) is the discrete-time controlled plant.
where
Using the sensitivity function, the constraint condition isdefined as follows:al.5 proposed a design method for controlling a discretetime FOPDT system. The present study discusses a
design method of the PID controller for controlling asecond-order plus dead-time (SOPDT) system, in which
the continuous-time plant is controlled using the discretetime controller. In the proposed method, both servo andregulation optimized control methods are designed.
Finally, the effectiveness of the proposed method isdemonstrated through numerical examples.

|

= max

Consider the continuous-time controlled plant given as
follows:
+2

(1)

+

(

(

(

(

) ( )+

)=

1+

)=

(

)=

(

)+

(

(1 −

(1 −

) ( )
)

3.1. Constraint condition) is defined as follows:

(

1+

(5)

| ( ) − ( )|The PID parameters are optimized for a normalized
system, and hence, dimensionless parameters are defined
as =
, ℎ=
,
=
,
=
, and
=
. The range of these parameters are set as
0.1 ≤ ≤ 1.0, 0.01 ≤ ℎ ≤ 0.10, and 0.3 ≤ ≤ 1.2.
In the proposed method, the constrained optimal problem
is preliminarily solved for a designated finite plant,
which is defined by discrete , ℎ, and , and the data set
in which the optimal normalized PID parameters fordiscrete , ℎ, and , is obtained. In Fig. 1, the obtained
normalized PID parameters are plotted by ∘, where= 1.4 and= 0.01.

As the constraint condition, the stability margin isdefined using the sensitivity function, and the evaluation
function for the tracking performance is also defined.

)=| ( )| =4. Controller Design

3. Definition of the Optimization Problem(

is defined

A trade-off relationship exists between the servo
performance and the regulation performance. In thepresent study, the PID parameters are optimized for the
servo and regulation control, respectively.

(2)

where ( ) is the control input, ( ) is the plant
output, ( ) = ( ) − ( ) is the control error, and( ) is the reference. Moreover, ,, , andare the sampling time, the proportional gain, the integral
time, and the differential time, respectively.

The sensitivity function)

In the present study, the evaluation function
as the integral absolute error:
=

)

)

(

(4)

3.2. Evaluation function

whereis the plant gain,is the natural angularfrequency,is the damping coefficient, andis thedead-time. In the present study, we discuss the design
method of the sampled-data control system using thefollowing discrete-time PID control law:
( )=

|=0

where
is the maximum value of the sensitivity
function, andis the desired value selected by thedesigner. The recommended range ofis from 1.4 to, the larger the stability
2.01. The smaller the value ofmargin. On the other hand, the larger the value of,the better the tracking performance, although the stability
margin becomes small.

2. Description of the Control System

( )=

−

(

1

)

(

)

(3)

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

311

Sampled-data PID Control System

Solving these equations, the desired, , andfor[ , , ℎ ] are obtained, where α, β, and γ are decided
based on the following equations:Fig. 1. Relationships among , , anddesign,= 1.4 and= 0.01),and−= ( − )+ ( − )+ ( − )−= ( − )+ ( − )+ ( − )ℎ − ℎ = (ℎ − ℎ ) + (ℎ − ℎ ) + (ℎ − ℎ )In Fig. 1, the interpolated parameters are plotted over thediscrete calculated optimal parameters. Furthermore,is calculated for both the preliminarily solved andinterpolated systems using the approximation method,
and the obtained
values are shown in Table 1. This
result reveals that the proposed decision method is
sufficiently effective.

(servoThe desired normalized PID parameters for an arbitrary
plant are decided by the linear interpolation from the data
set. Practically speaking, the interpolated parameters arecalculated using the nearest four points, as shown in Fig.
2. From this figure, the vector equation is obtained asfollows:OP⃗ = OA⃗ + OB⃗ + OC⃗
0≤

≤1

0≤

≤1

0≤

Table 1. Obtained
Servo design

(6)

≤1

, ℎ ,
,
,
], A [ ,,where point O [ ,ℎ ,,,], B [ ,, ℎ ,,, ], and C
[ ,
, ℎ ,
,
,
] are the nearest points of thedesired [ ,, ℎ ]. Then, Eq. (6) is rearranged asfollows:
=
=
=

+
+
+

−
−
−

+
+
+

−
−
−

+
+
+

1.4

Min
1.398

Mean

1.6
1.8

1.599
1.790

1.403
1.605
1.807

2.0

1.996

2.010

Regulation design
Max
1.440

Min
1.399

Mean
1.403

Max
1.453

1.668
1.909

1.597
1.798

1.605
1.807

1.663
1.897

2.156

1.997

2.009

2.137

5. Numerical Simulation
In this section, the effectiveness of the proposed method
is confirmed.

−−−

5.1. Control performance for various values ofFirst, the control performance is confirmed for . The
controlled plant is defined as= 4.2,= 1.13, and= 0.44 in Eq. (1), and= 0.018. Here, we consider
four pattern damping coefficients:= 0.451 ,=0.69,= 1.0, and= 1.199. The control results areshown in Fig. 3. The reference value is set to 1.0, and theunit step disturbance signal is added after 20 s. Figure 3
shows that the proposed method is effective for underand over-damping systems.

Fig. 2. Image of the linear interpolation© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

312

Ryo Kurokawa, Takao Sato, Ramon Vilanova, Yasuo Konishiparameters are designed for the normalized system, andthe tracking performance is optimized subject to theassigned . Finally, the effectiveness of the proposed
method is demonstrated through numerical examples.

5.2. Verification of stability margin
Next, the stability margin is confirmed. Here, thecontrolled plant is defined as= 2.02 ,= 0.91 ,Acknowledgements
The present study was supported by a Grant-in-Aid for
Scientific Research (C) 16K06425. The authors would
like to express their sincere thanks for the support fromthe Japan Society for the Promotion of Science.
References
1. K. J. Åström and T. Hägglund, Advanced PID Control,(ISA-Instrumentation, Systems, and Automation Society,2006).
2. R. Vilanova and A. Visioli, PID Control in the Third
Millennium (Springer, UK, 2012)
3. O. Arrieta and R. Vilanova, Simple PID Tuning Rules with
Robustness Achievement, IFAC
Guaranteed
Proceedings 44(1) (2011) 12042-12047.
4. O. Arrieta and R. Vilanova, Simple Servo/Regulation
Proportional-Integral-Derivative (PID) Tuning Rules for-Based Robustness Achievement, Industrial
Arbitrary& Engineering Chemistry Research 51(6) (2012) pp.
2666-2674.
5. H. Tajika, T. Sato, R. Vilanova and Y. Konishi, Optimal
PID Control in Discrete Time Using a Sensitivity Function,
23th Mediterranean Conference on Control andAutomation (MED) (2015) pp. 249-254.

Fig. 3. Output responses for each damping coefficient(servo design and= 1.4)= 0.33 , and = 0.98 in Eq. (1), and= 0.05 .
After 40 s, the dynamics is changed to= 2.6,=
1.3, = 0.43, and = 0.43 as the model variation.
Furthermore,is varied as 1.4, 1.6, 1.8, and 2.0,
respectively, and the control results are compared. The
obtained results are shown in Fig. 4. The reference valueis 1.0, and the unit step disturbance signal is added after
20 s. The model variation is caused at 40 s. Figure 3
shows that the smaller the value of, the larger thestability margin, and vice versa. On the other hand, thelarger the value of, the better the tracking
performance, and vice versa.
Conclusion
In the present study, we proposed a new design method
for controlling an SOPDT sampled-data system, where
the continuous-time plant is controlled by the discretetime PID control law. In the proposed method, the PID

Fig. 4. Output responses for each
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

313

Experimental Evaluation of a Data-Driven Control System using an Electronic Thermal
Regulator
Yuka Okubo

Graduate School of Engineering, Hiroshima University,
1-4-1, Kagamiyama, Higashihiroshima city, Hiroshima, Japan
Yoichiro Ashida

Graduate School of Engineering, Hiroshima University,
1-4-1, Kagamiyama, Higashihiroshima city, Hiroshima, Japan
Takuya Kinoshita

Graduate School of Engineering, Hiroshima University,
1-4-1, Kagamiyama, Higashihiroshima city, Hiroshima, Japan
Toru Yamamoto

Graduate School of Engineering, Hiroshima University,
1-4-1, Kagamiyama, Higashihiroshima city, Hiroshima, Japan
E-mail: okubo-yuka@hiroshima-u.ac.jp, ashida-yoichiro@hiroshima-u.ac.jp, kinoshita--takuya@hiroshima-u.ac.jp,
yama@hiroshima-u.ac.jp
http://www.hiroshima-u.ac.jp

Abstract
IoT (Internet of Things) attracting attention, the data of the controlled objects is expected to be utilized to control
systems in industries. Therefore, the data-driven scheme is hoped to be implemented to the electronic thermal
regulator. The technique is hardly to be implemented in it due to restricted memory capacity, hence, a computer
calculates PID gains by the data-driven and sends the gains to the electronic thermal regulator. The effectiveness ofthe proposed platform is demonstrated by the experiment.
Keywords: PID controller, data-driven controller, electronic thermal regulator, FRIT

the practical controlled objects have nonlinearity, and it
is difficult to control them suitably by using the fixed PID
controller. Therefore, the data-driven technique5 having
effectiveness for non-linear systems has been proposed.
Calculating a set of PID gains based on the databasedepending on current operating data, the data-driven
scheme can appropriately control such systems. The datadriven technique needs the large memory for database.
Therefore, it is hard to implement this technique to theelectronic thermal regulator due to its restricted memory
capacity. If the computer is utilized for implementing thedata-driven control scheme, the new interface between

1. Introduction
In recent years, IoT has been received attentions, and thedata of systems obtained through a lot of sensors is hoped
to be utilized to control systems in industries1. Such data,
which is collected by communication between sensors
and cloud, is stored in database.
On the other hand, the electronic thermal regulators2
are employed to control systems as general controllers, inwhich PID controller3, 4 is basically applied. In the caseof using PID controller, the PID gains must be tuned
considering the properties of controlled objects. Most of© The2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

314

Yuka Okubo, Yoichiro Ashida, Takuya Kinoshita, Toru Yamamoto

the computer and controlled object must be introduced
instead of the electronic thermal regulator. Hence, thedata-driven scheme is hardly implemented to theindustrial field.
In this paper, the construction of the data-driven
control platform using the electronic thermal regulator isfirstly explained, in which the database is introduced inorder to realize the data-driven PID controller. According
to the proposed platform, the computer calculates PID
gains by the data-driven technique and sends them to theelectronic thermal regulator. As a result, theaforementioned memory capacity restriction of theelectronic thermal regulator can be avoided. Furthermore,
the existing interfaces between electronic thermal
regulator and control objects do not need to be changed
for implementing the data-driven scheme.
In this paper, section 2 shows the schematic of thedata-driven control platform. The design of data-driven
controller is explained in section 3. In section 4, it isshown the effectiveness of the proposed platform byconducted experiment using actual equipment. The paper
ends with concluding remarks in section 5.

Fig. 1. The schematic of data-driven control platform.

where ( ) is the control input, and ,are the ordersof the system output and the control input, respectively.
In this paper, the following control law with a PID
structure is introduced:( )=2. Construction of data-driven control platform

( )−

+

( )

The schematic of data-driven control platform is shownin Fig. 1. In this paper, the electronic thermal regulator
plays a roll of a controller. The electronic thermal
regulator receives control outputs, e.g. temperature,humidity or pressure through a sensor and determines thecontrol input. In the proposed platform, the reference
signal and control I/O data are sent to the computer
having database from the electronic thermal regulator.
The computer sends a set of PID gains calculated on datadriven scheme to the electronic thermal regulator. Since
the computer sends not control input but PID gains, theelectronic thermal regulator can control systems using
last PID gains even if the communication is fault.

( )−
( )−
( ),
=
(3)
(
)
where
denotes the control error signal defined by
the following equation:( ) = ( ) − ( ),(4)inwhere ( ) is the reference signal. The , , andequation (3) express the proportional gain, integral time,and derivative time, respectively.
,
andsimilarly denote the proportional gain, integral gain andderivative gain. The
shows the backward operator.
and (≔ 1 −) are the sampling time anddifferencing operator.

3. Design of the data-driven controller[STEP1] Generate initial database: In the data-driven
control, the control parameters are calculated from thehistorical closed-loop data stored in a database. If there
are no stored information vectors in the database, thedata-driven scheme cannot work. To create initial
database, it is required to obtain the closed-loop data. The
data stored in the form by the following equation:(t) ≔ [ ( ), ( )], = 1,2, ⋯ , (0)(5)

3.2. Design of data-driven controller

3.1. Controlled Object
The controlled object can be described by the following
nonlinear system:( )=( − 1) ,(1)
where ( ) is system output, (⋅) denotes the function of
nonlinearity and ( − 1) is an information vector
showing the historical data of the system. ( − 1) isdefined by following equation:( − 1) ≔ [ ( − 1), … ,−,( − 1), … , ( − )], (2)( ) ≔ [ ( ), ( ), ( )],(6)where (0) is the number of information vectors stored
in the initial database. The query is defined by thefollowing equation:( ) ≔ ( + 1), ( ), ( ), … , ( −+ 1),( − 1), … , ( −(7)+ 1)],© The2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

315

Experimental Evaluation of a

[STEP2] Calculate distances and Select neighbors: The
distances between query ( ) and every information
vector ( ), where ≠ , in the database are calculatedby the following equation:( ), ( ) =( )− ( ),( ) − min ( )

max

(8)
( = 1, 2, ⋯ , ( ))where ( ) is the number of information vectors in the
database at . The max ( ) denotes the maximum
value ofelements in all information vectors. Similarly,
the min ( ) expresses the minimum value ofelements in all information vectors. The k -pieces vectors
with the smallest are selected from all information
vectors. The selected information vectors and PID gains
corresponding to the information vectors are neighbor
vectors.
[STEP3] Calculate PID gains: The suitable PID gains
around the query are calculated by applying the steepest
decent method to selected neighbor vectors in STEP 2.
( )=

( ),

= 1,

Fig. 2. Appearance of the temperature control equipment.

used so that the closed-loop system including a controller
has a desired characteristic of the reference model. ( )is shown by the following equation:
(1)
( )=
(14)
̃ ( ).
( )( ) is a user-specified polynomial included in thereference model and arbitrarily designed in advance suchthat the model has a desired characteristic.
(
)≔1+
(15)
+
,

(9)

= −2exp −()wherein equation (9) is weight forcorresponding to theinformation vector ( ) as
follows:=,= exp −

cos

≔

(10)

≔ 0.25(1 − ) + 0.51∑
where expresses the distance calculated by equation(8) between query andinformation vector.

⎫
⎪
⎬
⎪
⎭

.

(16)

denotes the rise-time, and is a parameter relevant todumping. It is required that is set to between 1/2 and1/3 sum of time constant and dead time. is desired to
set to between 0 and 2. When = 0, the step response is
the Binomial model response. When = 1 , the step
response shows the Butterworth model response.
In equation (14), the fictitious reference input ̃ ( ) is).
calculated by input-output relation of controller (The ( ) is expressed as follows:( )= +(17)+ ⋯+,where is the control law order. equals 2 in the casewhere PID controller is used. The input-output relationof controller is shown by the following equation:(){ ̃ ( ) − ( )},
( )=
(18)
∆
(
)∆ ( ) + ( ).
(19)
̃( ) =
the fictitious reference signal ̃ ( ) is obtained bydeformation of equation (18).

3.3. Off-line learning by FRIT
The fictitious reference iterative tuning (FRIT)6 is atechnique calculating directly the control parametersfrom a pair of I/O data. The new PID gains
areobtained by updating the database stored PID gains.
The following steepest decent method is applied tomodify the PID gains:( + 1)(11)=−,≔ [ , , ],wheredenotes the learning rates for each control
parameters. The ( + 1) expresses the error criterion as
follows:1( + 1) ≔ ( + 1)
(12)2
( + 1) ≔ ( + 1) − ( + 1).
(13)

4. Experimental control results

When ( + 1) is minimized, the PID gains can be( ) and( ) are equal. In
calculated such that( ) isequation (13), the fictitious reference output

In this paper, the data-driven control platform wasapplied to the temperature equipment shown in Fig. 2.

© The2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

316

Yuka Okubo, Yoichiro Ashida, Takuya Kinoshita, Toru Yamamoto

The electronic thermal regulator E5CC-QQ2ASM andthe interface converter K3SC-10 for connection between
the electronic thermal regulator and a computer was
utilized, which they made by OMRON Corporation. The
temperature equipment equips with a ceramic heater andan aluminum block. The control output ( ) is thetemperature of the aluminum and is measured with athermocouple of K-type. The control input ( ) is duty
ratio which determines how much to heat through a relay.
The reference signal ( ) was set to ( ) = 60 [℃] andthe sampling time
was set to 2 [s]. After the control
output can track to the reference signal, a disturbance wasapplied that non-heated aluminum contacts and cools theheated one.
The PID parameters for generating initial databasewere denoted as follows:(20)= 2.07, = 0.021,= 1.29.
The initial data vectors were stored database according toequation (5). User-specified parameters for theexperiment are shown in Table 1.
The desired polynomial ( ) was designed as() = 1 − 1.91
(21)
+ 0.91 ,Table 1: User-specified parameters for the experiment.
Orders of the information
vectorNumber of neighbors
Learning rates
Initial Number of data

andwere set to = 90 and = 0 . The
where
control results of using the fixed PID gains and the
proposed platform are respectively shown in Fig. 3. The
upper figure shows that solid line is the control output by
the proposed platform, dotted line is the desired control
output, dash line is the reference signal and chain line isthe control output using the fixed PID gains. The control
result of using fixed PID controller cannot track to thedesired control output and the disturbance let it cool 13℃
approximately at = 700 . In comparison with two
results, the rise-time of the proposed scheme is shorter
than one of the fixed PID controller. Therefore, theeffectiveness of data-driven control platform was verified.

=1
=2
=6
= 1.0 × 10
= 1.0 × 10
= 1.0 × 10(0) = 1138

Fig. 3. The control results by the proposed platform, and by
fixed PID controller.

References
1. T. Munaka, Newly Coming World by IoT-The Trend ofIndustrial Internet, System, Control and Information, Vol.
60, No. 10, (2016) pp. 425-430. (in Japanese)
2. Tateishi Electrric Co., Temperature Controller Thermac X
Seriesss, Journal of The Society of Instrument and Control
Engineers, Vol. 27, No. 12 (1988) pp. 1130-1131. (inJapanese)
3. K. J. Astrom, T, Hagglund, Advanced PID Control,International Society of Automation, (2005)
4. K.L. Chien, J.A. Hrones, and J.B. Reswick, On the
Automatic Control of Generalized Passive Systems,
Trans. ASME, Vol.74, (1972) pp. 175-185
5. T.Yamamoto, K.Takao, and T.Yamada, Desigh ofa Data-Driven PID Controller, IEEE Transaction on
Control Systems Technology, Vol.17, No.1 (2009), pp. 29-39
6. S. Soma, O. Kaneko, and T. Fujii, A New Approachto Parameter Tuning of Controllers by Using One-Shot
Experimental Data: A Proposal of Fictitious Reference
Iterative Tuning, Trans. Institute of Systems Control
and Information Engineers, Vol.17, No.12, (2004)
pp. 528-536

5. Conclusion
In this paper, the effectiveness of data-driven control
platform has been verified. In the proposed platform, it ishandily permitted advance control law that a computer
sends PID gains to the electronic thermal regulator using
an existing interface. Even if PID gains cannot be sent to
the electronic thermal regulator, a system is controlledusing the last PID gains. In this paper, the PID gains areadjusted at every step. In future work, the control design
method, in which PID gains are adjusted only the casewhere the control performance is deteriorated, will beconsidered.

© The2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

317

Self-repairing Adaptive PID Control for
Plants with Sensor Failures
Masanori Takahashi

Department of Electrical Engineering and Computer Science, Tokai University
9-1-1 Toroku, Higashi-ku, Kumamoto, 862-8652, Japan
E-mail: masataka@ktmail.tokai-u.jp

Abstract
This paper presents a new design method for a self-repairing adaptive PID control system. The control system has
the adaptive adjusting mechanisms for the PID gains, and also can detect the sensor failure by self-test using the
integrator in the PID controller. Hence, for plants with unknown parameters, self-repairing control can be successfully
attained. Furthermore, in this paper, the control performances are theoretically analyzed.
Keywords: PID control, Adaptive control, Fault-tolerant control, Sensor failure.

test using the integrator in the PID controller. Thus, no apriori information about the plant is needed to attain the
SRC. The proposed SRCS is expected to be widely
utilized as fault-tolerant PID control.

1. Introduction
Against sensor failure, several types of self-repairingcontrol systems (SRCS) have been developed in theprevious works. The SRCS can detect failure by selftesting using the internal signals, and automaticallyreplace the failed sensor with the healthy backup to
maintain its stability. They belong to active fault-tolerant
control (AFTC) based on dynamic redundancy1. The
main difference (feature point) from the conventional
AFTC is that the fault detector is quite simple, and its
structure does not depend on the mathematical model ofthe plant.2,3,4 For example, the integrator in the PID
controller can be utilized as fault detector for the SRCS5.

Furthermore, in this paper, the control stability, failure
detection and sensor-recovery are theoretically analyzed.
In addition, the numerical simulation is explored toconfirm the effectiveness of the proposed adaptive SRCS.
Throughout this paper, with ∈ ℝ, we define the “sgn”
function by1 ( ≥ 0)sgn[ ] =−1 ( < 0)Notice that this is slightly different from the ordinary one.

However, in most existing SRCSs, roughly estimatedparameters of plants are required to construct thecontroller. This paper presents a new design method fora self-repairing adaptive PID control system. The control
system has the adaptive adjusting mechanisms for the
PID gains, and also can detect the sensor failure by self-

2. Problem Statement
Consider the following linear time invariant system:
̇=
+
+
̇=
+
(1)

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

318

Masanori Takahashi

where ∈ ℝ is the actual output, ∈ ℝ is the input,
and ∈ ℝ
is the state of the plant. Here, assume thatthe plant is minimum-phase, that is, all eigenvalues of∈ ℝ( )×( ) lie in ℂ .

introduced as a scaling factor. These are given by
designers.
Here, we shall analyze the control stability on the time
period [0, ) where the sensor is healthy, i.e., = .
Now, define the augmented signal : ℝ → ℝ by:= − +(7)
Then, the signals, , , obey the following equations.

For measurement of the output , the two sensors are
exploited; one is the primary sensor #1 and the other isthe backup sensor #2 for occasion of failure. Then, themeasured feedback signal can be represented as( )( ≤ )( )=(2)( )( > )̇=

where ∈ ℝ, = 1, 2 is the output of the sensor # , and∈ ℝ is the detection time whose details will be
determined later. If the sensors are healthy, then themeasured signals are equivalent to the actual output, thatis, we have = . Unfortunately, the primary sensor
fails in the following way.
( )= , ≥(3)
where ∈ ℝ is the unknown failure time, and ∈ ℝ
is the unknown stuck value.

1
1+

∗

−

− (1 −∗{−(−−∆ ( − ) − ( ∗ +

̇=

−

+

̇ =− ∗ +

∗

∗)

∗

−

+

∗

−

sgn[ ]}

∗
∗)

∗)

− ∆

+ ∆
̇

(8)

− ∆ ( − ) − sgn[ ], ∆ := −and ∆ : = ∗ −where ∆ : =−, and ∗ ∈ ℝ , ∗ ∈ ℝ and ∗ ∈ ℝ are the ideal
gains for PID control.
∗∗

Consider the positive definite function : ℝ → ℝ :=The problem to be considered here, is to construct theactive fault-tolerant PID control system for plant withunknown parameters and sensor failures (3), that can
automatically replace the failed sensor with the backup to
maintain the control stability.

1
(1 +
2

∗)

+

+

1
+ ∆ + ∆ +∆(9)where ∈ ℝ( )×( ) is the positive definite matrix
which satisfies+= −2 for any positive
definite ∈ ℝ( )×( ) .

3. Basin Design of the Control System
First of all, the adaptive PID controller is designed by= (− + ) −̇(4)where ∈ ℝ is the output of the integrator:
̇ =−− sgn[ ](5)with a constant ∈ ℝ , which is introduced for detectionof failure. The fault detection using the signal will be
discussed in the next subsection.

Taking the time derivative of , giveṡ = −( ∗ − − ∗ + ∗ ∗ )
+ ∆

∗

−(

+

−(1 −
−

In the above adaptive controller, the PID gains, : ℝ →ℝ , : ℝ → ℝ and : ℝ → ℝ are adaptively tuned
as follows.
̇ =−+ (− + )̇ =−( −2 )−̇ =−−̇ (− + )(6)
where ∈ ℝ is an any small constant, and ∈ ℝ is apositive constant. Also, a small constant ∈ ℝ is

−

−

∗

−∆ ( − ) − ∆

−

−

∗

∗)

) sgn[ ]

∗

+

∗

−

+

− ∆ ( − ) − sgn[ ]1∆ ̇ − ∆ ̇ −

∆

−

−

̇

Furthermore, from (6), it follows that
1
1
1
̇=−
‖ ‖ −
−
2
2
2
2

∆

̇

−

2

∆

2

∆

+

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

319

(10)

(11)

Self-repairing Adaptive PID Control

where
=

=2

∗

2

1

+

∗

[ ]−3

= (1 −

=

∗

+

2

∗

+

∗)

− 3| | − 2 ∗ − ‖ ‖ − ‖∗− | |−‖+( ∗) +

Here, choose ,Then, we have
̇≤−

∗

,

∗

+ ,

= min

which yields,

‖

1

and

( ∗) +,,2∗( ) ≤ (0)exp(−2

so that,2

From the above discussion, on the time period [0, ),there exists Γ ∈ ℝ such that( ) < Γ, ∈ [0, )(14)However, if the sensor fails, and = 0, then the outputof the integrator can be represented by‖,= ( )−( ∗))+, ∈ [0,)) (15)

Clearly, is unbounded with respect to . Hence, theinequality (14) hold no longer. By using this unstablebehavior, we can find failure, and define the detection
time by(16): = min{ | | ( )| ≥ Γ }> 0, = 1,2,3.

2

dt = ( ) − ( −(12)Next, we consider the faulty situation where(13)

the solution ( ,) obeys≠ 0. Then,̇ =−− sgn[ ]̇ =−− ( −2 )Hence, the equilibrium point ̅ ,, is given by

Hence, from (9), it is verified that all the signals in theadaptive control system are bounded on [0, ).

̅=The block diagram of the proposed control system is
illustrated in Fig. 1.

2

+

=−

sgn[ ]

2sgn[ ]

Now, we suppose that Γ, , and are selected so that
Γ<

2

+

2

Then, even if approaches to the equilibrium point, ithits the threshold Γ . Therefore, the detection time
defined by (16) exists when ≠ 0. Failure detection canbe successfully achieved. After detection, the failed
sensor is replaced with the backup, and the control
stability can be maintained.

5.

Numerical Simulation

To confirm the effectiveness of the proposed method, thenumerical simulation is explored.
Consider the following unstable plant:

Fig. 1. Block diagram of the proposed adaptive PID control
system with self-repairing function.

4.

̇ = + 2 + , (0) = 1(0) = −0.5
̇=− + ,
The failure scenario is supposed that

Failure Detection & Repairing

This section shows the concrete detection method
utilizing the integrator (5).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan320

Masanori Takahashi= 25 [ ], = ( )(17)
In the simulation, they are assumed to be unknown.

6.

In this paper, a new adaptive PID control system has beendeveloped that has self-repairing function against sensor
failures, and the concrete failure detection is shown by
using the integrator in the PID controller.

For the above plant, we construct the adaptive PID
controller based on (4) and (5). By trial and error, thedesign parameters are selected as follows.
= 0.01, = 1, = 0.1
The other parameters for failure detection are given by= 1,Γ = 1

The applications to MIMO case, nonlinear systems andso on are still left in the future works.
Acknowledgment

The simulation results are shown in Figs. 2 and 3. In Fig.
2, the measured output , the actual output (top) andthe absolute value of (bottom) are shown. Also, in Fig.
3, the three adaptive gains,(top),(middle) and(bottom) are shown.

This work is supported by JSPS KAKENHI Grant
Number JP16K06429.
References
1. R. Isermann, R. Schwarz and S. Stolzl: Fault-tolerant,
Drive-by-wire systems, IEEE Control Systems Magazine,
Vol. 22, No. 5, pp. 64-81 (2002).
2. Y. Zhang and J. Jiang: Biographical review onreconfigurable fault-tolerant control systems, Annual
Reviews in Control, Vo. 32, No. 2, pp. 229-252 (2008)
3. K. Zhang, B. Jiang and V. Cocquempot: Fast adaptive fault
estimation and accommodation for nonlinear time-varying
delay systems, Asian Journal of Control, Vol. 11, No. 6,
pp. 643-652 (2009).
4. Y. Wan, D. Zhou, S. J. Qin and H. Wang: Active faulttolerant control for a class of nonlinear systems withsensor faults, Int. J. of Control, Automation and Systems,
Vol. 6, No. 3, pp. 339-350 (2008)
5. M. Takahashi: Self-repairing PI/PID control against
sensor failures, Int. J. Innovative Computing, Information
and Control, Vol. 12, No. 1, Feb. pp. 193-202 (2016)

From these results, it is clear that the failure (17) can be≅ 26 [s].
found and the failed sensor is replaced atFurthermore, the control system can be well stabilized,
and the actual output converges to a very small ballbefore and after the failure.

[s]

Sensorreplacedhere| |Conclusions

[s]
[s]

Fig. 2. Simulation results: the measured output, the actual
output and the absolute value of the integrator output.

[s]

[s]

Fig. 3. Simulation results: the three adaptive PID gains.
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan321

Design and Development of a Constant Temperature Reservoir
for a Database-Driven Smart Cultivation System
Shin Wakitani

Graduate School of Engineering, Hiroshima University, 1-4-1 Kagamiuyama
Higahsi-hiroshima, Hiroshima 739-8527, Japan
Sharma Sneha

Graduate School of Engineering, Hiroshima University, 1-4-1 Kagamiuyama
Higahsi-hiroshima, Hiroshima 739-8527, Japan
Toru Yamamoto

Graduate School of Engineering, Hiroshima University, 1-4-1 Kagamiuyama
Higahsi-hiroshima, Hiroshima 739-8527, Japan
E-mail: wakitani@hiroshima-u.ac.jp, sharma-sneha-atul@hiroshima-u.ac.jp, yama@hiroshima-u.ac.jp
https://www.hiroshima-u.ac.jp

Abstract
Decreasing the number of agricultural working population is serious problem. Production of agricultural products inplant factories is as one of the solutions to the above problems. However, it is not easy to set optimal environment
conditions for many kinds of plants in food factories. This research aims for realization of a smart cultivation systemthat uses a database-driven (DD) technique. In this work, a pilot scale constant temperature reservoir is developed,
and a DD nonlinear temperature control is performed.
Keywords: Cultivation System, Temperature control, Data-driven control, Nonlinear control.

and they can be applied to the system easily. Although,actual controlled variables that should be manipulated inplant factories are growth speed, taste, nutrition etc. ofplants, and the environment control system must be
designed to maximize a criterion including the above
variables, a uniform method that can deal with manykinds of plants has not been established.
A plant can be regarded as a controlled object which hasstrong nonlinearity from a point of view of control
engineering. Thus, it is difficult to manipulate them by alinear controller with fixed control parameters. Moreover,it may difficult to maintain ideal environment conditionsof a plant factory by the controller due to the changing
environment around the factory and aged deterioration ofequipment. For such systems, a database-driven (DD)

1. Introduction
Recently interests and requirements to domestic
production and pesticide-free production of agricultural
products have been increasing in Japan. However, it has
not putting the brake on the decreasing of agricultural
workforce. Because of this situation, a plant factory, thatcan make many production processes of agricultural
products automate and is able to produce them safely andstably, has been gotten much attention. A plant factory
can be said as one form of protected horticulture that cancontrol production schedule by manipulating its
environment conditions such as temperature, humidity,
CO2 concentration. Techniques for such environments
controls have been actively studied in process control,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

322

Shin Wakitani, Sneha Sharma, Toru YamamotoFig. 1. Appearance of constant temperature reservoir.
Fig. 2. Schematic of constant temperature reservoir.

control method1, that adaptively tunes control parameters
corresponding to environment conditions by utilizing a
database which is stored a lot of previous operation data,
is expected to achieve high control performance.
Moreover, as related researches of the DD control
technique, design methods of nonlinear modeling2, softsensor, expert controller3 (that can imitate several experts’control action) based on a database have been proposed.
From these circumstances, it can consider to be able torealize a smart cultivation system that utilizes DD
methods as core technique.
In this research, a pilot scale constant temperature
reservoir is developed as a first step. In this system, aPeltier device which is one of thermoelectric conversion
devices is implemented to control temperature. In the
Peltier device, by supplying current to the device, oneside exhausts the heat, and the other side absorptions theheat corresponding to the amount of the current.
Moreover, the exhaust and absorption face of the heat canbe switched by changing the direction of the current.
However, the amount of generated Joule heat follows thesquare of the amount of the current, thus a linear
controller with fixed control parameters does not
maintain a good control performance due to thenonlinearity. Therefore, in this paper, the DD
temperature controller is designed and applied to thesystem to track the actual temperature to wide range
temperature references. However, it spends much time toevaluate the effectiveness of the system due to the large
time constant of the system. In order to perform debug
and performance evaluation of an algorithm the DD
controller rapidly, a nonlinear simulator is first
constructed based on a principal of the heat balance.
This paper is organized as follows. In Section 2, the pilot
scale constant temperature and its nonlinear modeling is
explained. In Section 3, control results of a fixed PID

Table 1. System parameters.
Variablename

Explanation

Value

Resistance of Peltier device
Thermal conductivity of Peltier device
Peltier coefficient
Proportionality constant for T
Thermal resistance
Thermal capacity
Thermal resistance
Thermal capacity

3.85
10.0
1.00
5.00
0.83
5.00
39211
994.6

controller and the DD controller is shown. Finally,
Section 4 summarizes the research findings.
2. Constant Temperature Reservoir Design
Appearance of the constant temperature reservoir isshown in Fig. 1. The schematic and system parameters
are shown in Fig. 2 and Table 1. Specific values of systemparameters are identified by the genetic algorithm.
2.1. Constant Temperature Reservoir
A Peltier device is implemented on the back of the system,and temperature sensor and humidity sensor are putted on
the inside wall. Moreover, full color LEDs for growing
plants is putted on the ceiling. The Arduino is utilized asa controller, and it can manipulate the amount of current
from the power supply by changing a duty ratio of PWM
output voltage of the controller.
2.2. ModelingIn this section, the following couple of heat balance
relationships are derived:(1) Peltier device – Aluminum block.
(2) Aluminum block – Space of constant temperature
reservoir.
A simulator based on these heat balance models isconstructed by using MATLAB/Simulink.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

323

Design and Development of

2.2.1. Heat balance between Peltier device andaluminum block

A derivative equation related into the time variation of( ) is given as follows:
( )

( )−

=

(1)

( ).

Where the maximum voltage of the power supply
is set to 15 V, and the range of the manipulated value( ) is set between −100% ≤ ( ) ≤ 100% , theamount of the current in the Peltier device can be given
as follows:
( )

.

( )=

(2)

Fig. 3. Static property of simulation model.

( ) that is generated by the above current in the
Peltier device is described as follows:
( )=

( )+

( )+

( ) , (3)

( )}.

(6)

3. Application of DD Control Technique
Temperature of the constant temperature reservoir must
be controlled properly in order to produce many kinds ofplants. In this work, the DD control technique is applied
to control the temperature. In the DD controller design
scheme, the initial database is required. Thus, in order togenerate an initial database, a fixed PID controller isapplied to the system, and its operation data is stored into
a database.

( )=
( )−
( ).
(4)
Note that the exhaust heat face of the Peltier device
touches a water-cooling face, and it is assumed that thetemperature of the contact face is given the following
equation:
( )=
( ).
(5)
The
( ) is given as follows:
( )=

{

( )−

3.1. Control result by Fixed PID controller
Initial PID parameters are obtained based on a model
munching. Firstly, it is assumed that the controlled object
can be approximated as the following first order systemwith time delay around an equilibrium point:

2.2.2. Heat balance between aluminum block andSpace of constant temperature reservoir
A derivative equation related into the time variation of( ) is given as follows:
Where

( )

=

( )−

( )+

{

( )−

( )}.

( ) is given as follows:
( )=

( )

+

( )

+

(8)

( )

(9)

Where, , , and is the time constant, the system gain,
and the time-delay. By using the least squares method,
the system parameters around= 20 are obtained as= 934, = 0.282, and = 60 , respectively. The
following approximated model can be obtained byapplying first order approximation to the time-delay:( ). (7)The( ) is the heat flow from the full color LEDs.
Thus, the( ) is given as follows by introducing the
maximum heat discharge of each color LED (red, green,
blue)
,
,
and command signals to eachLED
, ,(0 % ≤ u , , ≤ 100 %):
( )=

.

( )=

( )≅

.

( )=

( )−

(10)

Secondly, the following I-PD control law is introduced:
( )

−

( )

,

(11)

( ) = ( ) − ( ).
(12)Where, ( ) and ( ) are the controller output and thesystem output, respectively. In this simulation, ( ) =_ ( ) and ( ) =( ) . r ( ) is the reference
temperature. The following transfer function is
introduced as a reference model:.

(9)

The static property between ( ) and( ) of theobtained simulation model is shown in Fig. 3. The
figure shows that the system has nonlinearity.

( )=

.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

324

(13)

Shin Wakitani, Sneha Sharma, Toru YamamotoFig. 4. Control result obtained by fixed PID controller.

Fig. 6. Trajectories of PID gains corresponding to Fig. 5.

3.2. Control result by DD-PID controller
The control result in Fig. 4 and PID gains are stored into
a database, and a DD controller is designed according to
the reference1. The control result using the DD controller
is shown in Fig. 5. And trajectories of PID gains areshown in Fig. 6. In this simulation, the rise time is set toσ = 210, that is faster than the fixed I-PD controller
design. These results show that a good control
performance is obtained by adaptively tuning PID gains
based on the DD control technique.
4.
Fig. 5. Control result obtained by DD-PID controller.

In this work, a constant temperature reservoir that canmanipulate several environmental variables wasdeveloped to establish a smart cultivation system basedon the DD technique. The mathematical model based ona principal of the heat balance was derived and asimulator is constructed by MATLAB/Simulink. The DD
controller was also designed, and the effectiveness of thecontroller was verified by using the simulation model.
The implementation of the control system to the realsystem and designing the structure of smart cultivation
system are future works.

σ is the required rise-time of the closed loop system,
which is arbitrarily set by a user. The following
relationship can be obtained by comparing the transfer
function of the closed loop system, constructed by (10)and (11), and the reference model (13):
=

=

，
(

)

,

=

，
=

(

=

)

(14)
,

=

(

)

Conclusions(15)

The control result by fixed the I-PD controller is shownin Fig. 4. In this simulation, the rise time is set to σ =500, and the discrete I-PD controller whose samplingtime= 10 is applied. Moreover, the reference
temperature is given as follows:(0 ≤ < 3600)
20(3600 ≤ < 7200)
35
( )=
.
(16)
25 (7200 ≤ < 10800)
15 (10800 ≤ < 14400)
From Fig. 4, the controller output around ( ) = 35
oscillates due to the nonlinearity of the system, and thetracking property is deteriorating.

References
1. T.Yamamoto, K.Takao, and T.Yamada, Design of a DataDriven PID Controller, IEEE Trans. Control Systems
Technology, 17 (1) (2009) pp. 29-39.
2. G. Bontempi, M. Birattari, and H. Bersini, Lazy Learning
for Local Modeling and Control Design, International
Journal of Control, 72 (7-8) (1999), pp. 643-658.
3. H. Matsumori, S.Wakitani and M. Deng, Design andApplication of a Data-Driven Expert Controller based on
the Operating Data of a Skilled Worker, Journal of
Robotics and Mechatronics (JRM), 28 (5) (2016), pp. 730738.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

325

Design of a Performance-Adaptive 1-Parameter Tuning PID Controller
Yoichiro Ashida

Graduate School of Engineering, Hiroshima University,1-4-1 Kagamiyama
Higashihiroshima city, Hiroshima, Japan
Shin Wakitani

Graduate School of Engineering, Hiroshima University,1-4-1 Kagamiyama
Higashihiroshima city, Hiroshima, Japan
Toru Yamamoto

Graduate School of Engineering, Hiroshima University,1-4-1 Kagamiyama
Higashihiroshima city, Hiroshima, Japan
E-mail: ashida-yoichiro@hiroshima-u.ac.jp, wakitani@hiroshima-u.ac.jp, yama@hiroshima-u.ac.jp
www.hiroshima-u.ac.jp

Abstract
Many self-tuning PID controllers have been proposed. However, these methods are rarely employed to actual systems
because of low reliability of an on-line PID parameter estimator. In this work, a performance adaptive controller thathas two stage of PID parameters tuning mechanisms is proposed. One of the tuners is a one-parameter tuner using
the recursive least squares. The other tunes PID parameters using ordinary least squares. The effectiveness of the
proposed method is evaluated by an experimental example.
Keywords: PID control, data-oriented, self-tuning control, performance-driven, 1-parameter tuning.

employed. The reasons why PID controllers have been
widely applied include that: (1) PID controllers have
simple structure, and (2) physical meanings of the control
actions are clear. Therefore, some self-tuning PID control
schemes2 have been proposed.
In this work, a performance-adaptive controller thathas two stage of PID parameters tuning mechanisms is
proposed. One of the tuners is a one-parameter tuner
which calculates only the proportional gain using therecursive least squares. The other is a PID parameters
tuner using least squares. When the one-parameter tuner
cannot maintain a good control performance, the PID
parameters tuner determines new PID parameters tomaintain a desired control performance. The
effectiveness of the proposed method is evaluated by anexperimental example.

1. Introduction
Most of the controllers in industries have fixed control
parameters. However, the controllers cannot obtain good
control performances against time -variant systems. To
tackle this problem, self-tuning controllers1,2 have been
proposed. In these controllers, control parameters are
tuned adaptively and can maintain a good control
performance to time-variant systems. However, it isdifficult to employ these methods to actual systems
because of low reliability of an on-line estimator. In
addition, it is not preferred to tune many control
parameters simultaneously in terms of the safeness of aclosed-loop system.
Especially in process industries, proportionalintegral-derivative (PID) controllers3 have been widely© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

326

Yoichiro Ashida, Shin Wakitani, Toru Yamamoto

2. The tuning method of PID parameters= −2 exp −The PID parameters tuning method is described at thissection. A velocity-type PID controller in discrete time
domain is given as follows:
( )=

( )−

( )−

( ) ,

= exp −

+2

( − 1) +( − 2).

4 −1
2

⎫
⎪
⎪

(10)
.
⎬
⎪
≔
⎪
⎭
≔ 0.25(1 − ) + 0.51
and denote the rise-time and the damping coefficient,
respectively, andis adjusted by . When δ = 0 ,( ) indicates the Binomial model, and when = 1,( ) indicates the Butterworth model. In practice,
the value of is considered to design between 0 and 2,
and by using bigger , the response becomes oscillating.
The evaluation function is defined as follows:(1), , andare the proportional gain, thewhereintegral time and the derivative time, respectively.
denotes the differencing operator defined by ≔ 1 −, where z
is the shift operator which means( ) = ( − 1) . In addition, ( ) is the input
signal and ( ) is the output signal. Furthermore,denotes the sampling time, and ( ) denotes the output
error signal defined by ( ) ≔ ( ) − ( ) where ( )denotes the reference signal. (1) can be rewritten by
( )=
( )−
( )−
( ),
(2)
where
, , andare proportional, integral, andderivative gains, respectively. By assuming ≠ 0, (3)can be obtained by rewriting (2).
1+ +( )= ∆ ( )+
( )
−

2

cos

() ,

=

(11)

where denotes the number of data and ( ) is defined
as follows:( )≔ ( )−( ) ( ).
(12)
By minimizing the evaluation function , the following
relationship can be obtained:() ( ) → ( ).
(13)
From (4), parameters to be optimized are ( = 1, 2, 3).
When the minimization has been finished, it leads thefollowing relationship:( )→( ) ( ),
(14)
Therefore, the reference response is realized by using the
optimized . To apply a PID controller, PID parameters
are derived from as follows.
2 + −2=⎫⎪(15)= (2 + − 2)
.
⎬
1− −
⎪
=
2 + −2 ⎭(3)

Generalized output ( ) is defined as follows:( ) ≔ ∆u( ) + {y( ) − ( − 2)}(4)+ + { ( − 1) − ( − 2)} + ( − 2).
Where coefficients , , andare defined as follows:
1
≔
⎫
⎪
+ + ⎪
(5)
≔
.
⎬
+2
⎪
⎪
≔−
⎭
From (3) to (5), the following relationship can beobtained:( ) = ( ).
(6)In the proposed method, it is desired to make the system
output ( ) track the reference model output ( )which is defined by( )=( ) ( ),
(7)
(
)
(1)
( )≔
(8)
,
( )
Where denotes the time-delay of the controlled object,and is assumed to be known.
( ) is given as follows:( )=1++.
(9)
Where and are determined by following equations:

3. Design of a performance driven PID
controller
3.1. 1-parameter tuning

Both 1-parameter tuning and retuning of all PID
parameters are realized by using the PID tuning method
described at the previous section. In the 1-parametertuning, only overall gain of the PID controller is tuned.
From (15), is rewritten as follows:
=

.

(16)

Furthermore, andare constant values, andandare also constant values. Therefore, 1-parametertuning can be realized by tuning only . In this paper, a© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

327

Design of a performancerecursive least squares method4 is applied to minimize( ). Estimated parameter vector ( ) and data vector( ) are defined as follows:( ) ≔ ( ),
(17)
( )≔
( ) ( ),
(18)
at steps.
where ( ) denotes estimated value of
The recursive estimation algorithm with forgetting
factor is shown as follows:( ) = ( − 1) + ( ) ( )()[ { ( ) − ( − 2)} (19)( )= ( )−+ { ( − 1) − ( − 2)} + ( − 2)],( − 1) ( )
(20)
( )=
,
( ) + ( ) ( − 1) ( )(21)1[ ( − 1) − ( ) ( ) ( − 1)].
( )=( )3.2. Control performance evaluation

Fig. 1. Appearance of the experimental temperature control
system.

(22)

When 1-parameter tuning cannot maintain good control
performance, all PID parameters should be tuned. In thissubsection, the evaluation criterion of control
performance is described. A reference signal is assumed
to be the step-type signal. The criterion of control
performance is defined as follows:
( )=

−

1

+1

|

Fig. 2. Schematic figure of the experimental temperature
control system.

) ( ) − ( )| , (23)(The usefulness of the proposed self-tuning method is
experimentally evaluated on a pilot-scale temperature
control system. Fig. 1 shows an appearance of thecontrolled object, and the schematic figure is illustrated
in Fig. 2. In this experimental system, two heaters are
secured on a steel plate. These heaters work
synchronously, corresponding to the input signal fromthe computer. One thermo-couple is also prepared, andthe measured temperature of the steel is sent to thecomputer as the system output signal. The control
objective is to regulate the temperature of the steel to thedesired reference signal by manipulating the power of theheater. In this experimental result, the sampling timeis 1 second and the reference signal is changed alternatelyfrom 70 [℃] in the temperature and 100 [℃].
To simulate heater deterioration, the relationshipbetween controlled input ( ) and the voltage of thepower conditioner ( ) is varied as follows:wheredenotes a recent step when a reference signal is( ) denotes an average of errors betweenchanged.
controlled output and reference model output. All PID( ) satisfies theparameters are tuned when
following condition:( )> |

(

)|,

(24)

where α is a user-specified parameter which determines( ) havea threshold. | ( )| is introduced because
proportional relationship to the | ( )|.
3.3. A method of PID parameters tuning

In order to tune all PID parameters, the PID tuning
method described at Sec. 2 are used. The optimized
parameters are , , and . In addition, least squares
method is employed to minimize (k). In the ordinary
least squares, the data between
and steps are used.
4.

An experimental result© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

328

Yoichiro Ashida, Shin Wakitani, Toru Yamamoto

4( ) ( ≤ 200) ⎫100⎪⎪
4 − 3( − 200)
( )=1+( )(25).
100 ⋅ 600
(200 < ≤ 800) ⎬⎪1⎪( )=1+( ) ( < 800) ⎭100
The control result by using the proposed performanceadaptive controller is shown as Fig. 3, and the trajectories
of PID parameters corresponding to Fig. 3 is shown in
Fig. 4. In this result, the initial PID parameters were
determined by using the PID tuning method described atSec. 2 as follows:( )=1+(0) = 0.70, (0) = 28.9,(0) = 4.67.

150
100
50
0

0

200

400

0

200

400

600

800

1000

1200

600

800

1000

1200

100
50
0

(26)

Time [s]Fig. 3. Experimental control result by using the proposed
control method.

Parameters of the proposed method were set as follows:= 1, = 0.2, = 8, ( ) = 0.995, = 30 , and= 0 . From (26), initial estimated parameter wasdetermined as (0) = 41.3.
Fig. 3 indicates that the control performance gradually
deteriorated after 200 steps. Fig. 4 shows that only
was tuned before 863 steps to maintain good control
performance, and all PID parameters are tuned at 863
steps. As a result, good control result was obtained after
863 steps.

4
2
0
32

0

200

400

600

800

1000

1200

0

200

400

600

800

1000

1200

0

200

400

600

800

1000

1200

30
28
8
6

4.1. Conclusions

4

In this paper, a performance-adaptive 1-parameter tuning
PID controller is proposed, and effectiveness of the
proposed method is evaluated by the pilot-scaletemperature control system. The proposed performance
adaptive controller has two stage of PID parameters
tuning mechanisms. Features of the proposed method aresummarized as follows:
The proposed method is the 1-parameter tuning
method, and only is tuned.
All PID parameters are retuned only when control
performance deteriorates.
Both the 1-Parameter tuning and the retuning
method of all PID parameters are based on the same
PID tuning scheme.
From these characteristics, the proposed method can beeasily understood. However, the effectiveness of the 1parameter tuning in this paper does not proved
analytically. This is a future work.

Time [s]Fig. 4. Trajectories of the PID parameters corresponding to
the Fig. 3.

References
1. D. W. Clarke, and P. J. Gawthrop, Self-tuning controller,
Proceedings of the Institution of Electrical Engineers
122(9) (1975) 929–934.
2. Y. Ashida, S. Wakitani and T. Yamamoto, Design of an
implicit self-tuning pid controller based on the generalized
output, Proc. of the 20th IFAC World Congress (2017)
14511–14516.
3. K. J. Åström and T. Hägglund, Advanced PID Control,(International Society of Automation, North Carolina,2006).
4. E. B. M. Costa and G. L. O. Serra, Self-Tuning Robust
Fuzzy Controller Design Based on Multi-Objective
Particle Swarm Optimization Adaptation Mechanism,
Journal of Dynamic Systems, Measurement, and Control
139(7) (2017).
5. G. C. Goodwin and K. S. Sin, Adaptive Filtering
Prediction and Control, (Prentice Hall, New Jersey, 1984).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

329

Sticking Fault Detecting Method for CARIMA Model
Toyoaki Tanikawa

Advanced Course in Industrial and Systems Engineering, National Institute of Technology, Kagawa College
355 Chokushicho, Takamatsu, Kagawa, 761-8058 Japan
Tomohiro Henmi

Department of Electrical and Computer Engineering, National Institute of Technology, Kagawa College
355 Chokushicho, Takamatsu, Kagawa, 761-8058 Japan
E-mail: toyoakipd0@gmail.com, henmi@t.kagawa-nct.ac.jp
www.kagawa-nct.ac.jp

Abstract
This paper proposes a sticking fault detecting method for CARIMA model which detect the sticking fault of control
input and feedback signal. It consists of model estimation using recursive least square method with the forgetting
factor and fault detection. In the fault detection, an evaluation function is introduced, and it generates a fault signal
from the input and output data. Numerical simulations are performed, and it is shown that this method can detect the
sticking fault.
Keywords: fault detection, sticking fault, CARIMA model.

model. Then, we discuss its effectiveness performing a
numerical simulation.

1. Introduction
If a system with a fault continues to be operated, it cancause a serious accident or a considerable damage. Thus,
it is important to detect faults and compensate them, andmany fault detection methods have been proposed.1 The
advantage of fault detection is that the safety is improved,
you can cope with the fault more promptly, andsometimes the system can be controlled compensating
the fault. There are two kinds in fault detection, signalbased detection and model-based one. The signal-based
detection is for example a method using spectral analysis,
statistical signal analysis or pattern recognition, while themodel-based detection uses an observer or a parameter
estimation. In model-based detection, a general method
detecting additive faults is proposed in Ref. 2.
In detecting a fault where control input or feedback
signal is fixed, a general detection method is notestablished. In this paper, we propose a model-based
detection method for the sticking fault of control input
and feedback signal on a system expressed as a controlled
auto-regressive integrated moving average (CARIMA)

2. Sticking Fault Detecting Method
In this section, we discuss how to detect sticking fault ofcontrol input and feedback signal on the general linear
systems.
2.1. Problem Statement
First, we assume the control object is single input andsingle output system and expressed as following
CARIMA model:
[

]

[
[
[

( )=

]=1+
]= +
]= 1+

[

]

+
+
+

( )+ [

+ ⋯+
+ ⋯+
+ ⋯+

]

( )
(1)

( ) represents a system output,( ) is awherecontrol input, ( ) is a white Gaussian noise with 0 meanandis a time delay.
is a backshift operator where( ) = ( − 1), and ∆ is a difference operator© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan330

Toyoaki Tanikawa, Tomohiro Henmi

Controller

( )

( )

+

( )
SW1

on CARIMA model is obtained by the following
recursive algorithm:( ) Controlled

Object( − 1) ( )ϵ(k)( ) ( − 1) ( )( − 1) ( ) ( ) ( − 1)1
( )=( − 1) −( ) ( − 1) ( )+( ) = ( ) − ( ) ( − 1)( ) = ∆ ( ) − ( ) ( ).
(5)( ) = ( − 1) +SW2Parameter
Estimation
( )

Fault
Detection+

Fault Signal( )( ) is a priori error and ( ) is a posterior error. The
parameter vector ( ) and the data vector ( ) aredefined as follows:

Fig. 1. Block diagram of sticking fault detection

defined as ∆ = 1 −. System parameters, , andin Eq. (1) changes gradually or do not change at all. We
suppose the reference value of the control systemchanges depending on time.
Let us define the sticking fault. When the sticking
fault of control input occurs, the input to the system( ) becomes a white noise and expressed as follows:
( )=

(6)

(0) =

(7)

(normal)( ) (fault)

(2)

( )
(normal)+ ( )(fault)(3)where I is an identity matrix. is usually set as a large
value like 10 or 10 .
The weight of a past input-output data in costfunction ( ) becomes smaller as the time goes by due
to the forgetting factor. Thus, the estimated parameter( ) is calculated using resent data mainly. Thus, thisparameter estimation method is also effective for systems
where its parameters change.

( )+is a constant value and ( ) is a white
where
Gaussian noise with 0 mean.
Fig. 1 shows a block diagram of the fault detection
for the above-mentioned faults. This method consists of
two parts, model estimation and fault analysis.

2.3. Fault Detection
Fig. 2 shows the outline of the fault detection. To detect
sticking fault, it uses the estimated system parameters
and the input-output data of past
steps called detecting
period. The fault state is evaluated by the following
function:

2.2. Model Estimation
Least square method with the forgetting factor3 isused for the model estimation. In this method, the costfunction is given by:( )=( ) = [ ( ), ⋯ , ( ), ( ), ⋯ , ( ),
̂ ( ), ⋯ , ̂ ( )]( ) = [−∆ ( − 1), ⋯ , −∆ ( − ),∆ ( − ), ⋯ , ∆ ( −− ),( − 1), ⋯ , ( − )]where , and ̂ represent the estimated parameter of ,b and c in Eq. (1) respectively. The initial value of thecovariance matrix ( ) is given by:where ( ) is a control input calculated by the controller,
is a constant value and ( ) is a white Gaussian
noise with 0 mean. When the sticking fault of feedback
signal occurs, the feedback signal to the controller ( )becomes a white noise and expressed as:( )=+[ ( ) − ( | )]

( )=

(4)

∑

∑

[

] ( )−
[

] ()

[

+

] ()

(8)

where is an infinitesimal value to avoid dividing by 0.
[ ] and [ ] are the estimated system parameter
polynomials at step −−which are gained from( −−) calculated in Eq. (5), whereis orderof the system defined by= max{ , } + 1 . It iswhere is a system parameter vector, is a forgetting
factor where 0 < ≤ 1, ( | ) is an estimated output
is the number of the input-output data. The
andestimated parameter which minimizes the cost function© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan331

System Output ( )Sticking Fault Detecting Method

Order of System

On the other hand, after the sticking fault of the input
has occurred, the evaluation value at a moment describedin fig. 2 is deformed using Eq. (2) of the fault state andEq. (1), (3) and (10), and expressed as follows:

Detecting Period

( )
∑ −
=

fault
−

−

−

+1

Time

Fig. 2. Outline of fault detection

( )=

(9)

3. Sticking Fault Evaluation Value
We discuss the size of the evaluation value ( ) on thenormal state and the fault one. First, let us define the error[] andpolynomials of the system parameters,[ ] as follows:
]= [
]= [

]− [
]− [

]
].

∑ {∆
∑{

( )−
∆
∆ ( )+∆

( )+
( )+

(10)

( )}
( )}

∑ −

∑

( )+

()

()

.

( ),
( )>
( )
( ),
( )≤ ( )≤
(k) =
( ),
( )<
( )
( ) = Asin( ) +( ) = Asin( ) +Using Eq. (1), (2), (3) and (10) and ignoring theinfinitesimal value , evaluation function in Eq. (8) on
the normal state can be deformed as follows:
( )=

(12)

(13)

Because the terms of error polynomials and white
noises are smaller than that of ( ) as assumed before,( ) ≳ 1 is obtained in both Eq. (12) and (13). Therefore,
the evaluation value becomes greater after the sticking
fault occurs, and the fault can be detected by Eq. (9).
However, the thresholdhas to be set based on an
input-output data on an actual system or a simulation
results because the evaluation value ( ) is dependent
on the estimation error and disturbances.
Systems usually have limitations of the control input.
When designing a controller considering the limitation,higher and lower constant limits are generally used.
However, Eq. (11), (12) and (13) state that ( ) has to( ) can be 0 when the input ishave a value, andlimited by the constant value. Thus, the limitation has tobe dependent on time, and is given by:

where
is a threshold value set as> 1 . After
detecting the fault, it stops updating the estimatedparameters in Eq. (5).

[[( ) +∆( )+ ( )( ) + ( )}∆while the evaluation value on a sticking fault of the
feedback signal is deformed using Eq. (3) of the fault
state, and expressed as following equation:) is a good estimation of theassumed that ( −−actual system at step because we are focusing on
systems whose parameters do not change abruptly.
We assume a situation that a fault has occurred. The
estimated parameters become useless values after thefault. However, at a moment like Fig. 2, estimated−) is still valid and only theparameter ( −input-output data after the fault are used for theevaluation value (k). Then ( ) becomes bigger than
1 as being mentioned in section 3, and a fault signal isgenerated after satisfying the following inequality:
( )>

( )−
( )+

∑ {∆

( )
(14)

where
( ) is the control input calculated by thecontroller, and , , and are setting parameters.

(11)

4.

where the polynomial expression is abbreviated like
instead of [ ], and ∑ means ∑. Assumingthat the terms of error polynomials and that of white noise( ) are relatively small compared to the term of ( ) in
Eq. (11), ( ) ≲ 1 is gained.

Simulation Example

To verify the effectiveness of this fault detecting method,
we apply it to a system expressed as follows:[[] = 1 − 1.307+ 0.3768] = 0.05158 + 0.03729© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

332

(15)

Toyoaki Tanikawa, Tomohiro Henmi= 1 . We set the disturbance characteristics
wherepolynomials [ ] as [ ] = 1 . The systemparameter polynomials do not change throughout the( ) are set as a zerosimulation. ( ) , ( ) andmean white Gaussian noise with the variance of 0.012,
0.12 and 0.0052 respectively.
In the model estimation, we assume that the order ofthe system parameter polynomials ,and and theare known. Then, the estimated parametertime delay
vector ( ) is given by:
( )=

( ),

( ),

( ),

( ) .

References
1. T. Hukuda, Diagnosis on Machinery(Part 1) -The Outline
and an Evaluation of its Diagnostic Reliability-(inJapanese), in Journal of Japan Society for Safety
Engineering, vol.36, no.4 (1997), pp. 247-252.
2. R. Isermann, Model-based Fault-detection and Diagnosis– Status and Applications, IFAC Automatic Control in
Aerospace, Saint-Petersburg, Russia (2004), pp. 49-60.
3. L. Ljung and T. Söderström, Theory and Practice of
Recursive Identification (MIT Press, Cambridge, MA,
1983).

(16)

We set to 10 and to 0.98. In the fault detection,= 20,= 1.5 and = 10
are used.
Generalized Predictive control which uses CARIMA
model to predict the output is used as a controller. Whendesigning it, Eq. (14) is used for the internal model, and=1,=5,= 5 , ( ) = 0.01 , and time
constant of the reference trajectory = 0.3 s are taken.
For the input limitation, = 1 ,= 0.1 ,
=4
and = −4 are used.
Fig. 3 shows a simulation result where the sticking
fault of control input occurs at a step 400. ( ) represents
reference value and the dashed line on the graph ( ) is
the threshold value. The evaluate value ( ) is
normally smaller than 1, and it becomes greater after thefault occurs. It surpasses
at a step 406, and then thefault signal is generated. ( ) takes relatively greatvalue on the beginning due to the error of the parameter
estimation.
Fig. 4 shows a result of the sticking fault of feedback
signal. Like Fig. 3, ( ) becomes greater after the fault,
and the fault signal is generated at a step 404.

Fig. 3. Detection result on control input sticking fault

5. Conclusion
We proposed the sticking fault detecting method in which
the evaluation function is introduced. It evaluates the
input-output data on a detecting period, using the pastestimated parameter calculated by the recursive leastsquare method. Performing a numerical simulation, it isconfirmed that the sticking fault of control input andfeedback signal can be detected.
Acknowledgements
Fig. 4. Detection result on feedback sticking fault

This work was supported by JSPS KANKENHI Grant
Number 15K21591.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan333

Design of Neural Network PID Controller Based on E-FRIT
and Online Learning
Kento Kinoshita*

Graduate School of Engineering, Hiroshima University, 1-4-1 Kagamiyama
Higashihiroshima city, Hiroshima, Japan
Shuichi Ohno

Graduate School of Engineering, Hiroshima University, 1-4-1 Kagamiyama
Higashihiroshima city, Hiroshima, Japan
Shin Wakitani

Graduate School of Engineering, Hiroshima University, 1-4-1 Kagamiyama
Higashihiroshima city, Hiroshima, Japan
E-mail: kinoshita-kento@hiroshima-u.ac.jp, ohno@hiroshima-u.ac.jp, wakitani@hiroshima-u.ac.jp
https://www.hiroshima-u.ac.jp/

Abstract
PID controllers have been widely used in industrial world. When a controlled object has a nonlinear characteristic, agood control result is not always obtained with fixed PID gains. To overcome the problem, a design method of anonlinear PID controller using a neural network has been proposed. In this paper, an online learning method of the
proposed controller to attenuate the effect of the system change of the controlled object has been presented.
Keywords: PID control, E-FRIT, neural network, data-driven, online learning.

The FRIT evaluates only the control response. Hence,
the FRIT is insufficient for controlling an object whose
stability is important.4 The E-FRIT (Extended Fictitious
Reference Iterative Tuning) has been developed toenhance the stability of the control system.4 The E-FRIT
considers variations of the manipulated variable in its
criterion. An algorithm for learning a database used in the
DD control based on the E-FRIT has been proposed.5
However, DD control requires a large amount of memory
and high processing speed to calculate its control
parameters, because DD control uses a database. This
paper proposes a design method of a nonlinear PID
controller which combines a neural network and the EFRIT. In addition, we also presented an online learning
method of the designed controller. The proposed

1. Introduction
The proportional-integral-derivertive (PID) control
has a long history and has been a main control method in
the industrial world. The fictitious reference iterative
tuning (FRIT) has been presented to determine PID
gains.1 By the FRIT, PID gains are tuned directly from a
set of operating data. However, when a controlled object
has a nonlinear characteristic, a good control result may
not be obtained with the fixed PID gains. Therefore,
methods that utilize the FRIT and effective control
methods for nonlinear systems, such as a data-driven(DD) control and a neural network control have been
proposed.2,3© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan334

Kento Kinoshita, Shuichi Ohno, Shin Wakitani

Neural Network Controller
Neural Network
r(t)

+

e(t)

-

PID Controller

Z-1
u(t)

SystemZ-1y(t)Fig. 2. Schematic diagram of a neuron.
Fig. 1. Block diagram of a neural network PID control system.

controller makes the closed loop system which has
desired response with low computational costs.
Moreover, the controller is adequately updated by the
online learning method to mitigate the effect of systemchange.

Fig. 3. Illustration of PID gain tuning.

1. Neural Network PID Controller
sigmoid function as the activation function because thesigmoid function is often used in practice. The standard
sigmoid function is given by
1
( )=
.
(5)
1+

Let us consider discrete-time systems. A neuralnetwork PID control system is illustrated by the block
diagram in Fig. 1. The neural network in Fig. 1 has one
hidden layer. At time t, the neural network calculates PID
gains from ( ), ( ), ( − 1) and ( − 2) which are
the reference and system output, respectively. The PID
controller controls the system with the calculated PID
gains.
Let z be a unit-time delay operator, that is,( ) = ( − 1),(1)and define the difference operator ∆ as
∆=1−
.
(2)
In this paper, the PID controller has velocity type PID
control law given by∆ ( ) = ( ) ( ) − ( )∆ ( ) − ∆ ( ), (3)where y(t) is the controlled variable at time t.
Moreover, ( ), ( )and( ) are the proportional
gain, the integral gain and the derivative gain at time t,
respectively.
A neural network consists of many neurons connectedeach other. Fig. 2 shows a schematic diagram of a neuron.
A neuron has multiple inputs and one output. If the output
is and the inputs are , , ⋯ , , the input-output
relation is described by=+,

The calculation of PID gains by the neural network
PID tuner is illustrated in Fig. 3. The outputs of neuralnetwork are in the range (0,1) because the range of Eq.
(5) is (0,1). Hence, the output of neural network are
scaled by constant when desired PID gains are larger
than 1.
2.

Controller Design

A schematic diagram of the E-FRIT is depicted in Fig.
4. The E-FRIT is a direct control parameter tuning
method by using operating data.4 The parameter tuning
by the E-FRIT does not require any mathematical modelsof controlled objects. In Fig. 4, ( ,) is a controller
consists of a control parameter vector . In the PID
controller, is,and. In addition, ( ) and( ) are the output and manipulated variables of theoperating data, respectively. The relation between the
input to a controller and the output from the controller isdenoted by
( ,
)
(6)
( )=
̃( , ) − ( ) ,∆where ̃ ( , ) is a fictitious reference signal which is
given by( ,)∆ ( ) + ( ).
̃( , ) =(7)(4)

where the variables (1 ≤ i ≤ ) are called synaptic
weights and the variable is a bias. The function (⋅) isthe so-called activation function. We use the standard© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

335

Design of Neural Network

The output ( ) is the response of the reference modelto ( ). In Eq. (14), variance is calculated by all control
result until time . The synaptic weight is updated todecrease the criterion by backpropagation. The update
rule of the synaptic weights is given by Adam.6 Adam isa stochastic gradient descent which has high convergence
speed.

Fig. 4. Block diagram of the E-FRIT.

We set( ) as our desired system and obtain( , ) which is the output of reference model()
for ̃ ( , ). Additionally, a difference of the manipulated
variable ∆ ( , ) is given by) ̃( , ) − ( , ) .
∆ ( , )= ( ,(8)Then, the parameters in are given by minimizing a( ) which is defined as
criterion
( )=

1

+

( + 1) −( , + 1)( )∆ ( , ) ,where N is the number of sample data andscaling parameter which is expressed asy ( )−

( )=

( , )

∆ ( , )

3.

To begin, the controlled object is discussed. We consider
the controlled object whose property are changed in thesimulation. Two Hammerstein model are denoted by thefollowing equations:[System 1]( ) = 0.6 ( − 1) − 0.1 ( − 2)
+1.2 ( − 1) − 0.1 ( − 2) + ( ) , (15)( ) = 1.5 ( ) − 1.5 ( ) + 1.5 ( )(9)( ) is a.

(10)( ⋅) which means variance.
with
The design method of the neural network control
system is based on the E-FRIT. In our neural network
controller, parameters in are PID gains which are tuned
by a neural network. Therefore, represents ( , )when the synaptic weight of the neural network is .
From the above, we define a criterion such as
( )=
where

1

+

( + 1) −( ( , ), + 1)

( )∆ ( ( , ), ) ,, (11)( )=( , ) .
(12)
is updated to decrease the criterion bysynaptic weight
backpropagation.
Our criterion of the online learning method is1( , )=( )− ( , )2+ ̅ ̅ ( )∆ ( , − 1) ,

(13)

where

̅( ) =y ( )− ( , )∆ ( , ).

Simulation Example(14)[System 2]( ) = 0.6 ( − 1) − 0.1 ( − 2)
+1.2 ( − 1) − 0.1 ( − 2) + ( ), (16)( ) = 1.6 ( ) − 2.0 ( ) + 3.0 ( )where ( ) is white Gaussian noise, whose arithmeticalmean is 0 and variance is 1.0 × 10 . The sample time isset as 1s. Moreover, the reference is set as follows:
1.0 ( 0 ≤ ≤ 50 )
2.0 ( 51 < ≤ 100)
( )=
.
(17)
2.5 (101 < ≤ 150)
1.0 (151 ≤ ≤ 200)
We assume the controlled object is System1 when thelearning data is obtained. We also assume the controlled
object is change from System 1 to System 2 at 30 s, when
the controlled object is controlled by the proposed
controller. In addition, we set the reference model( ) to0.2368()=. (18)
1 − 1.0268+ 0.2636
We set parameters of our neural network and thelearning method are set as follows: The number ofneurons in hidden layer of the neural network is set as 7.
The parameter and ̅ in the criteria is set to 1.
The learning data is obtained by a fixed PID controller.
The PID gains of the controller is set as=0.2502, = 0.0779,= 0.1137. The control resultof the fixed PID method is shown in Fig. 5. The neuralnetwork PID controller is designed using the data
depicted in Fig. 5. It approximates the response of the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

336

KD

u0

KI

y

KP

Kento Kinoshita, Shuichi Ohno, Shin WakitaniFig. 5. Control result by the fixed PID controller (System 1).

Fig. 8. Trajectories of PID gains corresponding to Fig. 7.

yimmediately after the system change occurs, the control
result follows the desired response gradually. From theabove, we conclude proposed method is effective.
5. ConclusionuThis paper proposed a design method of a neural network
PID controller based on the E-FRIT, whose performanceis evaluated by simulations. The output of the controlled
object by the proposed controller is close to the output ofthe desired reference model, even if the system changes.
This work was partly supported by JSPS KAKENHI
Grant Number JP16H02921.

Fig. 6. Control result without the online learning (Controlled
object is change from System 1 to System 2 at 30 s).

References
1. S. Soma, O. Kaneko, T. Fujii, A New Approach to
Parameter Tuning of Controllers by Using One-Shot
Experimental Data - A Proposal of Fictitious Reference
Iterative Tuning ，Transactions of the Institute of Systems,
Control and Information Engineers 17(12) (2004) 528536, (in Japanese).
2. K. Takao, T. Yamamoto, T. Hinamoto, A Design ofMemory-Based PID Controllers, Transactions of theSociety of Instrument and Control Engineers 40(9) (2004)
895-905, (in Japanese).
3. Y. Ohnishi, H. Kitagawa, S. Mori, S. Wakitani, and T.
Yamamoto, Design of Neural Networks Based FRIT PID
Controllers and Its Applications, 11th IFAC Workshop on
Adaptation and Learning in Control and Signal
Processing, (2013).
4. K. Tasaka, K. Manabu, M. Ogawa, S. Masuda, and T.
Yamamoto, Direct PID Tuning from Closed-Loop Dataand Its Application to Unstable Processes, Trans. of ISCIE
22(4) (2009) 137-144, (in Japanese).
5. S. Wakitani, M. Deng, A New Learning Algorithm for
Data-driven PID Controller Based on E-FRIT ， IEEJ
Transactions on Electronics, Information and Systems,
136(5) (2016) 683-689, (in Japanese).
6. D. P. Kingma, J. Ba, Adam: A Method for Stochastic
Optimization, Conference paper at ICLR 2015.

Fig. 7. Control result by the proposed method (Controlled
object is change from System 1 to System 2 at 30 s).

closed loop to the response of the reference model when
the controlled object is not changed from System1.
The control result by the designed controller without
the online learning is shown in Fig. 6. From Fig.6, it isclear that the output diverges.
The control result by the designed controller isdemonstrated in Fig. 7. The trajectories of PID gains arepresented in Fig. 8. Although the control result oscillates© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

337

Parameter Optimization with Input/Output Data via DE
for Adaptive Control System with Neural Network
Taro Takagi

Department of Control Engineering, Natl. Inst. of Tech., Maizuru College,
234 Shiroya, Maizuru, Kyoto 625-8511, Japan
Ikuro Mizumoto

Graduate School of Science and Technology, Kumamoto University,
2-39-1 Kurokami Chuo-ku, Kumamoto 860-8555, Japan
E-mail: t.takagi@maizuru-ct.ac.jp, ikuro@kumamoto-u.ac.jp

Abstract
In this paper, adaptive control system with neural network (NN) will be designed. At the beginning, parallel
feedforward compensator (PFC) will be designed by using one-shot experimental data of controlled system via
differential evolution (DE). From the obtained PFC and the ideal almost strictly positive real (ASPR) model, nominalmodel of controlled system can be obtained. Then, parameters of adjust law for NN will be optimized by using
obtained nominal model via DE.
Keywords: Adaptive Control, ASPR, PFC, Neural network, Differential evolution.

if the conditions are satisfied. However, those parameters
are directly affect to control performance. Therefore, it
has to be appropriate value in actual control systems.
Since there are no decision method for those parameters,
designer has been decided it by trial and error ofnumerical simulations. Therefore, it is quite difficult todecide appropriate value.
In this paper, the parameter optimization method of
PFC and NN by using input/output data via differential
evolution (DE) will be proposed. DE is one of theevolutionary computation methods and it has a good
convergence and uniqueness of solution8.

1. Introduction
In spite of their simple structure, Almost strictly positive
real (ASPR) based adaptive control systems have highrobustness with respect to disturbances and uncertainties1,
2. However, the ASPR conditions which controlled
system has to satisfy are very severe conditions for actual
systems. To overcome this problem, introduction ofparallel feedforward compensator (PFC) has been
proposed3. Several design methods of PFC which are
based on mathematical model of controlled system suchas relative degree have been proposed4. However, toobtain the mathematical model, iterative of identification
experiments have to be done and it cost time and money.
Therefore, a design method which is using system’s
input/output data has been proposed5, 6. For ASPR system,an adaptive control system with feedforward input
generated by neural network (NN) has been proposed7.
This input is applied to avoid tracking error caused by
PFC. In theory, PFC and NN parameters can be any value

2. Adaptive Control System with Adaptive NN
Feedforward Input
Consider the controlled system is described in equation
state model as follows.

.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

338

(1)

Taro Takagi, Ikuro Mizumoto

For this system, a PFC described as follows is introduced.
(2).
Then, supposed that the augmented system:

(3)
where
,

Fig. 2. Block diagram of adaptive control system with NN
feedforward input.

,,
is ASPR. Fig. 1. shows the block diagram of augmented
system. Then, for this augmented system, design anadaptive output feedback control system with adaptive
NN as follows.

Here, the problems are how to design the PFC andhow to decide the parameters of adjusting laws.
3. Design of PFC by Input/Output Data
Define the ideal output of the ideal augmented system byusing the input data as follows.
(9).
Then, the ideal output of PFC can be described by using
the output of controlled system as follows.

(4)

(10)
.
Here, suppose that the ideal PFC given by the following:

where
(5)

(11)

(6)
and

then the ideal output of PFC also can be given as

is the output of PFC obtained by(7)

By introducing a stable filter:.
Also,
is the estimated value of
ideal weight vector of NN and l is the number of NN
nodes. In addition,is aradial basis function vector generally designed by theGaussian functions such as

(12)

.

,

Eq. (4) can be represented as

(13)

(14)
By defining the ideal parameters as

(8)

(15)

where is the center of receptive field and is the widthof Gaussian function. Fig. 2. shows the block diagram ofadaptive control system which is shown in this section.

and the filtering signal as(16)then the ideal output of PFC can be represented as(17).
From eq. (9), for any parameters, the output of PFC canbe given as

Fig. 1. Block diagram of augmented system.

.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

339

(18)

Parameter Optimization with Input/Output

Finally, consider minimizing the following evaluation
function.

Select the next generation as follows.
(24)

(19)
.
Although the order of PFC and parameters of filter affectto the output of PFC, there are no theorical method todecide these parameters. Therefore, it will be optimized
by DE later. Remark that the ideal output of PFC is
known but parameters of ideal PFC and filter are
unknown.

.

The shown DE algorithm is so-called DE/best/1/bin
strategy9.
5. Control System Design via Differential
Evolution
Here, concrete designing method of control system via
DE will be shown.
(i) PFC design(a) Obtain the input/output data set of controlled
system.
(b) Creation of initial population
Randomly create an initial N sets of population
as follows

4. Differential Evolution
In this section, the basic algorithm of DE which is appliedin this study will be shown.
(i) Creation of initial population
Randomly create an initial N sets of population
as follows.

(25)

(20)

.
The superscript means a generation number ofthe vector, and D is the order of decision variables.
(ii) Extraction of base vector
Extract the best vector from nth generation
vectorwhich minimizing the evaluation
function. This is, set the base vector as follows when
the lth target vector minimize the evaluation function.
.

(c) Extraction of base vector
Extract the best vector from nth generation
vector as Eq. (21) which minimizing theevaluation function Eq. (19) by using obtained
input/output data.
(d) Mutation
Generate a mutated vector from randomly
chosen two vectors as Eq. (22).
(e) Crossover
Generate a trial vector from the target vector
and the mutated vector as Eq. (23).
(f) Selection
Select the next generation as Eq. (24).
(ii) Adaptive controller design(a) Obtain an approximation model of controlled
systemFrom ASPR modeland obtained PFC
in above step as(21)(iii) Mutation
Generate a mutated vector from randomly
chosen two vectors
asfollows.
(22).
Here, F is a scaling factor for scaling thedifference vector.
(iv) Crossover
Generate a trial vector from the target vector andthe mutated vector as follows.

(26)(b) Creation of initial population
Randomly create an initial N sets of population
as follows

(23)

(27)

.
Here, C is a crossover rate, randi,j is a random
uniform numbers in the range [0, 1] and jr is arandomly chosen integer number in the range [1, D].
(v) Selection(c) Extraction of base vector
Extract the best vector from nth generation
vector as Eq. (21) which minimizing theevaluation function as follows.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

340

Taro Takagi, Ikuro Mizumoto

.

(28)

(d) Mutation
Generate a mutated vector from randomly
chosen two vectors as Eq. (22).
(e) Crossover
Generate a trial vector from the target vector
and the mutated vector as Eq. (23).
(f) Selection
Select the next generation as Eq. (24).

Fig. 3. Control input/output data.

6. Numerical Simulation
Assume that input/output data shown in Fig. 3 are
obtained. From these data, by setting the parameters F=
0.8 and C=0.9 with ASPR model

third order PFC was obtained as

Fig. 4. Control result by initial parameters.

,.
Then the adaptive controller parameters were optimized.
The initial value of design parameters wereThen the optimized parameters were
Fig. 5. Control result by optimized parameters.

Fig.4 and Fig. 5 show the control result by initial
adaptive controller parameters and optimized parametersrespectively. It can be seen that the control performance
became better after optimizing the parameters.

4. Z. Iwai, I. Mizumoto and H. Ohtsuka, Simple Adaptive
Control SAC, (Morikita Publishing, 2008)
5. I. Mizumoto and T. Takagi, PFC design via FRIT
approach and adaptive PID control system design for
discrete-time systems, In proc. of IFAC Conference on
Advances in PID control, (2012) 64–69.
6. T. Takagi and I. Mizumoto, Adaptive output feedback
control with adaptive PFC and its stability analysis, ICIC
Express Letters, Part B: Applications, 6(12) (2015) 31893195.
7. I. Mizumoto, H. Tanaka and T. Tokimatsu, Adaptive
output regulation of a class of discrete-time nonlinear
systems based on output feedback and NN feedforward
control, In proc. of 49th IEEE Conference on Decision andControl, (2010) 3631-3636.
8. M. Ito and M. Tanaka, A comparative study of particle
swarm optimization, differential evolution, real-coded
genetic algorithm on function optimization, Konan
University Bulletin. Science and Technology, ed., 52(1)(2005) 125-135. (in Japanese)
9. G. C. Onwubolu, B. V. Babu, New Optimization
Techniques in Engineering (Springer Science & BusinessMedia, 2004).

7. Conclusions
In this paper, the parameter optimization method whichusing input/output data via DE was proposed. The
numerical simulation shows the usefulness of the method.
References
1. I. Mizumoto, S. Ohdaira and Z. Iwai, Output feedback
strictly passivity of discrete-time nonlinear systems andadaptive control system design with a PFC, Automatica,
46(9) (2010) 1503-1509.
2. T. Takagi and I. Mizumoto, Adaptive PID control systemwith adaptive NN feedforward for discrete-time systems,
IEEJ Transactions on Electronics, Information and
Systems, 134(9) (2014) 1175-1181 (in Japanese)
3. H. Kaufman, I. Barkana and K. Sobel, Direct Adaptive
Control Algorithms 2nd Ed., (Springer-Verlag New York,1997)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

341

Mathematical Expression of Minakata Kumagusu’s philosophy of Natural Science
Yasuhiro Suzuki

Graduate School of Informatics Nagoya, University, Furocho Chikusa
Nagoya City, 464 0814, Japan

Abstract
We interpret the mandalic epistemology of 南方熊楠 (Minakata Kumagusu) using Category theory and show thatit can be interpreted by set theory. Based on this result, we give philosophical foundation of Abstract Rewriting
System on Multisets, ARMS, a hybrid theoretical computational model, which synthesizes continuous and discrete
computation, based on Theoretical Chemistry and Computer Science.
Keywords: Natural Computing, Natural Philosophy, Philosophy of Computation, Minakata Kumagusu

1. Introduction
南 方 熊 楠 (Minamata Kumagusu , 1867-1941) is anaturalist, with research themes ranging fromhumanities, cultural anthropology, astronomy, biology,
bacteriology and others (he published 51 papers on
Nature magazine only).
Kumagusu stated scientific theory and scientificphilosophy in a round-trip with Horyu Togi (土宣法竜),
a high priest in the Buddhist world at the time. We callhis scientific theory as Kumagusu Mandala [1].
His mandalic epistemology consists of MONO, (物)corresponding to physics, KOKORO, ( 心 )corresponding to truth, and KOTO, (事) equivalent torecognition of things by mind. His idea can be clearlyshown using Category theory. Also, considering using
Category theory, we can complement the ambiguouspart of his thought. In this paper, we use only the basic
definition of Category theory (for example [2]) so no
advanced knowledge is required.

2. Elements of the Mandalic Epistemology

The text is to be typeset in 10 pt Times Roman, single
spaced with interline spacing of 13 pt. Text area
(excluding running title) is 6.75 inches across and 8.8
inches deep. Final pagination and insertion of running
titles will be done by the publisher, so make sure that nopage numbers are given in your paper and only therunning titles provided in this template (authors’ names
and paper title) are used.
His epistemology is composed of four basic elements,
心(KOKORO), 物(MONO), 事(KOTO), 名(NA) and an
extra element 印(IN).
心 is the collection of arrows and defined as Hom(物,事)=心, where “arrow” means operations such as
map or function and Hom(A, B) is a collection of allarrows from A to B; 物 can include empty element.
When the operation of 心 (KOKORO) disappears 事(KOTO)disappears, while 物 (物) does not disappear. If© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan342

Yasuhiro Suzuki(collection of) same 事(KOTO) appear repeatedly, 名(NA) emerges. 名 (NA) is a kind of 物(MONO) hence
if a 事 (KOTO), which produces the 名 (NA)disappeared, 名 (NA) can be remain as a set of 物
(MONO). Kumagusu did not describe the mechanismof emergence of NA(名) and gave examples of NA(名)as language or religion. So we can derive it naturally
from the examples of NA (名); Language or religion
have “ some structure ” inherent; if a language iscomposed by completely random KOTO( 事 ), such“ language ” would not be able to convey any
information. Language or religion must have somestructures; such structure is NA(名). KOTOs (事) that
produce NA(名) are generated and extinct repeatedly,that is because the MOMOs (物) that produce the NA( 名 ) through KOTO ( 事 ) will continue to exist.
Therefore, NA(名) can be regarded as a collection ofsuch MONOs(物 s).
Kumagusu argued that KOTO(事) and KOKORO(心)have internal interaction (it is called CHIKARA(力), itmeans “ force ” in English), we will denote
CHIKARA (力) interaction as CHI(x).
We will assume that KOTO(事) and KOKORO(心)are categories. Hence the definition of Category requiresthat CHI must satisfy the association law in compositionof CHI as CHIj(CHIi(MONO(物) (or KOKORO(心))))= CHIl(MONO( 物 ) (or KOKORO( 心 ))). When
KOTO( 事 ) and KOKORO( 心 ) possesses thisassociation law in composition, we will call it isconserved. Hence, if MONO(物) and KOKORO (心)are conserved, we can define KOTO(事) as Cartesian
product of MONO(物) and KOKORO(心) as follows;

and MONO. Because NA( 名 ) is obtained from
MONOs(物 s), we can regard the collection of MONOs
as NA( 名 ). IN( 印 ) is obtained from projections of
KOKORO(心)to NA(名), since MONO(物)is category,IN( 印 )also can be regarded as Cartesian product ofNA( 名 )and KOKORO( 心 ); then we can obtainfollowing proposition from the definition of Category
theory.
Proposition 1. When Categories of MONO( 物 ) and
KOKORO(心) are conserved and NA(名) exist, IN(印)is uniquely obtained.
Kumagusu did mention about MONO( 物 ),KOKORO(心), KOTO(事), NA(名), but did not about
IN( 印 ). Mathematically, IN( 印 )will be naturally
obtained from other four elements and can be ignored.
3. Isomorphism of 物 and 名
When 物(MONO) and 事(KOTO) share the same 名(NA), 物(MONO) and 名(NA) are isomorphic.
We define the equivalence of 物 (MONO) as follows:when Hom(物, 事) = G and Hom(事, 名)=H, if whenf ∘ h＝g ∘ h＝NA(名), f,g∈G, h∈H, then f=g, 物 1:
from f(物 1)=g(物 2), 物 1=物 2, which corresponds toMono morphism in Category.
And also we define the equivalence of KOTO(事) as
follows: when Hom(物, 事)=H, Hom(事, 名) = G, if
when H ∘ g＝H∘ f＝NA(名), f,g∈G then f=g, 事 1:

,where X is KOTO(事) and A and B are respectively
MONO(物) and KOKORO(心), f1 and f2 are called as
Projections and these projections index X as the KOTO

© The 2015 International Conference on Artificial Life and Robotics (ICAROB 2015), Jan. 10-12, Oita, Japan

343
1

Mathematical Expression of Minakata

from f(事 1)=g(事 2), 事 1=事 2, which corresponds to
Epi morphism in Category.
Hence, from the definition of Category theory we canhave the corollary;

References
1. Minakata Kumagusu, Toki Horyu, Round trip letters,
Yasaka Syobou, 1990.
2. Michael Barr, Charles Wells, Category Theory for
Computing Science, Prentice Hall, 1990

Corollary. Hom( 物 , 名 ) and Hom( 名 , 物 ) are bimorphism.
So, from this corollary we obtain proposition 2, which is
obtained straight forwardly from the corollary,definition of isomorphism in Category;
Proposition 2. If and only if 物 and 名 are category,
Hom(物, 名) is isomorphism.

Mathematically, in the mandalic epistemology, 名(NA)is representative element and 物 (MONO) is element,
while 心(KOKORO) and 事(KOTO) are recognitions ofmatters. Based on this epistemology, in order tointroduce numeric concept, we will expand thisframework by using multisets, which allows duplicateof the same representative elements. Hence, if we havefollowing arrows among 物, 事 and 名,;if we take the circle as 名, then 物 1, 2, and 3 are thesame, the rectangular 物 1 and 3 are the same. It is easy
to check with refereeing the definitions.
Acknowledgement
This research has supported by Grant in Aid for Scientific
Research (KAKENHI) No. 15K00268, 16H03093 and
15K02105.

© The 2015 International Conference on Artificial Life and Robotics (ICAROB 2015), Jan. 10-12, Oita, Japan344

Toward Artificial Intelligence by using DNA molecules
Yasuhiro Suzuki

Graduate School of Informatics, Nagoya University, Furocho
Chikusa, Nagoya city 464-0814, Japan
Rie Taniguchi

Graduate School of Informatics, Nagoya University, Furocho
Chikusa, Nagoya city 464-0814, Japan
E-mail: ysuzuki@nagoya-u.jp
http://ysuzuki.info

Abstract
Molecule Robotics have been realized by using bio molecules such as DNA, proteins, for example in Molecular
Robotics Research Project [2012-17, JSPS]; bio molecules, such as DNA or proteins are highly structured and they
already have a kind of “intelligence” and adapt environment change. Hence, we have tried to extract their “ability”
and induce Intelligence artificially. We have used well-known DNA reactions, Seesaw gate reaction and we foundthat DNA molecule can senses concentration of single strand input sequence or quasi-input (mismatch, point
mutation, including sequence) and chooses higher concentration one; this result shows DNA molecule (short DNA
sequence) can adapt its environment.
Keywords: DNA computing, Molecular Robotics, Strand Displacement Reactions

the membrane and the shape of body to be changed.
Gears of clutch is changed by a signal moleculecomposed of sequence-designed DNA.

1. Introduction

Adelman solved Hamilton Path Problem by using acomputer, which was synthesized by DNA molecules.
2. Molecular Artificial Intelligence
His research opens up novel era of molecular computing.
DNA computing has evolved as a method of Bonino
Artificial Intelligence has developed on electrical device,technology and realized a molecular robot.
where every information is coded as electrical signals.
For example, S. Nomura’s team has chemicallyOn the other hand, in a molecular robot every
implemented a molecular robot [1]. This robot is
information is coded as chemicals such as DNA orcomposed of bio molecules, which is composed of aproteins. Hence, if we can realize Artificial Intelligencebody, an actuator, and an actuator-controlling device
by using chemicals, we will be able to give intelligence(clutch); whose body is composed of a lipid bilayer, theto the molecular robots.
actuator consists of proteins, kinesin, and microtubules
and the clutch is made of designed DNA molecules.
This clutch transmits motor proteins generated force to© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan345

Yasuhiro Suzuki, Rie Tanicughimigration process exchanges a single strand DNA withdouble strand DNA through random-walk like; thecomplementary single-stranded domains is called as
toeholds. In this reaction, at least one single stranded
nucleic is released and this released strand can evoke
downstream reactions. Since branch migration does notchange the free energy during the reaction of
exchanging strand pairs, it can be described by arandom walk process with equal transitions probabilities
of displacement in both directions at each position of
the branch point [5].

2.1. Abstract Chemical Reactions by DNA
Several logical gates have realized by using DNA[quian]. Various simulations of Abstract Chemical
Reactions, ACR based on Chemical kinetic have been
made by using the DNA logical gates [3]. For
example, ???et. al. chemically implemented the LotokaVorterra, LV model and showed that their reaction canrealize the same behavior to analytic solution of LV
modeled by differential equations.
Quian et.al., have proposed collection of DNA
reactions as a “tool box” to implement ACR by using
DNA; they showed that an Artificial Neural Network
can be implemented by using the tool box [3].
3. Intelligence of Molecular Robot
Philosopher, prof. Kazuhisa Todayama has argued apossible development of philosophy as Concept
Engineering. Concept engineering provides novel
concept based on thinker’s assets. For example, in thecase when we have to face with difficult problems suchas create a regulation for robots in order to ready for acase when a fully autonomous intelligent robot kill ahuman. In such a case, philosophers can provide wide
varieties of concepts in different aspects and they can
support to create a novel concept through collaboratingwith specialists and citizens. Todayama points out that
Human right is a legacy of concept engineering, which
has invented by philosophers based on the asset ofphilosophy[2,3].
3.1. How to create intelligence in molecular robots

Fig.1. Strand displacement reaction; In this example, the shortstrand, which is denoted as an arrow binds to the toehold
domains, through branch migration process across region 2,
the strand is displaced and freed from the complex.

Todayama proposed how to create intelligence in amolecular robot from concept engineering view point; if
we let the molecular robot free as possible as we canand if the robot can have own purpose, intelligence tobe emerged, where any purpose is granted and no matter
how it is simple or not, it is important to have its own
purpose. In order to create molecular AI, we give a
purpose to a robot, which is maintaining the self.
4.

In order to implement molecular AI, we modified the
Seesaw gate reaction [4]; since this reaction is selfmaintaining reaction, we can give the reactions a
purpose as self-maintaining. This reactions arecomposed of basically two strand displacement
reactions; input + gate:fuel ↔ gete:input + fuel (1) andgate:input + output ↔ gete:output + input (2); in order
to confirm the reactions circulate, we add the reaction,
fuel + gate:marker gate:fuel + marker (3).

Methods

Chemical reactions in the toolbox are mainly
modification of the toehold strand displacement reactionof DNA. Strand displacement reaction is initiated froma complementary single-stranded domains and branch© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan346

Toward Artificial Intelligence by

Fig. 2. Seesaw gate reaction: (a) Abstract gate diagram. Red
numbers indicate initial concentrations. (b) The DNA gate
motif and reaction mechanism. S1, S2, S3 and S4 are therecognition domains; T is the toehold domain; T0 is the
Watson–Crick complement of T, etc. Arrowheads mark the 30
ends of strands. Signal strands are named by their domains
from 30 to 50, All reactions are reversible and unbiased; solidlines indicate the dominant flows for the initial concentrations
shown in (a), while the reverse reactions are dotted. (c)5.

Fig. 3. Time series of concentration of Input sequences
obtained by the real time PCR: top) input sequence in bulk
water only, below) Input sequence without toehold andgate:fuel, the concentration of Input sequences decreased
through binding to gate:fuel sequences.

Resultrom biochemical experiments, we observed that when
we added non-mutated input and mutated input, systemselect higher concentration one (Fig.3). And we also
confirmed behaviors in the time series of concentrationof normal input and mutated input sequences showedoscillations (Fig.3). And when the concentration
decreases, the concentration of another input increases,
hence we conjecture that the because of system selecthigher concentration one, selected input sequences are
switched among non-mutated and mutated input
sequences. These results illustrate that the system madeof DNA reaction networks are able to select input
sequences in higher concentration so it will able toadapt the environmental change. Therefore, from
Biological and philosophical points of views, thissystem realizes molecular Artificial Intelligence (Fig.3).

Acknowledgement
I am deeply grateful to prof. Kasuhisa Todayama, Nagoya
University. This research has supported by Grant in Aid for
Scientific Research (KAKENHI) No. 15K00268, 16H03093
and 15K02105.

References
1. Y. Sato, et.al., Micrometer-sized molecular robot changes
its shape in response to signal molecules Science
Robotics 01 Mar 2017: Vol. 2, Issue 4, eaal3735
2. R. G. Millican, Varieties of Meaning, (MIT press, 2002).
3. K. Todayama, Tetsugaku Nyumon (in Japanese,
introduction to philosophy, Chikuma shobo publishing,2014)
4. L.Qian and E. Winfree, A simple DNA gate motif forsynthesizing large-scale circuits, J R Soc Interface. 2011
Sep 7; 8(62): 1281–1297.
5. Luis P. Reynaldo, el.al., The Kinetics of Oligonucleotide
Replacements, J. Mol. Biol. (2000) 297, 511±520© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan

347

Differential and Integration of Sensation and its Application
Yasuhiro Suzuki

Graduate School of Informatics, Nagoya University, Furocho Chikusa
Nagoya City, 464-0841, Japan

Abstract
Our sense can be seen as a dynamical system in multi-dimensional sensory space, where each dimension corresponds
to the sort of sense such as visual, audio, tactile, and so on. Our sensory systems are composed of differentiators andwe perceive sensory stimulations through differentiating sensory stimulations. In order to calculate differentiation ofsense, we have to set measurable quantity of the sense, we can use not only quantity measure such as brightness ofpicture, loudness of sound but also can use quality measure such as value of Semantic Differential (SD) method formeasuring impressions or emotion such as beautiful, fear, happy. We propose a method of Differential and Integrationof sensation and we show the case when transforming a drawing of Piet Mondrian’s “1921” into a music piece byusing this method.
Keywords: Computational Aesthetics, Tactile Communication, Tactile Score, Kansei Computing

hand, there is no such a combinational principle fortactile sense.
We communicate through tactile sense in the form ofshaking hands to express your trust, hugging a grieving
friend instead of speaking to him or her, and stroking cutedogs and cats. These communications directly make an
appeal to the sensitivity and feeling, and we tactilely
grasp the innermost truth of who someone is from theway of using the language and behavior.
Tactile sense has been of interest to basic science, suchas psychology, psychophysics, cognitive science, andrecently it has also been of interest to engineering anddesign. In engineering, technologies of tactile sense havebeen developed in virtual reality, e.g., robotics andergonomics. One of the main issues they have addressed
is “how to regenerate the tactile sense mechanically” andthe necessary and desirable applications of suchtechnologies have been explored. Reviewing the large
amount of research on tactile engineering should
preferably be included elsewhere, in another book. In thepast, tactile engineering has mainly been applied toentertainment, such as the integration of a tactile sensing

1. Introduction
To see shapeless or unseen things and to hear unheard
things have been challenges in Arts and Sciences.
Visualizing shapeless things such as light, air, emotions,etc. or unseen things such as concepts, atmosphere, etc.
have been challenges in visual Art and Science. For
example, Leonard da Vinci, who is an artist, engineer,
biologist and a pioneer of data science, drawn anormalized human anatomical chart based on dissecting
many human dead bodies.
Tactile sense directly makes an appeal to the deep
feeling without language or words [3]. Tactile sense hasa language however the way of using the language is
obscure. If we could have full command of the languageof tactile sense, and express the feeling and tell it by using
the language, cold media would be changing into
intimate organic ones.
Visual media are systematized with colors and their
combinational principles such as color circles, and music[1] in auditory media is systematized with combinational
principles of theories of harmonics [2, 5]. On the other© The 2015 International Conference on Artificial Life and Robotics (ICAROB 2015), Jan. 10-12, Oita, Japan348

Yasuhiro Suzuki

device into a video game controller, and tocommunication technology, such as in mobile phones.

Fig.1 Spectral pitch, the horizontal axis illustrates time and the

2. Calculus of Senses
Our sense can be seen as a dynamical system in multidimensional sensory space, where each dimension
corresponds to the sort of sense such as visual, audio,
tactile, and so on. Our sensory systems are composed ofdifferentiators and we perceive sensory stimulations
through differentiating sensory stimulations.
In order to calculate differentiation of sense, we have to
set measurable quantity of the sense, we can use not only
quantity measure such as brightness of picture, loudnessof sound but also can use quality measure such as valueof Semantic Differential, SD method for measuring
impressions or emotion such as beautiful, fear, happy.
Major headings should be typeset in boldface with thefirst letter of important words capitalized.
The most important concept of the differentiation ofsense is measuring the differences in time or / and spaceof sense; it shares the concept of differentiation inmathematics but this is not mathematical concept and amathematical equation of sense is not required; incalculating differentiation of sense, we use differentiablemeasure. Algorithm of differentiation of sense is asfollows;
I. Differentiation of Sense1
select a sense to calculate differentiation,
2
set the width of differentiation,3
measure the differences by the time width (2.);
3.1(if it is required) measure of the differencesof 3.
For example, we will differentiate a music piece;
1.
2.
3.

vertical, frequencies that correspond to pitches of music notes.
D, D, G, G and duration of each note is the same as 0.5 second.
and by setting the duration of quarter note is 0.5, we obtain themusic score of D, D, G, G;, this differentiation of sound corresponds to drawn the musicpiece in musical notation.

II. Transform Sense1.
select a differentiated sense and the sense totransform (e.g. the volume of sound into thebrightness of movie)
2.
transform the sense and reconcile time width eachother,
Transform sensory such as audio sensory to visual has
been developed well, for example, the sound-on-filmrecording, which is the most prevalent method ofrecording analog sound on a film print; in short,transform the volume of sound (audio sense) intobrightness (visual sense) of a film and synchronize thesound and film. For example, by using this method wetransform a picture into a music score; we differentiate a
picture of Piet Mondrian, 1921 and transform into amusic score.

a sense to calculate differentiation = the frequencies
of sound,the time width of differentiation = 0.5 second,the differences by the time width (2.);

1.

2.

then we differentiation the sound measure and obtain the
spectral pitch, which is the amount of vibration at eachindividual frequency and it corresponds to pitch of musicnote (Fig. 1).

the differentiated sense = the area of the same color
grid square (square centimeter), space width = thearea of a grid square
the sense to transform = musical pitches, reconcile
space width;

we coarse grained the area as more than 90 squarecentimeters = whole note, between 90 and 45 = half note,
between 45 and 22.5 = quarter note, between 22.5 and11.25 = eighth note, between 11.25 and 5.625 = 16th note,
between 5.625 and 2.8125 = 32th note and less than

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan

349

Differential and Integration of Sensation

2.8125 = 64th note, respectively; and we are assigned
color and musical pitch as; black as C, blue, D, red E,
yellow G and white, A by borrowing Major Pentatonic
Scale (Table 1).

engineering that we have been developing for more than
two thousand years. For this reason, we chose toinvestigate massage as a way to devise a method of
describing the tactile sense [4]. Through ourinvestigations, we found massage to be composed ofpressure, an area of contact or touching, and the velocityof the movement of massager’s hands. We modeled ourtactile description on the musical score (Fig.3). In staff
notation of the tactile note, we defined the third line as
the basic pressure—the basic pressure with which wemight hold a baby or delicate, breakable object. Hence,
the basic pressure is not defined absolutely but may
change from person to person or for different types ofmassage.

Table 1, Area of grid square of Mondrian 1921, in squarecentimeter; 8.1 is black, 24.1 and 31.0, red, 58.5 yellow, 93.6
and 47.5 are blue, 1.5 is black and all others are white

III. Integral of Sense
And also once a sense is differentiated we can calculate
integral of sense as follows;
1. select the differentiated sense (time width was
defined),2. define the interval of calculating integral.
3. calculate an area of sense at each time width andcalculate the sum of areas in the interval.
Then we obtained a music score of Mondrian
1921(Fig.2);

Fig.3: (Top) An example of a tactile score, with special marking
above the staff notation. 1 denotes both hands moving the same;
2 indicates a small gap between hands; and 3 indicates a large
gap between hands. The slur-like marks illustrate a unit
component, a massage phrase; the integral-like marks illustratereleasing pressure; and the breath-like marks correspond to ashort pause in massaging, much like a breath in playing music

We also defined the part of the hand used in massage
and the kind of stroke used when massaging (see Fig. 4).
For example, the fingertip to the first joint is 1, the second
joint is 2, the third joint is 3, the upper part of the palm is
4, the center of the palm is 5, and the bottom of the palmis 6. Thus, a flowing motion from a fingertip to the third
joint is denoted as “1–3.”
For massage strokes, we analyzed the method of
massaging in face therapy and the stroking action of thehands’ movements. For example, A5 illustrates drawing
a circle on the cheek with the center of the palm.
In the tactile score (Fig.3), at the first count in the
beginning part, A5 circles are drawn on both sides of thecheeks using the center of the palm with weaker pressure
than the basic pressure; at the second count, the hands are
moved to the tails of the eyes and a small circle is drawn
using the center of the palm while keeping the samepressure as the first count; and, at the third and fourthcounts, the hands are moved to both sides of the cheeks

Fig 2. From the music score, which is transformed
P.Mondirian’s 1921;

3. Tactile Score

Massage, which is composed of various tactile senses of
movements of hands on the body, is a form of haptic© The 2015 International Conference on Artificial Life and Robotics (ICAROB 2015), Jan. 10-12, Oita, Japan350

Yasuhiro Suzuki

and circles are drawn using the fingertips with a stronger
pressure than the basic pressure.

Fig. 4 Tactile score transformed from the noise in a cabin ofJapanese bullet train (Shin-kan-sen train).

Acknowledgements
This research has supported by Grant in Aid for Scientific
Research (KAKENHI) No. 15K00268, 16H03093 and
15K02105.

References

Fig. 3: (Top) Strokes of massaging on a face. These strokes are
obtained from massage experiences in beauty shops. Strokes
that pass uncomfortable areas have been excluded. (Bottom)
Usage of part of the hand.

References are to be listed in the order cited in the text.
Use the style shown in the following examples. For
journal names, use the standard abbreviations. Typeset
references in 9 pt Times Roman.

3.1. Transform sounds to Tactile Score

1. J. Huang, D. Gamble, K. Sarnlertsophon, X. Wang, S.
Hsiao Feeling Music: Integration of Auditory and Tactile
Inputs in Musical Meter Perception, PLOS ONE Vol.7(10),Oct 2010
2. S. Merchel, M. E. Altinsony, Designing auditory-tactile
perception of music, Proceedings of ICMEM 2015, 2015
3. L. Skedung, M. Arvidsson, J. Young Chung, C. M.
Stafford, B. Berglund, M. W. Rutland, Feeling Small:
Exploring the Tactile Perception Limits, SCIENTIFICREPORTS Sep 2013
4. Y.Suzuki and R. Suzuki (2013) Tactile Score, Springer
Verlag5. J. Ward, Acquired Auditory-Tactile Synesthesia, Annalsof Neurology Vol.62, No.5 , Nov, 2007

According to the method of differentiate sense in thesection 2; we select a sense to calculate differentiation ispressure (step 1), the width of differentiation is the samethe range of pressures of sound (step 2) and measuring
the differences by the time width is the same as therhythm of sound (step 3).
Tactile score denotes pressure, contact area and rhythm.
In this example, we will denote pressure and rhythm,
where the contact area is defined as frequency of sound;
we assume that lower frequency has broad contact areaand higher frequency has narrow contact area so we takethe reciprocals of frequencies and multiply by amplitudes.
We show an example when the noise in the cabin ofJapanese bullet train (Shin-kan-sen train, Fig.4).
Fig.4. Tactile score, which is transformed from the noise of
cabin in Japanese bullet train (Shin-kan-sen train).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan

351

Discovering Successful Determinants of Efficiency of MICHINOEKI in Chugoku Area
Minoru Kumano

Faculty of Regional Innovation, University of Miyazaki
1-1 Gakuendai-Nishi, Miyazaki, 889-2192 Japan
E-mail:kumano@cc.miyazaki-u.ac.jp
Tsutomu Ito, Takao Ito

Graduate School of Engineering, Hiroshima University
1-4-1 Kagamiyama, Higashi Hiroshima, 739-7527
Toru Hiraoka

Department of Information Systems, University of Nagasaki
1-1-1, Manabino Nagayo, Nishisonogi, Nagasaki, 815-2195, Japan
E-mail: hiraoka@sun.ac.jp
Hirofumi Nonaka

Department of Information Management System Engineering, Nagaoka University of Technology
1603-1, Kamitomioka, Nagaoka, Niigata, 940-2188, Japan
E-mail: nonaka@kjs.nagaokaut.ac.jp
Masahara Hirota

Department of Information Science, Faculty of Informatics, Okayama University of Science
1-1 Ridaicho, Kita-ku, Okayama-shi, Okayama, 700-0005, Japan
E-mail: hirota@mis.ous.ac.jp

Abstract
Many theories and analyses of Michinoeki from the viewpoints of break function, information dispatch function, andregional cooperation function have been published in the past decades. Based on our investigation, additional
dimensions called disaster prevention function need to be added because a variety of activities have been developed
recently. Specific to the context of Michinoeki systems, this paper develops and empirically tests a mathematicalmodel of Michinoeki from the standpoint of how those four basic functions significantly influence its performance,thus uniquely contributing to extant knowledge. Using data drawn from Michinoeki in Chugoku area, this researchattempts to shed light on the relationship between four basic functions and its performance including sales revenue
and number of customers who passed through the shop cashier, to confirm the validity of the new four function model.
Based on the findings, the managerial implications are discussed, the study limitations are identified and directions
for further research are suggested.
Keywords: four-function model, regional economy, performance, Michinoeki, efficiency

function, and regional cooperation function have been
published in the past decades. Based on our investigation,
additional dimensions called disaster prevention functionneed to be added because a variety of activities have been

1. Introduction
Many theories and analyses of Michinoeki from theviewpoints of break function, information dispatch© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan352

Minoru Kumano, Tsutomu Ito, Takao Ito, Toru Hiraoka, Hirofumi Nonaka,

developed recently. Specific to the context of Michinoeki
systems, this paper develops and empirically tests amathematical model of Michinoeki from the standpointof how those four basic functions significantly influenceits performance, thus uniquely contributing to extant
knowledge. Using data drawn from Michinoeki in
Chugoku area, this research attempts to shed light on therelationship between four basic functions and its
performance including sales revenue and number ofcustomers who passed through the shop cashier, toconfirm the validity of the new four function model.
This paper is structured as follows: Section 2 introduces
the history of the Michinoeki and background literature.
In section 3, the paper explicates data collection and theestimation of the regression model using specific internaland external variables. Section 4 shows the results anddiscusses our findings. The conclusions, directions forfurther research and managerial implications areproffered in the final section..

Tsunami decimated Fukushima and other towns.
Therefore, the four-function model of Michinoeki could
be illustrated in Fig. 1.

Fig. 1 Four-function model of Michinoeki
This paper is try to discover successful determinants ofefficiency of Michinoeki in Chugoku area, Japan basedon the four-function model.
3. Data Collection and Model Building
In this paper, we selected ten determinants as thevariables to express the four functions in Michinoeki.
They are square meters of land space (SMLS), weekdays’
traffic near the station (WTNS), operating cost (OC),total parking number (TPN), total number of the restroom(TNR), square meters of free rest place (SMFRP), seatsof the free rest space (SFRS), number of items handled(NIH), number of local products items (NLPI), andpopulation of the city located (PCL). SMLS and OC
could be considered as the determinants of variables of
information dispatch function, WTNS, NIH, NLPI andPCL are designed as the indexes of regional cooperation
function, TPN, TNR, SMFRP and SFRS will play
important role in break function and disaster preventionfunction. Sales revenue and number of purchasers are
selected as the performance of Michinoeki. Thus, thisleads to the formation of the following regression
equation:

2. Background
Regional areas development plays important role in thenational economy. Storper stressed that regions are thecornerstones of the global economic organization—justas firms, sectors and nations do. Thus, innovative
solutions need to be developed to foster regional
economic revitalization [1]. Cooke and Morgan [2]indicate that the ability of a region to cope withcompetitiveness challenges increasingly depends on theextent of collaboration between its firms, and the abilityof regional governments in supporting such collaboration.
Michinoeki roadside rest stations for those driving
automobiles and/or trucks to deliver products andservices is a special case of a successful model of an
innovative ecosystem. These stations have beendeveloped under the support of residents, farms, and local
government. They are located along major national
highways and provide free parking space, restrooms, andregional and tourist information for automobile drivers
and travelers. Therefore, Michinoeki within a specificarea serve the three functions of providing rest space forvisitors, spreading information, and allying with regionalsociety. Recently, disaster prevention is has also become
an important consideration of Michinoeki after the 3.11

= 1
+ 2
+ 3
+ 4
+
+
+
SFRP+
+
5
6
8
7
+ 10
+
9

(1)

The data were extracted from the internal databases andprovided by senior administrators of Michinoeki
headquarters.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan353

Discovering Successful Determinants ofTable 1. Results of the regression models of sales and number of the purchasers.
ModelsSales
Standardized coefficient
SMLS
WTNS
OC
TPN
TNR
SMFRP
SFRS
NIH
NLPI
PCL
Intercept-0.1378-0.0204
0.06240.27310.07590.1788-0.075-0.10190.48780.22280

Coefficient of determination
multiple correlation coefficient
Adjusted R-square
Data number

Standardized Coefficients

2
1
0
OC

Tottori

ShimaneTPN

TNR SMFRP SFRS

NIH

NLPI

PCL

-2
-3
Okayama

Hiroshima

Yamaguchi

Fig. 2 Results of Sales revenue model.
Standardized Coefficients

2
1.5
1
0.5
0
-0.5

SMLS WTNSOCTPNTNR SMFRP SFRS

NIH

NLPI

PCL

-1.5
-2
Shimane

Okayama

Hiroshima

Yamaguchi

Fig. 3. Results of number of purchaser’s model
4.

0.14630.67390.47130.01980.26960.00010.78770.59240.01330.02770.76360.505290.71084
0.65230
73

The relationship between ten variables and sales and number ofpurchasers are calculated and discussed. Only part of thedeterminants such as SMLS and TPN have impacts on theperformance of Michinoeki. Only one-year Data set is sufficient.
Much more data set and investigation is required.

-1

Tottori

Probability-0.1988
0.0429-0.09320.30530.15590.4364-0.0304-0.0846
0.39
0.22940

In the regression model of number of purchasers, only
four parameters are significant: SMLS (Shimane), TPN(Okayama), SMFRP and NLPI (Yamaguchi). In the
regression model of sales revenue, Okayama’s TPN,SMFRP, and NIH, Yamaguchi’s TPN, NLPI, and PCL
are significant. Therefore, square meters of land space
total parking number, square meters of free rest place,
number of local products items, and population of the city
located could be considered as the important
determinants in the management of Michinoeki.
Moreover, break function and regional cooperation
function works in Okayama and Yamaguchi while
information dispatch function plays role in Yamaguchi.
5. Conclusion

3

SMLS WTNS

0.44050.87240.68510.07740.662
0.15680.5776
0.57850.00830.06720.7033Standardized coefficient

0.34596
0.58818
0.4867871

4-1

Number of the Purchasers
Probability

References

Calculation and Discussion

1. Successful examples of regional revitalization, http://www.
kantei.go.jp/jp/singi/sousei/meeting/top_seminar/h29-0113-haifu4.pdf#search=%27%E5%9C%B0%E6%96%B
9%E5%89%B5%E7%94%9F+%E4%BA%8B%E4%BE%8B%27, retried April 20, 2017.
2. Satoyama (2003), The Traditional Rural Landscape ofJapan, Takeuchi, et al. eds., Springer.

We first calculated the relationship between the ten
variables and sales revenue and the Number of thepurchasers. The results is shown as in Table 1. Next, we
calculated standardized coefficients of sales revenue andnumber of purchasers regression model respectively of eachprefecture. They are reported in Fig. 2 and 3.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

354

Relationship Analysis on the Number of Customers of Michinoeki in Kyushu Region
Toru Hiraoka, Shiori Nishimura

Department of Information Systems, University of Nagasaki,
1-1-1, Manabino Nagayo, Nishisonogi, Nagasaki, 815-2195, Japan
E-mail: hiraoka@sun.ac.jp, bs116025@sun.ac.jp
Hirofumi Nonaka

Department of Information and Management System Engineering, Nagaoka University of Technology,1603-1, Kamitomioka, Nagaoka, Niigata, 940-2188, Japan
E-mail: nonaka@kjs.nagaokaut.ac.jp
Minoru Kumano

Faculty of Regional Innovation, University of Miyazaki,
1-1, Gakuendai-nishi, Miyazaki, 889-2194, Japan
E-mail: kumano@cc.miyazaki-u.ac.jp

Abstract
We identify factors related to the number of customers at roadside stations (Michinoeki) in the Kyushu region, andanalyze the relation. We also consider the number of customers and the availability of various facilities and services
at Michinoekies, and analyze the relation. We pick up the following factors: land area of all facility, area of the retailfacility, parking capacity, traffic volume, number of products sold at the retail facility, total project cost, andpopulation of the local municipality. By correlating these factors and the number of customers, we find factors thatare related to the number of customers. We also pick up 44 types of facilities and services, and analyze the correlation
between the existence of these facilities and services. In addition we perform multiple regression analysis andquantification method I to estimate the number of customers from these factors or the existence of these facilities andservices, respectively.
Keywords: Michinoeki, roadside station, number of customer, relationship analysis, Kyushu region

usual roles. Sales levels are high at many Michinoekies,
but maintenance costs associated with age are becoming
a burden for some facilities. Future maintenance andgrowth of Michinoekies will require tax-derived
resources from MLIT and municipalities as well as
increased profits and higher sales amounts at
Michinoekies. To increase sales, Michinoekies must
increase the number of their customers.
In this situation, the purpose of this paper is to
identify factors related to the number of customers at
Michinoekies and analyze the relation. The availabilityof various facilities and services at Michinoekies is also
considered, and the relation between the number of

1. Introduction
Michinoekies in Japan are roadside stations that provide
information, offer facilities for resting, and promote local
cooperation in order to contribute to a safe andcomfortable road environment and create local activities1.
Municipalities build Michinoekies and register them with
the Road Bureau of the Ministry of Land, Infrastructure,
Transport and Tourism (MLIT). The first 103 locations
were registered in 1993, and 1,117 locations wereregistered as of April 21, 2017. Recently, Michinoekies
have begun playing fulfilling roles, such as promoting
disaster prevention2 and tourism3, in addition to their© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

355

Toru Hiraoka, Shiori Nishimura, Hirofumi Nonaka, Minoru Kumano

customers and the existence of facilities and services isanalyzed.
This paper focuses on the following factors related to
the number of customers at Michinoekies: land area of all
facilities, the area of the retail facility, parking capacity,
traffic volume, number of products sold at the retailfacility, total project cost, and population of the local
municipality. Factors that are strongly associated with thenumber of customers are identified through correlation
analysis. In addition, multiple regression analysis isperformed to estimate the number of customers based ona set of factors, and the goodness of the multiple
regression equation for each set of factors is examined.
This paper considers 44 types of facilities andservices provided at Michinoekies that include free rest
area and tourist information booth (see Table 1). For eachof these 44 types of facilities and services, the correlation
with the number of customers is analyzed to identify
which types of facilities and services are associated with
the number of customers. Quantification method I is used
to estimate the number of customers on the basis of thecorrelated facility and service types.
This paper identifies the factors that significantly
impact the number of customers. This information will
be useful in modifying the layout of existing
Michinoekies and in selecting appropriate sites for new
Michinoekies. This paper also examines the types offacilities and services that should be prioritized fordevelopment at Michinoekies to increase the number ofcustomers. This analysis considers 128 Michinoekies in
the Kyushu region, using the number of customers who
made a payment at a cashier counter in FY 2015.
Although many customers report using only a restroom

or vending machine, counting such customers is difficult.
The main data used in this paper are from FY 2015 andwere collected through a September 2016 questionnaire
survey of municipalities that administer Michinoekies.
2. Analysis of the Relation between the Numberof Customers and Various Factors
2.1. Data preparation
Data were collected for 128 Michinoekies in the Kyushu
region. For each Michinoeki, data on the number ofcustomers, land area of all facilities, area of the retailfacility, parking capacity, traffic volume, the number of
products sold at the retail facility, total project cost, andpopulation of the local municipality were collected. The
data on traffic volume and the municipal population wereobtained from the FY 2010 National Survey of the Roads
and Traffic and the FY 2010 National Census,
respectively. Sales data and other data are from FY 2014
and FY 2015, respectively, collected through theSeptember 2016 questionnaire survey of municipalities
that administer Michinoekies.
For the land area of all facilities and the area of theretail facility, total floor area is used, not site area. The
area of the retail facility refers to the area of the space
devoted to sale of local products (agricultural products,
wood products, fishery products, etc.); the number of
products sold at the retail facility refers to the number ofdistinct products sold in the space. Parking capacity ismeasured in the number of vehicles that can be parked
simultaneously. On a parking spot for a large, heavy
vehicle (e.g., truck, bus) is treated as equivalent to two
spots for ordinary vehicles. Traffic volume is measured
in the number of vehicles per day. Total project cost is
the initial cost of constructing Michinoekies. Complete
data were obtained for 111 Michinoekies, and thefollowing analysis is performed for these 111 locations.

Table 1. Facilities and services of Michinoekies collected inthis research.
Free rest area, Shower, Tourist information boot, Post, ATM,
Baby cot, Space for selling local products, Restaurant, Cafe,
Accommodation, Hot spring, Camp site, Car camping place,
Marina, Park, Observatory, Plaza, Athletic field, Cultural
facilities (Theater, Museum, etc.), Conference room, Local
product processing plant, Experience farm, Experience
workshop, Gas station, Hospital, Experience and exhibition
facility of disaster prevention, Function of the government
office, Nursing room, Powder room, Charging stations forelectric vehicles, Highway bus, Bus, Demand transportation,
Tourist bus, Tourist information staff, Services related totourist programs such as an agricultural experience program,
Delivery service to the elderly and others, information on
traffic congestion, Accident information, Information on
traffic regulation, Homepage, Blog, Twitter, Facebook

2.2. Method and result of correlation analysis
Correlation analysis was performed for the number ofcustomers at Michinoekies against a set of factors. Table
2 shows the calculated correlation coefficients. The
variables in the table are listed in order of descending
correlation coefficient.
Parking capacity has the strongest correlation with
the number of customers; the area of the retail facility
and traffic volume have weaker correlations; total project© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

356

Relationship Analysis on the

Table 2. Correlation coefficients between the number ofcustomers and a set of factors.
Factor name
Correlation coefficients
No.
1
Parking capacity
0.525
2
The area of the retail facility
0.460
3Traffic volume
0.428
4
Total project cost
0.204
5
The number of products sold at the retail facility
0.156
6Population of the local municipality-0.122
7
Land area of all facilities-0.111

Table 3. Absolute residual mean and multiple correlation
coefficient according to the usage factors in multiple regression
analysis.
Use factor
Absolute residual Multiple correlation
Case (No. in Table2)
mean
coefficient
100,4320.525
1
191,2440.645
2
1, 2
83,2630.692
31, 2, 384,1990.704
41, 2, 3, 4
82,4770.708
51, 2, 3, 4, 5
78,6510.742
6
1, 2, 3, 4, 5, 6
79,0670.742
71, 2, 3, 4, 5, 6, 7

cost, the number of products sold at the retail facility,
population of the local municipality, and land area of theentire facilities have relatively little correlation.
Increasing the number of vehicles that can be parked andexpanding the space for selling local products, instead ofsimply increasing land area of all facilities, is thus
expected to be effective in increasing the number ofcustomers. Building Michinoekies on high-traffic roads,
rather than on the basis of municipal population, is also
seen to be effective. Since the correlation coefficient
between total project cost and the number of customers
is not high, Michinoeki projects may need restructuring.
However, the appropriateness of such restructuring mustbe judged comprehensively, with consideration given to
the fact that Mitinoekies serve multiple purposes: they
offer information, provide facilities for resting, andpromote tourism, local cooperation, and disasterprevention. The relatively low correlation between thenumber of customers and the number of local products
sold at the retail space suggests a need for proper productselection.

absolute values of the differences between the estimatednumber of customers and the observed number ofcustomers (the absolute residual mean) is calculated.
Table 3 shows the results.
The absolute residual mean is smallest in Case 6, but
Cases 3-7 differ only a small mount, and so parking
capacity, the area of the retail facility, and traffic volume
are sufficient as explanatory variables for estimating thenumber of customers by multiple regression. Also, sinceadding total project cost or the area of the entire facilities
as an explanatory variable leads to an increase in theabsolute residual mean, these variables may as well be
excluded in estimating the effect on the number ofcustomers. The use of such a multiple regression
equation makes it possible to estimate the effect on thenumber of customers of a change in parking capacity or
the area of the retail facility at an existing Michinoeki.
Such estimation is also possible during the process of
selecting the appropriate site or designing facilities for aplanned Michinoeki.

2.3. Method and result of multiple regression
analysis

3. Analysis of the Relation between the Numberof Customers and the Existence of Specific
Facilities and Services

Multiple regression analysis is performed with thenumber of customers as the explained variable and thefactors as explanatory variables. The explanatory
variables are added to the regression equation one by one(see Table 1 for order of entry), starting with parking
capacity which had the strongest correlation with thenumber of customers. Case 1 is the model with only No.
1 in Table 2 included, Case 2 also includes No. 2, and soon, through Case 7. In each case, the regression equation
is estimated; the estimated number of customers ispredicted for each of the 111 Michinoekies from theincluded the explanatory variables; and the mean of the

3.1. Data preparation
Data were collected for 128 Michinoekies in the Kyushu
region. This paper considers the 44 types of facilities andservices listed in Table 1. The data are for FY 2015 andwere collected through a questionnaire survey ofmunicipalities administering Michinoekies, which was
conducted in September 2016. There are no missing
observation points for the 44 variables, but the number ofcustomers were collected for only 123 stations. This setof 123 Michinoekies was used in the following analysis.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

357

Toru Hiraoka, Shiori Nishimura, Hirofumi Nonaka, Minoru Kumano

Table 4. Correlation coefficient with the number of customers
and the number of installations of facilities and services.
Name of facilities and services The number of Correlation
installations
coefficient48
0.406
Information on traffic congestion69
0.277
Charging stations for electric vehicles
8
0.271
ATM
72
0.230
Cafe
Services related to tourist programs suchas an agricultural experience program

12

Table 5. Absolute residual mean and multiple correlation
coefficient according to the usage facilities and services inquantification method I.
Number of facilities Absolute residual Multiple correlationand services usedmean
coefficient
131,2840.052
1
126,5520.284
2
118,4200.409
3114,1310.447
4113,273
0.457
5

0.220

3.2. Method and result of correlation analysis

5 shows the mean differences between the number ofcustomers (the absolute residual mean), as well as thecorresponding multiple correlation coefficient.
For cases in which five or more facility and service
indicator variables are included, differences in theabsolute residual mean and the multiple correlation
coefficient are small. It therefore would suffice to include
four indicator variables (information on traffic
congestion, charging stations for electric vehicles, ATM,and cafe), when estimating the number of customers via
quantification method I. That is, the use of thisquantification method I makes it possible to predict theeffect of introducing these facilities and services atMichioekies.

This section shows correlation analysis of the relation
between the number of customers at Michinoekies andthe existence of certain facilities and services at thatlocation. Each facility and service was dummy coded,with value 1 if it exists and 0 otherwise. Table 4 shows
the correlation coefficients between the number ofcustomers and the facility and service indicator variables(Top 5 with large correlation coefficient). Missing values
for two variables (marina and space for selling local
products) reflected data-related irregularities: noMichinoeki has marina, and the number of customers
could not be obtained for the one Michinoeki without
space for selling local products.
Five variables have a relatively correlation (acorrelation coefficient with 0.2 or larger): information on
traffic congestion (0.406), charging stations for electricvehicles (0.277), ATM (automated teller machines)(0.271), cafe (0.230), and services related to tourist
programs such as an agricultural experience program(0.220). The number (percentage) of Michinoekies
offering these five facilities and services is 48 (37.5%),
69 (53.9%), 8 (6.3%), 72 (56.3%), and 12 (9.4%),
respectively. This result implies that there are many
Michinoekies that do not offer these facilities andservices. In summary, Michinoekies should prioritizethese five facilities and services when aiming to improve
service for their users and increase the number ofcustomers.

4.

Conclusion

This paper identified factors related to the number ofcustomers at Michinoekies and analyzed the relation. The
availability of various facilities and services atMichinoekies was also considered, and the relation
between the existence of facilities or services and thenumber of customers was analyzed.
We would like to express our sincere gratitude to
Kyushu-Okinawa “Michinoeki” Renrakukai that gave us
much cooperation in conducting questionnaire survey.
References
1. Michi-no-Eki Official Website, https://www.michi-noeki.jp/stations/ (2017).
2. A. Tanaka and K. Kawasaki, Current state and futureissues of the development of roadside stations as disasterprevention function –A case study of roadside stations inFukushima Prefecture-, Reports of the City Planning
Institute of Japan, 14 (2016) 236–241.
3. Y. Yamamoto and A. Yuzawa, A Study on Actual
Situation and Effect of Farmer's Market in Roadside
Station from Viewpoint of Regianal Promotion Function:
A Case Study in Kanto Region, Journal of the City
Planning Institute of Japan, 47(3) (2012) 985–990.

3.3. Method and result of quantification method I
Quantification method I was used to estimate the numberof customers according to the existence of the correlated
facilities and services. These variables are included one
at a time in the estimation process in descending order ofcorrelation coefficient. The 123 Michinoekies for which
the number of customers is known are considered. Table© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

358

Emotional Contribution Analysis of Online Reviews
Elisa Claire Alemán Carreón, Hirofumi Nonaka
Department of Information and Management System Engineering, Nagaoka University of Technology,1603-1, Kamitomioka, Nagaoka, Niigata, 940-2188, Japan
E-mail: nonaka@kjs.nagaokaut.ac.jp
Toru Hiraoka

Department of Information Systems, University of Nagasaki,
1-1-1, Manabino Nagayo, Nishisonogi, Nagasaki, 815-2195, Japan
E-mail: hiraoka@sun.ac.jp, bs116025@sun.ac.jp
Minoru Kumano

Faculty of Regional Innovation, University of Miyazaki,
1-1, Gakuendai-nishi, Miyazaki, 889-2194, Japan
E-mail: kumano@cc.miyazaki-u.ac.jp
Masaharu Hirota

Department of Information Science, Okayama University of Science,
1-1, Ridaicho, Kita-ku, Okayama, 700-0005, Japan
E-mail: hirota@mis.ous.ac.jp
Takao Ito

Graduate School of Engineering, Hiroshima University,
1-7-1, Kagamiyama, Higashi-Hiroshima, 739-8521, Japan
E-mail: itotakao@hiroshima-u.ac.jp

Abstract
In response to the constant increase in population and tourism worldwide, there is a need for the development ofcross-language market research tools that are more cost and time effective than surveys or interviews. Focusing on
the Chinese tourism boom and the hotel industry in Japan, we extracted the most influential keywords in emotional
judgement from Chinese online reviews of Japanese hotels in the portal site Ctrip. Using an entropy based
mathematical model and a machine learning algorithm, we determined the words that most closely represent thedemands and emotions of this customer base.
Keywords: Entropy, Support Vector Machine (SVM), Online Review Analysis, Sentiment Analysis

populations1. Now, there was a particularly notable
increase in the Chinese tourist population in Japan of
107.3% from 2014 to 2015, and a total number of touristsof 6,372,948 in 2016 (Japan National Tourism
Organization, 2017). However, while there are studies ofChinese tourists in several countries, for example, one
researching the satisfaction levels of Chinese tourists in
Vietnam2 or another studying their food preferences inAustralia3, these studies are done mainly outside Japan,

1. Introduction
Worldwide, the population of Chinese tourists has
recently increased significantly over the years. This
increase has led to a significant impact to several
industries over the world, most directly the hotel industry,
as their customer bases changed. In addition to this
impact on business, there is a current increase ofacademic research across the world about Chinese tourist© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

359

Toru Hiraoka, Shiori Nishimura, Hirofumi Nonaka, Minoru Kumano

which has been largely impacted by Chinese tourism.
Many if not most of these studies are performed based on
the results of surveys or interviews, as are the market
studies performed by companies. However, depending on
the formulation of the questions, the results ofquestionnaire based surveys can be altered and they alsopresent difficulties and costs of the survey by itself andin this case, the need for translations. Furthermore, thesample size of these survey studies is commonly limited
to tens or hundreds of people at most. On the other hand,
Data mining and text mining techniques can not only
cheaply and quickly gather tens of thousands if not more
samples, but the source of this extensive amount of
information also can be thought to be unaltered by any
part of the data extraction process. Considering this, inour study, we developed a text-mining method to analyze
the demands and needs of Chinese customers of JapaneseHotels, exploring our results in the perspective of
business or management decisions. In order to perform
this analysis, we have extracted a large quantity of text
reviews from a Chinese portal site Ctrip (http://
http://www.ctrip.com), and determined the mostcommonly used words that would contribute the most to
emotional judgement behind positive and negative
opinions in a review using an entropy based
mathematical extraction method. These extracted
keywords related to emotional judgement not only allow
us to perform a Support Vector Machine based emotionalclassification of the reviews, but conceptual words inthese lists bring insight into which concrete topics are theChinese tourists concerned with. After classifying thesentences in the extracted reviews as positive andnegative with an optimized SVM, we have analyzed theweight value assigned to them by the SVM. For words
that are not part of the support vector this is equal to 0;
however, the support vector lets us observe the words that,while potentially close to the border between positive andnegative sentiments, provide a strong and clear
distinction in the emotional classification. These words
allow us to analyze the writing tendencies of Chinese
customers in either positive or negative reviews. We also
observed the frequency of the terms in all of the reviews
to extract the most utilized words in either kind ofreviews. This is valuable information for making
business management decisions from the point of view ofthe hotel industry companies. Because these keywords
are most related to customer satisfaction, a company can

improve customer service or facilities to increase profit.
Methodology
2. Introduction
2.1. Data collection and processing
In this study, we used the HTML parsing python library
BeautifulSoup , and the local database management tool
SQLite using the python library sqlite3 to automatize
data managing processes. Different from the English
language, Chinese texts lack a separation between
different words, and as such, when collecting these texts,
they are all a single string of characters. To be able toperform a statistical analysis of each word, the Stanford
Word Segmenter4 program developed by the Stanford
NLP Group was implemented for this task using thepython nltk library. During the segmentation of all thewords in our corpus, irregularities occurred where thereviews were written in other languages or where unusualpunctuation marks were used. We designated a list of
characters that could be recognized as noise and thencleared all the text in the corpus of these characters.
2.2. Sentiment analysis
In order to determine the words that clearly impact theuser’s emotional judgement we calculated the entropy
value of each word in relation to each class. Shannon’s
Entropy, in the field of Information Theory, is defined tobe the expected value of the information content in asignal5 or it can be thought of as the grade of
impossibility of predicting an outcome. Using this value,
we can observe the probability distribution of each word
inside the corpus. A word that is included in many
documents, it will have a high entropy value for that setof documents, since it becomes uncertain to predict inwhich document it will appear. Opposite to this, a word
appearing in only one document will have an entropy
value of zero, since it is completely predictable. We show
this concept in the figure below. To apply this logic, we
retrieved a sample of our corpus and with thecollaboration of a group of Chinese students, tagged eachsentence as the classes positive or negative depending on
the emotion that the text conveyed, then calculated theentropy values for each word in relation to the set ofsentences from each class. Words with higher entropy
relating to the positive set than to the negative set by afactor of α were determined to be keywords influencing© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan360

Relationship Analysis on thepositive emotional judgement in Chinese reviews of
hotels. Likewise, words with higher entropy for thenegative set than the positive set by a factor of α’ were
determined to be keywords related to the negative
emotional judgement in our texts.
Support Vector Machines are supervised machinelearning models usually applied to classification or
regression problems6. We use it to classify the rest of our
corpus into emotional classes in our study. An SVM istrained to classify data based on previously labeled data,
generalizing features of the data by defining a separating(p-1)-dimensional hyperplane in a p-dimensional spacein which each dimension is a feature of the data. The
separating hyperplane, along with the support vectors,
divide the multi-dimensional space and minimize theerror of classification. In our study we used the linear
kernel of the SVM classification process. Each training
sentence is a point of data, a row in the vector x, where
each column represents a feature, in our case thequantities of each of the keywords in that particularsentence. The labels of previously known classifications(1 for positive, 0 for negative) for each sentence comprisethe f(x) vector. The Weight Vector w is comprised of theinfluences each point has had in the training process todefine the angle of the hyperplane and the bias coefficientb determines its position. In the field of corpus study andNatural Language Processing, each of the features of adata point is the number of times that a word is included
in a document. In our study we used a number of keyword
lists, defined by our entropy calculations with different
comparison coefficients, as the possible features; trained
the SVMs implementing the Support Vector Classifier
included in the python library scikit-learn ; managed thevector mathematics with the mathematical python library
numpy ; and tested for each one using the K-fold Cross
Validation method, which has been proven to provide
good results in small samples7. In each test we calculated
the Precision, Recall and F1-measure8 for our predictions.
As stated earlier in section, each point of data that isclassified incorrectly causes a change in the weightvector to better locate the separating hyperplane andclassify new data correctly. These changes to the weightvector are strong for features that needed to be taken
account of to classify with a minimal error, thosecontained in the support vectors, close to the separating
hyperplane. Sequentially, the weight vector can beinterpreted as a numerical representation of the effecteach feature, or in our case, each of these normallyambiguous words, had for the classification process andthe class it has decisive influence in. Because the weightvector assigns value to the words that comprise thesupport vectors, the words with higher weights arethought to be closer to the dividing hyperplane, while stillclearly and decisively belonging to one of the categories
the hyperplane divides the high-dimensional space in.
3. Results and Discussions
After having our training data tagged by a group of Chinese
student collaborators, we experimented with different
comparison coefficients for the entropy values calculated from‘positive’ and ‘negative’ emotional classes. The mutuallyindependent coefficients α and α’ were tested from 1.5 to 3.75
in intervals of 0.25. The result was 20 lists, 10 for eachemotional class.
At the beginning we experimented with different kernels for the
SVM, as well as some Ensemble Learning methods, like the
Boosting, Voting and Stacking. We ultimately decided to use
the linear kernel for the benefits of the weight vector
obtainability. We also experimented with different parameters
for the SVC, finding that the best performing value for C, a
constant that affects the optimization process when minimizing
the error of the separating hyperplane. Low values give some
freedom of error minimizes false positives, but depending on
the data it can increase false negatives. Inversely, high valuesof C will likely result in minimal false negatives, but apossibility of false positives. We found the ideal value for thisparameter was C=0.5 in our final classifier. We trained adifferent Support Vector Classifier with each of the 20 lists, andwe chose the best performing lists for each emotional class,resulting in a positive emotion classifier (positive or not
positive), based on the results of a k-fold cross validation
process in which we calculated their accuracy and F-measuremeans and standard deviations. Table 1 shows the results of oursystem performance. After observing the misclassification
behavior for the positive emotion classifier, which mostly
misclassified negative sentences, we decided to combine both
keyword lists into a single large list to train the positive emotion
classifier.
Below Table 2 we show some of our keywords from the subject
class (words that help us understand demands of the users) thathave a relatively high weight value for both positive andnegative extremes, and their translations in the relevant context.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan361

Toru Hiraoka, Shiori Nishimura, Hirofumi Nonaka, Minoru Kumanoof Chinese text translation and that they prefer hotels
where breakfast is included.
In future works we plan to investigate further into this
topic, extending our data set, researching for other factors,
such as time, scoring, revenue, and others. Anothersubject to investigate is to determine subjects that areclassifiable as general topics in tourism compared tothose only applicable in hotels. As mentioned before, wewould also like to study the food preferences of Chinese
tourists. Additionally, it would be interesting to study
further into more specific emotions than the positive andnegative classification we performed in our study.

Table 1. Performance of our method
EmotionC

AccuracyµAccuracy

F1 µF1Positive(α=2.75)

0.5

0.88

0.15

0.90

0.09

Analyzing the extracted subject keywords is the key toperforming a market study of Chinese customers of JapaneseHotels. They can be interpreted as explicit needs and demands.
Analyzing words with an influence of positive emotions, wefound that the most relevant subjects Chinese customers
perceive positively are cleanliness and size, very possibly of theroom they had stayed in. There is also the possibility that
reviewers were praising in general the cleanliness of Japan’senvironment, easily accessible and their culture of respecting
spaces.

Acknowledgements
This research is supported by Japan Construction
Information Center Foundation (JACIC).
References
1. Sun, Y., Wei, Y., & Zhang, L. (2017). International
academic impact of Chinese tourism research: a review
based on the analysis of SSCI tourism articles from 2001
to 2012. Tourism Management, 58, 245-252.
2. Truong, T., & King, B. (2009). An evaluation ofsatisfaction levels among Chinese tourists in Vietnam.
International Journal of Tourism Research, 11(6), 521535.
3. Chang, R., Kivela, J., & Mak, A. (2010). Food preferencesof Chinese tourists. Annals of Tourism Research, 37(4),989-1011.
4. Chang, P., Galley, M., & Manning, C. (2008). Optimizing
Chinese Word Segmentation for Machine Translation
Performance. Statmt '08 Proceedings of The Third
Workshop on Statistical Machine, 224-232.
5. Shannon, C. (1948). A Mathematical Theory ofCommunication. Bell System Technical Journal, 27(3),279-423.
6. Cortes, C., & Vapnik, V. (1995). Support-vector networks.
Machine Learning, 20(3), 273-297.
7. Kohavi, R. (1995). A study of cross-validation andbootstrap for accuracy estimation and model selection.
International Joint Conference on Artificial Intelligence,14(2).
8. Powers, D. (2011). Evaluation: From Precision, Recall and
F-Measure to ROC, Informedness, Markedness &Correlation. Journal of Machine Learning Technologies,
2(1), 37-63.

Table. 2. Positive Emotion Keyword
Word

4.

EnglishWeight

干净

Clean

0.638

大

big, wide

0.624

交通

Traffic

0.586

Conclusion

In our study, with the purpose to understandemotional responses of Chinese customers of Japanese
hotels, their demands and needs, we extracted keywords
from their reviews from the Chinese portal site Ctrip
using entropy calculations from a manually classified
sample of our data; then we used these keywords inmachine learning experiments. Using the keywords totrain a linear kernel Support Vector Classifier, we
obtained the highest performance entropy based
keywords.
Using the weight vectors of our classifiers, as well as
frequency of the words in our data set, we found that
Chinese customers have a preference for big and clean
rooms, big thermal baths or bathhouses, expect goodcost-performance regardless of price, that there is a lack

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan362

An approach to visualize place of interest andshooting spot using geo-tagged photographs
Masaharu Hirota

Faculty of Informatics, Okayama University of Science
Okayama, 7000005, Japan
Masaki Endo

Division of Core Manufacturing, Polytechnic University
Kodaira, 1870035, Japan
Hiroshi Ishikawa

Faculty of System Design, Tokyo Metropolitan University
Hino, 1910065, Japan
E-mail: hirota@mis.ous.ac.jp, endou@uitec.ac.jp, ishikawa-hiroshi@tmu.ac.jp

Abstract
Hotspots are interesting places where many people do sightseeing. Hotspots are classifiable into two types: area ofinterest and shooting spot. In this paper, we describe the method for visualizing the area of interest and shooting
spot from social media sites. Our method classifies an area into the area of interest or shooting spot using a
distribution of photographic orientation. We demonstrate our approach using photographs annotated with metadatafrom Flickr.
Keywords: Geographical characterization, Visualization, Photograph location, Photograph orientation

location. Photographs that include a photographorientation have become numerous recently.
Many photographer take photographs of subjects orlandscapes that satisfy their own interests. Subsequently,
some of them upload their photographs to web sites. As
places from which many photographs have been taken,
these locations might also be interesting places for many
people to visit. The main approach to extract theinteresting places uses huge photographs obtained fromphoto-sharing sites, because the photographs present theinterest of people. As described in this paper, we definesuch places as hotspots.
Hotspots are classifiable into two types: place ofinterest (area of interest, point of interest) and shooting
spot. We define a shooting spot as a type of hotspot in anarea where many photographs have been taken. A

1. Introduction
Because of the increasing popularity of mobile devices
such as digital cameras, numerous photographs taken by
photographers have been uploaded to photo-sharing web
services. Recently, those devices have included
embedded global positioning system (GPS). Using them,photographers annotate photographic metadata tophotographs. A photographic location, which is one typeof metadata observed by GPS, shows a place at whichphotographer took photographs. Also, modern cameras
and smartphone equipped with a digital compass can add
metadata of the photographic orientation when aphotograph is taken by a photographer. The photographic
orientation denotes the direction from which thephotograph was taken, and from the photographic© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan363

Masaharu Hirota, Masaki Endo, Hiroshi Ishikawa

2. Related Work
Some researchers study approaches to analyze
hotspots obtained from large data annotated withphotographic locations.
The method used to extract hotspots is used to findgeographical characterization. Spyrou et al. proposed amethod to elucidate the underlying semantics ofextracted hotspots using user-generated tags [1]. Omori
et al. evaluate geo-referenced photographs annotatedwith user-generated tags related to coastlines showing
actual coastlines [2]. In addition, some methods exist to
extract relations between hotspots and other hotspots
such as the relations between photographic subjects andshooting spots. Shirai et al. proposed a method to extract
hotspots using DBSCAN and to calculate the relations of
hotspots [3]. Also, Hirota et al. proposed a method todetect and to visualize various relations of hotspots using
photographic orientations and social tagging [4]. Those
researches extract hotspots based on density-based
approach. In this paper, we propose a method to extract
two types of hotspot of area of interest and shooting spot
uses the distribution of photographic orientation of huge
photographs in an area.

Fig.1. Examples of area of interest and shooting spot.

photographer who takes a photograph of an attractive
spot such as a landmark or landscape takes thephotograph at a place that is distant from the spot. As aresult, photographic orientation of photographs in theshooting spot is oriented in a specific direction. Also,
area of interest (point of interest or place of interest), theother type of hotspot, is usually attractive as a tourist spot
for many people (e.g., Colosseum, Statue of Liberty). In
such areas, many photographers take many photographs
inside or near places. As a result, photographic
orientation of photographs in area of interest is oriented
in various directions.
In this paper, we classify an area into the area ofinterest or shooting spot based on the distribution ofphotographic orientation of photographs taken at the area.
Here, we show Figure 1 of the examples of the area ofinterest and shooting spot. In this figure, an icon of thecamera shows the photographic location of a photograph,and the orange arrow shows the photographic orientationof the photograph. The distribution of photographic
orientation in an area of interest is various, because there
is a landmark around a place of the area of interest. As aresult, the direction of photographs is not specified like
Figure 1. On the other hand, the distribution ofphotographic orientation in a shooting spot is unbalanced,
because hooting spot is distant from a landmark. As aresult, the difference of the distribution of photographic
location in an area may be effective for the classificationof the area of interest and shooting spot.
In this paper, we address the method for extracting
shooting spot and area of interest from huge photographs
annotated with metadata. Our approach to extract theshooting spot and area of interest uses machine learningalgorithm based on photographic orientation.

3. Proposed method
We propose a method to visualize area of interest andshooting spot using classification algorithm based on
photographic location and orientation. Our approach
includes the following steps.
1.
For a particular area to analyze, we obtained vast
numbers of photographs with metadata ofphotographic orientation and location from a social
media sites.
2.
From photographs, we used to analyze theparticular area, we create a grid space and cellswith the size specified by a user.
3.
We extract the cells in which many photographs
were taken and create the histogram ofphotographic orientation in each area.
4.
In each area, we apply the classification method to
identify location which is area of interest or
shooting spot based on the distribution ofphotographic orientation.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, Beppu International Convention Center, Oita, Japan364

An approach to visualize

In this section, we visualize the area of interests andshooting spots using our proposed method from vast
photographs obtained from Flickr [6].

3.1. Calculation of grid
To identify location of area of interest and shooting spotfrom which many people have taken photographs, wecreate a two-dimensional grid that consists of square cells
based on photographic location.
Photograph p ∈ = {p , , … } is mapped to thecoordinate as shown below.
y = p − Lat∗M− Lat )/ (Lat∗M
x = p − Lng/ (Lng− Lng )
Here, y and x shows the coordinate of photograph p
in the grid. Lat
, Lat
,
and
respectively represent the maximum and minimum
values of latitudes and longitudes in a dataset.
and M
respectively denote
Additionally, M
the height and width of the grid. There are found usingparameter m for adjustment of how many cells we wantto make in this procedure. Using the obtained grid, we
extract the cells for which the number of photographs
including its cell is greater than threshold minP as a
hotspot.

4.1. Experimental conditions
We describe a dataset for experiments for visualizing
areas of interest and shooting spot. Photographs forexperiments are obtained from the Flickr photographic
search results. Those photographs have Exchangeable
image file format (Exif) metadata for latitude, longitude,
and orientation. We obtained 370,626 photographs
taken in London. To prepare a training dataset whichcontains the area of interest and shooting spot, weannotate labels (area of interest or shooting spot) toareas. Table 1 presents the number of each label.
Table 1 Number of labels
Number of label
Area of interest
45
Shooting spot
100
SVM employed for the experiments used a Gaussian
kernel. Additionally, we experiment using five crossvalidation, and we tune hyperparameters c and γ by gridsearch.
4.2. Classification

3.2. Classification of hotspots

We describe the performance of our classification
results. Table 2 presents our classification results. In thispaper, criteria of classification performance are F1measure, Precision and Recall.
Table 2 Results of classification
Score
F1-measure
0.928
Precision
0.937
Recall
0.931
In Table 2, our classification is high scores in eachcriterion. This result shows that using the distribution ofphotographic orientation of photographs in an area asfeature vector is effective for classification of area ofinterest and shooting spot. We use the classifier createdin this section at the following visualization steps.

To identify each hotspot in the area of interest andshootings, we apply machine learning algorithm to thehotspots. The classification is based on the feature of the
distribution of the photographic location of photographs
in a hotspot. In this paper, we use Support Vector
Machine (SVM) as classification algorithm to classify ahotspot into the area of interests or shooting spot.
To create a feature vector of a hotspot, we create ahistogram of the photographic orientation. We count the
number of photographs every 30 degrees, and we use theresult as a histogram. Then, the histogram is normalized
to the sum to 1. Also, the histogram is sorted based on the
count of photographs. This reason is the approach of thisclassification does not consider of ordering of thephotographic orientation. In other words, the feature forour classification is based only on the distribution. Afterthe learning of classification, we classify each hotspot
into the area of interest or shooting spot, and we visualize
classified results on OpenStreetMap [5].

4.3. Visualization
Figure 2 and 3 present the results of the area of interestand shooting spot extracted from photographs taken in
the same area of London using our proposed method.
The sides of cells in figure2 and 3 is 50 and 25 meter,
respectively. In those cells, red cell shows the area ofinterest. Black cell shows the shooting spot.
In a to c in figure 2 and 3, the point where figure 2
encloses figure 3 is classified as the area of interest ineach figure. In a and b of the figures, the large area of

4. Experiments© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, Beppu International Convention Center, Oita, Japan365

Masaharu Hirota, Masaki Endo, Hiroshi Ishikawa

Fig.2. Visualization result (50 meter cells).

Fig.3. Visualization result (25 meter cells).

interest is divided into small area of interests andshooting spots. Also, in c of the figures, the same area is
extracted as the area of interest. Therefore, the extractedarea of interest in figure 3 is smaller. These result shows
reducing the size of the cell, we can extract moredetailed points.
Also, in the d in figure 2 and 3, the classification resultchanges shooting spot to the area of interest according
to reducing the size of cells. This result shows even atpoints including the area of interest, if the cell size is not
appropriate, the approach cannot extract appropriate
geographical characteristics. Adjusting the cell size
automatically for extracting the area of interest andshooting spot is a future work.

Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number 16K00157, 16K16158, and Tokyo
Metropolitan University Grant-in-Aid for Research on
Priority Areas Research on social big data.
References
1. E. Spyrou, M. Korakakis, V. Charalampidis, A. Psallas,and P. Mylonas. A geo-clustering approach for thedetection of areas-ofinterest and their underlying
semantics. Algorithms, 10(1):35, 2017.
2. M. Omori, M. Hirota, H. Ishikawa, and S. Yokoyama. Can
geo-tags on flickr draw coastlines? In Proceedings of the
22Nd ACM SIGSPATIAL International Conference on
Advances in Geographic Information Systems, pages 425–
428.ACM, 2014.
3. M. Shirai, M. Hirota, H. Ishikawa, and S. Yokoyama. A
method of area of interest and shooting spot detection
using geo-tagged photographs. In Proceedings of The FirstACM SIGSPATIAL International Workshop on
Computational Models of Place, pages 34:34–34:41. ACM,
2013.
4. M. Hirota, M Shirai, H. Ishikawa, and S Yokoyama.
Detecting relations of hotspots using geo-tagged
photographs in social media sites. In Proceedings ofWorkshop on Managing and Mining Enriched Geo-Spatial
Data, pages 7:1–7:6.ACM, 2014.
5. OpenStreetMap, http://www.openstreetmap.org/
6. Flickr, https://www.flickr.com/

5. Conclusions
In this paper, we proposed a method for visualizing
area of interest and shooting spots using theclassification method. Our classification uses the
distribution of photographic orientation in an area. Afterusing our proposed method with photographs obtained
from Flickr, we presented and discussed example ofresult.
As a future work, we will recommend travel routs
considering this result, because such places that allow
people to take photographs around famous landmarks
are important for tourism.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, Beppu International Convention Center, Oita, Japan

366

Histogram Analysis Method Based on Gaussian Distribution and Curvature Computation (I)---- Peaks and Valleys Detection ---Yusuke Kawakami*

DynaxT Co., Ltd., 2217-6 Hayashi
Takamatsu City, Kagawa 761-0301, Japan
Tetsuo Hattori, Yoshiro Imai, Kazuaki Ando, Yo Horikawa

Graduate School of Kagawa University, 2217-20 Hayashi
Takamatsu City, Kagawa 761-0396, Japan
R. P. C. Janaka Rajapakse

Tainan National University of the Arts, 66 Daci
Guantian District, Tainan 72045, Taiwan
E-mail: riverjp2002@gmail.com, {hattori imai, ando, horikawa}@eng.kagawa-u.ac.jp, janakaraja@gmail.com

Abstract
This paper presents an analysis method of image histogram using curvature computation based on Gaussian
distribution. By the image histogram and curvature computing over the cumulative histogram, we can not only
detect the peaks and valleys in the image histogram, but also obtain the accompanying information on the peaks andvalleys such as those shapes and sizes. In this paper, we propose the information extraction method from image
histogram using the curvature computation, based on the assumption that the peaks and valleys are approximated by
the same number of Gauss density functions.
Keywords: Image processing, Histogram Analysis, Cumulative histogram, Curvature computation

illustrated that HMGD processing can improve feeling(or Kansei) impression of the original image. Moreover,
we have proposed the method for peaks detection of theimage histogram and the variance estimation of Gaussdensity function that is approximated at each of thepeaks, based on curvature computation of the
cumulative histogram.
Since we consider that the method for peaks detection
can also be applied to valleys detection and that the

1. Introduction
As we have already suggested, automated image
processing for the enhancement of color images has
become very familiar, for example, Digital Signage,
Smart Phone, etc 1-3.
In our previous papers, we presented a Histogram
Matching based on Gaussian Distribution (HMGD)
processing for the image arrangement and 4-6. And we*

1882-36, Yashima-nishi, Takamatsu City, Kagawa 761-0113 Japan© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

367

Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai, Kazuaki Ando, Yo Horikawa, R. P. C. Janaka Rajapakse

method can be developed for the whole histogram
analysis, then in this paper, we newly propose themethod based on the curvature computation over the
cumulative histogram as a useful one for the whole
histogram analysis. Also in this paper, repeating ourpreviously presented methods, we suggest the capabilityof our proposed method to make analysis of histogram
analysis.

a x

g x

2

Rx

1 g x

a x

3
2 2

2

g x.

(6)

From Eq. (6), we understand that the curvature R(x)varies the sign according to the value of x 9. That is, if
x<a → R>0 (downward convex shape), and if x<a →R<0 (upward convex shape).

2. Useful Principles
As for valley detection, we can approximate the shapeof the neighbor points at the valley point as 1.0 - g(x).
So, around the valley point, the curvature R(x) is shownas follows:

2.1. Peaks and Valleys Detection of Histogram
In the section, we firstly describe the principle of peakdetection of original image.
Let y be a function with respect to x, the definition
curvature R(x) is given by Eq. (1)6-9.
2

d y
1
dx 2

dy
dx

2

3
2

a

x
2

R x

1 g x

g x

x a

3
2 2

(6’)

g x.

2

(1)

Then, if x<a → R<0 (upward convex shape), and if x<a→ R>0 (downward convex shape).

Let g(x) and K be a Gaussian distribution function
approximated at a peak and its coefficient that are
defined by the following equations.

In any case, it holds that if R>0, the shape of the original
histogram curve is downward convex.
In this way, from the value of curvature over the
cumulative histogram, we can detect peaks and valleys
in the original histogram curve.

Rx

K

g x

2
K

L
2

2

0

2

x a2
2 2

exp

u a
2 2

exp

.

2

(2)

2.2. Variance Estimation
In this section, we explain two methods of varianceestimation for the Gauss density function.

du 1

Next, let y=f(x) be a function representing the
cumulative histogram which is represented Eq. (3).
That is, dy/dx and d2y/dx2 be described as Eq. (4) and (5),respectively. From Eq. (4) and (5), we obtain theapproximation of curvature R(x) as Eq. (6).
xf x

g u du
0

dy
dx

g x

2

d y
dx 2

x

K
2

2
K
2

dg x
dx

2

exp

0

u a
2 2
x a
2 2

exp

a x
2

2

.

2

2.1.1. Variance Estimation Based on Symmetry
Fig. 1 shows the conceptual image of the original image
histogram which is variance 2 and average a. And Fig.
2 shows its cumulative histogram. From Eq. (2), we
have the following Eq. (7) and Eq. (8).

du. (3)

ga

2

2

K
exp
2

a

K
.
2

(7)

(4)

ga
g x

K

(5)

2

2
2

a
2

gae1

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

368

(8)

Histogram Analysis Method Based

g(a)

H

2

2

Ra

y

S
1 S

2

3

2

2

S
3

1 S

2

gae

1

gae1

1

2

2

3

2

H.

(10)

.

That is, we can obtain the variance 2. For example, letv
2 be the distance from average a,
0

a

2

a

2

a

Ra v

x

L

2 2

Ra v

4
H,
v

H

2

v 2 (11).
2

2.1.2. Variance Estimation by Regression Analysis
Fig. 1. Conceptual image of the original image histogram 8, 9.

yR a

From Eq. (6), the R(x) can be shown as the following Eq.
(12).

a x

Rx

2

2

0

Rx

a

a

2

a

2

Thus, we can evaluatenThen, the curvature value R at a
Eq. (9).

2

2

a
2

ga

2

1 ga

2

C

1

gae1

2

3

i 1
n
i 1

2

3

2

i

2

,

2

i 1,

n

(14)

as following Eq. (15).

H i Ri
Hi

2

,

2

1
C

(15)

Both of the variance estimation methods are equally
useful for the whole histogram analysis.

(9)

gae1

2
2

is represented as

2

(13)

CH x

CH i

i ~ N 0,Fig. 2. Conceptual image of the original image cumulative
histogram 8, 9.

Raa x g x . (12)Now, we can calculate a constant C by using leastsquare regression analysis10 as in the following Eq. (14).

x
L

Ri

a

2

And, let C=1/ 2 and H(x) = (a-x)g(x) respectively, we
can derive Eq. (13).

R(a)
Ra

1

g x

2

3. Experimentation

2

Fig. 3 shows an example of the curvature computationand peak and valley points detection. Peaks are detected
at x= 53, x=91, x=151. Valleys are detected at x=61,
x=112. Moreover, the variances around those points will
be obtained by the aforementioned method. So, theLet S=g(a)e-1, we have Eq. (10).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan369

Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai, Kazuaki Ando, Yo Horikawa, R. P. C. Janaka Rajapakse

3. E. S. Umbaugh, Computer Vision and Image Processing:
A Practical Approach Using CVIP tools (Prentice Hall
PTR, 1998).
4. W. Burger and J. M. Burge, Principles of Digital Image
Processing: Fundamental Techniques (Springer 2009).
5. T. Izumi, T. Hattori, S. Sugimono, and T. Takashima,
Color Image Arrangement Using Elastic Transform on
Principal Component Axis (in Japanese), Journal ofJapan Society of Kansei Engineering 8(3) (2009) 667674.
6. Y. Kawakami, T. Hattori, D. Kutsuna, H. Matsushita,
Y. Imai, H. Kawano, R.P.C. Janaka Rajapakse,
Automated Color Image Arrangement Method Based on
Histogram Matching - Investigation of Kansei impression
between HE and HMGD -, International Journal of
Affective Engineering 14(2) (2015) ISSN 2187-5413, 8593.
7. Y. Kawakami, T. Hattori, Y. Imai, H. Matsushita,
H. Kawano, R. P. C. Janaka Rajapakse, Kansei
Impression and Automated Color Image Arrangement
Method, Journal of Robotics, Networking and Artificial
Life 1(1) (2014) ISSN 2352-6386, 60-67.
8. Y. Kawakami, T. Hattori, Y. Imai, H. Matsushita, H.
Kawano, and R. P. C. Janaka Rajapakse, AutomatedColor Image Arrangement Method Using Curvature
Computation in Histogram Matching, in Proceedings ofInternational Conference on Artificial Life and Robotics(ICAROB 2015) (Oita, Japan, 2015) ISBN 978-49902880-9-9, pp. 272-277.
9. Y. Kawakami, T. Hattori, Y. Imai, Y. Horikawa,
H. Matsushita, R. P. C. Janaka Rajapakse, AutomatedProcessing of Multiple-Brightness Peak Histogram Image
Using Curvature and Variance Estimation, Journal of
Robotics, Networking and Artificial Life 3(1) (2016)
ISSN 2352-6386, 55-60.
10. D. G. Kleinbaum, L. L. Kupper, A. Nizam, E. S.
Rosenberg, Applied Regression Analysis and Other
Multivariable Methods (Brooks/Cole Pub Co, 2013).
11. Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai,
Kazuaki Ando, Yo Horikawa, R. P. C. Janaka Rajapakse,
Histogram Matching Based on Gaussian Distribution
Using Regression Analysis Variance Estimation”,
Proceedings of International Conference on Artificial
Life and Robotics (ICAROB 2017), ISBN 978-49908350-2-6, pp.575-578, 2017.
12. Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai,
Kazuaki Ando, Yo Horikawa, R. P. C. Janaka Rajapakse,
Histogram Matching Based on Gaussian Distribution
Using Variance Estimation --- Comparison between
Curvature Computation and Regression Analysis --Proceedings of International Conference on Artificial
Life and Robotics 2017 (ICAROB 2017), ISBN 978-49908350-2-6, pp.579-582, 2017.

approximated Gauss density functions can be obtainedat each of those points.
We consider that this will be very useful for histogram
analysis.

(a)

(b)

(c)

(d)

(e)

(f)

Fig.3. Example of the curvature computation. (a)Original
image. (b) Histogram of the original image. (c) Cumulative
histogram. (d) Curvature of the cumulative histogram. (e)Peaks are detected at x= 53, x=91, x=151. (f) Valleys aredetected at x=61, x=112.

References
1. R. C. Gonzalez and R. E. Woods, Digital Image
Processing (Addison-Wesley Publishing Company,
1993).
2. B. Jahne, Digital Image Processing --Concepts,
Algorithms, and Scientific Applications-- 4th edition(Springer, 1995).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan370

Histogram Analysis Method Based on Gaussian Distribution and Curvature Computation (II)---- Experimentation ---Yusuke Kawakami*

DynaxT Co., Ltd., 2217-6 Hayashi
Takamatsu City, Kagawa 761-0301, Japan
Tetsuo Hattori, Yoshiro Imai, Kazuaki Ando, Yo Horikawa

Graduate School of Kagawa University, 2217-20 Hayashi
Takamatsu City, Kagawa 761-0396, Japan
R. P. C. Janaka Rajapakse

Tainan National University of the Arts, 66 Daci
Guantian District, Tainan 72045, Taiwan
E-mail: riverjp2002@gmail.com, {hattori imai, ando, horikawa}@eng.kagawa-u.ac.jp, janakaraja@gmail.com

Abstract
This paper presents an analysis method of image histogram using curvature computation based on Gaussian
distribution. By the image histogram and curvature computing over the cumulative histogram, we can not only
detect the peaks and valleys in the image histogram, but also obtain the accompanying information on the peaks andvalleys such as those shapes and sizes. In this paper, we expose the experimental results and show the effectivenessof the proposed method.
Keywords: Image processing, Histogram Analysis, Cumulative histogram, Curvature computation

This paper presents an analysis method of image
histogram using curvature computation based on
Gaussian distribution. By computing the curvature of
the cumulative histogram curve, we can not only detect
the peaks and valleys in the image histogram, but also
obtain the accompanying information on the peaks andvalleys such as those shapes and sizes.

1. Introduction
Nowadays,automated
image
processingforenhancement of color images and/or image arrangement
has become very familiar, for example, Digital Signage,
Smart Phone, etc1-3. In such automated image
processing, histogram analysis of original image plays avery important role4-12.
*

1882-36, Yashima-nishi, Takamatsu City, Kagawa 761-0113 Japan© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan371

Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai, Kazuaki Ando, Yo Horikawa, R. P. C. Janaka Rajapakse

In this paper we also provide the experimental results
for some color images and show the effectiveness.

From Eq.(7), we understand that the sign of thecurvature R(x) varies according to the value x (9). That is,
if x<a → R>0 (downward convex shape), and if x<a →R<0 (upward convex shape).
Here, R>0 (downward convex shape) and R<0 (upward
convex shape) correspond to the peak and valley in theoriginal histogram curve, respectively.

2. Useful Principles13
2.1. Peaks and Valleys Detection of Histogram13
Let y be a function with respect to x, the definition
curvature R(x) is given by Eq. (1)6-9.

d2y
1
dx 2

Rx

3
2

2

dy
dx

2.2. Variance Estimation by Regression Analysis13
From Eq.(7), the R(x) can be shown as the following
Eq.(8).

(1).

Let g(x) and K be a Gaussian distribution function
approximated at a peak and its coefficient that are
defined by the following equations.
K

g x

2

2
L

K
2

2

0

x a
2 2

exp

u a
2 2

exp

a x

Rx

2

2

Rx

(3)

Ri

Next, let yf x be a function representing the
cumulative histogram which is represented as Eq.(4).
That is, dy/dx and d2y/dx2 be described as Eq. (5) and (6),respectively. From Eq.(5) and (6), we obtain theapproximation of curvature R(x) as Eq. (7).
xKg u du

K

g x

2

d y
dx 2

2

2

dg x
dx
a x

Rx

2

2

0

dy
dx

x

2

x a
2 2

exp

a x
2

g x

1 g x

0

3
2 2

u a2
du.
2 2

exp

2

.

g x

a x
2

g x.

a x g x.

(8)

(9)

CH x

Now, we can calculate a constant C by using leastsquare regression analysis10 as in the following Eq.(10).

i

f x

2

And, let C=1/ 2 and H(x) = (a-x)g(x) respectively, we
have Eq.(9).

(2)

du 1

1

g x

2

CH i
~ N 0,

Thus, we can evaluate
n

C

(4)

i 1
n
i 1

(5)

i

2

,

2

i 1,

n

(10)

as the following Eq.(11).

H i Ri
Hi

2

,

2

1
C

(11)

3. Experimentation
In order to how far the methods works, we have applied
the aforementioned them to some images that aretransformed from the original color images into grey
level images.
Fig.1 and Fig.2 show the examples of the curvaturecomputation over cumulative histogram. And they also
show the peak and valley points detection. Those peaks
and valleys are very well detected.

(6)

(7)

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

372

Histogram Analysis Method Based

600

1400
1200

400

1000

200
800

0
1

600
400

-200

200

-400

23 45 67 89 111 133 155 177 199 221 243

0
1

(a)

23 45 67 89 111 133 155 177 199 221 243

(b)

(e)

-600

(c)

(d)

Fig.1. Example of peaks and valleys detection (I).
(a) Original color image. (b) Lightness Histogram.
(c)Cumulative Histogram. (d) Curvature from the
Cumulative Histogram curve. (e) Detected peak points
and their values in the Lightness Histogram. (f) Detectedvalley points and their values in the Lightness
Histogram.

(f)

800

1800
1600

600
1400
1200

400

1000

200

800
600

0
1

400

23 45 67 89 111 133 155 177 199 221 243

-200

200
0
1

(a)

23 45 67 89 111 133 155 177 199 221 243

-400

(b)

1800

1800

1600

1600

1400

1400

1200

1200

1000

1000

800

800

600

600

400

400

(c)

Fig.2. Example of peaks and valleys detection (II).
(a) Original color image. (b) Lightness Histogram.
(c)Cumulative Histogram. (d)Curvature from the
Cumulative Histogram curve . (e)Detected peak points
and their values in the Lightness Histogram. (f)Detectedvalley points and their values in the Lightness
Histogram.

200

200

0

0
1

23 45 67 89 111 133 155 177 199 221 243

(e)

1

(d)

23 45 67 89 111 133 155 177 199 221 243(f)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan373

Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai, Kazuaki Ando, Yo Horikawa, R. P. C. Janaka Rajapakse

5. T. Izumi, T. Hattori, S. Sugimoto, and T. Takashima,
Color Image Arrangement Using Elastic Transform on
Principal Component Axis (in Japanese), Journal ofJapan Society of Kansei Engineering 8(3) (2009) 667674.
6. Y. Kawakami, T. Hattori, D. Kutsuna, H. Matsushita,
Y. Imai, H. Kawano, R.P.C. Janaka Rajapakse,
Automated Color Image Arrangement Method Based on
Histogram Matching - Investigation of Kansei impression
between HE and HMGD -, International Journal of
Affective Engineering 14(2) (2015) ISSN 2187-5413, 8593.
7. Y. Kawakami, T. Hattori, Y. Imai, H. Matsushita,
H. Kawano, R. P. C. Janaka Rajapakse, Kansei
Impression and Automated Color Image Arrangement
Method, Journal of Robotics, Networking and Artificial
Life 1(1) (2014) ISSN 2352-6386, 60-67.
8. Y. Kawakami, T. Hattori, Y. Imai, H. Matsushita, H.
Kawano, and R. P. C. Janaka Rajapakse, AutomatedColor Image Arrangement Method Using Curvature
Computation in Histogram Matching, in Proceedings ofInternational Conference on Artificial Life and Robotics(ICAROB 2015) (Oita, Japan, 2015) ISBN 978-49902880-9-9, pp. 272-277.
9. Y. Kawakami, T. Hattori, Y. Imai, Y. Horikawa,
H. Matsushita, R. P. C. Janaka Rajapakse, AutomatedProcessing of Multiple-Brightness Peak Histogram Image
Using Curvature and Variance Estimation, Journal of
Robotics, Networking and Artificial Life 3(1) (2016)
ISSN 2352-6386, 55-60.
10. D. G. Kleinbaum, L. L. Kupper, A. Nizam, E. S.
Rosenberg, Applied Regression Analysis and Other
Multivariable Methods (Brooks/Cole Pub Co, 2013).
11. Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai,
Kazuaki Ando, Yo Horikawa, R. P. C. Janaka Rajapakse,
Histogram Matching Based on Gaussian Distribution
Using Regression Analysis Variance Estimation”,
Proceedings of International Conference on Artificial
Life and Robotics (ICAROB 2017), ISBN 978-49908350-2-6, pp.575-578, 2017.
12. Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai,
Kazuaki Ando, Yo Horikawa, R. P. C. Janaka Rajapakse,
Histogram Matching Based on Gaussian Distribution
Using Variance Estimation --- Comparison between
Curvature Computation and Regression Analysis --Proceedings of International Conference on Artificial
Life and Robotics 2017 (ICAROB 2017), ISBN 978-49908350-2-6, pp.579-582, 2017.
13. Yusuke Kawakami, Tetsuo Hattori, Yoshiro Imai,
Kazuaki Ando, Yo Horikawa, R. P. C. Janaka Rajapakse,
Histogram Analysis Method Based on GaussianDistribution and Curvature Computation (I) ---- Peaks
and Valleys Detection ----, Proceedings of International
Conference on Artificial Life and Robotics 2018
(ICAROB 2018), to be published.

Moreover, each variances of the approximated Gaussdensity function around those points are detected by the
aforementioned method as well.

4. Conclusion
This paper has described a histogram analysis methodfor Lightness histogram, using curvature computation
over the cumulative histogram based on the assumptionthat each peak in the Lightness histogram can be locallyapproximated to a Gaussian density function.
And by considering that the same curvaturecomputation also gives the valley detection at the sametime, we have proposed the analysis method as a very
promising one.
In fact, the experimentation results for the five images
have revealed that the method has correctly detected thepeaks and valleys.
As for the local approximation of Gaussian density
function, that is, as for the variance extraction, theregression method also has worked very well.
As a result, we consider that the proposed method isvery effective and promising.
As a further study, we think to take the following twostage strategy for the histogram analysis.
(i) First stage: Peaks and valleys detection and local
approximation of Gaussian density function (its averageand variance estimation) at each peaks that is equal tothe proposed method in this paper.
(ii) Second stage: The shape information extraction of
Mountains between neighboring two valleys and/or theperspective analysis using Scale Space Filtering.

References
1. R. C. Gonzalez and R. E. Woods, Digital Image
Processing (Addison-Wesley Publishing Company,
1993).
2. B. Jahne, Digital Image Processing --Concepts,
Algorithms, and Scientific Applications-- 4th edition(Springer, 1995).
3. E. S. Umbaugh, Computer Vision and Image Processing:
A Practical Approach Using CVIP tools (Prentice Hall
PTR, 1998).
4. W. Burger and J. M. Burge, Principles of Digital Image
Processing: Fundamental Techniques (Springer 2009).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan374

Experimental Evaluation of Change Detection Ability in New Sequential Probability Ratio
Yoshihide Koyama *1, Tetsuo Hattori*1, Yoshiro Imai*1, Yo Horikawa*1

Graduate School of Kagawa University, 2217-20 Hayashi
Takamatsu City, Kagawa 761-0396, Japan
Yusuke Kawakami*2

DynaxT Co., Ltd., 2217-6 Hayashi
Takamatsu City, Kagawa 761-0301, Japan
Hiromichi Kawano*3

NTT Advanced Technology Corporation, 19-18, Nakamachi
Musashino, Tokyo 180-0006, Japan
Takeshi Tanaka*4

Hiroshima Institute of Technology, 2-1-1 Miyake, Saeki-ku
Hiroshima 731-5193, Japan*1{hattori, imai, horikawa}@eng.kagawa-u.ac.jp,*2riverjp2002@gmail.com, *3hiromichi.kawano@ntt-at.co.jp,*4tanaka@cc.it-hiroshima.ac.jp

Abstract
Previously, we have already proposed a novel method using New Sequential Probability Ratio (NSPR) for the
structural change detection problem of ongoing time series data instead of using SPRT (Sequential Probability
Ratio Test). And in the previous paper, we have revealed the experimental results by applying the both methods, i.e.,
NSPR and SPRT, to time series data that are generated by a multiple regression model. In this paper, we present theevaluation results of NSPR more concretely in two cases.

Keywords: Time series data, structural change detection, SPRT, New Sequential Probability Ratio (NSPR)
monitoring data has changed or not, correctly and early
as much as possible.
We consider that it is very applicable to a wide rangeof fields. For the structural change detection problem,
we have already proposed some kinds of methods 6-13

1. Introduction
Change point detection (CPD) problem for ongoingtime series data is very important problem 1-7. It is for
identifying whether the generation structure of*2217-20 Hayashi-Cho, Takamatsu, Kagawa 761-0396 Japan© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan375

Yoshihide Koyama, Tetsuo Hattori, Yoshiro Imai, Yo Horikawa, Yusuke Kawakami, Hiromichi Kawano, Takeshi Tanaka

respectively, where S0 and S1 mean the state ofunchanged state and changed state, respectively.
~That is, it means that θi {R, 1 R} and θi {Rc , 1 Rc } ,respectively, where R P(OUT | S0 ) , Rc P(OUT | S1 ) .
And let P(Cn, H0) and P(Cn, H1) be the joint
probability of the symbol sequence Cn that happen
with the event H0 (the structural change is notoccurred) and H1 (the change is occurred), respectively.
Then, the following equations hold.

and have suggested that the New Sequential
Probability Ratio (NSPR) method is very promising.
In this paper, we compare and evaluate the two Tests
that are Sequential Probability Ratio Test (SPRT) and
New Sequential Probability Ratio (NSPR) by theexperimental result 6-13.
2. SPRT11-15
The Sequential Probability Ratio Test (SPRT) is used
for testing a null hypothesis H0 (e.g. the quality isunder pre-specified limit 1%) against hypothesis H1(e.g. the quality is over pre-specified limit 1%). And it
is defined as follows:
Let Z1, Z2, …Zi be respectively observed time series
data at each stage of successive events, the probability
ratio i is computed as follows.

λi

P ( Z1 | H 1 ) P ( Z 2 | H 1 )P ( Z1 | H 0 ) P ( Z 2 | H 0 )P ( Z i | H1 )P( Z i | H 0 )

1

, C2

1

a n , H 0 ) (1 γ) nθ1 θn

P ( a1

an , H1) γ

n

~
θi

(1 γ) nnΠθ (3)ii 1n((1 γ)θ1 )(γ

i 1

~
θi )

i 2

((1 γ) 2θ1θ2 )(γ

n

~
θi )

i 3
n

(1)

( (1 γ) k 1･

k 1

k 1

where P(Z | H0) denotes the distribution of Z if H0 istrue, and similarly, P(Z | H1) denotes the distribution of
Z if H1 is true.
Two positive constants C1 and C2 (C1 < C2) are chosen.
If C1 < i < C2, the experiment is continued by taking
an additional observation. If C2 < i, the process is
terminated with the rejection of H0 (acceptance of H1).
If i < C1, then terminate this process with theacceptance of H0.
C1

P ( a1j 0nθ j )(γ

~
θi )

i k

(4)

P(H0 | a1 an ) P(H0 | Cn )P(Cn , H0 )P(Cn )(5)

P(H1 | a1

P(Cn , H1)
P(Cn )(6)an )P(H1 | Cn )Using these above equations (Eq.(3)-(6)), the New
Sequential Probability Ratio (NSPR) that we propose
can be represented as follows.

NSPR

(2)

where
means type I error (reject a true null
hypothesis), and means type II error (accept a null
hypothesis as true one when it is actually false).

P(H1 | a1 an ) P(H1 | Cn ) P(Cn , H1)
P(H0 | a1 an ) P(H0 | Cn ) P(Cn , H0 )
n
k 1
n ~
((1 γ)k 1･ θj )(γ θi )

k 1

j 0
n
n

i k

(1 γ) Πθii 1

3. New Sequential Probability Ratio (NSPR)(7)

3.1. Definition of NSPR11-15

3.2. Recursive Form

Let C n a1a2 , , ai , an ai {IN, OUT} be a string (orsymbol sequence) obtained from the observed data.
~Let θi and θi be the conditional probability thatreceives the observed data ai in the state S0 and S1,When the NSPR is greater than 1.0, we can regard thatthe structural change has been occurred before thepresent time.
In the similar way as SPRT, the definition of NSPR
can also be described in a recursion formula. Let© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

376

Experimental Evaluation of Change

Λn be the value of NSPR defined by Eq.(7), we havethe following recursive equation.
~1(Λn 1 γ)( n )
n
1 γ

Λn

x1 20x2 5

y

y
250

(8)

y

x1 10x2 5change point tc = 70

200

150

0,

where Λ0

0

1,

~

0

1.

100

50

4. Experimental Evaluation

t

0
0

10

20

30

40

50

60

70

80

90

100

(a) (Case 1)

We have done the comparison experimentation by
SPRT and NSPR for a time series data that is generated
based on the following equations.

y

x1

y

x1

20 x 210 x 2

5

ε

5

(t
(t c

tc )
t)

(9)

where ε~N(0,σ2), i.e., the error ε is subject to theNormal Distribution with the average 0 and thevariation σ2, and tc means the change point. In addition,
we have set tc=70 and σ=5. The number of the time
points is 100 (i.e. .
Setting the two kinds (or cases) of coefficients inEq.(9) as shown in Eq.(10) for Case 1 and Eq.(11) for
Case 2, respectively. And we have generated 200 setsof time series data for the two kinds (or cases) of timeseries, respectively.
x1
x1

t, x2
random, x2

sin
sin

1
t
8
1
t
8

(b) (Case 2)

Fig.1. Example of two kinds of time series data
generated based on Eq.(9).
(a)Case1 (x1=t, and x 2 sin( t / 8) ).
(b)Case2 (x1=random, and x 2 sin( t / 8) ).
We define a two dimensional vector I=(I1, I2) as
follows.

(10)

t 70

I

(11)

( I 1, I 2 )fr ( t )t 41

t tc ,t 100
fr (t )
t 71

t

tc

(12)

where tc=70, and fr (t ) means the detection frequency
at time point t in 200 time series data.

For the two kinds of time series data (400 time series
data in total), we have examined the change detection
ability and the accuracy of each method (SPRT and
NSPR) by setting some various evaluation criteria.
In the experimentation, the prediction model isconstructed using the first 40 time points and the restof time points from t=41 to t=100 are used for change
detection.

We can consider that the detection ability and theaccuracy become higher as the absolute value of I1 andI2 approach to 0.
The all plotted data obtained by the experimentation in
the case for Case1 and Case2 are shown in Fig.2.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan377

Yoshihide Koyama, Tetsuo Hattori, Yoshiro Imai, Yo Horikawa, Yusuke Kawakami, Hiromichi Kawano, Takeshi Tanaka

As a result, in the sense of the abovementioned criteria,
the NSPR method has revealed that it is superior to
SPRT for the both cases of time series data used in theexperimentation.

6. H. Kawano, T. Hattori, and K. Nishimatsu, Structural
Change Point Detection Method of Time Series Using
Sequential Probability Ratio Test -Comparison with
Chow Test in the ability of early detection- (inJapanese), IEEJ Trans. EIS. 128(4) (2008) 583-592.
7. G. C. Chow, Tests of Equality Between Sets of
Coefficients in Two Linear Regressions, Econometrica.
28(3) (1960) 591-605.
8. K. Takeda, T. Hattori, T. Izumi and H. Kawano,
Extended SPRT for Structural Change Detection ofTime Series Based on Multiple Regression Mode, inProceedings of the 15th International Symposium on
Artificial Life and Robotics, (Feb. 2010), ISBN: 978-49902880-4-4, pp.755-758.
9. K. Takeda, T. Hattori, Tetsuya Izumi and H. Kawano,
Extended SPRT for Structural Change Detection ofTime Series Based on Multiple Regression Model,International Journal of Artificial Life and Robotics
Springer, ISSN: 1433-5298, 15(4), (2010) 417-420.
10. T. Hattori and H. Kawano, Change Detection Method ofTime Series as an Optimal Stopping Problem -Constructive Proof of Optimal Solution Theorem --, in
Proc. of 2011 International Conference on Biometrics
and Kansei Engineering, (2011), IEEE Computer
Society, ISBN: 978-0-7695-4512-7, pp.100-105.
11. Yoshihide Koyama, Tetsuo Hattori, Hiromichi Kawano,
Model Introduced SPRT for Structural Change
Detection of Time Series (I), in Proc. of The
International Conference on Artificial Life and Robotics(Jan. 2014), ISBN: 978-4-9902880-8-2, pp.252-255.
12. Yoshihide Koyama, Tetsuo Hattori, Hiromichi Kawano,
Model Introduced SPRT for Structural Change
Detection of Time Series (II) --- Kansei Channel andBayes’ Estimation ---, in Proc. of The International
Conference on Artificial Life and Robotics (Jan. 2014),
ISBN: 978-4-9902880-8-2, pp.256-259.
13. Yoshihide Koyama, Tetsuo Hattori, Hiromichi Kawano,“Model Introduced SPRT for Structural Change
Detection of Time Series (I)”, Journal of Robotics,
Networks and Artificial Life (ISSN: 2352-6386), Vol.1,
No.1, pp.55-60 (Atlantis Press, Jun. 2014)
14. Yoshihide Koyama, Tetsuo Hattori, Katsunori Takeda,
Hiromichi Kawano, “Model Introduced SPRT for
Structural Change Detection of Time Series (II)”,
Journal of Robotics, Networks and Artificial Life(ISSN: 2352-6386), Vol.1, No.3, pp.237-243 (Atlantis
Press, Dec. 2014)
15. Yoshihide Koyama, Tetsuo Hattori, Hiromichi Kawano,
Katsunori Takeda, Change Detection Experimentation
for Time Series data by New Sequential Probability
Ratio, Proceedings of International Conference on
Artificial Life and Robotics 2015 (ICAROB 2015),
ISBN 978-4-9902880-9-9, pp.278-281, 2015.

(a) Case 1(b) Case 2

Fig.2. All plotted data : (X,Y )=(I1,I2).
References
1. C. G. E. P. Box and G. M. Jenkins, Time Series
Analysis: Forecasting and Control (Prentice Hall, 1976).
2. Peter J. Brockwell and Richard A. Davis, Introduction
to Time Series and Forecasting, 2nd ed. (Springer;
2003)
3. C. Han, P. K. Willet and D. A. Abraham, Some methods
to evaluate the performance of Page's test as used todetect transient signals, IEEE Trans. Signal processing.
47(8) (1999) 2112-2127.
4. S. D. Blostein, Quickest detection of a time-varying
change in distribution, IEEE Trans. Information Theory.
37(4) (1991) 1116-1122.
5. A. Wald, Sequential Analysis (John Wiley & Sons,
1947).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

378

Application Proposal of Sequential Probability Ratio to Dynamic System State Estimation
Tetsuo Hattori *1, Yoshihide Koyama*1, Yoshiro Imai*1, Yo Horikawa*1

Graduate School of Kagawa University, 2217-20 Hayashi
Takamatsu City, Kagawa 761-0396, Japan
Yusuke Kawakami*2

DynaxT Co., Ltd., 2217-6 Hayashi
Takamatsu City, Kagawa 761-0301, Japan
Hiromichi Kawano*3

NTT Advanced Technology Corporation, 19-18, Nakamachi
Musashino, Tokyo 180-0006, Japan
Takeshi Tanaka*4

Hiroshima Institute of Technology, 2-1-1 Miyake, Saeki-ku
Hiroshima 731-5193, Japan*1{hattori, imai, horikawa}@eng.kagawa-u.ac.jp,*2riverjp2002@gmail.com, *3hiromichi.kawano@ntt-at.co.jp,*4tanaka@cc.it-hiroshima.ac.jp

Abstract
Previously, we have already presented a relation between the SPRT (Sequential Probability Ratio Test) based on
two hypotheses (i.e. Null hypothesis and Alternative hypothesis) and a binary channel in information theory. And
also we have shown that the Bayes’ updating process for obtaining a posteriori probability from the received signals
through binary channel is very similar to the SPRT computing. In this paper, extending the definition of SPRT tothe multi hypotheses condition, we propose to apply the extended SPRT to the internal state estimation of dynamic
system.

Keywords: Time series data, structural change detection, SPRT(Sequential Probability Ratio Test)
time series, we have previously proposed an
application of the SPRT (Sequential Probability Ratio
Test) to the change point detection problem.1-15 We
also have shown the relation between the SPRT basedon two hypotheses (i.e. Null hypothesis and1. Introduction
Time series data analysis is a very important problem
and it has a wide variety of application fields. For astructural change point detection problem in ongoing*2217-20 Hayashi-Cho, Takamatsu, Kagawa 761-0396 Japan© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan379

Tetsuo Hattori, Yoshihide Koyama, Yoshiro Imai, Yo Horikawa, Yusuke Kawakami, Takeshi Tanaka

Alternative hypothesis) and a binary channel in
information theory. Moreover we have shown that theBayes’ updating process for obtaining a posteriori
probability from the received signals through binary
channel is very similar to the SPRT computing.
In this paper, extending the definition of SPRT to themulti hypotheses condition, we propose to apply theextended SPRT to the internal state estimation of
dynamic system.

λnP( Z n | H1 )P( Z n | H 0 )P( y1 | x1 )x1

2.1 Definition
The Sequential Probability Ratio Test (SPRT) is used
for testing a null hypothesis H0 (e.g. the quality isunder pre-specified limit 1%) against hypothesis H1(e.g. the quality is over pre-specified limit 1%). And it
is defined as follows:
Let Z1, Z2, …Zn be respectively observed time series
data at each stage of successive events, the probability
ratio λ n is computed as follows.

P( Z1 | H1 ) P( Z 2 | H 1 )P ( Z1 | H 0 ) P ( Z 2 | H 0 )P( Z n | H1 )P( Z n | H 0 )P( y1 | x2 )

x2

1

, C2

y2

P( y 2 | x2 )The transition matrix (or channel matrix) PBinary isrepresented using conditional probability as follows.

(1)

P ( y1 | x1 ) P( y 2 | x1 )P ( y1 | x2 ) P ( y 2 | x2 )PBinary(4)

Let the prior probability distribution of input event X
be {P0(x1), P0(x2)} as initial values.
Let the observed output sequence { y j1 , y j2 , , y jn }Two positive constants C1 and C2 (C1 < C2) are chosen.
If C1 < n < C2, the experiment is continued by taking
an additional observation. If C2 < n, the process is
terminated with the rejection of H0 (acceptance of H1).
If n < C1, then terminate this process with theacceptance of H0.
C1

P( y 2 | x1 )y1

Fig.1. Binary channel.

where P(Z | H0) denotes the distribution of Z if H0 istrue, and similarly, P(Z | H1) denotes the distribution of
Z if H1 is true.

1(3)

2.2 Relation between SPRT and Bayes’ Updating on
Binary Channel
We regard that the observation and SPRT process forthe aforementioned structural change point detection oftime series data is essentially a kind of Bayes’
Updating by a binary channel as shown in Fig.1, where
input event X=(x1, x2) and output event Y=(y1, y2).

2. SPRT11-15

λn

λ n -1( y jky1 or y2 (k=1, 2,…, n)).

Then, by the first observation of output, the posterior
probability is given by Bayes’ theorem.

p ( x1 | y j1 )(2)

p ( x 2 | y j1 )wheremeans type I error (reject a true null
hypothesis), and means type II error (accept a null
hypothesis as true one when it is actually false).
The definition of λ n can be described in the following
recursive form.

where P ( y j1 )i

P0 ( x1 ) P ( y j1 | x1 )P ( y j1 )P0 ( x 2 ) P ( y j1 | x 2 )P ( y j1 )(5)

P0 ( xi ) P ( y j1 | xi ) .

According to the Bayes’ Updating rule, we have thefollowing updated prior probability {P1(x1), P1(x2)}.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan380

Application Proposal of Sequential

P１( x1 )P１( x 2 )that the conditional probability P(Z=z |S=Si) (i=1,2,3)
is computable.

P0 ( x1 ) p( y j1 | x1 )p ( x1 | y j1 )P ( y j1 )P0 ( x2 ) p ( y j1 | x 2 )p( x2 | y j1 )(6)

P ( y j1 )Similarly, by the second observation, we have

P2 ( x1 )p ( x1 | y j2 )P2 ( x 2 )P ( y j2 )P1 ( x2 ) p ( y j2 | x2 )p ( x 2 | y j2 )where, P ( y j2 )S1

P1 ( x1 ) p ( y j2 | x1 )
(7)

(a)

P ( y j2 )P1 ( xi ) P ( y j2 | xi )i

SiThus, by the nth observation of output, we haveP( zk | Si )P( zk | Si )Pn 1(xi )P( y jn | xi )Pn (xi ) p(xi | y jn )P( y jn )k 1nP( zl | Si )nPn ( x2 )Pn ( x1 )is as follows.

n

P0 ( x2 )P0 ( x1 )

n
k 1
n
k 1

If we assume that

P0 ( x1 )

(b)

(8)

k 1

Then the probability ratioP( zl | Si )(i 1, 2)P( y jk )

Z= zk

Z= zl

Si

n

P0(xi ) P( y jk | xi )S3

S2

P0 ( x 2 )S1

S2

P( y jk | x2 )P( y jk | x1)
1
2

(9)

Si

(10)

Then, we obtain the same ratio as that of SPRT.
Therefore, we consider that the Bayes’ Updating basedon binary channel has a strong relation to SPRT.

SM

(c)

Fig.2. Conceptual images of finite state automaton as adynamic system and the state outputs with conditional
probability. (a) Three states automaton model such asphase change material. (b) Conditional probabilities

3. Application to State Estimation of Dynamic
SystemWe extend the SPRT to apply it to the internal state
estimation of dynamic system.
For example, suppose that the dynamic system is given
the finite state automaton (for example, as shown in
Fig.2 (a)) and that the transition probability from astate to the other state is unclear. However, we assume

P(zk | Si ) and P(zk | Si ) where the state Si and the otherstates output Z (=zk), respectively. (c) General finite
state automaton model where the probability of thestate transition is unclear.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

381

Tetsuo Hattori, Yoshihide Koyama, Yoshiro Imai, Yo Horikawa, Yusuke Kawakami, Takeshi Tanaka

6. H. Kawano, T. Hattori, and K. Nishimatsu, Structural
Change Point Detection Method of Time Series Using
Sequential Probability Ratio Test -Comparison with
Chow Test in the ability of early detection- (inJapanese), IEEJ Trans. EIS. 128(4) (2008) 583-592.
7. G. C. Chow, Tests of Equality Between Sets of
Coefficients in Two Linear Regressions, Econometrica.
28(3) (1960) 591-605.
8. K. Takeda, T. Hattori, T. Izumi and H. Kawano,
Extended SPRT for Structural Change Detection ofTime Series Based on Multiple Regression Mode, inProceedings of the 15th International Symposium on
Artificial Life and Robotics, (Feb. 2010), ISBN: 978-49902880-4-4, pp.755-758.
9. K. Takeda, T. Hattori, Tetsuya Izumi and H. Kawano,
Extended SPRT for Structural Change Detection ofTime Series Based on Multiple Regression Model,International Journal of Artificial Life and Robotics
Springer, ISSN: 1433-5298, 15(4), (2010) 417-420.
10. T. Hattori and H. Kawano, Change Detection Method ofTime Series as an Optimal Stopping Problem -Constructive Proof of Optimal Solution Theorem --, in
Proc. of 2011 International Conference on Biometrics
and Kansei Engineering, (2011), IEEE Computer
Society, ISBN: 978-0-7695-4512-7, pp.100-105.
11. Yoshihide Koyama, Tetsuo Hattori, Hiromichi Kawano,
Model Introduced SPRT for Structural Change
Detection of Time Series (I), in Proc. of The
International Conference on Artificial Life and Robotics(Jan. 2014), ISBN: 978-4-9902880-8-2, pp.252-255.
12. Yoshihide Koyama, Tetsuo Hattori, Hiromichi Kawano,
Model Introduced SPRT for Structural Change
Detection of Time Series (II) --- Kansei Channel andBayes’ Estimation ---, in Proc. of The International
Conference on Artificial Life and Robotics (Jan. 2014),
ISBN: 978-4-9902880-8-2, pp.256-259.
13. Yoshihide Koyama, Tetsuo Hattori, Hiromichi Kawano,“Model Introduced SPRT for Structural Change
Detection of Time Series (I)”, Journal of Robotics,
Networks and Artificial Life (ISSN: 2352-6386), Vol.1,
No.1, pp.55-60 (Atlantis Press, Jun. 2014)
14. Yoshihide Koyama, Tetsuo Hattori, Katsunori Takeda,
Hiromichi Kawano, “Model Introduced SPRT for
Structural Change Detection of Time Series (II)”,
Journal of Robotics, Networks and Artificial Life(ISSN: 2352-6386), Vol.1, No.3, pp.237-243 (Atlantis
Press, Dec. 2014)
15. Yoshihide Koyama, Tetsuo Hattori, Hiromichi Kawano,
Katsunori Takeda, Change Detection Experimentation
for Time Series data by New Sequential Probability
Ratio, Proceedings of International Conference on
Artificial Life and Robotics 2015 (ICAROB 2015),
ISBN 978-4-9902880-9-9, pp.278-281, 2015.

Let P(Z z | Si ) be the conditional probability that

other states than the state Si output Z (=z) as shown in
the following equation.

P( Zz | Si )

m i

P( Z

z | Sm )(11)

Then the extended SPRT for the state Si is defined asfollows

λ in

λ in 1

P ( Z n | Si )P ( Z n | Si )(12)

where λ in means that the ratio of the conditional
probabilities P(Z | Si ) and P(Z | Si ) at the n time point.
For the internal state estimation of dynamic systemsuch as a finite state automaton, we follow the
behavior from the starting state and keep tracking thestate transition. Then the λ it at the time point t canalso be shown as the following equation by two
hypotheses H0 (the current state is unchanged) and H1(the state has changed into the other state).

λ it

λ it 1

P( Z t | H 0 )P ( Z t | H1 )(13)

Accordingly, like the structural change detection
problem, when λ it becomes less than a threshold, we
decide that the state has changed from the former state
S’ into the next state Snext. And we estimate the Snext
should be the state that satisfies the following.

S nextarg max
S S'

P( Z t | S )P( Z t | S )(14)

References
1. C. G. E. P. Box and G. M. Jenkins, Time Series
Analysis: Forecasting and Control (Prentice Hall, 1976).
2. Peter J. Brockwell and Richard A. Davis, Introduction
to Time Series and Forecasting, 2nd ed. (Springer;
2003)
3. C. Han, P. K. Willet and D. A. Abraham, Some methods
to evaluate the performance of Page's test as used todetect transient signals, IEEE Trans. Signal processing.
47(8) (1999) 2112-2127.
4. S. D. Blostein, Quickest detection of a time-varying
change in distribution, IEEE Trans. Information Theory.
37(4) (1991) 1116-1122.
5. A. Wald, Sequential Analysis (John Wiley & Sons,
1947).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan382

An Empirical Evaluation of Grid-based Path Planning Algorithms on Widely Used in Robotics Raspberry Pi
Platform
Anton Andreychuk

Peoples’ Friendship University of Russia (RUDN University),
Moscow, Russia
Andrey Bokovoy

Peoples’ Friendship University of Russia (RUDN University),
Federal Research Center “Computer Science and Control” of Russian Academy of Sciences,
Moscow, Russia
Konstantin Yakovlev

Federal Research Center “Computer Science and Control” of Russian Academy of Sciences,
Higher School of Economics,
Moscow, Russia
E-mail: andreychuk@mail.com, bokovoy@isa.ru, yakovlev@isa.ru

Abstract
We compare three widely used path planning algorithms, A*, Jump Point Search, Theta*, in terms of runtime
performance on a desktop personal computer and on the Raspberry Pi 2 embedded computer widely used inrobotics. We also evaluate the performance of recently introduced path planning algorithm, LIAN, on bothcomputers. Two principal questions are targeted: first) to what extent modern embedded computers are slower thanconventional desktops when solving path finding problems (the answer is – one order of magnitude); second) how
well the former scale up to larger problems (the answer is – in the same way as desktop PCs).
Keywords: path finding, path planning, grid, A*, LIAN, Raspberry Pi.

robotics researcher conducts the experiments, it is likelythat limited number of runs (usually involving a small
number of single-type scenarios) on a real robotic
system equipped with the real robotic hardware (which
can differ a lot from the desktop PC) will be performed(see Refs. 2-4). Both approaches are quite valuable as
the first one helps to answer the question “how good thealgorithm is in comparison to existing competitors (inlarge number of different scenarios)” and the second
one – “how well the algorithm actually works on a realrobot (in limited number of similar type scenarios)”.
Answering these questions is of great value to thecommunity, especially to those researchers aiming at

1. Introduction
Typically when a new grid-based path planningalgorithm is developed robotics is mentioned as one of
the main application. It’s also typical that researcherspresenting the new path finding algorithm evaluate it
empirically (experimentally) using one of the following
approaches (depending on their background). If theexperimental study is conducted by someone withstrong background in heuristic search it is likely thatthousands of tests will be performed on a desktop
computer with the use of the input data gained from thewidespread benchmark collections1. If more-into-the-© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan383

Anton Andreychuk, Andrey Bokovoy, Konstantin Yakovlev

creating robust and fully autonomous mobile robots
capable of solving complicated navigation tasks like theones mentioned in Refs. 5-7.
To the best of authors’ knowledge no survey has
been done so far on how long will it take to conduct
grid-based path planning on modern robot’s hardware incomparison to PC. Our work aims at filling this gap. We
have run plentiful of experiments on both desktop PC
and wide-spread in robotics Raspberry Pi 2 (see Refs. 89) computer. We have evaluated three well-known inpath planning community heuristic search algorithms
A*10, Jump Point Search11, Theta*12 as well as recently
introduced path finding algorithm LIAN13, which can beof particular value to robotics community as it iscapable of finding smoother paths with no sharp turns.

2.3. Algorithms
A* - heuristic modification of Dijkstra algorithm9.
Golden standard in the AI and robotics. A* path is thesequence of traversable adjacent cells and the algorithm
guarantees finding the shortest path.
Jump Point Search (JPS) – modification of A*
which allows the algorithm to skip large portions of thesearch space without reducing the quality of thesolution1. JPS finds paths much faster than A* without
sacrificing the optimality.
Theta* (or Basic Theta*) – any-angle modificationof A* which interleaves exploration of the search spacewith path smoothing11. Theta* path is the sequence ofpairs of traversable, but not necessarily adjacent cells, –sections.
LIAN – recently introduced angle-constrained path
planning algorithm that searches for a path as asequence of sections (like Theta*) holding the property
that an angle between any two consecutive sections
doesn’t exceed predefined threshold12. LIAN is ofparticular interest to the robotics community as it iscapable of finding smooth paths without sharp turns thatcan be easily followed by mobile robots. Due to thelarger state-space LIAN is notably slower thanA*/Theta*/JPS and in some cases its runtime exceeds
reasonable threshold (1-10s).
All algorithms were coded in OS independent C++.
In our implementation we explicitly generate elementsof the OPEN and CLOSE lists and use vector of lists
and unordered map to store them.

2. Experimental Setup
2.1. Hardware
Two computers and two operating systems were used.
First was a conventional desktop PC: Intel Core i7-4790
3.6 GHz with 16 GB of RAM run by Linux Debian 8
Kernel 3.16.7. Second was the compact, widely used inrobotics Raspberry Pi 2 Model B computer equippedwith 900MHz quad-core ARM Cortex-A7 CPU and
1GB of RAM run by Linux Raspbian Kernel 4.1.7
PREEMPT_RT.
2.2. Map and tasks
The following map bundles from the N. Sturtevant’s
movingai collection1 were used: Baldur’s Gate (75maps), Warcraft III (36 maps), Random obstacles 10%,25%, 40% fill (30 maps), Mazes corridor width 2, 4, 8
(30 maps), Rooms size 16, 32, 64 (30 maps). We also
run experiments on the City maps collection introduced
by us. We have extracted 100 fragments of the real city
environments from www.openstreetmaps.org sized 1
347 x 1 347 m each and discretized them to 501 x 501
grids. Cells corresponding to areas that overlap
buildings were marked un-traversable. Five different
start-goal locations for each map were fixed. Thus in
total 1505 path planning tasks comprised the testcollection.

3. Results
3.1. First series of the experiments
In the first series of the experiments JPS, A* and Theta*
were executed to solve all the available 1505 path
planning tasks. Here we present aggregated average
figures (all times are in milliseconds).
As one can see path planning on Raspberry Pi isgenerally 10 times slower than on a desktop PC, but theexact value of average runtime difference varies slightly
from collection to collection from algorithm toalgorithm. We believe that there are two reasons for
that: first) measuring timestamps for a single run is not© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan384

An Empirical Evaluation of

100% accurate, and second) operating system influences
the runtime (via interruptions, for example).
Strictly speaking, this means that it is not possible ingeneral to 100% accurately measure the algorithm’s
runtime. Definitely, this question should be investigatedfurther, but as for now, it goes beyond the scope of thepaper. We would like to mention though, that we
believe the figures obtained by us are relevant andcomparable as we run each single experiment (=particular algorithm + particular map + particular startand goal locations) several times and use the averageruntime for further analysis (see table 1).

on the map type and on the angle constraint) LIAN on
Raspberry was not able to find a solution within 10
seconds limit. As for the desktop PC – less than 1.2% oftasks were not processed within the cap. In general,
Baldur’s Gate collection is the toughest for LIAN towork on, while tasks from Warcraft III are handled by
LIAN more effortlessly and City tasks even more
effortlessly.
Table 2. LIAN averaged runtimes and success
rate.
W III

Table 1. A*, JPS, Theta* averaged runtimes (inmilliseconds).
Mazes

RandomObstacles

Rooms

2.52

11.58

36.54

5.82

JPS (ras)

23.46

110.37366.2950.95

A* (pc)42.86

164.9566.64

164.33A* (ras)509.311844.98739.001996.8083.29

258.3256.55

268.48749.522544.49582.102664.54

City/Game
Maps
JPS (pc)

Theta*(pc)
Theta*
(ras)

BG

CM

t

sr

t
20

sr

t

sr

LIAN (pc)

99.0a

100%

407.0

100%

28.5

100%

LIAN (ras)

336.2

97,6%

1 131.3

89,4%

140.2

95,5%

25

LIAN (pc)

77.1

100%

645.2

99.7%

21.7

99.9%

LIAN (ras)

424.3

98.3%

1 486.1

82.2%

115.7

93.2%

30

LIAN (pc)

79.6

100%

663.9

98.9

23.6

99.6

LIAN (ras)

321.2

96.5%

1 473.6

79.3

126,1

91.8%

Please note that runtimes are directly incomparable
between pc and ras as the number of solved tasks is different.

Next question we were interested in was how well
does Raspberry Pi scale up to larger problems. To
answer this question, the dependency between thenumber of steps (expansions) and the resultant runtime
should be analyzed – see figure 1.

3.2. Second series of the experiments
For LIAN evaluation the angle constraint was set
consecutively to 20 , 25 and 30 (these values wereadvised by peers involved in unmanned aerial vehicle(UAV) controllers design). LIAN’s other parameters
were set as advised in Ref. 15, i.e. Δ=5, w=2.
Experiments were run on Game and City maps only. As
for the Baldur’s Gate, Warcraft III and City maps
collections – more than 95% of all tasks within thesecollections are solvable for LIAN with angle constraints
equal or above 20 . As it’s known that LIAN is timeconsuming algorithm we set the cut-off time for eachexperiment to be 10 seconds. If the algorithm was notable to find a path within allotted time, the result for thisrun was considered to be failure. Averaged results arepresented in table 2 (failures were not taken into
account when measuring runtimes).
In the table t – is time (in ms), and sr – is successrate. As one can see in up to 20% of cases (depending

110
100
90
80
70
60
50
40
30
20
10
0

JPS (pc)

JPS (ras)

Fig. 1. Time (Y-axis, in milliseconds) needed to accomplish
the exact number of steps (X-axis) of JPS algorithm. Inflated
values for pc are used for better scaling.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan385

Anton Andreychuk, Andrey Bokovoy, Konstantin Yakovlev

semi-structured environments.” The International
Journal of Robotics Research, 29(5), 2010, pp. 485-501.
3. Y. Kuwata, S. Karaman, J. Teo, E. Frazzoli, J. P. How
and G. Fiore, Real-time motion planning withapplications to autonomous urban driving. IEEE
Transactions on Control Systems Technology 17(5), 2009,
pp. 1105-1118.
4. E. Magid and E. Rivlin, “CAUTIOUSBUG: A
competitive algorithm for sensory-based robot
navigation”. Proceedings of the 2004 IEEE/RSJ
International Conference on Intelligent Robots andSystems, Vol. 3, 2004, pp. 2757-2762.
5. A.I. Panov and K. Yakovlev, “Behavior and path
planning for the coalition of cognitive robots in smart
relocation tasks”. In Proceedings of The 4th International
Conference on Robot Intelligence Technology andApplications (RiTA 2015), 2015, pp. 3-20.
6. E. Magid, R. Lavrenov, A. Khasianov. Modified splinebased path planning for autonomous ground vehicle.
International Conference on Informatics in Control,
Automation and Robotics, (Madrid, Spain, 2017), pp.
132-141.
7. V. Karpov, A. Migalev, A. Moscowsky, M. Rovbo, V.
Vorobiev. Multi-robot Exploration and Mapping Basedon the Subdefinite Models. In: Ronzhin A., Rigoll G.,
Meshcheryakov R. (eds) Interactive Collaborative
Robotics (ICR 2016). Lecture Notes in Computer Science,
vol 9812. 2016, Springer, Cham.
8. J. Betthauser, D. Benavides, J. Schornick, N. O'Hara, J.
Patel, J. Cole and E. Lobaton, “WolfBot: A distributed
mobile sensing platform for research and education”.
Proceedings of IEEE 2014 Zone 1 Conference of theAmerican Society for Engineering Education, 2014, pp.
1-8.
9. J. D. Brock, R. F. Bruce and M. E. Cameron, “Changing
the world with a Raspberry Pi”. Journal of Computing
Sciences in Colleges, 29(2), 2013, pp. 151-153.
10. P.E. Hart, N.J. Nilsson and B. Raphael, “A formal basis
for the heuristic determination of minimum cost paths”.
IEEE Transactions on Systems Science and Cybernetics,
4(2), 1968, pp. 100–107.
11. D. Harabor and A. Grastien, “Online graph pruning for
pathfinding on grid maps”. Proceedings of the 25th AAAI
Conference on Artificial Intelligence (AAAI-11), 2011,
pp. 1114-1119.
12. A. Nash, K. Daniel, S. Koenig and A. Feiner, “Theta*:
any-angle path planning on grids. In Proceedings of the
22nd AAAI conference on Artificial intelligence (AAAI2007), 2007, pp. 1177-1183.
13. K. Yakovlev, E. Baskin and I. Hramoin, “Grid-based
angle-constrained path planning”. Proceedings of the
38th Annual German Conference on Artificial
Intelligence (KI-15), (Dresden, Germany, 2015), pp. 208221.

Figure 1 depicts results for JPS algorithm for theGame and City maps collections as otheralgorithms/collections demonstrate similar tendencies
and dependencies. PC runtimes are multiplied by 9,3(the exact average difference value between PC andRaspberry runtimes on the considered collections) for
better scaling.
As one can see dependency between number ofsteps and runtime is the same on PC and Raspberry, so
one can say that Raspberry Pi 2 scales to larger path
planning problems in the same way as desktop PCs.
Non-monotone pattern observed on the chart can be
explained in the following way. Cost (runtime) of asingle algorithm’s step is not the same across different
path planning tasks due to the fact that the number of“insert (update) to OPEN” and “lookup in CLOSE”
operations varies from one step to the other (anddepends on the properties of the exact map and startgoal locations).
4. Summary
Analyzing all the obtained experimental data we canmake the following conclusions. Solving path planningtasks on the wide spread in robotics community
Raspberry Pi 2 computer is one order of magnitude
times slower than on conventional desktop PCs. If anyangle (or angle-constrained) paths are targeted than it’s
likely that path planning will take more than 1-2
seconds (which can be sensible for some practicalapplication). Raspberry scales up to larger path planningproblems in the same way desktop PC does. Aninteresting question for further investigation is – how tocorrectly measure algorithm runtimes as it looks likethat they are subject to certain (or uncertain?) deviations
caused by the operating system’s influence.
Acknowledgments
This work was supported by the Russian Foundation for
Basic Research (Project No.17-07-00281).
References
1. N. Sturtevant, “Benchmarks for grid-based pathfinding”.
Transactions on Computational Intelligence and AI inGames, 4(2), 2012, pp. 144-148.
2. D. Dolgov, S. Thrun, M. Montemerlo and J. Diebel,“Path planning for autonomous vehicles in unknown© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Oita, Japan386

Development of the Insectoid Walking Robot with Inertial Navigation System
Vitaly Egunov, Andrey Kachalov, Michail Petrosyan, Pavel Tarasov, Elena Yankina
Computers and Systems Department, VSTU, Leninaav, 28,
Volgograd, 400005, Russia
E-mail: vegunov@mail.ru, dmn.info@mail.ru, mih95@inbox.ru, tarasradio@mail.ru
www.vstu.ru

Abstract
The article is devoted to the problem of controlling an insectoidwalking robot- hexapod. An inertial navigation
system based on the accelerometer ADXL345 and the gyro ITG-3200 is used in the developed robot. The options
for filtering data obtained from the inertial sensors and the methods of combining the accelerometer and gyroscope
data to determine the roll and pitch are under consideration in the article. The original software has been designed
to visualize the mobile robot position. For this purpose, the authors have developed a microcomputer program thattransmits the calculated total deviations (roll and pitch) over a wireless Wi-Fi network to a personal computer
which runs the developed program that process the data. The obtained data are processed and used to display an
object implemented with the help of OpenGL library.
Keywords: accelerometer, gyroscope, inertial navigation, mobile robot - hexapod, discrete integration,complementary filter.

such as debris, tough terrain, also inside buildings where
it is necessary to move along narrow corridors,
staircases and mines [4].
A team of authors have designed and made a walking
robot-hexapod, which has been tested on being
controlled by various approaches. The robot wasdeveloped on the basis of a single-board computer
Raspberry Pi 3 Model B and the inertial sensor GY-85,
which includes the accelerometer ADXL345 and thegyroscope ITG-3200. The designed 3D model of therobot prototype was printed with the 3D printer at theComputer and System Department, Volgograd State
Technical University. The functioning robot prototype
is shown in Figure 1.

1. Introduction
Currently, research in robotics is extremely relevant.
There is a huge number of robots that replace a humanin high risk areas (whether it is a manufacturing
workshop or technogenic catastrophe liquidation),deliver people from the necessity of doing monotonous
work, make our life easier. More sophisticated models
capable of analyzing information [1] and making
decisions, recognizing objects and manipulating themappear all the time.
Walking robots are actively designed andresearched[2,3]. This is due to the fact that walking
structures have a number of advantages compared toclassical wheeled and caterpillar vehicles, for example,
when moving on the difficult geographic terrain surface,©The 2018International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4,B-Con Plaza, Beppu, Oita,Japan387

Vitaly Egunov, Andrey Kachalov, Michail Petrosyan, Pavel Tarasov, Elena Yankina

enough to solve the task. In this project we havecombined the both sensors readings using acomplementary filter.
A personal computer program has been developed tovisualize the object position using the OpenGL library.
OpenGL is one of the most popular application softwareinterfaces for developing applications in both 2D and
3D graphics. In the given program, this library is used tovisualize the robot platform to vividly demonstrate theinertial navigation sensors functioning. The simulated
platform is a parallelepiped with faces of different
colors, which responds to sensor deviations (along the x
and y axes). The values of the roll and pitch aredisplayed on the screen together with the data obtained
as a result of processing the data from the accelerometer
along the x and y axes, as well as the angular values of
the gyroscope in three axes.
To display an object, the figure is drawn on the faces,
and the color is specified with the glColor3f (r, g, b)function. The simulated platform rotation of is
implemented using the function glRotatef (Angle, Xtrue,
Ytrue, Ztrue) (Figure 2).

Fig.1 Developedrobot-hexapod

2. Robot Control
One variant of robot controlling is to process the inertial
sensor data and then to generate control actions overservo drives. The interaction with the inertial sensor
connected to the Raspberry Pi microcomputer occurs
via the I2C interface. Before reading the data, themodule is configured to the required operating mode byrecording the initializing values to the registers of themodule. After this, the data are periodically read fromthe accelerometer ADXL345 and the gyroscope ITG3205.To remotely monitor the current state of the robot,
the resulting platform deviations are transmitted via thewireless Wi-Fi network from Raspberry Pi to a personal
computer where one can track the object deviations on
the constructed 3D object (a cube with multi-colored
faces) that is implemented using the OpenGL library inC++.
To determine the object tilt angles, one can use both thegyroscope and the accelerometer. In the case of theformer, the tilt angle is determined by discrete
integration of the rotation speed, which is determined by
the gyroscope. However, this sensor has notabledrawbacks which are zero drift and gradual error
accumulation in calculating the angle. In the case of theaccelerometer, its use is justified for determining the tilt
angles of fixed objects. If some force acts on the object,
one cannot determine the object orientation using theaccelerometer only. I.e. it can be concluded that using
the gyroscope or the accelerometer separately is not

Fig.2 The program result on a personal computer
In addition to the drawn figure, the display shows aconsole window, in which the roll and pitch values are
displayed for clarity, which in turn are written to the csv
file and can be used for the further analysis. According
to the obtained data, graphs are constructed (Fig. 3), thegraphs illustrate the platform deviation along thecorresponding axes. The charting program is written in
Python using the Matplotlib data visualization library.

©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4, B-Con Plaza, Beppu, Oita,Japan388

Development of the Insectoid

Referring to the diagram, we can see that the input of
the FRUND system receives the robot mathematicalmodel (the structure of its links, nodes). The link systemthat determine the type of the robot movements is added
to the mathematical model. Applied to the hexapod, it
can be walking forward with a certain gait, turning left,
etc. At the same time, a system of equations is formed,
the result of which is a set of trajectories for each robot
engine that can be used for direct control.
When forming motions (trajectories), the inverse
dynamic problem method is used. It allows by setting
the displacements of the robot finite nodes in time toreceive the movements of all the other nodes. Moreinformation about this method application forcontrolling an android can be found in.
To control the robot-hexapod various types of gaits andmotion directions that can be performed by the robot
have been generated.The mediator between these files
and the robot is the control system. Its structure isshown in modules in Fig. 5.
The purpose of the modules:
Hexapod Instance encapsulates the work logic withthe robot hardware, the input takes angles to control
the drives;
Hexapod provides methods for controlling drives, isused to perform robot configuration;
Packet master implements generating packets
methods for transmission to the robot; interface
allows to specify angles for one drive;
Serial Port Master implements transmitting
packages to the robot via the Serial port;
Storage stores the robot drives and system variables
configuration;
FRUND Generator implements reading the
movements file and execution the movements witha given speed;
Console Interface implements control from theconsole, allows running the program with the
motion file parameter.

Fig. 3 – Graph obtained from the data file res.csv
In this project sockets were chosen as a means of
networking between the PC and Raspberry Pi 3. And on
the server side the program runs on the Raspbian OS,
and the client uses Windows.
3. Alternative Robot Control using FRUND software
As an alternative approach to robot control, theMultibody Dynamics System FRUND was used, it
allows to solve this problem and, on the basis of thewalk generator, to obtain motions trajectories for robot
engines.By far, robots such as the anthropomorphic
robot [2, 5], the robot-hexapod [3] and other types ofrobots [6- 8] have been modeled using the FRUND
system[6]. In this paper, we will consider the motion
synthesis for controlling the robot-hexapod, thedescription of the software architecture for robot
control.In general, the robot control diagram with theuse of FRUND can be represented, as shown in Fig. 4.

Fig. 4 Diagram of robot control with the use of FRUND.
©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4, B-Con Plaza, Beppu, Oita,Japan

389

Vitaly Egunov, Andrey Kachalov, Michail Petrosyan, Pavel Tarasov, Elena Yankina

Serial port
masterPacket master

Hexapod CoreFRUNDGenerator

Administration of Volgograd region (project #16-47340385)

Generator
GUI

References

Storage

Hexapod

GUI Interface

Hexapod
Instance

ConsoleInterface

1. Robotic Complex Central Processing Node Performance
Requirements Assessment / Egunov V.A., Kirnosenko
S.I., Andreev A.E. // World Applied Sciences Journal(WASJ). - 2013. - Vol. 24, Spec. Issue 24 : Information
Technologies in Modern Industry, Education & Society. C. 37-42.
2. Gabbasov, B., Danilov, I., Afanasyev, I., and Magid, E.
(2015, December). Toward a human-like biped robot
gait: Biomechanical analysis of human locomotion
recorded by Kinect-based Motion Capture system. In
Mechatronics and its Applications (ISMA), 2015 10th
International Symposium on (pp. 1-6). IEEE.
3. Zhu, Y., Guo, T., Liu, Q., Li, Q., Yan, R.A study ofarbitrary gait pattern generation for turning of a bioinspired hexapod robot. // Robotics and Autonomous
Systems. – 2017. Vol. 97, с. 125-135.
4. Deng, H., Xin, G., Zhong, G., Mistry, M Gait andtrajectory rolling planning and control of hexapod robots
for disaster rescue applications //Robotics andAutonomous Systems. – 2017. Vol. 95, с. 13-24.
5. Yao, P., Li, T., Luo, M., Zhang, Q., Tan, Z. Mechanism
Design of a Humanoid Robotic Torso Based on Bionic
Optimization // International Journal of Humanoid
Robotics - 2017. Vol. 14(4),1750010
6. Gorobtsov A. S. Synthesis of parameters of controlled
motion of multi-link mechanical systems of arbitrary
structure by the inverse problem method / A. S.
Gorobtsov // Mechatronics, Automation, Control.– 2004.
– №6. – pp. 43-50.
7. Synthesis of Control Algorithm and Computer
Simulation of Robotic Manipulator-Tripod / Nesmianov
I., Vorob'Eva N., Dyashkin-Titov V., Zhoga V.,
Skakunov V., Terekhov S., Egunov V., Al-Hadsha F.A.H.
// Creativity in Intelligent Technologies and Data Science.
CIT&DS 2015 : First Conference (Volgograd, Russia,
September 15-17, 2015) : Proceedings / ed. by A. Kravets,
M. Shcherbakov, M. Kultsova, O. Shabalina. –[Switzerland] : Springer International Publishing, 2015. –P. 391-403. – (Ser. Communications in Computer andInformation Science. Vol. 535).
8. Sun, T., Xiang, X., Su, W., Wu, H., Song, Y. A
transformable wheel-legged mobile robot: Design,
analysis and experiment // Robotics and Autonomous
Systems. – 2017. Vol. 98, с. 30-41

Fig. 5 – Software structure for robot control
Thus, to control the robot means to run the control
program with a parameter (motion file), to search theport to which the robot is connected and, afterconnecting to the robot, to execute movements from thefile.It is also possible to control via the graphicalinterface. In this case, the operator is able to change theengine settings and their layout.
The development of the control process involveschanging the control scheme as follows:
files generated by the FRUND system will beplaced in a data storage device that will be installedin the robot;
motion control of the robot will be done directlyfrom the robot's board instead of files;
a remote control will be connected to the robot
board via a wireless channel, it will allow changing
the movement type, direction and switching robot
between manual and automatic control.
4. Conclusions
To sum up the following can be said:- the FRUND
system allows to control various robots of complex
structure;- in the simplest case, control can be done bywriting a short interface program that converts the
movement files into robot commands;- future tasks
include the implementation of feedback systems toenhance adaptive control capabilities and theimplementation of autonomous control.
Acknowledgements
Work is performed with the financial support of theRussian Foundation for Basic Research and the©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4, B-Con Plaza, Beppu, Oita,Japan390

Enhancing semi-dense monocular vSLAM used for multi-rotor UAV navigation in indoor
environment by fusing IMU data
Andrey Bokovoy

Lab “Intelligent dynamic systems”, Federal Research Center “Computer Science and Control” of Russian Academy of
Science, ul. Vavilova, 44/2, Moscow, Russia, 119333
Moscow, Russia
Faculty of Science, Peoples’ Friendship University of Russia (RUDN University), ul. Miklukho-Maklaya, 6, Moscow,
Russia, 117198
Moscow, Russia
Konstantin Yakovlev

Lab “Intelligent dynamic systems”, Federal Research Center “Computer Science and Control” of Russian Academy of
Science, ul. Vavilova, 44/2, Moscow, Russia, 119333
Moscow, Russia
Faculty of Computer Science, Higher School of Economics, ul. Myasnitskaya, 20, Moscow, Russia, 101000
Moscow, Russia
E-mail: {bokovoy,yakovlev}@isa.ru, 1042160097@rudn.university, kyakovlev@hse.ru
eng.rudn.ru
www.hse.ru

Abstract
We propose the enhancement of the modern vision-based monocular simultaneous localization and mapping(vSLAM) method, e.g. LSD-SLAM, used for compact multi-rotor UAV indoor navigation, by fusing inertial
measurement unit (IMU) data with camera images. We suggest removing the cost-expensive loop-closure
optimization algorithm from the vSLAM pipeline and replacing it with the computationally efficient flow
estimation procedure based purely on IMU data. The input IMU flow is being processed by the Extended Kalman
Filter (EKF) based techniques for localization purposes and used further in LSD-SLAM algorithm for UAV pose
estimation. We evaluate the proposed algorithm using the modeled indoor environment originally used for
RoboCup Rescue Simulation League 2013 competition and “hector_quadrotor” – commonly used in modelling
simulated UAV model. Evidently, implementation of the suggested enhancement results in significant drop-downof the runtime and leads to obtaining maps and trajectories of higher accuracy.
Keywords: vision-based SLAM, IMU, monocular camera, navigation, UAV.

associated with micro UAV, e.g. small size, low energy
battery capacity, low wireless channel capacity, low
performance of on-board computers a challenging task
is to improve the performance of such methods timewise while keeping the localization and mapping
accuracy at the same or better level then in original
methods.

1. Introduction
Nowadays, the vision-based monocular simultaneous
localization and mapping (vSLAM) methods andalgorithms are widely used for semi-autonomous andautonomous navigation of micro unmanned aerial
vehicles (UAV)1,2,3,4,5. Considering the restrictions© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

391

Andrey Bokovoy, Konstantin Yakovlev

One of the bottlenecks of the vSLAM methods is theloop-closure detection6. Loop-closure detection
algorithms perform global map and trajectory
optimization on the basis of comparison of previously
received and current data7,8, especially the data relative
to the position the system has already been to. Thatmechanism provides more accurate trajectory estimationand mapping in a long run but takes a significant
amount of time to proceed. The main question behind
this paper is whether we can replace the loop-closurealgorithm in modern vSLAM method with IMU data
fusion while decreasing processing time and keeping thesame or better map and trajectory accuracy.
The main motivation is to step towards using such amethod on board of micro UAV operating in indoor,
GPS-denied environments. We provide the experimental
evaluation of IMU+vSLAM method in simulated
environment.

3.2. Simulation
We use Gazebo for robot an environment simulation.
For indoor flight simulation we choose quadcopter“hector_quadrotor”. This UAV model has a wide arrayof onboard sensors (including IMU, cameras, laser
scanners and etc.) suitable for navigation tasks. IMU
operates with 200Hz frequency while camera is able toproduce a 640x480pix images with 30FPS. All thecommunication with quadcopter goes through ROS. The
algorithm is in charge of mapping, localization andnavigation processes. That way, we simulate the offboard computation scenario, when the heavy algorithms
processed on control station while UAV’s fly controller
only takes care of basic stabilization tasks, movement
and sensor data output.
Indoor environment involved in the experiment is
the model of National Institute of Standards andTechnologies that was used during RoboCup Rescue
Simulation League 2013 for Gazebo simulating
environment (fig. 1). Overall square of the environmentis 6 300m2 consisting of multiple rooms and doors.
We used 10 different for the simulation. The
trajectories’ length varied from 25m to183m. There
were 7 trajectories with loops and 3 without them. The
UAV was expected to fly with 1m/s speed. The IMU

2. Approach overview
We use LSD-SLAM9 as the core method for visionbased localization and mapping of micro UAV. We
have chosen this method based on evaluations from Ref.
10, as the one that produces highly detailed dense map
and accurate trajectory while processing video stream in
real-time on modern PC. In order to replace the loopclosure procedure inside the LSD-SLAM method, wefuse the micro UAV IMU data with the position
coordinates from received LSD-SLAM using Extended
Kalman Filter (EKF)11 techniques. This way trajectoryand map is computed considering the fused position
data and the loop-closure algorithm is no longer a partof LSD-SLAM processing pipeline.
3. Experimental evaluation
3.1. Algorithm implementation

Fig. 1. The model of National Institute of Standards andTechnologies for RoboCup Rescue Simulation League
competition.

We implemented our algorithm for wildly used inrobotics framework – Robot Operating System (ROS).
The framework provides a modular structure fordifferent components (nodes) and the communication
mechanism (topics, services), that keeps all the inneralgorithms independent and extendable. We modified
LSD-SLAM open-source code, provided by its authors,
to disable loop-closure algorithm and added some minorchanges and statistics gathering.

data flow is part of Gazebo UAV simulation model. For
IMU and vSLAM position data fusion we use the“robot_pose_ekf” node. The main criterions forevaluation are processing speed, the accuracy of theestimated trajectory and the accuracy of the map, that is© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan392

Enhancing semi-dense monocular vSLAM

Fig. 3 shows the average deviation from groundtruth of trajectories built by original LSD-SLAM andIMI+LSD-SLAM algorithms.
As we can see, the trajectory accuracy of originalalgorithm is on pair with the presented method. The

measured by median deviation from ground-truth map
over all the points of vSLAM’s output map.
4.

Experimental results

Table 1 presents the results of the experiment. Original

Table 1. The experimental results.
Scenario

Length(m)

Loopclosures

traj_1
traj_2
traj_3
traj_4
traj_5
traj_6
traj_7
traj_8
traj_9traj_10

25,3
28,9
39,2
58,3
59,2
91,5
97,2
101
110,2
125,3

+
+
+
+
+
+
+

Average processing time(sec)
LSDSLAM

0,189
0,211
0,231
0,288
0,29
0,192
0,291
0,295
0,185
0,315

IMU+LSDSLAM

0,137
0,141
0,156
0,135
0,143
0,149
0,147
0,143
0,159
0,141

Average deviation fromground-truth trajectory(m)
LSDIMU+LSDSLAM
SLAM

0,23
0,18
0,33
0,38
0,41
0,81
0,69
0,71
0,73
0,78

0,24
0,13
0,34
0,34
0,37
0,6
0,63
0,65
0,66
0,69

Average deviation fromground-truth map (m)LSDSLAM

0,35
0,52
0,57
0,56
0,62
0,93
0,85
0,86
1,12
0,98

IMU+LSDSLAM

0,34
0,5
0,51
0,53
0,57
0,71
0,77
0,78
0,89
0,91

only significant difference occurs with traj_6 scenario
due to complex path that consists of sharp rotations ofUAV. Original LSD-SLAM loses its tracking during
such movements and it forces LSD-SLAM to reallocate
the UAV’s position which leads to the reduced accuracy.

and modified (with IMU data fusion) LSD-SLAM
algorithms are compared. The results are also presented
as plots (fig.2, fig.3) for better visualization.
As shown on fig. 2, the original SLAM and
IMU+SLAM processing times grow with the length of
the trajectory (see table 1 for the lengths values).
However, the original LSD-SLAM’s processing time on
trajectories with loops is drastically higher, compared to
non-loop scenarios. That is the result of loop-closurealgorithm’s global map and trajectory optimization
process. In non-loop scenarios (traj_1, traj_6, traj_9)
IMU+LSD-SLAM’s processing time is slightly less that
the LSD-SLAM’s one.

Fig. 3. The average deviation from ground-truth trajectory of
LSD-SLAM and IMU-LSD-SLAM. Red dots represent thedeviation of non-loop scenarios.

Fig. 4 shows the average deviation of map,compared to ground truth. This measurement represents
the accuracy of the map built by LSD-SLAM and
IMU+LSD-SLAM algorithms.
IMU+LSD-SLAM is on pair with original LSDSLAM except the results of traj_6 and traj_9 scenarios.
The traj_6 map accuracy is influenced by sharp UAV

Fig. 2. The average processing time of LSD-SLAM and
IMU+LSD-SLAM algorithms. Red dots represents speed of
non-loop scenarios.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

393

Andrey Bokovoy, Konstantin Yakovlev

maneuvering and traj_9 represents the accuracy loopclosure algorithm during long length flights.

and Systems (IROS), 2012 IEEE/RSJ International
Conference on, pp. 4557-4564. IEEE, 2012.
3. Yang, Shaowu, Sebastian A. Scherer, and Andreas Zell.
Robust onboard visual SLAM for autonomous MAVs.
In Intelligent Autonomous Systems 13, pp. 361-373.
Springer International Publishing, 2016
4. Buyval A, Gavrilenkov M. Vision-based pose estimation
for indoor navigation of unmanned micro aerial vehicle
based on the 3D model of environment. In MechanicalEngineering, Automation and Control Systems (MEACS),
2015 International Conference on 2015 Dec 1 (pp. 1-4).
IEEE.
5. Yakovlev, K., Khithov, V., Loginov, M., Petrov, A.
Distributed control and navigation system for quadrotor
UAVs in GPS-denied environments. In: Filev D. et al.
(eds) Intelligent Systems'2014. Advances in Intelligent
Systems and Computing, vol 323. pp. 49-56. Springer,
Cham
6. Bokovoy A., Yakovlev K. Original Loop-closure
Detection Algorithm for Monocular vSLAM. In Analysis
of Images, Social Networks and Texts, 2017 6thinternational conference, AIST 2017, Moscow, Russia,
July 27-29, p.202-213
7. Angeli, A., Filliat, D., Doncieux, S. and Meyer, J.A.,
2008. Fast and incremental method for loop-closure
detection using bags of visual words. IEEE Transactions
on Robotics, 24(5), pp.1027-1037.
8. Vokhmintcev A., Yakovlev K. (2017) A Real-Time
Algorithm for Mobile Robot Mapping Based on
Rotation-Invariant Descriptors and Iterative Close Point
Algorithm. In: Ignatov D. et al. (eds) Analysis of Images,
Social Networks and Texts. AIST 2016. Communications
in Computer and Information Science, vol 661. pp. 357369.
9. Engel J, Schöps T, Cremers D. LSD-SLAM: Large-scaledirect monocular SLAM. InEuropean Conference on
Computer Vision 2014 Sep 6 (pp. 834-849). Springer,
Cham.
10. Buyval, A., Afanasyev, I., & Magid, E. (2017, March).
Comparative analysis of ROS-based Monocular SLAM
methods for indoor navigation. In Ninth International
Conference on Machine Vision (pp. 103411K-103411K).
International Society for Optics and Photonics.
11. Ljung L. Asymptotic behavior of the extended Kalmanfilter as a parameter estimator for linear systems. IEEE
Transactions on Automatic Control. 1979 Feb;24(1):3650.

Fig. 4. The average deviation from ground-truth map of LSDSLAM and IMU-LSD-SLAM. Red dots represent thedeviation of non-loop scenarios.

5.

Conclusion

We have proposed to enhance the original monocular
LSD-SLAM method by removing time-consuming loop
closure detection procedure and replacing it with the
IMU fusion technique. Conducted experimental
evaluation shows that such an enhancement leads tobetter runtimes (30% on average, reaching 44,5% forsome scenarios), while improving the accuracy of theestimated map and trajectory (7% and 10% accuracy
increase respectively compared to original LSD-SLAM
algorithm).
In future we plan to use the proposed method forreal-time micro UAV navigation in indoor, GPS-denied
environments.
Acknowledgements
The research was partially supported by RussianFoundation for Basic Research (RFBR), projects №№
15-07-07483, 17-29-07053.
References
1. Blösch, Michael, Stephan Weiss, Davide Scaramuzza,
and Roland Siegwart. Vision based MAV navigation in
unknown and unstructured environments. In Robotics andautomation (ICRA), 2010 IEEE international conferenceon, pp. 21-28
2. Fraundorfer, Friedrich, Lionel Heng, Dominik Honegger,
Gim Hee Lee, Lorenz Meier, Petri Tanskanen, and Marc
Pollefeys. Vision-based autonomous mapping andexploration using a quadrotor MAV. In Intelligent Robots© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

394

Method of Finding of Android Program Motion for the ZMP Trajectory of a Certain TypeAlexander Gorobtsov

Higher Mathematics Department, VSTU, Lenina av, 28,
Volgograd, 400005, Russia
Pavel Tarasov, Andrey Skorikov, Alexey Markov, Andrey Andreev

Computers and Systems Department, VSTU, Lenina av, 28,
Volgograd, 400005, Russia
E-mail: vm@vstu.ru , markovalex95@gmail.com, scorpion_energy@mail.ru,
tarasradio@mail.ru, andan2005@yandex.ru,
www.vstu.ru

Abstract
We consider the problem of finding of the programmed motion of biped robot with a given ZMP trajectory. It is
assumed that it provides a stable ZMP trajectory of the robot motion. The equations of the robot motion consist of
the equations of the robot dynamics itself and the stability conditions equations, which include the coordinates of
the ZMP trajectory. It is proposed to seek the solution of stability equations on a limited set of some concerted
movements of the robot. Several types of such coordinated movements are considered. The application of thismethod for the android with a mass of 60 kg and height of 160 cm is described. The parallel algorithms of thecomputational aspect of the method is presented.
Keywords: biped robot, ZMP trajectory, programmed motion, MBS modeling.

1. Introduction

2. Theoretical description

The control strategy of biped robots dynamic walking
is based on the walking pattern generation, which
considers the stable program zero moment point (ZMP)
trajectory. One of the reasons of difference between theactual ZMP trajectory and the program ZMP trajectoryis the imperfect dynamic model of the robot. There aremany research works on the walking pattern generationof humanoid robots [1-4] derived from simple dynamic
models of the robots. In this paper, a generator ofprogram ZMP trajectory for biped humanoid robots
using the multibody systems dynamic (MBS) methods
is investigated.

The dynamic model of the robot can be expressed in theform:
Mx DTpDxf (x, x, t ) u(t )h(x, x)(1)

In equations above x is the state vector of dimension
n, M is an inertia matrix, f ( x, x, t ) is the external
forces vector, u(t ) is a control forces vector, D is thematrix of the variable coefficients of the constraints
equations with the dimension k n , k is number of
constraints equations, h(x, x) is right side constraints
equations vector, p is Lagrange multipliers vector.

©The 2018International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4,B-Con Plaza, Beppu, Oita,Japan395

Alexander Gorobtsov, Pavel Tarasov, Andrey Skorikov, Alexey Markov, Andrey Andreev

Eq.(1) is the generalized equation of motion of any
control mechanical system and well known in
multibody dynamics [5]. Let control motion is the
movement of some prescribed point of the system alongprogram trajectories w(t ) , where w(t ) is vector withthe dimension m . Eq. (1) can be transformed bysubstituting control forces u(t ) on constraints equation
due to program trajectories w(t ) :Mx D T p DTwp w
Dxh(x, x)

Dwx

w(t )

Additional links equations described by the lastequation of system (3) can be divided into two groups equations of the trajectory of the ZMP, which
determines the stability of the robot and the equations of
the coordinated mutual displacement of the links
necessary for elimination of 2 extra degrees of freedom
from the system.
The main role in ensuring the sustainability of robots
the equations defining the trajectory of the ZMP in theplane of the supporting surface of the robot are playing.
For the x coordinate of the ZMP equation has the form:f (x, x, t )(2)nwhere D w is the matrix of the variable coefficients ofconstraints equations according to program trajectories
w(t ) with the dimension k m , w(t ) is second
derivation of a program trajectories, p w is the Lagrange
multipliers vector due to w(t ). Eq. (2) are equivalent to
Eq. (1), but (2) haven’t contain unknown control forcesu(t ) . Hence numeric integration of (2) provides
kinematics parameters of the program movement x* (t)and can be used for walking pattern generation.
For control of walking robot based on the system (1)and equation (2) in the last equation of the system
linkage (2) three types of equations must contain : theequations defining the motion of the body of thewalking robot, the equations specifying the motion of
the foot points of the walking propulsion and equations,ensuring the stability of the robot:i 1

0. (4)

Fig. 1. Robot AR600 (https://youtu.be/9BRt542QoSs)

Where xi , yi , zi are the coordinates in the respective
directions of the mass center of the i -th body, mi is themass of the i-th body, K y is projection of kinetic
moment on y axis, xzmp is coordinate of the trajectoryof the zero moment point.
The equation for the y coordinate of the ZMP is written
similarly, from the condition of equality of moments
regarding the x-axis.
Equations (4) are the equations which include both thecoordinates of the bodies and their second derivatives.
The direct inclusion of such link equations into system(2) using Lagrange multipliers is difficult.
To date for biped robots quite a large number ofapproximate methods for solving such a problem aredeveloped. First, we can distinguish a group of methods,
which neglect the members of the second derivatives,
considering them to be small, which corresponds to the

D wb x = w b
D wf x = w f((zi g)mi xi xi mi zi (zi g)mi xzmp) K y(3)

D ws x = w s
The function w b sets the program motion of thepoints of the robot body, w f – move of the foot pointsof the walking propulsors, w s – the movement ofsome additional points in an additional links equations.
Functionb contains six components and defines thespatial movement of the body.
The function w f sets the movement of the points ofthe stepping propulsor with regard to the given
kinematic parameters of the characteristic points of therobot body, in particular, the points of the fixing of thewalking mover to the body. A function w f implicitly
depends onb that is necessary for the step planningparameters.

w

w©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4, B-Con Plaza, Beppu, Oita,Japan

396

Method of Finding the Androidn
i 1(mi xi mi x zmp )

0.

(4)

Equations of such type are matched with the equationsof motion of a robot using Lagrange multipliers. In
equation (5) there are time functions xzmp (t ), yzmp (t )that describe the desired ZMP trajectory. To adjustprogram motion, the additional link equations areproposed of the form :ni 1

aij xihj

jt , j 1, m

(6)Here a ij are the coefficients of the auxiliary equationsof links, h j is a variable parameter of the j-th linkequation, j (t ) is some functions, m is the number ofauxiliary equations. Equation (6) can have the physicalmeaning of the slope of the hull, a wave of the hands,
etc.

Fig. 2. Design scheme of the MBS model of AR600.

quasistatic regime of motion. In this case, equation (4)becomes the usual equation linking the coordinates ofthe bodies and their impact are taken into account byusing Lagrange multipliers [6-9].
In our work we propose to look for a program motion ofrobot that meets the equations (4), as the sum of the
movements of a certain type. The contribution of eachmovement is determined by a random multiplier thatminimizes the residual error of equation (4).

3. Theoretical and experimental analysis
Theoretical and experimental studies were conducted on
the robot АR600 (Fig. 1). This platform is used invarious researches of robot’s gait to date (e.g. [10]). The
MBS model for this robot was elaborated, a design
scheme of model is shown in Fig.2. Figure 3 shows the
program functions that determine the trajectory of the
ZMP and which are included in the equation (5). On thebasis of the solution of the system of equations (3)program angle values of the actuators were found,which were implemented on the robot АR600 using PID
controllers. In addition to the program functions of the
movement of the robot mass center y zmp (t ) Fig. 3
shows the y coordinate of the ZMP trajectory calculatedby (4). As can be seen from Fig. 3, taking into accountthe accelerations in determining the trajectory of theZMP, makes a significant disturbance – the area
highlighted by a circle in Fig.3 – which increases withthe growth of the speed of the robot, which leads to lossof stability.
To compensate the mismatch between program andactual trajectory of ZMP the motion of lifting of therobot mass center at the start and the end of the loadtransfer from one foot to the other was taken as
equations (6):Fig. 3 The program trajectory of the center of mass and the
ZMP along the y coordinate.

As the basic equations to ensure stability of the robot,
are the equations of motion of the robot mass center,
which are obtained from equations (4) with the
exception of members from the second derivatives of
the coordinates. These equations are linear with respect
to the coordinates and have the form

©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4, B-Con Plaza, Beppu, Oita,Japan

397

Alexander Gorobtsov, Pavel Tarasov, Andrey Skorikov, Alexey Markov, Andrey Andreev
n
i 1

mi zi

h

t.

independent calculation of the partial derivatives of thecriterion function.

(7)

Where h t - the empirical function shown in Fig. 4.
The specified function is selected from the conditions
for creating a stabilizing moment due to the inertial
forces of vertical movement of the robot mass center.
Fig. 5 shows the comparative trajectories of the ZMP in
the lateral direction with regard to the equation (7). As it
can be seen from the figure, equations of the type (6,7)
allow you to change the trajectory of the ZMP in a fairly
wide range.

Fig. 4. Function h

Acknowledgements
Work is performed with the financial support of theRussian Foundation for Basic Research and theAdministration of Volgograd region (project #16-47340385)
References
1. Jin'ichi Yamaguchi, Noboru Kinoshita, Atsuo Takanishi
and Ichiro Kato. Development of a biped walking robot
adapting to the human's living floor. Proceedings of the1996 IEEE International Conference on Robotics &Automation, pages 232-239, Minneapolis, Minnesota,
April, 1996.
2. Graham Mann, Stability Control in an Actively-Balanced
Bipedal Robot. School of Computer & Information
Science, Edith Cowan University.
3. Masato Yirose, Kenichi Ogawa. Honda Humanoids
Robot Development. Phil. Trans. R. Soc. A 2007 365, 11-19.
4. Gabbasov, B., Danilov, I., Afanasyev, I., and Magid, E.
(2015, December). Toward a human-like biped robot
gait: Biomechanical analysis of human locomotion
recorded by Kinect-based Motion Capture system. In
Mechatronics and its Applications (ISMA), 2015 10th
International Symposium on (pp. 1-6). IEEE.
5. Shabana A.A. Dynamics of Multibody Systems/ A.A.
Shabana//New York , NY, Cambridge University Press,
2005.
6. Jung-Yup Kim, Ill-Woo Park, Jun Ho-Oh. Experimtntal
realization of dynamic walking of the biped humanoid
robot KHR-2 using zero moment point feedback andinertial measurement. Advanced Robotics, Vol. 20. No. 6.
pp. 707-736 (2006)
7. Kim J. H. and Oh J. H., Realization of dynamic walking
for the humanoid robot platform KHR-1, Advanced
Robotics 18, 749–768 (2004).
8. Kim, J. Y., Park, I. W. & Oh, J. H. J. Walking Control
Algorithm of Biped Humanoid Robot on Uneven andInclined Floor Journal of Intelligent & Robotic Systems(2007) 48: 457. DOI :10.1007/s10846-006-9107-8.
9. Ill-Woo Park, Jung-Yup Kim and Jun-Ho Oh. Online
Walking Pattern Generation and Its Application to a
Biped Humanoid Robot – KHR-3 (HUBO). Advanced
Robotics 22 (2008), p. 159–190
10. Magid, E., & Sagitov, A. (2017, June). Towards Robot
Fall Detection and Management for Russian Humanoid
AR-601. In KES International Symposium on Agent andMulti-Agent Systems: Technologies and Applications (pp.
200-209). Springer, Cham.

t.

Fig. 5. Program ZMP trajectory without and withconsideration of equation (7).

Equations included in (6) can be selected empirically or
through formal procedures, for example, by solving the
problem of parametric optimization for the set ofparameters aij and criterion (4). This task requires
significant computational costs and, therefore, can be
resolved by using parallel computing, since it allows©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4, B-Con Plaza, Beppu, Oita,Japan

398

Path planning for Indoor Partially Unknown Environment Exploration and Mapping
Aufar Zakiev, Roman Lavrenov, Evgeni MagidIntelligent Robotic Systems Laboratory, Higher Institute for Information Technology and Information Systems (ITIS),
Kazan Federal University, 35 Kremlyovskaya street, Kazan, 420008, Russian Federation
http://kpfu.ru/robolab.html
Vadim Indelman

Autonomous Navigation and Perception Lab, Technion-Israel Institute of Technology, Haifa, Israel
E-mail: zaufar@it.kfu.ru, lavrenov@it.kfu.ru, magid@it.kfu.ru, vadim.indelman@technion.ac.il

Abstract
This paper addresses a problem of partially unknown environment exploration and mapping. The proposed path
planning algorithm provides global and local goals search taking into account limited sensing range and visibility
constraints that arise from obstacles. Looking for local goals near a global path maximizes robot utility and helps
avoiding returns to regions with low potential gain. All stages were tested in ROS/Gazebo simulations and results
were compared with a naive algorithm that was proposed earlier.
Keywords: robotics, algorithm, modelling, mapping, ROS/Gazebo, indoor exploration, path planningalgorithms designed specially for indoor environment
exploration5. However, these approaches are tested insimulations with synthetic input maps or used highquality maps without artifacts that are impossible to
avoid during a mapping process (e.g., impulse noises,
incorrect sensory processing results, sensory limitationsof real robotic system, etc.). In this paper, we propose apath planning method for an indoor partially known
environment exploration and mapping. It was tested insimulations that were created using real sensory data. The
algorithm performance was compared with previously
proposed methods.

1. Introduction
Unknown environment automatic mapping is afundamental task for all kinds of mobile robots. It isessential for every autonomous robotic system to performmapping as precise as possible to make feasible further
effective usage of a generated map in navigation
procedures. A result of such mapping that is performedby one robot could be applied then for localization andpath planning by other robots. However, previously thesemaps are often incomplete since every robot has its own
operational limits (e.g., sensory limitations, time andpower limitations) and robots that reuse such imperfect
map are forced to operate in partially unknown
environment, which constrains robot capabilities andmay significantly decrease its effectiveness. Thus, it isimportant to have a good strategy to make possible
efficient map update when its part is still undiscovered.
The problem of partially unknown or uncertain
environment exploration1 is discussed in research
dedicated to single-robot exploration based on best
exploring position search2; in terms of multi-robot
exploration3 using greedy tactics4 or sophisticated

2. Partially unknown environments exploration
challenges
A popular and effective way of mapping is using laser
range finder (LRF) devices, which provide quite precise
data about local landscape around a robot. The obtained
data is represented in a form of height map images. Eachpixel of the height map stores data about corresponding
region of the environment. Pixels can have three possible
values: black (usually interpreted as occupied region),
white (obstacle-free) or gray (no occupancy data)6.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

399

Aufar Zakiev, Roman Lavrenov, Vadim Indelman, Evgeni Magid

Fig. 3. A region of the original map with highlighted space
between distinct laser rays.

Fig. 1. A region of the original map with odometry caused
artifact (obstacle-free space behind the wall).

Then, the robot takes the filtered map as an input and,assuming its initial position within the map, one of many
localization techniques could be applied to determine
robot location. For localization we had selected AdaptiveMonte Carlo Localization8 (AMCL) method. Next, therobot preforms as follows:
1. Reachable information gain regions are marked.
2. A global goal is selected with a greedy approachbased on information gain property of regions.
3. Local goals are selected taking into accountLRF limitations and a path toward the global
goal.
4. The robot sequentially travels through the local
goals toward the global goal.
5. Return to step No.1 until reachable information
gain regions are available within the map.

Fig. 2. A region of the original map with noisy regions (several
examples of noise are encapsulated within red rectangles).

However, mapping algorithms during mapping
procedures heavily rely on odometry data, which
decrease map precision and cause various artifacts in themap (Fig. 1). Scanning device imperfection leads toeventual noises in sensory data (Fig. 2) or/and incomplete
data (Fig. 3). Another problem is that a scanning LRF isconstrained by obstacles (e.g. walls, doors, etc.) and thisshould be taken into account during path planning to
unknown regions of the map. In addition, it is very
important to use predictable forms and assumptions onindoor environments, e.g., they are structured and walls
should surround every location (room, corridor, etc.).

3.1. Reachable information gain regions marking
Reachable information gain regions are unknown map
regions, which (possibly) could be explored by the robot(shown on Fig. 4). The definition of such regions isrecursive:
Every unknown cell of the map, which isadjacent to an obstacle-free cell, is included inreachable information gain region.
Every unknown cell, which is adjacent toanother reachable information gain cell is also
included in reachable information gain region.
Recursion depth is set in advance or controlled
manually, and depends on map resolution.
Using approximate wall thickness as a basedepth value produces good results in practice.

3. Proposed approachThis section describes our algorithm, which is partiallybased on our previous research work. First, noise
reduction within a map is performed using our modified
median filter built-in map preparation tool7.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

400

Path planning for Indoor

3.3. Local goals selection
Local goals are reachable information gain regions thatare located within a predefined radius from a global path.
Local goals allows the robot to explore small reachable
regions while following the global path toward the global
goal. Such algorithm prevents the robot from returning to
the previously explored locations, thus providing a more
efficient time and energy consumption. Figure 6
demonstrates an example of different behavior of thegreedy approach (top image) and our algorithm (bottom
image). The robot starts at red X-mark, but while with thegreedy approach it begins from exploring a large
unknown area (region 1) and then returns back to thesmall region (region 2), the exploring sequence of ouralgorithm is the opposite is, which saves time for passing
the same region (region 1) twice.

Fig. 4. A region of the original map with highlighted (in purple
color) reachable information gain regions.

This method takes into account the structured patternof an indoor environment to properly estimate the utilityof each position on the map: the farther the robot moves
from frontier cells, the more chances it has to find aparticular obstacle, which limits information gain.

Fig. 5. A region of the original map. A preferred observing
position by naive approach (left) and by the proposed approach(right). Circles and arrows show sensory radius of the robot.
Fig. 6. The robot starts at red X-mark. Greedy approach
exploring sequence (top) and our approach sequence (bottom).

3.2. Global goal selection
A global goal is computed with the following strategy:
the global goal is an obstacle-free cell with maximum
count of reachable information gain cells in the LRFsensing radius. This is a greedy approach that was usedin previous works4. Previous methods considered only
frontier points as candidates for the global goal. Our
approach uses a different strategy: the global goal could
be any obstacle-free point within the map. This makes theglobal goal selection more optimal in cases when several
information gain regions could be observed from a single
point. Figure 6 illustrates advantages of this approach.

Fig. 7. Red spline represents the global path. The line starting on
the global path is the LRF ray towards the local goal.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan401

Aufar Zakiev, Roman Lavrenov, Vadim Indelman, Evgeni Magid

It is important to emphasize that local goals must bewithin the LRF line of sight, i.e., there should be no
obstacles on the way of the LRF ray. This condition isverified by simulating LRF rays from a robot pose on the
global path toward a region of interest: when the ray does
not intersect any obstacle, it is possible to observe theregion of interest from the pose. Bresenham's line
algorithm9 is used to simulate such rays on occupancy
grid (illustrated in Fig. 7). This algorithm allows to rasterstraight LRF rays and visualize them on occupancy grid.

shows better performance than a naive greedy approach
for long-time exploration of indoor environments. As apart of future work, we plan to test the proposed
algorithm within various environment in ROS Gazebo
simulation10. The algorithm will be integrated into thecontrol system of a real robot “Servosila Engineer” andtested in real-world exploration scenarios.
Acknowledgements
This work was partially supported by the RussianFoundation for Basic Research (RFBR) and Ministry of
Science Technology & Space State of Israel (joint project
ID 15-57-06010). Part of the work was performed
according to the Russian Government Program ofCompetitive Growth of Kazan Federal University.

4. Comparison with the greedy approach
To prove the efficiency of our approach, a comparison
with the greedy method3 was performed. Both algorithms
were simulated in ROS Gazebo environment and for thesimulations we used a system with i7-4700HQ CPU,NVIDIA GeForce 770M GPU, and 24GB RAM.
The results are summarized in Table 1. Two
approaches were tested for a simulated exploration taskthat was run for 3, 5, 15 and 20 minutes within the same
map. The results demonstrated that the naive algorithm
performs better and collects more data if the exploration
time is limited; in such case it is more efficient to skip
exploration of low information gains and to proceed
directly toward high information gain regions exploration.
However, when only low information gain regions
remain, the naive method produces long distance paths
with multiple returns. This way, as exploration time
grows, our approach becomes more and more efficient.

References
1. V. Pshikhopov, Path Planning for Vehicles Operating,Uncertain 2D Environments (Butterworth-Heinemann,2017).
2. H. H. González-Banos and J. C. Latombe, Navigation
strategies for exploring indoor environments, The Int. J. of
Robotics Research (21.10-11.2002), pp. 829-848.
3. A. Andreychuk and K. Yakovlev, Applying MAPP
Algorithm for Cooperative Path Finding in Urban
Environments, Int. Conf. on Interactive Collaborative
Robotics (Springer, Cham, 2017) 1-10.
4. R. Simmons, D. Apfelbaum, W. Burgard, D. Fox, M.
Moors, S. Thrun and H. Younes, Coordination for multirobot exploration and mapping, 17th National Conf. on
Artificial Intelligence and 12th Conf. on Innovative
Applications of Artificial Intelligence (2000), pp. 852-858.
5. C. Stachniss, O. M. Mozos and W. Burgard, Efficient
exploration of unknown indoor environments using a teamof mobile robots, Annals of Mathematics and Artificial
Intelligence (2008), pp. 205-227.
6. A. Elfes, Using occupancy grids for mobile robot
perception and navigation, Computer (1989), pp. 46-57.
7. R. Lavrenov, A. Zakiev and E. Magid, Automatic mappingand filtering tool: From a sensor-based occupancy grid toa 3D Gazebo octomap, Int. Conf. Mechanical, System andControl Engineering (2017), pp. 190-195.
8. F. Dellaert, D. Fox, W. Burgard and S. Thrun, Monte carlo
localization for mobile robots, IEEE Int. Conf. Robotics
and Automation (1999), pp. 1322-1328.
9. J. E. Bresenham, Algorithm for computer control of adigital plotter, IBM Systems journal, (1965), pp. 25-30.
10. M. Sokolov, I. Afanasyev, R. Lavrenov, A. Sagitov, L.
Sabirova and E. Magid. Modelling a crawler-type UGV
for urban search and rescue in Gazebo environment, Int.
Conf. on Artificial Life and Robotics (2017), pp.360-363.

Table 1. Map exploration percentage depending on themethod and exploration time.
Approach3

Simulation time in min.
5
15

20

Naive
method

73%

77%

85%

88%

Proposedmethod

72%

73%

83%

95%

5. Conclusions and future work
Path planning for partially known environments is an
important task in robotics. The results of previous
explorations could be reused, and an efficient exploration
algorithm saves time and energy consumption of a robot.
In this paper we proposed an exploration algorithm that

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

402

Simulation of service robot swarm behavior
Alexei Lushnikov. Vlada Kugurakova, Arthur NizamutdinovHigher School of ITIS, Kazan Federal University, Kremlyevskaya str., 18,
Kazan, Russia
Timur Satdarov

KUKA Robotic,
Kazan, Russia
E-mail: alexkennyy@gmail.com, vlada.kugurakova@gmail.com,
yalopata@gmail.com, timur.satdarov@kuka.com
http://kpfu.ru/eng/itis

Abstract
In this paper, we present an implementation 3D simulation of a swarm of intelligent service robots in Unity3D.
Swarm’s task and purpose is clean-up of an assigned area. We compared and analyzed different possible algorithms
of such swarm’s behavior. Implementation includes prevention of collision between robots themselves and humans
that walk through their area of effect and can be expanded to any other obstacle, moving or stationary. The
implementation is suitable for serving as a basis for future real-life construction of a service robot swarm.
Keywords: robotics, algorithm, modelling, service robot, swarm behavior, simulation, virtual experiments

1.

applications of using a swarm of autonomous robots,
except for the transportation and storage of different
items, which is already well considered in the literatureand implemented in production. In this article, we areinvited to review our simulation of the behavior of aswarm of intelligent service robots clearing the territory
in an area.

Introduction

Systems of autonomous mobile robots are capable tosolve a numerous class of civil and military tasks,
including firefighting, rescue operations, agriculture andother tasks in different fields. Therefore, tasks ofreconfiguration of robot swarms are topical, studied
enough and using in practice. The used approaches are
based on different principles and, accordingly, based on
various mathematical methods. Application of multiagent decentralized methods of interaction of robot
groups allows creating robotic systems more flexible toreconfiguration, sustainable to counteraction and withlow cost. Releasing a person from routine “dirty” work
related to cleaning both streets and more complex
surfaces (roofs, walls, windows) is one of the simplest

2.

Relevance

Modern methods described in the literature of
implementing a robot swarm, offer systems, which
include direct connection of robots or using acentralized system for coordination of robots. Thesesystems are exposed to a hardware error and/or
malicious attacks on a centralized system [1]. The© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan403

Alexei Lushnikov, Vlada Kugurakova, Arthur Nizamutdinov, Timur Satdarov

algorithms are simulating the real ants behavior. Thisidea appeared during an observation how ants search theshortest way from anthill to a food source (Fig.6). For
the first time, these algorithms have been applied in the
combinatorial graph task, for the well-known travelingsalesman problem [10]. The algorithm is based on themechanism simulating the real ants’ behavior, when
they are looking for food. As in the classical ACO [11],this modification can be divided into conditional steps,
such as: placement and initialization, movement ofagents, pheromone update and breakpoint condition
checking. The idea of the algorithm is to model the
behavior of ants, their ability to find the shortest way
from the colony to the food source in short-term, andadaptation to new changeable conditions and to find the
new shortest way. They focus on the solution of various
graph problems, which makes it reasonable to apply
ACO to solve the optimization problems of the
trajectory, in order to move robots in the environment
with static obstacles. Some known applications of ACO[12] have shown competitive results.
Replacing ants with robotic cleaners wedemonstrate a simulation of the algorithm. This method
is far from an optimal method, and in the case of robots,
the movement of a bunch of robots will block amovement of pedestrians.

bioengineering approach suggests the application ofparallel and distributed behavior of social insects. This,in particular, the development of collective memory for
robots and zones covered by the test space, the selforganization of autonomous robots for continuous
dynamic coverage of the test space [2] and the solutionof other specific accompanying problems: unpredictable
dynamics of the external environment, up to conscious
counteraction [3]; errors in the algorithm of robots inconnection with incompleteness and contradictions on
the status of other participants and the external
environment; obstacles of different shapes (convex,
concave and curved) with varying velocities [4]; a lot ofways of development, achievement of goals, distributionof roles, structure of the collective; errors ofcommunication between objects in the served territory,
their interaction with each other; prevention of collision
between robots themselves and humans that walkthrough their area of effect and can be expanded to any
other obstacle, moving or stationary [5]; problems in thecommunication architecture, protocols, operating
facilities; technical breaking [6] of the autonomous
robot and ways of their elimination.
The rationale for using the bioinspired approach is given
in our other work [7], which, however, has little
correlation with the bioengineering approach proposed
below when using heuristic swarm algorithms.
The following implementation is suitable for serving as
a basis for future 3D modeling of a service robot andreal-life construction of a service robot swarm.
3.

Implementation algorithms

One of the problems for the realization of the idea ofusing a swarm of service robots capable of self-cleaningon the zoned territory is the “task of optimizing
navigation and route”. To solve it, we can use various
heuristic algorithms, because they provide acceptable
results for a sufficient time frame. There are a lot ofswarm algorithms: ant colony algorithm, bee colonyalgorithm; particle swarm method; firefly algorithm,
gravity search algorithm water drops algorithm, etc.

Fig.1, bee colony algorithm

3.2. Bee colony algorithms

3.1. The ant colony algorithm

Bees algorithm (BA) [13] and artificial bee colony(ABC) algorithm [14] are inspired by the foraging
behavior of bees. In spite of both algorithms having

One of the superior heuristic groups of algorithms is theant colony (ACO) group of algorithms [8, 9]. These© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan404

Simulation of service robot

informing other robots about its PBP, which go to
the specified point to collect garbage.

been developed for function optimization, there areseveral studies of bee colonies on solving discrete andcombinatorial optimization problems, such astransportation problem, scheduling problems, data
mining problems and resource allocation problem. An
extensive literature survey and a classification of thealgorithms on behavioral characteristics of the honeybees have been researched in [15].
The main idea of a bee colony in nature is an
exploration of an area around a hive in order to search
for nectar and its collection. According to the model
developed by Tereshko [14], the collective intelligenceof honey bee swarm contains three main components:“food” sources (with parameters: a proximity to the nest,
a richness of energy and an ease of extracting thisenergy), “scout” bees and “forager” bees (also in acolony there are drones and uterus, but they do notcollect nectar. Scout bees explore the space around andreport about perspective places, in which the greatest
amount of the nectar was found (bee dance –mechanism of sharing information in a hive) [14]. Then
forager bees collect the nectar in these places andcollect the information about amount of nectar in someneighborhood from the area indicated by the scout bees.
Works of these types of bees in a hive provide aneffective exploration of space around and collecting of
nectar (Fig.1).

5. Comparison of algorithms
Ant colony algorithm is difficult in implementation andunpractical: difficulty of the creation of visual andinformative trail; implementation of movement by trail
because of some problems; entanglement of trails andimpossibility to determine a path of a robot. A bee
colony algorithm is more flexible to sharing information
between robots.

Fig. 2, a schematic plan of the robot work

6. Street cleaning planA schematic work plan for robots was drawn up during
preparation for the implementation of 3D simulation(Fig. 2): robot’s storage location; the drain for snow;
robots that clean the streets (they are numbered);
information board with a description of the robot
parameters.
Creating a scene. For a realistic and visual
representation of the work of robots, the scene wascreated including a typical street attributes. The
crossroad, the bus stop, a dozen trees and several houses
were put on the stage as decorations. The working areaof the robots is located between houses and the fence
near the road.
Simulation of the robots work. Robots leave the box, in
which they are stored and charged, and then move totheir coverage areas. The first robot immediately begins
cleaning the street, while the second, third and fourth
occupy their position on the sidewalk, only after thatthey begin cleaning. For optimal coverage of therectangular area, the robot moves in a zigzag,
completely covering the area.
Precipitation, mud, garbage. Each cell of the robot's
advancement is a cell with garbage or precipitation, for

4. Implementation of simulation of service
robots by bee colony algorithm
Simulation of the algorithm is created replacing bees
with robotic cleaners:
1. Robots move, covering the whole selected surface
area. Each robot searches garbage and collects it. If
one of the robots finds more garbage, it saves thisvalue as the personal best position of collecting. If
the PBP of this robot essentially differs from the
PBP of other robots, this position became global best
position (GBP), and all robots aim to this point tocollect garbage. The PBPs are not erased, and aftercollecting garbage robots return to their positions. If
the robot does not find garbage during the N-time, it
starts helping another robot with higher PBP.
2. One robot moves from a starting point searching
garbage, and when it finds garbage, robot collects it.
If it cannot do it, robot returns to the starting point,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan405

Alexei Lushnikov, Vlada Kugurakova, Arthur Nizamutdinov, Timur Satdarov

example freshly fallen snow. With the passage of time,
the amount of snow in each cell increases uniformly.
The robots collect snow during the passage of the cell.
Snow accumulates inside robots. If on the road large
accumulations of snow then the robot collecting it asmuch as possible, informs the target's coordinate toother robots and they stop their work and rush to a newtarget. The robot which collected the snow returns to thebase for unloading. Also, a robot returns if has a criticalbattery charge level. The information board is locatedon the lower right corner of the simulation. It displays
data of robots by their number. It displays the robot's
number, the level of its charge, and the amount of snowthat it has already collected. Also, it displays thetemperature. If temperature is positive, then puddles are
displayed instead of snow on the roads.

5.

6.

7.

8.

7. Conclusion
The result of the research is the 3D simulation of service
robot swarm using of bee algorithm. The simulation
demonstrated an opportunity for implementation of thisrobot system. Further implementation of robot swarm
using this simulation is possible. Factors of the external
environment and possible errors of robot algorithm
should be taken into account in the implementation.

9.
10.

11.

Acknowledgements
This work was funded by the subsidy allocated to the
Kazan Federal University for the state assignment in thesphere of scientific activities.

12.

References

13.

1.

2.
3.

4.

T. Bontzorlos and G. C. Sirakoulis, Bioinspired

algorithm for area surveillance using autonomous robots,in Int J Parallel Emergent Distrib Syst. 2017; 32(4). – pp368-85.
A. Scheidler, D. Merkle and M. Middendorf, Swarm
controlled emergence for ant clustering, in Int J Intell
Comp Cybern. 6 (1), (2013). – pp. 62-82.
I. Afanasyev, A. Sagitov and E. Magid, ROS-based
SLAM for a Gazebo-simulated mobile robot in imagebased 3D model of indoor environment, in International
Conference on Advanced Concepts for Intelligent Vision
Systems. Springer, Cham. (2015, October). – pp. 273-283.
P. Raja and S. Pugazhenthi, Path Planning for Mobile
Robots in Dynamic Environments using particle swarm
optimization, in IEEE International Conference on
Advances in Recent Technologies in Communication andComputing,India, (2009). – pp. 401-405.

14.

15.

R. Khusainov, I. Shimchik, I. Afanasyev and E. Magid,
3D modelling of biped robot locomotion with walking
primitives approach in Simulink environment. In
Informatics in Control, in Automation and Robotics 12th
International Conference, ICINCO 2015 Colmar, France,
July 21-23, 2015 Revised Selected Papers. Springer
International Publishing. (2016). – pp. 287-304.
E. Magid, and T. Tsubouchi, Static balance for rescue
robot navigation: Discretizing rotational motion within
random step environment, in International Conference on
Simulation, Modeling, and Programming for Autonomous
Robots. Springer, Berlin, Heidelberg. (2010, November).
– pp. 423-435.
V. Kugurakova, M. Talanov, and D. Ivanov,
Neurobiological Plausibility as Part of Criteria for Highly
Realistic Cognitive Architectures, in Procedia Computer
Science, vol. 88, 2016, – pp. 217–223.
M. Dorigo and L. M. Gambardella, Ant colony system: acooperative learning approach to the traveling salesman
problem, in IEEE Trans Evol Comput. (1997).
M. Dorigo, G. Di Caro and L. M. Gambardella, Antalgorithms for discrete optimization, in Artificial Life,
1999.
A. A. Kazharov and V. M. Kureichik, Ant colony
optimization algorithms for solving transportation
problems, in Journal of Computer and Systems Sciences
International, 49(1), (2010). – pp. 30-43.
R. A. Neydorf, R., O. T. Yarakhmedov, V. Polyakh, I.
Chernogorov and D. Vucinic, Robot path planning basedon ant colony optimization algorithm for environmentswith obstacles. vol. 72, (2018). – pp. 175-184.
M. D. Toksari, Ant colony optimization for finding the
global minimum, in Appl Math Comput., vol. 176(1),(2006). – pp. 308-316.
D. T. Pham, A. Ghanbarzadeh, E. Koç, S. Otri, S. Rahimand M. Zaidi, The bees algorithm—a novel tool forcomplex optimisation problems, in Proceedings ofInnovative Production Machines and Systems Virtual
Conference (2006), pp. 454-461.
D. Karaboga, An Idea based on Honey Bee Swarm for
Numerical Optimization, Technical Report-TR06, Erciyes
University, Engineering Faculty, Computer Engineering
Department, (2005).
V. Tereshko, Reaction–diffusion model of a honeybeecolony's foraging behavior, in M. Schoenauer (Ed.), in
Parallel Problem Solving from Nature VI, Lecture Notes
in Computer Science, 1917, Springer-Verlag, Berlin(2000), – pp. 807-816.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan406

Smart Spline-Based Robot Navigation on Several Homotopies: Guaranteed Avoidance ofPotential Function Local Minima
Roman LavrenovIntelligent Robotic Systems Laboratory, Higher Institute for Information Technology and Information Systems (ITIS),
Kazan Federal University, 35 Kremlyovskaya street, Kazan, 420008, Russian Federation
E-mail: lavrenov@it.kfu.ru
http://kpfu.ru/robolab.html

Abstract
Potential function based methods provide powerful solutions in tasks of local and global path planning. They arecharacterized implementation simplicity, but suffer from navigation function local minima. In this paper we propose
a modification of our original spline-based planning algorithm. Voronoi-based approach provides a good initial pathas first iteration. A new safety criterion is integrated into path planning to guarantee path safety. The modified
algorithm was implemented in Matlab environment and demonstrated significant advantages over the originalalgorithm.
Keywords: robotics, algorithm, modelling, mobile robot, path planning, Voronoi graph, potential field, optimization
criteria, virtual experiments.

level with an attractive pole in a role of a target andobstacles being represented with repulsive surfaces2.
Then a robot follows the potential gradient toward its
minimum3. The major advantages of potential field
methods are simplicity and a capability of applying them
reactively for mobile and stationary obstacles negotiation.
Its typical and significant drawbacks include pathoscillations (for certain obstacle configurations) andexistence of local minima that attract the robot and thenkeep it captured inside.
Previously4 we had proposed a path planningalgorithm for a car-like mobile vehicle that could attain atarget configuration from its initial configuration withina well-known environment providing a smooth obstacle
free path. In order to improve the performance of theoriginal algorithm, to add flexibility for path optimization
and a possibility for further fast dynamic replanning, we
integrate Voronoi Graph approach into our algorithm5.
The new approach was tested in our previous work and1. Introduction
Path planning, which is a fundamental feature of a mobile
vehicle autonomous navigation, is concerned with
automatic planning of a collision-free path between
initial and target configurations. The classical path
planning problem (“piano movers problem”) is definedfor complete a priori knowledge about environment. A
robot is informed about its own shape, start and target
position and orientation, and a set of obstacles in theenvironment. The robot searches for a continuous pathfrom its initial position to the target position, while
avoiding collisions with static obstacles along its way.
The path should satisfy some particular criteria of its
optimality.
One of the classical yet still popular approaches for
dynamical solution of a collision avoidance problem is apotential field approach1. This technique is based on an
artificial potential field generated at a global or a local© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan407

Roman Lavrenov

shown the results, which significantly overperformed theoriginal algorithm.
2. Spline-based robot navigation with original
potential field approach
The spline-based method, proposed by Magid et.al. about
a decade ago4 navigates a car-like robot in a planar
known environment populated with static obstacles. It
considers an omnidirectional circle-shape robot that is
reduced to a point in a 2D configuration space. Eachobstacle is represented with a set of intersecting circles
of different sizes. To provide a collision free path, arepulsive potential function has a high value inside an
obstacle and on its boundary and a small value in freespace. High potential function value in the obstacle’s
center pushes all points of a path outside in order tominimize path cost during local optimization procedure.
The path cost function sums up three components:

F (q )T (q )

1

2

R (q)

3

L(q)

Fig. 1. A global potential field of the environment (left) and a
final path between start (S) and target (T) positions that wassuggested by the original algorithm.

then fail in escaping from a local minima at further
iterations (Fig. 1, on the right).
3. Adding path safety parameter
We introduce an additional path quality evaluation
criterion - average distance from obstacles. Whilenavigating in obstacle populated environments, a robot
should maximize its distance from the obstacles, and thisfeature is integrated into T(q) component of our costfunction. However, in some operations, there may be arequirement to stay at some defined distance fromobstacle boundaries. This criterion may be required forthe range-limited sensors of the robot. A distance of therobot from all obstacles of environment is calculated ineach configuration q(t) along the parametrically defined
path as follows:(1)

Where T(q) function takes into an account all N obstaclesof the environment and their influence (as potential field)on the robot along the entire path. The example ofinfluence of obstacles to potential field is demonstrated
in Fig.1. R(q) is a function for smoothness property of thepath. And L(q) function accumulated path length.
The original algorithm of spline-based path planningworks iteratively, beginning with start point S and target
point T, and utilizes the environment obstacles as its
input data. An initial path is suggested as a straight line
between points S and T. Eq. (1) sets a current path cost,which is further optimized with Nelder-Mead Simplex
Method6 in order to minimize the path cost and provide anext spline with +1 increase in its flexibility degree andcomplexity as well. A resulting better path serves as an
initial guess for the next iteration.
The original method succeeds to provide a collision
free smooth path for any complexity of the environment
only if each obstacle is approximated with a single circle.
However, when the obstacles are approximated with anumber of intersecting circles, the intersections introduce
local maxima of the global potential field. If an initial
spline passes through intersection of several circles thatform a single obstacle, the cost function F(q) may
succeed pushing the spline out of intersection area andr (c) (2)q(t ) min ( x(t ) x(c))2 ( y (t ) y (c))2c C

Here C is a set of all circular obstacles c with the centerat (x(c), y(c)) and radius r(c); further, these elementary
circular obstacles may intersect to form compound
obstacles. Average distance (α) is predefined. The effectof the average distance on the cost function Eq. (1) we
can calculate with:

A(q)

q(t )

2

(3)

t [ 0 ,1]

All four criteria are combined into the cost function:

F (q)

T (q)

1

2

R(q)

3

L(q)

4

A(q) (4)Figure 2 shows the result of path planning methods
with and without the new criteria. In both cases, wesearched a path from S to T. Blue points on the trajectoryindicate control points for constructing a spline10. Thus,in the environment in Figure 2, the path was found in 4

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan408

Smart Spline-Based Robot Navigation

iterations: top image shows a path without the newcriteria, while a path with new criteria with parameters
α=5, γ4=1 appears in the bottom. The algorithm with a
new criterion builds a safer path. The red arrows in the
figure indicate the places where the path became farther
from the obstacles.

4.

Voronoi graph based solution

In order to provide a good initial spline that could befurther improved locally with regard to user selection of
the cost weights, we apply Voronoi graph approach7.
With the three steps we prepare the environment bygrouping circles into obstacles10: register circle obstacles
to form a single compound obstacle, find outer and innerboundaries of each compound obstacle, and remove innerboundaries. For example, in Fig.3 (in the left image) the
three inner obstacle boundaries will be removed.

Fig. 3. Inner and outer boundaries of obstacles (left) andthe process of Voronoi graph calculating (right).

Next, Voronoi graph is constructed based on aclassical approach11. Upon obtaining Voronoi graph VG,
we connect start position S and target position T points
to VG and apply Dijkstra algorithm8. For example in Fig.
4 you can see created Voronoi graph as tiny red lines andcalculated path as fil red line. Any path (S, T) on Voronoi
graph VG is guaranteed to be collision free andmaximally safe with regard to distance from obstacle
boundaries, and thus could provide a good initial spline
for the original spline-based method4. A small set of
special points is utilized to form via points for the initial
spline10 (blue points in Fig. 4 and Fig.5).
Fig. 2. Path calculated by the original algorithm without (top)and with the new criteria (bottom). The red arrows mark thepoints of interest where the new criteria influence is clearlyvisible.

5. Conclusions and future work
In this paper, we have presented a method for calculating
a smooth and safe path for mobile robot in static planar
environment. A significant modification of our original
spline-based path planning algorithm for a car-likemobile vehicle navigation helps avoiding local minima
problem and adds more flexibility for path optimization.
We have integrated a Voronoi graph approach into thefirst iteration of spline optimization. Furthermore,
Voronoi graph provides an opportunity to vary paths

Unfortunately, despite the use of a new criterion route
optimization is performed only locally, the effect of theadditional parameter is also local. The path remains in thesame homotopy group even in the case when it is possible
to bypass any obstacle from the other side.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan409

Roman Lavrenov

from different homotopy classes with regard to dynamicchanges in optimization function criteria as well as
supports a fast dynamic replanning in a case when an
initially off-line selected path becomes unavailable. The
algorithm was implemented in Matlab environment andits results were compared with our original algorithm.
The new approach requires less optimization iterations
that the original algorithm due to a smart selection of an
initial spline. While the original algorithm fails to find an
existing path in complicated environments with multiple
concave obstacles, its smart version was successful in all
simulated tests. For example, original approach cannot
find path in environment shown in Fig.5. Voronoi-based
method find the path by 3 iteration.

Acknowledgements
This work was partially supported by the RussianFoundation for Basic Research (RFBR) and Ministry of
Science Technology & Space State of Israel (joint project
ID 15-57-06010). Part of the work was performed
according to the Russian Government Program ofCompetitive Growth of Kazan Federal University.
References
1. S. Fleury, P. Soueres, J.-P. Laumond, and R. Chatila,
Primitives for smoothing mobile robot trajectories. IEEE
transactions on robotics and automation, 11(3) (1995)
441-448.
2. O. Khatib and B. Siciliano, Springer handbook of robotics.
(Springer, 2016).
3. E. Magid, T. Tsubouchi, E. Koyanagi, and T. Yoshida,
Building a search tree for a pilot system of a rescue searchrobot in a discretized random step environment. J. of
Robotics and Mechatronics, 23(1) (2011) 567-581.
4. E. Magid, D. Keren, E. Rivlin, and I. Yavneh, Splinebased robot navigation. IEEE/RSJ Int. Conf. on Intelligent
Robots and Systems (2006), pp. 2296-2301.
5. E. Magid, R. Lavrenov and I. Afanasyev, Voronoi-based
trajectory optimization for UGV path planning. Int. Conf.
on Mechanical, System and Control Engineering (2017),
pp. 383-387.
6. J. C. Lagarias, J. A. Reeds, M. H. Wright and P. E. Wright,
Convergence properties of the Nelder-Mead simplex
method in low dimensions. SIAM J. on optimization, 9(1)(1998) 112-147.
7. R. Lavrenov, F.Matsuno and E. Magid, Modified splinebased navigation: guaranteed safety for obstacle avoidance.
In: Ronzhin A., Rigoll G., Meshcheryakov R. (eds)
Interactive Collaborative Robotics. Lecture Notes in
Computer Science, 10459 (Springer, Cham, 2017) 123133.
8. H. M. Choset, Principles of robot motion: theory,algorithms, and implementation (MIT press, 2005).
9. E. Magid, R. Lavrenov, M. Svinin and A. Khasianov,
Combining Voronoi graph and spline-based approaches
for a mobile robot path planning. In Lecture Notes inElectrical Engineering (Springer International Publishing,2018), in press.
10. E. W. Dijkstra, A note on two problems in connexion withgraphs. Numerische mathematik 1(1) 269-271.
11. A. Andreychuk and K. Yakovlev, Applying MAPP
Algorithm for Cooperative Path Finding in Urban
Environments. In: Ronzhin A., Rigoll G., MeshcheryakovR. (eds) Interactive Collaborative Robotics. Lecture Notes
in Computer Science, 10459 (Springer, Cham, 2017) 1-10.

Fig. 4. Path within Voronoi graph.

Fig. 5. Path on Voronoi graph (left) and resulting path after three
iteration with the modified algorithm (right).

As our future work, we plan to integrate a number of
additional parameters of 3D environments into a costfunction. We also plan to create a C++ library of themethod and further extend the proposed method for thenavigation of a heterogeneous robotic team operating inan urban search and rescue scenario9.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan410

Virtual Experimental Stand for Automated Fiducial Marker Comparison in Gazebo
Environment
Ksenia Shabalina1, Artur Sagitov1, Hongbing Li2, Edgar A. Martinez-Garcia3, Evgeni Magid1
1Intelligent Robotics Department, Higher School of Information Technology and Information Systems, Kazan Federal
University, Kremlyovskaya str. 35, Kazan, Russian Federation2
Department of Instrument Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
3
Institute of Engineering and Technology, Universidad Autónoma de Ciudad Juárez, Chih., 32310, Mexico
E-mail: ks.shabalina@it.kfu.ru, sagitov@it.kfu.ru, lihongbing@sjtu.edu.cn, edmartin@uacj.mx, dr.e.magid@ieee.org
http://kpfu.ru/robolab.html

Abstract
This paper presents experimental automated approach for comparing fiducial marker systems. Previously wecompared ARTag, AprilTag, CALTag systems under three types of adverse conditions: systematic occlusion,
arbitrary overlap with an object and marker rotation. In effort to improve statistical significance of our previous workwith manual experiments, we faced a challenge of conducting over a thousand additional experiments. Using Gazebo
environment we implemented virtual robot system that performs all necessary manipulations automatically. Further,
we investigate adding Gaussian noise in order to make simulations more realistic.
Keywords: Robotics, fiducial marker system, recognition algorithms, experimental comparison, ROS, simulation.

marker occlusion resistance since the occlusion is a rather
usual and unavoidable while working with humanoid
robot manipulators. If the markers are located on the
robot itself, they may become overlapped with varying
intensities by other workspace objects (e.g. its own
manipulators) and have some inclination angle withregard to a camera, which also affects its detectability6.
Previously, we conducted manual experiments7,8 toevaluate FMSs resistance to overlapping marker’s area(systematic occlusion and arbitrary overlap with anobject) and various rotations. Since it is time consuming
to conduct such experiments with thousands of trials
manually, we have decided to automate them and to startfrom experiments in ROS/Gazebo environment (Fig. 1).
This paper presents automated approach of
conducting experiments with marker systems to evaluate
occlusion resistance. Section 2 presents our related workof comparing FMS in presence of occlusions, in Section
3 we briefly overview ROS Gazebo environment and in

1. Introduction
Fiducial marker systems (FMS) are systems of markers
that are automatically detected by a camera with help ofcorresponding algorithms. FMS find their applications invarious areas: physics, medicine, robotics, augmented
reality, metrology etc. Taking in consideration only
application in robotics, marker systems are used in robot
navigation1, localization2 and mapping3, camera
calibration4 and tasks that require camera pose
estimation5. As our global goal is to calibrate Russian
humanoid robot AR-601M mono camera andmanipulators using FMS, we faced many FMS options.
Each FMS has it strengths and drawbacks that are derived
from the original area of its applications. Therefore, weneed to compare markers in order to select a single
suitable FMS from a large number of existing systems forour specific task. There are several marker performance
criteria6. In our work, we are particularly interested in© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

411

Ksenia Shabalina, Artur Sagitov, Hongbing Li, Edgar A. Martinez-Garcia, Evgeni Magid

Section 4 we describe our simulated world in Gazebo andexperiment setup. Section 5 shows the results ofexperiments and analyze of it. Finally, we conclude andplan our future work in Section 6.

CALTag system on the other hand showed high resistantto all types of occlusion: edge occlusion, arbitraryoverlap with an object and tag rotation.
3. ROS Gazebo Environment
Robot Operating System (ROS) is the most fast-growing
and popular framework for programming robots that was
initially developed by Willow Garage. The main idea of
ROS framework is a collaborative construction ofrobotics software. Everyone can use and improve ROS
packages - an atomic unit in ROS system that represents
one or more functionalities or ways to solve particular
problem - or create a new one.
Gazebo is a popular 3D simulator for simulation ofdifferent type of robots, helping with the creation of new
robots and testing algorithms in rather realistic conditions.
Gazebo includes physics simulation, robot andenvironment models, and custom plugins. We selected
Gazebo for our virtual experiments due to its integration
with ROS, accessible custom joint controllers and aconvenient way to design a robot with urdf-files.

Fig. 1. Gazebo world environment: a tag holding automatedstand (on the left) and a robot with a camera (on the right).

2. Related Work
During the selection of a most suitable FMS we haveselected three systems most applicable in our field:
ARTag5,6, AprilTag3 and CALTag9. As mentioned above,
initially we conducted all the experiments manually. Our
experimental work consisted of two main parts: pilot
experiments with web camera Genius FaceCam 1000X7
and experiments with AR-601M humanoid8 (Fig. 2). We
outlined two experiment designs: simple and compound.
First design (simple) consisted of systematic occlusion
and arbitrary overlap with an object. The second design(compound) consisted of several types of occlusion:
marker rotation, systematic occlusion, marker rotation
with systematic occlusion and arbitrary overlap with anobject. The examples of first set of experiments are
showed in Fig.3. Figure 4 demonstrates the scheme ofmarker rotation during second design experiments.
Figure 5 shows example of a marker rotation experiment.
These two experimental designs initially were carried out
using Genius FaceCam 1000X camera for investigating
FMS’s applicability using low-cost equipment. Afterinitial experiments were completed, the same
experiments were performed using humanoid robot AR601M mono camera Basler acA640-90gc.
Analyzing the results of our experiments, weconcluded that AprilTag and ARTag demonstrated highsensitivity to edge occlusions, while showing satisfactory
results in the experiments with arbitrary overlap andachieved perfect score in the marker rotation tests.

Fig. 2. AR-601M humanoid robot.

Fig. 3. Systematic occlusion experiments using
AR-601M mono camera.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, Beppu International Convention Center, Oita, Japan

412

Virtual Experimental Stand for

4.

were completed we received summary of experiments(number of errors in detection, rotation estimation etc.)
To make our experiments closer to real wordconditions, we added Gaussian noise parameters for thecamera of R2D2-like robot. We used three values fornoise standard deviation (stddev) parameters: 0.009, 0.09
and 0.1. Figure 6 shows examples of camera view withvarying levels of Gaussian noise. Image on the left
represents stddev parameter of 0.009 and it is considered
to be a common case for a real camera. In the central
image the camera has 0.09 stddev, which represents
moderate level of noise. Image on the right shows thecamera with 0.1 stddev, representing high levels of noise,
and its threshold value serves as a boundary condition ofmarker recognition possibility or failure.

Experiment Setup

Fig. 4. Scheme of marker rotation experiments.

We targeted to automate execution of thousands ofexperiments with each of the FMS families to minimize
time and effort that are required by real world
experiments. For this purpose, we created a Gazebo
world with two robots: R2D2-like robot (observer) andthe tag-managing robot (performer). The performer robot
is actually several interchangeable robot models designed
for each type of experiments. In this Section we discuss
the design and results of pilot virtual experiments with
AprilTag markers rotation around Z axes. Those type ofexperiments are presented in the Fig. 4 as Type 1A.
The performer was implemented using ros_control
package for its controllers. We tuned a PID controller anddesigned a procedure to perform rotation around Z axis
by a given angle (from 0 degrees to 90 degrees) in both
directions (implemented as tag_ rotation_node node).
Using already existing implementation of AprilTag in
ROS (apriltags_ros by Mitchell Wills) we detected a tagin a set of camera frames while the performer rotatedAprilTag publishing the results of the detection (ID of a
tag and an angle of its rotation). As the performer rotated
the tag, it published current angle of rotations. Thus, we
could deduce, which angles resulted errors in detection
or rotation estimation by comparing messages from
aforementioned nodes. The comparison process was
automated, as during launching virtual experimental
stand we specified the tag family and ID, experiment type,
number of repetitions, noise level and when experiments

Fig. 5. Manual marker rotation experiments around Z-axis.
Type 1A in scheme of rotation experiments.

5. Experimental Results
To test the virtual experimental stand we chose four
randomly selected AprilTag type marker with the sameIDs as in previous work7: ID 4, 6, 8, 9. The standperformed 3000 experiments for each marker with theresults of experiments demonstrated in Table 1. Everymarker could be successfully detected with up to 41
degrees of rotation with low nose and up to 25-38 degrees
with high level of noise. This factors explain the low
percentage of successful experiments in Table 1, because
the rotation was performed up to 89 degrees. As was
expected, with noise level increase the number ofsuccessfully detected and recognized markers decreased.
Moreover, virtual experiments with noise also

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, Beppu International Convention Center, Oita, Japan

413

Ksenia Shabalina, Artur Sagitov, Hongbing Li, Edgar A. Martinez-Garcia, Evgeni Magiddemonstrated that some markers are significantly more
successful that others (e.g., ID4), which emphasizes thenecessity of various markers performance comparison.

that some markers are significantly more successful that
others, which emphasizes the comparison necessity.
As a part of our future work we plan to construct anautomatic tool for marker rotation and test it with several
FMSs. The most resistant for all types of occlusion
marker system will be selected for further AR-601M
camera and manipulator calibration.

Table 1. Detection rate (in percentage) during rotation ofmarkers around Z-axis with varying camera noise levels.

Acknowledgements
This work was partially supported by the RussianFoundation for Basic Research and Ministry of Science
Technology and Space State of Israel (project IDs 15-5706010, 17-48-160879). Part of the work was performed
according to the Russian Government Program ofCompetitive Growth of Kazan Federal University.

Fig. 6. Camera image with low (on the left), moderate (in thecenter) and high level of noise (on the right).
Tag
ID

Low
(0.009)stddev
Moderate(0.09)

High
(0.1)

4
6
8
9

60,8%
49,8%
40,9%
43,3%

42,2%
35,4%
32,4%
20,8%

30,2%
22,7%
25,3%
17,4%

References
1. R. Kuriya, T. Tsujimura and K. Izumi, Augmented reality
robot navigation using infrared marker, Proc. 24th IEEE
Int. Symposium on Robot and Human Interactive
Communication (2015), pp. 450-455.
2. V. Dhiman, J. Ryde and J. J. Corso, Mutual localization:
Two camera relative 6-DOF pose estimation fromreciprocal fiducial observation, Proc. IEEE/RSJ Int. Conf.
on Intelligent Robots and Systems (2013), pp. 1347-1354.
3. E. Olson, AprilTag: A robust and flexible visual fiducial
system, in Proc. IEEE Int. Conf. on Robotics andAutomation (2011), pp. 3400-3407.
4. M. Fiala, Comparing ARTag and ARToolKit Plus Fiducial
Marker Systems, Proc. IEEE Int. Workshop on Haptic
Audio Visual Environments and their Applications (2005),
pp. 148-153.
5. M. Hirzer, Marker Detection for Augmented Reality
Applications, Seminar/Project Image Analysis Graz(2008), pp. 1-25.
6. M. Fiala, ARTag, a fiducial marker system using digitaltechniques, Computer Vision and Pattern Recognition(2005), vol.2, pp. 590-596.
7. A. Sagitov, K. Shabalina, R. Lavrenov and E. Magid.
Comparing fiducial marker systems in the presence ofocclusion, Proc. Int. Conf. on Mechanical, System andControl Engineering (2017), pp. 377-382.
8. K. Shabalina, A. Sagitov, L. Sabirova, H. Li, and E. Magid.
ARTag, AprilTag and CALTag Fiducial Systems
Comparison in a Presence of Partial Rotation: Manual andAutomated Approaches, Lecture Notes in ElectricalEngineering (Springer, in press).
9. B. Atcheson, F. Heide, W. Heidrich. CALTag: High
Precision Fiducial Markers for Camera Calibration, Vision,
Modeling, and Visualization Workshop (2010), vol.10, pp.
41-48.

6. Conclusions and Future Work
This paper presents experimental automated approach forcomparing fiducial marker systems (FMS). Previously
we compared ARTag, AprilTag, CALTag systems under
three types of adverse conditions: systematic occlusion,
arbitrary overlap with an object and marker rotation. In
effort to improve statistical significance of our previous
work with manual experiments, we faced a challenge ofconducting over a thousand additional experiments, andpilot experiments of such automated approach were
presented in this paper.
We constructed a world in Gazebo environment,
which consisted of R2D2-like robot and AprilTag marker
holding robot that rotates AprilTag for a given angle(from 0 degrees to 90 degrees) in both directions. We
integrated it with AprilTag node tag detector in such waythat we could detect the marker in each robot camera
frame. To make simulated experiments close to realworld conditions, we investigated three different
Gaussian noise standard deviation values. Then we
performed experiments applying different noise levels
and analyzing AprilTag system recognition underrotation. The system detected tags with up 40 degrees
rotation with a low level of noise and up to 25-38 degrees
with high. Virtual experiments with noise demonstrated© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, Beppu International Convention Center, Oita, Japan414

An Automated Optical Inspection system for a tube inner circumference state identification
Chung-Wen Hung*, Jhen-Gu Jiang, Hsien-Huang P. Wu, Wei-Lung Mao

Department of Electrical Engineering, National Yunlin University of Science & Technology , 123 University Road,
Section 3,Douliou Yunlin 64002, Taiwan, R.O.C
E-mail: wenhung@yuntech.edu.tw
www.yuntech.edu.tw

Abstract
An automated optical inspection, AOI, system for a chopstick tube inner circumference state identification is
proposed. This AOI system is installed after the material feed equipment to screen for cost reduction. The tube status
identification is based on the proposed machine vision software, and implemented with EmguCV library. The
algorithms of image processing are used to sort the material into five groups. In this paper, the algorithm will be
detailed, and the experimental results will show the proposed AOI system is workable.
Keywords: Machine Vision, AOI, EmguCV, Tube inner circumference.
telecentric lens are used to enhance the accuracy of the machine
vision system. In order to prevent the influence from theenvironment light, a stable light source is needed [1], and acircular light source is used to ensure uniform illumination ofeach tube. The proposed system is based on the machine vision
software library, EmguCV[2], to quickly confirm a tube wasimaged clearly and in compliance with the imaging
requirements of the defect inspection.

1. Introduction
As AOI technology advances, it is adopted in more and moreproduction lines, but due to cost, not for traditional industry
factories, such as chopstick factories. An expensive process forchopsticks is the chopstick defect detecting, and laborers checkthe good quality only in the final phase.. However, most of thedefective products are caused by the original flawed material.
Only rely on the naked eye to determine the defects, it is too
difficult and waste time. In the other hand, the optical detection
methods will not damage the surface of the tube. An AOI
system for chopstick material is proposed in this paper.
Machine vision technology is used to screen out the tube forflaws before they are processed. In order to have the recovery
of defective materials, the defects are sorted into four types:"Large", "Small", "Deformed" and "Empty".
In machine vision, the quality of object imaging can be a criticalpart. The complexity and accuracy of the algorithm are related
to it. Therefore, the Charge-coupled Device, CCD, and

2. Related researching
2.1. Region of Interest, ROI
The purpose of ROI is to separate the images captured by the
CCD in a particular area by filters, binarization, image cuts, etc.
Moreover, it is used to reduce the computational resources
required in image processing [3], and it can also reduce the error
in the operation of subsequent algorithms with the priorthreshold design.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan415

Chung-Wen Hung , Jhen-Gu Jiang, Hsien-Huang P. Wu, Wei-Lung Mao

Table 1. Specification of the CCD and lens

2.2. Edge detection
After the image filtering, grayscale, binarization, the black-andwhite boundary point in the ROI will been detected and marked,
and the marked line segment is the outline which is theperipheral line of the tube. In many algorithms, Cannydetector[4] is more effective in the edge extraction, so in thispaper, Canny detector is used to extract the edge of the tube.

OS
CPU
Development tools

ComputerWindows 7
I7-5500 x64
Visual Studio 2017

Interface
Sensor type
Shutter
Resolution

2.3. Component Labeling
When edges or binarization of an image is detected, labeling isused to cluster the pixels connected to the same area into thesame object. Its purpose is to isolate the noise in some images[5]or the non-interesting content that can not be completely
removed in the previous step. These noise and non-interesting
content can be ruled out by simple threshold judgment, and thenthe most complete image information for subsequent feature
extraction is got.

3.

Magnification
Working distance

CameraUSB 3.0
CMOS
Rolling shutter
3840 x 2748 Pixel
Lens
0.184
132.9mm

5. Algorithm and flow chart

The development library: EmguCV

There are many software packages developed for AOI, and they
are also popular. Such as Halcon, an image processing softwarefrom Germany, claims the most effective computer vision
software in Europe and Japan. However, due to cost issue, thispaper developed the system based on open source libraries.
EmguCV is an open source image processing development kit
that includes the many relevant libraries required for many
applications such as image processing, machine learning andcomputer vision. Users can choose according to their own needs
different programming languages such as C #, VB, C++.and soon. EmguCV solved the lack of OpenCV Graphical User
Interface, GUI, and is suitable for more application
development.

4. Hardware specifications and configuration
Table 1 shows the specifications of the CCD and lens used in
the system proposed this paper. The CCD with a telecentric lenscan effectively reduce the image deformation which caused by
the normal lens [1]. In order to average the light on each tube
and minimize the shadow [6, 7], the light source used in thissystem is a circular parallel ring light. Figure 1 is theconfiguration for camera, lens, light and tubes used in this paper.
The relative distance between the lens and the tube depends on
the working distance of the lens itself.

Figure.2 A flowchart of the system
The system flow chart is shown in Figure 2. The main purposeof the system is the inner circumference measurement of the of
the chopstick tube. There is no any image filter adopted in the
preprocessing stage in this proposed system. Because themaintenance of the image original information is mostimportant, and filters may remove some critical information.
The AOI system sorts the defect materials into five modes: one
empty situation, and four defect types. The inspection
algorithms are detailed in following:

5.1. Empty
By connecting the size of the region, the system could
determine whether the tube on the fixture. If the connected
region is bigger than the setting threshold, the system will
indicate no tube on the fixture, otherwise. The threshold setting
is based on analysis of the inner area of pre-shot photographs.

Figure.1 Schematic diagram of the AOI© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan416

An Automated Optical Inspection

5.2. Deformation tube

6.1. Experimental hardware system

The ellipse fitting algorithm is used to fit an under testing tube
to the most suitable ellipse [8]. The difference between the two
radii of the fitting ellipse is used to determine whether the tube
has a deformation situation. When the difference is bigger than
the setting threshold (about 8 pixels), the tube will be denoted
as “deformation”.

The block diagram and photo of the proposed system are shownin Figure 1 and 5. The results of the pre-processing are present
in Figure 3 and Figure 4, what are Edge detection andconnectivity region functions.

6.2. Circumference Threshold
The circumference threshold is a key parameter to sort the tube
into “Normal”, “Large” and “Small” types when not empty anddeformation. As Figure 6, the tubes of three types provided by
the factory are measured, then threshold could be decided.

5.3. Iron filing elimination algorithm
Except empty and deformation, iron filing effect has to beeliminated. The iron filings effect is caused in the process orfactory environment, and these iron filings have significant
influences on the accuracy in the precise identification. The ironfiling elimination algorithm is proposed to eliminate this effect.
The Inscribed and circumcircled circles are estimated [8]. Whenthe circumference difference of the two circles is big, the circle
fitting is used to estimate the tube inner circumference.
Contrary in small difference, what means minor iron filing
effect, the circumference is integrated from pixels directly.

Figure.6 Single state comparison graph

6. Experimental results

6.3. Mix state test results

1

2

3

4

5

6

7

8

9

10

11

12

Figure.3 Edge detection

Figure.4 Connectivity region

Figure.7 Serial Numbering

Figure.5 Photo of the AOI system
Figure.8 Mixed test 1© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan417

Chung-Wen Hung , Jhen-Gu Jiang, Hsien-Huang P. Wu, Wei-Lung Mao

7. Conclusion
An AOI system for a chopstick tube inner circumference
state identification is proposed, in this paper. The system
can effectively identify a variety of different types of the tube
by comparing various parameters and threshold design. The
experimental result indicates the 91.67% of the recognition
rate, and the system can be used in automated production lines
and effectively reduce the production loss.

Acknowledgment
This work is partially supported by the Ministry of Science andTechnology, ROC, under contract No. MOST 106-2221-E-224025, and 106-2218-E150-001.

Figure.9 Mixed test 2

Table 2. Test Results 1
Serial Number Real situation Identify results1
NormalNormal2
Deformed
Deformed
3
Small
Small
4
Small
Small
5
NormalNormal6
NormalNormal
7
Large
Large
8
Deformed
Deformed
9
Deformed
Deformed
10
Large
Large
11
Small
Normal
12
Large
Large
Table 3. Test Results 2
Serial Number Real situation Identify results
1
Large
Large
2
Deformed
Deformed3
NormalNormal4
Small
Normal
5
Large
Large
6
Deformed
Deformed
7
Empty
Empty
8
NormalNormal9
Deformed
Deformed10
NormalNormal11
Deformed
Deformed
12
Small
Small

References
1. H. Liao, Z. Chen and X. Zhang, “Calibration of Camera
with Small FOV and DOF Telecentric Lens”, Precedingof 2013 IEEE International Conference on Robotics andBiomimetics, Shenzhen, China, 2013.
2. EmguCV.org Available at www.emgu.com/wiki fourth
November 2017
3. W. C. Tham, S.I. Woolley, S. Cribbs and D. Anderson,“Diagnostically lossless compression of pipeline
inspection data”, Proceedings of Data Compression
Conference 2002, Snowbird, UT, USA, 2002
4. J. Canny, “A computational approach to edge detection”,
IEEE Transactions on Pattern Analysis and MachineIntelligence. Vol. PAMI-8, Iss. 6, Nov. 1986
5. J. Song, Z. Zhao, Q. Zeng and Y. Wei, “An algorithm foreliminating the isolated regions based on connected area
in image classification” Preceding of IGARSS 2004. 2004
IEEE International Geoscience and Remote Sensing
Symposium, Anchorage, AK, USA, 2004
6. K. W. Khawaja, A. A. Maciejewski, D. Tretter, and C. A.
Bouman, “Camera and Light Placement for Automated
Assembly Inspection” Preceding of Robotics andAutomation, 1996. Proceedings of IEEE International
Conference on Robotics and Automation, Minneapolis,
MN, USA, USA, 1996
7. Mónica Valenzuela-Delgado Wendy Flores-Fuentes,
Miguel E. Bravo-Zanoguera, Alejandro S. Ortiz-Pérez,Daniel Hernandez-Balbuena, Moises Rivas-López, Oleg
Sergiyenko, “Felix F. Gonzalez-Navarro, Machine visionsystem to measuring the velocity field in a fluid by Particle
Image
Velocimetry:SpecialCaseof
Magnetohydrodynamics” 2017 IEEE 26th International
Symposium on Industrial Electronics (ISIE), Edinburgh,
UK, 2017
8. Truc Le and Ye Duan, “CIRCLE DETECTION ON
IMAGES BY LINE SEGMENT AND CIRCLECOMPLETENESS”,
2016
IEEE
International
Conference on Image Processing, Phoenix, AZ, USA,2016

Figure 7 shows the serial number of tubes. The Figure 8 and 9
are the sampling photo, and the Tables 2 and 3 are theinspection results. There is only one error in every test, and therecognition rate of the algorithm reaches 91.67%.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

418

Implementation of the Mobile Based Robot Arm for Image Recognition
Ji-Hua Li

Graduate school Engineering Science and Technology, National Yunlin University of Science & Technology, 123 Sec. 3,
University Road, Douliou, Yunlin 64002, Taiwan
Jr-Hung Guo

Department of Electrical Engineering, National Yunlin University of Science & Technology, 123 Sec. 3, University Road
Douliou, Yunlin 64002, Taiwan
Kuo-Lan Su*

Department of Electrical Engineering, National Yunlin University of Science & Technology, 123 Sec. 3, University Road
Douliou, Yunlin 64002, Taiwan
E-mail: tpemail77@yahoo.com.tw, g9710801@yuntech.edu.tw, sukl@yuntech.edu.tw
Abstract
The paper develops a mobile based robot arm using KNRm system. The platform of the mobile based robot arm isbuilt as triangle style. The structure of the robot arm uses the Matrix elements that are manufactured by BardenPowell International Company. The mobile based robot arm integrates some sensors, three DC servomotor motors,
one RC servomotor, a controller, and an image recognition module. The mobile platform embeds a robot arm on thefront side. The robot arm is one degree of freedom. The driver device of the robot arm is a RC servomotor. The
cargo puts on the plank. The robot arm must rise up and put down the plank. In the experimental results, the mobile
based robot arm can search and recognize the assigned plank using image binanzation method and Otsu algorithm
by the image recognition system. Finally, the robot arm moves approach to the assigned color plank, and catches
the plank moving to the assigned position, and puts down the plank.
Keywords: Mobile based robot arm, KNRm system, image binanzation method, Otsu algorithm.

switched alternatively to adapt on different groundsituations [1]. Bloesch et al. presented a state estimation
approach for legged robots based on stochastic filtering,and designed an unscented Kalman filter and outliers
rejection methodology based on a consistent
formulation of the underlying tochastic model [2]. Su etal developed a mobile platform, based on KNR
controller. The mobile robot embed a robot arm withfour degrees of freedom, and used of light sensors andtouch sensors for line tracking and detects the initial
location [3].
The paper is organized as follows: Section II
describes the system architecture of the mobile based
robot arm, and explains the functions of the mobile
robot. Section III using an example explains the motion
planning method for the mobile robot. Section IV
presents the experimental results for the mobile based
robot arm to catch and put down the plank using the

1. Introduction
Autonomous mobile robot and robot arm is the
popular subject in recent years. The disadvantage ofautonomous mobile robot can not work with the humanbehavior in the dangerous environment and the robot
arm has not the mobility function. It is a good solution
to combine autonomous mobile robot and robot arm,and build up a mobile based robot arm. They have been
implemented widely in many areas. Thus, how to plan
the mobile robot efficiently to complete missions using
the robot arm in the unknown and complexity
environment, and how to control the robot arm tocooperate with the mobile robot are challenging tasks.
In the past literatures, many experts research on
mobile robot. Peng et al. designed a where/track mobile
platform to search and rescue in dangerous environment.
The motion modes of the mobile platform can be© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

419

Ji-Hua Li, Jr-Hung Guo, Kuo-Lan Su

image recognition system. Section VI presents the brief
concluding remarks.

The KNRm system controls each DC servomotor to
execute the rotation range and movement displacement
according to the feedback signal of the encoder sensor.
The control core of the KNRm system uses PID control
law and trapezoidal acceleration and deceleration
algorithm to control each DC servomotor, and tune themobile robot to follow the programmed trajectories. The
mobile robot uses three spring elements to decreasevibration and increase moving stability of the robot arm.
The robot arm can catch the assigned plank very easily.
Three spring elements are fixed on the driver axis.
The robot arm fixes on the frond side of the mobile
platform to be driven by one RC servomotor as shownin Fig. 3. The structure of the robot arm is built using
Matrix elements. The image system is embedded on the
top of the gripper to recognize position and color of thebox, and searches the assigned box on the competition
playground. Then the mobile robot moves to the box,
and catches the plank on the box using the robot arm,and moves the plank to put down on another box.

2. Mobile Based Robot Arm
The mobile robot uses omi-direction wheels to becontrolled by three DC servomotors. The prototype ofthe mobile robot is shown in the Fig. 1.The direction
angle of each servomotor is 120 degree for each other.
The axis location of the mobile robot is defined shownin Fig. 2. The moving direction describes asfollowing：
1. If the mobile robot moves along the Y axis, thenV1 V2 , V3
2.

(1)

If the mobile robot moves along the X axis, then

V1
3.

0

V2

2V3

(2)

If the mobile robot rotates along the clockwise
direction, then

V1 V2

V3

(3)

Figure 3. The prototype of the mobile based robot arm

Figure 1. The prototype of the mobile robot

3. Motion Analysis
We explain the motion scenarios of the mobile robot
moving on the competition playground (length 4m andwidth 2m). The start position “S” is assigned on the left
side of the competition playground. The size is length
60cm and width 60cm respectively shown in Fig. 4. The“B” is the target position. The mobile robot must catchthe assigned plank moving to the “B”.
For a example, we describe the motion scenario forthe mobile based robot arm to finish the assigned task.
First, the mobile robot stays at the start position shownin Fig. 5(a), and moves to the assigned box shown in
Fig. 5(b). The position of the box can be measured

Figure 2. The axis definition of the mobile robot© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

420

Implementation of the Mobile

before the competition. The mobile robot records theaxis values (X and Y) of each box, and programs the
motion trajectory moving to the assigned box according
to shape and color of the box. Then the robot arm of themobile robot catches the object that is located on the topof the box shown in Fig. 5(d), and moves to the centre
position of the competition playground. The mobile
robot puts down the object in the area shown in Fig. 5(e).
Finally, the mobile robot must finish the task in thelimitation time, and moves to the start position to stopshown in Fig. 5(f).

4.

We implement the experimental result on thecompetition playground using the mobile based robot
arm. First, the mobile robot searches the positions of allboxes using the image system to focus on the assigned
color and shape mark. The image system can set theparameters using the vision acquisition module andvision assistant module of the LabView software. We
put two boxes on the competition playground, and use
various graph on the front side of each box shown in Fig
6(a). One is rectangle; the other is circle. The visionsystem of the mobile robot catches the graph in thecentre of each flame, and computes the average value ofthe measured signals using add array element function.
The pre-processing of the vision system uses
LabView Software. The processing of image
recognition includes four steps, such as open objects,
remove small objects, convex hull and particle analysis.
The vision system can recognize shape and color of theassigned mark for all boxes shown in Fig.6(b), andcompute and remove small objects of the assigned markshown in Fig. 6(c). Then the vision system finishes
convex null and particle analysis for the mark of the box
shown in Fig. 6(d) and (e).

Figure 4. Competition playground and start position(a)Experimental Results

(b)

(a)
(c)

(d)

(e)
(f)
Figure 5. Motion processing of the mobile robot(b)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

421

Ji-Hua Li, Jr-Hung Guo, Kuo-Lan Su

flame. Then the mobile robot moves to the assigned box,
and controls the gripper of the robot arm to catch theassigned color plank show in Fig. 6(g). Finally, themobile robot moves to the target position “B” to put
down the plank, and moves to the start position to stopshown in Fig. 6(h).
5.

We designed a mobile robot with a robot arm using
KNRm system, and used the Matrix elements toconstruct the robot arm. The main core of the KNRm
system programs trapezoidal acceleration anddeceleration algorithm and Proportional-IntegralDerivative (PID) algorithm to control each DC
servomotor of the mobile robot, and used image
binanzation method and Otsu algorithm to classify colorand shape of the mark through image recognition
system. The mark was located on the frond side of thebox. In the experimental results, the mobile robot cansearch and move to the plank, and catch the assigned
plank moving to the target position to put down theplank.

(c)(d)

Conclusion(e)

Acknowledgements(f)This work was supported by the Ministry of Science andTechnology in Taiwan, (MOST 106-2221-E-224-019).

(g)

References
1. A. Peng, Y. Zhou, J. Hu and Y. Ou, “Mechanical design
for wheer/track transform mobile platform-search andrescue robot,” IEEE International Conference on
Robotics and Biomimetics. P.1787, 2014
2. M. Bloesch, C. Gehring, P. Fankhauser, M. Hutter, M. A.
Hoepflinger and R. Siegwart, State Estimation forLegged Robots on Unstable and Slippery Terrain,
IEEE/RSJ International Conference on Intelligent Robots
and Systems, pp.6058-6064, 2013.
3. J. H. Guo, K. H. Hsia, K. L. Su and J. T. Wang,
Development of the Mobile Robot With a Robot Arm,
International Conference on Industrial Technology(ICIT),
Taipei, Taiwan, March 14-17, pp.1648-1653, 2016.

(h)Figure 6. Experimental results
Finally we implement the experimental scenario
using the developed mobile based robot arm. The
mobile robot can search and move to the position of theassigned box using the vision system show in Fig. 6(f).
The control box of the KNRm system uses LabView
software to recognize shape and color of the mark to beright. The mobile robot tunes the direction of the robot
arm to catch the image of the mark in the centre of the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

422

Reinforced Quantum-behaved Particle Swarm Optimization Based Neural Networks
for Image Inspection
Li-Chun Lai

Computer and Intelligent Robot Program for Bachelor Degree, National Pingtung University
Chia-Nan Ko

Department of Automation Engineering, Nan Kai University of Technology
t105@nkut.edu.tw

Abstract
The paper combines the niche particle concept and quantum-behaved particle swarm optimization(QPSO) method with chaotic mutation to train neural networks for image inspection. When constructing
the reinforced quantum-behaved particle swarm (RQPSO) to train neural networks (RQPSONNs) for
image inspection, first, image clustering is adopted to capture feasible information. Then the database ofimage can be built. In this research, the use of support vector regression (SVR) method determines theinitial architecture of the neural networks. After initialization, the neural network architecture can be
optimized by RQPSO. Then the optimal neural networks can perform image inspection. In this paper, theprogram of RQPSONNs for image inspection will be built. The values of root mean square error (RMSE)
and peak signal to noise ratio (PSNR) are calculated to evaluate the efficiency of the RQPSONNs.
Moreover, the experiment results will verify the usability of the proposed RQPSONNs for inspecting
image. This research can be used in industrial automation to improve product quality and production
efficiency.
Keywords: Quantum-behaved particle swarm optimization, Niche particle, Support vector regression,
Image inspection

1.

better state.10-12 To overcome the premature phenomenon
in QPSO, a modified quantum-behaved particle swarm
optimization (MQPSO) is proposed to identify nonlinear
systems.13 Authors proposed niche particle swarm
optimization (NPSO) for image segmentation.14 In NPSO
algorithms, particles changing the place of the course tocenter the appropriate position (niche) of the particle will
be absorbed. Then, particles amalgamate the small
appropriate position (small niche) to become a large
appropriate position (big niche).15
This paper combines the niche particle concept,
quantum-behaved particle swarm optimization (QPSO)
method with chaotic mutation to train neural networks
for image inspection. When constructing the reinforced
quantum-behaved particle swarm (RQPSO) to train
neural networks (RQPSONNs) for image inspection.

Introduction

The application of image inspection is quite extensive
including medical science, machine vision, andpredicting analysis of patterns in smart automation
production fields.1–5 The input images are first filtered bya high-pass filter, which is used to remove direct current
and enhance high-frequency components. And then thefiltered input images, which are overlapped rather thandisplaced from each other in the plane, serve as the input
images.6 Neural networks have the capacities of learning,
adaption, and nonlinear mapping of images.7
The PSO algorithm possesses the ability of highconvergent speed, easily falling in some local optima is
its fatal defect. Many researchers have presented revised
PSO algorithms and obtained good results.8 Anotherimprovement on traditional PSO algorithm is quantumbehaved particle swarm optimization (QPSO).9 However,in QPSO, particles fall into local optimal state inmultimode optimization problems and cannot find any© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan423

Li-Chun Lai, Chia-Nan Ko

2.

Modified Quantum Particle Swarm

3.

Optimization

Quantum-Behaved

Particle

Swarm Optimization Neural Networks

From the view of classical dynamics, to avoid explosion
and guarantee convergence, particles must be bounded
and fly in an attractive potential field. Clerc andKennedy8 have proved that if these coefficients areproperly defined, the particle’s position pi will converge
tothecenterofpotential
field,
pf c [ pf1c , pf 2c , , pf nc ] , and is defined as:

pf i cReinforced( c1 r1 pil

c2 r2 p g )(c1 r1 c2 r2 ), i 1,2,,n .

3.1. Radial basis function neural networks
One can use a neural network to estimate theinput–output relation of a dynamic system. In this paper,radial basis function neural networks (RBFNNs) are
adopted because they have a simple structure, as shownin Figure 1. When the Gaussian function is chosen as theradial basis function, RBFNNs can be expressed in the
form

(1)

yˆ j (t 1)

where pil and p g are the best position of the ith

Li 1

Gi wijLi 1

wij exp

xˆ mi

2

(7)

2 i2

particle and the global best position; c1 and c2 are
cognitiveandsocialconstrictionT
where xˆ (t ) xˆ1 (t ), xˆ 2 (t ), , xˆ m (t ) is the input vector;Tyˆ (t ) yˆ1 (t ), yˆ 2 (t ), , yˆ m (t ) is the output vector; wij

coefficients,

respectively; r1 and r2 are random numbers between 0
and 1.
Inspired by the behavior that particles move in abounded state and preserve the global search ability, Sunet al.12 proposed the QPSO algorithm. In the QPSO
model, the solution of time-independent Schrödinger
equation for this system in one dimensional space can be
expressed as:10
L1
pi pf i c
ln
,
(2)
2
is a random number uniformly distributed on
where[0, 1] and L is the characteristic length of delta potential
well (called "Creativity" of particles) which specifies thesearch scope of a particle. The mainstream thought pointand can be expressed as the following forms:12

mbest

pi ,1 n pi , 2
,
,
i 1 n
i 1 n
n

,

pi , n, i 1,2,i 1 n
nis the synaptic weight; Gi is the Gaussian function;

mi andare the center and width of Giirespectively; and L is the number of the Gaussianfunctions, which is also equal to the number of hidden
layer nodes.
Given a set of training input–output pairs x ( k ) , y( k ) ,k 1,2, , N , the optimization problem of RBFNNs isto determine the values of wij , mi , andtoiminimize the index

when the input x̂ to the network is equal to x(k ), n, (3)The creative coefficientwith adaptive annealinglearning mechanism according to the change rate of
optimal estimation has the form:
fit ,
(5)
max

x2

mbest

fit

pg

pi

pil ,

,

(8)

where yˆ ( k ) is the corresponding output of RBFNNs

x1

2

2

y ( k ) yˆ ( k ) ,

k 1

(4)

L

N

J

xm

(6)

G 11

w11
w12

G22

ŷ2

wL 2
G LL

ŷ1

wLp

yˆ p

Fig. 1. The structure of RBFNNs.

whereis step length of, fit is the change
rate of optimal estimation so far. The mechanism ofadaptive annealing learning can overcome the stagnation
problem to accelerate the convergent speed.

3.2. Niche evolution© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

424

Reinforced Quantum-behaved Particle Swarm

The appropriate position evolves (niche evolution) cansolve the multimodal function optimization problems
effectively.16 Adopt the structure of appropriate positionand search some local extreme values synchronously andavoid early convergence. It is difficult to determine theappropriate radius (niche radius) share . Define the nicheradius as follows:17

5.

In the study, RQPSONNs combining the niche particle
concept, quantum-behaved particle swarm optimization(QPSO) method with chaotic mutation to train neuralnetworks is proposed to solve image inspection.
Moreover, the experiment results will verify the usabilityof the proposed RQPSONNs for inspecting image.
Acknowledgements

d avg
share

max ui

r

R

0 re

1 i N

(9)

N

R

i 1

ui

This work was supported in part by the Ministry of
Science and Technology, R.O.C., under grants MOST
106-2221-E-252-001.

(10)

i

N 1 N
2
d( i,
N ( N 1) i 1 j i 1

d avg

j

)

(11)

2

References
1. L. M. Chen, Z. Chen, and A. S. Feng, Image analysis
algorithm and verification for on-line molecular sieve sizeand shape inspection, Advanced Powder Technology 25(2)(2014) 508-513.
2. G. B. Massoud, R. R. Denis, and D. Carl, Multivariate
image analysis for inspection of multilayer films, Polymer
Testing 40 (2014) 196-206.
3. J. H. Chen, M. C. Su, R. Cao, S. C. Hsu, ans J. C. Lu, A selforganizing map optimization based image recognition andprocessing model for bridge crack inspection, Automationin Construction 73 (2017) 58-66.
4. J. Liu, Z. Tang, W. Gui, W Liu, P. Xu, and J. Zhu,
Application of statistical modeling of image spatial
structures to automated visual inspection of product quality,
Journal of Process Control 44 (2016) 23-40.
5. Y .G. Cen, R. Z. Zhao, L. H. Cen, L. H. Cui, Z. J. Miao,
and Z. Wei, Defect inspection for TFT-LCD images basedon the low-rank matrix reconstruction, Neurocomputing 149(Part C) (2015) 1206-1215.
6. X. P. Deng and D. M. Zhao, High-pass filtering andfrequency spectrum correlation for larger size image
recoginition, Optik 123 (2012) 924-927.
7. P. Banumathi, Artificial neural network techniques used for
identifying defects in woven fabric images, International
Journal of Latest Trends in Engineering and Technology
3(3) (2014) 248-252.
8. A. Ratnaweera, S. K. Halgamuge, and H. C. Watson,
Self-organizing hierarchical particle swarm optimizer with
time-varying acceleration coefficients, IEEE Transactions
on Evolutionary Computation 8 (2004) 240-255.
9. M. Clerc and J. Kennedy, The particle swarm: explosion,
stability, and convergence in a multidimensional complex
space, IEEE Transactions on Evolutionary Computation 1(2002) 68-73.
10. Y. Fang, Y. D. Zhao, M. Ke, X. Zhao, H. H. Iu, and K. P.
Wong, Quantum-inspired particle swarm optimization for
power system operations considering wind power

(12)

i

where is an adjusting parameter, R is norm of u, ,share . d avg is the average
0 is the relative parameter of
distance of particles.
3.3. Chaotic mutation
The Chaos phenomenon in nonlinear science means akind of definite but unpredictable motion state. It has
already been applied to optimizing stochastic
optimization problems efficiently. This study adopts
chaotic mutation operation to avoid falling into somelocal extreme value. The logistic equation of Chaos
iterating is expressed as follows:16
k 1
j

4.

k
j

(1

k
j

) k

1,2,

(13)

Simulation Results

In this paper, a work piece for image inspection is
performed by the proposed RQPSONNs. The experimentresults will verify the usability of the proposed
RQPSONNs for inspecting image shown as Figure 2.

(a)Conclusions(b)

Fig. 2. (a) work piece on the conveyer belt (b) workpiece finished inspection.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan425

Li-Chun Lai, Chia-Nan Ko

Computer Engineering and Applications 46(3) (2010)183-185.
15. G. J. Cheng, Y. An, Z. Wang, and K. Zhu, Oil wellplacement optimization using niche particle swarm
optimization, in Proceedings of 2012 Eighth International
Conference on Computational Intelligence and Security(CIS), pp. 61-64.
16. H. Li, Y. Zhang, A. Wang, Medical Image Registration
Based on JS Measure and Niche Chaotic Mutation
Quantum-Behaved Particle Swarm Optimization, inProceedings of 2010 6th International Conference on
Wireless Communications Networking and Mobile
Computing, pp.23-25.
17. O. L. Zhao, The research of the niche particle swarm
optimization based on self-adaptive radius technology, inProceeding of 2009 Asia Pacific Conference on Information
Processing, pp. 97-100.

uncertainty and carbon tax in australia, IEEE Transactions
on Industrial Informatics. 8(4) (2012) 880-888.
11. S. L. Ho, S. Y. Yang, G. Z. Ni, and J. Huang, A
quantum-based particle swarm optimization algorithm
applied to inverse problems, IEEE Transactions on
Magnetics 49(5) (2013) 2069-2072.
12. J. Sun, W. Fang, X. Wu, V. Palade, and W. Xu,
Quantum-behaved particle swarm optimization: analysis ofindividual particle behavior and parameter selection,
Evolutionary Computation 20(3) (2012) 349-393.
13. C. N. Ko, Y. M. Jau and J. T. Jeng, Parameter Estimation of
Chaotic Dynamical Systems Using Quantum-behaved
Particle Swarm Optimization Based on Hybrid Evolution,
Journal of Information Science and Engineering 31(2)(2015) 675-689.
14. M. M. Bai, H. Sun, and L. Y. Wu, Method for image
segmentation based on niche particle swarm optimization,.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan426

Development of IoT Module with Backup and Data-security Functions
Jr-Hung Guo*

Department of Electrical Engineering, National Yunlin University of Science & Technology, Taiwan
Kuo-Hsien Hsia†

Department of Electrical Engineering, Far East University, Taiwan
Kuo-Lan Su*

Department of Electrical Engineering, National Yunlin University of Science & Technology, Taiwan
E-mail: g9710801@yuntech.edu.tw, khhsia@mail.feu.edu.tw, sukl@yuntech.edu.tw

Abstract
Internet of Things (IoT) is one of the most popular research topics. There have been many studies and products about
this topic. However, there is few researches about the correctness of data and the mutual support of modules. In thispaper, we developed an IoT module with multi-sensor and communication interface. It debugs and confirms the datafrom multiple modules and sensors using the fusion and the redundant algorithms. It can also sustain or replace apossibly fail IoT module via multiple communication interface. The dynamic security key technology is used toensure the security of data transmission. Hence the IoT system can be more stable and more secure.
Keywords: Internet of Thing (IOT), data fusion, data redundant, dynamic security key.

were discussed to help IoT manufacturers secure theirproducts. Sivanathan et al undertook a deeper explorationof network-level security solutions for IoT, by comparing
flow-based monitoring with packet-based monitoring
approaches [2]. They showed that flow-based monitoring
can achieve most of the security benefits of packet-based
monitoring, and at dramatically reduced processing costs.
Premnath et al showed that a small size of key reduce thecryptographic computational processing requirements
quite drastically for IoT nodes [3]. It estimates the cost of
breaking public key crypto systems when the adversary
is limited by the available resources (i.e., dollar cost) andtime (i.e., number of days). It also presents the trade-off
between the processing loads for an IoT node versus the

1. Introduction
Internet of things (IoT), in the past decade, is a very hot
research topic. Many consumer and industrial products
have already used the IoT technology. However, thesecurity issue of IoT devices has only been taken
seriously for the past two years. It is mainly due to lesscomputation power or memory of the microprocessor
used in IoT devices. In previous studies, Wurm et al usedboth commercial and industrial IoT devices as examples
from which the security of hardware, software, andnetworks are analyzed and backdoors are identified [1].
Those security vulnerabilities are a common problem formost devices. Security solutions and mitigation methods© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan427

Jr-Hung Guo, Kuo-Hsien Hsia, Kuo-Lan Su

desired time span of privacy protection. Kuusijärvi et al
discussed the security challenges of IoT devices at thattime and proposed a solution to secure these devices via
a trusted Network Edge Device [4]. The major benefit ofthis approach is that the system can protect the IoT
devices with user-defined policies. Additional benefit is
the possibility to manage the countermeasures ofmultiple IoT devices/gateways at once, via a sharedinterface, thus largely avoiding the per-device
maintenance operations. Sadeghi et al gave an
introduction to industrial IoT systems, the related
security and privacy challenges, and an outlook on
possible solutions towards a holistic security framework
for industrial IoT systems [5]. From previous researches
we know that the security of IoT devices is very
important. Therefore, in this paper, we will develop a
dynamic data encryption method for IoT module of loworder microprocessors, so that most IoT devices cancommunicate securely.

Fig. 1. IOT module Architecture.

At the first communication of the modules, the sendermust first randomly generate the address of the Initial
Security Key Table. This address is generated by the
program counter in the 8051, or similar register or
random numbers in other chips. According to the lengthof ID, this address will be taken an AND operation with
01111111 or 00111111, and the result will be sent to the

2. System Architecture
In this paper, we use 8051 series microprocessor-based
IoT module used to develop communication dataencryption, and use redundant algorithms and data fusion
algorithms to confirm the accuracy of the data detectedby multiple modules/sensors with wrong data isolation.
The block diagram of this module is shown in Figure 1.
The IoT module has UART (Universal Asynchronous
Receiver/Transmitter), WIFI, Ethernet and 433/868/915
MHz tri-band multi-channel RF communication module.
It can use three kinds of communication interfaces at thesame time. So it can be used in different applications
because of the powerful communication function. In thisIoT module, we designed a communication protocol, assown in Table 1, to ensure the efficiency/accuracy of data
communication. We use the State byte to transmit theindex value of the initial dynamic password. Its structureis shown in Table 2. This protocol and the dynamic
security key method can ensure the securecommunications.

Table 1. Communication Protocol.
Byte 0 Byte 1
Byte 2Byte 335H
ID Sensor Kind Module

Byte 4State

4N(N=1~16)
Data/Parameter

Checksum
SUM( all byte)

Table 2. State Byte Structure.
bit 7
0
1

bit 6
Sensordisable

bit 5
bit 4
bit 3 bit 2 bit 1 bit 0
Electricity Deci
Sensor statuslowding factor
Initial password section

Bit7=0 Sensor status and deciding factor.
Bit7=1 Initial password section.

Table 3. Initial Security Key Table (Example).

3. Dynamic Security Key Method
In each IoT module will default to 64 ~ 128byte lengthof the initial security key Table, the structure shown in
Table 3.
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan428

Development of IoT Module

receiver via State byte of the communication packet.
After the confirmation of the correctness by the receiver,a new password generated by PC (Program Counter) and
DTPR (Data Pointer Register) will be sent back to thesender as the password for sending the next data
verification. The receiver will store these data including
sensor kind, module and 4-byte password. Each module
can store up to 64 groups (because both the sender andreceiver have to store), and overwrite if full. Therefore,
if the previously communicated module password isoverwritten, the receiver will request the sender toregenerate the encryption key again according to the wayof the first communication. The Initial Security Key
Table can be regenerated from time to time by the central
monitoring system. The dynamic security key process isshown in Figure 2.
Next, we use the actual examples to illustrate the processof generating and using the entire security key. Taking
Table 1 as an example, if the PC register is 01D1H, it isobtained after AND with 00000000 00111111.
00000001 11010001
AND) 00000000 00111111----------------------------------(1)00000000 00001011 =11H
At this point the sender will send 11H to the receiver, thereceiver will get from the address 11H 4 Byte password
11H 12H 13H 14H. Then combine the values of PC andDPTR two registers to get 4 Byte data and XOR with 11H
12H 13H 14H. If PC = 0824H, DPTR = 238AH, thefollowing way to generate encrypted security key.
08 24 23 8A
XOR) 11 12 13 14-----------------------(2)19 36 30 9E
Then the encrypted security key, back to the sender. Afterthe sender receives this set of security keys, it will fromthe generated initial segment 11H obtain the decryption
key 11H, 12H, 13H, 14H, and then use this set of keys tosolve the real security key.
19 36 30 9E
XOR) 11 12 13 14-----------------------(3)08 24 23 8A
The result of the solution will be 11H 12H 13H 14H, theresulting security key 08H 24H 23H 8AH XOR again.
Then compare the received data is 19H 36H 30H 9EH, if
the same to confirm that this group of security key
decoding is correct. Then use 4 byte to do data

Fig. 2. Dynamic Security Data Transmission Flow Chart.

transmission encryption. The sender in the data transfer
is completed, will re-use the value of PC and DPTR toobtain a new 4 Byte security key. And then sent to thereceiver. So every communication security key will bedifferent. And because PC, DPTR two register values are
unpredictable, so we can achieve a very good data
security key.
4. Algorithm
To analyze each sensor more accurately and reduce themisjudgment rate, we used the adaptive fusion method in
the monitoring system. This algorithm is very suitable for
digital detection signals, and the module sensor data after
redundant algorithm processing are all digital. We cananalyze the data using the adaptive fusion method.
(A) Redundant management algorithm
In each IoT module, we use redundant management
method to calculate exact sensor values, and isolate thefaulty measurement values. The redundant measurement
values l of the observed system are described asfollowing. For the ith measurement value mi and boundary
error bi, we can compute the indicator function I i k , and© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan429

Jr-Hung Guo, Kuo-Hsien Hsia, Kuo-Lan Su

find the power estimated value x̂ k of each power
detection unit at a given sample time k.
lxˆ (k )i 1

mi (k ) Ii k
l
i 1

Ii k

l
j 1, j i

wˆ i

(4)

bi (k ) b j (k )

wˆ i

1,
0,

if
if

*
*

is
is

wˆ

1
if ui
m1i
1
if ui
m0i

(9)

0 and H1

wˆ i

wˆ i

each updating. And the criteria that ŵi reach stable canbe set by the monitoring system.
true
false

(6)

5. Experimental Results(B) Adaptive Fusion Method
Consider a binary hypothesizing-and-testing system withn area detecting point, in which each detecting point
employs a predetermined decision rule. The two
hypotheses are H0 and H1, where H0 means that the event
does not occur, and H1 means that the event occurs. They
have a priori probabilities P0 = P(H0) and P1 = P(H1),respectively. The system structure is shown in Fig. 3.
Finally we can get the amended rule, detailed derivation
can refer the last two references [6,7]. Define the initial
weight value as:
P(7)wˆ 0 log 1
P0
Then we define the following parameters:
ui: results of individual modules or sensors detect events,
m: number of points that the event occurs
n: number of points that the event does not occur
m1i: number of points that ui = +1 and H1
m0i: number of points that ui = 0 and H0
n1i: number of points that ui = +1 and H0n0i: number of points that ui = 0 and H1
Because Adaptive Fusion Method is a digital type ofassessment method, so we can get the following
evaluation equation.
(1) If the module/sensor signal is reliable, then the weightchange is (set m1i 1, m0i 1 ):
1
m1i
m1i
1
moi
m0i

1 and H 0

(10)
where ŵi and ŵi represent the weight after and before(5)

i 1,2,..., j 1, j, j 1,... jf*

1 wˆ i wˆ 0
if ui
e
m1i
1 wˆ i wˆ 0
if ui
e
m0i

Finally, we can get the adaptive fusion rule of each sensor
weight as:Ii kf mi (k ) m j (k )11 wˆ i wˆ 0
n1i
e
n1i
n1i
m1i
1
1 wˆ i wˆ 0
noi
e
noi
n0i
m0i

1 and H1

We use MCS-51 series of single-chip microprocessor todevelop dynamic security key technology and theredundant method and adaptive fusion method to isolate
the problematic sensors or IoT modules. The module
could have communication interfaces of Ethernet, WiFi,
RF and others. This IoT module is a modular design. Up
to 3 communication interfaces can be installed on
demand at a time to ensure successful communication.
When there is a problem with the data in the IoT module,
the monitoring system will isolate the problematic IoT
module data. IoT module communication interface andgroup status are shown in Figure 3.
Figures 4(a) & (b) show the two IoT modules that theactual IoT module uses dynamic security key to connect
successfully. Figure 4(c) shows the real picture of the
dynamic security key in the monitoring system. In thisscreen we can see the status of each IoT moduleconnection and dynamic security key. And in this screen
can also be updated IoT module default dynamic security
key table. Figure 4(d) shows the monitoring screen of the
IoT module A/D and I/O status.
6. Conclusion Tables
We used the dynamic security key technology andredundant methods, and adaptive fusion method on the
IoT module of 8051 single-chip main controller. Because
the 8051 single-chip is a lower-order microprocessor, it
is difficult to use complex encryption techniques. We use
PC and DPTR registers of 8051 single-chip
microprocessor to generate the index of the dynamic
security key table. Then the dynamic security key will be
re-obtained in every communication. This method makes
the encryption key unpredictable, and can ensure data
communications security without heavy loading of a(8)0 and H 0(2) If the module/sensor signal is not reliable, then theweight change is (set n1i 1, n0i 1 ):© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan430

Development of IoT Module

single chip microprocessor. Practical test has confirmed
the effectiveness of this method. In addition, we also use
redundant methods and adaptive fusion methods on IoT
modules to identify and isolate IoT module data. The
effectiveness of this method has also been verified byactual test. The method developed in this paper can alsoeasily be transferred to other single-chip microprocessors.
In the future, we will continue to increase the complexityof the dynamic security key for more securecommunications. And we will also continue to strengthen
IoT module data confirmation and backup mechanism,
IoT system can be more stable for more applications.
Fig. 3. IOT Module communication interface, and group status.

References
1. J. Wurm, K. Hoang, O. Arias, A.R. Sadeghi and Y. Jin
Security analysis on consumer and industrial iot devices,
2016 21st Asia and South Pacific Design Automation
Conference (ASP-DAC), IEEE (2016) 519-524.
2. A Sivanathan, D. Sherratt, H.H. Gharakheili, V.
Sivaraman and A. Vishwanath, Low-cost flow-based
security solutions for smart-home IoT devices, 2016 IEEE
International Conference on Advanced Networks andTelecommunications Systems (ANTS) (2016) 1-6.
3. S.N. Premnath and Z J. Haas Security and privacy in theinternet-of-things
undertime-and-budget-limited
adversary model, IEEE Wireless Communications Letters
4(3) (2015) 277-280.
4. J Kuusijärvi, R. Savola, P. Savolainen and A. Evesti,
Mitigating IoT security threats with a trusted Network
element, 2016 11th International Conference for Internet
Technology and Secured Transactions (ICITST), IEEE
(2016) 260-265.
5. A.R. Sadeghi, C. Wachsmann and M. Waidner, Security
and privacy challenges in industrial internet of things,
Design Automation Conference (DAC), in: Proceedingsof the 52nd ACM/EDAC/IEEE Design Automation
Conference (DAC'15), IEEE, (2015) 1-6.
6. N. Ansari, J.G. Chen and Y.Z. Zhang, Adaptive decisionfusion for unequiprobable sources, IEE ProceedingsRadar, Sonar and Navigation 144(3) (1997) 105-111.
7. N. Ansari, E.S.H. Hou, B.O. Zhu and J.G. Chen, Adaptive
fusion by reinforcement learning for distributed detection
systems, IEEE Transactions on Aerospace and Electronic
Systems 32(2) (1996) 524-531.

(a) IoT Module 1(c) IoT Security Set Interface(b)IoT Module 2(d)IoT Monitor System

Fig. 4. IoT Conn. States and User Interface.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan431

Development of Auto-Stacking Warehouse Truck
Kuo-Hsien Hsia*, Ming-Guang Wu, Jun-Nong Lin, Hong-Jie Zhong, and Zh-Yao Zhuang

Department of Electrical Engineering, Far-East University, Taiwan
Tainan, 74448, Taiwan (R.O.C.)
E-mail: khhsia@mail.feu.edu.tw
www.feu.edu.tw

Abstract
Warehouse automation is a very important issue for the promotion of traditional industries. For the production oflarger and stackable products, it is usually necessary to operate the stacker for the stacking and storage of the products.
The general autonomous warehouse-truck does not have the ability of stacking objects. In this paper, we develop a
prototype of auto-stacking warehouse-truck that can work without direct operation by a skill person. Just commandedwith an RFID card, the stacker truck can move under the prior-planned route, and pick and deliver the product fromthe designated storage area or deliver and put the product to the designated storage area in the warehouse. It cansignificantly reduce the manpower requirements of the skilled-person of forklift technician and improve the safety of
the warehousing area.
Keywords: Warehouse automation, Stacking truck, Autonomous warehouse truck, RFID.

Although the automation of manufacturing processes andthe automation of manufacturing industries have beenable to evolve to unmanned factories, it is usually stillnecessary to manually drive the stacker from factory towarehouse, from warehouse to delivery, to inbound andoutbound delivery of the warehouse, etc. to carry out
stacking and picking operations. To drive a stacker
requires professional training. In Taiwan, a technician
license (Technician for fork lift operation) with the
Ministry of Labor is required to qualify for driving a
stacker. In addition, where people work, they must also
cooperate with lighting, air circulation, and various
conditions of work safety. Therefore, if it is not necessary
to work with manpower, then the cost of lighting and air
conditioning can be saved and the space can be used more
effectively. Hence many researchers have invested instorage automation research. Many logistics providers

1. Introduction
The industrial automation has been developed for morethan 60 years. It includes manufacturing automation,
warehouse automation, and logistics automation. No
matter in which one of these automations, robots always
play an extremely important and crucial role in theindustrial automation. For the manufacturing industry,
object movement is always inevitable in the process ofpurchasing of raw materials or components, loading ofmaterials, packing of finished products, import of goods
and the shipment of goods. A conveyor belt is usuallyused for delivering these components, semi-finishedproducts and finished product in the production line.
However, after the product is completed and packaged, it
is necessary to carry the manpower to carry and stackproducts when entering the warehouse or when shippingfrom the warehouse.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

432

Kuo-Hsien Hsia, Ming-Guang Wu, Jun-Nong Lin, Hong-Jie Zhong, Zh-Yao Zhuang

also include warehouse automation as one of the mostimportant work items available today.
Giordano et. al considered a matrix-based discrete event
control approach for a warehouse [1]. Xie et al use RFID
(radio frequency identifier) and wireless sensor network
in managing a warehouse [2]. In [3], Adinandra et al said
that autonomous mobile robots (AMR) have emerged asa means of transportation system in warehouses. Wang et
al proposed a positioning system composed of an RFID
reader equipped with forklift and terrestrial channels
embedded RFID tags to improve warehouse management
efficiency [4]. Amazon makes extensive use of robots for
product collation and warehouse management and had
45,000 robots across 20 fulfillment centers in the early2017 [5].
In this paper, we will develop a small warehouse truck
with the function of auto-stacking. Here “auto-stacking”
means the goods can be stored in stack manner with
detection of numbers of stacked goods and it will alarm
when the storage area is full.

Production
Line

Packing
Area

Warehouse
Area

Shipping
Area

Fig. 1. Flow of goods with RFID tag.

2. System Architecture
In the case of a typical factory, when the goods are
manufactured by the production line and packaged, they
are usually manually moved to a van and then stacked bya technician of folk-lifter to the warehouse area. When
there is a shipping requirement, the same way, they willbe transported in the same way from the warehouse area
to the shipping area for packing or into the container.
Different products will be placed in different warehouses
or different counter. We assume that on the product or its
package, there is an RFID (radio frequency identifier) tag
for identification. In addition, our scenario assumes thatthe warehouse is an unmanned environment and allorders are made in the shipping area or packing area.
Because an unmanned storage environment does notneed any lighting for vision and air-conditioning, it isenergy-saving. It also can reduce the possibility ofpotential danger of injury caused by the devices
movement or improper operation since no person isrequired to work in it. The flow of the goods with RFID
tag is shown in Figure 1.
Thus the warehouse truck with the auto-stacking ability
should be composed of three parts: the driver part todriving the motors of the truck and the mechanism of thefolk-lift device, the control part to command the truck thedestination and route, and the sensor part including RFID

Fig. 2. System architecture of the auto-stacking warehousetruck.

reader for commanding, sensors for obstacle avoidance
and tracking the route, and sensors for the folk of thetruck. The system architecture of the auto-stacking
warehouse truck is shown in Figure 2.
3. Simulation
The scenery of our simulation includes a shipping area
(also the packing area) and two warehouse area The
warehouse truck will be commanded to take the goods
from factory packing area to the correct location of thewarehouse and to leave the goods in the correct storage
counter or to take the correct good from the storage
counter of the warehouse to the shipping area. The goods
can be stacked in the designated warehouse. For
simplicity, the warehouse truck moves under the guide ofa marked curve on the floor. To distinguish different
warehouse, we marked a colored tag on the track. The
whole simulation scenery is shown in Figure 3.
Our experimental truck is based on aluminum extruded
material as a skeleton. A microchip 16-Bits MCU is usedas the kernel and an Arduino chip is supplemented tocontrol the truck. An infrared trace sensor is used to© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan433

Development of Auto-Stacking Warehousedetect the route planned in advance, so that the truck canfollow the route. A color sensor is used to read different
storage areas. Truck moving and the fork moving up anddown is driven by a 12VDC deceleration motor. The fork
stretching is to driven by a 5VDC stepper motor. Figure
4 shows our experimental auto-stacking warehouse truck.
The truck is initially in the packing area. We use an RFID
card to command the truck to pick up the goods and takeit to the designated storage counter. The goods will be
stored in a stacked manner.
There are two function buttons and an emergency stop
switch on the truck, with the green button for picking
goods up from the warehouse area to the shipping area,
and the yellow one for delivery goods to the warehousearea for stack. The different warehouses are distinguishedby two RFID cards. There is a stacking limitation in eachwarehouse area. The truck can move under the commandof pushed button and the read RFID message. Once thedestination warehouse is full of goods and a goods isplanned to be stacked into, the warning light will bebright. The alert can be released by changing thecommand mode. In our simulation, the goods can besmoothly transported from the packing area to the storage
area for stacking, or picked up from the storage area andtransported to the shipping area. Figure 5 shows the
scenario that a goods is in transportation to the warehousearea A.
4.

Fig. 3. Simulation Scenery.

Fig. 4. Our experimental auto-stacking warehouse truck.

Discussion and Conclusion

In this paper, we have made a brief discussion of theadvantages of an unmanned warehouse. In order toimprove the space efficiency of the limited storage area,
stacked storage of goods is a good idea. In order to realize
such a warehouse concept, we designed a simulated
warehouse truck with function of auto-stacking. It canmove on the specified route and store the goods in astacked way to the designated area. When the stack area
is full, it can alert the operator to change his instructions.
This idea can be realized in a real warehouse area withcubic packed goods such as televisions, refrigerators, or
air conditioners.

Fig. 5. A snap of the warehouse truck working.
matrix-based approach, IEEE Trans. Automation Sci. andEngr. 5(1) (2008) 53 – 70
2. Z. Xie, W. He, L. Lv and J. He, The implement of
warehouse management system based on RFID andwireless sensor network, IET Int’l Conf. on WirelessSensor Network 2010 (2010) 98-103.

References
1. V. Giordano, J.B. Zhang, D. Naso and F. Lewis, Integrated
supervisory and operational control of a warehouse with a© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

434

Kuo-Hsien Hsia, Ming-Guang Wu, Jun-Nong Lin, Hong-Jie Zhong, Zh-Yao Zhuang

3. S. Adinandra, J. Caarls, D. Kostić, J. Verriet and H.
Nijmeijer, Flexible transportation in warehouses, in:Automation in Warehouse Development (Springer,
London, 2012).
4. H. Wang, T. Cheng, and Z.A. Wu, Research on positioning
algorithm of forklift-mounted RFID reader, Sensors &Transducers 174(7) (2014)164.
5. S. Shead, Amazon now has 45,000 robots in its
warehouses, Business Insider, UK, Jan. 2017. Available at:
http://uk.businessinsider.com/amazons-robot-army-hasgrown-by-50-2017-1© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan435

Development of Four-axis SCARA Robotic Arm Built on Automation Control System
Jr-Hung Guo

Department of Electrical Engineering, National Yunlin University of Science & Technology, 123 Sec. 3, University Road
Douliou, Yunlin 64002, Taiwan
Kuang-Wei Chuang

Department of Electrical Engineering, National Yunlin University of Science & Technology, 123 Sec. 3, University Road
Douliou, Yunlin 64002, Taiwan
Kuo-Lan Su*

Department of Electrical Engineering, National Yunlin University of Science & Technology, 123 Sec. 3, University Road
Douliou, Yunlin 64002, Taiwan
E-mail: tpemail77@yahoo.com.tw, g9710801@yuntech.edu.tw, sukl@yuntech.edu.tw
Abstract
The paper presents a integral design method to integrate “DFM” (Design for Manufacturing) and “DFV” (Designfor Verification), and completes the production system from simulation to actual verification using the SCARA
robot arm. In the aspect of kinematics of horizontal joint, the 2D model is established by mathematical derivationand the analysis of the robot arm characteristics using “Jacobian” kinematic, and integrates into the verification
design. The electromechanical integration of the proposed system includes a SCARA robot arm, automatic control
components, sensors; electrical elements, pneumatic circuit design and software programming for implement. Then
user can lead the robotic arm to run the coordinates and movement path with proper accuracy, efficiency andreliability. The program language of the SCARA uses “DRL” (DELTA Robot Language) that is developed by
DELTA Company. This research also proposes a design method using automatic palletizing simulate layout for the
SCARA robot arm, and verifies the precious positioning of the SCARA robot arm.
Keywords: DFM, DFV, SCARA robot arm, DRL

presented a binocular stereo vision to decide the desired
location of the SCARA robot arm [2]. Cao et al.
designed a 5-DOF SCARA robot arm for welding, andbuilt the model and the kinematic equations using D-H
method [3].
The paper is organized as follows: Section II
describes the system architecture of the SCARA robot
arm, and explains the functions of the PLC-based
controller (ASDA-SM). Section III describes the
specifications and operation range of the SCARA robot
arm. Section IV presents simulation result andexperimental scenario for the SCARA robot arm tocatch a pen to verify the working precision. Section VI
presents the brief concluding remarks.

1. Introduction
A robot arm is a mechanical device that is driven bysome electronic motors or pneumatic actuators. A welltrained robot arm can be instead of laborers to complete
assigned tasks automatically. The purpose of the paper
is to implement a four-degree-of-freedom SCARA robot
arm, and finishes the assigned tasks. In the control
aspect, a PLC-based (ASDA-MS system) controller isused to control the SCARA robot arm.
There are some researches regarding the robot arm in inrecent years. For example, Su et al. designed a sevenjoint robot arm using PC-based controller. The robot
arm can catch the pencil to write the assigned words on
the platform, and plotted the house outlook, and built achurch using some wooden blocks [1]. Sim et al.

2. System Architecture© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

436

Jr-Hung Guo, Kuang-Wei Chuang, Kuo-Lan Su

The system architecture of the SCARA robot arm
system is shown in Fig. 1. The implementation system
contains a PC-based computer, a PLC-based controller(ASDA-SM), four AC servomotors, a solenoid and agripper and a human machine interface (HMI). We use a
ASDA-SM controller 、 a ball spline 、 four AC
servomotors、 a solenoid and a gripper to integrate the
SCARA robot arm. The solenoid drives the gripper cancatch the assigned object.
The core of the program language is DRAS. The DRAS
is developed by the DELTA Company, and is matchwith the national standard communication protocol(IEC61131-3), and integrates the PLC control system,and parameter setting of each servo-controller (ASDAMS), and the driver software of the HMI. The functionof DRAS can communicates with the PC system via
Modbus interface, and processes the vision signals
through Emgu system using Visual Studio C # language.

range of each joint. “J” part is the digital signals with 24
points for users. “K” part connect with the encoder of
each servomotor as feedback signal and measure thereal-time rotation angle.

Figure 2. PLC-based controller (ASDA-SM)

3. SCARA Robot Arm
The SCARA robot arm has four DOFs, (Degree ofFreedom) to be shown in Fig. 3. The first and second
joints rotate along the Z axis. The rotation radius of two
joints is the same to be 205mm. the rotation angle of the00
first joint is 130 , and the second joint is 150 .
The movement displacement of the third joint is 150mm.
0
the rotation angle of the fourth joint is 180 . The
specifications of the SCARA robot arm are shown in thetable1. The prototype of the SCARA robot arm is shownin Fig. 3. The operation range of the robot arm is shownin Fig. 4. User can program various working space ofthe robot arm using the simulation software.

Figure 1. The SCARA robot arm

The prototype of the controller (ASDA-SM) shows in
the Fig. 2. We explain each function of the controller.
The controller can use MODBUS, RS485 or RS232
interface to connect with the computer. “A” part is thecommunication port with the HMI device. The
controller can use EtherNet, USB1, USB2 or DMCNET
interface to connect with the computer in the “B”. “C”
part is the standard AC power input (R,S,T). “D” part isthe DC power input (24V) and enable of the servomotor.
“E” part is output terminal with digital signals.
“F” part is enable the DC power input (24V). “G” partconnects to the ground to protect the controller .Four
AC servomotors will connect with the part “H” of thecontroller. The limit positions of each servomotor
connect with the part “I”, and decide the moveable

Figure 3. Prototype of the SCARA robot arm© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan437

Development of Four-axis SCARA

In the assigned task, the SCARA robot arm cancomplete various assigned tasks such as coming andgoing on two points or many points. In the first task, wewill test the moving precision and positioning precious
for the SCARA robot arm to be right. Then user can use
the robot arm to work in automation field.

distance of each concentric circle is 20mm shown in Fig.
7. The operation processing of the robot arm is from top
to bottom, and touches each point is from left to right.
First we can make the simulation method to verify the
robot arm to do the assigned task. The simulation results
are shown in Fig. 5. The robot arm runs the processing
from the first point that locates on the upper left cornerof the working space. We can see the robot arm touch
the tenth point shown in Fig. 5(a). Then the robot arm
touch the twenty point、the thirty point、……、theseventy point and the eighty point shown in Fig. 5(b)(h).

Operation
range

Table 1. Specifications of the SCARA robot arm
Axes number
4
Degree of freedom
4 Dof*
Length(Join 1+Join 2)
410mm(205mm+ 205mm)
Nor.
1 kg
Loading
Max.
3 kg
J1
±130°
J2
±150°
J3
150mm
J4
±180°
J1+J2
±0.01mmJ3±0.01mm
Precision
J4±0.01°
Radius of ball spline
Weight

(a)

(b)

(c)

(d)

(e)

(f)

16ψ
About 18 kg(g)(h)
Figure 5. Simulation results of the SCARA robot arm

In the experimental scenario, the SCARA robot arm
catches a pen to touch eighty points with concentric
circles. The working space and the relation distance ofeach working position are shown in Fig. 6. The SCARA
robot arm must control the pen moving to known
position of each point, and falls down to touch theassigned point. The robot arm touches the first point
shown in Fig. 6(a). Then the pen of the robot must riseup and move to the next point to do the same processing.
The robot arm touches the tenth point、the fifty point

Figure 4. Operation range of the SCARA robot arm

4.

Experimental Results

We implement the operation precision of the SCARA
robot arm. In the experiment, the robot arm catches a
pen to touch on eighty positions with concentric circles,
and see the offset distance from the centre point. The
maximum radius of each concentric circle is 5mm. the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

438

Jr-Hung Guo, Kuang-Wei Chuang, Kuo-Lan Su

and the eighty point shown in Fig. 6(b)-(d). Finally we
can see the experimental result show in Fig. 7. The
robot arm can control the pen to touch the centre
position of each concentric circle.

5. Conclusion
The papers implemented the working precision for a
four joints SCARA robot arm, and controlled the robot
arm using PLC based system. The PLC based control
system is ASDA-MS that is produced by DELTA
Company in Taiwan. We calculated motion
displacement and rotation angle of each joint according
to the known position of each concentric circle. In theexperimental results, the SCARA robot arm used thesimulation method to verify the control processing to beright. Then the robot arm catch the pen to test thepositioning precious to be nice, and finished to toucheighty points on the working space.

(a)Acknowledgements
This work was supported by the Ministry of Science andTechnology in Taiwan, (MOST 106-2221-E-224-019).
References

(b)

(c)

1. S. T. Su and K. L. Su: Development of the Motion
Control for a Robot Arm, International Journal of
Research and Surveys (ICIC Express Letter, Part B：
Applications), Vol. 7, No. 3(2016), pp.717-724.
2. H. S. Sim, Y. M Koo, S. H. Jeong, D. K. Ahn and B. N.
Cha: A study on visual feedback control of SCARA robot
arm, International Conference on Control, Automationand System (2015), pp.1268-1270.
3. K. L. Su, B. Y. Li and C. H. Chang：Vision BasedMulti-pattern Formation Exchange of Mobile Robots,
International Symposium on Robotics (2012), pp.196-200.
4. K. L. Su, B. Y. Li and C. H. Chang：Vision BasedMulti-pattern Formation Exchange of Mobile Robots,
International Symposium on Robotics (2012), pp.196-200.

(d)Figure 6. Working process of the SCARA robot arm

Figure 7. Experimental result of the SCARA robot arm© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan439

Novel Detection Scheme for Stolen Password File
I-Hsien Liu

Department of Electrical Engineering, National Cheng Kung University, Tainan City, 70101, Taiwan
Chia-Hsiu Chen

Institute of Computer and Communication Engineering, National Cheng Kung University, Tainan City, 70101, Taiwan
Jung-Shian Li

Department of Electrical Engineering, National Cheng Kung University, Tainan City, 70101, Taiwan
E-mail: ihliu@cans.ee.ncku.edu.tw, chchen@ cans.ee.ncku.edu.tw, jsli@mail.ncku.edu.tw
www.cans.ee.ncku.edu.tw

Abstract
The use of traditional password is the most common and important authentication credential for today's online society.
How to avoid password cracking and detect the event of stolen password file has become an important information
security issues. Accordingly, our research improves the Honeyword system proposing a new password storage
mapping method. If the password file is stolen, an attacker could not know any user’s correct password. We apply toOpen LDAP Server to prove its feasibility.
Keywords: System Security, Password Security, Honeyword System, Open LDAP.

past few decades, the content of this certification
mechanism hasn’t been much change.
In recent years, information security issues have
gradually been taken seriously, system security has
become one of the considerations in the use of services
by users. In particular, the password is the first line ofdefense, but the file leaks are endless. These common
password leak events, causing the user's personal
information to be stolen. Not only the occurrence ofidentity theft, but also cause serious property damage.
According to Gemalto's 2016 report [1], the most serious
is the account access, followed by identity theft.

1. Introduction
Over the past decades, with the popularity of personal
computers and mobile devices, the convenience of theInternet, and the development of social networks, more
and more people rely on the Internet to deal with or sharethings on life, such as online payment, online shopping,etc. The school also provides a variety of online services,such as course enrollment system, online courses, etc., toprovide teachers and students more convenient campus
life. All of the above are need to use the user
authentication to achieve identity confirmation in order
to use the appropriate application.
Password, still the most important credential for today's
account certification. Therefore, it plays a very importantrole in human life. The evolution of the password so far
has developed a variety of forms to achieve certification.
For example, fingerprints, sound waves, retina and so on.
However, the traditional digital password authentication
credential is still widely accepted by the public, but in the

2. Background and Related work
2.1. Open LDAPOpen LDAP is an open source implementation of theLightweight Directory Access Protocol (LDAP). LDAP
is a subset of the X.500 directory standard protocol andis a cross-platform standard transport protocol. It is not
only available on demand, but also simpler than the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan440

I-Hsien Liu, Chia-Hsiu Chen, Jung-Shian Li

X.500 and supports TCP / IP. It not only retains theadvantages of the X.500 directory, but also reduces theoverall cost of management, so that the system access todirectory services can be more convenient.
The LDAP data model is stored in a tree-like structure,providing faster search efficiency. Ldap has many
advantages, such as the user can communicate with eachother through a common protocol, so that the host can
exchange the data between different protocols or nonuniform data format. In addition, the manager canclassify the data in the directory and control the user's
access rights, so that they can provide security to protect
personal data [2].
OpenLDAP stores the password in the userPassword
attribute, and provides a variety of password encryption
options, such as: SSHA, CRYPT, MD5, SHA, etc., sothat password storage is more secure. During theauthentication process, slapd in OpenLDAP will look forthe password stored in the list until it matches the
password provided by the user, or until all the values in
the userPassword are searched, and then given the user
related access rights according the result based on thequery.

Fig. 1. The password file structure

Gen (k) will produce a set of fake passwords Wi = ( wi,
1, wi, 2, ..., wi, k,), which contains the correct password
for the new user wi, ci = pi. And further for security
reasons, the index table c will be stored on a third-party
server, independent of the login server. The encrypted
password list will be stored as Hi = (vi, 1, vi, 2, ... , vi, k),where j-th will be the real user's password wi, j.
In the Honeyword system, a very important part is
Honeyword generator. In [3] this paper, the author
describes the three ways to generate the password,
respectively, as follows:

2.2. Honeyword System
The concept of Honeyword was first introduced by Juels
and Rivest [3]. Their main concept was to design adefense mechanism to detect whether a password file wasleaked. In the paper, the author assumes that the attacker
can obtain the password file and the ability to convert the
password into plaintext. The principle of Honeyword is
to store the user's real password with the (k-1) fakepassword generated by the Honeyword generator, so thepassword file is stored in a format with a user withmultiple possible passwords. As a result, even if thepassword file leaked, the attacker cannot know which one
is the user's real password. If he wants to log in as a fake
identity, there is a great possibility of entering a fakepassword, so the system will find the password file may
be compromised. At this point, the system can trigger thealarm based on the safety policy set by the administrator,and notifying the administrator to make thecorresponding deal. The password file structure is shownin Figure 1.
The principle of Honeyword is as follows: when a newuser registers, he will submit a set of account ui andpassword pi to the system, then the Honeyword generator

2.2.1. Chaffing by tweaking
A way to replace a selected character position, whichmust be replaced by the same type of character, such as
letter-to-letter, symbol-to-symbol, digit-to-digit, and soon. One of the common practices is "chaffing-by-tailtweaking", which is a way to change the last few digits togenerate Honeywords. For example: password123, if you
choose to change the last two digits and generate three
Honeywords, the Honeyword generator may generate apassword list of W = {password135, password148,
password123, password107}.
2.2.2. Chaffing-with-a-password-modelThis method is using the same syntax module to do thereplacement, that is, the group as a unit, such as: lettergroup, digital group. To the above example:
password123, will be divided into W8 | D3, that is, 8
letters 3 digital form. Assuming that you want to generate
two Honeywords, it is possible to generate a password
list of W = {chaffing345, keyboard987, password123}.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

441

Novel Detection Scheme forThis will be more effective than "Chaffing by tweaking"because it makes Honeyword more realistic and closer tothe real password, so that the attacker cannot tell thedifference between the real password and Honeywords,
increase the possibility of entering the fake password.
2.2.3. Chaffing with “tough nuts”
This kind of Honeyword is a way to confuse the attacker,
this password will increase the length of the characters or
the complexity of characters composition, it can improvethe difficulty of password cracking, thereby increasing
the attacker's attack time.

Fig. 2. System Architecturethen respond the privilege to the login. As described in
Introduction, it is possible to know that an attacker cansteal a user's password in different ways, such as sniffing,sending malware, and so on at different layers in the
network architecture, and then they can crack the
password by brute force, dictionary attack and different
password crack tool. So in this case, we assume that theattacker:Has the ability to obtain the real user-related accountpassword file in the system, andHas the ability to get the password in plain text andwants to log in the system.

There is also a way that Honeyword is produced in amixed way, such as "chaffing-by- tweaking-digits" with"chaffing-with-a-password-model". In this way, you canfirst generate a group of models, and then generate bpasswords by replacing digits, so that will form k = a x b
passwords. The strength of this approach will be stronger
than using only one way to generate Honeywords,
improving the security of the system. Another important
security device in the paper is a third-party security
server, where the author will set the honeychecker tostore the user and its index value. It is independent of thelogin server and is used to confirm the index value of theuser’s password.

Our system architecture is mainly composed of threeparts, including the user, administrator and database, thesystem architecture is shown in Figure 2.
The manager section mainly manages the registration of
the user and the login authentication, and makes
appropriate processing when the login abnormal. In orderto make the password authentication system more secure,the administrator needs to limit the password setting
according to the management policy, for example, theminimum characters of the password length, theminimum types of the password characters and so on.
The last is the database, which mainly contains two tables.
A data table to store the user's account, the hash function
after the password, another data table is the password
checklist, store the user ID and user’s true password
index.
In order to verify the feasibility of this certification
mechanism, we put this design running on the existing
OpenLDAP version as our server. We chose Open LDAP
because it is an open source directory server, not only
based on the X.500 standard, but also support TCP / IP,

3. Mapping-password System
With the importance of identity authentication to modern
networks, the security and reliability of cryptographic
authentication is becoming increasingly important.
However, the user and the password they use are still the
weakest part of the security mechanism since theappearance of password authentication [4]. Therefore, inorder to enhance the security of certification, we propose
a mapping-password system in accordance with theconcept of Honeyword system [3]. We put the user's
password in the system in random order, so that the user
password of the mapping account is not the original user
set on the surface. This not only retains the detection
mechanism of the password file leakage, but also reduces
the system storage cost.
When a real user wants to use the system service, they
will establish their own account, provide their own username and password to the system. The system will bebased on this name and password to verify the login, and© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan442

I-Hsien Liu, Chia-Hsiu Chen, Jung-Shian Li

Table 1. System Comparison.

and simplifies the implementation method. Whether incompanies or schools it is widely used, so it is important
to improve the security of OpenLDAP. Then we choose
MySQL as our database, which is because although
MySQL is a related database, but it is open source, andthe algorithm is efficient and reliable.

5. Conclusion
The user is not easy to remember too complex password,
and will use the same password in different accounts,
resulting in the password is still easy to be cracked. We
refer to the idea of Honeyword system [3], propose amapping-password system to improve it. According to
the experimental results, we can see that there is no
obvious performance ratio between the original systemand our system, so it will not bring too much overhead tothe system. We believe that our proposed system will beable to effectively improve the security of password
authentication, and can detect the account leak event,effectively prevent the attacker intrusion, so that the leak
event is no longer belatedly discovered.
Acknowledgements

4. Performance Evaluation
We compare the differences between the traditional
system, the Honeyword system, and the system weproposed. As you can see in Table 1, our passwords are
stored in the same way as traditional systems, and the
Honeyword system stores multiple passwords for a single
user. While our system requires additional ID and index
values compared to traditional system storage costs, but
we reduce the cost of storing additional t passwords foreach user compared to the Honeyword system, greatly
reducing the system's storage cost.
In the probability of attack success part, assuming thatthe attacker has obtained the password file, andsuccessfully cracked into a plaintext password, then theprobability of attack success is 100% in the traditional
system. That is because in the traditional way ofpassword storage, the username maps the real user
password. Honeyword system is based on the number offake passwords and the flatness of fake passwords todetermine the probability of attack success. The more thefake password, the smaller the probability that each user
will be successfully attacked, but the extra storage will be
required. As for our system, as the number of people rise,
the probability of attack success will become smaller, anddo not need to spend additional storage cost.

This work was supported by the MOST (Ministry of
Science and Technology) under contracts numbers
MOST 106-3114-E-006-003- and MOST MOST 1062221-E-006-033 -.
References
1. Gemalto, 2016 Mining for Database Gold - Findings fromthe 2016 Breach Level Index, (Gemalto, 2016).
2. The OpenLDAP Foundation, OpenLDAP Software 2.4
Administrator's Guide,( The OpenLDAP Project, 2017).
3. A. Juels and R. L. Rivest, Honeywords: Making passwordcracking detectable, in Proc. 2013 ACM SIGSAC (Berlin,
Germany, 2013), pp. 145–160.
4. L. Tam, M. Glassman and M. Vandenwauver, The
psychology of password management: a tradeoff between
security and convenience, Behaviour & Information
Technology, 29(3) (2010) 233-244.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan443

Honeypot System of SCADA Security Survey
Kuan-Chu Lu

Department of Electrical Engineering, Institute of Computer and Communication Engineering,
National Cheng Kung University, Tainan
I-Hsien Liu

Department of Electrical Engineering, Institute of Computer and Communication Engineering,
National Cheng Kung University, Tainan
Jung-Shian Li

Department of Electrical Engineering, Institute of Computer and Communication Engineering,
National Cheng Kung University, Tainan
E-mail: kclu@cans.ee.ncku.edu.tw, ihliu@cans.ee.ncku.edu.tw, jsli@mail.ncku.edu.tw

Abstract
Now the infrastructure employs SCADA system for monitoring, such as nuclear power plants, water conservancy
bureau, manufacturing, and logistics industry. With the rapid development of the internet, IOT (internet of things)
turns every massive infrastructure from a concealed network to an open access network for convenience.
Consequently, the system faces attack threats never occurred before. That is the reason why we need to build a safeand protected scheme for prevented the external attack and have to avoid SCADA system crashed. For protecting theimportant assets, we should understand the tools can preserve system safety to prevent the attackers invading the
SCADA system. This study surveys defensive mechanisms of varying SCADA system and proposed an effective
defensive structure to protect the SCADA system.
Keywords: Infrastructure security; SCADA; Industrial control Security; Network Security; IOT; Honeyd.

1.

capabilities are all developed through the development of
network infrastructure and information technology.
In other words, the convenience and rapid popularity of
the Internet which makes the critical infrastructure andusers’ key information rely on computers and networks
working properly. But, also be accompanied by multitypes network security threats and the challenge of how
to protect. In addition, hacker attacks have been on therise in recent years. In 2014, a report from Industrial
Control Systems Cyber Emergency Response Team,ICS-CERT pointed out that a total of 257 attacks had
been conducted on key infrastructure, such as energy
facilities, key manufacturing facilities, water resources

Introduction

With the rising of critical infrastructure and the thriving
Internet, all the energy production and distribution (suchas nuclear power plants, gas pipelines, and power grids
as well) to utilities (such as water, wastewater processing,water transport, transportation, and manufacturing) rely
on Industrial Control Systems, ICS. And the Supervisory
Control and Data Acquisition, SCADA, system (forremote control, remote monitoring, and data collection aswell) is a special type of industrial control system. Its
main orientation is for the wide area of industrial system[1]. The above data processing and communication© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan444

Kuan-Chu Lu, I-Hsien Liu, Jung-Shian Li

facilities,
transportation
facilities,
information
technology systems, nuclear power plant equipment,
communications
equipment,
andgovernment
departments in United States in 2013. Among them,energy facilities, key manufacturing equipment, water,transportation, science and technology and power plants
are account for a relatively high proportion. Obviously,
the hackers attacked mainly the targets these criticalinfrastructures [2].
At present, enterprises and government units usually
carried out the real-time monitoring and defense against
the complicated and changeable network attacks to resist
the invasion of attackers. Information security
departments mostly use information security equipment
such as Intrusion Detection System, IDS, or Intrusion
Prevention System, IPS, to resist potential threats.
However, this usually detects only the loopholes in themalware attacks on computers and devices. But asmalware attacks evolve in an endless stream, they will becompromised by malicious software programs [3].
The new technologies are needed to identify themalware- style attacks and at the same time to achieve
the effect of being able to deceive the attacker. In otherwords, its purpose is to entice an attacker's invasion,
expose the location of the vulnerability and monitor theattacker's location which can help the information
security managers fix quickly and learn about attacking
styles. And the honeypot technology can achieve theserequirements [4] [5]. An outstanding honeypot design
must have the professional ability and good technicalcapacity, this has made honeypot design a challenge forthe most experienced information security softwarearchitects [3]. Therefore, this article will introduce the
SCADA first, following by what is honeypot, theevolution of honeypot, and industrial system honeypotetc. Then, put forward how to install the honeypot on the
SCADA system structure, to introduce to the install
process. Finally, to discuss how to prevent and ensure
the system has a stable and secure environment.

2.1. Introduction of SCADASupervisory Control and Data Acquisition, SCADA, is amonitoring and data collection system. It is a structure
system dominated by information network technology inIndustrial Control Systems, ICS. The SCADA is mainlycomposed of Human Machine Interface (HMI),
Engineering Workstations, LAN, database system, routerdevice, Programmable Logic Controller (PLC),Intelligent Electronic Device, IED, a set of remoteterminal devices. As shown on Figure 1 [6].

Fig. 1. Basic SCADA System Structure Diagram [6].

2.2. The Evolution of Honeypot
Honeypot hardware was quite expensive before 2006.
Numerous researches, including honeynet project,needed to have the study topics and find workable cases[7], plus with the issues such as honeypot deployment
and maintenance cost, leading to the restricted
development. However, virtualization technology had
been gradually popularized from 2006 to 2008. This
allowed the virtual machine to act as a honeypot andcollect detailed data on network-threat attacks with low
cost and convenience, however, honeypot technology
was unable to be transferred to industrial control systems.
Therefore, there will be another dedicated honeypot on
the industrial control system. The following will
introduce the industrial control system honeypot andhoneyd.

2. Literature Review
This section will explore the SCADA's basic equipment
and structure, meanwhile to describe what is meant byhoneypot, honeynet origin, the evolution of honeypot,
honeynet industrial system, the formation of industrial
system honeypot etc.

2.2.1. Industrial System Honeypot
Honeyd could not be built into the industrial control
system environment because it needed to have a© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

445

Honeypot System of SCADA

dedicated honeypot software to build and run [8], forexample, conpot and gaspot are all honeypots speciallyused for industrial control systems. The main reason isthat the communication protocol on the two trapping
systems is Modbus rather than TCP / IP. In addition, thetest environment is usually a more special environment,such as power companies, water companies, oil farms,
transport industries, factories, etc. It needs to be adjusted
and set up according to different communication
protocols in different fields, and different with thetraditional honeypot structure.

data, with a huge amount of information and the ability
to influence national power causing as the target by cyberhackers, terrorist organizations and the competition
among countries. Therefore, this section will explore the
SCADA system on the security protection to understandthat the system has suffered serious incursions which
may threaten the livelihood of the people and the national
information security, also to provide the safety
information to the relevant researchers. And put forwarda honeynet structure diagram to prevent SCADA systemfrom being attacked.

2.2.2. Honeyd3.1. SCADA System Defense Method

Honeyd is an ideal framework developed by Niels Provos[9], a free software provided by GNU. Users do not needto deploy dozens of expensive devices. Users can set up
and run multiple virtual machines on their own computer
network. In other words, users can configure multiple
virtual machines, while simulating a variety of different
types of server which allow users to simulate theunlimited numbers of computer environments.
Mainly for the security software to avoid network virus
intrusion, including the famous Code Red, Slammer andBlaster worms.

It is urgently needed to prevent cyber attacks throughtaking steps to strengthen its defense because theenterprise network may attack the SCADA network.
Most of the security measures could not defend against
network attacks. Network attacks from the SCADA
agreement into the SCADA system [11] because thelimited computing power of current features of SCADA,
resulting in low data rates and difficulty in applying theexisting cryptographic techniques [12]. The National
Institute of Standards and Technology and European
Network and Information Security Agency developed
especially network security standards for SCADA
system to prevent hackers attack and control the disaster.
3.2. SCADA System Honeypot StructureThis study relied on the basic SCADA structure as shownin Figure 3. Because many network attacks will enter thecentral control of the SCADA system via the TCP / IP
network to achieve the control system or implant thevirus code for the purpose. Therefore, it needs to set up
the fake environment in honeypot interface of TCP / IP
which would let the attackers mistakenly think they haveentered the control system which could protect thedefense system itself. Furthermore, it can also collect theattackers’ attack paths and ways to achieve part of theouter network defense effect of SCADA system.
In the industrial system, the industrial system honeypot
could be set up in the industrial control system
environment to prevent attackers to skip TCP / IP andattack the system directly, and simulate the environmentof industrial control equipment, so that attackers could
not identify the authenticity of the system to achieve thepurpose of defense systems and data protection.

Fig. 2. Basic Honeyd Structure Diagram. [10]3. SCADA System Security
In recent years, the critical infrastructure in many
countries has been threatened by anonymousorganizations. As the times change, the SCADA system
has evolved into an open system from a previously closed
system, also it can be applied to the network tomanipulate other machinery and equipment, and collect© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

446

Kuan-Chu Lu, I-Hsien Liu, Jung-Shian Li

References
1. Serbanescu, A. V., Obermeier, S., & Yu, D. Y. (2015). A
flexible architecture for industrial control systemhoneypots. In e-Business and Telecommunications(ICETE), International Joint Conference, 4, 16-26.
2. ICS-CERT, (2014). ICS-CERT Year in Review. [onlie].
Available:https://ics-cert.uscert.gov/sites/default/files/Annual_Reports/Year_in_Revi
ew_FY2014_Final.pdf. [Access date: 18 Aug. 2017]
3. Ahmed, H. M., Hassan, N. F., & Fahad, A. A. (2017).
Designing a smartphone honeypot system using
performance counters. Karbala International Journal ofModern Science, 3(1), 46-52.
4. Ahmed, H. M., Hassan, N. F., & Fahad, A. A. (2013). A
Survey on Smart Phone Honeypot. International Journalof Computers & Technology, 11(4), 2476-2480.
5. Assmaa, A. F., Hanaa, M. A., & Nidaa, F. H. (2014).
Design requirements of a smart phone honeypot system.
International Journal of Computer Application, 6(4), 97104.
6. Stouffer, K., Falco, J., & Scarfone, K. (2011). Guide toindustrial control systems (ICS) security. NIST special
publication, 800(82), 16.
7. Spitzner, L. (2003). The honeynet project: Trapping thehackers. IEEE Security & Privacy, 99(2), 15-23.
8. Winn, M., Rice, M., Dunlap, S., Lopez, J., & Mullins, B.
(2015). Constructing cost-effective and targetableindustrial control system honeypots for production
networks. International Journal of Critical Infrastructure
Protection, 10, 47-58.
9. Provos, N. (2004). A Virtual Honeypot Framework.
In USENIX Security Symposium, 173, 1-14.
10. Openstuff Wiki, (2013). HowtoHoneypot. [onlie].
Available:
http://stankiewicz.free.fr/Wikka/wikka.php?wakka=Howt
oHoneypot. [Access date: 2 Nov. 2017]
11. Fovino, I. N., Carcano, A., Masera, M., & Trombetta, A.
(2009). An experimental investigation of malware attacks
on SCADA systems. International Journal of Critical
Infrastructure Protection, 2(4), 139-145.
12. Igure, V. M., Laughter, S. A., & Williams, R. D. (2006).
Security issues in SCADA networks. Computers &Security, 25(7), 498-506.

Fig. 3. Basic Honeypot Structure.

4.

Conclusion

Basically, SCADA systems could be used in criticalinfrastructure systems such as power generation, water
conservancy, and transportation. Therefore, a goodstructure and strategies to protect the system's network
security from threats and attacks is needed. The system
will be broken by new threats from the tremendous
changes of cyber attacks if the security technologies
adopted not mature enough. Therefore, this study
proposed a framework of using the honeypot for defense
plan approach. The main effect is to simulate thesimulation to deceive the enemy, in addition, the system
could also collect the attack paths and attack ways tounderstand the loopholes of system and safety.
In other words, system developers need to follow theplan-do-check-act procedure. First, to consider which
network vulnerabilities are the easiest to compromise
when coming the cyber attacks and use the attacker's
thinking view to take protective measures and plan thesystem security guidelines to resist threat attacks, then, tochoose the information security defense which is very
important to the system developer who is not only
familiar with the strategy of defense security, but need tounderstand the system threats that may pose an attack, so
as to be able to protect the security of SCADA system.
Acknowledgements
This research plan of MOST106-3114-E-006-003- andMOST106-2221-E-041-003-could not be completed
without the financial support of Ministry of Science andTechnology, Taiwan, R.O.C.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan447

An EtherCAT battery test system
Chung-Wen Hung*, Bo-Min Wang, Wen-Ting Hsu, Jhen-Gu Jiang

Department of Electrical Engineering, National Yunlin University of Science &Technology ,123 University Road,
Section 3, Douliou Yunlin 64002, Taiwan, R.O.C
E-mail: wenhung@yuntech.edu.tw

Abstract
A battery test system is a key equipment to ensure lifetime of batteries and portable device performance. On the other
hand, because the real time requirement of the in industry 4.0, EtherCAT (EtherNet Control Automation Technology)
communication is considered a standard interface of industry 4.0. The EtherCAT interface is based on Ethernet
hardware, and with the advantages such as larger transmission bandwidth, shorter update time and minimization on
synchronization time jitter. In order to upgrade battery test system competitiveness in the future, this paper proposed
an EtherCAT interface for an original battery test system. The TwinCAT (The Windows Control and Automation
Technology) environment Developed by Beckhoff is adopted to be the EtherCAT interface Master, and Renesas EC1 evaluation board is used as EtherCAT interface slave controller. The commands are sent to the battery test module
through the RS485 interface from the EC-1 evaluation board. Finally, C# software is proposed to provide friendly
GUI interface for user.
Keywords:Battery test system, EtherCAT, TwinCAT, Renesas EC-1

slave controller. The commands are sent to the battery
test module through the RS485 interface from the EC-1
evaluation board. Finally, C# software is coded toprovide friendly GUI interface for user. Market is a lackof battery test equipment with EtherCAT communicationinterface. This paper can help Battery package
manufacturers to establish new automation production
line based on Industry 4.0 standard. The systemarchitecture and software will be detailed, andexperimental results will be present to show system
workable.

1. Introduction
Due to global warming and Energy shortage, renewableenergy is becoming increasingly important. A battery test
system is a key equipment to ensure lifetime of batteries
and renewable device performance. On the other hand,because of the real time requirement of the in industry 4.0,
EtherCAT communication is considered a standard
interface of industry 4.0. The EtherCAT interface isbased on the Ethernet hardware, and with the advantages
such as larger transmission bandwidth, shorter update
time, and minimization on synchronization time jitter.
New automation equipment is asked to build in the
EtherCAT communication interface. In order to upgrade
battery test system, this paper proposed an EtherCAT
interface for a traditional battery test system. The
TwinCAT environment developed by Beckhoff is
adopted to be the EtherCAT system Master, and Renesas
EC-1 evaluation board is used as EtherCAT interface

2. Related Research
2.1. EtherCAT and CoE
There are some studies discussed about the advantages of
EtherCAT. EtherCAT is proposed to be a new industrial
communication protocol in these years. Compared toother industrial communication protocol like Modbus,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

448

Chung-Wen Hung1, Bo-Min Wang, Wen-Ting Hsu, Jhen-Gu Jiang

CAN, and Profibu, EtherCAT is more reliable and more
immediate. EtherCAT upgrade a level for network
performance by DMA technology of reading network
data, all data are processed by hardware [1]. For example,
EtherCAT master captures input/output signal, slaveconnect up to ten devices, but the EtherCAT system delay
less than a millisecond [2]. In EtherCAT, theCoE(CANopen over EtherCAT) protocol is used totransport PDO(Process Data Object) defined in
CANOpen. CANopen has been already used widely inmany field, including process data(CiA401), motor
control(CiA402), and other applications (CiA4XX).
EtherCAT follows CANOpen protocol and keeps its
advantages. [3] verified feasibility of CoE. A successful
CoE case, a TwinCAT-based servo motor control system,
was discussed in [4].

Fig. 1 is the TwinCAT-based EtherCAT battery test
system architecture proposed in this paper. EtherCAT
interface and system software is designed for a maturebattery test module. The commands are sent to thebattery test system through the RS485 interface fromthe EC-1 evaluation board. EC-1 is EtherCAT
communication interface and used to exchange thesystem information with EtherCAT Master,
TwinCAT. Finally, C# GUI software is coded tomonitor and control the battery test system.
Table 1. EtherCAT Specification
EtherCAT Master

Platform

Windows 7 PC

EtherNet Adapter

Intel I350-T2

Master Software

TwinCATGUI Software

2.2. Battery test system

Slave Hardware

A battery test system is used to charge or discharge thebatteries, and measure the parameters of a battery,
including voltage, current, temperature, and electricity. A
study proposed A battery automatic measurement system
which is based on personal computer to measure various
parameters and to build battery model automatically [5].
In recent years, there are some studies focused on the
resolution of the low voltage measurement for lithium
battery. [6] tested the different lithium-ion capacitor
batteries made by three different manufacturers, after
charging and discharging more than 10,000 times atdifferent voltages and temperatures, then evaluate theirlife and performance compared by recording lithium-ion
battery capacity. The above battery test system research
can be used as a reference system.

C#
EtherCAT Slave
Renesas EC-1

Table 1 is the specification of the proposed EtherCAT
system and environment. Master is a Windows PC
with Intel dual port EtherNet adapter I350-T2.
TwinCAT is adopted as EtherCAT master, and it
provides a bridge to C# which provides a GUI
interface. Slave communication is performed by aRenesas EC-1 board. EC-1 builds in an ESC(EtherCAT Slave Controller), it is used to decode andencode EtherCAT packets from master.
4.

Software and Hardware Environment

4.1. C#
C# is a multi-paradigm programming language
encompassing strong typing, imperative, declarative,functional, generic, object-oriented, and componentoriented programming disciplines. The battery test
software proposed in this paper is coding in C#. The
proposed C# program also can control EtherCAT devices
via TwinCAT environment.

3. System Architecture and Specification

4.2. TwinCAT
TwinCAT is an automated software framework
developed by Beckhoff. And it runs on Visual Studio in
the Windows PC. TwinCAT is adopted to be the
EtherCAT interface Master in the system proposed in thispaper.

Fig. 1. EtherCAT battery test system architecture© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan449

An EtherCAT battery test

Fig. 3 presents the flowchart of the proposed firmware.
The left half of the flowchart is the initializations of
EtherCAT Slave address and PDO object. After theconfiguration, the right half of the flowchart is a cyclic
task including EtherCAT PDO Read / Write and RS485
communication.

4.3. EC-1
EC-1 is the EtherCAT slave communication board
developed by Renesas, there is an ESC on the board. EC1’s development environment is recommended IAR
Embedded Workbench what is provided by IAR System.
5. Interface Implementation

5.3. GUI battery control software
The GUI application software is coded under theMicrosoft Visual Studio C# development environment. It
has powerful compilation capabilities such as C / C++. In
addition, it is also object-oriented and has a rich libraryof components that speed up application development.

5.1. EtherCAT masterIn this paper, TwinCAT is used to control the EtherCAT
system. And the describe file .xml for the battery test
system is necessary. Fig. 2 shows the operation interfaceof the TwinCAT program. The right portion indicatesconnected Slave objects, and the left screen shows theaccessing results with the selected object.

Fig. 2. TwinCAT

5.2. EtherCAT slave

Fig. 4. GUI battery control software

The EC-1 firmware proposed in this paper performs
communication interface between the battery test system
hardware and TwinCAT. The firmware includes the
EtherCAT data link layer and the Cia401 PDO object
dictionary function. Finally, the testing commands aresent to the battery test module through the RS485
interface from the EC-1 evaluation board.
Start

MasterReceive

EtherCAT
Config

PDO
Read

RS485
Write

Cyclic
Task

PDO
Write

RS485
Read

The proposed software shown in Fig. 4 provides a userfriendly human-machine interface. The battery control
software communicates with TwinCAT. The battery
control software can read battery parameters via
TwinCAT, it also sends charging or discharging
commands by TwinCAT. Moreover, the control program
also has battery protection settings, battery condition
setting, curve data display, and Excel data table functions.
6. Experimental Results
The experimental results are presented in this section.
After the TwinCAT system opening, and the EC-1 andbattery testing module are turned on, the proposed GUI
software can be used to control the battery test module.
The GUI software includes the TwinCAT connection
function, and it also can be used to set module in thestandby mode, charging mode or discharging mode. The
mode changes from standby mode to charging mode asshown in Fig. 5. The battery charging mode is CC-CV,
which indicate constant current and constant voltage,

MasterSend

Fig. 3. EC-1 firmware flow chart© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan450

Chung-Wen Hung1, Bo-Min Wang, Wen-Ting Hsu, Jhen-Gu Jiang

respectively. The battery charging runs in CC mode until
the battery voltage is charged to the target value, then in
CV mode.

Fig. 6 shows the operating mode is from CC to CV andthe battery current decreases and tends to be constant.
The system is set to discharge mode, as shown in Fig. 7,
and the battery runs in CC mode. In CC discharge mode,the battery current is fixed, and the battery voltage
decreases and the slop tends to be constant. The above
results show that the EtherCAT battery test system works
precisely and correctly.
7. Conclusion
The EtherCAT communication interface is proposed inthis paper, the ESC firmware for EC-1 and .xml file in
TwinCAT master for the battery test system aredeveloped. Also, the application C# program is provided.
The battery system is built in the EtherCAT interface, andit will have the EtherCAT advantages, like as great
reduction of the synchronization time between battery
channels and flexible topology. Moreover, the integrity
and richness of the proposed C# control program help
users to use this battery test system easily. The
experimental results show the proposed system workable.

Fig. 5. Battery start to charge

Acknowledgment
This work is partially supported by the Ministry of
Science and Technology, ROC, under contract No.
MOST 106-2221-E-224-025, and 105-2622-E-224-017 CC3.
References
1. EtherCAT Technology Group, “Ethernet & Fieldbus :Beckhoff Real-time Ethernet : Ultra High Speed I/O”,
EtherCAT Technology Group, Jul 01, 2005
2. J. Qi and L. Wang, “Networked Motion Control System
Design Based on EtherCAT”, Intelligent Computation
Technology and Automation, 2009.ICICTA '09. Second
International Conference on, Changsha, Hunan, Oct. 2009,
pp. 77-79
3. Yongming Chen, Hua Chen, Mingzhong Zhang and Yin
Li, “The relevant research of CoE protocol in EtherCAT
Industrial Ethernet,” Intelligent Computing and Intelligent
Systems, 2010 IEEE International Conference on, Xiamen,
Oct. 2010, pp. 67-70
4. Z. Chang-Cheng, X. Jian-Ming, J. Yao, M. Jie and X. LinTao, “Design of servo drive slaves based on EtherCAT,”The 27th Chinese Control and Decision Conference,
Qingdao, May 2015, pp.5999-6004
5. B. Schweighofer, K. M. Raab and G. Brasseur, “Modelingof high power automotive batteries by the use of an
automated test system,” in IEEE Transactions on
Instrumentation and Measurement, vol. 52, no. 4, Aug.
2003, pp. 1087-1091

Fig. 6. Battery charging in CC-CV mode

Fig. 7. Battery discharging in CC mode

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan451

An EtherCAT battery test

6. M. Uno and A. Kukita, “Cycle Life Evaluation Based on
Accelerated Aging Testing for Lithium-Ion Capacitors as
Alternative to Rechargeable Batteries,” in IEEE
Transactions on Industrial Electronics, vol.63, no. 3,
March 2016, pp. 1607-1617© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan452

Surface defect detection for tube object based on automated optical Inspection
Hsien-Huang Wu, Chang-Jhu He

Department of Electrical Engineering, National Yunlin University of Science and Technology
Douliou, Yunlin, Taiwan R.O.C.
E-mail: wuhp@yuntech.edu.tw, m104120 @yuntech.edu.tw
http://el404.ee.yuntech.edu.tw/

Abstract
Extrusion tube is a very popular product in the plastic industry produced by a extruder machine, for example, tubes
used in washer machine or in reverse osmosis (RO) water purification system. To guarantee the quality of the tube
produced for medical use, a surface defect detection system is needed. In this paper, we use a single camera combinedwith a telecentric lens and properly designed mirror set to achieve the complete coverage of the object surface for full
defect detection. Based on the properly designed image acquisition equipment and image analysis techniques, thefinal system for extrusion tube defect detection can successfully detect defects of black spots, bumps, injury, etc.,with a defect detection rate of about 95%, and the detection speed is up to 1578 mm/s. This speed of detection makes
the detection system can be deployed on the production line without interfering the operation of the tube machinery.
.Keywords: Automatic Optical Inspection, Extrusion Molding Tube, Defect Detection, Light Path

1. Introduction
With the gradual mature of industrial technology, thetraditional human control has been slowly transformed
into industrial automation. To achieve industrial
automation, vision is an indispensable technique andtherefore automatic optical inspection (AOI) technology
has gaining popularity in the traditional industry. With
the progress of science and technology, the required
precision and accuracy for product quality are getting
higher and higher, so the use of AOI to improve thetraditional manpower, has become increasingly
important. With the advent of Industry 4.0, combinedwith the reduced cost of the camera and faster processing
efficiency of the compute, the machine vision used in
AOI has also been paid more and more attention. In theimage recognition, applications are divided into four
major categories: detection, measurement, identificationand positioning. Automated detection with highefficiency, high stability, high reliability and so on, has
definitely became the industry's trend.
The tubular imperfections dealt with in this article also
appeared in a number of other applications, such as
cylindrical surface inspection using Line-Scan cameras
for surface scanning.1-3 Paper4 proposed a precision
grinding inspection system of metal surfaces for the

prototype cylindrical roller. Several research has been
conducted for non-flat surface, for example, flaw
detection on the surface of the steel5,6 where precision ishigher than 95%; evaluation of the cone of the trajectory
for a bullet7,8 and real-time defect detection for highreflection surface9 which use mirror for inspection of
specular reflection surface. Another system similar tothat of ours is the inspection of medical glass tube10,
which uses three 8k Line-Scan cameras to capture 360degree image for surface defect detection. All the above
systems were developed for not flat surface, and it showsthat defect detection for a tubular object11-14 need to use
a 360-degree system to capture the image15-18. This isalso true for the extrusion tube that was investigated inthis paper.
2. Hardware architectureThis section will provide a detailed description of thedefect detection system for the extrusion tube, such as thecamera, the lens and the light source, and the 360 degree
imaging system. Our goal is being able to acquire images
for a full coverage of the tube using only one single
camera and proper mirror design. Using the principle ofstraight line and reflection of light, we can obtain foursides of the object by the use of six mirrors. This design© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

453

Hsien-Huang Wu, Chang-Jhu He

is based on OPTO 360 degree optical System19 and was
modified to be used in a continuous object. The rule ofdesign is that all four optical (or light) paths of view needto have the same length, as is illustrated in Fig. 1 as pathL1 (blue), path L2 (light pink), path (green) L3, and path
L4 (red).

the start button for starting the defect detection process.
Load Config
Image Process
Init Camera
UpdateUIStartLiveN

StopLiveY
Y
END

Fig. 3. Main processing procedure
Fig.1. 6M4V Optical path design

According to the above rule that each light path would
be the same and the symbols used in Fig. 1, we have
L1=L2=L3=L4.

The image processing flow of the system is illustrated
in Fig. 4. It first extracts four ROI’s (Region of Interest)
which contain the region of the tube. Then a median filter
is applied to remove the candidate defect. Finally, wetake an absolute value difference (AbsDiff) between theoriginal image and the median filtered image and thedefect area will show up. One example of this processing
is shown in Fig. 5.
Image
Processing

Tube ROI

Fig. 2. Design of 4 view image acquisition system

The equipment implemented is shown in Fig. 2 for 4view acquisition based on six mirrors and one single
camera.

Median filter

3. Software architecture
As shown in Fig. 3, the software system will load theconfiguration file at the beginning of the execution
process. The settings of the parameter include the medianfilter mask size, the background threshold value, the size
range of the defect, and the repeated area of the test object.
The camera is then initialized to wait for the user to press

absDiff

Fig. 4. Flow of image processing© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan454

Surface defect detection for

5. Conclusions
In this paper, a successful design of the mirror to achievea single camera for panoramic ring shot has beenobtained. It can successfully detect the black spots, pits,
crush and other defects for the extrusion tube. This AOItechnology has proven to achieve automated testing,
reduce labor costs, and improve efficiency and quality.
The current speed of detection is 1578 mm/s which canmeet the production rate of a extruder machine for
medical extrusion tube.
Fig. 5 One example of defect detection process
6. Experimental results
The image acquisition device implemented by using 3D
printer is shown in Fig. 6, where six mirrors were used
for four views, and therefore, a 6M4V system. The
images taken with this 6M4V device and the results ofdefect detection are illustrated in Fig. 7.

Acknowledgements
The authors wish to express their appreciation for thefinancial support of the Ministry of Science andTechnology under project MOST 106-2221-E-224-018-.
The extrusion tubes used in the experiments were kindly
supplied by Everplast Machinery Co., Ltd., Taiwan.
Their generous support is greatly appreciated.

References

Fig. 6. 6M4V actual shooting architecture

Fig. 7. Results obtained by using 6M4V

1. Y. N. K. Y. H. MORIKAWA, "Image Processing System
for Visual Inspection of Cylindrical Parts."
2. Y.-C. Chiou and W.-C. Li, "Flaw detection of cylindricalsurfaces in PU-packing by using machine vision
technique," Measurement, vol. 42, pp. 989-1000, 2009.
3. M. Ali, M. Mailah, S. Kazi, and H. Tang, "Defects detectionof cylindrical object's surface using vision system," inProceeding of The 10th WSEAS International Conference
on Computational Intelligence, Man-Machine Systems andCybernetics (CIMMACS'11), Jakarta, 2011, pp. 1-3.
4. F. Pernkopf and P. O'Leary, "Visual inspection of machined
metallic high-precision surfaces," EURASIP Journal on
Applied Signal Processing, vol. 2002, pp. 667-678, 2002.
5. W.-b. Li, C.-h. Lu, and J.-c. Zhang, "A local annular contrast
based real-time inspection algorithm for steel bar surface
defects," Applied Surface Science, vol. 258, pp. 6080-6086,
2012.
6. J. P. Yun, S. Choi, B. Seo, and S. W. Kim, "Real-time visionbased defect inspection for high-speed steel products,"
Optical Engineering, vol. 47, pp. 077204-077204-8, 2008.
7. A. Zographos, M. Robinson, J. Evans, and C. Smith,"Ballistics identification using line-scan imaging
techniques," in Security Technology, 1997. Proceedings.
The Institute of Electrical and Electronics Engineers 31st
Annual 1997 International Carnahan Conference on, 1997,
pp. 82-87.
8. D. Li, "Ballistics projectile image analysis for firearm
identification," IEEE Transactions on image processing,vol. 15, pp. 2857-2865, 2006.
9. G. Rosati, G. Boschetti, A. Biondi, and A. Rossi, "Real-time
defect detection on highly reflective curved surfaces,"
Optics and Lasers in Engineering, vol. 47, pp. 379-384,
2009.
10. P. Foglia, C. A. Prete, and M. Zanda, "An inspection systemfor pharmaceutical glass tubes," WSEAS Trans. Syst, 2015.
11. C. C. Lam and J. G. Kemper, "Ultrasonic detection of flaws
in tubular members," ed: Google Patents, 2003.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

455

Hsien-Huang Wu, Chang-Jhu He

12. C. C. Lam, J. G. Kemper, and A. Manocchio, "Flawdetection in tubular members," ed: Google Patents, 2004.
13. B. I. Cantor, J. H. Flora, and P. J. Latimer, "Apparatus andmethod of discriminating flaw depths in the inspection of
tubular products," ed: Google Patents, 1992.
14. J. S. Cruickshank, "Method and apparatus for the inspectionof tubular members," ed: Google Patents, 1989.
15. J. A. Beckstead and S. R. Nordhauser, "360 degree/forward
view integral imaging system," ed: Google Patents, 2000.
16. Y. Yagi and M. Yachida, "Omnidirectional visual sensor
having a plurality of mirrors with surfaces of revolution,"
ed: Google Patents, 2000.
17 E. Driscoll Jr, H. Morrow, A. J. Steinhauer, and W. C.
Lomax, "Panoramic camera," ed: Google Patents, 2002.
18. S. G. Wu and Y. S. Kuo, "360-degree panoramic cylindricalobject imaging mirror design method, mirror and imaging
device," ed: Google Patents, 2014 (in Chinese).
19. OPTO, "OPTO 360° Optical System."© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

456

Client Searching Privacy Protection in Encrypted Database
I-Hsien Liu

Department of Electrical Engineering, National Cheng Kung University, Tainan City, 70101, Taiwan
Chuan-Gang Liu

Department of Applied Informatics and Multimedia, Chia-Nan University of Pharmacy and Science, Tainan City, 71710,
TaiwanCheng-Jui Chang

Institute of Computer and Communication Engineering, National Cheng Kung University, Tainan City, 70101, Taiwan
Jung-Shian Li

Department of Electrical Engineering, National Cheng Kung University, Tainan City, 70101, Taiwan
E-mail: ihliu@cans.ee.ncku.edu.tw, chgliu@mail.cnu.edu.tw, cjchang@ cans.ee.ncku.edu.tw, jsli@mail.ncku.edu.tw
www.cans.ee.ncku.edu.tw

Abstract
In recent years, Cloud Storage Providers (CSP) works hard to successfully achieve cheap cloud storage. To prevent
losing important data in their own device, some of the people backup their data to the cloud. It becomes much more
important for securing the data stored in the cloud. SESD (Searchable Encryption Support Dynamic), provides multikeyword based searchable encryption method for the semi-trusted cloud simulations.
Keywords: Multi-keyword Search, Searchable Encryption, Privacy Preserving.

in their own way and share the secret key to the user who
is authorized by the owner.

1. Introduction
Cloud service provider (CSP) started providing cloudstorage to general user. CSP like Dropbox providing freestorage space for users. User get used to upload theirpersonal file to the cloud such as photos, videos, medical
records, business documents, and so on. But, thehijacking event on CSP never stops. Yahoo [1], one of
the leader on search engine, Verizon tried to lower thedeal with Yahoo after the huge amount client accountstolen.
From SC Media [2], hacker can monitor 58.8 million databy the unsecured database in the cloud. Furthermore,some of the open source database becomes their next
target. Data storing in the cloud can not only rely on the
CSP privacy policy, but also control the authorization of
sharing data by the data owner which is the risen of
Searchable Encryption. Data owner can encrypt the file

2. Background and Related work
2.1. Searchable Encryption
The evolution of Searchable Encryption could learn fromthe growth of search engine. The pioneer announces the
Searchable Encryption and provide single keyword
search [3] [4]. But user type in keyword may not be thatexactly, fuzzy keyword search helps the system to solve
this situation [5] [6].
2.2. Dummy Keyword
By generating multi-keyword index to provide much
more accuracy of search result and data presentation, N.
Cao [7] announce the first multi-keyword and ranked© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan457

I-Hsien Liu, Chuan-Gang Liu, Cheng-Jui Chang, Jung-Shian Li

search function. Definite dummy keyword to secure
searching process.
2.3. Synonym Keyword
Sometimes the user types in not the exactly keyword but
the synonym keyword, adding semantic databasebecomes another trend of Searchable Encryption. Z. Fu[8] adds a synonym list, mapping synonym keywords tothe correct keyword in the metadata. To improve theefficiency of searching, clustering the data by using kNearest Neighbor. But in text mining, synonym
keywords in a 100% mapping policy isn’t the realcorrelation of keyword relationship. Another scheme [9]sets a semantic database in a private cloud, user send the
keywords to the private cloud to transfer the exactly
keyword and create the search trapdoor. While protecting
the trapdoor, apply a special encrypt type named one-tomany order preserving encryption (OM-OPE), sameplain-text can be encrypted to m kinds of ciphertext. It’sa great secure method but the time of encryption isanother problem.

Fig. 1. System Architecture

3.1.1 Data Owner
After the data preprocessing mentioned before, Data
Owner builds a metadata to map the keywords in vector
type data. Then constructs a searchable tree index I fromthe data vectors. Encrypting the source data collection F
to encrypted collection C before upload to the cloudserver. Data Owner sends the metadata and the secret key
to the data user in a safe tunnel, the authorization process
and the encrypt type of data are not considered in our
research.
3.1.2 Data User
The authorized Data User types in the interesting
keywords than create the trapdoor TD according to themetadata Data Owner shared. Data User chooses theamount of data k, and send the trapdoor TD to the cloudwith the coefficient k.
3.1.3 Cloud Storage Server
Cloud Storage Server stores the encrypted data collection
C and the searchable tree index I. While receiving the
trapdoor TD from the user, Cloud Storage Server
executes the special search algorithm to search therelevance files than response the user top k ranked file.
Besides, in dynamic scheme, Cloud Storage Server
received the update package P from the owner, Cloud
Storage Server updates the encrypted data collection C
and index I according to the attribute of the package P.

2.4. Dynamic Update
While the data upload to the cloud server, data owner
may insert or delete data often. The dynamic scheme in
Searchable Encryption becomes a serious problem. S.
Kamara [10] announces a keyword-based Red Black tree
to support dynamic scheme. Define the amount of
keywords are fixed, but using symmetric type ofencryption couldn’t prevent some cloud threat model. Z.
Xia [11] constructs a binary search index, putting somefake nodes for dynamic insert in order to prevent reconstruct tree.
Binary tree is not able to afford the huge amount of data.
B tree indexing in hard disk storage releases storage
space and accelerates the speed of searching in the index,
G.Graefe [12] introduces some B tree dynamic scheme.
M. Ondreička [13] keeps the rules of B tree, putting thedata in the leaf node, comparing data at each level of thetree.

3.2. Data Pre-Processing

3. Searchable Encryption Support Dynamic Scheme

While extracting keywords from documents, finding the
keyword representing the data is the main goal in text
mining. Before calculating TF-IDF of the keywords,
striping punctuation from text file, removing the stop

3.1. System Architecture
In our research, we set the three different entities show in
Figure 1: Data Owner, Data User, Cloud Storage Server.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan458

Client Searching Privacy Protection

words, and stemming to the original file helps to countthe correct word existing term frequency.
3.3. Threat Model
The semi-trust cloud server we considered is named as
Honest and Curious server. We could comprehend themeaning from the words Honest and Curious. The server
will honestly execute the process we set, but the server isalso curious during the execution in order to find somespecific information.
3.3.1 Known Ciphertext Model
The cloud server only has the encrypted data collection
C and the searchable tree index I. The server tries todecrypt the encrypted data collection by the well-known
attack model Ciphertext-only attack(COA). [14]3.3.2 Known Background Model
The cloud server with more intelligence and curious
about the process the user search processes, monitoring
the relation between trapdoor and search result. Using theresult to find the rules of trapdoor generation.
Furthermore, the server may compare two similar
trapdoor to figure out the difference of search result
corresponding to the difference keyword. Than using therelationship to help decrypting the data collection C. [15]Fig. 2. Tree Searching Efficiency in difference search require k

and balanced). In the evaluation section, the dummy
keyword length we added is 10.
In the first step, we compare the performance of tree
construction. The disadvantage of B tree structure is themore using comparison operator in tree construction
which retard the performance. In tree search step, themost time we spend is the multiplier in finding the most
relevance path.
To compare the search performance between B tree andbinary tree, we restrict two different condition. The first
condition, we evaluate the keyword dimension 3500
index with different search request number k as the
Figure 6. Searching in B tree spend half time of binary
tree. In other condition, we restrict the search request
number k as 40, than evaluate the search process indifferent keyword dimension index. As the Figure 2, theperformance is great because we spend half time ofbinary tree.

3.4. Design goal
While supporting multi-keyword ranked search, adding
dynamic scheme is the trend of Searchable Encryption.
We construct the index by B tree to accelerate the searchefficiency in the padding search request step. The
disadvantage of B tree is that we construct the index isslower, but we have close efficiency in dynamic scheme.

5. Conclusion
In our research, we choose the B tree structure to enhance
the search process. By changing the structure, theefficiency of dynamic scheme is close to the binary tree.
Also, in dynamic scheme we add a new parameter to findout the real path of target file and update the treeaccording to the path stack. The extra cost we spend is
the more time in tree construction. But in cloudcomputing, the action of dynamic and searching process
are much more importance while the tree is large enough.
And in data privacy, to prevent the less keyword in queryis the trend of user behavior, we add more random feature
to flatten the trapdoor generation.
In future works, adding the search result verification in
the data user side is the trend. Because the trusted of the

4. Performance Evaluation
The source data we take are the 315 IEEE 2014
INFOCOM papers. By extracting the top relevancekeyword for each file, picking the top 3 to 10 TF×IDF
value as the keyword represent the file. In the metadata,
we padding 0 value in the data vector until the keyword
dimension reach the common multiple of 500.
Comparing our scheme SESD efficiency with the binary
tree structure EDMRS & BDMRS [18]. Than we set the
attribute of B tree as 4 which means the father nodecontain 4 child nodes. And the total leaf node are 1024,
this makes B tree and the binary tree in perfect(both full

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan459

I-Hsien Liu, Chuan-Gang Liu, Cheng-Jui Chang, Jung-Shian Li

12. G. Graefe and H. Kuno, Modern B-tree techniques, in Proc.
ICDE 27th, (Hannover, Germany, 2011), pp. 1370-1373.
13. M. Ondreička and J. Pokorný, Extending Fagin’s
Algorithm for More Users Based on Multidimensional BTree, in Proc. of East European Conference on Advancesin Databases and Information Systems ‘08, (Pori, Finland,2008), pp.199-214.
14. TutorialsPoint, Attacks On Cryptosystems, (TutorialsPoint,
2017)
15. W. Sun, B. Wang, N. Cao, M. Li, W. Lou, Y. T. Hou and
H. Li, Privacy-preserving multi-keyword text search in thecloud supporting similarity-based ranking, in Proc. 8th
ACM SIGSAC, (Hangzhou, China, 2013), pp. 71-82.

user to the search engine may cause the user search only
once for the file interested, and the less search result user
view. Trying to rearrange the search result in the data userside can reach the reliability from the user.
Acknowledgements
This work was supported by the MOST (Ministry of
Science and Technology) under contracts numbers
MOST 106-3114-E-006-003- and MOST MOST 1062221-E-006-033 -.
References
1. T. Spangler, Verizon Hacks $350 Million From Yahoo
Deal After Massive Data Breaches, (Variety, 2017).
2. B. Barth, Unsecured database lets hacker expose 58
million plus records from data management firm,(SCMedia, 2016).
3. D. X. Song, D. Wagner and A. Perrig, Practical
Techniques for Searches on Encrypted Data, in Proc.
2000 IEEE Symposium on Security and Privacy ,(Washington, DC, USA, 2000), pp.44-55.
4. R. Curtmola, J. Garay, S. Kamara and R. Ostrovsky,
Searchable symmetric encryption: improved definitions
and efficient constructions, in Proc. the 13th ACM
conference on Computer and Communications Security,(Alexandria, Virginia, USA, 2006), pp. 79-88.
5. C. Wang, K. Ren, S. Yu and K. M. R. Urs, Achieving
usable and privacy-assured similarity search overoutsourced cloud data, in Proc. IEEE INFOCOM 2012,(Orlando, FL, USA, 2012), pp. 451-459.
6. B. Wang, S. Yu, W. Lou and Y. T. Hou, "Privacypreserving multi-keyword fuzzy search over encrypted
data in the cloud, in Proc. IEEE INFOCOM 2014,(Toronto, Canada, 2014), pp. 2112-2120.
7. N. Cao, C. Wang, M. Li, K. Ren and W. Lou, Privacypreserving multi-keyword ranked search over encrypted
cloud data, IEEE Transactions on Parallel and Distributed
Systems, 25(1) (2014) 222-223.
8. Z. Fu, S. Xingming, N. Linge and L. Zhou, Achieving
effective cloud search services : multi-keyword ranked
search over encrypted cloud data supporting synonym
query, IEEE Transactions on Consumer Electronics, 60(1)(2014) 164-172.
9. Z. Xia, Y. Zhu, X. Sun and L. Chen, Secure semantic
expansion based search over encrypted cloud data
supporting similarity ranking, Journal of Cloud
Computing, 3(8) (2014) 1-11.
10. S. Kamara and C. Papamanthou, Parallel and Dynamic
Searchable Symmetric Encryption, in Proc. International
Conference on Financial Cryptography and Data Security
2013, (Okinawa, Japan, 2013), pp. 258-274.
11. Z. Xia, X. Wang, X. Sun and Q. Wang, A Secure andDynamic Multi-Keyword Ranked Search Scheme over
Encrypted Cloud Data, IEEE Transactions on Parallel andDistributed Systems, 27(2) (2016) 340-352.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

460

Study of real-time biomimetic CPG on FPGA: behavior and evolution
Timothée Levi*1, Yanchen Guo1, Kazuyuki Aihara1, Takashi Kohno1

IIS, The University of Tokyo, 4-6-1 Komaba, Meguro-ku
Tokyo, 153-8505, Japan1
E-mail: levi@sat.t.u-tokyo.ac.jp, guoyanchen@sat.t.u-tokyo.ac.jpaihara@sat.t.u-tokyo.ac.jp, kohno@sat.t.u-tokyo.ac.jp

Abstract
Locomotion is one of the most basic abilities in animals. Neurobiologists have established that locomotion resultsfrom the activity of half-center oscillators that provides alternation of bursts. Central Pattern Generators (CPGs) areneural networks capable of producing rhythmic patterned outputs without rhythmic sensory or central input. We
propose a network of several biomimetic CPGs using biomimetic neuron model and synaptic plasticity. This network
is implemented on a FPGA (Field Programmable Gate Array). The network implementation architecture operates ona single computation core and in real-time. The real-time implementation of this CPGs network is validated bycomparing it with biological data of leech heartbeat neural network. From these biomimetic CPGs, we use them for
robotic applications and also for biomedical research to restore lost synaptic connections.
Keywords: Biomimetic neural network, CPG, FPGA, Silicon neuron

1.

task (neural network for image processing, data mining,etc.). A biomimetic system mimics the nature forsimulating the living behavior and/or replaces it(rebuilding the living part). Biomimetic system canperform bio-hybrid experiments. While multi-legged
robots need CPG to move or coordinate their movements,
researchers implement an Amari-Hopfield CPG4 or basic
CPGs like Van der Pol5, modeled as non-linear
oscillators. Those models provide sinusoidal oscillations
that are not biomimetic.
To be closer to the nature, we developed biomimetic
CPGs. These CPGs will mimic the animal locomotion
like swimming in xenopus, salamander6, and lamprey7,
as well as leech heartbeat system8,9. This article isdecomposed into three parts: description and results ofthe biomimetic network of CPGs, applications on robotic
field and biomedical neuroscience research.

Introduction

Locomotion in robotic and neuroscience field is highly
important, and its complexity depends on the degree offitting the nature. The locomotion is based on the activityof half-center oscillators. The first half-center oscillator
was proposed by Brown1 in 1914. These oscillators
which mimic the rhythmic movements are programmed
by central pattern-generating (CPG) networks consistingof neural oscillators2. These CPGs provides alternating
burst without rhythmic sensory or central input3.
Usually, in robotic field, the CPGs are made using simple
neuron-models, and are not in biological time scale(accelerated time). These two specifications do not able
bio-hybrid experiments between biological cells andartificial cells. Therefore, we could say that these CPGs
are bio-inspired and not biomimetic. A bio-inspired
system is inspired by nature for conducting engineering© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan461

Timothée Levi, Yanchen Guo, Kazuyuki Aihara, Takashi Kohno

2.

requires a large number of parameters and the LIF model
is too simple for reproducing the neural network behavior
in Hill9.
The Izhikevich model (IZH)14 represents a good trade-off
as it is based on two equations and is capable ofreproducing many different families of neurons bychanging four parameters. Furthermore, this model isresource-frugal, a key advantage when the aim is todesign a large CPG network embedded in the same board.
As we chose the neuron model, we have to choose
carefully the synapse model. This model should bebiomimetic with few resources. In case of excitatory(inhibitory) synapse, a positive contribution (negative
contribution) is added to the postsynaptic neuron when
the presynaptic neuron spikes. Furthermore, we add theshort-term plasticity described by Izhikevich15 that isdifferent from Spike-Timing Dependent Plasticity(STDP) and that manages the depression or thefacilitation of the synaptic strength. This synaptic
plasticity is similar to the one described in Hill9.

Biomimetic CPGs

We propose a network of hundreds biomimetic CPGs
using biomimetic neuron, synapse and short-term
plasticity models. For instance the salamander
locomotion needs 40 CPGs6.
2.1. Model of the Leech Heartbeat system
Our CPG is based on the Leech heartbeat neural network
system9. The simplest biomimetic CPG is the one
responsible for the leech heartbeat (Figure 1). This CPG
consists of two neurons, connected to each other via
inhibitory synapses. As this is the most basic network, it
is ideal for achieving a better understanding of various
phenomena that control its activity. This pair of neurons,
also known as an elementary oscillator, is also thesmallest network capable of generating oscillations undernormal conditions. We add two excitatory neurons
connected with excitatory synapses for detecting burst
activity of the elementary oscillator.

2.3. Design of real-time system
This network is implemented in a digital platform: Xilinx
Kintex-7 FPGA (Field Programmable Gate Array). The
network implementation architecture operates on a single
computation core. This digital system works in real-time,requires few resources and is low power consumption. It
is essential in robotic applications. The implementationof one CPG is validated by comparing it with biological
data of leech heartbeat neural network10. These CPGs
could reproduce biological activity of CPG neuralnetworks not only in terms of period but also in terms ofvariation of duty cycle and bursting period.
This implementation is low resources. Table 1 presents
the used resources for 128 CPGs in Xilinx Kintex-7
FPGA family. Using full resources, we can implement
984 CPGs. For instance, salamander locomotion needs
40 CPGs.

Fig. 1. A diagram of the CPG in the leech heartbeat system,composed of one elemental oscillator (N0 and N1).
Synapses between N0-N1 couple are inhibitory. N2 andN3 are burst-detector neurons. They are spiking just one
time when a burst occurs in their connected neuron. N2N3 are respectively connected to N0-N1 by excitatory
synapses.

2.2. Neuron, synapse and short-term plasticity
model
The biomimetic CPG that we designed is presented indetails in10,11. No CPGs network and no real-time period
modifications were possible in10.
The neuron model choice is very important, because wehave to choose between the biological plausibility and thecost computation. The choice of model was based on two
criteria: the family of neurons able to be reproduced andthe number of equations. These criteria were used tocompare several models, from the simple one: the Leaky
Integrate and Fire model (LIF)12 to the most complex
one: the Hodgkin-Huxley model (HH)13.
In the article by Hill9, HH model is used to reproduce theleech heartbeat system. Nevertheless, the HH model

Table 1. Utilization of the resources for the
implementation of 128 CPGs

Kintex-7
LUT
FF
BRAM
DSP

Utilization
26795
9479
43
40

Available
203800407600445
840

Percentage
13,1
2,3
9,6
4,7

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan462

Study of real-time biomimeticof the master CPG and value of the excitatory synapse
weight).

Fig. 2. Two real-time CPGs (N0-N1 and N4-N5) with theirburst-detection neurons (N2-N3 and N6-N7). Time axis is
one second by division.
Fig. 3. Forward locomotion with two examples of speed
using a chain of 7 CPGs. Time axis is one second by
division. Signal are the left neuron (N0, N2, N4, N6, N8,
N10 and N12) of each CPG.

Figure 2 describes the FPGA implementation results for
two CPGs (N0-N1 and N4-N5) with their two burstdetector neurons (N2-N3 and N6-N7).
The real-time behavior of CPG system is mandatory as
we can use for robotic applications (section 3), biohybrid experiments but also to simulate different
topologies and lost connections as it can exist in thenature (section 4). Using software, simulation time is toolong as neuron model and synaptic plasticity use complex
non-linear differential equations. Thanks to this real-timesystem, we can simulate several topologies of CPG
network and then find some treatment protocol whenneuronal connections are lost.

4. Biomedical applications
How to restore locomotion activity when someconnections are lost? We simulated several cases. Here,
we present one study where one CPG does not work
anymore as the inhibition connection between N6 and N7
is lost. In that case, N7 is always firing and N6 is always
inhibited as Figure 4 (top) describes it.

3. Robotic applications
Usually simple CPGs are often used for robotic
application but biomimetic CPGs are rarely used.
Nevertheless some few works6,16-20 developed
biomimetic motion for robotics. To demonstrate thefeasibility of using our biomimetic CPGs. We create
forward locomotion for artificial salamander withdifferent periods and different phases between CPGs. We
reproduce the forward locomotion explained in16 using aloop of seven CPGs (from CPG0 to CPG6) with crossed
inhibition connections between them. Figure 3 describes
two different forward locomotions with different periods.
We can modify the period in real-time by tuning theweight of the crossed synaptic connections between the
CPGs. We also investigated a master CPG which controls
the activity of seven CPGs with excitatory connections
between the master CPG and CPG0. Thanks to thismaster CPG, we can directly choose the frequency andphases between CPGs using just two parameters (period

Fig. 4. Reconstruction of a lost synaptic connection. The
couple N6-N7 recovers its CPG activity. Time axis is one
second by division.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan463

Timothée Levi, Yanchen Guo, Kazuyuki Aihara, Takashi Kohno

This result shows that thanks to new synaptic connections
from other CPG, we can restore a lost activity (figure 4
bottom). As our biomimetic CPGs can be connected toliving neurons, this research can be extent to bio-hybrid
experiments and uses these biomimetic CPGs asneuroprostheses during spinal cord lesion.

9.
10.

5. Conclusion

11.

This article demonstrates a real-time network ofbiomimetic CPGs. This system can be used in robotic
applications for smoother and closer behavior tobiological animals. It can also be used for bio-hybrid
robotic (living part and artificial part) as it wasdemonstrated that this system can be connected tobiological neurons21,22. Another advantage is that realtime systems can simulate several topology of CPGs andsimulate different hypothesis and protocol treatment forbiomedical applications like spinal cord injury23.

12.

13.

14.
15.

6. Acknowledgements
Authors thank Social Cooperation Program for BrainMorphic AI to Resolve Social Issues, IIS, UTokyo.

16.

7. References
1. T. Brown, On the nature of the fundamental activity of thenervous centres; together with an analysis of theconditioning of rhythmic activity in progression and atheory of the evolution of function in the nervous system.
Journal of Physiology, 48, 18-46, 1914
2. E. Marder and D. Bucher, Central pattern generators andthe control of rhythmic movements. Current Biology 11,
986-996, 2001
3. S. Hooper, Central Pattern Generators, Current Biology,
10, 176–177, 2000
4. S. Amari, Characteristic of the random nets of analog
neuron-like elements, IEEE Trans. on System, Man andCybernetics, 2, 643-657, 1972
5. B. Van Der Pol, The heartbeat considered as a relaxation
oscillation, and an electrical model of the heart.
Philosophical Magazine, 6, 763–775, 1928
6. A. Ijspeert, A. Crespi, D. Ryczko, J. Cabelguen, Fromswimming to walking with a salamander robot driven by aspinal cord model. Science, 315, 1416-1420, 2007
7. AH. Cohen, GB. Ermentrout, T. Kiemel, N. Kopel, KA.
Sigvardt, TL. Williams, Modelling of intersegmental
coordination in the lamprey central pattern generator for
locomotion. Trends in Neuroscience. 15, 434–438, 1992
8. GS. Cymbalyuk, Q. Gaudry, MA. Masino, R.L. Calabrese,
Bursting in Leech Heart Interneurons: Cell-Autonomous

17.
18.
19.
20.

21.

22.
23.

and Network-Based Mechanisms. J. Neuroscience, 22,10580-10592, 2002
AA. Hill, J. Lu, MA. Masino, OH Olsen, RL Calabrese, A
model of a segmental oscillator in the leech heartbeat
neuronal network, J Comput Neurosci., 10, 281-302, 2001
M. Ambroise, T. Levi, S. Joucla, B. Yvert, S. Saïghi, Realtime biomimetic Central Pattern Generators in an FPGA
for hybrid experiments. Frontiers in Neuroscience, 7, 215,
2013
M. Ambroise, T. Levi, Y. Bornat, S. Saïghi, Biorealistic
spiking neural network on FPGA, 47th Annual Conferenceon Information Sciences and Systems CISS, 1-6, 2013
G. Indiveri, Synaptic plasticity and spike-based
computation in VLSI networks of integrate-and-fireneurons. Neural Information Processing, Letters andReviews, 11, 135-146, 2007
AL. Hodgkin, AF. Huxley, A quantitative description of
membrane current and its applications to conduction andexcitation in nerve, The Journal of Physiology, 117, 500544, 1952
E.M. Izhikevich, Simple model of spiking neurons. IEEE
Transactions on Neural Networks, 14, 1569- 1572, 2003
E.M. Izhikevich and G. Edelman, Large-scale model ofmammalian thalamocortical systems, Proceedings of the
National Academy of Sciences of the United States ofAmerica, 105, 3593-3598, 2008
T. Matsuo, T. Yokoyama, D. Ueno, K. Ishii, Biomimetic
Motion Control System Based on a CPG for an
Amphibious Multi-Link Mobile Robot, Journal of BionicEngineering, 91-97, 2008
J.H. Barron-Zambrano and C. Torres-Huitzil, FPGA
implementation of a configurable neuromorphic CPGbased locomotion controller, Neural Netw.45, 50–61, 2013.
C. Li, R. Lowe, T. Ziemke, Humanoids Learning to Walk :a Natural CPG-Actor-Critic Architecture, Frontiers in
Neurorobotics, 7:5, 2013
E. Donati, F. Corradi, C. Stefanini, G. Indiveri, A spiking
implementation of the lamprey's Central Pattern Generator
in neuromorphic VLSI, IEEE BioCAS 2014,2014
C. Qi, W.Jiang, S. Yang, Y. Qin, B. Deng, W. Xile, A realtime FPGA implementation of a biologically inspired
central pattern generator network, Neurocomputing, 244,
63-80, 2017
S. Joucla, M. Ambroise, T. Levi, T. Lafon, P. Chauvet, S.
Saighi, Y. Bornat, N. Lewis, S. Renaud, B. Yvert,,
Generation of locomotor-like activity in the isolated rat
spinal cord using intraspinal electrical microstimulation
driven by a digital neuromorphic CPG, Frontiers inneuroscience, 10, 2016
T. Levi et al., Biomimetic neural network for modifying
biological dynamics during hybrid experiments, Journal
on Artificial Life and Robotics, May 2017
M. Capogrosso et al., A brain–spine interface alleviating
gait deficits after spinal cord injury in primates, Nature,539, 284-288, November 2016

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan464

A Metaheuristic Approach for Parameter Fitting in Digital Spiking Silicon Neuron Model
Takuya Nanami

The University of Tokyo, Institute of industrial Science
Tokyo, Japan
Filippo Grassia

LTI Lab., University of Picardie Jules VerneSaint-Quentin, France
Takashi Kohno

The University of Tokyo, Institute of industrial Science
Tokyo, Japannanami@sat.t.u-tokyo.ac.jp, filippo.grassia@u-picardie.fr, kohno@sat.t.u-tokyo.ac.jp

Abstract
DSSN model is a qualitative neuronal model designed for efficient implementation in digital arithmetic circuit. In
our previous studies, we developed automatic parameter fitting method using the differential evolution algorithm
for regular and fast spiking neuron classes. In this work, we extended the method to cover low-threshold spiking
and intrinsically bursting. We optimized parameters of the DSSN model in order to reproduce the reference ionicconductance model.
Keywords: Spiking neuron model, Low-threshold spiking, Intrinsically bursting, Differential evolution, FPGA

which approximates a spiking process by the resetting of
the state variables. The I&F-based models can be
implemented resource-efficiently, but are capable ofreduced reproducibility of complex neuronal activities.
For example, these models assume that the amplitude of
spikes is fixed, whereas the spike intensity is nonuniform
in the nervous system. 4
We have been studying qualitative neuronal models
for digital as well as analog circuit implementation thatsatisfy both the reproducibility of neuronal activities andlow computational cost. 5, 6, 7, 8 The core idea of ourqualitative-modeling-based approach is to reproduce thecore mathematical structures that a wide variety ofneuronal activities. In our previous studies, 9, 10 weextended the DSSN models7 to support various neuronalclasses; regular spiking (RS), fast spiking (FS),intrinsically bursting (IB), low-threshold spiking (LTS),

1. Introduction
The silicon neuronal network is composed of analog and(or) digital circuits that solve the differential equations ofneuronal and synaptic models. Different kinds ofneuronal models have been used in silicon neuronal
networks because of the trade-off between
reproducibility of neuronal activities and computational
efficiency. For example, Merolla et al.1 constructed adigital silicon neuronal network comprising 1 million
spiking neurons and 256 million synapses. They adopted
Leaky integrate-and-fire (LIF) model that is one of themost commonly used and simple neuron models. Thomas
et al.2 implemented 1024-spiking neurons of Izhikevich(IZH) model3 on a field-programmable gate array(FPGA). Those LIF, expanded LIF, and IZH models are
categorized as integrate-and-fire (I&F) based model© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan465

Takuya Nanami, Filippo Grassia, Takashi Kohno

elliptic bursting (EB), and parabolic bursting (PB). We
also studied an automatic parameter fitting method using
the differential evolution (DE) algorithm11 for the RS andFS classes. 12
In this work, we developed the method for the lowthreshold spiking and intrinsically bursting classes.
Parameters were optimized to reproduce waveforms of
the ionic-conductance models in Ref. 13 and reduce thecircuit resource requirements for implementation.
The remainder of this paper is organized as follows.
Section 2 introduces our neuron model, the DE algorithm,
and details of our parameter optimization procedure. The
result is shown in Section 3. Section 4 summarizes thiswork and suggests ideas for the future.

represents the input stimulus.
constant andParameters ∅ , ∅ , ∅ , ε , , and τ control the time,
constants of the variables. Parameters, , , ,, and , where is, ,,, ℎ , or ℎ , areconstants that adjust the shape of the -, -, - and andare used to select ∅nullclines. Parameters
depending on . All of the variables and constants in thisqualitative model are purely with no physical unit. Cubic
shaped -nullcline is a key to replicate the spiking
dynamics. Because multipliers are resource-consuming
in a digital arithmetic circuit, the DSSN model adopts apiecewise quadratic function for the -nullcline so thatits numerical integration step requires only one
multiplication between variables.

2. Method

2.2. Differential evolution algorithm(2)

The DE algorithm11 is a popular heuristic method to solve
optimization problems using a real number function. It is
one of the population-based optimization algorithms inwhich a number of parameter sets are converged to quasioptimal solutions with which the DSSN model
reproduces the target waveform. The initial randomparameter sets have to be selected carefully so that the
behavior of the DSSN model does not diverge. Detailed
explanation is written in the previous study. 12(3)

2.3. Parameter optimization procedure(4)

The LTS class neurons are found in the thalamus. It has
the spike-frequency adaptation as with the RS class, andhas the rebound bursting; neurons in the LTS classgenerate a burst firing just after the termination of a
sufficiently strong and long hyperpolarizing stimulus.
The IB class is a well-known bursting neuron class in thecortex. Neurons in the IB class generate a burst firing
immediately at the onset of a sufficiently strong stimulus,
then continue spiking until its termination.
The DSSN model has 28 parameters, and exploring
their spanning space is not realistic for the computerswith average power. To avoid this problem, we divided
the state variables of the DSSN model into two groups
based on their time scale, and tuned each group in
sequence. In our previous study, 12 the RS and FS classes
were characterized as the degree of spike-frequency
adaptation, and we just optimized parameters of fast statevariables in order to reproduce the saddle-

2.1. Digital Spiking Silicon Neuron model
The 4-variable DSSN model is a qualitative neuronmodel that can simulate various neuronal classes (RS, FS,
IB, LTS, EB, and PB) by Euler's method with fixed point
operation. Its equations are given by∅( )(1)( ( )− − + +
),
=
=

=

( )≡

( ( ) − ),= (ℎ( ) − ),∅∅(u) = ∅
∅

( )≡

1

( −

−

−

−

−
( −

−

),

+

( < 0)( ≥ 0),

+
) +c

≥

(

( <
)
< <
)
(
< ),

+

+c

<

(5)
(6)

,

(7)

( < )
(8)
( ≥ ),−+
where
corresponds to the membrane potential.
Variables and are the fast and slow variables which
abstractly describe the activity of the fast and slow ionchannels, respectively. Variable is the slowest variable
that modifies the structure of the fast subsystemcomprising v and n, and used in only the IB mode. In theRS, FS, and LTS modes, is fixed. Parameter is aℎ( ) ≡© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan466

A Metaheuristic Approach forcoefficient increases. Parameters were optimized for not
only reproducing waveforms of the ionic-conductancemodel but also reducing the circuit size. We defined thecost function as follows,
(9)
( ),
( )=
( )+
where
( ) is the function that calculates cost from the
waveforms of the ionic conductance model and the
DSSN model with parameter vector , based on the
difference of spike timing as well as the mean squareerror of the state variable corresponding the membranepotential. In this function, minimum and maximum
values of waveforms are normalized to 0 and 1 foris a number of digits withcomparison. Function
value 1 in the binary fixed-point expression of thecoefficients in . Constantis to balance these twocost functions. Parameters responsible for the dynamics
of the fast variables were determined to minimize the costfunction using the DE algorithm.

Fig. 1. Waveforms of the ionic-conductance model (red) andthe DSSN model (blue) of the LTS class in response to an
inhibitory (top) and weak excitatory (middle) and strong
excitatory (bottom) step stimulus input.

3. ResultFig. 2. Waveforms of the ionic-conductance model (red) andthe DSSN model (blue) of the IB class in response to a weak(top) and middle (middle) and strong (bottom) excitatory step
stimulus input.

We found two parameter sets with which the DSSN
model reproduces the activities of the ionic-conductancemodels for LTS and IB classes (Fig. 1). We evaluated thesimilarity of the spiking patterns between the ionicconductance model and the DSSN model by calculatingsome statistics while changing the stimulus intensity (Fig.
3). For the LTS class, we usedand 14 and these are
explained in our previous study. 12 Our fitted model’s− characteristics are not completely the same asthat of the reference model, but share the trend thatanddecreases as the stimulus intensity increases. For
the IB class, we measured the number of spikes in the
bursting duration and average firing rate after the
bursting. With both models, the number of spikes in the
bursting duration increases linearly as the average firing
rate increases, although the exact value does not coincidecompletely. The similarity of these characteristics will befurther estimated based on the requirements specified by
the circuits' applications.
The results in Figs. 1 and 2 were obtained bynumerical integration with fixed-point operation and theEuler's method ( dt = 2s). The same results are
expected to be observed in the following FPGA
implementation, because this condition is the same as our
VHDL codes. We compiled the DSSN models for Xilinx
XC7VX690T using Xilinx Vivado Design Suite.

node on invariant circle bifurcation. However, LTS andIB classes have more complex structure in their fasts
subsystem. For the LTS class, we firstly explored theparameter group responsible for the dynamics of the faststate variables and so that the fast subsystem of ourmodel is consistent with that of the ionic conductancemodel of the LTS class. Then, the parameters related tothe slow state variable were determined so that the
dynamical behavior of the whole ionic conductancemodel of the LTS class is reproduced. For the IB class,we firstly fitted the parameters responsible for the
dynamics of the state variables , , and so that the
dynamical behavior of the corresponding subsystem in
the ionic conductance model of the IB class isreporoduced. Then, the parameters related to thewere determined so that theremaining variable
dynamical behavior of the whole ionic conductancemodel of the IB class is reproduced.
In the DSSN model’s circuit implementation,
multiplication between a coefficient and a state variable
was realized by shifters and adders, and we needed alarger number of adders and shifters as the number ofdigits with value 1 in the fixed point representation of the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan467

Takuya Nanami, Filippo Grassia, Takashi Kohno

2.

3.

Fig. 3. Statistical properties of the ionic-conductance model(red) and the DSSN model (blue). In the LTS mode (left), thex−axis and y-axis are Cv and Lv. In the IB mode (right), x-axis
corresponds to the number of spikes in the bursting and y-axis
corresponds firing rate after the bursting.

4.
5.

Table. 1 : Device Utilization.

6.

Device utilization is listed in Table 1. In the column
labeled LTS (hand-fitted) and IB (hand-fitted), theresource usage for LTS and IB settings in Ref. 9 is shown.
The requirement for LUTs was reduced without any lossof the reproducibility of the neuronal activities.

7.

8.

4. Conclusion
In this work, we extended the parameter optimization
method for the DSSN model in the LTS and IB classmodes. These classes have complex mathematicalstructures of the fast subsystem, thus we carefully
designed our fitting procedure in order to reproduce
important structures in these classes. We could findparameter sets with which the DSSN model reproduces
the characteristics activities of LTS and IB classes using
average-power PC ( a Xeon E5-2667 CPU with a
NVIDIA P100 GPU ). Cost functions
were introduced
to reduce the necessary circuit size. In our future work,
we will apply our silicon neuronal networks to the spikepattern recognition. 15

9.

10.

11.
12.

13.

Acknowledgement
This work was partially supported by JSPS SAKURA Programand JST PRESTO and CREST

14.

References
1. P. A. Merolla, J. V Arthur, R. Alvarez-Icaza, A. S. Cassidy,
J. Sawada, F. Akopyan, B. L. Jackson, N. Imam, C. Guo,
Y. Nakamura, B. Brezzo, I. Vo, S. K. Esser, R.
Appuswamy, B. Taba, A. Amir, M. D. Flickner, W. P.
Risk, R. Manohar, and D. S. Modha, “A million spiking-15.

neuron integrated circuit with a scalable communication
network and interface”, Science, Vol. 345, pp. 668–673,
2014
D. B. Thomas and W. Luk, “FPGA accelerated simulationof biologically plausible spiking neural networks,” Proc. IEEE Symp. F. Program. Cust. Comput. Mach. FCCM
2009, pp. 45–52, 2009.
E. M. Izhikevich, “Which model to use for cortical spiking
neurons ?”, IEEE Trans. Neural Netw. vol. 15, pp. 1063–
1070, 2004
H. Alle and J. R. P. Geiger, “Combined analog and action
potential coding in hippocampal mossy fibers”, Science,
vol. 311, pp. 1290–1293, 2006
T. Kohno and K. Aihara “A three-variable ultralow-power
analog silicon neuron circuit” 2016 International
Symposium on Nonlinear Theory and Its Applications,Japan, November 27th–30th, pp. 190–193, 2016
T. Kohno, M. Sekikawa, J. Li, T. Nanami, and K. Aihara,"Qualitative-Modeling-Based Silicon Neurons and Their
Networks," Frontiers in Neuroscience, Vol. 10, No.273,
pp. 1–16, Jun., 2016.
T. Kohno and K. Aihara. Digital spiking silicon neuron:
Concept and behaviors in gj-coupled network.
Proceedings of International Symposium on Artificial Lifeand Robotics 2007, 2007.
J. Li, Y. Katori, and T. Kohno, "An FPGA-based siliconneuronal network with selectable excitability siliconneurons," Frontiers in Neuroscience, Vol. 6, pp. 1–13,
Dec., 2012.
T. Nanami and T. Kohno. “Simple cortical and thalamic
neuron models for digital arithmetic circuit
implementation”, Frontiers in Neuroscience, Vol. 10,
No.181, pp. 1–12, 2016
T. Nanami, K. Aihara, and T. Kohno, “Elliptic andparabolic bursting in a digital silicon neuron model”, 2016
International Symposium on Nonlinear Theory and Its
Applications, Japan, November 27th–30th, 2016
R. Storn, K. Price, “A Simple and Efficient Heuristic for
global Optimization over Continuous Spaces”, Journal of
Global Optimization, Vol. 11, pp. 341–359, 1997.
T. Nanami, F. Grassia, and T. Kohno, “A parameter
optimization method for Digital Spiking Silicon Neuronmodel”, Journal of Robotics, Networking and Artificial
Life, Vol. 4, pp. 97–101, 2017.
M. Pospischil, M. Toledo-Rodriguez, C. Monier, Z.
Piwkowska, T. Bal, Y. Fregnac, H. Markram, and A.
Destexhe. “Minimal Hodgkin-Huxley type models fordifferent classes of cortical and thalamic neurons”.
Biological Cybernetics, Vol. 99, pp. 427–441, 2008.
S. Shinomoto, K. Shima, and J. Tanji, “Differences inspiking patterns among cortical neurons.,” NeuralComput., vol. 15, no. 12, pp. 2823–2842, 2003.
T. Masquelier, R. Guyonneau, and S. J. Thorpe,“Competitive STDP-based spike pattern learning.,”
Neural Comput., vol. 21, pp. 1259–1276, 2009.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

468

Real-time Digital Implementation of HH neural network on FPGA:
cortical neuron simulation

Farad Khoyratee1, Sylvain Saïghi1, Timothee Levi1,2
1

IMS Lab, University of Bordeaux, France
2
IIS, The University of Tokyo, Japan

farad.khoyratee@u-bordeaux.fr, sylvain.saighi@u-bordeaux.fr, levi@sat.t.u-tokyo.ac.jp

Abstract
Research on neurological disorder led to an alternative treatment using biomimetic and real-time hardware asneuroprosthesis. These devices must follow several requirements such as the connection between cells andmachine or the neurons activities, the bio-hybrids experiments. Furthermore, timing and shape of the action
potential (AP) must reproduce the same dynamics of a real nerve impulse. The system must be real time,
biomimetic and tunable. Thus, a Field Programmable Gate Array (FPGA) has been chosen to carry the
implementation of cortical neurons. The Hodgkin-Huxley (HH) model is the most plausible and realistic one. A
model including Fast Spiking (FS), Regular Spiking (RS) and Low-Threshold Spiking (LTS) neurons has been
implemented in a Field Programmable Gate Array (FPGA) as a neural network. Some digital methods were used
like the CORDIC algorithm or Euler method to solve exponential and differential equations. Here a pipelined
implementation of a cortical neural network which contains 150 neurons is presented. This should allow futurestudies of neurological effect on cells and neurons replacement.
Keywords: Silicon neuron, Hodgkin-Huxley, FPGA, neurological diseases, CORDIC
1. Introduction

to the recorded electrical activity in order toreproduces the same dynamics and functions as thebrain.
Neurological illness affects cognitive and motor
capabilities aiming the Central Nerve System (CNS)
which contains more complex cells like corticalneurons. Several cells exist and have different firing
behavior. Fast Spiking (FS), Regular Spiking (RS),Intrinsically Bursting (IB) and Low Threshold
Spiking (LTS) are studied in this paper. RS, FS, LTS
and IB are the main cortical neurons15.
Implementations of a biodetailed and biorealistic
model into an embedded system have been a topic ofresearch for years. To fulfil these conditions, thebursting dynamic of neurons and the shape of the

The brain’s complexity raises different purposes
going from image recognition to the study of neurons
dynamics using mathematical model andcomputational tools. Also, the augmentation ofneurological diseases affecting people around theworld brings scientists to think about different
treatments using technology. Consequently, researchon neuroprosthesis has been increased and should be
used as a treatment or as a device allowing studying
and characterizing cells dynamics. Such devices
require studying the way in which cells and machineinteract, and, the intrinsic spontaneous activity ofneural network. Neural model should be also adapted© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

469

Farad Khoyratee, Sylvain Saïghi, Timothee Levi

Where gi is the maximum conductance; m and h
are gating variables for activation and inactivation.
Ei is the ion-specific reversal potential and p and q
are integers.
According to Hodgkin and Huxley13, the valuesof the activation and deactivation parameters are
updated by the equation (3).

action potential (AP) must be taken into account.
Different models have been compared2 such as the
Leaky integrate and fire3, Izhikevich3 and HodgkinHuxley (HH) model. It has been shown that the HH
model, especially the multicompartmental one4, isthe most realistic and the best choice.
Designs of biomimetic analog chips for robotics
or biomedical applications allow multiple
advantages like low consumption, a fast process andthe use of small devices5-6. However, thesetechnologies are difficult to create, to tune and are
expensive. Whereas, the increasing numbers ofdigital platforms like Field Programmable Gate
Array (FPGA) show that they are powerful, fast,
tunable and cheaper. Some implementations ofneural network have been done in a FPGA7-8 but few
used the HH model9-10 and among these last none ofthem aimed bio-hybrid experiments11-12.
In this paper we present the implementation of aneural network of cortical neurons using HodgkinHuxley equation in a FPGA dedicated for theneurological diseases study.

=

( )=

( )=

.ℎ .(

(1)

−

)

( )

( )

(5)

There are requirements concerning the electricalsystem for bio-hybrid platforms and neurologicaldiseases study. The device must be real-time, tunable,
light in resources and containing a large number ofneurons. The FPGA (Field Programmable Gate
Array) is the best component including flexibility,
speed and stability.
FPGA platform with a Kintex-7 FPGA
“xc7k325tffg900” family has been used. A Digital to
Analog Converter (DAC) is used to output thesignals. The DAC has a 12bits precision and a 2.5V
reference voltage. The frequency of the system is
200MHz.
Several methods have been used to store values
and perform complex operations such as exponential,differential equation resolution and multiplication.
All operations are fully pipelined. Fixed point
numbers of 39 bits have been used with 20bits forthe decimal part and 18bits for the integer part. The
last bit is for the sign. The choice of the length has
been set for an optimal concession between precisionand material cost. Xilinx 7 series of FPGA include“Digital Signal Process” (DSP) used to do
multiplications which have been chosen for the
implementation. One of the most used operators inthe HH equations is the exponential. The
implemented method is the Coordinate Rotation

Where Vm is the membrane potential, Cm is themembrane capacitance; Ii is the different ioniccurrents and Istim is the stimulation current. The
current is representing by the following equation:

.

( )

(4)

( )

2.2 Implementation

Fig. 1. Equivalent Electrical circuit of the membrane
according Hodgkin-Huxley=( )Where α and β are exponential-based equations.
The chosen equations11 include five ionic currents
which are sodium, potassium, leakage, slow
potassium (for spike-frequency adaptation) andcalcium.

2.1 Equations
The first plausible equation which reproduces theshape of the Action Potential (AP) was proposed by
Hodgkin-Huxley13. In order to reproduce the shapeand the bursting dynamics of a cortical neuron model
has been used14. The equations are able to simulate
Fast Spiking (FS), Regular Spiking (RS),Intrinsically Bursting (IB) and Low Threshold
Spiking (LTS) neurons. The dynamics bursting ofthese cells have been described and localized15.

−∑

(3)

( )

Where x ∈ {m, n, h, p, q, r, s} and τx(V) andx(V) are obtained by (4) and (5).

2. Materials and methods

=

( )

(2)

470

Real-time Digital Implementation of

Data was captured from an oscilloscope andimported on Matlab for the comparison (fig. 3).
Dynamics, amplitude and frequency are similar
between software simulations and the FPGA’s
output for all cortical neurons. 150 neurons arecomputed in real-time. All type of cortical cells canbe selected. Ressources taken by the FPGA is shownon (Table 1.). Use of Look Up Table (LUT) has been
increased from17. It can be explained by theutilization of the CORDIC algorithm which reduces
the use of DSP but increases the use of memory. Also,
more current, gating variables, parameters andfeatures have been added. Furthermore, the pipelinearchitecture and the different parallel computations
need shift registers for synchronization.
Simulations were done using NEURONS
software then implementing on Matlab. The
implemented model was compared with biological
data.

Digital Computer (CORDIC) algorithm1615 for his
low material cost, speed and precision. It allows thecomputation of sinh and cosh functions andindirectly the exponential by the addition of the two
hyperbolic functions. The Euler method is used forthe resolution of the differential equation (6) as it isa good accuracy and low cost of implementation.
( + )= ( )+= + . ′. ′( )(6a)(6b)

Where the chosen value of the time step dt is
10µs. The time step determines the precision of theresult. The smaller the value, the more accurate is theresult. However, it implies a limited number ofneuron computations between the moment t and t+dt.
Changing dt apply a difference of amplitude andfrequency on the spike trains. Then, 10µs is the bestcompromise for accuracy and calculation time.
The implementation structure (fig. 2) has
different parts. The HH module computing all
equations11 is piloted by a controller unit whichcalculates the time step, the period to enable reading
and writing into the Random Access Memory(RAM) and the neuron index. This module also
pilots the Neuron Type unit, allowing to choose theparameters according to the neuron types (FS, RS,
IB or LTS). The RAM stores the values of themembrane potential and the gating variables.
Previous Values of these are used to perform theEuler method.

A)

20mV
20 ms

B)

Vmi-1
xi-1
HH
Parameters

H-H

Vmi
xi

RAM

Nneur
Controller

RS
20mV
30 ms

C)
Neuron
Type
Type

FS

LTS
20mV
200 msEnable Reading
or writing
Fig. 3. Comparison of signals from matlab (bluelines) and from DAC Output (red dotted lines); A)
FS Neurons with a stimulation current of 0.5 nA; B)
RS neuron with a stimulation current of 0.75 nA; C)
LTS Neuron with a Stimulation current of -0.36nA;

Fig. 2. Implementation structure of the neuralnetwork; Vmi-1 and xi-1 respectively represents theprevious values of the membrane voltage and thegating variables; Nneuron is the index of the neuron
and HHparameters contain the values ofconductance, neuron size, reversal potential, andmembrane capacitance for FS, RS, IB and LTS
neurons.

3 Results

Table 1. Device resources for the kintex 7 FPGA
representing the number of Look Up Table (LUT), Flip
Flop (FF) and DSP used.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

471

Farad Khoyratee, Sylvain Saïghi, Timothee Levi
Used

Utilization (%)

LUT

16900382.93

LUTRAM

3141

4.91

FF

65059

15.96

DSP

332

39.52

9.

10.

4 Conclusions
11.

Cortical Neurons have been implemented using
Hodgkin-Huxley model in FPGA. The output isperformed in real-time and can be used for biohybrid experiments. The implementation was
validated by a comparison with software simulations
and FPGA outputs. More than 150 neurons can be
implemented including synapses. In the future,
networks
ofthe
Hodgkin-Huxley
multicompartmental model will be implemented.
The objective of this work can be split up in two
goals. First, the study of neurological diseases can bedone using this platform. The bio-hybrid
experiments for communication between machinesand cells are a step towards the implementation of a“diseases neurons” and the observation of theireffect on a biological culture. Second, the long term
objectives could be summarize by the developmentof neuro-prostheses which could replace affected
neurons or restore the lost communication between
different neural networks18.

12.

13.

14.

15.
16.
17.

References
1. Brette R. (2015), What is the Most Realistic Singlecompartment Model of Spike Initiation?, PLOS
Computational Biology
2. Indiveri G (2007), Synaptic plasticity and spikebased computation in VLSI networks of integrateand-fire neurons. Neural Information Processing,Letters and Reviews 11:135-146
3. Izhikevich, E.M. (2003), Simple model of spiking
neurons, IEEE Transactions on Neural Networks,
vol.14, no.6, 1569-1572
4. Davison A, Feng J, Brown D (2000), A reduced
compartmental model of the mitral cell for use in
network models of the olfactory bulb, Brain Research
Bulletin, 51:5, 393-399
5. Levi T, Lewis N, Tomas J, Saighi S, Renaud S,
Bornat Y, Alvado L (2008),
Neuromimetic
Integrated Circuits, VLSI Circuits for Biomedical
Applications, Iniewski Editor
6. Indiveri G. et al. (2001), Neuromorphic siliconneuron circuits, Frontiers in Neuroscience, 5:73
7. Ambroise M, Levi T, Joucla S, Yvert B, Saighi S.
(2013), Real-time biomimetic Central Pattern
Generators into FPGA for hybrid experiments,
Frontiers in Neuroscience, 7: 215
8. Grassia F., Levi T., Kohno T., Saighi S. (2014),
Silicon neuron: digital hardware implementation of

18.

472

the quartic model, Journal of Artificial Life andRobotics, 19:215-219
Bonabi Y, Asgharian H., Safari S. and Ahmadabadi
M. (2014), FPGA implementation of a biologicalneural network based on the Hodgkin-Huxley neuronmodel, Frontiers in Neuroscience, 8:379
Lu M, Wang J-L, Wen J, Dong X-W (2016),
Implementation of Hodgkin-Huxley neuron model in
FPGAs, Asia-Pacific International Symposium on
Electromagnetic Compatibility, 1:1115-1117
S. Joucla, M. Ambroise, T. Levi, T. Lafon, P. Chauvet,
S. Saighi, Y. Bornat, N. Lewis, S. Renaud, B. Yvert,,
Generation of locomotor-like activity in the isolated
rat spinal cord using intraspinal electricalmicrostimulation driven by a digital neuromorphic
CPG, Frontiers in neuroscience, 10, 2016
T. Levi et al., Biomimetic neural network formodifying biological dynamics during hybrid
experiments, Journal on Artificial Life and Robotics,
May 2017
Hodgkin AL, Huxley AF (1952), A quantitativedescription of membrane current and its applications
to conduction and excitation in nerve, The Journal of
Physiology, 117:500-544
Pospischil M, Toledo-Rodriguez M, Monier C,
Piwkowska Z, Bal T, Frégnac Y, Markram H,
Destexhe A (2008), Minimal Hodgkin-Huxley typemodels for different classes of cortical and thalamic
neurons, Biological Cybernitics, 99:427-441
Connors B. W, Gutnick M. J (1990), Intrinsic firing
patterns of diverse neocortical neurons, Trends
Neurosciences 13:99-104
Andraka R (1998), A survey of CORDIC algorithms
for FPGA based computers, Proc. ACM/SIGDA
Conf., p191-200
Levi T., Khoyratee F., Saïghi S., Ikeuchi Y. (2017),
Digital implementation of Hodgkin-Huxley neuronmodel for neurological diseases studies, Artificial
Life and Robotics
P. Bonifazi et al., In vitro large-scale experimental
and theoretical studies for the realization of bidirectional brain-prostheses. Frontiers in Neural
Circuits, 7:40, March 2013

Finding appropriate parameter voltages for driving a low-power analog silicon neuron circuit
Atsuya Tange

Department of Electrical Engineering and Information Systems, The University of Tokyo, 4-6-1 Komaba,
Meguro-ku, Tokyo, 153-8505, Japan
Takashi Kohno

Institute of Industrial Science, The University of Tokyo, 4-6-1 Komaba,
Meguro-ku, Tokyo, 153-8505, Japan
E-mail: tange@sat.t.u-tokyo.ac.jp, kohno@sat.t.u-tokyo.ac.jp
www.u-tokyo.ac.jp

Abstract
This research focuses on a silicon neuron circuit designed utilizing a qualitative neuronal modeling approach. In thiscircuit, temperature, fabrication mismatch, and secondary effects of transistors cause the difference between theintended characteristics and those in the implemented circuits. Therefore, we have to tune the bias voltages for eachneuron instance to realize the desired dynamical behavior after circuit implementation. We constructed an algorithm
to automatically find appropriate values for the bias voltages.

Keywords: neuromorphic hardware, neuromorphic chip, silicon neurons, analog VLSI

utilizing the qualitative modeling techniques, the model ofthis circuit was designed so that it can reproduce thedynamical structures in the excitable nerve membrane.
The parameter tuning procedure is supported by the
feedback amplifiers integrated in the silicon neuron circuit.
The characteristics of this circuit are influenced bytemperature, transistors’ fabrication mismatch, and theirsecondary effects (short channel effect, etc). They makethe expected circuit characteristics in designing stage andactual behavior of the individual circuit inconsistent.
Therefore, it is required to adjust bias voltages oftransistors in silicon neuron circuit to get the desired
dynamical behavior.
The procedure to do it by hand has been established3,
but it is not realistic to execute the procedure for eachsilicon neuron circuit when we build the large-scale siliconneuronal networks comparable to the human brain whichcontains about 100 billion neurons.

1. Introduction
Silicon neuron is electronic circuit that mimics
electrophysiological behavior of neurons. It is supposed to
be used as the basic elements of silicon neural networks,
whose aim is to simulate the behavior of the nervous
system in real-time or faster. They can be used not only as
the high-speed brain simulators for neuroscientificresearches but also as a basic technology for the next
generation low-power intelligent computing systems. In
addition, they can be used to construct bio-silico hybrid
systems in connection with neurons, which can be an ideal
technology for neuroprosthetic devices.
In this research, we focus on an analog silicon neuron
circuit designed by using the techniques of qualitative
neuronal modeling whose power consumption is as low as
about 3 nW.1 This circuit is composed of metal-oxidesemiconductor field-effect transistors (MOSFETs) in theirsubthreshold region for low-power consumption. By© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

473
1

Atsuya Tange, Takashi Kohno

Fig. 2. Automatic bias voltages tuning system forsilicon neuron

shallower gradient than the former. They are combined toconstruct the mathematical structures in the neurons.
andare constant current and input stimulus,
respectively. They are generated by transconductance
amplifiers.
There are nine bias voltages to tune this circuit, one ofwhich is related to the time constant of and the others
affect the shape of the sigmoidal curves.
Our circuit can reproduce Class1 and 2 neurons in the
Hodgkin’s classification6, by reconstructing specificdynamical structures for each class. In this work, weconcentrated on the Class1 neuron mode of our circuit.
To estimate the dynamical structures in our circuit, thetransconducance amplifiers (TAV and TAN in Figure 1)
are used to draw the nullcline of each variable. The
nullclines are drawn by measuring the current, theoutput current of TAV, and , the output current of ( ),while sweepingin DC analysis. Their equations are
represented as below.
= ( )− ( )++(3)Fig. 1. Block diagram of our qualitative-modelingbased low power silicon neuron circuit

To solve this issue, a system to fit characteristics ofsilicon neuron automatically is essential. In this work, thesame metaheuristic algorithm used in the previous similarworks2, 3, 4, the Differential Evolution (DE) method5, isused to find proper bias voltages. The DE method
performed better in terms of convergence speed andsimulation time, compared to other popular metaheuristic
algorithms, such as Genetic Algorithm and Simulated
Annealing.3 All the results in this work was obtained bycircuit simulation using ngspice.
Our silicon neuron model is explained in the next
section and our parameter fitting algorithm is in section 3.
The results and discussion are in section 4 and 5,
respectively.

=

2. A low-power analog silicon neuron model
Our silicon neuron circuit is composed of two blocks, the- and -blocks (Figure 1). Each block’s dynamics isrepresented as below.
=

( )−

=

( )+

( )−

( )+

−

( )+

−

( ),

,

( )−

( )+

(4)

3. Automatic parameter voltages fitting
algorithm
Figure 2 illustrates the block diagram of our automaticparameter fitting system. Here, we use circuit simulation
instead of actual circuit measurement because this work
focuses on the efficient verification of our fitting
algorithm in this system. It finds the appropriate parametervalues by optimizing the difference between the reference
data and the simulation result.
This optimization is done by using the DE method
which searches for the best-fitted parameter vector byrepeating the mutation, crossover, and selection ofparameter vectors for a number of generations. The

(1)
(2)

where and represents the membrane potential and the
dynamics of ionic channels, respectively.
For each function, ( ) ( = , ) is an ideal I-V
characteristics of a differential pair circuit, ( ) and ( )are ideal I-V characteristics of a cascoded transistor circuitwith source degeneration with a detached bulk voltage.
Both of them are in sigmoidal shape and the latter has© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan474
1

Finding appropriate parameter voltages

to produce the desired behavior similar to the reference
model, because our circuit’s dynamical properties are
sensitive to the shape of the nullclines and rn_Vm cannot
be fitted in Step1. In this step, we fit the dynamical
behavior in response to stimulus using the DE method.
Threshold current, spike width in response to pulse
stimulus, and spike frequency in response to sustained
stimulus are used for fitness function.
The definition of the threshold current is the minimum
amplitude of the pulse stimulus that makes the membrane
voltage exceed 400mV. The spike width is defined as time
width when the membrane voltage at the middle of the
spike amplitude.
Our silicon neuron circuit is equipped withtransconductance amplifiers for generating current
stimulus. Pulse stimulus with 500
time width was
generated by applying a pulse voltage to thetransconductance amplifier. We applied pulse stimuluswith a variety of amplitude by increasing the pulse voltage
by 7mV step for 20 pulses. The threshold current and the
spike width were measured with the firstly observed spikein this sequence.
For the spike frequency, two amplitudes of sustained
stimulus, 10pA and 15pA, were applied for 300ms using
the transconductance amplifier.
All of the 9 parameters are tuned in this step. The initial
parameter value was distributed in the range within thevalue obtained by Step1 ± 2 mV except for rn_Vm. For
rn_Vm, an initial value was found by sweeping until
spikes are generated in response to both pulse andsustained stimulus with the other parameters’ value
obtained by Step1. The range for this sweeping was
between the reference model’s parameter value (435 mV)-6mV to +12mV.
In this step, the fitness function is calculated as follows:= |( − ) |,where

Fig. 3. Nullclines in the Class1 neuron mode

automatic parameter fitting algorithm we suggest iscomposed of the 2 steps explained below.
3.1. Step1: Fitting nullclines
At first, we fit the shape of the - and -nullclines to agiven reference nullclines. Figure 3 shows the nullclines
in the Class1 neuron mode without stimulus. The two
nullclines intersect each other at three points, “S”, “T”,
and “U”. They represent a stable node, a saddle point, andan unstable node, respectively. Point “S” corresponds to
the resting state and “T” is responsible for the thresholdphenomena of neuronal spike generation.
The fitness function for the DE method is calculated bysumming the difference between the nullcines obtained in
the simulation and the reference nullclines. The fitness
function is written as follows:( )− , ,= ∑(5)= 1, 2 ⋯ , = 75, = , ,where ( ) and , represent the coordinate of a point on
the nullclines in simulation result and the reference
nullclines, respectively. is index of each point andrepresents or . The voltage range of the nullclines islimited to between 300 and 450 mV. This range covers thethree intersections of the nullclines in the Class1 neuron
mode. Four parameters, fx_Vb, fx_Vdlt, gx_Vm, Iax_Vin(x= , ) are tuned for each nullcline. Parameter rn_Vm isfixed at a random value in this step.
Initial vectors are generated in the range between 100mV and +100mV compared to the values used forgenerating the reference data. The DE method’s
optimization step was repeated for 400 generations and 40
vectors were contained in each generation.

⎡
⎢
=⎢
⎢
⎣

⎤
⎥
⎥,
⎥
⎦

⎡
⎢
=⎢
⎢
⎣
1
⁄
⁄
⁄

⎤
⎥
⎥,
⎥
⎦

(6)

⎡
⎤
⎢
⎥
=⎢
⎥.
⎢
⎥
⎣
⎦
is obtained by each transient simulation result andis obtained by the reference model. Elements of weight

3.2. Step2: Fitting neuron characteristics
The nullcline fitting procedure in Step1 was not sufficient© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

475
1

Atsuya Tange, Takashi Kohno

Table 1. Comparison of silicon neuron characteristics
Reference
Tuned
Error (%)
262.1
259.6
-0.953
[pA]
18.0
17.4
-3.30
[ms]
[Hz]
29.9
30.8
+3.01
[Hz]

42.9

43.5

+1.40

vector, , are calculated by dividing
by elementsof , respectively.
In this step, the DE method’s step was repeated for
200 generations and 45 vectors were contained in eachgeneration.

Fig. 4. Transient data of the reference (top) and thesimulation results of the tuned circuit (bottom)

neuronal classes such as the regular spiking, the squarewave bursting, and the elliptic bursting.
In the future, we will apply this algorithm to a realcircuit.

4. Results
Table1 shows the characteristic of our silicon neuron
circuit with the parameters obtained by our fitting
algorithm. For all the criteria, the error is less than 4%.
Figure 4 shows the reference transient data andsimulation results of the fitted circuit. The amplitude of
spikes seems to be well fitted.
The reference data was generated by using ngspicesimulation. Thus the ideal fitting result is identical to theparameter set used for the reference data generation, withwhich the error is zero. But as shown in Table 1, ouralgorithm could not find the original parameter set but
found a local optimum point.

Acknowledgement
This study was partially supported by JST PRESTO andCREST and JSPS-MAEDI SAKURA Program.
References
1. T. Kohno and K. Aihara, “A qualitative-modeling-based
low-power silicon nerve membrane,” 2014 21st IEEE Int.
Conf. Electron. Circuits Syst. ICECS 2014, pp. 199–202,
2015.
2. L. Buhry, F. Grassia, A. Giremus, E. Grivel, S. Renaud, and
S. Saighi, “Automated parameter estimation of the
Hodgkin-Huxley model using the differential evolution
algorithm: application to neuromimetic analog integrated
circuits,” Neural Comput, vol. 23, no. 10, pp. 2599–2625,
2011.
3. L. Buhry, S. Saighi, A. Giremus, E. Grivel, and S. Renaud,“Parameter estimation of the Hodgkin-Huxley model using
metaheuristics: Application to neuromimetic analog
integrated circuits,” 2008 IEEE-BIOCAS Biomed. Circuits
Syst. Conf. BIOCAS 2008, pp. 173–176, 2008.
4. E. Green, “A Multistage Heuristic Tuning Algorithm for an
Analog Silicon Neuron Circuit,” no. Icarob, pp. 4–6, 2017.
5. R. Storn and K. Price, “Differential Evolution – A Simple
and Efficient Heuristic for global Optimization overContinuous Spaces,” J. Glob. Optim., vol. 11, no. 4, pp.
341–359, 1997.
6. A. L. Hodgkin, “The local electric changes associated withrepetitive action in a non-medullated axon,” The Journal of
Physiology, vol. 107, no. 2, pp. 165–181, Mar. 1948.
7. T. Kohno and K. Aihara, “A three-variable ultralow-power
analog silicon neuron circuit,” no. 1, pp. 1–4.

5. Discussion
In this research, we proposed an algorithm to find properparameter values to obtain the desired dynamical
properties for individual silicon neuron circuit instance.
This algorithm will be crucial for operation of large-scale
silicon neuron networks. The error for the criteria in thefitting procedure was less than 4%. From the viewpoint ofengineering application, defining the maximum
acceptable error is a difficult problem, because it depends
on specific applications but they are not established yet.
From the viewpoint of neuromimetics, this error is
acceptable when our algorithm is used to tune a number ofsilicon neuron instances to Class 1, because there is a wide
distribution (far larger than 4%) of dynamical properties
in the same class of neuronal cells.
We will also try to apply this algorithm in fitting Class2
neuron. In addition, our algorithm will be improved to be
applicable to the three variable ultralow-power siliconneuron circuit.7 This circuit can realize a wide variety of© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan476
1

A low power silicon synapse with tunable reversal potential.
Ashish Gautam

Graduate School of Engineering, The University of Tokyo, 4-6-1 Komaba, Meguro-ku,
Tokyo, 153-8505, Japan
Takashi Kohno

Institute of Industrial Science, The University of Tokyo, 4-6-1 Komaba, Meguro-ku,
Tokyo, 153-8505, Japan
E-mail: asgautam@sat.t.u-tokto.ac.jp, kohno@sat.t.u-tokyo.ac.jp
www.u-tokyo.ac.jp

Abstract
Synapses are a building block of signal processing and computation in neuronal cell’s networks. We present
concept and simulation results of a pseudo five-bit, low power silicon synapse circuit capable of emulating both
excitatory (NMDA/AMPA type) and inhibitory (GABA type) responses. The post synaptic current generated in thiscircuit is proportional to the difference between the post synaptic membrane potential and a tunable synaptic
reversal potential.
Keywords: Synapse, neuromorphic, reversal potential, synaptic weights.

post-synaptic currents as pulses and others replicatedthe exponentially rising and decaying characteristic ofthese currents. Recently proposed models2 focused on
reducing the hardware footprint by sharing of synaptic
circuits having a single synapse circuit emulate multiple
synapses. These circuits are compact and powerefficient but do not reproduce the synaptic reversal
potential which is thought to be playing an importantrole in the information processing. In addition, theirstatic power consumption was not evaluated eventhough it has a large impact on the power consumptionof large-scale networks.
We propose a low-power synaptic circuit withsynaptic reversal potential that has pseudo five-bit ofsynaptic weights, implementing a familiar kinetic modelof synaptic conductance. Our design places its emphasis
on ultra-low power consumption and the exponential

1. Introduction
Research interest in the field of neuromorphic
engineering has grown significantly in the last decade.
The driving force behind the research is an attempt toarrive at a novel computing architecture whichsurpasses the limitations of the currently prevalent vonNeumann architecture by taking advantage of a highly
parallelized network structure of localized memory andprocessing units to emulate the information processing
in neuronal cell’s networks. Neuronal and synaptic
circuits that emulate the dynamics of neuronal cells andsynapses form the building blocks of this nascentcomputing architecture. Over the years a wide variety ofneuronal and synaptic circuits with a varied degree of
detail have been proposed. Earlier models1 simplified©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

477

Ashish Gautam, Takashi Kohno

profile of synaptic current emulating the currents
mediated by various neurotransmitters (AMPA, NMDA,GABAa, and GABAb) in the brain. In addition, thesynaptic current in our circuit is proportional to thedifference between the membrane potential and thesynaptic reversal potential.
2. Synaptic Model
It was shown that the kinetic models of the chemical
synapses which describe the neurotransmitter kinetics
can describe and reproduce the synaptic activities
precisely3. Following the analysis discussed in Ref.3,
the post-synaptic current in our circuit is given by thesimplest version of the kinetic models.
(1)
( )=
( )−
∗ ( )∗(),

Figure. 1: Schematic diagram of the proposed synaptic circuit.
Dimensions of the transistors: M4=0.3758*(w/l), M6=w/l,
M8= 2*(w/l) and M10=4*(w/l).Nets sVt0 and sVt1 areshorted and connected to a common node Vτ. Tail transistor
M14 is three times wider than M13.

where r represents the fraction of bound post-synaptic
receptors, gsyn is the maximal value of synaptic
conductance, Vsyn is the post-synaptic potential and Esyn
is the synaptic reversal potential.

to VddInt and current that flows through the device isgiven by (ignoring the short channel effect)
( ∗

=

)

,

(2)
where Iw0 represents the leakage current, sVw is thevoltage applied to the gate of M4 and Vsyn represents thesource voltage of the transistor M4. This current is thenintegrated by the RC network formed by capacitor Csyn
and the NMOS transistors M11 and M12 (as their gatesare shorted they act as a single transistor). During thecharging phase, these transistors are not in saturation as
Vsyn is initially 0. When the input pulse turns off, thegate of the transistor M1 goes high, the current Iw shuts
off and the capacitor begins to discharge linearlythrough the transistors M11-M12. When thesetransistors are saturated the current Iτ is given by

3. Proposed Synaptic Circuit
3.1. Description and analysis
The schematic diagram of the proposed synaptic circuit
is shown in Fig. 1. It consists of three major blocks, theinput stage (devices M1 to M10), the integrator (Csyn,
M11, and M12) and the transconductance amplifier(M13 to M19). The input stage is functionally similar tolog domain integrator synapse circuit proposed by
Merolla et al.4 with roles of PMOS and NMOS
interchanged. The PMOS transistor M1 acts as a switch,transistor M2 and the inverter I0 together provide thenecessary charge to be injected during the rising andfalling phase of the input pulse so as to avoid the
distortion of the current waveform at the switching
points of the input pulse due to the effect parasitic
capacitors inherent in the Metal-Oxide-Semiconductor
Field-Effect Transistor (MOSFET) device. Transistors
M3, M5, M7, and M9 act as switches and determine thestate of corresponding branches shown in the figure.
These branches are for implementing synaptic weights
discussed later. For the present discussion, we assume
only the branch consisting M3 is turned ON leaving M5,
M7, and M9 in the OFF state. An input pulse at the gateof transistor M1 pulls the drain terminal of NMOS M4

(

=

)

,

(3)
where Iτ0 represents the leakage current, Vτ controls thetime constant of the synaptic circuit. Voltage Vsyn is fed
into the tail transistor of the transconductance amplifier(M13 and M14), which generates a current proportional
to the difference of the voltage applied at the inputs ofthe differential pair (Esyn-Vmem). The generated current
Isyn is given by
=

(

)

∗

(

(

))

.

(4)

The source of the tail transistors is connected
to Vref, which minimizes the leakage current when Vsyn
is set to zero during the inactive state of the synapse©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan478
1

A low power siliconcircuit and turns off the tail transistor as soon as
possible during the discharging phase when thetransistors M11-M12 come out of saturation. By
differentiating Eq. (4) we get
=

∗

∗

,

∗

=(

−

),

∗

+

=

and the dynamics of the node Vsyn is given by
combining Eq. (5) and Eq. (6), we get∗

DAC. Because the number of synaptic circuits in siliconneuronal networks is very large, their footprint size has
to be minimized. We propose a more compact circuit
whose resolution is almost five-bit. In Fig.1 the four
branches consisting of transistors M3 to M10, with the
NMOS transistors sized appropriately provide 4 bit ofsynaptic weight. The fifth bit is represented by node
nWx4 which when active turns ON the switch M15 andincorporates transistor M14 along with M13 as the taildevice thus providing four times increase in themagnitude of synaptic current. Table 1 below lists down
the synaptic weights given by the five-bit input, nWa,
nW0, nW1, nW2, and nWx4. We call it pseudo five-bit
as the total number of weight values we get are 27
instead of 32. In our circuit, instead of using 32transistors to realize full five-bit, we used seven fullsized transistors for M6, M8 and M10 and one half sizetransistor for M4. This shrinks not only the footprint ofthese transistors but also that of M2 by reducing thecharge injection phenomenon.

(5)
(6)

∗

.

(7)
In the log domain integrator synapse describedin Ref. 4, the right-hand side of Eq. (7) comes out to bea constant due to the inverse relation between Iw and Isyn.
This is not exactly true in our circuit due to the bodyeffect of the NMOS transistor M4, but to avoid
complexity in our analysis we neglect the body effectand approximate the dynamics of our circuit as that of
the log domain integrator. By this Eq. (7) reduces to theequation of a first order low pass filter and its response
to a spike arriving atand ending at during thecharging phase is given by
( )=
−
−
,
(8)
and during the discharge phase
−
( )=
−
,
(9)

Table 1. Pseudo five-bit synaptic weights.

∗

.

nW0

nWa

nWx4

0
0
..
.

0
0
..
.

0
0
..
.

0
1
..
.

0
0
..
.

Weight
Value
0
1
..
.

1

1

1

1

1

26

1
1
1
..
.

1
0
1
..
.

1
1
0
..
.

0
1
1
..
.

15
16
17
..
.

4. Simulation Results(10)
During the discharge phase, a small current
flows out of the Vsyn node back into the synaptic input
stage due to the parasitic capacitance of the NMOS
transistors M4, M6, M8, and M10 which is comparable
to Iτ for higher values of time constant (100ms to200ms), and in this range to calculate the time constantthis additional current flowing out of the Vsyn node must
be added to Iτ in Eq. (10).
∗

nW1

1
0
0
..
.

where Iconst is the constant term on the right-hand side ofEq. (9),is the initial condition atand τ is thetime constant given by=nW2

We designed our synaptic circuit to be connected to thesilicon neuron circuit in Ref. 6. The scale of synaptic
current in our circuit was designed so as to interact withthis silicon neuron circuit, whose membrane capacitanceis about 900fF. Since the membrane capacitance of theneuronal cells is about several hundreds of picofarads
and the scale of synaptic currents is about several
hundreds of picoamperes, the synaptic current scale ofour silicon synapse circuit is set to about 10pA. This
range is far smaller than that in other silicon synapse
circuits in Ref.2 and Ref.5. The static power
consumption of our circuit was less than 2pW. To

3.2 Pseudo five-bit synaptic weights.
Wang et al. proposed a programmable five-bit synaptic
weight circuit5, where the weights are set by an on-chip©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

479

Ashish Gautam, Takashi Kohno

inhibitory setting (GABAa and GABAb in Fig. 2) will becompensated by silicon neuron circuit’s parameters.
5. Conclusion
We designed a silicon synapse circuit for a low powersilicon neuron circuit using Taiwan Semiconductor
Manufacturing Company (TSMC) 250nm CMOS
process and presented the simulation results. Our circuit
is designed to consume lower power compared to 2.8pJ7
which implements a DPI synapse discussed in Ref.2.
We were unable to find static power consumption value
for previously described synaptic circuits in Ref.2 andRef.5. The area occupied by a single synapse is
26.02x140.2 um2. In our future work, we plan to reduce
the area of synaptic circuits by significantly reducing
the size of the integrator stage and sharing one
transconductance amplifier with many input stages, aswell as improve the bit resolution of the synapse circuit.

Figure 2: Synaptic currents emulating the response of (a)AMPA (b) NMDA (c) GABAA (d) GABAB neurotransmitters.
sVw was set to 80mV for NMDA and GABAb mode and to
110mV for AMPA and GABAa mode.

6. Acknowledgements
This study was supported by JST PRESTO and CREST,
and VDEC, The University of Tokyo in collaboration
with Cadence Design Systems, Inc.
References
1.
2.
Figure 3: Synaptic current output over the dynamic range ofthe synaptic weights.

3.

process an input spike the dynamic energy consumption
was calculated to be about 500fJ. This value was arrived
at by subtracting the value of static energy dissipatedfrom the total energy consumed to process an input
spike, with the circuit configured to peak current valueof 3pA and time constant of 200ms. For the maximum
current of 10pA and maximum time constant of 200ms,
this value was calculated to be about 1.5pJ. Figure 2
shows the waveform of the synaptic currents emulating
the response of various neurotransmitters and Fig. 3
shows the plot of synaptic currents over the dynamic
range of synaptic weights described in Table 1. These
results were obtained by circuit simulation using
Spectre. All the traces are in typical shape of thesynaptic current waveforms. The offset observed in the

4.

5.

6.
7.

Mead, C. (1989). Analog VLSI and neural systems.
Reading, MA: Addison-Wesley.
Bartolozzi,C; Indiveri,G (2007). Synaptic dynamics
in analog VLSI. Neural Computation, 19(10):25812603.
Destexhe.et.al. (1998). Kinetic models of synaptictransmission. In Methods in neuronal modelling,from ions to networks (pp. 1–25). Cambridge, MA:
MIT Press.
Merolla P and Boahen K. 2004. A recurrent modelof orientation maps with simple and complex cells.
In: Advances in Neural Information Processing
Systems 16 (NIPS) MIT Press, Cambridge, MA. pp.
995-1002.
Wang, Y.,&amp; Liu, S. (n.d.). Programmable
synaptic weights for an aVLSI network of spiking
neurons. 2006 IEEE International Symposium on
Circuits &Systems.doi:10.1109/iscas.2006.1693637.
Kohno, T., & Aihara, K. (n.d.). A three variableultralow-power analog silicon neuron circuit. NOLTA
2016 (Yugawara, Japan, November 27-30, 2016).
Giacomo Indiveri, Federico Corradi, and Ning Qiao.
Neuromorphic architectures for spiking deep neuralnetworks. In IEEE International Electron Devices
Meeting (IEDM), 2015.

©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan480

New methodology of neural network reconstruction
for "in vitro" culture on MultiElectrode Array (MEA)
Timothee Leleu

Institute of Industrial Science, University of Tokyo, Japan
E-mail: timothee@sat.t.u-tokyo.ac.jp
Timothee Levi

Institute of Industrial Science, University of Tokyo, Japan
Takashi Kohno

Institute of Industrial Science, University of Tokyo, Japan
Kazuyuki Aihara

Institute of Industrial Science, University of Tokyo, Japan

Abstract
Techniques for inferring the network structure are generally based on cross-correlations and result in ambiguousreconstruction. Recently, we have proposed a method for which there is one-to-one correspondence between
statistical properties of packets of spikes (or avalanches) and the network structure. This method utilizes higher order
statistics of spike trains. We show using numerical simulations of biologically realistic neuron models that the
proposed method is general, and is well-fitted for the analysis of cultured neuronal networks coupled tomicroelectrode arrays.
Keywords: Network structure reconstruction, packets of spikes, neuronal avalanches, cultured neuronal networks.

From viewpoint of time series analysis, the task ofreconstructing the network structure requires solving the
network reconstruction inverse problem using theexperimental data. This is a very difficult problem and it
is well-known that classical reconstruction methods, suchas pairwise correlations3, cannot be used toreconstruction unambiguously the network structure4.
Indeed, cross-correlation techniques are not adapted for
determining the directionality of connections5 and cannot
distinguish between the effects resulting fromconnections between pairs of neurons and the ones
resulting from a shared input to these neurons4.
In the following, we describe a method that allows
utilizing the higher order statistics of spike trains in orderto reconstruct the network structure unambiguously6.
Higher order correlations in neuronal activity, called

1. Introduction
The realization of neural prostheses that could help
millions of people affected with motor and cognitive
disabilities requires engineering the interface between
neuronal cell assemblies and the non-organic machine1.
The proper simulation of neuronal activity that can
reproduce biologically realistic properties of neurons isnecessary for achieving this task. First, theelectrophysiological patterns should be simulated
accurately and efficiently in terms of computational
resources2. Second, it is necessary to reconstruct theneural network structure, such as micro-connectivity andthe strength of synaptic connections between pairs ofneurons. This requires developing efficient techniques
for inferring the network structure from recorded activity.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Beppu, Japan

481

Timothée Leleu, Timothée Levi, Takashi Kohno, Kazuyuki Aihara

neuronal avalanche, occur when the brain is operatingnear a critical state7. These neuronal avalanches occur
both “in vitro” 7,8 and “in vivo” 9 , but can be identified
more clearly “in vitro” experiments which havecomparatively lower noise levels due to smaller synaptic
density and better controlled experimental environment.
In particular, we show using numerical simulations ofIzhikevich spiking neurons2 that the proposed method
allows for accurate reconstruction compared to classicalnetwork reconstruction methods10.
2.

discrimination threshold is computed. Lastly,
neurobiological spike signals are extracted using, forexample, the wavelet transform.
2.2. Detection of neuronal avalanches
The second step consists in the time-binning of the
spiking activity. The detected spike train can bediscretized in time by considering the binary matrix A
such that A(n, j) = 1 if a spike triggered by the neuron jhas been detected during the time-bin n, and A(n, j) = 0
otherwise. We define a neuronal avalanche as a packet of
spikes, indexed by p and starting at the time-bin , as
the spiking activity in subsequent time-bins that ispreceded and succeeded by two empty time-bins indexedbyand, respectively, where ∆ , is the∆duration of the packet p. The spiking activity during apacket, noted , is given as follows10:

Proposed method

The proposed methodology for reconstructing the
network structure consists in the following steps (see Fig.
1): (1) the pre-processing of the raw data and binning of
spiking activity (2) the detection of spike packets orneuronal avalanches, (3) inferring the network structureusing the statistical properties of neuronal avalanches.
We note Ω = {ω } the matrix of the inferred network
structure with ω the inferred strength of synaptic
connection from neuron j to i. Note that the proposed
method is limited to the reconstruction of excitatory
connections for the sake of simplicity. Moreover, we
focus on the analysis of biological data recorded from
cultured neural networks coupled to microelectrodearrays.

= { ( , )}

∈{

,⋯,

,

∆

}

,

(1)

where A, j = 1 with j the index of the neuron thathas triggered the packet p, A
, i = 0 , and
A
,
i
=
0
,
∀i
.
The
spiking
activity
during a∆packet can be further summarized by summing the matrix
with respect to the time-bins, i.e., by considering the( )vectordefined as follows:
( )

∆

( )

=

( )

,

(2)

with
=∑
( , ) . Lastly, the averaged
composition of packets, noted, can be calculated by
distinguishing the packets as a function of the neurons jthat have triggered them as follows:

2.1. Pre-processing of raw data
First, spikes are detected from the recorded biological
data. Bandwidth reduction for selective band
amplification and noise reduction is applied. Next, aFig. 1. Schema illustrating the proposed method with (from left to right): pre-processing of raw data, detection of neuronal avalanches,
and application of the inversion formula.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Beppu, Japan482

Network reconstruction using avalanches

=

∑

( )

,

The proposed method for the reconstruction of network
structure from the analysis of neuronal avalanches is
illustrated in Fig. 1.

(3)

is the number of packets that start from thewhereneuron j.

3. Evaluation of reconstruction accuracy

2.3. Inversion formula

In order to test the proposed method, we simulate theactivity of neuronal networks using the Izhikevichmodel2,
which
reproduces
numerous
electrophysiological features of cortical neurons. The
parameter values chosen for the numerical simulations of
excitatory and inhibitory neurons correspond to regularspiking (RS) and fast spiking (FS) neurons, respectively.
Then, the accuracy of the network reconstruction
methods are measured using the Pearson correlation κ
between inferred weights and the ones used for thenumerical simulations of the Izhikevich model. The

The network structure can then be inferred from theaveraged composition matrix of packets M =using the following formula, called inversion formula6,11:
Ω∝ −

,

M∝∑

Ω .

(4)

where I is the identity matrix of size and M is invertible.
Note that eq. (4) can be rewritten as follows using the
Neumann series:(5)

Fig. 2. (a) Accuracy of the network reconstruction measured by the Pearson κ correlation between inferred and true weights vs.
averaged strength of excitatory to excitatory connections < ω >. (b1), (b2), and (b3) show the distribution P(L) of avalanche sizes L
in the subcritical, near critical, and supercritical regimes, respectively. In simulations of the Izhikevich model, synaptic noise is modeledby white noise with mean -2 (-2) and standard deviation 5 (6.25) for excitatory (inhibitory) neurons. The proportion of inhibitory neurons
and sparsity of connections are 0.1 and 0.2, respectively.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Beppu, Japan

483

Timothée Leleu, Timothée Levi, Takashi Kohno, Kazuyuki Aihara

window at proximity of the time t; and δ , the variationof firing rate given as δ ( ) = ( )−< >.

accuracy of the proposed method is compared to theclassical method based on pairwise correlations that is
described in Appendix A. Lastly, we keep a fix fractionof weights to be non-zero in all cases in order to avoid
larger reconstruction errors solely because a greaternumber of inferred weights are non-zero. This fraction is
given by the sparsity of the original network structure.

References

1. P. Bonifazi, F. Difato, P. Massobrio, G.L. Breschi, V.
Pasquale, T. Levi, M. Goldin, Y. Bornat, M. Tedesco, M.
Bisio, R. Galron, S. Kanner, J. Tessadori, S. Taverna, and
M. Chiappalone, In vitro large-scale experimental andtheoretical studies for the realization of bi-directional
brain-prostheses, Frontiers in Neural Circuits. 7(2013) 40.
2. E.M. Izhikevich, Simple model of spiking neurons, IEEE
Transactions on neural networks. 14(6) (2003) 1569–1572.
3. V.M. Eguiluz, D.R. Chialvo, G.A. Cecchi, M. Baliki, and
A.V. Apkarian, Scale-free brain functional networks,
Physical review letters. 94(1) (2005) 018102.
4. D.H. Perkel, G.L. Gerstein, and G.P. Moore, Neuronal
spike trains and stochastic point processes: II simultaneous
spike trains, Biophysical journal. 7(4) (1967) 419–440.
5. T. Schreiber, Measuring information transfer, Physicalreview letters. 85(2) (2000) 461.
6. T. Leleu and K. Aihara, Unambiguous reconstruction of
network structure using avalanche dynamics, PhysicalReview E. 91(2) (2015) 022804.
7. J.M. Beggs and D. Plenz, Neuronal avalanches in
neocortical circuits, Journal of neuroscience. 23(35)(2003) 11167–11177.
8. N. Friedman, S. Ito, B. Brinkman, M. Shimono, R. DeVille,
K.A. Dahmen, J.M. Beggs, and T.C. Butler, Universal
critical dynamics in high resolution neuronal avalanche
data, Physical review letters. 108 (20) (2012) 208102.
9. E.D. Gireesh, and D. Plenz, Neuronal avalanches organize
as nested theta-and beta/gamma-oscillations during
development of cortical layer 2/3, PNAS. 105 (21) (2008)
7576–7581.
10. T. Leleu. T. Levi, T. Kohno, and K. Aihara, Network
structure reconstruction using packets of spikes in cultured
neuronal networks coupled to microelectrode arrays,
NOLTA journal, in revision.

3.1. Reconstruction at the critical regime
Fig. 2 (a) shows that the proposed method performs better
than the ones using the pairwise correlations and twoconsecutive time-bins, notably near the critical regime.
The two consecutive time-bins method utilizes only thetwo first time-bins of each packet in order to reconstruct
the network structure. The distribution P(L) of theavalanche sizes L is shown in Fig. 2 (b1), (b2), and (b3)when the averaged recurrent excitatory connection
strength used in numerical simulation, noted< ω >, issmall (subcritical regime), intermediate (near criticalregime), and large (supercritical regime), respectively.
Smaller and very larger packets do not contain muchinformation about the network structure. In particular,larger packets become identical because the wholenetwork becomes active. It has been suggested that thecritical regime is the one at which neuronal networks
operate7-9. Thus, these results suggest that the proposed
method is well-fitted for the analysis of biological data.
Acknowledgements
This research was supported by ImPACT Program ofCouncil for Science, Technology and Innovation(Cabinet Office, Government of Japan). Authors thankthe Social Cooperation Program for Brain-Morphic AI to
Resolve Social Issues, IIS, UTokyo.
Appendix A. Pairwise correlation method
The network structure is classically reconstructed using
correlationof spiking activity within packets between
neurons i and j as follows:
=

,

(A.1)

where <> represent time averages; , the instantaneous
firing rate obtained by smoothing the spike train over a© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, Beppu, Japan

484

Technique of Recovery Process and Application of AI
in Error Recovery Using Task Stratification and Error Classification
Akira Nakamura, Kazuyuki Nagata

Intelligent Systems Research InstituteNational Institute of Advanced Industrial Science and Technology (AIST)
Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki, 305-8568 Japan
Kensuke Harada

Robotic Manipulation Research Group
Systems Innovation Department
Graduate School of Engineering Science, Osaka University
1-3 Machikaneyama, Toyonaka 560-8531, Japan
Natsuki Yamanobe

Intelligent Systems Research InstituteNational Institute of Advanced Industrial Science and Technology (AIST)
Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki, 305-8568 Japan
E-mail: a-nakamura@aist.go.jp, k-nagata@aist.go.jp,
harada@sys.es.osaka-u.ac.jp, n-yamanobe@aist.go.jp
www.aist.go.jp

Abstract
We have proposed an error recovery method using the concepts of task stratification and error classification. In thispaper, the recovery process after the judgment of error is described in detail. In particular, we explain how to change
the parameters of planning, modeling, and sensing when error recovery is performed. Furthermore, we apply artificial
intelligence (AI) techniques, such as deep learning, to error recovery.
Keywords: error recovery, task stratification, error classification, manipulation, artificial intelligenceparameters of planning, modeling, and sensing whenerror recovery is performed. This contributes tosystematic error recovery.
Performing extensive error recovery leads to theaccumulation of data about the recovery method.
Furthermore, we apply artificial intelligence (AI)techniques, such as deep learning, to error recovery.
The rest of the paper is organized as follows: Theconcept of error recovery is described in Section 2. We
modify the process of error recovery based on the size oferror. The recovery processes in the case of a small error
and a large error are explained in Sections 3 and 4,

1. Introduction
Error recovery is an important research topic in
robotic manipulation tasks in industrial production, plantmaintenance, and housework. However, systematicalmethods of error recovery have not yet been developed.
We propose error recovery using the concepts of task
stratification and error classification. In the proposed
method, the judgment of error is performed in theprocesses of the practice of a system. In this paper, therecovery process after the judgment of error is describedin detail. In particular, we explain how to change the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan485

Akira Nakamura, Kazuyuki Nagata, Kensuke Harada and Natsuki Yamanobe

Hand of
Manipulator

Cups

Hand-eye visionsystemManipulator

Hand of
Manipulator

Systemcomponents

Manipulator

BottlesFig. 2 Manipulators of robots working in daily life

2.2. Proposed error recovery method
Even though extensive research has been performedon techniques to recover an error that occurs in amanipulation system [4]–[6], systematization of error
recovery has not yet been achieved. Most of thesetechniques are applied for the recovery of a small errorthat occurs in the trajectory of a manipulation robot.
There are a few studies on recovery planning for a large
error, which causes failure of achievement of an orderedtask.
The proposed error recovery technique uses task
stratification and error classification, which we proposed
in Reference [7]. First, the cause of an error is estimatedwhen it occurs. Next, the appropriate correction for the
candidate of the cause is performed on a system. The
manipulation returns to the previous step, and the task is
re-executed successively (Fig. 3). The possibility of error
occurrence would be small as the system is corrected.
If an error is small, the process returns to the lowest
layer of a task hierarchy (Fig. 4). On the contrary, if anerror is large, the process moves to a high-ranking layerof the hierarchy, and it is re-executed from the previous
step (Fig. 4). Fig. 3 shows process flow for the former
case, in which stratification is not considered, and theflow of the latter case is provided in Reference [8].
In Reference [7], we proposed error recovery using a
backward correction process, which returns to theprevious step. However, if an error is small, we perform
error recovery using a forward correction process, inwhich an easy modification that shifts a trajectory isadded to the system without returning to the previous step.
In other words, in Reference [8], we propose an error
recovery method in which it is possible to select abackward correction process or a forward correction
process (Fig. 5).

Fig. 1 Manufacturing and maintenance robot
for audio-visual system components

respectively. We describe the usefulness of applying AI
to error recovery in Section 5.
2. Concept of error recovery
This section describes the proposed error recovery
method. The target task is the manipulation of a robot in
an industrial factory and a power generation plant (Fig.
1). Concrete instances are assembly, disassembly, part
replacement, and maintenance. In recent years, the roleof robots in human life and industrial facilities has
increased, and robotic manipulation during domestic
tasks such as cleaning, washing and cooking is also atarget of our error recovery system (Fig. 2).
2.1. Manipulation skill
First, we explain a fundamental unit and thecomposition of the manipulation task. Manipulation isconsidered based on a movement unit as a "skill".
We analyzed human motions in tasks such as
disassembly and reassembly and found that the
movements consisted of several significant motion
primitives. We refer to such motion primitives as “skills”[1], [2]. We considered three fundamental skills, i.e.,
move-to-touch, rotate-to-level, and rotate-to-insert, all ofwhich play an important part in such tasks. A specifictask is composed of sequences of skill primitives.
Moreover, several skills can be defined based on themodified versions of these three fundamental skills [3].
In addition, composite tasks can be stratified, and ahigher task possesses a lower task as a subtask; this is
described later.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

486

Technique of Recovery Process(Class 2) planning error, (Class 3) modeling error, and(Class 4) sensing error [7]. The process of handling anerror is explained in detail. However, as a process isrepeated without making modifications in (Class 1)
execution error (Fig. 3), we consider the remaining threekinds of errors in this study.

SensingChange of planningparameters

Modeling

Change of modelingparameters

Planning

3.2.1. (Class 2) Planning error

Change of sensing
parameters
task1
task2

Planning error occurs in cases where the equations
used to express a manipulation system do not express anactual system and the parameters used in planning areincorrect (Fig. 3). It includes the following corrections:

error
error

Classification of error

task k error( The error recovery
which returns to
the previous step )・Correction of the parameters of the equations used toexpress a manipulation system.

Class 1 : Execution error
Class 2 : Planning error
Class 3 : Modeling error
Class 4 : Sensing error

・Change in the initial condition for re-executing. For
example, revision of the position and orientation of
fingers.

Fig. 3 Fundamental process flow
with error recovery

・Change in the boundary condition and the thresholdin the control process.
・Change in size, the direction of power, and the torque

3. Recovery process in the case of a small error
This section describes the error recovery technique
for the case of a small error. In particular, the correctionof an error is described in detail.

Task layer (M)3.1. Conventional methods of controlling error

Task layer (2)Conventional control methods have been used tocorrect a minute error in a trajectory. Techniques that use
adaptive control, such as model reference adaptive
control (MRAC) and self-tuning adaptive control, are themost popular. In such methods, the error in a trajectory
can be minimized using a special control law while
operating it. Therefore, it is a kind of error recovery using
a forward correction process.

Task layer (1)Modification
Class N1

Modification

Class N２

Task layer (0)Task layer

Modification
Class NＭ

The error recovery which
returns to the previous step

Fig. 4 The expression of task stratification andthe process flow of the error recovery

to be provided.

3.2. Concept of proposed error recovery method

・Change in the planning technique to be used.

We proposed an error recovery method using task
stratification and error classification in Reference [7](Fig. 3, Fig. 4). In this method, a system is revised basedon the estimated cause of error, and the process isreturned to the previous step and re-executed. In otherwords, the method can be interpreted as recovery througha backward correction process. We considered thefollowing four kinds of errors: (Class 1) execution error,

3.2.2. (Class 3) Modeling error
Modeling error occurs when a geometry model insoftware does not express the actual geometry accurately.
A concrete correction (Fig. 3) includes the following:First, a change point about a single object is recorded.
・Change in position and orientation.
・Change in size.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

487

Akira Nakamura, Kazuyuki Nagata, Kensuke Harada and Natsuki Yamanobe

・ Change in various parameters of the sensing

・Change in shape.

Small(Lightweight)

Error
Large

(Heavyweight)

Recovered using past
modification data.

Use

Modified while
continuing workAI(a forward
correction process)No useRecovered with a steadyparameter modification.

Returned tothe previous
step andre-executed

Use

Recovered using past
return data.

AI(a backward
correction process)No useRecovered with a steadyreturn plan.

Fig. 5 Table about the error recovery techniqueequipment.

The models of all objects in a working environment
are considered as targets for these corrections. However,in several cases, the corrections are performed only forobjects directly related to work. For example, in work
involving touching an object grasped in two fingers
against a face of a board, the grasped objects, two fingers,
and the face of the board are considered as the subjectsof geometric models.
Next, it is important to be accurately model the
relation between the objects. The following aspects
should be considered for correctly modeling the relation
between the two fingers and the grasped object when thetask of touching the grasped object is performed:

・Change in the calibration parameters of the equations
used to express the relation between a manipulator
and a vision system.
We consider a correction process for the abovementioned error. It is desirable to know the ranges of thevalues of the parameters to be revised beforehand. The
value of a parameter is changed such that possibility oferror occurrence decreases, and re-execution is
performed. The values of parameters should be changed
as strategically as possible. In addition, it is necessary to
identify the interaction between parameters. If otherparameters depend on a parameter, then it is necessary tochange the values of these dependent parameters.

・It is important that the location of the face touching
the object is expressed correctly with respect toposition and orientation. It is corrected if necessary.

4.

・When the grasped object moves because of slipping,
it is important to express it using a correction.

Recovery process in the case of a large error

In this section, we explain the process of error
recovery by returning to a higher layer, as shown in Fig.
4. In Reference [7], we considered the repair task ofaudio-visual system components as an example andexplained the stratification of the task in detail.
In the case of a large error, it is frequently necessary toreturn to the previous step. When practice fails
completely, the process may have to return to the starting
point, i.e., the highest layer. The working environment
may change significantly because of failure. For example,
multiple objects may move, fall, and break in the working
environment because of failure of practice. In such a case,the contents of the re-execution task may have to bechanged significantly or entirely. Even when not

For example, the grasped object may slip between thetwo fingers owing to collision. Similarly, it is important
to express the relations between multiple objects
accurately.
3.2.3. (Class 4) Sensing error
Sensing error is caused by inaccurate sensing andinsufficient calibration. If an error occurs during working
practice, and its cause is judged to be a sensing error, thepractice task is stopped and modification is performed(Fig. 3). A concrete correction includes the following:© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

488

Technique of Recovery Processreturning to the highest layer, it is better to reconsider are-execution task in a few cases.
Plural passes may occur in a re-execution task. In sucha case, it is a suitable to select a pass based on an
evaluation standard to be established. There are various
evaluation standards; however, considering that an error
has caused the re-execution, a method of selecting a passthat ensures the achievement of the final goal of the task
is the most desirable.

judgment of error in detail. In particular, we explained
how to change the parameters that express various
conditions in planning, modeling, and sensing errors
when recovery is performed. Then, we changed theprocess of error recovery based on the size of an error.
The technique was derived based on task stratification. If
an error is large, the recovery process may move to ahigher layer. This implies returning to the previous step.
Our recovery method can be applied to various sizes oferrors. We explained the possibility of applying AI
techniques, such as deep learning, to error recovery.
Abundant available data on error recovery contribute to
the usefulness of AI. Future studies will include theapplication of our method to an actual system. Moreover,in error recovery using AI, we will study a technique ofmodifying system parameters in the case of a small error
and a method of constructing the re-execution task in thecase of a large error.

5. Possibility of the application of AI
In recent years, it has been shown that using AI
techniques, such as deep learning, is effective for solving
problems in several fields. In addition, the use of AI isefficient for error recovery.
When an error is small, it is easy to apply AI forparameter correction (Fig. 5). It is easy to estimate whichparameter should be changed and how much it should becorrected because there is a large amount of past data.
When an error is large, the use of AI contributes equally
to error recovery (Fig. 5). It is easy to estimate the higher
layer to which a pass should be returned as there is a large
amount of accumulated data.
It is effective to use AI in the following processes inour error recovery technique:

References
1. T. Hasegawa, T. Suehiro and K. Takase, A model-based
manipulation system with skill-based execution, IEEE
Trans. Robot. Autom., 8(5) (1992) 535-544.
2. A. Nakamura, T. Ogasawara, T. Suehiro and H. Tsukune,
Skill-based backprojection for fine motion planning, in
Proc. IEEE/RSJ Int. Conf. on Intell. Robots Syst., (Osaka,
Japan, 1996), 2, 526-533.
3. A. Nakamura, T. Ogasawara, K. Kitagaki and T. Suehiro,
Using robust and simplified geometric models inskillbased manipulation, in Proc. IEEE/RSJ Int. Conf. onIntell. Robots Syst., (Hawaii, USA, 2001), 1, 138-145.
4. B. R. Donald, Planning multi-step error detection andrecovery strategies, Int. J. Robot. Res., 9(1) (1990) 3-60.
5. E. Z. Evans and C. S. George Lee, Automatic generationof error recovery knowledge through learned reactivity, inProc. IEEE Int. Conf. Robot. Autom., (San Diego, USA,1994), 2915-2920.
6. C. M. Baydar and K. Saitou, Off-Line error prediction,
diagnosis and error recovery using virtual assembly
systems, in Proc. IEEE Int. Conf. Robot. Autom., (Seoul,
Korea, 2001), 818-823.
7. A. Nakamura and T. Kotoku, Systematization of error
recovery in skill-based manipulation, Artificial Life andRobotics, Springer, 14(2) (2009) 203-208.
8. A. Nakamura, K. Nagata, K. Harada, N. Yamanobe, T.
Tsuji, T. Foissotte and Y. Kawai, Error recovery using task
stratification and error classification for manipula-tion
robots in various fields, in Proc IEEE/RSJ Int. Conf. onIntell. Robots Syst., (Tokyo, Japan, 2013), 3535-3542.

・Estimation of the cause of error.
・ Parameter correction (which parameter should bechanged and how much)・Selection of a point in a higher layer to return to (i.e.,
return to which step)・ Determination of the contents of a pass for reexecution (by what kind of task)
Applying AI is effective for error recovery. However,as abundant data are necessary, it is required that there ismuch outbreak of the error, and demand contradicts ideal
performance in a sense. When there are only a few errors,
there is less experience; thus, it is important to plan andperform recovery steadily, and not to the extent to which
AI can be used for recovery.
6. Conclusion
Theoretical systematization of error recovery has not
yet been developed. We proposed error recovery using
task stratification and error classification. In this paper,
we described the recovery process carried out after the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

489

,

Motion selection for 3D robotic snap assembly
Peihao Shi, Kensuke Harada, Weiwei Wan, Ixchel G. Ramirez

Department of Systems Innovation, Osaka University, 1-3 Machikaneyama-cho
Toyonaka, Osaka, Japan.
Juan Rojas

School of Mechanical and Electrical Engineering, GDUT, 100 Waihuanxi Lu, Gongxue 2, 105
Guangdong University of Technology, Guanzhou, Guandong, China
Hiromu Onda

AIST, 1-1-1Umezono
Tsukuba, Ibaraki, Japan

Abstract
In this paper, we aim to provide an assembly method for a snap joints assembly task. We create a 3D cellphone modeland use ADAMS simulation environment to analyze the relative motion between screen and backer part. We focus
on two points in this research. 1) Two kinds of relative motion between the screen and backer parts, i.e., the rotationbased and the translation-based methods, are compared, and 2) difference between assembly and disassembly isanalyzed. By using the maximum elastic energy in an assembly process, we show that 1) the rotation-based assembly
motion has better robustness than the translation-based assembly motion in cellphone assembly tasks when we set
the same initial position error, and 2) The rotation-based assembly method is more effective for snap joint disassembly.
Keywords: Robotic Assembly, Snap joints, Strain Energy.

1. Introduction
simulation environment to analyze the assembly processof the cellphone. Using the Maximum Elastic Energy(MEE) as an index for evaluating the quality of a snap
joints assembly tasks, we compare two kinds of assembly
methods, i.e., rotation-based and translation-based
methods. We also compare the difference of MEE inassembly and disassembly when we choose different
assembly method. By using the maximum elastic energy
in an assembly process, we show that 1) the rotationbased assembly motion has better robustness than thetranslation-based assembly motion in cellphoneassembly tasks when we set the same initial position error,and 2) the rotation-based assembly method is more
effective for disassembly tasks.

Snap joints connection is one of the typical examples ofplastic parts assemblage. As the widely application andmass-production of plastic parts, snap joints assembly is
becoming more and more popular in assemble method ofIndustrial products. In snap joint assembly, theconnection between two parts is built by elastic
deformation of snap joints. [1] Fig.1 shows an exampleof the snap assembly task. Because of the elastic
deformation of the snap joints, it becomes relatively hard
for an industry robot to finish snap joints assembly task.
As for the research on robotic snap joints assembly. Rojaset al. have proposed a force control method [5] andidentification of assembly state [6-8] for the snap
assembly problem.
In this paper we aim to provide a better assembly
method for a cellphone snap joints assembly task. We
create a 3D model of a cellphone, and use ADAMS© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan490

Peihao Shi, Kensuke Harada, Weiwei Wan, Ixchel G. Ramirez, Juan Rojas, Hiromu Onda

By compare the MEE of different assembly methods,
we can choose the better assembly method which the
MEE is smaller.
3. Experiment and Analysis.
In this section, we set two kinds of relative motions
where one is translational assembly and the other isrotational assembly. These two motions are
representative human assembly motion. The rotation
based relative motion is shown in Fig. 3 where the snap
on one side of the screen part first contact the backer part,
then the screen part is rotated until the snap joints onanother side comes into contact with the backer part,finally the screen part translates until it completely fits
into the backer part. On the other hand, translation based
relative motion is shown in Fig. 4 where the screen partsimply translates to fit into the backer part. We computeand compare the MEE of rotation-based/translationbased relative motion. And the better assembly method is
be defined as the one bring smaller MEE in assembly
process.

Fig.1 Snap assembly task

2. Approach and Definitions
We create a 3D model of a cellphone used in a physics
simulation environment where the screen part contains
four snap joints as shown in Fig. 2. We simulate theassembly process by using the predefined relative motion
between two parts. Throughout a physics simulation of asnap joint assembly, we consider obtaining the function
curve of strain energy and obtaining the MEE from thestrain energy curve.

3.1 Initial Position Errors.
In robotic assembly tasks. Influence of initial positon
errors can be used as a criterion of robustness of assembly.
In our experiment, we set the initial position errors ofscreen part to 0mm, 1mm, 2mm to compare the rotationbased and translation-based assembly motion.

Fig.2 3D cellphone model.

We consider Ei(s) as the total elastic energy in theassembly process. Let s be the functions of a scalar,0≤s≤1, and s changes from 0 to 1 as the part is being
assembled. The maximum elastic energy (MEE) Ui is
being defined for snap assembly/disassembly task of Parti. [9]=

Fig.3 Rotation-based relative motion.

Fig. 5 and Fig 6 shows the simulation results when
we set the initial position errors to 0mm. The result(1)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

491

Motion selection for 3D

shown the MEE of translation based-motion (1.19J) isbigger than the MEE of the rotation-based motion (0.94J).
Fig. 7 and Fig. 8 shows the simulation result when we
set the initial position errors to 1mm. The result shown
the strain energy of translation based-motion (2.17J) isbigger than the rotation-based motion (1.12J).

shown the strain energy of translation based-motion(3.51J) is bigger than the rotation-based motion (1.13J).

Fig. 7 Stain energy of 1mm position error in translation-based
assemblyFig.4 Translation-based relative motion.

Fig. 8 Stain energy of 1mm position error in rotation-based
assembly

Fig. 5 Stain energy of 0mm position error in translation-based
assembly

Fig. 9 Stain energy of 2mm position error in translation-based
assembly

Fig. 6 Stain energy of 0mm position error in rotation-based
assembly

Fig. 9 and Fig. 10 shows the simulation result when
we set the initial position errors to 1mm. The result© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan492

Peihao Shi, Kensuke Harada, Weiwei Wan, Ixchel G. Ramirez, Juan Rojas, Hiromu Onda

Fig. 10 Stain energy of 2mm position error in rotation-based
assembly

Fig. 12 Stain energy of assembly/disassembly process inrotation-based assembly

The simulation result shown a conclusion: when we
set the same initial position errors, the MEE in rotationbased assembly process is smaller than translation-based
assembly process. And proved rotation-based assembly
method has better robustness than translation-based
assembly method.

The simulation result shown that the rotation based
assembly method is more effective when we disassembly
the cellphone screen part.
Conclusions.
This paper analyzes the 3D cellphone snap joints
assembly task. We compare two kinds of assembly
method from the following points. First is the initial
position errors of screen part. Second is difference
between assembly and disassembly.
Based on the simulation result. We can draw thefollowing conclusions: the rotation-based assembly
method has better robustness than translation-based
assembly method when we set same initial positon error.
The rotation-based assembly method is more effective
when we disassembly the cellphone screen part.

3.2 difference of assembly and disassembly.
As the second point, we analyze the difference ofassembly and disassembly in two kind of assembly
methods. The MEE is different in assembly anddisassembly when we use translation-based assembly
method. Fig. 11 shows the MEE in assembly/disassembly
process based on translation-based relative motion. The
MEE in assembly process is 1.19J, it is smaller than the
MEE in disassembly process (1.52J). Nevertheless, Fig.
12 shows the MEE in assembly/disassembly process
when we use rotation-based assembly method, the MEE
is almost same in assembly and disassembly process(0.94J).

References
1. P. Shi, K. Harada et al., Extended Directional Blocking
Graph (EDBG) for Snap Joint Assembly. (The 34-th
Annual Conference of the RSJ, 2016)
2. Randall H. Wilson and Jean-Claude Latombe, Geometricreasoning about mechanical assembly, (Artificial
Intelligence, 71, pp.371-396, 1994. )3. [3] H. Chang and T.-Y. Li, Assembly Maintainability
Study with Motion Planning, (Proc. of IEEE Int. Conf. on
Robotics and Automation, 1995. )4. D. Koditscheck, An Approach to Autonomous Robot
Assembly, (Robotica, vol. 12, no. 2, pp. 137-155, 1994. )5. J. Rojas, K. Harada et al., A Constraint-Based Motion
Control Strategy for Cantilever Snap Assemblies,
Proceedings of 2012 IEEE International Conference on
Mechatronics and Automation, 2012, pp.1815-1821.
6. J. Rojas, K. Harada et al., A Relative-Change-BasedHierarchical Taxonomy for Cantilever-Snap Assembly
Verification, IEEE/RSJ International Conference onIntelligent Robots and Systems, 2012, pp.356-363.
7. J. Rojas, K. Harada et al., Early Failure Characterizationof Cantilever Snap Assemblies using the PA-RCBHT,
IEEE International Conference on Robotics & Automation(ICRA), 2014, pp.3370-3377.
8. J. Rojas, K. Harada et al., Strategies, Controllers, andCoordination: Bi-Manual Snap Assembly Automation,
International Conference on Robotics and Biomimetics,
2014, pp.1266-1271.
9. P. Shi, K. Harada et al., Initial Motion Analysis for Robotic
Snap Assembly. (The 35-th Annual Conference of the RSJ,2017)

Fig. 11 Stain energy of assembly/disassembly process intranslation-based assembly© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

493

Control Performance Assessment Method as Assessment of Programming Learning
Achievement

Yoshihiro Ohnishi

Faculty of Education, Ehime University, 3, Bunkyocho
Matsuyama, Ehime 790-8577,JAPANE-mail: ohnishi@ehime-u.ac.jp,
www. ehime-u.ac.jp

Abstract
It is an important problem how to estimate the learning achievement of the programming learning. However, thequantitative evaluation method according to the learning achievement is difficult. On the other hand, "work" of acontrol program influences the control performance of the plant by petrochemical industry, and to have an influenceon the operation cost, Improvement of a program and am planning for productivity improvement are suggested byevaluating the control performance quantitatively. This research considers the using control performance assessment
method for the achievement value of the programming learning.
Keywords: programming learning, control performance assessment, minimum variance, learning achievement

1. Introduction

2. Control performance assessment

It is an important problem how to estimate the learningachievement of the programming learning. However, thequantitative evaluation method according to the learningachievement is difficult. On the other hand, "work" of acontrol program influences the control performance ofthe plant by petrochemical industry, and to have an
influence on the operation cost, Improvement of aprogram and am planning for productivity improvement
are suggested by evaluating the control performancequantitatively. The idea of control performanceassessment is becoming important in the process control
area[Huang and Shah, 1999], [Jelali, 2006],[Jelali, 2013].
This research considers the using control performanceassessment method for the achievement value of the
programming learning.

The judgement of whether a controller is showing thedesirable performance has been formed out of thesubjectivity of the expert operator familiar with a plant
concerned in the past. There is an index of rise time, theovershoot amount and an attenuation vibration
characteristic as the quantitative control performanceindex of the controller. However, these index can beobtained in the transient state which is at the time of a set
value change. The controlled value of the process control
system represented by the petrochemical plants can givethe reference value as the constant value to thetemperature, the pressure and the flow rate. In this case,the steady state control performance which is theminimizing the control error variance is more important© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan494

Yoshihiro Ohnishi

than the target value tracking performance or the
disturbance reply. For example, in the system oftemperature control to keep the constant value oftemperature as shown in Fig.1, it is required to keep thetemperature higher than some point. It's necessary to set
the high reference value to keep the minimum
temperature before the adjustment indicated on figure 1
left side. It's possible to reduce the output variance after
the adjustment indicated on figure 1 right side, and even
if the target value lower than the before adjustment, it's
possible to keep the minimum temperature. The energy
the adjustment back requires more than this case andbefore an adjustment can be done small. Thus it's
sometimes tied with efficiency of the energy to disperse
a control error small.
These ways have the method based on the benchmarkof the minimum variance control [Harris, 1989], themethod to consider a fluctuation of the manipulated value
as well as output error [Grimble, 2002], the method toexpress the control performance to compare with data inthe past in a plant concerned, and the way to limit thestructure of the controller for example the a model based
predictive control or PID control and express the control
performance in the structure.
This research defines the good program makes thegood control which realize the small output variance.

equation.

y(t) z d Gp (z 1 )u(t) Gn (z 1 ) (t)(1)
Where y(t) and d system output and minimum estimate
value of dead time , respectively. u(t) is control input,
and ξ (t) is probability noise including such as
observation noise. In addition, when information about
dead time can’t obtain nothing, d is set as 1. Feedback
control law is given as following equation for Eq.(1).
(2)
u (t ) G c ( z 1 ) e (t )Where, e(t ) means the control error signal which isdefined as:e(t ) : r (t ) y (t )(3)From substituting (1)(1) into (3):r (t ) Gn ( z 1 ) (t )1 z d G p ( z 1 )Gc ( z 1 )

e(t )

(4)

Here, by considering the steady state, control error
signal can obtain as following equation.

e(t )Gn ( z 1 ) (t )1 z d G p ( z 1 )Gc ( z 1 )(5)where, the noise transfer function Gn ( z 1 ) can bedivided the term about the dead time before and the term
about the dead time after by the following Diophantine
equation:Gn ( z 1 ) Gn1 ( z 1 ) z d Gn 2 ( z 1 )(6)1
where, G n1 ( z ) is polynomial given by following
equation.

Gn1 ( z 1 )n11 z 1n0

nd 1 z d 1

From eqns.(5) and (6)e(t )Gn1 ( z 1 ) z d Gd ( z 1 )

(7)

(t )

(8)

Where G d ( z ) is defined as following equation:
1

Gd ( z 1 ) :Fig. 1

G n 2 ( z 1 ) Gn1 ( z 1 )G p ( z 1 )G p ( z 1 )1 z d G p ( z 1 )Gc ( z 1 )(9)And, eqn.(8) is expressed as following equation
including

The example of the temperature
control systems.

e(t )n0 (t )

n d 1 (td 1)b (t )

d 0 (t d ) d 1 (t3. MV-index

d 1)

w( t d )This section introduces the calculation of MV-index
which is proposed by Desborough and Harris[Desborough and Harris, 1991].
The generalized controlled object described as followingb(t ) w(td)(10)
Here, b(t ) isn’t associated with control input and if

w(td ) is possible to be 0 by control input, let b(t)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

495

Control Performance Assessment Method

where, e and X are constructed by previous signals.
And,and b are the autoregressive parameter vector
and modeling error vector, respectively.
These are expressed asminimum variance output.
Here,shows actual variance of control error andshows achievable minimum variance of control
error at this system.
Furthermore, minimum-variance control performanceassessment index η is shown by following equation:

＝

e

(11)

The range is 0 ≤ ≤ 1 because≥.The large
value of MV-index
means that good control
performance.
MV-index _ can be calculated by eqn.(19). However, thenumerator of eqn.(11) cannot be calculated directly. This
term is obtained as following procedure（３.
The following equation can be derived bymultiplying
for both sides of eqn.(10)(t d )e(t d )
Gn1 ( z 1 ) z d Gd ( z 1 )(12)( − ) can be express from eqn.(12).
e(t d )
(13)

(t

d)

G n1 ( z 1 )z d Gd ( z 1 )

Gn1 ( z 1 ) (t )(14)Here, the following equation is also obtained because thefirst term of eqn.(14) is same as b(t) of eqn.(11) ,Since the coefficient of the second term of eqn. (15) can

d)

(15)
be expressed as the finite transfer function, (15) can be
expressed as the following AR model:

e(t )

b(t )
i 0

i

e(t

d

i)

e( n d )e( n dp 1)

e(n d 1)e( n dp)e( p )

(19)

e(1)
1

b

(18)

,

2

,

,

(20)

p

b ( n), b( n 1),

, b(d

2

e Χˆ

p)

(21)
where, n is the sample length.
The parameters of eqn.(20) can be obtained by:ˆ ( Χ Χ) 1 Χ e(22)Therefore, the minimum variance can be obtained by
MV

e ΧK̂

(23)

(14)
4. Simulation Example

0 .5
e Ls
20 s 1(24)

These models are discredited by 1.0[s] samplinginterval.
The following equations are obtained.

y(t ) 0.951y(t 1) 0.0243u(t 3)

(t )

(25)

Where, the mean value of ξ is 0，ξ_ = 0.05 . The
manipulate value u is limited as 0<u<100.
First, the ON-OFF control expressed as the following
rule was employed for the controlled object (25)b(t )Gd ( z 1 )e(tGn1 ( z 1 ) z d Gd ( z 1 )

p)

G (s)

G d (z 1)e(t d )
Gn1 ( z 1 ) z d Gd ( z 1 )e(t ), e( dThis section demonstrates the effectiveness of the
proposed scheme by a simulation example. The
following equations are used as the controlled object:

The following equation can be derived by eqn.(8) andeqn.(14).

e(t )

X

e( n ), e( n 1),(16)

The order p must be enough larger than the order of thecontrolled object. The relations of eqn.(16) arecomposed by the following matrix equation of the
form
e X
b
(17)

If(y(t)<30) then u(t)=100

else u(t)=0.0

This method is an example algorithm which is used by
beginner. This algorithm is Conditional branch typewhich is studied on technology class in Junior highschool.
Fig 2 is the control result by using the ON-OFF control.
And MV-index is 0.0208 which calculated on 200≦t≦
1200.
Next, the following PID control method is employed.

u (t )

k p {e(t )

e(t )
TI

TD e(t )}© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

496

(26)

(27)

Yoshihiro Ohnishi

programming learning. In this paper, a method based on
the controller performance assessment was proposed.
The effectiveness of the proposed method was evaluatedby the simulation example.

Where, kp is proportional gain, TI is the reset time, TD
is the derivative time. Each value was set as kp = 7.7，TI= 17，TD = 0.9, respectively. Fig.3 shows the control
result by this PID control method. And MV-index is0.2585 which is calculated on 200≦t≦1200.
From the result of Fig.2, the input value is very
oscillatory. The variance of the output value becomeslarge by the oscillatory input. From Fig.3, the varianceof the output signal become small by the PID control. The
MV-index can be the good metric to evaluate theprograming performance.

References
1. B. Huang and S. L. Shah: Performance Assessmentof Control Loops: Theory and Applications, Springer,
London (1999)
2. M. Jelali : An overview of control performanceassessment technology and industrial applications,Control Engineering Practice, Vol.14, pp. 441-466(2006)
3. M. Jelali : Control Performance Management inIndustrial Automation ,Springer, London (2013)
4. T. J. Harris: Assessment of Closed Loop Performance,
Canadian Journal of Chemical Engineering, Vol.67, pp.
856-861 (1989)
5. M. J. Grimble: Controller performance benchmarking andtuning using generalized minimum variance control 。
Automatica, Vol. 38, pp.2111-2119 (2002)
6. L. Desborough and T. J. Harris: Performance assessment
measures for univariate feedback control, The Canadian
Journal of Chemical Engineering, Vol 70, pp.1186-1197(1992)

Fig. 2

The result of the ON-OFF control.

Fig. 3

The result of the PID control.

5. Conclusions
This paper considers the using control performanceassessment method for the achievement of the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

497

Practice of Control Education by Experiment using Robot
Shinichi Imai

Graduate school, Tokyo Gakugei University,
4-1-1 Nukuikita-machi, Koganei-shi, Tokyo, Japan
Matsui Hideto

Graduate school, Tokyo Gakugei University,
4-1-1 Nukuikita-machi, Koganei-shi, Tokyo, Japan
Akira Yamada

Graduate school, Tokyo Gakugei University,
4-1-1 Nukuikita-machi, Koganei-shi, Tokyo, Japan
E-mail: shimai@u-gakugei.ac.jp, b122810x@st.u-gakugei.ac.jp, le@u-gakugei.ac.jp
http://www.u-gakugei.ac.jp/

Abstract
In this paper, for students who attend lectures in control engineering, there are not many students whose understanding
is insufficient. The reason why understanding is insufficient is to use many formulas. For that reason, various studies
have been conducted on control education. Therefore, in this research, propose a control education method using arobot so that interests and interests are given to students, practice lessons and verify its effectiveness.
Keywords: Control education, robot, Feedback control, Practical education

technology from 2012. Therefore, many reports arereported that aim to enhance control education 1,2.
However, although control engineering is amathematical content and contents of learning that isdifficult for beginners to understand, it is a practicaluniversity that is useful only when control theory
expressed by mathematical expression is used, and it is asmooth bridge between schooling and practicality
Experiments are essential academics. In learning control
engineering, many universities and colleges haveconducted methods to learn how to use control theory andlearn effectiveness by simulation on a computer, but do
not realize what the learned contents are used for Current
situation is that students' interests and interests are notgood. Also, in the "Industry" of "High School Study
Guidelines3" each subject which is mainly established in

1. Introduction
Today in sophisticated industrial society, globalizationof society advances, rapid progress of science andtechnology and remarkable change of industrial structure
are progressing. In recent years, with the development ofcomputer technology, computerized control is used invarious fields. Therefore, the importance of control
engineering in the industrial world in Japan is increasing,and training of teachers who can instruct control
education is necessary. In control technology as well,control technology is used for nearly all devices such as
air conditioners, electric pots, automobiles, etc. fromfamiliar objects to robots, aircraft and spacecraft. In
addition, classes of "measurement and control" are being
implemented in the teaching of junior high school© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

498

Shinichi Imai, Matsui Hideto, Akira Yamada

the Department of Specialization ", there is an item"
Electron Measurement Control "within the subject, andoutline of electronic measurement control, sequence
Control, feedback control, and learning the basics ofcomputer control are included. From this, it isindispensable to handle feedback control and sequencecontrol in industrial high school.
Therefore, in order to make students interested /interested, it is thought that it is desirable to learn throughexperiments, not only at the school. However, theexperimental apparatus related to control has many largescaled apparatuses and the price is very expensive and it
is difficult to conduct experiments with a small numberof people.
The purpose of this research is to develop controlled
educational materials which can learn each operation ofmanual control, automatic control, PID control
experientially using a robot.

Motor & tachometer
Ball

Goal

Distance Measuring

Fig.1

Control teaching materials robot

acquire knowledge and techniques concerning sequencecontrol. Typical equipment such as switches, relays, andtimers used for sequence control, types and uses of
sensors that detect physical quantities of light,
temperature, force, and logic elements are taken up. Also
learn about features, functions, characteristics, symbol
symbols and make them available for actual use.
Feedback control aims to handle the basics of feedback
control, control characteristics and utilization of
feedback control, and to acquire knowledge andtechniques related to feedback control. Here, it isdescribed that typical feedback control examples such asprocess control, servo control and the like are taken up,
understanding of the outline thereof is made and it ismade available for practical use.

2. Content on measurement and control
described in the guidelines for teaching highschoolIn the case of teaching the lesson of control, it is moredesirable to acquire expert knowledge related to theguidance contents as well as knowledge on the guidancecontents as well as to achieve smoother and moreadvanced instruction. Therefore, it is necessary toconstruct a lesson curriculum for students of teacher
training college based on the guidelines for teaching. In
the commentary on teaching guidelines for high school
educational guidelines in industrial edition, contents
related to control and items to be handled are clearlystated in the 24th electronics measurement control andothers as follows.
(1) Outline of electronic measurement control(2) Sequence control(3) Feedback control(4) Basics of computerized control
In addition, it is described that the above contents
comprehensively understand measurement technology,
automatic control technology and computer technology.
Sequence control aims to handle the fundamentals of
sequence control, equipment of sequence control, use ofbasic circuits and programmable controllers, and to

3. Educational robot teaching materials

Fig.1 shows the overall image of the educational
robot teaching material. The robot can blow ping-pongballs. The robot is composed of a motor for flying ping
- pong balls, a tachometer capable of detecting the
speed of the motor, and distance measuring(GP2Y0E03) for determining the distance to ping pongballs. This robot can control the number of revolutionsof the motor according to the distance of distancemeasuring, and can go to the goal with the ping-pong ball.
3.1 Precautions for production
In making a robot, cautioned is as follows.
(1) The operation of control can be visually
recognized(2) Produce at low price.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

499

Practice of Control Education

Motor & tachometer

Distance Measuring

Fig.3 circuit

Fig.2 robot

Motor drive

It is possible to actually observe how the control input
computed based on the theory is added to the controlled
object or the behavior of the controlled object. Also, bylistening to the operation sound of the device, touching
the switch or the button, you can feel the operation of thedevice. Furthermore, since teaching materials break
down when used, it is intended to facilitate the exchangeof parts. Fig.2 shows the actual appearance of the robot.
Parts of this robot are purchased at a 100 yen shop. The
parts cost is about 5000 yen per one.

P gainI gainD gainManualReference

3.2 Hardware
Hardware is shown in Fig.3 and the circuit diagram of
the robot is shown. The computer uses Arduino Uno
Rev3 (Next, Arduino). Using this computer, calculate theinput to the motor from commands from the operation
unit and information from the sensor. Connect Arduino
and personal computer (PC). The PC displays the resultsof input and output and various gains. In addition, as
shown in Fig.4, a volume is attached to the circuit board,
so that it is possible to adjust each gain. Next, there is aswitch that switches between manual control andautomatic control.

Fig.4 Operatione[V] is the output voltage of the sensor, and e is anumerical value from 0 to 1023. Skip the ping-pong balls
from the motor to the goal and measure the output voltageof the tachometer 10 times when the position of the robot
is blown at 10 cm intervals. The relationship between thevoltage and the position of the tachometer is found by theleast squares method.
4. Experience contents
Practice hands-on training using a robot. Specificcontents of education in experiential experiments are as
follows.
·Difference between manual control and automatic
control·Difference between ON-OFF control and PID control·How to adjust the PID gain of automatic control
Consider the block diagram as shown in Fig.5. However,r(t) is the target value, y(t) is the output, and e(t) is thedeviation (=r(t)-e(t)) between the target value r(t) and theoutput value y(t). u(t) is the input (manipulated variable).
Fig.5 Block diagram

3.3 Software
Arduino converts the input analog voltage to a numericvalue (integer) from 0 to 1023 in Arduino by 10 bit
resolution. The output voltage is controlled by PWM
control. The output can change the duty ratio by
specifying one numerical value (integer) of 0 to 255 with
8 bit resolution. When the numerical value is 0, the duty
ratio is 0%, in the case of the numerical value 128 it is50%, in the case of the numerical value 255 it is 100%.
From the voltage output by the tachometer, the rotation
speed of the rotor is calculated from the following
formula.
.
e=(1)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan500

Shinichi Imai, Matsui Hideto, Akira Yamadar(Distance Measuring)or Human(Eyes)‐eController(Arduino)
Human(Hand)u

System(motor)y

Switching
Structure

Sensor (tachometer)or Human (Eyes)

4.1 ON-OFF control
Learn that accurate control cannot be done in the sensethat it uses a simple ON-OFF control to make it perfectly
match the target value. We teach that control result not
only output, but also important to observe input. The ONOFF control is expressed by the following equation.
, ( )>0(2)u(t) =
0, ( ) ≤ 0
where,
is a constant value.

Fig.5 Gain unadjusted result

Fig.6 Gain adjusted result

4.2 P control
Design a proportional controller and give an appropriate
proportional gain
to observe the output. P control isexpressed by the following equation.
u(t) =( )(3)
In the case of stable control, steady state deviation occurs
and learns that it becomes unstable whenincreases.

As shown in Fig.5, since the gain is not appropriately
adjusted, it can be seen that it is an oscillatory response.
Fig.6 shows that the response is improved because thegain can be adjusted. The results of the questionnaire onclass practice will be reported on the day of presentation
due to space limitations.
5. Conclusion

4.3 PI control
In consideration of the problem in the proportional
controller, an integral controller is introduced in addition
to the proportional controller. The PI control is expressedby the following equation.
( )+ ∫ ( )u(t) =(4)
where, is the integral gain. In this case, unlike the P
control, learn that the steady-state error can be madezero.

In this paper, we developed teaching materials that canrealize the usefulness of measurement and control. As aresult, manual control and automatic control of the motor
could be performed. For future plans, we would like to
set issues such as equipment troubles and difficulty levelsetting in practice and want to improve.
References

4.4 PID control
Learn about PID control with proportional controller,
integral controller plus differential controller. PID
control is expressed by the following equation.
( )( )+ ∫ ( ) +u(t) =(5)
where,
is the differential gain. Understanding of PID
control is deepened by finding the optimum gain while
trying trial and error on the three gains. Fig.5 and Fig.6
show the robot output results.

1. K.Asato,
K.Teruya,
T.NagadoandS.Tamaki,
Development of Magnetic Levitation System for Scienceand Technology Education: Magnetic Levitation Control
by Using a Hall Element Displacement Sensor with Neural
Network, IEEJ journal paper(D), 136(10)(2016) 744-752
2. T. Asai, K. Osuka, M. Ishikawa and M. Inoue, A
Motivational Lecture Using an Experimental Control
System to Get Personal Experience and Its EffectivenessTransactions of the Society of Instrument and Control
Engineers, 48(10) (2012) 622-631.
3. Ministry of Education, Culture, Sports, Science andTechnology: High School Study Guidelines Explained(Industrial Edition), Jikkyo Shuppan Co., Ltd., (2010)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan501

Development of Support Teaching Material for Nurturing Cooperativity through Playing
Kazuo Kawada

Department of Technology and Information Education, Hiroshima University,
1-1-1 Kagamiyama, Higashi-Hiroshima, Hiroshima 739-8524, Japan
E-mail: kawada@hiroshima-u.ac.jp
Teruyuki Tamai

Matsuyama Municipal Kamogawa Junior High School,
2-7-19 Kamogawa, Matsuyama, Ehime 791-8004, Japan
E-mail: tama.wr@gmail.com

Abstract
We have been conducted educational activities utilizing robots corresponding to each stage from kindergartens tojunior high schools. In this study, we propose and verify an educational material that supports collaborative operatingby experiences related to people and things. Specifically, three children have a joystick for operating a robot. The
robot does not move unless you manipulate the joystick at the same time. I developed the supporting teaching material
that nurture cooperativeness through play.
Keywords: Cooperation, Teaching Material, Operation, Evaluationand solve problems, and shows how to improve teaching
methods for them. For this reason, there are many
researches on learning methods2,3,4 in educational
subjects.
However,
researchthat
nurturescooperativeness which is important for character
education using a robot is a little.
Therefore, in this study, we proposed and developed
cooperative
teaching
material
andevaluatecooperativeness. Specifically, we gave children joysticks
to operate a carrier, and made the mechanism that thecarrier did not move unless it was operated at the sametime.

1. Introduction
Recently, in Japan, it is a problem that coordination
through desirable group activities is not acquired bychange of social situation. Specific problems include thefollowing:
1. Unable to build a good human relationship with others.
2. Unable to work for a group.
3. Unable to solve problems with other people bydiscussing them.
Also, it is pointed out that there are Japanese children
who can not successfully adapt to groups in kindergartensand elementary schools connection.
On the other hand, in November 2014, Japanese ministerof education, culture, sports, science and technology
consulted Central education council in relation to anemphasis on the quality and quantity of knowledge,emphasis on the quality and deepening of learning,learning subjectively and collaboratively learning to find

2. Proposed Teaching Material
The proposed teaching material is shown in Fig.1 withthe concept as follows.
3 people cooperate in maneuvering
Remote control type (joystick type)
Data recording (using a data logger)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

502

Kazuo Kawada, Teruyuki Tamai

Figure 2 shows the system configuration of this teachingmaterial. The joystick states (0V or 5V) of 3 people aresent to a micro-controller4. Then, the states of eachjoystick are stored in the data logger by the microcontroller or the data is transferred to the tablet PC by
Bluetooth.
The algorithm for operation of this teaching material isadapted to adjust the speed of the crawlers of a carrier by
the following equations.

Speed MAX MDR
3
JS1R JS 2 R JS 3 R

Speed Right

(1)

MDR

(2)

Fig.2 Schematic figure of prototype equipment.

Speed MAX MDL
Speed Left
(3)
3
MDL JS1L JS 2 L JS 3 L(4)where Speed Right ( Speed Left ) is the speed of the rightcrawler (the left crawler) of the vehicle and Speed MAX
is the maximum speed of the crawler. Further, JS1R
( JS1L ), JS 2 R ( JS 2 L ) and JS 3 R ( JS 3 L ) denotesignals on the right side (the left side) of the joysticks ofthe operator A, the operator B and the operator C,
respectively. The left and right sides of joysticks are set
to 0 when not moving, 1 when moving to the forward
side, and -1 when moving to the backward side. MDR( MDL ) denotes the sum of JS1R , JS 2 R and JS 3 R (thesum of JS1L , JS 2 L and JS 3 L ), and the signal of amajority decision is output as shown in Fig.3. With thesesignals of majority decision, the carrier can not moveunless the operator cooperatively operates.

Fig.3 Majority decision.

3. Curriculum for Nurturing cooperation
The curriculum for nurturing cooperativity usingsuggested teachings is explained. Since it is supposed topractice in elementary school classes as shown in Table
1, it is a curriculum conducted in 90 minutes class (45minutes × 2 times).
The first 10 minutes explain the objectives of lesson. The
next 20 minutes explain the rescue activities using
proposed teaching material, rescue field (Fig.4) andrescue dummy (Fig.5). At this time, students are
instructed that the operated carrier goes to the dummy
place quickly for rescue. And, the students with joysticks

Fig.1 Proposed teaching material.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan503

Development of Support Teaching

Figs 4 and 5 show the behavior of joystick operation by
the students in Case 1. If the operations of three students
match, the values of MDR and MDL appear at -3 or 0
or 3. It is clear that the students are not cooperating,because there are many times that are numerical values
other than -3 or 0 or 3.
Figs 6 and 7 show the behavior of joystick operation by
the students in Case 2. Since the operations of three
students are almost in agreement, the numerical values of
MDR and MDL appear in -3 or 0 or 3. For thesereasons, it is clear that cooperative operations are
possible by instructions from the leader. Moreover, whencooperative operations were possible, it proved that the
joystick operation of three students synchronized.

rescue the dummy without talking for 15 minutes. For
next 15 minutes, the students freely rescue dummies withtalking. For another 20 minutes, the students rescue thedummy according to instructions of the leader. Finally,
the importance of cooperativeness is summarized.

Time

Table1 Curriculum.
Contents

10[min]Learning about cooperativeness20[min]Learning about rescue activities

15[min]Rescue activities without talking

15[min]Rescue activities with talking

20[min]Rescue activities by a leader's direction

10[min]Summarize the lesson

5. Conclusions
In this paper, we proposed the teaching material tosupport cooperative development by direct experiences.
The experimental results by using proposed teachingmaterial show that when students cooperatively are able
to operate, it proved that the joystick operations
synchronized. In the future, we plan to further develop
the teaching material and curriculum for nurturing
cooperativeness.
1

Fig.4 Rescue Field.

JS_1R

-1

JS_2R
1
23
45
67
89
111
133
155
177
199
221
243
265
287
309
331
353
375
397
419
441

0

JS_3R

3
2
1

Fig.5 Rescue dummy.

MD_R
1
23
45
67
89
111
133
155
177
199
221
243
265
287
309
331
353
375
397
419
441

0

4. Experiments and Results-1

We verified the effectiveness of teaching material.
Subjects are three students of an elementary school.
Experiments were conducted in the following 2 cases.
Case 1: Three students operate the carrier without talking
Case 2: Students operate carrier according to instructions
from a leader (selected from 3 children)-2-3

Fig.4 Behaviors of the right joystick operation in case 1.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

504

Kazuo Kawada, Teruyuki Tamai

0

JS_2L
1
23
45
67
89
111
133
155
177
199
221
243
265
287
309
331
353
375
397
419
441

-1

1

JS_1L

0

JS_3L

-1

3

3

2

2

1

1

MD_L

JS_2L

0

JS_3L

MD_L
1
20
39
58
77
96
115
134
153
172
191
210
229
248
267
286
305
324
343
362
381

1
23
45
67
89
111
133
155
177
199
221
243
265
287
309
331
353
375
397
419
441

0

JS_1L
1
20
39
58
77
96
115
134
153
172
191
210
229
248
267
286
305
324
343
362
381

1

-1

-1

-2

-2

-3

-3

Fig.5 Behaviors of the left joystick operation in case 1.

Fig.7 Behaviors of the left joystick operation in case 2.

References

1

JS_1R

-1

1. A. Omae, A Study on the Significance and Problems in the
Cooperation between Preschool Programs and Eleme,Institute for Culture and Human Research, 15 (Kyoto
Bunkyo University, 2015) pp.15-32. (in Japanese)
2. T. Minoshima, T. Kiminoka, M. Nagamatsu, K. Kawada,
N. Yoshida, and T. Yamamoto, Towards theimplementation of social development in early school
education by using robotics, The Annals of Education
Research, 40 (Hiroshima University, 2012) pp.285-288.
(in Japanese).
3. K. Kawada, M. Nagamatsu, and T. Yamamoto, An
Approach to Rescue Robot Workshops for Kindergarten
and Primary School Children, J. Robotics andMechatronics, 25(3) (Fuji Technology Press, 2013),
pp.521–528.
4. T. Yamashiro, K. Kawada, M. Nagamatsu, and T.
Yamamoto, A Practice of ‘Monozukuri’ Education
Featuring “Rescue Robots Production” in an Elementary
School, Trans. on JSME, 77(776) (2011), pp.1465-1476,(in Japanese).
5. M. Banzi, and M. Shiloh, Getting Started with Arduino,
3rd Edition, (Maker Media, 2014).

JS_2R
1
20
39
58
77
96
115
134
153
172
191
210
229
248
267
286
305
324
343
362
381

0

JS_3R

3
2
1
MD_R
1
20
39
58
77
96
115
134
153
172
191
210
229
248
267
286
305
324
343
362
381

0
-1
-2
-3

Fig.6 Behaviors of the right joystick operation in case 2.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan505

On methods for teaching in training for keeping tempo constant in music*
Hideyuki Tanaka †Graduate School of Education, Hiroshima University,
Kagamiyama 1-1-1, Higashi-hiroshima, Hiroshima, 739-8524, Japan
Keita Ueda‡

Department of Education, Hiroshima University,
Kagamiyama 1-1-1, Higashi-hiroshima, Hiroshima 739-8524, Japan
E-mail: tanakalpha@hiroshima-u.ac.jp†

Abstract
Ability of keeping rhythm is one of important factors in playing music. In this paper, the authors carry out experimentsof training for keeping steady tempo in music. They compare the effects of visual and verbal feedbacks for informing
the performance. Participants include experts and non-experts.
Keywords: Rhythm in music, ability of keeping steady tempo, visual feedback, verbal feedback

1.

teachers who are worrying about not knowing systematic
methods for teaching rhythms.
There have been researches4,5 of objectively evaluating
the performance of the rhythm in music. In Ref. 4, effects
of a feedback of information of music play4 has beenstudied, and the performance of playing the musicalinstrument has been visualized and used for feedbacks5.
Several researches have focused on the position of
striking the drum pad6,7,8. However,researches on
feedbacks for keeping tempo have not been done enough.
Though Ref. 7 has given players feedbacks by graphs,
languages, and numerical values, and it has not beenclarified which feedback is effective.
In this research, we carry out experiments of training
for keeping steady tempo in music. We compare theeffects of visual and verbal feedbacks and analyze theresults dividing participants into groups of experts andnon-experts.

Introduction

In music, rhythm is one of the important factors, andkeeping steady tempo is the basis for playing music.
Rhythm indeed plays a fundamental role in music, andthere is no music without rhythm1. Music is an art basedon a time factor, and rhythm forms a basic element inmusic rather than melodies and harmonies2.
Though rhythm is one of the most important factors, it
is not necessarily easy even for music teachers to give
instructions of rhythms to students in public schools. In
Ref. 3, researchers stated that capabilities of students fortapping rhythms depend on their interests in music, andthat the gap between students being good at or bad attapping rhythm is getting wider. They moreover reported
that there are many students who cannot correctly tap therhythm in singing difficult songs in junior high and highschool, and they further informed that there are music© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan506

Hideyuki Tanaka, Keita Ueda

interpolation because of missing data. The constant line 1 is thedesignated period; the participants tap the rhythm better, if all
the periods are close to the designated one.

2. Experimental setup and feedbacks
2.1. Experimental setup
We show a schematic picture of the experimental setup
in Fig. 1. The participants tap the rhythm, beating thedram pad and holding a stick by their dominant hand. The
microphone captures the sound of the beat and send thesignal to the laptop PC of the windows OS. The PC
analyzes the signal and obtain the time of the impacts byremoving reverberation and provides information for thevisual and verbal feedbacks. In an offline setup, thevisual feedback or the verbal feedback is given by thedisplay or by the language.

We inform participants how to read the graph and theycan visually understand how s/he tapped the rhythm after
the experiment.
2.3. Verbal feedback for informing performance
In order to obtain objective data, we give a feedback
based on numerical values calculated by the laptop
computer. To this end, we define evaluation criteria toautomatically judge the tendency of the beats by thecomputer. Dividing the experiment into the first half andthe second half, we measure the periods and analyze
whether s/he keeps the steady rhythm and taps the rhythm
in a designated tempo. We moreover observe average
tempos of the first half and the second half and computethe difference between them.
Let ( ) be the -th period, where = 1,2,3, ⋯ , .
The numberof the first half data is given by=/2 for even numberor= ( + 1)/2 for odd
number , and the numberof the second half is= − . To judge whether the rhythm is steady,we define the following criteria respectively for the firsthalf and the second half:

Fig. 1: A schematic picture of the experimental setup.

We assign the speed of the beat as 60 beats per minutes,
and the participants try to keep the designated tempo in
two minutes. Before the experiments, we ask theparticipants to maintain the tempo as close to thedesignated one as possible. We then carry out 10 timesof trials for each participants and give a visual or verbal
feedback after each trial in an offline setting.

max (| ( )|) −

=

, ,⋯,

=

max

,⋯,

min

, ,⋯,

(| ( )|) −min(| ( )|),,⋯,(| ( )|),

where ( ) is the error between the designated period
and the -th period ( ):( ) = − ( ).

2.2. Visual feedback for informing performance

The root mean squares (RMS) of errors between thedesignated period and ( ) for the first and second half
are respectively defined as

Fig. 2 shows a graph used for visually informing theperiods between impacts.

=

( ) ,

=

( ),

=

( ) .

The average periods of the first and the second half arerespectively given by=

Fig. 2: The graph showing the tempo and time. The horizontal
and vertical axes respectively express time and the periods
between impacts. The red line shows the result of theperformance, where black crosses are estimated from

1

1

( ),

and the absolute value of the difference between them by
where

is defined as

=

=|

−

|,

.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan507

On methods for teaching

We moreover compute the mean μ and the variancethe data ( ) as follows:

=

1

( ),

=

1

of

listening to the signal or beating the drum pad according
to the signal. They then try to maintain the tempo in 120
seconds. After the trial, we give a visual or verbal
feedback to the participants in the groups of under
feedbacks. We carry out 10 trials and give them 5minutes rests after the 4th and 7th trials, by taking
physical and mental fatigues of participants into account.

( ( )− ) .

We give the verbal feedbacks to the participants for the
first and second half as follows:
Good in the first (or second) half: Tapping the< 0.15 andrhythm in the designated tempo, if< 0.1 hold (or if< 0.15 and < 0.1 hold).
Fairly good in the first (or second) half: Tapping
the steady rhythm in the designated tempo but not< 0.15keeping it in the designated tempo, if
holds (or if< 0.15 holds).
Not good in the first (or second) half: Tapping therhythm not in the designated tempo, if> 0.15holds (or if> 0.15 holds).
Concerning the average speeds for the first and second
half, we moreover give the following feedbacks:
Slow in the first (or second) half: The beat is
slower than the designated tempo, if< 1.05holds (or if< 1.05 holds).
Fast in the first (or second) half: The beat is fasterthan the designated tempo, if< 1.05 holds (orif< 1.05 holds).
Comparing the speeds of the first and second half, we
further give the following feedbacks:
Getting slower: The tempo of the second half isslower than the first one, if< 0 and> 0.05.
Getting faster: The tempo of the second half isslower than the first one, if> 0 and> 0.05.

3.

Results of experiments

We first compare the abilities of experienced and nonexperienced participants of tapping the rhythm as thedesignated tempo, focusing on the evaluation values
and. The results indicated that the error ofexperienced participants between the designated tempo
and the tapped rhythm is smaller than that of nonexperienced participants in all the groups of the visual,verbal and no feedbacks.
We next compare the effects of visual, verbal, and no
feedbacks. Figs. 3 and 4 show the RMS and of theerror ( ) respectively for experienced and nonexperienced participants. These figures show that theparticipants under the visual or verbal feedback (F.B.)
can maintain the error smaller than those under nofeedback. The verbal feedback is more effective than thevisual one for the performance of the experienced
participants.

2.4. Experiments

The number of participants is 45, including 27 persons
who have enough experiences of playing music and 18
persons who have not. We divide them into theexperienced and non-experienced, by asking themwhether they have experiences of playing the instruments
not in elementary, junior high or high schools. We
moreover divide them 3 groups of giving a visualfeedback (9 from 27 experienced and 6 from 18 nonexperienced, a verbal feedback (9 from 27 experienced,
6 from non-experienced), and no-feedback (9 from 27
experienced, 6 from 18 non-experienced).
The participant first listen to the signal of 60 BPM (beatper minute) for the reference, where they can choose justFig. 3: The RMS

Fig. 4: RMS

and

and

of experienced participants.

of non-experienced participants.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, Beppu International Convention Center, Oita, Japan508

Hideyuki Tanaka, Keita Ueda

We further compare the effects of visual, verbal, and no
feedbacks by using the variance . Figs. 5 and 6 showof experts and non-experts, respectively.

confused by the difference between their own feeling ofrhythm and the feedback indicated by the graph. Ref. 9
indeed mentions that there are people who have their own
feeling of rhythms. For further researches, it should be
noted that the visual feedback may improve results if we
simplify the graph.
4.

Fig. 5: Variances

Conclusions

We have carried out experiments of training for keeping
steady tempo in music. We compared the effects of visualand verbal feedbacks for informing and improving theirperformance. The obtained results indicated that thevisual feedback by a graph seems to give too muchinformation to the experienced participants, and theinformation possibly disturbed expert participants inmaintaining the steady tempo, because they have theirown feeling of rhythms.

of experienced participants.

References
Fig. 6: Variances

1. Y. Akutagawa, Basics of music (in Japanese) (Iwanami
Shoten Publishers, 1971).
2. T. Umemoto, Music psychology (in Japanese) (Seishin
Shobo, Publishers, 1986).
3. N. Mizuno, H. Ando, and M. Yoshida, Developmental
musical beat stability of children (in Japanese), in bulletinof Gifu Women’s University (2015), pp. 53–61.
4. T. Yonekawa and A. Nishikata, Rhythm pattern accuracy
diagnosis system capable of objective evaluation andcommentary feedback, IEICE Transactions on
Information and System, E86-D(1) (2003), 71-78.
5. Y. Nishii and I. Kurimoto, Development for Detacted
system of error in the sense of hearing (in Japanese), IPSJSIG Technical Reports, 93(32) (1993), 17-23.
6. Y. Tsuji and A. Nishikata, Improvement of beginners'
drumming play by expert's teaching for drumming form,in Proc. of the 2004 IEICE General Conference, (2004),
p157.
7. Y. Tsuji and A. Nishikata, Analysis of drumming playbased on parallel measurement of rhythm and drumming
form (in Japanese), The transactions of the Institute ofElectronics, Information and Communication Engineers.
D-I, J88-D-I(1) (2005), pp.99-107.
8. Y. Tsuji and A. Nishikata, Development and evaluation of
drum learning support system based on rhythm anddrumming form, (in Japanese), The transactions of theInstitute of Electronics, Information and Communication
Engineers, D-I, J88-D-I(2) (2005), pp.508-516.
9. U. Ichikawa, This is the point for improving in playing thedrum (in Japanese), Ongaku no tomo sha, Corp.
Publishers, (1988).

of non-experienced participants.

We see from Figs 3. and 5 that experienced participants
can suppress RMS of the error ( ) using visualfeedbacks, but not the variance . The result implies
that they can recognize the error in average (in the senseof RMS) and that they can accelerate or decelerate bylooking at the graph, if they beat fast or slow respectively.
However, the result indicates that they cannot properly
control the variance of periods, by looking at the graph.
On the other hand, the verbal feedback, often used formusic plays, gives better results than the visual one.
Regarding the variance in the group of the visualfeedback, we have obtained information throughquestionnaire, by asking how they thought of the
feedback between trials; they answered: “To achieve
better performance, I don't worry about the details.”, “I
tapped the rhythm without thinking too much.”, and “I
just cared the intervals of the reference signals between
trials.” Several experienced participants thus consideredthat tapping the rhythm subconsciously leads to better
performances.
Summarizing the above results, the visual feedback by
the graph possibly provides too much information for theexperienced participants to maintain the variance of therhythm, whereas the verbal feedback does appropriate
information to them. Experienced participants are thus

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

509

Reinforcement Learning as a Theoretical Framework for Education*
Masayasu Nagamatsu†, Yuki Moriguchi

Fuculty of Education, Hiroshima University, 1-1-1 Kagamiyama
Higashihiroshima, Japan‡
E-mail: nagamatu@hiroshima-u.ac.jp, m176972@hiroshima-u.ac.jp
www.hiroshima-u.ac.jp

Abstract
Reinforcement Learning as a computational model of human learning, has recenttly become the subject ofintensive investigation(Barto,A.G.,2005;Vigortio,C.M.,2010). Recent findings include: neural signature ofhierarchical reinfocement learning, and multiple roles of reward signals in the human brain. These findings not
only provide further theoretical evidences that complementary for traditional statistical inference in educational
research, but also provide framework for model based design of educational processes. Types of evidence foreducation, and its relation to recent findings of human reinforcement learnings are discussed.
Keywords: reinforcement learning, reward signal, dopamine, intrinsic motivation, evidence, educational research.

researchers suggest12 that these topics are “ripe forattack by young synthesizers and theoreticians”. The
purpose of this study is to summarise these findings
relating to human experiential learning and aiming toserve them as theoretical evidence for educational
research.

1. Introduction
Researchers in many field have been attracted1,2 tothe study of reinforcement learning because of its
simplicity and high flexibility of adapting to newenvironment. The model has been successfully applied
to the learning of aritificial3,4 and natural5,6 systems, orboth7. But when we try to model the learning process ofhumans, a problem arise. Bananas, juice can be good
rewards for mammals such as monkeys. But for humans,
these tangible rewards are ralatively cheap to the humanvalue systems, and might bring negative side effects8-11
to the long time behavior.
Recent studies have focused on these higer orderprocessing of reward, enabling us to model the key
components of experiential learning for human. Some

2. Educational practice and neural science : abridge (still) too far?
In 1997, a paper titled “Education and the brain: abridge too far” has been publised and repeatedly citedsince then. The author of the paper suggested thepossible bridges are; (1) a bridge between educational
practice and cognitive psychology. This seems wellestablished. (2) a second bridge between cognitive
psychology and brain science. This model is feasible*‡© The 2015 International Conference on Artificial Life and Robotics (ICAROB 2015), Jan. 10-12, Oita, Japan510

Masayasu Nagamatsu, Yuki Moriguchi

Beliefs, by contrast, are relatively static (at least interms of their core applications —see thediscussion of 'unboundedness' below). When
beliefs change, it is more likely to be a matter of aconversion or gestalt shift than the result ofargumentation or a marshalling of evidence.
(Ref.18)
This observation is not only suggestive for teacher
education, but also suggestive for modeling humanreward processing.
For example, a teacher may believe that students
who fail are simply lazy; another teacher may
believe that learning math is a function of drilling.
(Ref. 19)
Such perception difficult to alter, because theseperception is not a result of conscious processing, butrather acquired from teacher’s experience.

because cognitive science has brought insight into theprocess of knowledge acquisition.
The remaining missing link is related to motivation,or emotional aspects, because motivations are not themain issue of cognitive science. For this reason, theseapproaches are sometimes called “cold” approaches13-14.
Things are drastically changing by two reasons. First,
computational method such as reinforcement learningand deep learning, both of them are biologicallyinspired or having some biological counterpart.
2.1. Conjoint reward processing
According to the comprehensive review by Schultz(Ref.15, p.889), reward processings are done by two
functionally different components. The firtst part isrelated to explicit reward signal. Second part is related
to conjoint reward signal, and processed largely at
cortex (dorsolateral prefrontal cortex and parietal
association cortex). Such conjoint reward signal is a
key to understand human reward processing.
Educational research such as Refs. 13-14 shows thatreward orientation and motivational beliefs plays
important role in educational practices.

3. Type of evidences in educational practiceIn educational studies, the types of evidence has long
been the matter of debate. One of the core issue isrelated to quantitative vs. qualitative evidence. The
above findings strongly indicate that even thequatitative evidence deeply relay on subjective
percetion of teachers and students. Recent studies in theprocessing of reward will afford the2.2. Subjective reward perception
Schultz(Ref.15) also explained about subjective
reward perception16. “Neuronal reward responces may
depend on the subjective perception of rewardpredicting stimli.” The fact that reward depends on
individual perception is big empirical finding. Ref. 17
Shows top 20 priciples for enhancing teaching / learning.
The first principle is as follows.
Principle 1: Students’ Beliefs or Perceptions aboutIntelligence and Ability Affect Their Cognitive
Functioning and Learning Student knowledge
about intelligence itself can vary and affect how
students process.
This imply that how student percieve their achivement
is sujective, theirefore depend on individual differenceof perception.
In the study of teacher education, similar problem ofperception plays essential role. Refs. 18-19 shows thatteacher’s beliefs about student motivation difficult toalter.

4. Future directions ?
Recent advances in the study of reward processing
has enabled us to model the key issues in educational
practice, likewise the empirical finding about the studies
of student belief and teacher belief shows that theseprocessings will play key roles in educational practice.

References
1. M Schlüter, A Baeza, G Dressler, K Frank, et al. A
framework for mapping and comparing behaviouraltheories in models of social-ecological systems
Ecological Economics, 131 (2017) 21–35.
2. Giuseppe Carleo and Matthias Troyer Solving thequantum many-body problem with artificial neuralnetworks, Science 355, 602–606 (2017)
3. Jürgen Schmidhuber: Deep learning in neural networks:
An overview, Neural Networks 61 (2015) 85–117© The 2015 International Conference on Artificial Life and Robotics (ICAROB 2015), Jan. 10-12, Oita, Japan

511

Reinforcement Learning as a

Psychology’s Top 20 Principles for Enhancing Teachingand Learning, Mind, Brain and Education 10(1) 55-67.
18. Pajares, M. F. (1992) Teachers‘ beliefs and educational
research: Cleaning up a messy construct, Review of
Educational Research 62, 307-332.
19. Nespor, J.(1987) The role of beliefs in the practice ofteaching, Journal of Curriculum Studies 19, 317-328.

4. Busoniu, Lucian, Babuska, Robert, De Schutter, Bart: A
comprehensive survey of multiagent reinforcement
learning, IEEE Transactions on Systems Man and
Cybanetics Part C-Applications and Reviews, 38(2)156172
5. Gershman, S.J.; Daw, N.D. Reinforcement Learning andEpisodic Memory in Humans and Animals: An
Integrative Framework, Annual Review of Psychology 68,
101-128(2017)
6. Sonuga-Barke, E.J.S., et al. Annual Research Review:
Transdiagnostic neuroscience of child and adolescentmental disorders - differentiating decision making in
attention-deficit/hyperactivity disorder, conduct disorder,
depression, and anxiety, Journal of Child Psychology andPsychiatry 57(3) 321-349 (2016)
7. Todd M. Gureckis and Bradley C. Love: Computational
Reinforcement Learning, In Oxford handbook ofcompuratiornal and mathematical psychology, pp.99-117,
Oxford University Press(2015)
8. Lepper, M.R., Henderlong, J, and Gingras, I. (1999)
Understanding the effects of extrinsic rewards on
intrinsic motivation - Uses and abuses of meta-analysis:
Comment on Deci, Koestner, and Ryan (1999),
Psychological Bulletin 125(6) 669-976.
9. Eisenberger, R. and Cameron, J. (1996) Detrimental
effects of reward - Reality or myth?, American
Psychologist 51(11) 1153-1166.
10. R Eisenberger, J Cameron (1998) Reward, Intrinsic
Interest, and Creativity: New Findings, American
Psychologist 53(6) 676-679.
11. Schmid, B & Adams, J (2008) Motivation in ProjectManagement: The Project Manager's Perspective, PrectManagement Journal 39(2) 60-71.
12. Montague,
PR.,
Hyman,SE;
Cohen,JD.(2004)
Computational roles for dopamine in behaviour, Nature
431, 760-767
13. Pintrich, P.R, Marx,R.W., Boyle, R.A.(1993) Beyond
Cold Conceptual Change: The Role of Mol Beliefs andClassroom Contextual Factors in the Process ofConceptual Change, Review of Educational Research,
63(2),167-199.
14. Wolters, C.A, YU, S.L, and Pintrich,P.R(1999)The
Relation Between Goal Orientation and Student’s
Motivational Beliefs and Self-regulated Learning,
Learning and Individual Differences, 8(3),211-238.
15. Schultz, W.(2015) Neuronal Reward and Decision
Signals: From Theories to Data, Physiological Review,
853-951.
16. De Lafuente O, Romo, R.(2011) Dopamine neurons code
subjective sensory experience and uncertainty ofperceptual decisions, Proc. Natl Acad Sci USA 49 1976719771.
17. Lucariello, J.M., et al.(2016) Science Supports
Education: The Behavioral Research Base for© The 2015 International Conference on Artificial Life and Robotics (ICAROB 2015), Jan. 10-12, Oita, Japan512

Negative Test Case Generation from an Extended Place/Transition Net-Based Mutants
Tomohiko Takagi

Faculty of Engineering, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
Tetsuro Katayama

Institute of Education and Research for Engineering, University of Miyazaki
1-1 Gakuen-kibanadai nishi, Miyazaki-shi, Miyazaki 889-2192, Japan
E-mail: takagi@eng.kagawa-u.ac.jp, kat@cs.miyazaki-u.ac.jp

Abstract
In negative testing, the large state space and feasibility problems cause the difficulty of generating negative test cases
from EPN (Extended Place/transition Net)-based mutants. This paper shows the overview of EPN-based mutants,
and the details of our negative test case generation technique. In the technique using ACO (Ant Colony Optimization),ants heuristically search better paths (i.e., negative test cases) to find foods (i.e., intended failures) on a field (i.e., an
EPN-based mutant).
Keywords: Software Testing, Negative Testing, Model-Based Testing, Test Case Generation

3, discussion in section 4, and, conclusion and future
work in section 5.

1. Introduction
EPN (Extended Place/transition Net)-based mutants areformal behavioral models of software that containintended failures.1 In negative testing, well-selected
EPN-based mutants are used to generate negative test
cases to confirm that software does not include serious or
possible failures. However, the large state space andfeasibility problems on EPNs cause the difficulty ofgenerating the negative test cases from EPN-based
mutants.
The aim of this research is to construct a technique inwhich effective negative test cases to reach intended
failures are generated from EPN-based mutants. In thistechnique using ACO (Ant Colony Optimization), ants
heuristically search better paths (that is, negative test
cases) to find foods (that is, intended failures) on a field(that is, an EPN-based mutant). This paper shows theoverview of EPN-based mutants in section 2, the detailsof our negative test case generation technique in section

2. Overview of EPN-Based Mutants
This section shows the overview of EPN-based mutants.
An EPN is a PN (Place/transition Net) that is
extended by VDM++ (one of specification description
languages that are used in VDM)2 in order to formallydefine actions and guards on transitions, and it is suitableto represent the essential behavior of complex software.
A simple example of an EPN is given in Fig. 1. This
example has three places (p1, p2, and p3) and fourtransitions (t1, t2, t3, and t4), and some transitions haveguards and actions. For example, t3 has "v>=p" as a guardand "v:=v−p;" as an action. v is an instance variable ofnat type that was initialized to 0, and p is an input
parameter (an argument) of nat type for t3. When p2
contains one or more tokens and also the guard of t3 issatisfied, the firing of t3 can be done with the executionof its action. Thus guards cause the feasibility problem© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan513

Tomohiko Takagi, Tetsuro Katayama

when specific transitions need to be selected to construct
good test cases. Additionally, a state is expressed as a setof a marking (an array of the number of tokens on places)and values of instance variables, and therefore theintroduction of instance variables accelerates the large
state space problem.
EPN-based mutants (hereinafter, referred to as
mutant models) are EPNs that contain intended failuresof software, and they can be constructed by applying one
or more existing mutation operators to an EPN that
represents the expected behavior of software (hereinafter,
referred to as an original model). Fig. 2 shows a simple
example of a mutant model that was constructed by anarc insertion operator.

t1 (nat p) [ ] / v:=v+p;
p1

t4

p2

t2

instance variables
public v : nat := 0;p3t3 (nat p) [v>=p] / v:=v-p;

Fig. 1. Simple example of an EPN.

t1 (nat p) [ ] / v:=v+p;p1

p2

3. Negative Test Case Generation TechniqueA negative test case in this study is a sequence of
successive markings and transitions that starts from an
initial marking and ends with an occurrence of an
intended failure. It can be created based on a mutant
model that were discussed in the previous section. For
example, the mutant model shown in Fig. 2 can produce
a negative test case "{[0,2,0], v=0} → t1(2) → {[1,1,0],v=2} → t3(1) → {[1,1,1], v=1}". The last marking of thisexample negative test case originally should be [1,0,1],
and it succeeds in reaching the intended failure. Effective
negative test cases are created as shorter sequences toreach intended failures, so as to be able to be executed bytest engineers within a limited period of time. However,the large state space and feasibility problems that werediscussed in the previous section cause the difficulty of
constructing such negative test cases.
In this section, we propose an effective negative test
case generation technique using ACO. ACO is one ofmeta-heuristic algorithms, and it is suitable to solve
complex and time-consuming problems. In this technique,ants heuristically search better paths (that is, shorter
negative test cases) to find foods (that is, intended
failures) on a field (that is, a mutant model).
This technique consists of the following six steps.
(i) a ants are created (a is a natural number), and they
are placed onto an initial marking of a mutant model.
Each ant remembers the initial values of instancevariables of the mutant model.
For example, when the mutant model shown in Fig.
2 is given to this technique, ants are placed onto
[0,2,0], and they remember "v=0".

t2

t4

instance variables
public v : nat := 0;

p3changedt3 (nat p) [v>=p] / v:=v-p;

Fig. 2. Simple example of EPN-based mutants.

(ii) Each ant starts searching a path as follows.
First, an ant finds all feasible transitions on its
current state (that is, a marking that the ant currently
stays on, and values of instance variables that the ant
remembers). If the ants find a new 3-tuple (frommarking, transition, to-marking), the pheromone
value of the 3-tuple is initialized to an initialpheromone value τ0 (τ0 is 0 or a positive real number).
The ant then selects a feasible transition at random
in proportion to the pheromone values of all feasible
3-tuples, and goes to a next marking. If the selectedtransition has actions, the ant executes the actions
and updates the values of instance variables that the
ant remembers.
For example, in Fig. 2, an ant on the state {[1,1,0],v=2} finds feasible transitions t1, t2 and t3. When thepheromone values of the 3-tuples ([1,1,0], t1, [2,0,0]),([1,1,0], t2, [0,2,0]) and ([1,1,0], t3, [1,1,1]) are τ1, τ2and τ3, respectively, the probabilities of selecting t1,
t2 and t3 are τ1/(τ1+τ2+τ3), τ2/(τ1+τ2+τ3) andτ3/(τ1+τ2+τ3), respectively.
The ant repeats the above-mentioned selection of atransition, until an intended failure has appeared or
the length of a path from the initial marking (that is,
the number of transitions that the ant has selected)
has reached l (l is a natural number) or the ant hasreached a final marking (the definition of a final
marking is optional). In this technique, an intended© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

514

Negative Test Case Generation

failure appears if there are differences on markings,
values of instance variables, and feasible transitions
between the mutant model and its original model.
(iii) If there are ants that have succeeded in reaching an
intended failure (hereinafter, referred to as effectiveants), an ant that has found the shortest path among
the effective ants (hereinafter, referred to as an
iteration-best ant) is found. If the path of theiteration-best ant is shorter than the path of a bestso-far ant (that is, an effective ant that has found theshortest path through all the previous iterations inthis technique), the iteration-best ant becomes a new
best-so-far ant, and it is kept as a candidate solution.
(iv) If the number of iterations of this technique reaches
i (i is a natural number), the path of the best-so-far
ant is determined as a final solution (that is, the best
negative test case on the mutant model), but if a bestso-far ant does not exist (that is, if there are noeffective ants through all iterations), it is concluded
that the mutant model cannot produce a negative testcase.
(v) The pheromone values of all 3-tuples that have beenfound are updated by the following equation.

this section, we discuss the effectiveness of thistechnique based on its experimental results.
The examples are based on an EPN of an electronicmoney charging system that was introduced to discuss apositive testing technique in Ref. 3. That is, four mutant
models were manually constructed by applying fourkinds of mutation operators to the EPN. The mutation
operators used in this experiment were two kinds ofmodel-based mutation operators (arc insertion and arc
omission) and two kinds of code-based mutation
operators (numeric constant replacement and inequality
sign replacement), and they were applied to different
transitions of the EPN.
Subsequently, we applied the prototype of the tool toeach mutant model, and successfully got a short negative
test case from each mutant model. The processing time
per one mutant model was about 80 seconds. The
environment of this experiment was a laptop computerwith i7-4650U processor (1.70 GHz, up to 3.30 GHz) and8 GB RAM. The parameter values of this technique is
a=2, τ0=0.1, l=20, i=50, and ρ=0.03.
The processing time will be acceptable to most of test
engineers. However, the performance of this techniquedepends on the structure of a mutant model,
characteristics of an intended failure, and parametervalues of this technique. Thus further experiments are
expected in future study. Test engineers need todetermine the parameter values for this technique by
using a trial-and-error method, but it may not be easy.
Additionally, the quality of negative test cases depends
also on the quality of mutant models, which should be
discussed in another study.

aτt (x +1) = (1 − ρ)⋅τt (x) +

∆τkt

(1)

k=1

In Eq. (1), τt(x) expresses a pheromone value of a 3tuple t in xth iteration. ρ is a positive real number thatis less than 1.0, and it means an evaporation rate ofpheromone. ∆τkt expresses a pheromone value thatan ant k adds onto a 3-tuple t, and it is given by thefollowing equation.
1, if the ant k is effective and t∈Pk
∆τkt = Lk
0 , otherwise(2)

5. Conclusion and Future WorkIn Eq. (2), Pk is a path that has been found by an ant
k, and Lk is the length of Pk (that is, the number oftransitions of Pk).
(vi) All the ants are removed from the markings of themutant model, and this technique returns to step (i).
This technique has five parameters (a, τ0, l, i, and ρ)to adjust its performance. There are no parameter values
that are appropriate for all software development projects,
and therefore test engineers need to determine them byusing a trial-and-error method.

In this paper, we proposed a new technique to generate
an effective negative test case from an EPN-based mutant.
The negative test case generation from an EPN-based
mutant is a complex problem, since guards and instancevariables in an EPN-based mutant cause the feasibility
problem and the large state space problem, respectively.
Therefore, ACO that is one of meta-heuristic algorithms
was introduced into this technique. In this technique, ants
heuristically search better paths (that is, shorter negative
test cases) to find foods (that is, intended failures) on afield (that is, an EPN-based mutant). We developed aprototype of a tool that supports this technique, andapplied it to examples on a trial basis. We found that short

4. Discussion
We developed a prototype of a tool that supports thistechnique, and applied it to examples on a trial basis. In© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan515

Tomohiko Takagi, Tetsuro Katayama

negative test cases could be generated within areasonable time. However, the performance of thistechnique depends on several conditions, and further
experiments are expected in future study. Also, it may not
be easy for test engineers to determine the parametervalues for this technique.
There is still room to improve the performance of thistechnique. For example, we plan to add a pheromone
value to each sequence of successive 3-tuples, in order toenable ants to search paths effectively. Subsequently, we
will apply the improved technique to non-trivial
examples in order to evaluate its effectiveness.
Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number 26730038.
References
1. T. Takagi, S. Morimoto and T. Katayama, Development ofa Tool for Extended Place/Transition Net-Based Mutation
Testing and Its Application Example, Journal of Robotics,
Networking and Artificial Life (JRNAL), Vol.4, No.2 (Sept.
2017), pp.168-174.
2. J. Fitzgerald, P.G. Larsen, P. Mukherjee, N. Plat and M.
Verhoef, Validated Designs for Object-Oriented Systems,(Springer-Verlag London, 2005).
3. T. Takagi, A. Akagi and T. Katayama, Heuristic Test Case
Generation Technique Using Extended Place/Transition
Nets, Applied Computing and Information Technology,
Studies in Computational Intelligence, Vol.727 (2017),
pp.103-115.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

516

Development of a Mutant Generation Tool Using a Genetic Algorithm
for Extended Place/Transition Nets
Tomohiko Takagi

Faculty of Engineering, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
Shogo Morimoto

Graduate School of Engineering, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
E-mail: takagi@eng.kagawa-u.ac.jp, s17g483@stu.kagawa-u.ac.jp

Abstract
In EPN (Extended Place/transition Net)-based mutation testing, mutant EPNs need to be created by inserting intended
failures into an original EPN, and we develop a tool using a genetic algorithm to generate better mutant EPNs. In thealgorithm, a set of mutant EPNs corresponds to a chromosome, and the fitness of each chromosome is evaluatedbased on an original EPN weighted by metrics. This paper shows the algorithm, the functions of the tool, and thediscussion about its effectiveness.
Keywords: Software Testing, Mutation Testing, Model-Based Testing, Mutant Generation

In our previous study5, we developed a tool toconstruct an original EPN, construct mutant EPNs,
execute a test suite, calculate a mutation score, and so on.
However, the systematic techniques to generate higherquality mutant EPNs had not been established, and thetool provides only the way to generate mutant EPNs at
random by use of model-based mutation operators andthe way to construct mutant EPNs by manual applicationof model-based or code-based mutation operators.
To address this problem, we construct an algorithm
to generate better mutant EPNs, and develop a tool to
execute the algorithm. In this algorithm using a GA
(Genetic Algorithm), a set of mutant EPNs corresponds
to a chromosome (an individual), and the fitness of eachchromosome is evaluated based on an original EPN
weighted by metrics. Test engineers can select suitable
metrics for the evaluation, that is, can define the meaningof "quality of mutant EPNs".

1. Introduction
An EPN (Extended Place/transition Net) is aplace/transition net that includes actions and guards
written in a specification description language of VDM(Vienna Development Method),1 and it is used as aformal model to represent the relatively detailed behaviorof software. When model-based mutation testing2 isperformed based on the EPN, failures are intentionallyinserted into an original EPN (EPN that represents the
expected behavior of software) in order to create mutant
EPNs, and then a test suite (a set of test cases) to beevaluate is executed on the mutant EPNs.3-5 The ratio of
killed mutant EPNs (mutant EPNs whose failures weredetected by the test suite) to all of the created mutant
EPNs is called a mutation score, and it gives test
engineers a measurement of the quality of the test suite.
A large number of higher-quality mutant EPNs areneeded to expect the higher degree of accuracy for amutation score.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

517

Tomohiko Takagi, Shogo Morimoto

The rest of this paper is organized as follows. In
section 2, we propose an algorithm to generate bettermutant EPNs. Section 3 shows the functions of the toolin which the proposed algorithm has been implemented.
Section 4 gives the discussion about the effectiveness ofthe tool, and finally section 5 shows a conclusion andfuture work.

t1 [v<10] / v:=v+2;

class EPN

0.42, 0.21, 0.07

instance variables
public v : nat := 0;p2p1t2 [v>5] / v:=v−1;

public t2: () ==> ()t2() ==
v := v - 1
pre v > 5;

p3

public t3: () ==> ()t3() ==
v := v - 1;

0.54, 0.97, 0.22

2. Mutant EPN Generation Algorithmt3 [true] / v:=v−1;
0.33, 0.32, 0.16

In this section, we propose an algorithm to generate bettermutant EPNs.

operations
public t1: () ==> ()t1() ==
v := v + 2
pre v < 10;

end EPN

Complete VDM specifications

Fig. 1. Example of an original EPN with weights.

2.1. Overview of the way to address problems
A mutant EPN is an EPN that includes an intentional
failure, and it is created by applying one or more mutation
operators to an original EPN that test engineers hadconstructed based on specifications of SUT (Software
Under Test). The mutation operators can be broadlyclassified into model-based ones3 and code-based ones.
The combination of the mutation operators and the partsof the original EPN to be mutated brings astronomical
numbers of possible mutant EPNs, and therefore theyshould be selected based on quality. However, it is not
obvious (a) how to evaluate the quality of mutant EPNs,
and (b) how to apply mutation operators in order togenerate high-quality mutant EPNs.
In this study, we introduce weights4 into an original
EPN in order to address (a). The weights are values
between 0.0 and 1.0 that represent fault-proneness andthe impact on software reliability. They are calculated
based on metrics relating to usage distribution,complexity, and test execution history, and then are given
to each transition of an original EPN. It is assumed that agood mutant EPN includes an intentional failure relating
to a highly-weighted transition, and therefore the qualityof mutant EPNs can be evaluated based on the weights.
The weights are calculated by the following steps.
(i) Test engineers select metrics suitable for SUT, andthen gather materials for evaluation of the metrics.
(ii) Test engineers define the relationships between theoriginal EPN and the materials.
(iii) The metrics are evaluated based on the materials.
(iv) According to the results of (ii), the results of (iii) are
given to each transition of the original EPN, and thenthey are converted to values between 0.0 and 1.0.
A simple example of an original EPN with weights isshown in Fig. 1. It consists of three places (p1, p2, and p3),

three transitions (t1, t2, and t3), and eight arcs. There is atoken on p1. Each transition has a guard, an action, andthree weights. For example, t1 has "v<10" as a guard,"v:=v+2;" as an action, and "0.42, 0.21, 0.07" as weights.
Also, we introduce a GA into our mutant EPN
generation algorithm in order to address (b). The GA is awell-known metaheuristic approach that imitates natural
selection, and it is suitable to solve a problem in which it
is difficult to formulate equations and get a best solution
within reasonable time and cost. In our algorithm, a setof mutant EPNs corresponds to a chromosome, and thefitness of each chromosome is evaluated based on an
original EPN weighted by metrics. Chromosomes withhigher fitness tend to remain in next generation (and thus
tend to take part in producing offspring). Finally achromosome with highest fitness is selected as a solution.
The details of our algorithm is shown in the next section.
2.2. Algorithm
The mutant EPN generation algorithm consists of thefollowing six steps.
(i) An initial population is generated for the first
generation. The population consists of Nc (Nc≥2)
chromosomes, and a chromosome consists of Ng(Ng≥2) genes. A chromosome represents a set ofmutant EPNs (that is, a candidate solution). Ng is thenumber of genes, that is, the number of mutant EPNs
that test engineers need for the evaluation of amutation score. Each gene is generated by applying
a mutation operator to an original EPN. A mutation
operator to be used is randomly selected based on a
uniform distribution, and a part of the original EPN
to be mutated is randomly selected according to a
distribution of weights.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

518

Development of a Mutant(ii) In order to generate new chromosomes (that is,offspring), mutation and crossover is applied to theexisting chromosomes (that is, parents) in the
population of current generation. In mutation,
chromosomes are randomly selected with a mutation
probability pm (0.0≤pm≤1.0), and a gene that is
randomly selected in the selected chromosome isreplaced with a new gene that is generated by thesame way as (i). In crossover, chromosomes are
randomly selected with a crossover probability pc(0.0≤pc≤1.0) in order to make pairs, and then genes
between randomly selected cut positions areexchanged in a copy of each pair. Finally, newchromosomes are added to the population of current
generation.
(iii) Fitness of each chromosome in the population ofcurrent generation is evaluated by the following
equation.

generation, and their fitness are compared. Then one
with higher fitness is selected and is moved to the
population of next generation, and another isreturned to the population of current generation.
These are repeatedly performed until the number ofthe population of next generation reaches Nc.
(vi) Generation is changed, and then this algorithm
returns to (ii).
3. Functions of a Mutant EPN Generation Tool
This section shows the functions of our tool in which theproposed algorithm has been implemented.
The tool includes (a) a function to construct an
original EPN, (b) a function to generate mutant EPNs, (c)a function to execute a test suite on mutant EPNs andcalculate its mutation score. (a) and (c) had already been
implemented and discussed in our previous study,5 andtherefore we concentrate on the implementation of (b) inthis study. (b) includes two new sub-functions, that is, a
sub-function to calculate weights based on metrics, and a
sub-function to generate better mutant EPNs by a GAbased algorithm.

M

impi ∙evali (c)fitness(c)=

(1)

i=1

In Eq. (1), c is a chromosome to be evaluated, M(M≥1) is the number of kinds of metrics that test
engineers introduced to calculate weights, impi
expresses the degree of importance of ith metrics,
and satisfies the following.

3.1. Weight calculation sub-function
As already discussed in section 2.1, we introduce weights
into an original EPN, and our tool provides a subfunction to calculate weights based on metrics. Fig. 2
shows a screen shot of our tool (a main window) that is
executing the sub-function. Its GUI (Graphical User
Interface) consists of (A) a pane to start calculation andoutput the results, (B) a pane to select materials for theevaluation of metrics, and (C) a pane to show an original
EPN with weights.
A user of our tool, that is, a test engineer first selects
materials by using (B). (B) consists of three tab pages thatspecialize in materials relating to usage distribution,complexity, and test execution history, respectively. For
example, in Fig. 2, an access log file of Apache isselected in order to evaluate usage distribution of a Web
application to be tested. Keywords of actions that appear
in the selected access log file are automatically extracted
based on a specified log format, and a test engineer
defines the relationship between the actions and thetransitions of an original EPN.
After selecting materials, a test engineer pushes abutton to start calculation on (A), and then confirms its
results on (C). In Fig. 2, weights based on usageM

impi =1.0i=1(2)0.0≤impi ≤1.0 , (i=1, 2, ⋯, M)
evali(c) in Eq. (1) expresses the result of evaluating
ith metrics in c, and it is defined as follows.
∑Tj=1 weighti, j ∙(1 − 2 num(c, j) )evali (c)=(3)
∑Tj=1 weighti, j

In Eq. (3), T is the number of transitions of theoriginal EPN, weighti, j is a value of a weight given
by ith metrics on jth transition, and num(c, j)
expresses the number of genes of c in which jthtransition and/or its ingoing/outgoing arc(s) aremutated.
(iv) If the number of alternation of generations reaches G(G≥1), a chromosome with the highest fitness
through all generations is selected as a final solution,and this algorithm is terminated.
(v) Nc chromosomes that the initial population of next
generation consists of are selected from the
population of current generation. To be precise, twochromosomes are randomly selected based on a
uniform distribution from the population of current© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

519

Tomohiko Takagi, Shogo Morimoto

distribution and complexity are displayed in blue and red
characters, respectively. The former is the ratio of thefrequency of executing each transition, and the latter was
given by 1.0−MI/100, where MI means a maintainability
index. If there are no problems on the results, a test
engineer pushes an output button on (A) in order togenerate an xml file of an original EPN with weights.
3.2. Mutant EPN generation sub-function
Our tool also provides a sub-function to generate bettermutant EPNs by the GA-based algorithm that was
discussed in section 2.2. After completing the calculationof weights, a test engineer specifies the parameter values
for the algorithm on a dialog box of our tool, and our tool
starts mutant EPN generation. Fig. 3 shows a screen shotof our tool (a sub-window) that gives the results ofmutant EPN generation to a test engineer. Its GUI
consists of (A) a pane to show the lists of all genes of all
chromosomes in all generation, and (B) a pane to show amutant EPN.
On (A), a test engineer selects an arbitrary generationfrom the upper left list, and all chromosomes with fitness
in the selected generation are shown in the lower left list.
Additionally, a test engineer selects an arbitrary
chromosome from the list, and all genes of the selected
chromosome are shown in the right list. If a test engineer
selects an arbitrary gene in the list, a mutant EPN that
corresponds to the selected gene is shown in (B). A
mutated part in the mutant EPN is highlighted in orderthat a test engineer can easily understand it.
After the sub-window that shows the results ofmutant EPN generation is closed by a test engineer, ourtool shows only a final solution (that is, a chromosome
with the highest fitness through all generations) on its
main window shown in Fig. 4. Our tool allows a test
engineer to modify the final solution, if needed. The
modification function was already discussed in ourprevious study. 5

Fig. 2. Screen shot of the tool in calculating
weights based on metrics.

Fig. 3. Screen shot of the tool that shows the results ofmutant EPN generation by the GA-based algorithm.

4. Discussion
This section gives the discussion about the effectivenessof our tool shown in the previous section.
In this study, we applied our tool to an original EPNof an OFMS (Online File Management System) that was
discussed in Ref. 5. The environment to run our tool wasFig. 4. Screen shot of the tool that shows a final solution.

a personal computer with i5-6500T processor (2.50 GHz,
up to 3.10 GHz) and 4 GB RAM.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan520

Development of a Mutant

First, we developed a prototype of the OFMS basedon the original EPN. The size of the prototype is about
830 lines of JavaScript code. We then selected two
metrics in order to calculate weights for transitions of theoriginal EPN. One is RFT (the Ratio of the Frequency of
executing each Transition), and another is MI(Maintainability Index). RFT relates to usage distribution,
and an access log file of Apache was gathered as its
material. MI relates to complexity, and an output filefrom an existing source code analysis tool was gathered
as its material. We opened the materials on our tool, anddetermined the setting to execute automatic weightcalculation. The effort of these manual operations on ourtool was about 10 minutes. We then started the automaticweight calculation by our tool, and successfully got theoriginal EPN with weights. The processing time for the
calculation was about 6.5 seconds. The effort and thetime seem to be acceptable for most of test engineers.
Subsequently, we generated mutant EPNs based on
the original EPN with weights, by using our tool. The
parameter setting for the mutant EPN generation
algorithm was Nc=30, Ng=30, pm=0.5, pc=1.0, G=30,
imp1=0.5, and imp2=0.5 (imp1 and imp2 are the degree ofimportance of RFT and MI, respectively). Our tool
executed the algorithm, and successfully outputted 30
mutant EPNs as a final solution. The processing time forthe algorithm was about 2.1 seconds, and it seems to be
acceptable for most of test engineers. The average quality(that is, average fitness) of candidate solutions in the first
generation is about 0.65, and the quality of the final
solution is about 0.97. The improvement of about 0.32
was achieved through the alternation of generations by
the GA-based algorithm, and therefore our tool seems tobe effective, at least in this case. We reviewed theoutputted mutant EPNs, and found that:
mutation operators were intensively applied tohighly weighted transitions and their related
elements (that is, parts that correspond to frequently
used and/or complex functions of the prototype),
but some mutant EPNs included non-effective
intended failures that would not be important or
would not be made in actual development, and also,
there were equivalent mutant EPNs.
An additional algorithm to select appropriate
mutation operators based on the structure of an original
EPN, and an additional algorithm to detect equivalentmutant EPNs need to be developed in order to make more
improvements of the effectiveness.

5. Conclusion and Future Work
In order to improve EPN-based mutation testing, we
introduced weights into an original EPN to evaluate thequality of mutant EPNs, and then we proposed a GAbased algorithm to generate high-quality mutant EPNs
within reasonable time and cost. Moreover, we havedeveloped a tool including the functions to calculate
weights based on metrics and to execute the GA-basedalgorithm. The results of applying the tool to a non-trivial
example indicated its effectiveness, and the room forfurther improvements.
Our future work includes the development ofadditional algorithms to select appropriate mutation
operators based on the structure of an original EPN, andalso to detect equivalent mutant EPNs.
Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number 26730038.
References
1. J. Fitzgerald, P.G. Larsen, P. Mukherjee, N. Plat and M.
Verhoef, Validated Designs for Object-Oriented Systems,(Springer-Verlag London, 2005).
2. F. Belli, C.J. Budnik and E. Wong, Basic Operations for
Generating Behavioral Mutants, in Proc. of 2nd Workshop
on Mutation Analysis in conjunction with ISSRE'06 (Nov.
2006), pp.9.
3. T. Takagi, R. Takata, Z. Furukawa, F. Belli and M. Beyazıt,
Metrics for Model-Based Mutation Testing Based on
Place/Transition Nets, in Proc. of Joint Conf. of 21st Int.
Workshop on Software Measurement and 6th Int. Conf. on
Software Process and Product Measurement (IWSMMENSURA) (Nov. 2011), pp.7-10.
4. T. Takagi and T. Teramoto, Extended Mutation Score
Based on Weighted Place/Transition Nets to Evaluate Test
Suites, in Proc. of 15th Int. Conf. on Computer andInformation Science (ICIS) (June 2016), pp.959-961.
5. T. Takagi, S. Morimoto and T. Katayama, Development ofa Tool for Extended Place/Transition Net-Based Mutation
Testing and Its Application Example, Journal of Robotics,
Networking and Artificial Life (JRNAL), Vol.4, No.2 (Sept.
2017), pp.168-174.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

521

Implementation of RETUSS to Ensure Traceability
between Class Diagram in UML and Java Source Code in Real Time
Keisuke Mori*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, and Naonobu Okazaki**University of Miyazaki, 1-1 Gakuen-kibanadai nishi, Miyazaki, 889-2192, Japan†Tokyo University of Technology, 1404-1 Katakura, Hachioji, 192-0914, Japan
E-mail: mori@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kitayshr@stf.teu.ac.jp,
yamaba@cs.miyazaki-u.ac.jp, aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp

Abstract
Ensuring of the traceability of deliverables is one of effective methods to secure the quality of software. It can verify
that the requirements are reflected in the programs. But it has two problems: taking much labor and time, and causing
mistakes by human handling. This paper has implemented RETUSS (Real-time Ensure Traceability between UML
and Source-code System) in order to solve the above two problems. RETUSS can ensure traceability between Class
diagram in UML and Java source code in real time.
Keywords: Software Quality, Traceability, UML, Java

RETUSS can ensure traceability between Class diagramin UML and Java source code in real time.
UML2 (Unified Modeling Language) is a visuallanguage for expressing the design and pattern ofsoftware. It is used for requirement specification andsystem design documents3. RETUSS targets Class
diagram, which is an important diagram expressing thestatic structure of the system in UML. It also targets
source codes in Java language.

1. Introduction
It’s increasing the importance of software in society, andit’s becoming more important to secure the quality ofsoftware. Ensuring of the traceability of deliverables is
one of effective methods to secure the quality ofsoftware1. It can verify that the requirements are reflectedin the programs, specify the scope of the impact due tothe modification in the requirements, and remove the gap
between the documents and the source code. But it has
the following two problems:

2. RETUSS
Figure 1 shows an overview of RETUSS implemented inthis paper. RETUSS has the following five areas:

Taking much labor and time to modify similarly
other related deliverables by modifying some in adeliverable
Having a risk that you cannot ensure traceability
because of causing mistakes to ensure traceability byhuman handling

Menu Bar
File List Area
Drawing Item Selection Area
UML Description Area
Source Code Description Area

This paper has implemented RETUSS (Real-time
Ensure Traceability between UML and Source-code
System) in order to solve the above two problems.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan522

Keisuke Mori, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

Fig. 1. Overview of RETUSSModifying Class name and Note content
Deleting Classes and Notes
Adding Attributes and Operations in Class
Modifying the Attributes and Operations in Class
Deleting Attributes and Operations in Class
Hiding Attributes and Operations in Class
Description of Composition and Generalization
between Classes

In addition, RETUSS has the following threefunctions:
Open java a file
Description of Class diagram
Description of Java source code
A user can describe Class diagrams in the UML
Description Area in the "Description of Class diagram"function. Items and functions that can be described are as
follows:

A user can describe Java source code in the Source
Code Description Area in the "Description of Java sourcecode" function. When the user describes Java source code,
RETUSS draws Class diagram corresponding to sourcecode’s contents in UML Description Area. RETUSS

Description of Classes and Notes
Moving Classes and Notes© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

523

Implementation of RETUSS to

Fig. 2. The correspondence between the Class diagram and Java source code in executing the application example

draws multiple Classes corresponding to Java sourcecode in the Source Code Description Area. Because one
tab corresponds to one Class, a user draws Class of thecorresponding the Java source code in the text area of
each tab.

Java source code, respectively. Similarly, by looking at
the ClassDiagramDrawer class, the Class name, theRelated end name of the Composition destination,
Operation in the Class diagram correspond to the class
name, field, and method in the Java source code,
respectively.
Hence, it is clear that RETUSS can ensure
traceability between Class diagram in UML and Javasource code.

3. Application Example
As an application example, a part of Class diagram
showing the structure of RETUSS itself is adopted toverify that the functions of it are operated correctly.
Figure 1 shows a screenshot of RETUSS in executing theapplication example, and Figure 2 shows thecorrespondence between the Class diagram and Javasource code at this time.
By looking at the ClassNodeDiagram class of theClass diagram and the Java source code in Fig. 2, theClass name, the Class name of the Generalization
destination, the Attributes and Operations in the Class
diagram correspond to the class name, the class name of
the inherited destination, the field and the method in the

4. Evaluation of Usefulness
To consider RETUSS usefulness, we experiment withexperimental participants. As an experimental method,the experimental participants modify Class diagrams andJava source code the ensured traceability. Specifically,
the experimental participants modify Class diagram andthe Java source code to ensure traceability forrequirements change when the Class diagram and Javasource code already exist. As a modification method, twocases are prepared: case A using RETUSS, and case B© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan524

Keisuke Mori, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

Table 1. The time required by three experimental
participants to ensure traceability in each case(seconds).
Trials count
One
Two
Three
Average

Case A
113
76
129
106

other hand, RETUSS can ensure traceability between
Class diagrams and Java source code in real time.
Therefore, RETUSS takes less time to modify
similarly other related deliverables by modifying some ina deliverable, compared to the other two tools.

Case B
292
148
229
223

6. Conclusion
In this paper, we have implemented RETUSS, that
ensures traceability between UML and source code inreal time with the aim of solving two problems inensuring traceability.
We applied RETUSS to a part of Class diagram of thestructure of itself, and our tool ensured traceability
between Class diagram in UML and Java source code.
Finally, as a result of experiments with the experimental
participants, the time could be reduced by 52.47% toensure traceability between the Class diagram and theJava source code, and elimination causing for humanhandling by using RETUSS. Therefore, RETUSS cansolve two problems in ensuring traceability.
Consequently, RETUSS is expected to support tosecure the quality of software.
Future works are as follows:
Correspond to the same Attribute and same
Operation
Correspond to incompatible syntax in Java source
code
Extension of Class diagram description function

without using. For each case, we measure and compare
the time required for the experimental participants toensure traceability between the modified Class diagramand Java source code.
Table 1 shows the time required by threeexperimental participants to ensure traceability in eachcase.
The average time required for modifying in case A
using RETUSS was 106 seconds. On the other hand, theaverage time required for modifying in case B without
using RETUSS was 223 seconds. Case A could be
reduced by 52.7% compared to case B.
Further, in case B without using RETUSS, somemistakes were caused because of ensuring traceability byhuman handling. During the modification, for example,
mismatch between Visibility of Class and accessmodifiers of Java source code, and deletion of curly
braces of a method body of Java source code. On theother hand, the above mentioned errors did not occur incase A using RETUSS, because it can ensure traceability
automatically.
Hence, it’s possible to reduce labor and time andeliminate causing for human handling to ensure
traceability by using RETUSS.

References
1. SQuBOK Sakutei Bukai, Guide to the Software Quality
Body of Knowledge, 2nd edn. Ohmsha, 2014 (in Japanese).
2. Welcome To UML Web Site!, http://www.uml.org/,(accessed December 7).
3. Dan Pilone, Neil Pitman, UML 2.0 in a Nutshell. O’Reilly
Media, 2009.
4. UML tools for software development and modelling EnterpriseArchitect
UML
modeling
tool,
https://www.sparxsystems.com/, (accessed December 7).
5. Astah - Software Design Tools for Agile teams with UML,ER Diagram, Flowchart, Mindmap and More | Astah.net,
http://astah.net/, (accessed December 7).

5. Related Research
EA4 (Enterprise Architect) and Astah5 are modeling toolsthat describe UML. Both EA and Astah can generate
source code from UML and UML from source code
automatically. EA and Astah can ensure traceability ofsource code are language such as Java, C# and C++.
EA, Astah and RETUSS support the design phase andmaintenance phase of software development in using
UML. EA and Astah can generate skeleton Java sourcecode from Class diagram and Class diagrams from
implemented Java source code in methods automatically.
But EA and Astah cannot ensure traceability between
Class diagrams and Java source code in real time. On the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

525

Prototype of a Tool to Defect Specific Comments
Satoshi Tanoue*, Tetsuro Katayama* Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, and Naonobu Okazaki**University of Miyazaki, Japan,†Tokyo University of Technology, Japantanoue@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kitayshr@stf.teu.ac.jp,
yamaba@cs.miyazaki-u.ac.jp, aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp

Abstract
This paper has developed a prototype of a tool to detect specific comments. With the prototype, you can use regularexpressions, describe patterns of detected strings in a configuration file, and then detect specific comments. The
prototype was applied to projects with 5,000 LOC (Lines of Code) or more. As a result, the recall rate achieved 100%and the precision rate achieved 80%. Reducing improper comments and comments detected by the prototype which
express tasks remaining in the source code can improve understandability of the source code.
Keywords: comments, static analyze, maintenance, understandability.

development delay. Since, supporting the detection bytools is required.
Therefore, this paper implements a prototype of a tool
to detect specific comments which users choose toimprove the quality of the source code.
The prototype can describe the pattern of the detectedcharacter string in the configuration file by using theregular expression. The configuration file is described in
XML (Extensible Markup Language) format1. The
prototype detects comments matching the pattern of thecharacter string described in the configuration file. It
described once can be reused in other projects. As for acomment to be detected newly, it can be flexibly copedwith by adding the pattern of the detected character string
to the configuration file. By using this prototype, it ispossible to detect specific comments which users choose.

1. Introduction
Comments can be described in a natural language without
affecting the behavior of software. Therefore, using
comments can describe contents that cannot be expressedby source code. For example, TODO comments can add
tasks that exist in the source code as comments. By
describing them appropriately, they give effective
information to programmers. Therefore, comments
indicating tasks remaining in the source code, such as
TODO comments, need to be detected as necessary tocheck that issues remain in the source code.
On the other hand, there are problematic comments ifthey exist in the source code, such as repair history
comments. The repair history comments have a problem
that it is difficult to confirm that the maintenance is done,
and the number of codes that need to be read increases. In
this paper, such problematic comments are called improper
comments.
Although improper comments and comments
indicating tasks remaining in source code such as TODO
comments can be found visually, it cannot be checkedsufficiently due to lack of human resources or

2. The Implemented Prototype
The implemented prototype has been developed in Java
language. It runs on any OS if it can be installed Javavirtual machine. An overview of the prototype in thisresearch is shown in Fig 1. The developed prototype© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

526

Satoshi Tanoue, Tetsuro Katayam, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, and Naonobu Okazaki<?xml version="1.0" encoding="UTF-8" ?><comments><comment type="TODO Comments"><RegExp>TODO</RegExp><RegExp>(?i)future</RegExp></comment><comment type="Anbigious Comments "><RegExp >\?</RegExp ><RegExp >maybe</RegExp ></comment></comments>

Menu Bar

Edit Area

Display Area

Fig. 2 An example of the configuration fileFig. 1. Overview of the prototype

Table 1. Details of project A

consists of a Menu Bar, an Edit Area, and a Display Area.
The user can utilize the function of the prototype by
selecting a menu item from the menu bar.
The function of the prototype is shown below.
Changing Character CodeOpen File
Save FileAnalyze Comments
Analyze Comments of Multiple Files
Clear Edit Area
Clear Display Area
Function to Support Description of Regular
Expressions

LOC
6854

3.

Comments
1990

Comments to be detect
24

Implements

The "Analyze Comments" function of the implemented
prototype consists of extraction comments and patternmatching.
3.1. Extract comments
JavaCC (Java Compiler Compiler) 2 is adopted to extractcomments. JavaCC is often used as a parser generator thatcan use Java language. The grammar of JavaCC is similar
to EBNF (Extended Backus Naur Form)3.
By using this parser generator JavaCC, the prototype
generates a parser to extract comments from the sourcecode and extracts all comments in the source code.

2.1. Analyze comments
Comments correspond to line comments (//) and blockcomments (/ * * /). In case of programming languages inwhich the notation of line comments (//) and blockcomments (/ * * /), you can analyze comments. "Analyze
Comments" first extracts all comments existing in thesource code. Next, it matches based on the pattern of thedetected character string described in a configuration file.
When the description content of the comments ismatched in the pattern of the detected character string, it isdetected. Finally, it is necessary to visually determine
whether or not the detected comments are a detection
target comment.

3.2. Pattern matching
To do pattern matching, the prototype obtains the contentsfrom the configuration file. The configuration file isdescribed in XML format. To extract the pattern of thedetected character string from the configuration file, theprototype uses the DocumentBuilderFactory class which isa Java XML parser. It determines whether the comments
obtained by the method of section 3.1 matches thecharacter string detection pattern, and displays comments
matched with the pattern on the "Detected Comments" tab.

2.2. A configuration file
An example of the configuration file is shown in Fig 2. In
the configuration file, it is possible to describe the patternof the detected character string with the regular expressionthat can be accepted by Java language.

4. Evaluation of the Prototype
We applied the implemented prototype to two projects A
and B with LOC (Lines of Code) of 5000 lines or more

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

527

Prototype of a Tool

Table 2. Results of the application to project A.
Executions

Comments to be detect

Detected comments

Detected comments to be detected

Precision (%)

Recall (%)

1

24

32

21

65.6

87.5

2

24

30

24

80.0

100

3

24

28

23

82.1

95.8

the processing itself affects the recall rate, the pattern ofdetected the character string of the comments explaining
the processing itself was deleted from the configuration
file. This project includes one comment explaining theprocess itself. As a result, it became impossible to detect
the comment, the precision rate increased to 82.1%, but therecall rate fell to 95.8%.
Even in other projects, in order to confirm that the same
configuration file can be used, the configuration file usedin project A was applied to project B. Details of project B
are shown in Table 3. The result of applying to project B
is shown in Table 4. In the first execution, the recall rate
was 100% and the precision rate was as high as 92.3%. In
project B, the recall rate became 100% because there were
no comments explaining the processing itself. It was
confirmed that the configuration file adjusted to obtain
good results can be used as it is for other projects.

Table 3. Details of project B
LOC
9394

Comments
1584

Comments to be detect
24

developed by SKYCOM Corporation, and examined theprecision and recall rate. The calculation formula of the
precision rate is as follows.

The calculation formula of the recall rate is as follows.

As an evaluation method, the total number of comments
and comments to be detected is visually checked, and theprecision rate and the recall rate are calculated from both
the result confirmed visually and the analysis results byusing the prototype. Comments to be detected were
defined by employees in SCYCOM Corporation. Many
comments on this evaluation were written in Japanese.
First, the prototype is applied to project A for
adjustment of a configuration file. Details of project A are
shown in Table 1. The results are shown in the Table 2. In
the first application of the project A, because the pattern ofthe detected character string described in the configuration
file is insufficient, the recall rate was 87.5%, the precision
rate was 67.5%, both of which were not satisfactory values.
Therefore, a keyword indicating that it was ambiguous
which was not described in the configuration file such as
ambiguous Japanese words meaning "probably" or"maybe" were added as a pattern of the detected character
string. As a result, at the second time, the recall rate can be100%.
On the other hand, the precision rate was 80%. We
examined the cause of lowering the precision rate, andfound that there are many false detections on comments
explaining the processing itself.
Therefore, as a third time, in order to check how muchthe character string detection of the comments explaining

5. Discussion
In this section, we consider related researches andcomparison with grep command.
5.1. Related researchTechnical debt is a metaphor coined to express nonoptimal solutions that are taken in a software project toachieve some short-term goals 4. Technical debt can bedeliberately or inadvertently incurred 5. One example ofthe deliberate technical debt is self-admitted technical debt.
It can be found from source code comments. In previous
studies, the detection of self-admitted technical debt was
heavily dependent on visual observation6.
Maldonado and et al. proposed a method toautomatically detect self-admitted technical debt using
natural language processing technology 7. In Maldonado's
method, the recall rate cannot achieve 100%. This meansthat there is a leak in the comments to be detected.
Although the detected comments are different from our
method, it is difficult to make the recall rate 100% since
Maldonado's method uses machine learning. In the
prototype, the recall rate can be surely made 100% by
preparing the detected character string pattern.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

528

Satoshi Tanoue, Tetsuro Katayam, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, and Naonobu Okazaki

Table 4. Results of the application to project B.
Executions

Comments to be detect

Detected comments

Detected comments to be detect

Precision (%)

Recall (%)

1

24

26

24

92.3

100

remaining in the source code and improper comments
existing in the source code. Reducing their comments ofthe source code can reduce the problems existing in thesource code and improve understandability of the sourcecode.
The future issues are shown below.

5.2. Comparison with grep command
Even if you use grep command8 or the text search functionof a text editor, you can do the same thing as the prototype.
So, we actually quantitatively evaluated how muchdifference is made with grep command. We prepare thesame pattern as the detected character string described in
the configuration file adjusted, and apply grep commandusing the pattern of this detected character string. As aresult, recall rate is 76.7% and precision rate is 100% in
project A, precision rate is 40.0% and recall rate is 100%in project B.
In the evaluation experiment of grep command, thepattern "?" of the detected character string matched "?" ofthe ternary operator existing in the source code.
In this prototype, since all comments are first extracted
and then pattern matching is performed, the pattern of thedetected character string does not match the ternary
operators in the source code. Therefore, the precision rate
can be higher than grep command.

Improvement of precision
Detection of only the comment explaining theprocessing itself
Detection of unmaintained comments undocumented
comments
Acknowledgements
I would like to thank S. Kai and Y. Uehara of SKYCOM
Corporation for cooperating in evaluating the prototype.
References

6. Conclusion
In this paper, we have developed the prototype of a tool todetect specific comments which users choose in order toimprove the quality of the source code. With the prototype,
you can use regular expressions and describe patterns ofdetected strings in the configuration file. Thus, it ispossible to detect a specific comment which users choose.
Even when another comment to be newly detected appears,
it can flexibly cope with by adding the pattern of thedetected strings to the configuration file.
The implemented prototype was applied to projects with
LOC (Lines of Code) of 5000 lines or more, and examined
the precision and recall rate. As a result, by adjusting thepattern of detected strings, the recall rate achieved 100%and the precision rate achieved 80%. Moreover, theconfiguration file which was determined once can beapplied as it is to other projects. That is, it can be
confirmed that it can be reused. By using the prototype, it
is possible to detect comments showing user's tasks

1. W3C: Extensible Markup Language (XML) 1.0 (Fifth
Edition),
W3C(online),https://www.w3.org/TR/
2008/REC-xml-20081126/ (accessed 2017-7-29).
2. JavaCC: The Java Parser Generator, JavaCC (online),
https://javacc.org (accessed 2017-7-29).
3. for Standardization, I. O.: Information technology – Syntactic metalanguage – Extended BNF, Technical Report ISO
14977:1966(E), International Organization for Standardization (1995).
4. Cunningham, W.: The WyCash Portfolio Management
System, SIGPLAN OOPS Mess., Vol. 4, No. 2, pp. 29– 30(online), DOI: 10.1145/157710.157715 (1992).
5. Fowler, M.: Techical debt quadrant, Martin Fowler (online),
https://martinfowler.com/bliki/
TechnicalDebtQuadrant.html (accessed 2017-7-29).
6. Potdar, A. and Shihab, E.: An exploratory study on selfadmitted techical debt, 2014 IEEE International Conference on Software Maintenance and Evolution, pp. 91–
100 (online), DOI: 10.1109/ICSME.2014.31 (2014).
7. Maldonado, E., Shihab, E. and Tsantalis, N.: Us- ing Natural
Language Processing to Automatically Detect SelfAdmitted Technical Debt, IEEE Transactions on Software
Engineering, (online), DOI: 10.1109/TSE.2017.2654244(2017).
8. Foundation, F. S.: GNU Grep 3.0, Free Software Foundation(online),https://www.gnu.org/software/grep/
manual/grep.html (accessed 2017-7-29).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan529

A Recipe Decision Support System with Recognition Ability Recording Function Using
Knowledge Information and Agent
Keita Saito, Taro Asada, Yasunari Yoshitomi, Ryota Kato, Masayoshi Tabuse

Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku,Kyoto 606-8522, Japan
E-mail: {yoshitomi, tabuse}@kpu.ac.jp, {t_asada, r_kato}@mei.kpu.ac.jp}
http://www2.kpu.ac.jp/ningen/infsys/English_index.html

Abstract
Herein, we report on the development of a recipe recommender system with a recognition ability recording function
based on collaborative filtering and impression words, and which uses MikuMikuDanceAgent (MMDAgent) as ahuman interface. Our previously proposed system has been modified to facilitate recording the recognition abilityof the user. In this process, the agent poses questions to the user at appropriate times during interactions in order todetermine if there is any noticeable deterioration in his/her recognition ability.
Keywords: Recipe recommendation, Collaborative filtering, Impression word, MMDAgent, Elderly persons

in turn, is based on our previously reported music
recommendation method3. That method for recipe
recommendation will be explained briefly below.
We began by preparing a database composed ofrecipes that had been already been subjectively scored atthree-levels with “1” being the most appealing to “-1”being the least appealing. We then transformed thesubjective scores of “-1” to “0”. Those values (“0” or“1”) are expressed as “evaluation scores” below.

1. Introduction
When caring for elderly persons, it is well known that if
dementia can be identified in its early stages, appropriate
treatment can be initiated to improve symptoms and/or
delay progression. Since declining memory acuity is one of
the initial symptoms of dementia, this study reports on thedevelopment and evaluation of a new system that can help
quantify a subject’s memory and is based on ourpreviously reported recipe recommender system1. In thecurrent study, we have adopted MikuMikuDanceAgent(MMDAgent)2 which is a toolkit for building voiceinteraction systems, as a human user interface (UI).

3. Recipe recommendation method using
impression words
Our recipe recommendation method also uses
impression words1. In this study, 10 impression words
pairs (see Table 1) extracted from “Sizzle Word Report
2014”4 were used. This method is also based on ourpreviously reported music recommendation method3.

2. Recipe recommendation method using
collaborative filtering
The recipe recommendation method1 proposed in ourcurrent study is based on collaborative filtering, which,© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

530

Keita Saito, Taro Asada, Yasunari Yoshitomi, Ryota Kato, Masayoshi Tabuse

Table 1. Impression word pairs1.
Good flavor －Refreshing flavor
Mellow － Spicy
Sweet － Gentle
Addicted － not Addicted
Rich － Crispy
Juicy － Scratchy
Warm －Cool
Melty － Chewy
Seasonal Limited－ not Seasonal Limited
Authentic － not Authentic

Each of the recipes prepared in advance was
assigned one of seven score levels ranging from “-3” to“+3” by the participating test subjects who used theselected impression word pairs to make theirevaluations. We then transformed these subjectivescores to a three-level scale (“-3” to “-2” as “-1”, “-1” to“+1” as “0” and “+2” to “+3” as “+1”). These values are
expressed as “impression evaluation scores” below.
More specifically, when a recipe that has not been
recommended to a user has the same values (except for a“0”) as that for at least one recommended recipe that has ahigh user evaluation based on the three-level score for atleast five impression words, that recipe is treated as having
a positive evaluation by the user. In contrast, when a recipe
that has not been recommended to the user has the same
scores (disregarding the “0”) as that for another recipe thathas just been recommended to the user, even though it has
a negative user evaluation based on the three-level score
for at least seven impression words, that recipe is treated ashaving a negative evaluation by the user.
4.

Recipe recommendation using thecollaborative filtering and impression words

Fig. 1 shows a flowchart of the proposed system, which
combines collaborative filtering and impression words
based on the authors’ recipe recommendation method1.
First, the system recommends recipes stored in the
database to the user. Both the recommendation processusing collaborative filtering and that using impression
words are terminated when the number of recommended
recipes reaches the pre-set upper limit (K). In our
proposed system, the recommendation process using
collaborative filtering is terminated when there no users
in the user reference list that show the same exact
evaluation for the recommended recipe as have theusers up to that point. The recommendation process thencontinues by identifying the most similar recipe, fromthe viewpoints of three-level scores (disregarding “0”)Fig. 1. Recipe recommendation using collaborative filteringand impression words.

based on impression words, to that successfully
recommended among recipes not yet recommended.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

531

A Recipe Decision Support

One requests answers to three simple questions after
displaying a sentence that has been excerpted from junior orsenior high school home economics textbooks. The otherasks the user to recall some recipes recommended via themethod discussed in Section 4.
We designed our proposed system based on theassumption that it would be used by elderly persons.
Accordingly, the input method was changed from “input
by voice recognition”1 to “input by computer mouse”
because of concerns that the speech recognition
accuracy of elderly persons could deteriorate.
6. Performance evaluations
6.1. Conditions
From a total of 49 recipes that exclude side dishes, soups,
and arranged dishes, main course dishes that one or moreuser had designated as “I have never tried” were extracted
from the 293 entries in the “Rakuten recipes”5 category foruse in our evaluation. The impression words selected werefour pairs of “taste expression”, four pairs of “texture
expression”, and two pairs of “situation expression”
extracted from the “Sizzle Word Report 2014”4. Synonyms
and antonyms were verified using the Weblio dictionary6.
Similar expressions in the “expression” rankings weregrouped together and chosen via the following procedure1:

Fig. 2. Recipe recommendation considering yesterday’s dinner.

Just before concluding the recommendation process, theuser database is updated with subjective evaluations of
the recipe by adding the viewpoints of those users
recommending the recipe. In addition, the proposedsystem ultimately narrows down the recipe choices
using methods that consider yesterday’s recipe.
First, recipes in groups that are similar to the recipe
that user selected the day before are recommended.
Next, recipes in groups that differ from previous day’s
selection are also recommended. As in the impression
word recommendation, judgments regarding a recipe
whose impressions are similar to yesterday’s dinnerrecipe are based whether the “impression evaluation
score” coincides with five or more of the impression 10item word pairs. Fig. 2 shows a recipe recommendation
flowchart that considers yesterday’s dinner.

Step 1: The top 10 words satisfying the condition that
their antonyms are within the top 25 words arechosen in each ranking.
Step 2: The words that ranked within the top 10 and theirantonyms within the same ranking are chosen.
Step 3: The words ranked within the top 10 are chosen under
the restriction that their antonyms are out of ranking.
Step 4: Processing repeats for words ranking below the top 10.
In this report, we changed the scoring method
used for collaborative filtering evaluation scores from“recipe preference”1 to “whether I want to eat” based on
the idea that “the recommendation accuracy” would be
improved due to the enhanced fitting of the question andthe database.
Next, the proposed system recommended recipes
to five new participating users who were not among the
14 users who entered evaluations into the database. K
was set to 15, and experiments were conducted under
the following two conditions:5. Proposed system
In this section, we propose a system that records the user’s
recognition function while recommending recipes. To checkthe user’s recognition capabilities, we use two question types.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan532

Keita Saito, Taro Asada, Yasunari Yoshitomi, Ryota Kato, Masayoshi Tabuse

Recommendation accuracy = Number of recipes towhich the user replied, “I want to try” divided by the
number of recommended recipes.
Final recommendation accuracy = Number of users thatrecommended the recipe that the user most wants to try
divided by the number of users.
For the questions, we selected two sentences from
Ref. 7 and three from Ref. 8.
The proposed system evaluates the user’s state as
being in one of four conditions according to the
cognitive function score, which is based on a scale from0 to 20 points. If the score is 14 points or more, ajudgement of “No problem” is issued; if the score isfrom 10 points to 13 points, a judgement of “slightly
deteriorated cognitive function” is tendered; if the scoreis from six to nine points, the system determines that“there is a decrease in cognitive function”; for five pointscores or less, the system decides that there is a“possibility that the cognitive function may have been
greatly deteriorated”. Scoring itself is based on the
cognitive function test9.
Our experiment was performed on a Dell Optiplex
780 personal computer (PC) equipped with an Intel Core
2 Duo E8400 3.00GHz central processing unit (CPU) and
4.0 GB of random access memory (RAM). The Microsoft
Windows 7 Professional operating system (OS) was
installed on the PC and Microsoft Visual C++ 2010 wereused as the development languages.

Fig. 3. Accuracy recommendations of five new users.

ability in addition to memory, we would like to develop
a system that can measure those abilities as well.
7. Conclusion
In this paper, we proposed a recipe recommendation
system that combines collaborative filtering andimpression words. Our proposed system is equipped with afunction for checking the memory condition of a user using
a recommendation system outlined in our previously study1.
The tests conducted to date indicate that our proposedsystem is more effective than random recommendations,and that it is capable of providing an accurate evaluation ofa user’s recognition ability.
Acknowledgements
We would like to thank the subjects for theircooperation in the experiments.
References
1. K. Saito, T. Asada, Y. Yoshitomi, R. Kato, and M.
Tabuse, A recipe decision support system using
knowledge information and agent, J. Robotics,
Networking and Artif. Life to appear.
2. MMDAgent. http://www.mmdagent.jp/ Accessed 25 July 2016.
3. S. Yoshizaki, Y. Yoshitomi, C. Koro, T. Asada T, Music
recommendation hybrid system for improving
recognition ability using collaborative filtering andimpression words. J. Artif. Life and Robotics 18(1-2)(2013) 109-116.
4. Sizzle Word Report2014. http://www.bmft.jp/pdf/services/
kotoba14.pdf/ Accessed 25 July 2016.
5. Rakuten recipe. http://recipe.rakuten.co.jp/category/
Accessed 17 December 2017.
6. Weblio dictionary. http://thesaurus.weblio.jp/ Accessed
25 July 2016.
7. Y. Niida and K.,Kusuda, Home economics textbook forelementary school (in Japanese), (Yama-kei Publishers,
Tokyo, 2008).
8. M. Miyamoto etc. (eds.), ,Home basis - future by
partnership (in Japanese), (Jikkyo Shuppan, Tokyo,
2012).
9. Dementia Net. https://info.ninchisho.net/check/ch20/
Accessed 20 December 2017.

6.2. Results and discussions
Fig. 3 shows the recommendation accuracy of five new users.
The mean value of the proposed system’s recommendation
accuracy was 77.3%, while that of the random
recommendation was 43.7%. The value of the final
recommendation accuracy of the proposed system was 40.0%.
As a result of experiments involving five users
without memory abnormalities, we found that the
average score of cognitive function was 18.0. In our
future work, it will be necessary to conduct experiments
involving elderly test subjects and then evaluate theresults obtained. In addition, since it is possible that
healthy persons will become irritated if repeatedly asked
questions aimed at gauging their memory abilities while
researching recipes, it will be necessary to think about
ways to measure memory more naturally during useragent interactions. Furthermore, since dementia
negatively impacts computational power and language© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan533

A System for Analyzing Facial Expression and Verbal Response of a Person while Answering
Interview Questions by AgentTaro Asada1, Daichi Kogi2, Ryoichi Shimada3, Yasunari Yoshitomi1, Masayoshi Tabuse1

1: Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: {yoshitomi, tabuse}@kpu.ac.jp, t_asada@mei.kpu.ac.jp}
http://www2.kpu.ac.jp/ningen/infsys/English_index.html
2: S.Ten Nines Kyoto Co.,Ltd.
13-1 Shinmei-cho, Nishikyougoku, Ukyo-ku, Kyoto 615-0864, Japan
3: JFE Systems, Inc.
4-1-3 Taihei, Sumida-ku, Tokyo 130-0012, Japan

Abstract
We have developed a system for analyzing the facial expressions of a person obtained while answering interview
questions posed by an agent. The image signal input from a video camera that is collected while the subject is
answering questions is analyzed by our real-time system. Additionally, the fundamental frequencies and the timeto-utterance of the answerer just after a question is terminated are measured in order to help estimate theinterviewee’s mental state at the time the answer is given.
Keywords: Facial expression analysis, Movement analysis, Mouth area, Interview on video, MMDAgent.

affected by (1) the conversation topic, (2) the
conversation partner, and (3) the facial expression of thepartner. Accordingly, in order to fix all of (1), (2), and(3), we previously developed a system for analyzing thefacial expressions of a person while answering theinterview questions on video.3
In the present study, in order to improve system
convenience, such as by allowing easy changes to theinterview questions, we developed a system capable ofanalyzing the facial expressions of a person in real-time
while he or she is answering interview questions posedby an agent. In this system, the moving image in our
method is captured by a webcam and analyzed in realtime via Open Source Computer Vision Library(OpenCV) image-processing software4 along with apreviously proposed feature parameter (facial expression
intensity) 1 that is based on the mouth area that we

1. Introduction
In Japan, society is aging at an unprecedentedly rapid
pace, the average age of the population has been
increasing, and these trends are expected to continue.
Simultaneously, the number of elderly people withdementia and/or depression living in in underpopulatedareas is increasing very rapidly. However, since there isa gap between the number of patients and the number ofavailable healthcare professionals, it is difficult toprovide adequate psychological assessments andsupport for all patients.
To improve the quality of life (QOL) of elderly
people living in care facilities or in their homes, wepreviously developed a method for analyzing the facial
expressions of a person who is having a conversation
with another person on a videophone.1,2 However, it was
acknowledged that facial expression intensities could be© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan534

Taro Asada, Daichi Kogi, Ryoichi Shimada, Yasunari Yoshitomi, Masayoshi Tabuse

focused on in that research. Additionally, in our new
system, the time-to-utterance of the answerer just after
an interview question and the fundamental frequenciesof his or her voice are measured to provide clues
regarding his or her mental state.

which is a vector of the difference between the feature
vector for the expressionless face and that for an
observed expression. The facial expression intensity
candidate, which is defined as the norm of thedifference vector between two feature vectors, was used
for selecting the reference frame.2

2. Proposed System and Method

2.3. Reference frame selection and error
reduction processing

2.1. System overview and method outline
We constructed two modules for use in our proposedsystem, which is based on our previously reported
research.5 The first is a module for showing aninterview by an agent on the computer screen for theanswerer, and the second is a module for real-time facial
expression analysis of the moving imagery captured inreal-time by a webcam. The frames of the moving
images are then changed from RGB image data into
YCbCr image data, after which the Y component
obtained from each frame in the dynamic image is used
to analyze the facial expressions.
The proposed method consists of: (1) extracting themouth area, (2) measuring the facial expression
intensities, (3) selecting the reference frame, (4)measuring the time-to-utterance of the answerer justafter an interview question, (5) calculating the featureparameters for the facial expression strength, and (6)measuring the fundamental frequencies for voices of theanswerer using a conventional method. In the following
subsections, (2) and (3) are explained in detail. For
details on (1), and the judgment of utterance mentionedin (4), see Refs. 1 and 2. For details on (5), see Ref. 1.

The reference frame selection method used is based on
our reported research.6 In this process, the first 20
continuous frames of mouth area data that are
successfully extracted after the webcam recording
begins are treated as reference frame candidates. Then,
beginning with the first reference frame candidate, thefacial expression intensity of all candidate frames ismeasured, and the sum of all facial expression
intensities is calculated. The candidate frame with theminimum value is selected as the reference frame. In
this method, by applying the minimum value filter to themeasured facial expression intensity of the last threeframes, the influence of mouth area extraction error
induced by OpenCV is reduced.
3. Simulated interview using MMDAgent
The interview video was made from a video recording of ascene in which a psychiatrist interviewed a subject.3
Although a video on which an actual psychiatrist isdisplayed would be ideal, it is impractical to create newvideos whenever it becomes necessary to update questions
and/or make intonation changes. Accordingly, in thisresearch, we used MikuMikuDanceAgent (MMDAgent),7
which is a freeware animation program that allows users tocreate and animate movies, to create an agent.
To produce the agent’s voice, we used the speechsynthesis function setting built into MMDAgent.7 Agentmotions during the interview included "nodding" and"blinking" movements that were designed to mimic theactual mannerisms of the psychiatrist who was recorded on
the interview video used in the previously reported study3.

2.2. Measurement of facial expression intensity
For the Y component of the selected frame, the feature
vector for the facial expression was extracted for the mouth
area. The extraction is performed by using a twodimensional discrete cosine transform (2D-DCT) for each 8
× 8 pixel section. In order to minimize the influence of facial
expression intensity due to the jaw and/or neck line(s), thebottom left and right areas of the mouth areas are excluded.8
To measure the feature parameters of the facial
expressions, we selected low-frequency components
from the 2D-DCT coefficients for use as the facial
expression feature vector elements. However, the direct
current component was not included. In total, 15 feature
vector elements were obtained. The facial expression
intensity is defined as the norm of the difference vector,

4. Experiments
4.1. Conditions
The agent interview was created while referring to ascene in which a psychiatrist interviewed a subject. In© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

535

15

Subject A

A1

Facial expression intensity

Facial expression intensity

A System for Analyzing

Subject B

B1

10
5
0
0

20

40

60

80

15

Reference frame

A2

B2

5
0
0

20

40

60

80

Time (s)Subject A

Subject B

A1

Subject B

10Time (s)Subject A

Subject A

Reference frame

Reference frame

B1

Subject B

A2

Reference frame

B2

Fig. 2. Changes in facial expression intensity of mouth areafor Subjects A and B (upper graph). Reference frame andmouth area images are shown during Experiment 2 (A2 andB2), as indicated on the graph (lower images).

Fig. 1. Changes in facial expression intensity of mouth areafor Subjects A and B (upper graph). Reference frame andmouth area images are shown during Experiment 1 (A1 andB1), as indicated on the graph (lower images).

Facial expression intensitythat interview, the agent asked the subject eight
fundamental questions generally used to ascertain thecondition a patient with potential depression. A
preliminary experiment was then conducted to decide
the appropriate interval time between questions.
In this study, we used a Dell XPS 9350 personal
computer (PC) equipped with an Intel Core i7-6560U
2.2GHz central processing unit (CPU) and 8.0 GB of
random access memory (RAM) for a computational
environment. The Microsoft Windows 7 Professional
operating system (OS) was installed on the PC andMicrosoft Visual C++ 2008 Express Edition was usedas the development language.
Five males (Subject A in his 30s, Subjects B, C, D,
and E in their 20s) participated in the experiments. Eachof the subjects was interviewed by the agent for about
95 seconds. As an initial experimental condition after
the start of the interview, the subjects were instructed tomaintain a neutral facial expression without speaking
for about five seconds. Then, in the first threeexperiments, Subjects A and B were requested tointentionally respond with expressions of three types of
emotions (Experiment 1, relaxed; Experiment 2,
excited; and Experiment 3, depressed). In Experiment 4,
subjects C, D, and E were requested to respondnaturally to the same interview as that used inExperiments 1 and 2.
The visual of mouth area and audio information of thesubjects in the experiments were saved as BMP and

Subject A

15

A3

B3

10

Subject B

5
0
0

20

40

60

80

Time (s)Subject A

Reference frame

Subject B

A3

Reference frame

B3

Fig. 3. Changes in facial expression intensity of mouth areafor Subjects A and B (upper graph). Reference frame andmouth area images are shown during Experiment 3 (A3 andB3), as indicated on the graph (lower images).

WAV files. The BMP files were used for measuring thefeature parameters of the facial expressions, while the
WAV files were used for measuring the time-to-utteranceof the subject just after an interview question was given
and the fundamental frequencies of the subject.
4.2. Results and discussion
The facial expression intensity for the mouth area becamehigh every time during the response to a question (Figs.
1-3). Note that the feature parameters for facial© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

536

Taro Asada, Daichi Kogi, Ryoichi Shimada, Yasunari Yoshitomi, Masayoshi Tabuse

expressions for the mouth area tended to be greater inExperiment 2 than in both Experiments 1 and 3 (Table 1).
In addition, the average of the fundamental frequencies ofthe voice of the subjects was greater in Experiment 2 thanin both Experiments 1 and 3 (Table 2). The order ofincreasing time-to-utterance of the answerer just after aninterview question was terminated was Experiments 2, 1,
and 3 (Table 3).
The time-to-utterance of the answerer just after aninterview question for Subjects C, D, and E in
Experiment 4 was longer than that for Subjects A and B
in Experiments 1, 2, and 3 (Table 3).
Based on these results, we conclude that it ispossible to use our interview by agent system for
medical examinations (Tables 1-3).

frequencies of his or her voice were measured in orderto help analyze his or her mental state. It is believed that
this system could prove useful in the treatment of persons
suffering from psychiatric ailments such as depressive
disorder and dementia.
Acknowledgements
The authors would like to thank Professor J. Narumoto of
Kyoto Prefectural University of Medicine for his valuable
support and helpful advice in the course of this research.
We would like to thank the students of the Graduate Schoolof Kyoto Prefectural University for their assistance in our
experiments. This research was partially supported by theCenter of Innovation Science and Technology based
Radical innovation and Entrepreneurship Program (COI
STREAM) of the Ministry of Education, Culture, Sports,
Science, and Technology of Japan.

Table 1. Feature parameters for facial expressions.
Exp.
Subject A
Subject B
Subject C
Subject D
Subject E1(relaxed)

3.5
2.9

2(excited)3(depressed)

4.1
4.8

3.1
3.1

4(naturally)

References

4.0
4.0
2.1

1. T. Asada, Y. Yoshitomi, R. Kato, M. Tabuse, and J.
Narumoto, Quantitative evaluation of facial expressions
and movements of persons while using video phone, J.
Robotics, Networking and Artif. Life 2(2) (2015) 111-114.
2. T. Asada, Y. Yoshitomi, R. Kato, M. Tabuse, and J.
Narumoto, A system for facial expression analysis of aperson while using video phone, J. Robotics, Networking
and Artif. Life 3(1) (2016) 37-40.
3. T. Asada, Y. Yoshitomi, and M. Tabuse, A system foranalyzing facial expression and verbal response of aperson while answering interview questions on video, J.
Robotics, Networking and Artif. Life 4(2) (2017) 114-117.
4. Open CV. http://opencv.org/ Accessed 7 December 2017.
5. T. Asada, D. Kogi, R. Shimada, Y. Yoshitomi, and M.
Tabuse, A system for facial expression analysis of aperson while answering interview questions by agent (inJapanese), in Proc. of Human Interface Symposium 2017
(Osaka, Japan, 2017), pp.533-536.
6. R. Shimada, T. Asada, Y. Yoshitomi, and M. Tabuse,
Real-time system for horizontal asymmetry analysis on
facial expression and its visualization, J. Robotics,
Networking and Artif. Life to appear.
7. MMDAgent. http://www.mmdagent.jp/ Accessed 25 July
2016.
8. Y. Yoshitomi, T. Asada, K. Mori, R. Shimada, Y. Yano, M.
Tabuse, Facial expression analysis and its visualization
while writing messages, in Proc. of Int. Conf. on Artif. Lifeand Robotics (Beppu, Oita, Japan, 2018), in press.

Table 2. Average of fundamental frequencies of subject
voices in Experiments 1-4.
Exp.
Subject A
Subject B
Subject C
Subject D
Subject E1(relaxed)2(excited)

137.9
110.7

157.9
145.4

3(depressed)

106.9
103.7

4(naturally)

102.3
143.6
132.1
(Hz)

Table 3. Time-to-utterance of the answerer just after aninterview question was terminated.
Subject
Exp.
Q.1
Q.2
Q.3
Q.4
Q.5
Q.6
Q.7
Q.8
Ave.

1
1.2
1.3
1.4
1.3
1
1
1
1.6
1.2

A
2
1
0.7
0.7
1
0.7
0.6
1
0.6
0.8

3
1.6
1.7
1
1.7
1
1.2
1.3
1
1.3

1
0.9
0.4
1.1
0.9
0.5
0.5
0.8
0.9
0.8

B
2
1
0.7
0.3
0.5
1.4
0.3
0.8
0.6
0.7

3
0.9
1.1
0.7
1
1.2
1.1
0.7
0.5
0.9

C
4
1.1
3.3
1.3
0.9
2.7
0.8
1.9
1.6
1.7

D
4
2.1
1.3
0.9
1.3
1
1.4
2.4
1.5
1.5

E
4
2.4
1.2
0.5
0.9
1
1.3
1.6
2.2
1.4
(s)

5. Conclusion
Herein, we reported on the development of a system foranalyzing the facial expressions of a person while he or
she is answering interview questions posed by an agent.
Moreover, the time-to-utterance of the answerer justafter an interview question and the fundamental© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

537

Facial Expression Analysis and its Visualization While Writing Messages
Yasunari Yoshitomi1, Taro Asada1, Kenta Mori2, Ryoichi Shimada3, Yuiko Yano1, Masayoshi Tabuse1

1: Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: {yoshitomi, tabuse}@kpu.ac.jp, t_asada@mei.kpu.ac.jp}
http://www2.kpu.ac.jp/ningen/infsys/English_index.html
2: Neyagawa City Hall
1-1 Honmachi, Neyagawa, Osaka Prefecture 572-8555, Japan
3: JFE Systems, Inc.
4-1-3 Olinas Tower 17F, 4-1-3 Taihei, Sumida-ku, Tokyo 130-0012, Japan

Abstract
We have developed a real-time system for expressing emotion as a pictograph selected according to the facial
expression while writing a message. The image signal is analyzed by our real-time system using image processing
software (OpenCV) and a previously proposed feature parameter. We applied the system to post a message and apictograph expressing the facial expression while writing the message on an SNS. The experimental results suggestthat our system can be useful for expressing emotions while writing messages.
Keywords: Facial expression analysis, Real-time system, Mouth area, Visualization, Writing messages, OpenCV.

1.

pictograph, and posting the pictograph on Twitter together
with the message. The moving image in our method is
captured by a webcam and analyzed in real-time via Open
Source Computer Vision (OpenCV) image-processing
software2 by using a previously proposed feature parameter(facial expression intensity)3 that is based on the mouth
area. The visualization method proposed in this paper isbased on a previously reported method.4

Introduction

Social network services (SNSs) have become very
popular worldwide as communication tools on theInternet. For example, Twitter, one of the most popularSNSs, has 328 million users active monthly.1 We canpost a message, a static image, or a moving image on
Twitter. However, we cannot post information on
Twitter about our real emotions while writing a message.
One of our current studies is aimed at expressing thereal emotions of persons writing messages for posting on
an SNS site by analyzing their facial expressions andvisualizing them as pictographs. In this paper, we proposea real-time system for analyzing a facial expression whilewriting a message, visualizing the facial expression as a

2. Proposed System and Method
2.1. System overview and outline of the method
In this system (see Fig. 1), a webcam moving image
captured in real-time is analyzed via the following
process.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

538

Yasunari Yoshitomi, Taro Asada, Kenta Mori, Ryoichi Shimada, Yuiko Yano, Masayoshi Tabuse

Fig. 2. Mouth area. The four 8 8-pixel sections excluded
from the measurement in mouth area appear as cross-hatchedsquares.
Fig. 1. Display of the proposed system at startup.

study, the first 20 continuous frames of mouth area data
successfully extracted after the webcam recording begins
are treated as reference frame candidates. The reference
frame selection method is explained in detail in
Reference 5.

The proposed method consists of (1) extracting themouth area, (2) calculating the facial expression feature
vectors, (3) determining the facial expression intensity
and calculating its average value while writing amessage, (4) and posting the message and an
automatically selected pictograph for the message on
Twitter. The details of these steps are explained in thefollowing subsections.

2.4. Posting a message and an automatically selectedpictograph for the message on Twitter
Facial expression intensity is measured using ourpreviously discussed method4, in which a straight line isdrawn on a graph image prepared using OpenCV. A
pictograph is automatically selected by comparing the
average value of facial expression intensity and athreshold decided experimentally beforehand, and thenthe message and the selected pictograph are posted on
Twitter when the user presses a button. In this system,
two pictographs, expressing neutral and smiling facial
expressions, are used (see Fig. 3).

2.2. Mouth area extraction
First, moving image data are changed from RGB toYCbCr image data, after which the face area is extracted
from the YCbCr image as a rectangular shape, and thelower 40% portion of the face area is standardized. Next,
the mouth area is extracted from that portion. The mouth
area was selected for facial expression analysis because itis where the differences between neutral and smiling
facial expressions appear most distinctly.
2.3. Facial expression intensity measurement
For the Y component of the selected frame, the facial
expression feature vector is extracted for the mouth area
using a two-dimensional discrete cosine transform (2DDCT) for each 8 8-pixel section. In this study, two 8 8pixel sections at each of the left and right lower corners(see Fig. 2) are not included for this measurement,
because these sections might cause errors due to theappearance of the jaw and/or neck line(s) there. To
measure the feature parameters of the facial expressions,
we select low-frequency components from the 2D-DCT
coefficients as the facial expression feature vector
elements; however, the direct current component is notincluded. In total, 15 feature vector elements are
obtained. The facial expression intensity is defined as
the norm difference between the facial expression
feature vectors of the reference and target frames. In thisFig. 3. Two pictographs expressing a neutral (left) or smiling(right) facial expression.

3. Experiment3.1. Conditions
The experiment was performed on a Dell XPS 9350 PC
equipped with an Intel Core i7-6560U 2.2 GHz central
processing units (CPUs) and 8.0 GB of random accessmemory (RAM). The Microsoft Windows 7
Professional operating system (OS) was installed on the
PC and Microsoft Visual C++ 2008 and 2013 Express
Editions were used as the development language.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

539

Facial Expression Analysis and

CoreTweet6, as a library, and the Twitter API were used
to post a message and pictograph on Twitter.
Seven males (subjects A to D in their 20s, subjects E
and F in their 30s, and subject G in his 50s) participatedin the experiments.
We first performed experiments with six males (allexcept subject G) under the two conditions listed below.
As an initial condition in the experiment, the subjects
were instructed to maintain a neutral facial expression
and face forward without speaking for about fiveseconds just after the start of the experiment. After the
initial state of a neutral facial expression was terminated,
the subjects were requested to intentionally respondwith one of two types of facial expressions (Experiment
1, neutral; Experiment 2, big smile) and write a message,'このぬいぐるみかわいくない' (in Japanese), whichmeans, ‘This stuffed toy is pretty, isn't it?’ Experiments
1 and 2 were performed three times for each subject. In
eachexperiment,
facialexpression
intensity
measurements were performed for each subject during
writing a message for 40 seconds and then the average
facial expression intensity for that message wascalculated. In order to distinguish between the neutral
and smiling facial expressions on the basis of average
facial expression intensity for a message, a threshold
was set as the average of the maximum among neutral
expressions and minimum among smiling expressionsof the six subjects.
Two additional experiments (Experiments 3 and 4)
were next performed with only subject A. The same
initial conditions were used in these experiments as inExperiments 1 and 2. After the initial state of a neutral
facial expression was terminated, the subject was
requested to intentionally respond with two types of
emotions (Experiment 3, neutral; Experiment 4,
smiling) and write a message, '明日は情報伝達システ
ム学サブゼミに参加します。時間は 5 時限目、場
所は先生の部屋です。' (in Japanese), which means, ‘Iwill attend the discussion section held at the professor’s
room of the information communication system lab. in
fifth period tomorrow.’ Fig. 4 shows snap-shots ofExperiments 3 and 4, which were extracted from thecaptured moving images of those experiments. In eachexperiment, facial expression intensity measurement
was performed for 30 seconds and then the average
facial expression intensity for that message wascalculated, after which both the message and theFig. 4. Snap-shots of subject A in Experiment 3 (neutral; left)
and Experiment 4 (smiling; right).

Fig. 5. Messages and pictographs posted on Twitter inExperiment 3 (neutral; upper) and Experiment 4 (smiling;
lower).

pictograph expressing the facial expression whilewriting the message were posted on Twitter. Then,
subjects B, C, D, and G participated in twoquestionnaire surveys comprising questions asking themto assign one of four adjectives (neutral, happy, sad, or
angry) to (a) each of the captured moving images, and(b) for each of the messages accompanied by
pictographs (see Fig. 5) in Experiments 3 and 4.
3.2. Results and discussion
The ranges of average values of facial expression
intensities in Experiments 1 (neutral) and 2 (big smile)
were 2.13 to 3.70 and 3.50 to 15.61, respectively.
Therefore, the threshold for distinguishing between the two
types of facial expressions was determined as 3.60. Thus,in the proposed system, a facial expression having an
average facial expression intensity under 3.60 is judged to
be a neutral facial expression, whereas one 3.60 or higher
is judged to be a smiling expression.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

540

Yasunari Yoshitomi, Taro Asada, Kenta Mori, Ryoichi Shimada, Yuiko Yano, Masayoshi Tabuse

The former value is under 3.60 and the latter is above
3.60, resulting in our proposed system selecting theneural pictograph (Fig. 5, upper) in Experiment 3 and thesmiling pictograph (Fig. 5, lower) in Experiment 4 forposting on Twitter. The results of the questionnaire
surveys show that our proposed system distinguishedcorrectly between the two types (neutral and smiling) offacial expressions for the subject, and that the pictographs
selected by the system correctly reflected the facial
expressions while writing messages for the subject.
Note, in the case of subject F, for a typical example,
the effect of exclusion of the four 8 8-pixel sections
from the measurement in mouth area (see Fig. 2) was
estimated to be about 22% in terms of difference infacial expression intensity.

A1
A2

Reference frame

A1(max)A2(min)Fig. 6. Changes in facial expression intensity of mouth area
for subject A (upper graph). Mouth images are shown for
two moments during Experiment 3 (neutral) (A1 (maximumof facial expression intensity) and A2 (minimum of facial
expression intensity)), as indicated on the graph (lower
images), along with that of reference frame.

4. Conclusion
We have developed a real-time system for expressing
emotion as a pictograph selected according to the facial
expression while writing a message. We applied thesystem to the posting on Twitter of both a message anda pictograph. The experimental results suggest that oursystem can be useful for expressing emotions during
writing messages.

A3

Acknowledgements
We would like to thank the subjects for theircooperation in the experiments.
A4

Reference frame

References

A3(max)

1. http://files.shareholder.com/downloads/AMDA2F526X/5451637324x0x951000/718EA233-DF7E-4ECBAC2B-BFAB3F067C72/Q217_Shareholder_Letter.pdf
Accessed 13 December 2017.
2. Open CV. http://opencv.org/ Accessed 13 December 2017.
3. T. Asada, Y. Yoshitomi, R. Kato, M. Tabuse, and J.
Narumoto, Quantitative evaluation of facial expressions
and movements of persons while using video phone, J.
Robotics, Networking and Artif. Life 2(2) (2015) 111-114.
4. T. Asada, K. Mori, R. Shimada, Y. Yoshitomi, and M.
Tabuse, Facial expression analysis in real time while writing
messages (in Japanese), in Proc. of Human Interface
Symposium 2017 (Osaka, Japan, 2017), pp. 469-472.
5. R. Shimada, T. Asada, Y. Yoshitomi, and M. Tabuse,
Real-time system for horizontal asymmetry analysis on
facial expression and its visualization, in Proc. of Int. Conf.
on Artif. Life and Robotics (Oita, Japan, 2017), pp. 120-123.
6. CoreTweet. https://github.com/CoreTweet/ Accessed 20
December 2017.

A4(min)Fig. 7. Changes in facial expression intensity of mouth area
for subject A (upper graph). Mouth images are shown for
two moments during Experiment 4 (smile) (A3 (maximumof facial expression intensity) and A4 (minimum of facial
expression intensity)), as indicated on the graph (lower
images), along with that of reference frame.

As shown in Figs. 6 and 7, our proposed system wasable to distinguish between the two types of facial
expressions, neutral and smiling. The average facial
expression intensities for the two types were 2.94 and
9.60, respectively, in Experiments 3 and 4, respectively.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan541

Recognition of Finger Spelling from Color Images Using Deep Learning
Yusuke Yamaguchi, Masayoshi Tabuse

Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail:y_yamaguchi@mei.kpu.ac.jp, tabuse@kpu.ac.jp

Abstract
We have developed a system to recognize finger spelling in Japanese sign language using deep neural networks. As
deep neural networks, we adopt Faster R-CNN. By defining an output class for each finger letter and learning theobject detection network, it is possible to output where the finger letter exists in the input image. This method does
not require depth cameras, magnetic sensors, or other special equipment when used. Furthermore, this does not
require preprocessing that extracts the hand region using human skin colored regions and color gloves used in othermethods using color images. We synthesized a training data set by processing images taken with Kinect. As the test
data, we input images of performing finger letters into the trained network and check the score of the output area andclass.
Keywords: Recognition of Finger Spelling , Deep Learning, Faster R-CNN, Color Image.

image, it is necessary to devise a method for extracting ahand region. A method of extracting a hand region froma color image using human skin colored regions [3] or a
special background [4] has been proposed. Thesemethods require restrictions on the use situation.
This paper deals with 41 kinds of finger spelling,
without movement, recognition using Faster R-CNN asan approach to these challenges. We trained the network
using the data generated based on the image captured by
Kinect, and in the experiment we recognized the video ofthe sign language teaching material video.

1. Introduction
Many deaf and hard of hearing people in Japan usually
use Japanese Sign Language (JSL). On the other hand,most hearing people use spoken language, and many ofthem do not know sign language. In order to break down
barriers between them, development of a sign language
interpretation system is urgent. Finger spelling is a visuallanguage that represents characters using hand shapes,
and is used mainly to express words of proper nouns orthat is not defined as sign language.
The method of acquiring finger spelling features isroughly divided into "contact type" and "non-contacttype" depending on the sensor to be used. In the "contacttype" method, equipment such as a data globe [1] or acolor globe [2] is attached to the user. These sensors taketime and effort to attach, and there is also a risk ofaffecting sign language operations. "Non-contact type"has burden on the user less than "contact type".
Especially a single color camera is the most plain andpopular sensor. In the method using only a single RGB

2. Faster R-CNN
Faster R-CNN [5], which is an object detection network,
is a method that speeds up R-CNN [6] and Fast R-CNN[7].
2.1. R-CNN, Fast R-CNN
Fast R-CNN is a method that reduces the number of timesof convolution processing with the highest calculation©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4,B-ConPlaza, Beppu, Oita, Japan

542

Yusuke Yamaguchi, Masayoshi Tabuse

These processes of Faster R-CNN can be trained
end-to-end as a single network.

cost in R-CNN. Like the R-CNN, region proposal isacquired by the selective search. Then, the entire image
is convoluted, and a feature map of the object proposed
region is extracted from the convolutional feature map.
The extracted feature map is pooled by the Region Of
Interest (ROI), and is input to fully connected layer(hereinafter referred to as subnets). ROI pooling pools
the feature map of any size into regions partitioned into
the set grid sizes, so that feature maps of arbitrary sizes
can be obtained.
In this method, since it is unnecessary to pass eachregion proposal to the CNN, it is possible to perform
object detection approximately 100 times faster than RCNN. However, the processing cost of selective searchused in the pre-processing of Fast R-CNN is a bottleneck.

3.

Experiments

First, we trained Faster R-CNN using the finger spelling
movement data set. After that, we input the finger
spelling image into the trained network and confirmed
whether the output was correct or incorrect. If a class of
output having maximum score is the same class as the
input image and the output area indicates the part of thehand performing the finger spelling on the image, let theoutput be correct.
3.1. Network details
In this experiment, we used Faster R-CNN connected to
the fifth convolution layer of VGG-16. The learningparameters are as shown in the table 1. We usedapproximate joint training [5] to train Faster R-CNN.

2.2. Faster R-CNN
Faster R-CNN is a network that performs "region
proposal" with Region Proposal Network (RPN), which
was done by selective search of Fast R-CNN. Faster RCNN has the structure shown in the Fig 1. In this method,
it is possible to detect the region proposals and determine
the object class of the region at the same time, and it ispossible to detect objects faster than Fast R-CNN.
As with Fast R-CNN, convolution is performed on
the input image to obtain a feature map. RPN raster-scansa detection window (called an anchor) against a feature
map to detect an object. The anchor is a method of
applying k detection windows around the region ofinterest and raster scanning. The area designated by theanchor is input to the RPN, and the score of theobjectness and the rectangular area on the input image are
outputted. The area designated by the anchor is also input
to the subnet of Fast R-CNN, and object recognition is
performed when it is judged as an object by RPN.

3.2. Experiments data
Deep neural network like Faster R-CNN requires a large
amount of various types of training data. However, there
are no popular data sets for training and recognizing sign
language and finger spelling.
Therefore, we created training data based on images
taken with finger spelling movements. We preparedtraining data by processing a small number of original
images and generating large amounts of various variation
data. For recognition experiments, images are used aslearning materials for sign language and finger spelling.
Table 1. Learning parameters.
Parameters

settingbase learning rate (lr)
Schedule of learning rate (lr_policy)
gamma
anchoraverage loss
momentum of SGD
weight decay

0.001
Step
0.1
9
100
0.9
0.00053.2.1. Training-validation data
We used Microsoft's motion sensor Kinect v2 to capture
images. Kinect is a multi-sensor device that can handle

Fig. 1. The overview of Faster R-CNN.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan543

Recognition of Finger Spelling

subjects sufficiently confirmed how to operate the finger
spelling described in the textbook [9].
3.2.2. Recognition data
Since there is no general evaluation data, we use imagesof commercial teaching materials [9] shown in Fig. 3 asdata for evaluation. The resolution of this movie is
720×480. This movie reproduces mirrored images forright-handed people and small mirrored images for lefthanded people at the same time. In this experiment,
training was carried out only with the movement by theright hand, so only the output to the lower right area wasrecorded.

Fig. 2. The flow to generate some training-validation data froma finger spelling image captured by Kinect with SUN397
dataset.

color camera, depth camera, audio, etc., and can detect
25 joints of a person [8]. When capturing finger spelling
movements, the subject stands facing forward withrespect to Kinect. When acquiring images, we use blue
or green clothing and background which iscomplementary color of human skin color.
We trimmed the hand image from the color image,centered on the coordinates of the hand obtained by
Kinect, erased the green and blue parts, and made a handonly image. Images with only the tip from the wrist were
deformed with the following random parameters andarranged at random coordinates of the background image.
At this time, we randomly selected one image from
SUN397 which is the image data set for scene
recognition as the background image and used it.
In order to increase the dispersion of the learning data,
image processing was randomly executed according to
the following list. The number shown in parentheses isthe probability of execution.

4.

Results and discussion

The left axis in Fig. 4 shows the mean Average Precision(mAP) for each 10000 iterations using test data. mAP
becames flat after 60,000 times.
Even when the recognition score of the lower limit
was set to 0.8, 0.5 and 0.2, the recognition rate after
140,000 times was 64.4%, 76.1% and 78.8% respectively.
33 classes are recognized correctly for almost allframes, and 8 classes are misrecognized for all frames.
We divided the 8 classes that failed to recognize into 3
groups that seemed to be similar misrecognition causes
as shown in the table 2. In group 1, finger shapes are
similar. Since Kinect needs to use about 2 meters away
to detect joints of a person, acquired images are low
resolution. Therefore, it is insufficient to train thecomplex feature of finger spelling. Group 2 has a shapethat the fingers protrude forward, and it seems thattraining data could not correspond to them because theappearance changes greatly as the viewpoint changes alittle. We would like to respond by brushing up to flexible
learning data that can correspond to this. In Group 3, therotation in the range of -10 degrees to +10 degrees(1),extend the height and width from 1.0 times to 1.2
times(1),contrast emphasis(1/3) or reduction(1/3),
Gamma correction in the range of 0.75 to 1.5(1),Blur in kernel size 1 to 5(1/2),
Gaussian noise with mean 0 and sigma 15(1/3),
salt-and-pepper noise for 0.4% of pixels in equal
ratio(1/4).
Based on samples from each subject, we generated
roughly 500 pairs of training data for each finger spellingand summed up to generate 66,281 data. 80% of thegenerated data was used as training data and theremaining 20% was used as validation data.
We captured sample images from three subjects
who do not know sign language. Before experiments, theFig. 3. One of frames of video used for recognition.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan544

Yusuke Yamaguchi, Masayoshi Tabuse

0.8

80

0.6

60
mAP
0.2 or more
0.5 or more
0.8 or more

0.4
0.2
0.0

0

5

10

15

40

recognition rate [%]100

mean Average Percision(mAP)

1.0

Fig. 5. Finger spelling 'na' overlaps the arm like 'su' and 'fu'.

teaching material of finger spelling, recognition rate of
78.8% was obtained.

20

20

0

6.

number of Itteration [unit: 10000]

1. T. Yamane, T. Funatomi, M. Iiyama, M. Minoh,
Estimation of Rigid Transformation of Finger Segmentsfrom Data Glove, IEICE Technical report, Vol.111,
No.353, pp.77-82, 2011.
2. M. Osato, M. Suzuki, A. ITO, S. Makino, Feature Value
Combination for Finger Character Recognition Using a
Color Glove, IEICE Technical report, Vol.105, No.375
pp.93-105, 2011.
3. K. Miura, Y. Chang, N. Mukai, Recognition ofFingerspelling in Japanese Sign Language based on Nail
Detection and Wrist Position, ITE Technical Report, Vol.
37, No. 17, pp. 199-202, 2013.
4. K. Hirose, Recognition of Japanese Manual Alphabet
Using Thinning Image, CVIM, Vol. 93, No. 66, pp.1-6,
1993.
5. S. Ren, K. He, R. Girshick, J. Sun, Faster R-CNN:
Towards Real-Time Object Detection with Region
Proposal Networks, NIPS 2015.
6. R. Girshick, J. Dnahue, T. Darrell, J. Malik, Rich feature
hierarchies for accurate object detection and semantic
segmentation, CVPR 2014.
7. R. Girshick, Fast R-CNN, ICCV 2015.
8. J. Shotton, A. Fitzgibbon, A. Blake, A. Kipman, M.
Finocchio, B. Moore, and T. Sharp, Real-Time Human
Pose Recognition in Parts from a Single Depth Image,
CVPR 2011.
9. K. Matsumoto, Introduction to sign language understoodfrom zero with DVD (in Japanese), Shufunotomo Co., Ltd.,
Chiyoda-ku, Tokyo, 2011.

Fig. 4. The right axis shows recognition rate by minimum score
to allow output and the left axis shows mean average precision
for each iteration.

erroneously recognized area outputted contained arms asshown in Fig. 5. It seems that the convolution featureincluding the arm resembled that class. In order to deal
with this, it is necessary to consider a learning method
considering the arm. In addition, it is necessary toexperiment the image other than the moving image usedin the experiment and confirm the usefulness of thismethod. In the sign language recognition system, it isalso necessary to have a system capable of recognizing
Table 2. Class group that was
misrecognized due to close cause.
groupgroup 1group 2group 3

input
chi'
hi'
ra'
ru'
so'
tsu'
na'
ma'

Reference

outputto', 'ra'u', 'ra'u'ka'
ha'u', 'me', 'wa'fu', 'su'ne'finger spelling with movement.
5.

Conclusion

We trained Faster R-CNN and conducted an experiment
to recognize JSL fingers without movement from a single
color image. In the experiment using the video of© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan545

Recognition of Texting While Walking Using Convolutional Neural Networks
Junpei Miyachi, Masayoshi Tabuse

Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Shimogamohangi-cho, Sakyo-ku, Kyoto 606-8522, Japan
E-mail:j_miyachi@mei.kpu.ac.jp, tabuse@kpu.ac.jp

Abstract
In recent years, texting while walking has become a problem with the spread of smartphones. Pedestrians who operatea smartphone while walking decrease attention to their surrounding environment. The purpose of this study is todetect the pedestrians using surveillance camera for the support of accident preventions in railways. We propose themethod of texting while walking recognition using convolutional neural networks. We show the classification resultsof a pedestrian who is texting while walking.
Keywords: Texting while walking, Pose estimation, CNN.

the data. Therefore, we generate training data from asmall amount of sample. We verify the effectiveness of
the recognition method using this dataset.

1. Introduction
The life of people becomes more convenient bysmartphone in recent year. However, the number ofpeople who operate a smartphone while walking isincreasing. Texting while walking is one of dangerous
acts because the pedestrian decreases attention to theirsurrounding environment. It is the cause of falling fromstation’s platform or bumping into someone in publicareas such as railway stations. A monitoring system is
one of measures for the support of accident preventions.
Therefore, detecting pedestrians who operate asmartphone using a surveillance camera is required.
Pedestrians which are texting while walking have thesimilar features of posture. They tend to bring their arm
up to chest level and walk with their head down. In orderto recognize the posture, we employ convolutional neuralnetworks (CNN) that is one of deep learning, which has
improved results in various fields in recent years. It is
expected to obtain effective features for recognition bylearning. In this paper, we propose the method of the
texting while walking recognition using CNN. Althoughdeep learning generally requires a large quantity of
training data for adequate learn, it is a burden to collect

2. Related works
2.1. Studies on recognition of texting while
walking
Some studies on recognition of texting while walking arereported. Minamoto et al. [1] proposed a detect methodof texting while walking using HOG features for guide
dog robot. The robot recognizes the pedestrian facing his
front to it. Shinmura et al. [2] recognized a pedestrian
which are texting while walking using HOG features andSVM classifier for a vehicle-mounted camera. Theytargeted a pedestrian facing his front, left and right side
to the camera. Pedestrians turning to each direction are
mixed at a platform of a railway station. Regarding theaccidents in a station, it may occur in any direction on theplatform. Therefore, recognition of texting while walking
is required regardless of the pedestrian direction.

©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4,B-ConPlaza, Beppu, Oita, Japan

546

Junpei Miyachi, Masayosi Tabuse

2.2. Convolutional neural networks

3. Recognition method of texting while walking

CNN is widely used in computer vision since ImageNet
Large Scale Visual Recognition Challenge in 2012(ILSVRC2012). The conventional method combined
hand-crafted features and statistical analysis. In contrast,
CNN can learn the feature value for recognition. It
composes of convolution, pooling and fully connected
layers.

We constitute a network as a classifier of normal walking
and texting while walking. The CNN learning uses
generated images for training data.
3.1. Training data generation
A large number of training data are needed for deeplearning. Therefore, we generate training data from asmall amount of samples by combining a human image
with a background image. Moreover, we apply
transformation processing such as rotation, scaling andgamma correction to the synthetic image. The
transformation parameters are randomly assigned.
Generating procedures of training data are as below.
(i) Rotation and scaling processing are applied forhuman images in a random order.
(ii) The human region of the image is cut out.
(iii) The human region and a randomized position of abackground image are combined.
(iv) Gamma correction is performed to the synthetic
image.

2.2.1. Convolution layer
In a convolution layer, it conducts sum-of-productoperation pixel valueof an input image and a weight. The sum-of-product operation is expressedfilter ℎusing the following equation:
(

=

,

)

, ℎ

+

,

(1)
where K denotes a number of channel, denotes a bias
in filter m. The output is a value which activation function
is applied. It plays a role of feature extraction in eachposition on the image.

3.2. Recognizing texting while walking using
CNN

2.2.2. Pooling layer
In a pooling layer, values of obtained feature maps areintegrated in a local area. It plays a role of immutable
output for local translation. In this study, we adopt max
pooling which selects a max value in the area. The pixel
value
obtained by max pooling is expressed using thefollowing equation:
= max
(2)
,

Classified pedestrian images are obtained by performing( , )∈

where
denotes a set of pixel, which takes a squaredomain centering the pixel (i, j) on an input image.

Fig. 1. A human image for cutting out the human region (left),background image (center), and generated image for training
data (right).

2.2.3. Fully connected layer

human detection to images including human. The image
is input to CNN.

Fully connected layer is a layer that all nodes isconnected with next layer’s them. It integrates
information of the former layer and calculate the output.
The calculation is expressed using the following
equation:
where

=

+

,

(3)

denotes a weight of the edge.

©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4,B-ConPlaza, Beppu, Oita, Japan547

Recognition of Texting While

The CNN structure used in this study is based on
Alexnet [3]. Alexnet consists of 5 convolutional layers, 3pooling layers and 3 fully connected layers, which isapplied to various works as high performance andversatility. It is shown in Fig. 2. In the input layer, input
image is resized to 227×227, and it is input to CNN. We
use ReLU function for the activation function in all
convolutional layers and two fully connected layers. In
output layer, the number of units in output layer is 2 toclassify into normal walking and texting while walking.
The softmax function is used for the activation function,and class probability is output. The classified result
selects the best probability class.

(a)
(b)
(c)
(d)
Fig. 3. Example of training data. (a)front normal walking,(b) left texting while walking, (c)right normal walking, and(d)back texting while walking.

(A)
(B)
(C)
Fig. 4. Example of test samples of each subjects. Let A, B
and C be them, respectively.

Fig. 2. The CNN structure used in this study. “conv”, “pool”,“norm” and “fc” denote convolutional, pooling, normalizationand fully connected layer, respectively. “NW” and “TW”
denote normal walking and texting while walking, respectively.

4.1. Conditions

images (= 8,000 images) are for validation data. Fig. 3
shows example of training samples. For test data, we took
videos for 3 subjects, 100 images for each type wererandomly selected from among images obtained by
performing human detection (Implementation: OpenCV)
to the videos. The number of test samples is 2,400 images(2 types × 4 directions × 100 images × 3 subjects) intotal. Example of test samples is shown in Fig. 4.

4.1.1. Datasets

4.1.2. Training setting

We collected pedestrian images taken with camera fordatasets used in experiments. We prepared the pedestrian
images for 11 pedestrians, which are both the state of
texting while walking and normal walking. The numberof pedestrian images are 10 images for 2 types (texting
while walking or normal walking) and 4 direction (front.
left, right, back), respectively. Therefore, there are 880
pedestrian images in total. For background images, weused SUN397 dataset in SUN database which is used forscene recognition. There are 458 images in “street”
category in the dataset. We randomly selected 100
images for background images.
According to section 3.1, we generated learning
samples. The number of learning samples is 88,000
images in total (880 pedestrian images × 100
background images). The 10 people images (= 80,000
images) of them are for learning data, while 1 person

Hyperparameters are used in learning using CNN. In thisexperiment, mini batch learning to shorten learning timeis performed, let 200 the number of mini batch, hence
epoch number is 100. Let 0.0002 be the initial value oflearning late, 0.1 times the rate every10 epochs. Learning
times is set to 40,000 that is 100 epochs.

4.

Experiments

We performed an experiment of recognizing a pedestrian
which is texting while walking using CNN.

4.1.3. Developing environment
The experiment was performed in the following
computational environment: the OS was Ubuntu 14.04
LTS; the CPU was Intel(R) Core(TM) i7-6700 CPU
4.00GHz; the GPU was Geforce GTX 1080; thedevelopment language was python 2.7.6 and C++. Caffe
which is framework for deep learning was used forlearning of CNN.

©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4,B-ConPlaza, Beppu, Oita, Japan

548

Junpei Miyachi, Masayosi Tabuse

4.2. Method

5. Conclusion

We classified human images for test data using CNN. To
evaluate the effectiveness of the recognition method, we
also experimented using HOG features and SVM
classifier (HOG+SVM), which is one of the conventional
method. F-measure which is calculated from precisionand recall was adopted for performance indicator, and wecompared two methods.

We proposed the recognition method of texting while
walking. In this method, we apply pedestrian images
obtained by performing human detection to input, and it
is classified into normal walking and texting while
walking using CNN. In this experiment, dataset including
training images which is generated human images wasused for training. As a result, average F-measure was
0.836, and it exceeded the conventional method.
Therefore, the present study has demonstrated that the
proposed method is effective for recognizing pedestrians
which is texting while walking.
As an area of future work, we intend to put emphasis
on features of the upper half of the body because theinformation of a pedestrian texting while walking issummarized for the part such as hand position and facial
direction. In addition, we will recognize other dangerous
acts in platform.

4.3. Results and discussion
Table 1 and Table 2 show results of classification using
HOG+SVM and CNN, respectively. In comparison withoverall average of each method, CNN obtained higher
results than HOG+SVM. It was verified that CNN havethe high performance for recognizing texting while
walking. In regard to the average of each direction, front
direction was the best result in all direction. On the other
hand, left and back direction was relatively low results.
In back direction, a pedestrian’s right arm hid with his
body. In left direction, because human images preparedin this experiment had a smartphone in right hand, thesimilar situation occurs. Subject C’s result was different
from subject A and B. The cause may be that subject C
was wearing short sleeves. Since used dataset have no
images such as a pedestrian wears short sleeves, it isassumed that appearance of skin part of arms have an
influence on recognition accuracy. High values was
obtained from subject A in each direction. This seems
that he placed his left hand on the smartphone.

References
1. H. Minamoto and M. Sano, Recognize Careless Behaviorof Walker –Detect Texting While Walking–, The 6th
IIEEJ SIG on Visual and Hearing Impaired Support(2015).
2. F. Shinmura, Y. Kawanishi, D. Deguchi, I. Ide, H. Murase
and H. Fujiyoshi, Pedestrian’s Inattention Estimation
based Recognition of Texting While Walking from InVehicle Camera Images, IEICE Technical Report (2015),
pp. 83–88.
3. A. Krizhevsky, I. Sutskever and G. E. Hinton, ImageNet
Classification with Deep Convolutional Neural Networks,
Advances in Neural Information Process. Systems 25(2012), pp. 1097–1105.

Table 1. The classification result using HOG + SVM.

Direction
Front
Left
Right
Back
Overall

A
0.802
0.355
0.577
0.790
0.669

F-measure
B
C
0.882
0.639
0.510
0.628
0.698
0.964
0.713
0.317
0.718
0663

Ave.
0.779
0.508
0.764
0.632
0.684

Table 2. The classification result using CNN

Direction
Front
Left
Right
Back
Overall

A
0.985
0.914
0.890
0.934
0.932

F-measure
B
C
0.985
0.853
0.645
0.623
0.919
0.930
0.772
0.338
0.844
0724

Ave.
0.938
0.739
0.914
0.718
0.836

©The 2018 International Conference on Artificial Life and Robotics (ICAROB2018),Feb. 1-4,B-ConPlaza, Beppu, Oita, Japan549

A Study on the Lumbar Burden Evaluation of Work using One Smartphone
Mizuki Maiguma†, Hiroki Tamura‡, Koichi Tanno

Faculty of Engineering, University of Miyazaki,
1-1, Gakuen-kibanadai-nishi, Miyazaki-shi, Japan*Department of Environmental Robotics, Faculty of Engineering, University of Miyazaki,
1-1,Gakuen-kibanadai-nishi, Miyazaki-shi, Japan
E-mail: hi13041@student.miyazaki-u.ac.jp†, htamura@cc.miyazaki-u.ac.jp‡

Abstract
In this paper, we propose the human lumbar burden evaluation method and state estimation system using asmartphone and its application on agricultural work. The proposed system consists of two functions, 1) "Stateestimation" has a function of estimating posture (Stand up, Sit down, Crouch, Walk, etc.,) of the subject. 2)"Lumbar burden estimation" has a function to estimate the angle of the waist from the angle of the subject's upper
body and calculate the lumbar burden in combination with other parameters. The aim of this paper is to get the dataof the subject's work status in the agricultural field in a simple manner and quasi-real time and help improve theagricultural work efficiency by constructing the agricultural work burden evaluation system were using asmartphone. In this paper, we show on the experimental results of two functions and the evaluation of the actual
agricultural work.
Keywords: smartphone, lumber burden, gyro sensor, state estimation, acceleration sensor, JACK

smartphone, unlike previous research that used two
smartphones.

1. Introduction
In recent years, agricultural workers in Japan havedecreased and aging rapidly so it is necessary to review
the physical environment of agricultural work. One
solution to reduce this problem, technologicaldevelopments has been made to reduce the burden of
agricultural workers and automation various agricultural
machinery. On the other hand, it is difficult toquantitatively compare how much newly developed
agricultural machinery reduces the burden of
conventional agricultural work. Therefore, in this paper,
we are focusing on lumbar burden to evaluate burden
reductions by introducing agricultural machinery by IT
and aiming to build a work burden evaluation system.
To construct the system, we used a smartphoneequipped with various sensors such as acceleration
sensor, gyro sensor, and so on [1]. This paper examines
whether the proposed system can be realized with one

2.

Previous Research

Our previous research, one smartphone (iPhone4s, iOS7.0) equipped with RFID compatible RW module jacket
installed in the breast pocket and one in the front pocketof the pants. The work burden evaluation carried out
with two smartphones. To evaluate the lumbar burden,
the pressure of lumbar intervertebral disc calculated
from the waist angle and knee flexion angle. At thistime, the waist angle can estimate from the device
inclination angle (upper body angle) of the smartphone
attached to the breast pocket and the device inclination
angle (thigh inclination angle) of the smartphone
attached to the front pocket of the pants. The upper
body angle is α, the waist angle is β, and the thighinclination angle is γ, which is shown in Fig.1. The
waist angle was calculated Eq. (1).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Oita, Japan

550

Mizuki Maiguma, Hiroki Tamura, Koichi Tanno

3. Proposed method
In this chapter, we describe the method of estimating
the loads that can be determined by the state and theestimated lumbar burden using a smartphone.
3.2 Algorithm for state determination
The flowchart of the smartphone that attached on thebreast pocket can be seen in the Fig. 3.
Fig.1 three angles used for calculation of lumber burden= 180 − ( + )(1)

The knee flexion angle can calculate from the thighinclination angle. The tilt angle of the device calculated
from the gravity acceleration that obtained from theacceleration sensor and it means the tilt angle of thedevice against the direction of the center of gravity ofthe body. The device inclination angle (slope angle θ)
showed in Fig. 2 and its calculation formula is shown in
the Eq. (2). In addition, grav-Y and grav-Z are defined
the gravitational acceleration of the Y-axis and the Zaxis. The unit is degree.

Fig.3 Algorithm flowchart

Subject attaches the smartphone to the breast pocket,
and resting for a few seconds in a standing state. The
criterion value L [deg] of the subject's state isdetermined after performing calibration. The α isdefined the moving average of the vector composite
value of acceleration x, y, z given by the user with 20
data. It assume that the case where α exceeds thethreshold value 0.05 is in operation. The operation is astate that waiting for determination of walking, rotation,
crouching, and sitting.When gyro sensor Y axis exceeds
0.06 [radian/10msec], the state turns to the rotation. In
addition, when the angle of the device is below a certain
value during the operation. this means in a crouching or
sitting state. When it is judged that the tilt angle of thedevice is stabilizes after the smartphone goes down has
been measured, the case where the tilt angle S [deg] ofthe device is equal or more than -15 [deg] of thejudgment value it is assumed to be sitting, the exception
is crouching.

Fig.2 Tilt angle of the device in the direction of gravity

Slope angle θ = arc tan (

)×

/180

（2）

The estimation formula used for calculating the
lumbar burden based on the estimation formula used in"MIYADAI Pro" developed by Professor Mitarai of
Miyazaki University. We substitute the calculated waist
angle and knee flexion angle into this lumbar burden
estimation formula to calculate the lumbar intervertebral
disk pressing force. In our research, burden evaluation
carried out using the same estimation formula.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Oita, Japan

551

A Study on the

3.3 Constantization of thigh inclination angle

4.2 Verification experiment of estimation
accuracy of lumbar intervertebral pressingforce

In this section, we describe methods to stabilize theangle of the thighs. The waist angle is calculated using
the slope angle of the device obtained from the attachedsmartphone and the slope of the thigh. Since themounting position is set in the breast pocket of thesubject, it is impossible to detect the thigh inclination
angle. Hence, it is necessary to make the constant thighinclination angle when it is determined operation.
Nevertheless since the way of walking and crouching isdifferent depends on the person, it is impossible todetermine obviously the inclination angle of the thigh.
Therefore, prior to conducting the experiment, the thighinclination angle in the walking, rotation, and crouchingis confirmed beforehand and used as the input
information.

Experimental tools are one smartphone, a mowing
machine (5.5kg), and an automatic mowing machine(80.0kg). Automatic mowing machine operates by theremote controller (0.5kg). The method is to wear asmartphone in the breast pocket of the subject and getthem to do farm work freely. We record that motion
with a video camera and confirm the moving state fromthis video data. In this experiment, the burden analysis
software JACK 6.1 version (JACK) was use forcomparison. JACK is the software that can create virtualhuman on the desktop manually, allocate work to that
virtual human, and can analyze its dynamic burden. We
create a digital human model with the agricultural
attitude, the load of the subject in the movie taken
during the experiment, and calculate the lumbar burden.
The calculated value taken is a true value.

4. Verification experiment
In this chapter, we describe the two methods. There are
the experiments verification, the performanceverification of the state determination and accuracy
verification of the compression force of the lumbarintervertebral disc.

5. Results
In this chapter, we describe the result verification
method. There are the experiments verification, theperformance verification of the state determination andaccuracy verification of the compression force of thelumbar intervertebral disc.

4.1 Performance experiment of state
determination
In this section, we describe the experimental method of
the performance verification of the state determination
to estimate lumbar burden. The state determination isnecessary for estimating lumbar burden is performed
using the thigh inclination angle in each state measured
on each subject. This experiment was conducted 3 times
on 8 subjects. Subjects work in the order of standing,walking (6 steps), rotation (6 times), stumbling (1 time),
sitting (1 time). The interval between each motion is 3
seconds. The motion at that time was recorded with avideo camera, and the motion in the movie was taken asa true value. The analysis method compares the state
determination (smartphone value) determined by theapplication installed in the smartphone with the true
value, and calculates the identification rate when thedetermination is made. The calculation method of theidentification rate is shown in Eq. (3).
Identification rate (%) =

( )

( )

5.1 The result of experiment performanceexperiment of state determination
In this section, we describe the experimental results ofthe performance verification of state determination.
Table 1 indicates the action (Action) actually performedby the subject, the motion estimation (Estimation) by
the proposed method, and its identification rate.
Table1. Distinguish rate of state determination

From the Table 1 we can see that the overall
identification rate was 71.9 %. The identification rate ofwalking and sitting state judgment was bad, and the× 100 （3）© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Oita, Japan

552

Mizuki Maiguma, Hiroki Tamura, Koichi Tanno

identification rate of the rotation and crouching
judgment was good.

Fig. 5a indicates that the image which using the remotecontroller and Fig. 5b shows that the image which using
the mowing machine. In addition, a digital humanmodel created with the model shown in Fig. 5a and Fig.
5b as a model, and shown in Figs. 5c and 5d. The arrowin the image represents the direction of the force. At thetime of remote control operation, 0.5 kg force applied tothe hand in the direction of gravity, and it is necessary
to distribute the weight of 5.5 kg during mowing
operation. Therefore, we are applying 4 kg to theshoulder and 1.5 kg to the arm. The momentary lumbar
burden calculated under the above conditions, and theresult showed in table 2.

5.2 The result of verification experiment of
estimation accuracy of lumbar intervertebral
pressing force
In this section, we describe the experimental result of
the verification accuracy estimation of the lumbarintervertebral disc pressing force. The lumbar burden
estimation result during using mowing machineindicated in the fig.4. The vertical axis represents thelumbar burden (N), and the horizontal axis represents
time (min).

Table.2 comparison the result of smartphone and JACK

Although the error between smartphone measurement
result and the JACK measurement result which using
the remote controller was small, the error in the result of
the mowing machine was large.

Fig.4 lumbar burden estimation graph

6. Conclusion
From the results in Fig.4, the maximum value was
3278 N, the minimum value was 749 N, and the averagevalue was 1704 N. During using the automatic mowing
machine, the maximum value was 2820 N, theminimum value was 484 N, and the average value was
882 N.
5.2.1

In this paper, we have estimated the lumbar burden in
agricultural work using one smartphone. Based on the
result, the error between the smartphone value and Jackvalue using remote controller was small, and using
Mowing machine was large. The reason for this result ismeasuring lumbar intervertebral pressing force when thesubject lifts an object in the direction of gravity as the
proposed method. However, in the experiment of thistime, it considered that the method of calculating the
lumbar burden is different in the action of pulling theobject and the operation using the mowing machine.

Comparison with JACKIn comparison with the JACK, JACK measurement
used the Fig.5 as a digital human model.

7. References
Fig.5a automatic mowing machine

Fig.5b mowing machine

Fig.5c automatic mowing machine

Fig.5d mowing machine(JACK)(JACK)[1] Hamano Masafumi, Arai Ismail "Proposal of
Authentication Method for Smartphone Using
Accelerometer and Gyro Sensor" Research Report
Ubiquitous Computing System (UBI) ,17 ,March 2014

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Oita, Japan553

A Study on High Accuracy Stride Estimation on Smartphone Combining Acceleration Sensor
and Gyro Sensor
Shunta Nonaka†, Hiroki Tamura‡, Koichi Tanno

Faculty of Engineering, University of Miyazaki,
1-1,Gakuen-kibanadai-nishi, Miyazaki-shi, Japan
E-mail: hi14028@student.miyazaki-u.ac.jp†, htamura@cc.miyazaki-u.ac.jp‡

Abstract

The stride is an important parameter in human motion analysis and has been extensively studied bymany researchers. This paper focuses on smartphones that are popular in the world, regardless of themeasurement environment. In previous research, the stride estimation using the acceleration sensor or
the gyro sensor mounted on the smartphone had been studied. However, the stride estimation accuracy
has an error rate of about over 10 %. In this paper, we propose a highly accurate stride estimation
method using stride correction parameter obtained from cross section movement in addition toconventional sagittal surface stride using acceleration sensor and gyro sensor mounted on a smartphone.
Based on the testing results, the error rate of our proposed method with the stride estimated by the
Kinect sensor (RGB-D sensor) as the true value was about 6%. It makes possible to estimate the stride
with high accuracy.
Keywords: Locomotive Syndrome, Acceleration sensor, Gyro Sensor, Stride Estimation Method, Smartphone

estimation. We aim to build a system using smartphonesthat have accuracy like kinect sensors.

1. Introduction
The Japanese Orthopedic Association proposed theconcept of Locomotive Syndrome (Locomo) in 2007 [1].
Locomo is a state that the movement function isdeteriorating due to the failure of the exerciser.
Therefore, the author’s group has been developing asystem to estimate locomo from walking motion using
the kinect sensor as a safer and less burdensome
locomotive method for locomotion [2]. However, there
are about 47 million people currently including thereserve army of the locomo population in Japan. The
simple locomo evaluation system is necessary toevaluate the number of people's motor functions that isabout half of the Japanese population.
In this paper, we examine the evaluation of walking
motor function using the smartphone. The smartphone is
one of the daily sensors. A wearable device such as asmartphone using an acceleration sensor and/or gyro
sensor has weak accuracy in a specific environment
compared with a motion capture system using the kinect
sensor. Therefore, in this paper, we propose the highly
accurate stride estimation method to apply the locomo

2.

Previous Research 1

In the previous research, a gyro sensor of a smartphone
is mounted on the thigh. The hip joint angle calculated
from the integrated value of the angular velocity. The
angular velocity calculated by using the X-axis of the
gyro sensor. Assuming that the angular velocity of Xaxis is “gx” and the time when the hip joint angle
becomes maximally is "θmax", the hip joint angle used
for stride estimation is expressed by Eq. (1). The
integration interval was determined by the slope of thedevice calculated by the arc tangent of the gravitational
acceleration of the Y-axis and the Z-axis. When issuingin front of the right foot, the inclination of the device isfrom time to maximize "t0" (right foot rearmost) to theminimum to become time "t1" (landing in front rightmost). Further, if the length of the foot is "Lleg" and thestride is "L", the calculation formula of the stride isexpressed as Eq. (2).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan554

Shunta Nonaka, Hiroki Tamura, Koichi Tannno

θ

3.

=

L=

(1)

g ( )

|

(2)

|

Proposal method

Fig.2. Correction amount by rotation of waist.

The method we propose aims to estimate steps withhigh precision. Therefore, the calculation of theconventional sagittal plane method is added by themethod for the stride correction parameters obtained
from the cross section movement. To improve thelength of the step obtained from the sagittal plane,
pelvic rotation motion is measured with a gyro sensor tothe hip joint. It is because the smartphone is fixed to thethigh. In Fig.1, L1 and L3, we can see that the stride
lengths obtained by the acceleration sensor, and L2 is acorrection amount that takes into consideration therotation of the lumbar region obtained by the gyro
sensor. First, the formula for obtaining the stride length
using L1 and L3 is shown in Eq. (3). When the
difference between the direction of gravity and the angleof the thigh inclination is θ1 and θ2, then L1 and L3 aregiven by Eq. (3).

+

=

(sin

+sin

)

Then, L2 is obtained using Eq. (4). The r isrepresenting the width of the waist.

= 2 sin

2

( )

=

(4)
(5)

In Fig.2, “θ” is representing an angle due to therotation in the transverse plane, and it can be calculatedusing Eq. (5). The rotation of the cross section isexpressed by a value "gy" of Y-axis gyro sensor. The
integration interval can be determined by the values thatform the Y-axis and Z-axis of the acceleration sensor. It
is from the moment t0 at which the foot is pulling themost backward until the moment t1 when the foot is
stepped forward and landed, or from the moment oflanding to the moment when the foot reaches the back.
After calculating the rotation angle, the distancecalculated from the rotation of the lumbar region can becalculated by using the expression for obtaining thestring of the circle. Therefore, if the desired stride is L,
the stride is estimated by Eq. (6). We call the proposed
method 1 about the stride length of Eq. (6).

(3)

=
4.

+

+

(6)

How to judge walking

To build a system that calculates the stride fromwalking motion, it is necessary to estimate the state thata person is walking. Therefore, in this section, we
describe the walking judgment method of a person. To
determine the walking state, the trend of the smartphone
is calculated from the value of the accelerated sensor

Fig.1. One stride image.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

555

A Study on Highpeak to the negative peak is taken as the stride length (1step).

used. In this research, we installed the smartphone to thethigh as in Fig.3.
The tilt of the device can be used directly to track the
movement of the thighs. It is possible to capture theperiodic movements of the thigh that is walking. The
smartphone iPhone 4S (iOS 7.0) used for this research
has a function capable of detecting gravitational
acceleration. The inclination angle (rad) of the device
can be calculated from the detectable gravitational
acceleration on the Y axis and the Z axis by Eq. (7).

5.

The Y-axis value and the Z-axis acceleration sensor areused to determine the landing (hereinafter "acceleration
YZ"). The YZ acceleration value is not the accelerationof gravity used in the determination of the subject
acceleration produced by the subject. As shown in Fig.3,
when the smartphone is installed on the thighs, themagnitude of the acceleration YZ represents theacceleration in the sagittal plane. Fig. 5 shows that theslope of the smartphone and YZ acceleration during
walking.

Fig.3. Position to install smartphone.

，= arctan

Method of judging landing

Fig.5. Changes in the magnitude of YZ acceleration

Fig. 5 shows that it is quite difficult to find features
when landing. However, by disassembling into eachquadrant as shown in Fig.6, the feature amount at
landing can be measured. Fig. 7 shows that the
waveform of YZ acceleration when Fig.5 is divided into
four quadrants.

(7)

The example waveform when a person walking can beseen in Fig. 4.

y-axis

1st quadrantz-axis

2nd quadrant

smartphone
3rd quadrant

Fig. 4. Device tilt during walking.

The periodic waveform indicates walking is occurring,but it may not be determined that the subject is walking.
It is because there are individual differences such as
forward inclination and backward inclination in the
posture. When the inclination angle of the device iswithin ± 10 (degree), it is judged that the subject isstanding. Fig.4 "walking" showed that a positive value
represents a state when the foot is stepping forward anda negative value represents a state when the foot ispulled backward. Therefore, in this paper, from thepositive peak to the positive peak or from the negative

4th quadrant

Fig.6. Device axis and quadrant as seen from sagittal plane

Fig.7. Quadrant decomposition of the magnitude
of YZ acceleration© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

556

Shunta Nonaka, Hiroki Tamura, Koichi Tannno

Error Rate

than a method that only focuses on the sagittal plane.

Fig.8. YZ acceleration in the third quadrant only

In order to see what type of YZ acceleration is
occurring in what kind of motion, it can compare the tiltof the device tracking the movement of the foot with the
YZ acceleration in Fig.7. First, when the inclination of
the device is near from the peak of the downward
convex, it means YZ acceleration enters the third
quadrant. When the tilt of the device is negative, thefoot is in the state of being pulled to the back most, andthat state can be said to capture the acceleration of the
kicking leg. Next, the YZ acceleration takes the fourth
quadrant. It means that the fourth quadrant is oriented
mainly in the direction of walking, which shows that it
captures the movement in the process of stepping
forward. Next, take the first quadrant. It can be seen in
Fig.7 that the first quadrant means upward when steps
are taken. It means that the force in the direction ofwalking weakened. Then, after the first quadrant, movethe foot in the third quadrant direction representing thelower part of the thigh. This means the moment oflanding. After the third quadrant, which means landing,the waveform is slightly disturbed, but the device
vibrates after landing.
Therefore, the landing assessment is performed using
the first peak of the third quadrant that occurs
temporarily because the slope of the device is 10(degree) or more (shown in Fig. 8). The value of 10(degree) is to prevent peak misrecognition.

20%
18%
16%
14%
12%
10%
8%
6%
4%
2%
0%

Previous
Research 1
Average Error Rate
8.95%
Max Error Rate
18.42%

Proposal
Method 1
7.50%
16.40%

Proposal
Method 2
6.20%
13.90%

Fig. 9. Estimation accuracy of each strides estimation method.

7. Conclusions
In this paper, the movement of the sagittal during
walking motion that focuses on the movement of thecross-section, a method for estimating the step with highaccuracy studied.
The results of proposed method 1 have theimprovement rate of about 1.5% as an average error rate
and the improvement rate of about 2.0% as a maximum
error rate from previous research 1. The stride
estimation method of proposed method 1 is moresignificant than the previous research.
Furthermore, the results of proposed method 2 havethe improvement rate of about 2.8% as an average error
rate and the improvement rate of about 4.5% as a
maximum error rate from previous research 1.
By classifying walking based on the amount ofrotation of the lumbar region, it was possible to estimate
according to individual differences, and improvement of
versatility was confirmed. From these results, the error
rate of our proposed method 2 with the stride estimatedby Kinect sensor (RGB-D sensor) as the true value wasabout 6.2%, making it possible to estimate the stride
with high accuracy.

6. Experimental Results
In this section, we show the result of stride estimation.
We experimented with simultaneous measurements
between kinect and smartphone sensors. The number ofsubjects is 54 (Men: 12, Women: 42) at the age of 20 to
60. The proposed method 2 is a hybrid with the previous
research 1 and proposed method 1. This is based on the
results calculated from preliminary experiments, when
the waist rotation amount exceeds 12 cm, the stride had
a tendency to decrease error.
Fig. 9 can be seen that the proposed method 1 is better
than in previous studies 1 and 2. From this fact, it isassumed that the method of combining the sagittal andtransverse fields more accurately captures motion rather

8. References[1] New concept "Locomo (exerciser syndrome)"
The Japanese
Orthopedic Association
https://www.joa.or.jp/jp/public/locomo/ (accessed 2017-12-7)[2] Kouta Momose, etc.，“Basic study on evaluation method for patientwith locomotive syndrome using a RGB-D sensor”， IEICE TechnicalRep ， vol.113, no. 486, NLP2013-182, pp. 103-106, March, 2014,(Japanese)© The 2018 International Conference on Artificial Life and Robotics (ICAROB 2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

557

Development of Multi-Sensory Smart Objects Tracking Module for Mobile Robot Platforms
B.A.D.J.C.K. Basnayake, Y.W.R. Amarasinghe

Department of Mechanical Engineering, University of Moratuwa.
Katubedda, 10400, Sri Lanka
E-mail: janakabasnayake@gmail.com, ranamajp@gmail.com
www.mrt.ac.lk

Abstract
This paper presents a development of a Multi-Sensory Smart Objects Tracking Module (MSSOTM) using a MEMSbased IR thermal array sensor combined with ultrasonic and laser-based distance measuring system. The system
consists of two-axis motorized rotatable sensor platform and controlled by a DSP based centralized controller. Further,
MSSOTM is capable of identifying, localizing and sorting of stationary and moving objects as well. The implemented
system was tested and validated using Robotino® mobile robot platform.
Keywords: IR thermal imaging, obstacle detection and segmentation, laser range finder, mobile robots

According to the literature, most of the researchers
focusing on develop obstacle identifications andenvironment mapping systems for mobile robot
navigations6. Objects separation such as live objects(humans, animals etc.) and positioning are difficultcompared with stationary obstacles identification,
because of their behaviors. But it is important for robotic
platforms which operate in living premises in order toperform collision-free navigation and localization
features. This research presents the development of
MSSOTM which capable of detecting, localizing andseparating of movable and stationary living object in acertain range. At the present stage, the system hasdeveloped to identify the live objects and control amobile robotic platform to follow the target. Developed
system will use in navigation, human following andobstacles avoiding applications in mobile robot systems.

1. Introduction
In mobile robot systems, the ability to navigate in its
environment is important. Often, the surrounding
environment is dynamic and unknown, so obstacle
detection and mapping are essential for mobile robot
platforms which are operating in a dynamic environment.
Ultrasonic, laser and IR technologies based non-contact
obstacle identification system and distance measurement
systems were used for robotics navigation systems. Dirk
Bank introduced a novel ultrasonic sensing system forautonomous mobile systems using a wide-angled
ultrasonic transducers array1. His proposed system wascapable to obtain substantial information about theobjects in the environment. Hyunwoong Park et al. haspresented a novel rotating IR range sensor system for amobile robot platforms2. Radially mounted rotatable IR
range sensors used to detect objects and map theenvironment. Some other researchers have proposed
Laser Detection and Ranging (LADAR) sensors for
tracking obstacles3, 4, 5. Different types of LADARs whichhave been used for develop obstacle detectors arecomparatively precise than the ultrasonic sensors. But
they are not capable of tracking live objects.

2. Proposed System and Working Principle of
MSSOTM
The proposed system mainly consists of none contact
Micro-Electro-Mechanical-Systems (MEMS) - based IR
thermal array sensor and range finding sensors which are© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

558

B.A.D.J.C.K. Basnayake, Y.W.R. Amarasinghe

mounted on a two axis rotatable platform as shown in Fig.
1. Non-contact thermal array sensor used to identify theheated objects (live objects) and their directions byanalyzing the thermal profile of the surfaces in within thesensor Field of View (FoV). This will generate important
information about the presence of live objects to themobile robot. Distances to the objects are anotherparameter essential to the mobile robot to maintaincollision free proper navigation in the action. Therefore
range finding sensors are included in the system which isbased on ultrasonic long distance ranging sensor andshort distance laser-ranging sensor mounted on the same
platform. Independently controllable two axes rotatable
platform provides facility to orient sensors to required
directions and it is facilitated to expand sensing coverageof the system. Combining these two sensory systems, thesystem can provide accurate directional and distanceinformation about the presence of the objects andobstacles as well. The designed system has a Digital
Signal Processor (DSP) as the central controller and awireless communication link to communicate with acomputer. And also it has an analog and digital inputoutput module which can be used for interface with amobile robot platform as shown in Fig. 2.

Rotatearoundaxis 1

Thermal arraysensor(8*1 array)

Ultrasoniclong distance
ranging sensor

2

2

UART

IC

IC

Sensor Interface controller(8 Bit Micro controller)
UART
Central controller
16 Bit-120MHzDSPIsolated analog and digital
input/output interface forrobot platform
UART

Bluetooth

UART
Motion controller(8 Bit Microcontroller)
Motor 1
Axis 1

Motor 2
Axis 2Motor 3Axis 2

2

IC
Angle Sensor(Accelerometer)

Fig. 2. Block diagram of developed MSSOTM

2.1. Development of IR thermal image based live
objects identification system

A

Since infrared radiation is emitted by all objects with atemperature above absolute zero according to the black
body radiation law, thermal imaging makes it possible tosee detect heat objects with or without visible
illumination. When viewed through a thermal detector,
warm objects stand out well against cooler backgrounds.

B

2.1.1. MEMS IR thermal sensor

C

MEMS IR thermal array sensors are precise tiny noncontact sensors which can be used to measure the surface
temperatures of the objects in different application areas.
This system is based on eight by one pixel array sensor(Omron D6T) which is having four frames per second
data acquisition speed and 62.80, six degree FoV along
the X and Y directions respectively. Measurement rangeof the selected sensor is five to 50 0C and its detecting
range is up to six meters. The sensor has provided eighttemperature values related to the array positions and one
for the sensor body temperature through an I2C serial bus.

Top View
Rotatearoundaxis 2

Laser shortdistance
ranging sensor

Sensor panel

Side View

2.1.2. Live object detecting algorithm

A – Detection range of IR thermal sensor
B – Detection range of ultrasonic ranging sensor
C – Detection range of Laser ranging sensor

The function of the detecting algorithm compares thesurface temperatures captured by the thermal sensor witha reference temperature7. The reference temperature is a
dynamic value which is related to the ambient
temperature at any location. The thermal sensor has

Fig. 1. Layout of installation arrangement and detection
ranges of sensors.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan559

Development of Multi-Sensory Smart

2.2. Ultrasonic and laser combined distancemeasurement system

Pixel [0] temp:
Sensor bodytemp:
Offset temp:

(a)

(b)

The distance measurement system is developed using
two types of range sensors for long distance and shortdistance measurement. Long distance range sensor isbased on the integrated ultrasonic probe (URM07) whichis having inbuilt temperature compensation. The sensor
is working up to 100 mm to 5000 mm of range with 60°
of detecting angle. It has one centimeter of resolution and
one percent of accuracy. The sensor can be read througha digital serial interface. VL53L0X is precise Time-ofFlight (ToF) based laser-ranging sensor which is having
two-meter detecting range and one millimeter ofresolution with a narrow detecting angle compared to theultrasonic range sensor. The importance of using
LADAR is, it is provision to take precise measurementson near objects and identify the motions as well.

Detectionof temperaturerise at pixel [0]Fig. 3. (a) Omron D6T IR Thermal array sensor (b) Block
diagram of heat object detecting algorithm

features to measure the inside temperature of the sensor
body. The system is taking this sensor body temperature
as ambient temperature and modifies it by adding someoffset value to generate reference temperature to thecomparators. If the surface temperature at any pixel
element is greater than the reference temperature value,then it is identified as a live object presence in a specificdirection as shown in Fig. 3. While scanning theenvironment for objects, after detecting the temperaturerise occurred in the environment by any of sensor pixel,
sensor platform will rotate around the axis 1 and take thedetected direction to the middle of thermal sensor FoV.
At this time distance measuring sensors are also in thesame direction with the thermal sensor around the axis 1.
While measuring surface temperature by thermal sensor
rotating around axis 2, distance measurement sensors
also take the distance to the heated object by rotating inthe same axis. This is done by two axes rotatable platform
with independently rotatable two platforms around theaxis 2. After measuring distance profile and thermal
profile in the same direction as shown in Fig. 4. This
information is fed to the sorting algorithm running on the
DSP in order to clarify the detected heated object as ahuman, animal or other object presence on the premises.

2.3. Two-axis rotatable sensor platform with selftilt compensation
Two-axis rotatable platform is developed to position
sensors to required angles and directions. Both axes are
driven by stepper motors with optical reference limit
switches. For the 2nd axis, two individual motors are used
to rotate two individual platforms. One is used for the IR
thermal sensor and the other one is for distancemeasuring sensors. While it’s operating, the central
controller can change the angles of the individual
platforms mounted on 2nd axis reference to the ground,which is done by using accelerometer-based feedback
control system developed on the motion controller.
3. Testing and Validation of MSSOTM
The developed MSSOTM as shown in Fig. 5. was tested
and validated using Robotino® robot platform.

Axis 2Heat Object

Axis 1aSensorpanelb

IR thermal
sensorRotaryplatforma - Distance and thermal profile 1
b - Distance and thermal profile 2
Ɵ – Measuring angle reference to ground
D – Calculated

Ultrasonicrange
sensorLaser range
sensorCentralcontroller

Fig. 4. Operation of measuring system and measuring
parameters

Fig. 5. Developed prototype of MSSOTM© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

560

B.A.D.J.C.K. Basnayake, Y.W.R. Amarasinghe

Robotino® is a flexible mobile robot platform forresearch and education manufactured by Festo.
Robotino® has inbuilt radially arrange IR ranging sensor
array which can be used for obstacle identification and it
has a tactile bumper sensor attached to the bottom edgeof the chassis which can be used for the safety. And also
it has the facility to connect external devices through theinbuilt input/output (I/O) interface.

(a)
(b)
(c)
Fig. 7. (a) Identify a person from the front direction of robot
platform. (b) Identify the motion of the person and move therobot along the +X direction to follow the person (c) Identify aperson from a different angle and change the moving directionof the robot.

3.1. Testing on developed experimental setup
using Robotino®

4. Conclusion

In this experiment, developed MSSOTM wasprogrammed to human following application and control
commands are taken from the device interface. The
prototype was mounted on the Robotino® and connects
the I/O lines as shown in Fig. 6. Robotino has
programmed using Robotino® View to read the defined
I/Os and perform the given actions. The deferent objects
with various shapes and dimensions such as chairs, walls,
and computers as the stationary objects and a person as alive object were scanned by MSSOTM. According to
objects locations, MSSOTM was generated moving
commands to the Robotino® for follow the person
avoiding the other objects as considering the obstacles asshown in Fig.7.
Move to +X dir
Move to -X dir
Move to +Y dir
MSSOTM
Deviceinterface

Move to - Y dir
Rotate to CW dir
Rotate to CCW dir
Speed Reference
Acknowledgement
(a)

DI-1
DI-2

In this research, a multi-sensory smart obstacles
identification system was proposed as an external sensor
module for mobile robot platforms. The prototype of the
proposed design was developed and tested with
Robotino® mobile robot platform. Future work of thisresearch includes improvement on control andidentification algorithms for working in the realenvironment with enhanced performance.
References
1. D. Bank, A novel ultrasonic sensing system forautonomous mobile systems, IEEE Sensors Journal 2 (6)(2002) 597-606.
2. H. Park, S. Baek, S. Lee, IR sensor array for a mobile robot,
in: Proceedings, 2005 IEEE/ASME International
Conference on Advanced Intelligent Mechatronics, 2005,
pp. 928
3. M. Kise, Q. Zhang, N. Noguchi, AN OBSTACLEIDENTIFICATION ALGORITHM FOR A LASERRANGE FINDER BASED OBSTACLE DETECTOR,
Transactions of the ASAE 48 (3) (2005) 1269-1278.
4. H. Kim, M. Miwa, J. Shim, An obstacle avoidance systemof an unmanned aerial vehicle using a laser range finder,
Journal of the Korean Society of Marine Engineering 37(7) (2013) 737-742.
5. T. T. Hoang, D. T. Hiep, P. M. Duong, N. T. T. Van, B. G.
Duong, T. Q. Vinh, Proposal of algorithms for navigationand obstacles avoidance of autonomous mobile robot, in:2013 IEEE 8th Conference on Industrial Electronics andApplications (ICIEA), 2013, pp. 1308-1313.
6. T.-J. Lee, D.-H. Yi, D.-I. Cho, A Monocular Vision
Sensor-Based Obstacle Detection Algorithm forAutonomous Robots, Sensors 16 (3) (2016) 311
7. Basnayake, B. A. D. J. C. K., Amarasinghe, Y. W. R.,
Attalage, R. A., & Jayasekara, A. G. B. P. Occupancy
identification based energy efficient Illuminancecontroller with improved visual comfort in buildings. In
Engineering Research Conference (MERCon), 2017
Moratuwa (pp. 304-309). IEEE.

Robotino I/OInterface

DI-3
DI-4
DI-5
DI-6
AI-1
DO-1

MSSOTM
Robotino®
mobile robot
platform(b)
Fig. 6.(a) Layout of the prototype interface with Robotino®
(b) Experiment setup© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

561

Design and Development of a Conductive Polymer Based 3D - Printed Tactile Sensor
with Square Type Spring Structure
W.H.P. Sampath, A.H.T.E. De Silva, T.A.U. Roshan, Y.W.R. Amarasinghe

Department of Mechanical Engineering, University of Moratuwa.
Katubedda, 10400, Sri Lanka
E-mail: peshan@ieee.org, ahteranga@gmail.com, tauroshan@gmail.com, ranamajp@gmail.com
www.mrt.ac.lk

Abstract
In this research, a novel tactile sensor (with 50mm×50mm×56mm dimensions) has been developed using a silicone
rubber based conductive polymer sensing element array incorporated to a novel 3D printed square type spring
structure. Initial characterisation has been completed for the sensor. The structure stated in this paper has been
designed for force scaling purpose and it’s numerically analysed prior to the fabrication to avoid any mechanicalfailures and ABS Plus and Polylactic acid were used in the 3D printing fabrication.
Keywords: Square type spring structures, Conductive Polymers, 3D Printing,the performance and the base material is a Silicone
Rubber which has enhancements by silica and carbon
black, with Silane as the coupling agent3.
The spring structure stated in this paper has been
designed for force scaling purpose and numericallyanalyzed using COMSOL Multiphysics prior to thefabrication. Circuitry for embedded electrodes was also
developed in this research with a wirelesscommunication between the sensor assembly and thedeveloped user interface which allows to use the sensor
as a plug and play device

1. Introduction
Tactile sensors are being used to capture physical world
data through tactile parameters with the assistance ofdifferent sensing principles. The advancement in thefields of robotics and automation requires further
attention in the development of tactile sensors. Humanskin plays a vital point as a tactile sensor1 as it allows tosense number of tactile parameters simultaneously while
conductive polymer based piezoresistive tactile sensors
allows the researchers to gain the flexible quality which
is prevailed in human skin. However, prevailing tactile
sensors are yet to identify more than two parameterssimultaneously.
In this research, design, development and testing of anovel tactile sensor with the dimensions of
50mm×50mm×56mm based on a 2×2 conductive
polymer based sensing element array incorporated to asquare type spring structure. 3D printing technology isutilized to fabricate this spring structure. Sensing element
described under this research is highly customizable in

2. Proposed design and working principle
When considering force sensors, it’s essential to define aworking range and controlling the working range can bedone by overseeing the deflection applied to the sensing
element3. With the introduction of an external square© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan562

W.H.P. Sampath, A.H.T.E. De Silva, T.A.U. Roshan, Y.W.R. Amarasinghe

type spring structure (described under Fig. 1) to sensing
elements, working range can be easily altered.

Fig. 1. Square type spring structure (a) A two-layered saw tooth
shape is used (b) Fillet is to enhance the mechanical strength (c)Single springelement during the load testing is obtained as 2mm forthe expansion of its range. ABS Plus which has
maximum tensile strength of 3.6x107 N/m2 is used withthe 3D printing fabrication and it’s analysed as a
maximum von-Misses stress of 2.79x107 N/m2 (Fig. 3(b))
in the simplified assembly.
Considering a series and parallel spring system (shown

Fig. 2. Quarter sectional view of the proposed design

The spring structure is designed to facilitate a 2x2 arrayof sensing elements and designed for three separate parts.
Furthermore, the proposed sensing structure consideredfollowing design features.
3 Degrees of Freedom due to the inclusion of
2×2 sensor array.
The initial compression for sensing elements toinitiate conductive paths, screw thread typemechanical fastening between the spring andinner support is used.
An overload protection to the sensor structure
is given by the horizontal and vertical clearance
between guide pin and the inner support.
Structure facilitates electrodes, routing forwires and a compartment for internal circuitry.

Fig. 3. Structural Analysis (a) Simplified model of the spring
structure (b) Structural analysis carried out using COMSOLMultiphysics.

in Fig. 4), an analytical model for total stiffness - K(Equation 1) also derived for the stiffness of the sensor
and the stiffness of the pill and structure under a force
component was calculated as 20270 N/m and 16258 N/m

3. Design and Simulation of square type spring
structure
3.1. Development of square type spring structure
A two-layered saw tooth shape (Fig. 2(a)) is featured in asingle square type spring and a fillet radius of R (shownin Fig. 3(b)) is introduced to enhance the mechanicalstrength of the structure.
3.2. Structural analysis and design constraints ofsquare type spring structure
A Finite Element Analysis (FEA) was performed using
COMSOL Multiphysics for the simplified structure
shown in Fig. 3(a) by compressing the combination ofspring structure and sensing elements (simulated as a
virtual spring), along the centre axis by 2 mm prescribed
displacement. Maximal deformation of the sensing

Fig. 4. Simplified to a series and parallel spring system

respectively. The clearance between the guide pin andinner support also derived using Fig. 5. for a tilt angle of© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

563

Design and Development ofθ for a force applied on the top surface which reached the
maximum allowable deflection of the guided pin.
K

Ks
m
n

f

Kp

A mechanical moulding method is used to develop thesensing element while the mould described under Fig.
6(a) is designed into three layers to ease the removal
process of sensing element from the mould.

(1)Fig. 6. Sensing element fabrication (a) Three-layer mould used
for the fabrication (b) Fabricated sensing pills

The mould is capable of developing sensing elementswith different geometries and with different thickness
ranges (Fig. 6(b)) in one batch of production. A key
feature of this production process is sensing elements canbe developed according to the required geometry and sizewith different performance levels.

Fig. 5. Clearance deceided as a overload protection to thestructure

Consider distance AB,ℎ = ( − cos )⁄tan
Vertical clearance y is given,= − + ( − cos )⁄tan
Horizontal clearance x is given,= ( − )⁄2

4.2. Fabrication of sensing structure(2)

Three parts shown under Fig. 6 is fabricated using 3D
printing technology as it eases the fabrication process ofcomplex geometries. Square type spring structure (Fig.

(3)
(4)

4. Fabrication of the Tactile sensor

Fabrication of the tactile sensor was carried out under
two stages. First, the fabrication of sensing element andthen the sensing structure.

Fig. 7. 3D Printed sensor parts (a) Bottom pad (b) Inner support(c) Square type spring structure

4.1. Fabrication of conductive polymer based
sensing elements

7(c)) is made from ABS Plus with 100% infill ratio while
PLA with 75% infill ratio is used for remaining two parts.

Conductive polymer based sensing element used under
this research is mainly based on Room Temperature
Vulcanization (RTV) Silicone Rubber. For theenrichment of the performance of the polymer, inclusionof Nano-materials such as Super conductive carbon black- KETJENBLACK EC-600JD and Nano Silica (SiO2) iscarried out and Si-69 Silane Coupling agent added to themixer as a boding element.

5. Assembling, Packaging and Testing
5.1. Assembling and packaging
Fabricated sensing structure and sensing elements havebeen packaged as shown in Fig. 8 using embedded fullelectrodes configuration with a common electrode at thetop, to the structure.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan564

W.H.P. Sampath, A.H.T.E. De Silva, T.A.U. Roshan, Y.W.R. Amarasinghe

Fig. 10. Output of a sensing element vs axially applied
force before the signal conditioning

Fig. 8. Packaged sensor (a) Electrode configuration of thesensor (b) Packaged sensor

5.4. Signal Conditioning, Data Acquisition andCommunication

5.2. Testing and characterisation
Sensing elements are tested for its characterisation using
a static load test. The basic characteristics such as
conductivity, force range, hysteresis and non-linearity
were found out with an apparatus shown in Fig. 8 and Fig.
9 shows the hysteresis curve for a sensing element.

Signal conditioning is carried out to convert the nonlinear sensor output data to a linear combination with useof ATmega328 development board based signal
conditioning circuitry (Fig. 11). Bluetooth uses tocommunicate with the GUI available at the users’ device.

Fig. 8. Testing apparatus

Fig.
11. Data acquisition, signal
communication

conditioning and6. Conclusion
This square type spring structure minimized deflection of
the sensing element while it allows the expansion to theoperation range. This sensor designed to overcome theinherent issues in conductive polymer based sensing
elements using an internal signal conditioning circuitry.

Fig. 9. Resistance change in a sensing elements vs applied load
plot

References
1. T. D. I. Udayanga et al, “Development of a quantum
tunnelling composite based 1-dof tactile sensor,
International Journal of Advanced Information Scienceand Technology (IJAIST).
2. M. Tiwana et al, "A review of tactile sensing technologies
with applications in biomedical engineering", Sensors andActuators A: Physical, vol. 179, pp. 17-31, 2012.
3. A H.T.E. De Silva et al, Design and development of anovel 3d printed 1-dof tactile sensors with conductive
polymer based sensing element, in: SENSORS, 2016 IEEE

5.3. Characterisation of Sensor
Assembled sensor is checked with static load test. Output
obtained from a single sensing element under an axial
loading condition is shown under Fig. 10.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

565

Design and Development of a Shape Memory Alloy Spring Actuated Gripper for Minimally
Invasive Surgeries
T.A.U. Roshan, Y.W.R. Amarasinghe

Department of Mechanical Engineering, University of Moratuwa.
Katubedda, 10400, Sri Lanka
N.W.N. Dayananda

Department of Electronic and Telecommunication Engineering, University of Moratuwa.
Katubedda, 10400, Sri Lanka
E-mail: tauroshan@gmail.com, ranamajp@gmail.com, nuwan@uom.lk
www.mrt.ac.lk

Abstract
A Shape Memory Alloy based Spring Actuated Gripper (SMASAG) has been designed and developed for Minimally
Invasive Surgeries. Further, the research is consist of design and development of an SMA spring and actuator force
characterization by a developed testing apparatus. The gripping motion is achieved by a compact mechanism.
Keywords: shape memory alloy, minimally invasive surgeries, gripper, 3D printing.

prototypes which are driven by SMA actuators6 withdifferent techniques. According to the literature in mostcases, activation method was direct heating (Joule
heating) of SMA material. In this research, the SMA
material was activated by a fluid flow which was undercontrolled temperature and flow rate corresponding to theactivation requirements. Also designed a compact
mechanism for transferring the force generated by theactuator to the gripping jaws with analyzing theKinematics. Because it is very much important in thesetype of tools in order to enhance the performance. Further
3D printing was used to develop the prototype forvalidation.

1. Introduction
Minimally Invasive Surgery (MIS) is an emerging andhighly favorable surgery procedure for similar open
surgeries. It has been developing since the late 80s andwidely accepted due to several advantages overconventional surgeries.1 The equipment used to performthese MIS are ranging from simple tools to high-end
remote-controlled devices with the observation of thesurgical field through a scope.2 The surgicalmanipulations are performed in a confined space and theminiaturization of actuators in such tools is a requirement.
There are several types of actuators which giveadvantages as well as disadvantages over miniaturization.
3
From those, Shape Memory Alloy (SMA) can be taken
as a possible actuation material.4 The best known SMA
for biomedical applications is called NiTiNOL. These
NiTiNOL alloys possess several advantages.5 which aremost important in actuator designs for MIS tools. Overthe years several research groups have developed gripper

2. Design of SMASAG and Working Principle
2.1. Working principle of SMASAG
The proposed gripper composed of several components
and the functionality and the importance of each andevery component is described as follow,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan566

T.A.U. Roshan, Y.W.R. Amarasinghe, N.W.N. Dayananda.

bore providing the smooth movement with the preventionof leakages and it is 33.7 mm in length. The End Nutfacilitates the final assembly of the actuator and has
15 mm across flats and threaded to the Cylinder by an
M12 standard thread. There are two mounts for SMA
spring attachment and both of them are threaded to the
End Nut and the Piston by M6 standard thread.
2.2.2.

The design of the SMA spring is related to the SMA shapesetting operation and there, a fixture is designed toconstrain the spring shape.

Fig. 1. Proposed gripper assembly

The gripper is actuated by a Linear actuator structure andthat is designed as a Piston/Cylinder assembly. It has an
End Nut which facilitates easy disassembly. Both End
Nut and the Piston have Mounts where the SMA spring is
attached inside the cylinder. There are openings in the
Cylinder body to pass water for heat and cool SMA
Spring actuator. The stroke length of the actuator was
limited to five millimeters.
The linear motion of the piston is converted to an angularmotion by a Link which is hinged to the Base structure
and consists of two slots that limit the motion of two Pins
connected to the Piston and the Driver Jaw. The slots aredesigned in a manner where there are no sudden lockups
occur with pins.
The rotational motion of the Link is transferred to the
Driver jaw by the reaction force between the pin-slot joint
and makes the Jaw to rotate around a hinge on the Base.
In order to achieve compactness, the Gripper pair wasconnected via a Sector Gear pair. The main two actuation
elements can be considered as the SMA spring and thesteel torsion spring which are supporting for the opening
and closing of gripper jaws respectively. The
development and the characterization of SMA spring andthe torsion spring is elaborated in the following
subsections.

Fig. 2. SMA shape setting fixture design

According to the design dimensions in Fig. 2.
(in millimeters) of this fixture, the SMA spring has four
turns with two point two four pitch and the spring
diameter is eight millimeters.
2.2.3.

Design of torsion springThis provides the closing action of the Gripper Jaws.
Choosing a High spring constant may cause excessive
forces on the object and partial opening of gripper jaws.
Low spring constant may cause partial gripping.
Following equations will facilitate to design the torsional
spring with correct spring constant.

C

2

L10

L1x

L10 DT

2

2

L1x

LCx

FJS

The design of SMASAG is followed by the designing ofactuator supporting structure or the outer packaging of theactuator, SMA based spring and the torsional spring.

RLJ

(1)

rA
FSMA cosrG

RLJ

2.2. Design of SMASAG

2.2.1.

Design of SMA spring

2 L10 L1x cos

LC 0

C

LH cos

FG LG
LCx

NE

L10 L20
3 DT

NA

NB

NE
4

Design of outer packaging of the actuator

FSJ

(6)
(7)
(8)

dT 4
10.8DT N A 2

(9)

FSJ

FJS

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

567

(5)

dT
10.8 DT N A

k per turnThe outer packaging or the structure of the actuatorconsists of a Cylinder which has an outer diameter of 20mm, a bore of 10 mm and it is 50 mm in length. Also, thePiston has a diameter which is matched to the cylinder

(2)
(3)
(4)

(10)

Design and Development of

Fixture need to be used in order to get them in the correct
Spring shape while shape setting at high temperature.

Here
and
are the Actual, Body and Effective
turns of the torsional spring.
Fig. 3. Presents the parameters used in the above
equations. Eq. (1) to Eq. (5) can be used to calculate theforce exerted on the torsional spring due to the actuation
force by the SMA spring and Eq. (6) to Eq. (10) related tothe calculation of a number of turns of the torsion spring.

Fig. 4. Fabricated fixture

As the NiTiNOL wire tend to regain its trained shapewhile increasing the temperature above Autunitic final
temperature, it is very much important to use fixtures (Fig.
4.) when inducing newly defined shape into the material.
So the fixture was placed in a Muffle furnace fortemperatures and aging times indicated in Table 1. andwater quenched to get the final spring shape.(Fig. 5.)
Table 1. NiTiNOL wire shape setting details
Wire
Diameter(mm)

Shape Setting
Temperature
(0C)

Aging
Time
(mins)

Transformation
Temperature
(0C)

1.0
1.0

500
500

25
30

60 ± 580 ± 5

Fig. 5. Shape set NiTiNOL spring

3.2. Fabrication of outer packaging and the
GripperFig. 3. Freeparameters

body

diagramand

The fabricated SMASAG is depicted in Fig. 6. The Basestructure and the Gripper jaws were made out of ABS Plusmaterial with the use of 3D printing technology.

calculation

3. Fabrication of SMASAG andCharacterization of SMA ActuatorForce

3.1. Fabrication of SMA actuator
The SMA material which was used to fabricate the spring
was a commercially available material called NiTiNOL
which is composed of Ni - 55%; H - 0.001%; O - 0.05%;
N - 0.001%; C - 0.05%; Ti – Balance. The obtained
NiTiNOL material was in the form of straight wires withdifferent diameters and transformation temperatures. So a

Fig. 6. Fabricated SMASAG© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan568

T.A.U. Roshan, Y.W.R. Amarasinghe, N.W.N. Dayananda.

Simultaneously the temperature of the water wasmeasured through a four and half true RMS Digital
Multimeter which is compatible with thermocouple
probes. Those obtained values were plotted and Fig.8
shows the variation of force with the temperature for3.3. Force characterization of SMA spring
After shape setting the NiTiNOL wire into a spring shape,
it is important to have an understanding of the amount offorce that can be exerted through it on the mechanism. So
an apparatus was developed to measure the force that canbe achieved when springs are constrained to recover totheir initial shape.

Fig. 8. Force measurement using the apparatus

NiTiNOL springs which has same coil and wire diameterwith different transformation temperatures.(Table 1.)
Conclusion
In this work, an SMA spring actuator based gripper was
designed and fabricated. The work carried out to achievethat task includes, force characterization of a NiTiNOL
spring. The results show that, as the transformation
temperature reduces with same wire diameter, therequired force could be achieved quickly as all the springs
were tested under same conditions. Future works include
testing and Gripper force analysis of the SMASAG.

Fig. 7. (a) Force measuring apparatus (b) Calibrating by
known weights (c) NiTiNOL spring attachment

The apparatus shown in Fig. 7 (a). consists of a Fullbridge load cell and a container which was used toimmerse the Spring in water while one end connecting to
the base of the container rigidly and the other end
connecting to the load cell attachment. In the first placeload cell was calibrated by known weights as in Fig. 7 (b).
and after that NiTiNOL spring was attached to measure
the force as in Fig. 7 (c). Finally, the container was filledwith water and gradually increased the temperature by aheater.
The Full bridge load cell was connected to a 24-bit loadcell amplifier (HX711). And the HX711 was connected toa microcontroller (ATmega 2560) based development
board in order to acquire data. After obtaining raw data
values from the amplifier, the development board was
sending those raw data to a computer via serial
communication and all the data processing tasks wereconducted from a LabVIEW generated program. Also, aGraphical User Interface (GUI) was developed to plot anddisplay data.

References
1.
2.

3.

4.
5.
6.

C. Song, “History and Current Situation of Shape Memory
Alloys Devices for Minimally Invasive Surgery,” OpenMed. Devices J., vol. 2, pp. 24–31, 2010.
R. H. Petersen, “Video-assisted thoracoscopic
thymectomy using 5-mm ports and carbon dioxide
insufflation,” Ann. Cardiothorac. Surg., vol. 5, no. 1, pp.
51–55, Jan. 2016.
J. Mohd Jani, M. Leary, A. Subic, and M. A. Gibson, “A
review of shape memory alloy research, applications andopportunities,” Mater. Des. 1980-2015, vol. 56, pp. 1078–
1113, Apr. 2014.
D. Mantovani, “Shape memory alloys: Properties andbiomedical applications,” JOM, vol. 52, no. 10, pp. 36–44,
Oct. 2000.
D. Stockel and A. Melzer, “The Use of NiTi Alloys for
Surgical Instruments,” Conflu. Med., 1995.
Shaoze Yan, Xiajie Liu, Feng Xu, and Jinhui Wang, “A
Gripper Actuated by a Pair of Differential SMA Springs,”
J. Intell. Mater. Syst. Struct., vol. 18, no. 5, pp. 459–466,
May 2007.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

569

Design and Development of Quantum Tunneling Composite based Tactile Sensors
T.D.I. Udayanga, D.A.M.R. Fernando, H.L.P.L. Chaturanga, B.A.D.J.C.K. Basnayake and Y.W.R. Amarasinghe

Department of Mechanical Engineering, University of Moratuwa
Katubadda, Sri Lanka.
t.i.udayanga@ieee.org
www.mrt.ac.lk

Abstract

Tactile sensor is a device which acquire information about physical entities such as pressure, force, texture,etc., with the aid of physical touch and convert them in to useful electrical signals. With the advancementsof robotics and mechatronics, currently, there is a great demand for tactile sensors so as to use them invarious applications to acquire tactile data. In this study, a miniaturized novel enclosed tactile sensor was
designed and developed using Quantum Tunneling Composite (QTCTM), which is a conductive polymer
composite, as the sensing element. An enclosed novel structure was proposed so that the sensing element
and spring will be omitted from the environmental effects. Proposed sensor structure was analysed andmanufactured. Experiments were carried out and results shows that the sensitivity of this developed tactile
sensor to be 0.02 V/N and repeatability of ±3 N.
Keywords: Sensors, Tactile Sensing, Quantum Tunneling Composite, FEA.

are identified which can be incorporated with tactile
sensors such as Piezoresistive,10, 11 Piezoelectric12, 13,
Optical,14Capacitive,15,16
Conductive
Polymer/Composite.17, 18 Each of these sensing principles
have their own advantages and disadvantages.1, 19 Hence,use of them in applications should be consideredaccording to the performance required.
At the early stage of this research, as published in Ref.
20, a tactile sensor was developed. But further in to theresearch identified issues associated with that design
such as poor maintenance of geometrical anddimensional tolerances, low dimensional accuracy andprecision associated with the manufacturing technique.
Due to these manufacturing errors, alignment Problems
Occurred during assembling stage and offsetting errors
occurred while transferring force to sensing element.
Also due to the insufficient physical contact between
electrode and the sensing element, noise was added to the
output signal and it might has been one of a cause for thehysteresis error of the developed sensor.20 So in this

1. Introduction
Tactile sensors is a device which uses tactile sensing toacquire data from the physical environment. Tactile
Sensing is measuring tactile parameters such astemperature, vibration, texture, shape, force, pressure
with the aid of physical touch. Tactile parameters may
often include, temperature, vibration, softness, texture,shape, composition shear and normal force.1, 2 Even
though tactile sensing came to the attention of researchers
in the 1970s, tactile sensors haven’t been developed
much comparing with the non-contact sensors.3, 4 But
with the advancements of robotics and automation,
currently, there is a need of tactile feedback systems with
optimum performance.5 Application areas of tactile
sensing has been expanded as new fields of applications
has emerged recently regarding tactile sensing.
Robotics,6 medical,7,8 entertainment and communication9
are some of these identified application areas of tactile
sensors. Over the past decades, many sensing principles© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

570

T.D.I. Udayanga, D.A.M.R. Fernando, H.L.P.L. Chaturanga, B.A.D.J.C.K. Basnayake and Y.W.R. Amarasinghe

(a)

(b)

X

Top CoverBase
SpringQTCTM Pills
Electrode
Base Cap
Piston Bolt
Fig. 1. Propsed Concpetual Of the Novel Enclosed Tactile Sensor; (a) Isometric View , (b) Sectional Front Elevation

paper a novel enclosed design was discussed, which was
designed to overcome the issues identified, to add
overload protection characteristics and to minimize theenvironmental effect to the sensing element.
Following chapters will discuss about the structuraldesign of the proposed sensor, sensing principle andFinite Element Analysis of structure and experiments andresults conducted regarding the developed sensor.

3. Analysis and Simulation of Proposed
Structure
To validate the proposed sensor structure, Finite Element
Analysis (FEA) was performed solely for validating the
proposed sensor structure and to verify whether it follows
the proposed working principle at development of 1-DOF
Tactile Sensor.20 Analysis was performed using
COMSOL Multiphysics software, which is a FEA
software.
Fig. 2 shows the results of the FEA analysis for sensor
structure. According to the results shown in Fig. 2. (c),proposed structure with sensing element shows a linear
relationship between the force applied and the structural
displacement. This validate that the proposed structure is
working on the working principle described at Ref. 20.

2. Proposed Structural Design
Conceptual Design and the components of the designed
novel enclosed sensor structure is shown in Fig. 1. In thissensor structure, force is applied to the top cover and thenit is transferred to the sensing element while scaling
down the force through the spring. In this design, space
between the top cover and the base (dimension X asshown in Fig. 1) is controlled such that it will provide an
overload protection to the sensing element. Overload
condition can be set by adjusting the piston bolt and thebase cap in the structure. This will affect the force rangeof the sensor, as this controls the allowable maximum
displacement for the QTCTM pill. Hence, with this newly
added feature now force range of the sensor can be
adjusted suitable to the application not only by changing
the spring used in the structure but also by adjusting thepiston bolt and the base cap of the structure.
Another advantage of this structure is that the sensing
elements (QTCTM pills) are not open to the environment
as they are enclosed in the structure. This will ensure thatthe environmental conditions such as dust, humidity will
not affect the performance of the sensing element.

4. Fabrication of Enclosed Tactile Sensor
When fabricating electrode as discussed in Ref. 20, oneof the major issue identified was create the physicalcontact between the sensing element and the electrode.
Since the solution from previous stage of sensor
development is not an optimum, it is required to find thebest method to create an optimum physical contact. Fromthe literature, it was found that vacuum depositing
Aluminium on both surfaces will give the optimum
results.21 Since vacuum depositing facilities are notavailable in Sri Lanka, next optimum available solution,
which is to use silver paste to create the physical contact
between electrode and sensing element is used whendeveloping sensing element configuration.
Using CNC machining techniques each structuralcomponents were fabricated, as such machining
techniques ensures high dimensional precision andaccuracy. After fabricating each and every structural© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

571

Design and Development of

(a)

(b)

(c)

Fig. 2. Results of the FEA performed for the Proposed Structure; (a) Total Deformation Variation, (b) Equivalent Stress Variation,(c) Force vs Deformation Plot

components, they were assembled together withelectrode configuration. Fig. 3 shows the fabricatedstructural components and assembled and finalized novel
enclosed 1-DOF tactile sensor.

Table 2. Sensor Characteristics Values Calculated forDeveloped sensor
SensorCharacteristics
Sensor Range
Sensitivity
Resolution

5. Experiment and Results
Experiments were carried out to find the relationshipbetween the developed sensor output and input, to findthe sensitivity of the sensor, to find the sensor range, tofind the resolution of the sensor and the repeatability ofthe sensor. The experiment was designed using Universal
Tensile Testing Machine. Purpose of using this machine
to conduct experiment for the sensor is to deliver agradually increasing and decreasing force to the sensor
Fig. 4 shows the results of these experiments carried out
using the above developed experimental setup. Table 2
shows the sensor characteristics values calculated fromthe experimental results.
(a)

RepeatabilityValue
0 – 170 N
0.2209 V/N
0.5 N
±3N

6. Conclusion and Discussion
In these results Fig. 4. (a) shows the relationship between
sensor input and the sensor output with standard
deviation at each data point. Also Fig. 4. (b) shows therepeatability analysis for the developed sensor.
According to these results, it can be seen that thehysteresis error of the developed sensor was increased
comparing with the experimental results of Ref. 20.
Reasons behind this phenomena may be due to errors
cause by the universal tensile testing machine and errors
present in the QTCTM material use for the experiment.

(b)References
1. M. I. Tiwana, S. J. Redmond, and N. H. Lovell, “A review
of tactile sensing technologies with applications inbiomedical engineering,” Sens. Actuators Phys., vol. 179,
pp. 17–31, Jun. 2012.
2. H. I. Jaafar, “Current Trend of Tactile Sensor in Advanced
Applications,” Sens. Transducers, vol. 143, no. 8, pp. 32–
43, Aug. 2012.
3. M. H. Lee and H. R. Nicholls, “Review Article Tactile
sensing for mechatronics—a state of the art survey,”
Mechatronics, vol. 9, no. 1, pp. 1–31, Feb. 1999.
4. M. H. Lee, “Tactile Sensing: New Directions, New
Challenges,” Int. J. Robot. Res., vol. 19, no. 7, pp. 636–
643, Jul. 2000.

Fig. 3. (a) Fabricated Sensing Configuration, (b) FabricatedEnclosed Sensor© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

572

T.D.I. Udayanga, D.A.M.R. Fernando, H.L.P.L. Chaturanga, B.A.D.J.C.K. Basnayake and Y.W.R. Amarasinghe

(a)

(b)

Fig. 4. Experiment Results; (a) Force vs Sensor Output, (b) Repeatability Analysis for the Developed Sensor
5. P. Estevez, J. M. Bank, M. Porta, J. Wei, P. M. Sarro, M.
Tichem, and U. Staufer, “6 DOF force and torque sensor
for micro-manipulation applications,” Sens. Actuators
Phys., vol. 186, pp. 86–93, Oct. 2012.
6. H. Yousef, M. Boukallel, and K. Althoefer, “Tactile
sensing for dexterous in-hand manipulation in robotics—
A review,” Sens. Actuators Phys., vol. 167, no. 2, pp. 171–
187, Jun. 2011.
7. A. Sarvazyan and V. Egorov, “Mechanical Imaging – a
Technology for 3-D Visualization and Characterization ofSoft Tissue Abnormalities. A Review,” Curr. Med.
Imaging Rev., vol. 8, no. 1, pp. 64–73, Feb. 2012.
8. A. Hamed, S. C. Tang, H. Ren, A. Squires, C. Payne, K.
Masamune, G. Tang, J. Mohammadpour, and Z. T. H. Tse,
“Advances in Haptics, Tactile Sensing, and Manipulation
for Robot-Assisted Minimally Invasive Surgery,
Noninvasive Surgery, and Diagnosis,” J. Robot., vol. 2012,
p. e412816, Dec. 2012.
9. “iPod,”Apple.
[Online].
Available:
http://www.apple.com/ipod/. [Accessed: 18-Apr-2016].
10. P. Estevez, J. M. Bank, M. Porta, J. Wei, P. M. Sarro, M.
Tichem, and U. Staufer, “6 DOF force and torque sensor
for micro-manipulation applications,” Sens. Actuators
Phys., vol. 186, pp. 86–93, Oct. 2012.
11. B. A. D. J. C. K. Basnayake, Y. W. R. Amarasinghe, R. A.
Attalage, T. D. I. Udayanga, and A. G. B. P. Jayasekara,“Artificail intelligence based smart building automation
controller for energy efficiency improvements in existingbuilding,” International Journal of Advanced Automation
Science and Technology, vol. 40, no. 40, 2015.
12. Javad Dargahi, Mojtaba Kahrizi, Nakka Purushotham Rao,and Saeed Sokhanvar, “Design and microfabrication of a
hybrid piezoelectric‐capacitive tactile sensor,” Sens.
Rev., vol. 26, no. 3, pp. 186–192, Jul. 2006.
13. J. Brugger, D. Briand, T. Polster, and M. Hoffmann,“Aluminum nitride based 3D, piezoelectric, tactile sensor,”
Procedia Chem., vol. 1, no. 1, pp. 144–147, Sep. 2009.
14. M. Ohka, H. Kobayashi, and Y. Mitsuya, “Sensing
characteristics of an optical three-axis tactile sensor
mounted on a multi-fingered robotic hand,” in 2005

15.

16.

17.
18.

19.

20.

21.

IEEE/RSJ International Conference on Intelligent Robots
and Systems, 2005, pp. 493–498.
H.-K. Lee, S.-I. Chang, and E. Yoon, “A Flexible Polymer
Tactile Sensor: Fabrication and Modular Expandability forLarge Area Deployment,” J. Microelectromechanical
Syst., vol. 15, no. 6, pp. 1681–1686, Dec. 2006.
H. B. Muhammad, C. M. Oddo, L. Beccai, C. Recchiuto,
C. J. Anthony, M. J. Adams, M. C. Carrozza, D. W. L.
Hukins, and M. C. L. Ward, “Development of abioinspired MEMS based capacitive tactile sensor for arobotic finger,” Sens. Actuators Phys., vol. 165, no. 2, pp.
221–229, Feb. 2011.
S. Nambiar and J. T. W. Yeow, “Conductive polymerbased sensors for biomedical applications,” Biosens.
Bioelectron., vol. 26, no. 5, pp. 1825–1832, Jan. 2011.
Y. W. R. Amarasinghe, A. L. Kulasekera, and T. G. P.
Priyadarshana, “Design and simulation of 1-DOF tactile
sensor for gripping force measurement,” in 2013 IEEE 8thInternational Conference on Industrial and Information
Systems, 2013, pp. 399–402.
M. R. Cutkosky, R. D. Howe, and W. R. Provancher,“Force and Tactile Sensors,” in Springer Handbook of
Robotics, B. S. Prof and O. K. Prof, Eds. Springer Berlin
Heidelberg, 2008, pp. 455–476.
T.D.I. Udayanga, D.A.M.R. Fernando, H.L.P.L.
Chaturanga, Y.W.R. Amarasinghe and D.V. Dao, 2015.
“Development of a Quantum Tunneling Composite based
1-DOF Tactile Sensor,” International Journal ofAdvanced Information Science and Technology, vol. 40,
no. 40, 2015.
M. Ohmukai, Y. Kami, and R. Matsuura, “Electrode forForce Sensor of Conductive Rubber,” J. Sens. Technol.,vol. 2, no. 3, pp. 127–131, 2012.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

573

Analog Circuit Design of a Novel 4D Chaotic System
Dechu Tan, Hong Niu

College of Electronic Information and Automation, Tianjin University of Science & Technology
80 Mailbox, Tianjin University of Science & Technology, No. 1038 Dagu Nanlu, Hexi District,Tianjin, 300222, China
E-mail: spots@163.com
www.tust.edu.cn

Abstract

In this paper, a novel four-dimensional (4D) autonomous chaotic system is reviewed, and its
corresponding new analog circuit is presented based on the modified module-based approach to chaotic
circuit design. The chaotic phase portraits of the new circuit are given to illustrate the good qualitative
agreement between the numerical simulation and the experimental realization.
Keywords: novel 4D chaotic system, analog circuit design, modified module-based approach, experimental
realization

Section 4, the phase portraits of the experimental circuit
are given. The conclusions are drawn in Section 5.

1. Introduction
Circuit design of chaotic system has been an importantfield in the study of chaos. Many methods have been
proposed.1-3 The modified module-based approach isone of the common methods for analog circuit design ofchaotic system.4 This method does not need to convert
the differential equations of the chaotic system to theintegral ones, and the value of each element isdetermined by comparing the coefficients of theequations of the system with those of the equations of
the circuit correspondingly. The advantages of themethod are high accuracy of the simulation andexperimental results, but less number of elements. In
this paper, the analog circuit of the 4D chaotic system
proposed in Ref. 5 is designed via the modified
module-based approach.
The rest of this paper is organized as follows. In
Section 2, the model of the novel 4D chaotic system andits numerical simulation phase portraits are reviewed. In
Section 3, the analog circuit of the system is constructedbased on the modified module-based approach. In

2. Review of the Novel 4D Chaotic System
The novel 4D chaotic system has been discussed in Ref.
5. It is formulated asx a y x ,y c x y z xw,
z mx y hz ,1
w xy bw,
where x, y, z , w
are state variables, and a = 25,
b = 3, c = 18, m = 19 and h = 14.
The Lyapunov exponents of the system (1)respectively are
13

2.6686 0, 2

11.7885 0, 4

0.0003 0,14.8804 0,

when the initial values are (x0, y0, z0, w0) = (1, 1, 1, 1). It
indicates that the system (1) is chaotic. The phase
portraits of the system (1) are shown in Fig. 1(a)-(f).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan574

Dechu Tan, Hong Niu

3.2. Time Scale TransformationLet(2). Then,

Fig. 1. Phase portraits of the novel 4D chaotic system: (a) x-y(b) x-z (c) x-w (d) y-z (e) y-w (f) z-w

3. Analog Circuit Design of the Novel 4D
Chaotic System0t , where

100 , and substitute it into Eq.

0dx
a 0 x a 0 y,dtdy
c 0 x c 0 y 0 z 10 0 xw,
dt
dz
m 0 x 0 y h 0 z,
dt
dw
10 0 xy b 0 w.
dt

3

3.3. Normalization for Equations

The analog circuit of the system (1) is constructed basedon the modified module-based approach to chaotic
circuit design, which consists of scale transformation ofvariables, time scale transformation, normalization forequations, circuit module design and determination ofparameters.3, 4Consider that the signals are all put into the negative
inputs of the operational amplifiers, then Eq. (3) should
be normalized to

dx
dt
dy
dt
dz
dt
dw
dt

3.1. Scale Transformation of Variables
The active analog elements used in this paper are
TL082CP and AD734AN, whose supply voltages are±15V, so that the output saturation voltage is 13.5V.
From the numerical simulation results shown in Fig. 1,
it can be found that the maximum absolute values of thevariables are all greater than the output saturation
voltage. It means that the distortion will occur whenobserving the phase portraits of the design circuit, andeven the elements could be destroyed. Therefore, thescale transformation of variables should be performed.
Let x 10 x , y 10 y , z 10 z and w 10 w ,then the system (1) is represented asy ,a 0x a 0
c 0

x

c 0

m 0

x

0

10 0

y

10 0 xw,z0

y h 0 z,x y b 0 w,

4

to make the equations of the system consistent withthose of the circuit.
3.4. Circuit Module Design
The circuit schematic diagram of the novel 4D chaotic
system is designed based on Eq. (4), as shown in Fig. 2.
The equations of the circuit can be described asdx
ax ay,ddy
cx cy z 10 xw,
ddz
mx y hz,ddw
10 xy bw,2dwhere x , y , z and w are still denoted as x, y, z

dx
dt

1
1
x
R1C1
R 2 C1dydt

1
R 3C 2y ,1
R 4 C2

x

y

1
R 5C2

z

1
xw,
10R 6 C2

and w for convenience.

dz
dt

1
R 7 C3dw
dt

1
10R10C 4

x

1
1
y
z,
R 8 C3R 9 C3x y

1
w.
R 11C4© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

575

5

Analog Circuit Design of

4. Experimental Circuit of the Novel 4D Chaotic
System
4.1. Experimental Circuit
Substitute the TL082CP operational amplifiers and theAD734AN analog multipliers into the circuit schematic
diagram shown in Fig. 2. The experimental circuit of thenovel 4D chaotic system is presented in Fig. 3. The
parameters of the elements have been marked in the
figure.

Fig. 2. Circuit schematic diagram of the novel 4D chaotic
system

3.5. Determination of Parameters
Comparing the corresponding coefficients of Eq. (5) andEq. (4) yields

R1

R2

1
a 0 C1

40kΩ,

R3

R4

1
c 0C2

55.5kΩ,

R5

R8

1
0C2

1MΩ,

R6

R 10

1
100 0 C2

R7

1
m 0 C352.6kΩ,R9

1
h 0 C371.4kΩ,Fig. 4. Phase portraits of the experimental circuit: (a) x-y (b)x-z (c) x-w (d) y-z (e) y-w (f) z-w

R 11

1
b 0C4

333.3kΩ,4.2. Experimental Results

Fig. 3. Experimental circuit of the novel 4D chaotic system

10kΩ,The phase portraits of the experimental circuit are
drawn via Multisim, as shown in Fig. 4(a)-(f). The

where C1 = C2 = C3 = C4 = 10 nF.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan576

Dechu Tan, Hong Niu

scales of x, y, z and w are 2 V/Div, 2 V/Div, 1 V/Div
and 1 V/Div, respectively. By comparing the phase
portraits in Fig. 4(a)-(f) with those in Fig. 1(a)-(f)correspondingly, it can be confirmed that there is a goodqualitative agreement between the numerical simulationand the experimental realization. It demonstrates thatthe experimental circuit of the novel 4D chaotic system
is correct.

References
1. R. Kiliç, A comparative study on realization of Chua’s
circuit: Hybrid realizations of Chua’s circuit combining
the circuit topologies proposed for Chua’s diode andinductor elements, International Journal of Bifurcation
and Chaos 13(6) (2003) 1475–1493.
2. Z. Xu, C. Liu and T. Yang, Study on a new chaotic
system with analysis and circuit experiment, Acta
Physica Sinica 59(1) (2010) 131–139.
3. S. Yu, Chaotic Systems and Chaotic Circuits: Principle,
Design and Its Application in Communications (XiDian
University Publishing House, Xi’an, 2011).
4. H. Niu, Stability control and circuit implementation of anovel 3D chaotic system with variable coefficient,
Dynamical Systems and Control 5(1) (2016) 31–40.
5. H. Niu, Analysis and control of a novel 4D chaotic
system, in Proceedings of the 2017 International
Conference on Artificial Life and Robotics, eds. M.
Sugisaka, Y. Jia, T. Ito and J. Lee (Miyazaki, Japan,2017), pp. 165–168.

5. Conclusions
In this paper, the analog circuit of the novel 4D chaotic
system is designed via the modified module-based
approach. The experimental results obtained byMultisim demonstrate the feasibility of the design
circuit. Furthermore, the actual hardware circuit of thenovel 4D chaotic system is being implemented. The
study in this paper has some engineering significance.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan577

A method of end-to-end self-understanding of Chinese paper-dictionaries
Zhijian Lv1, Yizhun Peng2
1

Beijing Institute of Science and Technology Information (BISTI), China;
2

College of Electronic Information and Automation,

Tianjin University of Science and Technology, China
E-mail:*luzj@bjstinfo.ac.cn
Abstract

This paper introduces a method of end-to-end self-understanding of Chinese paper-dictionaries. In thismethod, a page of Chinese paper-dictionaries is scanned into an electronic image. And then theelectronic image is preprocessed, including un-distortion, side scrapping, binarization, and so on.
Finally, using an end-to end deep learning method, the pre-processed electronic image is intelligently
segmented, text recognized, and context understood. Our method has been applied to self-understand aserial of Chinese paper-dictionaries, which have more than 13000 pages and 2.1 millions of entries. And
its correct rate of self-understanding of Chinese phrases is more than 99.5%. Its performance has proved
it’s availability.
Keywords: end-to-end self-understanding, image-segmentation,

The model transfers recent success in classification[9,14, 15] to dense prediction by reinterpretingclassification nets as fully convolutional and fine-tuningfrom their learned representations.
In the next section, we review related works on FCN
for semantic segmentation, and then introduce the FCN
to self-understand a serial of Chinese paper-dictionaries.
The following section demonstrate its performance.
Finally, we conclude our method of self-understand
Chinese Dictionaries using FCN.

1. Introduction
Convolutional networks (Convnets) have madeprogresses in recognition, including whole-image
classification [9, 14, 15] and local tasks with structured
output, including bounding box object detection [13, 5,8], part and keypoint prediction [16, 10], local
correspondence [10, 3], and semantic segmentation [11,1, 2, 12, 7, 6, 4].
Semantic segmentation faces an inherent tension
between semantics and location: global information
resolves what while local information resolves where.
Deep feature hierarchies encode location and semantics
in a nonlinear local-to-global pyramid. J. Long et al[18]proposed a fully convolutional network (FCN)
trained end-to-end, pixels-to-pixels on semantic
segmentation[17] and defined skip architecture to takeadvantage of this feature spectrum that combines deep,coarse, semantic information and shallow, fine,appearance information. The approach does not makeuse of pre- and post-processing complications, including
superpixels [2, 7], proposals [7, 6], or post-hoc
refinement by random fields or local classifiers[2, 7].

2. FCNs and its corresponding architectures forsemantic segmentation
2.1. Fully convolutional networks
Convnets is defined as follows:
y =f
x δ,
, 0 ≤ δi, δj ≤ k(1), where xij for the data vector at location (i, j) in aparticular layer; and yij for the following layer; k iscalled the kernel size; s is the stride or subsampling
factor; and fks determines the layer type, such as a matrix
multiplication for convolution or average pooling, a© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

578

Zhijian Lv, Yizhun Pengspatial max for max pooling, or an elementwise
nonlinearity for an activation function, and so on forother types of layers.
These basic components (convolution, pooling, andactivation functions) of convnets operate on local input
regions, and depend only on relative spatial coordinates.
Each layer of data in a convnet is a three-dimensional
array of size h×w×d, where h and w are spatial
dimensions, and d is the feature or channel dimension.
The first layer is the image, with pixel size h×w, and dcolor channels. Locations in higher layers correspond to
the locations in the image they are path-connected to,which are called their receptive fields.
This functional form is maintained under composition,with kernel size and stride obeying the transformation
rule(2)
f °g ′ ′ = (f°g) ′ ( ) ′ , ′While a general deep net computes a general
nonlinear function, a net with only layers of this form
computes a nonlinear filter, called a deep filter or fully
convolutional network. An FCN naturally operates on
an input of any size, and produces an output ofcorresponding spatial dimensions.
A real-valued loss function composed with an FCN
defines a task. If the loss function is a sum over the
spatial dimensions of the final layer, l(x; θ) =∑ l′ x , θ , its gradient will be a sum over the gradientsof each of its spatial components. Thus stochastic
gradient descent on computed on whole images will be
the same as stochastic gradient descent on l’, taking allof the final layer receptive fields as a minibatch. Whenthese receptive fields overlap significantly, both
feedforward computation and backpropagation aremuch more efficient when computed layer-by-layer overanentireimage
insteadofindependently
patch-by-patch.

intersection over union, with the mean taken over allclasses, including background are introduced intotraining. The training ignores pixels that are masked out(as ambiguous or difficult) in the ground truth.
3. Self-understanding of Chinese dictionaries
Information technologies (abbr. IT) have applied into
our lives. In IT systems, entries, usually captured frompaper based dictionaries, are basic elements and used
for text processing. However, It is a hard work thatbillions of entries are captured from paper-based
dictionaries. And then many manual works have done inthese field. There are still so many paper-based
dictionaries to be processed.
Convnets are driving advances in semantic
segmentation and object recognition. So FCNs are
introduced to transfer paper-based dictionaries into
electronic dictionaries for various kinds of applications.
Firstly these dictionaries are scanned into electronic
images. And then these image are pre-processed,
semantic segmented, recognized with an OCR engine,
finally indexed, as shows in Fig1-A.

2.2. Segmentation Architecture of FCNs

A: The workflow of
self-understanding.

ILSVRC classifiers are cast into FCNs andaugmented for dense prediction with in-network
upsampling and a pixelwise loss. Segmentation istrained by fine-tuning and skips between layers to fusecoarse, semantic and local, appearance information are
add into FCNs. And then a skip architecture is learned
end-to-end to refine the semantics and spatial precisionof the output. a per-pixel multinomial logistic loss andvalidate with the standard metric of mean pixel

B an electronic image
fromapaper-based
dictionary with a scanner.

Fig 1. The workflow of self-understanding of apaper-base dictionaries and an example of its electronic
images .

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan579

A method of end-to-end

3.1. Pre-processing

3.3. Semantic segmented using FCNs

An standard processed image is very important for
FCNs, which are used for segmentation andrecognition.
An image, captured from a page of a paper-based
dictionary with a scanner, is pre-processed before it issegmented and recognized.
These pre-processes
include removal of header and footer, distortion, andbinary.

According to the FCNs provided by J. Long et al,each net is decapitated by discarding the final classifierlayer, and convert all fully connected layers toconvolutions. a 1 × 1 convolution with channel
dimension 21 is appended to predict scores for each of
the PASCAL classes (including background) at each of
the coarse output locations, followed by a (backward)
convolution layer to bilinearly upsample the coarse outputs
to pixelwise outputs.
In our FCNs, the perimeter of the contour of an entry
are appended into loss functions.
4. Experiments

A: The original image.

B: removal of edges
including footer andheader.

C: a transformed image

D: Binary image andprimary contours

We selected a serial of paper-base Chinese
dictionaries, which have more than 13,000 papers and2.1 millions of entries. In our method, a page of Chinese
paper-dictionaries is scanned into an electronic image.
And then the electronic image is preprocessed,
including un-distortion, side scrapping, binarization, andso on. Finally, using an end-to end deep learning
method, the pre-processed electronic image isintelligently segmented, text recognized, and context
understood.
In the procedure of scanning, about 1000 pages weremanually adjusted. In pre-processing. In addition,
pre-processing is vital to the final recognition
performance.
And its correct rate of self-understanding of Chinesephrases is more than 99.5%. Its performance has proved
it’s availability.

Fig.2 Performance of pre-processing of a captured image

3.2. Contour templates of entries
5. conclusion

In Chinese dictionaries, entries are classified into
different types and nested in other entries. Some entries
have child entries. And child entries have their own
child entries.
Contour templates of entries are introduced into thesemantic segmentation. Before segmentation is done, anentry contour template set is built according to the entry
types, such as follows in Fig 3.

This paper apply FCNs into self-understanding ofChinese paper-dictionaries. The procedure includes
scanning, pre-processing, semantic segmentation, andOCR recognition. Its performance has proved it’savailability.
But this method will be improved in the following
fields, including pre-processing, loss function of FCNs,
and contour templates.

Fig 3. Contour templates of entries© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

580

Zhijian Lv, Yizhun Peng

Acknowledgements
The research is partly supported by Information
Production Line Construction (Phase I), Beijing
Municipal Financial Grant: Information Production Line
Construction (Phase I)(PXM2017_178214_000005),
Special Grant for Reform and Development of BISTI(2017).

8.
9.
10.
11.

References
1. D. C. Ciresan, A. Giusti, L. M. Gambardella, and J.
Schmidhuber. Deep neural networks segment neuronalmembranes in electron microscopy images. In NIPS,
pages 2852–2860, 2012. 1, 2, 4, 7
2. C. Farabet, C. Couprie, L. Najman, and Y. LeCun.
Learning hierarchical features for scene labeling. PatternAnalysis and Machine Intelligence, IEEE Transactions on,
2013. 1, 2, 4, 7, 8
3. P. Fischer, A. Dosovitskiy, and T. Brox. Descriptor
matching with convolutional neural networks: acomparison to SIFT.CoRR, abs/1405.5769, 2014. 1
4. Y. Ganin and V. Lempitsky. N4-fields: Neural network
nearest neighbor fields for image transforms. In ACCV,2014. 1, 2, 7
5. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Richfeature hierarchies for accurate object detection andsemantic segmentation. In Computer Vision and Pattern
Recognition,2014. 1, 2, 7
6. S. Gupta, R. Girshick, P. Arbelaez, and J. Malik.
Learning rich features from RGB-D images for object
detection and segmentation. In ECCV. Springer, 2014. 1,
2, 8
7. B. Hariharan, P. Arbel′aez, R. Girshick, and J. Malik.
Simultaneous detection and segmentation. In European

12.
13.

14.
15.

16.

17.

Conference on Computer Vision (ECCV), 2014. 1, 2, 4, 5,
7, 8
K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid
pooling in deep convolutional networks for visual
recognition. In ECCV, 2014. 1, 2
A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
NIPS, 2012. 1, 2, 3, 5
J. Long, N. Zhang, and T. Darrell. Do convnets learncorrespondence In NIPS, 2014. 1
F. Ning, D. Delhomme, Y. LeCun, F. Piano, L. Bottou,
and P. E. Barbano. Toward automatic phenotyping ofdeveloping embryos from videos. Image Processing,IEEE Transactions on, 14(9):1360–1371, 2005. 1, 2, 4, 7
P. H. Pinheiro and R. Collobert. Recurrent convolutional
neural networks for scene labeling. In ICML, 2014. 1, 2,
4, 7, 8
P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R.
Fergus,and Y. LeCun. Overfeat: Integrated recognition,
localization and detection using convolutional networks.
In ICLR, 2014. 1, 2, 4
K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. CoRR,
abs/1409.1556, 2014. 1, 2, 3, 5
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D.
Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.
Going deeper with convolutions. CoRR, abs/1409.4842,
2014. 1, 2, 3, 5
N. Zhang, J. Donahue, R. Girshick, and T. Darrell.
Partbased r-cnns for fine-grained category detection. In
Computer Vision–ECCV 2014, pages 834–849. Springer,
2014.1
J. Long, E. Shelhamer, T. Darrell. The IEEE Conference
on Computer Vision and Pattern Recognition (CVPR),
2015, pp. 3431-3440.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan581

A Multi-robot Rescuing System
Huailin Zhao1,Zheng Wu2,Xiaoxing Wang3
123

School of Electrical and Electronic Engineering,
Shanghai Institute of Technology, Shanghai 201418, China
E-mail:964003999@qq.com
Abstract
The study subject is a multi-agent system consisting of a few mobile robots. In the system, each robot collaborates
with the other ones, and they all work together to complete the disaster rescuing tasks such as the fire fighting. The
multi-agent system control theory is applied in the system, where each mobile robot is an independent intelligent
individual. The robots communicate wirelessly among them. Even though there is trouble with the individual robots,
the others will collaborate among them in time and adjust their control policy to ensure the system itself working
normally and complete the disaster rescuing task. In the system, an upper computer is designed for supervising,human-machine interfacing and being the transferring station of the data exchange. The experiment shows that themulti-robot distributed control system based on WIFI, GPS and INS information is able to achieve the consistentaction such as rescue gathering.
Keywords: multi-agent system, cooperative control, WIFI communication.

or more in the future. In the rescue collaboration, eachsmart vehicle needs to know the motion information of
itself and the others in time, so it needs to configure
GPS navigation module, inertial navigation module and
WiFi communication module7.
The project temporarily uses an upper computer as thecommunication interface between the two smart
vehicles, and is used to assist system monitoring at thesame time.
Each smart vehicle uses the microcontroller STM32
as the control core, which is the 32-bit of 64kb or 128kb
memory microprocessor based on ARM core, with good
performance and cheap price. The GPS navigation
module selects ATK-NEO-6M, the inertial navigation
module MPU6050, and the WiFi module ATKESP8266, which are produced by the company
ALIENTEK. The three modules communicate with
STM32 seperately.
The STM32 has 4 serial ports, with 4 USART dataregisters and 8 pairs of pins for the chip. In this project,serial port 1 connects to the WiFi module, serial port 2
to the inertial navigation module, and serial port 3 to the
GPS navigation module.
The specific hardware structure is shown in Figure 1.
Figure 2 shows the serial communication pin
relationship of the controller STM32 which uses 4 pins
including PA9, PA10, PB10 and PC11.

1.

Introduction
With the development of the robot and artificial
intelligence, the intelligent robot technology has
gradually become an important strategy in the high-techfield1. Its application has covered many aspects,
including service robots in daily life, exploring robots
used in the ocean or space, the rescuing robot used inthis research, and so on2. Compared to the single robot
with limited performance and high cost, the multi-robot
cooperation system can meet different occasions,improve work efficiency, and reduce cost. The otherrobots in the multi-robot system can change theircontrol strategy autonomously in case that there istrouble with a small amount of robots3. This project uses
smart vehicles as the robots. In the collaboration system,
the smart vehicles exchange their motion informationsuch as speed, position, and the moving direction by
wireless communication, make judgment and control
decision independently, collaborate actively, so as toimprove the system efficiency4.
2.

System overview
The multi-vehicle system applies the distributed
control theory that every smart vehicle is an integrated
individual and can carry out information processing anddecision-making on its own5. The redundancy andscalability of the system is significantly increased,
relative to the centralized top-down control one. Evensome individual smart vehicles become invalid, the rest
adjust their control strategies timely by active
collaboration to avoid paralysis of the whole system6.

2.2 Design of the smart vehicle software
The colaberation algorithms and the related data arecompleted in the main loop. And the data with thewireless module, GPS navigation module, inertial
navigation are completed by the serial interrupt
programs. Figure 3 shows the flow chart of the smart
vehicle software.

2.1 Design of the smart vehicle hardware
The whole system is made up of two smart vehicles
temporarily that will be extended to three smart vehicles© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan582
1

Huailin Zhao,Zheng Wu,Xiaoxing Wang

communicate with the upper computer9. Figure 4 shows
the diagram of the system communication.

Fig. 1. Hardware structure of the smart vehicle
Fig. 4. Diagram of the system communication
3.1 Design of the WiFi communication protocol

First, the AT instruction is sent to the WiFi module
by a serial helper to carry on the configuration of thelayer aggregation module. The configuration contents
are saved to the flash of the module so that theconfiguration can be the default later. The
communication protocol is very important for this kindof data volume, multi-point communication system. The
communication protocol is as follows:
Check whether the data is correct by agreement.
Determine the data source, such as vehicle No.1
or vehicle No.2.
The agreement applied in this project includes thefollowing items:
The general agreement: frame end with \r\ n.
The STM32 sends the GPS address to the upper
computer, and the data starts with alphbat ‘R’,
longitude, latitude, and ends with \ r \n.
From the upper computer to the STM32, the GPSdata is sent three times. The data format is asfollows: latitude 1, longitude 1: latitude
2, longitude 2: latitude 3, longitude 3. A comma
is used to separate the latitude and the longitude.
And a colon is used to speprate each group of thedata.

Fig. 2. Pin connection of the controller

3.2 Data formation
Because there is no pass through mode on the service ,there must be an AT instruction before the upper
computer sends the data. Command: AT+CIPSEND=
channel number, character length \r\ n. In other words,
the number of data characters sent by the upper
computer must be fixed10.

Fig. 3. Flow diagram of the smart vehicle
3 Design of the communication program
ATK-ESP8266 is the high-performance UART WiFi module made by the company ALIENTEK, it is
the core unit of ESP8266 module made by the company
AI – THINKER. The module certified by FCC and CE
meets the European and American standard8.
The WiFi module supports the communication
modes including AP, STA, AP+STA. And it supports
any IP node as a server or client. This project uses the
PC as AP and the server, and the smart vehicles as STA
and the client. The server supports multiple connections
and the client supports passing through. In other words,
the client is set to pass through and connected to theserver. Therefore the STM32(client) can directly

3.3 Data receiving and processing
Regardless of that the upper or lower computer isreceiving data, the frame tail should be judged.
Starting from the upper computer, the protocol formatof the STM32 is R, latitude, longitude, because theupper computer does not exist pass through, the module
will add a message, so the actual effect received is+IPD: channel number: character length: R,latitude,
longitude \ r \ n.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan583

A Multi-robot Rescuing System

For example:
AT+CIPMUX=1
OKAT+CIPSERVER=1,5000
OK
0,CONNECT+IPD,0,1:1+IPD

ChannelnumberCharacters
longR

latitudelongitude\r

degree difference, the actual distance is 111 * cosθ
kilometers. Θ refers to the Shanghai is located in east
longitude 120 ° 51 '- 122 ° 12', north latitude 30 ° 40 '31 ° 53'.The decimal is the latitude 120.85-122.02,
longitude 30.67-31.8. The units are degrees, which are
the integer parts that the GPS module reads, whichmeans you can also read the decimal point five placeson this basis.
Accuracy is:\nTable 1. Data receiving format
The key to the recognition is that +IPD can be
represented by the smart vehicle. The channel number
can identify the data from that vehicle, and the R
represents the beginning of the data, and finally thelatitude and longitude are separated by a comma.

(1)

In fact, the accuracy of the module fluctuates within acertain range, and the measured accuracy is about 3-5
meters.
4.2 Design of the data processing and filtering program with
the GPS module

4. Design of the navigation module
GPS is based on satellite positioning and time transfer,and GPS positioning function is widely used in modern
measurement and modern construction.

The reading of the whole data is interrupted by
USART serial port. After I receive my cache, according
to NMEA 0183 to judge strings on the cache. judging
the USART3_RX_BUF[t] received a 'G' and 'C' in theinterrupt, if it is, then received a string "$GPRMC",
then enter the Data_change () function to determine
character and the last to turn a string into a U32.
The Data_change() is function that according to thepostion of the comma to judge what data and the lengthof the string. For example, the data after the thirdcomma representation of latitude, and the data between
third and fourth comma is the length of the latitude data
string. In the process need to put the string into digitalquantity.
Due to the precision of the GPS module, the GPSdata will jump in a small range , so after successfully
stored data, the data of the STM32 will averagefiltering. Data will be received into a length of 5 array
DataN [I], after average filtering through the sum
divided by five.
Code:((DataN[0]+DataN[1])/2+(DataN[2]+DataN[3])/2+Data
N[4])/3
Here is not divided by five directly. This is to prevent
overflow.

4.1 Introduction to the NMEA-30183 protocol

NMEA-30183 is the standard format developed by
the national Marine electronics association used ofelectronic equipment in the sea. It has become a unified
RTCM standard protocol for GPS navigation devices.
This project uses $GPRMC standard location
information.
The basic format of the $GPRMC is as follows:$GPRMC,(1),(2),(3),(4),(5),(6),(7),(8),(9),(10),(11),(12)*hh(CR)(LF)
Among them:(1) UTC time, HHMMSS(hour,minutes,second)(2) Positioning status, A= valid location, V= invalid
location(3) Latitude ddmm.mmmmm(4) Latitude hemisphere N (northern hemisphere) or
S (southern hemisphere)(5) Longitude ddmm.mmmmm(6) Longitude hemisphere E (east longitude) or W
(west longitude)(7) The rate of the ground(8) The moving direction of the ground(9) UTC date ddmmyy(10) Magnetic declination(11) Direction of magnetic deflection, E (east) or W
(west)(12) Mode indication (A = autonomic positioning, D=
difference, E= estimation, N= data invalid)

4.3 Design of the data processing and filtering program with
the inertial navigation module

Due to the accuracy and response speed of the GPS,
can't timely feedback direction, so the introduction ofinertial navigation module MPU6050 compensation.
MPU6050 is the 6 axis accelerometer gyroscope. threeaxis acceleration, three axis gyroscope. STM32 reads
MPU6050 measurement data. MPU6050 module withkalman filtering, therefore, STM32 reads the data has
been through filtering processing.

The GPS module used in this experiment can be accurate to
5 decimal places. The accuracy of STM is accurate to meters.
The conversion process is listed below:

On the longitude line, the latitude is one degree
difference, the the actual distance is about 111
kilometers; On the latitude line, the longitude is one© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan584

Huailin Zhao,Zheng Wu,Xiaoxing Wang

prevent the two cars from colliding with each other, weneed a safe distance. Taking into account the accuracyof the GPS module, the safety distance is set to 2.5
meters. In the process of gathering two cars, S1 and S2
will be smaller and smaller, when the distance of S2 isless than or equal to 2.5, the number two car will takethe position of number one car as the setting value15.

5 Motion control and collaboration algorithm

STM32 direct output PWM wave and can adjust thepulse width, so as to control servo motor rotating speed
and direction. Motor servo system can realize highprecision control, make sure that motion state of the
motor can be timely follow the specified input values
change. Servo motor receives many pulse, will turn thecorresponding angle, so as to realize position control.

6 Experimental tests and conclusions
6.1 Kp setting

5.1 Structure of the control system

Let’s set Kp firstly. We do this way: changing the
car's orientation and then let it move self-driven. The
software Tracker is used to track its tracking. The
waveform is shown in Fig.6.

Fig. 5.Servo motor control system
The whole movement control structure as shown in
the figure 5. With double closed-loop control, the rightmotor control direction and location at the same time,
the left motor control position. Inner ring MPU6050
inertial navigation module corrects the angle, The
current value of the GPS plus angle offset of MPU6050
to measure minus GPS target direction as input values .
The right motor only responsible for the control
direction. But due to the left motor movement, rightmotor must have the output to a stable direction11.
As the direction control make PI, continuous
correction of the course of the deviation, call function
math.h () and the inverse trigonometric function atan (),with latitude divided by precision to calculate thedeclination, the same method of calculating the heading
angle, and the heading angle plus MPU6050 as theoffset, finally, the offset minus the heading angle12.
Parameter tuning using the first P after I method. Due tothe relationship between motor speed and GPS sampling
speed, the pure proportional link of the outer ring has
little effect on the system. The inner parameters havelarger impact. First through the test to get theappropriate Kp then add Ki13.

Fig. 6. Stability curve with Kp = 10

Fig. 7. Stability curve with Kp = 20
Comparing with Figure 6 and Figure 7, the system isstable when Kp = 20. Therefore, we set Kp=20.
6.2 Ki setting

Now Let’s set Ki. Too large integration may result inpoor stability. Figure 8 shows that with Ki=5.
Figure 9 shows that with Ki=2. Comapred to Figure 8,
the system is more stable. Therefore we simply set Ki=2.

(2)
angle=atan(latitude/longitude)

The above formula,MV is output,angle is deviation ofangle, 32767 is the length of the short.To cooperate with
MPU6050 output and use range14.
5.2 Collaborative clgorithm

Assuming the car is S1 away from the target locationand the distance between the two cars is S2, In order toFig. 8. Stability curve with Ki = 5© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan585

A Multi-robot Rescuing System

3.

4.
5.

Fig. 9. Stability curve with Ki = 2
6.3 Collaborative experiment

In the end, the two vehicles can communicate witheach other, keeping the distance of 70 cm inmotion.Normally, two vehicles stay within 2m.

6.

7.

8.

Fig. 10.Collaborative experimental

9.

6.4 Conclusion

The robot rescue system, consisting of two smart
vehicles, is able to realize the inner communication
between the two vehicles and coordinate movement.
The two vehicles don’t collide with each other during
the following-up movement. The next step is to achieve
collaboration among three smart vehicles or more. And
the whole system will remove the upper computer toachieve direct communication among the smart vehicles.

10.
11.

12.

13.
14.

Fig. 11. Smart vehicle
1.
2.

References:

15.

Weston J WratkiIlS C ． Multi-class Support Vector
Machines．RoyalHolloway College，1998： 726．732
Yawei Hou, Huailin Zhao. Handwritten Digit
Recognition Based on Improved BP Neural Network.

The 13th Chinese Intelligent Systems Conference(CISC2017, Oct. 14-15, Mudanjiang, China).
Huailin Zhao，Yawei Hou，Shifang Xu, Congdao Han,
Masanori Sugisaka. Research on a Method of Character
Recognition for Self-learning Errors. The 2017
International Conference on Artificial Life and Robotics(ICAROB 2017), Jan.19-22, Miyazaki, Japan, p190-193.
Krebel U Pairwise classication and support Vector
machines． Cambridge MA：MIT Press， 1999：255～256
Huailin Zhao, Shunzhou Wang, Shifang Xu, Yani Zhang，
Masanori Sugisaka, A Method of Detecting Abnormal
Crowd Behavior Events Applied in Patrol Robot.
Journal of Robotics, Networking and Artificial Life. Vol.
4(1), 2017, p77–80.
Huailin Zhao， Yangguang Guo，Masanori Sugisaka,
Design of the Multi-Car Collaboration System, the 2017
International Conference on Artificial Life and Robotics(ICAROB 2017), Jan. 19-22, Miyazaki, Japan, p186-190.
Qiwen Zheng, Zhongyu Shen.Research on collaborative
control system of multi-smart car based on wirelesscommunication network.Nanjing normal university,
2012
Huailin Zhao, Xuyao Hao, Albert Wang, Chao Li,
Design and implementation of an intelligent cooking
robot based on Internet of things. Lecture Notes inElectrical Engineering. Vol.338, 2015, p423-430.
Huailin Zhao, Masanori Sugisaka, “Simulation study ofCMAC control for the robot joint actuated by McKibben
muscles”, Applied Mathematics and Computation(ISSN0096-3003), Vol.203, 2008, p457-462.
Wang zhiwen, guo ge. 2003. Mobile robot navigation
technology and prospect [J]. Robot. 25(5):470-474.
Shunzhou Wang, Huailin Zhao, Weiren Wang, Huijun
Di, Xueming Shu. Improving Deep Crowd Density
Estimation via Pre-Classification of Density. The 24th
International Conference on Neural Information
Processing (ICONIP2017, Nov.14-18, 2017, Guangzhou,
China).
Yawei Hou, Huailin Zhao. Handwritten Digit
Recognition Based on Depth Neural Network. 2017
International Conference on Intelligent Information and
BioMedical Science (ICIIBMS2017, Nov.24-26, 2017,
Okinawa, Japan).
Haitao Ou, WeiDong Zhang, Wenyuan Zhang, etc.,
based on the intelligent traffic control system of multiagent technology.Electronic journal, 2000(12) : 52.55
Huailin Zhao, Bei Wang, “Configuration of the
Mckibben Muscles and Action Intention Detection for
an Artificial Assistant Suit”, International Journal ofAdvanced Robotic Systems (ISSN 1729-8806), Vol. 9,
2012, p1-7.
Huailin Zhao, Wei Ren, Consensus Problem ofDistributed Multi-agent Systems, Proceedings of the2015 International Conference on Artificial Life andRobotics (ICAROB 2015), Japan, p201-206.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan586

Dynamic Analysis and FPGA implementation of A Novel Hyper-Chaotic System with One
Equilibrium Point
Shanfeng Wang, Hongyan Jia, Zhiqiang Guo

Department of Automation, Tianjin University of Science and Technology,
1038 Dagunanlu Road, Hexi District, Tianjin 300222, PR China
E-mail: jiahy@tust.edu.cn
www.tust.edu.cn
Abstract

In recent decades, the study on the chaotic system and hyper-chaotic system attracts more and more interest with developing of non-linear subjects. This paper firstly presents dynamic numerical analysis of anovel hyper-chaotic system with one equilibrium, which includes, bifurcation diagram, Lyapunov exponent and so on. Then, based on the method of converting continuous-time differential equation to discretetime differential equation, FPGA (Field Programmable Gate Array) implementation for the novel hyperchaotic system is finished, and the results from FPGA implementation are consistent with those fromnumerical analysis. The work in this paper may provide a method for the application of the hyper-chaotic
system.
Key words: Hyper-chaotic system; one equilibrium; dynamic characteristics; FPGA

1 Introduction
Since Lorenz chaotic system is found in 1963[1], Chao research becomes gradually a hot topic. Especially, in recent
decades, with developing of science and technology, many
new chaotic systems are presented, including Chen system[2], Liu system[3], and so on[4-6]. On the other hand,some hyper-chaotic systems also are generated on the basisof the chaotic systems, such as Lorenz hyper-chaos[7, 8],
four-dimensional non-autonomous hyper-chaos[9], and soon[10, 11].
At the same time, circuit implementation of chaotic systems attracts more and more interest, such as chaotic systems control circuit, chaotic synchronization circuit[12],chaotic generation circuit[8, 11], and chaotic digital circuit[13-15], etc. Particularly, because the hyper-chaotic systems have unpredictability more than chaotic system, theapplication of hyper-chaotic system has become a hot topic.
So, it is necessary for application of chaotic system to design chaotic circuit.

In this paper, a novel one equilibrium hyper-chaotic system based on Lü chaotic system is presented. Although this
hyper-chaotic system has one equilibrium, it has complex
dynamic character. Therefore, the chaotic characteristics isanalyzed, including bifurcation diagram, Lyapunov exponent diagram and so on. On the basis of numerical analysis,
the FPGA implementation of the novel hyper-chaotic system will be introduced. According to the result of experiment, it may provide a method for the application of thehyper-chaotic system.
2. The hyper-chaotic system
A novel hyper-chaotic system with only one equilibrium
point is introduced by ref[16], which is described by (1):© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

587

(1)

Shanfeng Wang, Hongyan Jia, Zhiqiang Guo

Whereis status variable, andis parametric variable of the system. The new system shows chaotic dynamic characteristics, when the initial value isand, and thehyper-chaotic attractors of the system (1) are shown in Fig.
1,chaotic character, when points are dense in bifurcation di, chaos is not discovered inagram. Obviously, when
this new system; when,or,the system shows periodicity or cycle; and whenor
, the system shows chaos.
Next, to further explain the character of system (1), theLyapunov exponent is presented in this paper.
3.2 Lyapunov exponent
Lyapunov exponent can describe the characteristics of chaotic motion. Because the fourth Lyapunov exponent (LE4)is always negative value on region of, thethree Lyapunov exponents (LE1, LE2, LE3) are provided.
The Lyapunov exponent of system (1) is shown in Fig. 3.

Fig. 1. Phase portraits of hyper-chaotic system (1):(a) y-z plane; (b) x-w plane
3. The dynamic character of the novel hyper-

LE1

LE2
LE3

chaotic system

0
Lyapunov

Now,

Lyapunov property

5

when

parameters
is
fixed
as
, the dynamic behaviors of novel
hyper-chaotic system (1) are discussed with variety of parameter .

-5

-10
-10

-5

0

5

10

c

15

20

25

30

35

Fig. 3. Lyapunov exponent of system (1)According to Lyapunov exponent theorem, the systemshows chaotic dynamics, when it only includes a positive
Lyapunov exponent. Therefore, from the Fig. 3, chaos can;
be not discovered in system (1), when parameter
the system (1) shows quasi periodicity or cycle, when parameter,or; the system (1)shows chaos, when parameteror.
the chaotic behavior of system (1) can be proved from
Lyapunov exponents and bifurcation diagram. Similarly,
the phenomenon can also be described by phase trajectory,
as shown in Fig. 4.

3.1 bifurcation diagram
Bifurcation diagram can describe the property of systemwith variety of parameter. The bifurcation diagram of system (1) is shown in Fig. 2.

Fig. 2. Bifurcation diagram of system (1)Generally, according to the bifurcation theory, the system (1) has no chaotic attractor, when bifurcation diagram
is one line; the system (1) is periodic, when points is several lines in bifurcation diagram; the system (1) will show
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan588

Dynamic Analysis and FPGA

4.2 FPGA implementation
As compared with other processors, FPGA is a dedicated hardware in processing logic. And the working way
of FPGA is parallel compute, which using hardware description language to achieve the design of systems[13, 15].
FPGA is a better tool to show chaotic or hyper-chaotic attractor[14]. The process of FPGA implementation is shownin Fig. 6. The experiment results are obtained, as showed
in Fig 6. The results are in agreement with the results of
the numerical analysis, which physically proves the system(1)Fig. 4. The phase trajectory of y-z phase plane
; (b)
; (c)
; (d)
(a)
4. FPGA implementation of hyper-chaotic system
At present, some analog circuits of chaotic system are introduced[9, 16]. However, some disadvantages are discovered in the implementation of analog circuit. Therefore,
implementation of digital circuit attracts more and moreinterest. In this paper, the method of digital circuit is introduced.
4.1 Bilinear Transformation Method

The method of converting a continuous-time integer orderintegral operator to discrete-time differential equations is

Fig. 6. The process of FPGA implementation

called IIM (Impulse Invariance Method). According to thismethod, the step is as follows.
The continuous-time integer order operator is described,
as follows:(2)

The discrete-time integral operator is obtained by using
IIM. As follows:(3)

Where T is sample frequency.
According to discrete-time transfer function(3), themodel of integer order integral operator is found by using DSP Builder Library on Matlab/Simunlink platform[14]. As follows:

Fig. 7.The results of FPGA implementation
y-z phase plane: (a)
; (b)
; (c)
(d)
;
5. Conclusion;

In this paper. Firstly, a hyper-chaotic system with only one
equilibrium point is introduced. Secondly, the dynamic
characteristics of the hyper-chaotic system is discussed, including bifurcation diagram, Lyapunov exponents diagramand phase trajectory diagram. Thirdly, the method ofFig. 5. The model of integer order integral operator© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

589

Shanfeng Wang, Hongyan Jia, Zhiqiang Guo

FPGA implementation is presented, and the experimental
results is obtained in FPGA hardware platform. Finally, it
can be concluded that the hyper-chaotic system with only
one equilibrium can show chaos dynamic, and the FPGA
implementation provides a method for the application of
the hyper-chaotic system.
Acknowledgements

system [J]. Nonlinear Dynamics, 2010, 59(3): 515-27.
10 Cheng Hsien Chen, Long Jye Sheu, Hsien Keng Chen,etal. A new

hyper-chaotic

systemanditssynchronization [J]. Nonlinear Analysis Real World
Applications, 2009, 10(4): 2088-96.
11 ShiJian Cang, ZengQiang Chen, ZhuZhi Yuan.
Analysis and circuit implementation of a new four-

The research is partly supported by the Research Fundfor the National Natural Science Foundation of China(Grant No.61573199). It is also supported by the Young
Scientists Fund of the National Natural Science Foundation of China (Grant No.11202148).
References

dimensional non-autonomous hyper-chaotic system [J].
2008,
12 G. Grassi, S. Mascolo. Design of nonlinear observers
for hyperchaos synchronization using a scalar signal;
proceedings of the IEEE International Symposium on
Circuits and Systems, F, 2002 [C].

1 Edward N. Lorenz. Deterministic Nonperiodic Flow

13 M. F. Tolba, A. M. Abdelaty, N. S. Soliman, et al.

[J]. Journal of Atmospheric Sciences, 1963, 20(2): 130-FPGA implementation of two fractional order chaotic

41.

systems [J]. Aeu-Int J Electron C, 2017, 78(162-72.

2 Guanrong. Chen, T. Ueta. Yet another chaotic attractor

14 Guangyi Wang, Xulei Bao, ZhongLin Wang. Design[J]. Int J Bifurcat Chaos, 1999, 9(7): 1465-6.

and FPGA Implementation of a new hyperchaotic

3 Chongxin Liu, Tao Liu, Ling Liu, et al. A new chaotic

system [J]. Chinese Physics B, 2008, 17(10): 3596-602.

attractor [J]. Chaos Solitons & Fractals, 2004, 22(5):15 Roger Woods, John Mcallister, Richard Turner, et al.

1031-8.

FPGA-based Implementation of Signal Processing

4 Jinhu Lu, Guanrong Chen. A new chaotic attractor

Systems [J]. Wiley Publishing, 2008, 15(15): 153-68.

coined [J]. Int J Bifurcat Chaos, 2002, 12(3): 659-61.

16 HongYan Jia, ZengQiang Chen, ZhuZhi Yuan. A novel

5 Leon O Chua, Motomasa Komuro, Takashi Matsumoto.

one equilibrium hyper-chaotic system generated upon

The double scroll family [J]. IEEE Transactions on

Lü attractor [J]. Chinese physics B, 2010, 19(2):Circuits & Systems, 1986, 33(11): 1072-118.

020507.

6 Sundarapandian Vaidyanathan. Adaptive Controller
and Synchronizer Design for the Qi-Chen Chaotic
System [M]. Springer Berlin Heidelberg, 2012.
7 Yuxia Li, Wallace KS Tang, Guanrong Chen.
Generating hyperchaos via state feedback control [J].
Int J Bifurcat Chaos, 2005, 15(10): 3367-75.
8 Yuxia Li, Guanrong Chen, W. K. S. Tang. Controllinga unified chaotic system to hyperchaotic [J]. IEEE
Transactions on Circuits & Systems II Express Briefs,
2005, 52(4): 204-7.
9 Shijian Cang, Guoyuan Qi, Zengqiang Chen. A fourwing hyper-chaotic attractor and transient chaos
generated from a new 4-D quadratic autonomous© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

590

Analysis and Circuit Implementation for a New Fractional-Order Chaotic System
Zhiqiang Guo, Hongyan Jia, Shanfeng Wang
1
College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;

E-mail: jiahy@tust.edu.cn

Abstract

This paper firstly recommends a new fractional-order chaotic system, only including six terms and threemultipliers, which is different from the Lorenz and other existing fractional-order systems. Then thephase trajectory diagrams, Lyapunov exponents diagrams and bifurcation diagrams will be showed toinvestigate the different dynamical characteristics based on this chaotic system. At last, an analog circuit
is designed to implement the system, further verify the effectiveness in practical application.
Keywords: fractional-order chaotic system, Lyapunov exponents, bifurcation, circuit implementation

characteristics. And the phase trajectory diagrams,
Lyapunov exponents diagrams and bifurcation diagrams
are showed to verify them. At last, an analog circuit isdesigned to implement this fractional-order chaotic
system, and the results of circuit implementation andnumeral simulation are similar. Thus, this new
fractional-order chaotic system can be applied in the
project.

1. Introduction
Chaos is a special form of motion of nonlinear dynamicalsystems, which has been widely studied in the past few
decades1-3. Since the chaos phenomenon has been foundin large number of fractional-order systems, the dynamic
characteristics of the fractional system have begun toattract more and more interest. At present, somefractional-order chaotic systems have been proposed andresearched, such as the fractional-order Lorenz system4,
the fractional-order Chen system5, the fractional-order Lüsystem6, and the fractional-order Qi system7.
Now, fractional-order chaotic systems have been
investigated in some factors, such as dynamic
characteristics8-9,
circuitimplementation10-11,
12-13
synchronization, and secure communication14 of thechaotic system. Furthermore, the fractional-order systems
are more complex to be implemented by analog circuits
than integer-order systems. Therefore, in order toinvestigate the fractional-order systems and apply them insome practical engineering, it is very necessary andmeaningful to design hardware circuits to implement
them.
This paper based on a new fractional-order chaotic
system, firstly discusses and analyzes the basic dynamic

2. A new fractional-order chaotic system
Recently, a new chaotic system is investigated15, and thebasic dynamic characteristics have been showed, such asphase trajectory, Lyapunov exponents, bifurcation, etc.
The dynamic equation of the new chaotic system is
̇=− +
̇=
−
(1)
̇=− +
Where, a , b , c , d , k and m are the systemparameters, and when a = 4 , b = 3 , c = 1 , d = 7 ,k = 1, m = 2, the system is chaotic. The system (1) canbe translated into fractional-order form as:© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

591

Zhiqiang Guo, Hongyan Jia, Shanfeng Wang

⎨
⎪
=− +
⎩
Where, a , b , c , d , k and m are the systemparameters, x, y, z are state variables, and α =β = γ = 0.9 is fractional order. In order to analyze
the different states of the chaotic system, theLyapunov exponents and bifurcation diagrams are
firstly given to investigated them. When fixing thesystem parameters a = 5 , b = 3 , d = 5 , k = 1 ,m = 2, selecting c ∈ [0, 6], and setting the initialvalue of the system to (1, 1, 1) , the Lyapunov
exponent diagrams and bifurcation diagram of thissystem are showed in Fig.1 and Fig.2, respectively.
Obviously, when selecting c ∈ [1.4, 4], the systemshows chaos. Particularly, when a = 5, b = 3, d =5, k = 1, m = 2, and fixing c = 3, the chaotic
phase trajectory diagrams are given in Fig.3.

(2)

3

2.5

2

2

1.5

1.5

1

1

0.5

0.5

0
-4

-3

-2

-1

0

1

2

3

4

0
-3

5

-2

-1

0
x

y

(a)

1

2

3

(b)

3
2.5
2
1.5
1
0.5
0
5
4
2

0

0
-2
-5

y

-4

x

(c)

Fig.3. Chaotic phase trajectory diagrams, when a = 5,
b = 3, c = 3, d = 5, k = 1, m = 2: (a) y − z plane;(b) x − z plane; (b) x − y − z plane.

1

0

-1

Furthermore, it can be seen that the fractional-ordersystem shows period state, when a = 5 , b = 3 ,d = 5 , k = 1 , m = 2 , and selecting c ∈ [4, 6] ,from Fig.1 and Fig.2. Particularly when fixing c =5, the period phase trajectory diagrams are showed
in Fig.4.

LE

-2

-3

-4

-5

1

2

3
c

4

5

2

2

1.9

1.9

1.8

1.8

1.7

1.7

6

z

Fig.1. Lyapunov exponents diagram when a = 5, b = 3,
d = 5, k = 1, m = 2, varying c.

z

-6
0

3

2.5

z

−

z

=

+

z

=−

⎧
⎪

1.6

1.6

1.5

1.5

1.4

3

1.3
-2

1.4

-1.5

-1

-0.5

0
y

0.5

(a)

2

1

1.5

2

1.3
-1.5

-1

-0.5

0
x

0.5

1

1.5

(b)

1

0

-1

-2

-3

-4
0

1

2

3
c

4

5

6

Fig.2. Bifurcation diagram when a = 5, b = 3, d = 5,
k = 1, m = 2, varying c.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

592

Analysis and Circuit Implementation

1.9
1.8

z

1.7
1.6
1.5
1.4
1.3
2
1

2
1

0

0

-1

-1
-2

y

-2

x

(c)

Fig.4. Period phase trajectory diagrams, when a = 5, b =3, c = 5, d = 5, k = 1, m = 2: (a) y − z plane; (b)x − z plane; (b) x − y − z plane.

Fig.6. Analog circuit of system (2)Where the electronic components mainly include resistors,
capacitors, the amplifier LF347BN and the analog
multiplier AD633 (output gain is 0.1). Parameters of theanalog circuit for the system are
=
=
=
=
=
=
=
=
=
=
=
= 10KΩ ,= 3.33KΩ ,= 3.33KΩ ,= 5KΩ ,=
20KΩ ,= 100KΩ ,=== 63.10MΩ ,=== 1.6M ,=== 15.8KΩ ,=== 0.4410μ ,=== 0.49μ ,=== 0.3μ , and theis variable resistor.
When= 33.33KΩ, the system (1) shows chaos, andthe results of circuit implementation are showed in Fig.7.
If changing= 20KΩ, corresponding system shows
period state, the results are showed in Fig.8. The results ofnumerical simulation and circuit implementation are
similar, Comparing Fig.1, Fig.2 with Fig.7, Fig.8,
respectively. Thus, this new fractional-order chaotic
system can be implemented.

3. Circuit implementation

The method of approximation conversion from time
domain to frequency domain is adopted to design acircuit for the novel system (2). Based on Bode diagrams,
the approximation of 1/ . with discrepancy 2dB is
described as follows:
.

≈(

.

.

(

)(

.

)(

. )

. )(

. )

There, the unit circuits can be used to achieve the 1/
unit, as shown in Fig.5.

Fig.5. Circuit unit of the 1/

(3)
.

.

Then, an analog circuit is designed to implement thisfractional-order system by the circuit unit of the 1/ . ,which is showed in Fig.6.

(a)

(b)

Fig.7. The results of circuit implementation, when a = 5, b =3, c = 3, d = 5, k = 1, m = 2: (a) y − z plane, (b) x − zplane.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

593

Zhiqiang Guo, Hongyan Jia, Shanfeng Wang(a)

6. Junguo Lu, Chaotic dynamics of the fractional-order Lüsystem and its synchronization. Physics Letters A Vol. 354(2006), p. 305
7. Yanling Guo, Guoyuan Qi, Topological horseshoe in a
fractional-order Qi four-wing chaotic system. Journal ofApplied Analysis and Computation Vol. 5(2015), p. 168
8. Junguo Lu, Guanrong Chen, A note on the fractional-order
Chen system. Chaos, Solitons and Fractals Vol. 27 (2006),
p. 685
9. Yongguang Yu, Hanxiong Li, Sha Wang, and Junzhi Yu,
Dynamic analysis of a fractional-order Lorenz chaotic
system. Chaos, Solitons and Fractals Vol. 42 (2009), p.
1181.
10. Hongyan Jia, Qinghe Wang, Qian Tao, Wei Xue, Analysis
and Circuit Implementation for the Fractional-order Chen
System. Chaotic Modeling and Simulation 2016; 3:
299-308.
11. Hongyan Jia, Qian Tao, Zengqiang Chen, Analysis andcircuit design of a fractional-order Lorenz system withdifferent fractional orders, Systems Science & Control
Engineering: An Open Access Journal 2014; 2: 745–750.
12. Cang Shijian, Chen Zengqiang, et al., Projective
synchronization of fractional-order memristive systems
with different structures based on active control method.
International Journal of Sensor Networks 2013; 14:102-108.
13. Weihua Deng, Changping Li, Chaos synchronization of thefractional Lü system, Physica A, 353(2005), 61–72.
14. Hongyan Jia, Qinghe Wang. Synchronization of two
four-wing fractional-order chaotic system and its
applications in secure communication, Atlantis Press 2016;
60: 1005.
15. P. Gholamin, A.H. Refahi Sheikhani. A new
three-dimensional chaotic system: Dynamical properties
and simulation, Chinese Journal of Physics, 55 (2017):
1300–1309.

(b)

Fig.8. The results of circuit implementation, when a = 5, b =3, c = 5, d = 5, k = 1, m = 2: (a) y − z plane, (b) x − zplane.

4. Testing and conclusion
In this paper, a new fractional-order chaotic system isinvestigated. By the numerical simulation, phase orbits
diagrams, the Lyapunov exponents diagrams, andbifurcation diagrams of the proposed system are also
given to verify its dynamitic characteristic. At last, ananalog circuit is designed to implement this system, andsome circuit results observed by the oscilloscope are
given to show chaos and period can be generated in the
new fractional-order system from the viewpoint ofphysics.
Acknowledgements
The research is partly supported by the Research Fund for
the National Natural Science Foundation of China (Grant
No.61573199). It is also supported by the Young
Scientists Fund of the National Natural Science
Foundation of China (Grant No.11202148).
References
1. Shihua Li, Haixing Cai, Research on circuitry realizationand synchronization of Chen chaotic systems, Acta Phys.
Sin, 53(2004), 1687-1693.
2. Chunguang Li, Guanrong Chen: Chaos in the fractional
order Chen system and its control. Chaos, Solitons andFractals Vol. 22 (2004), p. 549
3. Guanrong Chen, Ueta T, Another chaotic attractor, Int. J.
Bifurc. Chaos, 9(1999), 1465-1466.
4. Yongguang Yu, Hanxiong Li, Sha Wang, and Junzhi Yu,
Dynamic analysis of a fractional-order Lorenz chaotic
system. Chaos, Solitons and Fractals Vol. 42 (2009), p.
1181.
5. Changpin Li, Guojun Peng, Chaos in Chen's system with afractional order. Chaos, Solitons and Fractals Vol. 22(2004), p. 443© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan594

Application of a Conservation Chaotic System in Image Encryption
Shilong Liu, Mei Zhang, Wei Xue *College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
E-mail: * xuewei@tust.edu.cn
771398586@qq.com
www.tust.edu.cn

Abstract

In this paper, the dynamic behaviors of a conservative system are analyzed based on its phase trajectory,
Poincaré map, Lyapunov spectrum. On this basis, the conservative system is applied to digital image
encryption and the security performance of the algorithm is analyzed and verified. The conservative
system also has the characteristics of pseudo randomness and similar noise, whose safety is high, and is
not easy to be cracked. Therefore, the conservative system has a quite high research value andapplication prospect.
Keywords: Conservative; Chaotic system; Numerical simulation; Algorithm; Image encryption.

has potential application value in image encryption andsecure communication6.
With the rapid development of 4G network and periodof big data, a variety of image information spread rapidly
on the network. However, these information are facedwith the threat of stealing and destruction. How totransmit information safely and effectively becomes thefocus of attention. Due to high initial sensitivity,
unpredictability and similarity of the broadband power
spectral density and other prominent features, the chaotic
signal has natural concealment. Even if two identicalchaotic systems with a very small difference in initialconditions, their movement orbit will soon become
irrelevant. In early 1990, Matthews7 first applied thechaotic system to the image encryption algorithm. In theearly stage, the low-dimensional chaotic system isappliedtoimage
encryption.
Althoughthelow-dimensional chaotic system was designed easily, therandom sequence acquired was fast.8 However, lowcomplexity and low security is easy to be captured byreconstructing the attractor's embedded delay to decipher

1. Introduction
In 1954, Kolmogorov, the former Soviet Union master ofprobability, delivered the famous paper "Hamiltonian
function with minor changes in conditions of periodic
motion to maintain", which lays the foundation for theconservative system to produce chaos theory. In 1963,
Lorenz E N, an American meteorologist, discovered thefirst chaotic attractor-Lorenz attractor in dissipative
systems and then delivered a paper "Deterministic
Nonperiodic Flow"1, which reveals a series of basic
properties of chaotic motion, such as deterministic
aperiodicity, extreme sensitivity to initial values,
long-term behavior unpredictability, and so on. His
research opened a new door for the development of chaos
and greatly inspired the enthusiasm of scholars for the
study of chaos at home and abroad. Subsequently, many
new systems have been proposed to promote the rapid
development of chaotic dynamics theory, such as Chensystem 2, Lü system 3, Liu system 4, Qi system 5, and soon. The chaotic signal based on the conservative system© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

595

15

15

10

10

5

5

0

0

w

the information.9 The conservative chaotic system has
the characteristics of pseudo-randomness, similar to noise.
10
More importantly, there is no obvious chaotic attractor
in the conservative system. Different initial values canlead to different kinetics, which makes conservativechaos for image encryption with obvious advantages.11
Image encryption based on chaotic system processes
digital image that makes the cipher-text image disorder,
covers the plaintext information and achieves the effectof image encryption.12 In this paper, the application of theconservative chaotic system proposed in literature 13 hasbeen studied in image encryption.

z

Shilong Liu, Mei Zhang, Wei Xue

-5

-5

-10

-10

-15
-15

-10

-5

0
x

5

10

-15
-15

15

-10

1

In the literature 13, a new four-dimensional autonomous
dynamical conservative chaotic system is constructed.
The system contains four quadratic terms. The sum of thequadratic term and the primary term coefficients is also 0.
The mathematical model as follows:

0.5

0.5

w
w

0

1.5

0

15

0

-0.5

-0.5

-1

-1

-1.5

-1.5

-2

-2
-2

-1.5

-1

-0.5

0
x

0.5

1

1.5

2

-2.5
-2.5

2.5

-2

-1.5

-1

-0.5

0
z

0.5

1

1.5

Fig.2 Poincaré map

22.5

The Lyapunov exponent can display motion
characteristics of the system. Its positive and negativevalues and size in one direction, denotes the degree ofdivergence or convergence of the adjacent orbits in theattractors for a long time. 14 In order to describe theoverall effect of nonlinear dynamics, the Lyapunov
exponent is obtained by averaging the stretching orcompressing rate of each point of the motion track after along time. The Lyapunov exponent of the system isshown in Fig.3. When b=5.3, there are L1=1.7075,
L2=0.0975, L3=-0.0781 and L4=-1.7375, respectively. The
results show that the exponential value are approximately
symmetric with the X-axis, and the system (1) isconservative.

(1)

z
z

2

1.5

-2.5
-2.5

On evaluating the dissipativitye of the new system, it isfound that the divergence flow of (1) is conservative if
and only if V 0 .

y
y

10

2.5

2

w

y

1

x
x

5

Fig.1 Phase portraits of system (1)

2.5

V

0
x

Fig.2 is the Poincaré map of the system (1) with theabove parameters at z=1 and y= 1. It can be seen that thex-y cross section and z-w cross section generate a dense
and fractal point set. The system is chaotic.

2. Numerical analysis of a conservative chaotic
system

dx
ayw
dt
dy
xz
dt
dz
xy bw
dt
dw
axy bz
dt

-5

(2)

2.5

Calculation manifests that the theoretical value of V
is 0. Hence the system is conservative.
When the system parameters are taken a = 10, b = 5,
and system initial value is (1, -1, 1, -1), the conservative
system phase trajectory show in Fig.1:

2

Lyapunov exponents

1.5
1
0.5
0
-0.5
-1
-1.5
-2
-2.5
4.5

5

5.5

6

6.5

7
b

7.5

8

8.5

9

9.5

Fig.3 Lyapunov exponent spectrum of system (1)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

596

Application of a conservation

3. Research on encryption algorithm based onconservation chaotic system

4.1. Histogram analysis

900
600
800

500

700

Number of pixels

Color images are made up of red, green, and blue, while
they are stored in the form of matrix in the computer. The
image is encrypted by using the space-based scrambling
diffusion algorithm. Firstly, three primary colors are
separated from the color image. We can get three primarycolor matrixes IR1、IG1、IB1. Secondly, the key, which iscomposed of system parameters and initial values, isinserted into the chaotic system to generate chaotic
sequence. The three primary color matrices IR2、IG2、IB2
are obtained by XOR encryption operation with IR1、IG1、
IB1 trichromatic matrices and chaotic sequences. Finally,
the encrypted image is synthesized by the encrypted three
primary color matrix. The original image encryption
results are shown in Fig.4:

Number of pixels

3.1. Encryption process

The grayscale histogram describes the number of pixels
in each gray scale. The G component before and after
encryption are shown in Fig.6. It can be seen from thediagram that the distribution of the pixels on each graylevel fluctuates greatly before encryption. Because thedifference of points is large, the distribution of image
information is obvious. After encryption, the distributionof each grayscale pixel points is more uniform and lessfluctuant. It can effectively covers the distribution rule ofpixels in the original image. Meanwhile, it also can resist
the attack of statistical analysis and enhance the securityof image information.

600

500

400

300

400

300

200

200
100
100

0

0
B分 量 的 原 始 图 像 的 直 方 图
0

50

100

150

利 用 混 沌 序 列 y进 行 空 域 图 像 的 直 方 图 结 果

200

250

0

50

100

150

200

250

Gray level

Gray levelFig.6 Histograms before and after image encryption
Fig.4 Encryption results of the original image

4.2. Adjacent pixel correlation analysis
Tab.1 shows the correlation coefficients of the image
pixel in various directions. From the original image andencrypted image, the pixels of one row and one column
are taken to calculate the correlation coefficient ofadjacent pixels. The correlation coefficients of theadjacent pixels in the horizontal and vertical directions
are shown in Tab.1:
Tab.1 The correlation of adjacent pixels
Different
Verticaldirections Horizontal
Different
direction
direction
imagesThe original image
0.97757
0.9555R pixel matrix
Encrypted image
0.0042708
0.001601
R pixel matrix
The original image
0.97161
0.94387
G pixel matrix

3.2. Decryption process
The process of decrypting a ciphertext image is theinverse of the encryption. First of all, the encrypted three
primary color matrixes IR2、IG2、IB2 are separated from theciphertext image. The key is inserted into the chaotic
sequence generated by the chaotic system. According to
the sequence matrix, the image is performed by XOR
inverse operation. Then the sequence generated by the
chaotic sequence is used to scramble the pixel position of
the three primary color matrix after XOR decryption.
Finally, the original image is synthesized by the three
primary color matrix. Decryption results are shown in
Fig.5.

Encrypted image
G pixel matrix
The original image
B pixel matrix

Fig.5 Decryption results of the original image
4. Algorithm security performance analysis

0.00021523

0.0022991

0.95435

0.92538© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

597

Shilong Liu, Mei Zhang, Wei Xue

No.61573199). It is also supported by the Young
Scientists Fund of the National Natural Science
Foundation of China (Grant No.11202148).

Encrypted image
0.00039443
0.003629
B pixel matrix
It is easy to see that the correlation coefficient of theadjacent pixels in original image approaches 1. The
coefficient indicates a high correlation between an
adjacent pixel and the other. The correlation coefficientof the encrypted image is close to 0, and the adjacentpixel points are basically irrelevant. Therefore, thealgorithm has good security for statistical analysis.

References
1. Lorenz E N. Deterministic non-period follow [J]. Atoms
Sci. 1963, 20(2): 130-141.
2. Chen G R, Ueta T. Yet another chaotic attractor [J].
International Journal of Bifurcation and Chaos. 1999,
9(7): 1465-1466.
3. Lü J H, Chen G R. A new chaotic attractor coined [J].
International Journal of Bifurcation and Chaos. 2002,
12(3): 659-661.
4. Liu C X, Liu L, Liu T etal. A new butterfly-shaped
attractor of Lorenz-like system [J]. Chaos, Solitons andFractal. 2006, 28(5): 1196-1203.
5. Qi G Y, Chen G R, van Wyk M A, et al. A four-wingchaotic attractor generated from a new 3-D quadratic
autonomous system [J]. Chaos, Solitons and Fractals.
2006, 28(5): 1196-1203.
6. CANG Shijan, WU Aiguo, WANG Zhonglin, XUE Wei.
Chaotic Behavior of a Generalized Hamiltonian System
and Its Circuit Implementation [J]. Complex Systems andComplexity Science. 2017 , 14 (1) :103-110
7. Matthews. On the Derivation of a “Chaotic” encryption
algorithm [J]. Cryptologia. 1989, 13(1): 29-42.
8. Huang L L, Xin F, Wang L Y. Circuit implementation andcontrol of a new fractional-order hyper-chaotic system[J].
Acta Physica Sinica. 2011, 60(1): 010505-1–010505-9.
9. Grigorenko I, Grigorenko E. Chaotic dynamics of thefractional Lorenz system [J]. Physical Review Letters.
2003, 91(3): 034101-1–034101-4.
10. Q. Zhang, H. Zhang, Z. Li, One-way Hash function
construction based on conservative chaotic system, in:Information Assurance and Security, 2009
11. Shijian Cang, Aiguo Wu, Ruiye Zhang, Nose-Hoover-likesystem: Theoretical analysis and numerical demonstrations,
Preprint submitted to Applied Mathematical Modelling,2017
12. Wang Q Y, Zhou S B. Image encryption algorithm basedon fractional-order Chen chaotic system [J]. Journal ofComputer Applications. 2013, 33(4): 1043-1406.
13. Cang
Shijian,
Wu
Aiguo,
Wang
Zenghui.
Four-dimensional autonomous dynamical systems withconservative flows: two-case study [J]. Nonlinear
Dynamics .2017, 89:4
14. XUE Wei, XIAO Hui, Xu Jinkang, JIA Hongyan. A
fractional-order hyper-chaotic system and its application inimage encryption [J]. Journal of Tianjin University of
Science & Technology. 2015(5): 67-71

4.3. Key sensitivity test
The system parameter a= 10, b= 5, and the initial value
are set to (1, -1, 1, -1). In this condition, we encrypt theimage. When you enter the correct key, the image can be
accurately decrypted. If we change the initial value as y0= -1.000000001, other initial values and parameter values
are unchanged, the image decryption fail. As shown in
the Fig.7, it can be seen that extremely small key changes
cannot successfully decrypt the image, which indicatesthe high sensitivity to the key in this system.

Fig.7 Decryption result of the wrong key
5. Conclusion
In this paper, the chaotic characteristics of a conservative
chaotic system are analyzed by plotting the phasetrajectories, Lyapunov exponential graph and Poincaré
cross section map. According to the chaotic sequence
generated by the system (1), the digital image isencrypted and decrypted by pixel scrambling. The
application of image encryption in image encryption is
studied. Finally, the security performance of the image
before and after encryption are analyzed, such as thehistogram, the correlation of neighboring pixels and the
key sensitivity. The results show that the encryption
algorithm based on this system has a good security,
reliability and application value.
Acknowledgements
The research is partly supported by the Research Fund for
the National Natural Science Foundation of China (Grant© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

598

Study on Plant Disease Detection Based on Near Field Acoustic Holography
Xiuqing Wang *, Jiangfan Wang, Jimin Zhao, Qing Liu, Qi Chen

College of Electronic Information and Automation,
Tianjin University of Science and Technology, China
E-mail: * lwxqly@163.com
www.tust.edu.cn

Abstract
In the process of crop cultivation, pests and diseases have the most obvious impact on crop growth, yield andquality. In order to quickly and effectively prevent pests and diseases, real-time understanding of crop growth, inorder to reasonably accurately and accurately spraying pesticides based on disease information is particularly
important. The key to implementing precision spraying is accurate positioning of crop diseases. In this paper, themain stem of plant is taken as the research object. Through the acquisition of acoustic emission signal, the acoustic
field model of disease stress is established based on near-field acoustic holography, and the sound field of
three-dimensional space is reconstructed by algorithm to analyze the sound field of the sound source. Identify theplant's sound source signals, locate the sound source location, make an effective judgment of the plant damage
status, determine the disease situation, and establish the optimal control strategy.
Keywords: Disease detection, Near-field acoustic holography, Statistical best, Distribution source boundary point
method

1. Introduction

2. Statistics optimal cylindrical near-field acoustic
holography

Although China is a big agricultural country, the qualityof grain is declining year by year. The main reason is thatcrop diseases are more serious and the technology for
detecting plant diseases is not advanced enough. In order
to effectively and accurately spray pesticides on plants,
we need to develop a new method that can break theprevious constraint of detecting plant diseases.
The study found that the acoustic emission signals
produced by crops under certain conditions are one of theimportant sensitive information for understanding their
growth status and disease status. Through acoustic
holography [1-3], the study of acoustic emission signals
can identify the source of acoustic emission and locate
the plant disease defect site in the early stage without
obvious symptoms, and can spray pesticides precisely
and efficiently, effectively manage disasters and increase
economic benefits.

2.1Fundamental
Taking into account the stem of the plant as a cylinder,
the use of cylindrical measurement can obtain more andmore accurate sound field information to improve thereconstruction accuracy, it uses the cylindrical surface asa holographic reconstruction. From the wave equation,
the Helmholtz equation can be obtained for a steady-statesound field that does not depend on time variables.
2
p(x, y, z)
k 2 p (x, y, z)
0
（1）
2
Where
is the Laplace operator, p ( x , y , z ) is thecomplex sound pressure at the space point, p ( x , y , z )is a function of Cartesian coordinates x , y , z;k
/ c 2 / is the acoustic wave number; c is theacoustic velocity; λ is the acoustic wavelength.
If the integral operation is regarded as a summation
operation, r=a is equivalent to the minimum cylinder© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan599

Xiuqing Wang, Jiangfan Wang, Jimin Zhao, Qing Liu, Qi Chen

radius surrounding the sound source, the holographic
cylinder radius r=rh, and the position relationship between
the reconstructed cylinder radii r=rs and r=a are shown in
Fig.1.

Fig.1. Cylindrical acoustic holographic simulation model

According to the above formula and the relationshipbetween the holographic surface and the reconstructedsurface and the smallest cylinder, the relationshipbetween the two-dimensional Fourier transform of thesound pressure on the reconstructed cylinder and theholographic cylinder can be obtained asN（2）p( rs )
Cn(rs) p( rH ) pT c (rs ) pT ( A A 2 I ) 1 A b

2.2.1 Single source simulation and analysis
The main variables affecting reconstruction accuracy
are: reconstruction radius rs, holographic measurement
radius rh, measurement height L, Holographic
measurement point M*N, where M is the number ofmeasurement points in the circumferential direction and
N is the number of measurement points in the axial
direction. This topic select the sound source frequency of
60 kHz simulation experiments, modify the relevant
experimental parameters to find the best recognition
effect. Without changing the number of holographic
points, the number of holographic points selected is 9 * 9.
The radius of reconstructed radius, the radius and heightof the holographic measurement surface, the distance
between the reconstructed surface and the holographic
surface are continuously adjusted, and the influence on
the reconstruction effect is observed.

nn 1

Equation (2) is NAH reconstruction formula based on
the statistical optimal cylinder [4-6].

Fig.3. Reconstructed sound pressure map for
rs = 0.045 m, rh = 0.05 m, L = 0.9 m

It can be seen from Fig.3 that the amplitude of thesound pressure of the sound source is conspicuous, andthe reconstruction sound pressure of each measuring
point is close to the corresponding theoretical soundpressure, which can realize sound source recognition.

2.2Numerical simulation
Simulation process is as follows:
Measurement point sound source radiation to the holographic surface of the actual

Set the parameters in the preparation of a good reconstruction algorithm

Sound Pressure Reconstruction with Statistical Optimal Reconstruction Algorithm

Fig.4. Reconstructed sound pressure diagram for
rs = 0.025m, rh = 0.08m, L = 1.45m

Reconstruction of the sound field to obtain three-dimensional map, confirm the sound source location

As can be seen from the experimental simulations in
Fig.3, Fig.4 and more, several changes to the holographic
measuring radius and height of the plant lead to thefollowing conclusion:
The maximum radius of the experimental subject(main stem of the plant) is 0.1 m and the maximum
height is 1.5 m. When the change of the holographic
cylinder radius exceeds 0.1m and the cylinder heightexceeds 1.5 m, the adjustment of other parameters
cannot achieve the expected reconstruction effectand the sound source cannot be identified.
The reconstructed result is better when the ratio ofreconstructed radius to holographic radius is greaterthan or equal to 0.4, which can accurately identify
the sound source position. As the distance between
the holographic surface and the reconstructedsurface decreases, the reconstruction error decreases
and the position of the sound source becomes more

Fig.2. acoustic holographic algorithm flow chart

The experiment uses R15a acoustic emission sensor tomeasure, the resonant frequency of 150 kHz, theoperating frequency range of 50 ~ 200 kHz, so choose 50~ 200 kHz as the simulation frequency range. By
continuously modifying experimental parameters such asholographic cross-section radius and holographic
cylindrical height, measuring point spacing andreconstructing cross-section radius, the parameter range
to achieve the best simulation results is explored torealize sound source reconstruction.
Assuming that the acoustic emission signal is at thecenter of the plant axis, the acoustic field generated isassumed to be independent of the corner of the cylinderand changes only as the z-coordinate changes to reveal
the cylinder surface. The raised position is the AE signal
source position.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

600

Study on Plant Disease

and more prominent.
Under the same parameter configuration, thereconstruction accuracy will decrease with theincrease of the sound source frequency. Although
the reconstruction results deviate from thetheoretical results, the overall trend is unchanged
and the sound source localization can still beachieved.
The parameters of holographic measurement points,
the height of holographic measurement surface, theradius of reconstruction surface and the radius ofholographic surface are mutually influential. The
accuracy of obtaining sound pressure information
will increase with the height of holographic cylinder
and the number of measurement points but themeasurement points and cylinder height is too large
or too small, will have an impact on thereconstruction results.
2.2.2 Multiple source simulation and analysis
Due to the special physiological characteristics ofplants, acoustic emission signals do not occur singly.
Therefore, noise sources with multiple frequencies should
be simulated. In order to eliminate the influence of
ambient noise on the reconstructed results, filtering needs
to be performed. In this study, the windowing function ofsimultaneous filtering in the axial and circumferential
directions proposed by Chen et al.
Based on the single-source simulation in thisexperiment, this section introduces noise at multi-source
frequency f1=80kHz ， f2=100kHz ， f3=120kHz whenrh=0.05m，rs=0.045m, the measurement point is 9 * 9,and the height of measurement surface is 0.9m. The
signal to noise ratio is 35 dB, and the introduction of thewindow function to filter noise.
Specific simulation results shown in Fig.5, Fig.6.

sound source can still be judged.
In summary, the sound pressure information is
acquired through the R15 acoustic emission sensor array,and the distribution of acoustic emission source locations
under different disease levels can be obtained through thereconstruction algorithm. The localization of a soundsource containing random noise can be achieved byadding a window function to perform spatial wave
number domain filtering.
3. Distribution of source boundary point method
3.1. Fundamental
According to the consolidation deduction of the known
formula, the sound pressure value on the reconstruction
surface is obtained by singular value decomposition andgeneralized inverse transformation of the transfer matrix
Gp, and the specific reconstruction formula is as follows:（3）
pc Gp pq Vp p 1Up H pq
In the formula, the superscript "+" represents thegeneralized inverse; pq is the M-order acoustic pressure
vector on the holographic surface; and pc is thereconstructed surface acoustic pressure.
3.2. Numerical simulation
3.2.1Single source simulation and analysis
According to the size of the general plant radius, theradius of the holographic cylindrical surface is selected tobe 0.01m, the radius of the reconstructed sphericalsurface is 0.009m, and the sound source is located indifferent positions for f = 60kHz.f = 60 kHz, assuming
the sound source is located d=0.006m from the center ofthe sphere (at the origin) in the direction of the 7th node.
Respectively h=0.001m; h=0.003m; h=0.005m;
h=0.007m; compare the reconstructed value with thetheoretical value, the result is shown in Fig.7.

600

500

600

450
500

400
350

400

声 压 幅 值 /Pa

300

200

300
300

250
200

200

150
100
100

100
0

0

0.4 0.3 0.2 0.1
-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0 -0.1-0.2 -0.3

-0.4

0-3.1416

高度/m

高度 /m

3.141650

弧度/rad800

表 面声 压 幅 值 /Pa

重建值

500

声 压 幅 值 /Pa

声 压 幅 值 / Pa

400

0

0.1

0.2

0.3

0.4

高 度 /m

200

200

2

4

6

8
结 点 个 数 /N

10

12

0.1

0 -0.1-0.2

高 度 /m-0.3-0.4

0-3.1416

50

14

0

2

4

6

8

重建声压

理论声压

1200

表面声压幅值/Pa

300
250
200

800
600

200

100
50

1000

400

150

As can be seen from Figure 2-6 and Figure 2-7,although the filtered reconstruction value and thetheoretical value may have errors, the location of the

理论声压

1400

350

= 0.2, kc = 20

14

1600
重建声压

400

弧 度 /rad12(b) h = 0.003m

450

50

10

结 点 个 数 /N

100

0.4 0.3 0.2

3.1416

Fig.6. Reconstructed sound pressure map of filteredThree sound sources when

200(a) h = 0.001m

150

-0.1

250

100

250

0

-0.2

300

150

0

300
300

100

-0.3

300

0

350

400

-0.4

400

100

450

0

500

500

600

100

350

200

400

理论声压

400

600

理论值

500

200

重建声压

理论声压

700

600

300

450
重建声压

Fig.5. Reconstructed sound pressure map of filteredThree sound sources when = 0.2, kc = 17

表面声压幅值 /Pa

声压幅值/ Pa400

表 面声 压 幅 值 /Pa

理论值
重建值
500

0

2

4

6

8

10

12

14

0

0

2

4

6

8

10

结 点 个 数 /N

结 点 个 数 /N(c) h = 0.005m(d) h = 0.007

12

14

Fig.7. Comparison of reconstructed and theoretical values© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

601

Xiuqing Wang, Jiangfan Wang, Jimin Zhao, Qing Liu, Qi Chen

It can be seen from the simulation diagrams (a)，(b)，(c) and (d) that when f = 60 kHz, the sound source isassumed to be located d=0.006m from the center of thesphere in the direction of the 7th node with an interval of
0.002m from 0.001m to 0.008m, the final h=0.003m, thereconstruction accuracy is high, the reconstructed value isconsistent with the theoretical value, the other values of
the reconstruction error is larger.
3.2.2Dual sound source simulation and analysis
Since acoustic emission signals do not necessarily
occur uniquely, different frequency sources can be
represented by multiple frequency signals, each of which
represents the signal after each acoustic emission signal
passes through a narrow-band filter. According to theexperimental results of single-source simulation
parameters selection, f1 = 100 kHz and f2 = 150 kHz
multi-source reconstruction, the reconstruction results in
Fig.8.
4500

However, in this method, the measuring range of themain stem of the plant is larger than that of other methods,
which makes it easier for us to study the disease situationof more plants.
Distribution of source boundary point method can
avoid the problems of interpolation, numerical quadratureand singular integral in the algorithm. The reconstruction
result is highly sensitive to introducing noise and can be
regularized by singular value truncation filtering. The
method can realize the reconstruction of any shape of thesound source, but also can make the holographic surface
have more choices. The advantage is that high precision,
fast, widely used.
Acknowledgements
The research is partly supported by the Research Fund for
Tianjin Applied Basic and Advanced Technology
Research Program（14JCZDJC39000）.

重建声压
理论声压

4000
3500

0.32

4000

0.315

3500

References

3000

0.305

2500

z/m

表面声压幅值/P a

0.31

3000

2500

0.3
0.295

2000

2000

0.29

1500

1500

0.285

1000

1000

-0.02

500
0

500

0
y/m

1

2

3

4

5

6
结 点 个 数 /N

7

8

9

10

0.02

-0.02

-0.01

0

0.01

0.02

x/m

Fig.8. Reconstructed sound pressure diagram for
Two -source (100,150 kHz)

4. Conclusion
In this paper, for the characteristics of plant disease stress,
near-field acoustic holography is applied to the detectionof plant diseases. The main stem of plant is taken as theresearch object. Based on the acoustic emission signal
acquisition and near-field acoustic holography, theoptimal cylindrical near-field acoustic holography anddistributed source boundary point method were used toreconstruct the acoustic field information of the soundsource surface so as to identify and locate the location of
the acoustic emission source and determine the plantstress state. Through research, timely detection andidentification of diseases, so as to establish the optimal
prevention and control strategies to improve the control
effect and reduce the cost of prevention and treatment.
The statistical optimal cylindrical near-field acoustic
holography technology can accurately identify the
position of the sound source, eliminate the window effectand winding error, and improve the reconstruction
accuracy. However, this method can only be applied tothe conformal surface, which is unavoidable limitations.

1. Haibin Zhang. Non - conformal surface near - field
acoustical holographic theory and experimental study on
steady circumstance sound field [D]. Shanghai: Shanghai
Jiao Tong University, 2008. (in Chinese)
2. Yongbin Zhang. Near - field acoustic holography based onequivalent source method and particle velocity
measurement [D]. Hefei: Hefei University of Technology,
2010. (in Chinese)
3. Jinquan Zhang. Noise source identification and sound fieldvisualization based on beam forming technology [D].Hefei:
Hefei University of Technology, 2010. (in Chinese)
4. Williams E G ． Regularization methods for near-field
acoustical holography[J]．Journal of the Acoustical Societyof America，2001，110(4):1 976～1 988．
5. Williams E G，Houston B H , Herdic P C，et al．Fast
Fourier transform and singular value decomposition
formulations
for
patch
near-field
acousticalholography[J] ． Journal of the Acoustical Society ofAmerica，2003，114(3):1 322 ～1 333．
6. Moohyung Lee，Stuart Bolton J． A one - step patch near- field acoustical holography procedure[J]．Journal of the
Acoustical Society of America，2007，122(3): 1 662 ～1
670．

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

602

Simulation of Cell Dielectric Properties Based on COMSOL
Shudong Li, Xiaoyan Chen*, Fengze Han

College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
E-mail: * cxywxr@tust.edu.cn
www.tust.edu.cn

Abstract
The dielectric properties of cells can be observed by injecting a low amplitude current at different frequencies(1MHz~100MHz). The simulation research is taken on the software platform named COMSOL Multiphysics. The
electric field and the cell model is created with prior information. By simulation, it’s verified that at low frequencies,
the region of interest (ROI) behaves the conductivity characteristic while the electrical signal cannot pass through thecell membrane due to its capacitor properties. With the excitation frequency increasing, the ROI behaves more
permittivity characteristic that the current flowing through the cell membrane becomes more and the current density
increases. The research of the cell dielectric properties provides an auxiliary method to diagnose the status of the cell.
Keywords: dielectric properties, cell model, excitation frequency, finite element method (FEM), auxiliary method

The cell membranes are composed of insulating bilayer
phospholipid and proteins. The membranes separate theintracellular and extracellular fluid. However, water andsome micro-molecular proteins can exchange between
intracellular and extracellular fluid through membranes.
This is because membranes are permeable selective due tothe structure of itself.

1. Introduction
1.1 Cell structure
From the biological point, cells are the smallest units that
constitute the biological tissues. Most biological cells
consist of cell fluid and membrane. The cell fluid can bedivided into intracellular fluid and extracellular fluid asshown in Fig.1. Intracellular fluid can be regard as saline
solution which consists of water mainly, inorganic salt andother metabolite. Extracellular fluid provides the living
environment for cells with appropriate PH value, saline
concentration and so on. Both of the two fluids have goodelectrical conductivity [1].

1.2 Cell dielectric properties
The specified cell structure determines its special electricalcharacteristics. When a direct current or a low frequency
alternating current is applied to the cell, the current cannot
flow into the cell because of the insulated membrane. Thisbehavior is called low-frequency dielectric property.
When the excitation frequency gradually increasing, themembrane becomes electrically invisible and the current
can inject into the cell by passing through the membrane.
This performance can be defined as high-frequency
dielectric property[2]. The current lines flow as shown in
Fig.2.

Fig.1. Simplified single cell structure model© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

603

Shudong Li, Xiaoyan chen, Fengze Han

1.3 Finite element method
The finite element method (FEM) was defined andpublished by Clough in 1960. With the development ofcomputer, the numerical solution of differential equations
has been developed and applied a lot. The main idea of
FEM is to approach an entire continuous domain by finite
discrete domains. The unknown function on the entire
domain is represented by functions on each discrete
domain approximately[4]. A commercial software named
COMSOL Multiphysics is a powerful solution engine.
2. Method
This section uses the software COMSOL Multiphysics tobuild the model of single cell. The geometric shape ofmodel and the value of dielectric properties of cell are
selected referring to blood cells[5].

Fig.2. Current flows through the cell

According to the theory of cell dielectric properties, cell
membrane has the characteristic of capacitance and thetwo cell fluids has the characteristic of resistance[3]. The
electrical characteristics of single cell can be equaled to anelectrical circuit as shown in Fig.3(a), where theparameters represent different parts of the cell.
: equivalent resistance of membrane: equivalent capacitance of membrane: equivalent resistance of intracellular fluid: equivalent capacitance of intracellular fluid:equivalent resistance of extracellular fluid: equivalent capacitance of extracellular fluid
While
,
and
are much larger than,and , less current would go through them, so these canbe neglected to achieve a simplified equivalent electricalmodel as shown in Fig.3(b).

2.1. Single cell model
2.1.1 Modeling and parameter settings
The parameters of the single cell model is shown in Fig.4.
The meaning of the parameters is as follows:εe: permittivity of extracellular fluid
σe: conductivity of extracellular fluid
εi: permittivity of intracellular fluid
σi: conductivity of intracellular fluid
εm: permittivity of membrane
σm: conductivity of membrane
R: radius of single cell
Tm: thickness of membrane

Fig.4. Geometric and electrical parameters of single cell model

Fig.3(a). A equivalent circuit model of single cell(b). Simplified equivalent circuit model of single cell

As shown in Fig.4, the excitation current with current
density of 1A/m2 enters the region of interest (ROI) from© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

604

Simulation of Cell Dielectricparameters of two cell fluids and the four boundaries’
electrical characteristics are the same with the single cell
model in Fig.4.

the left boundary and flows out from the right boundary.
The above and bottom boundaries are electrical insulation.
2.1.2 Meshing grid
Meshing is the process by which the active geometricalspace (domain) of a model is sub-divided into a collectionof sufficiently smaller spaces so that linear or higher orderpolynomial approximations can be used as a reasonableanalog of the functional physical behavior being modeled,which is the core of finite element method.
The software COMSOL is used to generate the grid. If
using coarse meshing method (CMM) to structure the grid,
the accuracy of calculation results is not enough becauseof the less grids. But too refined meshing grid takes a lotof memory which leads to long time calculation. All things
considered, this paper used standard meshing method(SMM) that the number of generated elements is between
coarse meshing methods (CMM) and refine meshingmethod (RMM) to create the meshing grid as shown in
Fig.5 and Fig.7.

(a)(b)
Fig.6(a). Geometric model of multi-cell(b) Meshing grid of mutlti-cell model (2962 elements)
Table.1. Membrane electrical parameters of multi-cell modelGroup A
εm
1
2
3

5

σm
（S/m
）
10-2
10-5
10-8

Group B

Group C

εm

σm
（S/m
）

εm

3
5
7

10-5

3
5
7

σm
（S/m
）
10-8
10-5
10-2

3. Simulation result
3.1. Single cell simulation result
The current density distribution at three typical different
frequencies (1MHz, 50MHz and 100MHz separately) as
shown in Fig.8. It can be observed that there is no current
flowing through the cell model at 1MHz. It is because themembrane works as capacitance in parallel with a large
resistance. The membrane has high insulation and low
conductivity at low frequency. With the frequency
gradually increasing, a part of current starts to pass throughthe membrane at frequency 50MHz.. It's apparent that thecell membrane becomes electrically invisible when thefrequency reaches 100MHz .The current density in thewhole domain is homogeneous.

Fig.5. Meshing grid of single cell model(3014 elements)

2.2. Multi-cell model
Because of the special electrical properties of cell
membranes, cells show different characteristics in theelectric field with different frequencies and the changes ofcell membrane’s electrical properties lead to changes on
physiological state of cells.
In order to observe the dielectric properties of cells
under different physiological states, this paper establisheda multi-cell model based on the single cell model in section
2.1 as shown in Fig.6(a) and the meshing grid using SMM
is shown in Fig.6(b).
And three simulation works were implemented to watch
the influence on cells that caused by the changes on themembrane electrical parameters. Table.1 shows theparameters setting of three groups of experiments. The

3.2. Multi-cell simulation result
Table 2 shows the three sets of simulation results, in thesame current field, the three cells with different membrane
have respective frequency response. This because themembrane works as a capacitance in parallel with a large
resistance, when setting different electrical parameters© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

605

Shudong Li, Xiaoyan chen, Fengze Han(permittivity ε and conductivity σ) to the membrane, thevalue of the capacitance and the resistance changed. The
simulation results proves that the change of cell’s
physiological status can affect its dielectric properties on
account of changes on membrane.

1MHz

50MHz

This paper preliminarily build single cell model andmulti-cell model from microcosmic view and conduct atentative research at the dielectric properties of thebiological cell with finite element method. Comparing
with traditional method, this simulation method only needs
to set geometric and electrical parameters without building
a whole mathematical model. The simulation method
based on COMSOL was proved to be successful and thispaper provides an auxiliary method to diagnose the statusof the cell.

100MHz5. Acknowledgements

Fig.8.Current density distribution of single cell at different
frequency

The authors would like to thank the EIT research groupfrom Tianjin University of Science and Technology. This
paper was supported by a grant from the National Natural
Science Foundation of China (Grant No.61301246) and a
grant from the Science and Technology Project of Tianjin(12JCYBJC19300). It was also supported by the Reformin Education from Tianjin Municipal Education
Commission of China (171005704B).

Table.2. Multi-cell model simulation results

Group
A

(changes on
σm)

1Mhz

50MHz

100MHz

1Mhz

50MHz

100MHz

1Mhz

50MHz

100MHz

References
1. RJ Yerworth, RH Bayford, G Cusick, M Conway, DS
Holder, Design and performance of the UCLH Mark 1b 64
channel electrical impedance tomography (EIT) system,
optimized for imaging brain function, Physiol Meas,
2002,23: 149-158.
2. XY Chen, XQ Lv, Meng Du, Experimental verification of
depolarization effects in bioelectrical impedance
measurement, Bio-Medical Materials and Engineering,
2014,24: 3675-3683.
3. JR Wang, BY Sun, HX Wang, Qing Sun, Experimental
Study of Dielectric Properties of Human Lung Tissue inVitro, Journal of Medical and Biological Engineering, 2014,34(6): 598-604.
4. Jiaming Jin, The finite element method in electromagnetics,
John&Sons, 2014.
5. K Asami, Dielectric dispersion in biological cells ofcomplex geometry simulated by the three-dimension finite
difference method, Journal of Physics D: Applied Physics,
2006, 39(3): 492.
6. A Tchako, A M Sadeyh, Stress Changes in intervertebral
discs of the leruical spine due to partial discectomies of andfusion, Journal of Biomechanical Engineering, 2009,
131(5):1-11.
7. C Ding, X Liu, P Shang, Progress of the technology andapplication of dielectric spectroscopy of cell suspensions,
Beijing Biomedical Engineering, 2014, 33(1):94-99.

Group
B

(changeson εm)

GroupC(changeson εmand σm)4. Discussion
The finite element method has been widely used inmechanics of biological tissue[6]. But at the cell level, there
are few papers that introduce the electromagnetic
characteristics [7].

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

606

Research on the Method of Electrical Impedance Tomography Based on Conjugate Gradient
Iterative Algorithm
Yuanli Yue 1*, Xiaoyan Chen2, Ze Liu1, Fengzhi Dai 2
1

College of Electronic Information Engineering ,Beijing Jiaotong University, China;
2 College of Electronic Information and Automation,

Tianjin University of Science and Technology, China;

E-mail: * 17120205@bjtu.edu.cn
www.bjtu.edu.cn

Abstract

Electrical impedance tomography is a new technology of medical imaging, which can be used to detect
the real time internal electrical impedance. This article is doing research on the method of electrical
impedance tomography by conjugate gradient iterative algorithm. The finite element analysis software
COMSOL is utilized to make a two-dimensional model about a circular field with 16 electrodes. The
software MATLAB is used to finish the tomography reconstruction work by applying two different
reconstruction algorithms. After tomography reconstruction, a variety of evaluation methods andnumerical analysis is applied to estimate the reconstructed image quality, such as the correlation
coefficient and the similarity relative error evaluation.
Keywords: EIT, conjugate gradient iterative algorithm, COMSOL, reconstruction, MATLAB

different when human body in different pathologicalstates4. Based on this feature, EIT can analyze theabnormal electrical conductivity in the tissues or organs,and then determine the lesions of organs and tissues. For
the existing medical imaging technology, such as
Computed Tomography (CT), type-B ultrasonic, andnuclear magnetic resonance techniques. These diagnose
methods use harmful X-rays and harmful radionuclides
when performing body scans and tests, and have thedisadvantage of high technical cost. On the contrary, EIT
is non-toxic and harmless to human body5.
EIT has outstanding advantages, large technicalpotential and broad application prospects. The main
advantages of EIT technology can be broadly described in
the following aspects:
Electrical impedance tomography equipment is smallin size, simple in structure and easy to carry.

1. The introduction of EIT
Electrical impedance tomography (EIT) is a newly
developed technique that can be used to detect internal
images1. It is used to detect electrical impedance of thebiological tissues. The principle of EIT is based on theelectrical characteristics of the medium inside the field. By
injecting exciting current, the voltages on the boundary aremeasured. It can estimate the distribution of the
conductivity changes2. By detecting the collected voltage
data, using an algorithm to reconstruct the distribution of
the electrical conductivity inside the field. Because theconductivities of different human tissues are different, theconductivity of different regions located at same section is
also different. The data measured by EIT system candirectly reflect the distribution of conductivity and its
variations3.
The conductivity of tissues and organs in human bodyis different, and the characteristic value of conductivity is© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

607

Yuanli Yue,Xiaoyan Chen,Ze Liu, Fengzhi Dai

Electrical impedance tomography has the characteristics
of portability and flexibility, and the equipment can beminiature the size of the whole system to a laptop size.
Moreover, the operating system is in graphicalvisualization6. Because of the small size, the EIT
equipment can be transported to the scene of the accident
in time, saving time for first aid.
The imaging speed is fast and in real-time
EIT has fast imaging speed, so it can be used as a realtime medical equipment for clinical monitoring andbedside monitoring. Other medical imaging techniques
have hysteresis and their imaging speed is slower. At thisstage, many researchers are studying electrical impedance
tomography.

effective evaluation method of electrical impedance image
in the future.
3. The principle of EIT
Principle of electrical impedance tomography is anextension of Ohm's law, setting the resistance of the object
is R, the exciting current applied to the resistor R, and themeasurement of surface resistance voltage R on target canbe obtained the resistance value. After the weak excitation
current is injected, the system can detect different current
flows on the object with different resistances. Through themeasurement of a large number of current data, theappropriate algorithm can be used to reconstruct the imageof the resistance and conductivity of the target in different
regions.
A certain number of electrodes are attached to thesurface of the body, and the current is used as theexcitation signal to the measured region or object. Measure
the voltage on the surface of the object, select different
excitation and measurement mode, and observe theconductivity of the measured area. The working system
diagram is shown in the Fig.1:

2. Research of the field
EIT has a broad application prospect and advantages. The
researchers are mainly focus on the accuracy ofreconstruction and the speed of reconstruction. While theresearch are still in the initial stage at China, and there is
still a big gap between the research progresses. The
researches in China mainly focus on the improvement of
the algorithm. At present, the technology should be closely
integrated with the clinical and practical. In the future,researcher need to develop high-quality algorithms andeffective incentive model7.
At present, the development of EIT has somedeficiencies compared with other technologies in image
resolution and measurement accuracy. But it has its
advantages, it can be designed according to the needs of its
system. In real-time monitoring, it has great advantage.
These advantages makes it becomes the main direction ofcurrent imaging technology. In addition to the applicationof medical imaging and clinical medicine, EIT can also be
applied into industry to do industrial non-destructive
testing, fluid detection, nondestructive testing and fault
diagnosis. Up to now, EIT systems have been involved in
the field of industrial testing and medical applications.
The research of EIT mainly include forward problem
research, inverse problem research and hardware systemdesign. There has include conjugate gradient iterativealgorithm, sensitivity coefficient method, NewtonRaphson method, Landweber iterative method, Tikhonov
regularization, equipotential back projection method8. It isa difficult problem to evaluate the effectiveness of EIT in
the future, and that is an important subject to establish an

Fig.1. System work diagram of EIT

3.1. Forward problem
According to the electrical conductivity of the target, theexcitation signal is applied to the boundary. By solving the
Laplace equation, the node voltage in the region is
obtained, and can get the electric potential distribution in
the field. Then, the boundary voltage of the measured
object is obtained, so that the electric field can be analyzed
step by step, which is the forward problem of EIT.
The EIT forward problem is shown in the Fig.2:© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

608

Research on the Method

inverse problem of EIT, also known as the inverse problem
of EIT. The inverse problem process is divided into two
kinds of imaging methods, dynamic imaging and static
imaging. The solution of the inverse problem is that thevoltage solved by the forward problem is the same as theactual measured potential difference within the specified
deviation, and the reconstructed image solved by the
inverse problem is the approximation of the real image.
Finally only can get approximate solution, it is difficult to
get an accurate solution. The EIT inverse problem isshown in the Fig.3.

Fig.2. Solving the forward problem of EIT

The target of EIT is the electric field under the special
boundary condition, and the electric field is described by apartial differential equation. In the process of solving theequation, the image object is equivalent to the resistance,
so it can be converted to the process of solving theMaxwell equations. The excitation frequency used in themeasurement is set from 10 to 100Hz, and the free charge
movement and the dielectric constant in the target are
ignored in the experiment. The Maxwell equations are
obtained:
D

B

O

H

Fig.3. EIT inverse problem

B
t

E
j

The inverse problem of EIT is mainly based on the
known conditions and other information to reverse theunknown parameter information in the target object, so as
to reconstruct the more realistic image. In the process ofstudying the inverse problem, it is mainly to find theappropriate algorithm to improve the stability of thereconstruction operation to reduce the amount ofreconstruction. In this paper, using the conjugate gradient
iterative algorithm to reconstruction.

Dt

In the formula, H is the strength of the magnetic field, J
is the current density, D is the potential shift, B is themagnetic induction, E is the field strength, is the Poisson
operator)
The numerical method is mainly used in solving the EIT
forward problem, the main methods are finite volume
element method, boundary element method, and finite
element method and so on.

4. Introduction of Conjugate Gradient Iteration
Algorithm

3.2. Inverse problem

At present, the algorithm for the study of electrical
impedance imaging technology has a certain foundation,
there are many mature electrical impedance imaging
algorithms. The focus for different algorithms is different.
These algorithms have their own advantages and theirshortcomings, select the algorithm according to the target
needs, to estimate the effect of the image to choose. The
method used in this paper is conjugate gradient algorithm9.
Conjugate gradient iteration is an iterative method forsolving the numerical solution of sparse linear equations.
It is an algorithm between Newton algorithm and gradientalgorithm. Conjugate gradient iterative method is a more

The inverse problem of EIT is the process of solving theillicit nonlinear inverse problem. The key technique lies in
the reconstruction of the impedance image by giving thetarget boundary measurement and solving the electrical
impedance distribution in the model. By applying a weakexcitation signal to the measured object, the distribution of
the object inside can be obtained, and can get the boundary
voltage of the measured area. In essence, the solution of
the inverse problem is the process of reconstructing theimage, the reconstruction of the forward problem and thecharacteristic of the conductivity inside the object region
can be obtained. Therefore the problem is known as the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

609

Yuanli Yue,Xiaoyan Chen,Ze Liu, Fengzhi Dai

effective method to solve the nonlinearity. The
convergence speed is faster than the steepest descentmethod, and its jagging phenomenon is overcome. The
convergence is better than the Newton method and thecomputation is smaller. This algorithm is simple, easy toprogram, small storage space to make it an important way
to solve the optimization problem10. This algorithm only
needs to solve the first derivative, overcome theshortcomings of convergence, it has widely used.
The conjugate gradient iterative algorithm is a linear
search based on n (n is the highest order of the equations)conjugate directions. Theoretically, the iterative number ofsteps is n, so that we can find the optimal solution. It
applies to the coefficient matrix that is symmetric positive
definite.
Suppose there is a linear system Ax = b, A is a N × N
dimension symmetric positive definite coefficient matrix,
N 1
N 1
X R ,b R .In order to obtain the exact solution of
Ax = b, we define the following function：
f

x

1 T
x Ax
2

the inhomogeneous field set to 1:10. In order to enhance
the contrast, the homogenous field and the inhomogeneous
field are established, which are shown in the Fig.4.
Adjacent excitation measurement mode of operation tosolve the problem. The conjugate gradient iterativealgorithm is used to solve the inverse problem.

Fig.4. Two - dimensional model

The finite element analysis of the established twodimensional model can be divided into general meshing
grid and fine meshing grid. Split the target field into 957
nodes and 1707 units. Using relative excitation andadjacent modes, space-field and full-field simulation, theresults shown in Figure 5.

bT x

Then the gradient of f（x）is f
Ax b , or it can be
expressed by the margin function r Ax b When thegradient f or margin function r is equal to 0, and thefunction f (x) has a minimum value.

957 nodes,1707 units

5. COMSOL software two-dimensional model and
MATLAB simulation

3620 nodes,6828 units

Adjacent incentives
relative incentives
Fig.5. Different pattern of division and incentive mode

EIT problem mainly refers to the process of electrical
impedance tomography reconstruction. In this paper, thesoftware COMSOL is used to simulate the Jacobi matrix,
and reconstruct the conductivity distribution in the target
field. Using COMSOL to model, building a uniform
distribution of 16 around the electrode, the establishmentof a circular area. This paper solve the forward problem of
EIT accurately by COMSOL modeling to improve thequality of the image.
First, a two-dimensional model is created by COMSOL,
circular field is plotted, and 16 electrodes are distributed
around the circular field. The field conductivity is 0.037 s/ m2, the conductivity of the target small circle is 0.138
s/m2, and the electrode is selected as the titanium electrode.
The normal current density of the injected current is set to+1 A/m2 and -1 A/m2, and other electrodes are set to beelectrically isolated. The ratio of the homogenous field to

In the forward problem, the sensitivity matrix and thereference potential vector are generated. Then change thenumber of different iterations, imaging in MATLAB. The
number of iterations from 10 times to 90 times wereimaging, to see their imaging results. The result is shownin the Fig.6.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan610

Research on the Method

6.2. Relative error
The relative error is used to describe the reconstructed
image quality. The relative error is the deviation between
the actual image and the reconstructed image. Generally
speaking, when the value of the relative error is smaller,
the quality of the reconstructed image is better, the larger
the relative error value indicates that the reconstructedeffect of the image is poor. The relative error can be
expressed by:ER
is the true conductivity of the original image;isthe conductivity of the reconstructed image.
According to the results obtained in the image
reconstruction under MATLAB, the relative error anumerical image is shown in the Fig.7.

Fig.6. The images reconstructed by different iterations

6. Reconstruction of image quality evaluation
indicators and analysis
For the reconstruction images of MATLAB, evaluate thequality of their imaging results and images, the generalevaluation of electrical impedance tomography indicators
are similarity, the maximum correlation coefficient and therelative error. Using different metrics to evaluate thequality of reconstruction.

Fig.7. Relative error after 100 iterations

By comparing the number of iterations, the number of
iterations is 75, the relative error is the smallest.

6.1. Similarity
The structural similarity index refers to an evaluationindex that can be used to compare the similarity of two
different images. The range of similarity values ranges is 1 to +1. In the evaluation of the reconstruction, if the two
images are exactly same, the similarity value is 1. For thereconstruction, the more the value of the similarity iscloser to 1, the better the quality of the reconstruction is,
and the better the reconstructed image is consistent withthe original model.
Comparing and analyzing the similarity values obtainedwith different iterations is shown in Table 1-1. Increasing
the number of iterations to reconstruct the image accuracy
is getting better.

6.3. Correlation coefficient
The correlation coefficient is the evaluation index
proposed by the application mathematician Karl Pearson,
and it is mainly used to illustrate the degree of correlation
between two variables. The correlation coefficient iscalculated based on the difference between the twovariables and their average. The results of the two
deviations is defined to describe the correlation between
the two variables.
Compare the data of the largest correlation coefficient,
the result is shown in Fig. 8.It can be concluded that thecorrelation coefficient is close to 1 as the number of
iterations increases.

Table 1-1 SSIM and the number of iterations

iterations
SSIM

100
0.2450300
0.3034500
0.3380700
0.3388© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

611

Yuanli Yue,Xiaoyan Chen,Ze Liu, Fengzhi Dai

paper was supported by a grant from the National Natural
Science Foundation of China (Grant No.61301246) and a
grant from the Science and Technology Project of Tianjin(12JCYBJC19300). It was also supported by the Reformin Education from Tianjin Municipal Education
Commission of China (171005704B).

References

Fig.8. The correlation coefficient at 100 iterations

1. Ying Qian, New Advances in Medical UltrasoundEquipment, Medical equipment information, 2002(11): 2527.
2. Yan Wang, Study on Performance and Evaluation Methodof Electrical Impedance Tomography System. Beijing:
China Union Medical College, 2009.
3. Yongbo He, Tao Song, Huixian Wang. Magnetic resonanceelectrical impedance tomography. Beijing Biomedical
Engineering, 2006.2: 99-102.
4. Tang Mingxin, Dong Xiuzhen, Electrical impedance
tomography reconstruction impedance tomography,
Proceedings of a meeting on electrical impedance
tomography, Copenhagen, 1990, 5:212-216.
5. Shujun Chen. Image Reconstruction Algorithm and
Implementation of Electrical Impedance Tomography.
Nanjing: Nanjing University of Science and Technology,
2008.
6. S. M. Huang, A. Plaskowski, Tomography imaging using
two-component flow using capacitance sensors. J. Phys. E,
1989, 22(3): 173-177.
7. Lian Gong, Ping Huang, Yin Liu.An Effective Image
Reconstruction Algorithm for Electrical Impedance
Imaging.
China
ElectromechanicalSocietyofElectromagnetic Field Theory and Application Symposium,
1987.
8. Xiuzhen Dong, Mingxin Qin, Mengxing Tang. A Practical
Dynamic Electrical Impedance Tomography Algorithm.
Chinese Journal of Biomedical Engineering, 2000, 19 (3):348-352.
9. Seo J K, Kwon O, Ammari H, et al. A mathematical model
for breast cancer lesion estimation: Electrical impedancetechnique using TS2000 commercial system. IEEE. Trans
BME, 2004: 76-79.
10. Ruigang Liu, Xiuzhen Dong, Feng Fu.An Image
Postprocessing Algorithm for Electrical Impedance
Tomography, Medical and health equipment, 2006(08):1898-1906.

7. Conclusion
Through the analysis of the effect of imaging andevaluation of three indicators. It can be found that byincreasing the number of iterations within a certain range
to improve the quality and speed of imaging, can improve
the resolution. The correlation coefficient is increasing
infinitely close to 1, the relative error a is getting smallerand smaller, the value of similarity is getting bigger, whichindicates that the image quality of the image is getting
higher and higher. Finally, it can be seen in 75 times near
the best imaging results.
The shape of the circular field under 75 iterations tends
to be smooth, the contour is gradually clear, the artifacts
are gradually reduced, and the imaging effect becomes
better, which indicates that the quality of image
reconstruction is getting higher. As the number of
iterations continues to increase, the imaging effect does
not change significantly.
The wide application of EIT in medicine, can be used tosee the body of the internal tomography. The shortcomingof this paper is that only one imaging algorithm is used toimage, and then the algorithm can be improved andimproved, and the method of combining the various
algorithms is used to study the electrical impedance
imaging technology.
8. Acknowledgements
The authors would like to thank the EIT research groupfrom Tianjin University of Science and Technology. This

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan612

Research on Acoustic Emission Wireless Sensing System
Xiuqing Wang 1*, Qing Liu 1, Yang Li 2, Jimin Zhao 1, Jiangfan Wang 1
1
College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
2

TEXAS A and M University, U.S.A
E-mail: * lwxqly@163.com
www.tust.edu.cn

Abstract

Piezoelectric ceramic acoustic emission sensor and fiber Bragg grating acoustic emission sensor areused to build two kind of acoustic emission wireless sensing systems. The acoustic emission signal
source is simulated by breaking the automatic pencil lead core. The acoustic emission signal ismonitored by the acoustic emission sensor, and the characteristics of acoustic emission signal are
extracted and uploaded to the host computer platform through wireless sensor network. A comparative
experiment of piezoelectric ceramic acoustic emission sensing system and fiber Bragg grating acoustic
emission sensing system is carried out.
Keywords: acoustic emission, piezoelectric ceramic sensor, fiber Bragg grating sensor, wireless sensor

network

piezoelectric materials. With the development of fiber
Bragg grating technology, the advantages of fiber Bragg
grating acoustic emission sensor have been exploredand developed, and the sensor has been widely used in
the field of acoustic emission detection1. Acoustic
emission signals are changeable and have a wide
dynamic range and the signals are divided into burst andcontinuous acoustic emission signals2. In this paper,
acoustic emission signal source is simulated by breaking
the automatic pencil lead core, which will produce burst
acoustic emission signal. Considering the large amountof acoustic emission waveform data and the limited
speed of wireless network transmission, the method of
extracting
theacoustic
emission
waveformcharacteristic parameters is adopted to realize real-timemonitoring the acoustic emission signal through thewireless sensor network, which has been extensive
applied in acoustic emission detection since 1950s. The
characteristic values such as amplitude, rise time,descent time and duration are uploaded to the host

1. Introduction
Acoustic emission testing is an important nondestructive
testing technology, which has been widely used in many
fields, such as petroleum, electric power, railway, etc.
Because wired routing is too cumbersome and notsuitable for remote and harsh environment applications,
wireless sensor networks are widely used in industrial,
agricultural and other fields of automatic monitoring.
This paper builds acoustic emission wireless sensing
system combined wireless sensor network technology
for monitoring of the acoustic emission signal. Besides，
through the contrast experiment of piezoelectric
ceramics acoustic emission sensing system and fiber
Bragg grating acoustic emission sensing system, it canbe seen the unique advantages of the latter.
2. The Hardware Structure Design
The traditional acoustic emission system is mainly
based on piezoelectric ceramic sensors, whose working
principle is based on the piezoelectric effect of© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

613

XiuQing Wang , Qing Liu, Yang Li, Jimin Zhao, Jiangfan Wang

adopts the principle of intensity demodulation. In signal
processing module the electrical signals produced by thephotoelectric converter are amplified, filtered andextracted for characteristic values. Next, the processed
results are transmitted to the host computer through thewireless network for display, analysis, storage and otheroperations. At the same time, the host computer can also
issue commands to the system acquisition andprocessing module, such as starting or stopping
collection, change the configuration parameters and soon.

computer software through wireless sensor network tocomplete the subsequent processing.
2.1. Fiber Bragg grating
wireless sensing systemacoustic

emission

The principle of fiber Bragg grating sensing detection isthat the light propagating in the fiber core will scatter at
each grating surface. If the Bragg condition is satisfied,
the reflection of each grating plane will gradually
accumulate, and finally a reflection peak will be formedin the direction. In this paper, an acoustic emission
wireless sensing system based on fiber Bragg grating
technology is presented, whose block diagram is shownin Fig.1.

2.2. Piezoelectric ceramic
wireless sensing systemacoustic

emission

The structure of piezoelectric ceramic acoustic emission
system is similar to that of the system based on fiber
Bragg grating sensor on the whole, as shown in Fig.2.
The center response frequency of the piezoelectric
ceramic sensor is 151.37 kHz. The piezoelectric ceramic
sensor transmits the acquired acoustic emission signal tothe amplifying and filtering circuit, and then acquisitionand processing circuit collects and processes the signal.
The characteristic parameters are transmitted to the hostcomputer software through the wireless network.

Fig.1. block diagram of fiber Bragg grating acoustic emission
wireless sensing system

As shown in Fig.1, the sensing system is mainlycomposed of signal acquisition module, signal
processing module, wireless transmission module andhost computer. The signal acquisition module consistsof a fiber Bragg grating acoustic emission sensor with acenter wavelength of 1553 nm and some opticalcomponents. The sensor is fixed on the experimental
wood plank, and the acoustic emission signal source issimulated by breaking the automatic pencil lead core.
The optical components are narrow linewidth laser
source, optical fiber coupler and high-speed
photoelectric converter. A narrow linewidth laser
emitted from laser source is incident to the fiber grating
through an optical fiber coupler. It is reflected by thefiber grating and then incident to the high-speed
photoelectric converter through the optical fiber coupler.
The optical signal is converted into the electrical signal
output by the photoelectric converter. The system

Fig.2. block diagram of piezoelectric ceramic acoustic
emission sensing system

3. Contrast Experiment
Piezoelectric ceramic sensor is a traditional acoustic
emission sensor, while fiber Bragg grating acoustic
emission sensor is a novel one. In order to compare thedifferences between two acoustic emission sensing
systems, this part makes a comparative experiment. As
shown in Fig.3, a piezoelectric ceramic acoustic© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan614

Research on Acoustic Emission

2

2

1.5

1.5

1

1

0.5

0.5
Voltage/V

Voltage/Vemission sensor with center response frequency of
151.37 kHz and a fiber Bragg grating acoustic emission
sensor with central wavelength of 1553 nm are fixed on
the same wood plank. Before fixing, the plank should bepolished，the surface of the contact of piezoelectric
ceramic sensor should be coated with corresponding
couplant and the fiber Bragg grating sensor should beglued to the plank, so that the two sensors can be fully
contacted with the plank to reduce the error. The
acoustic emission signal source is simulated by breaking
the automatic pencil lead core at the same distance withthe two sensors, and the acquisition results of the twosystems are compared.

0
-0.5

0
-0.5

-1

-1

-1.5

-1.5

-2

-2
2

3

4
5
Time/s

6

2

3

-4

x 10

4
5
Time/s6-4x 10(a) Original waveform (b) Recovered waveform
Fig.4. Time domain waveform detected by piezoelectric
ceramic acoustic emission sensing system
-3

-3

x 10

1.5

1.5

1

1

0.5

0.5

Voltage/V

Voltage/V

x 10

0

0

-0.5

-0.5

-1

-1

2

3

4
5
Time/s

6

2
-4

x 10

3

4
5
Time/s6-4x 10(a) Original waveform (b) Recovered waveform
Fig.5. Time domain waveform detected by fiber Bragg grating
acoustic emission sensing system

Fig.3. Schematic diagram of contrast experimentBecause the waveform data of acoustic emission
signal is very large, it will occupy a lot of memory
space. A time-domain waveform storage method of
acoustic emission signal is proposed. The feature pointsof the waveform are stored on the acquisition processor,
and the software simulates the signal waveform bypolynomial interpolation. Fig.4 (b) and Fig.5 (b) arerecovered results. We can see that the method cansuccessfully recover signal waveforms.

The experimental results show that there aredifferences in the acoustic emission signal waveform
between two sensing systems. Next, a typical acoustic
emission event is introduced as an example. In thispaper, breaking a lead core is temporarily defined as anacoustic emission event.
3.1. Time domain waveform of acoustic emission
Fig.4 (a) and Fig.5 (a) are the waveforms in the time
domain collected by processors of two systems when an
acoustic emission event occurs. It can be seen that the
waveform obtained by fiber Bragg grating system has
fewer oscillations in the duration. It is pointed out thatsuch difference is related to the inherent structuralcharacteristics of the sensor itself3. The upper and lower
surfaces of the piezoelectric ceramic sensor can cause
the superposition of the acoustic emission signals, while
the fiber Bragg grating acoustic emission sensor itself
does not have such problem. Therefore, the waveform
will not produce distortion, which is the advantage ofthe system based on fiber Bragg grating acoustic
emission sensor than the other in principle.

3.2. Estimation of power spectral density by Welch
methodThe power spectrum of the signal reflects many
important features of the signal, and the extraction of
the signal characteristics using the power spectrum is an
important method of the signal processing field. This
paper calls the PSD function of the Matlab signal
processing toolbox, and estimates the Welch power
spectral density of the two acoustic emission waveforms
shown in Fig.4 (a) and Fig.5 (a). The result is shown in
Fig.6. In order to facilitate the power spectral density
analysis, some characteristic parameters of power© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

615

XiuQing Wang , Qing Liu, Yang Li, Jimin Zhao, Jiangfan Wang

spectrum are defined in this paper. High energy
frequency band is a frequency band with high density ofrelative power spectrum4, which is expressed in H. H of
the piezoelectric ceramic system and the fiber Bragg
grating system are respectively expressed as H1, H2. X
represents the energy in a certain frequency band; N
represents the total frequency band energy; R represents
the energy ratio, that is R=X/N.
It can be starkly found that the power spectral densityof piezoelectric ceramics system on the vertical axis ismuch larger than that of fiber Bragg grating system on
the whole. This is due to the amplitude difference of
acoustic emission signals collected by the two systems,
and the amplitude of the time domain waveform
collected by the piezoelectric ceramic system is larger
than that of the fiber Bragg grating system.
From the view of Fig.6, H1 is mainly concentrated in
the range of 0-200 kHz, while H2 is mainly
concentrated in the range of 0-80 kHz. That is to say, inthe frequency range of 80-200 kHz, R value of thepiezoelectric ceramic system is larger than that of thefiber Bragg grating system. This phenomenon may berelated to the characteristics of piezoelectric ceramic
sensor, whose frequency response in the 80-200 kHz
frequency range near its center response frequency ishigher than that in the 0-80 kHz frequency range.

the piezoelectric ceramic system. The reason may bethat the upper and lower surfaces of the latter willsuperimpose the oscillations of acoustic emission
signals. Compared with the other, the fiber grating
acoustic emission sensor does not have the upper andlower surfaces, and can avoid the problem of the
waveform distortion.
In the power spectral density estimation by Welch
method, there is a difference in H between the twosystems, the low frequency distribution is similar. Thismay be caused by the frequency response characteristicsof the piezoelectric ceramic sensor whose response todifferent frequency signals is different. When thefrequency of the detected signal is near its center
response frequency, it can restrain the noise effectively,
and has a good detection result. However, when thefrequency of the detected signal is unknown, its
frequency may not be near the central response
frequency, the signal detected may bring errors.
From the above analysis, the fiber Bragg grating
acoustic emission system is superior to the system basedon piezoelectric ceramic sensor for monitoring acoustic
emission signals. The causes of the differences in time
domain waveform and power spectral density estimation
by Welch method need further study.
Acknowledgements

Welch Power Spectral Density Estimate-40This work is supported by Tianjin Research Program ofApplication Foundation and Advanced Technology（14JCZDJC39000）.

Power/frequency (dB/Hz)-60

Piezoelectric ceramic
acoustic
emission sensing
system

-80

-100

-120

-160

References

Fiber Bragg grating
acoustic
emission sensing
system

-140

0

50

100

150

200

250

300

350

400

450

1. Faye Zhang, Mingshun Jiang, Qingmei Sui, Acoustic
emission localization system based on FBG sensors andsupport vector machine. Journal of Vibration,
Measurement and Diagnosis, 2017, 37 (1): P189 - P194.
2. Gongtian Shen, Rongsheng Geng, Shifeng Liu, Parameter
analysis of acoustic emission signals. Nondestructive
Testing, 2002, 24(2): P72- P77.
3. Ning Li, Peng Wei, Hong Mo, Bearing state monitoring
using a novel fiber Bragg grating acoustic emission
technique. Journal of vibration and shock, 2015, 34(3):P172- P177.
4. Jianwei Liu, AE-infrared characteristics and damage
evolution generated from rock under variable angle
shear (Jiangxi, Jiangxi University of Science andTechnology, 2014)

500

Frequency (kHz)Fig.6. Welch power spectral density estimation of acoustic
emission signals detected by two sensing systems

4. Conclusion
In this paper, two acoustic emission wireless sensing
systems are built, one is based on Fiber Bragg grating
sensor, and the other is based on piezoelectric ceramic
sensor. With the signal produced by the breaking leadcore as research object, contrast experiment is carriedout, and differences of the two systems are found:
The acoustic emission signal collected by the fiber
Bragg grating system has fewer oscillations than that of© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

616

ORiN PAC (PC-based Automation Controller)- Integrated Development Environment for Manufacturing System Toshihiro Inukai

DENSO WAVE INC., 1, Yoshiike, Kusagi, Agui-cho
Chita-gun, Aichi, 470-2297, Japan
E-mail: toshihiro.inukai@denso-wave.co.jp

Abstract
This paper introduces application examples of ORiN PAC (PC based control system utilizing ORiN as middleware).
The manufacturing system is composed of huge kinds of devices, and it is necessary to make device programs forcontrolling these devices in cooperation. The mfg. system becomes increasingly complicated, and along with theincrease in programming cost, the improvement of programing efficiency became a big issue. In this paper, we
propose ORiN PAC as a solution to realize "one software architecture" applicable from small scale to large scale mfg.
systems. This solution has been used in the real factories in the world (not just a demo), and it will be quite importantalso for realizing "Industrie 4.0" world.
Keywords: Industrie 4.0, Manufacturing System, Cyber Physical System, ORiN

The target field of this study is “low production
volume (1 to below around 1000) and wide variety zone”
of mfg. system in Figure 1. Because the productivity ofthis zone is basically low, and for “high production
volume products”, mfg. system should be optimized tominimize the direct cost.

1. Introduction
PC-based Automation becomes gradually popular in theindustrial control field, with the spread of open standard
industrial
networks
suchas
EtherCAT[1],
POWERLINK[2] and PROFINET[3]. The background is
as follows:(i) Increase of mfg. software development cost
Improve the efficiency by using IT tools(ii) Increase of difficulty to ensure mfg. software
engineers Use IT engineers of non-mfg. fields(iii) Increase of mfg. system complexity
Utilize open
standard IT hardware and software
To cope with the situation, PAC (PC-based
Automation Controller) is spreading recently, and it will
be a key component of Industrie 4.0 world.
However most of mfg. software engineers hesitate touse PAC because it’s too general. So they want aframework suitable for PAC (that is ORiN PAC).

Fig. 1. Production volume and scale of mfg. system© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan617

Toshihiro Inukai

2. Automation System Architecture

3. ORiN

A main goal of Industrie 4.0 is flexibility. Many
engineers have been trying to realize it with several
architectures such as PLC-centric, Robot-centric and
CNC-centric. However it is difficult to achieve theflexibility for them because of the complexity and theversatility of mfg. systems. Each device has own strategy,
management and control parts in Figure 2. In this
scenario, it is difficult to make the whole system
optimized and flexible.

To make PAC easy, a framework suitable for PAC isrequired. Therefore, Fraunhofer IESE (Germany) started
a middleware project (BaSys 4.0[4]) in 2016.
Japan also started an “open standard middleware for
automation” project (ORiN[5]) in 1999, and ORiN
Forum[6] members have been maintaining andimproving it continuously. ORiN is an acronym of “Open
Resource interface for the Network), and the features are
as follows.
ORiN provides three standards: CAO (Controller
Access Object), CRD (Controller Resource
Definition) and CAP (Controller Access Protocol),CAO provides a standard program interface for bothof device and application,
CRD provides a standard data schema to describe thestatic information such as profiles of device,
CAP provides a standard communication protocol tocooperate with remote CAO systems,
These standards have a unified concept, but it can beused individually.

Fig. 2. Typical architecture of current mfg. system
To realize flexibility (reusability, changeability),
separation of the three parts is required first. Also,
simplification and encapsulation (i.e. object-oriented
approach) are also required as shown in Figure 3.

Fig. 4. ORiN architecture
4.

ORiN PAC is a “PC-based automation controller
utilizing ORiN as middleware.” In this section we show
two examples.
All devices in Figure 5 are connected to ORiN PAC,
and one Visual Basic program is controlling wholesystem. The programmer could make a program for FA
devices as for PC devices (peripherals).
The next in Figure 6 is a multi-vender robot control
system which was difficult to develop conventionally.
Four different manufacture’s robot (DENSO, Kawasaki
YAMAHA, Yaskawa) were connected.

Fig. 3. Proposed architecture of mfg. system
On the other hand, that is a standard approach for IT
world. Therefore several companies (e.g. Beckhoff
Automation, B&R, Phoenix Contact a) had alreadystarted PAC business. However, it’s still difficult for mfg.
system engineers to control the whole system that
includes legacy devices and/or the other platform subsystems.

aORiN PAC Application Examples

Phoenix Contact calls it “PLCnext” concept.
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

618

ORiN PAC Integrated Development

Fig. 5. Integrated cell control

Fig. 9. ORiN Multi-purpose controller
6. ORiN PAC with OSS
The success of OSS (Open Source Software) in the officeautomation brought impact to the factory automation.
ROS[7] is one of them, and it becomes popular in the lab.
However it is still difficult to be used in an “industrial”
solution because of the luck of the following issues:
safety certification (e.g. ISO, UL), field network
certification(e.g.
EtherCAT,PROFINET),
communication support for various automation devices(e.g. PLC, Robot) and various upper-level systems (e.g.
SCADA, MES). OSS community is basically notinterested in supporting those proprietary technologies.
By combining ORiN PAC with OSS, the applicable
field is expanded. In the following examples, ORiN PAC
provides ROS libraries (CAP-ROS node and ROSSerialCAO gateway provider).

Fig. 6. Multi-vender robot control
5. Realtime Extension Examples
ORiN is not a standard for realtime system. Therefore in
the case realtime control is required, some kind of
realtime extension should be combined with ORiN PAC.
Figure. 7,8 and 9 show three realtime extension
examples: for Robot (Fig. 7), for Automation (Fig. 8) andfor Multi-purpose (Fig. 9). ORiN PAC provides gateway
libraries (TwinCAT3 ADO provider and INtime NTX
provider).

Fig. 7. ORiN Robot controller

Fig. 10. Manipulator (above) and mobile robots (below)
applications by ORiN and ROS
Fig. 8. ORiN Automation controller© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan619

Toshihiro Inukai

7. Additional Advantages
There are several additional advantages in ORiN PAC
approach. In this section, we describe three examples.
7.1. Integrated Development Environment
Automation device venders are basically providing aproprietary development tool. The mfg. programmers
must learn how to use it , and that is one reason of poor
productivity of mfg. software.
At four examples in Figure 5, 6, 8 and 9, the mfg.
programmers can use Microsoft Visual Studio, as with IT
software developers. Of course, software asset
management tools such as Subversion and Git are very
useful for them.
Fig. 12. ORiN IoT architecture
8. Conclusion
We proposed ORiN PAC architecture and its possibility
in this paper. We hope this will be a hint for realizing asmart factory.
Acknowledgements
This work is supported by NEDO (New Energy andIndustrial Technology Development Organization) and
ORiN Forum members.

Fig. 11. Integrated development environment
7.2. Digital Twin
Digital Twin is one of key topics of Industrie 4.0. We
thought that is a technology for high volume production
products. But it is useful for the mfg. system which istypically low volume. For example, in case of the mfg.
line design, the combination of real devices and virtualenvironment will yield quite big benefit to reduce thedevelopment cost [8].

References
1.
2.
3.
4.
5.

EtherCAT, https://www.ethercat.org/
POWERLINK, http://www.ethernet-powerlink.org/
PROFINET, http://www.profibus.com/
BaSys 4.0, https://www.basys40.de/
Makoto MIZUKAWA and Satoshi SAKAKIBARA,
Implementation and applications of open data network
interface ORiN, in Proc. of the SICE Annual Conference,
pp. 1340–1343, 2004
6. ORiN, http://www.orin.jp/
7. ROS, http://www.ros.org/
8. Toshihiro INUKAI, Hironori HIBINO and Yoshiro
FUKUDA, Simulation Environment Synchronizing Real
Equipment for Manufacturing Cell, in Journal ofAdvanced
Mechanical
Design,Systems,
and
Manufacturing, Vol. 1, No. 2, 2007

7.3. Combination with ORiN IoT
CPS (Cyber Physical System) is also one of key topics ofIndustrie 4.0. A CPS provides connections to upper-levelsystems for strategic communication. ORiN IoT
architecture shown in Figure 12 provides most of global
standard communication ways for each IoT application
platform. And ORiN PAC systems can be easily
connected to ORiN IoT systems with CAP. This meansthat ORiN PAC doesn’t need to have IoT specificfunctions.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

620

Research on the Synchronization and Circuit Realization of a Four-Wing Chaotic System
Yiqiao Qin, Fengzhi Dai *, Yuxing Ouyang, Qijia Kang, Ce Bian, Baochang Wei, Runhua Mao, Shengbiao Chang

Tianjin University of Science and Technology,
1038 Dagunanlu Road, Hexi District, Tianjin, 300222, China;
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
This paper introduces a three-dimensional four-wing chaotic system and the system is analyzed and circuit verified.
The paper adopts two synchronous control strategies to control the three - dimensional four-wing chaotic system.
One is the feedback synchronization controller based on the observer, and the other is the synchronization controller
based on theory of stability. The feasibility of these two methods is proved by analyzing the numerical data and thecircuit implementation. On this basis, we conducted signal encryption research. The results verify the feasibility ofthe synchronous design and encryption design.
Keywords: Chaos, Chaotic synchronization, Secure communication, Circuit realization

1. Introduction
x ax ky yz
yby z xzz
x cz xy(1)
When the system variables are selected as b=12, c=5,
and k=1, we get the bifurcation graphs as shown in Fig.1.
The research of encryption and chaotic communication
should be based on the chaotic state of the system, so theselected parameters are a=5, b=12, c=5, k=1.

The meaning of chaotic secure communication is thatthrough a series of algebraic operations, the signal thatneeds to be transmitted can be superimposed on a chaotic
signal which is generated by a chaotic system. Because of
the pseudo randomness of the chaotic signal, the mixed
signal will have no regularity, and be similar to the noise
signal. The characteristic guarantees the security of theinformation that it is difficult for eavesdroppers to restore
the original information even if they intercept the mixed
signal. After receiving by legitimate people, the useful
information can be distinguished from the mixed signal
through the synchronization between systems of the senderand the recipient.
2. Analysis of four-wing chaotic systems
The mathematical expression of a 3D four-wing chaotic
system is as Formula 1:

Fig.1. Bifurcation graphs of four-wing chaotic systems

When the parameters are a=5, b=12, c=5, and k=1, thex-y-z plane phase diagram, the x-y plane phase diagram© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

621

Yiqiao Qin, Fengzhi Dai, Yuxing Ouyang, Qijia Kang, Ce Bian, Baochang Wei, Runhua Mao, Shengbiao Chang

and the y-z plane phase diagram of the system are shownin the following Fig.2.

Fig.3. synchronization diagram of The first method

Fig.2. plane phase diagram

3.2. The 2nd method of synchronization control

There are two kinds of synchronization control ofchaotic systems proposed in this paper. The first is
feedback synchronization controller based on the observer,
and the second is the controller based on stability theory.

The driving system for the four wing chaotic system isshown in Formula 2. Its response system is as Formula 6.

3. Design of synchronization system

When the parameter value is (a, b, c, k) = (5, 12, 5, 1), theobserver of drive system is Formula 2.

Set

5
0

1
12

0 x1
1 y1

y1 z1
x1 z1

z1

1

0

5 z1

x1 y1

D

10 0 0
0 0 0
0

Ax2

dt

(2)

y2

y2

12 y2

z2

x1 z1

z2

x2 5 z 2

x1 y1

z2

x2 z 2 U 2

z2

x2 5 z 2x2 y 2 U 3

y2 z 2 U1
(6)

e1e1

e2 e2

e3e3

U1

10e1 e2 e3 e2

U2

0

U3

e1 e2

(8)
When V is always negative, through Lyapunovstability theory, the error system is globally asymptoticallystable at the origin. In this case, the system as shown in
Formula 2 is synchronized to the system as shown in
Formula 6.
Through algebraic calculation, a proper controller canbe constructed to synchronize the two systems completely.
The controller in the system is selected as Formula 9. It iseasy to prove that is always negative.
Through the Formula 9 and the Formula 6, the response
system can be obtained as Formula 10.

(3)

BF ( x2 ) C s ( x1 ) s( x2 )10 x1 5 x2

12 y2

V

0 0

x2

y2

y2

Construct Lyapunov exponent and its derivative can beobtained as Formula 8.

C

Because A-D matrix has only negative real parts,
according to Routh criterion, it is a Hurwitz stable matrix.
According to formula 4, and corresponding to formula
2, the synchronization system can be get as Formula 5.

d x2

5 x2

Ui (i=1, 2, 3) is the controller. The error variables
between the response system and the drive system are as
Formula 7.
e1 x2 x1 , e2 y2 y1 , e3 z2 z1(7)

3.1. The 1st method of synchronization control

x1
y1

x2

(4)

y1 z1
(5)

The following is the simulation of this design. Set thetime of simulation is [0 - 5], the time step is 0.0001, X0 is(3, -6, 7), Y0 is (3, 0, 0). the result of the MATLAB
simulation is shown in Fig.3. Through synchronous timing
diagram, it can be observed that although the initial value
is different, the drive system and the synchronization
system have been fully synchronized as the time goes by.

x2

5 x2 2 y2 z 2 10 x1

y2

12 y2

z2

x2 z 2

z2

5z2

y2

y1 x1

2 x1e2(9)y1y 2 z1y1 z 2y1 z1

x2 y 22 x1 y1 2 x1 y2(10)

Matlab simulation results are as Fig.4, and the relatedparameters are the same as those in the previous.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan622

Research on The Synchronization

The values of some basic components in this circuit are
as follows:
R1=R2=R4=R5=R16=R18=R19=10kOhm,
R6=R9= R12=R20=R23=R26=1000kOhm,
R3=R11=R13=R14= R25=200kOhm,
R8=R22=83.3kOhm,
R7=R10=R17= R21=R24=R27=100kOhm,
C1=C2=C3=C4=C5=C6 =10nF.
The second synchronous control method mentioned
above is the controller based on stability theory. According
to the response system shown in formula 10 and the drive
system and the formula 2, the communication circuit isdesigned as shown in Fig.6.
The values of some basic components in this circuit are
as follows:
R14=R28=200kOhm,
R4=R28=200kOhm,
R18=R19 =R23=R32=100kOhm,
R15=R20=R21=R22=R26=R27 =100kOhm,
R7=R17=R33=R34=50kOhm,
R16=R25 =R29=R30=R31=1000kOhm,
R24=83.3kOhm,
C1=C2 =C3=C4=C5=C6=10nF.

Fig.4. synchronization diagram of The second method

4. Circuit design of four wing chaotic system
The first synchronous control method mentioned above isfeedback synchronization controller based on the observer.

Fig.5. synchronization diagram of The second method

According to the response system shown in Formula 5
and the drive system and the Formula 2, thecommunication circuit is designed as shown in Fig.5. It is
composed of operational amplifiers LF347, analog
multipliers AD633, resistors and capacitors to perform
additive, subtraction and multiplication operations.

Fig.6. synchronization diagram of The second method© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

623

Yiqiao Qin, Fengzhi Dai, Yuxing Ouyang, Qijia Kang, Ce Bian, Baochang Wei, Runhua Mao, Shengbiao Chang

5. Testing and conclusion
We set the original sine signal as S1, the decrypted signal
as S2, the mixed signal as H, the signal produced by thedrive system as X1, and the signal error as deviation. The
initial value X0 is (3, -6, 7), Y0 is (3, 0, 0), and thesimulation time is (0s - 5s).
The simulation result of feedback synchronization
controller based on the observer is shown as Fig.7.

(a) X1(a) X1(b) S1(c) H(d) S2(b) S1(c) H(d) S2(e) Deviation
Fig.8. simulation result of The second method

References
1. Shu-Yi S, Fu-Hong M, Xue-Hong W, et al. Implementationof a new chaotic system based on field programmable gatearray, Acta Physica Sinica, 63 (6) (2014) 60501-60502.
2. Yang-Zheng L, Chang-Sheng L, Xin-Chao L. A newsmooth quadratic chaotic system and its digital signal
processing implementation, Physics, (6) (2011) 60507.
3. Bao B, Xu J. Generalized Projective Synchronization for
Multi-wing Chaotic Attractors, ICECE, (2010) 5234-5237.
4. Y Lin, C Wang, HE Haizhen, LL Zhou, A novel four-wing
non-equilibrium chaotic system and its circuit
implementation, Pramana, 86(4) (2016) 801-807.
5. K Guo, Chaos synchronization and its application in securecommunication (Xi'an, Xi'an Technological University,
2013)
6. FF Li, Research on chaotic synchronization control andApplication (Nanjing, Nanjing University of Aeronautics
and Astronautics, 2007)
7. X G Liu, W Yang, W Liu, Y Dai, Designing S-boxes basedon 3-D four-wing autonomous chaotic system, Nonlinear
Dynamics,84 (4) (2015) 1867-1877.

(e) Deviation
Fig.7. simulation result of The first method

The simulation result of the controller based on stability
theory is shown as Fig.8.
We set the original sine signal as S1, the decryptedsignal as S2, the mixed signal as H, the signal produced by
the drive system as X1, and the signal error as E. The initialvalue X0 is (3, -6, 7), Y0 is (3, 0, 0), and the simulation
time is (0s - 10s).
Acknowledgements
The research is partly supported by the Research Fund for
the Reform in Education from Tianjin Municipal
Education Commission of China (171005704B), and theScience & Technology Fund Planning Project of Tianjin
Higher Schools (20140710-1400020005).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

624

Design of Control System for Reboiling Part of Distillation Process
Lingran An1, Fengzhi Dai 1*, Yujie Yan1, Yuxing Ouyang1, Zhongyong Ye1, Xia Jin1, Baochang We1, Hongwei Jiao2
1

College of Electronic Information and Automation,

Tianjin University of Science and Technology, China;
2

Tianjin Technology School of Printing and Decoration, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

This paper selects the control system for re-boiling part of distillation process for major research. The
control scheme is determined according to the influence of the relevant parameters on the distillation
process. The hardware part uses the Siemens S7-200 series PLC as the main control machine. The lower
position machine uses STEP 7-Micro / WIN programming software, and writes the control program
through the ladder diagram. It combines the PID algorithm and single closed-loop control to complete
the field data collection and processing and communication with the host computer. The host computeruses the configuration king named kingview6.55 monitoring system software to complete the automatic
control system monitoring.
Keywords: Distillation Column, Reboiler, Control System, PLC

Classic distillation column reboiler control methods
are steam and condensate, they have their own
advantages and disadvantages. The researchers
optimized it to form a cascade circuit with thetemperature of the kettle, thus improving the regulation
efficiency and increasing the stability of the control
system3. In view of the problems such as the delay ofthe sensitive plate temperature and poor operability
during the reboiler control, the researcher reformed thereboiler control system based on the existing control andsolved the problem of equipment operation stability4.

1. Introduction
In the distillation process, the reboiler not only affects
the vapor phase load of the distillation column system,
but also has a direct effect on the temperature of the
kettle and the product quality of the tower kettle.
Therefore, the proper choice of reboiler control scheme
can play an important role in the overall energy
consumption, stability and safety of the distillation
process.
Some researchers have attempted to use intelligent
control methods such as the neural network and fuzzycontrol for the distillation column in order to solve theproblem of the time denaturation and the coupling ofvariables in the distillation control system1. Some
researchers also analyzed the advanced control
algorithm of the distillation column theoretically, andproposed a multivariable dynamic decoupling control
algorithm combined with the predictive PI algorithm topropose a dual control scheme for the stripping section2.

2. Overall scheme design
The process piping and control flow diagram of thereboiler part are shown in Fig.1. A portion of the liquid
at the bottom of the column is heated by reboiler E102
and then returned to the column to maintain thecontinuous distillation process. The other part iscontrolled by the regulating valve VA120, as a product© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

625

Lingran An, Fengzhi Dai, Yujie Yan, Yuxing Ouyang, Zhongyong Ye, Xia Jin, Baochang We, Hongwei Jiaoof the bottom of the column. The liquid level of thereboiler is controlled by LIC101. The reboiling part uses
the electric heating rod to heat the liquid, and theheating power is controlled by the TIC101. The coolingwater starts from the cooling water pump P102, passes
through the valve VA109, then flows through the rotor
flowmeter F101, then enters the top condenser E101,
then passes through the kettle cooler E103, then enters
the cooling tower T102, finally returns to the coolingwater tank V103.

The detection point LI102 is the on-site display
parameter, so the glass tube liquid level meter isselected.
The detection point LIC101 needs electric remotetransmission, so the remote transmission magnetic
flap level gauge is selected.
The three temperature detection points TIC101,
TI102 and TI103 choose the Pt100 temperature
sensor.
2.2. Determination of control scheme
First of all, determine the control points of the control
system as follows:
LIC101: Control the liquid level of the reboiler
E102 by controlling the opening degree of theregulating valve VA120.
TIC101: Control the vapor phase temperature of thereboiler E102 by controlling the heating power ofthe heating rod.
Then we complete the selection of actuators
according to the selected control points:
The control point LIC101 needs to control the
opening degree of the regulating valve through theanalog output of the PLC, so the electric
proportional regulating valve is selected.
The actuator of the control point TIC101 selects thesolid-state voltage regulator.
The control scheme chooses the single closed loop
control system, and the control algorithm of theregulator adopts PID operation adjustment. The control
block diagrams of LIC101 and TIC101 are as follows:

Fig.1 The process piping and control flow diagram of thereboiler part2.1. Determination of detection scheme
First, determine the detection points of the control
system as follows:
FI101: Detect the cooling water flow flowing into
the tower top condenser E101 in the cooling water
circulation system.
LI102: Detect the liquid level of the tower kettle
discharge tank V102.
TIC101: Detect the vapor phase temperature of thereboiler E102.
TI102: Detect the temperature of the cooling water
inlet.
TI103: Detect the temperature of the cooling wateroutlet.
LIC101: Detect the liquid level of the reboiler
E102.
Then, we select the measuring instruments according
to the detection point:
The detection point FI101 selects the glass rotor
flowmeter, for the On-site display of the coolingwater flow.

DisturbanceSetliquid level

Deviation

Regulator

Regulating
valve

Reboiler

Reboiler level

Liquid leveltransmitterFig.2. The control block diagram of liquid level of the reboiler

Fig.3. The control block diagram of vapor phase temperatureof the reboiler© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

626

Design of Control System

cycle of the PLC work needs to initialize the system,and the subsequent system programs are periodically
invoked by a timer. When the timing time arrives, it
reads the field detection data and calls the data
conversion and scaling conversion subroutines toprocess and store the collected data. A part of the data iscompared with the set value to calculate the deviation,
for the PID instruction wizard to perform the operation.
Finally, the control output conversion subroutine iscalled, which transforms the data after the PID
operation and then outputs it to the actuator.

3. System circuit module design
The input and output points of this control system
include: 2 digital input points (start and stop switches),
1 digital output point (LED indicator), 4 analog input
points (cooling water inlet / outlet temperature, reboiler
level and reboiler temperature) , 2 analog output points(solid-state voltage regulator and control valve opening
degree). Therefore, the CPU222 module is selected,
which has 8 native digital inputs and 6 native digital
outputs, allowing up to 2 I / O expansion modules. One
EM231 analog expansion input modules and an EM232
analog expansion output modules are selected as
required. The hardware wiring diagram is shown in
Fig.4.
LED

Temperature
Transmitter

Temperature
Transmitter

+

+

-

1L

0.0 0.1 0.2 2L 0.3 0.4 0.5

RA

A+

A-

Temperature
Transmitter

-

RB

B+

B-

Liquid Level
Transmitter

+

+

-

RC

C+

C-

Solid
State
Voltage
Regulator

Regulating
Valve

RD

D+

D-

M0

I0

V0

M1

2M 0.4 0.5 0.6 0.7 M

L+

Input DC24V

+
Stop
Start
SwitchSwitch

M

L
+

+

DC24V
- Power
Supply

V1

I1

M2

V2

I2

M3

V3

I3

EM232

EM231
1M 0.0 0.1 0.2 0.3

In STEP7-Micro/WIN V4.0, we can use the wizard togenerate a PID instruction or program, and then call it in
the main program. After completing the configurationof the PID loop through the wizard, the specified PID
regulation subroutine PIDx_INIT and interrupt program
PID_EXE will be automatically generated. The
PIDx_INIT instruction performs the PID function basedon the inputs and outputs we have previously set in the
PID wizard, and it can be invoked every time the scan is
performed. In this design, the vapor phase temperature
control system of the reboiler and the liquid level
control system of the reboiler adopt PID wizard togenerate the instructions6.

-

CPU222

DC24V

4.2. PID operation subroutine design

M

L+

+

DC24V
- Power
SupplyFig.4. The hardware wiring diagram

4. Programming software design
The control layer of the control system of reboiling partof distillation process is the PLC system. It is used toreceive on-site data and control the site equipment
according to the pre-designed control strategies, while
send the on-site data and equipment state to the PC
machine5. This design selects the S7-200 series PLC as
the controller, and the programming software uses the
STEP 7-Micro/WIN software of the German SIEMENS
company S7-200 series PLC.

5. Monitoring software design
This design adopts kingview6.55 monitoring system
software. Generally, such a system can be divided into
three layers: control layer, monitoring layer andmanagement layer. The monitoring layer is connected to
the lower connection control layer and the upper
connection management layer. It not only realizes thereal-time monitoring and control of the site, but also
completes the important function of uploading andrelease, configuration development in the automatic
control system. The configuration software also
provides visual monitoring images for the participants,
which is helpful for the real-time monitoring of the test
subjects.
PLC and PC are connected by PC-PPI cable hardwire,
and the monitoring software of PC is communicatedwith PLC through PPI7. First of all, we complete thedefinition of the external equipment and thecommunication between the Kingview6.55 and theexternal equipment in the software. Secondly, we define
the variables in the control system. Finally, we design
the configuration screen, which are: reboiler operation
monitor main interface, production report and data
curve.

4.1. The main program design
The control layer of the control system of reboiling partof distillation process is the PLC system. It is used toreceive on-site data and control the site equipment
according to the pre-designed control strategies, while
send the on-site data and equipment state to the PC
machine5. This design selects the S7-200 series PLC as
the controller, and the programming software uses the
STEP 7-Micro/WIN software of the German SIEMENS
company S7-200 series PLC.
The PLC program is designed in a structured way.
The main program completes all functions of the system,
and calls each module in turn according to the system
requirements and subroutine functions. The first scan

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

627

Lingran An, Fengzhi Dai, Yujie Yan, Yuxing Ouyang, Zhongyong Ye, Xia Jin, Baochang We, Hongwei Jiao

6. Summary and prospect

Acknowledgements
The research is partly supported by the Research Fundfor the Reform in Education from Tianjin Municipal
Education Commission of China (171005704B).

This design adopts DCS control system based onprogrammable controller Siemens S7-200. Its contents
mainly include the overall scheme design of the reboiler
part of the distillation process control system and thedesign of hardware and software.
This paper focuses on the design of the reboiler
control system in the distillation process. However, thecomplete distillation process also includes the feed partand the reflux part, and this design is not combined withthe two links. Therefore, the small S7-200 series PLC
selected in the implementation of the hardware can notsatisfy the whole complete distillation control. In theoverall control, a medium or large PLC can be selectedto achieve complete control.
Because the control points selected in this design are
only to control a link, the selection of software andalgorithms is relatively simple. The algorithms ofcomplete distillation control is relatively complex, andthe nonlinear control technology can be applied to the
distillation column. In addition, adaptive control
algorithms, predictive control algorithms, inference
control algorithms and robust control algorithms can be
developed. It is also possible to consider intelligent
control methods such as neural network and fuzzycontrol for distillation tower.

References
1. Jiangpeng Wang, Simulation study on temperature
control of rectification tower based on fuzzy decoupling,(Yunnan, Kunming University of Technology, 2015)
2. Yanming Zhou, Theoretical analysis and design ofadvanced control algorithm of rectification tower,(Shanghai, Donghua University, 2014)
3. Jun Shen, Analysis and optimization of typicaltemperature control scheme for rectification towerreboiler, Chemical and pharmaceutical engineering,
2015, 6: P1-P5.
4. Zhongwu Zhu, Dabo Jiang, The process and piping
design of two reboiler and distillation column, Fertilizer
design, 2016, 3: P15-P18.
5. Liguo Wen, The design of the control of distillation
column based on PLC, (Heilongjiang, Northeast
Petroleum University, 2013)
6. Xueyong Shi, Realization of PID control in the liquid
level of distillation tower kettle, Science and Technology
Economic Guide, 2017, 3: P72.
7. Yahui Mu, The practical technology of Kingview
software, Yellow River Water Conservancy Press, 2012:
P34-P35.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

628

Research on intelligent control system based on machine vision
Ce Bian1, Fengzhi Dai 1*, Meili Li2, Yuxing Ouyang1, Yiqiao Qin1, Runhua Mao1, Baochang Wei1, Shengbiao Chang1
1College of Electronic Information and Automation, Tianjin University of Science and Technology,
1038 Dagunanlu Road, Hexi District, Tianjin, 300222, China;
2

China University of Petroleum-Beijing, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
Machine vision is widely used in today's factories. This paper studies the motion control system based on machine
vision. In the past, it is difficult for machine vision to extract valid information from the lens which has plenty ofcomplex shapes, transparent and reflective elements in the electronic industry. This paper studies precise motion
control based on machine vision.
Keywords: machine vision, motion control system, image process, servo motor, precision

1. Introduction
In LED industry, the lens is transparent and has acomplex shape. It is necessary to determine the outline
and attitude of the lens when acquiring image
information. The first thing is to collect the outline oflens. Image edge belongs to high-frequency signal. It
can produce great influence when the edge and theshape capturing part of irregular shape of lens.
1: industrial computer, 2: monitor, 3: industrial camera,
4: lens, 5: LED light source, 6: PCB, 7: mobile platform,
8: raster ruler, 9: servo motor, 10: motor drive.

2. The main hardware analysis
Machine and computer vision usually refer to the digitalprocessing of the image by the computer, thus toanalyze the pixels in the picture. The application ofmachine vision in the industry field totally is different
from ordinary computer vision. Machine vision system
can adapt to the most of the industrial environment.
Machine vision system is an integrated system that
includes the digital image processing, industrial control,
LED lighting, optical imaging, sensors, analog anddigital video, computer hardware and software,human-machine interface, as shown in Fig.1.

Fig.1 Hardware design of machine vision

2.1. Components of Machine Vision
1. Lighting mode. LED, as a new kind of green lightsource, is applied to the machine vision system withluminous stability and can be controlled by manual andprogramming adjustment.
2. The Image sensor. CCD and CMOS sensor are used
to convert image information into digital information by© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

629

Ce Bian, Fengzhi Dai, Meili Li, Yuxing Ouyang, Yiqiao Qin, Runhua Mao, Baochang Wei, Shengbiao Chang

using photodiode. They transfer digital signal in adifferent way. Through the transferring way of CCD
sensor, each pixel data was transmitted to the next pixel
line by line. Through the amplifier amplify the date thenoutput at the end; CMOS sensors are not suitable forlong-distance data transmission. Each signal cantransmits data through an amplifier and A/D conversion
circuit when period) collected by each pixel in a circuit
similar way.
3. Image storage. The period time that the camera taking
photos is small relative to the data processing time
which is less than 40 milliseconds. Unprocessed
information deposit onto a buffer zone, and then by way
of first in first out for processing.
4. The image collection card can collect standard or
non-standard GRB signals that simulate black and white
component. The card support camera which scans eachline one by one or scans each line the GRB component
separately.

low-pass filter which blocking the high-frequency noise
signal and leaving the low-frequency useful signal.
2.2. Motion control system
The XY mobile platform discussed in this article is that
motion control card sends pulses separately to two servo
motors control each axial movement. It's going to be thedirection of the vector X and Y. After superimposed
motion trajectory, the motion control system cannot
control the head of trajectory. In order to control thehead of the trajectory, interpolation is an excellent
solution to this problem. There are four types ofinterpolation, comparison method, numerical integral
method, time division method and the splineinterpolation method. This paper studies thepoint-by-point comparison interpolation method. Itmeans starting from the origin point. System compares
the given trajectory at every feed step. Through theresult of deviation, it will judge the axis feed for the
next step.

Fig.2 Lens image
5. Image pre-processing. The image information
processing in the machine vision system is to highlightthe needed elements in the image instead of considering
the image quality. So it is unnecessary to be similar tothe original image after in a pre-processing of images.
Image processing means to separate the objects fromthe background. Firstly, we need to identify the edges.
Edge is a collection of pixels that have a step change in
the adjoined pixel. The edge is usually determined by
the sudden change of the gray level. Commonly there
are three edges, the stepped edge, the roof edge and thelinear edge. To get the information of the image, thesmoothness image is mainly to deal with the noise in the
pixel and reduce the noise to improve the image quality.
The smoothing treatment is handled primarily by a

Fig.3 The structure of SMT

1. Stepping motor is an actuator that converts electricalpulse into angular displacement, and controls pulse
number and pulse frequency. The permanent magnet isthe rotor of the servo motor. The driver adjusts therotation speed and angle according to the comparison
between feedback value of encoder and the target value.
2. The servo motor is controlled by encoder feedback
when stepping motor work speed pulse and direction
pulse are required.
3. The step motor controls the angle of rotation bycontrolling the number of pulses. One pulse corresponds
to one step angle. The servo motor controls the rotation
angle by controlling the length of pulse time.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

630

Research on intelligent control

4. Stepping motor occurs low-frequency vibration at
low speed.
5. The output torque of stepping motor decreases with
the increase of speed and decreases sharply at highspeed. The servo motor will be running smoothly at low
speed.
6. Stepping motor cannot be overloaded. Ac servo
motor is the constant torque output，and the torque will
work in a constant way when running at the rated speed.
Ac servo motor has strong overload capacity.
7. The stepping motor from the stationary acceleration
to the working moving speed needs 200 ~ 400ms. The
ac servo system accelerates from rest to its rated speed
only a few ms.

Fig.5 Decision of Lens direction

3. The algorithm
3.1. Algorithm analysis
There are a lot of information that is not necessary in
the images. So it is essential to separate the useful
information.
There are three kinds of image segmentation. Theedge segmentation is to use the sharp change of pixel
gray degree to separate the helpful information.
Region-dependent segmentation is to select several
primary pixel points and then extend them to makejudgments. Threshold segmentation is a traditional
segmentation method. The central principle is to selectthe appropriate threshold for segmentation.
This article explains the method to delete the uselessinformation in the image after segmentation, as shownin the following figures Fig.4 to Fig.7.

Fig.6 Decision of Lens edge

Fig.7 Lens image after removing the superfluous

3.2. Coordinate conversion
This study selects traditional camera calibration method.
There are four coordinate systems that will be used in
the process of camera calibration. They are the world
coordinate system, camera coordinate system, image
physical coordinate system and image pixel coordinate
system, as shown in Fig.8.

Fig.4 Ideal image information© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

631

Ce Bian, Fengzhi Dai, Meili Li, Yuxing Ouyang, Yiqiao Qin, Runhua Mao, Baochang Wei, Shengbiao Chang(3) The semi-closed loop control system adds the
feedback to realize the precise closed-loop control.
Acknowledgements
The research is partly supported by the Research Fundfor the Reform in Education from Tianjin Municipal
Education Commission of China (171005704B).
References
1. Zhi-Yuan Zhang. Research on robot motion control basedon machine vision. North China Electric Power
University, 2014.
2. Niblett Timothy J, Church Richard L. The Shortest
Covering Path Problem. International Regional Science
Review, 391, 2016.
3. S.Y.Chen, Han-Tang Tong, Carlo Cattani. Markov
Models for Image Labeling. Mathematical Problems in
Engineering, 2012: 163-170, 2012.
4. Ting-Ting Ding, Zhou Fang, Bo Liu, Junan Zhang. Fastcalibration and error analysis based on machine vision
detection. Manufacturing automation, 2014.
5. Lei-Hong Li, Qun Guan, Kai-Heng Hu, Yu Yang. The
development of camera calibration system based on
OpenCV in vc (VC) ++ environment. Computer
applications and software, 6: 19-21+31, 2011.
6. Li-Tao Wang, Chun-Xiao Zhuang, Qiu-Cheng Wang.
Digital image geometry transformation based on VC
programming. Science and technology innovation andapplication, (17): 53-53, 2015.
7. Zheng Zhang, Chao Zhang, Shuxia Zhang, Hailing Han.
Digital image processing and machine vision--VisualC++ and Matlab realization. The people's mail andtelecommunications press, 2014.
8. Chang-Hao Dong. Study on interactive PCB image
segmentation based on graph theory. The PLA
Information Engineering University, 6: 22, 2013.

Fig.8 Four coordinates relative to the position

4. Conclusion
This article uses SMT in the electronics as research
object to analyze the intelligent control system based onmachine vision. The visual system analyzes the image
by collecting different images information. The control
system drives the actuating parts to obtain optimalclosed-loop precise control. The most useful
information collection and the precision of the motion
control directly affect the performance of SMT machine.
The moving control system selects performs component
according to the comparison between stepping motorand servo motor.
The paper presents the following points:(1) Image segmentation is used to remove the complex
interference information.
(2) Double processing of image acquisition information
is used to get the accurate component coordinates.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan632

Development of NC Power Based on Buck Circuit
Yuxing Ouyang, Fengzhi Dai*, Runhua Mao, Ce Bian, Baochang Wei, Yiqiao Qin, Shengbiao Chang, Qijia Kang

College of Electronic Information and Automation, Tianjin University of Science and Technology,
1038 Dagunanlu Road, Hexi District, Tianjin, 300222, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
This paper develops a kind of power-supply module which uses digital control and buck circuit. The module canconvert 220v alternating current into 0-30V direct current. In the aspect of circuit design, it is mainly AC-DC,
DC-DC conversion circuit and SCM minimum system circuit. In the aspect of control circuit, for feedback signal of
current sensor, the power-supply module can obtain average value by multiple means and remove the mutation
value.
Keywords: Buck circuit, SCM, AC-DC conversion circuit, DC-DC conversion circuit

circuit has a complicated structure, low stability, easy togenerate errors, and consumes a certain amount ofenergy because it needs to withstand a large amount ofresistance.
The main circuit of the NC power based on Buckcircuit is divided into the following three modules: The
first module is AC-DC conversion circuit. Its function is
to convert the 220V AC to the 30V DC through atransformer, a bridge rectifier module and filter
capacitor; the second module is the DC-DC conversion
circuit. Its function is to convert the 30V DC to the
0-30V DC through the Buck circuit; the third module isthe microcontroller control circuit, its main function isto protect the special requirements of constant voltage
or constant current, through reading the current sensor
feedback signal and adjust the PWM duty cycle tochange the voltage output2.
This paper is organized as follows. Section 2
describes the hardware circuit design, and Section 3
introduces the control strategies. The main circuit
simulation is implemented in Section 4, and theconclusions are presented in Section 5.

1. Introduction
With the development of power electronic technology
and digital control technology, more and more devices
combine these two technologies. This paper develops a
power module which combines MCU with Buck circuit
to enhance the digital control of the power supply
module to a certain extent to make up for the lack of
traditional power supply module1.
At present, the traditional power conversion circuit
uses TL494 as the PWM signal generator and the MOS
tube as the switch element. The voltage regulation isrealized by building analog circuits and adjusting theresistance of sliding rheostat or potentiometer to change
the duty ratio of PWM.
A closed-loop system can also be formed by building
an analog circuit, but this closed-loop system has asimple structure and low anti-interference ability. The
most important thing is that this conversion method
which based on analog circuits has its own
insurmountable shortcomings, such as the inability todisplay parameters such as voltage and current in realtime and the disadvantage of not being able to adjust inreal time. Moreover, the conventional power conversion© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

633

Yuxing Ouyang, Fengzhi Dai, Runhua Mao, Ce Bian, Baochang Wei, Yiqiao Qin, Shengbiao Chang, Qijia Kang

2. The hardware circuit design
The basic circuit of the system is divided into step-down
rectifier filter circuit, Buck circuit, measurement circuit,
SCM supply circuit, 430F149 minimum system circuit,
LCD liquid crystal display circuit and independent key
circuit. The step-down rectifier filter circuit is discussed
in Section 2-1. Section 2-2 describes the Buck circuit.
Section 2-3 introduces the 430F149 minimum system
circuit3.

Fig.2 The buck circuit

2-3. The circuit of the system
Other circuits are related to the control chip. 430F149
microcontroller peripheral circuits include reset circuit,
crystal oscillator circuit and power supply circuit. Powersupply circuit mainly uses the LM7805 regulator
module and LM117 regulator module. This part of thecircuit can be about 30V DC voltage into 3.3V DC
voltage, so as to provide a stable microcontroller
voltage. The circuit of the system is shown in Fig.3.

2-1. The step-down rectifier filter circuit
This circuit includes the transformer, full-bridge
rectifier and filter capacitor. The role of the transformer
is to convert 220V AC to 30V AC. After therectification action of the full bridge rectifier module,
the alternating current is converted to direct current, but
the waveform of the direct current is unstable. Using thefilter capacitor can make the DC waveform more stable4.
The step-down rectifier filter circuit is shown in Fig.1.

Fig.1 The step-down rectifier filter circuit

Fig.3. The circuit of the system

2-2. The Buck circuit
3. The control strategies

Buck circuit is mainly composed of the MOS tube
IRF540, inductor, capacitor and freewheeling diode.
MOS tube IRF540 mainly used in the circuit as ahigh-power switching components, is the core of the
DC-DC circuit. The diode protects the entire circuit by
preventing the reverse flow of current. Inductance andcapacitance form a low-pass filter, making the circuit
output current waveform smoother. The buck circuit isshown in Fig.2.

Good hardware circuits and powerful control ICs play a
crucial role in NC power, but control strategies are also
an integral part of maintaining the proper operation of adevice5. For the NC power developed in this paper, thecontrol strategy is mainly used in the chips on the
ACS712 current sensor module feedback signal
processing. The system can be based on the signal toadjust the PWM signal output. However, due to thesensor itself may be some problems, we can improve thesituation to some extent through data processing. The
PWM signal output must also have a certain control© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan634

Development of NC Powerstrategy, in order to improve the voltage conversion
capability of the entire system6.
Because the ACS712 current sensor feedback is ananalog signal, when subjected to external interference
may be abrupt changes in value, making themeasurement error. The control strategy adopts themethod of filtering the mutation value and obtaining the
average value. We remove each group of five data withlarger absolute deviation value and retain the five datawith smaller absolute deviation value. A total of 25 setsof data for the five groups, we again average these 25
data, the average as the true value of the feedback
signal.
After obtaining the feedback value of the dataprocessing, we use the difference between the feedback
value and the set value to adjust the duty cycle of the
PWM. Because the load will have a certain voltage andcurrent limit, so we adjust the PWM duty cycle, the step
value should not be too large, to prevent overshoot.
However, if the step value is too small will affect theadjustment time, making the adjustment time increases,
which will not be conducive to real-time response of theentire system. Therefore, we set the threshold method toadjust the PWM duty cycle. When the difference
between the actual feedback value and the set value islarger, we adopt a larger step value to adjust the duty
cycle of the PWM, and when the actual feedback value
approaches the set value, reduce the step value.
Through this strategy, the voltage conversion abilityof the entire system can be well improved, and thereal-time response of the whole system is good.

4-1 The AC-DC conversion simulation circuit
In the AC-DC conversion simulation circuit, we set
three measurement points, namely the input voltage Ui,transformer output voltage U1i and rectified output
voltage U1. We use voltmeters and oscilloscopes asmeasurement tools. The oscilloscope detects the
waveform of the final output voltage U2 and the time ittakes for the voltage to reach steady-state time. The
simulation result of AC-DC circuit is shown in Tab.1.

Ui (V)

Tab.1 The simulation result of AC-DC circuit
n1:n2
U1i (V)U1 (V) C(uF)
T(ms)

W

220
7:1
31.149 42.357
500
4.924
S
220
8:1
27.492 36.768
500
4.924
S
220
9:1
24.437 32.688
500
4.924
S
220
10:1 21.993 29.234
500
4.924
S
(Note: C: Filter capacitor, T: steady-state time, W:
Waveform, S: stable.)
If we hope that the final output voltage of the overall
circuit U2 range between 0-30V, then its AC-DC circuit
part of the output voltage values U1 which after
rectification must be above 30V. Taking into accountthe MOS tube IRF540 protection and as much as
possible to reduce the heating power, AC-DC circuit
output voltage values should be as close to 30V.
According to the above two points, when we select thetransformer turns ratio is 9: 1 turns the circuit can makethe output voltage value U1 is 32.688V.
4-2 The buck circuit simulation
Buck circuit as the core circuit of the NC power supply
studied in this paper will directly affect the efficiency ofthe entire system, so the parameters of each component
is particularly important7. We through the circuit
simulation to determine the Buck circuit capacitance,
inductance parameters. Select the optimal parameters tokeep the output waveform stable, reducing the time ittakes for the Buck circuit to reach steady state.
From the experimental data in Tab.2 we can knowthat actual simulation values and theoretical values aredifferent. Experimental data show that when we selectthe capacitance and inductance of the larger values, thelonger the time required to reach steady state, but the
waveform is relatively smooth. When the selected
capacitance and inductance is small, although the

4. The main circuit simulation
The maximum input voltage of LM78XX series voltage
regulation module is 36V, so the voltage after the filter
module of variable voltage rectifier must be reduced
below 36V. Buck circuit which capacitance andinductance parameters will directly affect the time thatsystem circuit to reach steady-state. So, we need to go
through a software simulation to find one or more setsof optimization data. Section 4-1 introduce the AC-DC
conversion simulation circuit. Section 4-2 introduce thebuck circuit simulation.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

635

Yuxing Ouyang, Fengzhi Dai, Runhua Mao, Ce Bian, Baochang Wei, Yiqiao Qin, Shengbiao Chang, Qijia Kang

processing and control strategies to ensure the stabilityof the equipment and formed a closed-loop control
system. In the whole system, the analog circuit part is
responsible for the voltage conversion, the digital
control part is responsible for the regulation of thesystem, and the two parts work independently andcomplement each other, so that the power supply
module is intelligentized and the device can be moresafely and steadily for the load Power supply8.

steady-state time is shorter, but the waveform isunstable. So, we choose the capacitance and inductanceparameters should choose the middle value, so wechoose 470uF and 100mH.
Tab.2 The simulation result of Buck circuit
U1(V)

D

TV(V)

SV(V)

L1(mH)

C12(uF)S(ms)32.688

20%
40%

6.538

5.915

100

470

203.4

13.07512.590

100

470

247.6

60%
80%

19.61319.388

100

470

255.2

26.24025.981

100

470

272.1

90%
90%

29.41229.285

100

470

250.5

29.41229.149

1000

470

397.4

90%
90%

29.41229.275

10

470

101.7

29.41229.275

100

47

68.7

32.68832.68832.68832.68832.68832.68832.688

Acknowledgements
The research is partly supported by the Research Fundfor the Reform in Education from Tianjin Municipal
Education Commission of China (171005704B).
References

90%
29.41229.338
100
4700
799.9
(Note: D: Duty cycle, TV: Output theoretical value, SV:
Output simulation value, S: Steady-state time)
32.688

1. Chun-Wei Feng, Wei-Feng Min, Juan Lei. Design of aHigh-precision Numerical Control LED Constant Current
Driver. Electronics World, 14(14): 113-114, 2017.
2. Yin-qiao Li, Yu-Ling Liu, Xue-Feng Liu, Shan Huang.
Design of NC DC power supply based on MSP430.
Computer Measurement andControl, 24(12):231-233+236, 2016.
3. Zhen-Feng Xiao, Rong-Xiang Yuan, Xiang-Tian Deng,
Xiao-Lei Liu. Remote Intelligent Fault Monitor Based on
MSP430F169. Electric Power Automation Equipment,
33(01): 160-164, 2013.
4. Zheng-Sen Jiang. Homemade Raiders of Precision CNC
Power Supply. Journal of Radio, (9): 58-63, 2012.
5. Xiu-Hua Wang. NC Constant Current Source Based on
Integrated Switching Regulator Chip. Power Technology,
5: 738-739, 2012.
6. Tian-Chen Huang, Song Jia, Jian-Hua Yu. Design andRealization of High Precision NC DC Current Source.
Instrumentation Technology and Sensor, (6): 27-29,
2013.
7. Zhong-Hua Yun, Huan-huan Yin, Yong-Feng Li.
Simulation and Research of DC Buck Chopper in
Multisim. Application Engineering of Computer
Engineering, (16): 216-218, 2015.
8. Zhao-Lin Huang, Wen-Tong Wu, Xiao-Jun Jing, et al.
Design of precision numerical control direct current
power supply. Control Automation, 8(8): 134-137, 2011.

From the circuit simulation results we can see that we
can determine the transformer turns ratio of 9: 1, and in
the Buck circuit, the specific value of the capacitor is
470uf and the inductor value is 100mH.
5. Conclusion
Developed in this paper based on Buck circuit NC
power supply, which can output any value between
0-30V stable voltage, but also real-time display current
and voltage values and other parameters. The device can
not only be used as a constant current source to a certain
extent, but also can freely switch between a constantcurrent source and a constant voltage source.
We are free to set the working hours of the equipment
programmatically and enable the equipment toautomatically calculate and display the voltage, current
and consumed energy values across the load and enable
the automatic alarm function when these values reachthe limit. At the same time, we incorporated data

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

636

Simulation of PID Temperature Control System Based on Neural Network
Yujie Yan, Fengzhi Dai *, Lingran An, Yuxing Ouyang, Zhongyong Ye, Xia Jin, Ce Bian

College of Electronic Information and Automation,
Tianjin University of Science and Technology, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

The system designed in this paper is a combination of neural network and PID control. BP neuralnetwork has a great advantage in solving the control of nonlinear and uncertain systems. It can use thesteepest descent learning method to adjust the threshold and weight values by back propagation. The
ultimate goal is to adjust the PID controller adjustable parameters by the neural network learningalgorithm. Using MATLAB software to simulate, the results show that the system has a strong
following performance and a high anti-interference ability. Experiments show that the PID temperature
control system based on neural network has some practicality.
Keywords: neural network, PID control, MATLAB, temperature

1. Introduction
Control theory has undergone several stages ofdevelopment and made historic contributions toaerospace, military and industry. However, both theclassical control theory and the modern control theory
rely too much on the mathematical model, which limits
these applications greatly1.
Main plant temperature control system is a typicallarge delay, nonlinear and time-varying system. If we use
a conventional PID controller, we will not achieve thedesired control effect. The work of this paper is to selectthe most widely used BP neural network algorithm and tocontrol the main steam temperature of the power plant by
combining the control strategy of three-layer BP neuralnetwork with PID classic incremental numerical control
algorithm. Because the main steam temperature of thepower plant is a time-varying uncertain object, it has
different mathematical models under different load values.
Basing on the data of the literature, we conductedsimulations under different loads. In addition to verifying

the following performance of the neural network PID
control system, it also increases the test of anti-jamming
capability. It is another practical attempt and exploration
to combine the traditional PID control and neural network
control to achieve the main steam temperature control2.
2. PID temperature control system based on BP
neural network
BP neural network is a kind of multi-layer feed-forward
network. It follows the error backwards and can learn a
lot of input and output information. To minimize the errorof network, It use the steepest descent learning method toadjust the threshold and weight values by backpropagation.
2.1. PID control system based on neural network
In the actual control system, the PID controller still
occupy more than 90%. In this paper, the design of thecontrol system, which has not only the advantages ofclassical PID controller, but also the advantages of neural© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

637

Yujie Yan , Fengzhi Dai , Lingran An, Yuxing Ouyang , Zhongyong Ye, Xia Jin, Ce Bian

Select the performance index function E(k), our goal is
to minimize E (k):network control3. As long as the input-output error of thesystem is not 0, the system continuously adjusts theneural network weights to obtained optimal PID
controller parameters. PID control system based onneural network shown in Fig.1.

Ek

1
rin k
2

yout k2(5)

Use the gradient descent method to correct neuralnetwork weight coefficient:

Ek
wli3

wli3 k

2.2. Algorithm of PID control system based on BP
neural network
Use incremental PID control algorithm, the expression is:

uk 1

kd e kkp e k2e k 1ek 1

wli3 kki e kek 2(1)

wli2 k

(2)

g x

ex
ex e x(3)

The activation function of hidden layer neurons is:f xtanh x

ex
ex

e x
e x

2
i

k

wli3 k 1

(7)

i

2

1
j

k

wli2 k 1(8)

Summarize the algorithm steps are as follows:
Determine the structure of BP neural network, that is,determine the number of input layer, output layer andhidden layer node. Give the initial value and thelearning efficiency of each layer's weight coefficient.
Give the input rin (k) and output yout (k) of thesystem and calculate the error error (k).
Calculate the input and output of neurons in eachlayer of neural network, and the output of output
layer is the three adjustable parameters of PID
controller.
Calculate the PID controller output u (k).
Learning through neural networks, online adjustmentof the weighted coefficient, to achieve adaptive
control of PID parameters.
Set k = k + 1, return to the first step.

Among them, f(*) is a nonlinear function trained byneural network. This design chooses 4-5-3 BP neuralnetwork structure. Because the three coefficients must be
positive, the activation function of the neurons in the
output layer is a non-negative4. The formula is as follows:1
1 tanh x
23l

The same can be introduced hidden layer weight value
formula:Because the kp, ki, kd are adjustable coefficients, theabove formula can be expressed as:u ( k ) f [u( k 1), kp, ki, kd , e( k ), e( k 1),e( k 2)](6)

In the formula α is the inertia coefficient and η is thelearning rate. The above formula adds the inertia term to
make the system converge quickly. Through calculationand simplification, get the output layer weight calculation
formula5:

Fig.1 PID control system based on neural networkuk

wli3 k 1(4)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

638

Simulation of PID Temperature

3. Application of neural PID control system formain steam temperature system in power plant
Thermal power plant power supply system by the boiler,
steam turbine generator and other equipment. There aremany factors that determine the stability and efficiency of
the thermal power plant, and the main steam temperatureof the boiler is the most important factor that affects thestability and accuracy of the power plant.
Fig.2 Power plant main steam temperature control system

3.1. Main steam temperature control characteristics
It is usually required that the main steam temperature becontrolled as accurately as possible rather than within acertain safety margin. In fact there is a saying that about
every 5 degrees lower thermal efficiency decreased by
1%. If the steam temperature is too high, the steam pipe
will react at high temperature and even rupture. If thesteam temperature is too low, the humidity will increase
and affecting the safe operation of the system. Moreover,long-term operation of the unit in the inappropriate
temperature range will greatly reduce the service life6.

The role of the main circuit of the control system is tomaintain the steam temperature given by the superheater,
the role of the secondary circuit is to eliminate theinterference. R in the above picture is the input of thesystem steam temperature, mh1, mh2 are the temperaturetransmitter transfer function. G1 (s) is the transfer
function of the lead-in area of the control system, G2 (s)is the transfer function of the inert zone of the control
system, and K is the transfer function of the actuator.
3.3 Simulation of main steam temperature control
based on neural network.

3.2. Main steam temperature control model andmethod in power plant

The main steam temperature control of thermal power
plants varies dynamically under each load. According to
the literature, the transfer function of the inert region canbe approximated as e-10s. When the system load is 100%,
the transfer function of the lead zone is7:

Main steam temperature control of power plants has threebasic disturbances. The amount of water to be reduced isoften used as an adjustment signal, and the system
structure under this adjustment signal is relatively simple
and easy to control. Although the load disturbance canalso affect the main steam temperature, it is an external
disturbance. Control system under smoke disturbanceboiler structure is too complicated for practicalapplication. After the water temperature changes have a
disturbance, the temperature released by the superheateris greatly reduced. However, the desuperheater outlet
temperature adjustment is faster, with outstanding
advancement, this section we call the lead zone.
Therefore, the main steam temperature control systemcommonly used in power plants is a PID cascade control
system.
Neural network-based power plant main steam
temperature control scheme, is to combine the neuralnetwork with the traditional PID controller to make up
for the artificial tuning PID parameters inaccuracy. The
block diagram of main steam temperature control systemof power plant controlled by neural network is shown in
Fig.2.

G1( s )

0.815
(1 18s ) 2(9)Carry on MATLAB simulation under this load, theresult is as follows:

Fig.3 100% step load response curve© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

639

Yujie Yan , Fengzhi Dai , Lingran An, Yuxing Ouyang , Zhongyong Ye, Xia Jin, Ce Bianat this time it is best to reduce the learning rate andreduce the set initial value. The design of this simulation
also verified, although only reduce the learning efficiency,
the adjustment time and overshoot are smaller, greatly
improving the control effect. All the simulation results
show that this design has a good anti-interference effectin addition to good follow-up performance.
4. Conclusion

Fig.4 Parameter curve

Due to space limitations, this article will not repeat thesimulation results under other loads. In fact, under
different loads, the simulation results show that thesystem follows well. Especially when the system isrunning at full capacity, it can perfectly determine thethree adjustable parameters of the PID to achieve the best
control effect.
A good control system needs to achieve stable, fast,accurate control effect. Due to the interference of main
steam temperature control system in thermal power plants,
the anti-interference ability of the system should also be
improved in practical application. In the simulation can
add a 0.1 interference to simulate the actual power plantinterference signal. Although there are some disturbancesin the disturbance, the system quickly returned to normal.
The result is as follows:

Fig.5 Step response curve plus interference

In addition, it is found in the other load simulations
that there is fluctuation in the front of the curve, and theovershooting is relatively large. According to experience,

At present, the use of neural network PID control system
is widely used in industry, agriculture and other fields.
This control system also controls the drying of the barleymalt, the raw material for beer. In addition can also be
used in car medical thermostat control system. After wecontinue to explore, in the future neural network control
must have a greater application prospects.
Acknowledgements
The research is partly supported by the Research Fund for
the Reform in Education from Tianjin Municipal
Education Commission of China (171005704B).
References
1. Peng Ge, PID algorithm based on neural network in the
furnace temperature control application. Northeastern
University, 2005: 10-15.
2. Weiguang Zhao, Ying Yang, Improved algorithm of PID
controller in cascade control system of power plant mainstream temperature. Power System Engineering, 2008,
24(6): 61-63.
3. Lei Ouyang, Based on self-tuning PID controller
temperature control system. Anhui: Anhui University of
Science, 2009: 5-8.
4. Linfang Sun, Mengmeng Shao, Xuying Liu, Superheatedsteam temperature modeling based on improving BP
algorithm. Techniques of Automation and Applications,2010, 29(4):1-3.
5. Chunxia Dong, Neural network fuzzy PID control in the
furnace temperature control application. Hebei: Hebei
University of Technology, 2004: 8-12.
6. Wei Qin, Based on the AM2301 temperature and humidity
meter design, Hunan Agricultural Machinery, 2012, 9: P59- P60.
7. Zhi Wei, Dongjie Gao, The control of main steam
temperature for supercritical once-through boilers. Control
Systems, 2008,24 (4): 17-18.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan640

Design and research on real-time material management system based on
production process
Baochang Wei, Fengzhi Dai *, Yuxing Ouyang, Haifang Man, Yiqiao Qin, Runhua Mao, Ce Bian

College of Electronic Information and Automation,
Tianjin University of Science and Technology, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

The paper primarily introduces the real-time material management system in thecourse of machinery production. This system is developed and designed using C#
language and SQL2008 database, using C/S and B/S mixed mode, the C#
language-related programming skill and database information together, through the
C/S and B/S architecture for network communication, the database Server-related
production material information sharing management system feedback to the specificuser, thus enhancing the production efficiency.
Keywords: database, C# language, ASP.NET, module design, real-time material system.

1. Introduction
With the rapid development of science andtechnology, the automation level of enterpriseproduction is higher and higher. For corporations,
Production management is the main part of its
development.
Anefficientproduction
management system plays a key role in thecontinuous development of enterprises and theimprovement of their overall competitiveness.
In the whole enterprise production, the use ofmaterial amount is an indispensable part ofproduction management, because in mostindustry production cost, material cost accounts
for a large proportion. Therefore, to achieve
efficient management of materials can save a lotof cost for enterprise production. In addition, thedegree of information sharing has become an
important symbol of the level of enterprisemodernization. The standardized design of thedatabase is considered as one of the effective
links to solve the problem of enterprise data
management [1].Through the data visualization
analysis to achieve the collection and sharing of
information in the production management stage,
Enterprises can more effectively arrange
material inventory, improve the utilization of

production machines, reduce inventory losses,
improve staff efficiency and customer service
levels.
In this paper, based on computercommunication technology, we create material
production information system, establish
database and data communication system.
This system is developed using C # language an
d SQL2008 design database, SQL Server 2008
database management system for data storageand management, performs data warehouse
functions [2].C # programming, combined with
ASP.NET technology, the system has a safe andreliable performance[3]. Network communication
is carried out through C/S and B/S architecture,and the information of production materials
related to the database is shared to the user torealize real-time monitoring of the target in the
production process. The company realizes
production process automation through network
technology, computer technology and databasetechnology.
2. System overall functional framework design
The producer needs to know and monitor the
production status in real time, and can master thewhole production process without going to the

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan641

Baochang Wei , Fengzhi Dai , Yuxing Ouyang , Haifang Man , Yiqiao Qin , Runhua Mao , Ce Bian

production site. In addition, enterprises hope thatthrough the information management systemaccurate and timely control of production
operations, in order to be able to quickly andaccurately refueling work to achieve the fullrange of product production control. Prior to thisproject, we reviewed the relevant materials andvisited the actual operation of the machinery in
the workshop and the actual use of the materials.
The Production site is shown in Fig.1.

2.2 System demand analysis
The system is mainly used for informationinteraction between production speed andmaterial management in workshop production.
In order to ensure the security of information, it
works mainly within the company's internal
local area network. System requirements in the
PC terminal and mobile terminal devices on thedisplay, according to the display function to
design a different interface. Describe the mainfunctions of each module and to achieve. The
system should realize the information sharing in
the company's production process.
2.3 Overall design planning

Fig.1. production site

2.1 The overall system design goals
To build a software system, the overall
framework model of the software system(blueprint) is first planned [4]. The main objectiveof this project is to design a production material
real-time management communication system.
System users are divided into two categories,
one is the operation workers, They mainly
operate the production process in real time; The
other group is the production manager, which ismainly responsible for looking at the overall
production situation and determining the specificproduction requirements of the product. The
system development process includes thedevelopment of the mobile front desk and thecomputer PC terminal and the design of the
background database. Among them, the worker
through the mobile terminal for inquiries,
Production manager through the PC side to view.
Then, according to the actual demand design anddevelopment of the foreground procedure, therelevant functions of the production schedule
information in the production process are
realized. System function module connection
diagram is shown in Fig 2.

Fig.2. System function module connection diagram

After the system function analysis, it can bedivided into three modules: Backgrounddatabase server system module, User front desk
device system module, User mobile Android
device system module. Project overall module
diagram is shown in Fig 3.

Fig 3. Project overall module diagram

2.3.1 Background database server systemmodule design
The module is mainly to store some data in the
production process, such as production materials
information. According to the actual situation of
production, the background database server
system module can divide itself into several
sub-modules: Production information moduleand equipment management module
Production information module
The function of this module is mainly to provide
the user with some information in the production
process, including production equipment list,
production record, production material situation,
production machine and material inquiry and soon.
Equipment management module
The module is controlled by the databaseadministrator, according to the operation status,
set the product, input the material information,and modify the production speed.

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

642

Design and research of

information,
including
product
number,production schedule, running machine code, therequired materials. Through the softwaredevelopment, based on the working conditionsof the production workshop, the real-time
information of machinery production is precisely
understood, and the information is summarized
into the database server

2.3.2 User front desk device system module
designThe module is used to view production
information. The manager needs to check theoverall demand of the production workshop
through the computer, Do not need to go to thescene to be able to know the production
workshop need material situation, facilitate
timely and accurate delivery. After the material
is delivered, the worker will log on the PC side,
feed the data back to the database, and update
the relevant display information.
2.3.3 User mobile Android device systemmodule design
The module mainly provides production
information for production operators, the mobile
terminal window is developed using jQuery
Mobile, Through the AJAX technology toreceive the return data and analysis. Loading theterminal APP into the phone allows the operator
to easily query the real-time production status.

3.2 Database Conceptual Structure Design
Currently widely used and can effectively
represent the data conceptual model is theentity-contact diagram, it is also called E-R chart.
E-R chart reflects the relationship between entity
type and attribute. The entity and properties ofthe database system design are shown in the Fig
4, where the rectangle represents the entity set
and the ellipse represents the properties of theentity.

3. Design and Implementation of Database
Management System
Based on the comprehensive consideration ofmaterial production information data volume,
operation platform support degree, softwaredevelopment cost and other aspects, the systembackground database is developed using
Microsoft SQL Server 2008 relational database.
The following are the main steps in database
design:
A: According to the actual production
requirement design database to store data, carryon the overall frame design, grouping.
B: According to the data packet design to create
the form.
C: Set the properties of the data in the table
D: Determine the relationship between tables
and optimize the data structure.
Generally speaking, the design database system
mainly includes data demand analysis,
conceptual model design, logical structure
design. As a database application management
system, all production data involved in the
production process should be stored in the
database system.

Fig 4 entity and properties of the databasesystem design

3.3 Database logical structure design
The logical structural design of data is totransform the data conceptual model into a
concrete data model supported by DBMS on thecomputer. According to the analysis of dataneeds, we can get the relevant entities that areavailable for production and determine therelationship between the entities.
4. System implementation and testing
In the first half of the article, the system isanalyzed and the overall framework is designed,
and the design of each functional module and thecreation of database are realized. This section
will implement the design of the corresponding
interface based on the previous analysis.

3.1 data demand analysis
Through the analysis of system function
requirements, need to establish a database server
system that can store the front-end client display

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan643

Baochang Wei , Fengzhi Dai , Yuxing Ouyang , Haifang Man , Yiqiao Qin , Runhua Mao , Ce Bian

4.1 System background database administrator

4.2 Background database login module test

interface implementation

System can distinguish between different
users, according to the user's login information
to determine whether to allow access to thesystem, when the user name and password exist
an incorrect, the user cannot enter, to prevent
non-professionals from making arbitrary
changes to the database. Randomly selected
multiple user names and passwords for testing,and found that only the user name and password
match to enter the system, achieving the desired
results.

This interface is used by database administrators
to modify the data in the update production. The
database administrator is required to log in to
determine the identity before entering the system,
after logging is eligible to modify and query the
data
and
other
operations.
Databaseadministrator login interface is shown in Fig 5.

5. Conclusion
Through continuous analysis and improvement,
the overall functional framework of the system
was established, and an effective databasemanagement display system for enterprises was
implemented. Through the system module test,
all achieved the expected effect, satisfied theenterprise to realize the various functional
requirements.

Fig 5. Database administrator login interface

4.1.1 Mobile terminal information display
interface
The interface displays the real-time
production status of the machine in the
production workshop, including the machinecode of each machine, production status, andmaterial consumption (expressed as a percentageof the material). For ease of differentiation, theproduction status of each machine is represented
by rectangles, the whole rectangle represents thetotal amount of materials (background set toyellow), and the consumption of materials is
expressed in red. Information display interface isshown in Fig 6.

Acknowledges
Thanks to Tianjin University of Science andTechnology, it provides me with a good learning
environment so that I can devote myself tostudying. Thanks to my graduate tutor, he helped
me a lot in study and life.
References
1. Jun Chen, Research on Data Standards andDatabase Design of Urban Rail Network.
Guangzhou: South China University ofTechnology, 2015:27-32.
2. Fuqiang Luo, Visual C#. NET program design
tutorial.
Beijing:
people'spost
andtelecommunications press, 2012.
3. Jinjin Zhu, Tuhao Shen, Kejin Bao, Therealization of Ethernet communication between
computer and PLC industry based on.net
4. Zhang F, Cui Y. Design of distributed
warehouse management system based on
CSLA.NET // Pacific-Asia Conference on
Circuits.2010:292-295.
5. Lijuan Liu, The application of interaction designin man-machine interface design. Electronic
Technology
andSoftwareEngineering,2015(6):76-76.

Fig 6. Information display interface

4.1.2 Prepare material information display
interface
The interface is distributed on the lower left ofthe mobile information display interface, and thereal-time data status of the parts of the machine
is displayed in the same interface for the
production personnel to view. Prepare material
information display interface is shown in Fig 7.

Fig 7. Information display interface

The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan644

Stability Control of Inverted Pendulum Based on Multi-loop PID
Zhongyong Ye 1, Fengzhi Dai 1*, Yuxing Ouyang 1, Yujie Yan 1, Xia Jin 1, Lingran An 1, Hongwei Jiao 2
1 College of Electronic Information and Automation,
Tianjin University of Science and Technology, China

2

Tianjin Technology School of Printing and Decoration, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
The traditional PID control can be very good for the stability of the pendulum angle control, but for thelocation control of car that loads the pendulum, it can do nothing. That is, the car will go along the rail
without restrictions on the movement. The car and inverted pendulum system is a single-input multiple
output instability system, so the multi-loop PID controller is designed based on the output feedback thatis particularly important. Simulation shows the feasibility of using the multi-loop PID control, indicatingthat improved PID control is entirely feasible.
Keywords: Inverted pendulum, Multi-loop PID control, Output feedback, Simulink

1. Introduction
2. Mathematical Modeling

Inverted pendulum as a typical instability, a serious nonlinear system, used to test the control methods of
instability, nonlinear and rapid control. Using different
control methods to control different types of inverted
pendulum has attracted the attention of many scientists allover the world and has become one of the most challenging
topics.
Inverted pendulum system has two balance points: thestability of the suspended state equilibrium point and theinverted state of the unstable equilibrium point. Control of
inverted pendulum includes pendulum control and stability
control. Pendulum will swing from the stable equilibrium
point to the unstable equilibrium point, and the stable
control will be inverted the pendulum system stabilizednear the unstable equilibrium point.
Now there have been many papers published about thependulum and stability control, and these results adopt
different control ideas for swinging and stabilization andimplement these two processes with unified control theory.

An inverted pendulum system consists of an inverted
pendulum shown in the Fig.1, and the car along the rail
movement. The inverted pendulum is fixed on the car.

Fig.1 Model of Inverted pendulum system

2.1 Parameters

M Car quality© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

645

ZhongYong Ye, FengZhi Dai, YuXing OuYang ,YuJie Yan, Xia Jin ,LinRan An, Hongwei Jiaom Pendulum quality
b Car friction coefficient
I Pendulum rotation axis to The length of the rod

xycenter of mass
J Pendulum inertia
F The force added to the car
x Car displacement
The angle between the pendulum and the vertical
downward direction
The angle between the pendulum and the vertical
upward direction

3.

.
(b) Swing rod force

Fig.2 Force analysis of trolley and pendulum in the system
.

.

..

'

Assume X {x, x, , } , u
x , we can get the car
acceleration as the input inverted pendulum system state
space expression:
.

x
..

x
.

..

0 1
0 0
0 0

0
0
0
3g
0 0
4l

0 x
.
0 x
1
0

.

0
1
'
0 u
3
4l

0

0 0 1 0

0

.

u'

(2)

PID controller design

In the traditional PID controller design, the control canwell control the stability of the pendulum angle to thelimitations of the traditional PID. However, the position of
the car cannot be well controlled. That is to say, the car
will move along the unrestricted movement of the guide
rail.
Therefore, it is particularly important to design adouble-loop PID controller based on the inverted
pendulum system with output feedback. By modeling thependulum angle and car acceleration, the pendulum
displacement and car acceleration transfer functions are:Fig.2 is the system of car and pendulum force analysis.
Among them, N and P are the horizontal and verticalcomponents of the interaction force between the car andpendulum.
In the actual inverted pendulum system, the positive andnegative directions of detecting and executing the device
have been completely determined. Therefore, thedefinition of the variable direction is as shown in Fig.2,
and the directions of the figure are all positive vectors.

Car force analysis.

1 0 0 0 x

3.1 Inverted double loop PID controller design

2.2 Mathematical modeling

(a)

x

(1)

( s)
V ( s)ml( I ml ) s 2 mgl

G2 ( S )X (S )V (S )

2

1
S2

(4)

The double loop PID controller is actually a kind of
feedback correction idea. Under the condition that the
open-loop value of the local feedback loop is far greaterthan 1, the characteristics of the local feedback loop
mainly depend on the feedback device, and irrespective of
the enclosed part, proper selection of the form andparameters of the feedback correction device can make theperformance of the corrected system that meets therequirements of a given indicator.
However, it should be pointed out that when designing
the feedback correction system, the stability of the inner
loop needs to be taken into consideration and the choice ofthe feedback correction parameters should be madeimproper so that the inner loop loses its stability. Therefore,
it is difficult for the entire system to work stably andreliably and not convenient for the system to be open loop
debugging.
Therefore, the feedback loop formed within the system,
the best is stable. Still with the car's acceleration as input© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

646

(3)

Stability Control of Inverted

to the pendulum angle and the car's displacement as theoutput, the resulting dual loop PID correction control
system block diagram shown in Fig.3.

Fig.4 Double loop PID control system model diagram

Fig.3 Dual loop PID control system structure

Substituting the system parameters into G1 (S) and G2(S) gives the transfer function:
Transfer function of pendulum angle and car
acceleration

G1 ( S )

(s)
V (s)

0.027250.0102125s2 - 1.26705(5)
Transfer function of car position and car acceleration
G2 ( S )X (S )V (S )

1
S2

Fig.5 Two-loop control system pendulum angle simulation curve(6)

3.2 Double-loop inverted pendulum simulation
This paper considers the pendulum angle as the outer loop,
and the car displacement as the inner loop. The outer loop
control object (transfer function Eq.(3)) and the inner loop
control object (transfer function Eq.(4)) into the control
system that is shown in Fig.3. Then we can get the
Simulink simulation shown in Fig.4.
The simulated curves are shown in Fig.5 and Fig.6
respectively. The simulation results show that the dualloop PID control scheme can achieve the double stabilityof the car position and pendulum angle well and achieve
the desired control purpose and requirements. Moreover,during debugging, we found that this system has low
sensitivity to the change of PID parameters. When thespecified parameters change within a certain range, thecontroller can effectively control the inverted pendulum
and make it run stably.

Fig.6 Double loop control system car position simulation curve© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

647

ZhongYong Ye, FengZhi Dai, YuXing OuYang ,YuJie Yan, Xia Jin ,LinRan An, Hongwei Jiao

4.

about inverted pendulum, mastered a certain theoreticalknowledge, completed the design of the inverted
pendulum controller, exercised my own ability to analyze
and solve problems, and laid the foundation for future
work solid foundation.

Summary

The inverted pendulum control has the importance ofengineering to verify many control laws. Of course, theproject most commonly used the PID control law that is
simple and reliable, low cost, with more importantpractical. The PID control law is also based on the idea of
feedback, which means that the feedback control occupies
a fundamental role in the entire automatic control theory.
In this paper, the first level inverted pendulum is using
the conventional PID control. But we found that one PID
control can only control one variable, and an inverted
pendulum is a typical single-input dual-output nonlinear
system. The traditional PID cannot do both the amount ofcontrol, so we use the dual-loop PID control, and it is sureenough that the simulation shows that the dual-loop PID
controller is feasible, and can meet our requirements for
the PID controller, thus to solve the shortcomings of the
traditional PID.

References
1. Yixiao YIN, Study on Linear Inverted Pendulum System.
Shandong, Shandong Agricultural University, 2015.
2. Hailong Liu, Design and Research of Inverted Pendulum
Experiment System. Dalian, Dalian University ofTechnology, 2006.
3. Li D Y. The cloud control of triple inverted pendulum andits dynamic balance mode. Engineering Science, 1999,
1(1):41-46.
4. Wang P H, Li X B, Fu F Q, Wang Q. The full process
variable structure control of 3-stage system. Machinery
Design & Manufacture, 2009, 7(1):127–129.
5. Yude Wang, PID Parameter Adjustment Simulation Study.
Electric Automation, 2011, Vol.33, No.6: 1-6.
6. Lin Li, Double Circuit PID Control of Inverted Pendulum.
Journal of University of Shanghai for Science andTechnology, 2012. Vol 32: 75-76.
7. Li Z S. Artificial Intelligence: Retrospection andExpectation. Beijing: Science Press, 2006:174-207.
8. Li Z S, Wang Y X, Tan Z, Wen Y L. The swing-up andhandstand control and its implementation of cat double
pendulum. Proc of the 5th world congress on intelligent
control and automation, Hangzhou: IEEE, 2004: 23602364.

Acknowledgements
The research is partly supported by the Research Fund for
the Reform in Education from Tianjin Municipal
Education Commission of China (171005704B). And at
this point, I would first like to thank to my tutor, Dr.
Fengzhi Dai. From the beginning of the paper to the final
draft, I have always received the meticulous guidance andhelp from Dr. Dai. His pragmatic and rigorous scholarshipand work style deeply infected me, so I benefit greatly!
During the research, I consulted a large number of papers© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

648

Design of Intelligent Saving Robot Based on Six-legged Robot
Xinyu Zhang, Xiaokun lin, Feng Dai *College of Electronic Information and Automation,
Tianjin University of Science and Technology, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

In recent years, along with the humanity to have high mobility in the complex environment, and highreliability and easy to expand increasingly urgent demand for mobile platform, also the multi legged robot
research and algorithm improvement of gait, scientists have considered the research to the application of
biped multi legged robot. The motion of the biped robot is confined to the plane, unable to overcome many
mountain disaster or rough terrain. Therefore, a rescue robot based on six legged robots is considered. This
robot can use mobile phone app to switch robot mode or remote control robot through Bluetooth or wirelessmode. And it can be controlled by switch between the automatic and manual mode, and the robot can realize
obstacle jumping through sensors to identify the types of obstacles. The robot is loaded with an electronic
compass, a camera and a gyroscope, which can intelligently record the path of travel and automatically return
to its starting position along the path.
Keywords: Six-legged Robot, Saving robot, Remote Control

robot's mechanical flexibility and a variety of sensors,
programming algorithm, which completely turned into
intelligent robots, autonomous unmanned control
through complex the environment and a variety ofadditional high-end convenient features to make it more
effective to complete the task. Its purpose is to make footrobot can be applied to practical engineering applications,so that it can accomplish rescue, disaster relief and otherhigh-risk work, so as to solve the traditional obstacles
such as robots, traffic difficulties and other issues.
Through innovation, robots can serve human beings more
effectively.

1. Introduction
Nowadays, the research on the six legged robot is popularin the world. In addition to the industrial six legged robot
is indirectly applied to sorting, most of the six legged
robots are only used for entertainment and teaching. Six
legged robot with wheeled and tracked robot while
showing great advantages in the common way, but insome practical engineering applications, such as disaster
relief and rescue, have not appeared the figure of six
legged robot. Our most common dance robots are written
only through the upper computer, into the action group,and then perform the action group to complete thevarious dance movements. This robot can only be used
for teaching and entertainment, and cannot be applied topractical engineering applications. The robot is in theoriginal dance of the six legged robot based ontransformation and special rational design of it and add it
to the main control chip, the dance of the six legged

2. The hardware structure design
To better fit the look of the six legged robot, we placed
the relevant sensors and hardware modules on apreviously designed 15*15 PCB. Multiple modules© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

649

Xinyu Zhang, Xiaokun lin, Fengzhi Dai

contact each other and perform their duties. At the sametime, we have set aside some blank modules on the basisof independent design, which is convenient for latermodification and replacement. Taking into account thelimited space inside the robot and the central area, wetake the upper and lower double layer design to provide
a good environment for the operation of the LCD and
GPS modules. The design of the structure is shown in
Fig.1.

2.2. Main control chip
STC12C5A60S2/AD/PWM series microcontroller is asingle clock / machine cycle macro crystal technology
production (1T) chip is a high speed, low power
consumption, strong anti-jamming and a new generationof 8051 single-chip instruction code is fully compatible
with the traditional 8051, but the speed is 8-12 times
faster. Internal integrated MAX810 dedicated reset
circuit, 2 way PWM, 8 way high-speed 10 bit A/Dconversion (250K/S), for motor control, strong
interference occasions. STC12C5A60S2 has several
important characteristics:
chip integrates 1280 bytes RAM.
each I/O port drive capacity can reach 20mA, but
the whole chip should not exceed 120ma.
has the EEPROM function.
There is an independent baud rate generator
A/D conversion, 10 bit precision ADC, a total of 8channels, the conversion speed of up to 250K/S(250 thousand times per second)
The robot uses independent STC12c5a60s2
microcontrollers, each of which is responsible forthe operation of each function of the robot.
The main control chip is shown in Fig.2.

Fig.1 The design of the structure

2.1. Servo control panel
The robot has 18 digital servos, so we have to design a
32-way steering engine panel. The steering engine
control panel uses the ATmega16 as the main control.
The ATmega16 AVR kernel has a rich instruction set and
32 general-purpose working registers. All registers are
directly connected to the arithmetic logic unit (ALU), sothat one instruction can access two separate registers
simultaneously in one clock cycle. This architecture
greatly improves code efficiency and has a data
throughput rate of up to 10 times higher than the average
CISC microcontroller.
ATmega16 has the following features: 16K bytes of insystem programmable (Flash has the ability to read andwrite at the same time, namely RWW), 512 byte
EEPROM and 1K byte SRAM, 32 common I/O port line,
32 general-purpose working registers, JTAG interface forboundary scan, supports on-chip debugging andprogramming, compared with three the flexible mode oftimer / counter (T/C), internal / external interrupt chip,
programmable serial USART, serial interface with initialcondition detector, 8 Road 10 with selectable differential
input stage programmable gain (TQFP package) ADC
has an on-chip oscillator Programmable Watchdog Timer,
a the SPI serial port, and the six can be selected by thesoftware power saving mode.

Fig.2. STC12C5A60S2

2.3. L3G4200D three axis digital gyroscope sensor
Simultaneously determine the position of 6 directions,move track, accelerate. A single axis can only measure
two directions, that is, a system requires three gyroscopes,
and one of the 3 axes can replace the three single axis
ones. 3 axis small size, light weight, simple structure,
good reliability, is the basic component of thedevelopment of laser gyro. The biggest function of thethree axis gyroscope is to measure angular velocity to
distinguish the motion state of the object, so it is also
called motion sensor. Digital three axis gyroscope can
support ISP communication and IIC communication.
This robot takes IIC communication. And the L3G4200D
three axis gyroscope digital characteristics that we foundin practical applications to a certain angle gyroscopefrom a horizontal position, the detected module tilt angle© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

650

Design of Intelligent Saving

and then to send to the microcontroller module thecurrent location for the starting position, the robot to step
provides a convenient. The L3G4200D is shown in Fig.3.

Bluetooth wireless Wi-Fi and 720P HD camera for image
acquisition and transmission, the remote control method
is more stable than ordinary safe and reliable.
The HD camera and wireless Bluetooth module is shown in
Fig.5.

Fig.3. L3G4200D

2.4. GY-271 three axis digital electronic compass

Fig.5.HD camera and wireless Bluetooth module

HMC5883L uses anisotropic magnetoresistive (AMR)
technology, which has the advantage of being unmatchedby other magnetic sensor technologies. The anisotropic
sensor has characteristics of high sensitivity and highprecision linear axis. Solid phase structure for theorthogonal axis line sensor with low sensitivity can beused to measure the size and direction of the earth's
magnetic field, the measurement range from 8 to milli
Gauss (Gauss). Honeywell's magnetic sensor is the mostsensitive and reliable sensor in the low magnetic fieldsensor industry. The GY-271 three axis digital electronic
compassis shown in Fig.4.

2.6. Steering engine
The servos are 18 1501 digital servos. Six legged robots
with 3 legs each. 1501 metal gear torque, fast speed, low
noise, virtual and dead are very small, only need to senda signal to the locking angle, with high control precision,
good linearity, in strict accordance with the control
protocol, the output angle accurately and fast response
and other advantages. At the same time programmablefeatures also make the digital steering gear in dealing
with jitter, and offside to more convenient and accurate,
slow and more gentle, more smooth, more effective forthe motor to provide the torque needed to start. Whenusing servo controller, PC slider index 500~2500(0.5ms~2.5ms), corresponding to the 1501 digital servo
0~180. The steering engine is shown in Fig.6.

Fig.4. GY-271 three axis digital electronic compass

2.5. HD camera and wireless Bluetooth module
In order to make the robot better with the mobile phone
and other mobile terminal communication, we adopt

Fig.6. Steering engine© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

651

Xinyu Zhang, Xiaokun lin, Fengzhi Dai

estimate of the robot landmark location, although the
GPS module including the design of hardware, but thesignal is often terrain and object occlusion, leading to theaccuracy greatly reduced the effectiveness of GPS signal
under complex terrain condition is only 60%. And, in still
conditions, the GPS cannot give the heading information.
In order to make up for this deficiency, the method of
combining electronic compass and GPS for directionalnavigation is adopted. It can effectively compensate theGPS signal, and ensure that the navigation directional
information 100% is effective, even after the GPS signal
is lost, the lock can also work normally, and the "lost star"is not lost".
L3G4200D three axis digital gyroscope and masterchip communication mode is IIC communication. In any
leg of six legged robot gyroscopes in robot leg and lower
leg corresponding tilt direction axis has two times thenumber of changes per two times the number of changes
is counted as one step. Then, the robot moves through themain control of the digital gyroscope repeatedly twodigital changes step. The combined measurement bydirectional and pedometer can estimate distance, and thecentral chip data exchange through WIFI and Bluetoothtransmission, real-time display of the completion
schedule and tasks can be mobile terminal.

3. System circuit module design
The basic modules of the system include power module,
obstacle passing module and route memory module.
Through the cooperation of three modules, the basic
functions of transmission, obstacle avoidance and route
feedback are completed.
3.1. power module design

In order to provide energy effectively, the 5V DC power

supply is adopted, and the DC-DC exchange module isshown in Fig.7.

Fig.7. power module design

3.2. Robot obstacle passing module (sensor distribution
design)In order to exercise effective and efficient real-time
control and monitoring of the robot, selects the best
multiple sensor mounting points, the perfect realizationof the linkage and communication module, and ensure
the smooth motion. The sensor distribution design isshown in Fig.8.

4. Introduction of functional module
In the course of development, the robot has applied somebasic scientific principles, and through scientificcalculation and adjustment, the machine has beenrunning smoothly and accurately in the plan.
4.1. Robot gait principle
The gait of the six legged walking robot is varied, inwhich the triangle gait is the typical gait of the six legged
walking robot. "Six feet" insects usually do not walk six
feet at the same time while walking, but instead divide
three pairs of feet into two groups, alternating with atriangular support structure.
At present, most of the six legged robot adopts the
structure of insects, 6 legs distributed on both sides of thebody, front and rear foot and the right side of the body on
the left side of the foot as a group, before and after theleft foot and right midfoot for another group, werecomposed of two triangular bracket, rely on the leg
before and after the implementation of paddling supportand swing, this is a typical triangle gait, as shown infigure. In the diagram, the robot hip moves horizontallyand vertically. At this point, B, D, F feet swing feet, A,
C, and E feet do not move, just support the body forward.

Fig.8. sensor distribution design

3.3. Robot route memory module (electronic compass
and gyroscope application design)
In order to accurately measure the route and facilitate thedistance when moving, the application of electronic
compass and gyroscope is introduced. According to theelectronic compass readings, according to a rough© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

652

Design of Intelligent Saving

The XYZ value used for azimuth calculation must
remove the drift value so that the center of the circle
returns to the origin. The specific method is: the level of
uniform rotation, XY axis rotation data collection
equipment, 90 degrees (at the level of Z axis Z axis)
uniform rotation to collect data, the data is read into theshaft of the maximum value and minimum value by 2,
you get a offset value of each axis.

Because the body gravity center is low, not coordinatedZ movement, easy stability, so this walking program canbe widely used. The Sketch map of triangle gait is shownin Fig.9.

Xoffset= Xmax+Xmin /2
Yoffset= Ymax+Ymin /2
Zoffset= Zmax+Zmin /2
Then, the naked values of each axis read by themagnetometer subtract the offset value calculated
previously, and the Heading values used for angle
calculation can be obtained
If only horizontal measurements are used, azimuth=arctanYH/XH. If our device is not in a horizontal
position at the time of measurement, the azimuth of thecompass will deviate when the above formula is applied,
and then the acceleration sensor will be used tocompensate the tilt angle of the magnetometer. As shown.
To compensate for the inclination of the electronic
compass, the roll angle (theta) Roll and pitch angle (Pitch)
must be calculated first .Enter the following formula andcalculate the Heading value.

Fig.9. Sketch map of triangle gait
4.2. Azimuth calculation of electronic compass
When the 3 axis magnetometer work can read XYZ threeaxis magnetic field intensity, and cannot directly used fornumerical calculation of azimuth angle, because thereadings may be some other devices layout containing
magnetic materials, the formation of hard iron drift
coordinates, as shown in the following figures. The
Azimuth calculation of electronic compass is shown in
Fig.10 and Fig.11.

Xh=X*cos(φ)+Y*sin(θ)*sin(φ)-Z*cos(θ)*sin(φ)
Yh=Y*cos(θ)+Z*sin(θ)
Considering the 4 quadrants of the angle, the formulaof the heading angle can be changed into the following
formula.
for(Xh<0)=180-[arctan(Yh/Xh)*180/π]for(Xh>0,Yh<0)=-[arctan(Yh/Xh)*180/π]for(Xh>0,Yh>0)=360-arctan(Yh/Xh)*180/πfor(Xh=0,Yh<0)=90
for(Xh=0,Yh>0)=270

Fig.10. Xh-Yh Figure

5. Testing and conclusion
The robot can achieve the following main functions in theactual test: the robot can pass all kinds of obstacles, suchas self-climbing, stiles, obstacle avoidance, obstacle etc.
We can switch to the man-machine control mode, and theoperator can use the mobile phone APP to control therobot wirelessly.

Fig.11. X-Y Figure© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

653

Xinyu Zhang, Xiaokun lin, Fengzhi Dai

The robot also has a one-button automatic return
function, and the controller automatically returns to thepoint of departure by simply pressing the automatic
return key. In the design process, the test was moved
outdoors, in the natural environment as the experimental
site, the environment obstacle robot testing in the outdoor
space of this complex capacity, if the robot should beapplied to the engineering practice, the robot should be
able to adapt to the requirements of the design
environment.
The actual error under the design of subtle errors andhardware, the robot automatically return to the starting
point of the position deviation distance error issometimes not the expected range, and the distancecoefficient of nonlinear proportional distance error withthe robot walking in the actual test for an average of
0.0112. The deflection error is within the range of 4 to 20
degrees.

References
1. Yusuke OTA, Yoshihiko INAGAKI, Kan YONEDA, et a1.
Research on a Six—Legged Walking Robot with Parallel
Mechanism. Proceedings of the 1998 IEEE/RSJ Intl.
Conference on Intelligent Robots and Systems, Victoria.
2. Bares J E. Whittaker W I. Configuration of autonomous
walkers for extreme
3. Xu Xiaoyun, Yan Guozheng, Ding Guoqing. Research on
micro six legged bionic robot and its triangle gait. Optics
and precision engineering, 2002, 8(10): 392-393.
4. Fu Li, Liu Weiguo, Iran, multiple servo control with PwM
microcontroller wave generating method, micro motor,
2006, 34(2).
5. Hu Xia. Research on Key Technologies of six legged
walking robot. Hangzhou: Zhejiang University, 2008)
6. ray Jingtao, peak, Cui Ying. The research status andProspect of multi legged walking robot. Beijing: Journalof mechanical design, 2006.
7. Zhang Haitao. Energy analysis of walking machines underhorizontal static stability. Calculation of joint drive power.
Journal of Tianjin University of Science and Technology,
2001 (2):13.15.
.
8. Gan Jianguo, Zhu Wei, the stem. Six legged walking
machine with guiding gait research of Robot, 1996, 16(4):235.

Acknowledgements
The research is partly supported by the Research Fundfor the Reform in Education from Tianjin Municipal
Education Commission of China (171005704B).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan654

Research on multi-object recognition algorithm based on video
Yong Hou 1, Runhua Mao 1, Yan Yu 1, Yuxing Ouyang 1, Ce Bian 1, Binhu Song 1, Baochang Wei 1,
Yiqiao Qin 1, Shengbiao Chang 1, Fengzhi Dai 1*, Hongwei Jiao 2
1
College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
2 Tianjin Technology School of Printing and Decoration, China

E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

This paper mainly studies and improves the detection and tracking algorithm of therobot fish video and presents a background extraction method based on HSV color
space. This method is more efficient than the background extraction method of RGB
color space, which improves the recognition, anti - jamming ability and robustness.
Through the experiment, it has a strong adaptability to external influences such as
background water surface fluctuation, reflective, light change shadow.
Key words: Robotic fish, tracking recognition, machine vision, color spacecolor is blue, so adjusting the coefficients to 1:
2: 7 according to the Lagrange multiplieralgorithm makes a better distinction between
foreground and background.
First take the static background without fish,save as a static background image. Then makearithmetic of subtraction on the background. Inorder to eliminate the influence of reflections
and shadows, use the edge average method to set
threshold and median filtering to filter out theseinterference points. Then used the templatematching algorithm to match the previously
designed oval template with the extracted fish.
And then fuzzy positioning4of the fish in thewater within the camera's video recording range
by using the basic means of human’s eyes’recognition. The robotic fish's position in thewater is transmitted to the computer by theoperator using the mouse frame selection
method. To identify the scope of this fuzzy
framework for the recognition of the position
and attitude of the robotic fish in the water5.
In order to make the matching more robust,
the template matching algorithm uses a double
template, the foreground template matches thewhite part, and the background part matches the

1. Introduction
The current tracking algorithm of moving
objects1 improve the matching degree of theobject and the correctness of the matching by
optimizing the characteristics of the tracked
object, and predicts the possible occurrence ofthe object at the next moment or determine theobject's search direction by the manner ofeffective narrowing the object search range .
Meanwhile, in order to get a better prospect, thispaper presents color-space model based on
HSV2. After remodeling the original color-space,
we remove the main light interference in theimage and the shadow interference of the robotic
fish effectively, which makes the segmentation
and recognition of multiple robotic fish3 moreaccurate. So the improvement of this algorithm
makes the original matching tracking algorithm
more robust.
2. Object Recognition
RGB channel modeling is used in theexperimental platform. The background mainly© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

655

Yong Hou, Runhua Mao, Yan Yu, Yuxing Ouyang, Ce Bian, Binhu Song, Baochang Wei, Yiqiao Qin, Shengbiao Chang, Fengzhi
Dai, Hongwei Jiao

black background part, so that our matchingalgorithm is faster and more robust6.

Then load our selected image, select the area
as far as the robot fish or to be separated fromthe center of the object;
Put all of these points in the HSV
three-dimensional space, and then draw these
20*20 points in the 3 dimensional diagram.
These values are collected and arranged, andthe actual color is distributed as table 1.

Fig.1 Double layer formwork

Table 1 spatial numerical distribution of HSV

But in the actual competitions, it was foundthat imperfect pool modeling, changing
competition environment, reflection of the pool,and some environmental factors seriously
affected the players' play7. Therefore, it isnecessary to improve the large platform
algorithm.
The influence of robot fish pool modeling has
the following points:
The influence of the light in the pool, suchas uneven lighting, the pool edge blocking
the light, resulting in the background of theprevious collection is invalid;
The shadow of the fish has an influence on
the fish's marking;Irregular reflection caused by watersloshing, serious distortion of color causedby the collection of many irregularreflective areas;
Not suitable for color fish.
3.

Gray

White

H（min） 0

0

0

0

11

155

H（max） 180

180

180

11

25

180

S（min） 0

0

0

43

43

S（max） 255

43

30

255

255

V（min） 0

46

221

46

46

V（max） 46

220

255

255

255

Black

Red

OrangeContinuous table 1 spatial numerical distribution of
HSV
Yellow

Green

Cyan

Blue

PurpleH（min） 26

35

78

100

125

H（max） 34

77

99

124

155

S（min） 43

43

43

43

34

S（max） 255

255

255

255

255

V（min） 46

46

46

46

46

V（max） 255

255

255

255

255

Data analysis:
As the laboratory pool is glass, the reflection
is more serious than the competition pool. Under
sunshine, the pool color is distributed between
cyan and blue. We adjust the full range of color
to 0, also is black background.
There is also a problem to be solved, part of
the red distribution in the 0-10 and 156-180,
which will affect our identification of red Koi
fish, make part of the fish's foreground show
black and some show white. We need to adjust
this part of the red but show the black tohighlight range, while the black fish's
foreground needs other ways to handle it.

Algorithm improvement

3.1 Algorithm Introduction
In order to recognize the color of fish, and tobetter filter out the light and fish shadows8, thispaper presents a better classification algorithm
based on the color feature of HSV feature space.
Compared with the RGB channel, HSV iscloser to human perception of color. The color
property H can reflect the variety of the color
more accurately, and the sensitivity of thechange of the environment is relatively low, so it
is very suitable for the image processing.
First we need to analyze how these different
colors of fish, pool, and back light are
distributed in the HSV space so that they can beclassified later to realize the concept of program
algorithm as follows:First we manually select 20*20 pixels range
area as the analysis object;

3.2 Improvement of experimental parameters
We have adjusted the desired part of the color
to the highlighted, and then adjusted thechromaticity of the background and theilluminated part to zero. We can see the effect asfollows: we still cannot see the black fish (leftone).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

656

Research on multi-object recognition

and the operation is convenient. In addition,
solve all the problems caused by the previous
environment, so it will be integrated into theglobal machine fish platform to further test
results in various universities.
Fig.2 Parameter adjustment of H channel
V S R B color channel (top right of S)

Fig.5 Pool test of Guangxi University of Science andTechnology

From the test results of Guangxi science andTechnology University, we can see even if thehuman eye cannot see the yellow fish , thecomputer can also identify part of the fish body .

Fig.3 Other color channel testing (V S R B channels
respectively)

In Fig.3, we can find that the B and V
channels in these figures can be used because
black fish can be clearly extracted without
introducing other effects such as lighting, poolbackground, etc.

Fig.6 Pool test of China University of Geosciences

It can be seen that the fish and the ball in thepool are well separated.

3.3 Algorithm experiment and improvement
At this time, we tried to merge these channels
with the reclassified H channels9, and came tothe following results:

Fig.7 Glass pool test results of Peking University

It can be seen from pool video in different
environment that the red fish and yellow fish are
extracted and the light part turned black.

Fig.4 Dual channel fusion

We can see the black fish again, but there aresome watery impurities. Here, the median filter
is used to filter the image.
4. Testing and conclusion

Fig.8 Pool test results of Beijing University ofTechnology

4.1 Test under different circumstances
Compared to the previous foregroundextraction method, the present method is simpler,
the complexity of the algorithm is low. It is notnecessary to collect the background, just open
the camera to go to select the fish for the game,

It can be seen that there are many impurities
in the pool, the reflection light is very serious,
the whole pool's pixel value fluctuates violently,
but the method can still extract the fish
effectively.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

657

Yong Hou, Runhua Mao, Yan Yu, Yuxing Ouyang, Ce Bian, Binhu Song, Baochang Wei, Yiqiao Qin, Shengbiao Chang, Fengzhi
Dai, Hongwei Jiao

the robotic fish cannot make a good distinction,
the follow-up test will further study to solve thisproblem.

4.2 Different light interference test
Highlights test:

Acknowledgements
The research is partly supported by the Research
Fund for the Reform in Education from Tianjin
Municipal Education Commission of China(171005704B).

Fig.9 Pool test results (highlights) of Peking
University

We experimented with maximizing light
disturbances, partially whitening the fish, andmanipulating the video without changing any
parameters. We can see that this method is still
effective and can extract the outline of the fish.
Lack of light test:

References
1. Chang Xiangkui The research on Tracking
Algorithms of Video Moving Objects. Henan
University. 2007
2. Zhang Jun, Dai Ke-xue, Li Guo-hui. HSVcolor-space and codebook model based
moving objects detection, 2008,(03):423-427.
3. Wang Yaowei, Ji Zhijian, Zhai Haichuan. A
survey on motion control of the biomimetic
robotic fish. CAAI Transactions on Intelligent
Systems, 2014, 9( 3) : 276-284.
4. Tang Yufeng. Research and realization of themulti-target tracking system for robotic fish
based on global vision. East China Jiaotong
University. 2013.6
5. Ding Guangshuai, Lei Yunhong. Algorithm for
Localization of Underwater Robot Based on
Regional Treatment in Machine Vision.
Ordnance Industry Automation. 2010,29(11):74-78.
6. Jia Guimin, Wang Xiangiun, Zhang Shihai.
Target Tracking Algorithm Based on
Adaptive Template Update in Complex
Background. Acta Optica Sinica. 2009.3.
7. Xu Yuru. Difficulties and Development Trendof Intelligent Water Robot Technology. China
Automation Conference and the summit of thetwo integration. 2009.
8. Zhao Xuechun, Qi Feihu. Automatic
Recognition of Vehicle License Based on
Color Segmentation. Journal of Shanghai
Jiaotong University. 1998.10.
9. Song Xiaowei. Color Information Fusion for
Visual Object Tracking. Xidian University.
2009.1.

Fig.10 Pool test results (lack of light) of Peking
University

It can be seen that the fish can be extracted
without any indoor light source.
Above, this algorithm has been proved to beeffective and can completely replace theprevious foreground extraction algorithm.
5. Conclusion
The main light interference in the image andthe shadows interference of the robot fish iseffectively removed after the original color
space re-modeled, it make segmentation of therobot fish more accurate. Due to theimprovement of the algorithm, the original
matching tracking algorithm is more robust. The
deficiency is that this method only suitable forbackground pure environment. Collision with© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

658

Reinventing the Flavor Wheel
Hiroki Fukushima

Kyushu Women’s University, 1-1 Jiyugaoka, Yahata-nishi-ku
Kitakyushu City, 805-006, Japan

Abstract
In this paper, the author proposes a new type of flavor wheels. The tasting words of former flavor wheels arearranged based on the taxonomical classification of the objects. In this study, the author proposes the new structureof flavor wheels: a) arrange tasting words based on the co-occurrence relationships, and b) not a hierarchicalstructure, but the network structure.
Keywords: flavor wheel, taste, odor, sake, luxury grocery items

1. Introduction
1.1. General Flavor Wheels
A flavor wheel, or a fragrance circle is a chart that isused in the domain of evaluating alcohol drinks or otherluxury grocery items. There are various patterns offlavor wheels in the various domain, such as wine,
coffee, whiskey, beer, and so on. The flavor wheel for
chocolate shown in Figure. 1 is a typical example offlavor wheels, that has the hierarchical structure of theexpression words.
As for Japanese sake, a flavor wheel based on theclassifications of scientific flavor components waspresented recently(Utsunomiya, 2012) (Figure. 2).
1.2. Drawbacks of the Traditional Flavor Wheels
The author points out mainly two drawbacks of ordinalflavor wheel: a) the lack of the relationships among thewords and genres. And b) the hierarchical structure does
not reflect the cognitive process of our representation of
the tastes.
This study aims to overcome these two drawbacks andpropose a new type of flavor wheel.

Figure. 1
An example of a general form of flavor wheels. This
wheel is made by Chocolopolis – a famous chocolatecompany in Seattle. Various expression-words forflavors in chocolate is arranged in the outer circle, andabstract genres are set as the inner circle.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan659

Hiroki Fukushima

Table 1. Details of Sake Corpus
Tokens
Types
Sentences
Paragraphs (brands of sake)
Average Frequency
Standard Deviation

Details
120,789
6,018
5,582
2,388
10.50
(64.55)2.2. Selecting Basic Tasting Words
From the nouns in the corpus, the author picked up thebasic tasting words. The conditions of the selection are:a) name of concrete foods (or objects) for expressing thetastes; b) the frequency is over three.
Under these conditions, following words are
selected as the tasting words of sake: white cedar, plum,
caramel, cacao, chocolate, liche, grape fruit, lactate,
raw cream, yoghurt, raisin, muscat grape, banana,
vanilla, melon, grape, apple, peach, strawberry, pear,
flower, nectar, truffle, cheese, cream, plum, maple syrup,
candy, mineral, nut, Yubari king melon, hazelnut,almond, milk, cashew, walnut, butter, kaki (persimmon

Figure. 2
A flavor wheel for sake, presented by Utsunomiya(2012). Translated by the author from Japanese to
English for this paper.

1.3. Reinventing the Flavor Wheels
In order to overcome the drawbacks of the traditionalflavor wheels, the author proposes a new format of theflavor wheel, applying natural language processing
technologies.
The features of the new format are: a) arrange
tasting words based on the co-occurrence relationships,
and b) not a hierarchical structure, but the network
structure.
2. Method
2.1. Corpus
In this study, the author used Sake Corpus: the corpus ofJapanese sake tasting expressions. The corpus is basedon books and magazines written entirely in Japanese,and Table 1 summarizes the details of each. Note thatparagraphs refer to the different sake brand descriptions.
The size of the Sake Corpus was 120,789 words.
The corpus is based on the (a) sake-reviewing books
and magazines sold in Japan, and (b) the tasting
comments and expressions provided given by 6 tasters
including the author.

Figure. 3
The co-occurrence network of the basic tasting words
in the sake corpus.
fruit), apricot, pineapple, mango, and cherry.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

660

Reinventing the Flavor Wheel

caramel is used for the same sake in order to express thecomplex flavor of sake.

2.3. Analyzing Co-occurrence networks
Co-occurrence networks are generally used to provide agraphic visualization of potential relationships between
people, organizations, concepts or other entities
represented within written material. The generation andvisualization of co-occurrence networks have become
practical with the advent of electronically stored text
amenable to text mining.
In this study, in order to determine the order andarrangement of the tasting words on the flavor wheel,
the author uses the strength of co-occurrence
relationships among the tasting words.
Based on the Jaccard index, the author made the cooccurrence network. Note that the Jaccard index is thecorrelation index in the corpus content words.
For drawing the co-occurrence networks andcalculating the Jaccard index, the author used KH Coder(Higuchi, 2004) as a coding tool.

The Structure: Inner Circle and Outer Circle
In the new flavor wheel, the tasting words are arrangedon the double layer. The words of the inner circle areprototype members of the category. And the words ofouter circle are peripheral members of the category.
The prototype member is determined based on thenumbers of the links each word has in the co-occurrence
network. More links a word has, more typical the word
would be in the category.
3. Result
As a result, the author produced the reinvented Flavor
Wheel for sake, as shown in Figure. 4.

2.4. Arranging the Basic Tasting Words
In this section, the author presents the new type offlavor wheel with the basic tasting words and the cooccurrence network.
Clusters
In the former style of the flavor wheels, the clusters of
the expression words are determined based on thehierarchical structure of scientific features or thetaxonomical classifications.
In this study, the author tries to generate the newstyle of flavor wheel, in which the words and theclusters are arranged based on the relationships of cooccurrence networks.
Thus, in the new flavor wheel, the cluster reflects
not the taxonomical classifications, but the relationshipsof the words in the reviewing sentence. This means that
this study stresses the importance of human expressions
rather than scientific measurement.
The composition of the word clusters may differ
from our knowledge: the cream and raw cream are set
in a separate category, on the other hand, ume-plum andcaramel are set in the same category. These
categorizations would never occur as long as we accept
the taxonomical classifications. However, in the realcontext of the tasting reviews, ume-plum, and the

Figure. 4
Newly reinvented Flavor Wheel of Sake. This flavor
wheel is based on the sake tasting expression corpus(120,000 words in Japanese). The category (cluster) andthe arrangement of the words are determined by cooccurrence relationships (Jaccard index). Inner words
are prototype members of the category, and the outerwords are peripheral members of the category.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

661

Hiroki Fukushima

Reference

Categories

1. Fukushima, H. (2018). A Phenomenological Modelfor Generating the Tasting Description of Japanese Sake.
In T. Ogata (Ed.), Content Generation ThroughNarrative Communication and Simulation. IGI Global.

The flavor wheel consists of nine clusters, but it would
be almost nonsense to give an abstract name to eachcategory. The category is the “result” of the expressions,and only we have to do is not naming it but considering
the hidden relationships and guessing why the twowords are arranged side by side on the base of our senseof taste.

2. Fukushima, H., Imai, M., & 2Tanaka, S. (2017). The
Usage Mechanism of Japanese Ideophones in the
Description of Taste: Morphological and co-occurrence
analysis of the description of wines and sakes.
International Journal of Computational Linguistics
Research, 8(3).

The Composition of the Categories
As same as the distance and connections of the words,
the composition of the nine categories reflects the
distance of the usage tendency (co-occurrence
frequency).

3. Higuchi, K. (2004). Quantitative Analysis of Textual
Data : Differentiation and Coordination of Two
Approaches. Sociological Theory and Methods, 19(1),101–115. http://doi.org/10.11218/ojjams.19.101
4. Tanaka, S., & Fukaya, M. (1998). <Imizukeron> no
Tenkai (The Evolvement of the <Sense-Making
Theory>). Kinokuniya.

4. Discussions
Our sensory domain of taste or odor is not the carbon
copy of the sensor machine. And the taxonomicalclassifications and the perceptual category of taste do
not share the same structure.
The taxonomical classifications and the category of
the natural language have the hierarchical structure.
This would be because the natural language is a vision
and auditory-dominant system (Fukushima, Imai, &Tanaka, 2017).
The author has argued that the cognitive system of
the taste is not the hierarchical process from basic levelto upper/lower level, but the gradational categorizationon the plain surface (Fukushima, 2018). The nouns of
the natural languages have the hierarchical concept
structure. On the other hand, verbs, adjectives, andadverbs have the gradational category (Tanaka &Fukaya, 1998). Thus, the author presumes that thecategory of the sensory domain of taste is the adjective
category. From this viewpoint, the supporting tools forexpressing the sense of taste should be focused on
adjectives rather than nouns. The future development of
the flavor wheel would be the reflection of the cooccurrence relationships between the tasting words andadjectives.

5. Utsunomiya, H. (2012). Flavor Wheel. Kagaku To
Seibutsu (Chemistry and Biology), 50(12), 897–903.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

662

Acquiring Short Scripts and Setting a Case Frame in Each Acquired Script:
Toward Random Story Generation
Jumpei Ono

Graduate School of Software and Information Science, Iwate Prefectural University
Takizawa, Iwate, 020-0693, Japan
Takashi Ogata

Faculty of Software and Information Science, Iwate Prefectural University
Takizawa, Iwate, 020-0693, Japan
E-mail: g236m001@s.iwate-pu.ac.jp, t-ogata@iwate-pu.ac.jp

Abstract
The integrated narrative generation system, that the authors have developed, generates a story to translate the story
into the surface representation. In the story generation process, the INGS uses narrative knowledge that was
automatically acquired from existing narrative works. This paper presents a method to acquire short scripts, which
are a kind of narrative knowledge, from existing works in Aozora Bunko for the story generation. This paper presents
a mechanism to generate random story-like event sequences by using 23,751,142 bi-gram scripts acquired based on
the method proposed below. The authors aim to use the scripts generated by the method as a first set to be revised
through the next learning process.
Keywords: Case frame, Integrated narrative generation system, Knowledge acquisition, Random story generation,
Script, Verb concept.

automatically.3 This paper details the attempt to acquire
short scripts based on bigrams acquired from existing
narrative works for story generation.
A script that Schank has mentioned involves
knowledge about human actions in a particular situation.4
For example, a restaurant script describes the procedure
procedures for ordering and eating dinner at a restaurant.
We attempt to not only acquire Schank’s scripts but toalso acquire the temporal sequences of events.
The INGS consists of one part mechanisms and one
part knowledge. Knowledge refers to dictionaries forconcepts and language notation1, and to knowledge basesthat build partial structures of a story (e.g., the narrative
content knowledge base). By using this knowledge, themechanisms generate various aspects in a narrative, such

1. Introduction
The integrated narrative generation system (INGS)1 is asystem that generates a narrative automatically. The
system not only generates a story but translates the story
into the surface representation. In story generation, the
INGS uses knowledge about story structure anddictionaries2 that systematically store nouns, verbs, andso on. The authors call story content knowledge, whichis stored knowledge bases of the INGS. The dictionaries
are called conceptual dictionaries. A variety of generated
stories depend on the scale of the knowledge base. If theknowledge base is small scale, story generation by the
INGS fails frequently. However, expanding theknowledge base in manually has limitations. Therefore,
the authors have attempted to acquire knowledge© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan663

Jumpei Ono, Takashi Ogata

as a story, discourse, and surface representation (i.e.,
sentences, music, or images).

Fig. 3 shows an example of automatically-acquired
knowledge. This knowledge is applied to an event that
includes “eat,” and the story technique adds an event that
includes “drink” to the structure of the story. (&v age1)or (&v obj1) are variables described by the cases. Thesevariables denote the position or location where thegenerated character, thing, or location is inserted basedon the restrictions. Cases that have the same variables
also have the same character, thing, or location.

2. A Method for Story Generation in the INGS
Fig.1 shows a story structure that is generated in the
INGS. The structure has a hierarchical structure thatconsists of states, events, and relations. A state describes
static information regarding characters, things, locations,and times in a story. An event describes dynamic
information that represents the difference between two
states. A relation links several events by somerelationship such as Causal relationship.

(script (食べる 2[eat] (食べる 2[eat] 飲む
2[drink]))((event 食べる 2[eat] (agent (&v age1))(counter-agent (&v age2)))((1 (event 食 べ る 2[eat] (agent (&vage1))(counter-agent (&v age2))))(2 (event 飲 む 2[drink] (agent (&vage1)) (object (&v obj1))))) ))Relation1

Relation2

Relation3

Fig. 3. Example of a short script.
Event1State
1

Event2State
2

Event3

State
3

Potential selection points in story generation involve
mainly the “application point of the story technique,”“applying a story technique,” and “using knowledge.”
The “application point of story technique” and “applying
a story technique” steps can be controlled based on aparameter that is inputted into the INGS. For example, if“parameter length” is increased, a story technique isselected that increases the number of events within thestory.

Event4

State
4

State
5

Fig. 1. An example of a story structure.

Story generation in the INGS is the expansion andtransformation of a story structure using story techniques.
A story technique is a formal procedure for expanding thestructure of a story and draws on a knowledge base that
corresponds to the particular technique. Fig. 2 shows anexample of using a story technique; the technique selects
applicable knowledge and expands the structure of thestory by using that knowledge.

3. Acquiring Short Scripts from Existing Works
The mechanism for acquiring short scripts has threesteps: acquiring bigrams, making short scripts, andsetting a case frame. This section explains each of thethree steps.
3.1. Acquiring bigrams from existing text

$Cause-effect$Sequence $Sequence
E1 E2 E3 E4

E5Story technique($Cause-effect)

Bigrams that consist of two verbs are acquired byapplying morphological analysis to existing text. The
acquired bigram is called a verb bigram. For example,
verb bigrams are acquired as shown in Fig. 4. The
procedure for acquiring bigrams is as follows:$Cause-effect
E5

E6$Cause-effect$Sequence $Sequence

substitution(i) Morphological analysis: The mechanism analyzes atext using a morphological analyzer.
(ii) Extracting verbs from results: The mechanism
extracts verbs from the results of morphologicalanalysis.

E1 E2 E3 E4 $Cause-effect
E5

E6

Fig. 2. A method for expanding a story structure (Fig. 8 in Ref.
1).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan664

Acquiring short scripts and(iii) Making bigrams: The mechanism creates pairs ofverbs in the order of the results of extraction. For
example, if the result of extraction is (A B C), thenthe mechanism combines two pairs of verbs (A B)
and (B C). N − 1 verb bigrams are created from N
verbs.
(iv) Removing a part of the bigram: The mechanism
removes bigrams that include verbs that are notstored in the conceptual dictionary, because the
INGS cannot use such bigrams in story generation.

3.3. Setting a case frame
By setting a case frame in a short script, the INGS cangenerate a story using the short script. Fig. 6 shows theprocedure for setting the case frame in short scripts (XnYk). A case frame consists of cases that require a verb.
For example, a verb concept has a description as seen in
Fig. 7. Here, the elements of the “case-frame” are the caseframe. A case frame has restrictions based on the noun
conceptual dictionary. The restrictions define theprobability of cases that are included in a verb concept.

<Target text>
男がウィスキーを運ぶと、女はパスタを食べていた。
彼はウィスキーを机に置き、椅子に座った。
[When a man carried whisky, a woman was eating pasta. He
puts the whisky on the table and sits down on the chair.]<Acquired bi-grams>(運ぶ 食べる) (食べる 置く) (置く 座る)[(carry eat) (eat put) (put sit)]

Start
i=1
Targeting i-th case in
Xn

Fig. 4. Example of acquired bigrams.

Giving a not usedvariable to targetcase

3.2. Making short scripts
The acquired verb bigrams are made into short scripts.
This step connects the verb concepts to verbs that areincluded in verb bigrams. Verb concepts are assigned anumber in the INGS because the INGS provides theplural of the verb. For example, the verb concept “食べる 1” has a meaning like “earn,” while the verb concept“食べる 2” has a meaning like “eat.”
Short scripts (Xn Yk) were created from verb bigram(X Y) ((Xn Yk) consists of verb concepts Xn and Yk; (XY) consists of verb X and verb Y). In this case, verb X
has i kinds of meaning, verb Yk has j kinds of meaning,
and the number of short scripts is equal to i multiplied by
j. In making short scripts, we need to consider themeaning of a verb. However, in this paper, the authors
created all patterns of (Xn Yk) with the procedure shownin Fig. 5.

N
i = i +1

X1

X2

・・・

Y
i=1

Targeting i-th case in
Yk

Y
Giving a same
variable to target
case

Is therecase in
Xn ?

N

Giving a not usedvariable to target
case
Last casein Yk?

Verb concept “X”
Verb “X”

Last casein Xn?

YN

Xn

i = i +1
Verb “Y”

Y1

Y2

・・・

Yk

Verb concept “Y”

EndFig. 5. An image of making short scripts by a bigram.

Fig. 6. A flowchart for setting a case frame.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan665

Jumpei Ono, Takashi Ogata((name 食べる 2[eat])(sentence-pattern “N1 が N2 を 食べる”[“N1 eat N2”]))(case-cons-set((case-frame((agent N1) (counter-agent N2) (location N3) (object nil) (instrument nil) (fromnil) (to nil) (adverb nil) (possessive nil) (situation nil) (purpose nil)(experiencer nil) (source nil) (idiom nil) (information nil) (as nil)))(constraint((人[human] -死人[corpse] -人間〈人称〉[human<person>]-準人間[semi-human])(鯨[whale] 馬[horse] 牛[cattle] 豚[hog] 山羊[goat] 羊[lamb] 鹿[deer] 猪[wild boar]
兎[rabbit] 家禽[poultry] 猟鳥[game bird] 魚[fish] たこ・いか・えび・かに[octopus/squid/shrimp/crab])(場所[locaiton] -交通路[traffic route] -公共施設{その他}[public building{etcetera}]- 地 域 [local] - 崖 [cliff] 山 { 部 分 }[mountain{parts}] 平 地 [flatland] 岸[bank]))))((case-frame((agent N1) (counter-agent nil) (location N3) (object N2) (instrument nil) (fromnil) (to nil) (adverb nil) (possessive nil) (situation nil) (purpose nil)(experiencer nil) (source nil) (idiom nil) (information nil) (as nil)))(constraint((人[human] -死人[corpse] -人間〈人称〉[human<person>]-準人間[semi-human])(食料[food] -調味料[flavoring] -飲物・たばこ[drink/cigarette] –汁[soup])(場所[locaiton] -交通路[traffic route] -公共施設{その他}[public building{etcetera}]- 地 域 [local] - 崖 [cliff] 山 { 部 分 }[mountain{parts}] 平 地 [flatland] 岸[bank])))))(is-a (v 身体動作[physical action]))))
Fig. 7. Example of a verb concept.

4. Result

5. Using Short Scripts for Story Generation

Please provide a shortened running head (not more thanfour words, each starting with a Capital) for the title ofyour paper. This will appear with page numbers on the
top right-hand side of your paper on odd pages. In thispaper, the authors acquired short scripts from 13,331
works in the Aozora Bunko.5 The acquisition mechanism
used MeCab in the morphological analysis. We acquired
short scripts from each work. In analyzing each work, we
did not consider structures such as paragraphs in eachwork. Table. 1 shows the number of acquired verb
bigrams and short scripts. (MeCab was downloaded from“http://taku910.github.io/mecab/”)The INGS uses a technique that allows the generated
story to converge on one result based on several
parameters that are set beforehand. Specifically, the
INGS performs story generation by using selection rules
at several points of the generation process. The authors
have adopted random generation in order to graduallychange a narrative. A summary regarding random
narrative generation is presented in Ref. 6. In this section,a generated story is presented.
Currently, parameters have not been applied by theauthors in order to evaluate whether acquired knowledge
alone suffices for random story generation. This method
accomplishes generation by applying various story
techniques that INGS has available, and uses scripts thatwere acquired automatically which are stored
hierarchically, based on verbs. When the INGS selects anevent that randomly, and if the event can be expanded byany story technique, the INGS selects a story technique

Table 1. Amount of Short scripts.
Type

AmountVerb bigram

552,773

Short script

23,751,142© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

666

Acquiring short scripts andin those story techniques randomly. However, if the event
cannot be expanded by any story technique, the INGS
then selects another event randomly. Fig. 8 shows how agenerated story is transformed into a natural language
sentence by a sentence generation mechanism in the
INGS. Although the authors have experience with the
topic of random generation, previous attempts to generate
a story through random methods failed owing to the lackof a knowledge base.

6. Conclusion
This paper showed short script acquisition from existing
works. The authors acquired 23,751,142 short scripts
from existing works that were stored the Aozora Bunko.
The acquired short scripts were used for story generationin the INGS. In future works, the authors will acquire
longer scripts. In addition, they will attempt to acquire
scripts semantically, because the results in this paper did
not consider the semantic structure of a story.
The authors’ goal is the random generation of a story.
We presented story generation by using knowledge,
which was obtained automatically. Based on thediscussion above, a method to change narratives and toset parameters for a narrative generation was presented.
For future research, the aim is to increase automationby using learning mechanisms. For example, we propose
a generation strategy composed of a preliminary story
generation, a valuation of the story, and elements
conjecture where operating parameters are not input
manually, and instead are selected automatically.
Through the learning operations at some selection points,
generation results improve using a bottom-up process. A
second option would be the construction of a system
although maintaining the consistency of a story in anarrative generation process is very difficult, andmaintaining consistency would be our goal.

若主人は溝で酩酊しました。若主人は溝で粛清の夢を見ました。特急が溝で滑りました。若主人は溝でわれに帰りました。引力がトレーシングペーパーより起こりました。若主人は特急をトレーシングペーパーと反対にしました。若主人はトレーシングペーパーで地政学へ駄目を押しました。若主人はトレーシングペーパーでカメレオ
ンを抜きました。若主人は派出所で酩酊しました。若主人は派出所で派出所を放浪しました。若主人は派出所で酩酊しました。若主人は派出所で酩酊しました。若主人
は派出所で酩酊しました。若主人は派出所で寝ました。
若主人は上で上を放浪しました。若主人は上で専攻に音を上げました。若主人は上でカメレオンを捕獲しました。審理に上で競争力がありました。カメレオンは上で稲刈りに干渉しました。若主人に上でカメレオンは居ました。審理は上で審理との矛盾がありました。若主人は上で審理を発音しました。若主人は上で働きました。若
主人は鉄塔で働きました。若主人は鉄塔で働きました。
若主人は鉄塔で働きました。若主人は鉄塔で居残り手当を得ました。若主人は溝で働きました。若主人は溝で働きました。若主人は溝で居残り手当を得ました。若主人
は縄暖簾でビアを飲みました。[A young host is extremely
drunk in a ditch. The young host dream of liquidation in the ditch.
A super‐express slips by a ditch. The young host comes to theyoung host' self in a ditch. A tracing paper is generated byattraction. The young host reverses the super‐express against aditch. The young host makes sure of the tracing paper. The young
host passes a Chamaeleon. The young host is extremely drunk in
a police box. The young host wanders in a police box. The young
host is extremely drunk in the police box. The young host is
extremely drunk in the police box. The young host is extremely
drunk in the police box. The young host goes to bed in the policebox. The young host wanders through the upper part. The young
host is at the young host's wits' end for a special study. The young
host catches the Chamaeleon. A judge is competitive. The
Chamaeleon interferes in the upper part. The young host has the
Chamaeleon. The judge contradicts the judge. The young hostpronounces the judge. The young host works in the upper part.
The young host works in a pylon. The young host works in thepylon. The young host works in the pylon. The young host earns
in the pylon. The young host work s in the ditch. The young hostworks in the ditch. The young host earns in the ditch. The young
host drinks a beer.]

References
1. T. Ogata, Computational and Cognitive Approaches toNarratology from the Perspective of Narrative Generation,
in T. Ogata & T. Akimoto (eds.), Computational andCognitive Approaches to Narratology (IGI Global,
Pennsylvania, 2016), pp. 1-74.
2. T. Ogata, Building Conceptual Dictionaries for an
Integrated Narrative Generation System, Journal of
Robotics, Networking and Artificial Life, (2015), pp. 270284. doi:10.2991/jrnal.2015.1.4.6
3. T. Arai and T. Ogata, Acquiring and Using Event
Sequences Based on Verbs from Novels, Proc. of The 31st
Annual Conference of the Japanese Society for Artificial
Intelligence, (2017), 1D2-OS-29a-5in2.
4. R. C. Schank, Tell Me a Story: A New Look at Real andArtificial Memory (John Brockman Associates, 1990).
5. Aozora Bunko, http://www.aozora.gr.jp, 1998. (Last
access date, Dec. 2016)
6. T. Ogata, T. Arai and J. Ono, Random Narrative
Generation in an Integrated Narrative Generation System,
in Proc. of Forum on Information Technology, (2016), pp.
133-134.

Fig. 8. Generated sentence using a story structure© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

667

A Method of Naimaze of Narratives Based on Kabuki Analyses and Propp’s Move Techniques
for an Automated Narrative Generation System
Takashi Ogata

Department of Software Informatics, Iwate Prefectural University
Takizawa, Iwate, 020-0693, Japan
E-mail: t-ogata@iwate-pu.ac.jp

Abstract

Naimaze in kabuki has been known as a narrative creation method, or a group of techniques thatcombines components in existing kabuki works or in the narratives from other genres for making a newwork. Various elements, such as stories, plots, characters, and places, can be used as components innaimaze. The author aims to use the naimaze method in automated narrative generation systems that wehave been developing, namely the Integrated Narrative Generation System (INGS) and the Geino
Information System (GIS). In this paper, the author first presents an approach to design the narrativetechniques of naimaze in the INGS (and GIS) by applying the method for combining “moves” asexplained in V. Propp’s narratological study, “morphology of the folktale.” A move by Propp means anarrative macro level unit, or a kind of sequence that is composed of several “functions;” and he showedvarious ways to combine several moves to construct an entire narrative structure. The next section
discusses the many possibilities of naimaze techniques and the implementation of experimental
programs based on various kabuki analyses; and the utilization of the Propp’s move method as apreliminary attempt for naimaze techniques in the INGS and GIS. This paper also outlines the directionof future research for designing and implementing an organized naimaze technique group.
Keywords: Kabuki, Naimaze, Multiple narrative structures, Integrated narrative generation system, Geino
information system.

generation and production through multiple narrative
structures. For example, a narrative’s generation flow
comprises the multiple structures of a story at asemantic level and a plot at a constructive level.
The author considers a kabuki narrative as aparticularly rich object for the multiple narrative
structures model. In the first step of studying kabuki as ameans of narrative generation and production, theauthor has analyzed the multiplicity of kabuki withregard to the following fifteen kinds of elements shownin Fig. 1. This figure also shows the overview of therelationship between the INGS and GIS. For instance,
the element of a “person” in kabuki can be divided into

1. Introduction
The author has been studying kabuki1,2,3,4,5 in order tointroduce the acquired narrative knowledge and a wide
range of techniques into two types of mutually related
narrative generation systems, called the Integrated
Narrative Generation System (INGS)6,7 and the Geino
Information System (GIS).8 In particular, the author has
been studying kabuki from the viewpoint of multiple
narrative structures or the multiple narrative structure
model.6,7 The generation process of a narrative work andthe repetitive production process of several narrative
works can be considered from the viewpoint of© The 2017 International Conference on Artificial Life and Robotics (ICAROB 2017), Jan. 19-22, Seagaia Convention Center, Miyazaki, Japan

668

Takashi Ogata

the following three aspects: the real “person” who lives
in the actual world; the “person” as an actor who has ahistory of performing roles; and the “person” of the
kabuki character that appears on stage. Each person has
their own history or story.

A kabuki play

Scenario
RecordCriticism
etc.

Previous kabuki studies by the author include thefollowing topics:(1)
Preliminary and comprehensive considerations
regarding kabuki and narrative generation(2)
An entire framework of the multiple narrative
structures of kabuki(3)
Studying and considering several topics,
including “person,” “story,” and “tsukushi,” in themultiple narrative structures of kabuki.
(4)
Introducing knowledge acquired from research,
analyses, and considerations of kabuki into ournarrative generation systems (INGS and GIS).

Stage direction
Scenario
production
etc.

Stage
Seating

Text

2. Background

Audience

Process

TheatreReal
incident

Work

Genre

Material
and topic

Story
and plot

PersonActorTime or
era

Place

Style

In (1),10,11,12 the author has studied the history andcharacteristics of kabuki and considered the narrative
methods or techniques for connecting them to ournarrative generation research.
In the next point (2),13,14,15,16 the author haspresented an entire framework of the multiple narrative
structures of kabuki that comprise the multiple
structures among the components, the multiple
structures in each component, and the multiple
structures for synthesizing their multiplicities, in orderto understand and deal with a whole system of kabukifrom the viewpoint of the multiple narrative structuresmodel of kabuki.
The objectives of (3) include deeper analysis andstudy of each component of the above-mentioned
multiple narrative structures and an in-depth
consideration of the multiple connections or
relationships of these kabuki components. In particular,
the author has considered through investigations
regarding “person”17 and “story”.18 However, tsukushi
or zukushi19 is not included in the current list of themultiple narrative structures (Fig. 1). By the term
tsukushi (or zukushi), the author means a thorough, oroften exhaustive, description of a group of objects,
including places such as mountains and cities, objects
such as fish and foods. Naimaze descriptions frequently
appear in Japanese classical literature, such as novels,
stories, and long poems; and geino works, such as
dances, ningyo joruri works, and kabuki works. Gunji20
regarded basho zukushi (the tsukushi of places), such as
michiyuki (a symbolic travel scene), as a type of kabuki
esthetics that is propagated through the Japanese literary
and geino traditions. Gunji discussed basho zukushi inkabuki from the viewpoint of unique arrangements
between spatial and temporal aspects in a narrative,
different from the emakimono style in which thetemporal aspect is superior. The author understands

Fig. 1. Multiple narrative structures of kabuki and INGS
and GIS. [Source: Ogata (2016).8]
Naimaze in kabuki has been known as a narrative
creation method or a group of techniques that combinescomponents in existing kabuki works or in the narratives
from other genres for making a new work. Various
elements, such as stories, plots, characters, and places,
can be used as components in naimaze. The author aims
to use the naimaze method in the INGS and GIS. In thispaper, after the author describes our previous studies inkabuki narrative generation as background information,
the author presents an approach for designing thenarrative techniques of naimaze for the INGS (and GIS)
by applying the method for combining “moves” asexplained in V. Propp’s narratological study,“morphology of the folktale.”9 A move by Propp meansa narrative macro level unit, or a kind of sequence thatis composed of several “functions;” and he showedvarious ways to combine several moves to construct anentire narrative structure. The next section displays
many possibilities of naimaze techniques and theimplementation of experimental programs based on
various kabuki analyses; and the Propp’s move method
is utilized as a preliminary attempt for naimaze
techniques in the INGS and GIS. This paper, further,shows the direction that will be taken by future researchwhen designing and implementing an organizednaimaze technique group in these systems based on our
future kabuki studies.

© The 2017 International Conference on Artificial Life and Robotics (ICAROB 2017), Jan. 19-22, Seagaia Convention Center, Miyazaki, Japan

669

Kabuki Multiple Narrative Structureszukushi or tsukushi as an important element in themultiple narrative structures of kabuki. On the other
hand, it is located in a meta-level element that crosses
fifteen kinds of elements shown in Fig. 1.
The final point (4) aims to introduce knowledge
acquired from research, analyses, and considerations ofkabuki into our narrative generation systems (INGS and
GIS).18 One of the objectives of the study of kabuki’s
narrative generation by the author is to comprehensively
and precisely describe a “kabuki narrative generation
system model.” According to this macro objective, oneof the present goals is to introduce the acquired kabuki
narrative knowledge, methods, and techniques into
mutually related systems, the INGS and GIS.

before the other is.”5
“Two moves may have a common ending.”

6“Sometimes a tale contains two seekers.”

The following descriptions provide the explanations
and texts for three types of examples of naimaze using
Propp’s method. These examples are not the results ofautomatic generation, but human attempts:

3. Naimaze: The Usage of Propp’s Move
Method for Mixing Narratives[Example 1: Narukami ＋ Sanmai no Ofuda]
The first example shows that a folktale story,
Sanmai no Ofuda22 is based on the story of akabuki work, Narukami23 using the method 1 inTable 1. In particular, Sanmai no Ofuda (secondpart) is connected to Narukami (first part) and themain characters in Sanmai no Ofuda are changed
to the characters in Narukami.
[Example 2: Kanjinchō in Sanmai no Ofuda]
The second is an example in which, using method
2 in Table 1, the story of Kanjinchō24 is inserted
into the first move by the story of Sanmai no
Ofuda, and the main characters in Kanjinchō,
Benkei and Minamoto no Yoshitsune, appear.
[Example 3: Characters in Sukeroku →Characters of Gogo no Eiko] The final example
deals with the naimaze of characters only. This
example does not use the Propp’s method. In
particular, in the story of Sukeroku25 of kabuki, thecharacters are replaced by the main characters in
Gogo no Eikō26 by Yukio Mishima, the chief,
Fusako, Ryūji, Noboru, and Yoriko.

The author presents a method for simulating naimaze inkabuki based on our previous paper21 related to Propp’s
folktale theory.9 Propp presented a method of
combining stories using several patterns of combining“moves.” In Propp’s narratological theory, the mostbasic narrative units for folktale-like stories are thirtyone kinds of “functions.” A move is composed of theselective sequence of functions and many actual
folktales are combinations of several moves. Table 1
shows examples of the combination patterns of moves
as proposed by Propp. The author’s idea is to implement
the naimaze in kabuki by using Propp’s move method.
Table 1. Move combination patterns [Source: Ogata &Hosaka (2004)21.]

W*

1

W2

“One move directly follows another move.”

2

G
a

W*

[Example 1] The rain has stopped falling and the wholecountry is suffering from a drought. This is because
Narukami Shonin (a priest) holds a grudge against theEmperor for not keeping his promise, and has caged thedragon god of rain behind a waterfall. A beautiful
young woman appears, heading for Shonin’s hermit
cave in the northern mountain. She says that she wants
to wash the clothes of her dead husband in which her
feelings for him linger, but because there is no water,
she has come to the waterfall to find the water to washthe clothes. To say farewell to her husband, the woman
crosses the river toward Shonin, and lifts up the hem ofher skirt to reveal her white legs, and without thinking,“A new move begins before the termination of thefirst one.”3“An episode may also be interrupted in its turn, andin this case fairly complicated schemes may result.”

4

14
2

S∨

9
1

“A tale may begin with two villainies at once, of
which the first one may be liquidated completely© The 2017 International Conference on Artificial Life and Robotics (ICAROB 2017), Jan. 19-22, Seagaia Convention Center, Miyazaki, Japan

670
1

Takashi Ogata

young monk. The young monk goes to the mountain andbecomes so absorbed in collecting chestnuts that he isstill on the mountain when the sun goes down. Then, an
old woman appears and invites the young monk to stayat her home. But during the night, the boy wakes up andsees the woman in her true form of a mountain witch,
sharpening a knife in the kitchen preparing to eat him.
The young monk says “I want to use the toilet,” so themountain witch thinks about it, and ties him up with arope and takes him to the toilet. The young monk
attaches one of the paper charms to a pillar in the toilet,
and says “Pretend to be me and answer the witch,” andhe flees out of the window. The witch asks “Are you
finished?” and the paper charm replies in the voice ofthe young monk, “Just a little longer.” The witch thenasks again, “Are you finished?” and the paper charm
again replies in the voice of the young monk, “Just alittle longer.” Finally the witch cannot stand it any
longer and breaks down the door of the toilet, to findthat the young monk has disappeared without a trace
and there is only the torn paper charm. Realizing she
had been tricked, the witch chases after the young monk.
The young monk, who can see the witch chasing after
him, throws down the second paper charm and says acurse, “Make a big river appear behind me,” and sureenough, a big river appears. But the witch uses her
magic powers to swallow the water in the river. Next,
the young monk uses his final charm and says the curse,“Make a sea of fire appear” and a sea of fire appears.
But the witch uses the water she has swallowed to put
out the sea of fire. The young monk hurries back to thetemple and seeks the help of the priest. The young monkand his party are trying to pass, with the priest taking
the lead, wearing the disguises of yamabushi mountainpriests. The priest says that they are travelling to seek
donations for the rebuilding of the destroyed Todaiji
Temple. However, already the information has beendelivered to the barrier keeper, Saemon Togashi, thatthe young monk and his party are travelling disguised
as mountain priests, and he gives the strict order that nomountain priests are to be allowed to pass. Indignant at
this, the priest and his friends chant an esoteric
Buddhist prayer to Togashi, to attempt to clear his
suspicion. This impresses Togashi and is reminded ofthe previous words of the priest, and he orders him toread the kanjincho (a subscription list of people whohave donated). The priest pulls out a blank scroll that
he happens to have and spontaneously begins to read it,
pretending it is the kanjincho. But Togashi is still
suspicious, and asks him difficult question about the lifeof a yamabushi disciple and about esoteric prayers, but
the priest answers without hesitating. Togashi lets themShonin bends forward toward her. To the suspicious
Narukami Shonin, the woman turns and says that she
wants to become a nun and his disciple. Shonin sends
out his two disciples, Hakuunbô (the white cloud bonze)
and Kokuunbô (the black cloud bonze) to bring her nun
garments, and then the woman suddenly becomes sick.
The surprised Shonin attends to her, and while doing so
he touches her breast. He has never touched a woman
before and he is gradually unable to control his desire
due to the softness of her skin. Seizing her chance, thewoman brings out to drinking cups and gets Shonindrunk, and in his drunkenness he reveals that he keeps
the dragon god imprisoned behind a sacred rope acrossthe waterfall. As Shonin sleeps drunkenly, the woman
creeps away and cuts the rope and releases the dragon
god, and immediately it begins to rain heavily. The
woman was in fact Kumonotaema Hime (Princess), themost beautiful woman at the court, sent by the Emperor
to break the power of Narukami. When he revives andbecomes aware of what happened, his hair stands on
end and he becomes enraged, he turns into flames, hefights and knocks down the disciples who attempt tostop him, and he chases after the princess. The princess,who can see Narukami chasing after her, throws down
the second paper charm and says a curse, “Make a big
river appear behind me,” and sure enough, a big riverappears. But Narukami uses his magic powers toswallow the water in the river. Next, the princess uses
her final charm and says the curse, “Make a sea of fireappear” and a sea of fire appears. But Narukami uses
the water he has swallowed to put out the sea of fire.
The princess hurries back to the temple and seeks thehelp of the priest, who makes her promise to train more
seriously after that, and hides her in a large pot. The
priest begins baking rice cakes in the hearth. Then
Narukami enters the temple and demands that the priest
give him the princess. The priest says that before that,let’s see which one of us is the best at changing shapes.
He asks “Can you make yourself as big as a mountain?”“Yes I can” replies Narukami, and makes himself as big
as a mountain. The priest then asks, “Can you makeyourself as small as bean?” “I can do that too”
Narukami replies, and makes himself as small as a bean.
The priest picks up Narukami, who is now the size of abean, stuffs him in the rice cake he is baking, and eatshim.
[Example 2] Once upon a time there was a temple in avillage, in which lived a young monk and a Buddhist
priest. One day the young monk said to the priest that hewanted to go to the mountain to collect chestnuts. The
priest takes out three charm cards and gives them to the© The 2017 International Conference on Artificial Life and Robotics (ICAROB 2017), Jan. 19-22, Seagaia Convention Center, Miyazaki, Japan

671

Kabuki Multiple Narrative Structures

Tomokirimaru and tries to persuade him to betray Genji.
Sukeroku of course does not agree to this, cuts-down
Ikyuu, retrieves Tomokirimaru, and leaves Yoshiwara.

pass, but still has strong doubts about one of the lowerranked men in the group, who is actually the young
monk. The priest beats his master the young monk withhis pilgrim’s staff, which clears him of suspicion. The
young monk, who had escaped from danger,compliments the priest for his quick thinking, but thepriest bursts into tears and apologizes to his master forthe rude way he had to save his life. Responding to this,the young monk gently takes the priest by the hand andhas him think of the story of the battle in which they
pursued Taira together. Then Togashi appears andapologizes for his previous rudeness and offers themsake to drink. The priest drinks the sake and performs adance. The young monk and his party escape while thepriest dances. The priest puts on his oi (a wooden box
carried on one's back to store items for a pilgrimage)takes his leave of Togashi, and hurries after his master.
Then the mountain witch enters the temple and demands
that the priest give her the young monk. The priest saysthat before that, let’s see which one of us is the best atchanging shapes. He asks “Can you make yourself as
big as a mountain?” “Yes I can” replies the witch, andmakes herself as big as a mountain. The priest then asks,“Can you make yourself as small as bean?” “I can dothat too” the witch replies, and makes herself as small
as a bean. The priest picks up the witch, who is now thesize of a bean, stuffs her in the rice cake he is baking,and eats her.

4. Future Research Directions
In the future, the author will design and implement thenaimaze techniques for narrative generation in the INGS
and GIS in accordance with the following directions:
Positioning of naimaze in the multiple narrative
structures model of kabuki: Naimaze is made tocorrespond to the positioning of the meta-level
concept for fifteen topics of the multiple narrative
structures. Naimaze is not an element that isincluded in the topics and is regarded as a kind ofnarrative technique that, so to speak, utilizes
several elements in the topics in the meta-level andtraverse. For example, although applying anaimaze method or technique to a story or plot is
the most general usage, naimaze is also related tothe element of a “character” as in a “person.” Inparticular, though it is a possibility of naimaze tomix “stories” or “plots” of two or more narratives,
the usage of the naimaze of “stories” or “plots” isrelated to the naimaze of “characters” of different
works. Similarly, introducing “characters” fromdifferent existing works into a new work enables
the associative relationship of the new work to pastworks.
Narrative elements that enable diverse
naimazes: The most basic mechanism of naimaze
is related to, or needs, the consideration of thefollowing aspects: (1) object, (2) basic method ortechnique, and (3) strategic method or technique.
The first necessary step for implementing naimaze
techniques is to make these issues clear at themacro design level. Regarding (1) “object,” in theelements of the multiple narrative structures ofkabuki shown in Fig. 1; “real incident,” “work,”“material and topic,” “story and plot,” “person,”“time or era,” and “place” can be considered theobjects of naimaze in the ordinary sense.
Furthermore, as a possibility, “text,” “genre,”“actor,” and “style” may also become the objects.
Regarding (2), “basic method or technique”
comprises a group of formal methods ortechniques for conducting the naimaze of eachobject. Further, there are naimaze methods ortechniques for simultaneously processing several
objects. At the most formal level, namely methods
or techniques in the program’s implementation[Example 3] Soga no Goro, who appears as a guest
named Hanakawado Sukeroku, goes to Yoshiwara (theentertainment district) to search for Genji’s treasured
sword “Tomokirimaru.” In Yoshiwara, which is a place
where various men gather, he intentionally tries to start
fights with men so that they will draw their swords,
revealing if they have the sword he is searching for.
Agemaki is a courtesan who is frequented by Sukeroku,
and the magnificently bearded Ikyuu also appears with
Agemaki. Sukeroku thinks that Ikyuu might have
Tomokirimaru and tries to get him to draw his sword,
but he is unable to. Sukeroku’s older brother, Soga noJuro, than appears as a seller of white sake, and on
hearing the opinion of his younger brother, he himself
starts trying to pick fights. Soon Agemaki returns to thestage accompanied by a samurai. Sukeroku tries to picka fight with the samurai, but surprisingly that samurai is
his mother, Manko, in disguise, who has come because
she is worried about the two brothers. Manko dresses
Sukeroku in an easily breakable paper kimono andwarns him against a fierce fight, and they return to Juro.
Ikyuu once again appears on the stage. Ikyuu sees thatSukeroku is actually Soga no Goro, draws© The 2017 International Conference on Artificial Life and Robotics (ICAROB 2017), Jan. 19-22, Seagaia Convention Center, Miyazaki, Japan672

Takashi Ogata

level here, functions called narrative techniques,
which expand or transform a story structure or
narrative discourse structure generated by the
INGS, can also be utilized for naimaze. Thismeans that naimaze’s techniques can also belocated as a group of narrative techniques in the
INGS. Additionally, concrete knowledge of kabuki
and the other narratives is required for designingand implementing naimaze methods or techniques.
This kind of knowledge, namely, kabuki-oriented
knowledge, can be stored in the GIS as a part of
the author’s framework of narrative generation.
Naimaze’s methods and techniques can be
designed and implemented through mutuallyrelated narrative generation systems, the INGS and
GIS. Finally, (3) “strategic method” means a kindof control mechanism that enables the effective useof naimaze. In addition to an ordinary plan inwhich previously defined strategic control methods
are used for naimaze generation, another idea of
the author concerning this problem is to prepare adifferent control method that executes random
narrative generation, naimaze in this case, andincrementally narrows down the range ofgenerated narratives through repetition and storageof the evaluation of the results and process27.

Information Science Advancement Research Grant(2017).
References
1. S. L. Leiter, Historical Dictionary of JapaneseTraditional Theatre, 2n’d Edition. (Rowman &Littlefield, Lanham/Boulder/NY/London, UK,2014).
2. T. Imao, Henshin no Shisō [The Theory of
Metamorphosis]. (Hōsei University Press, Tokyo,
Japan, 1970).
3. T. Kawatake, Kabuki: Baroque Fusion of the Arts,trans. Connel Hoff, F. and J. (The International
House of Japan, Tokyo, Japan, 2003).
4. T. Watanabe, Kabuki - Kajo Naru Kigo no Mori
[Kabuki: The Forest of Excessive Signs](Shinyosha, Tokyo, Japan, 1989).
5. T. Watanabe, Kabuki - Kata no Shinzui [Kabuki:
The Essence of Kata] (Kadokawa-GakugeiShuppan, Tokyo, Japan, 2013).
6. T. Ogata, Computational and Cognitive Approaches
to Narratology from the Perspective of Narrative
Generation, in T. Ogata & T. Akimoto (eds.),
Computational and Cognitive Approaches toNarratology (Information Science Reference (IGI
Global), Hershey, PA, USA, 2016), pp. 1-74.
7. T. Ogata & S. Asakawa, Aspects of Content
Generation Through Narrative Communication and
Simulation. in T. Ogata & S. Asakawa (eds.),
Content
Generation
Through
Narrative
Communication and Simulation (Information
Science Reference (IGI Global), Hershey, PA, USA,in press-b).
8. T. Ogata, Kabuki as Multiple Narrative Structures.
in T. Ogata & T. Akimoto (eds.), Computationaland Cognitive Approaches to Narratology(Information Science Reference (IGI Global),
Hershey, PA, US
A, 2016), pp. 391-422.
9. V. Y. Propp, (1968). Morphology of the Folktale,trans. L. Scott. (University of Texas Press, Austin,
TX, USA, 1968). (Original work published 1928).
10. T. Ogata, A Preliminary Consideration of theMultiple Narrative Structures of Kabuki. in Proc. of
the 49th Special Interest Group on Language Sense
Processing Engineering in Japanese Society for
Artificial Intelligence, (2015), pp. 59-64.
11. T. Ogata, Towards Kabuki (1): From Narratives ofArbitrariness and Edition to Narratives ofMultiplicity and Existence. in T. Ogata, Y.
Kawamura, & A. Kanai, in Informational
Narratology:
Artificial
Intelligence/Cognition/Social
Process
and5. Conclusion
The author has been conducting a synthetic study ofnarrative generation systems based on mutually relating
two systems, the INGS and GIS. Naimaze methods ortechniques of kabuki dealt with in this paper will be
introduced into these narrative generation systems in the
future. In this paper, the author proposed several basic
methods of naimaze using our previous research basedon the move connection theory by Propp. Next, theauthor listed several future issues for introducing
naimaze as a group of narrative techniques into the
INGS and GIS. These issues, at the same time, can be
designed and implemented on the basis of the author’s
research results and ideas shown in this paper, including
the multiple narrative structures of kabuki and thenarrative generation through random generation andcontinuous evaluation.
Acknowledgements
This work is supported by the Support Center for
Advanced Telecommunications Technologies Research
Grant (2015-2017) and Kayamori Foundation of© The 2017 International Conference on Artificial Life and Robotics (ICAROB 2017), Jan. 19-22, Seagaia Convention Center, Miyazaki, Japan673

Kabuki Multiple Narrative Structures

22. Sanmai no Ofuda - Wikipedia [The Three Charms:
Wikipedia]. (n.d.). Retrieved from the Japanese
Wikipedia:
https://ja.wikipedia.org/wiki/%E4%B8%89%E6%9
E%9A%E3%81%AE%E3%81%8A%E6%9C%AD
23. Japan Actors’ Association, & The Organization for
the Preservation of Kabuki (2017). Narukami Kabuki Enmoku An’nai [Narukami: Kabuki Play
Guide]. Retrieved October 20, 2017, fromhttp://enmokudb.kabuki.ne.jp/repertoire/502
24. Kanjinchō - Wikipedia [The Subscription List:
Wikipedia]. (n.d.). Retrieved from the Japanese
Wikipedia:
https://ja.wikipedia.org/wiki/%E5%8B%A7%E9%
80%B2%E5%B8%B3
25. Sukeroku - Wikipedia [Sukeroku: Wikipedia]. (n.d.).
Retrieved from the Japanese Wikipedia:
https://ja.wikipedia.org/wiki/%E5%8A%A9%E5%
85%AD
26. Y. Mishsima, Gogo no Eikō (Kōdansha, Tokyo,
Japan, 1963). (trans. S. Mitchell, The Sailor Who
Fell from Grace with the Sea (Knopf, NY, USA,1965).
27. T. Ogata, T. Arai, & J. Ono, Random Narrative
Generation Using an Integrated Narrative
Generation System. in Proc. of the 15th
Information Science and Technology Forum,(2016), pp. 133-134.

Narrative Generation (Hakutoshobo, Tokyo, Japan,
in press-b).
12. T. Ogata & J. Ono, Distribution of Contents andMedia in Narratives: A Preliminarily Considerationof Games, Kabuki, and The Tale of Genji. in Proc.
of the 50th Special Interest Group on Language
Sense Processing Engineering in Japanese Society
for Artificial Intelligence, (2015), pp. 97-115.
13. T. Ogata, (2017). Analyzing Multiple Narrative
Structures of Kabuki Based on the Frameworks ofNarrative Generation Systems, in Proc. of the 2017
International Conference on Artificial Life andRobotics, (2017), pp. 629-634.
14. T. Ogata, On Narrative Multiplicity of Kabuki andthe Other Narratives. in Proc. of the 53th Special
Interest Group on Language Sense Processing
Engineering in Japanese Society for Artificial
Intelligence, (2017b), pp. 19-29.
15. T. Ogata, The Multiple Narrative Structures of
Kabuki and Geino Information System. in Proc. ofthe 31th Annual Conference of the Japanese Society
for Artificial Intelligence, (2017c), 1D2-OS-29a-4.
16. T. Ogata, Towards Kabuki (2): Aspects of Multiple
Narrative Structures. in T. Ogata, Y. Kawmura, &A. Kanai, (eds.), Informational Narratology:
Artificial Intelligence, Cognition and Social
Process, and Narrative Generation (Hakutoshobo,
Tokyo, Japan, in press-b).
17. T. Ogata, The Multiple Narrative Structures of“Person” in Kabuki. in Proc. of the 52th Special
Interest Group on Language Sense Processing
Engineering in Japanese Society for Artificial
Intelligence, (2017a), pp. 113-118.
18. T. Ogata, An Integrated Approach to Narrative
Generation: From Mishima and Kabuki to Narrative
Generation Systems. in T. Ogata & S. Asakawa(eds.), Content Generation Through Narrative
Communication and Simulation, (Information
Science Reference (IGI Global), Hershey, PA, USA,in press-a).
19. T. Ogata, T. (2016c). Narrative Generation of“Zukushi”: Towards Its Systematization ThroughSurveying Kabuki and Developing a Proper Noun
Conceptual Dictionary in INGS. in Proc. of the
30th Annual Conference of the Japanese Society for
Artificial Intelligence, (2016c), 1K5-OS-06b-2.
20. M. Gunji, Kabuki no Bigaku [Aesthetics of Kabuki].
(Engeki Shuppansha, Tokyo, 1963).
21. T. Ogata, & Y. Hosaka, Transformation of Story in
the Story Generation System Based on Narratology,
in Proc. of the 9th International Symposium on
Artificial Life and Robotics Vol.2, (2004), pp. 593596.

© The 2017 International Conference on Artificial Life and Robotics (ICAROB 2017), Jan. 19-22, Seagaia Convention Center, Miyazaki, Japan

674

Narratology goes to Creativity ---Cognitive Content Generation
Akinori Abe*

Division of Behavioral Science, Faculty of Letters, Chiba Univesity, 1-33 Yayoicho, Inageku,
Chiba, 263-8522, Japan†
Dwango Artificial Intelligence Laboratory, Tokyo, Japan
E-mail: ave@ultimaVI.arc.net.my

Abstract
In the previous studies, I discussed the creativity in IMDJ (Innovators Marketplace on Data Jacket). The InnovationGame seems a game where a new production will be obtained during the combination of various techniques, materials
and previous products. Accordingly I discussed IMDJ performance from the perspective of abduction. In addition,
more important matter is the effect of communication among participants which is plays a significant role in creativity.
That is, for creativity, important matters are ``action of a novel relational product'' and ``the role of interaction andcollaboration with other individuals.'' Then I discussed the role of comfortable communication (narratology) by robots
in the creative situation as well as usual situation. In this paper, I will discuss the role of narratology which is basedon the cognitive content generation in the creative task.
Keywords: creative task, Innovators Marketplace on Data Jacket, interaction, .narratology, cognitive content
generation

analyzed the conversation data by the frequency of
evoked laughter in each topic and in all participants. The
reproduced stories presented by the robot created morelaughter than the original stories presented by human.
They concluded that the robot successfully enlivened
group conversation through evoking laughter. The
frequency of laughter will not be connected with thecreativity. However this result show the possibility of theintroducing of a proper robot for the activation of thegeneral meeting/conversation which will cause us to thebetter creativity. Therefore, in this paper, I also discuss
this type of conversation with a robot. Of course theconversation can be discussed in the context ofnarratology or story telling.

1. Introduction
In this paper, I will discuss the role of narratology or story
telling in the creative tasks (contents generation).
Narratology was regarded as a concept in story telling or
generation. For instance, literary works generation can beachieved by the concept of narratology [1,2]. Previously
I discussed the possibility of the literary works generationby computer [1]. Also Rolf Jensen showed theimportance of story in selling products [3]. These can be
regarded as creative tasks and by the story or narrative,
the creativity enhanced in several situations. Thus story
or narratology is very important in creativity. For theconversation activation with a robot, Yamaguchi, Neguri
and Otake showed very interesting results [4]. They

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

675

Akinori Abe

2. Computational literature generation

4.2. Computational literary work generation

Usually several techniques are used in the computational
literary works generation. In fact, one of the special
feature can be used in the computational literary works
generation is the “intertextuality” proposed by J. Kristiva[5]. Kristiva defined intertextuality as follows:
The word's status is thus defined horizontally(the word in the text belongs to both writing
subject and addressee) as well as vertically(the word in the text is oriented towards ananterior or synchronic literary corpus)...
each word (text) is an intersection of words(texts) where at least one other word (text)
can be read... any text is constructed as amosaic of quotations; any text is theabsorption and transformation of another.
This is a very promising feature. Because computers aregood at such as cutting, searching, selecting, andcombination of small materials.

For the computational literary works generation, I
adopted the concept of “intertextuality.” Its feature of
text as a mosaic of quotations construction andabsorption and transformation of another text is easily
achieved by abduction. That is, a randomized generationis quite easyi but not so interesting. A logical generation
can be achieved by abduction. For instance, if the final
goal or result is known, for the explanation of the final
goal or result, several quotations which can be regarded
as hypotheses are abduced. I applied abduction to asimple discourse generation which dealing with humour[9]. In this framework, the structure of the previous
conversation is referred to to generate a new conversation.
In addition, a humourous conversation is generated by
AAR [10] (the extension of Theorist). This type of
application was developed for the activation of
conversation. This procedure can be applied to thenormal conversation as well. Of course an additionalemotion can be added to the conversation by this method.
In addition, the above application is based on situational
inference change. A suitable situation change can leadour thinking in the better direction. Accordingly, if we
use this type of situation change not only for making joke
but also for giving a certain stimulation, innovative
activities can be encouraged.

3. The role of narratology in creativity
Several applications and activities of narratology andstory telling are shown in [6]. In fact narratology andstory telling are for literary works. However, recently
they have been applied to medical interview andtherapies. As shown in the facts in product selling andadvertisements, by the story an additional value can be
added to the products. During the dinner in the Restaurant
Story in London, we can enjoy the story as well as thetaste of the food. The story might be regarded as an
atmosphere, but it can also be regarded as an additional
value. We can say more that without story no product canhave value inside. Thus story or narratology is very
important in creativity. By the story or narratology, thecreativity will be enhanced in several situations.
4.

5.

Emotional robot

In [11], we discussed what type of communication tool
can encourage user's communication and make users feelenjoyable and comfortable. We conducted experiments
by using the following six types of bots:.
1. standard (adjustment) type: many responses (aizuchi)
and introductions to the conversation.
2. conversation introduction type: many introductions tothe conversation are given.
3. response type: many responses are given.
4. non-sympathy type: blame and non-sympathy words
are given.
5. puppet type: imagine the feature of the puppet andspeak according to it.
6. humour type: more playful, and no context is
considered.
For the type-2 robot (conversation introduction type) andthe type-6 robot (humour type), an evaluation as
enjoyable is high. On the other hand, for the type-3 robot(response type) and type-4 robot (non-sympathy type), an
evaluation as boredom is high. From the results, type-2
and type-6 robots will activate and enhance the

Abduction as story generator

4.1. Computational abduction
Peirce defined abduction from a philosophical point ofview as the operation of adopting an explanatory
hypothesis [7]. For the computational abduction, several
systems have been proposed. One of them is Theorist [8].
Theorist is considered as a computational hypotheticalreasoning system. A hypothetical reasoning which is an
explanatory reasoning generates (collects, selects) a
consistent hypothesis set from hypothesis candidates to
explain the given observation. The generated hypothesis
set can be regarded as an answer (solution) which can
explain the observation.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

676

Narratology goes to Creativity.

collaboration with other individuals” are pointed out.
These types of creativity can be achieved by a simple
abduction. Accordingly, the abductive process shown in
the above can include creativity. However, for the more
attractive creativity, a “storytelling” will be necessary. In
addition, the value of products will depend on the story
they tell. Thus narratology is also the important factor for
creativity. It is pointed out that “creative individuals areoften thought of working in isolation, the role ofinteraction and collaboration with other individuals is
also critical.” During collaboration, it is more necessary
to consider the existence of a storyteller. Abduction canhave a role of the collaboration between participants andstorytellers. Thus by abduction a certain story can begenerated as well as a combination of several techniques
to explain a requirement.

communication and creativity. Communication andcreativity might have a strong relationships.
6. IMDJ procedure
Innovation Marketplace on Data Jacket (IMDJ) is called
as Innovation Game. The Innovation Game seems a gamewhere a new production will be obtained during the
combination of various techniques, materials andprevious products. To summarize the game, when arequirement is given, a proposal will be generated
accordingly. Even if the proper techniques are not on thegame board, techniques which were used in the previous
game can be searched and used. In this sense, the strategy
used in this game can be regarded as abduction. Here, I
will show the brief strategy of the Innovation Game. In
the Innovation Game participants do their best to explain
the certain requirement by generating or collecting a setof hypothesis (in this case, a set of techniques) in thecurrent environment. Many creative activities can be
performed as a case of abduction. This type of inference
strategy can be considered as abduction. Of course, if thecurrent environment changes, the inference strategy has
the characteristic of non-monotonic reasoning. A strategy“generating or collecting a set of hypothesis (in this case,a set of techniques) to explain a certain requirement.” canbe formalized as hypothetical reasoning. For theformalization, please see [12]. When we make aninference in IMDJ, we basically search or generatetechniques by abduction and combine the consistenttechniques to produce our proposal that can satisfy therequirements. This type of inference can be regarded as acreative inference. In addition, when we cannot findproper techniques to satisfy the requirements, we perform
an analogical mapping from techniques in other context
to those in the current context. Then we can generate aproper hypothesis set (techniques) to create proposal as a
combination of consistent techniques.

7.2. The narratology in IMDJAn abductive procedure will be performed in the productgeneration (creativity) in IMDJ. Where the possible
products can be generated as explaining the requirement.
In the above, I show the effect of the communication
robots in conversations. According to my definition, forthe creativity, a sort of intuition or special sense will benecessary. An intuition or special sense will emerge withsome stimulus such as information (backgroundknowledge). In emergence, knowledge will function as acatalyst. The catalyst can be obtained during
conversation or listening to the story told by the otherparticipants. That is, the eﬀect of communication among
participants is plays a significant role in creativity. A
communication among users is very important toreorganize or improve their thinking. In addition,
sometimes even a new idea might appear. This type ofcommunication can be discussed from the perspective of
narratology or story telling. As discussed in the previous
section, narratology is a very important concept invaluable activities as well as literary works generation. It
is necessary to consider the effect of narratology evenduring a technical discussion. For instance, for atechnical discussion, there should be a certain story.
Moderators or communication robots (if applicable)
should produce a certain story during the discussion. Ifthey can properly generate or add a story to the technicaldiscussion, participants can produce the better products.

7. The role of narratology in IMDJ
7.1. Creativity
In [13], I defined creativity as follows:
Creative means novel and creative work
comes from nothing or from something
relative but whose relation to the work was
hidden. Actually, for creativity, a sort of
intuition or special sense will be necessary.
An intuition or special sense will emerge withsome stimulus like information (backgroundknowledge). In emergence, knowledge will
work as a catalyst.
For the creativity, the importance of “action of a novel
relational product” and “the role of interaction and

8. Conclusions
In this paper, I mainly discuss the eﬀect of narratology orstory telling in several situations including IMDJ.
Recently, it has been discussed the importance of
narratology or story telling in several applications suchas an advertisement and intellectual tasks. In [12], I
discussed and formalized the role of abduction in the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

677

Akinori Abe

product generation. Abduction can also has the role of thecollaboration between participants and storytellers. Thusby abduction a certain story can be generated as well as a
combination of several techniques to explain arequirement. For the better product generation in IMDJ
the importance of narratology or story telling should beconsidered.
References.

5. Kristeva J., Desire in Language: A Semiotic Approach to
Literature and Art. Columbia University Press (1980).
6. Abe A.: Narratology and Creativity, In Content
Generation through Narrative Communication and
Simulation (Ogata T. and Asakawa S. eds.) (2018). toappear
7. Peirce C. S., Abduction and Induction, chap. 11,
Philosophical Writings of Peirce, Dover (1955)
8. Poole D., Goebel R., and Aleliunas R., Theorist: A Logical
Reasoning System for Defaults and Diagnosis, in The
Knowledge Frontier: Essays in the Representation of
Knowledge (Cercone N.J., McCalla G. Eds.), (1987) p.
331-352, Springer-Verlag
9. Abe A., Applications of Abduction, Proc. of ECAI98
Workshop on Abduction and Induction in AI, pp. 12-19(1998)
10. Abe A., Abductive Analogical Reasoning, Systems andComputers in Japan, 31(1), (2000) pp. 11-19.
11. Abe A. and Hayashi M., On communication via bots forthe computational therapy, Proc. of Int’l Workshop on
Language Sense on Computer in IJCAI2016 (2016) pp.
61-66.
12. Abe A., The Role of Abduction in IMDJ, Proc. of
IJCAI2015 International Workshop on Chance Discovery,
Data Synthesis and Data Market, (2015) pp. 59--64
13. Abe, A., User's interests change as Chance Discovery,
Proc. of KES2002 (2002), pp. 1291-1295.

1. Abe A., The possibility of the Generation of Literary
Works by Computer. In (Ogata T. and Akiomoto T.
eds.), Computational and Cognitive Approaches toNarratology, (2016) pp. 75-89.
2. Ogata T., Computational and Cognitive Approaches
to Narratology from the Perspective of Narrative
Generation in Computational and Cognitive
Approaches to Narratology (Ogata T. and Akiomoto
T. eds.), (2016) pp. 1-74.
3. Jensen R., The Dream Society: How the Coming
Shift from Information to Imagination Will
Transform Your Business. Mcgraw Hill (1999)

4. Yamaguchi K., Nergui M., and Otake M., A Robot
Presenting Reproduced Stories among Older Adults inGroup Conversation. Applied Mechanics and Materials(HJ. Patthi Bin Hussain, Zhu Yuesheng, Murthy Rallapalli
and Jamaluddin Mahmud eds.), Volumes 541–542 (2014)
pp. 1120-1126.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

678

A Framework for Haiku Generation from a Narrative
Takuya Ito
Faculty of Software and Information Science, Iwate Prefectural University,
Sugo 152-52, Takizawa, Iwate, Japan*
Takashi Ogata
Faculty of Software and Information Science, Iwate Prefectural University,
Sugo 152-52, Takizawa, Iwate, Japan
E-mail: g031n019@s.iwate-pu.ac.jp

Abstract
The primary goal of the authors’ research related to the generation of narratives and haikus is to complete
a mutual transformation mechanism between a narrative and a haiku. In this work, the authors propose aframework for transforming a narrative into haiku-like sentences. The paper shows a basic method ofgenerating haiku-like sentences from a fragmented narrative by selecting, arranging, and modifying theelements in the narrative. The authors have been developing an Integrated Narrative Generation System(INGS) that automatically generates narratives. The authors aim to introduce the developed haiku-relating
mechanism into the INGS in the future.
Keywords: Artificial intelligence, Narrative generation, Haiku, Haiku-like sentence, Haiku generation.
have developed an Integrated Narrative Generation
System (INGS)4 that automatically generates narratives.
The authors also aim to use a haiku-generation
mechanism in the INGS. Based on the above backgroundand purposes, the authors have conducted studies related
to the generation of haikus or haiku-like sentences andnarratives. In earlier studies, the authors have presented
the following methods for haiku generation: a method
using the transition patterns of word categories in haikus5,
a method based on appearance frequency and cooccurrence of concepts and words in haikus6, and amethod using deep learning for the transition patterns ofwords and characters in haikus7.
This paper deals with haiku generation from anarrative. The next section proposes a framework to
generate a haiku from a narrative. In section 3, theauthors demonstrate haiku generation using the proposed
framework.

1. Introduction
The authors have earlier presented a method forgenerating a narrative from a haiku.1 By analyzing therelationships between a haiku and the interpreted
sentences of a haiku, these relationships can be used toobtain new interpreted sentences. Conversely, Nitta2
proposed methods to generate a haiku from a narrative.
This method establishes a meta-sentence and coresentences about the narrative.
The primary goal of the authors’ research related to
the generation of narratives and haikus is to complete a
mutual transformation mechanism between a narrative
and a haiku3, a kind of a catchphrase about the narrative.
The authors aim to generate a narrative from a haiku or acatchphrase, as well as a haiku or a catchphrase from anarrative. A haiku is the shortest form of poetry in theworld and expresses a scene or feeling using only
seventeen syllables. The authors consider that a haiku canfunction as a kind of catchphrase for representing theessence or theme of a narrative. In addition, the authors© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan679

Takuya Ito, Takashi Ogata

2. A Haiku Generation Framework from a
Narrative

父が家屋で狼に困る．[A father is embarrassed awolf in his house.]
息子のアランが家屋で父を励ます．[His son, Alain
encourages his father in his house.]
父が「アランが外出する」ことを禁止する．[Thefather forbids Alain from going out.]
アランが家屋で泥を洗う．[Alain washes off mud inhis house.]
アランが牧場で衣服を乾かす．[Alain dries clothes
in a stock farm.]
狼が牧場で羊を襲う．[The wolf attacks sheep in a
stock farm.]
アランが狼を待ち構える．[Alain waits for thewolf.]
アランが狼を襲う．[Alain attacks the wolf.]
アランが狼を殴る．[Alain hits the wolf.]
狼が逃げる．[The wolf escapes.]
アランが狼を逃がす．[Alain misses the wolf]
父がアランを称える．[The father praises Alain.]
Fig. 1. An input narrative.

A haiku is composed of three parts with five syllables,
seven syllables, and five syllables. To develop aframework for generating a haiku or a haiku-like
sentence from a narrative, the authors focus on the factorsof “selection,” “arrangement,” and “modification” of theelements of a narrative. Because haikus generated by the
proposed mechanism do not include kigos words
describing seasons) that are, traditionally, necessary
elements in most haikus, the authors call a haiku
generated by the mechanism a “haiku-like sentence.”
Based on these factors, the authors present the following
algorithm to generate a haiku-like sentence from anarrative.
Selecting a format for a haiku-like sentence. For
generating a haiku according to various methods, thesystem preliminarily prepares several formats andselects one of the formats that suits the purpose.
(ii) Extracting one or more sentences in a narrative,
which is the input information in this system,
according to the condition of the selected method.
These conditions are shown in the examples in thenext section.
(iii) Extracting one or more elements from the sentences
extracted in (ii) and substituting the extracted
elements in the sentence format selected in (i).
(iv) Adjusting the above information according to thestandard haiku form of three parts of five, seven, andfive syllables. If necessary, the system adds one ormore kirejis to the selected elements in the haiku to
be generated. (Kireji is a category of words used toregulate the sounds of a haiku and especiallyindicates emotional emphasis in the psychologicalaspect of linguistic rhetoric.)(i)

3.1. A haiku-like sentence that summarizes acharacter’s action
This method is expected to generate a haiku-like sentencethat expresses an overview or summary of a character’s
action in the narrative. It selects two sentences in the
narrative that includes a character as a designated agent
to generate a haiku-like sentence using the elements inthe selected sentences. The process is explained asfollows.
(i) Selecting a format of haiku-like sentence: In thisexample, the system selects a format, “S V1+O1 V2.”
In this format, S means a designated character (oragent) in the narrative. V1 corresponds to a verb inthe selected first sentence and O1 is the object of V1.
V2 corresponds to a verb in the second sentence thatwas selected.
(ii) Extracting two sentences that have a designatedcharacter in the narrative: For example, according
to a character, “wolf,” “The wolf attacks sheep in a
stock farm.” and “The wolf escapes.” are extracted.
(iii) Extracting the agent, verb, and object cases fromthe sentences selected in (ii) and substituting themwith each element in the format type selected in(i): A character or agent “the wolf,” which isincluded in the sentences selected in (ii), is
substituted with S in the format selected in (i). Then,a verb, “attacks,” which was extracted from the first
sentence selected in (ii), is substituted with V1.
Furthermore, “sheep” is substituted with O1 and“escapes” is substituted with V2. At last, “狼 (theIn the above algorithm, step (i) corresponds to the“arrangement” of narrative information. Steps (ii) and(iii) correspond to the “selection.” Step (iv) corresponds
to “modification.”
3. Generation Processes of a Haiku-like Sentencefrom a Narrative
The authors present the following three types of methodsof haiku-like sentence generation from a narrative using
the above general framework. A story shown in Fig. 1
was used as the input narrative for all the methods.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

680

A Framework for Haiku

with V1 and “escapes” is substituted with V2. Thus,“狼 (the wolf), アラン (Alain), 殴る (hits), 逃げる(escape)” are acquired.
(iv) Adjusting syllable number in each part according
to a haiku format: As “狼 (Ookami)” has foursyllables, it is extended using a one-syllable kireji,“や (ya).” Similarly, the one-syllable kireji, “や (ya)”is added to “アラン殴る (Alain Naguru)” that has
six syllables. For “逃げる (Nigeru)” that has threesyllables, a two-syllable kireji, “か な (kana)”, isadded. Finally, the following haiku-like sentence iscreated.

wolf), 襲 う (attacks), 羊 を (sheep), 逃 げ る(escapes)” are acquired.
(iv) Adjusting syllable number in each part according
to a haiku format: For “狼 (Ookami)” that has foursyllables, a one-syllable kireji, “や (ya),” is added.
Similarly, “ 襲 う 羊 (Osou hitsuji)” that has six
syllables is extended with a one-syllable kireji, “や(ya).” Further, as “ 逃 げ る (Nigeru)” has threesyllables, a two-syllable kireji, “か な (kana),” isadded. Finally, the following haiku-like sentence iscreated.
Ookami ya / Osou Hitsuji ya / Nigeru kana
狼や 襲う羊や 逃げるかな[The Wolf / Attacks Sheep / (Who) Escapes.]

Ookami ya / Alain Naguru ya / Nigeru kana
狼や アラン殴るや 逃げるかな[The Wolf / (Who is) Hit by Alain / Escapes.]

3.2. A haiku-like sentence that expresses a scene inwhich a character appears

3.3. A haiku-like sentence that focuses on alocationIn the next method, the authors expect to generate ahaiku-like sentence that describes a scene in a narrative.
This method extracts a sentence that has the character as
a designated agent. Further, it selects the sentence before
or after and generates a haiku-like sentence using theelements of the selected sentences. The detailedalgorithm is described here.

This method enables to generate a haiku-like sentencethat focuses on a location in a narrative according to thefollowing process.
(i) Selecting a format of haiku-like sentence: The
system selects a format, “S1+V1 S2+V2 L.” S1
corresponds to a character (or agent) in the first
sentence selected and V1 is a verb that shows S1’s
action. S2 is a character in the second sentence andV2 is a verb that shows the action by S2. L is a word
representing a location or place.
(ii) Extracting sentences that have a word
representing a location or place and selecting two
sentences from them: When “his house” is defined,
the system extracts the sentences containing “hishouse.” The sentences “A father is embarrassed awolf in his house.;” “His son, Alain encourages hisfather in his house.;” and “Alain washes off mud inhis house.” are extracted. In this example, “A father
is embarrassed a wolf in his house.” and “Alain
washes off mud in his house.” are selected.
(iii) Extracting the agent, verb, and object cases anda word representing a designated location fromthe sentences selected in (ii) and substituting themwith each element of the format type selected in(i): An agent, “A father,” which is included in thesentences selected in (ii), is substituted with S1 in theformat selected in (i). Then, a verb unit, “isembarrassed,” which is extracted from the first
sentence selected in (ii), is substituted with V1.
Furthermore, “Alain” is substituted with S2 and(i) Selecting a format of haiku-like sentence: In thisexample, the system selects a format, “S1 S2+V1
V2.” S1 is a designated character (or agent) in the
narrative. S2 is a character who appears in a sentence
before or after the sentence containing S1, and V1 isa verb showing the action of S2. V2 is a verb thatshows S1’s action.
(ii) Extracting a sentence that has a designatedcharacter in the narrative and selecting asentence that is located before or after of theselected sentence: When “the wolf” is selected as acharacter or agent and a sentence, “the wolf escapes.”
is extracted, the system extracts sentences, “Alain
hits the wolf.” and “Alain misses the wolf.” In thisexample, the former is selected.
(iii) Extracting the agent, verb, and object cases fromthe sentences selected in (ii) and substituting themwith each element in the format type selected in(i): A character or agent, “the wolf,” which isincluded in the sentences selected in (ii), is
substituted with S1 in the format selected in (i). Then,the character or agent, “Alain,” which is extracted
from the second sentence selected in (ii), is
substituted with S2. Moreover, “hits” is substituted© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

681
1

Takuya Ito, Takashi Ogata“washes off” is substituted with V2. “his house” issubstituted with L. As a result, “父 (A father), 困る(is embarrassed), アラン (Alain), 洗う (washes off),
家屋 (his house)” are acquired.
(iv) Adjusting syllable number in each part according
to a haiku format: For “アラン洗う (Alain Arau)”
that has six syllables, an one-syllable kireji, “や (ya)”is added. Moreover, “家屋 (Kaoku)” is extended bya two-syllable kireji, “か な (kana).” Finally, thefollowing haiku-like sentence is created.

are extracted from a narrative used as the input and theseelements are arranged and modified according to the
haiku form. Three types of methods were shownaccording to the framework in this paper. Future work
will include various topics, including evaluating thegenerated haiku-like sentences, completing thedevelopment of a mutual transformation mechanism
between narrative and haiku, and applying the results tovarious applications such as advertising narrative
generation.

Chichi Komaru / Alain Arau ya / Kaoku kana
父困る アラン洗うや 家屋かな
[A Father is Embarrassed / Alain Washes / at His
House.]
4.

References
1. T. Itou, T. Arai, and T. Ogata, Narrative Generationfrom Semantic Networks of Haiku, in Proc. of the
31st Annual Conference of the Japanese Society for
Artificial Intelligence, (Nagoya, Japan, 2017), 1D3OS-29b-1in2.
2. Y. Nitta, Interpretation and Generation of Haiku byRegular Expression, IEICE Technical Report,
115(69) (2015) 31-36.
3. T. Ito, and T. Ogata, On Haiku/Narrative Generation,
in Proc. of the 56th Special Interest Group on
Language Sense Processing Engineering, (Morioka,
Japan, 2017), pp. 35-37.
4. T. Ogata, Computational and Cognitive Approaches
to Narratology from the Perspective of Narrative
Generation, in T. Ogata and T. Akimoto (eds.),
Computational and Cognitive Approaches toNarratology (IGI Global, PA, USA, 2016), pp. 1-74.
5. T. Ito, T. Ogata, Analysis of Transition of Part of
Speech in Haikus: Toward Using in Haiku
Generation, in proc. of the 34th Annual Meeting of
the Japanese Cognitive Science Society, (Kanazawa,
Japan, 2017), P2-48.
6. T. Itou, T. Ogata, Haiku Generation Using
Appearance Frequency and Co-occurrence ofConcepts and Words, in Proc. of the 11th
International Conference on Cognitive Science,(Taipei, Taiwan, 2017), Presentation Number:
121.03.
7. K. Igarashi, T. Ito, and T. Ogata, Haiku Generation
Using Deep Learning Based on Words andCharacters, in Proc. of the 56th Special Interest
Group on Language Sense Processing Engineering,(Morioka, Japan, 2017), pp. 33-34.
8. Y. Zhang, J. Ono, and T. Ogata, Single Event andScenario Generation Based on Advertising
Rhetorical Techniques Using the Conceptual
Dictionary in Narrative Generation System, in Proc.
of the 4th IEEE International Conference on DigitalGame and Intelligent Toy Enhanced Learning,(Takamatsu, Japan, 2012), pp. 162-164.

Future Works

A short-term plan is to evaluate the generated haiku-like
sentences or the processes according to specific goals.
These goals are related to “what in a narrative must be
represented through a haiku or haiku-like sentence?”
One of the macro goals is, as described in the firstsection in this paper, to design and implement a mutualtransformation mechanism between narrative and haiku.
Fig. 2 shows the overview. This mechanism also enables
us to use one or more haiku or haiku-like sentences as the
input information to the INGS system. Furthermore, in
the relationships with advertising application systems
that the authors have been developing8, such a circulative
mechanism could contribute to the implementation of a
new advertising method in which both long narrative
explanations and short haiku-like representations canmutually be generated for a product or brand.

Fig. 2. An entire image of mutual transformation between
narrative and haiku.

5. Conclusion
The authors presented a framework to generate a haikulike sentence in which one or more designated elements© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

682
1

r

Cognitive Content Generation for Healthy Ageing
Yuki Hayashi

Graduate School of Science and Engineering, Chiba University, 1-33 Yayoi-cho, Inage-ku, Chiba-shi
Chiba, 261-0004, Japan*
E-mail: yu@chiba-u.jp
Akinori Abe

Faculty of Letters, Chiba University, 1-33 Yayoi-cho, Inage-ku, Chiba-shi
Chiba, 261-0004, Japan / DWANGO Artificial Intelligence Laboratory, Tokyo, Japan †E-mail: ave@ultimaVI.arc.net.my

Abstract
From the viewpoint of preventing dementia, balanced meal, moderate exercise, intellectual activities and social
interaction are important. We considered shopping activities were proper to encourage those factors because theactivities were intellectual in human daily activities. In this paper, we are going to construct Shopping Game anddiscuss how to evaluate the activities in the game from the viewpoint of preventing dementia. In addition, we will
discuss content generation in Shopping Game.
Keywords: Prevention of dementia, Shopping activities, Shopping Game, Content generation.

shopping activities are one of the essential daily activities
for many people. Therefore, in the situation of shopping
activities, we thought we could activate intellectual
activities to prevent dementia. In addition, we have totake psychological stress into consideration. One of thesolutions is gamification. Gamification is a method
which applies some ingredients or ideas in the field ofgame to the other field4. For example, gamification isused for rehabilitation or learning support.
In this paper, we construct Shopping Game and discuss
how to evaluate the activities in the game from theviewpoint of preventing dementia. In addition, we will
discuss content generation in Shopping Game.

1. Introduction
Dementia is a general term for a decline in mental ability
enough to interfere with daily life. Alzheimer's is themost common type of dementia. Otake pointed out
balanced meal, moderate exercise, intellectual activities
and social interaction were important from the viewpointof preventing dementia1.
We considered shopping activities were proper toencourage those factors because the activities wereintellectual in human daily activities. Shopping activities
are one of the daily activities and people should do thatusually by themselves. Some people visit a real grocery
store and the others use online store or delivery service(e.g. COOP). In addition, convenient stores are visited
not only by the young and also by the elderly in Japan, so

2.

Method - Shopping Game

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

683

Yuki Hayashi, Akinori Abe

In many researches about consumer behavior3, theyusually observe consumers in a real grocery shop.
However, this way of observation may be difficult for
observers and for participants. Abe said that “The way ofobservation in my research requires lots of time and labor,
so it is difficult to collect lots of shopping activities data.
3”. Accordingly, we developed Shopping Game.
In this game, user can move a character and walk aroundvirtual store (Fig. 1). Upper part means a basket which
the character has and lower part means a floor in thevirtual store. If user wants a particular item, he/she candrag the picture of item from floor to basket. On the other
hand, if user stops purchasing an item, he/she can dragthe picture of item from basket to floor. There are many
areas in the store, for example, vegetables, bakery or
snacks. When a character approaches a certain area, user
can see information on each item, which information
fundamentally consists of name, amount and price.

choice of milk was caused by a choice of meat. Thisparticipant showed her ability of budget management[In front of beverage area] I see … oh, I want milkX. milk … My family like milk-X. We will notpurchase the others. I know milk-X is more
expensive than the others, but milk-Y is not better
than milk-X … meat … [She gave up to purchasebeef at meat area previously] I chose pork and saved
my money. So I can spend my money on milk!
Fig. 2. The utterances of participant (A)Participant (B) was house-worker and (C) was herdaughter (Fig. 3). They discuss which vegetables aregood for yakisoba. In this case, they could not find anoodle for yakisoba in the end, so they planned topurchase a noodle at the other shop.
B: What’s best for our lunch? … ok, let’s cookyakisoba. First, I need green peppers …
C: Why?? [She dislikes green pepper]
B: I also need a cabbage.
C: How about carrots?
B: Carrots remain in a refrigerator.
B: oh … where is noodle [for yakisoba]?? This shop
doesn’t have one??
B: Now, let’s drop in at Shop-P in our way to home.
[She said to her daughter] Just remember it!
C: Shop-P?
B: Yes, I don’t want to change my plan [to cookyakisoba]. If I do so, I should return all thesevegetables to vegetable area. It’s troublesome!!
Fig. 3. The utterances of Participant (B) and (C)When our participants play the game, they showed some
types of utterances / conversation. We summarized
intellectual activities of our participants into Table. 1.
This game will be used as the following purposes. To
prevent dementia, cognitive functions would be trained.
In the future, evaluation of cognitive functions would bealso added. The game would be useful to activate acommunication. Otake said “The shortage of social
communication is one of the main factor of dementia2”.
So we plan to make a situation where some users cansimultaneously play the game to communicate with eachother.

Fig. 1. The structure of Shopping Game

Participants were 34 people, and 12 of them were houseworkers, 11 were boarders and 11 were home-students.
They can play the game depending on their purposes.
They often told us some comments. So we recorded theiractivities and utterances.
3.

Results

As shown in Fig. 2, participant (A) has a stickle
preference to a specific item (milk-X). You can see thatthe specific item activated utterances. In addition, a© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

684

r

Table. 1. A list of units observed in shopping activities.

4.

generated in oriental medicine has been accepted aroundthe world. This concept means inarticulate zone between
health and disease and encourages healthy life style. So
preventing dementia is the problem not only for theelderly but also for the young or middle-aged people. We
have to enhance our game taking diversity of users into
account.

Discussion - Shopping Game with content
generation

In the future, Shopping Game has to change its contents
automatically depending on its user. Some functions will
be added to the Shopping Game. From the viewpoint ofcognitive training, out of stock and change of price areneeded. In addition, it is good for user to be sometimes
shown some special items with discount or with best
season. Furthermore, other shoppers would influence onuser so some virtual shoppers which move automaticallyare needed in the virtual store.
Recently, healthy ageing has been an important concept
to decrease risk of disease, care independence or any
other troubles. For example, WHO has recommended
healthy lifestyle to younger and middle-aged people. In
addition, a concept “ME-BYO” which was originally

5. Conclusions
In this paper, we proposed Shopping Game for healthy
ageing. It played a role of cognitive training for users andit activated their utterances / conversations. In the future,
we consider it will be also used for evaluation of
cognitive functions so we have to make a criteria forusers’ performance.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

685

Yuki Hayashi, Akinori Abe

References
1. M. Otake, Coimagination method for nursing care,
Chuohoki Publishing CO.,Ltd., (2011).
2. M. Otake, Conversation Support Robot for Promoting
Cognitive Activity of Older Adults, Journal of theJapanese Society for Artificial Intelligence. 29(6) (2014)591-598.
3. S. Abe, Shopping Activities and Information Processing-a
trial of protocol analysis-, Yokohama management study.
4(2) (1984) 33-48.
4. S. Deterding, M. Sicart, L. Nacke, K. Ohara, D. Dixon,
Gamification. using game-design elements in non-gaming
contexts. CHI’11 Extended Abstracts on Human Factors
in Computing Systems. (2011) 2425-2428.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

686

The Influence of Story Writing Worksheets on Art Appreciation
Kotone Tadaki

Graduate School of Humanities and Public Affairs, Chiba University
1-33 Yayoi-cho, Inage-ku, Chiba 263-8522, Japan
Akinori Abe

Graduate School of Humanities and Public Affairs, Chiba University
1-33 Yayoi-cho, Inage-ku, Chiba 263-8522, Japan
E-mail: kotonetadaki@yahoo.co.jp

Abstract
In Japan, many art museums prepare various types of worksheets for visitors. Usually, these worksheets include somekinds of guides to make visitors to understand and appreciate the artwork more. One typical approach is encouraging
to create a story about the artwork. In this paper, we conducted the experiment by using the worksheet to determine
the influence of this kind of worksheets while appreciating abstract paintings.
Keywords: art appreciation, story, worksheet, education

T. Okada and J. Inoue (1991) suggested the difference
between two novice strategies is caused by a lack of
specific forms in abstract paintings. From this point ofview, they conducted the experiment using a survey todetermine whether differences in evaluation andimpression exist between abstract paintings andrepresentational paintings. The research question was“will people use different strategies to evaluate thepainting for each abstract painting and representational
painting?". From their result, they conclude that theartistic evaluation for abstract paintings is highly affectedby the tenderness and the taste and the artistic evaluation
for representational paintings is highly affected by thestrangeness and the impact.
These two papers can be one possible answer for D.
O’hare (1976) which showed un-trained subject (novice)prefer representational paintings than abstract paintings.
K. Ishibashi and T. Okada (2010) tried to overcome thissituation by making novices copying and creating theartwork. K. Ishibashi and T. Okada (2010) pointed outthat representational paintings are easy to understand fora novice because when they appreciate representational

1. Introduction
1.1. Background
For museum educator and teachers, the very simple
question “How do we appreciate the artworks?” has beena major challenge for years. J.A. Schmidt et al (1989)
described novice strategies for understanding paintings
by using verbal protocol. Novice can be defined as people
who have never thought about visual analysis for art. This
research was aiming to build a computer-based education
program for the student. J.A. Schmidt et al (1989)
suggested that it is possible that art instructor or
educator’s teaching strategies did not fit novice strategies.
Novice frequently uses the semantic features or contents(e.g. theme and symbolism) than formal elements (e.g.
lines, shapes and colour) which instructor usually uses
when teaching them to understand the artwork. However,when novice watching the abstract paintings, the strategy
changes. Novice frequently uses the formal element thansemantic features.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan687

Kotone Tadaki, Akinori Abe

paintings they have more background knowledge
available to use. These background knowledges weremade up in education usually in school. K. Ishibashi andT. Okada (2010) call this phenomenon the reality
constraint (写実的制約: translated by Y. Tanaka and S.
Matsumoto (2013)). According to K. Ishibashi and T.
Okada (2010)’s experiment, copying can relax the reality
constraint because by copying the painting, novice getsthe artist’s point of view.
Y. Tanaka and S. Matsumoto (2013) also suggested thepossible solution to relax the reality constraint. Y. Tanaka
and S. Matsumoto (2013) used a commentary to relax theconstraint instead of making novices to copy the painting.
The word “commentary” in this research means a shorttext to explain the artwork which is usually displayedwith artworks in museums. They made two types ofcommentaries which were object commentary and formal
commentary. The object commentary provides
information about the object drawn on the specificpainting. The formal commentary provided information
about style, colour, technique and various technicalaspects of the artwork. When they compare the survey(free description) between two conditions, reading thecommentary with technical information (formal
commentary condition) relaxed the reality constraint.
When reading a formal commentary, novice tried to think
deeply beyond what was drawn on the painting. Also, thedescription of the painting was more diverse in formal
commentary condition.
Two possible solutions to relax the reality constraint wassuggested. K. Ishibashi and T. Okada (2010)’s idea canbe used for developing an effective educational programon museums. Y. Tanaka and S. Matsumoto (2013)’s idea
may possibly develop the quality of commentary.

Therefore, if they need to write a story about the abstract
painting, they possibly create characters and situationfrom the painting. Once they find the characters or an
emotion expressed in the paintings, we can say they madean imaginary outline of the painting to form a characteror emotion.
Our research questions are (1) are elements of theartwork do they used to create a story? (2) If (1) was true,
will story creating worksheet helped to image a form on
the painting? and (3) how story creating worksheetchanges the attitude to the painting?
We should note that the experiment in this paper was
conducted in Japanese. All stimuli used in the experiment
was written in Japanese. Therefore, although procedures
and results of this paper may improve art museums inJapan, it might not fit non-Japanese museums.
2. Worksheets
When we visit a museum and trying to understand thedisplayed item such as the Old Master’s painting, bonesof dinosaurs or ancient accessories, we first look for some
information to help us. Usually, they are installing asmall piece of paper called “caption” (“commentary” on
Y. Tanaka and S. Matsumoto (2013)) next to the item.
Visitors can use captions to obtain information byreading a few sentences written in the caption.
However, the information which can installed as captions
is very limited. For example, by using captions, museum
staffs can inform the exact name of the artwork but
cannot inform how curators understand and appreciate
the artwork.
To overcome this weakness of captions, museums
providing many types of tools. One of the typicalexamples of the tools is the worksheet.
In this research, we focused on worksheets. Worksheets
are the easiest tool to install among educational programs
and captions. Since worksheets are the easiest to install,the content included on the worksheet is flexible. The
equipment we need to create the worksheets is only aprinter and papers. Some museum such as The
Metropolitan Museum of Art１, Guggenheim Museum２
and Yokohama Museum of Art (“The Elegant Other:
Cross-cultural Encounters in Fashion and Art”
Exhibition)３ distributed their worksheets (called “family
guide” in The Metropolitan Museum of Art, “family

1.2. Research Aim
We will conduct the experiment to determine whether thestory making experience using worksheets relax thereality constraint or not.
As T. Okada and J. Inoue (1991) suggested, abstract
paintings lack the forms. Both of two experiments (K.
Ishibashi and T. Okada (2010), Y. Tanaka and S.
Matsumoto (2013)) tried to relax the constraint by
obtaining the artist’s point of view not by filling the lackof forms.
In this paper, we aimed to overcome the lack of forms inabstract paintings by creating imaginary forms. Whencreating a story, a writer needs the character and situation.
１
２
３

The Metropolitan Museum of Art. (2018, December 8). Retrieved from https://www.metmuseum.org/learn/kids-and-families/family-guides
The Solomon R. Guggenheim Foundation. (2018, December 8). Retrieved from https://www.guggenheim.org/for-families
Yokohama Museum of Art. (2017, November 30). Retrieved from http://yokohama.art.museum/education/school/worksheet.html (in Japanese)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

688

The Influence of Story

activity guide” in Guggenheim Museum) on their website.
These uploaded worksheets are available for print andvisitors themselves, visitors’ parent or teacher can print
the worksheet before visiting the exhibition. If themuseum thought their target visitors have enough ability
to use online resources, the equipment they need is only
a computer.
In this section, we will introduce three worksheets withstory creating activity used at museums in Japan.

can make their own Karuta by filling their own rhyme
and pictures. Creating sets of sentences and pictures area similar process with creating a story and adding theillustration to the story. Therefore, we can regard thisKaruta style worksheet as a story creating worksheet.
The second worksheet is a blank newspaper style
worksheet. By using this worksheet, visitors can tell
about the favourite item they found in Tokyo National
Museum. When visitors filling this worksheet, they are
asked to research or image about the history of a specificitem. The worksheet can be separated into four small part.
In each part, visitors asked to fill the blank box. The firstpart (Fig.1 left page’s top-right) is asking the information
about Tokyo National Museum. The second part (Fig.1
left page’s top-left) is about the impression of thefavourite item and history of the item. The Third part(Fig.1 left page’s middle) is to list up the three favourite
things about the item. The Fourth part (Fig.1 left page’s
bottom) is to sketch the item.
The first worksheet encourages to create a story about
what was expressed on the item or the artwork. The
second worksheet encourages to create a story about theobject’s history.

1.1.2. Example1: Main building Tour Guide(Tokyo National Museum)
There are two floors called the Japan gallery in Tokyo
National Museum’s main building. The Japan gallery is
the largest permanent collection exhibition in Tokyo
National Museum showing the Japanese history from
Yayoi period to Meiji period (about B.C.300-A.D.1850).
We can guess that this worksheet was made for school
visitors or elementally school students since there areboxes to write the school name and grade on the handout.
This handout was made of two completely separable
pages. Since there are two floors in Tokyo National
Museum, each page represents each floor. Both of
handouts include the map on the front page andworksheet on the back.

1.1.3. Example2: Family Guide “Shall We
Dance?” (The Metropolitan Museum of Art)
This worksheet is one of 18 worksheets uploaded on The
Metropolitan Museum of Art’s website４. The worksheet
is about three artworks related to dancing. Each page
includes a detailed commentary and fun fact. One ofthese three artworks is a stained-glass window. For thisstained-glass window, the worksheet instructed to create
own story. The worksheet also suggested to eight
sentences which can be used to create story such as“Once upon a time there was a young girl and boy. Theirnames were…and…”, “They went to a costume party tocelebrate the…” or “The party was held at…”.

Fig. 1 Tour guide of Tokyo National Museum

The worksheet about the first floor includes Karuta(traditional playing card) style worksheet. Karuta is a
traditional card game in Japan. One player will read up
the rhyme written on Yomihuda (cards with short
sentences) and the others will touch the corresponding
Ehuda (cards with pictures which show the situation
written in corresponding Yomihuda). On the worksheet,
there is empty frame of Yomihuda and Ehuda and visitors４

1.1.4. Example3: MOMAT Collection Children’s
Self Guide (The National Museum of Modern Art,
Tokyo)The National Museum of Modern Art, Tokyo prepared aset of worksheets for their permanent exhibition. This setof worksheets contains six pages and each page contains
activity for one painting. Therefore, by using thisThe Metropolitan Museum of Art. (2018, December 8). Retrieved from https://www.metmuseum.org/learn/kids-and-families/family-guides© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan689

Kotone Tadaki, Akinori Abe

worksheet, visitors can think deeply about every six
artworks.

3.2. Stimuli
We used 18 artworks as stimuli. six of them are
representational paintings and others are abstract
paintings. Every artwork will be displayed with simple
caption and label. All participants will answer theworksheet and questionnaire during appreciating theartwork.
3.3. Conditions
We will have three conditions for each participant(within-subject factor). The participant will see six
abstract paintings without creating a story (abstract
condition), six representational paintings without
creating a story (representational condition) and six
abstract paintings using a worksheet to create a story(story condition). All participants will follow this order.
If the story creating worksheet helped to image a form on
the painting, the result of story condition will be similar
to representational condition and differ from the result of
abstract condition.

Fig. 2 MOMAT Collection Children’s Self Guide

One page about Ryusei Kishida’s artwork includes thequestion encouraging to create a story. On the painting,there is a brown coloured road crossing the square canvas
bottom to up. The question on the worksheet is “If you
walk up this road, what will you see?” and “You areclimbing up the hill road step by step. You can see blue
sky in front of you.” There is a blank box to write ananswer at the bottom of the page.

References
1. J. A. Schmidt, J. P. McLaughlin and P. Leighten, Novice
Strategies for Understanding Paintings, Applied Cognitive
Psychology. 3(1) (1989) 65-72.
2. T. Okada and J. Inoue, A psychological analysis about theelements of artistic evaluation on viewing paintings, The
educational sciences Journal of the Yokohama National
University. 31 (1991) 45-66.
3. D. O’hare, Individual differences in perceived similarityand preference for visual art: A multidimensional scaling
analysis, Perception & Psychophysics. 20(6) (1976), 445452
4. K. Ishibashi and T. Okada, Facilitating Creative Drawing
by Copying Art Works by Others, Cognitive studies:bulletin of the Japanese Cognitive Science Society. 17(1)(2010), 196-223.
5. Y. Tanaka and S. Matsumoto, Cognitive Constraints and
its Relaxation in Appreciation of Painting, Cognitive
studies. 20(1) (2013), 130-151.

3. Experiment
Our research questions are listed below.
(1) Are elements of the artwork do they used to create astory?
(2) If (1) was true, will story creating worksheet helped
to image a form on the painting?
(3) How story creating worksheet changes the attitude to
the painting?
Since this experiment will be conducted on 23rd
December 2017, we could not report the result on thispaper. Results will be presented at the conference.
3.1. ParticipantsParticipants will be a group of university student
including both of undergraduate and graduate. All of theparticipant do not have an art major which means we canregard them as novices.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

690

Rolling Bearings Fault Diagnosis Method Using EMD Decomposition
and Probabilistic Neural Network
Caixia Gao*, Tong Wu, Ziyi Fu
School of Electrical Engineering and Automation, Henan Polytechnic University
Jiaozuo 454000, China
E-mail: gcx81@126.com, 249419152@qq.com,Fuzy@hpu.edu.cn
http://www.hpu.edu.cn/www/index.html

Abstract
Aiming at the problem that the vibration signal of the early fault is weak. A fault diagnosis method of rolling
bearing combined with empirical mode decomposition (EMD), principal component analysis (PCA) andprobabilistic neural network (PNN) is proposed, in which the energy, kurtosis and skewness of first few IMFs are
extracted as fault feature, the dimension of feature set is reduced by PCA, the new set is put into the PNN to
identify fault recognition. The simulation shows that this method has higher fault diagnosis accuracy.
Keywords: Rolling bearing, fault recognition, empirical modal decomposition, principal component analysis,
probabilistic neural network.

proposed to extract the feature, but the adaptability ofwavelet analysis is not as strong as EMD and EEMD
when dealing with non-stationary signals. It can be seenfrom the above bibliographies that how to better distill
the fault features from non-stationary and noisy bearing
signal is the key to fault diagnosis of rolling bearings.
In the operation of rotating machinery, the vibration
signal usually has non-stationary and nonlinearity
characteristics, so it is difficult to obtain good effect infeature extraction by using traditional Fourier transformas the theoretical basis. And the collected signal isaccompanied by a certain degree of noise. Therefore,
the EMD is used to deal with the vibration signal, andthen the energy, kurtosis and skewness are calculated
for each IMF component to form feature set. After that,
use the principal component analysis to remove theredundant information effectively and reduce thedimension of the feature set on the basis of preserving

1. Introduction
Rolling bearings are common and vulnerable parts inrotating machinery. According to statistics, 30% of therotating machinery faults are caused by the bearing
failure, so the condition of the rolling bearing is closely
related to the operation of the machinery, and it is ofgreat importance to detect and diagnose the fault ofrolling bearings.1
At present, there are many scholars studying thefault diagnosis of rolling bearings. The Ref.1 uses amethod of combining EMD decomposition withsingular value difference spectrum. The Ref.2 uses thefast independent component analysis to extract the fault
feature, but it does not extract the fault information in
depth, and the Ref. 3 uses the wavelet packet de-noising
method combined with LMD to extract fault
information. In Ref.4, discrete wavelet transform isCorresponding author: Caixia Gao (1981- ), female, associate professor, master tutor, fault diagnosis research. Tel (Tel.): 0391-3987580. E-mail:gcx81@126.com.
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan*691

Caixia Gao, Tong Wu, Ziyi Fu

the integrity of the main features of the feature set.
Finally, the feature set of the reduced dimension is put
into the PNN, and the state classification of the rolling
bearing can be got.

Energy： E

i 1(1)
Kurtosis： K

2. Fault Feature Extraction Based on EMD(2)

EMD is a self-adapting time-frequency analysis method,which is different from the wavelet analysis in that isthat there is no basis function system, that is, whenanalyzing the series, the original sequence is divided
into a limited number of intrinsic functions rather thanwith certain basis functions, and each component shows
uniqueness and reflects the information at each uniquetime scale in the signal. The decomposed signal is astationary signal. In essence, EMD is a method to turnnon-stationary signals into stationary ones. The main
calculation procession sees Ref.1.Use EMD todecompose the bearing signal, taking a number of highfrequency components and abandoning the last several
IMF components which are generally low-frequency
noise-based.1
The method of extracting characteristic parametersof signals with multiple characteristic parameters is
improved on the basis of intrinsic mode energy method.
As a single energy method is the overall energy
calculation for each component of IMF, it may lose
some useful fault information for the extracted feature
signal. Therefore, this paper adds two feature quantities,
namely, the kurtosis and skewness of the signal.5
The kurtosis index is a dimensionless parameter
which bears no relationship to the speed, size and loadof the bearing and is particularly sensitive to theimpulse signal, so it is quite suitable for pitting damage
fault of the surface. Generally the normal bearing
kurtosis index is close to 3.
The skewness index reflects the asymmetry of thevibration signal, indicating the degree of deviation of
the center of the signal probability density function fromthe standard normal distribution and reflecting theasymmetry of the signal amplitude distribution
compared with its ideal mean. Except the machinerywith quick-return characteristics, if there is friction or
collision in a certain direction, it will cause theasymmetry of the vibration waveform so that the degreeof bias index increases.
They are calculated as follows:

Skewness： K
(3)

n

X i2

1

n

n 1i 11nn 1i 1( Xi

X )4 /

4(Xi

X )3 /

3

The flow chart of multi-feature extraction is shownin Fig1.

Fig.1. The flow chart 1 of multi-feature extraction.

If taking the first n IMF components after EEMD
decomposition, then the number of dimension of the
final extraction of the feature set is 3n.
3. Principal Component Analysis of Fault
FeatureIn the process of modern production industry, generally
a lot of process variables will be collected to detect andcontrol the process. Principal component analysis is acommonly used analytical method in multivariate
statistical analysis, and there’s a fundamental difference
between this method and the mentioned Fourier system
based method or other time-domain and frequencydomain analysis methods. It is characterized by thesimultaneous processing of multiple dimension
variables data from which finds the hidden statisticalinformation characteristics and can eliminate thecorrelation between different dimensions well, so that a
number of related variables can be turned into a few
independent variables, that is, a new feature set of lessdimensions rather than the original feature set reflects
most of the information that should be originallyavailable. And the principal component analysis can© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan692

Rolling Bearings Fault Diagnosis

eliminate the noise and redundancy in the fault data andimprove the accuracy. The main calculation procession
sees Ref. 6.
4. Probabilistic neural network

Fig.2. Troubleshooting flowchart.

PNN was first proposed by Dr. Specht, and it can get
the Bayesian optimal results when used for patternclassification. It has the following advantages: ① Easy
for training with fast convergence; ② strong faulttolerant characteristics; ③ network nodes are up to thenumber of training samples and the number of patterns
to decide.
The structure of the PNN network is divided into
four layers, namely the input layer, hidden layer, sum oflayers and the output layer. The raw data is processed asa PNN diagnostic model after a series of previous
processing. The main calculation procession sees Ref. 7.

After EMD decomposition of the fault data, alimited number of IMF components and the last residual
amount are generated. Their frequencies are arranged
from high to low, so the low-frequency components are
generally discarded. The first five components are taken
here.
Five IMF components are obtained after EMD foreach group of data, forming five-dimensional data, andthen calculate each IMF component for the energy,
kurtosis, and skewness, getting 15-dimensional data toform a multi-feature data set.
Then, the principal component analysis is performedon the multi feature data sets, and the following
principal component analysis table 1 is obtained

5. The classification of rolling bearing fault
5.1. Description of the experimentTab.1. Part of the main component contribution table.

In order to verify the effectiveness of the diagnostic
method in this paper, the data is taken from the
Department of Electrical Engineering and Computer
Science, Case Western Reserve University, USA.8
The analyzed data is obtained with the motor in 2
horsepower, and there are the inner ring fault data, outerring fault data, rolling element failure data, and normal
bearing data. The above four kinds of data form theoriginal data of this experiment. Select 1024 data as agroup from the original data and 40 groups for eachstate from the above four states, totally 160 groups ofdata, of which 10 groups randomly selected for eachstate as the training group and the remaining 30 groups
as the test group.

Main component
numberContribution
rate %

1

47.5

Cumulative
contribution
rate %
47.5

2
3

24.5
7.6

72.0
79.6

4

4.8

84.4

5
6

4.5
3.9

88.9
92.8

7

3

95.8

In general, the cumulative contribution rate of themain component analysis up to 85% can keep most of
the original signal information. 6As can be seen from theabove table, starting from the 7th main element, thecumulative contribution rate reaches 95.8%, meeting therequirement of dimensionality reduction for the original
data set.

5.2. Feature extraction and principal component
analysis
In this experiment, EMD is used to decompose theabove-mentioned vibration signals, and then each IMF
component is calculated for the energy, kurtosis andskewness. Next, PCA method is used to reduce thedimension, and finally input into PNN for stateclassification. The classification flow chart is as follows.

5.3. Pattern recognition based on PNN
This paper is based on the EMD intrinsic mode energy
method to improve. First, the energy of the first 5 IMF
is calculated and then input into PNN, and the
diagnostic accuracy is 85.33%; and then adds the othertwo features to form a 15-dimensional feature set, enter
the PNN, the diagnostic accuracy is 91%. It can be© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan693

Caixia Gao, Tong Wu, Ziyi Fu

concluded that after adding two features, PNN will
increase the diagnostic accuracy.
Finally, the data analyzed by the principal
component analysis are input to the PNN in turn forpattern recognition.

improve the fault diagnosis accuracy, which proves that
PCA can effectively remove the noise and redundancy
in the vibration signal.
6. Conclusion(i) The feature set of the principal component
analysis has higher accuracy rate than that of the feature
set without main component analysis, which indicatesthat the principal component analysis can eliminate theredundancy and noise in the information to a certain
extent.
(ii) When the principal component analysis is used
to reduce the dimensionality of the data, the higher thecontribution rate is, the higher the accuracy is, but it
remains within a certain range after reaching a certain
limit.
Acknowledgements
Fund Project: National Key Research and Development
Program (2016YFC0600906)

Fig.3. The number of different principal components todiagnose the correct rate.

References
1. C. Zhang, J. J. Chen and Y. L. Xu, A bearing fault
diagnosis method based on EMD and difference spectrum
theory of singular value, J. Vib. Eng. 24(5) (2011) 540545.
2. Q. Yang, L. Wang and Y. C. Liu. FICA-IPNN ensemble
fault diagnosis approach of rolling bearing, J. Electric
Machines and Contr. 18(3) (2014) 74-78.
3. W. Sun, B. S. Xiong and J. P. Huang. Fault diagnosis of arolling bearing using Wavelet packet de-noising and
LMD, Vib. Shock. 31(18) (2012) 153-156.
4. J. Chebil, M. Rairi and N. Bushikhah. Signal analysis ofvibration measurements for condition monitoring of
bearings, Australian J. Basic and Applied Sci. 5(1) (2011)70-78.
5. F. J. Pei, K. L. Bi and M. R. Lv. Fault diagnosis of roller
bearings based on characteristic parameters andprobabilistic neural network, China Mechanical Eng.
25(15) (2014) 2055-2058.
6. Q. Q. Liang，H. Han and X. Y. Cui. Fault diagnosis forrefrigeration system based on PCA-PNN, J. Chem. Ind.
67(3) (2016) 1023-1031.
7. L. X. Yang and Y. L. Zhu. High voltage circuit breaker
fault diagnosis of probabilistic neural network, Power
Syst. Prot. Contr. 45(10) (2015) 63-67.
8. K. A. Loparo ． Bearings Vibration Data set, (Case
Western Reserve University, Cleveland, Ohio,USA, 2003

As can be seen from the figure above, when thenumber of principal components reaches five, thecorrectness rate reaches 95%, and the cumulative
contribution rate at this time is 88.9%. Then as thenumber of principal components increases, thecumulative contribution rate increases. The correct rateof accuracy fluctuates about 95%. Therefore, taking fivemain elements is the final solution, which can ensure theaccuracy and efficiency. The comparison of the threemethods is shown in Table 2.
Tab.2. Comparison of three methods for correct diagnosis.
MethodAccuracy

Energymethod85.33%

Multi - feature
method91%

Multi - feature
PCA
95%

It can be seen that the accuracy of the energy
method is low, and the accuracy is improved effectively
when two features are added. It is thus obvious that the
kurtosis and skewness of vibration signal can be used tomake up the deficiency of energy feature in fault
information completeness. Finally, on the basis ofmulti-feature method, the PCA is processed to further© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan694

Detection of Dangerous Driving Behavior via Fuzzy Inference System
Shangzheng Liu*, Qinghui Zhu

School of Electric and Electrical Engineering, Nanyang Institute of Technology, 80 Changjiang Road
Nanyang, 473004, Henan, P.R.China
Fuzhong Wang

School of Electrical Engineering and Automation, Henan Polytechnic University, 2001 Century Avenue
Jiaozuo, 454003, Henan, P.R.China
www.nyist.edu.cn,www.hpu.edu.cn

Abstract
Aiming at identifying fatigue driving, a new detection scheme for the analysis of fatigue driving was proposed inthis paper. On the basis of a comprehensive analysis of the factors that influenced fatigue driving, the proposedsystem uses driver’s eye’s closure time, the beat frequency of sight line, pupil diameter size as 3 indices. The
proposed system is evaluated, and the results indicate that it is a feasible method for the evaluation on fatigue
driving.
Keywords: fatigue driving, fuzzy inference system, fatigue detection, eye movement.

caused by fatigue driving has been becoming a serious
societal problem in recent years. Therefore, it is of greatsignificance to detect fatigue driving and to develop an
efficient intervention for driving mental fatigue not only
to reduce the social cost of traffic safety but to improve
the health of drivers and passengers.
Fatigue driving detection measures include
subjective evaluation method and objective evaluation
method. The objective method can be categorized into:
1) evaluation of biomedical signals, including pulse rate,
EEG, changes of head position, eye-closure rate, andeyelid movement; 2) evaluation of driver–vehicle data,
including steering angle, throttle/brake input, and speed.
The major drawback of the techniques based on
biomedical signals is that they require directly placing
sensors on the subject’s body.

1. Introduction
Driving safety is influenced by two key factors: 1) the
mental states of the driver 2) the external environment.
Various psychological conditions, e.g., fatigue,
distraction and motion sickness can affect driving safety.
Fatigue driving has a close relation with traffic
accidents such as collision, personal injury and so on.
Investigations show that up to 37% of all vehicle
fatalities involve a fatigue driving or drowsy driving in
the United States1. In Europe, drowsy driving
contributes to one fourth to one third of road accidents2.
Some researchers found that fatigue driving has been
estimated to be involved in 2% to 23% of all crashes3,4.
The degradation in driving performance because of
fatigue accounts for a small, but significant, percent of
highway accidents5. The growing number of accidents*

Corresponding author© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

695

Shangzheng Liu, Qinghui Zhu, Fuzhong Wang

relatively steady while a person is in sober state.
Therefore, through changes of pupil, the driver fatigue
can be effectively identified.

2. Fatigue Driving Detection Method
2.1. Percentage of eyelid closure
The time of eye closure/opening is a good indicator of
fatigue. PERCLOS (Percentage of Eyelid Closure Overthe Pupil Over Time) is defined as percentage of eyesclosing time over a particular time. Studies show that it
is a good indicator of fatigue. Figure (1) shows the
PERCLOS principle, it is calculated as following
formula:=2.3. Percentage of saccade
Humans don’t look at a scene in fixed steadiness;
instead, the eyes move around, locating interesting partsof the scene. When scanning immediate surrounding，
human eyes make jerky saccadic movements and stop
several times, moving quickly between each stop.
Saccades to an unexpected stimulus normally take about
200 milliseconds (ms) to initiate, and then last fromabout 20–200(ms). When human is fatigue, the numberof eye jump declines.

(1)

× 100%
Largest pupil

Pupil
diametert3

t4

Defuzzification

t2

Fuzzy Inference

Percentage
of saccade

20%

t1

Fuzzification

Eye open extent

80%

PERCLOUSt

Fig.1. PERCLOS Principle

The ps is the value of PERCLOS. T1 is the timeneeded that eyes closed totally to 80%. T2 is the timethat eyes closed from 80% to 20%. T3 is the time from
20% closed to 20% open, t4 is the time needed from 20%
open to 80%.
In specific tests there are three measurements: P70,
P80 and EYEMEA(EM). P70 is the time eyelid closure
over pupil more than 70%. P80 is the time eyelid
closure over pupil more than 80%. EM is the meansquare value of the percentage of eye closure speed. The
study shows that P80 can reflect the degree of fatigue
well, so that P80 can be used as a determine indicator in
the evaluation of fatigue driving.
2.2. Pupil diameter
When people are in fatigue and non-fatigue state, thedifference in pupil is obvious. While a person is fatigue,
the pupil diameter reduces significantly, and it isFig.2. Fuzzy Inference Model

3. Fuzzy Fatigue Driving Detection Model
Obviously, alert state and sleep are crisp. But fatigue isvague. Therefore, fuzzy inference system is used to
identify fatigue driving. The fuzzy inference model offatigue driving detection is shown in Fig. 2. The inputs
are the PERCLOS, percentage of saccade and the size of
pupil diameter. The output is the membership of fatigue.
3.1. Fuzzification
The universe of PERCLOS is [0%, 100%]. For
convenience, multiplying scaling factor 100, theuniverse is mapped to [0,100]. Pupil Diameter ischanged from 10 to 50(pixels), the corresponding
scaling factor is 2. Percentage of saccade varies from 0
to 0.005(1/ms), the scaling factor is 20000. The
membership of PERCLOS, pupil diameter andpercentage of saccade are showed in Fig.3, Fig.4 and© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan696

Detection of Dangerous Driving

Fig.5 respectively. The linguistic variables for inputs
output are F (Fatigue), D (Distraction) and N (Normal).

3.2. Rules
The model has 3 inputs. Therefore, we have 27 fuzzy

rules. The rule is as follow:
IF PERCLOS is N, Percentage of saccade is N and pupil
diameter is N, then status is N. The rules are shown in
Table 1.
Table 1. Rule table for fatigue detection
PERCLOSFig.3. Membership function of PERCLOS

N
D
F

pupil

N
N
N
N
N
N
D
N
D
F

N
D
F
N
D
F
N
D
F

saccade
D
N
N
D
N
D
D
D
D
F

F
N
D
F
D
D
F
F
F
F

Fig.4. Membership Function of Percentage Saccades

Fig.5. Membership Function of Pupil DiameterFig.7. Surface of FIS

3.3. Defuzzification
Defuzzification operates on the implied fuzzy sets
produced by the inference mechanism and combines
their effects to provide the “most certain” output. We
consider “center of gravity” (COG) defuzzification
method for combining the recommendations represented
by the implied fuzzy sets from all the rules.
Fig.6 Membership of output

4. Case Analysis
In order to verify the practicality and accuracy of the
proposed model in fatigue driving, this paper takessome simulation in MATLAB. The GUI of experiment© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

697

Shangzheng Liu, Qinghui Zhu, Fuzhong Wang

results is shown in Fig.7. 7 samples are examined6. The
results are shown in Table 2.

5. Conclusions
Nowadays, with the sharp increasing of vehicles, toimprove overall safety is becoming more important. In
this paper, we present a fuzzy inference system forestimating fatigue driving. The results show that the
proposed system can accurately identify the driver’s
fatigue and distraction.

Table 2. Testing results
Data

FIS output Evaluation of state

PERCLOS 42
1 Percentage of saccade 38
Pupil diameter size 27
PERCLOS 51
2 Percentage of saccade 36
Pupil diameter size 28
PERCLOS 63
3 Percentage of saccade 32
Pupil diameter size 25
PERCLOS 72
4 Percentage of saccade 16

32.7

N

50

D

71.4

F

80.4

F

71.4

F

80.4

F

30

N

Acknowledgements
This paper was supported by key technology R & D
program of Henan (Project No.122102210243) and
Basic and frontier projects of Henan (ProjectNo.162300410194).
References
1. H. Summala, and T. Mikkola, Fatal accidents among car
and truck drivers, Human Factors, 36(2) (1994) 315–326.
2. M. Dangelmaier, D. Spath, A. Bekiaris, and C. Marberger,
Effective warning of a drowsy driver - the AWAKE
experience, In Proc. 10th Int. Conf. Human-Computer
Interaction, Theory and Practice(Part II), eds. J.A. Jackoand C. Stephanidis (Grete, Greece, 2003), pp. 43-47.
3. C. Craye and A. Rashwan et al, A multi-modal driver
fatigue and distraction assessment, Int. J. ITS Res. 14(2)(2016) 173-194.
4. J. D. Lal, and A. Craig, A critical review of thepsychophysiology of driver fatigue, Biol. Psychol., 55(3)(2001) 173-194.
5. X. Gao and Y. Zhang et al, Evaluating driving fatigue
detection algorithms using eye tracking glasses, in 7th Annu.
Int. IEEE EMBS Conf. Neural Eng. (Montpellier, France,2015), pp.767–770.
6. Liu Qing and Li Zhi-feng, Evaluation studies on ship
driving fatigue based on BP artificial neural network, in
Proc. 8th Int. Conf. Nat. Comput. (Chongqing, China, 2012),
pp.217–221.

Pupil diameter size 21
PERCLOS 63
5 Percentage of saccade 28
Pupil diameter size 25
PERCLOS 85
6 Percentage of saccade 19
Pupil diameter size 21
PERCLOS 38
7 Percentage of saccade 39
Pupil diameter size 41

The results of test show that the system output agrees
with the expected. Therefore, the evaluation value of the
FIS can approximately be regarded as the value offatigue driving state. The result shows that using fuzzy
inference model to detect fatigue driving is feasible.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

698

Feature points designing and matching for the target spacecraftin the final approaching phase of rendezvous and docking
Wenjing Pei, Yingmin Jia*
The Seventh Research Division and the Center for Information and Control
School of Automation Science and Electrical Engineering, Beihang University (BUAA), 37 Xueyuan Road, Haidian
District
Beijing, 100191, China
E-mail: wenjingpei@buaa.edu.cn; ymjia@buaa.edu.cn
www.buaa.edu.cn

Abstract
In this paper, the images are acquired by CCD, the noise in which is removed by Gaussian Filter. Canny Edge
Detection is used to extract the edges of the target spacecraft, and Progressive Probabilistic Hough Transform(PPHT) is applied to detect the lines. After that, feature points are designed on two solar panels. Thus 48 featurepoints can be obtained. Feature points matches by Scale-invariant feature transform (SIFT). Experimental results
show that our method has better running speed and accuracy.
Keywords: Lines detection, feature points designing and matching, Space-craft rendezvous and docking, the target
spacecraft, solar panels

preserving the images feature. It depends on whether or
not the image features are extracted accurately.
Edges often carry a lot of important information of
the images. Nowadays, many edge detection methods
have been proposed 2-3. Classical edge detection
techniques include Sobel Operator4, Laplacian Operator,and so on 4-5. Canny Operator was put forward in 1986
by John F. Canny 6. Moreover, setting proper upper andlower threshold, we can extract all possible edges. Thus,
the contours of the target space-craft are got.
Numerous methods for lines detection have been
proposed 7-8. Traditional lines detection methods include
Hough Transform, improved Hough Transform and
LSD (Line Segment Detector) 9. However, there arelittle studies for vision systems of space-craft
rendezvous and docking.

1. Introduction
Spacecraft rendezvous and docking is a very complex
and challenging proceeding. It involves the vision
measurement of the chaser spacecraft to the target
spacecraft. The relative velocity, location and attitude
between two space-crafts must be measured accurately
and rapidly, which will directly determine the success or
failure of rendezvous and docking, especially in the
final approaching phase.
As a result of the weak light in space, images
acquired are easier to underexpose. Furthermore, spatial
images often inevitably mix with different levels ofnoise and distortion. In this way, it is necessary tosmooth the obtained images 1. The main purpose ofimages smoothing is to eliminate the noise andCorresponding author© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

*699

Wenjing Pei and Yingmin Jia

In this paper, we propose an algorithm for space
images to detect the lines, design and match featurepoints for images. It will directly affect further
extraction of motion information.

2.1. Image preprocessing
Images preprocessing mainly consists of two steps:
images denoising by Gaussian Filter and Canny Edge
Detector. It is necessary for the obtained images toreduce noise. Gaussian Filter can remove most of thenoise, while Canny Edge Detector can extract the edges
by setting upper threshold and lower threshold.

2. Methodology
There are two solar panels and the fore-end cabin in thetarget spacecraft model, which correspond to the lines
and characteristic circle, as shown in Fig. 1.

2.1.1. Gaussian Filter
Gaussian Filter is a low-pass filter and can attenuate
high frequency signals. Furthermore, Gaussian Filter isa linear filtering method, which can smooth out thenoise by a Gaussian function. It has been widely used in
the field of image processing.
One-dimensional Gaussian function can be
expressed as:
1
G ( x, y )exp( x 2 / 2 2 )(1)2Similarly, two-dimensional Gaussian function canbe expressed as:
1
G( x, y )exp( ( x 2 y 2 ) / 2 2 )(2)2Here, x, y are represented the distance from theorigin to the horizontal and vertical axis respectively; σ2is the variance of the Gaussian distribution. The value ofeach pixel is as a weighted average in this pixel's
neighborhood. It will result in some blur, so it canpreserve edges and boundaries of objects better thanother methods.

Fig.1. the surface of target spacecraft model
Start
Start
Lines
Acquire
SIFT
Preprocessing◇Gaussian Filter◇Canny Edge

Feature points designingand matching

PPHT
End

2.1.2. Canny Edge Detection

Fig.2. the overall process of feature extraction

Canny Edge Detection can extract all possible edges by
setting upper threshold (T1) and lower threshold (T2);
The detail description see in 6.

Fig.2 shows the overall process of the proposed
method for feature extraction. Using CCD imaging
sensor acquired space images. After that, the detection
process starts with preprocessing the images. First, theimages are grayed and denoised by Gaussian Filter.
Then Canny Edge Detector is applied for obtaining all
possible edges of the target spacecraft. Second, PPHT is
utilized to detect the lines. Select all of the correct lines
and compare it with other detection method. Finally,
SIFT is used to feature points designing and matching
and 48 feature points are obtained. The detail process is
as follows.

2.2. Lines detection
PPHT is a modified method on the basis of Hough
Transform (HT). It has obvious advantages, which
doesn’t accumulate all possible points but only a part ofthem are accumulated. In this way, it can reduce thecomputed quantity greatly and shorten calculating time.
Parametric equations of lines are defined (3), where( , ) is the parameter space in the pixel space. Theirrelationship is as shown in Fig. 3.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

700

Feature points designing and

x cos

y sin

original image. Finally, feature points are matched. The
details are as follows.
The two-dimensional image is represented as
I ( x, y ) and its scale space at different scales isL( x, y, ) , WhereL ( x, y , ) G ( x, y , ) I ( x, y )
(4)

(3)

Difference of Gaussian (DOG) Pyramid is used todetect the extreme, the details is as follows:
D ( x, y, ) (G ( x, y, k ) G ( x, y, )) I ( x, y )Fig.3. relationship ( x, y ) and ( , )L ( x, y , k ) L ( x, y , )(5, 6)
2
2
2
1
G ( x, y , )e( x y )/ 2
2 2Here, G ( x, y, ) is Gaussian convolution kernel;( x, y ) is coordinates of pixels and σ2 is the varianceof Gaussian normal distribution. One point is a featurepoint, if it is the extreme point. The gradient magnitude
and orientation are assigned as m( x, y ) and ( x, y ) :m( x, y)The detail proceedings are as follows:①Select a new point randomly as voting accumulator (ρ,θ)), then remove this pixel from the input image.
② If the highest peak ,that is ( , ) is greater than T3(a predefined threshold), go to ③. If not, go to ①.
③Find all lines and select the longest segment that canbe denoted from starting point P0 to ending point P1.
④All the points of the longest line are removed fromthe input image.
⑤Remove all points of the selected lines in ③(P0−P1)
from the accumulator, which means they do not needvoting process.
⑥If the segment is longer than a predefined minimum
length, then take P0−P1 as one of the output results
directly.
⑦Go to ①.

( L( x 1, y ) L( x 1, y))2 ( L( x, y 1) L( x, y 1)) 2( x, y )arctanL( x, y 1) L( x, y 1)
L( x 1, y) L( x 1, y )(7, 8)The same parts of the two images have the same
eigenvectors, so their distance is the nearest. Let
a (a1 , a2 ,..., an ), b (b1 , b2 ,..., bn )aretwoeigenvectors, Uab is denoted as the distance between aand b.
U MIN / U1 R, 0 R 1
n(9, 10)U ab ( (ai2 bi2 ))1/ 2

2.3. Feature point designing and matching
In this paper, we design 48 feature points on two solar
panels of the target spacecraft. There are 24 featurepoints on the left solar panel and the right solar panel
respectively. They consist of three groups with 8 featurepoints which are almost on the same line.
Then, they are both numbered from 0 to 7
symmetrically, as shown in Fig.4.

i 1

UMIN is the nearest distance, and U1 is the next
nearest distance. R is matching threshold. If UMIN /U1<R, we can think it is right matching and vice versa.
3. Experimental Results
In this experimental, we use Shenzhou-7 spacecraftmodel as the target spacecraft. Vision system consists of
two CCD cameras and a computer, which is used aschasing spacecraft. The ratio of the targe spacecraft tothe original size is 1:40. Figure 5 displays two original
images.
Fig.6. shows the results of Canny Edge Detection. In
this processing, the original image is grayed andsmoothed by Gaussian Filter first. Fig.7. shows theresults of detection of lines with PPHT which represent

Fig.4. Feature points designing

Feature point matching means finding out therelationship between feature points adjacent images.
SIFT algorithm is used to match images. Firstly, featurepoints on the left solar panel and the right solar panel
are matched by SIFT algorithm with adjacent frame
respectively. Then, the obtained images merged with the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan701

Wenjing Pei and Yingmin Jia

two solar panels. This method can improve runningspeed greatly. The comparison table is as follows:(a)

we utilize Canny Edge Detection to obtain the possibleedges of the target spacecraft. Then PPHT is used tocomplete the lines detection. Finally, feature points
designing and matching are implemented on the left andthe right solar panel respectively. In this way, 48 featurepoints are obtained on two solar panels. In the future,
we will continue to improve matching algorithm.
Besides, we will study to feature points designing andmatching on the fore-end cabin in the target spacecraft.

(b)
Fig.5. the original image

5. Acknowledgements
This work was supported by the NSFC (61327807,
61521091, 61520106010, 61134005), and the National
Basic Research Program of China (973 Program:
2012CB821200, 2012CB821201).
6. References

Fig.6. Canny Edge Detector

1. D. T. Kuan, A. A Sawchuk, T. C. Strand, Adaptive noisesmoothing filter for images with signal-dependent noise.
IEEE Trans. Pattern Anal. & Machine Intelligence,
PAMI-7(2) (2009) 165-177.
2. M. C. Shin, D. B. Goldgof, K. W. Bowyer and S.
Nikiforou, Comparison of edge detection algorithms
using a structure from motion task, IEEE Trans. Syst.
Man & Cybernetics Part B: Cybernetics, 31(4) (2001)
589-601.
3. D. Demigny, On optimal linear filtering for edge
detection, IEEE Trans. Image Process., 11(7) (2002)
728-737.
4. W. Gao, X. Zhang and L. Yang, An improved sobel edge
detection, IEEE Int. Conf. Comput. Sci. Inf. Technol.
(Chengdu, China, 2010), pp. 67-71.
5. S. Tai and S. Yang, A fast method for image noise
estimation using laplacian operator and adaptive edge
detection, IEEE Int. Symp. on Commun., Contr. Signal
Process., (Malta, 2008), pp. 1077-1081.
6. P. Bao, L. Zhang and X. Wu, Canny edge detection
enhancement by scale multiplication, IEEE Trans.
Pattern Anal. Machine Intelligence, 27(9) (2005) 14851490.
7. H. Ye, G. Shang, A new method based on houghtransform for quick line and circle detection, Int. Conf.
Biomed. Eng. Informatics, (Shenyang, China, 2015), pp.
52-56.
8. R. Grompone, J. Jakubowicz, J. Morel and Gregory
Randall, LSD: a fast line segment detector with a false
detection control, IEEE Trans. Pattern Anal. MachineIntelligence, 32(4) (2010) 722-732.
9. K. Murakami and T. Naruse, High speed line detection
by hough transform in local area, Conf. Pattern Recognit.,(Barcelona, Spain, 2000), pp. 467-470.

Fig.7. the lines detection

Gaussian Filter just needs 67.23ms, while it takes
154.17ms with no Filter. It shows that using Gaussian
Filter can improve running speed greatly. Besides, we
use Visual C++ and OpenCV programming to realize
the lines detection.

Fig.8. Feature points matching

Fig. 8 shows that SIFT algorithm is used to adjacentimages. Feature points matching are implemented on theleft solar panel and the right solar panel respectively.
Then add them to the original image.
4. Conclusion
In the final approaching phase of spacecraft rendezvous
and docking, features extraction accurately and fast
directly determine whether the whole proceeding is
successful or fail. In this paper, the algorithm of lines is
proposed. After smoothing images by Gaussian Filter,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan702

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan703

Construction and Visualization of Atmospheric Corrosion Level Map
Dongmei Fu*, Gaoyuan Wang, Tao Yang, Chao Wu, Mengchen Cui

School of Automation and Electrical Engineering, University of Science and Technology Beijing, 30 Xueyuan Road
Haidian District, Beijing, China
E-mail: fdm_ustb@ustb.edu.cn
www.ustb.edu.cn

Abstract
To solve the problem of how to construct the Chinese atmospheric corrosion level map of carbon steel based on theatmospheric environment and ISO 9223-2012 standard has a major practical value. Specific to the uneven
distribution of data samples, this paper proposes an improved inverse distance weighted method based on density torealize the data interpolation. On the basis of interpolation, the paper constructs the carbon steel atmosphericcorrosion level map of China.
Keywords: atmospheric corrosion; corrosion map; inverse distance weighted.

practical. Fathoni U, et al.4 plotted the atmosphericcorrosion map of Malaysia based on 10-year historical
data using inverse distance weighted interpolation. Ivan
Cole et al.5 plotted nearly 2000 data samples of the
exposed materials using inverse distance weighted
interpolation Vietnam atmospheric corrosion map;
Chico B et al.6 adopted the carbon steel, zinc and othermetal materials corrosion rate data and used the methodof spatial interpolation to depict the atmosphericcorrosion map of Spanish. Around 1990s, the Instituteof metal corrosion and protection, Chinese Academy of
Sciences used spatial interpolation to plot theatmospheric corrosion maps of Shenyang City7 andHainan Province8. Tidblad J et al.9 established thecorrosion rate prediction models of carbon steel andzinc and further plotted the atmospheric corrosion mapsof the future European. Carbon steel is the mostcommonly used metal material for industrial andagricultural production and people's life, but so far,there’s not any carbon steel atmospheric corrosion map

1. Introduction
Atmospheric corrosion is the interaction result of
metallic materials with their surrounding atmosphericenvironment. There are two factors affecting
atmospheric corrosion1: one is the metal itself, the otheris the atmospheric environment. When the material isdetermined, the atmospheric corrosion is only related tothe atmospheric environment. According to ISO92232012 standard2, these factors are the time of wetness,
sulfur dioxide concentration and chloride ion
concentration.
As metal materials will corrode in the atmosphere, it
is of great significance to obtain the corrosion level map
and the corrosion rate map, which accord with thepractical geographical distribution, for materials
selections and anti-corrosion3. Through the atmosphericcorrosion map, the corrosive state of various regions canbe easily checked and an intuitive understanding of theatmospheric corrosivity distribution is provided, making
the materials development and anti-corrosion measures*

Corresponding author.
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan704

Dongmei Fu, Gaoyuan Wang, Tao Yang, Chao Wu, Mengchen Cui

Data density
Specific to the point A(x,y) with its local domain, thedata density is defined to be =S/N . Setting a thresholdt while low densityt，high density domain ∅ has
domain ∅ hast.
Interpolation for the point A(x,y)
As for any point A(x,y) to be interpolated, when( , ) ∈ ∅ , taking the average value of all the points
in domain ∅ as one known data with location to be thecentral point in the domain. According to Eq.(1) to
calculate the final interpolation result. When ( , ) ∈∅ , the result can be obtained directly by Eq.(1)..

of China being constructed based on ISO9223-2012
standard.
In order to construct the carbon steel atmosphericcorrosion maps according to ISO9223-2012 standard,
the problem to be firstly solved is to procure the valuesof time of wetness, sulfur dioxide concentration,
chloride ion concentration of any sites in the map.
However, the places for continuous monitoring thesedata are very limited; therefore, the data interpolation isnecessarily needed. This paper used the inverse distanceweighted method and the density-based improved
inverse distance weighted method to calculate the datainterpolation respectively. Upon the basis ofinterpolation, the carbon steel atmospheric corrosion
level map of China is to be constructed.

3. Atmospheric Corrosion Level Map of China
Based on ISO9223-2012

The figure 1 below is the flow chart of the map
construction.

2. Inverse Distance Weighted Method andImprovement
The inverse distance weighted interpolation states thatas to getting the values of unknown points, those known
points which are closest to the unknown ones contributemost while the contribution is inversely proportional tothe distance10, as the following formula shows:

z0

n

zj

j 1

d jp

1
p
j 1 djn(1)where z 0 is the interpolation value of the unknown
point, z j is the value of the j th known point, d j is the
distance from the unknown point to the j th known point,
and p is the power of that distance.
Inverse distance weighted interpolation is one of themostly used methods with simplicity and easily
understandability. However, there is one obvious
problem in Eq. (1): when the data density is seriously
uneven, the interpolation result will greatly weaken the
global distance factor but tend to adopt the data locallywith higher density. In view of this, this paper proposes
the following improved algorithm:
Local domain for any point A(x,y)
Set S is the local area, and N is the number ofpoints within local domain. Define that the local domain
S/N .
for the point A(x,y) is a circle with radius R

Fig. 1. Process of corrosion level map construction

The sulfur dioxide concentration data come from
China Statistical Yearbook and there are statistical dataof totally 49 major cities in China. The distribution of
those cities is shown in Figure 2(a). Using inverse
distance weighted interpolation method results in Figure2(b), and using density-based improved inverse distanceweighted interpolation method results in Figure 2(c)(with parameter R 4, t 5 , and the calculation
afterwards takes the same parameters.)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan705

Construction and Visualization ofhighest in the coastal areas and gradually degrades
along with the distance from the coast to the interior
land. Further, in a very short distance, the chloride ion
concentration decreased rapidly while then the decline
rate slowed. The resulting national chloride
concentration level map is shown in Figure 4.

(a). SO2 sampling distribution map(a). Time of wetness distribution(b). SO2 level map(original)(b). Time of wetness level map(original)(c). SO2 level map(improved)
Fig. 2. SO2 sampling and level map of China

Figure 3 (a) shows the distribution of time ofwetness in 135 major cities nationwide. It is obvious to
see that the data points of time of wetness are densely
distributed and relatively even compared with the
distribution of sulfur dioxide concentration.
There is no significant difference between Fig.3(b)and Fig.3(c) because the distribution of data points fortime of wetness is much more uniform.
The Chloride ions in the atmosphere mainly comefrom the ocean. The distribution of chloride ion
concentration is directly related to the distance awayfrom the sea. Chloride ion concentration reaches the(c). level map(improved)
Fig. 3. Time of wetness sampling and level map of China

According to ISO9223-2012 standard and the flow
chart of Fig.1, it is ready to get a map of atmosphericcorrosion level map of carbon steel in China, as shownin Fig.5 and Fig.6.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

706

Dongmei Fu, Gaoyuan Wang, Tao Yang, Chao Wu, Mengchen Cuicentral domain of the map is kind of sparse without
improvement, while the eastern and southern domains
tend to severe. The reasons are that the data points in theeastern domain are relatively denser with high sampling.
Though in the central and western domain, data points
increased, but except for high sampling in Urumqi,
Xinjiang province, the other areas still have low density.
Acknowledgements
This work was supported by National Material
Environment Corrosion Field Scientific Observation
and Research Platform.

Fig. 4. Cl- concentration level map

References
1. M. Dai and Z. Liu, Analysis of the conjunction between
environmental factors and atmospheric corrosion ofcarbon steel, Corros. Prot. 21(4) (2015) 147–148.
2. ISO 9223-2012, Corrosion of metals and alloys Corrosivityofatmospheres
Classification,
determination and estimation (Avenue Marnix 17, B1000 Brussels, 2012).
3. X. Li, D . Zhang, Z . Liu, et al, Share corrosion data,
Nature 527(7579) (2015) 441–442.
4. U. Fathoni, C. M. Zakaria, C. O. Rohayu, Developmentof corrosion risk map for Peninsular Malaysia using
climatic and air pollution data, in IOP Conf. Ser.: EarthEnviron. Sci., (Mashhad, I. R. Iran, 2013), pp. 2088–2092.
5. C. Ivan, H. N. Viet, Steel corrosion map of Vietnam, Tạpchí khoa học và Công Nghệ 48(6) (2014) 109–115.
6. B. Chico, D. L. Fuente, et al, Corrosivity maps of Spainfor zinc in rural atmospheres, Revista De Metalurgia
46(6) (2010) 485–492.
7. Z. Wang, H. Chen, An investigation on atmosphericcorrosiveness in Hainan province, J. Chin. Soc. Corros.
Prot. 16(03) (1996) 225–229.
8. H. Chen, G. Yu, et al, Atmospheric corrosion map
development in Shenyang, Corros. Sci. Prot. Technol.
4(3) (1992) 195–201.
9. J. Tidblad, Atmospheric corrosion of metals in 20102039 and 2070-2099, Atmos. Environ. 55 (2012) 1–6.
10. F. W. Chen, C. W. Liu, Estimation of the spatial rainfall
distribution using inverse distance weighting (IDW) in
the middle of Taiwan, Paddy Water Environ. 10(3)(2012) 209–222.

Fig. 5. Corrosion map(original)

Fig. 6. Corrosion map(improved)

4. Conclusions
In this paper, an improved inverse distance weighted
interpolation method based on density is proposed.
According to ISO9223-2012 standard, the paper
constructed the carbon steel corrosion level map basedon original inverse distance weighted interpolation
method and the improved one. In view of therepresentation of the corrosion map, there is no major
impact on most areas; however, in the local view, the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

707

Multiple-Model Adaptive Estimation With A New Weighting Algorithm
Weicun Zhang*, Sufang Wang, Yuzhen Zhang

School of Automation and Electrical Engineering, University of Science and Technology Beijing, 30 Xueyuan Road
Haidian District, Beijing, China
E-mail: weicunzhang@ustb.edu.cn, zhangyz_ustb@163.com
www.ustb.edu.cn

Abstract
This paper presents a new scheme of weighted multiple-model adaptive estimation for a discrete-time stochastic
dynamic system with large parameters uncertainty, in which the classical weighting algorithm is replaced by a newweighting algorithm to reduce the calculation burden and to relax the convergence conditions.
Keywords: multiple model adaptive estimation; weighting algorithm.

algorithm to generate weights for each 'local' KF. After
that, each 'local' KF generates its own state estimate anda corresponding output error (residual) to feed theweighting algorithm. The 'global' MMAE state estimate
is then a weighted summation of each 'local' KF's
estimate. Figure 1, which was borrowed from reference
20, depicts the logic structure of an MMAE system.

1. Introduction
There are many approaches to address the modelinguncertainty and nonlinearity problems in state
estimation, filtering, and control. Among others, themultiple model adaptive estimation (MMAE) or
multiple model adaptive control (MMAC) schemes havereceived much attention. The use of multiple models foradaptive estimation originated from Magill 1. Later on,
Lainiotis 2, Athans et al. 3, 4, Anderson and Moore 5, andLi and Bar-Shalom 6 studied MMAE/MMAC for
different purposes.
In recent years from late 1990's, MMAC is a
research focus in adaptive control area 7-18. In parallel,
there are also more and more studies conducted in
MMAE 19-21.
There are mainly three tasks (or steps) to complete
an MMAE system in design stage, Task 1: to construct
a 'local' model set to cover the parameters uncertainty or
nonlinearity of the system as described in (1); Task 2: todesign a 'local' Kalman filter (KF) set according to the'local' model set, in which each 'local' KF is designed
for each 'local' model; Task 3: to devise a weighting*

Fig. 1. The block diagram of an MMAE 20

In this paper, a new weighting algorithm adaptedfrom reference 17 is adopted with two purposes, one isCorresponding author.
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

708

Weicun Zhang, Sufang Wang, Yuzhen Zhangxˆ i ( k | k

to simplify the classical weighting algorithm that
depends on dynamic hypothesis test concept andBayesian formula 1, 9; the other is to relax theconvergence condition of the classical weighting
algorithm.
The reminder of the paper is organized as follows.
Section 2 give a brief description of an MMAE system;
Section 3 presents the weighting algorithm; Section 4 isthe main result about the convergence of the state
estimation; Section 5 includes two cases of simulations;Finally, some conclusions will be drawn in Section 6.

Pi ( k | k
K i (k )xˆ i ( k | k )y(k ) C(k, ) x(k )

(k )

(1)

n
m
is the state of the system, u (k )
the
x(k )
q
control input, y (k )the system output, (k ) R r an
measurement noise that cannot be measured withp
uncertain variance R {R1 , , RN } , and ( k )system noise that cannot be measured with uncertain
variance Q {Q1 , , QN } . The matrices A(k , ) ,B(k , ) , and C (k , ) are assumed piecewise
continuous, uniformly bounded in time, and containlunknown constant parameters denoted by vector.
The initial condition x (0) of (1) is assumed
deterministic but unknown. Consider a finite set ofcandidate parameter values: 1 , 2 ,..., N indexedby i 1,..., N .
The MMAE can be described as follows:xˆ (k )Ni 1

pi (k ) xˆi (k )1)

Ai Pi ( kBi u ( k1) AiT

1)
Qi(I

K i ( k )C i ) P ( k | k1)(3)
where Ai : A( k , i ) , Bi : B (k , i ) , Ci : C (k , i ) ,xˆi ( k / k ) is the xˆi (k ) in equation (2).
We expect that if the j th model M j in the model
set is (or close to) the real plant model, then thecorresponding j th KF will generate the correct state
estimation xˆ j ( k ) . In addition, if the j th weight p j ( k )converge to 1, and others to 0. Then the state estimatesof the MMAE will converge to xˆ j ( k ) .
Thus the key point for an MMAE system is toconstruct an effective weighting algorithm as well as an
appropriate model set to include the real model or theclosest model to the plant. The weighting algorithm
will be described in the next section.

Consider a discrete-time system described by thefollowing state-space equation

(k )

1)

Pi ( k | k 1) C iT ( C i Pi ( k | k 1) C iT R i ) 1xˆ i ( k | k 1) K i ( k )( y ( k ) C i xˆ i ( k | k 1))
P (k )

2. The Multiple-Model Adaptive Estimatorx(k 1) A(k, ) x(k ) B(k, )u(k )Ai xˆ ( k1)

3. Weighting Algorithm
We adopt a novel weighting algorithm which was put
forward in reference 17 for MMAC systems, to replace
the classical weighting algorithm based on dynamic
hypothesis test and Bayesian formula 1, 9.
li (0)

1; p i (0)Nli ( k )1lmin ( k )li ( k )(2)pi ( k )where xˆ( k ) is the estimate of the state x ( k ) at time k ,andpi (k ); i 1,..., N are time-varying weights
generated by the weighting algorithm, witch will begiven in the next section. In (2), each "local" state
estimatexˆi (k ), i 1,..., Nis generated by acorresponding 'local' KF, which is described as follows.

li (0)(4)

1 k‖ ei ( p ‖
) 2
k p 1

(5)

min i li ( k )(6)li ( kl (k )
1) minli ( k )(7)li ( k )(8)Nl (k )i 1 i

where ‖‖ denotes the Euclidean norm.
According to reference 17, we have the following
convergence result of weighting algorithm (4) - (8).
Theorem 1. If M j M is the model closest to the true
plant in the following sense with probability one© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan709

Multiple-Model Adaptive Estimationkk‖e j (r ) ||2

r 1

1
k
1
k

k

‖e j (r ) ||2

Sj

2

Si

r 1
k

‖ei (r ) ||

r 1

Sj

Si , i1; pi ( k )p j (k )(9)xˆ(k )

1, ... N , i

j

(10)

k

‖ e j ( r ) ||

r 1

1
k
1
k

k

2

‖ ei ( r ) || , k

r 1‖ e j ( r ) ||

2

Sj

r 1
k

‖ ei ( r ) ||2

Sj

Si , ik‖ e j ( r ) ||21k1
kxˆ j (k )k‖ ei ( r ) ||2 , k

r 1‖ e j ( r ) ||2

Sj

2

Si

r 1
k

‖ ei ( r ) ||

r 1

Sj

(14)

(15)

Si , i1(17)jwhere S j is a constant, S i may be a constant or infinity.
Then the state estimates of MMAE will converge tothe estimates given by the j th KF corresponding to M j ,i.e.,xˆ ( k )xˆ j ( k )(18)

Proof. According to Theorem 1, condition (17) leads top j (k )1

1; pi ( k )0, i 1,...N , i

j

(19)

Then we havexˆ(k )

(12)

N
i 1

Si

r 1

pi (k ) xˆi (k )r 1

In this section, the convergence analysis of MMAE
systems is conducted. The results are delivered in thefollowing theorems.
If model set M includes the only real model ofsystem (1), we have the following result.
Theorem 2. If M j M is the only real model of system(1) in the following sense with probability onekjThat completes the proof.
If the real model of system (1) is not included inmodel set M , but there is a closest model to system (1),
we have the following result.
Theorem 3. If M j is the model closest to the true plantin the following sense with probability one

Main Results

20, i 1,...N , i

Further, by Lemma 2 in reference 17 we know that thevariance of state estimation error will converge to its
minimum value, i.e.,(16)E[ xˆ (k ) x(k )]T [ xˆ (k ) x(k )] E T (k ) (k ) Q

It is worth pointing out that the convergencecondition for the weighting algorithm (4)-(8) is fewer
than that for the classical weighting algorithm. To be
specific, the convergence condition (9) means thediscriminability of ei (k ) , while the convergenceconditions for classical weighting algorithm include
ergodic, stationary, and discriminability of ei (k ) , formore details please be referred to reference 9.

k

N
i 1

To adapt the above-mentioned weighting algorithm
to MMAE problem, we need only to redefine the error
signal as estimation error or residual of each 'local' KF,
i.e.,ei (k ) y(k ) yi (k ) y(k ) Ci xˆi (k )
(11)

4.

1; pi ( k )Then we have

j

0, i

(13)

Proof. According to Theorem 1, condition (12) leads towhere S j is a constant, S i may be constant or infinity.
Then the weighting algorithm (4)-(8) leads top j (k )xˆ j ( k )xˆ ( k )‖ei (r ) ||2 , k 1

r 1

pi (k ) xˆi (k )xˆ j (k )(20)That completes the proof.
To get sharper convergence rate, we make two
modifications in weighting algorithm (4) - (8), i.e., weadopt the following equations to replace equation (5),and equation (7), respectively.
1 kli (k ) 0.001‖ei ( p‖
)2
(21)
kp1

j

where S j is a constant, S i may be a constant or infinity.
Then the state estimates of MMAE will converge tothe optimal estimates given by the j th KF
corresponding to M j , i.e.,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

710

Weicun Zhang, Sufang Wang, Yuzhen Zhangl (k )li (k ) li (k 1){ min }2li (k )

(22)
9.

5. Conclusions
10.

A new MMAE scheme is proposed with an improved
weighting algorithm which was borrowed from MMAC
systems. Both theoretical analysis and simulation results
verified the effectiveness of the proposed MMAE
scheme. The future work will be focused on improving
further the algorithm to get more rapid convergence
rate; adapting the weighting algorithm for state
estimation of time-varying and non-linear systems.

11.
12.
13.

Acknowledgements

14.

This work was supported by National Key Technologies
Research and Development Program of China (No.
2013BAB02B07) and National Science Foundation ofChina (No. 61520106010).
References

15.

1. D. T. Magill, Optimal adaptive estimation of sampled
stochastic processes, IEEE Trans. on Automat. Contr. 10(1965) 434–439.
2. D. G. Lainiotis, Partitioning: A unifying framework foradaptive systems I: Estimation II: Control, IEEE Trans.
Automat. Contr. 64 (1976) 1182–1198.
3. M. Athans, et. al, The stochastic control of the F-8c aircraftusing a multiple model adaptive control(mmac)
method part I: Equilibrium flight, IEEE Trans. Automat.
Contr. 22 (1977) 768–780.
4. M. Athans (eds.), Investigation of The Multiple Method
Adaptive Control (MMAC) Method For Flight Control
Systems (Technical Report NASA-CR-2916, NASA
Technical Reports, 1979).
5. B. D. O. Anderson and J. B. Moore, Optimal Filtering(Prentice Hall, New Jersey, USA, 1979).
6. X. R. Li and Y. Bar-Shalom, Multiple-model estimation
with variable structure, IEEE Trans. Automat. Contr. 41(1996) 478–493.
7. B. D. O. Anderson, et. al, Multiple model adaptive
control: PartI: Finite controller coverings, Int. J. Robust
Nonlinear Contr. 10 (2000) 909–929.
8. M. Athans, S. Fekri and A. Pascoal, Issues on robust
adaptive feedback control, in Preprints 16th IFAC World

16.
17.
18.
19.

20.

21.

Congress (Plenum, Prague, Czech Republic, July 2005),
pp. 9–39.
S. Fekri, M. Athans and A. Pascoal, Issues, progress andnew results in robust adaptive control, Int. J. Adapt.
Contr. Signal Process. 20(10) (2006) 519–579.
S. Fekri, M. Athans and A. Pascoal, Robust multiple
model adaptive control (RMMAC): A case study, Int. J.
Adapt. Contr. Signal Process. 21(1) (2007) 1–30.
J. Hespanha, D. Liberzon, A. S. Morse, et. al, Multiple
model adaptive control: Part II: Switching, Int. J. Robust
Nonlinear Contr. 11 (2001) 479-496.
A. S. Morse, Supervisory control of families of linear
set-point controllers - Part I: Exact matching, IEEE Trans.
Automat. Contr. 41 (1996) 1413-1431.
A. S. Morse, Supervisory control of families of linear setpoint controllers - Part II: Robustness, IEEE Trans.
Automat. Contr. 42 (1997) 1500–1515.
K. S. Narendra and J. Balakrishnan, Adaptive control
using multiple models, IEEE Trans. Automat. Contr. 42(1997) 171–187.
K. S. Narendra and O. A. Driollet, Stochastic adaptive
control using multiple models for improved performancein the presence of random disturbances, Int. J. Adapt.
Contr. Signal Process. 15 (2001) 287–317.
G. J. Schiller and P. S. Maybeck, Control of a large spacestructure using MMAE/MMAC techniques, IEEE Trans.
Aerosp. Electron. Syst. 33 (1997) 1122–1131.
W. Zhang, Stable weighted multiple model adaptive
control: discrete-time stochastic plant, Int. J. Adapt.
Contr. and Signal Process. 27(7) (2013) 562–581.
M. Huang, X. Wang and Z. Wang, Multiple model selftuning control for a class of nonlinear systems, Int. J.
Contr. 88(10) (2015) 1984–1994.
A. P. Aguiar, M. Athans and A. Pascoal, Convergenceproperties of a continuous-time multiple-model adaptive
estimator. In Proc. of ECC'07-European Contr. Conf.,
eds. S. G. Tzafestas, E. E. Camacho and P. G. Groumpos(Kos, Greece, July, 2007), pp. 1530–1536.
A. Pedro Aguiar, V. Hassani, A. Pascoal and M. Athans,
Identification and convergence analysis of a class ofcontinuous-time multiple-model adaptive estimators, In
Preprints 16th IFAC World Congress (Plenum, Seoul,
South Korea, July 2008), pp. 8605–8610.
J. Bernat and S. Stepien, Multi-modelling as new
estimation schema for high-gain observers, Int. J. Control.
88(6) (2015) 1209–1222.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

711

Revisit Constrained Control of Chaos
Yunzhong Song†, Ziyi Fu, Fuzhong Wang

College of Electrical Engineering and Automation, Henan Polytechnic University, 2001 Century Avenue, Jiaozuo
454003, P.R.China
E-mail: songhpu@126.com, fuzy@hpu.edu.cn, wangfzh@hpu.edu.cn
www.hpu.edu.cn

Abstract
Only two conditions were considered in Physics Letters A296 (2002) 87, and thereafter a special problem wasinvestigated in Physics Letters A359 (2006) 624. In this paper, a soft control scheme dominated and featured justby one single flexible factor is suggested to improve the already existing results.
Keywords: chaos control, soft control, flexible factor, Lorenz system

The motivation to come to this paper lies in thetwo following reasons. The first one is from theincompleteness of the policy itself. It is well known thatthe strict bang-bang control action can bring thedisruptive damage to the actuator, especially for its
mechanical parts. To overcome that limitation, theimproved version of the control policy should be
introduced in. And the second one is from the system
point. The newly proposed policy is made possible, andthe alignment among the different suggestions should bejoined in. From the original idea to the transient processproceeding strategies, and including the newly coming
ones here, should be encapsulated together, so as tofacilitate their future employment.

1. Introduction
An additive, scalar, and inequality constrained control
method has been introduced to the well known chaotic
system Lorenz system by Tian, Tade, and Levy1. The
resulting control law is bang-bang one and can stabilize
the unstable equilibria, where two cases such that
control action added at the first and second equations of
the Lorenz system are reported. Some special
phenomena like difficulty in stabilizing equilibria
induced by control action added on the third equation of
the Lorenz system, is touched upon and overcome by
the combination of a logic cell and the switching control
law which appeared in Song, Zhao and Qi2. In this paper,
we will consider to use a soft control law which is
chattering free in steering the chaotic Lorenz system toits desired unstable equilibria.

†

2. Control of Lorenz System with SoftTechnology
This part will start with the retrospect of the optimal
control of chaos system, which will be covered by

Corresponding author© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan712

Yunzhong Song, Ziyi Fu, Fuzhong Wang

subsection 2.1, then the main results will be followed at
2.2, after that some comments will be provided toclarify the idea further in 2.3.

worsen when they approach their steady states so nearby,
this phenomena is not permitted in many situations.

2.1. Problem formulation

In order to get rid of chattering in control action, wesuggest the following control action

2.2. The main results

For Lorenz systemux1

f1 ( x1 , x2 , x3 )

( x1

x2 ),

x2

f 2 ( x1 , x2 , x3 )

rx1

x3

f3 ( x1 , x2 , x3 )x1 x2 bx3 .

(1)x1 x3 ,

x2

Parameters like
10, r 28, and b=8/3 are taken,
which results in chaotic motion. And the corresponding
three system equilibria arexe(1) [0, 0, 0]T ,xe(2,3) [ b(r 1), b(r 1), r 1]T [ 6 2, 6 2,27]T .
All of them are unstable. The control purpose is tostabilize xe(2) or xe(3) . Now we recall control on the third
equation of Lorenz system. Using an additive andconstrained control like references [1-3], we have
0 and 3 0 . Suppose [ 1 , 2 ]T has non-zero
3
solution, the following switching surface can be gotten.
S ( x)

f1

f2
x3

f2

f1
x3

x12

x1 x2

10 x12 10 x1 x2(2)e

M sgn[ S ( x)], u

M sgn[ S ( x)]Table 1. Flexible factor values
corresponding control actions.
1/9
1/5
1/3
3/7

(3)
e

It can be easily checked that both x or x are on
this surface. So the two possible bang-bang control lawsareuwhere 10 , is named as flexible factor. Control
action as Eq.(5) can be named as soft control, this will
be explained later. It is more interesting that whenchanges from a smaller positive value to a relatively
bigger positive value, u transfers from u to u ,naturally and simultaneously (according to Ref. [1-3],
control action renders xe(3) stabilization is named as u ,and vice versa). Some typical values ofis listed as
Tab.1.

1/4

0. (2)

4/9
1/2
4/5

(3)

(5)

M [ S ( x )]

and

the

u
u

u
u
u
u
u
u
u

Where M=3. According to the already existing
conclusions derived by references [2-3], when u
successfully stabilizes xe(3) , u fails in stabilizing xe(2) .
In virtue of the ergodic characteristic of the chaotic
system, the control action u can be improved asu

M sgn[ S ( x)], x1 c0, x1 c.

,(4)where x1 is the first state coordinate of the chaos(2)system, c=8 is around xe(2)1 , xe1 is the first coordinateof xe(2) . Constant c can also be chosen as xe(2)1 itself，
1 . From the known results, we can find that thecontrol action for stabilizing both xe(3) and xe(2) switches
frequently between +M and –M. And that will become

Fig.1 Control results of u+ in (5) with M=3,

=1/9.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

713

Revisit Constrained Control of

The control results about selected as 1/9, 1/4 and 4/9,
1/2, are demonstrated as Fig.1, Fig.2, Fig.3, and Fig.4,
respectively. And the listed values of like 3/7 and 4/5
in Tab.1, their counterparts results are omitted here.
Please notice that control actions are impelled to zero invery short time, compared to their bang-bang
counterparts.
Where, among Fig.1 and Fig.2, the control actions
are corresponding to u , while among Fig.3 and Fig.4,
the control actions are correlated with u (Please refer toTab.1). Since control action ceases at last, we coin soft
control to declare its peculiarity.
2.3. Some comments
Fig.2 Control results of u+ in (5) with M=3,

=1/4.

Fig.3 Control results of u- in (5) with M=3,

=4/9.

To be specific, the soft control introduced with theflexible positive factor, which is constrained among 0
and 1, enables the possibility of chattering reduction andsmoothly switching between the two different control
directions, simultaneously. And besides that, the starting
point of control is set free from the neighbouring target
points, and this will reduce the burden of theunnecessary barrier, which is expressed via constraints
of inequality. And even more, prospective control
strategy for chaos systems can be enriched according to
the unification of the ideas covered by the series papers
as [1-3] and this one. The initiation began at theswitching surface and the optimal control policy, andfollowed by full states targeting through theintroduction of constrained inequality. And the stay to
go trick came as the third one. To drive the idea further,in the present paper, our aim is try to move the control
action from the positive ones to the negative ones
seamlessly, and to facilitate the potentially applicationof the optimal control strategy. Of course, the mild costmust be paid to turn the flexible factor from one intoanother.
3. Conclusion

Fig.4 Control results of u- in (5) with M=3,

In conclusion, with flexible factor joined in, a soft
control law, not only chattering has been shut down
greatly but also the adjustment of control action
becomes easily, is realized to stabilize the unstableequilibria of Lorenz system.
Chaos system is just one kind of complex systems,
and it is far from the toughest one. Examples like highperformance expected vehicle control, which waspioneered in Jia4 and the customarily designed manner=1/2.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan714

Yunzhong Song, Ziyi Fu, Fuzhong Wangof regulation in Jia5, can be the solid instantiations of
this. As to the massive volume and large scale complex
systems, not only the plant itself, but also thesurrounded hostile environment factors, will impede thesuccessful manipulation of them. It is so hard for us tospell out some adversary factors of the harshenvironments. They can be the time varying delays,
incomplete noisy uncertain and unreliable information,
augmented annoying disturbances, and multi-inputs
multi-outputs. And these make adaptive schemes
become the first candidate, please referring to Jia6. For
the single large quantity system, which was just comeinto our view in recently years, philosophy of regulation
has been varied sharply. The main stream has beenshifted from hardware to software, and contemporarily,
the new trend occurs, i.e. from both of them to thewetware, that pushes strongly in learning from theorganized behavior of group habited living beings andour mankind society system. Without any doubt, thiscan be said as a large jump in regulation in complex
systems deployment. This can be easily seen in Song,Fu and Wang7, Song8 as well as Meng and Jia9. So, thesoft strategy demonstrated here deserves its further
exploration in the near future, to the plant we face, thetask we try to overcome, the unpleasant disturbance wemust cope with, all of these must be set via balanced
negotiation, co-operation.

Acknowledgements
This work is partially supported by NSFC Grant(61340041 and 61374079) and the Project-sponsored by
SRF for ROCS, SEM to Yunzhong Song.
References
1. Y. Tian, M. Tade and D.Levy, Constrained control of chaos,
PLA, 296 (2002) 87-90.
2. Y. Song, G. Zhao and D. Qi, Some comments on constrained
control of chaos, PLA, 359 (2006) 624-626.
3. Y. Song, Z. Fu and F. Wang, Targeting chaos system via
minimum principle Control, JRNAL, 3-4 (2017) 249-252
4. Y. Jia, Robust control with decoupling performance forsteering and traction of 4WS vehicles under velocity-varying
motion, IEEE. T. Contr. Syst. T, 8(2000) 554-569.
5. Y. Jia, Alternative proofs for improved LMI representations
for the analysis and the design of continuous-time systems
with polytopic type uncertainty: a predictive approach, IEEE
Trans. Autom. Contr.l, 48(2003) 1413-1416.
6. Y. Jia, General solution to diagonal model matching control
of multi-output-delay systems and its applications in adaptive
scheme, Prog Nat Sci, 19( 2009) 79-90.
7. Y. Song, Ziyi Fu, Fuzhong Wang, Socialized multi-agent
system rendezvous via networks of networks, JRNAL, 3(2016) 136-139.
8. Y. Song, Consensus of agents with mixed linear discrete
dynamics, IJCAS, 14 (2016) 1139-1143.
9. D. Meng, Yingmin Jia, Robust consensus algorithms for
multiscale coordination control of multivehicle systems with
disturbances, IEEE Trans. Ind. Electron., 63 (2016) 11071119.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan715

Research on Filtering for Random Data Packet Dropouts and Delays in Wireless Sensor
Networks
Sumin Han*, Fuzhong Wang

School of Electrical Engineering and Automation, Henan Polytechnic University, Jiaozuo 454000 P.R.China
hansumin@hpu.edu.cn; wangfzh@hpu.edu.cn
www.henan polytechnic university.edu

Abstract
A wireless transmission data model is proposed to solve uncertain packet losses and delays in the wireless sensor
network. It constructs a transmission output by the Bernoulli distribution sequence to express the random
packet dropouts and delays. Based on the minimum variance estimation, the optimal fusion filter and covariancematrix is derived. The effectiveness of the system is verified by a simulation example. It is indicated that he higher
the packet loss probability is, the lower the estimation accuracy is.
Keywords: Fusion filter, packet dropouts, delays, wireless transmission.

published analyzing random time delays of the wirelesstransmission system with the related Markov chain
model. However, the approach is limited not to be
applied in more fields for the system’s complexity andswitching difficultly. Moreover, it requires stable states
and constant Kalman gain, which is very difficult tomeet.
The literatures6-10 are aimed to express the systemmodel in the relevant forms of Bernoulli random
sequence, and to study the random data drop and delay
problems of linear or nonlinear systems with complex
derivation.
In this paper, a transmission delay model based on
Bernoulli distribution sequence is proposed to consider
the random two-step error that can occur in linear timevarying systems. For simplifying the complex formuladerivation and calculation, it is assumed that one step
variance error matrixes are orthogonal with each other.
The last section demonstrates the simulation of the
optimal fusion Kalman filter.

1. Introduction
Data losses, delays and bandwidth constraints havebecome a common problem in the fields of unreliablewireless communications. Especially in wireless sensor
networks, since data from the measurement node to the
network is confronted with the conditions such as nodesaturation, network congestion, stochastic delays, packet
losses or random noise measurements and transmission
errors, they will ultimately lead to the packet losses anddelays of data transmission problem1. Therefore, theresearch of distribution filter and fusion filter for thisproblem is widely concerned.
In recent years, lots of results are reported
frequently. About data dropouts, delays and randommeasurement errors in the wireless sensor networks.
Generally, two main aspects are concerned: one is themodeling of stochastic system; the other is the design ofthe filter. There are two main approaches aboutmodeling random systems: Markov chains andBernoulli sequences. Many papers2-5 have been*

Corresponding author© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

716

Sumin Han, Fuzhong Wang

( + 1| + 1) = ( + 1| ) + ( + 1) ̃( + 1) (3)By minimizing the covariance error matrix
P(k+1|k+1)，the optimal filter and the Kalman gain canbe derived, whereP(k+1|k+1)= E{[ ( + 1) − ( + 1| + 1)] ×[ ( + 1) − ( + 1| + 1)]}
The innovation is(4)̃( + 1) = ( + 1) − ̂ ( + 1| )where
̂ ( + 1| ) = ( + 1) ( ) ( + 1| )+1 − ( + 1) ( + 1) ( ) ( | − 1) +1 − ( + 1) (1 − ( + 1) ×
( ) ( − 1| − 2)(5)
According to the recursive projection theorem andthe orthogonality of the innovation sequences, the basic
equations of the optimal filter is derived by theminimum variance.
( + 1) = ( + 1|k) ( ) ×[ ( )+ ( )+ ( )+ + + ]
(6)
with
( )=
( ) ( + 1| ) ( )( ) = (1 − ) ( ) ( | − 1) ( )( ) = (1 − ) ( ) ×( − 1| − 2) ( )== (1 − )= (1 − )where = , = 0,1.
Covariance matrix can be renewed：( + 1| + 1)=[I − ( + 1) ( + 1) ( )] ( + 1| ) ×[I − ( + 1) ( + 1) ( )] + ( + 1)
× [ ( ) + ( ) + + + ] ( + 1) (7)Considering two-step packet losses or delays,
Kalman gain and covariance matrix will be updated
only by attaining the last two steps’ states. So, only the
initial values of three points are known, iteration will
start. As (5) and (8) shows, the initial values of optimal
states (2|2) , (1|1) , (0|0) and of the covariancematrix (2|2), (1|1), (0|0) must be given.

2. Problem Formulation
Consider the following linear time-varying stochastic
sensor systems with stochastic uncertainties of white
noises, the system model is:( + 1) = ( ) ( ) + ( )(1)( )= ( ) ( )+ ( )where ( ) ∈ ℜ is the state vector, and ( ) ∈ ℜ is
the measured output for m sensors. ( ) , ( ) aremutually uncorrected gauss white noises. For system (1),
we make the following assumptions.
[ ( )] = 0, [ ( )] = 0[ ( ) ( )] =，[ ( ) ( )] =， [ ( ) ( )] = 0
In order to avoid network congestion, we assume
that the packets from the sensor end will be sent only
once in each moment. When the data is delayed, it ispossible that the adjacent two measurement data arrives
at the local processor at the same time. Namely, if thelocal processor doesn’t receive a packet at k moment,
the packet at k-1 or k-2 moment will be received at thismoment. Certainly, it means that the packets will bedelayed or delayed. The data received by the local
processor is referred as transmission output through thewireless transmission, expressed by the following
model:( ) = ( )y( ) + 1 − ( ) ( ) ×( − 1) + (1 − ( ))(1 − ( )) ( − 2) (2)where ( ) ∈ ℜ are mutually uncorrected randomvariables obeying Bernoulli distributions with theprobabilities [ ( ) = 1] = , [ ( ) = 0] = 1 −, 0 ≤ ≤ 1,i=0,1. Formula (2) can be explained asfollows. The measured value y(k) arrives the local
processor with a certain probability . If ( ) =0, ( )=1, the local process receives the last one-stepdata y(k-1). If ( ) = 0, ( ) =0,the local process
receives the last two-step data y(k-2). In this way, therandom transmission errors can be simply revealed. For
the existence of a lifetime of network nodes, the data
will naturally disappear after a period of time with theenergy exhausted. Hence, the transmission model in thispaper mainly considers two step delays.

4. Simulation Example

Consider the discrete linear system including one sensor
node. The corresponding time varying discrete-timemodel is given as follows:

3. Optimal Filter
Based on the transmitted outputs z(1), z(2)… z(k-1), ouraim is to design the optimal filter based on the minimum
variance estimation. The optimal state is© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

717

Research on Filering for( + 1) =

0.724

0.2

−0.77880( )+ ( )First state50( ) = [1 1] ( ) + ( )For formula (2), ( ) is obeyed Bernoulli distribution
with the probability of p.
1.044
0
R=0.2847
Q=00.9392(0|0) = (1|1) = (2|2) = 0,(0|0) = (1|1) = (2|2) = 0.1I,
where I is the unit matrix
Base on the initial values, iteration will begin at thenode k=2.The main operations of each iteration are
generalized as follows:
1) Compute prediction state ( + 1| );
2) Compute the transmission out prediction statê ( + 1| )3) Compute one step prediction covariance matrix( + 1| );
4) Compute the filter gain ( + 1) by (6);
5) Compute the optimal state ( + 1| + 1);
6) Compute covariance matrix ( + 1| + 1).
When p=0.8, we obtain the predicted values by themodel shown in Fig. 1. Fig.1 demonstrates that thevalues of the two states by the fusion filtering and true
values. Fig.2 gives the estimation errors of the systemby the optimal fusion. It is illustrated the estimation
errors can be quickly stable.
As mentioned above, p is the probability ofBernoulli sequence, which indicates the data arrival
probability, so it will influence the estimation accuracyof the model to a certain extent. In this paper, theaccuracy of the model is analyzed as shown in Fig. 2.
The following points can be seen from Fig. 1 andFig. 2:
1) When the data arrival probability p is more than 0.8,
the state estimation of this filter is in line with the realvalue;
2) The model and the corresponding filter has goodcharacteristics of dynamics and stability.
3) With the increase of the data arrival probability p, theestimation error has been improved
According to the above analysis, it can be seen thatthe data arrival probability p reflects the degree ofrandom packet dropouts and delays. The larger p is, theless the degree of the packets and delays happens. Whenp is relative small, the prediction accuracy is low, whichindicates the measurement error is relative large.

(a)8
Kalman filter
true value

Second state

6
4
2
0
-2
-4
-6

0

50

100

150

200

t/step(b)
Fig.1 Fusion filter

Fig.2 The covariance error of the first estimated value whendata arrival probability p=0.5,0.8,0.98

5. Conclusion
The paper describes the transmission output modelusing a Bernoulli sequence with the random packet
dropouts and delays in a wireless network. We derive
the optimal filter and covariance error matrix based on
the minimum variance criterion. From the discussion
and simulation results, some conclusions can be reached.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

718

Sumin Han, Fuzhong Wang

4. B. Yin, G. Yin. Stability of Markov modulated discretetime systems with delays, Automatica, 39(8) (2008)
1339-1351.
5. X. M. Zhang, Q. L. Han. Network-based H∞ filtering
using a logic jumping-like trigger, Automatica. 49(5)(2013) 1428-1435.
6. M. Moayedi, Y. K. Foo, Y. C. Soh. Adaptive Kalmanfiltering in networked systems with random sensor delays,multiple packet dropouts and missing measurements, in
Proc. 48th IEEE Con. on. IEEE, (China, Shanghai, 2009),
pp.2058-2063.
7. B. Chen, W. A. Zhang, L. Yu. Distributed fusion
estimation with missing measurements, randomtransmission delays and packet dropouts, IEEE Trans.
Autom. Contr., 59(7) (2014) 1961-1967.
8. J. Ma, S. Sun. Optimal linear estimators for systems withrandom sensor delays, multiple packet dropouts anduncertain observations, IEEE Sensors J., 59(11) (2011)5181-5192.
9. J. Ma, S. Sun. Linear estimators for networked systems
with one-step random delay and multiple packet dropouts
based on prediction compensation, IET Signal Processing,11(2) (2017) 197-204.
10. H. Rezaei, R. E. Mahboobi, M. Farsi. Robust filtering for
uncertain networked systems with randomly delayed andlost measurements, IET Signal Processing, 9(4) (2015)
320-327.

Firstly, the model and the corresponding filter have avaluable performance for the random packet dropoutsand delays in the wireless network. Secondly, the packet
probability p is an important parameter influencing theestimation accuracy. It is main challenge in the future
research because the smaller p needs a higher accuracy.
Acknowledgements
This work was supported by National Key Research andDevelopment Program (2016YFC0600906).
References
1. I. F. Akyildiz, W. Su, Y. Sankarasubramaniam, et al.
Wireless sensor networks: a survey, Computer Networks,
38(4) (2002) 393-422.
2. O. Costa. Stationary filter for linear minimum meansquare error estimator of discrete-time Markovian jumpsystems, IEEE Trans. Autom. Contr., 47(8) (2002) 13511356.
3. S. Smith, P. Seiler. Estimation with lossy measurements:
Jump estimators for jump systems. IEEE Transactions on
Automatic Control. 48(12) (2003) 2163-2171.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

719

Optimal Hohmann-Type Impulsive Ellipse-to-Ellipse Coplanar Rendezvous
Xiwen Tian, Yingmin Jia*

The Seventh Research Division and the Center for Information and Control, School of Automation Science andElectrical Engineering, Beihang University (BUAA), 37 Xueyuan Road, Haidian District
Beijing, 100191, China
E-mail: tianxiwen123@163.com; ymjia@buaa.edu.cn
www.buaa.edu.cn

Abstract
This paper devotes to the problem of ellipse-to-ellipse coplanar rendezvous, where the solution and distribution of
Hohmann-type optimal impulsive rendezvous are investigated. The analytical relation between the initial states andrendezvous time are derived for Hohmann-type, and the optimal impulse amplitudes are given thereupon. The
distribution boundary of Hohmann-type model is obtained according to the Hohmann transfer and Hohmann withcoasts. Simulations are demonstrated to analyze the influences of the solution and distribution.
Keywords: Optimal impulsive rendezvous, Hohmann-type rendezvous, ellipse-to-ellipse, optimal distribution.

on the selection of reference frame for optimal
impulsive rendezvous, and investigated the effect on theclassification, distribution and guidance precision. For
the case of elliptic orbit rendezvous, Wang6 used thestate transition matrix given by Yamanaka7 to calculate
the optimal solution of four-impulse model, but theanalytical solution and the distribution are difficult to beachieved. Chen8,9 studied the ellipse-to-circle coplanar
rendezvous based on his results on the dynamical
equations for elliptic orbit rendezvous in low
eccentricity, and provided the solutions anddistributions of all types optimal models. Motivated bywhich, our previous work10 considered the ellipse-toellipse coplanar rendezvous and obtained the analyticalsolution and distribution of four-impulse model. In thispaper, we will further investigate the Hohmann-typemodel for optimal impulsive ellipse-to-ellipse coplanar
rendezvous.

1. Introduction
Optimal impulsive rendezvous is aimed at obtaining
minimum-fuel guidance strategy for spacecraft
rendezvous, which has attracted considerable attention.
Despite that Lawden’s necessary conditions1 for optimal
impulsive trajectories and Lion’s improving methods2
for non-optimal trajectories have provided someguidelines to solve the optimal problem where the initialstates and rendezvous time are specified, thedistributions of optimal models cannot be obtained
clearly in these way. So far, only Prussing’s theory3 of
optimal impulsive rendezvous on close circular orbits iscomplete in its theoretical system, which derives thesolutions and distributions of optimal impulsive models
by solving the primer vector equations and boundary
value problem. A reference frame in mean velocity orbit
was built by Frank4, and showed better performance indescribing the impulse locations and magnitudes thanthe mean radius orbit in Prussing’s results. Xie5 focused*

Corresponding author© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

720

Xiwen Tian, Yingmin Jia

2. Dynamics Description

3.1. Solution of Hohmann transfer

The relative motion between two spacecrafts in elliptic
orbits was derived in our previous work10, which is stillused in this paper and given as follow:r 3 r 2(1)
2 r

The impulse direction can be obtained from the solution(6), while the impulse time and magnitudes needed becalculated according to the following boundary value
problem:
21 C
0

4S

The initial and terminal states of system (1) are

x0

x01 , x02 , x03 , x04

xF

0, 0, 0, 0

where

1

kc 1 ec cos f c

x01

T

(2)

T

kt 1 et cos ft

1

(3)

1

kc 2 ec sin f c kt 2 et sin ft

x03
x04

kc

3
2

1 ec cos f c

2

3
2

kt

1 et cos ftx

2d 3 sin

then

3d 4

x03 , d 2

d3

d12 d 22 , d 4

2 x01 x04 ,arcsind1
d3

2

2 A sinand ∆are theis the rendezvous

d3 cos
2 1 C

sin

F

(9)
F

2d 4 sin F

(10)

F

d1

(11)

x01
2
x01
d12

,sin

d1

(12)

2
x01
d12

Combining (5), (9) and (13), it has(5)

3 F x01C

2C

3C

D

d1S

21 C

2d1

fc

ft

(14)

The appropriate initial states and rendezvous time which
satisfy the necessary conditions of Hohmann transfer
can be obtained by solving (11) and (14) together, andthen, the second impulse ∆ can be obtained as

The solution to primer vector equations corresponding
to system (1) can be given in the following form:

2 B cos

d3 sincos

3. Optimal Hohmann-Type Rendezvous

B sin

V2

(8)

F

On the other hand, from the second row of (8), it has
3d4 2d3 sin F
2d3 sin(13)
V1
4S 3 F3 x01 2 x04

A cos

x

where

2d 3 cos

1

2d 4

2
x01
d12 sin

where

d1

1

V1

Substituting (5) into (10), it can be obtained that(4)d3 sin

0

3 4C

V1

2d 4 d 3 cos
2d 3 sin

2S

2is the difference of phase angle between two
spacecrafts; , and , are their eccentricities andtrue anomalies, respectively.
The states at phase angle τ was also deduced[10]:

3d 4

0

where
= sin , = cos , ∆magnitudes of two impulse, andtime. From (4) and (8), we haved3 sin F
V1
2S

1

x02

3 F

V2

(6)

Hohmann-type model is a special case of optimal twoimpulse rendezvous, where the coefficients of (6) are(7)
A B C 0, D
1d4

d3 sin

F

2S

(15)

3.2. Distribution of Hohmann-type model
The distribution of optimal Hohmann-type rendezvous
is to illustrate the existence of feasible solution. To
investigate the distribution, rendezvous time is chosen
as the X-coordinate and the special phase angle defined
below as Y-coordinate:then= 0, | | = 1 . It can be verified that thenecessary conditions of optimal impulsive rendezvous
are satisfied for any phase angle τ.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

721

Optimal Hohmann-Type Impulsive Ellipse-to-Ellipse

1.5d4 F d3 sinFd3 sinF(16)

Let τ be rendezvous time solved by (11) and (14),and δθ is the corresponding special phase angle. If=
and=, then it is just the Hohmanntransfer. The two impulses are implemented at = 0
and = . However, when the real rendezvous time islonger than
, the coasts are needed to save the fuel.
If τ > τ and δθ = δθ , it is a Hohmann modelwith terminal coast. The two impulses are implemented
at τ = 0 and τ = τ , and the residual time τ − τ
is for terminal coast. The special phase angle δθ andrendezvous time τ should satisfy the following relation
F

1.5d 4Fhd3 sinFhIf>satisfiesFhd3 sin

Fh

1, F

2

F

Fig. 1. Distribution of Hohmann-type model.

F

4. Simulations(17)

In this section, simulation examples are presented toshow the guidance performance and distribution of
Hohamman-type impulsive rendezvous.

Fh ,and the special phase angle
Fh1.5d 4d3 sin

F

F

Fh

2

1

, F

4.1. Hohmann ellipse-to-ellipse rendezvousFFhd3 sinFhFh

It is assumed the semi-major axis and eccentricities of
the target orbit and chaser orbit are initially=
6730 (km),= 6750 (km),= 0.0005 and=
0.0004 , respectively. Letand(rad) be theappropriate rendezvous time and initial difference ofphase angle, respectively, which satisfy (11) and (14).
And denote (m) as the optimal radius of reference
frame, (m) as the initial relative distance, ∆ (m), ∆and ∆ (rad) as the guidance errors.

(18)

,

then, after the initial coast for time −, the special
phase angle will become exactly. This case is a
Hohmann model with initial coast, and the impulses are= −and = .
implemented at
Let
1.5d 4 F
Fh1
Fh
Fh

Fh 2

d3 sinFh

Fh

1.5d 4d3 sinFd3 sin
F
Fh

F

(19)

Table 1. Results of Hohmann impulsive rendezvous

Fhd3 sin

f
τ
β
R
R
∆a
∆e
∆θ
∆R

If >
and
<
<
, then there exists
a Hohmann model with both initial coast and terminal
coast. As shown in Fig.1., this case is illustrated in the
middle of the curves expressed by (17) and (18), that is) as the intersection
the shadow part. Denote ( ,point of the curves determined by (16) and (18), then
the two impulses are implemented at=−. The initial and terminal coast last for time
and =
−
and
−
, respectively.
From the above, the optimal Hohmann-type
impulsive rendezvous has four models, all of whose
impulse magnitudes are determined by (9) and (15), andimpulse direction is along the tangential direction.

1
30°
3.62
7.12e-03
6.70e+06
4.99e+04
6.80e+01
3.38e-05
5.82e-06
1.29e+02

2
150°
3.44
9.68e-03
7.92e+06
7.05e+04
3.38e+04
4.51e-03
2.87e-04
5.14e+03

3
210°
2.80
6.04e-03
7.81e+06
4.83e+04
3.09e+04
4.22e-03
3.50e-05
5.71e+02

4
330°
2.68
3.46e-03
6.71e+06
2.71e+04
2.49e+01
8.92e-06
3.80e-06
5.08e+01

Simulation results of Hohmann transfer for ellipseto-ellipse rendezvous are demonstrated in Table 1,
which shows that: (1) with different true anomalies,
even if the other initial states are the same, the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan722

Xiwen Tian, Yingmin Jia

rendezvous time and initial difference of phase angle
which satisfy (11) and (14) varies much; (2) the optimal
radius of reference frame also changes with the true
anomaly; (3) the guidance precision is high when thechaser initially stays around the perigee.

defining the special phase angle, we derived theanalytical solution for Hohmann transfer, and obtainedthat the optimal Hohmann-type impulsive rendezvous
has four models, i.e. Hohmann transfer, Hohmann withinitial coast, Hohmann with terminal coast andHohmann with both coasts. In further research, we will
integrate all optimal models in one map, including fourimpulse, three-impulse, three-impulse with coasts, twoimpulse, two-impulse with coasts, and Hohmann-type.

4.2. Distribution of Hohmann-type model
To investigate the distribution of Hohmann-type ellipseto-ellipse rendezvous, we take rendezvous timeas
⁄ as Y-coordinate. Fig.2
the X-coordinate andshows the distribution of Hohmann-type model withdifferent true anomalies and eccentricities.

Acknowledgements
This work was supported by NSFC (61327807,
61521091, 61520106010, 61134005) and the National
Basic Research Program of China (973 Program:
2012CB821200, 2012CB821201)
References(a) [ ,

] = [0.0004,0.0005](b) [ ,

] = [0.004,0.005](c) [ ,

] = [0.051,0.05](d) [ ,

] = [0.004,0.005](e) [ ,

] = [0.004,0.005](f) [ ,

] = [0.004,0.005]

= 30°

= 30°

= 210°

1. D. F. Lawden, Optimal Trajectories for Space Naviation(London: Butterworhs, 1963).
2. P.M. Lion and M. Handelsman, Primer vector on fixedtime impulsive trajectories, AIAA J., 6(1) (1968) 127-132.
3. J. E. Prussing, Optimal Multi-Impulse Orbital rendezvous,
Sc.D. Thesis. (Cambridge: Dept. of Aeronautics andAstronautics, Massachusetts Institute of Technology,1967).
4. C. L. Frank and L. D. Plexico, Improved solution of
optimal impulsive fixed-time rendezvous. J. Spacecraft
and Rockets, 19(6) (1982) 521-528
5. Y. C. Xie, C. Q. Chen, H. Zhang and J. Hu, Selection ofreference systems for optimal impulsive rendezvous
between two near circular orbits (in Chinese), ChineseSpace Sci. Technol., 28(2) (2008) 8-14.
6. J. X. Wang, H. X. Baoyin, J. F. Li and F. C. Sun, Optimal
four-impulse rendezvous between coplanar ellipticalorbits. Phys. Mech. Astron., 54(4) (2011) 792-802.
7. K. Yamanaka and F. Ankersen, New state transition
matrix for relative motion on an arbitrary elliptical orbit,
J. Guid. Contr. Dynam., 25(1) (2002) 60-66.
8. C. Q. Chen and Y. C. Xie, Dynamical equations for akind of elliptic orbit rendezvous (in Chinese),
Aerospace Contr., 26(5) (2008) 12-17.
9. C. Q. Chen and Y. C. Xie, Optimal impulsive ellipse-tocircle coplanar rendezvous, Sci. in China Series E: Tech.
Sci., 52(5) (2009) 1435-1445.
10. X. W. Tian, Y. M. Jia and C. Q. Chen, Optimal fourimpulse ellipse-to-ellipse coplanar rendezvous, Contr.
Conf. IEEE, (2016) 5498-5503.

= 30°= 150°= 330°

Fig. 2. Distribution of Hohmann-type model.

5. Conclusion
This paper extends our previous work10 to the
Hohmann-type optimal impulsive rendezvous. By© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan723

Research on SVG Control Method Under Unbalanced Conditions
Zheng Zheng, Yousong Zhou*, Guopeng Zhang

School of Electrical Engineering and Automation, Henan Polytechnic University
Jiaozuo, 454000, China
E-mail: zhengzh@hpu.edu.cn,z2410767638@163.com,hpoyz@163.com
http://www.hpu.edu.cn/www/index.html

Abstract
In order to reduce the complexity of traditional methods, a PCI (Proportional Complex Integral) controller isadopted to control SVG under unbalance-grid condition, which has better performance than PI controller toeliminate steady-state error to compensate nonlinear loads. Based on the αβ frame, the proposed control scheme
considerably reduces total algorithm complexity. The current control loop is designed based on mathematicalmodeling under unbalanced conditions. Simulation and experimental results are provided to validate the correctnessof the proposed methods.
Keywords: SVG; complex filter; steady-state error; proportional complex integral control;

1. Introduction

2. Control Method

In recent years, the research of SVG (Static Var
Generator) mainly focuses on both the rapid andaccurate detection of reactive current and thesimplification of reactive power control methods1,2.
Proportional resonance control method used in paper3-4
can be realized the non-static input current. The method
has high requirements for the accuracy of the hardware
and parameters of the device, which is difficult toachieve the ideal state. Non-beat control method andpredictive control method are used in paper5-7.The
precise model designs of these control methods has
some difficulties, which easily lead to the instability ofthe system.
This paper presents PCI control method based on αβ stationary reference frame. The control method not
only effectively eliminates the steady-state error of AC
current, but also improves the compensation
performance of the equipment.

The main circuit structure of SVG of the voltage sourceis shown in Fig.1.Three-phase power grid phase voltage
is expressed as ea, eb and ec; Three-phase load sidecurrent is represented by ila, ilb and ilc ; The DC side
voltage of the SVG is expressed as udc.
ea
eb
ec

Load

ea
SignalDetection

SignalDetection

Control
Circuit

DriverCircuitFig.1. Main circuit topology of SVG*Corresponding author© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

724

Zheng Zheng,Zhou Yousong, Zhang Guopeng

calculation is increased due to complexity of thecoordinate transformation. In this paper, according to
the equivalent transformation of variables, proportional
complex integral control method which can be directly
applied to three-phase coordinate system is designed.

2.1. Positive and negative component separation
method based on complex filter
A method, used complex filters to the separation for
positive and negative sequence components of the gridcurrent, which is proposed in this paper. The principle
block diagram is shown in Fig. 2. In figure, ωc is the
cut-off frequency of the filter.

Fig. 2.Current Positive and Negative Sequence Component
Separation Calculation

Fig.4. PCI Control Based on Coordinate System

From Fig. 2, I ( s ) and F ( s ) are satisfied withI ( s)

c

s

2

(s

j o)
2
o

2 cs

I ( s)(c

F ( s)

2(1)

F ( s) I ( s)

)
o
2

( o

2

(2)

2

) 4 c

The phase of the three phase coordinate systemvaries by 120°, we can obtain

By Clarke transformation, we can obtain
I

2

I

3

1
0

1

1

2

2

Ia

3

3

2

2

Ib
Ic

I

I

I

I

j 3xa

xb

xc

j 3xb

xc

xa

j 3xc

xa

xb

(4)

The control block diagram of the three phase
coordinate system is obtained by (4), as shown in Fig. 5.

(3)

A specific block diagram is obtained, as shown in
Fig. 3.

I

c

I

s

j

o

s

j

o

Ia
Ib

c

I

c

abc

Ic

c

I

c

s

j

o

s

j

o

c

I

I

c

Fig. 5.PCI Control Based on Three-phase Coordinate Systemc

3. Analysis of simulation result

Fig.3. Three-phase Unbalanced Current Positive and Negative
Sequence Component Separation Calculation

The relevant system parameters are shown in Table 1.

2.2. PCI control method

Table 1.Parametric results figure

The traditional proportional complex integral control
method is based on α-β stationary reference frame, as
shown in Fig. 4. When the current is controlled in athree-phase coordinate system, the amount of control

Parameter
Value
Grid Line Voltage 400V
Grid Frequency
50Hz
Grid Reactance
5mH

Parameter
DC side voltage
DC bus capacitor
The rated Capacity

Value
700V
6μF
200kVar© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

725

Research on SVG Control …

In the system, the bandwidth is usually selected tobe 10 times higher than the base wave frequency, andalso less than 1/5 of the switching frequency, so therange of bandwidth fb is 500 to 2000. The transfer
function of the closed loop of the system is obtained as：
T (s)I o ( s)I r ( s)K ( k p s ki
Ls

T ( s)T (s)- state error comparison of the output current waveformof SVG. When the system is controlled with PCI control
after 0.5s, the SVG output current is close to thecommand current, and the error is controlled to 0.1 A.
So the PCI control has a good effect in the steady state
error control.

2( Kk pj okp )

j o L) SR

K k2 (oarctan(

L

kp (

2

Kki )
o

ki

)

j o ( Kk p

)2

ki2

o

p

(L

Kki

2

( Kk p) arctan(

2

R) (

o

( Kk p

R )(

L

L

o

(5)

R)

)

(6)

2

o
2

)

Kki

Fig.7. (b) Load side three phase current positive and negative
sequence component) (7)When only kp is considered, the closed loop
amplitude frequency features are
T ( s)

Kk p
(L )

2

( Kk p(8)R)2Where fb=650Hz, ωb=4100rad/s, (8) give rises to
kp=0.1. Under the requirements of the system bandwidth,
the design make fb=690Hz, ωb=4330rad/s, (6) give rises
to kp=20. Under the condition of unbalanced working
conditions, the control strategy diagram of the static
reactive power generator is shown in Fig. 6.

plural filter
for separation
calculation

Fig. 8. PI control and PCI control steady-state error
comparison

4. The Experimental Test
In this paper, RT-LAB semi-physical platform isverified to the proposed strategy, as shown in Fig. 9. The
SVG control circuit is based on the core of DSP28335,
and the main part of the circuit is realized by the RTLAB platform. For the system, the input grid phasevoltage is 311V, the DC voltage is 700V, and the pureresistance load resistance is 100 Ω.

SVPWM
pulse
modulation

plural filter
for separation
calculation

Control System and Peripheral Circuit
Standard Digital Signals

Fig. 6. The Model of SVG Based on Complex Filter and PCI
Control

DSP Control System

A/D
Converter

Various Sampling
Signals

Control Pulse
Signals
SVG main circuit
Load

Fig.7. (a) Load side three phase current waveform diagram

According to above model, Matlab software is used
to simulate. It is a three-phase current on the load sidein Fig. 7. Fig.8 shows PI control and PCI control steady

RT-LAB

Fig.9. Schematic diagram of the experimental system© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

726

Zheng Zheng,Zhou Yousong, Zhang Guopeng

In order to avoid the disturbance caused by the startof the equipment, the equipment is cut off after thenormal running time of 1s, and the experimental
waveforms of this system are shown in Fig. 10, Fig. 11
and Fig. 12.

can realize real-time compensation for reactive andnegative sequence components under unbalanced
conditions.
5. Conclusion
Under unbalanced condition, the PCI controller isstudied using in SVG to compensate nonlinear loads.
Compared with traditional control method, the proposed
scheme can considerably reduces total algorithm
complexity. The current control loop is designed basedon the mathematical modeling under unbalanced input
conditions, and parameters of PCI controller is designed.
An experimental platform is built based on DSP 28335
and RT-Lab. Simulation and experimental results are
given to demonstrate the effectiveness of the proposed
control scheme.

Fig. 10. Three - phase current waveform on grid side

Acknowledgements
Fund projects: National Natural Science Foundation(U1504518).
Fig. 11.Grid side a phase voltage current waveform

References
1. Y. Liu and C. Shao, Reconstruction strategies for phasecurrents in three phase voltage source PWM inverters, J.
Inf. Contr.36(4) (2007)506-513.
2. Y.N.Yu, R.F. Yang, et al. Attractive beat control ofcascadedH-bridge
SVG,
J.
Trans.
China
Electrotechnical Soc. 01(25) (2017) 50-58.
3. L. Ma, X.M. Jin, et al. Analysis of three-phase gridconnected inverter proportional resonant control and gridvoltage feed-forward, J. Trans. China ElectrotechnicalSoc.08(27) (2012) 56-62.
4. S.Bhowmik, R.Spee, J.Enslin. Performance optimization
for doubly-fed wind power generation systems. J. IEEE
Trans. Ind. Appl. (1999) 2387-2394．
5. X.G. Zhang, R.Wang and D.G. XU. A dead-beat control
strategy for circuiting-current in parallel connection
systems of three-phase PWM converters, J. Proceedingsof the CSEE, 02(33) (2013) 31-37.
6. Z.J.Fang, S.X. Duan,et al. Formation mechanism andsuppression strategy of prediction control error applied ina battery energy storage inverter, J. Proc. CSEE, 10(30)(2013) 1-9.
7. W. Chen and J. Wu. Adaptive control of ASVG by using
diagonal recurrent neural network, J. Proc. CSU-EPSA,(1999) 33-37.

Fig. 12.Side of the three - phase current imbalance waveform

Fig.10 shows that three-phase current state canachieve the balance when the device is in normal
operation; while the device is disconnected, the state
becomes unbalanced. As shown in Fig.11, after A Phase
is compensated, the phase of the voltage and current is
the same, which achieve a better compensation effect.
From Fig.12, it can be seen that unbalancedness degree
is reduced from 0.28 to 0.06 for the three phase currentof the load, which achieve the state's demand forunbalancedness degree.
The experimental results show that the SVG control
method based on complex filter and PCI control method© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

727

Dynamics Analysis of Payload On-orbit Catapult Separation Based on ADAMSYi Li, Yingmin Jia*, Jianheng Ling

The Seventh Research Division and the Center for Information and Control
School of Automation Science and Electrical Engineering
Beihang University (BUAA), 37 Xueyuan Road, Haidian District
Beijing, 100191, China
E-mail: liyi309@163.com; ymjia@buaa.edu.cn; lingjianheng1987@163.com
www.buaa.edu.cn

Abstract
In this paper, a catapult separation device equipped with guide mechanism is considered, and the corresponding
dynamic model is descripted firstly. Then, employing the dynamic simulation software ADAMS, the virtual
prototype model of the separation device is established, and the whole process of on-orbit separation is simulated.
Finally, the structure parameters of the guide mechanism such as the number of roller group or distance between
groups are studied. The simulation results could provide references for the optimization of the separation device.
Keywords: Guide mechanism, catapult separation, virtual prototype, angular velocity.

force in the process of catapult separation. The
additional torque introduced by the eccentric separation
would affect the attitude or even cause tumbling of the
payload, so it is very important to study the separation
dynamics of eccentric payload. Jeyakumar2 establishedthe dynamic equations for the satellite separation systemusing the helical compression spring mechanism. In
Jiang’s research3, characteristics of the constrained andcentroid biased on-orbit satellite separation wereanalyzed, and then the dynamic model of the separating
motion was formulated. Based on multi-body dynamic
theory, Shen4 built the eccentric separation model, andsimulated the dynamic during the separation by
ADAMS software. Zhang5 built virtual prototype modelof separation mechanism to simulate the whole processof payload on-orbit launch, and analyzed the stability ofon-orbit launch considering the influence of separation
mechanism error. Shu6 focused on the multi-bodydynamics during separation/release, and established the

1. Introduction
On-orbit separation refers that satellite platform
separates payload with a separation device, and the
payload enters its preset orbit successfully1. The
common schemes for on-orbit separation include
pyrotechnic separation, electromagnetic separation andcatapult separation. Pyrotechnic separation has security
problem since the separation process with explosive
material would produce large impact and interference
on onboard devices; electromagnetic separation requires
more power consumption, meanwhile, it's easily
influenced by external magnetic field; however, catapult
separation has the following advantage: simple structure,less power consumption and easy to control, therefore,
the catapult separation is chosen in this paper.
Due to the deviations of compressing springparameters, machining errors or external disturbances,the mass center of the payload is not on the resultant*

Corresponding author© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan728

Yi Li, Yingmin Jia, Jianheng Ling

simulation model based on ADAMS. The above
researches stressed on dynamics analysis of tube-shapeeccentric sub-satellite separation, and analyzed theinfluences of separation mechanism parameters on
satellite platform by employing MATLAB or ADAMS
platform. On basis of the previous studies, in this paper,a new type of catapult separation device equipped withguide mechanism is presented, and the influences of theguide mechanism's parameters on the payload attitude
are analyzed in ADAMS/View.

2.2. Dynamics description
During the process of catapult separation, the separation
impulse is provided by release of the energy stored in
the compressing springs. Based on the structure of theseparation device, the force of the payload include
spring force and the contact force produced by therollers. By means of multi-body dynamics theory, theforce and the moment of payload are analyzed andcalculated as follows:nF

2. Problem Formulation

i 1

2.1. Description of catapult separation device

Rbr (

M

As shown in Fig. 1, the catapult separation device
includes driving motors, differential gear mechanism,
four compression springs, the guide mechanism, a carused for loading the payload, and so on, the guidemechanism consisting of rollers plays a crucial role inrestraining angular velocity of the payload. According
to the general design scheme, the whole separation
process can be divided into three phases:(i) the payload moves together with the car, in otherwords, they maintain relative static,(ii) the car collides with the buffers in the end of wholedistance, the payload moves along the guidemechanism under the effect of inertial force, andthe payload is still not out of the car in this phase,(iii) the payload is completely out of the guidemechanism’s constraints, that means the car and the
payload move respectively.

i 1

i 1

rSj

k

Fri

n

k
j 1

n

Fli

j 1

FSj ,

(1)

n

rli

Fli

i 1

rri

Fri

(2)

FSj ).

where Fli , Fri are the contact forces of the i th group of
the rollers, and FSj is the spring force of the j th spring.
Rbr is transformation matrix, from inertia reference
coordinate system to the payload coordinate system. rli ,rri and rSj are the position vectors mapped in inertia
reference coordinate system.
Based on Hertz contact theory, the nonlinear spring
damping contact model is adopted to model the contact
force which can be expressed as
Fli

FNi

F0

K li C li li

Fsi

FNi

,

(3)

where FNi represents the normal pressure, Fsi
represents the friction force. F0 is the preload, C andK represent the damping factor and equivalent system
stiffness, respectively,
denotes the normallideformation andsatisfiessign(v)f,v

step ( v , v f , f , vs ,step (v, vs , s , vs ,vf
) sign(v) , vsvss ), vsv

v f (4)

where f states the dynamic frictional coefficient, ands represents the static frictional coefficient. v is thevelocity of the payload, v f denotes the dynamic slip
velocity while vs denotes the static slip velocity.

Fig. 1. Schematic drawing of the catapult separation device© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

729

Dynamics Analysis of Payload

analyzing the influence rules of the parameters under
different conditions, we can optimize and redesign theguide mechanism.
We assume that the initial angular velocity of thesatellite platform is zero, and the orbit motion of the
payload is neglected in the process. The mass of theplatform is M 500kg , the mass of the payload is
m 5kg ; the stiffness coefficients of four compression
springs are set as k1 2.43N / mm , k2 2.46 N / mm
k3 2.47 N / mm k4 2.48 N / mm respectively; thespring stroke length is x 90 mm (the velocity of the
car will be 4m / s ), the gravity is neglected; and thesimulation time is 0.1s , the step is 200, the lock isreleased at t 0.034 s , otherwise.
The guide mechanism primitively has two groups ofrollers, and now we add one group of rollers between
the two groups on the premise of little change of theseparation velocity as shown in Fig. 3, the distance
between the added group and the first group is set as
l 75mm .

3. Virtual Prototype Modeling
Using the ADAMS software, the virtual prototype
model including the satellite platform, the payload andthe catapult separation device is established as shown in
Fig. 2. In general, there is a constraint relationshipknown as joint between two components, this model has
three kinds: Fixed joint, Revolute joint, andTranslational joint.

Fig. 2.Virtual prototype model of separation system

Then, we should define force, torque, and contact
for the virtual prototype model. The defined contacts aredemonstrated in Table 1, calculating for which there are
two methods: restitution method and impact method.
According to the dynamics analysis in chapter 2, we
adopt the impact method in this model.

lTable 1. Contacts of the model
Schedule

Rigid 1

Rigid 2

Contact 1
Contact 2
Contact 3
Contact 4
Contact 5
Contact 6carcarroller_11
roller_12
roller_21
roller_22

payloadbufferpayloadpayloadpayloadpayloadadded group

Fig. 3. Installation of the groups of rollers

Fig. 4 shows the final roll angular velocity (rotation
about x axis) decreases from 0.849°/s to 0.553°/s, theyaw angular velocity (rotation about y axis) decreases
from 1.063°/s to 0.870°/s, and the pitch angular velocity(rotation about z axis) decreases from0.594°/s to 0.4°/s
after adding one group of rollers. So adding groups ofrollers on the premise of little change of the separation
velocity could restrain the separation angular velocity.
Considering the guide mechanism has three groupsof rollers, we try to adjust the distance between the
middle group and the first group, that is, the distance isimplemented at l1 50mm , l2 35mm , respectively.
The simulation results are illustrated in Fig. 5.

4. Simulation
The payload’s angular velocity is the main target ofmeasuring the stability of on-orbit catapult separation.
Many factors influence the separation angular velocity,
including various structure parameters of the device andthe motion of the satellite platform. Here, the structure
parameters of the guide mechanism are discussed. By© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan730

Yi Li, Yingmin Jia, Jianheng Ling

When the distance decreases to 50 mm, the angularvelocities including roll, yaw and pitch decrease as well(the roll angular velocity decreases to 0.280°/s, the yaw
angular velocity decreases to 0.353°/s and the rollangular velocity decreases to 0.327°/s); however, if the
distance continues to decrease to 35 mm, the angularvelocities will increase instead.
5. Conclusion
In this paper, a new type of catapult separation device
equipped with guide mechanism is considered; using the
ADAMS software, the virtual prototype model isestablished. The simulation results show that adding one
group of rollers and adjusting it to the center of theother two groups could restrain the angular velocity
effectively.

Fig. 4. The angular velocity after adding one group of rollers

Acknowledgements
This work was supported by the National Basic
Research Program of China (973 Program:
2012CB821200, 2012CB821201), the NSFC (61327807,
61520106010, 61134005, 61521091).
References
1. J. P. Yuan, et al. Spacecraft Counter Orbit Dynamics(China Astronautic Publishing House, 2014).
2. Jeyakumar. D, B. N. Rao. Dynamics of satellite
separation system, J. Sound Vib., 297(1) (2006) 444-455.
3. C. Jiang, Z. K. Wang, L. Fan, et al. Dynamics analysis of
the constrained and centroid biased on-orbit satellite
separation (in Chinese), Flight Dyn., 28(1) (2010) 76-79.
4. X. F. Shen, Y. Z. Xiao, et al. Dynamic simulation of
eccentric sub-satellite based on Monte Carlo method (inChinese), Aerospace Shanghai, 31(1) (2014) 12-17.
5. L. Zhang, Y. Liao and G. Tang. Dynamic simulation of
payload on-orbit launch (in Chinese), J. Projectiles
Rockets Missiles Guid., 28(4) (2008) 186-189.
6. Y. Shu and Z. Li. Dynamics simulation of on-orbit
release and separation of payload (in Chinese),
Spacecraft Environment Eng., 29(1) (2012) 18-22.
7. Nohmi M, Tanikawa J and Hosoda T. Simulation
analysis of a tethered space robot for space experiment on
sounding rocket by JAXA/ISAS, in Proc. Int. Conf.
Mechatronics and Automation. IEEE, (China, Xi’an,2010) 264-269.
8. Nenchev D N, Nohmi M and Uchiyama M. Tethered
robot casting using a spacecraft-mounted manipulator, J.
Guid. Contr. Dyn., 24(4) (2001) 827-833.
Fig. 5. The angular velocity with different distances© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan731

A Land Testbed for Experimental Research on Autonomous Ship Navigation
Keisuke Watanabe

Department of Navigation and Ocean Engineering, Tokai University, 3-20-1 Orido Shimizu-ku
Shizuoka, Shizuoka, 424-8610, Japan*
Kazumasa Harada

Department of Navigation and Ocean Engineering, Tokai University, 3-20-1 Orido Shimizu-ku
Shizuoka, Shizuoka, 424-8610, Japan†
Koshi Utsunomiya

Department of Navigation and Ocean Engineering, Tokai University, 3-20-1 Orido Shimizu-ku
Shizuoka, Shizuoka, 424-8610, Japan
E-mail: keisuke@tsc.u-tokai.ac.jp
www.u-tokai.ac.jp

Abstract

In the development of automated ship technology, experimental system is essential. The main navigation
system is supposed to consist of GPS, AIS (Automatic Identification System), RADAR and cameras. In
this paper, we introduce the setup of our land based test robot which simulates several essential functions
needed for experiments of our research. In our land test robot, 2 ultrasound distance sensors are used as amodel of GPS, which determine the coordinate of the test robot measuring the distances from 2 walls. A
swinging ultrasound ranging sensor with a servo motor is used to simulate the RADAR. A xbee wirelesscommunication module simulates the AIS. The camera is used for image processing to determine thecontrol algorithm to avoid the collision from other moving obstacles. In this paper, we describe our test
robot’s system assembly and several experiments to confirm functions of our concepts for our future
experimental research on autonomous ship navigation.
Keywords: autonomous ship navigation, land based test robot, unmanned vehicle control

autonomous navigation technology is one of the mostimportant key aspects. The research on automated shipnavigation and control is not so new1,2, which were
investigated intensively during 1980s in accordance withthe advancement of modern control theories as well as
the navigation tools like GPS etc.
Unmanned and autonomous navigation have to beelevating to a new level of research these days. At present,
there published many papers related to the autonomous

1. Introduction
The unmanned vehicle technology has been investigatedintensively for these years from a car to an aircraft andthe development of unmanned ship is also being
investigated by major shipping companies from the view
point of being able to use them on commercial base.
Although there are many aspects to be investigated forthe development of unmanned ship technology,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

732

Keisuke Watanabe, Kazumasa Harada, Koshi Utsunomiya

Table 1. Correspondence of ship and land test
robot modules

navigation3,4, most of them are based on the computer
simulations or commercial simulators. In thesesimulations, taking into account all uncertainties which
occur during the maneuvering of the ship is almost
impossible.
The importance of experimental research on this field
becomes more important than before because theuncertain incidents like malfunction of sensor ordynamics modeling errors are possible to cause major
fatal accidents such as ship collision. So we need anexperimental system which can simulate various realsituations on the sea. In the near future, we will develop
a tank test system using plural ship models.
As the cost of tank tests is not negligible, wedeveloped a land robot whose systems are the same
components with the tank test system as the first step ofour research. In this paper, we introduce the concept ofour test system and the modules we developed. We also
explain the collision avoidance experiment which is
introduced to check the assembled modules.

Real ship
RADAR

Test robot
Ultrasound ranging sensor
With a servo motor

XBee

Wireless communication for
AIS emuration

GPS
Camera2 ultrasound ranging sensors
USB Camera

Ultrasoundranging sensors

CameraXBee

2. Components of the land robot system
The essential navigation systems are GPS, AIS, RADAR,
and cameras. Every ship has GPSs to measure theirglobal position. AIS is used to send the ship status toother ships. The ship status includes the position of theship, velocity vector, ship name, ship size, ship’s countryetc. On the sea, we can check the AIS information on themonitor of RADAR. RADAR is used to detect obstaclesor ships around us to avoid collision. We use not only
RADAR but also AIS information to avoid collision. If aship is entering our possible collision zone, we
communicate each other and one of us must change thespeed and course. The velocity vector of AIS is very
important information to calculate the relative position
between ships in the near sea area.
To emulate these systems, we set up the modules asshown in Table 1 and Fig.1. We will explain each module.

Fig. 1. Schematic view of the land test robot

2.2. Ultrasound collision avoidance moduleThis module emulates the RADAR of a ship. An
ultrasound ranging sensor is attached to the front of therobot with a servo motor. By swinging it with a servo, weemulated a kind of front RADAR. Fig.2 shows acalibration result to verify this function. In Fig.2, upper
image shows 2 obstacles, which are recognizedsuccessfully as shown in the lower graph.

2.1. Ultrasound coordinate detection module

2.3. Xbee module

In our land or tank experiments, we cannot use GPS
because the experiments are in the room. So we decided
to use ultrasound raging sensors to detect the distancefrom 2 walls to calculate the coordinate in the test field.

AIS (Automatic Identification System) is very important
for autonomous navigation algorithm. We implemented
Xbee Series1 wireless serial communication module toemulate the AIS. A robot can broadcast its position andvelocity to other robots as well as it can receive theseinformation from others.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan733

A Land Testbed for2.4. Camera image processing module
In the manned ship, the watch is very important. As forthe unmanned ship, the image processing module will be
used as the watch. We implemented a USB camera andare developing the collision avoidance program using
OpenCV.
3.

Experiments of the total system
No obstacle

The total system architecture is shown in Fig.3. The
hardware modules are controlled by Arduinos. The main
computer of the robot is a Raspberry Pi and the robot canbe controlled or monitored through Wi-Fi from outside
PC which emulates the computer of the control room
where all the ships are monitored in land.

Obstacles
Detected

3.1. The mission of experiment
To develop the program which integrates each module,
we made the mission of experiment as shown in Fig.4.
At first, the robot moves forward using its compass
and ultrasound ranging RADAR. If the robot finds an
obstacle by this RADAR and the distance from the
obstacle becomes less than 10cm, the control modechanges to camera sensing mode. In the camera sensing
mode, the camera image is processed to find a red object
whose area is greater than a specific threshold value. If a
red obstacle is confirmed, then the robot moves backward
and turns 90 degrees left to avoid the collision to the
obstacle. Then it keeps moving forward until it finds awall and stops if the distance to the wall becomes 10cm.

Fig. 2. RADAR emulated by a front ultrasound ranging sensor

3.2. The actuators
We will change the platform from this land based robot
to a ship model in the near future. In this research, we
hired crawlers as the actuator of the robot, considering its
slow response in changing its heading angle. Usually, aship’s yaw motion response is not quick. That’s why wedidn’t hire normal wheels.
Fig. 3. The architecture of the robot

3.3. The experimental result
The experimental field is as shown in Fig.5. The field iscovered by 4 walls whose side length is around 1m, andthe coordinate of the robot is measured by using 2 ultrasound ranging sonars which are attached to the left
direction and the backward direction.

All the data were successfully transmitted from therobot to the monitoring PC by a Xbee module whichemulates the AIS. We can also monitor the inner state of
the robot and communicate through Wi-Fi connection.
Fig.6 shows the coordinate of the robot. The startpoint was around (87, 17). Then it moves parallel to the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan734

Keisuke Watanabe, Kazumasa Harada, Koshi Utsunomiya

Fig. 5. The experimental field and the robot
Fig. 4. Mission in experimentY-axis and found the red obstacle at around (80, 60). So
it stopped and turned left then moved along parallel to theX-axis until it stopped around (12, 45), where the front
sensor detected the wall to stop from 10cm distance.
4. Summary and future work
In this study, we developed a land robot which is
supposed to emulate a ship model for our experimental
research on autonomous ship navigation technology.
The essential navigation modules are GPS, AIS,RADAR and cameras. We emulated these modules byapplying ultrasound ranging sensors, Xbee module, andUSB cameras.
To confirm whether our system can work as we
intended, we experimented the obstacle detection andavoidance test. The robot moves forward with measuring
its coordinate and finding obstacles by the RADAR.
When the robot finds an obstacle in front of it, then theimage processing module is activated and if it finds a red
obstacle, it turns 90 degrees left to avoid collision. Xbee
module data communication as AIS emulation also
succeeded. Although we still have several subjects to be
improved, from the experimental result, all the modules
we implemented functioned well as we intended.
We will fabricate more robots to develop the software
for autonomous navigation algorithm. After brushing up
the algorithm, we will transplant the system on several
ship models to experiment in the tank.

Fig. 6. The coordinate result of the experiment

References
1. K. Hasegawa, A. Kouzuki et al., Ship Auto-navigation
Fuzzy Expert System (SAFES), Proc. of the Society ofNaval Architecture Japan, vol.166 (1989), pp.445-452.
2. T. Koyama and K. Shimizu, Fundamentals of Automatic
Harbour Maneuvering, 7th SCSS, vol 3 (1984), pp.123-145.
3. Lloyds Register, Cyber-enabled ships: Deploying
information and communications technology in shipping –Lloyds Register's approach to assurance. (2016).
4. Kongsberg, Automated Ships Ltd. and KONGSBERG tobuild first unmanned and fully-automated vessel foroffshore operations. (2016) [Online]
5. Levander, O., Autonomous Ships on the High Seas. IEEE
Spectrum (2017), Issue February.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

735

Seafloor Image Color Enhancement Method based on Retinex modeland Experiment Report in the undersea environment
Jonghyun Ahn

Dept. of Human Intelligence Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, Japan
E-mail: ahn@brain.kyutech.ac.jp
Shinsuke Yasukawa

Institute of Industrial Science, University of Tokyo
4-6-1 Komaba Meguro, Tokyo, Japan
E-mail: yskw@iis.u-tokyo.ac.jp
Yuya Nishida

Dept. of Human Intelligence Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, Japan
E-mail: ynishida@lsse.kyutech.ac.jp
Takashi Sonoda

Dept. of Human Intelligence Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, Japan
E-mail: t-sonoda@brain.kyutech.ac.jp
Keisuke Watanabe

Dept. of Navigation and Ocean Engineering, Tokai University
3-20-1 Orido Shimizu, Shizuoka, Shizuoka, Japan
E-mail: keisukejapan@gmail.com
Kazuo Ishii

Dept. of Human Intelligence Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, Japan
E-mail: ishii@brain.kyutech.ac.jp

Abstract
Underwater imaging is an important part to seafloor observation by underwater robot. Underwater robots provide
many images, which contain seafloor environment, and theses images contribute to various fields such as marine
biology, marine geology and fishery science. However, underwater images has different properties, such as uneven
illumination and light attenuation, compared to images which taken in air, and these properties impede image
checking by researchers. In this paper we propose an image enhancement based on scale-gain Retinex, and the
proposed method is experimented on the Suruga-bay in Japan.
Keywords: underwater imaging, underwater robot vision, seafloor image enhancement, seafloor observation

robots contribute to various fields such as marine
biology, marine geology and fishery science. However,seafloor images have low visibility because of the lightattenuation. The obtained images by underwater robots
have properties such as uneven illumination andattenuated color by the light attenuation. These
problems are shown in Fig.1.

1. Introduction
Underwater Robotics is one of the solutions forobservation sea environment. Recently, various
underwater robots are developed to explore sea, andunderwater robots provide seafloor images toresearchers [1-3]. The obtained images by underwater© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

736

Jonghyun Ahn, Shinsuke Yasukawa, Yuya Nishida, Takashi Sonoda, Keisuke Watanabe, Kazuo Ishii

Obtained image is taken by underwater robot: TUNASAND, and the many red snow crabs are contained inimages. However, main colors of images are blue andgreen because red light is highly attenuated; left side ofimage looks dark because of light source in the intensityof image. These problems show attenuated color anduneven illumination, and theses low visibility images
are difficult to check whole area of seafloor images.
In this paper, we propose a seafloor image
enhancement method, which can improve visibility ofimages, and the proposed method is experimented on
Suruga-bay (Depth: 33m).
2. Proposed Image Enhancement Method
The proposed image enhancement method using
Retinex theory [4,5]. The Retinex theory is a famous
method to improve images against uneven illumination.
In the Retinex theory, one image comprises reflectancefactor and illumination factor. The equation of relation
between reflectance factor and illumination factor is;( , )=( , )∙( , ),(1)

where I denotes an image,is color space such as red
space, green space and blue space in RGB color space,
x,y are coordinates in image, R is reflectance factor and
L is illumination factor from light source. R is the factor
related real color, which is uninfluenced by illumination,of object. The illumination factor is estimated by
Gaussian filter and the equation is;( , )=∙ exp∗ ( , ),

(2)( , )} + .

(3)

Fig.1. Example of underwater imaging problems by robot

restore color. But the each color intensity is normalized
in processed image by Retinex, therefore it is difficult torestore color.
The proposed image enhancement method scale-gain
Retinex, which presented by Kotera [7], is used. The
scale-gain Retinex maintains the color factor in images,
and only uneven illumination is improved. The
estimated illumination in scale-gain Retinex is;

K is a normalization factor andis the standard
deviation of the distribution. I and the estimatedillumination log-transform, and estimated illumination
is subtracted I.

( , ) = {log ( , ) − log( , )=

O is Retinex output and is scaling gain and is offset
parameter.
Recently, underwater image enhancement methods
using Retinex are presented [6]. These methods show
highly enhanced visibility of seafloor images. However,Retinex theory is not considered light attenuation inunderwater environment, therefore the colors is notexactly same. Therefore, the underwater image
enhancement method based on Retinex is necessary to∗ ( , ),

(4)

.

(5)

∙ exp

Where Y is luminance of I. The scale-gain Retinex is
don’t use log-transform and it is;( , )=

( . )

( , )

A is scaling gain.
As the process results of Retinex and scale-gain Retinex
is shown in Fig.2.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

737

Seafloor Image Color Enhancement

The both images show even illumination compared to
raw image in Fig.1. Processed image by Retinex shownormalized RGB color (used parameter :64, :256,: 128); processed image by scale-gain Retinex show
maintained RGB balance (used parameter :64, :80).
The proposed method combines the scale-gain Retinex
with color attenuation model in water. In the proposed
method scaling gain is;( , )=(∙ ( , )∙ )∙ ( , )∙.

(6)

Where P is proposed method output, C is camera gain
parameter of each color, a is attenuation constant and d
is distance between seafloor and camera.
The proposed method based on the Beer-Lambert law[8] which attenuation model. The process result P isFig.3. Process results by proposed image enhancement

shown in Fig.3.
We can check natural color in image in the Fig.3
compared to Fig.2, and even illumination.
3. Experiment in Suruga-bay
The sea experiment presents to check proposed image
enhancement method. The concept of experiment isshown in Fig.4.
In the experiment, we sink the camera system whichcontains one camera and 2 light sources (LED array,
40W) and then the operator check the seafloor image
via the cable on real-time. The experiment equipment,
which contain camera and light, bring back seafloor
sand to compare seafloor color values of photographed
image and color value of sand in air condition.
The detail of system is shown in Fig.5. For thecommunication between ship and camera system, PowerLine communication (PLC) is used, and LED1, 2
controlled by CPU via Arduino.
The photographed image, the sand image, whichbrings from seafloor, in air condition and processed
result by proposed method are shown in Fig.6.
As the results, the averages each color value of sandimage in air are red 47, green 49 and blue 44; theaverages each color value of processed image by
proposed method are red 70, green 67, and blue 68. The
processed value show more high RGB values, but RGB
balance show similar values compared to seafloor image.

Fig.2. Process results by Retinex and scale-gain Retinex© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

738

Jonghyun Ahn, Shinsuke Yasukawa, Yuya Nishida, Takashi Sonoda, Keisuke Watanabe, Kazuo Ishii

Ship
Cable

50m

Light
sourceCamera1.5m
Fig.4. Concept of sea experiment
Ship
PC

Ethernet
Serial

PLC

AC
24V
12V
5V

Hull
PLC

HUB

CameraAC/DC

Fig.6. Process results by proposed image enhancement
LED 2

4. E. H. Land, J. J. McCann. "Lightness and retinex theory,"
Journal of the Optical Society of America, Vol. 61, pp.111, 1971.
5. Z. Rahman, D. J. Jobson, G. A. Woodell, “Retinex
processing for automatic image enhancement," Electronic
Imaging 2002, International Society for Optics andPhotonics, pp.390-401, 2002.
6. J. Ahn, et al. "Enhancement of deep-sea floor images
obtained by an underwater vehicle and its evaluation by
crab recognition." Journal of Marine Science andTechnology, vol.22, pp1-13, 2017
7. H. Kotera and M. Fujita, “Appearance improvement ofcolor image by adaptive scale-gain Retinex model,” 10thColor Imaging Conference, pp.166-171, 2002.
8. H.R. Gordon, "Can the lambert-beer law be applied to the
diffuse attenuation coefficient of ocean water?."
Limnology and Oceanography, vol. 34, no. 8, pp. 13891409,1989.

LED 1
CPU
ArduinoDepth sensor

LED controller

Fig.5. System of the experiment equipment

4. Conclusion
In this paper, we proposed image enhancement method
using scale-gain Retinex and light attenuation model. As
the result of experiment, we get similar RGB balancevalues; this result show possibility of color restorationof seafloor image. As the future work, we will apply
proposed method to various colors image, and it ismaking easy to evaluation of color restoration.
References
1. T. Ura, "Observation of Deep Seafloor by Autonomous
Underwater Vehicle." Indian Journal of Geo-Marine
Sciences, vol.42, pp.1028-1033, 2013.
2. T. Nakatani, et al. "AUV TUNA-SAND and its
Exploration of hydrothermal vents at Kagoshima Bay."
Oceans 2008-MTS/IEEE Kobe , 2008.
3. K. Ishii, et al. "A New Tool to Access Deep-Sea Floor“Sampling-AUV”. ", International Conference on
Artificial Life and Robotics 2017, pp.301-303, 2017.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

739

Simulation of Horizontal Vibration Suppression of a Suspended Structure for Seabed Mining
Keisuke Watanabe

Department of Navigation and Ocean Engineering, Tokai University, 3-20-1 Orido Shimizu-ku
Shizuoka, Shizuoka, 424-8610, Japan
Kazuo Ishii

Department of Intelligence Systems, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku
Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: keisuke@tsc.u-tokai.ac.jp, ishii@brain.kyutech.ac.jp
www.u-tokai.ac.jp

Abstract

In a project of subsea mining in the near future, many work class robots such as crawlers, drillingmachines, or ore collectors will be installed on the seabed. These robots are heavy but they must be landedon the seabed very softly to avoid impact damage from the seabed. So the installation method is very
important and effective installation method should be investigated. In this paper, we examined vibration
motion of a suspended heavy machine by a crane vessel through dynamics modeling and simulation in
the horizontal plane. Descending speed of the machine affects the amplitude of the vibration, that is, if
the descending speed is higher, larger amplitude will be observed. To suppress this vibration, we
examined a velocity feedback control by attaching a thruster to the machine. From simulation results, wefound horizontal vibration will be effectively suppressed by using simple feedback control.
Keywords: subsea mining, installation of subsea machines, control of suspended structure

through the experience of many ultradeep projects
worldwide. For example, Subsea production system(SPS), also named as ‘subsea tie-back’ (SSTB) or subsea,
consists of subsea wells, subsea processing facilities,
subsea flowlines, subsea controls, and surface Host, hascapability to produce oil & gas without an offshore
structure straightly up the subsea wells. A typical Subsea
production system is shown as following Fig.1.
Subsea processing facilities, including manifolds,
metering, separators, boosters, and subsea pumps e.g.
extend the capabilities of subsea system, allows deeper
water exploration, more wells producing and longer
distance tie-back.
Subsea flowlines and risers, to transport the
production flow to downstream process facility likesurface host platforms, could be rigid pipes or flexible
pipes. The water depth of the development becomes

1. Introduction
The mineral resource on or under the seabed has been
told it will improve Japanese self-sufficiency of mineral
resource. However, there had been no project realized
due to two reasons. One reason is a technical issue that
there hasn’t been mature technology to develop theunderwater deposit. The other reason is the project hasn’t
been profitable because subsea mining needs much moreamount of investment than the land based project. Only
recently these situations are changing better because
many subsea oil production projects made technologies
advanced and cheaper. So the subsea mining project will
be possible to realize in the near future. As the offshore
oil development has been flourished for these 20 years,
the subsea production system have been fully matured© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

740

Keisuke Watanabe, Kazuo Ishii

deeper, and in some field like offshore Brazil pre-salt, thetarget water depth is generally more than 2000m.
In these projects, as the oil price were high, they caninvest much money. But in the subsea mining project, we
can’t expect such investment as oil/gas development, so
we need to investigate how we can minimize the total
project cost more strictly than the oil/gas field1,2.
In this research, we considered a possibility of using
active control technique for deep water installation using
a crane vessel3. The module or structure consists of apendulum system whose arm length is becoming longer
as the suspended structure is going down to the sea bed.
An initial offset angle from the vertical axis will cause avibration motion whose horizontal amplitude develops
according to time during installation. Two dimensional
equation of motion is derived and several simulation
results will be shown for the cases of both free suspended
situation and actively controlled situation. The control
algorithm is LQR and a ROV is assumed attached to thestructure as an actuator to suppress the vibratory motion
to install the structure on the desired target point on thesea bed.

Fig. 1. Schematic View of Subsea Production System.
(1) Subsea wellhead & trees, (2) Subsea manifold, (3) Subsea
umbilical termination units, (4) Subsea flowlines, (5)Umbilicals & Risers, (6) HOST

2. Derivation of the equation of motion
The coordinate system is shown in Fig.2. The module
structure is supposed to be suspended from a crane on thevessel. The offset angle from the vertical axis is aparameter of the nonlinear pendulum motion of thestructure.
It is assumed that the crane cable is paid out fromthe initial length l0[m] with the velocity of v [m/s]. The
initial length corresponds to the height from the initial
submerged point of the structure to the suspended pointof the top of the crane arm. In this research, the force
displacement excitation on the top of the crane arm due
to wave induced motion of the vessel was not taken into
consideration, the excitation is assumed forced
oscillation by harmonic vibration of the angle of the cable.
The Lagrangian is, using the cable length l(t),
̇ +
̇ +
cos
(1)
L=T−U=

Fig. 2. Coordinate System of the Suspendee Structure.

, where, f is the control force.
3.

The analytical solution of the next free oscillatory motion
becomes Bessel function in accordance with a parameter
lθ.
(1 +

Considering the added mass and fluid resistanceforce, the equation of motion of the pendulum system is,
(1 +
+

)

sin

=

̈ + 2

+

1
2
̇

Simulation for free suspended situtaion

)

̈ + 2

+

1
2

̇

(3)
We made a simulation program using MATLAB withcoding 4th order Runge-Kutta ODE solver which can beconsidered the effect of feedback force.
+

(2)

sin

=0

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

741

Simulation of Horizontal Vibration

The simulation is using next parameters. The
structure size is 5m wide 5m long 4m height. The mass is
assumed 50tf. The water depth is assumed as 2500m.The
initial offset angle is 20 deg. The paid out rate of the cableare 2 cases, which is 1m/s and 0.5 m/s. Added masscoefficient is assumed 1.0, whereas drag force coefficientis assumed as 3.0.
The simulation result of the paid out rate of the cable
is 1.0m/s is shown in Fig.3. The horizontal displacement
which means the fluctuation from the desired installation
point is shown in Fig.4. As Fig.3 shows, the offset angle
is decreasing according to time, which is due to damping
effect of fluid resistance force. However, as the crane
cable length is increasing, the horizontal offset from thedesired point is developing. We assumed the dragcoefficient higher assuming that some damping device is
attached to the structure. But from the result shown in
Fig.4, it seems very difficult to reduce the offset vibration
amplitude only using passive fluid damping force.
To see the effect of cable paid out rate, we examined
another simulation where the cable rate is 0.5m/s. If thepaid out rate is slower, we can expect the offset becomesslower, whereas installation time becomes longer, which
will cause higher operational cost.
Fig.5 shows the offset angle fluctuation when thecable speed is 0.5m/s. Compared to the result of v=1.0m/s,
this result shows if the cable paid out rate is smaller, thenthe fluctuation amplitude is possible to be suppressed.
Thus if we need to put the descending speed higher toreduce the installation time, it might be possible that thesuspended structure’s oscillatory motion becomes larger,
which will cause degradation of installation accuracy.
As Fig.5 shows, if the descending velocity is set tolower, the horizontal displacement amplitude is stillaround 40m due to very long cable length.
Fig.6 shows the amplitude comparison between
different cable paid out rate and the time taken from sea
surface to sea bed. There is a trade-off relation between
the installation speed and the natural offset amplitude
around the desired installation point. These results show
some new technique are to be deployed for more costeffective and precise, safer installation.

Fig. 3. Angle and Angular Velocity of the Pendulum Motion v=1m/s.

Fig. 4. Horizontal Amplitude when v=1m/s

4. Simulation of LQR controlled installation
We considered LQR optimal control algorithm in thisresearch. Because the EOM is non-linear, linearized
EOM is used as a nominal model for control gain
computation.

Fig. 5. Angle and Angular Velocity of the Pendulum Motion
v=0.5m/s.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

742

Keisuke Watanabe, Kazuo Ishii

Fig. 6. Comparison of Horizontal Displacement

Fig. 7. Horizontal displacement in controlled case

The cable length is time variant, however, in thenominal model for controller design, we fixed the cable
length 1000m as the representative value for linearization.
The simulation is done by using non-linear EOM, where
the control force is calculated using the linearized LQR
model. As the typical ROV thrust force is less than
7000N, the feedback gain is adjusted the maximum thrustforce becomes less than 7000N.
Fig.7 shows the controlled horizontal displacement
when the descending speed is 1m/s. These results
indicate that the control feedback works very well tosuppress the self-vibration of the structure. The offset
from the desired installation point is only 6m around,which means the very precise installation of the structureon the sea bed can be achievable. Fig.8 shows the control
force result. The control force is less than 7000N andseems feasible using existing ROV as an actuator.

Fig. 8. Control force estimation

References
1. Watanabe Y., Watanabe K, et al., “A Self-Walking
Vertical Mining System Concept Using DTH for Seafloor
Mining and Its Preliminary Design”, OTC-27003-MS,
Offshore Technology Conference (2016)
2. Watanabe K., Watanabe Y, et al. “A Self-Walking Vertical
Mining System Concept Using DTH”, OTC-25825-MS,
Offshore Technology Conference (2015)
3. Watnabe K., “A Possibility of Active Controlled
Installation for Deep Water Subsea Production Modules”,
IEEE, ICETET2012,(2012)

5. Summary
We investigated the possibility of applying active control
to installation of subsea modules supposing subsea
mining project in the near future. The suspended
structure consists of a pendulum system with a cranevessel whose oscillation amplitude is affected by thedescending speed of the structure. From our two
dimensional simulation, we found if the descending
speed is faster, then the oscillatory motion near theseabed becomes worse. However, if we deploy a ROV as
an actuator for actively controlled installation, it ispossible to suppress the oscillatory motion effectively.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

743

Analysis of Team Relationship using Self-Organizing Map
for University Volleyball Players
Yasunori Takemura

Nishinippon Institute of Technology, 1-11 Aratsu Kanda
Miyako, Fukuoka 800-0344, Japan*
Kazuya Oda

Kyushu Institute of Technology, 2-4 Hibikino Wakamatsu
Kitakyushu, Fukuoka 808-0896, Japan
Michiyoshi Ono

Nishinippon Institute of Technology, 1-11 Aratsu Kanda
Miyako, Fukuoka 800-0344, Japan†
E-mail:takemura@nishitech.ac.jp, oda-kazuya@edu.brain.kyutech.ac.jp, s148414@nishitech.ac.jp

Abstract
In Japan, sports efforts are actively being carried out to host the 2020 Olympic games. Especially in the field ofsports science, researches on ergonomics, development of sports equipment and pattern recognition technology using
artificial intelligence are actively researched.
In previous research, we developed a clustering algorithm for
positioning adaptation and relationships in team sports using Self-Organizing Maps in university rugby players.
However, I have not yet confirmed whether the developed algorithm can be applied to other team sports. For thisreason, we applied the same algorithm to an university volleyball player. Then, as an algorithm, we verify whether it
can be generally used for team sports.
Keywords: Self-Organizing Maps, pattern recognize, sports science

psychological condition. Similarly, it can be said that thesuitability of an individual to a certain position in theteam affects the team’s performance [13].
In previous research, Barry and Cureton [1], Nicks andFleishman [2], Larson [3], McCloy and Young [4] andothers clustered physical features and conducted factor
analysis in investigating sports performance. In Japan,
Tokunaga studied the diagnosis criteria for athletic
adaptation (i.e., suitability) in sports [5]. These works
showed that physical features are one of the strongest
factors determining athletic adaptation. However,Matsuda [6] showed that an athlete, no matter how good

Introduction
In Japan, sports efforts are actively being carried out tohost the 2020 Olympic Games. Especially in the field ofsports science, researches on ergonomics, developmentof sports equipment and pattern recognition technology
using artificial intelligence are actively researched. The
budget of Japanese sports in 2016 is 32.4 billion yen, andconsidering that the budget for 10 years ago was 19
billion yen, it is doubling momentum.
By the way, victory or defeat in team sports depends oneach player’s individual technique, physical strength, and© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan744

Yasunori Takemura, Kazuya Oda, Michiyoshi Ono

his/her physical features, is not athletically suited to teamsports without having good motivation in terms of setting
goals and training. That is to say, for an individual orteam to be successful, a player needs to have both good
physical features (e.g., techniques, balance, height, andweight) and good psychological features. For example,
Saijo [7] presented the psychological features ofJapanese and New Zealand rugby players.
In this way, the suitability of a player in a certain
position and the relationships between different positionsin team sports are related to physical and psychologicalfeatures. As it stands now, a coach or selector decides theplayer suitability and relationships between positions
him/herself. However, does it follow that good decisions
are made? Previous research has not clarified athlete
adaptation to positions and relationships between
positions considering both physical and psychologicalfeatures[8].
In the present study, we develop a clustering
algorithm for positioning adaptation and relationships inteam sports. We consider the two main types of features,
which are physical and psychological features, andintroduce the concept of using an algorithm to cluster
player features in team sports. In previous works, wedeveloped clustering algorithm using SOM for teamrelationship map [8,9]. Previous work is adapted inuniversity rugby players. However, I have not yet
confirmed whether the developed algorithm can beapplied to another team sports. For this reason, weapplied the same algorithm to our volleyball players
Then, as an algorithm, we verify whether it can begenerally used for team sports.

Describe the physical data and analysis using
statistics
Physical data of members of the N university
volleyball team were recorded in 2016 December. The N
university volleyball team comprised 22 students. There
were 3 RTs, 6 LTs, 7 CNTs, 2 STs and 4 LBRs. Physicalfeatures recorded were height [cm], weight [kg], grasping
power (left hand, right hand) [kg], ante flexion [cm], toe
touch [cm], jumping side to side [times], standing longjump [cm], 9 meters dash (3 round trips) [seconds],
vertical jumping [cm], and back muscle strength [kg].
The data set thus had 11 physical dimensions. Data
showing statistically characteristic results could beshown below. It shows the adaptation result of theposition derived from the data.
Figure 1 shows the height data. The horizontal axis
denotes individual positions. The vertical axis shows theheight [cm].
As you can see from the Fig.1, LT, RT and CNT showthat players with relatively high stature are selected, andST and LBR can be said that it is not necessary to be aplayer with a high height. This shows that the height of
LT, RT and CNT is required as an attack such as attack.
Figure 2 shows the result of jumping side to side.
The vertical axis shows the how many times they can do[times]. The horizontal axis denotes individual positions.
As you can see from the Fig.2,
It shows that only CNT shows a large value. In otherwords, it can be seen by looking at this figure that thereflexes are necessary for the CNT position. In this way,
the suitability of the position can be understood byanalyzing several items statistically, but in many items
there is not a big difference, it is impossible to decidesuitability of the position unconditionally. Also, since thenumber of population is different as statistical data, it isdifficult to find the suitability of the position only bycomparing statistical data unconditionally.

Research plan and method
Selection of sport
We apply a team-sport clustering algorithm tovolleyball. Volleyball is selected because a volleyballteam is one of the most necessary cooperative works andinteraction between a player and teammates.

185.0
180.0
175.0

The text is to be typeset in 10 pt Times Roman, single
spaced with interline spacing of 13 pt. Text area
(excluding running title) is 6.75 inches across and 8.8
inches deep. Final pagination and insertion of running
titles will be done by the publisher, so make sure that nopage numbers are given in your paper and only therunning titles provided in this template (authors’ names
and paper title) are used.

170.0
165.0
160.0
LT

RT

CNT

ST

LBR

Fig.1 Heights of volleyball players (N university 2016 data)© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

745

Analysis of Team Relationshipmethodology of DIPCA.3 is such that each personanswers the 52 questions on a scale of 1 to 5 (1: I don’t
think so, 2: I rarely think so, 3: I sometimes think so, 4: I
often think so, 5: I always think so) within 10 to 15
minutes.
Figure 3 shows the evaluation result of DIPCA.3.
The horizontal axis denotes each evaluation criteria. The
vertical axis shows evaluation value. Each evaluation
maximum value is 20 [point]. The color bars describe theeach position (LT, RT, CNT, ST, and LBR).
Comparing the psychological feature quantities,
we can see that there is little difference between thepositions. On the whole, however, RT knows that athleteswith very psychological characteristics are selected.
Moreover, as a whole, the ability as fighting sprit, selfrealization and motivation to win is high in the team as awhole. Meanwhile, items such as confidence,
deamination, endurance, and judgment need to belowered in capability as whole team members.

68.0
66.0
64.0
62.0
60.0
58.0
LT

RT CNT ST LBR

Fig.2 Jumping side-to-side result of volleyball players(N university 2016 data)

Describe the psychological data and analysis
using statistics
Psychological data of members of N university
volleyball club were recorded in 2016 December. The N
volleyball club in 2017 comprised 22 students (There aresame members as physical data).
Psychological features measured were those of
the Diagnostic Inventory of Psychological Competitive
Ability for Athletes (DIPCA.3) [8]. DIPCA.3 measures
12 types of psychological ability in a 48-item
questionnaire. DIPCA.3 is often used before mental
training, because it reveals athletic strengths andweaknesses. The DIPCA.3 check sheet consists of 48
questions that measure psychological ability and fourquestions that measure reliability. These questions havealready been analyzed by good–poor analysis (G-P
analysis), and the answers provide 12 criteria relating tofive factors. The factors are motivation in sport, mental
stability and concentration, confidence, operation
capability and cooperativeness. Additionally, wemeasure the reliability of the answers by comparing
answers to similar questions. To examine these factors indetail, 12 criteria are described. Motivation in sportconsists of four criteria: endurance, fighting spirit, selfrealization motivation and motivation to win. Mental
stability and concentration consist of three criteria: theability to relax, capacity to concentrate and self-control.
Confidence consists of two criteria: determination andconfidence. Operation capability consists of two criteria:predictive capability and judgment. Cooperativeness has
only one criterion, which is simply described ascooperativeness. DIPCA.3 provides a total score (ability).
Each criterion is scored, and the score of each factor isthe sum of scores for the related criteria. The total scoreis calculated by summing the score for each factor. Table
I gives the total scores, which are classified into fivelevels. A classification of 1 indicates a very low overallmental ability and a classification of 5 indicates a very
high ability. Each mass describes the total score. The

20

LT

RT

CNT

ST

LBR

18
16
14
12
10
8
6
4

Fig.3

Psychological Result using DIPCA.3(N university 2017)

Experimental Result
To evaluate the performance of the SOM algorithm,
individually, at first, only x input data sets (only physicaldata sets) are used same as previous works[8]. Figure 4
shows the feature maps used for the 2017 physical data
sets. Each lattice cell describes a unit. If a unit is the
BMU for a data set, it is labeled as such as 1 line: numberof player and 2nd line: position). The color of the latticeindicates the Euclidian distance to the neighboring unit
(only fig.4) and figs.5 and 6 are described vector values.
For example, blue indicates that the neighboring unit isnearby; red indicates that it is far away (Figs.5 and 6 arehigh score), and green indicates that it is at mid-distance(Figs. 5 and 6 are low score).
Fig.4 shows the feature map of physical data.
For example, on the left side of the feature map, attackrole people is gathered. On the other hand, right side is© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

746
1

Yasunori Takemura, Kazuya Oda, Michiyoshi Onogathered to deffensive positon people and ST position.
Figure 5 shows the same feature maps, however, thecolor label describe about the height. Looking at thisfigure, high-height (red lebeled) players are gathered on
the left side, and players with relatively low height aregathered on the right side. That is, the same result asstatistical data can be obtained also in the feature map
using SOM. Next, Fig. 9 shows the results of coloring theresults of jumps side to side. The jumps side to side does
not have a relatively large difference. However, it turns
out that the CNT person shows a relatively highnumerical value (green). In this way, similar results tostatistics can be found in SOM feature map. However, thesuitability of the position within the team was not able toobtain distinctive results even if examining other items.
Fig.5 the feature maps of physical data sets
which is colored by jump side to side

Fig.4 the feature maps which is inputted physicaldata sets
Fig.6 the feature map of psychological data sets
resultSecond, we input y data (psychology data sets).
Figure 6 shows the feature map which is inputted
psychological data sets. Each lattice cell describes a unit.
If a unit is the BMU for a data set, it is labeled as such as
1st line: number of player and position and 2nd line: total
score evaluation (1: low level to 5: high level using
DIPCA.3 statistics result). In this figure, right up side isgathered to low-level players. On the other hand, left
bottom side is gathered to high-level people. Therefore,
players in the upper right corner of the map are mentally
immature players and need mental training. On thecontrary, many of the players belonging to the lower leftare elected regular members, and the director also knowsthat they consider mental reach. In addition, because
mental elements are still being analyzed, we will explain
the details in the conference.

Fig.5 the feature maps of physical data sets
which is colored by height data

CONCLUSIONS© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

747

Analysis of Team RelationshipIn this paper, we adopt team relationship clustering
algorithm using SOM for volleyball team players. In
previous works, we adopt team relationship clustering
algorithm using SOM for university rugby players[13,14]. However, I have not yet confirmed whether thedeveloped algorithm can be applied to another teamsports. For this reason, we applied the samealgorithm to our volleyball players.
In previous research [8,9], We succeeded in electing
the position of the rugby players using the phisical datastes. From the psychological data sets, we were able
to find out
hat kind of people the regular members
are. In addition, if two data groups are usedcomprehensively, the suitability of positioning and theselection of regular members were possible. In this paper,
we confirmed whether we can obtain the sameknowledge as statistical data by using physical data groupaccording to the method. We also examined whether
statistically similar results can be obtained forpsychological data group as well. In the future, we needto analyze the suitability of the position and therelationship with the company. We also plan to analyze
integrated physical data group and psychological datagroup.
In addition, as a future task, it is necessary toconsider a method for general use because there is adifference in results, compared with the result of theclustering in the rugby player performed in the preceding
research [13,14] and the result of the volleyball player.

8. O.Saijo, K.Suda, et., al., “On the Psychological Aptitude of
Rugby Foot Ball Players –From Comparison New Zealand
University Selected Team with Nippon College ofPhysical Education Team –“, Journal of Nippon College ofPhysical Education, Vol22-2, pp.135-138, 1993
9 Y.Takemura, M.Yokoyama, S.Omori, R.Shimosaka,“Evaluation and Analysis of Relationship between Rolesand Individual’s Role Adaptation in Team Sports using
SOM ~1st Report Data Analysis and Introduction ofTheory”, In: Cho Y., Matson E. (eds) Soft Computing
in Artificial Intelligence. Advances in Intelligent
Systems and Computing, vol 270. Springer, Cham
10 Y.Takemura, “Development of SOM algorithm for
Relationship between Roles and Individual’s Role in
Rugby ~2nd Report : University Rugby teams analysis
using physical data and Psychological data”,
International conference on Artificial life and Robotics(ICAROB2017), pp.412-415, Miyazaki, 2017

Acknowledgement
This work was supported by a Grant-in-Aid forYoung Scientists (B) 15K21570 from JSPS KAKENHI.
References
1.
2.

3.
4.
5.
7.

Barry, A.J. and T.K. cureton, “Factor Analysis of Physiqueand Performance”, Res. Quart., 23-3:100 ~ 108, 1961
Nick, D.C. and E.A. Fleishman, “What Do Physical
Adaptation Test Measure?,”, A Review of Factor Analysis
Studies, Educational and Psychological Measurement, 221:77-95, 1962
Larson, L. A., “A Factor Analysis of Motor Ability
Bariables and Tests for College Men, “, Res. Quart., 12:499-517, 1941
Mcloy, C.H. and R.D. Young, “ Test and Meassurement in
Health and Physical Education”, New York, ApplentonCentury-Crofts, Inc., pp.51-65, 1954
Tokunaga, “Factor Analysis about sports adaptation”,
Japan Journal of Physical Education, Health and Sport
Sciences, Vol.22-2, pp.71-80,1977
Matsuda, R.N. Singer, “Motor learning and humanperformance”, ISBN978-4-469-26119-6, Taishukanshoren, 1986, In Japanese© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

748
1

Optimization for Line of Cars Manufacturing Plant using Constrained Genetic Algorithm
using Constrained Genetic Algorithm
Keiji Kamei*

Department of Production Systems, Nishinippon Institute of Technology, 1-11 Aratsu
Kanda, Miyakogun, Fukuoka 800-0394, Japan†
Takafumi Arai

Nissan Motor Kyushu Co. Ltd., 1-3 Shinhama-cho
Kanda, Miyakogun, Fukuoka 800-0395, Japan
E-mail: kamei @nishitech.ac.jp
www.nishitech.ac.jp/~kamei/

Abstract
Recently, improvement of production efficiency on cars manufacturers is required. However, that improvements
under existing circumstances are depending on experience and intuition by workers. We propose to objectively andefficiently improve a production line based on a GA. The difficulty of applying a GA is the number of racks andboxes is predetermined, and so we apply constrained GA. The results of simulation for virtual production line showthat our proposal succeeded in reducing about 10 seconds per car compared with random positioning.
Keywords: Constrained Genetic Algorithm, Plant Optimization, Industrial Application, Car manufacturing

area is that the number of racks to put assembly boxes
and that boxes is predetermined.
Our optimization problem is similar to several path
planning problems such as “Traveling Salesman Problem(TSP),” “Minimum Spanning Tree problem(MST).”
Workers pick up assemblies while they goes back andforth between racks and an AGV. Accordingly, our
optimization problem becomes synonymous withminimization of total length of moving path of a worker.
The difference between path planning problems and our
problem is that positions of waypoints for path planningproblems are fixed; on the other hand, our problem
changes positions of waypoints that are racks and boxes.
Since the number of racks and boxes ispredetermined as mentioned, our approach has to put

1. Introduction
Recently, improvement of production efficiency on cars
manufacturers is required owing to increasing of
demands in developing countries1. On the other hand,that improvements under existing circumstances are
depending on experience and intuition by workers at a
production line. To overcome this situation, we propose
to objectively and efficiently improve a production line
without depending on experience and intuition. The
production line of candidate for optimization is “Picking
up assemblies” area, so that our aim is to optimize thepositions of racks and boxes for assemblies. The
difficulty of optimization of “Picking up assemblies”*

In 2007 he obtained his PhD at the Department of Brain Science and Engineering, Kyushu Institute of Technology. From April, 2007 to March, 2014,
he was a lecturer. Since April, 2014, he has been an associate professor, Nishinippon Institute of Technology.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan749

Keiji Kamei, Takafumi Arai

predetermined number of racks and boxes into a
production line. For this, there is also a problem of“Knapsack Problem(KP)” in our optimization problem.
It can be said that our optimization problem combines
TSP, MST and KP for the reasons stated above.
To overcome difficulties, we propose to apply a
Genetic Algorithm(GA)2 to optimization for positions ofracks and boxes. A GA is suitable for global search in ahigh dimensional space because a GA has also the abilityof probabilistic random search based on genetic
operations in addition to local search. A GA has many
individuals(populations), and an individual has achromosome that cords the parameters for optimization.
Optimization processes are on the basis of genetic
operations, i. e., selection, crossover and mutation. A
process of crossover and mutation does not be
constrained because parameters for optimization in achromosome are independent in each; therefore,
generated parameters of optimization are able to take allvalues within a set condition. By contrast, parameters of
optimization in our problem are dependent on each otherbecause the number of racks to put assembly boxes andthat boxes is predetermined. Consequently, we are notable to apply a conventional GA to optimization. For this,
we apply a “constrained GA” to optimization for
positions of racks and boxes in a production line.
The details of our constrained GA are below. Geneloci correspond to racks and boxes positions in aconstrained GA. Selection process in genetic operations
is same way as a conventional GA. In crossover process,loss or duplication of some racks and boxes will occur.
The way of adjustment of the number of racks is that allof duplicative racks are deleted then missing racks are
randomly inserted to those positions. Mutation process
for racks is executed after the adjustment of crossoverprocess. Mutation process in a constrained GA changes
gene loci in order to meet the conditions for the numberof racks. Crossover and mutation process for positions ofboxes in racks is executed after crossover and mutation
process for racks. Crossover process for boxes is
executed on the same stage each other, e.g., boxes in top
stage of a rack are changed with boxes in top stage ofanother rack. In mutation process, genes for boxes are
mutated randomly as same as a conventional GA in first.
Following that, excess boxes are randomly chosen, thenthose boxes are deleted. Subsequently, shortfall in boxes
are inserted randomly to meet the conditions for thenumber of boxes. From a constrained GA in above, ourproposal is able to meet the conditions for our
optimization problem.
The results of simulation for virtual production line
show that our proposal succeeded in reducing about 10
seconds per car compared with random positioning.
2. Virtual Environment of Production Line
We construct a virtual environment of production line tovalidate the effectiveness of our proposal. Table 1 shows
the conditions for virtual environment, and Table 2
shows the number of assembly boxes for each car.
Table 1. Conditions of virtual environment. Length unit ismillimeter.
Kinds of Rack(Length)
Stages in a Rack

3(1000, 1500, 2000)
3(Top, Middle, Bottom)

Kinds of Assembly Boxes
Length of a Line

3
15000

Distance of AGV and Racks
Moving Speed of AGV

1500
200[mm/s]

Walking Speed of a Worker
Number of racks

2000[mm/s]
1000×3, 1500×4, 2000×3

Types of Cars(Car names)
Mixing Rate of Cars in
Productions

4(Car1, Car2, Car3, Car4)
Car1:0.5, Car2:0.25,
Car3:0.15, Car4:0.1

Table 2. Number of assembly boxes
for each car. “300”,”600” and “1000”
correspond to width of assembly boxes.
Width unit is millimeter.
Width of assembly boxes
Car1

300
15

600
4

1000
2

Car2
Car3

15
10

3
3

1
1

Car4
Shared

10
30

0
5

1
5

A worker in virtual environment picks up the all of
assemblies which correspond to a designated car in the
Table 2. Besides, 20 shared assemblies are randomly
designated from the all of shared assemblies.
The Fig. 1 shows the bird’s-eye view of an exampleof virtual production line. The condition in that figure isinitial state of picking up assemblies.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

750

Optimization for Line of

1000

1500

1500

2000

1500

1000

1500

2000

1500

2000

individuals(populations), and an individual has achromosome that cords the parameters for optimization.
Optimization processes are on the basis of genetic
operations, i. e., selection, crossover and mutation.
In a conventional GA, a process of crossover andmutation does not be constrained because parameters for
optimization in a chromosome are independent in each;
therefore, generated parameters of optimization are able
to take all values within a set condition. On the other hand,parameters of optimization in our problem are dependent
on each other because the number of racks to put
assembly boxes and of that boxes is predetermined.
Consequently, we are not able to apply a conventional
GA to optimization. For this, we apply a “constrained GA”
to optimization for positions of racks and boxes in a
production line as mentioned in the next section.

1000

AGV
15000

Fig. 1. Bird’s eye view of an example of virtual production line.
Values are distance and size of rack. Unit is millimeter. The redcircle indicates a worker. The blue, yellow and magenta squares
correspond to 1000, 1500 and 2000 size of racks, respectively.

A worker goes back and forth between racks and an
AGV. An AGV moves at a constant speed on a lane.
Consequently, it is safe to say that this optimization
problem is same as minimization of the moving distanceof a worker.
3. Optimization Problem
From the above section, our optimization problem issimilar to several path planning problems such as“Traveling Salesman Problem (TSP),” “Minimum
Spanning Tree problem(MST).” Our optimization
problem becomes synonymous with minimization of
total length of moving distance of a worker. The
difference between path planning problems and our
problem is that positions of waypoints for path planningproblems are fixed; on the other hand, our problem
changes positions of waypoints that are racks and boxes
in order to minimize length of moving distance.
In addition, since the number of racks and boxes ispredetermined as mentioned in the table 1 and 2, ourapproach has to put predetermined number of racks andboxes into a production line. For this, there is also a
problem of “Knapsack Problem(KP)” in our optimization
problem. It can be said that our optimization problem
combines TSP, MST and KP for the reasons stated above.

4.2. Constrained Genetic Algorithm
We explain the details of a “constrained Genetic
Algorithm” in this section. Gene loci correspond to racks
and to boxes positions in a constrained GA. Selection
process in genetic operations is same way as aconventional GA. In crossover process, loss or
duplication of some racks and boxes will occur. The wayof adjustment of the number of racks is that all of
duplicative racks are deleted then missing racks are
randomly inserted to those positions. The Fig. 3
illustrates an example of the way of adjustment.

(a)4. Genetic Algorithm and Constrained

P1
P2
C1
C2

Racks
1000 1000 1000 1500 1500 1500 1500 2000 2000 2000
9
4
2
5
7
6
8
1
0
3
5
3
9
6
0
8
2
7
1
4
9
3
2
5
0
6
8
1
0
3
5
4
9
6
7
8
2
7
1
4

P1
P2
C1
C2

Racks
1000 1000 1000 1500 1500 1500 1500 2000 2000 2000
9
4
2
5
7
6
8
1
0
3
5
3
9
6
0
8
2
7
1
4
9
4
2
5
0
6
8
1
7
3
5
0
9
6
7
8
2
3
1
4

(b)
Fig. 3. An example of crossover process for racks. P1, P2, C1
and C2 correspond to parent 1, 2 and child 1, 2, respectively.
(a)Duplication and loss. (b)After adjustment.

To overcome difficulties mentioned in the section 3, we
propose to apply a GA to optimization for positions ofracks and boxes. We explain the details of a conventional
GA, and of a constrained GA to solve our problem.

Mutation process for racks is executed after the
adjustment of crossover process. Mutation process in aconventional GA changes genes; by contrast, that process
in a constrained GA changes gene loci in order to meetthe conditions for the number of racks. The Fig. 4
illustrates an example of mutation for racks.

4.1. Genetic Algorithm
A Genetic Algorithm(GA) is suitable for global search ina high dimensional space because a GA has also theability of probabilistic random search based on genetic
operations in addition to local search. A GA has many

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan751

Keiji Kamei, Takafumi Arai

C1
Before
C2
After

C1
C2

Fig. 5 shows the changes in the fitness value of the bestindividual.

Racks
1000 1000 1000 1500 1500 1500 1500 2000 2000 2000
9
4
2
5
0
6
8
1
7
3
5
0
9
6
7
8
2
3
1
4
9
5

4
0

1
9

5
6

0
4

6
8

8
2

2
3

7
1

3
7

66

Fig. 4. An example of mutation process for racks. C1 and C2
correspond to child 1 and 2, respectively. Colored cells
represent the mutated positions.
Fitness(sec）

61

1
1113
2225
3337
4449
5561
6673
7785
8897
10009
11121
12233
13345
14457
15569
16681
17793
18905
20017
21129
22241
23353
24465
25577
26689
27801
28913
30025
31137
32249
33361
34473
35585
36697
37809
38921
40033
41145
42257
43369
44481
45593
46705
47817
48929

60

Generation

Fig. 5. Optimization results. “Blue line” shows “Case 1” and“Orange line” shows “Case 2” in Table 3.

Positions for racks and assembly boxes are
randomly determined in the first generation; therefore,
the maximum time for picking up in the first generation
is about 70.9 second and the minimum is about 65.3
second. In contrast with the first generation, the time forpicking up assemblies from the optimized positions inlast generation is about 61.7 second from Fig. 5. For this,our proposal achieved to reduce about 10 seconds per carcompared with random positioning.
6. Conclusions

In this section, we validate the effectiveness of aconstrained GA for optimization of a production line.
The parameters for a constrained GA are shown in Table
3. The fitness function is as the Eq. (1).

We propose to optimize a production line of cars
manufacturing plant by GA in this paper. The difficulties
of our aim are that parameters of optimization in our
problem are dependent on each other because the numberof racks to put assembly boxes and that boxes ispredetermined. Consequently, we are not able to apply
conventional GA to optimization. For this, we apply a“constrained GA” to optimization for positions of racks
and boxes in a production line.
From the results of experiment, our proposal
achieved to reduce about 10 seconds per car comparedwith random positioning.

Table 3. The parameters for a constrained GA.
Case 2
Case 1
4000
50000
0.08
0.02

0.07
0.025

Prob. of Crossover(Boxes)
Prob. of Mutation(Boxes)

0.08
0.02

0.07
0.025

Num. of Evaluation Cars

63

62

5. Experiments

Prob. of Crossover(Racks)
Prob. of Mutation(Racks)

Case 264

Crossover and mutation process for positions ofassembly boxes in racks is executed after crossover andmutation process for racks. Crossover process for boxes
is executed on the same stage each other, e.g., boxes in
top stage of a rack are changed with boxes in top stage ofanother rack. In mutation process, genes for boxes are
mutated randomly as same way as a conventional GA.
Following those processes, the adjustment process inorder to meet conditions is proceeded. Firstly, excess
boxes that do not meet conditions in Table 2 are
randomly chosen, then those boxes are deleted.
Subsequently, shortfall in boxes are inserted randomly tomeet the conditions for the number of boxes. If a box may
not be able to insert to anywhere, that individual isdeleted and an individual is newly generated.
From a constrained GA in above, our proposal isable to meet the conditions for our optimization problem.

Num. of Individuals
Num. of Generations

Case 165500

References=∑(1)
where is individual number, is total time of picking
up of all of assemblies for car ,is the number of
evaluation cars for an individual.

1. K. Kamei et. Al., Development of the Recognition Method
for Black Objects under Black Background Color(SISA2017, Fukuoka, 2017), pp.352–357.
2. R. Pfeifer and C. Scheier, Understanding Intelligence,(MIT Press, Cambridge, 1999).

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

752
1

Slip Model of Ball Driven by Two Rollers
Kenji Kimura, Shota Chikushi, Kazuo Ishii

Department of Human Intelligence Systems
Kyushu Institute of Technology
2-4 Hibikino ,Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196 Japan
E-mail: kimura_kenji@edu.brain.kyutech.ac.jp,

Abstract
Spherical wheel motion driven by rollers attract attention due to the possibility of uneven surface. RoboCup isa platform designed to promote the research fields such as Artificial Intelligent (AI) and robotics, and one ofexamples is the ball dribbling mechanism of RoboCup Middle size league robot, and this mechanism controls
the ball with two driving rollers. As the result of the survey at the 2017 World Cup, all teams in the world havedetermined the roller arrangement heuristically, and no mathematical consideration has been made. In this study,
we propose a model considering slip, discuss the relationship between slip ratio and robot speed using
evaluation function, and verify the model by experiment.
Keywords: Spherical wheel motion, the ball dribbling mechanism, slip ratio,

1.

League soccer robot, in recent years, a ball handlingmechanism for further controlling the rotation of the ballis mounted. In the ball handing mechanism, two drivingrollers are symmetrically arranged on the upper
hemisphere. Tech United Turtles uses general rollers[5],CAMBADA uses omni rollers[6].These roller placement
problems rely on a lot of experimental data for thesecontrol. And all teams in the world have determined theroller arrangement heuristically, and no mathematicalconsideration has been made. Kimura introduced themathematical non slip model[7],and discuss about nonslip condition and the relationship between angularvelocity vector and movement angle of the sphere isclearly in arbitrary two rollers arrangement .Chikushi
experiments on non slip kinematic in two rollers
symmetric arrangement using prototype[8].
In this research, we propose a kinematics slip modelthat takes slippage into account, clarify the relationshipbetween the roller speed loss ratio and robot speed loss
ratio depend on the arrangement of roller the world team,seeking accurate robot speed, Evaluate the placement.

Introduction

In recent years, development of a transmission
mechanism of an actuator in locomotion and drivingusing a sphere has been actively conducted. At first, M.K.
West proposes a holonomic mobile robot combining
multiple spherical wheels like the Omni wheel[1], thishas unconstrained in the direction orthogonal to the
propulsion force. Omni-Ball wheels developed by
Tadaguma solve the design restrictions of the abovementioned spherical wheels[2].The omni-directional
locomotion function is realized by the passive rotation of
the pair of hemispheres, and shows a capability toovercome a step difference higher than that of an omnirollers.As a mobile robot that drives and moves
monocytes.A ball balanced robot of Kumagai ismentioned[3], and by rotating the omni rollers
independently by equally arranging the three omni rollers
in an equilateral triangle shape, rotation of the sphere in
an arbitrary direction is realized. Wada proposed two
spherical power transmission which is a rotation pair ofspherical bodies driven by two rollers[4], the upper
spherical body is driven by two rollers, the lower spheretransmits power, the caster's, it is proposed an active
caster that realizes steering and driving, and by driving
the wheel shaft and the steering shaft, it is possible tomove in all directions. In the RoboCup Middle Sized

2.

Kinematics of Sphere by Rollers

2.1. The Existence of the Angular Velocity Vectorof the Sphere−, is fixed in the center of a sphere of radius ,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan753

Kenji Kimura, Shota Chikushi, Kazuo Ishii

as shown in Fig.1. indicates a position vector of the ith
wheel from the with respect to the . Center of the rollers
and,are on the same straight line. And whenisrepresented by using polar coordinate( , , , ),denoted as follow
=

cos

, cos

, , sin, cos, , sinspan{ , } , is expressed as intersection point of( ), ( )1−(

let be the angular velocity vector of the sphere ofas the starting point , let be a unit vector along the.The admissible
roller axis as the starting point
angular velocity that satisfies the non slip condition is
involved in the linear subspace span{ , }(spanned by
the , denoted). ∈ span{ , }, if the peripheral
roller speed is given. the inverse kinematics is givenby∙= −( / )

[(

+
2

cos

)

−(

cos

)

×
×

‖

]

(4)

with
(5)

‖

Using (1) (3) (4), slip-kinematics is given by
=

(2)

1

−(

∙

)

[(

cos

+

sin

)

+
2

−(

sin

cos
‖

)

×
×

]

(6)
‖

And, the sphere rolling velocity on the horizontal(robot speed) denote , using,(components of), ‖ ‖ is given by

As shown in Fig.1(a), the end-point set ofwith
expect to input lies on a straight line ( ) parallel to
P_i and distance ν_i/r. ω is not uniquely determined. As
shown in Fig.1(a), the case of using two rollers ( =1: left side , = 2: right side ), ( ), ( ) has anintersection point. Therefore, is uniquely determined.

‖ ‖=Fig. 1. (ⅰ) ( ) is the end-point set of ( is notuniquely determine) (ⅱ) is uniquely determined as anintersection point.

+(7)

Fig. 2. when ( ), ( ) has no intersection, is defined
the center of the smallest radius sphere touching( ), ( ).

2.2. Slip Model of sphere driven by two rollers

2.3. Roller speed loss ratio and evaluation
function

In general, roller arrangement exists such that( ), ( ) have no intersection, as shown in Fig.2.
(| | ≤ /2) denotes the angle between span{ , }and , denotes distance between span{ , } and( ), let ( ) be the orthogonal projection of ( )with respect to span{ , }, the distance between
and ( ) is ( / ) cos , therefore is given by,= ( / )sin)

The orthogonal projection component of
× , is given by
exspect to

(1)

,

∙

if ∉ span{ , } , the sphere slips with expect torollers at.Usingand, the sphere rolling speed− ( ∙ ) , roller speed loss ratio
is defined asfollows− ( ∙ )
(8)
= 100(1 −)(3)

is function depend on , , in particular, if ∈span{ , } ,then =0[%] and, if ⊥ ( ∉span{ , }), then = 100[%]. The evaluation
function is defined as follow

It is assumed that it is the optimum point of the end
point of , Consider the center of the smallest radius( ), ( ) . The orthogonal
sphere touching
projections component ofwith respect to© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

754

Slip Model of Ball

=(
3.

+

)/2

(9)

Experiment

We experiment to estimated Robot speed and roller
speed loss ratio. an experiment was conducted to measure
Fixed parameters are given by, = 0.75[ /], , = 215, , = 275, , = , = 60[deg], andvariable ( = , = − ; two rollers are placed insymmetry) is each of case = 0, 10, 20, 30 [deg], Thetarget values are shown in Table 1. As a result,experimental value 100‖ ‖ , ,(estimated value
from the motor using (7) (8)) is closed to theoretical value100‖ ‖ ,,, As shown in Fig. 4-7, The meanerrors are showed in Table2, therefore these values
extremely small and the validity of the slip model was
proved.

(b) Roller speed loss ratio
Fig. 5. Comparison theoretical value and experimental
value = 10[deg](a)Robot speed ratio(b) Roller speed loss ratio
Fig. 6. Comparison theoretical value and experimental
value = 20[deg](a) Robot speed ratio(b) Roller speed loss ratio
Fig. 4. Comparison theoretical value and experimental
value = 0[deg](a) Robot speed ratio(b) Roller speed loss ratio(a) Robot speed ratio© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan755

Kenji Kimura, Shota Chikushi, Kazuo Ishii

Fig. 7. Comparison theoretical value and experimental
value = 30[deg]5.

In this study, we proposed a kinematic model ofspheres by roller driving considering slippage.
Experiments measured the roller speed loss ratio in surge
motion and the speed of advance of the robot and verified
the accuracy of kinematic. We also clarified therelationship between the roller speed loss ratio and therobot speed loss ratio and discussed the speed loss ratioof the robot in the rollers arrangement of the world team,and clarified optimal arrangement of the ball dribblingmechanism. As a future task, we would like to consider
the robot speed loss ratio in the all directions.

Table1. Theoretical value

Table2. Experimental value (average error)

4.

Conclusion

Consider of world teams

References

According to survey of the 2017 World Cup,world
Roller arrangement angles are close to =0 , 10 , 20 , 30[deg].The horizontal axis shows the roller speed loss ratio,
the vertical axis shows the robot speed loss ratio, asshown in Fig.8. four white mark is ( , 100(1 − ‖ ‖)),
these are on the evaluation curve, and four back mark( , 100(1 − ‖ ‖) ), these areclose to the evaluation andupper part curve, therefore, the validity of evaluation
curve was proved. Moreover, RV-infinity ( =0),
Tech-U, Falcons( =10), The robot speed loss ratio iswithin 5[%], H-Musashi, NuBot( =20) is 9[%],Water( =30) is 18[%]Therefore, if = 0〜10[deg] ,itis optimal roller arrangement to be possible in efficientmotion.

1. M. West and H. Asada. Design of a holonomic
omnidirectional vehicle. In Proceedings of the 1992 IEEE
International Conference on Robotics and Automation,Vol.(1992),pp.97-103,
2. Berengeres. Development of holonomic omnidirectional
vehicle with “omni-ball”: spherical wheels. In
Proceedings of the 2007 IEEE/RAS-EMBS International
Conference Robots and systems, (2007), pp.33-39,
3. M. Kumagai, T. Ochiai: “Development of a robot balanced
on a ball – Application of passive motion to transport”,
Proc. ICRA IEEE(2009), pp.4106-4111
4. M. Wada, Y. Inoue and T. Hirama, ”Kinematics andMechanical Design of an Active-caster with a Ball
Transmission” The Robotics Society of Japan, vol.31,
No.6,(2013),pp.591-598,
5. Rob. Hoogendijk: “Design of a Ball Handling Mechanism
for Robocup” Bachelor End Project, department
Mechanical Engineering, Technische Universiteit
Eindhoven
6. B. Cunha, A.J.R. Neves, P. Dias: “CAMBADA'2015:
Team Description Paper” RoboCup2015Hefei
7. K. Kimura, K. Ishii, Y. Takemura, M. Yamamoto,“Mathematical Modeling and motion analysis of the wheel
based ball retaining mechanism”, SCIS & ISIS (2016),
pp.4106-4111,
8. S.Chikushi,T.Weerakoon,T.Sonoda,K.Ishii, “Kinematics
of Two-Roller-Driven Ball for RoboCup Soccer Robot”,
Journal of Robotics, Networking and Artificial
Life,Vol.4,No.3(2017),pp.248-253,

Fig.8 Relationship between evaluation value andexperimental value in world teams© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan756

Strategy Analysis of Multi-Agent Games
Using Self-Organizing Map
Moeko Tominaga

Graduate School of Life Science Systems Engineering, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: tominaga-moeko@edu.brain.kyutech.ac.jp
Yasunori Takemura

Faculty of Engineering, NishiNippon Institute of Technology
1-11, Aratsu, Kanda-town, Miyako-gun, Fukuoka, 800-0397, Japan
E-mail: takemura@nishitech.ac.jp
Kazuo Ishii

Dept. of Human Intelligent Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka, Japan
E-mail: ishii@brain.kyutech.ac.jp

Abstract
As technology develops, realization of a symbiotic society between human beings and robots sharing theirenvironment has become one of most important subjects. For example, soccer is a multi-agent game that require
strategies such as each members’ position and defense. The behavior of a player changes depending on the gamesituation such as winning or losing, score gap, remaining time. The players may play offensive when their team islosing, or be defensive when their team is winning with minimum score difference. In this paper, human soccer
game is observed and analyzed to determine how the state vector of each players affect team behavior. The teamstrategy is analyzed based on parameters such as the positions, the number players, score, time and actions ofsoccer robots using Self-Organizing Map(SOM).Keywords: Strategy, Self-Organizing Map, team behavior, SOM.

deviation between social expectation and direction ofresearch and development. Therefore, it is essential
discuss how to achieve the coexistence with robots andwhat of the symbiotic society should be.

1. Introduction
Recently, social implementation of robot is highly
expected as a solution to many social problems such as
ensuring safety of a sustainable society, respond to rapid
population decrease and aging population, establishingindustrial base including the primary industry. In order toproperly implement the robot society, it is necessary toreturn and present the research outcome to our society ina manner that is easy to understand while avoiding

1.1. Cooperative Behavior
A cooperative behavior can be generated when multiple
autonomous agents perform a single large task whileinteracting to each other. Multi-agent system (MAS) has© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

757

Moeko Tominaga, Yasunori Takemura, Kazuo Ishii.

human beings, alternatively not leading to proper
behavior selection.

been conducted by many researchers in recent years as asolution to solve difficult problems that are challenging
when using one agent. As test bed of MAS, RoboCup
encourages the cooperation of multiple agents using
method of learning such as reinforcement learning andneural network. Sandholm and Crites1 shows thatapplication of reinforcement learning in the Iterated
Prisoner’s Dilemma require sufficient measurement data
and action for successful learning. In addition, Arai2
compares the tracking problem in a lattice environment
with Q-learning and Profit Sharing properly andconcludes that Profit Sharing is suitable for learning in a
multi agent environment. However, these studies have
not yet considered application toward robots that operate
in a real environment.

1.3. Human-Robot Cooperation
As mentioned in the previously, various discussions havebeen made in acquiring cooperative behaviors among
multiple real robots. However, when looking at thesymbiotic society between humans and robots, wequestion the fact that the observation target for learning
so far is a robot. In fact, even without the interface
problem, it is not easy to communicate between humans
and robots using the current multi agent system. This
research consider cooperative behaviors between humans
and apply it to robot. (see Fig.1) In learning of a real robot,
experience bias and selection of input vector elementsfrom state space in environment recognition isconsidered using neural network. In this paper, positionin human cooperative behavior is analyzed using SelfOrganizing Maps. Futsal is used as an object ofobservation of humans’ cooperative behavior since thegame’s objectives is also cooperation of multiple agentsin dynamic environment which includes all multiobjective/multiple constraint problems and real-timeplanning / reasoning is required.

1.2. Robot Learning
In robot learning, information of the environment andprerequisite knowledge of robot itself are not primarily
required. However, since learning requires a large
number of trials, securing the robot’s stable state and its
action apace is crucial in order to reduce the time cost andenable learning that fits the intention of the user. In a realrobot, the configuration of state space containing thesufficient information to perform a task depends on the
behavioral capacity of the robot. Likewise, the action
space also depends on the perception ability. In responseto this problem, Asada3 proposed a method of
constructing the state space with recursive segmenting
initial state vector. However, the behavior and state spacecomposition method is based on the robot’s experience
which may raise concerns if the experience is biased, thebehavior may be considered as strange habits like in

2. Setting Observation Elements in Situation
Analysis
In observing the human futsal match, firstly, we set as an
object the elements which can be easily observed. Astarget, in addition to the coordinates and the speed of theathlete and the ball, the elapsed time of the game and thegoal difference are considered. In this case, as theobservation method, since the method of photographing
the entire game field using the monocular camera wasselected, the state space is established based on various

Fig.1. Human-Robot Symbiosis System

Fig.2. The Observation Environment of the Game

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan758

Strategy Analysis of Multi-Agent

coordinate data in the field. Human futsal games wereplayed on 5 vs 5 players familiar with team play and thecoordinates of all players and balls were observed. (seeFig.2) Also, the observation vector is expressed by theEq. (1).
= [

,

, ⋯

,

]

multidimensional input vectors can be reduced
dimension in the output layer, and input vectors withsimilar features are placed closer to each other. As anapplication example of SOM, there is visualization thateasily expresses complex data in two dimensions, etc. Aswith the clustering technology, creation of abstraction ispossible. Since excellent interpolation performance ismaintained, generalization of learning data is possible,and highly valid output is expected for unknown input
data.

(1)

The player position coordinate
and the ball position
coordinate in the field are shown in the Eq. (2) and (3).
=

= [

,

,

]

(2)

3.1. Algorithm
In this research, we used Batch type SOM which loadtaking into consideration all inputs generated after thecompetition phase of all input vectors, instead ofincremental learning which load updating each time a
new input vector is input. By using this method, it
becomes unnecessary to consider the influence on the
output by the input order of input vectors. The definitionof batch type SOM in this research is shown in thefollowing Eq. (4) to Eq. (5). Where is the input vector
ID and is the ID of each neuron in the output layer.
1. Competitive phase
Selection of the best matching neuron ∗ for each
input vector .

(3)

is an ID for identifying the player. We observed a gameconsisting of a team of = 1, 2, 3, 4, 5 players and ateam consisting of = 6, 7, 8, 9, 10 players. Incidentally,= 6, 10 is the goalkeeper of both teams.
3. Self-Organizing Maps
Self-Organizing Maps is unsupervised learning, and it is
known as competitive learning similar to information
processing by the neural circuit of life. Competitive
learning is an important concept in hierarchical neuralnetworks. Each input neuron (input vector ) in the input
layer (input space ) is connected to all output layer
neurons, and the coupling weightis given to thecoupling. (see Fig.3) The algorithm of SOM4 consists of
two hierarchies, a competitive hierarchy and acooperative hierarchy. For the input, the best matchingneuron is selected, and the load of that neuron and its
immediate neuron is changed. With this structure,

∗

2.

= arg

−

( + 1) =

( )+

(4)

Coordination phase
Updating the load of each neuron.

(5)

∙∆

Incidentally, the increase / decrease value ∆ofthe coupling load is expressed by the following
equation (6).
∆

( + 1) = ∆( ) + Λ( ,

∗

,

)∙

(6)
In the second item on the right side, thecombination of the minimum Euclidean distancerule and the neighborhood function is applied asa load update rule.

Fig.3. Overview of SOM algorithm© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

759−

Moeko Tominaga, Yasunori Takemura, Kazuo Ishii.

4. Experiment
Experiments were conducted with the parameters shownin Table 1
Table. 1: Parameters setting SOM
Learning timet

Map size

1030×30

4.1. Input Vector: Team1 (The input vector is expressed by the following Eq. (7),except for the goalkeeper of the same team, the player
coordinates are = 1, 2, 3, 4 when = 1 − 428 In thecase of = 429 − 856, = 6, 7, 8, 9 are used, and thenumber of input vectors is M = 856. In this time, 6challenges from set play to shoot were used as input data.
= [4.2. Experimental Result,]= 1, 2, 3, 4): Team2 (= 6, 7, 8, 9)

Fig.5. The unit selected by input data after learning(Left figure: the try who never took the ball to the opposing
team / Right figure: the try which the ball has been taken isshown on the right figure)outside the red unit in the right figure. As a result, we canclearly distinguish on SOM when one team consistently
ran from monopolizing the ball, from the set play to theshoot, and when it was not possible.

(7)

The outputted feature map is shown in Fig.4. The color
gradation represents the Euclidean distance from theweight of the peripheral unit. Also, after learning,observe the unit selected by input vector again. From theset play to the shoot, the try who never took the ball tothe opposing team is shown on the left figure, and thetried thing is shown on the right figure. It is color-codedby team. Referring to Fig. 4, it can be seen that the
selected unit is inside the red unit in the left figure,5. Conclusion
In this research, the team strategy is analyzed based onparameters such as the positions of human player, timeand actions using Self-Organizing Map (SOM). We
developed the SOM map from the position of eachplayer and ball on the field. As a result of experiments,
it was found that discrimination is possible on thefeature map when the pass turn was successful or failed.
In the future, I would like to also examine the change inplayer's positioning when the pass turn fails.
References
1. T. W. Sandholm, R. H. Crites: On Multiagent Q- learning
in a Semi-Competitive Domain, Workshop Notes of
Adaptation and Learning in Multiagent Systems Workshop,
IJCAI-95, 1995.
2. S. Arai，K. Miyazaki，S. Kobayashi: Methodology inMulti-Agent Reinforcement Learning : Approaches by QLearning and Profit Sharing，Journal of Japanese Society
for Artificial Intelligence, 13(4), pp.609-618, 1998.
3. Y. Takahashi，M. Asada: Behavior Acquisition by MultiLayered Reinforcement Learning ， Journal of the
Robotics Society of Japan, vol. 18, No.7, pp.1040-1046,
2000.
4. Kohonen. T, Self-Organizing Maps, Springer Japan
5. Van Hulle, Marc M: Faithful representations andtopographic maps from distortion-to information-based
self-organization, Kaibundo Publishing Co., Ltd., p9-92,
2001.

Near

Far

Fig.4. The feature map expressed by the gradation display
after learning the coordinates of the athlete and the ball as
the input vector© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

760

Recognition of Tomato Fruit Regardless of Maturityby Machine Learning Using Infrared Image and Specular Reflection
Takuya Fujinaga

Dept. of Human Intelligence System, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: fujinaga-takuya@edu.brain.kyutech.ac.jp
Shinsuke Yasukawa

Center for Integrated Underwater Observation Technology,
Institute of Industrial Science, The University of Tokyo
4-6-1 Komaba, Meguro-ku, Tokyo 153-8505 JAPANE-mail: yskw@iis.u-tokyo.ac.jp
Binghe Li

Dept. of Human Intelligence System, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: li-binghe@edu.brain.kyutech.ac.jp
Kazuo Ishii

Dept. of Human Intelligence System, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: ishii@brain.kyutech.ac.jp

Abstract

This paper presents a tomato fruit recognition method using plant characteristics of tomato and infrared
images. Labor shortage and aging are problems in Japanese agriculture field. We aim to realize automatic
harvesting and production management system of tomato. For that, it is necessary to detect the position
and maturity of tomato fruit. Tomato fruit shows high reflectance against infrared light. The specularreflection part of the tomato fruit in the infrared image is used as training data. The Tomato harvesting
robot can focus only on tomato fruit in the harvestable range by using infrared image. We use the images
acquired at the actual tomato greenhouse to evaluate this proposed method. As a result of machinelearning, Precision is 0.940, Recall is 0.808, and F-measure is 0.868.
Keywords: Tomato Harvesting Robot, Infrared Image, Specular Reflection, Machine Learning

becoming obvious in the agricultural field. Under suchcircumstances, attempt of smart-agriculture is being
conducted. The smart-agriculture is to positively
introduce ICT (Information and Communication
Technology), IoT (Internet of Things) and etc. to improve
work efficiency.

1. Introduction
Stable production of crops by farmers is very
important not only for our lives but also for naturalenvironment conservation. However, problems such asaging, decrease of workers, shortage of successor are© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan761

Takuya Fujinaga, Shinsuke Yasukawa, Binghe Li, Kazuo Ishii

In tomato cultivation, there are a high technology
greenhouse where temperature, humidity, carbon dioxide
concentration and culture liquid are controlled and highwire cultivation is introduced for efficiency of harvestingwork. Furthermore, the automation of harvesting andmanagement in greenhouses by robot is expected as anext-generation technology to reduce the burden on
workers.
Fruit detection technology using image information isnecessary to predict fruit yield [1] and automaticallyharvest fruit [2, 3]. As a detection method of tomato fruit,study using optical properties of tomato has been done
[4-7]. Since these require additional illumination andspecial sensors, the system is generally expensive. Studyon fruit recognition by machine learning has also been
actively done [1, 8-11]. Although there are attempts topredict the yield of tomatoes from RGB images, it isnecessary to method of detecting only mature tomato
fruit [9] or device such as acquiring image byilluminating light during the night [11].
We aim to introduce the robot can not only harvests
tomato fruit but also monitors tomato growth state [12,13] in the large-scale greenhouse. In particular, we focus
on research and development of management system intomato greenhouse. We proposed a generation method ofa tomato growth state map using infrared image [12].
This map is used not only for planning the harvesting
operation of the robot but also for grasping the state of
the tomato fruit. We also proposed a tomato fruit
detection method of template matching based using
infrared images and specular reflection [13].
Although the infrared image is used in our proposed
methods, it is easier to introduce the system because it
adopts the sensor that can be used more easily than themethod using other special sensors. The outline of robot,
system and sensor are described in Chapter 2.
In this paper, we propose a tomato fruit detection
method with specular reflection part of tomato fruit on
infrared image as training data. Training results areevaluated by images acquired by robot in the real largescale greenhouse.

Fig.1 Tomato cultivation environment.
Tomato is cultivated in Cultivation Area. Laborers harvest
and manage tomatoes on Rail in Working Area.

Fig.2 Overview of tomato harvesting robot.
Tomato harvesting robot consists of manipulator,
end-effector, camera, computer and mobile mechanism.

greenhouse which we assume introduction of robots. As
an example, there is Hibikinada greenhouse in
Kitakyushu. This greenhouse area is about 8.5 ha. The
outline of the cultivation environment in Hibikinada
greenhouse is shown in Fig.1. As shown in Fig. 1, thetomatoes are arranged at a predetermined interval in thecultivation area. Length of this area is about 70 m, andthere are hundreds of this area in greenhouse. There is theworking area between this cultivation areas, and laborers
move within this working area to harvest and manage(pruning, pest control) tomatoes. In order to control
greenhouse temperature, the pipe through which warm
water flows is installed on the working area. This pipe
also has role as a rail for cart which is move way todecrease burden of workers.
We develop and research robot that can move on thisrail as the first step to realize tomato automatic harvestingand production management system. The overview oftomato harvesting robot is shown in Fig.2. This robot

2. Tomato automatic harvesting and production
management system
2.1. Cultivation environment and robot
Describe the environment of the large-scale tomato© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan762

Recognition of Tomato Fruitconsists of a manipulator and an end-effector for
harvesting tomato fruit, a camera for acquiring the image
in the greenhouse, a computer for controlling the robot,
and a moving mechanism for moving on the rail.
2.2. 2.

Tomato automatic harvesting

Describe the movement of robot until automatic
harvesting of tomato fruit. It is assumed that multiple
robots are installed in the greenhouse as shown in Fig.1.
The cultivation places are divided to section according to
the type of tomato, and the robots harvest and monitor inplace predetermined.
The robot decides the target cultivation area, moves tothe working area, and moves on the rail. When the robot
moves on the rail, images of the greenhouse are acquiredin order to grasp growth state of tomato fruits. At thistime, the position where image is acquired is measured
from the encoder mounted on the moving mechanism andis saved. When the robot moves to the end of the rail, thetomato growth state map of the cultivation area isgenerated on the spot. Harvestable tomato fruits aredetected from the map, and the robot harvests the tomato
fruit.

Fig.3 Image acquired in the tomato greenhouse.
Tomato fruit within and outside the harvestable range are
confused.

Not only can be light the processing of the robot in
the sensing stage but also the greenhouse administrator
who confirms the tomato growth state map can focus on
only the front row, leading to the prevention of double
confirmation of the tomato in the back row.
In addition, we aim to detect tomato fruits from thistomato growth state map and predict the yield and harvest
time of the tomato fruit. Prediction of yield and harvest
time leads to shortening of working hours, stabilizationof market, and continuation of agricultural management.
As a previous step for that, it is necessary to detect
the position of tomato fruit. In this paper, we propose amethod which recognize tomato fruit regardless ofmaturity by applying tomato fruit detection method of
template matching base using infrared image andspecular reflection [13].
Using an infrared image not only removes tomatoes
and background in the back row but also can apply theplant characteristics of tomato. The plant characteristics
of tomato is described in Chapter 3.

2.3. Production management system
Fig. 3 is the image inside the greenhouse captured by
the robot moving on the rail as shown in Fig.1. Not only
the tomato fruits in the front row but also the tomato fruitsin the back row are confirmed. However, since themovable range of the manipulator is in the range up to thetarget cultivation area (the front row), it is not possible toharvest the tomatoes of behind (he back row).
Therefore, it is important to measure the 3dimensional position of tomato fruit. So far, a method of
get the 3-dimensional position of tomato fruit by thestereo vision system [6] or the 3-dimensional visualsensor using two wavelengths [7] is proposed. However,as mentioned in Chapter 1, these methods require
illumination and special sensors.
In our system, Time of Flight (ToF)-based Kinect(Kinect v2) which can obtain three images multi-modal(Infrared image, RGB image, depth image) at a low priceis used. The robot can ignore the tomatoes (fruits, stems,
leaves) and the background in the back row by using theinfrared image, and can focus on only the tomato fruits inthe front row.

3. Machine learning using infrared and specularreflection
3.1. Specular reflection of tomato on infrared
image
It is possible to select parts fruit and other than fruit
using spectral reflection characteristics of plants [14].
Tomato has high reflectance for fruit, stem and leaf
against the wavelength band of near infrared light(around 800 nm). Regardless of maturity, the reflectanceof tomato fruit is high in this wavelength band, andfurther it is possible to judge maturity (ripe fruit or unripe
fruit) of the fruit.
In this paper, infrared image is used to detect tomato
fruit regardless of maturity. For acquiring infrared image,© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan763

Takuya Fujinaga, Shinsuke Yasukawa, Binghe Li, Kazuo Ishii

Kinect v2 is used, and this wavelength band of the nearinfrared light is 830 nm, which is considered to be awavelength band effective for detecting tomato fruit.
On the infrared image, the tomato fruit shows theconcentric circle with center is strong response andaround is weak response. In the greenhouse, response
intensity varies at the time of acquiring image due to the
influence of sunlight and shadows of stem and leaf.
Therefore, in previous study [13], the gradient orientationof the specular reflection part is used so that it does notdepend on the response intensity. In this paper, we report
tomato fruit recognition method by machine learning,
using specular reflection part of tomato fruit as training
data.
Here, specular reflection is generally the reflection of
perfect light by a mirror, water surface, or the like. The
specular reflection of tomato fruit described in this papermeans reflection of infrared light from tomato fruit andstem and leaf with high reflectance when infrared light isprojected on tomato.

In order to search the position of tomato fruit, a sliding
window method is adopted, which extracts a candidate
region while moving a constant small region for eachconstant pixel in the image. In the sliding window
method, even one fruit may be recognized more thanonce. Therefore, the position of the tomato fruit isobtained using the Mean Shift method [17] that detects
the maximum value of the density of the pixel group
recognized multiple times.
4. Recognition result of tomato fruit regardlessof maturity
4.1. Outline of training data and test data
Evaluate the proposed method using 70 images
acquired at Hibikinada greenhouse. 40 images as training
data and 30 images as test data. The illuminance in thegreenhouse is 2800 to 9660 lx when acquiring the image.
The training data are classified as shown in Table 1.
Here, two training data sets are prepared in order toevaluate the recognition accuracy due to thecharacteristic bias of the specular reflection part of thetomato fruit. Each training data set is selected by 20
different images from 40 acquired images, and positive
data and negative data of 21 * 21 pixels are made. Sample
images of positive data and negative data are shown in
Fig.4. Fig. 4 (a) is positive data, which is a specularreflection part of tomato fruit on the infrared image. Fig.4(b) and (c) are negative data, Fig.4 (b) shows a part where
the stems are densely packed, and Fig. 4 (c) shows a partof stems and leaves. The time of the classifier is about 6.6
seconds in the training data set 1 and about 5.7 seconds inthe training data set 2.
In this report, only tomato fruit that can be visuallyconfirmed specular reflection part is subject ofrecognition. As mentioned in Section 3.1, specularreflection part of tomato fruit shows the concentric circle
with center is strong response and around is weak
response, so it can be visually confirmed. On the 30 test
data, there are 604 tomato fruits, among which 480
tomato fruits can be confirmed specular reflection.

3.2. Recognition method of tomato fruits using
Support Vector Machine
In order not to depend on environment light, HOG
features [15] of specular reflection part is used as training
data. HOG features is a histogram of the gradientdirection of the brightness, and it is known that it is robust
to variation of illumination. Also, even when images ofdifferent resolutions are targeted for HOG features, they
can be compared by resizing to the same resolution, sothat a reduction in calculation amount can be expected.
For the training data and the test data, as shown in Fig.
1, the image acquired by the robot moved in thegreenhouse is used.
As the training data, use the image of 21 * 21 pixel
centered on the pixel with the highest response intensityof specular reflection part of tomato fruit. 21 * 21 pixel
is almost of the same the size of the target tomato fruit in
the image acquired from the camera position mounted on
the robot.
The specular reflection part of tomato fruit is positive
data, stem and leaf, planter is negative data. Support
Vector Machine [16] which is one of pattern recognition
models is used to classify training data.
For the test data, use an infrared image of 424 * 512
pixels focusing only on the tomatoes in the front row by
removing the tomatoes in the back row and background.

Table1 Outline of training data.
Positive
Negative
Training data set 1
224
1184
Training data set 2
246
1030

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan764

Recognition of Tomato Fruit

5. Discussion
The Precision of training data set 1 is 0.959, and thePrecision of training data set 2 is 0.922. The Precision isthe evaluation index that the result is also positive,
predicted as positive. Since the Precision exceeds 90% in
both training data sets, the misrecognition is less than
10% of the total. This method for recognition of tomato
fruit is imply to be effective.
As an example of incorrect recognition, there is FP in
Fig. 5. As tomato not only fruit but also stem and leaf
have specular reflection. Depending on how the stem andleaf are attached, it is thought that the misrecognition isdone because there may be a case where the response issimilar to that of tomato fruit.
Regarding the Recall, among the 480 tomato fruits,374 (0.779) in the training data set 1 and 401 (0.835) inthe training data set 2. Even though the specularreflection part can be confirmed in the tomato fruit byvisually, approximately 20% of the tomato fruits are notrecognized.
Examples which are not recognized as tomato fruit
include FN1 and FN2 in Fig. 5. In FN1, due to the
influence of the sunlight, it fades out and it is notrecognized as specular reflection part. In FN 2, it slightly
has specular reflection. However, the response of
specular reflection part is not concentric circle, so it is notrecognized as tomato fruit.
In the infrared image acquired by Kinect v2, there is atendency that the accuracy of the edge portion of theimage deteriorates. In Kinect v2, infrared light isprojected, and it measures the time until reflected light
returns to the device. Since the distance to the object islonger at the end part of the image than in the center ofthe image, the light attenuates due to the influence ofenvironment light. Therefore, it is thought that accuracy
drops.
We plan to apply this method to mosaic images. In the
proposed method [12], mosaic image is generated fromimages acquired at regular intervals. However, we thinkthat this problem can be solved by devising thegeneration of the mosaic image. In the case that tomato
fruit dense in the center of the image, reducing theoverlapping area at the time generation of mosaic image.
We also consider a solution that infrared light is projectedfrom the back of Kinect v2 and images are always
acquired under the same illumination condition.
Tomato fruit which specular reflection cannot be
confirmed is excluded, which is about 21% of the total.
In this case, the bunches are dense and the fruits areoverlapped with each other or the tomato fruits are
hidden by stems and leaves. We consider two possible
ways to deal with this problem.

(a)
(b)
(c)
Fig.4 Sample image of positive and negative data.
(a) is positive data. (b) and (c) is negative data.

Fig.5 Recognition result.
Points is recognition result, solid line frame is False
Negative, broken line frame is False Positive.

4.2. Result
For evaluation of recognition, evaluation index ofPrecision, Recall, F-measure is used. Equations (1), (2)and (3) show the calculation method for each evaluationindex. True Positive (TP) is judged to be correct positive,
False Positive (FP) judged as positive by mistake, False
Negative (FP) judged erroneously negative.
Since only tomato fruit is targeted for recognition,
True Negative judged to be correct negative not subjectof evaluation.
(1)
(2)
(3)
An example of the recognition result of tomato fruit isshown in Fig.5. In Fig.5, points are the recognition
position of the tomato fruit, the solid line frames are False
Negative, and the broken line frame is False Positive. Table
2 shows the results of each evaluation index in eachtraining data set.
Table2 Result of evaluation index.
Precision
Recall F-measure
0.959
0.779
0.860
Training data set 1
0.922
0.835
0.877
Training data set 2© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan765

Takuya Fujinaga, Shinsuke Yasukawa, Binghe Li, Kazuo Ishii

First, similarly to the problem at the time of acquiring
the infrared image described above, devising thegeneration of the mosaic image. Track the tomato fruitfrom the acquired video and cut out the image when
specular reflection part can confirm. When generating themosaic image, change the combine position between
images.
The second is to use the method [18] to estimate theshape of the tomato based on the geometric and physicalinference of the tomato. It is possible to grasp the tomato
fruit which cannot be confirmed from the image by
considering that tomato maintains stable state as a tomato
bunch with the interaction force or gravity of theindividual fruits of tomato.

5.
6.

7.

8.

6. Conclusion
In this paper, as a previous step of the tomato growthstate map, the method of recognizing tomato fruits is
proposed. By using the specular reflection part of thetomato fruit on infrared image as training data, Precision
is about 0.940. The effectiveness of tomato fruit
recognition method using tomato plant characteristics isshown. Since it is confirmed that the illuminance in thegreenhouse is effective at 2800 to 9660 lx, we evaluate
this method by changing the weather and illumination
conditions in the future.
We decide the maturity of tomato fruit by comparing
the position of tomato fruit on the infrared image and the
RGB image. Furthermore, the proposed method isapplied to the mosaic image which is the basic image ofthe tomato growth state map, and not only the position of
the tomato fruit but also the maturity information isadded. We aim to realize automatic harvesting of tomato
fruit and tomato production control system that can
predict the yield and harvest time of tomato fruit.

9.
10.

11.

12.

13.

14.

References
1. Bargoti, S., & Underwood, J. P. (2017). Image
segmentation for fruit detection and yield estimation inapple orchards. Journal of Field Robotics, 1039-1060.
2. Lili, W., Bo, Z., Jinwei, F., Xiaoan, H., Shu, W., Yashuo,
L., Qianbinb, Z., & Chongfeng, W. (2017). Developmentof a tomato harvesting robot used in greenhouse.
International Journal of Agricultural and BiologicalEngineering, 10(4), 140-149.
3. Yaguchi, H., Nagahama, K., Hasegawa, T., & Inaba, M.
(2016, October). Development of an autonomous tomato
harvesting robot with rotational plucking gripper. InIntelligent Robots and Systems (IROS), 2016 IEEE/RSJ
International Conference on (pp. 652-657). IEEE.
4. Kondo, N., Yamamoto, K., Shimizu, H., Yata, K., Kurita,
M., Shigi, T., Monta, M., & Nishizu, T. (2009). A machine
vision system for tomato cluster harvesting robot.

15.
16.
17.

18.

Engineering in Agriculture, Environment and Food, 2(2),60-65.
Hatou, K., Hashimoto, Y. (2002). Recognition system for
tomato fruits based on the wire size reduction.
Environmental Control in Biology. 40(1), 75-80.
Ota, T., Yamashita, T., Hayashi, S., & Komeda, T. (2010).
Development of tomato harvesting robot with a visionsystem using specular reflection (Part 1) – Selection of
Hand-Approach Direction by Image Processing-.
Agricultural Machine and Food Engineers, 72(6), 587594.
Fujiura, T., Nakao, S., Kondo, N., Dohi, M., & Yamashita
A (1995). Dichromatic 3-D vision sensor for the
agricultural robot. Agricultural Technology Management,
2(1), 59-64.
Sa, I., Ge, Z., Dayoub, F., Upcroft, B., Perez, T., &McCool, C. (2016). DeepFruits: A fruit detection systemusing deep neural networks. Sensors, 2016, 16, 1222.
Rahnemoonfar, M., Sheppard, C. (2017). Deep Count:
Fruit Counting Based on Deep Simulated Learning.
Sensors, 2017, 17, 905.
Arefi, A., & Motlagh, A. M. (2013). Development of anexpert system based on wavelet transform and artificial
neural networks for the ripe tomato harvesting robot.
Australian Journal of Crop Science, 7(5), 699.
Yamamoto, K., Guo, W., Yoshioka, Y., & Ninomiya, S.
(2014). On plant detection of intact tomato fruits using
image analysis and machine learning methods. Sensors,
14(7), 12191-12206.
Fujinaga, T., Yasukawa, S., Li, B., & Ishii, K. (2017,
September). Generation of mosaic image in tomato
greenhouse using infrared image. Smart Info-Media
System in Asia. 15-20.
Yasukawa, S., Li, B., Sonoda, T., & Ishii K. (2017,
January). Development of a tomato harvesting robot. The
2017 International Conference on Artificial Life andRobotics (ICAROB 2017), 408-411.
Kondo, N. (1988), Selection of suitable wavelength bands
for discrimination between part body their spectralreflection. Agricultural Technology Management, 24(4),175-183.
Dalal, N., & Triggs, B. (2005). Histograms of oriented
gradients for human detection. Computer Vision and
Pattern Recognition. 1(1), 886–893
Cortes, C., & Vapnik, V. (1995). Support-vector networks.
Journal of Machine learning, 20, 273–297.
Comaniciu, D., & Meer, P. (2002). Mean shift: a robust
approach to ward feature space analysis. IEEE
Transaction on Pattern Analysis and Machine Intelligence,24(5), 603-619.
Chen, X., Chaudhary, K., Tanaka, Y., Nagahama, K.,
Yaguchi, H., Okada, K., & Inaba, M. (2015, September).
Reasoning-based vision recognition for agricultural
humanoid robot toward tomato harvesting. In Intelligent
Robots and Systems (IROS), 2015 IEEE/RSJ International
Conference on (pp. 6487-6494). IEEE.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan766

Comparative Study of Sequential Processing Terrain Referenced Navigation
Hyun Cheol Jeon

Department of Mechanical & Aerospace Engineering/ASRI, Seoul National University, 1 Gwanak-ro, Gwanak-gu,
Seoul, 08826, South Korea
Chan Gook Park

Department of Mechanical & Aerospace Engineering/ASRI, Seoul National University, 1 Gwanak-ro, Gwanak-gu,
Seoul, 08826, South Korea
E-mail: smartjhcz@snu.ac.kr, chanpark@snu.ac.kr

Abstract
This study performs comparative analysis of sequential processing terrain referenced navigation (TRN) by applying
various filtering techniques. TRN that prevents divergence of inertial navigation system using terrain information isan alternative to GNSS. In this paper, we consider extended Kalman filter (EKF) and point mass filter (PMF) which
are widely used in the TRN. The EKF estimates state variables by linearizing a nonlinear models but it has risk ofdivergence due to linearization errors. On the other hands, the PMF estimates the state variables robustly but it has a
disadvantage of high computational load. Through this study, the advantages and disadvantages of the TRN when
applying the EKF and PMF are compared and analyzed.
Keywords: INS, Terrain Referenced Navigation, Sequential processing, nonlinear filtering, EKF, PMF,

The TRN estimates the position of the vehicle bycomparing the measured terrain elevation with digital
elevation map (DEM) values of the flying area. Generally,a barometric altimeter and a radar altimeter are used formeasuring terrain information, and the TRN algorithms
are divided into two ways, depending on how themeasurements are processed; batch processing andsequential processing TRN [1-3]. Batch processing TRN
performs a position fix by matching a measured profile
consisting of terrain elevation measurements collectedalong the trajectory of the vehicle with candidate profiles
composed of DEM values. On the other hand, sequential
processing TRN performs position correction every timea measurement is input by using a filtering techniquesuch as extended Kalman filter (EKF) or point mass filter(PMF) as shown in Fig. 1.

1. Introduction
As a traditional method for estimating the positionof vehicle, inertial navigation system (INS) has been used.
However, since errors are accumulated in the process of
integrating acceleration and angular velocity, the positionof the vehicle calculated by the INS diverges over time.
To overcome this shortcomings of the INS, INS/GNSS
integrated navigation system that combining the INS andglobal navigation satellite system (GNSS) has been
widely used. Although this system can provide theposition information of the vehicle with a bounded error
and a fast update rate, the GNSS is vulnerable to externalsignal disturbances such as jamming and spoofing.
Therefore, terrain referenced navigation (TRN) has beendeveloped which is robust to the external signal
disturbance and can perform accurate navigation for along time.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan767

Hyun Cheol Jeon, Chan Gook ParkEKF is a nonlinear filtering method of estimating
the error states by linearizing the nonlinear system andmeasurement models [4]. It is relatively simple and has asmall amount of computation, but it has a disadvantage
in that it can be degraded or diverge when the
nonlinearity is large or the initial position error is large.
The PMF is a representative numerical Bayesian filtering
technique that expresses probability distributions bydividing the state space into grids at regular intervals andestimates the states using this probability distribution [5].
Since the PMF uses the models as it is, it has anadvantage that the state variable can be estimatedeffectively even for nonlinear and non-Gaussian noisemodels. However, because the amount of computation
increases exponentially with the number of statevariables, it is impossible for the PMF to estimate manystate variables.
In this study, these two nonlinear filtering
techniques are applied to the TRN and the results arecompared in terms of computational complexity andperformance. It is shown that which filter should beapplied to the sequential processing TRN depending onwhether a number of state variables are estimatedinaccurately or a small number of state variables are
estimated accurately.

xk

f k 1 (x k 1 ) w k 1ykhk ( x k ) v k(1)

where f k 1 , hk 1 represent nonlinear system andmeasurement model, respectively. wk 1 , vk 1 represent
system and measurement noise, respectively.
In the EKF based TRN, f k 1 represents navigation
equation and hk 1 represents the vertical distance
between the vehicle and the surface measured by theradar altimeter. The linearization of (1) is expressed as,
fk 1
xk
xk 1 w k 1
xk 1
(2)
hk
yk
x k vkxk
where x k expresses the error states.
The measurement residual
yk represents thedifference between the actual radar altimetermeasurement and the estimated radar altimetermeasurement at the estimated vehicle position, which canbe expressed asLterrainterrain
hDEM
hDEM
yk
1
l
vk
L
l
(3)
h
H k xkvk

vk ~ N (0, Rradar

RDEM )L, hl represent a latitudinal
where, hdirection and a longitudinal direction slope, respectively.
L , l , h represent latitude, longitude and altitude,
respectively.
It is possible to estimate the position of the vehicle
by applying the following Kalman filter to the linearized
model like (2), (3).
Pk Fk Pk 1FkT Q kterrainDEM

2. EKF based TRN
The EKF is a nonlinear filtering technique that applies astandard Kalman filter to the error states by linearizing
the nonlinear system and measurement models. Because
the computational load of the EKF is smaller than that ofother nonlinear filters, it is possible for the EKF toestimate many state variables such as position, velocity
and attitude of the vehicle using the navigation equation
as a system model. The following is a general nonlinear
system and measurement model. The following is general
nonlinear system and measurement model.

terrain
DEM

Kk

Pk HTk H k Pk H Tk

Pk

I K k H k Pkxk

K k ykRk

1(4)

3. PMF based TRN
The PMF is a numerical solution of the following
Bayesian filtering technique, which makes it possible toobtain robust and accurate estimation results fornonlinear and non-Gaussian problems.

p( xk | Yk 1 )p( xk | Yk )Fig. 1. Block diagram of the sequential processingTRNp ( xk | xk 1 ) p( xk 1 | Yk 1 )dxk 1p( yk | xk ) p ( xk | Yk 1 )p ( yk | xk ) p( xk | Yk 1 )dxk

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

768

(5)

Comparative Study of Sequential

However, in the process of performing theprediction, it is necessary to consider the relation with all
propagated points in order to calculate the priorprobability or the weight at one redesigned grid point asshown in (6). Therefore, in general, a large amount ofcomputation corresponding to O (N2) , where N is thetotal number of grid points.
Np( x | Yk 1 )ikp( x | Yk )p( x | x

j 1

i
k

j
k 1

) p( xj
k 1| Yk 1 )the state variables is computed through the weighted sumof each grid point and its corresponding probabilityxˆ k 1

i
k

(6)

i
k

T

and
where ξ k
Lk lk
ξ k 1 represents travel
distance calculated by the INS. g k represents terrain
elevation.
Unlike the EKF based TRN, the PMF based TRN
does not estimate altitude, so it use terrain elevation as ameasurement, not a vertical range. Therefore, abarometric altimeter measurement hk ,baro and a radar
altimeter measurement hk , radar are used to measureterrain elevation.
zk hk ,baro hk , radar

htrue

vk ,baro

hk , radar

htrue

hterrain(8)

vk , radar

Assuming that all measurement errors follow theGaussian distribution, zk is compared with the DEM
value at the grid point z DEM ξ ik , and the likelihood value
is calculated as follows.
z ξ ikp ( zk | ξ ik )z DEM ξ ik

zk
Nz ξ ik ;0, Rradar

Rbaro

RDEM

(10)

To compare the performance of EKF based TRN
and PMF based TRN, the TRN simulation was performed
under the following conditions. It is assumed that thevehicle flies toward north at constant altitude withconstant velocity as shown in Table 1. The INS was
performed to simulate the movement of the vehicle, anda navigation grade IMU was applied. It was assumed thata barometric altimeter and a radar altimeter were used as
the sensor for measuring the terrain information, and the
SRTM 3arcsec is used as DEM. All measured noise isassumed to follow the Gaussian distribution, and its 1
sigma values are shown in the Table 2. To compare thesimulation results, each algorithm was performed 30
times with Monte Carlo simulation, and the result was
expressed as root mean square error (RMSE) as shown in
Fig. 2.
As mentioned above, the PMF uses a nonlinear
model without any assumptions such as linearization, so
it can be seen that the estimation results are more accurate
than that of the EKF. In addition, since the PMF does not
estimate the altitude, it is necessary to use a barometricaltimeter, so the inaccuracy of the measurement is greaterthan that of the EKF. However, better results are obtained
due to the use of a more accurate model. The calculation
time was 11.09 seconds for the EKT based TRN and19.34 seconds for the PMF based TRN for one run. This
is not a big difference because the PMF uses the kernel
to calculate prior distribution. If the PMF is performed
with iterative operation, much more computation time isrequired. Moreover, the EKF estimates 15 state variables,
whereas the PMF estimates only 2 state variables. Thus,
the EKF is much less computationally intensive for thesame number of state variables.

Therefore, the PMF based TRN generally uses thelinear system and nonlinear measurement model like (7),which estimates latitude Lk and longitude lk and uses
terrain elevation z k as a measurement. In particular, inthe case of TRN using the linear system model, it ispossible to calculate the prior distribution using thekernel, so it is possible to greatly reduce the calculation
amount of the PMF [6].
ξk ξk 1
ξk 1 wk 1(7)
zk g k (ξ k ) vk

hk ,baroi 1

ξ ik 1 ki 1|k 1

4. Simulation Resultsp( yk | x ) p ( x | Yk 1 )

i
k

N

(9)

Table 1. Trajectory conditions
Parameters
LatitudeLongitude
Velocity
Altitude
Table 2. Sensor conditions

where ξ represents arbitrary redesigned grid point.
Rradar , Rbaro , RDEM represent measurement error variances,respectively.
The posterior distribution is calculated based on (6)using the calculated prior and likelihood. Estimation ofikValues35.2° - 35.75°
127.75°240m/s2000m© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan769

Hyun Cheol Jeon, Chan Gook Park

measurements has obtained much more accurate
estimates than that of the EKF. It is because the PMF uses
a nonlinear model without any assumptions such as
linearization. On the other hand, it can be seen that thecomputational load of the PMF that estimates only two
state variables is larger than that of the EKF which
estimates 15 state variables, and it is found that the PMF
requires much more computational load in estimating thesame number of state variables. By this research, it canbe shown that which filter should be applied to thesequential processing TRN depending on whether anumber of state variables are estimated with relatively
degraded accuracy or a small number of state variables
are estimated with relatively accurate performance.

35
EKF
PMF

30

25

20

15

10

5

0

50

100

150

200

250

300

time [sec](a) Latitude error
40
EKF
PMF

35

Acknowledgements
This work has been supported by the Ministry of Science,
ICT & Future Planning of Republic of Korea under SpaceCore Technology Development Program (Project
number NRF-2013M1A3A3A02042468).

30

25

20

15

References
1. L. Zhao, N. Gao, B. Huang, Q. Wang and J. Zhou, A Novel
Terrain-Aided Navigation Algorithm Combined with the
TERCOM Algorithm and Particle Filter, IEEE SENS J.,
15(2) (2015), 1124-1131.
2. P. J. Nordlund and F. Gustafsson, Marginalized Particle
Filter for Accurate and Reliable Terrain-Aided Navigation,
IEEE Trans. Aerosp. Eletron. Syst., 45(4) (2009), 13851399.
3. N. Bergman, Recursive Bayesian estimation: Navigationand tracking applications, Ph.D. dissertation, Linköping
Studies in Science and Technology, 1999.
4. D. Simon, Optimal State Estimation: Kalman, H infinity,
and nonlinear approaches (Hoboken, NJ, John Wiley &Sons Inc., 2006)
5. M. Šimandl, J. Královec and T. Söderström, Advanced
point-mass method for nonlinear state estimation,
Automatica, 42(7) (2006), 1133-1145.
6. T. Furukawa, B. Lavis and H. F. Durrant-Whyte, Parallel
grid-based recursive Bayesian estimation using GPU forreal-time autonomous navigation, in 2010 IEEE
International Conference on Robotics and Automation,(Anchorage, AK, 2010), pp. 316-321.

10

5

0

50

100

150

200

250

300

time [sec](b) Longitude error
Fig. 2. RMSE results of the sequential processing TRN
Sensor
Baro altimeter
Radar altimeter
DEM

5.

Error (1σ)
5m
15m
5m

Conclusions

In this paper, sequential processing TRN using
various nonlinear filtering algorithm is studied. Since the
TRN uses nonlinear terrain measurement, it is necessary
to apply a nonlinear filter. Two nonlinear filters areapplied to the TRN and the results of simulation arecompared. The simulation results shows thecharacteristics of the two filtering techniques. Althoughboth filters robustly estimate the vehicle position in thesimulated area, the PMF using more inaccurate© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

770

Road Marking Map Matching for Road Vehicle Localization
Kyu Won Kim

Department of Electronic Engineering, Konkuk University, 120 Neungdong-ro, Gwangjin-gu,
Seoul, 05029, South Korea
Gyu In Jee

Department of Electronic Engineering, Konkuk University, 120 Neungdong-ro, Gwangjin-gu,
Seoul, 05029, South Korea
E-mail: kkw1125@konkuk.ac.kr, gijee@konkuk.ac.kr

Abstract
In this paper, we propose map matching algorithm using road marking intensity map made of 3D LIDAR. The roadmarking is very useful in terms of availability because it is existed anywhere on the vehicle. The map matchingalgorithm is used phase correlation using FFT to enable very fast matching. In addition, the road marking map isgenerated by binarizing the intensity. As a result, a large-scale global map can be stored with a small capacity. Also,
LIDAR scan data was accumulated for generating a high precise density local map. Accumulated local map can besupported sufficient information, especially longitudinal information. Therefore, it can be more robust and precise
vehicle localization.
Keywords: autonomous vehicle, 3D LIDAR, binary road marking map, map matching.

information of the global map is binarized to generate 2D
binary road map for map matching. As a result, a large
global map was created with a small capacity. Map
matching algorithm using FFT-based phase correlation
was performed to enable real-time matching. In order toderive a robust matching performance, sensor dataobtained from 3D LIDAR is accumulated to generate aprecise local map. Through this, we supplemented therelatively weak longitudinal information and derived
road marking information that can be matched even in therush hour.

1. Introduction
Recently, localization is becoming an issue ofautonomous vehicle research. Generally, GPS is used asa sensor for localization. However, GPS has large error
in urban area. To solve this problem, error correction
using map is used. And the position of the vehicle isestimated by correcting the error by matching the dataobtained through the map and the sensor. In general,maps are used for location estimation by storing roadenvironments such as lanes [1], traffic lights [2], andcurbs [3]. Among them, the road marking map using thereflectivity of LIDAR is highly available because it uses
the road markings existing everywhere on the road.
In this paper, we propose a method of generating a
map using the reflectance of road marking and matching
the position with the 3D LIDAR scan data. First, the map

2. 2D Binary Road Marking Map Generation
First of all, we generate Global road marking map. M
ap must be Accurate and precise for accurate location es
timation for accurate localization. Therefore, the initial t
rajectory was generated using the expensive GPS-RTK© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan771

Kyu Won Kim, Gyu In Jee(Real Time Kinematic) / INS for precise map generation.
However, when the map is generated using the basic SLAM [4], the road marking in the map is not consistent. To solve this problem, we performed trajectory optimizati
on using Graph SLAM [5] to generate precise and accur
ate map trajectories. Based on the generated map trajectory, the map data was generated by accumulating the int
ensity data scanned with 3D LIDAR. At this time, the reflectivity information was calculated by averaging eachgrid in the map.
However, in order to calculate the reflectivity average, itis necessary to put the average and the number of scanseach grid. In such a case, the map capacity increases
exponentially when generating a wide range of global
map. In order to prevent this, the average of thereflectance in the map is binarized. Also, we applied the
Otsu Threshold method to generate threshold. Fig. 1
shows the result of generating the road markings withintensity and binary maps. The road markings are clearly
extracted. And, Table 1 shows the capacity of each map,the map resolution is 0.1m. The intensity map shows 121
MB of storage capacity by storing the average and thenumber of scan. On the other hand, capacity of binary
map is greatly reduced. And if you make a binary map

3. Precise Local Map Generation
The localization using the map is performed bymatching the global map and the local map obtained
through LIDAR. Also, we have accumulated the LIDAR
data [6] according to the progress of the vehicle and we
use this to create a precise local map. A precise local map
compensates for insufficient longitudinal information
because it uses accumulated data according to thetraveling direction of the vehicle. Fig. 2 shows how
LIDAR data scanned from the road marking accumulates
as the vehicle travels. As shown in figure, the previously
scanned data is pushed backward as the vehicle moves
and you can see the accumulation of the current scanned
data. Also, it fills in the empty space between the layers
as the data accumulates. So, the problem of reducing thedensity of the road marking is solved. Fig. 3 shows theprocess of generating a precise local map. As theaccumulation progresses, it can be seen that the roadmarking information increases so as to clearly distinguishthe road surface display in the longitudinal direction.
4.

Map Matching Algorithm

4.1.
FFT-based Phase Correlation
Localization of the vehicle is performed by matching aprecise local map with the global map. The map matching
method is based on FFT-based phase correlation. The FFTbased phase correlation equation is as follow.
corr (m, z )F 1{

M ( f ) : F {m(t )}(a)Fig. 1. Road Marking Map Generation Result.
(a) Intensity Map (b) Binary Map
Table 1. Capacity of Map
Intensity
Map

Binary Map

Binary Map(Sparse Matrix)

Capacity

121MB`

2.04MB

449KB

Z ( f ) : F{z (t )}(1)

Fig. 4 shows the FFT-based correlation results. (a) is a crosscorrelation result, and (b) is a phase correlation result. As shownin this figure, it can be seen that the (b) Correlation Peak is
sharper than (a). However, even if phase correlation is used,noise is generated. And if Peak is 2 or more, wrong matching
result can be obtained. To solve this problem, we applied a
Gaussian filter based on the previously estimated position. As aresult, noise is removed and only Peak results near the estimatedlocation can be used, even if peaks occur at other locations. The
Gaussian filter equation is as follow.

(b)

Map area(1.4km x 2km)

M ( f )Z ( f )}M ( f )Z ( f )with sparse matrix, you can remove the place where 0 isincluded. So, the capacity will be further reduced.

F

F© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan772

Road Marking Map Matching

Ppeak max(corr )[ x[ x peak y peak ] find (corrc cov([ xPpeak )[ xmean ymean ] [ E[ x peak( t k : t ) ] E[ y peak(t k : t ) ]]Pgaussexp((xxmean ) 2 ( y2 2ymean )2

Pw corr Pgaussy ] [( xtpos

xt 1 ) ( ytyt 1 )]y ])max(eigenvalue(c))

K Rayleigh 1 (1 Pmd ))Pmd : probability of false matching
PL K

(2)

(3)

pos

In addition, matching reliability can be judged through peak
sharpness. Because it means that the matching result is certain.
Therefore, we used the PSR to determine the sharpness of thepeak. PSR is the ratio between the peak and the noise aroundpeak. The larger the PSR value, the sharper the peak. The
equation for obtaining PSR is as follow.
PSR(a)Fig. 4. FFT-based Correlation Result.

(b)peak mean( sidelobe)
std ( sidelobe)(4)

When the PL and PSR are obtained described above, thereliability of the matching result is determined based on the twoparameters. Matching result is more reliable when PL is lower
and PSR is higher. Fig. 6 shows an example of threshold setting

Fig. 5. Example of Gaussian Filter application result.

Fig. 5 shows an example of Gaussian filter application result.
As shown in this figure, the noise except Peak is disappeared.
Also, peaks at other positions are removed, and a sharp peakcan be extracted based on the estimated position.

Fig. 6. Example of PL & PSR Threshold.

of PL and PSR. The results in (a) of Fig. 6 show high matchingreliability. Therefore, this study used map matching results in (a)range, and if the results of other ranges were derived, Dead
Recoking was performed through LIDAR Odometry using ICP
until reliable results were obtained.

4.2.
Matching Confidence Judgement
In the case of the localization using the map, if the matchingis wrong, a large error may occur. Especially, in the case of
autonomous vehicles, wrong positioning results can lead toserious accidents. Therefore, if the matching result is wrong, itshould be detected and the wrong matching result should not be
used. In this study, two parameters were generated to determine
the reliability of the matching result. One is the PL (Protection
Level), and the other is a PSR (Peak to Sidelobe Ratio) that
determines the sharpness of the correlation. Equation of PL is
as follow.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan773

Kyu Won Kim, Gyu In Jee

Fig. 8. Position Error.
Fig. 7. Experimental Area.

Table 2. RMS Position Error

5. Experimental Result
In order to verify the localization performance using the map
matching algorithm proposed in this paper, we performedexperiments in the urban area where the GPS error is large asshown in Fig. 7. And, we used the data of the rush hour time toestimate localization performance when there are many nearby
vehicles. Also, 3D LIDAR used in the experiment was
Velodyne HDL-32E. The reference trajectory used for the error
analysis is based on the vehicle trajectory optimized throughGraph SLAM in Section 2.
Fig. 8 shows the position error. As shown in Table 2, theGPS / DR RMS error occurs more than 6m, while the RMS
error is less than 15cm in both lateral and longitudinal whenusing the map matching algorithm. Therefore, we confirmedthat the map matching algorithm of this study has very precise
and reliable positioning result.

RMS Error[m]LateralLongitudinal

GPS/DR

6.79

6.28

Map Matching

0.149

0.146

References
1. D. Kim, T. Jung, and K.-S. Yi, “Lane map-based vehicle
localization for robust lateral control of an automatedvehicle,” Journal of Institute of Control, Robotics andSystems (in Korean), vol. 21, no. 2, pp. 108-114, 2015.
2. J. Levinson, J. Askeland, J. Dolson, and S. Thrun, “Traffic
light mapping, localization, and state detection forautonomous vehicles,” Proc. of Robotics and Automation(ICRA), 2011 IEEE International Conference on. IEEE,May 2011.
3. A. Y. Hata, F. S. Osorio, and D. F. Wolf, “Robust curb
detection and vehicle localization in urban environments,”
2014 IEEE Intelligent Vehicles Symposium Proceedings.
IEEE, 2014.
4. C.-C. Wang and C. Thorpe, “Simultaneous localizationand mapping with detection and tracking of moving
objects,” Robotics and Automation, 2002. Proceedings.
ICRA'02. IEEE International Conference on, vol. 3. IEEE,
2002.
5. Thrun, Sebastian, and Michael Montemerlo. “The graph
SLAM algorithm with applications to large-scale mappingof urban structures,” The International Journal of Robotics
Research, 25.5-6, pp. 403-429, 2006.
6. Z. J. Chong, B. Qin, T. Bandyopadhyay, M. H. Ang, E.
Frazzoli, and D. Rus, “Synthetic 2d lidar for precise
vehicle localization in 3d urban environment,” Proc. of
Robotics and Automation (ICRA), 2013 IEEE
International Conference on. IEEE, 2013.

6. Conclusion
In this paper, we propose a map matching algorithm using
binary road marking map. And, we used a binary road marking
map to store map information on a wide area in a small capacity.
In addition, a precise local map is generated to supplement thelacking longitudinal road marking information, and matching ispossible even when there are many obstacles such as a vehicle.
And we applied the map matching algorithm using FFT - based
Phase Correlation to improve the processing time and to obtainaccurate and reliable localization result using the reliability
determination algorithm.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan774

Control System Design of Directionally Maneuvering Multicopter with Constant Yaw Rate
Jaehyu Bae, Dong-Gyun Kim, Byoungjin Lee and Sangkyung Sung

Department of Aerospace Information Engineering/Konkuk University, 120 Neungdong-ro, Gwangjin-gu, Seoul, 05029,
Korea
Joon Goo Park

School of Electronics Engineering/Kyungpook National University, 80 Daehakro. Bukgu. Daegu. 41566. Korea
E-mail: schumir@konkuk.ac.kr, sksung@konkuk.ac.kr

Abstract
This paper presents the controller design, implementation and flight test results of a multicopter system for achieving
vertical axis rotating motion and a separated maneuvering in horizontal axis, simultaneously. For this, formulation
for position and velocity control of the multicopter is designed and the trajectory guidance method is exploited.
Especially, the quaternion - based motions and thrust control equations are newly derived, which can generate an
independent differential thrust in the horizontal axis while maintaining a constant vertical axis rotation. In order toverify the performance of the designed controller, the algorithm is implemented in an onboard flight control computer.
Flight test results demonstrate trajectory errors within 2m during cyclic motion.
Keywords: Multicopter, Rotating Motion, Separated Maneuvering, Differential Thrust, Flight Test,

Therefore, it is not necessary to fix the head of the gas
depending on the shape of the gas, and the instantaneous
horizontal posture change can be used even when an
arbitrary start is required.
Also, when using the sensor to acquire theinformation of the surrounding environment by using
these characteristics, it is possible to obtain more
information by rotating the sensor only by the operationof the gas separated from the horizontal maneuver
without adding an actuator such as a gimbal for rotating
the sensor can do.
In this paper, we propose the automatic flight control
experiment including the cyclic heading maneuver using
the Hexacopter. Section 2 introduces the dynamic modelof Hexacopter which is used as a platform, and Section 3
explains the principle of the design of quaternion-based

1. Introduction
Multicopter is a type of unmanned aerial vehicle
capable of vertical takeoff and landing. It is simpler instructure than a helicopter, which is the same verticaltakeoff and landing aircraft, and is easily used byindividuals, contributing to the private use of drones.
Recently, small drones with the shape of a multicopter
have been developed and their utilization is getting
higher. Small Multicopter is used for leisure sports,
spraying of pesticides, delivery service, and is often used
to acquire information about the environment such as
aerial photographing and 3D spatial informationacquisition. [2]On the other hand, the multicopter type gas has asymmetrical structure about the front, rear, left and rightreference axes in terms of shape and kinetic model.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

775

Jaehyu Bae, Dong-Gyun Kim, Byoungjin Lee, Sangkyung Sung, Joon Goo Parkin the figure. And is a constant due to the relationshipbetween the thrust and the rotation moment of the rotor.
The sum of the thrusts generated by each rotor by the total
thrust equations is equal to the magnitude of the total
force, as shown below.

horizontal start separation cyclic heading controller andthe rotation maneuver. Section 4 introduces vehicle andflight experimental results, and conclusions are made in
Section 5.

6

Fcon

2. Multicopter Dynamics Modeling
The multicopter system representatively considered inthis paper is hexacopter, while it can be extended to othersymmetric multicopter system model. The dynamics ofthe hexacopter are derived by the Moment Equation,
Total Thrust Equation and Energy Equation. Figure 1
shows the defined hexacopter model. [5]

i 1

Ti

(2)

Formulated as the total thrust equation in (2), the force to begenerated for the trajectory control is derived from the position/ velocity controller in Section III. And energy equation isderived to efficiently distribute the total amount of thrust to begenerated to each rotor.

E

6
i 1

Ti 2 , min( E ) condition(3)

The energy is assumed to be the sum of the squares of
the thrusts of each rotor and is used under the condition
that energy is minimum. The thrust to be generated ineach rotor to satisfy all three conditions; that is moment,
total thrust, and energy equation is Eq. (4). [3]Figure 1 Hexacoptor System Model.

Each rotor was ordered in the counterclockwise
direction to the front right by 1 and the distance from thecenter of gravity to the center of each rotor was defined.
Moment Equation can be used as the moment of rotationof each rotor by the thrust of the rotors.
l
Mx
M con

My
Mz

T2
2

T3 T4

T1
2

T5
2

T1

Mx
6 l

T2

Mx
6 l

2 3 l
My
2 3 l

Mz
6

Fcon

Mz
6

Fcon

6
6

T3

Mx
3 l

Mz
6

Fcon

T4

Mx
6 l

My

Mz
6

Fcon

T5

Mx
6 l

Fcon

2 3 l

Mz
6

T6

Mx
3 l

Mz
6

T6

3
l T1 T2 T4 T52
T1 T2 T3 T4 T5 T6

My

2 3 l
My

6

(4)

6
6

Fcon
6

3. Control System Design for Maneuvering(1)

The controller generates the control moment via theposition and velocity controller and the attitude controller
based on the information provided in the guidance andnavigation. To generate the control moment, we calculate
the thrust to be generated in each rotor by Hexacopter
dynamics in Section 2.

M con is the moment that should be generated for thecontrol target, which is derived from the attitudecontroller in Section 3I. Ti is the thrust generated by eachrotor, and the subscript is the number of the rotor defined© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

776

Control System Design of

an
cn

a
, bn
a
an

bn

an

bn

b
b

(6)

q

an cn , an cn

qcon

finner _ qua

Figure 2 Control System Structure.

The position and velocity controller is the process ofgenerating the force that the hexacopter must takethrough the position and speed control, Eq. (5) is theequation.
Fcon

K P, Pos

Pos conPos

K I , Pos

Pos conT

0
0 , Fcon1(7)

Pos dtCdrag Vel con Vel con
K P,VelVel con Vel con

K D,Vel 03 1

Vel Vel(5)

Figure 3 Attitude Quaternion for Generating Control
Force.

Acc

For the ease of attitude regulation, perturbation
quaternion q is computed as (8), which in fact isdifferential quaternion between the Z axis direction
vector of the q-reference body and the Z axis direction
vector of the qcon -reference body.
q is the magnitude of the attitude to be controlled
and required to be converted into the Euler angle. Finallythe moment M con is generated by the PD controller
implementation.

m g

The position control system takes a generalformulation of PI control. PI control is performed on the
difference between the control input value generated by
the guidance and the actual position of the vehicle
provided by the navigation. K P, Pos is the proportional
control gain for the position, and K I , Pos is the integral
control gain.
The velocity control requires a force proportional to
the square of the speed. Cdrag is a drag coefficient and is

M con

a coefficient by aerodynamic force, which is multipliedby the square of the velocity magnitude such that thevehicle can actually actuated. K P,Vel is the proportional

K P , Att f qua 2euler q

K D , Att

03 1

(8)

b

where b is the body rate obtained by gyroscope
For generating constant rotating system, rate
controller is the Z-axis is designed.

control gain for the velocity, and K D,Vel is designed toreact to sudden disturbances by multiplying theacceleration by the differential control gain for thevelocity. And m g is the vertical drag of gravity.
For designing the attitude controller, it is proposed a
new function for deriving the required control force, asin (6). The output of the function q is the quaternion ofrotating vectors between a and b . s rotating in. [1] With
this equation, the objective attitude quaternion for vehicle
to have the desired Fcon is computed as (6)M con, z

K P, Att f qua 2euler qz

K D , Att

rcon

r

(9)

where rcon is the target rate in the rotation axis.
When the constant angular velocity of a vehicle
produces a constant moment, the moment magnitude of
the horizontal axis of the body takes a sinusoidal
waveform, and the output of the motor is also sinusoidal.
Figure 4 shows the moment of the rotating Hexacopter
while generating the moment, and Eq. (10) shows themoment equation at that time.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

777

Jaehyu Bae, Dong-Gyun Kim, Byoungjin Lee, Sangkyung Sung, Joon Goo Park

maneuvering, a sinusoidal PWM input is generated,which is contaminated by external disturbance such asside wind effect.

Figure 63 Rotation Angle (Yaw) .
Figure 4 Moment via Vehicle Rotation.

Mx

M x cosM y sinMy

M x sinM y cos(10)

It can be observed that the formula for the momentchanges in sinusoidal form by (10). Consequently, thegeneral thrust equation of hexacopter is obtained by
T
T

A cos

A Mx

B My

B sin

Mx

Figure 7 Motor control Input.

C Mz

A sin

B cos

My

(11)

5.

C Mz

Conclusions

In this paper, a control system design of
directionally maneuvering is presented with a constantrate motion. Experimental demonstration is presented forits verification.

4. Experimental Results
Figure 5 shows the hexacopter system used in theexperiment. Onboard FCS system is implemented forrealizing the directional maneuvering under fixedrotating motion.

Acknowledgements
This work was supported by the Ministry of Science, ICT& Future Planning of Republic of Korea (NRF2016M1B3A1A01937380 and 2016R1A2B4010719).
References
1. B. J. Lee, “Design and Performance Analysis of an
Attitude Reference System using GPS/INS Velocity
Differential Vectors”, Ph.D. Dissertation Dept. of
Aerospace Information Engineering Konkuk University,2017.
2. D. G. Kim, B. J. Lee, Y. J. Lee, and S. Sung, "Flight
Control Test of Quadrotor-Plane with Hybrid Flight Modeof VTOL and Fast Maneuverability." Journal of Instituteof Control, Robotics and Systems, 22.9, pp. 759-765, Sep.
2016.
3. T. S. Jin, "Dynamic Model and P-PD Control based Flight
Performance Evaluation for Hexa-Rotor Type UAV." Thetransactions of The Korean Institute of ElectricalEngineers, 64.7, pp. 1074-1080, Jul. 2015.

Figure 5 Hexacopter For Flight Demonstration.

A simple trajectory consisting of straight lines isused for flight test. Figure 6 and 7 shows the rotating
angle of the vehicle and the control input at the motor
while directional maneuvering. A constant rate of about
45deg/sec is achieved for rotating motion. For horizontal© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan778

Automatic recognition of benthic species using image processing
Yuki Soejima, Yuya Nishida, Takashi Sonoda and Kazuo Ishii

Department of Human Intelligence Systems
Kyushu Institute of Technology
2-4 Hibikino,Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196 Japan
E-mail: ishii@brain.kyutech.ac.jp,

Abstract
For sustainable use of fisheries resources, it is important to estimate the amount of resources such as

TAC (Total Allowable Catch), TAE (Total Allowable Effort) and their investigations have been
conducted by bottomed net fishing, however, the method suffers environmental damages. The survey
method for wider area that does not affect the ecosystem is needed. Recently, research using anautonomous underwater robot (AUV) is introduced as a new survey method as the solution. We also
introduced an AUV for the survey of fish and crabs in the Sea of Okhotsk. The number of marine lives
is measured manually, and the technique of automatic measurement of the number is needed for theobservation. In this research, we propose an automatic marine life recognition method from seafloor
images.
Keywords: Recognition, Benthic species, Underwater image processing

1.

Introduction

2.

Fishery resources that exist in the deep sea have been
attracting attention, and many marine technologies havebeen developed to acquire resources. Among them, TAC(Total Allowable Catch) and TAE (Total Allowable
Effort) are listed from the Fisheries Agency, and there isa trend to make it possible to use finite marine biologicalresources for a long time. Aquatic biological resources
become sustainable resources that will not be depleted bycontinuing fishing at an appropriate catch. There is apossibility of depletion if catch of fish keeps in freecompetition. Therefore, it is necessary to manage
resource amount to some extent.
In recent years, surveys using autonomous underwater
robots (AUV) are proceeding, and among them, fisheryresources can be observed in local waters. However,since the image captured by AUV is observed by a humanbeing and the number of organisms is measured, atechnique for automatically measuring the number oforganisms in an image becomes necessary. In this study,
we propose a pretreatment detection method to recognize
benthos organisms from deep sea bottom images.

Recognition of benthic species

Images of the ocean floor surface photographed in the
deep sea may be recognized as different organisms even
if they are the same organism because the manner inwhich light is reflected in the image is different.
Therefore, as a preprocessing of object recognition
processing, a detection method was examined to generate
an object divided object.
In the object detection method, object candidate
detection (shown in Fig. 1, an algorithm is shown) for
detecting objects with high attention by sarienzies, object
extraction for dividing living organisms and the seabedsurface from the detected object candidates (an algorithm
is shown in Fig. 2) , As shown in FIG.
In the object candidate detection processing, a color
correction image (showing the original image and thecolor correction image in Fig.3) is input, a saliency map
is generated from the Fourier transform, and the object
candidate area is generated from the binarization of thesaliency map. The values of the sagitti within thecandidate area are summed up, and the average value ofthe pixels is taken as the attention degree of the candidatearea. We will detect candidates with high degree ofattention (Figure 4 shows attention degree and candidate© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

779

Koji Masuda, Yuya Nishida, Takashi Sonoda, Kazuo Ishii

area). In the next extraction processing, since the spaceof the detected candidate image is perceived at the color
distance, it is converted into the Lab color space which isa psychological space which makes the color change
uniform. Reduce color resolution by color reduction
processing, and generate cluster mask for each color
system. As assumptions to determine the number ofclusters, we assume that "There are up to two kinds ofinterested organisms and other organisms at the sametime in the overlooked candidate image, and the ocean
floor surface is visible". The result of extracting theobject is shown in Fig.5.
Input enhanced images

Resize the image
Separate the image to
RGB imeges
Generate Saliency Map

Binalization

Fig.2 Flow of object extraction

Generate Object Map
Detect the object
positions

3.

Recognition Experiments

We verify the recognition rate by the object extracted
data using object detection method of this research.
For the experiment method, prepare 30 seabed surface
images as original images. Input the seabed surface
image and perform color correction processing. Thosewith high degree of attention were detected from thecolor corrected image and the feature vectors
generated from the database by the BoK method werecompared with the feature vectors of the teacher databy the histogram intersection method. As a result ofrecognition, the detection rate of the object candidate
was 90%, the extraction rate of the object was 89%,
and the recognition rate of the object was 67%. Table
1 shows experimental results.

Evaluate the object
candidates
Output the top 10
candidates

Fig.1 Flow of image processing© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan780

Automatic recognition of benthic

seafloor surface image became possible, so extraction offeature amount such as the seabed surface etc. appearing
in other than object is reduced, only the feature amountof the living thing is extracted and object recognition
Accuracy improved. Although the recognition rate isabout 70%, it is about 55% in terms of overall processing
accuracy. Because there are misrecognition of small
objects and small shrimps and small shrimps in asituation where the target organisms are overlooked, it isnecessary to utilize the morphological features of eachkind of organism for classification.

Fig.3 Deep sea image using color correction.

References
1. Y. Junku, T. Ura and G. Bekey, Underwater robots,
Springer Science & Business Media, 2012.
2. L. L. Whitcomb, "Underwater robotics: Out of theresearch laboratory and into the field," ICRA'00. IEEE
International Conference on. Vol. 1., 2000.
3. Y. Nishida, T. Ura, T. Sakamaki, J. Kojima, Y. Ito and K.
Kim, "Hovering Type AUV “Tuna-Sand” and Its Surveys
on Smith Caldera in Izu-Ogasawara Ocean Area," 2013
OCEANS-San Diego. IEEE, 2013.
4. T. Maki, Y. Sato, T. Matsuda, R. T. Shiroku and T.
Sakamaki, "AUV Tri-TON 2: An intelligent platform fordetailed survey of hydrothermal vent fields," IEEE/OES
Autonomous Underwater Vehicles (AUV), 2014.
5. Y. Nishida, K. Nagahashi, T. Sato, A. Bodenmann, B.
Thornton, A. Asada and T. Ura, "Development of anautonomous underwater vehicle for survey of cobalt-richmanganese crust," OCEANS 2015-MTS/IEEE Washington,
2015.
6. T. Ura, "Observation of Deep Seafloor by Autonomous
Underwater Vehicle," IJMS, 42(8), 1028-1033, 2013.
7. M. Bewley, B. Douillard, N. Nourani-Vatani, A. Friedman,
O. Pizarro and S. Williams, "Automated species detection:
An experimental approach to kelp detection from sea-,"
Australasian Conference on Robotics and Automation,
2012.
8. M. Boudhane and B. Nsiri, "Underwater image processing
method for fish localization and detection in submarine
environment," Journal of Visual Communication andImage Representation, 2016.
9. E. O. Hulburt, "Optics of Distilled and Natural Water,"
Journal of the Optical Society of America Vol. 35, Issue
11, pp. 698-705, 1945.
10. T. Nakatani, T. Ura, Y. Ito, J. Kojima, K. Tamura, T.
Sakamaki and Y. Nose, "AUV" TUNA-SAND" and its
Exploration of hydrothermal vents at Kagoshima Bay,"
OCEANS 2008-MTS/IEEE Kobe Techno-Ocean, 2008.
11. R. Szeliski, Computer Vision: Algorithms andApplications, Springer-Verlag London, 2010.

Fig.4 Object candidate detection processing

Fig.5 Object extraction processing
Table 1 Results of Recongition

Kich
iji
Shri
mp
Crab

Num
of
Obje
cts

Num
Of
Detect
ion

Num.
Of
Candi
date

Num
Of
Recogni
tion

Succ
ess
rate
[%]

15

15

15

10

67

15

14

11

7

64

10

7

6

4

67

.

4. Conclusion
With the detection method of this research, object
extraction processing using color information in the© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan

781

Koji Masuda, Yuya Nishida, Takashi Sonoda, Kazuo Ishii

12. B. L. McGlamery, "A computer model for underwater
camera systems," Ocean Optics VI. International Society
for Optics and Photonics, 1980.
13. J. S. Jaffe, "Computer modeling and the design of optimalunderwater imaging systems," IEEE Journal of Oceanic
Engineering, 1990.

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan782

AUV Homing Using Acoustic Chirp Signal
Koji Masuda, Yuya Nishida, Takashi Sonoda, Kazuo Ishii

Department of Human Intelligence Systems
Kyushu Institute of Technology
2-4 Hibikino ,Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196 Japan
E-mail: ishii@brain.kyutech.ac.jp,

Abstract
For autonomous underwater vehicle (AUV), high autonomy is required in order to accomplish mission such as
inspection, observation, manipulation under extreme environments, deep-ocean. One of necessary function for
AUV is acoustic navigation. In this paper, we conducted homing experiments using a small AUV and acoustic
pinger with chirp signal. The chirp signal has the feature that the frequency of signal change gradually, so thatthe signal is suitable to calculate the correlation between the time difference between hydrophones. The homing
experiments are carried out and evaluated through tank tests.
Keywords: AUV, homing control, charp signal

1.

developed a compact acoustic positioning system thatmounting is possible to the AUV. It has threehydrophones and receives signal from pinger. Sound
source detection is calculated by Cross Correlation
Function and Super Short Base Line method. In addition,
we introduce chirp signal as the sound source to develop
a robust acoustic positioning system against noises. In theexperiment of sound source detection with hydrophonearray, we measured ±90 [deg] of area in front. According
to the experimental result, the detection accuracy of thedeveloped system module is within 10 [deg] of average
error and 5 [deg] of standard variation, and the present
result suggested that sound source detection is feasible inthis system.
The positioning of AUV is important. However, acoustic
positioning system is used since GPS cannot be used
underwater. It is grouped into three classes such as Long
Base Line (LBL), Short Base Line (SBL) and Super Short
Base Line (SSBL). LBL is extremely accuracy comparedwith other methods but this method requires someultrasonic beacon at the bottom of sea and its calibration.
SBL and SSBL requires only one beacon. SBL detects
the position of sound source from time difference ofarrival. SSBL does it from phase difference and receiver
array of AUV become compact. Moreover, SBL and
SSBL are suitable for AUV because these methods don’t
require the calibration [3, 4]. In this research, SSBL
method is applied for AUV for ease of operation. This

Introduction

In recent years, observation with underwater robots
have been carried out. However, deep sea is dangerous
environment for human beings due to high pressure,darkness and low temperature. Therefore, the exploration
activity of underwater environment with underwater
vehicle is expected. Especially Autonomous Underwater
Vehicle (AUV) has been attracting attention to work or
research in the water on behalf of the human. In addition,
there is a growing need for a wide range search such as
biological ecology observation and maintenance of theunderwater structure. AUVs can operate without anumbilical cable and can be utilized in various fields havebeen attracting attention [1]. On the other hand, AUV is
restricted the activity time due to battery capacity. Thus,
operation of charging station in water is required for abattery-driven AUV to conduct observations forextended durations [2]. Utilization of charging station in
the water is required for long-term operation to solve thisproblem.
The navigation system of AUV is one of important
things for utilizing charging station. Acoustic positioning
system is used because it is difficult to use radio wave
such as GPS in water. In this paper, we describe anavigation system to the station with acoustic positioning
system. In this system, sound source pinger is mounted
the station and AUV is one meter class. Therefore, we© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan783

Koji Masuda, Yuya Nishida, Takashi Sonoda, Kazuo Ishii

shown in Fig. 2. The experimental pool has a diameterof 6 [m], the distance between pinger and hydrophonearray is r and the relative angle between pinger andhydrophone array is θ_xy. In this experiment, themeasuring relative angle is set in total nine points from-90 [deg] to +90 [deg] in steps of 22.5 [deg].
Furthermore, the distance r is set in 2, 3 and 4 [m]. The
number of samples of each angle are 15.
Figure 3-5 show the results of sound source detection.
The Average in the figures shows average value of 15
data. In this figures, the maximum error at distance of2, 3 and 4 [m] was 9.74, 7.67 and 6.56 [deg] and the
distance between pinger and hydrophone array iscloser, the detection accuracy is lower.
Figure 6 shows one of results of AUV homing
experiments. Target position is about 10 [m] in frontof AUV and the target depth is 3 [m]. AUV succeeded
to reach the target using acoustic sensors.

paper describes a navigation system with acoustic
positioning system for AUV.
2.

Acoustic Positioning System

The block diagram of the acoustic positioning system
is shown in Fig. 1. This system consists of threehydrophones, amplifier, bandpass filter and microprocessing unit (MPU). The acoustic wave received byeach hydrophones is converted into an electrical signal.
The signals are performed amplification and noise
removal by the signal processing circuit. MPU takes asample the signals at 250 [kHz]. The sampling data issent to AUV’s single-board computer and phasedifference is obtained by calculating following Cross
Correlation Function (CCF).
1

𝑅𝑥𝑦 (𝑚) = 𝑀 ∑𝑀
𝑛=1 𝑥(𝑛)𝑦(𝑛 + 𝑚)

(1)

x is value of reference signal voltage. y is value of
received signal voltage. The argument m is obtained as
the phase difference when Rxy is maximum.
CCF show a strong correlation when the target signals
include many frequency range. Therefore, chirp signal isused as sound source with robustness against noises inmind. The chirp signal has a pulse wave form whose
carrier frequency changes linearly with time and it ispossible to obtain many frequency range within a certain
time [5]. The sweep frequency range of pinger is from28.75 [kHz] to 21.25[kHz], the duration time is 1 [msec].

Fig. 2. Experimental environment.

Fig. 1. Block diagram of acoustic positioning system.

3.

Experiments

An experiment of sound source detection with
developed acoustic positioning system module wascarried out. Pinger and hydrophone array are set as

Fig. 3. Result of sound source detection at distancer=2[m].

© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan784

AUV Homing Using Acoustic

4. Conclusion
The acoustic positioning system module mountable
AUV was developed for navigation to charging station.
The detection accuracy was verified in the experimentand the result suggest that it is available as navigation
system. In the future, mounting the developed module on
an AUV and implementation of motion for returning thestation will be conducted.
References
1. H. Kondo, T. Maki, T. Ura, and T. Sakamaki, “AUV
Navigation based on Multi-Sensor Fusion For Breakwater
Observation,” ISARC2006 International Symposium on
Automation and Robotics in Construction, 2006.
2. J. Han, A. Asada, T. Ura, Y. Yamauchi, Y. Yagita, T. Maki,“Noncontact power supply for seafloor geodetic observing
robot system,” J Mar Sci Technol, pp183-189,2007.
3. Y. Eriguchi, I. Kazuo, K. Watanabe, “A Sound Source
Localization System for Underwater Robots,”
Proceedings of the 2008 JSME Conference on Robotics
and Mechatronics, Nagano, Japan, 2A1-A10, 2008.
4. Y. Watanabe, H. Ochi, T. Shimura, T. Hattori, “A Studyon SSBL Underwater Positioning Using SS
Communication,” Proceedings of Symposium on
Ultrasonic Electronics, Vol. 29, pp. 193-194, 2008.
5. I. Arai, Y. Tomizawa, M. HIROSE, “Pulse Compression
Subsurface Radar,” IEICE Trans, Commun, Vol. E83B.No.9, 2000.

Fig. 4. Result of sound source detection at distance
r=3[m].

Fig. 5. Result of sound source detection at distance
r=4[m].

Fig.6 Experimental results of AUV homing.
© The 2018 International Conference on Artificial Life and Robotics (ICAROB2018), Feb. 1-4, B-Con Plaza, Beppu, Oita, Japan785