ｇｓ６

ICAROB
2021

PROCEEDINGS OF THE 2021
INTERNATIONAL CONFERENCE
ON ARTIFICIAL LIFE AND ROBOTICS
January 21 to 24, 2021
On line26th AROB International Meeting Series
Editor-in-Chief
Masanori Sugisaka
Editors: Yingmin Jia, Takao Ito, Ju-Jang Lee
ISBN 978-4-9908350-6-4

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Contents
1

Organization, etc.

1

2

Messages12

34Time Table
Opening Ceremony

17
20

5

Technical paper index

21

6

Abstracts
PS abstracts
OS abstracts
GS abstracts

41
42
89

Authors index

100

6-1
6-2
6-3

7

©ICAROB 2021 ALife Robotics Corp. Ltd.

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

SPONSERED

ALife Robotics Corporation Ltd.

ORGANIZED BYInternational Steering Committee of International Conference on Artificial Life and Robotics(ICAROB)SUPPORTED BYIEEE Fukuoka Section (Japan)

Higashihiroshima City (Japan)

CO-ORGANIZED BY
Chinese Association for Artificial Intelligence (CAAI, P. R. China)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-1

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

ADVISORY COMMITTEE CHAIRS
Moshe Kam (New Jersy Institute of Technology, Former IEEE President, USA)
Kensuke Harada (Osaka University, Japan)
ADVISORY COMMITTEE
Adam Grzech (Wroclaw University of Technology, Poland)
Bruce Eisenstein (Drexel University, Former IEEE President, USA)
Hidenori Kimura (RIKEN & Wasada University, Japan)
Jeffrey Johnson (The Open University, UK)
Jerzy Świątek (Wroclaw University of Technology, Poland)
Joshua M. Epstein (The Johns Hopkins University, USA)
Kai-Tai Song (National Chiao Tung University, Taiwan)
Ken-ichi Tanaka (Executive Fellow, Mitsubishi Electric Corporation, Japan)
Masato Nakagawa (Executive Fellow, DENSO CORPORATION, Visiting Professor of Hiroshima
University, Japan)
Masayoshi Tomizuka (University of California Berkeley, USA)
Moshe Kam (New Jersey Institute of Technology, Former IEEE President, USA)Paul Kalata (Drexel University, USA)Paul Oh (Drexel University, USA)Peter Herczfeld (Drexel University, USA)
Steen Rasmussen (University of Southern Denmark, Denmark)Zengqi Sun (Tsighua Univesity, P. R. China)
GENERAL CHAIR
Masanori Sugisaka(Alife Robotics Corporation Ltd., Japan)(Visiting Professor, The Open University (UK))
CO-GENERAL CHAIRS
Yingmin Jia (Beihang Univesty, P. R. China)
Takao Ito (Hiroshima University, Japan)
Ju-Jang Lee (Honorary professor, KAIST, Korea)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-2

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

VICE GENERAL CHAIR
Ang, Chun Kit (University of UCSI, Malaysia)
Henrik. H. Lund (Technical University of Denmark, Denmark)John. L. Casti (International Institute for Applied Systems Analysis, Austria)
Jangmyung Lee (Pusan National University, Korea)
Luigi Pagliarini (Technical University of Denmark, Denmark)(Academy of Fine Arts of Macerata, Italy)
Mohd Rizon bin Juhari (University of UCSI, Malaysia)
PROGRAM CHAIRMAN
Makoto Sakamoto (University of Miyazaki, Japan)
CO-PROGRAM CHAIR
Marion Oswald (Technische Universität Wien, Austria)
INTERNATIONAL ORGANIZING COMMITTEE
Akira Fukuda (Kyushu University, Japan)
Akira Nakamura (AIST, Japan)
Akinori Abe (Chiba University, Japan)
Andrew Gadsden (University of Guelph, Canada)
Caoli Wang (University of Shanghai for Science and Technology, P. R. China)
Chan Gook Park (Seoul National University, Korea)
Eiji Hayashi (Kyushu Institute of Technology, Japan)
Evgeni Magid (Kazan Federal University, Russia)
Fuzhong Wang (Henan Polytechnic University, P. R. China)
Hazry Desa (University of Malaysia, Perlis, Malaysia)
Hidehiko Yamamoto (Gifu University, Japan)
Hideyuki Suzuki (The University of Tokyo, Japan)
Hiroki Tamura (The University of Miyazaki, Japan)
Hiroshi Kage (Mitsubishi Electric Corporation, Japan)
Hiroshi Matsuno (Yamaguchi University, Japan)
Jiwu Wang (Beijing Jiaotong University, P. R. China)
Jovana Jovic (CNRS-AIST JRL, Japan, France)
Junping Du (Beijing University of Posts and Telecommunications, P. R. China)
Katsunori Shimohara (Doshisha University, Japan)
Kazuo Ishii (Kyushu Institute of Technology, Japan)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-3

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Kenichi Tanaka (Nagasaki Institute of Applied Science, Japan)
Kenji Hashimoto (Waseda University, Japan)
Kevin Voges (Canterbury University, New Zealand)
Kunikazu Kobayashi (Aichi Prefectural University, Japan)
Kuo-Hsien Hsia (Far East University, Taiwan)
Kuo-Lan Su (National Yunlin University of Science and Technology, Taiwan)
Kyungho Park (U.S. ARMY, Japan and USA)
Masao Kubo (National Defense Academy of Japan, Japan)
Masanao Obayashi (Yamaguchi University, Japan)
Maxim Sokolov (Innopolis University, Russia)
Mehta Rajiv (New Jersey Institute of Technology, USA)
Minoru Kumano (University of Miyazaki, Japan)
Peter Sapaty (Ukrainian Academy of Science, Ukraine)
Pierre Parrend (University of Strasbourg, France)
Qiang Cai (Beijing Technology and Business University, P. R. China)
Qu Yanbin (Harbin Institute of Technology, P. R. China)
Shin Jisun (Pusan National University, Korea)
Singo Mabu (Yamaguchi University, Japan)
Takashi Kohno (The University of Tokyo, Japan)
Takashi Ogata (Iwate Prefectural University)
Teruhisa Hochin (Kyoto Prefectural University, Japan)
Tetsuro Hattori (Kagawa University, Japan)
Thi Thi Zin (University of Miyazaki, Japan)
Thomas S. Ray (University of Oklahoma, USA)
Toru Yamamoto (Hiroshima University, Japan)
Toshihro Inukai (DENSO WAVE INCORPORATED, Japan)
Victor Berdonosov (Komsomolsk-on-Amur State University of Technology, Russia)
Yasunari Yoshitomi (Kyoto Prefectural University, Japan)
Yi Chai (Chongqing University, P. R. China)
Yoshifumi Morita (Nagoya Institute of Technology, Japan)
Yoshiro Imai (Kagawa University, Japan)
Zengqiang Chen (Nankai University, P. R. China)
INTERNATIONAL PROGRAM COMMITTEE
Abdul Rahman bin Dullah (Universiti Teknikal Malaysia Melaka, Malaysia)
Akinori Abe (Chiba University, Japan)
Akira Utsumi (The University of Electro-Communications, Japan)
Akihiro Yamaguchi (Fukuoka Institute of Technology, Japan)
Akihito Kanai (Hosei University, Japan)
Ali Selamat (University of Technology of Malaysia, Malaysia)
Aminurafiuddin bin Zulkifli (Multimedia University, Malaysia)
Amornphun Phunopas (King Mongkut’s University of Technology, North Bankok, Thailand)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-4

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Andre Rosendo (ShanghaiTech University, P. R. China)
Anne Jeannin-Girardon (University of Strasbourg, France)
Anton Shiriaev (Norwegian University of Science and Technology, Norway)
Artur Sagitov (Kazan Federal University, Russia)
Ashi Gautam (The University of Tokyo, Japan)
Atsuya Tange (The University of Tokyo, Japan)
Ayumu Tominaga (Kyushu Institute of Technology, Japan)
Bin Zhang (Beijing University of Posts and Telecommunications, P. R. China)
Cecilia Zanni-Merk (INSA-Rouen, France)
Chaoli Wang (University of Shanghai for Science and Technology, P. R. China)
Chia-Nan Ko (Nan kai University of Technology, Taiwan)
Chung-Wen Hung (National Yunlin University of Science & Technology, Taiwan)
Congdao Han (Shanghai Institute of Technology, P. R. China)
Cui Zhang (Beihang University, P. R. China)
Cynthia Matuszek (University of Maryland, USA)
Donglian Qi (Zhejiang University, P. R. China)
Dongmei Ai (Universty of Science and Technology Beijing, P. R. China)
Duangiai Jitkongchuen (Dhurakij Pundit University, Thailand)
Dunwei Gong (China University of Mining and Technology, P. R. China)
Jiao Jia (Beihang University, P. R. China)
Junping Du (Beijing University of Posts and Telecommunications, P. R. China)
Endra Joelianto (Bandung Institute of Technology, Indonesia)
Fabio Guigou (University of Strasbourg, France)
Fei Hao (Beihang University, P. R. China)
Fuzhong Wang (Henan Polytechnic University, P. R. China)
Haibo Li (Royal Institute of Technology, Sweden)
Haisheng Li (Beijing Technology and Business University, P. R. China)
Haruhisa Okuda (Mitsubishi Electric Corporation, Japan)
Haruka Tsuboi (The University of Miyazaki, Japan)
Haruna Matsushita (Kagawa University, Japan)
Heeje Kim (Pusan National University, Korea)
Hidetsugu Suto (Muroran Institute of Technology, Japan)
Hiroyuki Iizuka (Osaka University, Japan)
Hongbo Li (Tsinghua University, P. R. China)
Hongji Ma (Shandong University of Science and Technology, P. R. China)
Hongjiu Yang (Yanshan University, P. R. China)
Hongpeng Yin (Chongqing University, P. R. China)
Hussein Abbass (University of New South Wales, and ADFA, Australia)
I-Hsien Liu (National Cheng Kung Univrsiity, Taiwan)
Ilya Afanasyev (Innopolis University, Russia)
Istvan Harmati (Budapest Institute of Technology and Economics, Hungary)
Ivan Tanev (Doshisha University, Japan)
Jiandong Zhao (Beijing Jiaotong University, P. R. China)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-5

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

JJ Merelo (University of Granada, Spain)
Joono Cheong (Korea University, Korea)
Julio Navarro Lara (University of Strasbourg, France)
Jun Kobayashi (Kyushu Institute of Technology, Japan)
Jung-Shian Li (National Cheng Kung University, Taiwan)
Junping Du (Beijing University of Posts and Telecommunications, P. R. China)
Keiji Kamei (Nishinippon Institute of Technology, Japan)
Keisuke Watanabe (Tokai University, Japan)
Keita Honda (Gifu University, Japan)
Ke Zhang (Chongqing University, P. R. China)
Kenichi Tanaka (Nagasaki Institute of Applied Science, Japan)
Kensuke Ogata (Osaka City University Japan)
Khairul Salleh bin Mohamed Sahari (Universiti Tenaga Nasional, Malaysia)
Khoyratee Farad (The University of Tokyo, Japan)
Kouichi Takeuchi (Okayama University, Japan)
Konstantin Yakovlev (Russian Academy of Sciences, Higher School of Economics, Russia)
Kui Xiang (Wuhan University of Technology, P. R. China)
Leonid Freidovich (Umea University, Sweden)
Levi Timothée (The University of Tokyo, Japan)
Liming Chen (Beihang University, P. R. China)
Lin Li (University of Shanghai for Science and Technology, P. R. China)
Lin Zhao (Qingdao University, P. R. China)
Mamoru Yoshimura (The University of Miyazaki, Japan)
Manabu Yamada (Nagoya Institute of Technology, Japan)
Masahide Ito (Aichi Prefectural University, Japan)
Masahiro Ushio (Kyushu Institute of Technology, Japan)
Masahiro Yokomichi (The University of Miyazaki, Japan)
Masamichi Hori (The University of Miyazaki, Japan)
Masanori Takahashi (Tokai University, Japan)
Masayoshi Kano (Chukyou University, Japan)
Masayoshi Tabuse (Kyoto Prefectural University, Japan)
Masaomi Hatakeyama (university of Zurich, Switzerland)
Max Talanov (Kazan Federal University, Russia)
Meng Duan (Beihang University, P. R. China)
Moeko Tominaga (Kyushu Institute of Technology, Japan)
Mohammad Al-Shabi (University of Sharjah, United Arab Emirates)
Mohammad Biglarbegian (University of Guelph, Canada)
Mou Chen (Nanjing University of Aeronautics and Astronautics, P. R. China)
Nan Xiao (Northeastern University at Qinhuangdao, P. R. China)
Nicolas Monmarché (university of Tours, France)
Noriaki Kuwahara (Kyoto Prefectural University, Japan)
Noritaka Sato (Nagoya Institute of Technology, Japan)
Norrima Mokhtar (University of Malaya, Malaysia)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-6

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Palakorn Tantrakool (King Mongkut’s Institute of Technology, North Bankok, Thailand)
Ping Wang (North China Electric Power University, P. R. China)
Pierre David (University of Strasbourg, France)
Pierre Willaume (University of Strasbourg, France)
Rizauddin bin Ramli (Universiti Kebangsaan Malaysia, Malaysia)
Roman Lavrenov (Kazan Federal University, Russia)
Ruztamreen bin Jenal (Universiti Teknikal Malaysia Melaka, Malaysia)
Ryohei Anshi (Kyushu Institute of Technology, Japan)
R.K.P.S. Ranaweera (University of Moratuwa, Sri Lanka)
Satoshi Ikeda (The University of Miyazaki, Japan)
Sanjay S. Joshi (University of California, USA)
Seong-Ik Han (Pusan National University, Korea)
Shahriman Abu Bakar (Universiti Malaysia Perlis, Malaysia)
Shanbi Wei (Chongqing University, P. R. China)
Shihao Sun (Beihang University, P. R. China)
Shin-ichi Asakawa (Tokyo Woman's Christian University, Japan)
Shin Wakitani (Hiroshima University)
Shumin Fei (Southeast University, P. R. China)
Shyi-Ming Chen (National Taichung University of Education, Taiwan)
Stephen Wilkerson (York College of Pennsylvania, USA)
Takashi Kuremoto (Yamaguchi University, Japan)
Takayoshi Yamada (Gifu University, Japan)
Takuya Fuginaga (Kyushu Institute of Technology, Japan)
Takuya Nanami (The University of Tokyo. Japan)
Taishiro Kishimoto (Keio University, Japan)
Taisuke Akimoto (Kyushu Institute of Technology, Japan)
Takashi Iwamoto (Mitsubishi Electric Corporation, Japan)
Tarik bin Abd Latef (Universiti Malaya, Malaysia)
Taro Shibanoki (Ibaraki University, Japan)
Teruhisa Hochin (Kyoto Prefectural University, Japan)
Tetsuro Katayama (The University of Miyazaki, Japan)
Thomas Noel (University of Strasbourg, France)
Thunyaseth Sethaput (Thammasat University, Thailand)
Tianping Zhang (Yangzhou University, P. R. China)
Tomohiko Takagi (Kagawa University, Japan)
Toru Hiraoka (University of Nagasaki, Japan)
Toshihiro Inukai (DENSO Wave Incorporated, Japan)
Toshinori Nawata (Kumamoto National College of Technology, Japan)
Tsunehiro Yoshinaga (Tokuyama National College of Technology, Japan)
Ussath Martin (The University of Potsdam, Germany)
Wan Khairunizam Wan Ahmad (Universiti Malaysia Perlis, Malaysia)
Weicun Zhang (University of Science and Technology Beijing, P. R. China)
Wenhao Zheng (Beihang University, P. R. China)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-7

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Wenlin Li (Beihang University, P. R. China)
Wisanu Jiviriya (King Mongkut’s University of Technology North Bankok, Thailand)
Xiaocan Wang (Technical University Munich, Germany)
Xiaofeng Su (Beijing Institute of Astronautical Systems Engineering, P. R. China)
Xiaoyan Chen (Tianjin University of Science and Technology, P. R. China)
Xiaoyan Fu (Capital Normal University, P. R. China)
Xuemei Ren (Beijing Institute of Technology, P. R. China)
Xuhui Lu (Beihang University, P. R. China)
Yan Cui (Shanxi Normal University, P. R. China)
Yasunori Takemura (Nishinippon Institute of Technology, Japan)
Yo Horikawa (Kagawa University, Japan)
Yongqiang Qi (China University of Mining and Technology, P. R. China)
Yoshihiro Kita (Tokyo Institute of Technology, Japan)
Youji Kawamura (Kindai University, Japan)
Yu-an Zhang (Qinghai University, P. R. China)
Yue Lin (Beijing Institute of Control Engineering, P.R. China)
Yueqian Liang (China Academy of Elecronics and Information Technology, P. R. China)
Yuji Minami (National Institute of Technology, Ube College, Japan)
Yunju Chen ((Shiga University, Japan)
Yunzhong Song (Henan Polytechnic University, P. R. China)
Zacharie Mbaitiga National Institute of Technology, Okinawa College, Japan)
Zakri bin Ghazali (Universiti Malaysia Pahang, Malaysia)
Zengqiang Chen (Nankai University, P. R. China)
Zhao Peng (Huazhong University of Science and Technology, P. R. China)
Zhengquan Yang (Civil Aviation University of China, P. R. China)
Zhongxin Liu (Nankai University, P. R. China)
Zuradzman bin Mohamad Razlan (Universiti Malaysia Perlis, Malaysia)
LOCAL ARRANGEMENT COMMITTEE
Makoto Sakamoto (University of Miyazaki, Japan)
Masanori Sugisaka (ALife Robotics Corporation Ltd., Japan)
Takao Ito (Hiroshima University, Japan)
HISTORY
The International Conference on Artificial Life and Robotics (ICAROB) resulted from the AROBsymposium (International Symposium on Artificial Life and Robotics) whose first edition was held in1996 and the eighteenth and last edition in 2013. The AROB symposium was annually organized by
Oita University, Nippon Bunri University (NBU), and ALife Robotics Corporation Ltd., under thesponsorship of the Science and Technology Policy Bureau, the Ministry of Education, Science, Sports,
and Culture (Monbusho), presently, the Ministry of Education, Culture, Sports, Science, and©ICAROB 2021 ALife Robotics Corp. Ltd.

P-8

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Technology (Monkasho), Japanese Government, Japan Society for the Promotion of Science (JSPS),
the Commemorative Organization for the Japan World Exposition ('70), Air Force Office of Scientific
Research, Asian Office of Aerospace Research and Development (AFOSR/AOARD), USA. I would liketo express my sincere thanks to not only Monkasho (annually fund support from 1996 to 2013) but
also JSPS, the Commemorative Organization for the Japan World Exposition ('70), and various otherJapanese companies for their repeated support. The old symposium (this symposium has been held
every year at B-Con Plaza, Beppu, Oita, Japan except in Oita, Japan (AROB 5th ’00) and in Tokyo, Japan(AROB 6th ’01).) was organized by the International Organizing Committee of AROB and was cooperated by the Santa Fe Institute (USA), RSJ, IEEJ, ICASE (Now ICROS) (Korea), CAAI (P. R. China),
ISCIE, IEICE, IEEE (Japan Council), JARA, and SICE. The old AROB-symposium expanded much byabsorbing much new knowledge and technologies into it. This history and character of the former
AROB symposiums are passed on the current ICAROB conference and to this journal, International
Journal of Robotics, Networking and Artificial Life (JRNAL). From now on, ALife Robotics Corporation
Ltd. is in charge of management of both the conference and the journal. The future of the ICAROB isbrilliant from a point of view of yielding new technologies to human society in the 21st century. We
also expect to establish an international research institute on Artificial Life and Robotics in the futurewith the help of Japanese Government and ICAROB. This conference invites you all.

AIMS AND SCOPE
The objective of this conference is the development of new technologies for artificial life and robotics
which have been recently born in Japan and are expected to be applied in various fields. Thisconference presents original technical papers and authoritative state-of-the-art reviews on thedevelopment of new technologies concerning robotics, networking and artificial life and, especiallycomputer-based simulation and hardware for the twenty-first century. This conference covers abroad multidisciplinary field, including areas such as:
Artificial intelligence & complexity
Artificial living
Artificial mind research
Artificial nervous systems for robots
Artificial sciences
Bipedal robot
Brain science and computing
Chaos
Cognitive science
Computational Molecular biology
Computer graphics
Data mining
Disasters robotics
DNA computing
Empirical research on network and MOT©ICAROB 2021 ALife Robotics Corp. Ltd.

P-9

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Environment navigation and localization
Evolutionary computations
Facial expression analysis, music recommendation and augmented reality
Foundation of computation and its application
Fuzzy control
Genetic algorithms
Human-welfare robotics
Image processing
Insect-like aero vehicles
Intelligence in biological systems
Intelligent control
Management of technology
Medical surgical robot
Micro-machines
Multi-agent systems
Nano-biology
Nano-robotics
Networking
Neural circuits
Neuro-computer
Neuromorphic Systems
Neuroscience
Pattern recognition
Quantum computing
Reinforcement learning system & genetic programing
Robotics
Software development support method
System cybernetics
Unmanned underwater vehicles
Unmanned Aerial Systems Technologies
Unmanned Aerial Systems designing, controls and navigation
Unmanned Aero vehicles
Virtual reality
Visualization
Hardware-oriented submissions are particularly welcome. This conference will discuss new results inthe field of artificial life and robotics©ICAROB 2021 ALife Robotics Corp. Ltd.

P-10

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

COPYRIGHTS
Accepted papers will be published in the proceeding of The 2021 International Conference on
Artificial Life and Robotics (ICAROB2021) by ALife Robotics Corp. Ltd. Copyright belongs to ALife
Robotics Corp. Ltd. Some of high-quality papers in the proceeding will be requested to re-submit theirpapers for the consideration of publication in Journal of Robotics, Networking and Artificial Life &Journal of Advances in Artificial Life Robotics under agreement of both Editor-in-Chief Dr. Masanori
Sugisaka and 3 reviewers. All correspondence related to the conference should be addressed to
ICAROB Office.
ICAROB Office
ALife Robotics Corporation Ltd.
4-Go, 8-Ban, Higshi 2Cyome, Handadai, Oita 870-1108, JAPANTEL/FAX：+81-97-597-7760
E-MAIL：icarob@alife-robotics.co.jp
Home Page：https://alife-robotics.co.jp/©ICAROB 2021 ALife Robotics Corp. Ltd.

P-11

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

MESSAGES

Masanori Sugisaka
General Chair of ICAROB

Masanori Sugisaka
General Chair(President, ALife Robotics Co.,
Ltd., Japan)(Visiting Professor, The Open
University, UK)

It is my great honor to invite you all to The 2021 International Conferenceon Artificial Life and Robotics (ICAROB 2021). This Conference is changed as
the old symposium from the first (1996) to the Eighteenth (2013) annuallywhich were organized by Oita University, Nippon Bunri University(NBU),
and ALife Robotics Corporation Ltd. under the sponsorship of the Scienceand Technology Policy Bureau, the Ministry of Education, Science, Sports,
and Culture (Monbusho), presently, the Ministry of Education, Culture,
Sports, Science, and Technology (Monkasho), Japanese Government, Japan
Society for the Promotion of Science (JSPS), The Commemorative
Organization for the Japan World Exposition (’70), Air Force Office of
Scientific Research, Asian Office of Aerospace Research and Development(AFOSR/AOARD), USA. I would like to express my sincere thanks to not only
Monkasho (annually fund support from 1996 to 2013) but also JSPS, theCommemorative Organization for the Japan World Exposition (’70),
Japanese companies for their repeated support.
The old symposium was organized by International Organizing Committeeof AROB and was co-operated by the Santa Fe Institute (USA), RSJ, IEEJ,ICASE (Now ICROS) (Korea), CAAI (P. R. China), ISCIE, IEICE, IEEE (Japan
Council), JARA, and SICE. The old AROB symposium was growing up by
absorbing many new knowledge and technologies into it. This history andcharacter was inherited also from ICAROB2014(The 2014 International
Conference on Artificial Life and Robotics, included a series of ICAROB
proceedings in SCOPUS and CPCI-Web of Science now. From now on, ALife
Robotics Corporation Ltd. is in charge of management. This year we have
The 2021 International Conference on Artificial Life and Robotics(ICAROB2020) (26th AROB Anniversary). The future of The ICAROB isbrilliant from a point of view of yielding new technologies to human society
in 21st century.
I hope that fruitful discussions and exchange of ideas between researchers
during Conference (ICAROB2021) will yield new merged technologies forhappiness of human beings and, hence, will facilitate the establishment of
an international joint research institute on Artificial Life and Robotics infuture.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-12

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Yingmin Jia

Yingmin Jia
Co-General Chair

Yingmin Jia
Co-General Chair(Professor, Beihang University,
R .P. China)

Co-General Chair of ICAROB
It is my great pleasure to invite you to The 2021 International Conferenceon Artificial Life and Robotics (ICAROB 2021), in Higashi Hiroshima Arts &Culture Hall Kurara, Hiroshima, Japan from January 21-24, 2021.
ICAROB develops from the AROB that was created in 1996 by Prof.
Masanori Sugisaka and will celebrate her 26th birthday in 2021. So far
many important results have been presented at the past meetings andhave a profound impact on artificial life and robotics. Doubtless, it is reallyone of the most famous international conferences in the field of artificial
intelligence and attract wide interests among scientist, researchers, andengineers around the world.
For a successful meeting, many people have contributed their great efforts
to ICAROB. Here, I would like to express my special thanks to all authors
and speakers, and the meeting organizing team for their excellent works.
Looking forward to meeting you at ICAROB in on line and wishing you enjoy
your stay in on line.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-13

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Takao Ito
Co General Chair of ICAROB

Takao Ito
Co-General Chair(Professor Hiroshima
University, Japan)

It is my great honor to invite you all to the 2021 International Conferenceon Artificial Life and Robotics (ICAROB 2021), to the wonderful city ofHigashi-Hiroshima, Hiroshima Prefecture, Honshu Island, Japan.
The ICAROB has its long history. First launched in 1996 as ISAROB, thisformer organization of ICAROB, was developed under the strong leadershipand yeoman efforts of the President—the internationally famous Professor
Masanori Sugisaka, who is widely acknowledged as the father of AROB. Ourconference has brought together many research scholar, faculty members,
and graduate students from all over the world, and published many
manuscripts in high-quality proceedings as well as highly-reputed journals
every year.
Over the years, dramatic improvements have been made in the field ofartificial life and its applications. ICAROB has provided a foundation forunifying the exchange of scientific information on the study of man-madesystems that exhibit the behavioral characteristics of natural living systems,
including software, hardware and wetware. Our conference shapes thedevelopment of artificial life, extending our empirical research beyond theterritory circumscribed by life-as-we-know-it and into the domain of lifeas-it-could–be. It will provide us a good place to present our new research
results, innovative ideas, and valuable information about artificial
intelligence, complex systems theories, robotics, and management oftechnology.
The conference is on line. I eagerly look forward to personally meeting you
in on line, during the ICAROB 2021 and to sharing a most pleasant,interesting and fruitful conference with you. Do come and make thisconference a fruitful, productive as well as enjoyable event.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-14

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Ju-Jang Lee
Co-General Chair of ICAROB

Ju-Jang Lee
Co-General Chair(Honorary professor, KAIST)The First International Conference on Artificial Life and Robotics (ICAROB)
was held in Oita City, Oita, Japan from Jan. 11th to 13th, 2014. This
year’s Conference will be held amidst the high expectation of theincreasingly important role of the new interdisciplinary paradigm ofscience and engineering represented by the field of artificial life androbotics that continuously attracts wide interests among scientist,
researchers, and engineers around the globe.
Distinguished researchers and technologists from around the world arelooking forward to attending and meeting at ICAROB. ICAROB is becoming
the annual excellent forum that represents a unique opportunity for theacademic and industrial communities to meet and assess the latest
developments in this fast-growing artificial life and robotics field. ICAROB
enables them to address new challenges, share solutions, discussresearch directions for the future, exchange views and ideas, view theresults of applied research, present and discuss the latest developmentof new technologies and relevant applications.
In addition, ICAROB offers the opportunity of hearing the opinions ofwell-known leading experts in the field through the keynote sessions,provides the bases for regional and international collaborative research,and enables to foresee the future evolution of new scientific paradigms
and theories contributed by the field of artificial life and robotics andassociated research area. The twenty-first century will become thecentury of artificial life and intelligent machines in support of humankind
and ICAROB is contributing through wide technical topics of interest that
support this direction.
It is a great honor for me as a Co-General Chair of the 8th ICAROB 2021
to welcome everyone to this important event. Also, I would like to extend
my special thanks to all authors and speakers for contributing theirresearch works, the participants, and the organizing team of the 8th
ICAROB.
I’m looking forward to meeting you at the 8th ICAROB in on line andwishing you all the best.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-15

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

GENERAL SESSION TOPICS
GS1 Chaos & Application (4)GS2 Image & Signal Processing 1 (4)GS3 Image & Signal Processing 2 (3)GS4 Image & Signal Processing 3 (3)GS5 Genetic Algorithms & Robotics 1 (3)GS6 Robotics 1 (4)GS7 Robotics 2 (4)GS8 Medical Data Processing (6)ORGANIZED SESSION TOPICS
OS1 Human-Machine Interface Application (4)OS3 Robot Control (9)OS2 Media Information Processing, Music
Recommendation and Artificial Intelligence (4)OS4 Intelligent UAV and Systems (3)OS5 Advanced Information Processing Applications (6)OS6 Intelligent Control (6)OS7 Information Applications and Security (5)OS8 Intelligent Systems and Control (5)OS9 Intelligent Systems and Life (7)OS11 Robotic Manipulation (3)OS10 Advanced Control Systems and Signal Processing(7)
OS12 System and Control (18)OS13 Intelligent Systems and Robotics (14)OS14 Image Processing (4)OS15 Bridging the Gap Between AI, Cognitive Science,and Narratology (7)OS17 Robotics and machine vision (3)OS16 Software Development Support Method (5)OS19 Artificial Intelligence for Embedded Systems andRobotics (4)OS21 Educational Application of Control Technology(4)
OS23 Advances in Field Robotics and Their
Applications (6)OS20 Virtual Reality and Intelligence Interactions (3)©ICAROB 2021 ALife Robotics Corp. Ltd.

OS18 Natural Computing (3)OS22 Robot Competitions and Education (5)OS24 Mathematical Informatics (5)P-16

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

1/21(Thu.)

Group meeting for the conference (ZOOM ID: 984 9796 8944)

1/22(Fri.) – 1/24(Sun.)ICAROB Secretariat

1/24(Sun)Group meeting for the next conference (ZOOM ID: 914 3254 0230)

TIME TABLE (1/22)
1/22(Fri.)

Meeting Room 1(ZOOM ID: 993 2436 1751)

Meeting Room 2(ZOOM ID: 986 1953 6512)

8:409:00-10:00

Meeting Room 3(ZOOM ID: 984 4367 9770)

Registration
OS19 Artificial Intelligence forEmbedded
Systems
andRobotics (4)Chair: Hakaru Tamukoh

OS21 Educational Application ofControl Technology (4)Chair: Kazuo Kawada

GS2 Image & Signal Processing 1(4)
Chair: Tohru Kamiya

10:00-10:20

Coffee break

10:20-10:50

Opening Ceremony (ZOOM ID: 925 6396 7074)

11:00-12:00

Chair: Takao Ito
Plenary Speech PS1 (ZOOM ID: 966 3705 6919)
Kazuo Ishii
Lunch

12:00-13:00
13:00-14:15

OS2 Media Information
Processing, Music
Recommendation and Artificial
Intelligence (4)Chair: Chair: Yasunari Yoshitomi

OS8 Intelligent Systems andControl (5)Chair Chung-Wen Hung

14:15-15:15
15:15-17:00

Meeting Room 4(ZOOM ID: 971 5454 5491)

OS24 Mathematical Informatics(5)
Chair: Takao Ito

GS5 Genetic Algorithms &Robotics 1 (3) Chair: Hideyuki
TanakaGS3 Image & Signal Processing
2 (3)Chair: Keiko Sakurai

Coffee break
OS9 Intelligent Systems and Life
(7)
Chair: Kuo-Hsien Hsia

©ICAROB 2021 ALife Robotics Corp. Ltd.

OS10 Advanced Control
Systems and Signal Processing(7)
Chair: Takuya Kinoshita

P-17

OS15 Bridging the Gap Between
AI,
Cognitive
Science,andNarratology (7)Chair: Jumpei Ono

OS23 Advances in Field Robotics
and Their Applications (6)Chair: Keisuke Watanabe

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

TIME TEBLE (1/23)
1/23(Sat.)

Meeting Room 1(ZOOM ID: 993 2436 1751)

Meeting Room 2(ZOOM ID: 986 1953 6512)

8:409:00-10:15

Meeting Room 3(ZOOM ID: 984 4367 9770)

Registration
OS14 Image Processing (4)Chair: Joo Kooi Tan

OS16 Software Development
Support Method (5)Chair: Tetsuro Katayama

OS22 Robot Competitions andEducation (5)Chair: Yasunori Takemura

10:15-11:00

Coffee break

11:00-12:00

Chair: Masayoshi Tabuse
Plenary Speech PS2 (ZOOM ID: 912 8494 3957)
Yasunari Yoshitomi

12:00-13:00

Lunch

13:00-14:30

OS5 Advanced Information
Processing Applications (6)Chair: Toru Hiraoka

OS6 Intelligent Control (6)Chair: Yingmin Jia

14:30-15:30
15:30-16:45

Meeting Room 4(ZOOM ID: 971 5454 5491)

OS7 Information Applications
and Security (5)Chair: Kuo-Hsien Hsia

GS6 Robotics 1 (4)Chair: Noritaka Sato

GS1 Chaos & Application (4)Chair: Masato Nagayoshi

Coffee breakOS11 Robotic Manipulation (3)Chair: Kensuke Harada©ICAROB 2021 ALife Robotics Corp. Ltd.

OS17 Robotics and machine
vision (3)Chair: Jiwu Wang

P-18

OS18 Natural Computing (3)Chair: Marion Oswald

GS8 Medical Data Processing(6)
Chair: Shingo Mabu

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

TIME TABLE (1/24)
1/24(Sun.)

Meeting Room 1(ZOOM ID: 993 2436 1751)

Meeting Room 2(ZOOM ID: 986 1953 6512)

8:409:00-9:45

OS4 Intelligent UAV and
Systems (3)Chair: Young Im Cho

GS4 Image & Signal Processing
3 (3)Chair: Taro Shibanoki

OS3 Robot Control (9)Chair: Yizhun Peng

OS13 Intelligent Systems andRobotics (14)Chair: Fengzhi Dai

OS20 Virtual Reality andIntelligence Interactions (3)Chair: R.P.C. Janaka Rajapakse
Lunch

OS3 Robot Control (9)Chair: Yizhun Peng

OS12 System and Control (18)Chair: Huailin Zhao (18)OS1 Human-Machine Interface
Application (4)Chair: Norrima Mokhtar

Coffee break

12:00-13:00
13:00-14:15

Meeting Room 4(ZOOM ID: 971 5454 5491)

Registration

9:45-11:00
11:00-12:00

Meeting Room 3(ZOOM ID: 984 4367 9770)

Group meeting for the next conference (15:00-15:30) (ZOOM ID: 914 3254 0230)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-19

GS7 Robotics 2 (4)Chair: Wisanu Jitviriya

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

The 2021 International Conference on
ARTIFICIAL LIFE AND ROBOTICS(ICAROB2021)
January 21 (Thursday)
17:30-19:30

Group meeting for the conference (ZOOM ID: 984 9796 8944)

January 22 (Friday)
10:20-10:50

Opening Ceremony (ZOOM ID: 925 6396 7074)
Chair: Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
Welcome Addresses
1. General Chairman of ICAROB
2. Co-General Chairman of ICAROB
3. Co-General Chairman of ICAROB
4. Co-General Chairman of ICAROB

Masanori Sugisaka (ALife Robotics Co., Ltd., Japan)
Yingmin Jia (Beihang University, China)
TaKao Ito (Hiroshima University, Japan)
Ju-Jang Lee (KAIST, Korea)

January 23 (Saturday)
Banquet: on line (ZOOM ID: 971 4311 0144)
18:30-20:30
Chair: Takao Ito (Hiroshima University, Japan)
Welcome Addresses
Prof. Yingmin Jia (Beihang University, China)
Prof. Ju-Jang Lee (KAIST, Korea)
Prof. Jangmyung Lee (Pusan National University, Korea)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-20

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

TECHNICAL PAPER INDEXJanuary 22 (Friday)
8:40-Registration
Conference Room:
10:20-10:50 Opening Ceremony (ZOOM ID: 925 6396 7074)
Chair: Tetsuro Katayama (The University of Miyazaki, Japan)
11:00-12:00 (ZOOM ID: 966 3705 6919)
Plenary Speech PS1,
Chair: Takao Ito (Hiroshima University, Japan)
PS1 Tomato-harvesting-robot competition towards smart agriculture
Kazuo Ishii (Kyushu Institute of Technology, Japan)
Meeting Room 1 (ZOOM ID: 993 2436 1751)
9:00-10:00 OS19 Artificial Intelligence for Embedded Systems and Robotics (4)Chair: Hakaru Tamukoh (Kyushu Institute of Technology, Japan)
OS19-1

Convolutional Network with Sub-Networks
Ninnart Fuengfusin, Hakaru Tamukoh (Kyushu Institute of Technology, Japan)

OS19-2

Influence of FPGA implementation methods in High-Level Synthesis
1
Yusuke Watanabe, 2Hakaru Tamukoh
1( RAFT WORK Co., Ltd, Japan, 2Kyushu Institute of Technology, Japan)

OS19-3

A Hardware-Oriented Random Number Generation Method and A Verification
System for FPGA
Sansei Hori, Hakaru Tamukoh (Kyushu Institute of Technology, Japan)

OS19-4

Synthesis of realistic food dataset using generative adversarial network (GAN)
based on RGB and depth images
Obada Al Aama, Hakaru Tamukoh (Kyushu Institute of Technology, Japan)

13:00-14:00 OS2 Media Information Processing, Music Recommendation and Artificial Intelligence(4)
Chair: Yasunari Yoshitomi (Kyoto Prefectural University, Japan)
Co-Chair：Masayoshi Tabuse (Kyoto Prefectural University, Japan)
OS2-1

Comparison of Data Augmentation Methods in Pointer-Generator Model Using
Various Sentence Ranking Methods
Tomohito Ouchi, Masayoshi Tabuse (Kyoto Prefectural University, Japan)

OS2-2Music Recommendation System Driven by Variations in Fingertip Skin Temperature
Mayuka Wada1, Taro Asada2, Yasunari Yoshitomi2, Masayoshi Tabuse2©ICAROB 2021 ALife Robotics Corp. Ltd.

P-21

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

(1Shimadzu System Development Corp., Japan, 2Kyoto Prefectural University, Japan)
OS2-3Music Recommendation System Driven by Interaction between User and Personified
Agent Using Speech Recognition, Synthesized Voice and Facial Expression
Ayumi Matsui1, Miki Sakurai2, Taro Asada3, Yasunari Yoshitomi3, Masayoshi Tabuse3(1Sumitomo Mitsui Card Co., Ltd., Japan, 2TIS Inc., Japan, 3Kyoto Prefectural
University, Japan)

OS2-4

Wallet Operation Evaluation System Using Deep Learning
Junichiro Yamawaki1, Yasunari Yoshitomi2, Masayoshi Tabuse2, Taro Asada2(1SKY Co., Ltd., Japan, 2Kyoto Prefectural University, Japan)

15:15-17:00 OS9 Intelligent Systems and Life (7)Chair: Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
Co-Chair: Evgeni Magid (Kazan Federal University, Russia)
OS9-1

Architecture of a student training computer program for preparing professional outpatient
consulting skills within an electronic medical records system during COVID-19 alertness situation
Sergey Bulatov1, Evgeni Magid2, Enzhe Kharisova1, Roman Lavrenov2, Vitaly Dudin1, Artur
Khazetdinov2 (1Kazan State Medical University, Russia), (2Kazan Federal University, Russia)

OS9-2

Satisfaction Assessment on the Counseling Service System for Full-Time Teacher-Counselor in
Tainan Elementary School
Hsiu-Hao Liu (Chang Jung Christian University, Taiwan)
Yun-Syuan Jhang (National University of Tainan, Taiwan)

OS9-3

Assess The Critical Factors for the Counseling Service System Usage Intention
Hsiu-Hao Liu (Chang Jung Christian University, Taiwan)

OS9-4

Design and Implementation of EMI Suppression Filter for Electronic Commutation
Fans
Ching-Chun Chuang, Chih-Chiang Hua, Chung-Wen Hung, Chun-Jen Yao(National Formosa University, Taiwan)

OS9-5

Application of IOT to Forest Management Taking Fushan Botanical Garden as an
ExampleShuo-Tsung Chen, Chih-Chiang Hua, Ching-Chun Chuang(National Formosa University, Taiwan)

OS9-6

Development of the IoT Module using MQTT Protocol and AES
Jr-Hung Guo, Tzu-Yuan Lin, Kuo-Hsien Hsia
(National Yunlin University of Science and Technology, Taiwan)

OS9-7

Exploring the Intention to Continuance of Learning Programming at Elementary School of Rural
Area by the mBot Robot
Yung-Hsin Cheng, Jia-Ming Hsiao(National Yunlin University of Science and Technology, Taiwan)

Meeting Room 2 (ZOOM ID: 986 1953 6512)
9:00-10:00 OS21 Educational Application of Control Technology (4)Chair: Kazuo Kawada (Hiroshima University, Japan)
Co-Chair: Yoshihiro Ohnishi (Ehime University, Japan)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-22

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS21-1Development of Shock Sensitive Tiny Dummy Robot for Junior High School Rescue
Robot Challenge
Kazuo Kawada, Keisuke Iuchi, Keita Murai, Hiroyuki Y. Suzuki(Hiroshima University, Japan）
OS21-2

Basic Research on Parameter Tuning Skills Evaluation Based on Sensor Car Behavior
Data in Technology Education
Teruyuki Tamai (Hiroshima University / Ehime University),
Yoshihiro Ohnishi (Ehime University), Kazuo Kawada(Hiroshima University)OS21-3

Mini Windmill Generator Kit for homework for Hiroshima Univ. Monozukuri JuniorDoctor Special Educational Program
Hiroyuki Y. Suzuki, Kazuo Kawada, Masayasu Nagamatsu(Hiroshima University, Japan)

OS21-4Development of Teaching Materials to Learn the Efficient Use of Energy
Yoshihiro Ohnishi1, Teruyuki Tamai1, Shinnosuke Mori1 Kawada Kazuo2(1Ehime University, 2Hiroshima University, Japan）

13:00-14:15 OS8 Intelligent Systems and Control (5)Chair: Chung-Wen Hung (National Yunlin University of Science and Technology, Taiwan)
Co-Chair: Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
OS8-1

Networking Integration Application of an Intelligent Production Line for Aerospace
Precision Manufacturing
Chau-Chung Song, Chun-Chi Wang, Chen-Pang Chen, Chung-Wen Hung
(National Yunlin University of Science and Technology, Taiwan)

OS8-2

Low-Cost Indoor Localization Using Sound Spectrum of Light Fingerprints
Chung-Wen Hung, Hiroyuki Kobayashi, Jun-Rong Wu, Chau-Chung Song(National Yunlin University of Science and Technology, Taiwan)

OS8-3

Air Valve Fuzzy Control Combined with Sheet Music Recognition Techniques Applied
to Autoplaying Soprano Recorder Machines
Chun-Chieh Wang, Guang-Ming Jhang (Chienkuo Technology University, Taiwan)

OS8-4

Image Inpainting Techniques Combined with Isolated Pixel Filtering Applied toMultifunctional Drawing Robots
Chun-Chieh Wang, Zhan-Xian Ye (Chienkuo Technology University, Taiwan)

OS8-5

Mobile Robot with Image Recognition -- Using LabVIEW and KNRm
Kuo-Hsien Hsia, Bo-Jung Yang, Jr-Hung Guo, Chang-Sheng Xiao
(National Yunlin University of Science and Technology, Taiwan)

15:15-17:00 OS10 Advanced Control Systems and Signal Processing (7)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-23

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Chair: Takuya Kinoshita (Hiroshima University, Japan)
Co-Chair: Shinichi Imai (Tokyo Gakugei University, Japan)
OS10-1

Nonlinear Internal Model Controller based on Local Linear Models, and its
Application
Shinichi Imai (Tokyo Gakugei University, Japan)

OS10-2

Design of a data-driven control system for reference model design using
predicted signals
Yuki Nakatani, Takuya Kinoshita, Toru Yamamoto (Hiroshima University, Japan)

OS10-3

Design of a Data-driven GMV Controller Using the Nelder-Mead Method
LiYing Shi, Zhe Guan, Toru Yamamoto (Hiroshima University, Japan)

OS10-4

Design of a Databased-Driven GPC for Nonlinear Systems
Zhe Guan, Toru Yamamoto (Hiroshima University, Japan)

OS10-5

Design of a Data-Driven Controller using Open-Loop Data
Y. Nishiya, Takuya Kinoshita, Toru Yamamoto (Hiroshima University, Japan)

OS10-6

Improved Estimation of Sway-Angle for Overhead Crane based on Phase
Difference of Acoustic Signals in Frequency Domain
Hanako Ogawa, Takeshi Yamada, Masayoshi Nakamoto(Hiroshima University, Japan)

OS10-7

Study on an Optimal Design Method for Control Systems based on Bayesian
Optimization
Koichi Hirota, Shin Wakitani, Toru Yamamoto (Hiroshima University, Japan)

Meeting Room 3 (ZOOM ID: 984 4367 9770)
9:00-10:00 GS2 Image & Signal Processing 1 (4)Chair: Tohru Kamiya (Kyushu Institute of Technology, Japan)
GS2-1

Unconstrained Measurement of Heart Rate Considering Harmonics of a Respiratory Signal by
Tactile Sensor
Kazuya Matsuo (Kyushu Institute of Technology, Japan), Toshiharu Mukai (Meijo
University, Japan), Shijie Guo (Hebei University of Technology, China)

GS2-2

Image Registration Method for Chest MDCT Images Based on 2-D Finite Element Method
Takuji Ogimoto, Tohru Kamiya (Kyushu Institute of Technology, Japan)
Takatoshi Aoki (University of Occupational and Environmental Health, Japan)

GS2-3

Detection of Abnormal Shadows in Low-dose CT Images Using CNN
Hiromu Ikeda, Tohru Kamiya (Kyushu Institute of Technology, Japan)

GS2-4

EEG signal extraction method based on HHT and CSP
Lei Gong (Tianjin University of Science and Technology, China)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-24

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

13:00-14:15 OS24 Mathematical Informatics (5)Chair: Takao Ito (Hiroshima University, Japan)
Co-Chair: Makoto Sakamoto (University of Miyazaki, Japan)
OS24-1

Verification of CG character operation by brain wave discrimination
Kenji Sakoma1, Kodai Miyamoto1, Taketo Kamasaka1, Makoto Sakamoto1, Amane
Takei1, Tsutomu Ito2, Takao Ito3 (1University of Miyazaki, Japan)(2National Institute of Technology, Ube College, Japan)(3Hiroshima University, Japan)

OS24-2

Approach to tourism support by aerial video using CG
Kenji Sakoma1, Noritake Seto1, Kodai Miyamoto1, Taketo Kamasaka1, Makoto
Sakamoto1, Amane Takei1, Tsutomu Ito2, Takao Ito3 (1University of Miyazaki, Japan)(2National Institute of Technology, Ube College, Japan)(3Hiroshima University, Japan)

OS24-3

Method for detecting eye misalignment based on movement near the center of the
pupil
Uchida Noriyuki1,2, Kayoko Takatuka2, Hisaaki Yamaba2, Masayuki Mukunoki2,
Naonobu Okazaki2(1Kyushu University of Health and Welfare, Japan), (2University of Miyazaki, Japan)

OS24-4

Basic research on video production for educational support by virtual technology
Kodai Miyamoto1, Taketo Kamasaka1, Kenji Sakoma1, Makoto Sakamoto1, Amane
Takei1, Tsutomu Ito2, Takao Ito3 (1University of Miyazaki, Japan) (2National Instituteof Technology, Ube College, Japan) (3Hiroshima University, Japan)

OS24-5

There is a movement towards the development of hula costume CAD
Taketo Kamasaka1, Kenji Sakoma1, Kodai Miyamoto1, Makoto Sakamoto1, Amane
Takei1, Tsutomu Ito2, Takao Ito3 (1University of Miyazaki, Japan), (2National Instituteof Technology, Ube College, Japan) (3Hiroshima University, Japan)

15:15-17:00 OS15 Bridging the Gap Between AI, Cognitive Science, and Narratology (7)Chair: Jumpei Ono (Aomori University, Japan)
Co-Chair: Hiroki Fukushima (Kyushu Womens’ University, Japan)
Co-Chair: Takashi Ogata (Iwate Prefectural University, Japan)
OS15-1
OS15-2OS15-3

Analysis and Construction of Elements of the Stage Performance Structure in a Kabuki-dance
Miku Kawai (Iwate Prefectural University, Japan), Jumpei Ono (Aomori University, Japan), Takashi
Ogata (Iwate Prefectural University, Japan)
Unchiku Generation with Moving Illustration Using Kabuki Knowledge
Jumpei Ono (Aomori University, Japan), Miku Kawai (Iwate Prefectural University, Japan), Takashi
Ogata (Iwate Prefectural University, Japan)
Implementing Story Units of Japanese Folktales with Conceptual Dictionaries
Takuya Ito (Iwate Prefectural University, Japan), Jumpei Ono (Aomori University, Tokyo) Takashi
Ogata (Iwate Prefectural University, Japan)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-25

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS15-4The Story Generation Process and Cognitive Biases
Jun Nakamura (Chuo University, Japan)

OS15-5Time in an Aesthetic Experience of a cup of Sake
Hiroki Fukushima (Kyushu Womens’ University, Japan)

OS15-6

Extension of Clinical/Psychological Approach Using Post-Narratology: Possibility of application
on Artificial Intelligence and Robot
Kai Seino (National Rehabilitation Center for Persons with Disabilities, Japan)

OS15-7A relationship between narratology and marketing
Akinori Abe (Chiba university, Japan)

Meeting Room 4 (ZOOM ID: 971 5454 5491)
9:00-9:45 GS5 Genetic Algorithms & Robotics 1 (3)Chair: Hideyuki Tanaka
GS5-1

Layout decision system for multiple production lines using work-flow-line and GA
Masato Noda1, Hidehiko Yamamoto1 (Gifu University, Japan),
Hirohumi Tsuji2, Yasuhisa Terawa2, Yoshinori Nakamura2, Masayuki Tsuchida2(Infofarm Corporation, Japan),
Katsuaki Yamada3, Yukiyasu Kuriyama3 (Kai Industries Corporation, Japan)

GS5-2

Autonomous decentralized FMS’s AGVs moving control by mind change with deeplearning
Ryunosuke Yamane, Hidehiko Yamamoto (Gifu University)

GS5-3

Secondary School Robotics Education in Camarin High School: Developments andChallenges for Improvement
Jeffrey Rivera Galino (Camarin High School, Philippines and Hiroshima University,
Japan), Hideyuki Tanaka (Hiroshima University, Japan)

13:00-13:45 GS3 Image & Signal Processing 2 (3)Chair: Keiko Sakurai (University of Miyazaki, Japan)
GS3-1

Deep-Learning Based Segmentation Algorithm for Defect Detection in Magnetic
Particle Testing
Akira Ueda, Huimin Lu, Tohru Kamiya (Kyushu Institute of Technology, Japan)

GS3-2

A Method for Improving Recognition of Lying Postures Using a Measured Signal
Intensity of Respiration and Heartbeat by Flexible Tactile Sensor Sheet
Kazuya Matsuo (Kyushu Institute of Technology, Japan)
Toshiharu Mukai (Meijo University, Japan),
Shijie Guo (Hebei University of Technology, China)

GS3-3

Basic research for the realization of online MEG using SSD
Kazuhiro Yagi, Yuta Shibahara, Lindsey Tate, Keiko Sakurai, Hiroki Tamura©ICAROB 2021 ALife Robotics Corp. Ltd.

P-26

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

(University of Miyazaki, Japan)

15:15-17:00 OS23 Advances in Field Robotics and Their Applications (6)Chair: Keisuke Watanabe (Tokai University)
Co-Chair: Kazuo Ishii (Kyushu Institute of Technology)
OS23-1Three-dimensional Measurement Using Laser Pattern And Its Application to Underwater Scanner
Yuya Nishida, Tomoya Shinnoki, Shinsuke Yasukawa, Kazuo Ishii(Kyushu Institute of Technology, Japan)

OS23-2

Motion control of a cable-restricted underwater vehicle for long-term spot
observation
Yoshiki Tanaka, Yuya Nishida, Kazuo Ishii (Kyushu Institute of Technology, Japan)

OS23-3Development of current sensors for digitizing expert knowledge in fish feeding
Towards sustainable aquaculture
Dominic B. Solpico, Yuya Nishida, Kazuo Ishii (Kyushu Institute of Technology, Japan)

OS23-4

Underwater image reconstruction using convolutional auto-encoder
Shinsuke Yasukawa, Sreeraman Srinivasa Raghura, Yuya Nishida, Kazuo Ishii(Kyushu Institute of Technology, Japan)

OS23-5

Spherical Robot Transfer Problem with Minimal Total Kinetic Energy
Kenji Kimura1, Kazuo Ishii2(1Fukuoka Daiichi High School, Japan, 2Kyushu Institute of Technology, Japan)

OS23-6

Acoustic Impedance Measurement through the Modelling of Ultrasonic Wave Transmission
Ryuugo Mochizuki, Yuya Nishida, Kazuo Ishii (Kyushu Institute of Technology, Japan)

OS23-7Installation Method of In-situ Drilling Platform(withdraw)
Keisuke Watanabe (Tokai University, Japan)

January 23 (Saturday)
8:40-Registration
Conference Room (ZOOM ID: 912 8494 3957)
11:00-12:00
Plenary Speech PS2
Chair: Masayoshi Tabuse (Kyoto Prefectural University, Japan)
PS2 Human-computer Communication Using Facial Expression©ICAROB 2021 ALife Robotics Corp. Ltd.

P-27

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Yasunari Yoshitomi (Kyoto Prefectural University, Japan)
Meeting Room 1 (ZOOM ID: 993 2436 1751)
9:00-10:15 OS14 Image Processing (5)Chair: Joo Kooi Tan (Kyushu Institute of Technology, Japan)
Co-Chair: Seiji Ishikawa (Kyushu Institute of Technology, Japan)
OS14-1

Detection of a Fallen Person and Estimation of Their Head Position from UAV Images
Haruka Egawa, Seiji Ishikawa, Joo Kooi Tan (Kyushu Institute of Technology, Japan)

OS14-2Development of a Pedestrian Crossing Navigation System for Visually Impaired
People Using MY VISION
Kohei Kitagawa, Seiji Ishikawa, JooKooi Tan (Kyushu Institute of Technology, Japan)

OS14-3

Abnormal Human Action Detection Based on GANTomoya Sano, Joo Kooi Tan (Kyushu Institute of Technology, Japan)

OS14-4

3-D Position Measurement of a Cargo Using Epipolar Geometry for Logistics
Automation(withdraw)
Kazuki Fukuda, Joo Kooi Tan (Kyushu Institute of Technology, Japan)

OS14-5Bus Line Number Detection Employing MY VISIONYe Zhou, Yosiki Hamasaki, Joo Kooi Tan (Kyushu Institute of Technology, Japan)

13:00-14:30 OS5 Advanced Information Processing Applications (6)Chair: Toru Hiraoka (University of Nagasaki, Japan)
Co-Chair: Masaharu Hirota (Okayama University of Science, Japan)
OS5-1

A Method for Patterns of Cell-Like Images Based on Distance Transformation
Toru Hiraoka, Kohei Maeda (University of Nagasaki, Japan)

OS5-2

Generating Striped Animations by Inverse Line Convergence Index Filter
Toru Hiraoka, Ryosuke Takaki (University of Nagasaki, Japan)

OS5-3

A Method for Estimating Home Location of Foreigners in Japan Using Photograph
Location
Masaharu Hirota, Tetsuya Oda (Okayama University of Science, Japan)

OS5-4

A Proposal of Online Map-matching Based Trajectory Compression Algorithm Using
Road Networks
Shota Iiyama, Tetsuya Oda, Masaharu Hirota (Okayama University of Science, Japan)

OS5-5

Case Study and Direction of Share Cycle System in Japanese Cities
Minoru Kumano (University of Miyazaki, Japan)
Toru Hiraoka (University of Nagasaki, Japan)

OS5-6

The IoT Solution to Play English Word Learning Tool
Shogo Aizawa, Motohide Yoshimura (University of Nagasaki, Japan)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-28

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

15:30-16:15 OS11 Robotic Manipulation (3)Chair: Kensuke Harada (Osaka University, Japan)
Co-Chair: Tokuo Tsuji (Kanazawa University, Japan)
Co-Chair: Akira Nakamura (AIST, Japan)
OS11-1

Using Various Evaluation Standards to Determine an Error Recovery Process in an
Automation Plant
Akira Nakamura*1, Natsuki Yamanobe*1, Ixchel Ramirez Alpizar*1,
Kensuke Harada*2, Yukiyasu Domae*1(*1 National Institute of Advanced Industrial Science and Technology (AIST), Japan*2 Osaka University, Japan)

OS11-2

Robotic Picking for Piled Sushi Topping
Kenta Matsuura*1*2, Keisuke Koyama*1, Weiwei Wan*1, Kensuke Harada*1(*1 Osaka University, Japan, *2 Currently with Yaskawa Electric Co. Ltd., Japan)

OS11-3

Motion Generation by Learning Relationship between Object Shapes and Human
Motions
Tokuo Tsuji1*, Sho Tajima*1, Yosuke Suzuki*1, Tetsuyou Watanabe*1,
Shoko Miyauchi*2, Ken’ichi Morooka*2, Kensuke Harada*3, and Hiroaki Seki*1(*1 Kanazawa University, Japan, *2 Kyushu University, Japan, *3 Osaka University,
Japan)

Meeting Room 2 (ZOOM ID: 986 1953 6512)
9:00-10:15 OS16 Software Development Support Method (5)Chair: Tetsuro Katayama (University of Miyazaki, Japan)
Co-Chair: Tomohiko Takagi (Kagawa University, Japan)
OS16-1

Simulation and Regression Testing for Behavior of Software Models Based on
Extended Place/Transition Net with Attributed Tokens
Tomohiko Takagi, Ryo Kurozumi (Kagawa University, Japan)

OS16-2Development of an Early Prototype Tool for Learning Software Modeling Using
Extended Place/Transition Net
Tomohiko Takagi, Akio Usuda (Kagawa University, Japan)

OS16-3

Test Suite Reusability Measurement Based on Frequency and Coverage of Reused
Test Cases
Mochamad Chandra Saputra*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, Naonobu Okazaki*(*University of Miyazaki, Japan) (†University of Nagasaki, Japan)

OS16-4The Seven Information Features of Class for Blob and Feature Envy Smell Detection
in a Class Diagram
Bayu Priyambadha*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, Naonobu Okazaki*(*University of Miyazaki, Japan) (†University of Nagasaki, Japan)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-29

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS16-5

Improvement of RETUSS to Ensure Traceability between Sequence Diagram in UML and Java
Source Code in Real Time
Kaoru Arima*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, Naonobu Okazaki*(*University of Miyazaki, Japan) (†University of Nagasaki, Japan)

13:00-14:30 OS6 Intelligent Control (6)Chair: Yingmin Jia (Beihang University, China)
Co-Chair: Weicun Zhang (University of Science and Technology Beijing, China)
OS6-1

Faster R-CNN Based Defect Detection of Micro-precision Glass Insulated Terminals
Qunpo Liu1, Mengke Wang1, Zonghui Liu1, Naohiko Hanajima2, Bo Su1(1Henan Polytechnic University, P.R.China) (2Muroran Institute of Technology, Japan)

OS6-2

Adaptive Sliding Mode Control for a Constant Tension Suspension System
Yuxin Jia, Yingmin Jia, Kai Gong (Beihang University, China)

OS6-3

Weighted Multiple Model ADRC for Uncertain Linear System
Weicun Zhang, Jing Ge (University of Science and Technology Beijing, China)

OS6-4

Trajectory Tracking Control of Differential Wheeled Mobile Robot Based on Rhombic Input
Constraints
Kai Gong, Yingmin Jia, Yuxin Jia (Beihang University, China)

OS6-5

Maneuvering target tracking based on improved interacting multiple model algorithm
Weicun Zhang, Meiyu Zhu (University of Science and Technology Beijing, China)

OS6-6

Encapsualted Agents of Hybrid Order Discrete Dyanmics
Yunzhong Song (Henan Polytechnic University, P.R.China)

15:30-16:15 OS17 Robotics and machine vision (3)Chair: Jiwu Wang (Beijing Jiaotong University, China)
Co-Chair: Junxiang Xu (Beijing Jiaotong University, China)
OS17-1

Kinematic modeling and Simulation of humanoid dual-arm robot
Jiwu Wang, Junxiang Xu (Beijing Jiaotong University, China)

OS17-2Kinematics analysis and simulation of 6R Robot Based on MATLAB/Simulink
Jiwu Wang, Shuo Han (Beijing Jiaotong University, China)

OS17-3

Research on planar ranging system based on binocular stereo vision
Jiwu Wang, Xin Pei (Beijing Jiaotong University, China)

Meeting Room 3 (ZOOM ID: 984 4367 9770)
9:00-10:15 OS22 Robot Competitions and Education (5)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-30

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Chair: Yasunori Takemura (Nishinippon Institute of Technology, Japan)
Co-Chair: Kazuo Ishii (Kyushu Institute of Technology, Japan)
OS22-1

Robustness Verification Against Noise of Self-localization Method Using Omni-directional Camera
for Soccer Robot
Yuehang Ma, Kaori Watanabe, Hidekazu Suzuki(Tokyo Polytechnic University, Japan)

OS22-2

Tomato-Harvesting Robot Competition: Aims and Developed Robot of 6thCompetitions
Takayuki Matsuo1, Yasunori Takemura2, Takashi Sonoda2, Shinsuke Yasukawa3, Yuya Nishida3,
Kazuo Ishii3 (1National Institute of Technology, Kitakyushu College, Japan, 2Nishinippon Instituteof Technology, Japan, 3Kyushu Institute of Technology, Japan)

OS22-3

Smart Agriculture IoT Education Course in enPiT-everi (Education Network for
Practical Information Technologies - Evolving and Empowering Regional Industries)
Yasunori Takemura1, 2, Keiji Kamei1, 2, Atsushi Sanada1, 2, Kazuo Ishii1(1Kyushu Institute of Technology, Japan, 2Nishinippon Institute of Technology, Japan)

OS22-4Development of a Handy Autonomous Underwater Vehicle “Kyubic”
Toshimune Matsumura, Yuuichiro Uemura, Kentaro Yanagise, Yoshiki Tanaka,
Yuya Nishida, Kazuo Ishii (Kyushu Institute of Technology, Japan)

OS22-5A Greenhouse Project toward Smart Agriculture
Kazuo Ishii1, Yuya Nishida1, Shinsuke Yasukawa1, Kanako Shirahashi1,
Yasunori Takemura2, Takayuki Matsuo3(1Kyushu Institute of Technology, Japan, 2Nishinippon Institute of Technology, Japan, 3National
Institute of Technology, Kitakyushu College, Japan,)

13:00-14:15 OS7 Information Applications and Security (5)Chair: Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
Co-Chair: I-Hsien Liu (National Cheng Kung University, Taiwan)
OS7-1

The Security Challenges with the Widespread Use of IT Infrastructure in ICS
Kuan-Ming Su, I-Hsien Liu, Jung-Shian Li(National Cheng Kung University, Taiwan)

OS7-2

A Communication System with Equipment's Characteristics
Chia-Chun Lai1, I-Hsien Liu1, Chi-Che Wu1, Chuan-Gang Liu2, Jung-Shian Li1(1National Cheng Kung University, Taiwan)(2Chia Nan University of Pharmacy and Science, Taiwan)

OS7-3

Application of the Self-Organizing Map (SOM) to Analyze the Multiple Perspectives on CrossNational Culture
Li-Ming Chuang, Yu-Po Lee, Shu-Tsung Chao
(Chang Jung Christian University, Taiwan)

OS7-4

Threats Hidden in Employee Workstation through Office Files
Tung-Lin Lee, I-Hsien Liu, Jung-Shian Li (National Cheng Kung University, Taiwan)

OS7-5

Microsatellite Attitude Control Approach: Combined with Generation Adversarial Networks FaultDetection and Cerebellar Model Articulation Controller Fault-Tolerant Control©ICAROB 2021 ALife Robotics Corp. Ltd.

P-31

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Ho-Nien Shou (Air Force Institute of Technology, Taiwan)

15:30-16:15 OS18 Natural Computing (3)Chair: Marion Oswald (Vienna University of Technology, Austria)
Co-Chair: Yasuhiro Suzuki (Nagoya University., Japan)
OS18-1

Contribution to the Theory of Periodic Reaction of Three Bodies Systems
Yasuhiro Suzuki (Nagoya University Japan)

OS18-2

Deep Micro Vibrotactile, DMV and its Applications
Yasuhiro Suzuki (Nagoya Universit, Japan)

OS18-3

Inter-Induce computation and its Philosophical Foundation
Yasuhiro Suzuki (Nagoya University, Japan)

Meeting Room 4 (ZOOM ID: 971 5454 5491
9:00-10:00 GS6 Robotics 1 (4)Chair: Noritaka Sato (Nagoya Institute of Technology, Japan)
GS6-1

Digital Testing Device for Active Range of Motion of Finger Joints Utilizing Artificial
Neural Network
Huu-Hieu Quang, Yoshifumi Morita (Nagoya Institute of Technology, Japan),
Makoto Takekawa (everfine, Japan)

GS6-2

Position and Force Teaching Method for 6 DoF Manipulator Using Contact Teaching
Tool and Teaching Data Editor
Duy-Do Bui, Hiroki Tanaka, Quang-Trung Chu, Hideki Inuzuka, Yoshifumi Morita(Nagoya Institute of Technology, Japan) Masao Sakai (Aichi Prefecture, Japan)
Haptic Device that Presents Sensation Corresponding to Palm on Back of Hand for
Teleoperation of Robot Hand Report2: Consideration on Decided Specification
Kyosuke Ushimaru,Noritaka Sato,Yoshifumi Morita(Nagoya Institute of Technology, Japan)

GS6-3

GS6-4

Robot Assisting Water Serving to Disabilities by Voice Control
Yang Chunxin, Sakmongkon Chumkamon, Eiji Hayashi(Kyushu Institute of Technology, Japan)

13:00-14:15 GS1 Chaos & Application (4)Chair: Masato Nagayoshi (Niigata College of Nursing, Japan)
GS1-1

A Method of Role Differentiation Using a State Space Filter with a Waveform
Changing Parameter in Multi-agent Reinforcement Learning
Masato Nagayoshi, Simon Elderton (Niigata College of Nursing, Japan)
Hisashi Tamaki (Kobe University, Japan)

GS1-2

The research about editing system of performance information for player piano. - Make
inferences about both handed musical composition by using DP matching system -©ICAROB 2021 ALife Robotics Corp. Ltd.

P-32

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Ryo Kinoshita, Eiji Hayashi (Kyushu Institute of Technology, Japan)
GS1-3

Development of a LiDAR based Navigation System for Tree Harvester
Ayumu Tominaga, Ryusuke Fujisawa, Eiji Hayashi (Kyushu Institute of Technology,
Japan), Abbe Mowshowitz (The City College of New York, US)

GS1-4

Numerical study on a class of chaotic financial chaotic systems(withdraw)
Lei Gong, Minghan Song (Tianjin University of Science and Technology, China)

GS1-5

Dynamic characteristics analysis of a multi-scroll conservative chaotic system withsinusoidal nonlinearity
Zhonggao Chen (Tianjin University of Science and Technology, China)

15:30-17:00 GS8 Medical Data Processing (6)Chair: Shingo Mabu (Yamaguchi University, Japan)
GS8-1

Development of in-Home Wireless Continuous Temperature Data Logging andAlarming System for Fever Monitoring in Pediatrics
Ali S. AlMejrad(University of Hail, Kingdom of Saudi Arabia)

GS8-2

Design and Development of in-home Wireless Crucial Events Logging and Alarming
System for Elderly and Disabled People Care
Ali S. AlMejrad(University of Hail, Kingdom of Saudi Arabia)

GS8-3

Virtual bird’s-eye view for remote operation of unmanned construction machinery
Noritaka Sato, Akihiro Fukuda (Nagoya Institute of Technology, Japan)

GS8-4

Domain Transformation of Chest CT Images Using Semi-Supervised Cycle GAN for
Opacity Classification of Diffuse Lung Diseases
Masashi Miyake1, Shingo Mabu1 , Shoji Kido2, Takashi Kuremoto1(1Yamaguchi University, Japan, 2Osaka University, Japan)

GS8-5

Anomaly Detection of Lung Sounds Using DAGMM
Ryosuke Wakamoto1, Shingo Mabu1, Shoji Kido2, Takashi Kuremoto1(1Yamaguchi University, Japan, 2Osaka University, Japan)

GS8-6

A ground reaction force analysis in walking and running gaits in horse leg model on viscoelastic
hoof-ground contact
Dondogjamts Batbaatar, Hiroaki Wagatsuma (Kyushu Institute of Technology, Japan)

January 24 (Sunday)
8:40-Registration
Meeting Room 1 (ZOOM ID: 993 2436 1751)
9:00-9:45 OS4 Intelligent UAV and Systems (3)Chair: Young Im Cho (Gachon University, Republic of Korea)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-33

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Co-Chair: Jang-Myung Lee (Pusan National University, Republic of Korea)
OS4-1

Satellite Image-based UAV Localization using Siamese Neural Network
Seong-Ha Ahn, Ho-Sun Kang, Jang-Myung Lee(Pusan National University, Republic of Korea)

OS4-2

Analysis Based on CNN for Automated Vehicle Parking Occupancy
Muksimova Shakhnoza, Young Im Cho (Gachon University, Republic of Korea)

OS4-3

On the boundary layer part of the asymptotics of the solution of a singularly
perturbed boundary value problem
Assiya Zhumanazarova, Young Im Cho (Gachon University, Republic of Korea)

11:00-12:00 OS3-1 Robot Control (4)Chair: Yizhun Peng (Tianjin University of Science and Technology, China)
Co-Chair: Huailin Zhao (Shanghai Institute of Technology, China)
OS3-1

A Design and Implementation Intelligent Flowerpot
Songyun Shi, Yizhun Peng (Tianjin University of Science and Technology, China)

OS3-2

Research of Attention-LSTM Model for Baby Cry Detection Robot
Tianye Jian, Yizhun Peng, Wanlong Peng, Zhou Yang
(Tianjin University of Science and Technology, China)

OS3-3

A Design and Implement of an Automatic Intelligent Car
Chengzhi Liu, Yizhun Peng, Jikai Zhao, Dezhi Yang
(Tianjin University of Science and Technology, China)

OS3-4

A Target Detection in Remote Sensing Image based on Deep Learning
Lianchen Zhao, Yizhun Peng, Di Li, Yuheng Zhang(Tianjin University of Science and Technology, China)

13:00-14:15 OS3-2 Robot Control (5)Chair: Yizhun Peng (Tianjin University of Science and Technology, China)
Co-Chair: Huailin Zhao (Shanghai Institute of Technology, China)
OS3-5

A Design of Home Smart Nursing Robot based on Raspberry Pi 4
Yuqi Zhao, Yizhun Peng, Jie Liu, Xiaowei Wu, Jikai Zhao
(Tianjin University of Science and Technology, China)

OS3-6

Design and Simulation of Indoor Tour Guide Robot Based on ROS
Yuheng Zhang, Yizhun Peng, Lianchen Zhao
(Tianjin University of Science and Technology, China

OS3-7

Research on Path Planning of Manipulator©ICAROB 2021 ALife Robotics Corp. Ltd.

P-34

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Nana Wang, Yizhun Peng, Zhou Yang, Yuheng Zhang(Tianjin University of Science and Technology, China)
OS3-8

Research on White-Line-tracking Walking Technique of NAO Robot
Wanlong peng, Yizhun peng, Yuheng zhang,Tianye jian
(Tianjin University of Science and Technology, China)

OS3-9

A Design and Implement of Portable Epidemic Detection Device based on STM32
Yuqi Zhao, Yizhun Peng, Xiaowei Wu, Yusong Zhou
(Tianjin University of Science and Technology, China)

Meeting Room 2 (ZOOM ID: 986 1953 6512)
9:00-9:45 GS4 Image & Signal Processing 3 (3)Chair: Taro Shibanoki (Ibaraki University, Japan)
GS4-1

Landslide Area Detection from Synthetic Aperture Radar Images Using Convolutional
Adversarial Autoencoder and One-class SVM
Shingo Mabu, Soichiro Hirata, Takashi Kuremoto (Yamaguchi University, Japan)

GS4-2

Anomaly Detection in Time Series Data Using Support Vector Machines
Umaporn Yokkampon1, Sakmongkon Chumkamon1, Abbe Mowshowitz2,
Eiji Hayashi1 and Ryusuke Fujisawa1 (1Kyushu Institute of Technology, Japan)(2The City College of New York, USA)

GS4-3

Relationship Between Tactile Stimuli and Human Body Sway
Masaya Tadokoro, Taro Shibanoki (Ibaraki University, Japan)

11:00-12:00 OS13 Intelligent Systems and Robotics (14)Chair: Fengzhi Dai (Tianjin University of Science and Technology, China)
Co-Chair: Yunzhong Song (Henan Polytechnic University, China)
OS13-1Analysis of the Consensus Protocol of Heterogeneous Agents with Time-Delays
Jichao Zhao1, Fengzhi Dai1,3, Yunzhong Song2(1College of ElTianjin University of Science and Technology, China, 2Henan
Polytechnic University, China, 3Tianjin Tianke Intelligent and Manufacture
Technology CO., LTD, China)

OS13-2Burrows-Wheeler Transform Acceleration based on CUDA
Chang Sheng, Fengzhi Dai (Tianjin University of Science and Technology, China)

OS13-3

Design of Automatic Water Supply Upper Computer System
Peng Lu, Fengzhi Dai, Tianyi Zhang(Tianjin University of Science and Technology, China)

OS13-4

Analysis of Boiler Water Level System based on the Fuzzy Control
Tianyi Zhang, Fengzhi Dai, Peng Lu(Tianjin University of Science and Technology, China)

OS13-5Development of the Circuit System for Greenhouse Environment Regulation
Yuhui Cheng, Fengzhi Dai, Chengxu Ji, Peng Lu©ICAROB 2021 ALife Robotics Corp. Ltd.

P-35

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

(Tianjin University of Science and Technology, China)
OS13-6

Design of an Intelligent Car based on MSP430
Ruming Kang, Fengzhi Dai (Tianjin University of Science and Technology, China)

OS13-7

Design of a WIFI Video Car
Qianqian Zhang1, Fengzhi Dai1,2, Jichao Zhao1, Haokang Wen1, Hongbo Hao1
(1Tianjin University of Science and Technology, China)(2Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China)

OS13-8Design of WebGIS Transportation and Distribution System based on the Genetic
Algorithm
Hongbo Hao, Fengzhi Dai (Tianjin University of Science and Technology, China)

OS13-9A Study of YOLO Algorithm for Target Detection
Haokang Wen, Fengzhi Dai, Yasheng Yuan
(Tianjin University of Science and Technology, China)

OS13-10

Research on Recognition and Application of EEG Signal based on SSVEP-BCI
Di Yin1, Fengzhi Dai1,3, Mengqi Yin2, Yasheng Yuan1, Yuxuan Zhu1
(1Tianjin University of Science and Technology, China)(2Hebei University of Chinese Medicine, China)(3Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China)

OS13-12

Research on Crop Image Recognition Technology - Take Daylily as an Example
Jichao Zhao, Fengzhi Dai (Tianjin University of Science and Technology, China)

OS13-13

Design of Daylily Agricultural Picking Robot
Jichao Zhao, Fengzhi Dai (Tianjin University of Science and Technology, China)

OS13-14Development of Intelligent Public Trash Can based on Machine Vision and Learning
Longyu Gao, Fengzhi Dai, Zhiqing Xiao, Jiangyu Wu, Zilong Liu(Tianjin University of Science and Technology, China)

13:00-14:15 OS12 System and Control (18)Chair: Huailin Zhao (Shanghai Institute of Technology, China)
Co-Chair: Jichao Zhao (Tianjin University of Science and Technology, China)
OS12-1The Optimized Intelligent Algorithms on Face Recognition and Tracking for
ROS-based Robots
Yue Chen, Shuhao Tian, Huailin Zhao, Shengyang Lu(Shanghai Institute of Technology)

OS12-2Control of a Novel 5D Hyperchaotic System
Qiang Wei1, Hong Niu2
(1Army Military Transportation University, China)(2Tianjin University of Science & echnology, China)

OS12-3

Design of Multifunctional Vehicle Interior Environment Monitoring System Based on©ICAROB 2021 ALife Robotics Corp. Ltd.

P-36

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Wireless Communication
Yuqi Yan, Jialin Yang, Zhongxu Qin (Wuhan University of Technology, China)
OS12-4

Robot Structure and Motion control Design Based on UG and ProductYuhang Sheng (Tianjin University of Science and Technology, China)

OS12-5

Design of Brushed Motor Position Loop Control System Based on Incremental PID
Tianyi Zhang, Peng Lu (Tianjin University of Science and Technology, China)

OS12-6

Design of a Fire alarm system
Yuhui Cheng, Tianyi Zhang (Tianjin University of Science and Technology, China)

OS12-7

Design of intelligent curtain control circuit based on single chip Microcomputer
Qianqian Zhang, Jichao Zhao, Haokang Wen, Hongbo Hao(Tianjin University of Science and Technology, China)

OS12-9A Survey of Low Visibility Image Enhancement Based on MSRCR Algorithm
Haokang Wen, Hongbo Hao (Tianjin University of Science and Technology, China)

OS12-10

Design of a Taxi Meter Based on Single Chip Computer
Haokang Wen, Min Wang (Tianjin University of Science and Technology, China)

OS12-11

Design of a High Precision Digital Clock Based on Single Chip Microcomputer
Haokang Wen, Qiang Zheng (Tianjin University of Science and Technology, China)

OS12-12

Design of Manchu Image Acquisition System Based on STM32
Hongbo Hao1, Fulin Zheng2(1Tianjin University of Science and Technology, China)(2 Dalian Minzu University, China)

OS12-13

Design of timing socket based on single chip microcomputer
Hongbo Hao1, Shuailin Chen2(1Tianjin University of Science and Technology, China)(2Qingdao University of Science and Technology, China)

OS12-14

Study on the effect of physical fitness training on children’s cognitive ability
Jianhua Deng, Lei Ning (Yanshan University, China)

OS12-15

Effects of high heels on plantar stress in women
Jianhua Deng, Lei Ning (Yanshan University, China)

OS12-16

A research on Intelligent Classification of Urban Trash Bins based on Machine
Learning
Longyu Gao, Zilong Liu, Luqi Shen, Songyun Shi, Yongzheng Lv(Tianjin University of Science and Technology, China)

OS12-17

A Design on Intelligent Public Trash Can based on Machine Vision and Auxiliary
Sensors
Longyu Gao, Leixin Han, Jiangyu Wu, Mingfei Liu, Ruming Kang
(Tianjin University of Science and Technology, China)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-37

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS12-18

A research on Front-end Garbage Classification based on Machine Vision
Longyu Gao, Zhiqing Xiao, Junlong Hao, Luqi Shen, Manqian Hu(Tianjin University of Science and Technology, Tianjin, China)

Meeting Room 3 (ZOOM ID: 984 4367 9770)
11:00-12:00 OS20 Virtual Reality and Intelligent Interactions (3)Chair: R.P.C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)
Co-Chair: Yoshimasa Tokuyama (Tokyo Polytechnic University, Japan)
OS20-1

NeuroPhyllotaxis: An Interactive Application for Generative Art Based on EEG Data
Chien-Tung Lin, R.P.C. Janaka Rajapakse(Tainan National University of the Arts, Taiwan)
Yoshimasa Tokuyama (Tokyo Polytechnic University, Japan)

OS20-2

VRMAZU: VR Visualization of Mazu Temple for Passive Interaction with Generated
Sound from the ML Technique(withdraw)
Yi-Li Liang. R.P.C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)
Jen-Tun Lee (Japan Advanced Institute of Science and Technology, Japan)
Yoshimasa Tokuyama (Tokyo Polytechnic University, Japan)

OS20-3ThoughtMix: Interactive Water Color Generation and Mixing Based on EEG Data
R.P.C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)
Yoshimasa Tokuyama (Tokyo Polytechnic University, Japan)

OS20-4

HaptWarp: Soft Printable and Motion Sensible Game Controller
Jen-Tun Lee, Kazunori Miyata(Japan Advanced Institute of Science and Technology, Japan)
R.P.C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)

13:00-14:00 OS1 Human-Machine Interface Application (4)Chair: Norrima Mokhtar (University of Malaya, Malaysia)
Co-Chair: Heshalini Rajagopal (Manipal International University, Malaysia)
OS1-1

Gray Level Co-Occurrence Matrix (GLCM) and Gabor Features Based No-Reference
Image Quality Assessment for Wood Images
Heshalini Rajagopal1, Norrima Mokhtar1, Anis Salwa Mohd Khairuddin1, Wan Khairunizam2,
Zuwairie Ibrahim3, Asrul Bin Adam3, Wan Amirul Bin Wan Mohd Mahiyidin1(1University of Malaya, Malaysia) (2University of Malaysia Perlis,
Malaysia) (3University of Malaysia Pahang, Malaysia)

OS1-2

Design and Development of Automated Seeding and Irrigation System
Anirban Kumar, Heshalini Rajagopal (Manipal International University, Malaysia)

OS1-3

Investigation of A Real-Time Driver Eye-Closeness for the Application of Drowsiness
Detection
Muhammad Zubir bin Kamazlan1, Wan Khairunizam1, Abdul Hafiz Halin1,
M. Rudzuan M. Nor1, Azian Azamimi Abdullah1, Norrima Mokhtar2(1University Malaysia Perlis, Malaysia) (2University of Malaya, Malaysia)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-38

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS1-4

Towards Establishing Path Planning Strategies For Autonomous UAVs; A Brief
Survey-Summary on Recent TechniqueAnees ul Husnain, Norrima Binti Mokhtar, Noraisyah Binti Mohamed Shah, Mahidzal Bin Dahari(University of Malaya, Malaysia)

Meeting Room 4 (ZOOM ID: 971 5454 5491)
11:00-12:00 GS7 Robotics 2 (4)Chair: Wisanu Jitviriya
GS7-1

Development of Interactive Robot – Emotion Estimation System Using Speech by
1dCNN Yugo Kawachi, Eiji Hayashi (Kyushu Institute of Technology, Japan)

GS7-2

Deep Learning Methods for Semantic Segmentation of Dense 3D SLAM MapsPei Yingjian, Sakmongkon Chumkamon, Eiji Hayashi(Kyushu Institute of Technology, Japan)

GS7-3

Robot Motion Generation by Hand Demonstration
Sakmongkon Chumkamon, Umaporn Yokkampon, Eiji Hayashi, Ryusuke Fujisawa(Kyushu Institute of Technology, Japan)

GS7-4

Medical Telerobotics: IRAPs SHaRE-aGIVeR
Noppadol Pudchuen, Jiraphan Inthiam, Wisanu Jitviriya, Amornphun Phunopas,
Chirdpong Deelertpaiboon(King Mongkut’s University of Technology North Bangkok, Thailand)
Aran Blattler (Kyushu Institute of Technology, Japan)

Group Meeting©ICAROB 2021 ALife Robotics Corp. Ltd.

P-39

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-40

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Abstracts
PS abstracts
PS1 Tomato-harvesting-robot competition towards smart agriculture
Kazuo Ishii (Kyushu Institute of Technology, Japan)
In agriculture, the aging and depopulation of farmers cause the shortages of farmers andmanpower. Most of commercialized robots are industrial robots for factory automation,and most of robots for the first industry, agriculture, forestry and fisheries are still underdeveloping. The reasons are cost-efficiency of the robotization, safety of the works using
robots, difficulty of outdoor operations, and knowledge transfer problem from farmers
to computer, etc. As one of solutions for the problems, robot technology into theagriculture is expected to contribute to the laborsaving, improvement of production,
production line automation, and also the management toward smart-agriculture. We
organize the Tomato-harvesting-robot competition to offer the research field andwelcome researchers into agricultural robotics.

PS2 Human-computer Communication Using Facial Expression
Yasunari Yoshitomi (Kyoto Prefectural University, Japan)
To develop a computer system, such as a robot, that can communicate smoothly withhuman, it is necessary to equip the system with a function of understanding humanemotion. Moreover, it is also necessary to equip the system with a function of expressing
emotional signal to human. From both points of view, facial expression is a promisingtarget as a research field. With co-researchers, I have been researching both aspects on
facial expressions. For developing a method for recognizing facial expression, we haveused infrared-ray images in addition to visible-ray images. For expressing emotionalsignal to human, we have developed a personified agent. Our challenges for humancomputer communication using facial expression will be introduced in my presentation.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-41

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS abstracts
OS1 Human-Machine Interface Application (4)Chair: Norrima Mokhtar (University of Malaya, Malaysia)
Co-Chair: Heshalini Rajagopal (Manipal International University, Malaysia)
OS1-1 Gray Level Co-Occurrence Matrix (GLCM) and Gabor Features Based NoReference Image Quality Assessment for Wood Images
Heshalini Rajagopal1, Norrima Mokhtar1, Anis Salwa Mohd Khairuddin1, Wan Khairunizam2,
Zuwairie Ibrahim3, Asrul Bin Adam3, Wan Amirul Bin Wan Mohd Mahiyidin1(1University of Malaya, Malaysia) (2University of Malaysia Perlis, Malaysia)(3University of Malaysia Pahang, Malaysia)

Image Quality Assessment (IQA) is an imperative element in improving theeffectiveness of an automatic wood recognition system. There is a need to develop a NoReference-IQA (NR-IQA) system as a distortion free wood images are impossible to beacquired in the dusty environment in timber factories. Therefore, a Gray Level CoOccurrence Matrix (GLCM) and Gabor features-based NR-IQA, GGNR-IQA algorithm
is proposed to evaluate the quality of wood images. The proposed GGNR-IQA algorithm
is compared with a well-known NR-IQA, Blind/Referenceless Image Spatial Quality
Evaluator (BRISQUE) and Full-Reference-IQA (FR-IQA) algorithms, Structural
Similarity Index (SSIM), Multiscale SSIM (MS-SSIM), Feature SIMilarity (FSIM),
Information Weighted SSIM (IW-SSIM) and Gradient Magnitude Similarity Deviation(GMSD). Results shows that the GGNR-IQA algorithm outperforms the NR-IQA and
FR-IQAs. The GGNR-IQA algorithm is beneficial in wood industry as a distortion freereference image is not required to pre-process wood images.

OS1-2 Design and Development of Automated Seeding and Irrigation System
Anirban Kumar, Heshalini Rajagopal (Manipal International University, Malaysia)
Malaysia, has a very conducive environment for agriculture. Six percent of the
population are employed to some of the agriculture activities making agriculture themost curtail industry in Malaysia. Hence, there is a need to develop the agricultural
facilities by incorporating latest technological advancements. Conventional seeding
process is time consuming and requires additional labor cost. In this project, an
automated system is proposed for seeding as well as irrigation process in agriculture
which reduces the labor cost. This system aims to increase the efficiency of theseeding process without affecting the nature of soil. The proposed system isequipped with Arduino MEGA and Arduino UNO which acts as the main control
unit while ultrasonic and soil moisture sensors are used to detect the obstacles andsoil moisture level, respectively. The robot consists of a funnel like arrangement toperform the seeding procedure. The field is equipped with moisture sensors placedat different areas that monitors the moisture level of the soil on a regular interval for
irrigation purposes. The proposed system will be of great benefit to the futureendeavor of agricultural business as well as it will be able to optimize the seeding
and irrigation.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-42

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS1-3 Investigation of A Real-Time Driver Eye-Closeness for the Application ofDrowsiness Detection
Muhammad Zubir bin Kamazlan1, Wan Khairunizam1, Abdul Hafiz Halin1, M. Rudzuan M.
Nor1, Azian Azamimi Abdullah1, Norrima Mokhtar2
1( University Malaysia Perlis, Malaysia) (2University of Malaya, Malaysia)

The increase in accident and death rates due to drowsiness while driving raises concerns
to the community. An efficient solution is vital to ensure the safety of all drivers on theroad. Most previous studies have analyzed drowsiness using head tilt, yawning, and eye
condition. Face detection applied in drowsiness detection from previous research notincluded distances between subject and camera. The features used for eye detection
required large storage and long-term process which are not applicable in a real-timesystem. This study uses Haar algorithm and analysis is performed based on the size ofthe region of interest for face detection. Eye monitoring uses facial landmark features
and the evaluation is dependent on the width of the eye. The percentage of eye closure
is used to describe the eyes as closed. This study only takes into account the normal rateof blinking eyes while driving because of the long-time constraints required for a person
to be in a drowsy state. In this research, the Raspberry Pi 3B+ and Pi cameras are usedas processing and vision devices. The highest accuracy of face detection achieved basedon the ROI area at a distance of 80 cm is 98.33%. The lowest difference between eye
width and the intercanthal distance is 0.36%. The overall normal eye blink rate whiledriving is in the range of the normal eye blink rate which does not exceed 20 blinks/min
as reported by the previous researcher.

OS1-4 Towards Establishing Path Planning Strategies For Autonomous UAVs; A
Brief Survey-Summary on Recent TechniqueAnees ul Husnain, Norrima Binti Mokhtar, Noraisyah Binti Mohamed Shah, Mahidzal Bin Dahari(University of Malaya, Malaysia)
The extent of autonomy in path planning for a UAV primarily depends upon thecapabilities of its algorithm. The diversity in UAV applications and an abundance ofchoices in autonomous path planning algorithms are swelling every day, so the selectionof most appropriate algorithm gets baffling. The past two decades of research on UAVs
revealed that seventy percent of it had been published in the previous three and a halfyears. Hence, a comprehensive survey study was proposed and conducted to obtain an
overview of the recent developments in autonomous path planning applications and theirrespective algorithms. This article presents a summary of the survey and suggests mostsuitable path planning algorithms for a UAV application.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-43

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS2 Media Information Processing, Music Recommendation and Artificial
Intelligence (4)Chair: Yasunari Yoshitomi (Kyoto Prefectural University, Japan)
Co-Chair: Masayoshi Tabuse (Kyoto Prefectural University, Japan)
OS2-1 Comparison of Data Augmentation Methods in Pointer-Generator Model
Using Various Sentence Ranking Methods
Tomohito Ouchi, Masayoshi Tabuse (Kyoto Prefectural University, Japan)
In the existing research, we proposed a data augmentation method using topic model for
Pointer-Generator model. In this study, we add to the sentence ranking method in thedata augmentation method. Specifically, we add two ranking method using LexRank andLuhn. LexRank is based on Google's search method and Luhn defines sentence featuresand ranks sentences. We compare three data augmentation method. We considered which
method is suitable for data augmentation. Nowadays, it is very difficult to extract
information due to the flood of information on the Internet. Therefore, we propose
Pointer-Generator model using a data augmentation method using various ranking
methods.

OS2-2 Music Recommendation System Driven by Variations in Fingertip Skin
Temperature
Mayuka Wada1, Taro Asada2, Yasunari Yoshitomi2, Masayoshi Tabuse2( Shimadzu System Development Corp., Japan, 2Kyoto Prefectural University, Japan)
1

In recent years, music therapy has been performed to recover cognitive function ofelderly people. In the previously reported system, it is necessary to input the user's
subjective evaluation to determine the next recommended song. In this study, variations
in fingertip skin temperature were used as input instead of subjective evaluation,
focusing on evaluation of emotion expressed as physiological response throughvariations in fingertip skin temperature. Our system can be used with no support forusers, such as severe dementia patients or bedridden patients, who cannot respondactively.

OS2-3 Music Recommendation System Driven by Interaction between User andPersonified Agent Using Speech Recognition, Synthesized Voice and Facial
Expression
Ayumi Matsui1, Miki Sakurai2, Taro Asada3, Yasunari Yoshitomi3, Masayoshi Tabuse3( Sumitomo Mitsui Card Co., Ltd., Japan, 2TIS Inc., Japan, 3Kyoto Prefectural University, Japan)
1

We propose a system for music recommendation through an interaction between a user
and a personified agent using speech recognition, synthesized voice and facial
expression. The speech is recognized using a speech recognition system called as Julius,
followed by facial expression synthesis of the agent using preset parameters dependingon the vowel each. We used MikuMikuDanceAgent (MMDAgent) to create an agent. To
produce the agent’s voice, we use the speech synthesis function setting built into MMD
Agent. We add the agent a new function of changing its facial expression according touser’s response to music recommended by our system. The effectiveness of the proposedsystem is verified.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-44

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS2-4 Wallet Operation Evaluation System Using Deep Learning

Junichiro Yamawaki1, Yasunari Yoshitomi2, Masayoshi Tabuse2, Taro Asada2(1SKY Co., Ltd., Japan, 2Kyoto Prefectural University, Japan)

In Japan, the average age of the population has been increasing, and this trend is expected
to continue. Because of this trend, the number of older people with dementia isincreasing very rapidly. It is necessary to find out persons at the early stage, so called
MCI (Mild Cognitive Impairment), of dementia, for taking care of them while
suppressing their dementia progression. For investigating MCI, wallet operation test has
been receiving considerable attentions. In the present study, we propose a system forwallet operation evaluation using deep learning. By the system, a wallet, bills, and coins
are automatically recognized and the ability of correctly picking up bills and coins in thewallet within the reasonable time is estimated.

OS3 Robot Control (9)Chair: Yizhun Peng (Tianjin University of Science and Technology, China)
Co-Chair: Huailin Zhao (Shanghai Institute of Technology, China)
OS3-1 A Design and Implementation Intelligent Flowerpot
Songyun Shi, Yizhun Peng (Tianjin University of Science and Technology, China)
Aiming at the characteristics that people often neglect the care of potted plants at home,
this article designs a smart flowerpot, which can make the potted plants survive and growbetter without supervision. This is a smart home product based on the Internet of Things
technology. The flowerpot collects data from temperature sensors, humidity sensors, soil
humidity sensors, harmful gas sensors, and photosensitive modules through STM32
single-chip microcomputers, and cooperates with smart tracking systems and automatic
irrigation systems. To achieve the purpose of unmanned cultivation of potted plants,
beautify and improve the living environment.

OS3-2 Research of Attention-LSTM Model for Baby Cry Detection Robot
Tianye Jian, Yizhun Peng, Wanlong Peng, Zhou Yang
(Tianjin University of Science and Technology, China)
In order to achieve the effective acquisition of frame-level speech features under
different emotional needs of baby, a speech emotion recognition model for baby basedon improved long-term and short-term memory (LSTM) network is established. The
frame-level speech features are used instead of the traditional statistical features topreserve the temporal relationships in the original speech, and the traditional forgetting
and input gates are transformed into attention gates by introducing an attention
mechanism, in order to improve the performance of speech emotion recognition, thedepth attention gate is calculated according to the self-defined depth strategy. The results
show that, in Fau Aibo Children’s emotional data corpus and baby crying emotional
needs database, compared with the traditional LSTM based model, the recall rate and F1score of this model are 3.14%, 5.50%, 1.84% and 5.49% higher, respectively, comparedwith the traditional model based on Lstm and GRU, the training time is shorter and the
speech emotion recognition rate of baby is higher.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-45

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS3-3 A Design and Implement of an Automatic Intelligent Car
Chengzhi Liu, Yizhun Peng, Jikai Zhao, Dezhi Yang (Tianjin University of Science andTechnology, China)
In order to study the situation that some unmanned vehicles need to drive repeatedly on
fixed lines, this article introduces an autonomous smart car based on the STM32
platform. Infrared detectors are used to determine whether the vehicle is driving on aprescribed route. Ultrasonic detectors can prevent the vehicle from colliding with otherpeople or vehicles. Inexpensive trolleys can help express companies to transport
packages and letters to reduce labor costs. Large vehicles can be used to carry large cargosuch as containers in docks and other places.

OS3-4 A Target Detection in Remote Sensing Image based on Deep Learning
Lianchen Zhao, Yizhun Peng, Di Li, Yuheng Zhang (Tianjin University of Science andTechnology, China)
For high-resolution optical remote sensing images, there are still many challenges intarget detection. In this paper, deep learning algorithm is used to detect the target inremote sensing image. Improve and optimize the deep learning target detection
algorithm. When the selected data set is used for target detection, the AP value isimproved, which leads to the concept of multi-scale feature fusion feature pyramid andresidual network. By improving the selected Yolov3 network model, the detection effectof the two targets of aircraft and ships in remote sensing images has been significantly
improved.

OS3-5 A Design of Home Smart Nursing Robot based on Raspberry Pi 4
Yuqi Zhao, Yizhun Peng, Jie Liu, Xiaowei Wu, Jikai Zhao
(Tianjin University of Science and Technology, China)
This product is based on Bluetooth, WIFI and other wireless technologies, with the
Raspberry Pi 4 as the core, to meet the health, safety and entertainment needs of theserviced users, mainly the elderly, while assisting the leisure and entertainment of users
when traveling outdoors. A multi-functional escort robot designed for safety and otherissues. It has many methods such as temperature and humidity monitoring, harmful gas
monitoring, and noise monitoring to ensure the safety of users. At the same time, it also
has TV projection, audio-visual entertainment touch screen interaction, mobile phone
projection and other entertainment functions.

OS3-6 Design and Simulation of Indoor Tour Guide Robot Based on ROS
Yuheng Zhang, Yizhun Peng, Lianchen Zhao (Tianjin University of Science and Technology,
China
Aiming at the problems of the small number of lecturers in museums and other venues
and the large demand for audience consultation, a ros-based indoor guide robot was
designed. The robot consists of a mechanical system, a motion control system, and asensor system. In order to improve the efficiency of development and debugging, andreduce the cost of experiments, it is necessary to test related algorithms in a virtual
simulation environment before the robot enters the actual working state. Experiments
have proved that the robot can autonomously guide guests to the destination and explain
according to a preset path; the robot has multiple sensors to sense obstacles, and canautonomously avoid obstacles during the explanation and continue to move; the robot
can accurately and efficiently recognize faces and provide accurate services Fast.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-46

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS3-7 Research on Path Planning of Manipulator
Nana Wang, Yizhun Peng, Zhou Yang, Yuheng Zhang (Tianjin University of Science andTechnology, China)
In order to further realize the flexibility, intelligence and human-machine friendliness,the robot must have the ability of flexible motion planning. This paper mainly introduces
the general steps of manipulator path planning and common path planning algorithms.
First, the path planning of the manipulator is reviewed from three aspects: environment
modeling, path search, and path smoothing. In addition, compared various manipulator
path planning algorithms, such as RRT algorithm, ant colony algorithm, genetic
algorithm, etc. Finally, the development trend of path planning technology issummarized and analyzed.

OS3-8 Research on White-Line-tracking Walking Technique of NAO Robot
Wanlong peng, Yizhun peng, Yuheng zhang,Tianye jian
(Tianjin University of Science and Technology, China)
Liner-tracking walking technique is an essential part of intelligent robot technology. This
paper selected NAO robot as the research platform to study the line-tracking walking
problem. Firstly, NAO white shell increased the difficulty of white line recognition, thisis different from other robot. A method based on image preprocessing is presented,
sorting white edge with cut-point which is non-differential or derivative is zero, thendesigned the controller based on the improved digital incremental PID(proportion
integration differentiation)algorithm and determined the controller parameters throughthe concise test method on Simulink. Finally, it was realized by programming in Python
language.

OS3-9 A Design and Implement of Portable Epidemic Detection Device based on
STM32
Yuqi Zhao, Yizhun Peng, Xiaowei Wu,Yusong Zhou (Tianjin University of Science andTechnology, China)
The equipment is mainly designed for epidemic detection. It is an intelligent detection
box device. It collects and analyzes data through cameras and sensors. It uses LBP
algorithm and LAB color model to enable the detection box to identify the temperatureand mask of the entered personnel. Wearing conditions, preliminary comparison of
epidemic prevention standards for passers-by, whether to wear masks, whether the bodytemperature exceeds the standard, and at the same time enter the identity of new
personnel and store them in the database, and detect the incoming personnel through facerecognition to find out in time Those who are not entered shall take corresponding
measures in time to improve the efficiency of epidemic prevention and control. The
device does not need to be held in hand, can be fixed anywhere, can effectively avoid
cross-infection, and supports self-set temperature alarm thresholds, support voicebroadcast reminders, fast and accurate non-contact temperature measurement.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-47

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS4 Intelligent UAV and Systems (3)Chair: Young Im Cho (Gachon University, Republic of Korea)
Co-Chair: Jang-Myung Lee ((Pusan National University, Republic of Korea))
OS4-1 Satellite Image-based UAV Localization using Siamese Neural Network
Seong-Ha Ahn, Ho-Sun Kang, Jang-Myung Lee (Pusan National University, Republic of Korea)
We present a method for UAV localization using pre-existing satellite images. The useof Unmanned Aerial Vehicles (UAVs) has rapidly increased in several applications suchas surveillance, search, and defense. When in GPS-denied situations, however, theonboard GPS signal may be noisy or inaccurate. The proposed method is based on a
Siamese Neural Network that contains two instances of the same neural architecture andweights. Siamese Neural Network learns the similarity metric so that can recognize thesame place from two raw images. Convolutional Neural Network is used as a backbone
in Siamese Neural Network to overcome variation due to differences such as perspective,
shadow angle, and presence of vehicles. We describe UAV localization pipeline and adataset for training and testing our networks. Finally, the performance of the proposed
method was shown in accuracy.

OS4-2 Analysis Based on CNN for Automated Vehicle Parking Occupancy
Muksimova Shakhnoza, Young Im Cho (Gachon University, Republic of Korea)
The noteworthy growth of the automotive trade besides an absence of urban
development has caused issues like traffic congestion, air contamination, and driving
difficulties. To facilitate consumers, most car manufacturers provide a pre-installed oraftermarket navigation system in vehicles. This enables drivers to easily navigate to theirdestination. However, navigation systems provide a rough estimate of the remaining
distance to the destination. With the recent technological advancements in sensing andintelligent technologies, companies are interested to find out whether theseadvancements can assist in reducing the time spent to search for vacant parking spaces.
Locating an unoccupied parking space is a major predicament that drivers face sincesearching for it is a tedious process. We investigations showed that, in comparison withprevious approaches, for the task of classifying given parking spaces as vacant or
occupied, the proposed approach is more robust, stable, and well-generalized for unseen
images captured from completely different camera viewpoints, which has strong
indications that it would generalize effectively to other parking lots.

OS4-3 On the boundary layer part of the asymptotics of the solution of a singularly
perturbed boundary value problem
Assiya Zhumanazarova, Young Im Cho (Gachon University, Republic of Korea)
In this paper, we study the boundary layer part of the asymptotics of a singularly
perturbed integro-differential boundary value problem with an initial jump. A part of theboundary layer is defined as an expansion in powers of a small parameter, and thecoefficients of this expansion are solutions of a third-order differential problem. The
purpose of this work is to describe in detail the properties of the boundary layer part of
the asymptotics and to prove estimates of their coefficients.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-48

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS5 Advanced Information Processing Applications (6)Chair: Toru Hiraoka (University of Nagasaki, Japan)
Co-Chair: Masaharu Hirota (Okayama University of Science, Japan)
OS5-1 A Method for Patterns of Cell-Like Images Based on DistanceTransformation
Toru Hiraoka, Kohei Maeda (University of Nagasaki, Japan)
A non-photorealistic rendering method has been proposed to generate cell-like images
that cell-like patterns are represented in photographic images. The cell-like patterns areautomatically generated according to changes with edges and shades of the photographic
images. However, the cell-like patterns are irregularly arranged. In this study, we propose
a method to arrange the cell-like patterns along the edges of the photographic images.
Our method improves on the conventional method by using Euclidean distance from theedges. We show that the cell-like images with the cell-like patterns arranged along theedges can be generated by our method through experiments using various photographic
images.

OS5-2 Generating Striped Animations by Inverse Line Convergence Index Filter
Toru Hiraoka, Ryosuke Takaki (University of Nagasaki, Japan)
A non-photorealistic rendering method has been proposed for generating a striped image
which is overlaid striped patterns in a photograph. The conventional method generates
the striped image by an iterative process using an inverse line convergence index filter.
When a striped animation is generated by converting each frame of a movie by theconventional method, flickering occurs in the generated striped animation. In this letter,
we propose a method for generating a striped animation that has characteristic with lessflicker from the movie. The effectiveness of the proposed method is investigatedexperimentally. As a result of the experiments, the proposed striped animation had lessflicker than the conventional striped animation.

OS5-3 A Method for Estimating Home Location of Foreigners in Japan Using
Photograph Location
Masaharu Hirota, Tetsuya Oda (Okayama University of Science, Japan)
The attributes of travelers such as home location and age could be useful for many
applications such as information recommendation and targeted advertisement. However,this information is not accessible in most Web services because users do not reveal them.
We propose a method to estimate a foreign tourist's home location based on eachcountry's tendency, in which tourists from certain regions tend to visit certain places
when traveling abroad. The feature for the estimation uses the frequency of photographlocation in a geohash. In this paper, we use foreigners in Japan as a case study. We
evaluate the performance of our proposed method by using photographs obtained fromtheir user accounts on Flickr.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-49

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS5-4 A Proposal of Online Map-matching Based Trajectory Compression
Algorithm Using Road Networks
Shota Iiyama, Tetsuya Oda, Masaharu Hirota (Okayama University of Science, Japan)
As the data size of GPS logs increases, the amount of data transferred from mobile
devices to the server, and the computational cost of analysis of GPS logs increase. Oneof the solutions to these problems is to compress the GPS logs. However, it is difficult
to compress a sparse GPS log while preserving the feature points in the GPS log, suchas the user's movement speed, the shape of the GPS log trajectory, and the direction of
movement of the GPS log. In this study, we propose an online compression algorithm
for GPS logs that improves the compression ratio while preserving GPS logs' featurepoints. Our proposed method compresses GPS logs by using information from the road
network to identify users' roads. We evaluate the performance of this method using the
GPS data of the bus.

OS5-5 Case Study and Direction of Share Cycle System in Japanese Cities
Minoru Kumano (University of Miyazaki, Japan), Toru Hiraoka (University of Nagasaki, Japan)
In the share cycle, people can share a bicycle with others, use and return it using multiple
cycle ports, and register and borrow it with just an IC card or smartphone. The sharecycle system is still new in Japan, and there are few papers even if you search the
National Institute of Informatics. There is no case study. Therefore, in this paper, weselected 16 advanced cases nationwide and conducted a questionnaire survey in 2019.
The purpose was to clarify the business purpose, issues, effects, features, numericalvalues such as the number of ports and the number of bicycles, and the direction. As aresult, it was clarified that the purpose is to improve the ease of migration and to
supplement public transportation, and to improve the profitability and the number ofusers are many issues, which contributes to tourism.

OS5-6 The IoT Solution to Play English Word Learning Tool
Shogo Aizawa, Motohide Yoshimura (University of Nagasaki, Japan)
We innovate an IoT solution to play the English word learning tool. It uses a Raspberry
Pi and two NFC readers. You can learn English words in three steps by using our tool.
First, prepare NFC tags corresponding to English words and illustrations. Next, select apair of the English word and illustration. Finally, hold them over the NFC reader, thenthe correct answers are displayed. Our solution has two aspects. The one is an
educational tool for children and the other is an IoT toy playing with children. As to theformer, the children can learn English words by intuitive operation such as operating thetouch panel and holding the NFC tags. As to the latter, you can experience a part of an
IoT technology such as NFC. In this paper, we report findings through the constructionof the English word learning tool.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-50

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS6 Intelligent Control (6)Chair: Yingmin Jia (Beihang University, China)
Co-Chair: Weicun Zhang (University of Science and Technology Beijing, China)
OS6-1 Faster R-CNN Based Defect Detection of Micro-precision Glass Insulated
Terminals
Qunpo Liu1, Mengke Wang 1, Zonghui Liu1, Naohiko Hanajima2, Bo Su1( Henan Polytechnic University, P.R.China) (2Muroran Institute of Technology, Japan)
1

Micro-precision glass-encapsulated electrical connectors are the core components usedin precision electronic equipment. As an electrical connector, its quality has a huge
impact on the performance of precision electronic equipment. Due to limitations inmaterials and production processes, some of the micro-precision glass connectors
produced have defects such as missing blocks, bubbles, and cracks. At present, it isdifficult to ensure product quality and production efficiency with manual inspection
methods. However, the defect characteristics of micro-precision glass connectors arequite different, and it is difficult for traditional defect detection technology to design anideal feature extractor for detection. Therefore, this paper proposes to use deep learningtechnology to detect missing blocks. Firstly, the sample pictures of missing defects ofelectrical connectors are preprocessed, and then be used to train the deep learning
network of Faster RCNN. According to the test results, the algorithm has an accuracy ofover 80% in detecting missing defects in glass connectors.

OS6-2 Adaptive Sliding Mode Control for a Constant Tension Suspension System
Yuxin Jia, Yingmin Jia, Kai Gong (Beihang University, China)
The constant tension suspension system can be used to counteract the gravity of the test
object, and is widely used in the ground test of spacecraft control schemes. In this paper,the dynamic model of a constant tension suspension system is established, and a newadaptive sliding mode controller is designed. The simulation results show that thetransient time and steady-state error of the control system designed in this paper are
small, which satisfies the requirements of microgravity simulation.

OS6-3 Weighted Multiple Model ADRC for Uncertain Linear System
Weicun Zhang, Jing Ge (University of Science and Technology Beijing, China)©ICAROB 2021 ALife Robotics Corp. Ltd.

P-51

2
参考输入

LADRC
单个LADRC

1

y

系统输出

For uncertain linear systems such as parameter jumping, this paper establishes a fixedmodel set to cover the uncertainties of the real plant to be controlled, and for each local
model with minor uncertainties, the corresponding local controller is designed based on
Active Disturbance Rejection Controller (ADRC) approach. As a result, a weighted
multiple model ADRC adaptive control framework is proposed. Some simulations havebeen conducted based on MATLAB to verify the effectiveness of the proposed weighted
multiple model ADRC adaptive control algorithm.

0

-1
0

20

40

60
时间 t/sec

80

100

120

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS6-4 Trajectory Tracking Control of Differential Wheeled Mobile Robot Basedon Rhombic Input Constraints
Kai Gong, Yingmin Jia, Yuxin Jia (Beihang University, China)
This paper focuses on the trajectory tracking control algorithm for the differential
wheeled mobile robot (DWMR) based on rhombic input constraints. The kinematics anddynamics model of the DWMR are Established, and vector analysis method is used todesign the time-varying parameters of the controller when the linear velocity and angularvelocity of the DWMR were not independent of each other. Through the trajectory
tracking simulation of the 8-shaped curve, a good control performance is obtained.

OS6-5 Maneuvering target tracking based on improved interacting multiple model
algorithm
Weicun Zhang, Meiyu Zhu (University of Science and Technology Beijing, China)
In recent decades, with the rapid development of artificial intelligence, target tracking
technology has become a major aspect of scientific and technological research. Mobile
target tracking technology is widely used in military and civilian fields. For example, in
air defense and air traffic control, reliable and accurate tracking of targets is always themain purpose of target tracking system design. This paper presents a new interacting
multiple model (IMM) algorithm for maneuvering target tracking. Some simulation
based on MATLAB have been conducted to verify the effectiveness of the proposed
algorithm.

OS6-6 Encapsualted Agents of Hybrid Order Discrete Dyanmics
Yunzhong Song (Henan Polytechnic University, P.R. China)
This paper is targeting to investigate the encapsulated cell realization of the hybrid orderagents in discrete time, where it is composed by two different dynamic order agents, one
is the first order and the another one is the second order, the work is the continued caseof its continuous counter part of the encapsulated agents. To be specific, the explorationof the sampling speed is made distinguished from each other, where the fist order section
is assigned at low speed, and the second order section is selected at comparatively highspeed. To compensate the distinct sampling speed, the lifting technology is taken to
analyze the encapsulated system. Theoretical analysis and simulation results are madeavailable for the further reference.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-52

Fig.1 Target tracking
performances

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS7 Information Applications and Security (5)Chair: Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
Co-Chair: I-Hsien Liu (National Cheng Kung University, Taiwan)
OS7-1 The Security Challenges with the Widespread Use of IT Infrastructure in ICS
Kuan-Ming Su, I-Hsien Liu, Jung-Shian Li (National Cheng Kung University, Taiwan)
The communication established by Ethernet is becoming more and more common in theindustrial control systems (ICS), and it brings not only pros but also cons likevulnerabilities. To hijack or get access to industrial devices, the attacker must know the
information about the devices first, but most of the devices, equipment, and systems in
ICS are not public, which is like a black box to the attacker. Therefore, we generalized astandard procedure to attack ICS with Ethernet-enabled, which is able to get the
information and access of the devices in an unknown ICS like identifying the
manufacturer of programmable logic controllers (PLC) and overwriting theconfiguration of PLCs. Also, we will implement the procedure to demonstrate thefeasibility of this procedure with the system obtained from the real industrial control
field whose devices’ manufacturer is Schneider and protocol is Modbus TCP.

OS7-2 A Communication System with Equipment's Characteristics

Chia-Chun Lai1, I-Hsien Liu1, Chi-Che Wu1, Chuan-Gang Liu2, Jung-Shian Li1( National Cheng Kung University, Taiwan) (2Chia Nan University of Pharmacy and Science,
Taiwan)1Over the past few years, the application on internet has grown rapidly and the devices
that can connect to internet also have an explosive increase. In the past, some informationsuch as IP address, MAC address and communication port are commonly used to identify
a specific device on the internet. However, using this kind of information is not enoughfor the purpose to identify precisely because the information can simply be imitated byany other devices. In our research, we focus on the communicational characteristics ofdevice. And we also provide a mechanism that can analyze these characteristics andgenerate configuration references which other devices can utilize. With the customized
communication module and the references mentioned above, we can make other device
imitate the behavior of analyzed device and provide a better effect on device emulation.

OS7-3 Application of the Self-Organizing Map (SOM) to Analyze the Multiple
Perspectives on Cross-National Culture
Li-Ming Chuang, Yu-Po Lee, Shu-Tsung Chao (Chang Jung Christian University, Taiwan)
This study integrates the previous cross-cultural literature and aims to construct ananalysis model of cross-national culture with multiple dimensions from three important
cultural dimension theoretical models commonly used in cross-cultural studies:
Hofstede, Global Leadership and Organizational Effectiveness (GLOBE) and World
Values Survey (WVS). This study uses a self-organizing map (SOM) as an analysis
method to integrate 17 cultural variables from this multicultural dimension for cluster
analysis and explains the cultural types in 26 countries based on the results. This study
explores the differences and similarities of different countries in different culturaldimension analyses and provides a comparative model of multicultural analysis. This
study takes samples from three cross-cultural analysis databases as data sources andemploys the self-organizing map for analysis based on a neural network algorithm. The
results identify the cross-cultural groups of 26 countries and reveal their key culturalsimilarities and differences. We also elaborate upon the findings of these culturalcharacteristics and multi-cultural dimensions.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-53

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS7-4 Threats Hidden in Employee Workstation through Office Files
Tung-Lin Lee, I-Hsien Liu, Jung-Shian Li (National Cheng Kung University, Taiwan)
Due to the single sign-on implementation of windows operating systems, users could
save a lot of time without having to enter their account and password again and again.
However, this setting also allows hackers to Use some dedicated constructed malicious
files to steal the user's identity verification information (NTLM hash), and then launch
attacks like pass the hash, and then log in to the user's computer remotely, which becomesa way to gain access to the organization's intranet. Although there are many ways tocause an NTLM hash leak, fortunately, most of them cannot be exploited by hackers.
For example, this paper will meet various common NTLM hash leak methods, and focus
on the most commonly used method of hacking: Office files, and discuss them The
possibility of extended application and under what circumstances will the existing
protection mechanism be bypassed.

OS7-5 Microsatellite Attitude Control Approach: Combined with Generation
Adversarial Networks Fault-Detection and Cerebellar Model Articulation
Controller Fault-Tolerant Control
Ho-Nien Shou (Air Force Institute of Technology, Taiwan)
A new attitude control architecture for microsatellite is proposed. Based on the deeplearning fault-detection method, the Cerebellar Model Articulation Controller(CMAC)
is used as the fault-tolerant control. With the function of Generation Adversarial
Networks(GAN) to recognize images, the microsatellite attitude fault wavelet spectrum
is used as a guide for the training of generators and discriminators, and for system realtime fault diagnosis and classification. When the system fault diagnosis determines thatthe fault occurs, the cerebellar neural network is involved in fault-tolerant control. The
GAN learning ability of the generative confrontation network is used to solve theproblem of insufficient sample data and sample labeling respectively. The CMAC is used
as a local learning network, which has generalization ability, strong convergence speed,
and easy software and hardware implementation. The simulation results show thatcompared with other methods, the GAN method of fault-detection combined with
CMAC can achieve higher accuracy and robustness.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-54

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS8 Intelligent Systems and Control (5)Chair: Chung-Wen Hung (National Yunlin University of Science and Technology,
Taiwan)Co-Chair: Kuo-Hsien Hsia (National Yunlin University of Science and Technology,
Taiwan)OS8-1 Networking Integration Application of an Intelligent Production Line for
Aerospace Precision Manufacturing
Chau-Chung Song, Chun-Chi Wang, Chen-Pang Chen, Chung-Wen Hung
(National Yunlin University of Science and Technology, Taiwan)
In this paper, the networking integration application of an intelligent production line isstudied and developed for aerospace precision manufacturing. The technicaldevelopment of the performance equalization and control for EDM machine is also
introduced with the related technologies of the machine network and EtherCAT network.
In addition, the development of intelligent production lines is integrated to improve thesystem stability and production efficiency of production line systems for aerospacecomponent manufacturing. The key technologies of Cyber-Physical System (CPS) andintelligent networking is also integrated to constructs the real-time intelligent systemmonitoring system (SCADA) for data record and management. The key technologies ofthrough-hole process are developed for intelligent diagnosis and through-hole inspectionof related production processes. The integrated robot arm is responsible for the loading
and unloading of the material, and the entire production line is connected by EtherCAT
network. The automatic optical inspection (AOI) is introduced at the rear end of the
production line to realize the quality judgment of the finished product. Furthermore, thevirtual cloud network and intelligent factory is achieved with the information collection
and intelligent monitoring system.

OS8-2 Low-Cost Indoor Localization Using Sound Spectrum of Light Fingerprints
Chung-Wen Hung, Hiroyuki Kobayashi, Jun-Rong Wu, Chau-Chung Song(National Yunlin University of Science and Technology, Taiwan)
A low-cost indoor localization system using sound spectrum of light fingerprint is
proposed in this paper, an artificial intelligence, AI, algorithm will be implemented in a
low-cost micro-control unit, MCU, to perform the localization. The unit, complex andtiny differences of the light fingerprints are caused by the different characteristics of thediscrete components used in lighting devices. To reduce the memory size for low-cost
MCU, only sound spectrum of light fingerprint is used to identification the lighting
device. Moreover, the grid search also is used to optimize the hyperparameters tocompact the AI model. The system architecture and algorithm development are discussed
in this paper, and the experimental results will be present to show the proposed system
workable.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-55

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS8-3 Air Valve Fuzzy Control Combined with Sheet Music Recognition Techniques
Applied to Autoplaying Soprano Recorder Machines
Chun-Chieh Wang, Guang-Ming Jhang (Chienkuo Technology University, Taiwan)
In the past research, there are many disadvantages to score recognition and flute
performance. Therefore, this paper would improve on these two points. For the part ofmusic score recognition, we use the y-axis projection method to detect the staff position
and remove it to replace the erosion and expansion in morphology. After research andobservation, it was found that the notes have a specific writing style on the staff.
Therefore, we use this feature to distinguish notes. Detect note stem by x-axis projection
to distinguish them in the first stage, then remove the stems to distinguish them in thesecond stage. After the notes are distinguished, the pixel clustering method is used to
identify the scales. The center point of the rune head is compared with the staff position.
The scale codes are arranged in order from bass to treble. Moreover, the performance isarranged according to basic music theory. The soprano recorder performance part, using
the motors to control the finger to press the blow hole will not keep up the tempo. So wechanged the motor to a solenoid valve to control the pneumatic cylinder to press the blow
hole. In addition, the recorder will have different pressures depending on the pitch. So
we increased the part of the control from one air valve to three air valves. Not only that,
we have divided the range of the recorder into three different ranges of bass, midrange
and treble, which greatly improves the situation that the original sound is generated whenplaying. Finally, we use fuzzy control theory to control the air valve. Experiments provethat the air valve fuzzy control combined with sheet music recognition techniques canfully realize the functions of autoplaying soprano recorder machines.

OS8-4 Image Inpainting Techniques Combined with Isolated Pixel Filtering Applied
to Multifunctional Drawing Robots
Chun-Chieh Wang, Zhan-Xian Ye (Chienkuo Technology University, Taiwan)
The purpose of this thesis is to assist street painters to improve their work efficiency.
That is, the simpler part of the drawing is drawn by the robot. Once the initial picture iscompleted, the artist will take over the painting. In this paper, HSI color space is used to
improve the effect of color simplification such that the recognition of image processing
results is enhanced. Moreover, the isolated pixel filtering is used to replace the lessaffected isolated point color with the surrounding color. Furthermore, Image Inpainting
Techniques are utilized to reduce the distortion caused by the isolated pixel filtering.
Besides, we adjusted the path planning as well as reduced isolated points to dramaticallyreduce drawing time. At the same time, LabVIEW can issue commands directly tocontrol the robot by adding the communication function. In addition, in order to achieve
the multifunctional drawing robotic mode, this system adds the sketch function. Throughthe image resolution adjustment as well as the shortening of the spacing of the drawing
lines, the robot can draw more detailed pictures in the same size of drawing space.
Simultaneously, we reserved some space in the organization to provide the use of
subsequent development of other functions. The measured results confirm that theapplication of the technology in this paper can shorten the drawing time by about 55%
on the multifunctional drawing robot system.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-56

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS8-5 Mobile Robot with Image Recognition -- Using LabVIEW and KNRm
Kuo-Hsien Hsia, Bo-Jung Yang, Jr-Hung Guo, Chang-Sheng Xiao
(National Yunlin University of Science and Technology, Taiwan)
The main purpose of this paper is to use the image recognition of LabVIEW software toconstruct mobile robots with various functions, and make the robots applicable to theindustry and have web monitoring applications. The core of the robot is mainly the
KNRm controller. This controller is suitable for beginners, and can be connected to DC
servo motor, RC servo motor, infrared, ultrasonic and camera to achieve various
functions of the robot. The structure of the robot uses metal parts sold by Studica
company, which can be in accordance with the desired function to assemble the robot.
Since the company is a designated equipment sponsor company for World Skills
competitions, it can also be in line with international standards. Finally, PID control andsensors are added to make the robot movement and position more accurately.

.

OS9 Intelligent Systems and Life (7)Chair: Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
Co-Chair: Evgeni Magid (Kazan Federal University, Russia)
OS9-1 Architecture of a student training computer program for preparing
professional outpatient consulting skills within an electronic medical records system
during COVID-19 alertness situation
Sergey Bulatov1, Evgeni Magid2, Enzhe Kharisova1, Roman Lavrenov2, Vitaly Dudin1, Artur
Khazetdinov2 (1Kazan State Medical University, Russia), (2Kazan Federal University, Russia)

One of the important goals of a medical education in the context of the COVID-19
pandemic is a broad introduction of simulated-based teaching methods, which might
include elements of robotics and artificial intelligence. We analyzed computer programs
that are currently used for maintaining medical records of patients by various polyclinics
of Kazan. Based on these results, we summarized requirements for a training computer
program that could provide students with medical records maintaining skills anddeveloped program’s preliminary architecture. Our main attention focused on modelingsituations that are associated with pre-hospital stage processing of medical records forpatients with suspected or confirmed COVID-19.

OS9-2 Satisfaction Assessment on the Counseling Service System for Full-Time
Teacher-Counselor in Tainan Elementary School
Hsiu-Hao Liu (Chang Jung Christian University, Taiwan)
Yun-Syuan Jhang (National University of Tainan, Taiwan)
As the most important case management system in the school counseling works, thecounseling service system must meet the needs and expectations of the school teachercounselor (the main users). This investigation conducted a questionnaire among teachercounselor to survey their satisfaction when using the counseling service system in Tainan
City. The results of satisfaction items are as follows: "The items are provided abundantand complete by the system" was the highest of all items. "The items are provided tomeet the needs to be collected in the counseling works by the system" was secondly. The
least satisfactory item was "Using the system can help me communicate and interact
with others in the counseling works". Additionally, system usage intention was
effectively explained by perceived ease of use and usage attitudes. The conclusion was
to discuss the implications of this research and suggest several future research issues.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-57

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS9-3 Assess The Critical Factors for the Counseling Service System Usage
Intention
Hsiu-Hao Liu (Chang Jung Christian University, Taiwan)
As the most important case management system in the school counseling works, thecounseling service system must meet the needs and expectations of the school teachercounselor (the main users). Semi-structured interviews were used to interview teachercounselor who have experience in using systems in Tainan. The data are recorded andsemantic analyzed after interviewing, summarize the critical factors that affect the usage
intention of the teacher-counselor. The conclusion was to discuss the implications of thisresearch and suggest several future research issues.

OS9-4 Design and Implementation of EMI Suppression Filter for Electronic
Commutation Fans
Ching-Chun Chuang, Chih-Chiang Hua, Chung-Wen Hung, Chun-Jen Yao(National Formosa University, Taiwan)
This paper presents an electronically commutated fan has a complex digital control
compared to conventional alternating current fans. Since serious electromagnetic
interferences occur in the electronic commutation fan, a filter is required and its
mechanical design is very important. A digital control for an electronically commutated
fan is used to reduce the components of the EMI filter, and the suppression of conducted
and radiated electromagnetic interference is discussed. Moreover, the fan system is
integrated with proper mechanical design to reduce the radiated electromagnetic
interference. Fig.1 shows a single-phase BLDC motor driver for the proposed fan. The
electromagnetic interference produced by the electronically commutated fan is analyzed
and the suppression strategy is provided. The experimental verification for theelectronically commutated fan will be made available to the public.

.

OS9-5 Application of IOT to Forest Management Taking Fushan Botanical Garden
as an Example
Shuo-Tsung Chen, Chih-Chiang Hua, Ching-Chun Chuang(National Formosa University, Taiwan)
In recent years, the Internet of Things (IoT) technology has developed rapidly and has
been successfully applied in different fields, and has begun to expand the application
context of the Internet of Things. This work aims to study how to apply the IoTtechnology to forestry management, including: 1. Forestry management using wirelessnetwork communication technology of Low Power Wide Area Network (LPWAN) suchas LoRa and NB-IoT; 2. Apply different sensing technologies to survey resource of forestand monitor the microclimate changes in forest. In order to actually verify the proposed
feasible communication technology and sensor arrangement, we chose the Fushan
Botanical Garden with the most diverse and complex terrain in Taiwan as theexperimental site. Fig. 1 shown that the evaluation of a situation and setting for Forest
Resources Survey. We actually built LoRa and NB-IoT communication equipment(including relay equipment) and various sensors to test the influence of terrain, climate,
and trees on signal transmission efficiency and equipment installation. The returned dataalso verifies the successful operation of various communication devices and sensors.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-58

.

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS9-6 Development of the IoT Module using MQTT Protocol and AES
Jr-Hung Guo*, Tzu-Yuan Lin, Kuo-Hsien Hsia
(National Yunlin University of Science and Technology, Taiwan)
The efficiency and safety of the Internet of Things have always been the focus of thedevelopment of IoT devices. Because the chips used in IoT devices generally have poor
computing power, they cannot transmit data quickly and in large amounts, and use more
complex security algorithms. Therefore, this thesis is to develop an IoT module with
STM32 chip as the main controller. This module uses MQTT Protocol and AES
encryption technology, and this IoT module can be operated directly with a browser.
MQTT is a communication protocol for the Internet of Things, which was developed by
IBM and Eurotech, and officially became an OASIS international standard in 2014. The
purpose of development is to send and receive processing messages under narrow
bandwidth and low energy consumption conditions. To ensure the security of IoT
communications, we use AES encryption technology. Through this design, thecommunication of the entire IoT module is more efficient and safe. Finally, we applied
this IoT module to the security system, and the overall efficiency and safety have beenverified. In the future, we will continue to improve related software and hardware so that
this IoT module can be used in different fields.

.

OS9-7 Exploring the Intention to Continuance of Learning Programming at
Elementary School of Rural Area by the mBot Robot
Yung-Hsin Cheng, Jia-Ming Hsiao (National Yunlin University of Science and Technology,
Taiwan)Since Curriculum Guidelines of 12-Year Basic Education implemented by the Ministryof Education in 2018, the program learning courses have been added to junior highschool education. However, there are no programming course in the elementary school.
This study is proposed to explore the continuity and intentions of the rural area students
in the programming course with mBot robot and mBlock programming tool through the
Post-Acceptance Model of IS Continuance. It is indicated that enlightenment educationof computational thinking should be implemented during the elementary school bymeans of graphical programming software and robot practice. Through the graphicalsoftware and teaching robots, we can cultivate problem solving skills for students' logic,
creative thinking and communication, as well as through the task-guided way to train thestudents' concentration and perseverance.

.

OS10 Advanced Control Systems and Signal Processing (7)Chair: Takuya Kinoshita (Hiroshima University, Japan)
Co-Chair: Shinichi Imai (Tokyo Gakugei University, Japan)
OS10-1 Nonlinear Internal Model Controller based on Local Linear Models, and its
Application
Shinichi Imai (Tokyo Gakugei University, Japan)
In this paper, nonlinear internal model controller based on local linear models, and its
Application. The internal model control has a simple structure and has a high robustness
for system uncertainties. However, there are few studies of internal model control
schemes for nonlinear systems. On the other hand, many controlled systems have thenonlinearity. The effectiveness of the newly proposed control scheme is numericallyevaluated on experiment examples in comparison with the conventional control methods
for nonlinear systems.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-59

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS10-2 Design of a data-driven control system for reference model design using
predicted signals
Yuki Nakatani, Takuya Kinoshita, Toru Yamamoto (Hiroshima University, Japan)
In recent years, data-driven control schemes that do not require system identification
have been actively studied. Generally, it is easy to give a reference model focusing only
on the output response. In contrast, it is difficult to give a reference trajectory considering
the input signals based on the controlled system's characteristics. Furthermore, it isnecessary to consider the output signal and the input signal since there is a limit of theactuator performance in the control design of the actual machine. This paper proposes adata-driven control system that can predict the input/output response of an unknown
system in offline using operation data. The effectiveness of the proposed scheme isnumerically verified.

OS10-3 Design of a Data-driven GMV Controller Using the Nelder-Mead Method
LiYing Shi, Zhe Guan, Toru Yamamoto (Hiroshima University, Japan)
This paper presents a design scheme that can obtain the optimized generalized minimum
variance (GMV) control parameters by applying the Nelder-Mead (NM) method basedon proportional-integral-derivative (PID) controller for linear systems. The NM method
is used to find the most suitable parameter λ of GMV controller, then the PID parameters
can be obtained. In the previous GMV controller, the PID parameters are calculated bysimply changing λ manually. Therefore, it is hard to get desirable control performance.
The application of NM method can optimize the calculation of the most suitable λ andget better PID parameters. Furthermore, the estimation of closed-loop response uses
data-driven approach. The effectiveness of the proposed scheme is verified by using asimulation example.

OS10-4 Design of a Databased-Driven GPC for Nonlinear Systems
Zhe Guan, Toru Yamamoto (Hiroshima University, Japan)
This paper presents a Generalized Predictive Control (GPC) design scheme considering
databased-driven approach for nonlinear systems. In several conventional design
methods, model parameters are required to calculate control parameters. However, it istime- and cost-consuming to identify the model in real practical systems, especially forchemical or thermal industries with unknown delay time and strong nonlinearity.
Recently the database-driven approach has been attracted considerable attention to tackle
the model identification issue. It is widely applied in nonlinear systems. On the other
hand, GPC is considered in the cases with unknown delay time. As a result, the controller
is designed based on GPC by using databased driven approach. The proposed scheme isverified by a numerical simulation which demonstrates the effectiveness.

OS10-5 Design of a Data-Driven Controller using Open-Loop Data
Y. Nishiya, Takuya Kinoshita, Toru Yamamoto (Hiroshima University, Japan)
In recent years, data-driven control has been proposed as a control system design that
does not require system modeling. Furthermore, it is extended to a nonlinear systemusing a database. It is necessary to collect several data to obtain good control
performance. However, collecting sufficient data incurs both time and human costs. This
paper proposes an offline scheme for generating several data from a set of open-loop
data is proposed. Additionally, the controller is designed considering the input signal toprevent a heavy burden on the actuator. In this paper, the numerical examples verify theeffectiveness of the proposed scheme.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-60

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS10-6 Improved Estimation of Sway-Angle for Overhead Crane based on Phase
Difference of Acoustic Signals in Frequency Domain
Hanako Ogawa, Takeshi Yamada, Masayoshi Nakamoto (Hiroshima University, Japan)
When operating crane, sway of the payload is an issue that should be controlled. For theanti-sway control, it is essential to detect the sway-angle θ. However, it is difficult tomeasure the sway-angle directly. In this paper, we propose a method to obtain the phasedifference (λ1 - λ2) of the acoustic signals s1 and s2 in the frequency domain by using
the discrete Fourier transform (DFT). Based on the phase difference, we compute thearrival time difference and convert that to the sway-angle of deflection. To improve theaccuracy, we discuss a method for generating the frequencies of the sound sources andthe signal processing with filters. In the experiments, the sway-angle measured by thecamera are compared with the proposed method.

OS10-7 Study on an Optimal Design Method for Control Systems based on
Bayesian Optimization
Koichi Hirota, Shin Wakitani, and Toru Yamamoto (Hiroshima University, Japan)
Most product systems consist of a combination of multiple subsystems. Appropriate
setting and design of operating goals (functional goals) of these subsystems areimportant to achieve and maintain the target performance of the system. In recent
years, model-based development (MBD) has been attracting attention in the industrial
world. In the MBD process, a user can evaluate the subsystem’s performance and easily
improve their design. In many situations of control system design, plant design iscompleted in advance. In subsystem design, a plant and a controller are required to be
optimized at the same time for the functional goals determined upstream design process.
Based on the background, this study considers the simultaneous optimum design of thecontroller and the plant.

OS11 Robotic Manipulation (3)Chair: Kensuke Harada (Osaka University, Japan)
Co-Chair: Tokuo Tsuji (Kanazawa University, Japan)
Co-Chair: Akira Nakamura (AIST, Japan)
OS11-1 Using Various Evaluation Standards to Determine an Error Recovery
Process in an Automation Plant
Akira Nakamura*1, Natsuki Yamanobe*1, Ixchel Ramirez Alpizar*1, Kensuke Harada*2, Yukiyasu
Domae*1*1( National Institute of Advanced Industrial Science and Technology (AIST), Japan*2
Osaka University, Japan)

In an automation plant, not only simple repetition tasks but also complicated tasks arecarried out. An error is more likely to occur in such difficult work, so the improvementof the technique to perform recovery of an error is needed. The task often is re-executed
after returning to previous step, in the case of a big error. Therefore, it becomes theimportant problem to decide both the past step that it should return to and the recovery
planning after returning. In this paper, error recovery planning taking these two factors
is proposed by using various evaluation standards.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-61

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS11-2 Robotic Picking for Piled Sushi Topping

Kenta Matsuura*1*2, Keisuke Koyama*1, Weiwei Wan*1, Kensuke Harada*1( Osaka University, Japan, *2 Currently with Yaskawa Electric Co. Ltd., Japan)*1This paper proposes a method for picking piled sushi topping. By observing a humanmotion picking a sushi topping, we propose two picking strategies where one is to insert
a finger into the separation among two toppings with shaking the finger, and the other is
to insert a finger into the separation between a topping and a table. Along with twosegmentation method, i.e., plane segmentation and LCCP segmentation methods, we
experimentally verify the effectiveness of the proposed approach.

OS11-3 Motion Generation by Learning Relationship between Object Shapes andHuman Motions
Tokuo Tsuji1*, Sho Tajima*1, Yosuke Suzuki*1, Tetsuyou Watanabe*1, Shoko Miyauchi*2,
Ken’ichi Morooka*2, Kensuke Harada*3, and Hiroaki Seki*1*1( Kanazawa University, Japan, *2 Kyushu University, Japan, *3 Osaka University, Japan)(This paper presents a method for planning a robot motion of daily tasks by learning
the relationship between object shapes and human motions. Robots are required to beable to deal with multifarious objects in various categories. However, it is difficult for
robots to plan motions automatically for performing a task because objects even in thesame category have different shapes. In our method, the motions are estimated bylearning the relationship between object shapes and human motions using linear
regression analysis. We evaluate the estimated motions and the experimental results oftasks which are performed by a robot to verify the effectiveness of our proposed method.

OS12 System and Control (18)Chair: Huailin Zhao (Shanghai Institute of Technology, China)
Co-Chair: Jichao Zhao (Tianjin University of Science and Technology, China)
OS12-1 The Optimized Intelligent Algorithms on Face Recognition and Tracking
for ROS-based Robots
Yue Chen, Shuhao Tian, Huailin Zhao, Shengyang Lu (Shanghai Institute of Technology)
With the development of artificial intelligence, face recognition and tracking technology
have been widely used in many fields such as target positioning, automatic driving, andhuman-computer interaction. Recently, a large number of face detection, recognition andtracking algorithms have emerged, but there are still many shortcomings in practicalapplications, such as slow face detection, low detection accuracy, and face recognition
and tracking for ROS robots Algorithms are rare. This paper improves the traditional
Haar-like algorithm and LK optical flow tracking algorithm, and designs a ROS robot
platform based on the improved algorithm. By comparing the accuracy and timelinessof face detection and tracking between the improved algorithm and the traditionalalgorithm, the superiority of this design algorithm is obtained.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-62

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS12-2 Control of a Novel 5D Hyperchaotic System

Qiang Wei1, Hong Niu2
( Army Military Transportation University, China)
2( Tianjin University of Science & echnology, China)
1

In this paper, a novel five-dimensional (5D) autonomous hyperchaotic system ispresented, and the characteristics of the 5D system are given in brief. For control of the
5D hyperchaotic system, a linear feedback controller is designed via the Lyapunovstability theory, so that the 5D system is no longer hyperchaotic but globally
asymptotically converges to the equilibrium point at the origin. The numericalsimulation results are given to illustrate the feasibility and effectiveness of the method.

OS12-3 Design of Multifunctional Vehicle Interior Environment Monitoring
System Based on Wireless Communication
Yuqi Yan, Jialin Yang, Zhongxu Qin (Wuhan University of Technology, China)
The interior environment of the car affects the driver's mental state to a certain extent,
so it is necessary to design a multi-functional interior environment monitoring system.
In this paper, STM32103 single chip microcomputer is used as the core controller, which
integrates multiple sensors to collect various environmental information in the vehicle,
and transmits the data to the monitoring platform through WiFi wireless communication.
When the detected data exceeds the preset threshold, the monitoring system will generate
an alarm to remind the user to carry out relevant operations. The system has theadvantages of low cost, convenient use and high precision, and has certain commercial
value and market prospect.

OS12-4 Robot Structure and Motion control Design Based on UG and ProductYuhang Sheng (Tianjin University of Science and Technology, China)
The paper is to design a six-degree-of-freedom biped robot by the research on thehumanoid characteristics of the biped robot. Our biped robot chooses the steering gear
ASMC-03B as its power unit, we restricts the model parameters by analyzing thefunction relationship between the steering gear torque and the volume, and regulates thesize of every parts, and create a motion analysis model. In the hardware part, Arduino
UNO, which is used as the main control chip, realize the communication between themain control chip and the servo drive module PAC9685 through the IIC bus protocol,
which saves the main control chip resources and ensures the execution efficiency. In thesimulation part, it is to simulate the steering angle of the steering gear, and output it in
the form of a waveform.

OS12-5 Design of Brushed Motor Position Loop Control System Based on
Incremental PID
Tianyi Zhang, Peng Lu (Tianjin University of Science and Technology, China)
Motors are an important part of the transmission and control system. With thedevelopment of technology, the focus of motors in practical applications has shifted toprecise control of speed, position and torque. This paper focuses on the control systemdesign of the brush motor position loops, using STM32 microcontroller and incremental
PID to achieve accurate control of the motor stopping moment position. The software isdesigned in C programming. It is divided into four parts: System Clock Configuration
Functions, Principal Functions, System Tick Timer Interrupt Callback Function, Position
closed loop PID Control Design Function. The hardware part is based on the
STM32F103VET6 core board.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-63

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS12-6 Design of a Fire alarm system
Yuhui Cheng, Tianyi Zhang (Tianjin University of Science and Technology, China)
This paper introduces the hardware and software design of the fire alarm system, andanalyzes its working principle. This paper uses a distributed sensor network based on
ZigBee to achieve fire alarm and real-time evacuation. Select infrared flame sensor and
MQ2 smoke sensor as the information source. The core control board is the Arduino
Mage2560 development board. Through the ESP8266 module and the GPRS module,
the multi-channel information transmission function can be realized. The alarm can beclassified according to the fire situation. System status monitoring software, written in
C#, can view the status of each node in real time.

OS12-7 Design of intelligent curtain control circuit based on single chip
Microcomputer
Qianqian Zhang, Jichao Zhao, Haokang Wen, Hongbo Hao(Tianjin University of Science and Technology, China)
Since entering the 21st century, high technology has promoted the development of
human beings and artificial intelligence has been popularized gradually. In this paper,
the structure and principle of crystal oscillator circuit, reset circuit and photosensitive
sensor circuit are introduced with 89C51 single chip microcomputer as the main control
unit. The whole circuit system is analyzed in this paper. The circuit module ofphotosensitive sensor can be used to detect the external light intensity and automaticallycontrol the curtain. The user can set the temperature threshold with the remote control.

OS12-8 Research on the algorithm of flue gas desulfurization system
Hongbo Hao (Tianjin University of Science and Technology, China)
In this paper, the process of flue gas desulfurization and denitrification, which is anonlinear, time-varying, large lag and strong coupling complex variable process, isanalyzed and studied in depth. Based on the analysis of the coupling characteristics ofthe adsorption tower, the control model of the desulfurization and denitrification processof the adsorption tower is established, and the system identification adaptive PID
Decoupling control algorithm is adopted. The Simulink toolbox corresponding to Matlab
software is used to simulate and verify the effectiveness of the algorithm. The RBF
neural network identifier is used to identify the plant model on-line to improve theadaptive ability of the controller.

OS12-9 A Survey of Low Visibility Image Enhancement Based on MSRCR
Algorithm
Haokang Wen, Hongbo Hao (Tianjin University of Science and Technology, China)
With the development of computer vision systems in the fields of traffic and safety
monitoring, image enhancement has become an important research direction ofcomputer vision. After development, the Retinex algorithm has a better enhancement
effect, and the MSRCR algorithm has now developed into one of the important methods
in image enhancement. This article introduces the development of the Retinex algorithm,
and focuses on the main process and steps of the MSRCR algorithm, and uses thealgorithm to enhance the low-visibility images of haze and night in multiple scenes. The
actual results show that the MSRCR algorithm has a better enhancement effect and has
a wide range of application values.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-64

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS12-10 Design of a Taxi Meter Based on Single Chip Computer
Haokang Wen, Min Wang (Tianjin University of Science and Technology, China)
As a convenient means of transportation, taxis greatly facilitate people's daily travel.
Based on such a background, this article designed a simple taxi meter based on
STC89C52 microcontroller. This taximeter uses a single-chip microcomputer as thecore, combines display module, clock module, storage module, key module, andcombines software and hardware to build the required taxi meter model. The taxi meter
introduced in this article has the functions of displaying time, starting price, and realtime display of the cost according to vehicle mileage. Theoretically, the meter has higher
accuracy and has a wider practical value in daily life.

OS12-11 Design of a High Precision Digital Clock Based on Single Chip
Microcomputer
Haokang Wen, Qiang Zheng (Tianjin University of Science and Technology, China)
Compared with traditional mechanical clocks, digital clocks have higher accuracy anddurability and are widely used in people's daily lives. This paper designs a digital clock
which uses AT89S52 single-chip microcomputer as the main control chip. It can count
hours, minutes, and seconds, and can calibrate the time. It can switch between 24-hourand 12-hour systems. As a smart clock, while displaying the time, it also adds thefunction of temperature display. The intelligent digital clock designed in this paper has
stable performance in theory and has certain practical value.

OS12-12 Design of Manchu Image Acquisition System Based on STM32
Hongbo Hao1, Fulin Zheng2( Tianjin University of Science and Technology, China)(2 Dalian Minzu University, China)
1This paper designs a Manchu image acquisition system based on STM32 to protect andutilize Manchu ancient books. The system uses stm32f407 as the core controller, withthe help of image sensor, LCD screen, SD storage module to complete the function ofimage acquisition, storage and upload of Manchu ancient books. This paper describes
in detail the hardware circuit design, software programming and overall function
debugging of the image acquisition system. In order to achieve the purpose of digitalprotection of Manchu ancient books, we can collect, display, save and upload images indifferent situations.

OS12-13 Design of timing socket based on single chip microcomputer
Hongbo Hao1, Shuailin Chen2( Tianjin University of Science and Technology, China)(2Qingdao University of Science and Technology, China)
1

In this paper, the design of timing socket is mainly used for intelligent timing control ofhousehold appliances, and the working time of timing is set by pressing the key. In theworking period, the external connection device and the power supply are in the
conduction state. Outside the set time period, the external connector is disconnected fromthe power supply. This design uses STC89C52 as the driving chip. DS1302 clock module
provides accurate timing. LCD1602 LCD module displays time and operation interface.
The relay module controls the connection between the socket and the power supply. DC
power supply module is converted to DC power supply module.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-65

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS12-14 Study on the effect of physical fitness training on children’s cognitive
ability
Jianhua Deng, Lei Ning (Yanshan University, China)
With the word physical fitness getting more and more attention in our country, physicalfitness has been widely developed in our country. Therefore, in recent years, the students'
physique has received attention from all aspects of society, and improving the students'
physique has also become a new topic for physical education workers. I
try
todiscuss the effect of physical fitness training on children's cognitive ability from theperspective of physical fitness training. To provide theoretical basis and reference forintervention of children's cognitive ability training methods. So as to promote theimprovement of children's cognitive ability and provide guarantee.
This
paper
studies the influence of physical fitness training on children's cognitive ability by using
the methods of literature, observation and experiment. Finally, the conclusions are
drawn.

OS12-15 Effects of high heels on plantar stress in women
Jianhua Deng, Lei Ning (Yanshan University, China)
OBJECTIVE: to investigate the changes of Plantar pressure and gait in women wearing
high-heeled shoes for different years. METHODS: from March 2019 to April 2019, a
total of 30 female faculty members in Yanshan University were surveyed byquestionnaires and plantar pressure was measured by static and dynamic tests. RESULT:
1. The peak value of plantar pressure of the arch of foot increased with the increase ofwearing time (P & Lt; 0.05). 2. The plantar pressure and impulse of the second metatarsal
and the fourth metatarsal increased with the increase of wearing time (P & Lt; 0.05).
CONCLUSION: 1. Long-term wearing high-heeled shoes will cause foot discomfort,
easy to cause flat foot and Hallux Valgus Lesions; 2. The distribution of plantar pressure
and the change of arch shape and gait may be the main reasons for wearing high-heelshoes.

OS12-16 A research on Intelligent Classification of Urban Trash Bins based on
Machine Learning
Longyu Gao, Zilong Liu, Luqi Shen, Songyun Shi, Yongzheng Lv(Tianjin University of Science and Technology, China)
Aiming at the problems of inaccurate, insensitive and general performance of the current
intelligent garbage sorting bins, an intelligent garbage sorting bin based on machine
vision is proposed. The trash can is mainly divided into five parts: the main control
module, machine vision module, classification module, overflow reminder module, and
Wi-Fi Internet of Things module. The trash can uses convolutional neural networks tobuild an intelligent garbage classification model and classification algorithm to achieve
rapid and accurate garbage classification. The experiment will be done based on theidentification of waste bottles, analyze the recognition characteristics of machine vision,and then propose methods to improve the accuracy of recognition.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-66

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS12-17 A Design on Intelligent Public Trash Can based on Machine Vision and
Auxiliary Sensors
Longyu Gao, Leixin Han, Jiangyu Wu, Mingfei Liu, Ruming Kang
(Tianjin University of Science and Technology, China)
In order to improve the correct rate of front-end recognition in the garbage classification
process, based on the machine vision technology, the automatic garbage classification
system is designed and it has a significant improvement in recognition accuracy
compared to traditional smart garbage cans. But in the case of identifying irregular
garbage, the recognition accuracy is greatly reduced. In order to solve this kind ofproblem, four types of auxiliary sensors are added to the trash can, through the mutual
cooperation between the sensors, combined with the results of machine vision
recognition, comprehensive judgment, greatly improved the recognition accuracy ofirregular garbage.

OS12-18 A research on Front-end Garbage Classification based on Machine Vision
Longyu Gao, Zhiqing Xiao, Junlong Hao, Luqi Shen, Manqian Hu(Tianjin University of Science and Technology, Tianjin, China)
Adding a machine vision recognition module to the traditional smart trash can caneffectively improve the efficiency of trash recognition. The intelligent garbage
classification model constructed by the convolutional neural network can accurately
identify the types of garbage, with an average accuracy rate of 0.87. Deploy the trainedmodel on openMV and test it on the produced physical trash can. After the system isstable, the average time to complete a sorting and recovery is 2s. Experiments show thatthe system can effectively identify the types of garbage and complete garbage
classification and recycling.

OS13 Intelligent Systems and Robotics (14)Chair: Fengzhi Dai (Tianjin University of Science and Technology, China)
Co-Chair: Yunzhong Song (Henan Polytechnic University, China)
OS13-1 Analysis of the Consensus Protocol of Heterogeneous Agents with TimeDelays

Jichao Zhao1, Fengzhi Dai1,3, Yunzhong Song2( College of ElTianjin University of Science and Technology, China, 2Henan Polytechnic
University, China, 3Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China)
1

In practical engineering applications, the research on the consensus of heterogeneous
multi-agents is of great significance. The consensus of multi-agents mainly includes
average consensus, maximum consensus, and minimum consensus. The paper studies
the average consensus of heterogeneous multi-agents, including continuous-timeconsensus protocol, discrete-time consensus protocol, consensus with time delay, andconsensus of switching topology. The knowledge of graph theory is used to describe thesystem, and the results are simulated and analyzed for the consensus of time delay andswitching topology to verify the correctness and effectiveness of the consensus protocol.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-67

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS13-2 Burrows-Wheeler Transform Acceleration based on CUDA
Chang Sheng, Fengzhi Dai (Tianjin University of Science and Technology, China)
Burrows-Wheeler transform (BWT) is a commonly used transform in compression or
text comparison. For example, in bzip2, BWT is used to preprocess the original data,then the same characters in the original data are close to each other, which improves thecompression rate. Because the prefix tree of the original string can be easily obtained
from the result of the BWT, BWT is also applied to the search and comparison of strings.
For instance, the comparison of DNA sequences uses the BWT algorithm. However,
BWT is not a fast algorithm, only Tens of megabytes per second on CPU. This article
uses the GPU to sort the original string by the base of the 4-byte key size radix sort. Afterradix sort, the part with insufficient length is sorted again to complete the BWT
algorithm.

OS13-3 Design of Automatic Water Supply Upper Computer System
Peng Lu, Fengzhi Dai, Tianyi Zhang (Tianjin University of Science and Technology, China)
Aiming at the unstable water supply and fluctuating water pressure in the water supply
system, an upper computer system of automatic water supply system based on Siemens
PLC and KingView software was proposed. The pressure sensor in the water supply
pipeline is used to detect the pressure in the pipeline, and the liquid level sensor monitors
the liquid level in the tank. The sensor transmits the data to the PLC, and the PLC issues
the control instruction after the computation processing. KingView software can realize
real-time monitoring and fault alarm of the system. The system can effectively improve
the stability of water supply, and can avoid human entering the dangerous environment
to search for unknown faults.

OS13-4 Analysis of Boiler Water Level System based on the Fuzzy Control
Tianyi Zhang, Fengzhi Dai, Peng Lu (Tianjin University of Science and Technology, China)
Based on the three-stroke water supply system, this paper analyzes the performance
characteristics of the boiler water level control system in reality, so as to reason out theappropriate fuzzy control rules, design fuzzy controller, and applied to the control
system, so that the system for self-adjustment of PID parameters, constitute a fuzzy PID
control system. On this basis, this paper analyzes the performance, advantages andcharacteristics of two control systems: the traditional PID control system and the fuzzy
PID control system, and simulates the parameters of the input variables for comparison
and analysis.

OS13-5 Development of the Circuit System for Greenhouse Environment
Regulation
Yuhui Cheng, Fengzhi Dai, Chengxu Ji, Peng Lu(Tianjin University of Science and Technology, China)
Aiming at the demand of agricultural modernization, this paper proposes a greenhouseenvironment regulation system based on single-chip microcomputer. For the greenhouse,
due to the closed space and the requirements of the crops for the stability of the growthenvironment, the cheap single-chip control system can be used to complete this work
well. The system can obtain the current environmental data in the greenhouse throughthe DHT11 temperature and humidity sensor and the light sensor. The system can keep
the environment in the greenhouse in a proper state without people, and is suitable forlarge-scale greenhouse planting.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-68

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS13-6 Design of an Intelligent Car based on MSP430
Ruming Kang, Fengzhi Dai (Tianjin University of Science and Technology, China)
The system takes the MSP430 single chip microcomputer as the control core, uses thereflective photoelectric sensor TCRT5000 module to track the line, and realizes theautomatic tracking. In the experiment, the black-and-white line is used as the route. The
system is driven by the L298N module and controlled by the PWM DC motor. This paper
introduces the principle of the reflective photoelectric sensor and the circuit diagram of
the tracking module. It also explains how to realize the automatic tracking based on the
MSP430 single chip microcomputer. The technology can be used in warehouse,
unmanned production line, intelligent service robot and other fields.

OS13-7 Design of a WIFI Video Car

Qianqian Zhang1, Fengzhi Dai1,2, Jichao Zhao1, Haokang Wen1, Hongbo Hao1
(1Tianjin University of Science and Technology, China)(2Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China)

This subject is mainly composed of motor, frame, STC89C51 single-chip
microcomputer and other auxiliary modules. Through the android client broadcast video
car cameras, wireless video signal acquisition by the client to send commands to the car
at the same time, wireless router transmission instruction by wireless router, single-chip
microcomputer for processing, then driven by single chip microcomputer control motor,
drive motor rotation, steering device, so as to realize the control of motor sports andvideo acquisition.

OS13-8 Design of WebGIS Transportation and Distribution System based on the
Genetic Algorithm
Hongbo Hao, Fengzhi Dai (Tianjin University of Science and Technology, China)
With the concept of "smart logistics" put forward, the transformation of the logistics
industry facing information technology is facing great challenges. At present, China's
logistics industry is facing problems such as extensive resource allocation, lowmanagement level and high cost. Aiming at the transportation problem in logistics
industry, this paper designs a solution method of balanced / unbalanced transportation
problem based on Monte Carlo similarity and the genetic algorithm. Two coding
methods, Prufer number and matrix are adopted. On this basis, dynamic mutation rate
and random mutation strategy are designed, and Monte Carlo similarity receiving
method is introduced. Finally, from the perspective of system requirements, the WebGIS
transportation and distribution system based on the genetic algorithm is designed anddeveloped.

OS13-9 A Study of YOLO Algorithm for Target Detection
Haokang Wen, Fengzhi Dai, Yasheng Yuan
(Tianjin University of Science and Technology, China)
With the development of deep learning, target detection has become one of the research
directions of many scholars. As one of the more mature algorithms, the YOLO series ofalgorithms have been widely used in real life. Combining the development history of the
YOLO algorithm, this article focuses on the main framework and main content of thecurrent latest YOLOv5 algorithm, and uses the YOLOv5 model to identify and detectfootballs. This article evaluates its detection effect. The test results show that YOLOv5
has a wider application meaning in real life.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-69

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS13-10 Research on Recognition and Application of EEG Signal based on
SSVEP-BCI
Di Yin1, Fengzhi Dai1,3, Mengqi Yin2, Yasheng Yuan1, Yuxuan Zhu1
(1Tianjin University of Science and Technology, China)(2Hebei University of Chinese Medicine, China)
3( Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China)

In recent years, brain-computer interface systems based on steady-state visual evoked
potentials (SSVEP) have attracted attention due to their high information transfer rate
(ITR) and more and more targets. The current mainstream algorithms for SSVEP
recognition have greatly improved the accuracy and target detection time. This paper
designs a robotic arm application system based on the eCCA-Y method for multi-target
recognition. The phase characteristics of CCA's sine and cosine signals are added to the
EEG signal. Compared with mainstream algorithms, research shows that this method
can improve the SSVEP-based BCI performance. And choose a six-degree-of-freedom
manipulator as the actuator of the brain-computer interface, and use a phase-encoded
stimulation paradigm for multi-target recognition to conduct experiments on theapplication of the proposed method.

OS13-11 Research on Bad Driving Detection based on Behavior Recognition
Yasheng Yuan, Fengzhi Dai, Di Yin, Yuxuan Zhu(Tianjin University of Science and Technology, China)
Dangerous driving behavior is considered to be the direct or indirect reason of roadaccidents. Although artificial video surveillance is good to prevent bad driving, it wastes
too much time and manpower. How to effectively identify behavior become the focus of
the research. In recent years, deep learning showed the huge advantage in the field ofcomputer vision. This paper adopt a number of deep learning network models, mining
video integration of space and time features, introduction of analogy in human visual
attention mechanism, improve the model deeply, using the LSTM to accurate andefficient video behavior analysis technology.

OS13-12 Research on Crop Image Recognition Technology - Take Daylily as an
ExampleJichao Zhao, Fengzhi Dai (Tianjin University of Science and Technology, China)
Digital image recognition technology is the core technology of agricultural robots. The
key research content is to judge the maturity of crops through image recognition. In thispaper, the daylily is used as the crop to be identified, and the digital image processing
technology is used to separate the image of daylily from the background, and then send
the processed information to the picking actuator of the robot to assist the actuator inpicking tasks. This method can be used not only in the recognition of daylily, but also in
the recognition of tomatoes, cucumbers and other crops by adjusting the recognition
pictures and recognition parameters. It has high scalability.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-70

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS13-13 Design of Daylily Agricultural Picking Robot
Jichao Zhao, Fengzhi Dai (Tianjin University of Science and Technology, China)
Agricultural picking activities occupy an important role in daily life, and the researchand development of agricultural picking robots play a more crucial role and increasingly
become an advantageous tool to improve agricultural productivity. Daylily, as a crop fordaily consumption, can significantly reduce serum cholesterol and has extremely highnutritional value. However, the picking conditions of day lily are harsh, and long-term
picking is likely to cause extremely high harm to the human body, especially the hands.
This paper designs an agricultural picking robot that can pick daylily automatically,
which can greatly free hands and improve labor productivity. Especially in health, it cansignificantly reduce the harm to the human body.

OS13-14 Development of Intelligent Public Trash Can based on Machine Visionand Learning
Longyu Gao, Fengzhi Dai, Zhiqing Xiao, Jiangyu Wu, Zilong Liu(Tianjin University of Science and Technology, China)
At present, with the maturity of machine vision technology and the continuous expansionof application fields, there have been many intelligent trash cans based on machine
vision, which can realize certain garbage identification and automatic classification.
However, due to certain technical limitations of machine vision, it is impossible toidentify all garbage. In this paper, a smart public trash can based on machine vision andauxiliary sensors is proposed. In addition to realizing machine vision to identify andautomatically classify garbage, sensors will also be used to assist in identifying garbage
to solve problems such as the same garbage classification of different shapes. At thesame time, enhanced learning will be added to realize the self-learning of the trash can,so as to achieve the goal of continuously increasing identifiable types.

OS14 Image Processing (5)Chair: Joo Kooi Tan (Kyushu Institute of Technology, Japan)
Co-Chair: Seiji Ishikawa (Kyushu Institute of Technology, Japan)
OS14-1 Detection of a Fallen Person and Estimation of Their Head Position from UAV
ImagesHaruka Egawa, Seiji Ishikawa, Joo Kooi Tan (Kyushu Institute of Technology, Japan)
In recent years, aerial photography has often been used for finding victims in the eventof a disaster. Searching from the sky enables quick rescue activities in places that are
difficult for human to approach. In this paper, we propose a method of detecting a person
fallen on the ground from the images taken by a camera mounted on a UAV(multicopter).
Unlike pedestrians, the head orientation of a fallen person in an image is not identical.
Therefore, in the proposes method, combination of Ri-HOG features and Ri-LBP
features are employed for representing a fallen person, and the fallen person is detectedby a classifier constructed using Random Forest. In addition, a head position of a fallen
person is estimated by using the peak of the gradient direction histogram. The
effectiveness of the proposed method was verified by experiments.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-71

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS14-2 Development of a Pedestrian Crossing Navigation System for Visually
Impaired People Using MY VISION
Kohei Kitagawa, Seiji Ishikawa, JooKooi Tan (Kyushu Institute of Technology, Japan)
In this paper, we propose a system for a visually impaired person to cross a pedestriancrossing safely by the employment of the state of a traffic light and the remaining
distance to the other side obtained from MY VISION images. The traffic light at apedestrian crossing is detected by a discriminator using HOG (Histograms of Oriented
Gradients) features and Random Forest. The possible actions for a visually impaired
person at a pedestrian crossing are classified into six types of action patterns byanalyzing the traffic lights and the distance information. The proposed system chooses
one of the six patterns so that the visually impaired person may cross the pedestriancrossing safely. The effectiveness of the proposed method was verified by experiments.

OS14-3 Abnormal Human Action Detection Based on GANTomoya Sano, Joo Kooi Tan (Kyushu Institute of Technology, Japan)
Recently, surveillance systems using cameras have been widely used, according tofrequent outbreak of crimes. One of the important roles of such a camera surveillancesystem is to detect abnormal human actions or events. In this paper, we propose a methodof abnormal human actions/events detection using Generative Adversarial Nets (GAN).
In anomaly action detection, the main problem is that the image data of abnormal humanactions/events is more difficult to obtain than normal human actions/events. To solve thisdifficulty, we use only normal human action data in the employed training networks.
The GAN architecture trains two deep networks which compete with each other; agenerator that performs conversion of Pixel-to-Pixel and a discriminator that supports it.
In the experiment, a single class classification using KTH dataset and anomaly detection
using Avenue dataset were performed for detecting abnormal human actions andanomaly events, respectively. Our experimental results show the effectiveness of the
proposed method in detecting abnormal human actions and events.

OS14-4 3-D Position Measurement of a Cargo Using Epipolar Geometry for
Logistics Automation(withdraw)
Kazuki Fukuda, Joo Kooi Tan (Kyushu Institute of Technology, Japan)
In recent years, with the expansion of the sales market such as online shopping, labor
shortage has become a problem in logistics industries. Logistics is roughly composed offive tasks, in which transport vehicles such as forklifts are mainly used for cargo storage
and cargo handling work. Since the operations of forklifts are still performed manually
in many factories, automating the operations is strongly requested. This problem can besolved by the acquisition of the 3-D position information of the cargo. In this paper, we
propose a method of estimating the 3-D position of a cargo from the images obtained
from (i) two fixed cameras and (ii) a single camera mounted on a transport vehicle formeasuring the relative distance between the transport vehicle and the cargo and for
position control of the vehicle. The effectiveness of the proposed method is shown by
experiments.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-72

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS14-5 Bus Line Number Detection Employing MY VISIONYe Zhou, Yosiki Hamasaki, Joo Kooi Tan (Kyushu Institute of Technology, Japan)
In response to the requests from visually impaired people that they wish higher QoL intheir daily activities, and to the facts that they find difficulty in using public
transportation, we propose we propose a bus boarding support system using MY
VISION. For the support, we detect a bus approaching a bus stop from MY VISION
images employing the characteristics of Haar-like filters, and extract the frontal upperarea displaying bus line number by random forest. After having obtained the bus line
number area, we use Contrast Limited Adaptive Histogram Equalization (CLAHE) toenhance the LED light of the area, and then recognize the line number using templatematching. The effectiveness of the proposed method is shown by experiments.

OS15 Bridging the Gap Between AI, Cognitive Science, and Narratology (7)Chair: Jumpei Ono (Aomori University, Japan)
Co-Chair: Hiroki Fukushima (Kyushu Womens’ University, Japan)
Co-Chair: Takashi Ogata (Iwate Prefectural University, Japan)
OS15-1 Analysis and Construction of Elements of the Stage Performance Structure
in a Kabuki-dance
Miku Kawai (Iwate Prefectural University, Japan),Jumpei Ono (Aomori University, Japan),
Takashi Ogata (Iwate Prefectural University, Japan)
We conducted a detailed analysis of the stage performance structure of Kyōganoko
Musume Dōjōji and reproduced it on the animation tool KOSERUBE with the music,
lyrics, and images aligned. In this paper, 11 scenes of Kyōganoko Musume Dōjōji areorganized. The actual flow is compared with the movement on the system. The goal ofthis paper is to understand the entire stage structure of Kyōganoko Musume Dōjōji indetail. In addition, this research aims to use the stage performance structural system for
an integrated narrative system that the authors are developing. We believe that thissystem will be the basis for the construction of the Kabuki stage. Further, we want toapply it not only to Kabuki but also to other genres, such as computer games andcommercials.

OS15-2 Unchiku Generation with Moving Illustration Using Kabuki Knowledge
Jumpei Ono (Aomori University, Japan),
Miku Kawai (Iwate Prefectural University, Japan),
Takashi Ogata (Iwate Prefectural University, Japan)
In this study, we combined the unchiku generation system and an animation system, andadded the animation system as a moving illustration. We have been studying the unchiku
generation system and animation system for a long time. The first system is an unchiku
generation system. Unchiku is the unchiku of deeply preserved knowledge. Unchiku inthe story is one of the rhetorical techniques of the story. The second system is an
animation system that expresses the stage performance structure. This system is basedon the Kabuki-dance Kyōganoko Musume Dōjōji. In this study, we created a prototype
and considered a new system that combines both the systems. The system provides ananimated description for the generated story. Its purpose is to provide an explanation thatdeepens the understanding of the story.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-73

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS15-3 Implementing Story Units of Japanese Folktales with Conceptual
Dictionaries
Takuya Ito (Iwate Prefectural University, Japan),Jumpei Ono (Aomori University, Tokyo)
Takashi Ogata (Iwate Prefectural University, Japan)
This paper reports the development of story units that are knowledge representing story
structure for story generation. A story structure in our narrative generation study isformed by a tree that consists of events connected based on the relationships, such as thecausal and temporal relations of events. We have been implemented story units based on
the “Type of Folktale” in the Nihon Mukashi-banashi Taisei (The Concluded
Compilation of Japanese Folktales). The current main theme is to use our conceptual
dictionaries, including verb and noun conceptual dictionaries, to combine the current
version of story units to our “integrated narrative generation system,” which is a generalframework of our narrative generation study. In this paper, we explain the above
mechanism.

OS15-4 The Story Generation Process and Cognitive Biases
Jun Nakamura (Chuo University, Japan)
When it comes to designing a product, a service, or a new technology, whatever the casemay be, author believes that in order to attract people, it is important to tell a story about
the subject matter in a narrative style, that makes people impressed with the subject
matter. In this paper, author asked several groups to create stories based on the same
given data set. As expected, a variety of stories were created for each group. Cognitive
biases were evident in the process of creating the stories. It is assumed that cognitive
bias is a sort of driving force behind the creator's intentions, then it can be interpreted
that the creator must have a certain strong intention to create a story.

OS15-5 Time in an Aesthetic Experience of a cup of Sake
Hiroki Fukushima (Kyushu Womens’ University, Japan)
There are two broad genres of artworks depending on the nature of their appreciation:
arts of space and arts of time. In the aesthetic appreciation of taste, which is the focus of
the experience of a cup of Sake? Is there the dimension of time in the experience of taste,
and if so, what is it? Because taste has never been considered a formal subject ofaesthetics, there is little discussion of describing and appreciating the aesthetic
experience. This study deals with the concept of time in the appreciation of Sake, withsome reference to the concept in narrative studies. As with wine, the time associated with
Sake is multifaceted, and each is intrinsic to the aesthetic experience. Consider someexamples: Time from the Sake is brewed to the time it is opened on the table (so-called"vintage"; for some years). The time from opening the bottle to pour into the glass (forminutes to tens of minutes). The time between pouring into the glass and entering themouth (for a few minutes). Time to swallow in the mouth (a few seconds). The flavor
felt after swallowing ("after flavor"; tens of seconds). This study focuses on the period
from the Sake is placed in the mouth to the time it is swallowed. The purpose of thisstudy is to reveal how time is narrated (explicitly or metaphorically) in the tens ofseconds of a sake tasting.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-74

Story units
about Japanesefolktales

INGS
Verb conceptual
dictionary

Narratives like
Japanese folktales

Noun conceptual
dictionary

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS15-6 Extension of Clinical/Psychological Approach Using Post-Narratology:
Possibility of application on Artificial Intelligence and Robot
Kai Seino (National Rehabilitation Center for Persons with Disabilities, Japan)
A purpose of this resarch is to examine the theories and methods of the psychologicalapproach in treatment and research from a narrative viewpoint. In addition, we discuss
the possibility of expanding collaboration with AI and Robot. This paper provides aliterature review of the narrative psychological approaches in the fields of psychology
and psychiatry. It addresses not only narrative therapy but also related therapeutic
approaches that use the narrative framework. And Psychotherapy and Psychological,
Practice and research of Self-help group, Open Dialogue, Tohjisha-Kenkyu (Selfdirected studies), etc. are outlined, and the possibility of integration to A.I. Artificial
Intelligence or application on Robots is proposed. In addition, the actual research which
aimed at integration to A.I. Artificial Intelligence is reviewed. In discussions. First,
literary narratology as basic theory is compared to psychological narratology as apractice or application theory. Second, the extension and possibility of a psychologicalnarratology are considered.

OS15-7 A relationship between narratology and marketing
Akinori Abe (Chiba university, Japan)
We have been studying an expression of the taste of Japanese sake. Actually, thedescriptions of the taste can be regarded as stories. Accoringly, we conducted anexperiment, in which we asked participants
to draw descriptions of the taste of theJapanese sakes and to design labels of them. From the results of the experiment, we will
analyse the relationship between narratology and marketing of Japanese sake.

OS16 Software Development Support Method (5)Chair: Tetsuro Katayama (University of Miyazaki, Japan)
Co-Chair: Tomohiko Takagi (Kagawa University, Japan)
OS16-1 Simulation and Regression Testing for Behavior of Software Models Based on
Extended Place/Transition Net with Attributed Tokens
Tomohiko Takagi, Ryo Kurozumi (Kagawa University, Japan)
Extended Place/transition Net with Attributed Tokens (EPNAT) is a formal specification
description language for modeling the expected behavior of state transition-based
software that consists of multiple objects. Each attributed token corresponds to an object,
and the values of its variables are updated while moving on the net. Engineers need tounderstand such complex aspects of EPNAT models when constructing, validating andrefining them. In order to address this problem, we propose a simulation and regression
testing technique for the behavior of software EPNAT models represent, and then show
an early prototype tool to support it. In the tool, fireable transitions and values ofvariables are indicated for the simulation, and also the process of the simulation isrecorded as test cases for the regression testing.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-75

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS16-2 Development of an Early Prototype Tool for Learning Software Modeling Using
Extended Place/Transition Net
Tomohiko Takagi, Akio Usuda (Kagawa University, Japan)
Extended Place/transition Net (EPN) is place/transition net that includes some additional
elements written in VDM++ to enhance its representation power, and can be used tomodel the state transition-based behavior of software in development processes. The useof EPN is based on technical knowledge and skills, and therefore engineers will need tolearn them. This paper shows an early prototype tool for learning software modelingusing EPN, and then gives the discussion about its effectiveness and problems. A user of
the tool, that is, an engineer tries to construct his/her EPN model based on given software
requirements by selecting and putting the components of EPN. The behavior of the EPN
model is partially visualized by using animated graphics, and finally its correctness is
automatically checked.

OS16-3 Test Suite Reusability Measurement Based on Frequency and Coverage of
Reused Test Cases
Mochamad Chandra Saputra*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, Naonobu Okazaki*(*University of Miyazaki, Japan) (†University of Nagasaki, Japan)
A test suite that consists of a set of test cases is an important element in software testing.
The objectives of white-box testing process are to execute all the test cases in the test
suite and also achieve 100% code coverage. One of the good criteria for test suite ispossible to reuse it for testing another program. Reusability of a test suite is one ofimportant factors to decrease the cost and time in testing the program. This researchdefines and measures the test suite reusability score expressed with the degree of
reusability of the test suite. It is calculated by considering both frequency and code
coverage of successful reused in the test suite. Test suite reusability measurement
provides useful information to improve the efficiency of software testing, especially in
regression testing and automated testing.

OS16-4 The Seven Information Features of Class for Blob and Feature Envy Smell
Detection in a Class Diagram
Bayu Priyambadha*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, Naonobu Okazaki*(*University of Miyazaki, Japan) (†University of Nagasaki, Japan)
Measuring the quality of software design artifacts is difficult due to the limitation of
information in the design phase. The class diagram is one of the design artifacts produced
during the design phase. The syntactic and semantic information in the class is important
to consider in the measurement process. The class information is used to detect the smell
as an indicator of a lack of quality. All information related to the class is used by several
classifiers to prove how informative it to be used to detect the smell. The smell types thatare a concern in this research are Blob and Feature Envy. The experiment using threeclassifiers (j48, Multi-Layer Perceptron, and Naïve Bayes) confirms that the information
can be used to detect the smell. The average of correctly detected by the classifiers isabout 80.67%.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-76

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS16-5 Improvement of RETUSS to Ensure Traceability between Sequence
Diagram in UML and Java Source Code in Real Time
Kaoru Arima*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, Naonobu Okazaki*(*University of Miyazaki, Japan) (†University of Nagasaki, Japan)
Ensuring traceability of software deliverables is one of the methods to ensure software
quality. It has two problems: taking much labor and time, and causing mistakes by humanhandling. In order to solve them, our laboratory developed RETUSS (Real-time Ensure
Traceability between UML and Source-code System). RETUSS ensures traceability
between UML and source code by transforming them to each other in real time.
However, RETUSS is not useful in ensuring traceability between sequence diagram andJava source code due to its limited scope of application. This paper improves theusefulness of RETUSS by extending the scope of application of sequence diagram andJava source code. The experimental results confirmed that the improved RETUSS also
saves labor and time, and eliminates mistakes by human handling.

OS17 Robotics and machine vision (3)Chair: Jiwu Wang (Beijing Jiaotong University, China)
Co-Chair: Junxiang Xu (Beijing Jiaotong University, China)
OS17-1 Kinematic modeling and Simulation of humanoid dual-arm robot
Jiwu Wang, Junxiang Xu (Beijing Jiaotong University, China)
In recent years, dual-arm robots have attracted more and more attention due to theiradvantages such as strong cooperation ability and high flexibility. With the improvementof real-time requirement, the inverse kinematics solution of robot becomes a key
problem to be solved urgently. In order to solve the time-consuming problem of inverse
kinematics of robot arm, a closed inverse kinematics solution algorithm for humanoiddual-arm robot was proposed. The effectiveness of the algorithm was verified bysimulation.

OS17-2 Kinematics analysis and simulation of 6R Robot Based on
MATLAB/Simulink
Jiwu Wang, Shuo Han (Beijing Jiaotong University, China)
The intelligent robot has played an important role in improving working conditions andproduction efficiency with the development of intelligent robot technology. Here therobot kinematics was studied and simulated based on the MATLAB/Simulink. The
forward and inverse kinematics equation were established based on the standard
Denavit-Hartenberg method, and eight solutions regarding angle variables of robot joints
were obtained. Then the motion trajectory of the robot was simulated in the joint space,
and the smooth curves of angular displacement, angular velocity. Furthermore, thevisualization of the robot kinematics model was realized based on the Simulink, and thekinematics simulation control system was established. The paper provides necessary
theoretical basis and correct kinematics model for the research on kinetic analysis of 6R
robots.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-77

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS17-3 Distance measurement system based on binocular stereo vision
Jiwu Wang, Xin Pei (Beijing Jiaotong University, China)
In order to solve the problem of low precision and time consuming in traditional ranging,
this paper studies key technologies such as binocular ranging principle, camera
calibration, and stereo matching.On this basis, a planar ranging system based on parallelbinocular vision is designed. Firstly, through the stereo camera calibration toolbox of
MATLAB, the internal and external parameters are obtained. Then stereo correction isused to make the two images coplanar and aligned by line and performing stereo
matching on the corrected image to obtain the parallax. Finally, it is able to select any
two target points through the mouse to obtain their spatial coordinates and calculate thedistance between the two points. Experimental results show that the method meets theprecision requirements.

OS18 Natural Computing (3)Chair: Marion Oswald (Vienna University of Technology, Austria)
Co-Chair: Yasuhiro Suzuki (Nagoya University., Japan)
OS18-1 Contribution to the Theory of Periodic Reaction of Three Bodies Systems
Yasuhiro Suzuki (Nagoya University Japan)
A chemical ecosystem is an ecosystem in which chemical substances mediate
interactions. When herbivores feed, plants analyze herbivore saliva and produce volatile
chemicals that attract herbivores' natural enemies. Natural enemies are attracted tovolatile chemicals and eliminate herbivores. This system is a trytrophic system
consisting of herbivores, carnivores and volatile chemicals. The basic equation ofmathematical ecology is the Lotoka Volterra equation, which is a bitrophic system ofherbivores and carnivores. This paper proposes the fundamental equations of chemical
ecosystems and shows their mathematical features.

OS18-2 Deep Micro Vibrotactile, DMV and its Applications
Yasuhiro Suzuki (Nagoya University, Japan)
Deep Micro Vibrotactile, DMV is infrasound. DMVs produce sound effects when mixed
with audio. Since the DMV has a transcendental low frequency, mixing it with soundsuch as music does not affect the frequency spectrum in the audible range. However,through demonstration experiments at concerts, it was confirmed that DMV changes thetactile qualities of the sound. This paper introduces the acoustic features and case studies
of DMVs

OS18-3 Inter-Induce computation and its Philosophical Foundation
Yasuhiro Suzuki (Nagoya University, Japan)
Set theory is based on the distinguishability of elements. How to recognize and identify
the world is the essence of set theory. If each element cannot be identified, all theelements are one set. So the set does not make sense. The Heart Sutra is highly rationaland can be interpreted mathematically. The mathematical interpretation of the Heart
Sutra shows the divergence of how to discriminate. Based on this world view of Heart
Sutra, we propose Inter-Induce computation, IIC as a novel calculation paradigm that
does not depend on set theory. This paper gives an overview and philosophicalfoundation of IIC.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-78

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS19 Artificial Intelligence for Embedded Systems and Robotics (4)Chair: Hakaru Tamukoh (Kyushu Institute of Technology, Japan)
Co-Chair: Sansei Hori (Kyushu Institute of Technology, Japan)
OS19-1 Convolutional Network with Sub-Networks
Ninnart Fuengfusin, Hakaru Tamukoh (Kyushu Institute of Technology, Japan)
We propose a convolutional network with sub-networks (CNSN), a convolutional neuralnetwork (CNN) that can be detached in a depth-wise direction into sub-models. Due toa conventional design of CNN which feature map shapes are varied throughout themodel, a hidden layer may not able to receive an input image without changing the last
fully-connected layer. To deal with this problem, we attach a step-down convolutional
layer as an additional layer to each sub-model. The step-down layer acts as an input layerof the sub-model which reduces input to the preferred representation to the sub-model.
To train CNSN, we treat the base- and sub-models as separable models and by using
multi-losses, a combination of losses from the base- and sub-model, we can updateweights to able to be utilized to both base- and sub-models.

OS19-2 Influence of FPGA implementation methods in High-Level Synthesis
1

Yusuke Watanabe, 2Hakaru Tamukoh (1RAFT WORK Co., Ltd, Japan, 2Kyushu Institute ofTechnology, Japan)
We explain about how the difference of implementation methods written in C++ in HighLevel Synthsis(HLS) influences on time it takes for tiny You only look once(YOLO) v2,
a real-time object detection system to infer on an FPGA. To utilize features of FPGA, weneed to implement hardware-oriented algorithms such as the weight binalization. We
primarily focus on convolution in tiny YOLO v2 network and we report execution resultson the Xilinx SDSoC development environment to know whether methods areappropriate or not.

OS19-3 A Hardware-Oriented Random Number Generation Method and A
Verification System for FPGA
Sansei Hori, Hakaru Tamukoh (Kyushu Institute of Technology, Japan)
We propose a new random number generation method for digital hardware such as an
FPGA. A random number generator is one of the important components to implement
neural networks (NNs) such as a generative model on FPGA. However, conventional
methods of random number generators (RNGs) require huge hardware resources. The
proposed method employs underflow bits obtained during calculations of NN on the
FPGA instead of the random number from the conventional RNGs to reduce the hardware
resources. In this paper, we employed the proposed method to a restricted Boltzmann
machine to evaluate the method and trained MNIST dataset on a software environment.
Furthermore, this paper proposes an FPGA infrastructure that can easily connect NNs toa host computer.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-79

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS19-4 Synthesis of realistic food dataset using generative adversarial network(GAN) based on RGB and depth images
Obada Al Aama, Hakaru Tamukoh (Kyushu Institute of Technology, Japan)
Constructing a large food dataset is time and effort consuming due to the need to cover
the feature variations of food items. Hence, a huge data is needed for training neuralnetworks for robot vision systems. Current study aims to advocate the Cycle-GAN tobuild up large food dataset based on large number of simulated images and relatively few
real captured images thus obtaining more realistic images effortlessly compared withtraditional capturing. Real RGB and depth images of variant food samples allocatedover turntable were captured in three different angles using real-sense depth camera withdifferent backgrounds. Furthermore, for simulated images, the Autodesk 3D_Maya
software was employed using the same parameters of captured real images. Results
showed that generally, realistic style transfer on simulated food objects was obtained asa result of employing Cycle-GAN. GAN proved to be an efficient tool that could
minimize imaging efforts resulting in realistic images.

OS20 Virtual Reality and Intelligent Interactions (4)Chair: R.P.C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)
Co-Chair: Yoshimasa Tokuyama (Tokyo Polytechnic University, Japan)
OS20-1 NeuroPhyllotaxis: An Interactive Application for Generative Art Based on
EEG Data
Chien-Tung Lin, R.P.C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)
Yoshimasa Tokuyama (Tokyo Polytechnic University, Japan)
Generative art is produced by procedural techniques. It has obtained a lot of attention
from the beginning of computer graphics. Many works of art are inspired by nature,among which phyllotaxis is more It is a combination of mathematics and the beauty ofnature. Not only can it be seen everywhere in nature, but also often appear in man-madeobjects, becoming part of culture or religion. This paper presents the development of aninteractive generative art application that is created from a phyllotaxis pattern by using
the user's EEG data. When people are using it, it is easy to relax the mood and achieve
the function of art therapy. We tried to use EEG data to make an interactive installation
art which creates phyllotaxis patterns and project them on the wall. Everyone has adifferent state, the generated patterns are also different from person to person, which
creates interesting interactive contents. In addition, sound can also be changed by EEG
to become dynamic and real-time contents.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-80

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS20-2 VRMAZU: VR Visualization of Mazu Temple for Passive Interaction with
Generated Sound from the ML Technique(withdraw)
Yi-Li Liang. R.P.C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)
Jen-Tun Lee (Japan Advanced Institute of Science and Technology, Japan)
Yoshimasa Tokuyama (Tokyo Polytechnic University, Japan)
Passive interaction based on virtual reality applications has the potential to identify
personalized user activities. Through personalized interaction methods to access thevirtual scene, the user gets a better sense of immersion. This paper introduces a VR
visualization of Taiwan traditional Mazu Temple for passive Interaction with generated
sound from the machine learning (ML) technique. In this VR application, user cancontrol their own situation through the feedback information of the generated soundbased on the user's location in the VR scene, so that their physical condition can reach agood state. This development that personalized interaction in VR applications is very
important to get different experiences in each time of the user's temple activities, so an
ML approach to design sound ambiance is needed. This paper also concerns the different
perspectives on understanding Taiwan traditional temple activity knowledge and userinteraction.

OS20-3 ThoughtMix: Interactive Water Color Generation and Mixing Based on
EEG Data
R.P.C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)
Yoshimasa Tokuyama (Tokyo Polytechnic University, Japan)
In modern art, new types of installations and artworks use sensor-based inputs as aninteractive method from which to create a new form of media art. Most of thoseinstallation artworks have static and pre-captured contents; they tend to be played as
dynamic, and by some concerned as interactive creations. However, most of theinstallations were not be able to capture the visitor's rich feedback which could trigger
the contents. In order to identify the relation between the cybernetic-aesthetics andvisitor's brain wave activities, we propose an electroencephalogram (EEG) data-driven
approach to generated watercolor effects and mixing application which visualizes thechanges in brain wave activities in real-time. Our proposed application transforms thebrain waves into real-time watercolor effects that are wrapped around by generating thedata about emotion, attention, meditation, and neural mechanisms with EEG data.
"ThoughtMix" creates watercolor effects as an immersive experience to influencecreativity and art therapy by actually visualizing the colors user's mind creates.

OS20-4 HaptWarp: Soft Printable and Motion Sensible Game Controller
Jen-Tun Lee, Kazunori Miyata (Japan Advanced Institute of Science and Technology, Japan)
R.P.C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan)
Many interaction techniques have been introduced for controlling virtual reality contents
and navigation. Traditional controllers need to map between two-dimensional inputs andthree-dimensional actions while interactive surfaces enable more natural approaches,
they still lack tactile feedback. We present real-virtual bend and twist, a tactile feedback
technology that delivers the twist and bends effect into a virtual environment. The
method uses two inertial measurement unit (IMU) sensors fixed on both sides on the 3D
printed soft material bar to acquire object deformation to simulate the virtual
deformation based on the changes in the angle exerted on a physical device. People cancustomize the 3D printed connection shape to apply to different virtual scenarios.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-81

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS21 Educational Application of Control Technology (4)Chair: Kazuo Kawada (Hiroshima University, Japan)
Co-Chair: Yoshihiro Ohnishi (Ehime University, Japan)
OS21-1 Development of Shock Sensitive Tiny Dummy Robot for Junior High School
Rescue Robot Challenge
Kazuo Kawada, Keisuke Iuchi, Keita Murai, Hiroyuki Y. Suzuki(Hiroshima University, Japan）
Junior High School Rescue Robot Challenge” is an annual tournament held by our groupin Hiroshima Univ. by sponsorship of a construction machinery company. The
tournament has 16 years of history and the rescue theme has been changed in each year.
The theme in 2019 was to convey injured people “tenderly” from top of half collapsedbuilding to ground, without giving any shocks. We developed shock sensitive tiny
dummy robot, controlled by M5STACK microcomputer since it rigged withaccelerometer. The dummy can remember how much shocks were given duringtransportation. The remote (wired) controlled robots developed by junior high school
student teams conveyed the dummy and evaluated the “tenderness” duringtransportation.

OS21-2 Basic Research on Parameter Tuning Skills Evaluation Based on Sensor Car
Behavior Data in Technology Education
Teruyuki Tamai (Hiroshima University / Ehime University), Yoshihiro Ohnishi (Ehime
University), Kazuo Kawada (Hiroshima University)
In this research, the evaluation of programming included in the contents of junior highschool technology education is explained. In that case, a new evaluation method using
the sensor car as a teaching material is proposed. First, the guidance of programming injunior high school technology education is explained. In addition, the evaluation methodof the programming which has been made so far is explained. Next, the new evaluation
method is explained. Quantitative data obtained by operating the sensor car used as ateaching material will be described. And how to analyze the data will be described. Next,
the method to evaluate based on the data is explained. Finally, the usefulness of theevaluation method is examined.

OS21-3 Mini Windmill Generator Kit for homework for Hiroshima Univ.
Monozukuri Junior Doctor Special Educational Program
Hiroyuki Y. Suzuki, Kazuo Kawada, Masayasu Nagamatsu (Hiroshima University, Japan）“Hiroshima Univ. Monozukuri Junior Doctor” is a special educational program foryoung (5-6 grades in elementary school and junior high school) volunteered students.
We originally prepared around 10 seminars on various topics and completed 2/3 of them.
We had to give up, however, all remained seminars because of sudden attack of COVID19. Since then, we introduced a set of homework in which students could study bythemselves. We modified commercially available “Mini Windmill Generator Kit” for the
homework, to make easy to measure generated voltages with three color LEDs and atester. Students tried to find best sizes and shapes of tail fin and windmill, using a manualof “Procedure of quantitative research” that we made. Majority of the students could findappropriate sizes and shapes of fins and windmills, through the data collection, drawing
figures and by discussion.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-82

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS21-4 Development of Teaching Materials to Learn the Efficient Use of Energy
Yoshihiro Ohnishi1, Teruyuki Tamai1, Shinnosuke Mori1 Kawada Kazuo2（1Ehime University, 2Hiroshima University, Japan）

This paper introduces the programming learning at junior high school. In this research,an experimental device using the fan which driven by EDLC was developed. Learning
activities using this device is to maximum the driving time of fan by the charged energy
on EDLC. The goal is to learn programming as a technique for efficiently using electricalenergy.

OS22 Robot Competitions and Education (5)Chair: Yasunori Takemura (Nishinippon Institute of Technology, Japan)
Co-Chair: Kazuo Ishii (Kyushu Institute of Technology, Japan)
OS22-1 Robustness Verification Against Noise of Self-localization Method Using
Omni-directional Camera for Soccer Robot
Yuehang Ma, Kaori Watanabe, Hidekazu Suzuki (Tokyo Polytechnic University, Japan)
The focus of the RoboCup competitions is the game of soccer, where the research goals
concern cooperative multi-robot and multi-agent systems in dynamic environments. In
the field of RoboCup, self-localization technique is important to estimate own position
including goal and other robot positions and to decide strategy. This paper presents a
self-localization technique using an omni-directional camera for an autonomous soccer
robot. We propose the self-localization method with white line information of soccer
field, and recognizes the robot position by optimizing the fitness function using Genetic
Algorithm. Moreover, we also verified the robustness of the proposed method against
noise through experiments.

OS22-2 Tomato-Harvesting Robot Competition: Aims and Developed Robot of 6thCompetitions
Takayuki Matsuo1, Yasunori Takemura2, Takashi Sonoda2, Shinsuke Yasukawa3, Yuya Nishida3,
Kazuo Ishii3 (1National Institute of Technology, Kitakyushu College, Japan, 2Nishinippon
Institute of Technology, Japan, 3Kyushu Institute of Technology, Japan)

Tomato is one of the important fruit vegetables and most tomatoes are produced in thegreenhouses, or large-scale farms, where the high temperature and humidity, and long
harvest age force the farmer heavy works. To develop the tomato harvesting robot, many
research issues exist such as manipulator design, end-effector design, collaborative
behavior, artificial intelligence, motor control, image processing, target recognition andso on. With an aim to promote the automation of tomato harvesting, we have organized
the tomato harvesting robot competition since 2014. In this paper, we discuss the aim
and the results of 6th tomato harvesting robot competition in 2019.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-83

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS22-3 Smart Agriculture IoT Education Course in enPiT-everi (Education
Network for Practical Information Technologies - Evolving and Empowering
Regional Industries)
Yasunori Takemura1, 2, Keiji Kamei1, 2, Atsushi Sanada1, 2, Kazuo Ishii1( Kyushu Institute of Technology, Japan, 2Nishinippon Institute of Technology, Japan)
1

Fostering human resources who can solve social issues by using Information Technology
is an important issue in Japan. The Ministry of Education, Culture, Sports, Science andTechnology (MEXT) has formed enPiT (Education Network for Practical Information
Technologies) programs a practical education network for industry-academia
collaboration and practice problem-based learning for the purpose. We have planned asmart agriculture IoT course in the program enPiT-everi (Evolving and Empowering
Regional Industries) directed by Kitakyushu university. In the smart agriculture IoT
course, the programming of IoT devices to measure the temperature of the green houseand soil, soil moisture and CO2, image processing using AI technology to detect maturityof fruits, green worms was carried out.

OS22-4 Development of a Handy Autonomous Underwater Vehicle “Kyubic”
Toshimune Matsumura, Yuuichiro Uemura, Kentaro Yanagise, Yoshiki Tanaka, Yuya Nishida,
Kazuo Ishii(Kyushu Institute of Technology, Japan)
Ocean is one of big challenging and extreme environments, and hard for human to accessdirectly. As the tool for ocean survey, Autonomous Underwater Vehicles: AUVs are
expected and developed from ‘80s. The recent rapid progress of computer andinformation technologies makes the development of AUVs easier and more practical.
We had developed a handy AUV “Kyubic” for the observation of shallow water andartificial structures. In this paper, we describe the system architecture of Kyubic and theexperimental results in Underwater Robotics Competition in Okinawa 2020.

OS22-5 A Greenhouse Project toward Smart Agriculture
Kazuo Ishii1, Yuya Nishida1, Shinsuke Yasukawa1, Kanako Shirahashi1, Yasunori Takemura2,
Takayuki Matsuo3(1Kyushu Institute of Technology, Japan, 2Nishinippon Institute of Technology, Japan, 3National
Institute of Technology, Kitakyushu College, Japan,)In the future estimation of 2050, the food demand increases 70% and the production
decreases 15% caused by global warming, and farmer population be 1/5 of current
workers. Until 2050, we need an agricultural system of twice production with the same
farmland area and 5 times effective operations. To realize the sustainable society, smart
agriculture including robot technology, AI, IoT is one of the solutions for food issues.
We have been working for a greenhouse project for evaluation of robotic, AI and IoTtechnologies. In this paper, we introduce the project and robotic applications.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-84

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS23 Advances in Field Robotics and Their Applications (7)Chair: Keisuke Watanabe (Tokai University)
Co-Chair: Kazuo Ishii (Kyushu Institute of Technology)
OS23-1 Three-dimensional Measurement Using Laser Pattern And Its Application
to Underwater Scanner
Yuya Nishida, Tomoya Shinnoki, Shinsuke Yasukawa, Kazuo Ishiii(Kyushu Institute of Technology, Japan)
Underwater infrastructure and facilities are getting increasingly degraded and needinspections for cracks, scouring, however, the inspections are not carried out because ofunderwater environment and water turbidity. As a solution for the inspections,
underwater vehicles attract attentions and expected to take precise photos of brides andtheir seabed. In this paper, to realize efficient underwater infrastructure inspection,
automatic crack detection by image processing is proposed. In first process of our
method generates enhanced image based on the absorbance from the turbidity meter andremoves background component, and then detects crack from the enhanced image byusing decision tree learning algorithm. This paper explains the algorithm of our method
and shows evaluation experiment results.

OS23-2 Motion control of a cable-restricted underwater vehicle for long-term spot
observation
Yoshiki Tanaka, Yuya Nishida, Kazuo Ishii (Kyushu Institute of Technology, Japan)
There are many marine resources in the Japanese EEZ, and recently, un-tethered
underwater vehicles, AUVs (Autonomous Underwater Vehicles) are used as the practical
tools for wide area seafloor survey. However, AUVs have the risks of their losing by
hardware or software failures, AUVs have the risks to be trapped on the seafloor and not
to be back on surface. The current trend of observation is 3D data in time series, 4 D data
sampling. Underwater station can measure for long period in a certain point, not for wide
area survey. This research aims to develop a long-term observation device with a low
operation risk and wide area. The proposed system consists of a base station for
supplying electric power and an underwater vehicle. The underwater vehicle is tethered
to the base station with a cable. In this paper, we explain the developed observation
device and report the result of motion control of the vehicle.

OS23-3 Development of current sensors for digitizing expert knowledge in fish
feeding towards sustainable aquaculture
Dominic B. Solpico, Yuya Nishida, Kazuo Ishii (Kyushu Institute of Technology, Japan)
Improving the efficiency of fish feeding contributes to achieving sustainable expansionof the aquaculture industry. However, expert knowledge on efficient feeding remains askill acquired through experience. This paper presents a new approach of digitizing suchknowledge by measuring underwater currents induced by fishes as indicator of their
behavior and appetite. Initial measurements to be made shall provide additional insightto the sensor system design. For this, a prototype current sensor suite has been
constructed to measure the current around the fish cage, especially during feeding times.
It consists of two modified flow sensors to measure the current flowing into or out of thecage at two target depths, and an IMU sensor to detect other induced water movementson the sensor mounting.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-85

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS23-4 Underwater image reconstruction using convolutional auto-encoder
Shinsuke Yasukawa, Sreeraman Srinivasa Raghura, Yuya Nishida, Kazuo Ishii(Kyushu Institute of Technology)
One of the main tasks of AUVs is to capture deep-sea images like fishes, crabs, otherliving organisms and resources for information leading to research on deep-sea
ecosystems.
Acoustic transmission are used to establish wireless underwatercommunications between the AUV and the ship. However, there are some limitations in
the communication channels due to limited bandwidth, multi-path, temperature
distribution and change in the direction of transmitting source and receiving sensor
which results in losses in data being transmitted. Initially, the captured images areenhanced to reduce the effect of light attenuation and then compressed for transmission
through acoustic modems. Only an important part of image is being transmitted throughset of data packets. The received data packets in the ship will be reconstructed to predictthe presence of living organisms. The loss in data during transmission creates a difficulty
for the operators to predict the exact information. In this research, to compensate thistransmission loss, an efficient compression and reconstruction technique using
convolutional autoencoder with minimal distortion is proposed. Finally, for evaluationof the proposed image compression technique, the quality of reconstruction of images
with and without data loss will be compared using the quality metrics signal to noise
ratio(PSNR), structural similarity index(SSIM) and perceptual quality of image.

OS23-5 Spherical Robot Transfer Problem with Minimal Total Kinetic Energy

Kenji Kimura1, Kazuo Ishii2 (1Fukuoka Daiichi High School, Japan, 2Kyushu Institute ofTechnology, Japan)

Previous spherical mobile robots were driven by two rollers with a fixed rotational axis,
which restricts the angular velocity vector of the sphere to two dimensions. Threedimensional freedom is expected to improve the rotational diversity of the sphere. This
study proposes a spherical mobile robot with a variable roller-rotational axis that allows
three-dimensional freedom of movement. Furthermore, the kinetic energy oftransporting the sphere by the roller is minimized by an optimization procedure.

OS23-6 Acoustic Impedance Measurement through the Modelling of Ultrasonic
Wave Transmission
Ryuugo Mochizuki, Yuya Nishida, Kazuo Ishii (Kyushu Institute of Technology, Japan)
In food industry, shortage of workers is one of a serious problem. Automation of foodhandling is one of critical issues. To alleviate the damage caused by food picking
operation by robotic hand, we propose non-contact acoustic impedance estimation withultrasonic wave, which is preceded before the picking for optimization of picking force
control. By assuming the correlation between hardness and acoustic impedance, theimpedance is calculated by the product of sonic velocity and density of a medium. We
derive an ultrasonic transmission model considering attenuation by ultrasonic reflection
and absorption, then, made an experiment to estimate unevenness of the impedance. As
the result, we succeeded in detecting the difference of acoustic impedance from theultrasonic reflection.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-86

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS23-7 Installation Method of In-situ Drilling Platform(withdraw)
Keisuke Watanabe (Tokai University, Japan)
Drastic cost reduction for subsea drilling is desired for scientific research and subsea
mining field exploration. The traditional drilling method needs a special drilling platform
which equips massive drilling facilities with many trained crews. Low cost in-situ
drilling platform is one of the possible alternatives and we are studying about a selfwalking jack up platform with a down the hole drill. In this paper, I focus onconsideration on installation method to reduce its installation cost. The platform issuspended from a vessel and its motions in the horizontal plane are controlled bythrusters attached to the platform. The dynamics of a suspended platform with thrusters
is formulated and a simulation program to estimate position accuracy is developed.
Through simulations the effectiveness of the method is confirmed.

OS24 Mathematical Informatics (5)Chair: Takao Ito (Hiroshima University, Japan)
Co-Chair: Makoto Sakamoto (University of Miyazaki, Japan)
OS24-1 Verification of CG character operation by brain wave discrimination

Kenji Sakoma1, Kodai Miyamoto1, Taketo Kamasaka1, Makoto Sakamoto1, Amane Takei1,
Tsutomu Ito2, Takao Ito3 (1University of Miyazaki, Japan)
2( National Institute of Technology, Ube College, Japan) (3Hiroshima University, Japan)

Since 2016, VR technology has been researched and developed all over the world andhas made significant progress. In this research, I studied whether VR space could beplayed simply by thinking in my head, and conducted research. The equipment used atthat time is an electroencephalograph, but since this equipment is generally very
expensive, we used Neurosky's Mind Wave Mobile to determine if it could be done ascheaply as possible. This time, we used the fast Fourier transform and the support vector
machine as the methods of EEG analysis. As a result, various problems were found. In
the future, we will improve this problem and try to improve it further by changing themethod of EEG analysis and other conditions.

OS24-2 Approach to tourism support by aerial video using CG

Kenji Sakoma1, Noritake Seto1, Kodai Miyamoto1, Taketo Kamasaka1, Makoto Sakamoto1,
Amane Takei1, Tsutomu Ito2, Takao Ito3 (1University of Miyazaki, Japan)
2( National Institute of Technology, Ube College, Japan) (3Hiroshima University, Japan)

In recent years, rural areas have been declining due to the concentration of work in cities
and the aging of the population. So, there are local governments that are focusing on
tourism to regain the vitality of the past. As a first step in the tourism business, we willconsider whether it is possible to increase the number of tourists by incorporating IT
technologies such as CG in videos introducing PVs and commercials. This research ispart of a joint development with Takachiho-cho.In this paper, we created a CG image ofa bear flying in the sky combined with a real Takachiho video taken with a drone.
However, various other problems have been identified, and I hope to improve them infuture research to produce even better ones.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-87

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS24-3 Method for detecting eye misalignment based on movement near the centerof the pupil
Uchida Noriyuki1,2, Kayoko Takatuka2, Hisaaki Yamaba2, Masayuki Mukunoki2, Naonobu
Okazaki2(1Kyushu University of Health and Welfare, Japan), (2University of Miyazaki, Japan)

Only ophthalmologists and orthoptists are eligible to diagnose misalignment of the eyesamid persistent understaffing. We automated the traditional eye misalignment evaluation(Cover-test) in our earlier studies and developed a technique to distinguish themisaligned eyes from its behavior. This method, however, had a problem with thevertical eye movements being affected by the eyelids and eyelashes, which prevented
thorough detection. Therefore, by observing only the movement near the center of thepupil, where it is less likely to be disturbed by the eyelids and eyelashes, we arranged analternative method to recognize anomalies. Based on this process, we achieved aninspection system and carried out a performance experiment. We confirmed that we
could observe not only horizontal motion, but also that of the vertical eye.

OS24-4 Basic research on video production for educational support by virtualtechnology
Kodai Miyamoto1, Taketo Kamasaka1, Kenji Sakoma1, Makoto Sakamoto1, Amane Takei1,
Tsutomu Ito2, Takao Ito3 (1University of Miyazaki, Japan) (2National Institute of Technology,
Ube College, Japan) (3Hiroshima University, Japan)

As a result of conducting a questionnaire about science classes to high school students
in 2016, the percentage of high school students who answered "I like science" and"Science is important" is lower than other subjects. However, more than 80% of
elementary and junior high school students said they like experiments and observations.
In addition, the 2019 smartphone penetration rate survey found that it is popular among
about 90% of students. In addition, VR technology has recently made remarkableprogress. From the above, I researched the idea that creating a simulation application
using VR technology using smartphones would change the way high school students
think about science classes. In this paper, we have developed a simulation application
for science experiments. Subjects were asked to experience the newly created app andcomplete a questionnaire. As a result, the average score is 4 out of 5 and it is not bad.
But at the same time, a problem was found. The problem was that this app was asimulation app, so there wasn't much user operability, so I wanted a little more
operability. I want to make apps in other fields while improving the problem.

OS24-5 There is a movement towards the development of hula costume CAD

Taketo Kamasaka1, Kenji Sakoma1, Kodai Miyamoto1, Makoto Sakamoto1, Amane Takei1,
Tsutomu Ito2, Takao Ito3 (1University of Miyazaki, Japan), (2National Institute of Technology,
Ube College, Japan) (3Hiroshima University, Japan)

In this paper, the collision between the pow skirt and the body is determined. We verified
each simple animation using the Euler method, the FB Euler method, and the RungeKutta method. As a result of verification, the Euler method and Runge-Kutta methodanimations showed contact with the body, movements away from the body of clothes,and violent movements, and the FB Euler method completed a simple and stableanimation. However, I encountered problems such as heavy calculations that required
manual parameter changes. In addition, issues such as research on texture mapping
methods for pow skirts were also raised. Therefore, we will proceed with future research
and aim for more realistic reproduction and practical research.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-88

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

GS abstracts
GS1 Chaos & Application (5)Chair: Masato Nagayoshi (Niigata College of Nursing, Japan)
GS1-1 A Method of Role Differentiation Using a State Space Filter
with a Waveform Changing Parameter in Multi-agent Reinforcement Learning
Masato Nagayoshi, Simon Elderton (Niigata College of Nursing, Japan), Hisashi Tamaki (Kobe
University)
Recently, there have been many studies on the multi-agent reinforcement learning(MARL) in which each autonomous agent obtains its own control rule by RL. Here, it
is considered that different agents having individuality is more effective than uniform
agents in terms of role differentiation in MARL. Therefore, we have proposed a
promoting method of role differentiation using a wave-form changing parameter in
MARL. In this paper, we confirm the effectiveness of role differentiation using a state
space filter with a waveform changing parameter through computational experiments
using "Pursuit Game" as one of multi-agent tasks.

GS1-2 The research about editing system of performance information for player
piano. - Make inferences about both handed musical composition by using DP
matching system Ryo Kinoshita, Eiji Hayashi (Kyushu Institute of Technology, Japan)
Playing the piano expressive by player piano, it is necessary to adjust the volume, length,
and timing of music. In the case of piano music, there are often 1000 or more notes inthe score of even a short piece of music. So, to edit music data manually requires not
only knowledge but also a huge amount of time and effort. Therefore, we aimed todevelop a system that, like a skilled pianist, can perform even the first musical scorebased on information related to previous skills and experience. So, we developed asystem that automatically estimates the performance expression of unedited music usingedited performance data and score data. In this paper, we described a phrase search using
DP matching and a method for selecting an optimal phrase, how to infer parameters ofnotes, evaluation of an inferred entire both-handed song.

GS1-3 Development of a LiDAR based Navigation System for Tree Harvester
Ayumu Tominaga, Ryusuke Fujisawa, Eiji Hayashi (Kyushu Institute of Technology, Japan),
Abbe Mowshowitz (The City College of New York, US)
This study deals with an autonomous moving system for the automation of felling
process by high-performance machines in the forestry. Many fatal accidents are caused
by logging operations. In this research, a navigation system was developed to realize
autonomous movement between accumulation sites and trees to be harvested in order toimprove productivity and safety. A three-dimensional map is generated by LiDAR
observation, and harvester moves autonomously to the tree as specified by the operator.
A test of the harvesting process was conducted in an experimental environment. The
evaluation was focused on the required time of autonomous movement in the process.
The result showed that the effectiveness of the system was confirmed in operations suchas row-thinning.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-89

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

GS1-4 Numerical study on a class of chaotic financial chaotic systems(withdraw)
Lei Gong, Minghan Song (Tianjin University of Science and Technology, China)
In this paper, a class of chaotic financial system is studied based on numerical simulation.
Firstly, chaotic dynamics of the chaotic financial system is briefly introduced. Then, thestability of the equilibrium points is analyzed by using the Routh Hurwitz criterion.
Finally, the influence of different parameters on the system is studied by means of phase
diagram, Lyapunov exponent diagram and bifurcation diagram, which show a theoreticalmethod to control chaotic dynamics of financial market.

GS1-5 Dynamic characteristics analysis of a multi-scroll conservative chaotic
system with sinusoidal nonlinearity
Zhonggao Chen (Tianjin University of Science and Technology, China)
In this paper, a multi-scroll conservative chaotic system with sinusoidal nonlinearity is
studied. Based on theoretical analysis and numerical analysis, such as equilibrium point
and its stability analysis, Lyapunov exponent spectrum and bifurcation diagram, thesystem is found to show some complex dynamics. In addition, NIST test and FPGA
implementation verify that the pseudo randomness and feasibility of the proposed systemis satisfactory.

GS2 Image & Signal Processing 1 (4)Chair: Tohru Kamiya (Kyushu Institute of Technology, Japan)
GS2-1 Unconstrained Measurement of Heart Rate Considering Harmonics of a
Respiratory Signal by Tactile Sensor
Kazuya Matsuo (Kyushu Institute of Technology, Japan),Toshiharu Mukai (Meijo University, Japan),
Shijie Guo (Hebei University of Technology, China)
Measurements of the sleeping state would be useful to monitor the health of a nursed
person. The sleeping state can be estimated from biological information such as respiration
rate, and heart rate. This study describes a heart rate measurement method by considering
the harmonics of a respiratory signal. Respiration and heart rates are measured by applying
frequency analysis to the time-series data of body pressure. The harmonics of a respiratory
signal serve as noises in heart rate measurement. Therefore, heart rate measurement is
improved by eliminating the effects of the harmonics. The average of the frequency errorsof heart rate measurement by our proposed method is 0.144 Hz. The results of experiments
demonstrated that our proposed method enhances the precision of heart rate measurement.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-90

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

GS2-2 Image Registration Method for Chest MDCT Images Based on 2-D Finite
Element Method
Takuji Ogimoto, Tohru Kamiya (Kyushu Institute of Technology, Japan)
Takatoshi Aoki (University of Occupational and Environmental Health, Japan)
Multi detector-row computed tomography (MDCT) device has been used for early
detection of lung cancer. However, there is concern that an increase in the burden on
doctors will be caused by improvements in CT performance. Therefore, developing acomputer aided diagnosis (CAD) reduces the burden on doctors. A temporal subtraction(TS) technique is one of the CAD systems and it is the subtraction operation of the current
image and the previous one of the same patients that emphasizes temporal changes. In the
TS image, when the position of the current image and the previous image are misaligned,
subtraction artifacts are remained. In this paper, we propose a registration method basedon the 2-D finite element method (FEM). In our proposed method, to improve the highcomputed cost that is the biggest problem of the FEM, we introduce the 2D FEM
registration. We applied this method to 31 series MDCT image sets which was obtained
previous and current from the same subjects and evaluated. As a result, full width at half
maximum (FWHM) of 28.0, artifact to lung volume ratio of 5.77% and computational timeof 140 sec were obtained.

(c)output image
Fig.4 Experimental result

GS2-3 Detection of Abnormal Shadows in Low-dose CT Images Using CNN
Hiromu Ikeda, Tohru Kamiya (Kyushu Institute of Technology, Japan)
CT imaging is a very effective way to detect abnormal shadows in the lungs. Considering
the physical burden on the patient, it should be taken at a low dose. Our approach consistsof two main elements. First, candidate abnormal shadow areas are extracted from lowdose CT images of the chest by threshold processing and filtering. Second, we implementa CNN model to classify candidate region on images. To improve identification accuracyin CNN, we devised a method to add spatial information to the input image. In this paper,
the proposed method is applied to 13 abnormal shadow images and 100 normal tissue
images, and true positive ate of 84.6%, false negative rate of 15.4% were achieved
respectively.

GS2-4 EEG signal extraction method based on HHT and CSP
Lei Gong (Tianjin University of Science and Technology, China)
In this paper, the Hilbert Huang transform (HHT) is used to extract the marginal spectrum(MS) of the four-task motor imagery of EEG signals. For the first time, a spatial filter
constructed by common spatial pattern (CSP) is used to extract the feature of signal
marginal spectrum, namely the marginal spectrum CSP feature (MSCSP). The MI EEG
Data2a provided by The University of GRAZ is verified by the experiment, and the results
show that compared with the traditional single CSP feature extraction method, the effectof MSCSP feature extraction was significantly improved. Which provide a theoretical andexperimental foundation for the application of MI based on BCI system.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-91

0.5

0.5

1
2

0

0

0.1

0.2

0.3

0.4

0.5

0.6

1
3

0

0

0.1

0.2

0.3

0.4

0.5

0.5
1
4

0

0.6

0.5

0

0.2

0.4

0.6

0.8

2
3

0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.5
0.2
2
4

0

0

0.2

0.4

0.6

0.8

3
4

0

0

0.2

0.4

0.6

0.8

1

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

GS3 Image & Signal Processing 2 (3)Chair: Keiko Sakurai (University of Miyazaki, Japan)
GS3-1 Deep-Learning Based Segmentation Algorithm for Defect Detection in
Magnetic Particle Testing
Akira Ueda, Huimin Lu, Tohru Kamiya (Kyushu Institute of Technology, Japan)
Magnetic Particle Testing (MPT), also referred to as magnetic particle inspection, is a
nondestructive examination (NDE) technique used to detect surface and slightly
subsurface flaws in most ferromagnetic materials such as iron, nickel, and cobalt, andsome of their alloys. In a bad environment, the procedure is complicated, and automationof MPT is strongly desired. To find defects in the formed magnetic powder pattern, it isrequired to be highly skilled and automation has been considered difficult. In recent years,
many defect detection methods based on deep learning have been proposed, and theeffectiveness of deep learning has been shown in the task of automatically detecting
various types of defects having different shapes and sizes. In this paper, we describe thedevelopment of deep learning based segmentation algorithm for defect detection in MPT
images. We have achieved a F2 score of 84.04% by using U-Net as the segmentation modeland by utilizing a strong backbone network and an optimal loss function.

.

GS3-2 A Method for Improving Recognition of Lying Postures Using a Measured
Signal Intensity of Respiration and Heartbeat by Flexible Tactile Sensor Sheet
Kazuya Matsuo (Kyushu Institute of Technology, Japan), Toshiharu Mukai (Meijo University,
Japan),
Shijie Guo (Hebei University of Technology, China)
Measurements of the sleeping state would be useful to monitor the health of a nursed
person. The sleeping state can be estimated from biological information such as respiration
rate, heart rate, and lying postures. This study describes a method for improving
recognition of lying postures using a measured signal intensity of respiration and heartbeat.
We can obtain respiration and heartbeat by means of using the time series data of the bodypressure measured at the suitable location determined by a lying posture. Therefore, arecognition rate of lying postures and a measured signal intensity of respiration andheartbeat have positive correlation. In the experiments, we show that recognition of lying
postures is improved by means of using a measured signal intensity of respiration andheartbeat.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-92

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

GS3-3 Basic research for the realization of online MEG using SSD
Kazuhiro Yagi, Yuta Shibahara, Lindsey Tate, Keiko Sakurai, Hiroki Tamura (University ofMiyazaki, Japan)
Neurofeedback systems have been found to be effective in the clinical rehabilitation ofparalysis. However, most systems exist only for use with EEG, which is cumbersome toapply to patients and has lower spatial resolution than MEG. Furthermore, the bestpractices for neural data feature extraction and feature selection are not well established.
The inclusion of the best performing feature extraction algorithms is critical to thedevelopment of clinical neurofeedback systems. Using simultaneously collected MEG andaccelerometer data before and during 10 spontaneous finger movements, we performed anin-depth comparison of independent components analysis (ICA) and spatio-spectral
decomposition (SSD) algorithms for their individual abilities to isolate movement-relevant
features in brain activity. Having restricted raw data to that from sensorimotor rhythm(SMR) frequencies in select MEG sensors over sensorimotor cortex, we compared ICA
and SSD components using: (1) 2D topographies, (2) activations over time, (3) andcorrelations with accelerometer data at both 0ms and 60ms time delays. SSD performed
more quickly and produced components that were more highly correlated with the
behavioral data than ICA. We will discuss these results and suggestions for application toneurofeedback systems. In particular, we will present detailed visualizations of SSD
results and discuss potential strategies and pitfalls for feature selection.

GS4 Image & Signal Processing 3 (3)Chair: Taro Shibanoki (Ibaraki University, Japan)
GS4-1 Landslide Area Detection from Synthetic Aperture Radar Images Using
Convolutional Adversarial Autoencoder and One-class SVM
Shingo Mabu, Soichiro Hirata, Takashi Kuremoto (Yamaguchi University, Japan)
In this paper, an anomaly detection system using deep learning for detecting disasterstricken (landslide) areas in satellite images is proposed. We used synthetic aperture radar(SAR) images because SAR can observe a wide area of the earth even at night or in bad
weather. However, it is difficult to obtain a large number of training images withannotations. Especially it is much more difficult to obtain disaster area images than normal
area images. To overcome this problem, we proposed an anomaly detection system that
only uses normal area images for the training. The proposed system combines a
convolutional adversarial autoencoder (CAAE) and one-class SVM, which realized good
feature extraction and anomaly detection. In the experiments, the detection ability ofnormal and abnormal areas was evaluated.

GS4-2 Anomaly Detection in Time Series Data Using Support Vector Machines
Umaporn Yokkampon1, Sakmongkon Chumkamon1, Abbe Mowshowitz2,
Eiji Hayashi1 and Ryusuke Fujisawa1 (1Kyushu Institute of Technology, Japan)(2The City College of New York, USA)

Analysis of large data sets is increasingly important in business and scientific research.
One of the challenges in such analysis stems from uncertainty in data, which can produce
anomalous results. In this paper, we propose a method of anomaly detection in time series
data using a Support Vector Machine. Three different kernels of the Support Vector
Machine are analyzed to predict anomalies in the UCR public data set. Comparison of thethree kernels shows that the defined parameter values of the RBF kernel are critical for
improving the validity and accuracy in anomaly detection. Our results show that the RBF
kernel of the Support Vector Machine can be used to advantage in detecting anomalies.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-93

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

GS4-3 Relationship Between Tactile Stimuli and Human Body Sway
Masaya Tadokoro, Taro Shibanoki (Ibaraki University, Japan)
This paper describes the relationship between tactile stimulation and human body sway.
We previously proposed a body sway mitigation system based on tactile stimulation andrevealed that simultaneous stimulation behind both auricles were significantly improve
human balance function, however, the mechanism of it were not fully discussed. In thispaper, some stimulation patterns were applied to participants and COP eccentricities
before/after stimulation was extracted. The results showed that COP values after
stimulation could be inclined to the opposite of the stimulation site. It indicates that tactile
stimuli can control human balance function.

GS5 Genetic Algorithms & Robotics 1 (3)Chair: Hideyuki Tanaka (Hiroshima University, Japan)
GS5-1 Layout decision system for multiple production lines using work-flow-line
and GA
Masato Noda1, Hidehiko Yamamoto1 (Gifu University, Japan),
Hirohumi Tsuji2, Yasuhisa Terawa2, Yoshinori Nakamura2, Masayuki Tsuchida2 (Infofarm
Corporation, Japan),
Katsuaki Yamada3, Yukiyasu Kuriyama3 (Kai Industries Corporation, Japan)

We develop the system to decide the efficient layout of assembly production line by using
Genetic Algorithm (GA). The target production line has some operators. We call thesystem as System of Production-line-layout Decision by Chameleon-code and GA(PDCG). PDCG decides the efficient layout of production line by using GA, work flowlines acquired by Chameleon code and the machine failure data. Specifically, PDCG
evaluates the layout efficiency by calculating the operator’s walking time in order to fix
the machine failures which sometimes occur on the production line. By the evaluation,
PDCG can find the good layout which has multiple production lines with one operator. We
applied PDCG to the actual razors assembly line’s layout in order to verify theperformance of PDCG. We compare the layout obtained by PDCG with the parallel layout
currently used in razors assembly factory. As a result, PDCG’s layout reduced the walking
time, compared with the parallel layout.

GS5-2 Autonomous decentralized FMS’s AGVs moving control by mind changewith deep learning
Ryunosuke Yamane, Hidehiko Yamamoto (Gifu University)
We have developed the system of autonomous decentralized flexible manufacturingsystem (AD-FMS) factory. Especially, automated guided vehicles (AGVs) in the AD-FMS
factory avoided path interferences between AGVs by inserting the mind model into theAGVs. By using the mind model, the interferences can be avoided by repeating to change
two types of mind, the arrogant mind and the modest mind. However, the problem is stillleft that it takes time to make the decision which way an AGV moves to avoid theinterference. In order to solve the problem, we develop the new mind model. The newmind model includes the deep learning system. The new model is called Minimum Unitof Mind with Deep learning (MUMD). We applied the new mind model including the deeplearning system to AD-FMS and ascertained MUMD is useful in the time of pathinterferences avoidances.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-94

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

GS5-3 Secondary School Robotics Education in Camarin High School:
Developments and Challenges for Improvement
Jeffrey Rivera Galino (Camarin High School, Philippines and Hiroshima University, Japan)
Hideyuki Tanaka (Hiroshima University, Japan)
Camarin High School is one of the schools in the Philippines that is making an initiativein improving STEM education through robotics education. In this paper, the authors report
the development of a robotics program for students in junior high school. Through the
Teacher Training Program of the Japanese Government for international teachers,
localized challenges were identified and advances in technology education were examinedfor consideration. This report is hoped to provide useful insights to both practitioners ofrobotics education and initiators in a similar situation.

GS6 Robotics 1 (4)Chair: Noritaka Sato (Nagoya Institute of Technology, Japan),
GS6-1 Digital Testing Device for Active Range of Motion of Finger Joints Utilizing
Artificial Neural Network
Huu-Hieu Quang, Yoshifumi Morita (Nagoya Institute of Technology, Japan), Makoto
Takekawa (everfine, Japan)
This paper proposed a digital goniometer device for measuring the active range of motion(A-ROM) of finger joints by artificial neural network (ANN). Although a therapist
typically measures the A-ROM by a physical goniometer, it consumes a significant amountof time, and imposes a relatively heavy burden on patients. The proposed device comprises
an Intel RealSense depth camera and a computer. The camera is set up toward the patient`sfinger joints. Subsequently, the therapist captures to collect RGB-D data. The threedimensional position coordinates of the specified joint are acquired from the RGB-D data.
These data are used as inputs to an ANN model for predicting the joint`s angles. The
effectiveness of the proposed device is confirmed experimentally on the third joint of the
middle angle.

Digital Testing device for
A-ROM of finger joints

GS6-2 Position and Force Teaching Method for 6 DoF Manipulator Using Contact
Teaching Tool and Teaching Data Editor
Duy-Do Bui, Hiroki Tanaka, Quang-Trung Chu, Hideki Inuzuka, Yoshifumi Morita(Nagoya Institute of Technology, Japan) Masao Sakai (Aichi Prefecture, Japan)
This paper proposed a method for teaching tasks performed by a six degree-of-freedom
robot manipulator while the robot’s tool makes contact with a workpiece. By using ourpreviously developed direct teaching device, the time to teach pressing angle and force ofthe tool is greatly reduced compared to using the teaching pendant. In this study, a contact
teaching tool and a teaching data editor were proposed on the assumption that thecompliance characteristics of the robot hand are known. The tool allows us to accurately
grasp the shape of the workpiece. The editor generates the target trajectory to realize thedesired pressing angle and force on the basis of the teaching data and the known
characteristics. The effectiveness was confirmed experimentally from the viewpoint of
improving the teaching accuracy and reducing the teaching time.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-95

PAWTED with a contact
teaching tool

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

GS6-3 Haptic Device that Presents Sensation Corresponding to Palm on Back ofHand for Teleoperation of Robot Hand Report2: Consideration on Decided
Specification
Kyosuke Ushimaru,Noritaka Sato,Yoshifumi Morita (Nagoya Institute of Technology, Japan)
Recently, teleoperated rescue robots are required. However, it is known that the
teleoperation of a robot hand mounted on a rescue robot is difficult. Therefore, we propose
a new haptic device that presents haptic sensation for teleoperation of a robot hand. The
device stimulates the back of the hand instead of the palm of the operator. Determinationof required specifications by an experiment with subjects is written in this paper. To design
the device, the interval of the stimulation cue (i), the diameter of the stimulation cue (d),and the force of the stimulation (f) should be optimized. As a result of the experiment, wefound that the accuracy rate is highest, when (i, d, f) = (30mm, 6mm, 0.9kgf). Moreover,
we deeply considered on the decided specification with an additional experiment.

|

GS6-4 Robot Assisting Water Serving to Disabilities by Voice Control
Yang Chunxin, Sakmongkon Chumkamon, Eiji Hayashi (Kyushu Institute of Technology,
Japan)ROS is an open-source robot operating system. In this paper, we use ROS to control Conberobot arm. By introducing the YOLACT real-time instance segmentation, we trained ourown model for Object Detection. Secondly, the Speech-Recognition system is establishedthrough Deep speech and Mozilla Text-To-Speech with Tacotron2 DDC model. Deep
speech is an end-to-end speech system, where deep learning supersedes these processing
stages. Combined with a language model, this approach achieves higher performance than
traditional methods on hard speech recognition tasks while also being much simpler. In
this way, we create an artificial intelligence, which accomplished a simple conversation
with people. And the voice control system is established based on Speech-Recognition
system. In the experiment, we successfully control the robot arm move positions and do
water serving for disabilities by voice command. With this research, voice control robot
arm can be apply in the life support area, it will be more convenient for disabilities in dailylife

GS7 Robotics 2 (4)Chair: Wisanu Jitviriya (Kyushu Institute of Technology, Japan)
GS7-1 Development of Interactive Robot – Emotion Estimation System Using
Speech by 1dCNN Yugo Kawachi, Eiji Hayashi (Kyushu Institute of Technology, Japan)
With the expansion of the market for the robotics industry, the development of service
robots is becoming more popular. These robots are designed to be used in homes and in
medical and welfare setting where they can communicate with people, and they need toact and talk in friendly. In this research, we are developing an interactive robot that pursues
human-like movements by focusing on non-verbal interactions such as facial expressions
and body language. We can guess the other person’s emotions from the intonation of
speech. In addition to the facial expressions and body language used in previous studies,
we considered that a function for emotion estimation from speech is necessary. Therefore,
we developed a machine learning system using 1dCNN for emotion estimation of speech.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-96

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

GS7-2 Deep Learning Methods for Semantic Segmentation of Dense 3D SLAM
Maps
Pei Yingjian, Sakmongkon Chumkamon, Eiji Hayashi (Kyushu Institute of Technology, Japan)
Most real-time SLAM systems can only achieve semi-dense mapping, and the robot lacks
specific knowledge of the mapping results, so it can only achieve simple positioning andobstacle avoidance, which may be used as an obstacle in the face of the target object to be
grasped, thus affecting the realization of motion planning. The use of semantic
segmentation in dense SLAM maps allows the robot to better understand the map
information, distinguish the meaning of different blocks in the map by semantic labels,
and achieve fast feature matching and Loop Closure Detection based on the relationshipbetween semantic labels in the scene. There are many semantic segmentation datasets
based on street scenes and indoor scenes available for use, and these datasets have somecommon tags. Based on these training data, we can derive a semantic segmentation model
based on RGB images by using the Pytorch platform for training.

GS7-3 Robot Motion Generation by Hand Demonstration
Sakmongkon Chumkamon, Umaporn Yokkampon, Eiji Hayashi, Ryusuke Fujisawa(Kyushu Institute of Technology, Japan)
Since traditional robot teaching requires time and instruction to the robot motion, we
present a system framework and experiment for generating robot motion trajectories fromhuman hand demonstration, which the worker could teach robot easier rather thanassigning the instruction to the robot controller manually. Therefore, the robot can imitate
the action in a new situation instead of directly teaching the robot arm. Our contributions
are including three-point 1) the extracting method of hand movement with marker-lessusing hand detection in 3D from human 2) the motion generalization of the hand
trajectories from human 3) Robot path planning for grasping and place the object to thetarget. We also present the experiment conducted by the user movement for real data andevaluate the system using the manipulator's gripper. The experiment shows the pick-andplace task of the robot for food by hand demonstration.

GS7-4 Medical Telerobotics: IRAPs SHaRE-aGIVeR
Noppadol Pudchuen, Jiraphan Inthiam, Wisanu Jitviriya, Amornphun Phunopas, Chirdpong
Deelertpaiboon(King Mongkut’s University of Technology North Bangkok, Thailand)
Aran Blattler (Kyushu Institute of Technology, Japan)
The Coronavirus disease 2019 (COVID-19) pandemic has affected the global population.
In particular, the medical personnel in direct contact with patients have been exposed tohigh risk. To reduce the spread of COVID-19 and protect healthcare workers and patients,
we would like to present the fully automated medical telerobot as the IRAPs SHaREaGIVeR robot. Our robot is capable of generating both 2D and 3D maps automatically,
delivering medical supplies, food, or medical devices such as blood pressure monitors,
pulse oximeters, and so on. In addition, the user interface system is also vital part. Users
are able to connect and control the robot using a computer, a mobile device, or a tablet via
the wireless network which is installed inside the robot. Currently, our robots are being
operated on with medical personnel at hospitals, regarding the feedback from the formal
caregivers that can prove our robot's efficiency in reducing the risk of COVID-19 spread.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-97

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

GS8 Medical Data Processing (5)Chair: Shingo Mabu (Yamaguchi University, Japan)
GS8-1 Development of in-Home Wireless Continuous Temperature Data Logging
and Alarming
System for Fever Monitoring in Pediatrics
Ali S. AlMejrad(University of Hail, Kingdom of Saudi Arabia)
Fever is one of the most common concern to be taken care by parents that sometimes force
them to bring their child to the hospital emergency for continuous follow up especially atnight when sleeping. Fever is defined when a rectal temperature reaches over 38.0 C (100.4
F). Temperatures measured from other sites of body are usually less. There is no defined
threshold for fever due to variation of body temperatures between different individuals asmuch as 1 F. Fever itself is not life threatening unless the rectal temperature is persistentover than 41.6 C (107 F). Fever may be due to a serious illness and usually is caused bycommon infections that are not serious. However, a fever can be caused by other causes
other than infections. Since continuous temperature monitoring is inapplicable at the rectal
site, the paper aims at developing a prototype system using Wireless Sensor Network(WSN) for wireless continuous fever monitoring system from the head skin or wrist thatare the most applicable area even so one must add 2 degrees for the read temperaturecorrection. The system can record the temperature at the patient terminal and transmit
the fever status and critical situations to an alarming system in the side of taking-carepeople that will help in in-home care for pediatrics suffering from fever.

Patient

Temperature
SensorData
Acquisition

Lin k fr om thebounded transmitter
node

Transmitter Node

Optional
Data Access Module

Link from thean y transmitternode

USB Interface

USB Interface

Data Logger

Web pageWeb Server

SD Card Reader

Alarm Notification SMS

System Block Diagram

GS8-2 Design and Development of in-home Wireless Crucial Events Logging andAlarming System for Elderly and Disabled People Care
Ali S. AlMejrad(University of Hail, Kingdom of Saudi Arabia)
This paper aims at developing a system that will help in in-home care for elderly or
disabled people. The system can record the patient vital parameters such as temperature,
SpO2, ECG, heart rate, bed moisture and fall off sensor which could be installed at thebathroom or shower area at the patient terminal and transmit the critical situations such as
high or low values to an alarming system. The system will be developed using the National
Instrument Wireless Sensor Network (NI-WSN) to control the crucial events transmission
and LabVIEW software to design the user interface.

System Block Diagram

GS8-3 Virtual bird’s-eye view for remote operation of unmanned construction
machinery
Noritaka Sato, Akihiro Fukuda (Nagoya Institute of Technology, Japan)
Recently, remote operation of construction machinery is required to release humans fromdangerous places. However, the operator only watches a video from a camera mounted ona construction machine in general and lacks the sense of distance. Therefore, risk ofcollisions and operation time increase. In order to solve those problems, we propose amethod to display virtual bird’s-eye view which is created by superimposing a CG modelof a construction machine on pre-captured pictures. The operator can recognize positional
relationship between the machine and the environment by using the proposed method.
Experiments are carried out to verify the effectiveness of the proposed method which can
reduce the number of collisions and operation time comparing to the conventional method.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-98

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

GS8-4 Domain Transformation of Chest CT Images Using Semi-Supervised Cycle
GAN for Opacity Classification of Diffuse Lung Diseases
Masashi Miyake1, Shingo Mabu1 , Shoji Kido2, Takashi Kuremoto1(1Yamaguchi University, Japan, 2Osaka University, Japan)

The aim of this research is to perform domain translation of chest CT images so that
medical institutions can effectively use a computer-aided diagnosis (CAD) system trained
at a different institution. We propose a semi-supervised Cycle GAN for domaintransformation by combining the standard Cycle GAN and the trained CAD. In detail, aftersome images are domain-transformed by Cycle GAN, the trained CAD classifies them to
examine whether they are appropriately transformed or not. Then, the results are fed backto the Cycle GAN to re-train. In the experiments, classification of normal opacity and fivekinds of abnormal opacities of diffuse lung diseases in CT images are carried out and theeffectiveness of the domain transformation of the proposed method is clarified.

GS8-5 Anomaly Detection of Lung Sounds Using DAGMM

Ryosuke Wakamoto1, Shingo Mabu1, Shoji Kido2, Takashi Kuremoto1(1Yamaguchi University, Japan, 2Osaka University, Japan)

There are few studies using deep learning for auscultation, and also, there are only a few
small-scale benchmark datasets of lung sounds that are annotated for the training ofmachine learning. Therefore, we aim to build an anomaly detection system that only uses
normal data that can be obtained more than abnormal data. To realize the anomaly
detection system, we propose an algorithm that improves the Deep Autoencoding Gaussian
Mixture Model (DAGMM), that simultaneously implements feature extraction andclustering. In this study, various types of neural networks are applied to DAGMM as thecompression networks for efficient feature extraction. From the experimental results, it isclarified that the proposed methods show effective classification performance for lungsounds.

GS8-6 A ground reaction force analysis in walking and running gaits in horse leg
model on viscoelastic hoof-ground contact
Dondogjamts Batbaatar, Hiroaki Wagatsuma (Kyushu Institute of Technology, Japan)
In this paper, we focused on the hoof-ground interaction in the simplified horse leg model
because walking and running gaits are known to be different in trajectories; however, theforce analysis still remains as unsolved issues. The computational experiments in Matlab,
elastic and inelastic impact with the ground was resolved by using the dissipative contact
force model and the ground reaction force was clearly examined in four different
conditions from the combination of walking/running and elastic/inelastic contact. The
proposed method contributes to the establishment of the detail time course analysis of theground reaction force for animal and human gaits in a consistent manner.

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-99

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

AUTHORS INDEX
Notation of session name
PS: Plenary Session IS: Invited Session, OS: Organized Session, GS: General Session,
Note: 33/90 = (page no. in Technical Paper Index) / (page no. in Abstracts)[A]
Abdullah

Azian

OS1-3

39/43

Azamimi

OS4-3

34/48

Chen

Chen-Pang

OS8-1

23/55

Chen

Shuailin

OS12-13

38/65

Abe

Akinori

OS15-7

26/75

Chen

Shuo-Tsung

OS9-5

22/58

AburadaKentaroOS16-3

30/76

Chen

Yue

OS12-1

37/62

OS16-4

30/76

Chen

Zhonggao

GS1-5

33/90

OS16-5

30/77

Cheng

Yuhui

OS12-6

37/64

OS13-5

36/68

Adam

Asrul

OS1-1

39/42

Ahn

Seong-Ha

OS4-1

34/48

Cheng

Yung-Hsin

OS9-7

22/59

Aizawa

Shogo

OS5-6

29/50

Chu

Quang-Trung

GS6-2

32/95

Al aamaObada

OS19-421/80

ChuangChing-Chun

OS9-4

22/58

AlMejradAli S.

GS8-1

33/98

OS9-5

22/58

GS8-2

33/98

ChuangLi-Ming

OS7-3

32/53

OS11-1

29/61

Chumkamon

Sakmongkon

GS4-2

35/93

GS6-4

33/96

AlpizarIxchelRamirez

Aoki

Takatoshi

GS2-2

24/91

GS7-2

39/97

Arima

Kaoru

OS16-5

30/77

GS7-3

40/97

Asada

Taro

OS2-2

22/44

OS2-3

22/44

OS2-4

22/45

[D]

[B]

Dahari

Mahidzal

OS1-4

39/43

Dai

FengzhiOS13-1

36/67

OS13-2

36/68

Batbaatar

Dondogjamts

GS8-6

34/99

OS13-3

36/68

Blattler

Aran

GS7-4

40/97

OS13-4

36/68

Bui

Duy-Do

GS6-2

32/95

OS13-536/68

Bulatov

Sergey

OS9-1

22/57

OS13-6

36/69

OS13-7

36/69

[C]
Chao

Shu-Tsung

OS7-3

32/53

OS13-836/69

ChoYoung Im

OS4-2

34/48

OS13-936/69©ICAROB 2021 ALife Robotics Corp. Ltd.

P-100

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

OS13-1036/70

OS13-1137/70

OS13-1237/70

OS13-13

37/71

OS13-14

37/71

[H]

Guo

Shijie

OS9-6

22/59

GS2-1

24/90

GS3-2

27/92

Deelertpaiboon

Chirdpong

GS7-4

40/97

Hamasaki

YosikiOS14-5

28/73

Deng

Jianhua

OS12-14

38/66

Han

Shuo

OS17-2

31/77

OS12-15

38/66

Han

LeixinOS12-17

38/67

Domae

Yukiyasu

OS11-1

29/61

Hanajima

Naohiko

OS6-1

30/51

Dudin

Vitaly

OS9-1

22/57

Hao

HongboOS12-7

37/64

OS12-837/64

OS12-9

38/64

[E]
Egawa

HarukaOS14-1

28/71

OS12-1238/65

Elderton

Simon

GS1-1

33/89

OS12-13

38/65

OS13-7

36/69

OS13-836/69[F]Fuengfusin

Ninnart

OS19-1

21/79

Hao

Junlong

OS12-18

38/67

Fujisawa

Ryusuke

GS1-3

33/89

HaradaKensukeOS11-1

29/61

GS4-2

35/93

OS11-2

29/62

GS7-3

40/97

OS11-3

29/62

FukudaAkihiro

GS8-3

33/98

Halin

Abdul Hafiz

OS1-3

39/43

FukudaKazukiOS14-428/72

Hayashi

Eiji

GS1-2

33/89

Fukushima

HirokiOS15-5

26/74

GS1-3

33/89

GS4-2

35/93

GS6-4

33/96

GS7-1

39/96

GS7-2

39/97

GS7-3

40/97

OS5-1

28/49

[G]
GalinoJeffrey

GS5-3

26/95

RiveraGaoLongyuOS12-16

38/66

OS12-17

38/67

OS12-18

38/67

OS5-2

28/49

OS13-14

37/71

OS5-5

29/50

Hiraoka

Toru

Ge

Jing

OS6-3

30/51

HirataSoichiro

GS4-1

35/93

Gong

Kai

OS6-2

30/51

HirotaKoichiOS10-7

24/61

OS6-4

30/52

HirotaMasaharu

OS5-3

29/49

GS1-4

33/90

OS5-4

29/50

GS2-4

25/91

Hori

SanseiOS19-3

21/79

OS10-3

24/60

Hsia

Kuo-Hsien

OS8-5

24/57

OS10-4

24/60

OS9-6

22/59

OS8-5

24/57

OS9-7

22/59

Gong

Guan

Guo

Lei

Zhe

Jr-Hung©ICAROB 2021 ALife Robotics Corp. Ltd.

Hsiao

P-101

Jia Ming

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Hu

ManqianOS12-18

38/67

Jhang

Guang-Ming

OS8-3

23/56

Hua

Chih-Chiang

OS9-4

22/58

Jhang

Yun-Syuan

OS9-2

22/57

OS9-5

22/58

Ji

Chengxu

OS13-5

36/68

OS8-1

23/55

Jia

Yuxin

OS6-2

30/51

OS8-2

23/55

OS6-4

30/52

OS9-4

22/58

OS6-2

30/51

OS6-4

30/52

OS3-2

34/45

OS3-8

35/47

Wisanu

GS7-4

40/97

TaketoOS24-1

25/87

Hung

Chung-Wen

Jia

Yingmin

Jian[I]

TianyeIbrahim

Zuwairie

OS1-1

39/42

Iiyama

Shota

OS5-4

29/50

Ikeda

Hiromu

GS2-3

25/91

Imai

ShinichiOS10-1

24/59[K]InthiamJiraphan

GS7-4

40/97

Kamasaka

Inuzuka

Hideki

GS6-2

32/95

OS24-2

25/87

Ishii

Kazuo

PS1

21/41

OS24-425/88

OS22-2

31/83

OS24-525/88

OS22-3

31/84

OS1-3

39/43

OS22-431/84

OS22-5

31/84

Kamei

Keiji

OS22-331/84

OS23-1

27/85

Kamiya

Tohru

GS2-2

24/91

OS23-2

27/85

GS2-3

25/91

OS23-3

27/85

GS3-1

26/92

OS23-4

27/86

Kang

Ho-Sun

OS4-1

34/48

OS23-5

27/86

Kang

RumingOS12-17

38/67

OS23-6

27/86

OS13-6

36/69

OS14-1

28/71

OS16-330/76

OS14-2

28/72

OS16-430/76

OS24-1

25/87

OS16-530/77

OS24-2

25/87

OS21-1

23/82

OS24-425/88

OS21-2

23/82

OS24-5

25/88

OS21-3

23/82

OS21-423/83

OS15-1

25/73

OS15-2

26/73

Ishikawa

Ito

Seiji

Takao

Jitviriya

Kamazlan

Muhammad
Zubir

Katayama

KawadaTetsuro

Kazuo

Ito

TakuyaOS15-326/74

Ito

TsutomuOS24-1

25/87

OS24-2

25/87

OS24-425/88

Kawachi

Yugo

GS7-1

39/96

OS24-525/88

Kharisova

Enzhe

OS9-1

22/57

OS21-1

23/82

Khazetdinov

Artur

OS9-1

22/57

Kido

Shoji

GS8-4

34/99

GS8-5

34/99

Iuchi

Keisuke

Kawai

[J]

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-102

Miku

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

KimuraKenji

OS23-5

27/86

OS7-4

32/54

Kinoshita

TakuyaOS10-2

24/60

Liu

Jie

OS3-5

35/46

OS10-524/60

Liu

MingfeiOS12-17

38/67

Kinoshita

Ryo

GS1-2

33/89

Liu

Qunpo

OS6-1

30/51

Kita

Yoshihiro

OS16-330/76

Liu

ZilongOS12-16

38/66

OS16-430/76

OS13-14

37/71

OS16-530/77

LiuZonghui

OS6-1

30/51

Kitagawa

Kohei

OS14-2

28/72

Lu

Huimin

GS3-1

26/92

Kobayashi

Hiroyuki

OS8-2

23/55

Lu

ShengyangOS12-1

37/62

KoyamaKeisukeOS11-2

29/62

Lu

Peng

OS12-5

37/63

KumanoMinoru

OS5-5

29/50

OS13-3

36/68

Kumar

Anirban

OS1-2

39/42

OS13-436/68

Kuremoto

Takashi

GS4-1

35/93

OS13-5

36/68

GS8-4

34/99

YongzhengOS12-16

38/66

GS8-5

34/99
Ma

Yuehang

OS22-1

31/83

Mabu

Shingo

GS4-1

35/93

GS8-4

34/99

GS8-5

34/99

Kuriyama

Yukiyasu

GS5-1

26/94

Kurozumi

RyoOS16-1

29/75

Lv

[M]

[L]
Lai

Chia-Chun

OS7-2

32/53

Lavrenov

Roman

OS9-1

22/57

Lee

Jang-Myung

OS4-1

34/48

Lee

Jen-TunOS20-2

39/81

Jen-TunOS20-4

39/81

LeeTung-Lin

OS7-4

32/54

Lee

Yu-Po

OS7-3

32/53

Li

Di

OS3-4

35/46

Li

Jung-Shian

OS7-1

32/53

OS7-2

32/53

OS7-4

32/54

Liang

Yi-Li

OS20-239/81

Lin

Chien-Tung

OS20-1

38/80

Lin

Tzu-Yuan

OS9-6

22/59

Liu

Chengzhi

OS3-3

34/46

Liu

Chuan-Gang

OS7-2

32/53

Liu

Hsiu-Hao

OS9-2

22/57

OS9-3

22/58

OS7-1

32/53

OS7-2

32/53

Liu

I-Hsien©ICAROB 2021 ALife Robotics Corp. Ltd.

Maeda

Kohei

OS5-1

28/49

Magid

Evgeni

OS9-1

22/57

Matsui

Ayumi

OS2-3

22/44

MatsumuraToshimune

OS22-431/84

MatsuoKazuya

GS2-1

24/90

GS3-2

27/92

OS22-2

31/83

OS22-531/84

MatsuoTakayukiMatsuura

Kenta

OS11-2

29/62

MiyakeMasashi

GS8-4

34/99

Miyamoto

Kodai

OS24-1

25/87

OS24-2

25/87

OS24-425/88

OS24-525/88

MiyataKazunori

OS20-4

39/81

MiyauchiShoko

OS11-3

29/62

Mochamad

SaputraOS16-330/76

RyuugoOS23-627/86

ChandraMochizuki

P-103

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

M. Khairuddin

Anis Salwa

OS1-1

39/42

Mohamed ShahNoraisyah

OS1-4

39/43

MokhtarNorrima

OS1-1

39/42

OS1-3

39/43

OS1-4

39/43

[O]
Oda

Tetsuya

Ogata

Takashi

OS5-3

29/49

OS5-4

29/50

OS15-1

25/73

Mori

Shinnosuke

OS21-423/83

OS15-2

26/73

MoritaYoshifumi

GS6-1

32/95

OS15-3

26/74

GS6-2

32/95

Ogawa

HanakoOS10-6

24/61

GS6-3

33/96

OgimotoTakuji

GS2-2

24/91

Ohnishi

YoshihiroOS21-2

23/82

OS21-423/83

OS16-330/76

MorookaKen’ichiOS11-3

29/62

Mowshowitz

Abbe

GS1-3

33/89

GS4-2

35/93

GS2-1

24/90

OS16-4

30/76

GS3-2

27/92

OS16-530/77

OS24-325/88

OS15-1

25/73

OS15-2

26/73

OS15-3

26/74

Tomohito

OS2-1

21/44

Mukai

Toshiharu

Mukunoki

MasayukiOS24-3

25/88

Murai

Keita

OS21-1

23/82

OkazakiOno

Naonobu

Jumpei[N]
M. NorM. Rudzuan

OS1-3

39/43

Nagamatsu

Masayasu

OS21-3

23/82

Nagayoshi

Masato

GS1-1

33/89

Nakamoto

Masayoshi

OS10-6

24/61

Pei

Xin

OS17-3

31/78

Nakamura

Akira

OS11-1

29/61

Pei

Yingjian

GS7-2

39/97

NakamuraJun

OS15-4

26/74

Peng

Wanlong

OS3-2

34/45

Nakamura

Yoshinori

GS5-1

26/94

OS3-8

35/47

Nakatani

Yuki

OS10-2

24/60

OS3-1

34/45

Ning

Lei

OS12-14

38/66

OS3-2

34/45

OS12-15

38/66

OS3-3

34/46

OS22-2

31/83

OS3-4

35/46

OS22-4

31/84

OS3-5

35/46

OS22-5

31/84

OS3-6

35/46

OS23-1

27/85

OS3-7

35/47

OS23-2

27/85

OS3-8

35/47

OS23-3

27/85

OS3-9

35/47

OS23-4

27/86

Phunopas

Amornphun

GS7-4

40/97

OS23-6

27/86

Priyambadha

Bayu

OS16-430/76

Pudchuen

Noppadol

GS7-4

40/97

NishidaYuya

NishiyaY.

OS10-5

24/60

Niu

Hong

OS12-2

37/63

Noda

Masato

GS5-1

26/94

©ICAROB 2021 ALife Robotics Corp. Ltd.

Ouchi

[P]

Peng

Yizhun[Q]

P-104

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Qin

Zhongxu

OS12-3

37/63

Quang

Huu-Hieu

GS6-1

32/95

[R]
RaghuraSreeraman

OS23-4

27/86

Srinivasa
Rajagopal

Rajapakse

Heshalini

R.P.C. Janaka

OS12-16

38/66

Shibahara

Yuta

GS3-3

27/93

Shibanoki

Taro

GS4-3

35/94

Shirahashi

KanakoOS22-5

31/84

Shou

Ho-Nien

OS7-5

32/54

SoharaNaoto

OS23-1

27/85

OS1-1

39/42

SolpicoDominic B.

OS23-3

27/85

OS1-2

39/42

Song

Chau-Chung

OS8-1

23/55

OS20-1

38/80

OS8-2

23/55

OS20-2

39/81

Song

Minghan

GS1-4

33/90

OS20-3

39/81

Song

Yunzhong

OS6-6

30/52

OS20-4

39/81

OS13-1

36/67[S]

SonodaTakashiOS22-2

31/83

Su

Bo

OS6-1

30/51

Sakai

Masao

GS6-2

32/95

Su

Kuan-Ming

OS7-1

32/53

Sakamoto

MakotoOS24-1

25/87

Suzuki

Hidekazu

OS22-1

31/83

OS24-2

25/87

SuzukiHiroyuki Y.

OS21-1

23/82

OS24-425/88

OS21-323/82

OS24-5

25/88

OS18-1

32/78

OS24-1

25/87

OS18-2

32/78

OS24-2

25/87

OS18-332/78

OS24-4

25/88

YosukeOS11-329/62

OS24-525/88

Masayoshi

OS2-1

21/44

SakomaKenji

Suzuki

Yasuhiro

Suzuki

Sakurai

Keiko

GS3-3

27/93

[T]

Sakurai

Miki

OS2-3

22/44

SanadaAtsushiOS22-3

31/84

OS2-2

22/44

Sano

TomoyaOS14-3

28/72

OS2-3

22/44

Sato

Noritaka

GS6-3

33/96

OS2-4

22/45

GS8-3

33/98

Tadokoro

Masaya

GS4-3

35/94

TabuseSeino

Kai

OS15-6

26/75

TajimaSho

OS11-3

29/62

Seki

Hiroaki

OS11-3

29/62

TakagiTomohiko

OS16-1

29/75

Seto

NoritakeOS24-2

25/87

OS16-2

30/76

Shakhnoza

Muksimova

OS4-2

34/48

Takaki

Ryosuke

OS5-2

28/49

Shen

Luqi

OS12-16

38/66

Takatuka

KayokoOS24-325/88

OS12-18

38/67

Takei

Amane

OS24-1

25/87

Sheng

Chang

OS13-2

36/68

OS24-2

25/87

Sheng

YuhangOS12-437/63

OS24-425/88

Shi

LiYingOS10-324/60

OS24-525/88

Shi

Songyun

OS3-1

34/45

GS6-1

32/95

©ICAROB 2021 ALife Robotics Corp. Ltd.

Takekawa

P-105

MakotoThe 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Takemura

Tamai

Yasunori

TeruyukiOS22-231/83[W]OS22-3

31/84

W. M. Mahiyidin

Wan Amirul

OS1-1

39/42

OS22-5

31/84

Wada

Mayuka

OS2-2

22/44

OS21-223/82

Wagatsuma

Hiroaki

GS8-6

34/99

OS21-4

23/82

Wakamoto

Ryosuke

GS8-5

34/99

Tamaki

Hisashi

GS1-1

33/89

Wakitani

Shin

OS10-7

24/61

Tamukoh

HakaruOS19-1

21/79

Wan

Khairunizam

OS1-1

39/42

OS19-2

21/79

OS1-3

39/43

OS19-3

21/79

Wan

WeiweiOS11-2

29/62

OS19-4

21/80

Wang

Chun-Chi

OS8-1

23/55

Wang

Chun-Chieh

OS8-3

23/56

OS8-4

23/56

OS17-1

31/77

TamuraHiroki

GS3-3

27/93

Tan

Joo Kooi

OS14-1

28/71

OS14-2

28/72

OS14-328/72

OS17-2

31/77

OS14-428/72

OS17-3

31/78

OS14-5

28/73

Wang

Mengke

OS6-1

30/51

Wang

Jiwu

TanakaHideyuki

GS5-3

26/95

Wang

Min

OS12-10

38/65

TanakaHiroki

GS6-2

32/95

Wang

Nana

OS3-7

35/47

TanakaYoshiki

OS22-431/84

WatanabeKaori

OS22-1

31/83

OS23-2

27/85

Watanabe

KeisukeOS23-7

27/87

Tate

Lindsey

GS3-3

27/93

Watanabe

TetsuyouOS11-3

29/62

TerawaYasuhisa

GS5-1

26/94

Watanabe

YusukeOS19-2

21/79

Tian

ShuhaoOS12-1

37/62

Wei

Qiang

OS12-2

37/63

Tokuyama

Yoshimasa

OS20-1

38/80

Wen

Haokang

OS12-7

37/64

OS20-2

39/81

OS12-938/64

OS20-3

39/81

OS12-10

38/65

Tominaga

Ayumu

GS1-3

33/89

OS12-11

38/65

Tsuchida

Masayuki

GS5-1

26/94

OS13-7

36/69

Tsuji

Hirohumi

GS5-1

26/94

OS13-9

36/69

Tsuji

Tokuo

OS11-3

29/62

Wu

Chi-Che

OS7-2

32/53

Wu

JiangyuOS12-17

38/67

OS13-14

37/71[U]
UchidaNoriyukiOS24-3

Ueda

Akira

GS3-1

26/92

UemuraYuuichiroOS22-431/84

Ul Husnain

Anees

OS1-4

39/43

Ushimaru

Kyosuke

GS6-3

33/96

Xiao

Usuda

Akio

OS16-2

30/76

Xiao

©ICAROB 2021 ALife Robotics Corp. Ltd.

Wu

Jun-Rong

OS8-2

23/55

Wu

Xiaowei

OS3-5

35/46

OS3-9

35/47

Chang-Sheng

OS8-5

24/57

Zhiqing

OS12-18

38/67

OS13-14

37/71

[X]

P-106

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Xu

Junxiang

OS17-1

31/77

Yoshimura

Motohide

OS5-6

29/50

Yoshitomi

Yasunari

PS2

28/41

OS2-2

22/44

[Y]
Yagi

Kazuhiro

GS3-3

27/93

OS2-3

22/44

YamabaHisaaki

OS16-3

30/76

OS2-4

22/45

OS16-430/76

OS13-936/69

OS16-530/77

OS13-1036/70

OS24-325/88

OS13-1137/70

OS12-7

37/64

OS13-7

36/69

OS12-5

37/63

Yuan

Yasheng

YamadaKatsuaki

GS5-1

26/94

YamadaTakeshi

OS10-6

24/61

Yamamoto

Hidehiko

GS5-1

26/94

GS5-2

26/94

OS10-2

24/60

OS10-324/60

OS12-6

37/64

OS10-424/60

OS13-3

36/68

OS10-524/60

OS13-436/68

OS10-7

24/61

OS6-3

30/51

OS6-5

30/52

OS3-4

35/46

Yamamoto

Toru

[Z]
Zhang

Znang

Zhang

Qianqian

Tianyi

WeicunYamaneRyunosuke

GS5-2

26/94

Yamanobe

Natsuki

OS11-1

29/61

Yamawaki

Junichiro

OS2-4

22/45

OS3-6

35/46

Yan

Yuqi

OS12-3

37/63

OS3-7

35/47

Yanagise

KentaroOS22-4

31/84

OS3-8

35/47

Yang

Bo-Jung

OS8-5

24/57

OS12-7

37/64

Yang

Chunxin

GS6-4

33/96

OS13-1

36/67

Yang

Dezhi

OS3-3

34/46

OS13-7

36/69

Yang

JialinOS12-3

37/63

OS13-12

37/70

Yang

Zhou

OS3-2

34/45

OS13-13

37/71

OS3-7

35/47

OS3-3

34/46

OS3-5

35/46

Zhang

Zhao

Zhao

YuhengJichaoJikai

Yao

Chun-Jen

OS9-4

22/58

Yasukawa

ShinsukeOS22-2

31/83

Zhao

HuailinOS12-1

37/62

OS22-5

31/84

Zhao

Lianchen

OS3-4

35/46

OS23-1

27/85

OS3-6

35/46

OS23-4

27/86

OS3-5

35/46

OS3-9

35/47

Zhao

Yuqi

Ye

Zhan-Xian

OS8-4

23/56

Yin

Di

OS13-10

36/70

Zheng

Fulin

OS12-12

38/65

OS13-11

37/70

Zheng

Qiang

OS12-11

38/65

Yin

MengqiOS13-10

36/70

Zhou

Ye

OS14-5

28/73

Yokkampon

Umaporn

GS4-2

35/93

Zhou

Yusong

OS3-9

35/47

GS7-3

40/97

Zhu

Meiyu

OS6-5

30/52

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-107

The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), on line, Japan, 2021

Zhu

Zhumanazarova

YuxuanAssiyaOS13-1036/70

OS13-11

37/70

OS4-3

34/48

©ICAROB 2021 ALife Robotics Corp. Ltd.

P-108

Tomato-Harvesting-Robot Competition Towards Smart Agriculture
Kazuo Ishii1, Takayuki Matsuo1,2, Yasunori Takemura1,3, Takashi Sonoda1,3,
Yuya Nishida1, Shinsuke Yasukawa1, Takuya Fujinaga1
1Center for Socio-Robotic Synthesis, Kyushu Institute of Technology, 2-4, Hibkinino, Wakamatsu
Kitakyushu, Fukuoka 808-0196, Japan
National Institute Technology, Kitakyushu College
2-20-1, Shii, Kokuraminami-ku, kitakyushu-shi, Fukuoka, Japan2

Department of Engineering, Nishinippon Institute of Technology, 1-11 Aratsu, Kanda,
Miyakogun, Fukuoka 800-0394, Japan3

E-mail: ishii@brain.kyutechg.ac.jp, matsuo@kct.ac.jp, takemura@nishitech.ac.jp, sonoda@nishitech.ac.jp,
y-nishida@brain.kyutech.ac.jp, s-yasukawa@brain.kyutech.ac.jp, fujinaga.takuya835@mail.kyutech.jp 1

Abstract
In agriculture, the aging and depopulation of farmers cause the shortages of farmers and manpower. Most of commercialized
robots are industrial robots for factory automation, and most of robots for the first industry, agriculture, forestry and fisheries
are still under developing. The reasons are cost-efficiency of the robotization, safety of the works using robots, difficulty ofoutdoor operations, and knowledge transfer problem from farmers to computer, etc. As one of solutions for the problems, robot
technology into the agriculture is expected to contribute to the laborsaving, improvement of production, production line
automation, and also the management toward smart-agriculture. We organize the Tomato-harvesting-robot competition to offer
the research field and welcome researchers into agricultural robotics.
Keywords: tomato harvesting, agriculture robot, smart agriculture, competition.

the robotization, safety of the works using robots, difficulty ofoutdoor operations, and knowledge transfer problem fromfarmers to computer, etc. If we can overcome these difficulties
and implement the robots into agricultural fields, robots cancontribute to the laborsaving, improvement of production,
production line automation. Also, the management ofagricultural products such as quality, quantity, and condition ofenvironment become possible and the smart-agriculture will be
realized.
For the autonomous operation of agricultural machinery,
Noguchi proposed a robot farming system [1] that consists ofintelligent robot vehicles to automate farming activities fromplanting to supplying products to consumers and described theimportance of simultaneous operations by developing a
multiple robot system. For the automation of harvesting,
Kawamura et. al. developed a mobile robot with manipulator
and proposed the tomato harvesting method using image

1. Introduction
In agriculture, a decrease in the number of farmers, an aging
population, and a shortage of successors are significant
problems. Ministry of Agriculture, Forestry and Fisheries ofJapan reported that Japanese self-sufficiency ratio for food isabout 40 percent, which is lowest level among developed
countries. For such urgent problems, smart agriculture is
expected to be an innovative method of utilizing robot
technology and information communication technology, andresearches aiming to realize smart agriculture have been carriedout such as the autonomous operation of agricultural machinery,automation of harvesting, and monitoring of the fieldenvironment and crops, etc. Most of commercialized robots arefor factory automation of automobile, and most robots for
agriculture, forestry and fisheries are under developing. The
reasons for the late social implementation are cost-efficiency of© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

1

Kazuo Ishii, Takayuki Matsuo, Yasunori Takemura, Takashi Sonoda, Yuya Nishida, Shinsuke Yasukawa, Takuya Fujinagaof the greenhouse, and pipes are also used as the rails for theplatform trucks on which the workers sit down, harvest andcarry tomatoes. The diameter of pipes is 50 mm and the
distance between pipes is 600 mm. The position of tomato isalways adjusted between 800 to 1200 mm heights as the tomato
plants are growing up. The free-style area is for the robots ofgeneral tomato fields in outdoor environment in future. In thefirst to 6th competitions, artificial grasses are put on the fieldwith the sizes of 3600 x 3600 mm instead of soils. The settingof tomato position is the same with the rail-style area.
The required specifications for robots are shown in Table 1.
The projected area of the robot on the ground is within the 800
x 800 mm square including a box for storing tomatoes, and noheight limitation. The robots should have an emergency stop
switch on the easy-to-find location of robots. As
recommendations, the weight of the robot is less than 50 kg andthe electric power of each motor is less than 70 W.
The robots are classified mainly two types, manual control
and autonomous control, and the former robots are classified
by whether the operator observes tomato directly or indirectly
using cameras mounted on robots, and by robot locomotion
whether the robot uses rail or not. Totally, the robots are
categorized into 6 types depend on operation and locomotion
method as shown in Table 2. The success points for a tomatochange depending on robot categories. the point for one tomato
of category (1) manual control, direct tomato observation, andrail-type is 2, and that of category (6) autonomous control, freetype is 16.
The senior league consists of 3 stages as shown in Fig.2. In
first stage, the basic function of the robot is evaluated whether
the robot can move, have a manipulator and an end-effector,
and recognize tomato. Single tomatoes are hanged from a bar
and the robot that can touch a tomato proceeds to the second
stage.
In the second stage, the robot harvests a tomato from atomato cluster. If the robot gives damage to tomato, get -1 point
for each damage tomato, and the damage to tomato plant is big
minus points. And the target tomatoes should be redder than thereference tomato indicated by the committee. In the final stage,
the robots harvest tomato from plant body as shown in Fig. 3,
for the 1st competition.

processing and visual feedback [2][3]. Kondo et. al. proposed
the method to improve the success ratio of tomato harvestingand speed-up technique [4][5]. Ota et. al. proposed the path
planning method to pick up tomato [6]. Yaguchi et al.
developed an end-effector that grasped a fruit using grippers
and plucked it from the separation layer in the peduncle [7].
They described the results of harvesting experiments conducted
on an actual farm, demonstrating that the harvesting time wasshortened by improving the harvesting motion [8]. Yoshida etal. developed a robot that har- vested a cluster of cherry
tomatoes [9][10]. They focused on detecting the peduncles thatthe robot required when harvesting the clusters. For themonitoring of the field environment and crops, Fukatsu et al.
described a remote monitoring method that responded flexibly
and dynamically to changes in the field environment for longterm field monitoring [11] which consists of field servers, smallmonitoring sensor nodes. Fukui et al. developed a robot that
estimated the volume of tomato fruits to create a database offruit growth [12] by detecting fruits based on a human attention
model, saliency [13].
Tomato is one of important fruit vegetables and most
tomatoes are produced in the greenhouses, or large-scale farms,
where the high temperature and humidity, and long harvest age
force the farmers heavy works. With an aim to promote theautomation of tomato harvesting, we have organized the tomato
harvesting robot competition. In this paper, we introduce theTomato-Harvesting-Robot Competition.
2. Tomato-Harvesting-Robot Competition
The Tomato-Harvesting-Robot competition started in 2014
and 7th competition was held in 2020, which consists of two
leagues, the Senior League and the Junior League. The target
competitors for Senior League are supposed to be the teams
with automated robots, and the Junior League are for highschool or junior high school students, whom are asked to build
robots with LEGO Mindstorms.
2.1 Senior League
The Senior League supposes that teams composed ofundergraduate, graduate students or developers join the league
with tomato harvesting robots, and compete the accuracy andspeed of harvesting. The two kinds of competition field aredesigned, the one is the rail-style area and another is the freestyle area. The rail-style area is designed to have the similar
environment with the large-scale tomato factory, where pairsof pipes are arranged on the ground to control the temperature

Table 1 Required Specifications for Robots© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

2

Tomato-Harvesting-Robot Competition Towards Smart

Table 2 Robot Classification by Operation Method

Fig. 2 The overview of the final stages of the senior league inthe 1st competition
2.1. Ju 2.2 Junior League
Figure 3 shows the concept of Junior League. We set the
subjects to make the model of an autonomous transport systemof tomatoes harvesting, such as the transportation and sortingof harvested tomatoes. Competition subjects are shown below.
(1) Line Trace
The robot moves along the white course line using the color
sensor. In the harvest field, Tomato boxes are arranged. The
robot can find three RGB colored tomato boxes if moves along
the line.
(2) Color Identification
The robot explores the area and recognize color signs alongcourse line, and find three RGB colored tomatoes in boxes(tomato box).
(3) Mechanism Design and Control
To pick up the tomato boxes, participants are expected todesign and make a robot arm to get a tomato box. The robot isrequired to store, transport and relocate the boxes to theassigned storage area. After picking up the box, the robot is toreturn to the course line. Then, the robot carries the box to thespecific storage location.
(4) Object detection
After putting the tomato boxes to the storage locations, therobot should return to the course line. Finally, the robot goesthe charging station which is put at the position with a certain
distance away from the course line. The robot must find thestop position by the distance measuring sensor.

(A) FIRST STAGE: PICK UP SINGLE TOMATO(B) SECOND STAGE: PICK UP A TOMATO FROM TOMATO CLUSTER

Each team consists of 3 or 4 students and makes a robot
using LEGO Mindstorms. The basic specification for robot isthat the sizes of the robot are within 300 x 300 [mm] on the
ground. The robot can have structural change after starting, sothat the initial robot structure is regarded as the robot size.
(C) FINAL STAGE: PICK UP A TOMATO FROM TOMATO PLANT

Fig. 1 The three stages of the senior league.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

3

Kazuo Ishii, Takayuki Matsuo, Yasunori Takemura, Takashi Sonoda, Yuya Nishida, Shinsuke Yasukawa, Takuya Fujinaga

grows up to harvest tomatoes autonomously in spite of thelighting condition changes. Moreover, some journal papers are
published based on the results of the competitions such as[7][8][14]15].
Table 3 History of regulation changes

Fig. 3 The concept of Junior League
3.

Regulation Changes toward Smart Agriculture

The regulation of the competition is changed by evaluating
the achievement of teams toward actual greenhouse toward
smart agriculture.
Table 3 shows the history of regulation changes. The firstcompetition was held in 2014 with Senior League. From thesecond competition, Junior League started as the outreachactivity of agricultural robotics. From the third competition, abump was put on the free-style area to motivate theimprovement of mobility for rough terrain. In the fifthcompetition, the black board shown in Fig.2, which is placed
for making the image processing easy, was removed and therobot are asked to find tomatoes in various colors. From the 7thcompetition in 2020, the working field moved to an
experimental greenhouse, so that the robot need the color
consistency algorithm against sunlight environment. Juniorleague was held as in each school via internet.
The winner teams are shown in Table 4. The first winnerrobot from team JSK, the Univ. of Tokyo (Fig.4) joined thecategory (T-4) and the robot was operated by human in the
remote place and succeeded to have 5 tomatoes in 10 min. 2
teams joined to autonomous categories could not harvest atomato. In the 6th competition, the robot of Kyutech teamwhose category is T-5 autonomous robot for the rail-style field(Fig.5), succeeded to harvest 13 tomatoes in 10 min in the 2ndstage and 5 tomatoes in the final stage. The teams selected forthe final stage of 7th competition, all teams entries for the
category T-5, autonomous robot in the rail-style field. The
winner is the team of National Institute Technology,
Kitakyushu College (Fig.6) succeeded to take about 20
tomatoes in the greenhouse. The performance of each team

Table 4 Results of competition and Winner teams

Fig.4 The winner robot of the 1st competition, JSK team from
Univ. of Tokyo.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

4

Tomato-Harvesting-Robot Competition Towards Smart

4. Kondo, et.al., A Machine Vision System for Tomato Cluster
Harvesting Robot, Engineering in Agriculture, Environment andFood, Vol. 2, No. 2, pp.60-64, 2009.
5. Kondo, et.al, Development of an End-Effector for a Tomato
Cluster Harvesting Robot, Engineering in Agriculture,
Environment and Food, Vol. 3, No. 1, pp.20-24, 2010.
6. Ota et. al., Development of a Tomato Harvesting Robot with a
Vision System Using Specular Reflection (Part 1), the journal of
JASM, vol. 72 No.6, pp.595-603, 2010.
7. H. Yaguchi, K. Nagahama, T. Hasegawa, and M. Inaba,“Development of An Autonomous Tomato Harvesting Robot
with Rotational Plucking Gripper, 2016 IEEE/RSJ Int. Conf. onIntelligent Robots and Systems, pp. 652-657, 2016.
8. H. Yaguchi, T. Hasegawa, K. Nagahama, M. Inaba, A Researchof Construction Method for Autonomous Tomato Harvesting
Robot focusing on Harvesting Device and Visual Recognition, J.
Robot. Soc. Jpn., Vol. 36, No.10, pp. 693–702, (in Japanese withEnglish summary), 2018.
9. T. Yoshida, T. Fukao, and T. Hasegawa, Fast Detection ofTomato Peduncle Using Point Cloud with a Harvesting Robot, J.
Robot. Mechatron., Vol.30, No.2, pp. 180-186, 2018.
10. T. Yoshida, T. Fukao, and T. Hasegawa, Cutting Point Detection
Using a Robot with Point Clouds for Tomato Harvesting, J.
Robot. Mechatron., Vol.32, No.2, pp. 437-445, 2020.
11. T. Fukatsu and M. Hirafuji, Field Monitoring Using SensorNodes with a Web Server, J. Robot. Mechatron., Vol.17, No.2,
pp. 164- 172, 2005.
12. R. Fukui, K. Kawae, and S. Warisawa, Development of a Tomato
Volume Estimating Robot that Autonomously Searches an
Appropriate Measurement Position – Basic Feasibility Study
Using a Tomato Bed Mock-Up –, J. Robot. Mechatron., Vol.30,
No.2, pp. 173-179, 2018.
13. L. Itti et al., A model of saliency-based visual attention for rapid
scene analysis, IEEE Trans. on Pattern Analysis and MachineIntelligence, Vol.20, No.11, pp. 1254-1259, 1998.
14. Takuya Fujinaga et al., Image Mosaicing Using Multi-Modal
Images for Generation of Tomato Growth State Map, J. Robot.
Mechatron., Vol.30, No.2, pp. 187-197, 2018.
15. Takuya Fujinaga, et al., Tomato Growth State Map for theAutomation of Monitoring and Harvesting, J. Robot. Mechatron.,
Vol.32, No.6, pp. 1279-1291, 2020.

Fig. 5 The winner robot of the 6th competition, Kyutech team.
The robot harvested 5 tomatoes in the final stage.

Fig.6 The winner robot of the 7th competition, National
Institute of Technology, Kitakyushu College. The robot
harvested 20 tomatoes in the experimental greenhouse.
4.

Summary

We have organized the Tomato-Harvesting-Robotcompetition to promote agricultural robots and provide the test
field for tomato harvesting. The performance of robots are
growing up gradually and the regulation of the competition isrevised to be closed to the actual working environment.
Currently, the winner robot can harvest a tomato within 30
seconds in the sunlight condition. In the next, we expect to haverobots working in the soil field and in the night time.
Acknowledgements
The Tomato-Harvesting-Robot competition is supported by FAIS, Kitakyushucity, Hibikinada Green Farm, and YASKAWA Electric Corporation.

References

1. N. Noguchi, “Agricultural Vehicle Robot,” J. Robot. Mechatron.,
Vol.30, No.2, pp. 165-172, 2018.
2. Kawamura et. al.，Study on agricultural robot (1st report), The
journal of JASM , vol. 46 No.3,pp.353-358, 1984 (in Japanese).
3. Kawamura et. al.，Study on agricultural robot (2nd report), The
journal of JASM，vol.47 No. 2, pp.177-182, 1985 (in Japanese).

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

5

Convolutional Network with Sub-Networks
Ninnart Fuengfusin

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196, Japan
Hakaru Tamukoh

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: fuengfusin.ninnart553@mail.kyutech.jp, tamukoh@brain.kyutech.jp
https://www.brain.kyutech.ac.jp/~tamukoh/

Abstract
We propose a convolutional network with sub-networks (CNSN), a convolutional neural network (CNN) that can be
detached into sub-models on fly. Due to a conventional design of CNN, shapes of feature map are varied throughout
the model. Therefore, the hidden layer within CNN may not directly process the input image without any
modifications. To address this problem, we propose a step-down convolutional layer, a convolutional layer whichacts as an input layer for the sub-model. The step-down convolutional layer reshapes the input image to a preferred
representation to the sub-model. To train CNSN, we treat the base-model and sub-models as different models. We
separately forward- and back-propagate each model. By using multi-model loss, a linear combination of losses frombase- and sub-models, we can update weights that can be utilized in both base- and sub-models.
Keywords: Convolutional Neural Networks, Supervised Learning, Model Compression.

introduces a layer-wise detach ability to MLP. By
detaching weight layers, this allows NSN to operate withless amount of MAC, therefore it decreases an overalllatency of model with the trade-off in the model
performance. However, to apply the methods in NSN to
CNN, there are three compatibility issues. The first issue
is NSN requires a targeted model to contain a same shapeof feature map throughout the model. This creates amajor constraint to the CNN model. The CNN cannot
contain any spatial reduction layers, for example poolinglayers or convolutional layers without padding. Without
any spatial reduction layers, this may lead the CNN tooperate with higher MAC than the layer detaching canreduce. The second issue is NSN requires to train N+1
models in parallel, where N is a number of hidden layers.
This indicates that NSN method is not feasible with therecent CNN models that contains more than hundred
hidden layers. The last issue is NSN require to manuallycontrol the gradients across base- and sub-models. This
causes the high complexity in programming.

1. Introduction
In recent years, a convolutional neural network (CNN)
has been achieved state-of-art results in varied tasks froman image recognition1, sematic segmentation2, and object
detection3 tasks. However, when comparing the CNN
with a multilayer perceptron (MLP) with same amount ofweight parameters, the down-side of CNN is it
significantly utilizes more multiply–accumulate
operations (MAC) than MLP. This may lead into abottleneck when deploying a CNN with a mobile device
that has lower computational capacity. To address thisproblem, there are a several research directions attempt
to address this problem from a structure pruning4,
network sliming5, and early-exit network6.
In this research, we focus on the network sliming
direction. The network sliming focuses on create a neuralnetwork model that can be detached into smaller models
during inference. In our previous work in this direction,
we purposed Network with Sub-Networks7 (NSN) that

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

6

Ninnart Fuengfusin, Hakaru Tamukoh

In this research, we propose CNSN to address all
mentioned issues from our previous work. CNSN allows
the usage of different shapes of feature map by attachinga step-down convolutional layer as the input layer of submodels. This layer pre-processes the input image to apreferred representation to the sub-model. To allow
CNSN to operate more depth CNN models, CNSN
reduces a number of sub-models by allowing only few
selected sub-models to be detached with. Instead ofmanually control the gradients, we introduce a multimodel loss, the combination of losses from a base-modeland sub-models. This loss allows the base model to beable to detached into sub-models during inference andreduces overall complexity to the user comparing withNSN.
By detach convolutional layers on demand, CNSN
promises the greater reduction in MAC than NSN which
can detach only fully connected layers. Our main
contributions in this research are listed as follows:• We propose CNSN, a CNN with ability to detach abase-model into sub models.
• We introduce multi-model loss, a combination oflosses from base-model and sub-models.
• We introduce a step-down convolutional layer thatacts as the input layer to sub-models.

Fig. 1. Overview of CNSN. The top row represents a base modelof CNSN while two bottom rows represent the sub-models. Eachmodel can receive the image and produce the prediction. During
training, losses from all model are collected as multi-model loss.

2.1. Step-down Convolutional Layer
The step-down convolutional layer is a convolutional
layer that is designed to solve the spatial difference
between the input and hidden layer in the CNSN. By
solving this problem, this allows CNSN to reuse a hidden
convolutional layer of base model as an input layer of sub
model. However, to achieve this, there are some concerns
as follows.
The first problem is the number of input filters of
hidden layer may not be a same comparing to the input
layer. With this issue, the hidden layer cannot process the
input image. To solve this problem, we can place thestep-down convolutional layer with the same amount of
input filters to the input layer. Another issue is the
expected size of the input feature is difference between
base- to sub-models. By adjusting the number of stride
and size of kernel of step-down convolutional layer, we
can adjust the shape of output feature maps to be processfurther in the sub-model.

2. Convolutional Network with Sub-Networks
CNSN consists of a base-model and several sub-models.
The sub-model is a subset of the base-model with an
exception, step-down convolutional layer that is notincluded in the base-model. Therefore, all parametersfrom sub-models except the step-down convolutional
layer is identical to the base-model. In term of memory
usage, we only required to store the base-model andseveral step-down convolutional layers.
Overview of CNSN is illustrated in Fig. 1. Fig. 1
illustrates a base-model at the top row and two submodels at the two bottom rows. Each base- and submodels can directly receive the input image andindependently product the prediction to other models.
With less amount of parameters in the model, the submodel has less capacity comparing to the base-model.
However, with the same reason, it promises to performthe inference faster than the base-model.
With both base- and sub-models, CNSN consists ofthe three components to make CNSN operate-able with
CNN. These three components are as follows: step-down
convolutional layer, sub-models, and multi-losses.

2.2. Sub-models
Instead of allowing every layer to be detach-able as in
NSN, CNSN only allows few sub-models to be detached.
In each batch trainings, both NSN and CNSN requires
base- and sub-models to be propagated in the same
iteration. This causes the problem that is parameters frombase- and sub-models may not be fitted in a single GPU
if the base-model is large and the base-model contains
many sub-models. Therefore, to able to train on the
bigger CNN with CNSN, we reduce the number of detachability to few sub-models instead.
2.3. Multi-model loss
To allow both base-model and sub-models to operate as
an individual model, we purpose a method called multi-© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

7

Convolutional Network with Sub-Networks

for 50, 100 and 150 epoch. We warmed-up the learning
rate for an epoch and trained for totally 300 epochs using
the training batch size as 32. We reported the best test
accuracy that occurred during the training.
The experimental results are as shown as Table 1. In
Table 1, we compared the CNSN base model with thebaseline model with the same setting as CNSN except for
the weight decay. We found out the optimized weightdecays are differed between the baseline model.
and CNSN. Therefore, we applied the different weightdecay to each model.

model loss. The multi-model loss is designed to balance
between the loss from base-model and sub-models. By
assume there are N sub-models in the base-model, themulti-model loss or 𝑙𝑙𝑎𝑎𝑎𝑎𝑎𝑎 is formulated as shown in Eq. (1).
Where 𝑙𝑙𝑁𝑁 is the loss from a N sub-model, and 𝑙𝑙𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 is aloss from the base-model. To utilize this loss, all models
must be forward propagated in the same batch training
first to find 𝑙𝑙𝑁𝑁 and 𝑙𝑙𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏 . Then, we can use 𝑙𝑙𝑁𝑁 and 𝑙𝑙𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏
to find 𝑙𝑙𝑎𝑎𝑎𝑎𝑎𝑎 .
𝑁𝑁

𝑙𝑙𝑎𝑎𝑎𝑎𝑎𝑎 = � 𝑙𝑙𝑁𝑁 + 𝑙𝑙𝑏𝑏𝑏𝑏𝑏𝑏𝑏𝑏(1)

1

Table 1. Results of CNSN models comparing a base-line
model on CIFAR10 dataset.

3. Experimental results and Discussion
We implemented our CNSN using VGG-168-like modelas the base-model. Since VGG-16 is designed for
ImageNet Large Scale Visual Recognition Challenge9(ILSVRC) 2014, some layers in VGG-16 are required tobe modified to able to operate with images from
CIFAR1010. We removed first two fully-connected layers
and modified the last fully-connected layer to contain 512
input neurons and ten output neurons. To stabilize thetraining process, we also attach a batch normalization11(BN) layers after each weight layers except on the last
fully-connected layer. We utilized ReLU as an activation
function except the last fully-connected layer that we
assigned with the log-softmax instead.
In this experiment, the overall setting is illustrated inFig. 1. We assigned with two sub-models within the basemodel. Without including any of activation, BN, andpooling layer, the first sub-model or sub-model0 is thebase-model after removing first two convolutional layers.
The second sub-model, sub-model1 is the base-model
after removing first seven convolutional layers. The stepdown convolutional layer for sub-model0 is the
convolutional layer with stride two and kernel size two.
For sub-model1, it is the convolutional layer with stride
eight and kernel size eight. We defined the baselinemodel as a VGG-16-like model that was trained without
any modifications.
We conducted a benchmark with CIFAR10 dataset.
CIFAR10 consists of 10 different classes. Each classconsists of 5,000 training and 1,000 test images. We
performed the data augmentations by padding 4 pixels
into the training image and randomly crop back tooriginal size. The training images were further
augmented by horizontal flipping and normalizing with achannel-wise mean and standard deviation of CIFAR10
dataset. We trained all models with stochastic gradientdescent with momentum of 0.9. We set an initial learning
rate as 10−2 and step-down to one-tenth after we trained

sub-model0
sub-model1
base-model
baseline
model

Test
accuracy
0.77380.91150.9267MAC

Weight decay

0.124 G
0.275 G
0.314 G

6 × 10−4

0.93160.314 G

6 × 10−3

Our base-model achieved the loss in test accuracy for
0.0049 in exchanging to the ability to detach into submodels. The sub-models1 able to reduces more than halfof MAC comparing with the baseline model, however
this came with the significantly drop in term of test
accuracy.
4. Conclusion
We propose CNSN, a CNN that can be detached into
smaller CNNs on fly. To gain the detach ability to CNN,
we propose the multi-model loss and step-down
convolutional layer. The base-model of CNSN candeliver the performance that is compare-able with theregular trained models, while sub-models significantly
reduces the amount of MAC, however with the trade-off
in loss in test accuracy.
Acknowledgements
This research was supported by JSPS KAKENHI
Grant Numbers 17K20010.
References
1. Tan, Mingxing, and Q. V. Le. "Efficientnet: Rethinking
model scaling for convolutional neural networks." arXiv
preprint arXiv:1905.11946 (2019).
2. Chen, Liang-Chieh, Yukun Zhu, George Papandreou,
Florian Schroff, and Hartwig Adam. "Encoder-decoderwith atrous separable convolution for semantic image
segmentation." In Proceedings of the European
conference on computer vision (ECCV), pp. 801-818. 2018.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

8

Ninnart Fuengfusin, Hakaru Tamukoh

3. Tan, Mingxing, Ruoming Pang, and Quoc V. Le.
"Efficientdet: Scalable and efficient object detection."
In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pp. 10781-10790. 2020.
4. Li, Hao, Asim Kadav, Igor Durdanovic, Hanan Samet, andHans Peter Graf. "Pruning filters for efficientconvnets." arXiv preprint arXiv:1608.08710, 2016.
5. Yu, Jiahui, Linjie Yang, Ning Xu, Jianchao Yang, andThomas Huang. "Slimmable neural networks." arXiv
preprint arXiv:1812.08928, 2018.
6. Teerapittayanon, Surat, Bradley McDanel, and HsiangTsung Kung. "Branchynet: Fast inference via early exitingfrom deep neural networks." In 2016 23rd International
Conference on Pattern Recognition (ICPR), pp. 24642469. IEEE, 2016.
7. Fuengfusin, Ninnart, and Hakaru Tamukoh. “Networkwith Sub-Networks.” Proceedings of International
Conference on Artificial Life and Robotics, vol. 25, 2020,
pp. 191–194.
8. Simonyan, Karen, and Andrew Zisserman. "Very deep
convolutional
networks
forlarge-scale
image
recognition." arXiv preprint arXiv:1409.1556, 2014.
9. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S.,
Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M.
and Berg, A.C., 2015. Imagenet large scale visual
recognition challenge. International journal of computer
vision, 115(3), pp.211-252.
10. Krizhevsky, Alex, Vinod Nair, and Geoffrey Hinton. "Thecifar-10
dataset."online:http://www.cs.toronto.
edu/kriz/cifar. html 55 (2014).
11. Ioffe, Sergey, and Christian Szegedy. "Batch
normalization: Accelerating deep network training by
reducing internal covariate shift." arXiv preprint
arXiv:1502.03167, 2015.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

9

Influence of FPGA implementation methods in High-Level Synthesis
Yusuke Watanabe

CRAFT WORK Co.,Ltd,
5F OS Bldg 3-5-15 Shibasaki-cho, Tachikawa, Tokyo, 190-0023, Japan
Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196, Japan
Hakaru Tamukoh

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: watanabe.yusuke898@mail.kyutech.jp, tamukoh@brain.kyutech.jp
http://www.lsse.kyutech.ac.jp/english/

Abstract
We explain about how the difference of implementation methods written in C++ in High-Level Synthesis (HLS)
influences on latency for tiny You only look once (YOLO) v2, a real-time object detection system to infer on an
FPGA. To utilize features of FPGA, we need to implement hardware-oriented algorithms such as the weightbinalization. We primarily focus on convolution in tiny YOLO v2 network and we report execution results on the
Xilinx SDSoC development environment to know whether methods are appropriate or not.
Keywords: Convolution, FPGA, High-Level Synthesis, Hardware-Oriented Algorithm

In conclusion, when we use HLS, hardware-oriented
implementation is preferable and brings dramatically
improved hardware.

1. Introduction
When a robot which power supply is generally limited
detects objects, a deep neural network is often used.
FPGAs are good choices to implement neural networks
while reducing energy consumption. To reduce
implementation time in FPGAs, High-Level Synthesis(HLS) has been used recently. It automatically creates
digital hardware from C++ source codes and we don't
need to use time-consuming hardware description
languages. But typical C++ implementation is nothardware-oriented and therefore created hardware don't
bring results as we would think.
In this paper, we experiment with some convolution
operations in an object detection algorithm to utilize
FPGAs better in HLS.

2. Method
We explain about our implementation methods ofconvolution operations which occupy much ofprocessing latency in object detection. As an object
detection system, we use binarized tiny YOLO v2 whichbinarizes original tiny YOLO v2 input and weights dataaccording to Ref. [1]. Convolution operations are
executed in three dimensions of height, width and input
channels. Due to input and weights data binarization, we
can use bitwise operations which reduce FPGA resources
utilization and latency and therefore can be regarded as
hardware-oriented.
Eventually we use them as convolution operations andexperiment with the following four methods.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

10

Yusuke Watanabe, Hakaru Tamukoh(1) no bitwise operation(2) width dimension(3) channel dimension(4) mix of width and channel dimension
The first method doesn’t use bitwise operations. The
second one uses bitwise operations along the width
dimension and the third one along the channel. The last
one is a mix of the second and the third. We apply bitwise
operations to the dimension which has the largest value
among height, width and input channel. We don't try
along a height dimension because height values arealways smaller than width ones.

mix ofwidth andchanneldimension
Available

5. Conclusion

Table 1. Latency comparison.
min2,546,563,540

max
3,072,073,924

471,558,114

4,811,180,162

159,132,162

3,695,079,650

115,836,258

4,634,795,522

Table 2. Resources usage comparison
BRAM

DSP

no bitwise op.
width
bitwise dimension
op.
channeldimension

1,364

181,935 79,569
548,160 274,080

As for latency, if we use bitwise operations, the min clock
cycles become lower and the actual execution time on
the FPGA board also becomes shorter although max
clock cycles become higher. This fact means that mostprocessing doesn't take the max clock cycles paths.
Further if we apply bitwise operations to the mix of widthand channel dimension as shown in Table 1, clock cycles
reduction becomes higher.
About resources, if we use bitwise operations, we cansubstantially reduce BRAM usage. If a network becomesdeeper, we use more BRAM and BRAM shortage islikely to happen. It is important for BRAM not to get
wasted.

To experiment, we prepare for a specific input RGB
image file which width and height are 384 and 288 pixels
respectively. HLS was executed on 2018.3 release of
Xilinx SDSoC Development Environment. Our
targetFPGA board is a Zynq UltraScale+ MPSoC
ZCU102 evaluation kit. We used automatically createdsynthesis reports to evaluate the performance of our
implementation.
Table 1 shows the clock cycles to produce output.
Differences between min and max come from thepresence of conditional branches in source codes. Table
2 shows resources used to implement the binarized tiny
YOLO v2 design in percentage.

method15
2,520

4. Results

3. Experiment

methodno bitwise op.
width
dimensionchanneldimension
bitwise op.
mix ofwidth andchanneldimension

328
1,824

FF

LUT

67

31,35554,777328

22

22,38842,965328

10274,116

75,125

We conclude that hardware-oriented algorithms likebitwise operations in this paper are essential even when
we design hardware from C++ using HLS. This isbecause HLS environments doesn't automatically create
the best hardware yet. There are challenges to exactly
grasp which implementation brings better hardware. So
we need to become used to how to write hardwareoriented codes in C++. Actually our current
implementation is far from the best and we need to makeit more hardware-oriented.
References
1. Courbariaux, Matthieu & Hubara, Itay & Soudry, Daniel& El-Yaniv, Ran & Bengio, Y.. (2016). Binarized Neural
Networks: Training Deep Neural Networks with Weights
and Activations Constrained to +1 or -1.
2. J. Redmon, S. Divvala, R. Girshick and A. Farhadi, "You
Only Look Once: Unified, Real-Time Object Detection,"2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), Las Vegas, NV, 2016, pp. 779-788,
doi: 10.1109/CVPR.2016.91.
3. Nakahara, Hiroki & Yonekawa, Haruyoshi & Iwamoto,Hisashi & Motomura, Masato. (2017). A Batch
Normalization Free Binarized Convolutional Deep Neural
Network on an FPGA (Abstract Only). 290-290.
10.1145/3020078.3021782.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

11

A Hardware-Oriented Random Number Generation Method and
A Verification System for FPGA
Sansei Hori, Hakaru Tamukoh

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: hori-sansei@edu.brain.kyutech.ac.jp, tamukoh@brain.kyutech.ac.jp
https://www.kyutech.ac.jp

Abstract
Deep learning technology has made remarkable progress in recent years and has been applied to a variety of
applications such as smartphones and cloud servers. These systems employ dedicated processors to save power
consumptions and process massive data. In this paper, we introduce a hardware-oriented restricted Boltzmann
machine and propose a field-programmable gate array (FPGA) infrastructure for easy verification of user circuits.
The infrastructure makes it easy to communicate and control between the host PC and the user circuit.
Keywords: FPGA, Hardware Accelerator, Xillybus, Deep Learning, Random Number Generator, RBM

Field-programmable gate arrays (FPGAs)5 can also
be used to create dedicated circuits and can be rewritten,
making it possible to build more general-purpose systems.
We have proposed resource-saving hardware
implementation of a restricted Boltzmann machine(RBM)6,7, which is a building block of Deep Belief
Networks. However, the hardware implementation of theuser circuit on an FPGA is costly work. In this paper, we
introduce the hardware-oriented RBM, and we propose
an FPGA infrastructure for verifying the modules
implemented as IP in FPGA. Also, to verify the operationof the proposed FPGA infrastructure, we implemented ahardware circuit for learning an RBM8.9 and confirmedthat the circuit could be controlled from a host computer.

1. Introduction
Recent days, neural network technologies such as deeplearning1 have been utilized many applications such asimage processing and natural language processing.
Especially, these technologies have been actively applied
to various embedded fields, including smartphone
applications.
However, most training of deep neural networks(DNNs) require a massive amount of calculation
resources and are often performed on high-performancecomputers with GPUs2. On the other hand, someembedded systems have restrictions such as power
consumption and physical size to implement highperformance computers. Therefore, application-specificintegrated circuits (ASIC) and system-on-a-chip (SoC)
dedicated DNNs3,4 is actively developed to accelerate theprocessing and reduce power consumptions. These someof the technologies have already yield practicalapplications.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

12

Sansei Hori, Hakaru Tamukoh

Fig. 2. Resource-saving random number generation method.
Fig. 1. Restricted Boltzmann machine.

units. In this case, a large number of random number
generators are required, but it is difficult to implement
them in the hardware such as an FPGA. Therefore, theauthors have proposed a method using truncated bits
generated during fixed-point arithmetic operations as a
substitute for random numbers6,7.
In general, when some operations are implemented indigital hardware such as FPGAs, the various operations
are implemented as fixed-point operations. In theoperations, if there are variables with M bits and N bitsin the integer and fractional parts, respectively, the resultof multiplication of these values is 2M bits in the integer
part and 2N bits in the fractional part, as shown in figure
2. Furthermore, the bit width of the integer part isincreased by the addition process. When the result of thisoperation is stored in the register, the incremented bits
are truncated. In the method, this truncated bit is used asa substitute for a random number.

2. Restricted Boltzmann machine
Restricted Boltzmann machine (RBM) is one of thegenerative model and a part of the element structures
DNNs. An RBM has two layers called visible layer andhidden layer, as shown in figure 1. The probability
distribution of an RBM calculated by Eq. (1), where 𝒗𝒗
and 𝒉𝒉 represent the states of visible and hidden units, and𝜽𝜽 is a set of the network parameters. Z(𝜽𝜽) shown in Eq.
(2) is a normalized term, and Φ shown in Eq. (3) is anenergy function, where 𝑎𝑎𝑖𝑖 and 𝑏𝑏𝑗𝑗 represent the biases ofvisible and hidden units, and 𝑤𝑤𝑖𝑖𝑖𝑖 is the weight.
1 −Φ(𝒗𝒗,𝒉𝒉,𝜽𝜽)(1)
𝑝𝑝(𝒗𝒗, 𝒉𝒉|𝜽𝜽) =𝑒𝑒.
𝑍𝑍(𝜽𝜽)
𝑍𝑍(𝜽𝜽) = � 𝑒𝑒 −Φ(𝒗𝒗,𝒉𝒉,𝜽𝜽) .

(2)

𝒗𝒗,𝒉𝒉

Φ(𝒗𝒗, 𝒉𝒉, 𝜽𝜽) = − � 𝑎𝑎𝑖𝑖 𝑣𝑣𝑖𝑖 − � 𝑏𝑏𝑗𝑗 ℎ𝑗𝑗 − � � 𝑤𝑤𝑖𝑖𝑖𝑖 𝑣𝑣𝑖𝑖 ℎ𝑗𝑗 . (3)
𝑗𝑗

𝑗𝑗

𝑖𝑖

4.

𝑗𝑗

Figure 3 shows the configuration of a proposed user logic
verification infrastructure. In this system, an FPGA isconnected to the host PC via PCI Express bus tocommunicate and control a user logic. When configuring
the FPGA, this system uses a joint test action group(JTAG). The user can connect the user logic by two AXI
interfaces to verify the user logic. In this chapter, we
describe the main components of this system.

The firing probabilities of the visible and hidden unitsof the RBM are calculated by the following equations,where σ(x) is a sigmoid function. This firing probability
determines the state of each unit in the RBM.
𝑝𝑝�ℎ𝑗𝑗 = 1�𝒗𝒗, 𝜽𝜽� = 𝜎𝜎 �𝑏𝑏𝑗𝑗 + � 𝑤𝑤𝑖𝑖𝑖𝑖 𝑣𝑣𝑖𝑖 � .

(4)𝑝𝑝(𝑣𝑣𝑖𝑖 = 1|𝒉𝒉, 𝜽𝜽) = 𝜎𝜎 �𝑎𝑎𝑖𝑖 + � 𝑤𝑤𝑖𝑖𝑖𝑖 ℎ𝑗𝑗 � .

(5)

𝑖𝑖

𝑗𝑗

Verification Infrastructure on FPGA

4.1. Interface between the host PC and the FPGA
We use PCI Express bus to connect the host PC to the
FPGA to control the user circuit and to communicate

3. Resource-saving random number generator
When training an RBM, it is necessary to sample eachstate from the firing probability of the visible and hidden© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

13

A Hardware-Oriented Random Number

5. Verification of the infrastructure
In order to verify the operation of the infrastructure, we
synthesized an RBM using a conventional random
number generators by Xilinx Vivado HLS, which is ahigh-level synthesis tool and implemented it as a user
logic.
In this verification, the RBM trained the MNIST11
dataset. The experimental conditions are listed below.
•
Visible neuron: 784•
Hidden neuron: 150•
Without HLS optimization options•
Integer bit width: 14 bits•
Fraction bit width: 18 bits
Table 1 shows the resource utilization report of theRBM under the conditions. The target device is Xilinx
Kintex 7 evaluation board KC705 (XC7K325T).

Fig. 3. User logic verification infrastructure.

data with the software on the host PC. This systemapplies Xillybus10 to realize the PCI Express
connection, which can convert the PCI Express data
communication to first in first out (FIFO) or AXIStream. The host PC can access to the FPGA by reading
or writing device files on its operating system.

Resource
LUT
LUTRAM
FF
BRAM
DSP
IO
GT
MMCM

4.2. Internal bus and user logic interface
This system applies AXI bus as an internal bus toconnect all modules except the interfaces for Xillybus
core. The user logic has AXI-Stream and AXI interface.
The AXI-Stream interface connects to the Xillybus coredirectly, and this data path is used for data transfer. The
AXI interface connects to AXI interconnect is used forthe internal bus to control the user logic from the hostPC through the special function registers (SFRs).

Table 1. Resource utilization report.
Utilization
Available
Utilization %
13202
580
16279
276.5
108
5
8
2

20380064000
407600
445
840
500
16
10

6.48
0.91
3.99
62.13
12.86
1.00
50.00
20.00

We trained the RBM logic by the MNIST dataset on
25 epochs, and after the training, we obtained the weightsof the RBM to verify the infrastructure. The weights areshown in Figure 4, and we can see the infrastructure is
working in this application.

4.3. Control registers
The software running on the host PC can control andmonitor the user circuits on the FPGA by accessing thecontrol and status registers called SFRs. The user circuiton the FPGA can also access the SFRs and return its
status, such as in-process or completed, to the software.
4.4. AXI-Stream to AXI bridge
The AXI-Stream to AXI bridge module extracts
addresses and data from the 8-bit stream data sent fromthe Xillybus core to access the SFR. It is a necessary
module to connect the core to the SFRs implemented inblock RAM (BRAM) because the Xillybus coreprovides AXI-Stream interfaces.

Fig. 4. Visualized weight parameter calculated on the FPGA
infrastructure.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

14

Sansei Hori, Hakaru Tamukoh

6. Conclusions and future works

7. S. Hori, and H. Tamukoh, A random number generation
method for hardware implemented neural networks,
IEICE Tech. Rep., vol. 119, no. 78, SIS2019-1, pp. 1-4,
June 2019.
8. A. Fischer and C. Igel, An introduction to restricted boltzmann machines, Progress in Pattern Recognition, Image
Analysis, Computer Vision, and Applications, pp.14–36,
Springer, 2012.
9. G.E. Hinton, A practical guide to training restricted boltzmann machines, Technical Report UTML TR 2010-003,
Department of Computer Science, University of Tronto,
2010.
10. Xillybus PCIe IP, http://xillybus.com, access: 14th
December, 2020.
11. Y. LeCun, B. Boser, J.S. Denker, D. Henderson, R.E.
Howard, W. Hubbard, and LD. Jackel, Backpropagation
applied to handwritten zip code recognition, Neural computation, vol.1, no.4, pp.541–551, 1989.

In this study, we proposed and constructed a system thatconnects a host PC and an FPGA via PCI Express, andcontrols a user logic from software on the host PC. By
using this system, the user does not need to construct thecommunication and control part by ownself except themodule which user wants to verify, and the verificationof the hardware-oriented system becomes possible more
quickly.
Our future work is as follows. Firstly, to implement
an RBM with the resource-saving random number
generators in hardware, and to verify its operation in theverification infrastructure. Secondly, to connect external
memory, such as an SDRAM to the verification
infrastructure to be able to handle larger network
parameters on the FPGA. In addition, since the user
circuit is connected to the peripheral circuits only by AXI
bus and AXI-Stream bus, we aim to create anenvironment in which the user circuit can be reconfigured
while the peripheral circuits are running by utilizing thepartial configuration technology. If this technology
becomes available, the circuit can be verified and tested
more easily and quickly. The goal of this project is tocreate an environment that enables easier and fastercircuit verification and experimentation.
References
1. G.E. Hinton, S. Osindero, and Y.-W. Teh, A fast learningalgorithm for deep belief nets, Neural computation, vol.18,
no.7, pp.1527–1554, 2006.
2. NVIDIA, “NVIDIA Tesla V100 GPU Architecture”,
https://images.nvidia.com/content/voltaarchitecture/pdf/volta-architecture-whitepaper.pdf, 2017.
3. N.P. Jouppi et al. In-datacenter performance analysis of atensor processing unit, Proceedings of the 44th Annual
International Symposium on Computer Architecture. 2017.
p. 1-12.
4. S. Wang, P. Kanwar, BFloat16: The secret to highperformance
on
Cloud
TPUs,
https://cloud.google.com/blog/products/ai-machinelearning/bfloat16-the-secret-to-high-performance-oncloud-tpus, 2019.
5. S. Trimberger, A reprogrammable gate array andapplications, Proceedings of the IEEE, 1993, 81.7: 10301041.
6. S. Hori, T. Morie, and H. Tamukoh, Restricted boltzmann
machines without random number generators for efficient
digitalhardwareimplementation,
International
Conference on Artificial Neural Networks, Springer,
pp.391–398 2016.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

15

Synthesis of realistic food dataset using generative adversarial network based on RGB anddepth images
Obada Al aama

Department of Life Science and Systems Engineering, Graduate School of Life Science and Systems Engineering,
Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Hakaru Tamukoh

Department of Human Intelligence System, Graduate School of Life Science and Systems Engineering,
Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
E-mail:aama.obada-walidal425@mail.kyutech.jp, tamukoh@brain.kyutech.ac.jp
www.lsse.kyutech.ac.jp

Abstract
Constructing a large food dataset is time and effort consuming due to the need to cover the feature variations of food items.
Hence, a huge data is needed for training neural networks. This paper aims to advocate the Cycle-GAN to build up large fooddataset based on large number of simulated images and relatively few real captured images thus obtaining more realistic
images effortlessly compared with traditional capturing. Real RGB and depth images of variant food samples allocated overturntable were captured in three different angles using real-sense depth camera with different backgrounds. Furthermore, forsimulated images, the Autodesk 3D_Maya software was employed using the same parameters of captured real images. Results
showed that generally, realistic style transfer on simulated food objects was obtained as a result of employing Cycle-GAN.
GAN proved to be an efficient tool that could minimize imaging efforts resulting in realistic images.
Keywords: Cycle-GAN, Food dataset, RGB-D images.

1. Introduction

This paper aims to employ the automated Cycle-GAN
technique to synthesize a large food dataset by the meansof converting the simulated images to more relevant
realistic images instead of only using large number ofreal images, hence, the Cycle-GAN produces the nearest
real images based on processing a few numbers of real
images along with the huge number of simulated images
compared with the traditional imaging techniques.
Results obtained from this study represent preliminary
results that could be further used to train neural networkfor object detection which could be applied in various
robotics applications.

Recently, generative adversarial networks (GANs) haveattracted particular attention. A GAN can generate animage that resembles a real image. A human face image
dataset such as CelebA and a numeric character image
dataset such as MNIST have been used for training the
GAN as the target domains 1. GANs have produced
promising results in many generative tasks, such as
photo-realistic image generation. One of the importantobstacles of generating a large dataset is the need of ahuge number of real images which is time and effort
consuming. Meanwhile, few have reported food image
generation or transformation using the GAN thus far.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), Jan. 21-24, Japan

16

Obada Al aama, Hakaru Tamukoh

2.

Research Concept

The main concept focuses over using the depth camera tocapture both RGB and depth images simultaneously, on
the other hand, employing Autodesk 3D-Maya software
to obtain the simulated images from 3D food models.
Thereafter, training and testing the Cycle-GAN using
those above mentioned images to investigate thecapability of Cycle-GAN to produce the realistic images
based training and testing few real images and hugesimulated images.
3.

(a)

(b)

Methodology

3.1. Real image capturing
This experiment was carried out through adjusting theIntel® RealSense™ Depth Camera D435i to be in frontof turntable where the food objects including Apple,
Kiwi, Mini-tomato and Mushroom were allocated. BothRGB and depth images were captured using this camera.
The settled parameters of image capturing were as
follows; distance from the object was 50 Cm, different
capturing angles (0°, 45°, and 90°), turntable rotation rate
was 1.22 rpm, capturing rate was 30 fps and the image
resolution was 640×360 pixels. Images obtained were
further extracted and processed as shown in Figure 1. All
images for each food class were categorized in specificfolders. Each class has 6570 images for RGB and thesame number for depth in Jet-color mapping throughobtaining 2190 images from each selected angle. In
image capturing step the Rosbag_record was employed
and accordingly the Region of Interest ROI cropping at
150×150 pixels and normalization were applied
simultaneously resulting in Bag file format 2. Extractionof images was carried out using the CV_bridge code 3, 4
which role is to re-save the images from RVIZ simulator
to the proposed directory. Moreover, the saved images
were processed through removing the background after
the manual detection of the background color range for
RGB via Chroma-key masking 5, and applying the samemask on the depth images.

Fig. 1. Schematic diagram illustrating the different
procedures of real image capturing beginning from the(a) image capturing process, and (b) different steps ofextracting and processing of the obtained images.
3D_Maya software 6 to obtain the simulated images using
the same parameters applied for capturing the real images(capturing angles, object capturing distance, turntablerotation rate, capturing rate, and resolution), resulting inthe same number of captured images as those obtained
from Real capturing. Finally, the background was
automatically removed for both RGB and depth during
the render step. However, the simulated depth images
were obtained in grey-scale color model. The whole
setup of the experiment was illustrated in Figure 2.

3.2. Simulated image generation

Fig. 2. Simulator experiment setup.

The 3D simulated model for each candidate wasdownloaded from open source as an obj format.
Thereafter these models were imported to the Autodesk

3.3. Cycle-GAN image generation
For each food class 500 real images and 500 simulated
images were used to train the Cycle-GAN neural network.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), Jan. 21-24, Japan17

Synthesis of realistic food

The epoch was setup to be 200 epochs, and the training
time was about 11 hours. Thereafter 6070 images were
tested to validate Cycle-GAN neural network. ThroughCycle-GAN training the equation used were obtained
from 1. The main concept and various processes applied
through Cycle-GAN was illustrated in Figure 3.

Fig. 3. Cycle-GAN main concept.
4.

Fig. 5. Real captured RGB and depth images for kiwi,
and the output after masking.

Results and discussion

4.1. Real imaging
Four classes were selected as food objects which weremushroom, mini-tomato, apples, and kiwi. For eachobject both RGB and depth were captured in threedifferent capturing angles 0°, 45°, and 90°, resulting in
26280 total images as 2190 images for each category ateach capturing angle. In Figures 4, 5, 6, and 7 the real
images of different food objects were shown, along withthe applied mask, and the obtained output. Moreover, at
0° the turn table surface appeared in all captured images
and this require an extra cropping step to remove theinterfered part before applying the Chroma-key masking.
However, at 45° and 90° this process was not required
and the Chroma-key masking was applied directly.

Fig. 6. Real captured RGB and depth images for minitomato, and the output after masking.

Fig. 4. Real captured RGB and depth images for apple,
and the output after masking.

Fig. 7. Real captured RGB and depth images formushroom, and the output after masking.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), Jan. 21-24, Japan18

Obada Al aama, Hakaru Tamukoh

same manner the rest of the total obtained images wereused to test the Cycle-GAN as illustrated in Figure 10
showing the obtained depth Cycle-GAN output for
mushroom food class as example.

4.2. Simulated imaging
The same number of simulated images were obtained ateach angle for both RGB and depth as those obtained inreal imaging. However, the obtained depth images weregrey-scale. Figure 8 shows the obtained RGB and depth
simulated images for each investigated food class.

Fig. 10. Samples for Cycle-GAN output for
mushroom class depth images.
5.

Conclusion

This paper proposed to synthesize a large food dataset
based on the use of few number of real images along witha huge number of simulated images utilizing the CycleGAN to produce realistic images. We used only 500 Realand another 500 simulated for both RGB and depth fortraining Cycle-GAN for each food class and this number
represent a few number compared with that used fortesting which was 6070 simulated images. Accordingly,
our proposed method proved to be efficient insynthesizing of large food dataset. However, our data
represent a preliminary result and a future study focusing
on the effect of number of training images on the realistic
quality of the output of Cycle-GAN, should be conducted,
in addition to focusing on widening of the investigatedfood dataset including Japanese food.

Fig. 8. Simulated RGB and depth images for apple,
mushroom, mini-tomato, and kiwi.
4.3. Cycle-GAN
500 RGB Real images in addition to 500 RGB simulated
for each object class were used to train the Cycle-GAN.
Thereafter rest of the total obtained simulated images(6070) were used in Cycle-GAN testing step rather thanthose used in the training step as shown in Figure 9. It

References
1.

2.

3.
4.

Fig. 9. Samples for Cycle-GAN output for different
food classes RGB images at variant angles.

5.

was noticed from the obtained RGB Cycle-GAN output
that Cycle-GAN represent an efficient tool in modellingof near to real images in various angles. On the other
hand, for the depth images 500 images were selected also
for each object class to train the Cycle-GAN, and in the

6.

HORITA, Daichi, et al. Food category transfer withconditional cyclegan and a large-scale food image dataset.
In: Proceedings of the Joint Workshop on Multimedia for
Cooking and Eating Activities and Multimedia Assisted
Dietary Management. 2018. p. 67-70
SHI, Wuzhen, et al. Reference image based method ofregion of interest enhancement for haze image. In: 2015
IEEE International Conference on Image Processing(ICIP). IEEE, 2015. p. 735-739.
Website:"http://wiki.ros.org/cv_bridge/Tutorials/Conver
tingBetweenROSImagesAndOpenCVImagesPython",
15/12/2020.
MARTINEZ, Aaron; FERNÁNDEZ, Enrique. Learning
ROS for robotics programming. Packt Publishing Ltd,2013.
C. Teng, Y. Liao, Y. Chou and S. Lin, "Removing blue
screen background under non-uniform illumination,"2017 IEEE International Conference on Consumer
Electronics - Taiwan (ICCE-TW), Taipei, 2017, pp. 307308, doi: 10.1109/ICCE-China.2017.7991118.
Autodesk, INC. (2019). Maya. Retrieved from https:/
autodesk.com/maya.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), Jan. 21-24, Japan

19

Comparison of Data Augmentation Methods in Pointer-Generator Model Using Various
Sentence Ranking Methods
Tomohito Ouchi, Masayoshi Tabuse

Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Shimogamohangi-cho, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: t_ouchi@mei.kpu.ac.jp, tabuse@kpu.ac.jp

Abstract
In the existing research, we proposed a data augmentation method using topic model for Pointer-Generator model.
In this study, we add to the sentence ranking method in the data augmentation method. Specifically, we add two
ranking methods using LexRank and Luhn. LexRank is based on Google's search method and Luhn defines
sentence features and ranks sentences. We compare three data augmentation methods. We considered which
method is suitable for data augmentation. We confirm that most accurate model is the model using data
augmentation method by topic model.
Keywords: automatic summarization, data augmentation, Pointer-Generator Model, Extractive summarization.

order to make the generative summarization model.
Therefore, we focused on data augmentation as a methodthat can be applied to any model. This is to create
extended data from existing data. As a result, it was
confirmed that the accuracy of the evaluation metric
ROUGE [4] of Pointer-Generator model applied by thedata augmentation method is improved by about 1%
compared to baseline model.
Next, we explain the method of data augmentation
simply. We decide the importance of each sentence ineach article. And the sentence with the lowest importanceis removed to obtain extended data. In the existing
research [5], the topic model was used to measure theimportance of sentences. In this study, in addition to that,the method called Luhn[6] and LexRank[7] was used.
These three techniques are described in Section 2.
Experiments and results are described in Section 3. And
discussions are given in Section 4.

1. Introduction
Currently, the amount of information on the Internet isincreasing exponentially, and it is said that it will reach
59ZB in the research in May 2020[1]. It is also said thatthe total amount of data created in the next three years
exceeds the total amount of data in the past 30 years, andthe total amount of data created in the next five years ismore than three times the total amount of data created in
the past five years. Under such circumstances, the issueof selecting information is an urgent problem. Automatic
summarization struggles that issue. However, it can besaid that extractive summarization that only made up
with sentences is not sufficient. Since the sentence-tosentence connection is not taken into consideration,
readability is lacking. Therefore, it is needed generative
summarization as a technology that looks ahead. A
generative summarization basically uses the EncoderDecoder model, which learns the relationship between
input and output and generates one word at a time in the
output when a new input comes in during the test.
Various models have been proposed [2,3]. In this study,
the Pointer-Generator model [2] uses as the baselinemodel. One of the issues with the generative
summarization model is that data maintenance is costly.
We have to attach a manual summary to each article in

2. Data Augmentation Method
This section describes the three models used in thedata augmentation method. In each method, eachsentence is scored in an article, and the sentence with thelowest score is removed to obtain extended data.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

20

Tomohito Ouchi, Masayoshi Tabuse

2.1. Luhn

3. Experiment and Results

We measure position of the top 100 in most frequentwords removed stop-words. We define words with adistance of 5 or less as one cluster. The score of a cluster
is the square of the number of important words in acluster divided by the distance between the first and lastwords of the cluster. Finally, the maximum score of eachcluster becomes the score of the sentence.

In this section, the experimental conditions and theresults of additional experiments using the Luhn method
and the LexRank method are included.
3.1. Parameter Setting
The CNN / DailyMail dataset is used as the dataset fortraining, evaluating, and testing. The training data,
evaluating data, and test data are 287,226 articles, 13,768
articles, and 11,490 articles, respectively. The model
used for the experiment is the Pointer-Generator model,
which is divided into a copy mechanism and a coveragemechanism when learning. The Copy mechanism
calculates the error of the evaluating data each time theepoch ends and we uses the model of the epoch with thelowest error in Early Stopping. Early Stopping what wemean here, uses a model that waits twice as many epochs
as the error seems to be the minimum, unless theminimum value is updated. Next, in the coveragemechanism, the same processing is performed in the
coverage loss. We use ROUGE as using for evaluation
on existing research.
The program used in this research uses PyTorch. It has
been confirmed that this program can achieve the same
result as [2]. The hidden layer vector size was set to 256
and the embedded vector size was set to 128. The batch
size was set to 8. In the original paper, the batch size is16, so double learning is required to learn the same
number of articles. The beam size was set to 4. The beam
search will be described later. The number of vocabulary
was set to 50,000. The learning rate was set to 0.15.
In this program, the number of words used to encode
an input article is limited to 400. This setting has no effecton learning an extended data. Specifically, an extended
data is the same as an original data. This is because theextracted sentence may not be within 400 words from the
beginning. We must confirm that the extracted sentence
is present in the input article. Therefore, I found thearticle with the most number of words among the articles
used in the training data. The number of words with themost words was 2,380. And the upper limit of the numberof words used in encoding the input article was set to2,380. Table 1 shows the values of ROUGE when the
maximum number of words is 400 and 2,380. In the
Table 1, f, r, and p represent the F value, recall, andprecision, respectively.

2.2. LexRank
First, we explain PageRank, which is the basis of
LexRank. The basic idea of PageRank is that linked
pages are good pages, and links from more linked pages
are evaluated highly. This rating is equivalent to the user
inflow to the page. This is because if links are provided
from many pages, it is easy to flow in, and the inflow
from popular pages is larger than the inflow from normal
pages. Links between pages can be represented by amatrix, which is the probability that the user willtransition from that page to another linked page. The
matrix is made with dividing by the total number of links
on each page. The purpose of PageRank is to use thismatrix to determine the probability that a user will stayon each page, that is, the rating of the page. PageRank isbased on the premise that the page stay probability will
eventually stabilize if the page transition is repeatedmany times, so that the transition matrix multiplied by
the stay probability vector becomes closer to thetransition matrix.
In LexRank, the transition matrix is the matrix of thecosine similarity of the Tf-Idf score between sentences.
The basic idea of LexRank is that sentences similar to
many sentences and sentences similar to importantsentences are considered to be important sentences.
2.3. Topic Model
This method was used in the existing method. For how
to determine the importance of sentences using the topic
model, we referred to existing research [8]. The topic
model is one of the language models that assumes that
one document consists of multiple topics. In addition,
each topic has an appearance word distribution. The
method of determining the importance of a sentence is asfollows.
1.
2.
3.
4.

Calculate the frequency of occurrence in a topic
with words that make up a sentence
Sum of all the words that make up the sentence
Divide by the square root of the sentence length
Sum on all topics© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

21

Comparison of Data Augmentation

new : `` a person who has such a video needs to
immediately give it to the investigators '' robin 'scomments follow claims by two magazines , german
daily bild and french paris match , of a cell phone video
showing the harrowing final seconds from on board
germanwings flight 9525 .
paris match and bild reported that the video wasrecovered from a phone at the wreckage site .

Table1 the values of ROUGE when the maximum
number of words is 400 and 2,380
400
2380

ROUGE-1-f ROUGE-1-r ROUGE-1-p ROUGE-2-f ROUGE-2-r ROUGE-2-p
0.39350.43720.38000.17090.18910.16620.39580.41810.39940.17410.18320.1770400
2380

ROUGE-L-f ROUGE-L-r ROUGE-L-p
0.36160.40140.34930.36440.3846
0.3679

Table 1 shows when the upper limit of the numberof words is increased from 400 to 2,380, the value of
ROUGE increases slightly. In the following, theexperiment is performed with the upper limit of thenumber of words set to 2,380.

extendedmarseille prosecutor brice robin says he was not aware ofany video footage from the plane .
robin 's comments follow claims by two magazines ,german daily bild and french paris match .
`` one can hear cries of ` my god ' in several languages ,'' paris match reported .

3.2. Beam search
Greedy method contrasts with beam search. This isbecause, when generating a word, one word with thehighest generation probability is selected, while in beam
search, processing is performed while holding the top K
words. Then, we make the final summarizations bymultiplying the probabilities of each word generation,and make the highest one the final summarization. In thisexperiment, this K value is set to 4.

rimarseille prosecutor brice robin told cnn that `` so far novideos were used in the crash investigation '' robin 'scomments follow claims by two magazines , german
daily bild and french paris match .
all 150 on board were killed .
srnew : `` it is a very disturbing scene , '' official says .
new : `` one can hear cries of ` my god ' in several
languages , '' prosecutor says .
`` one can hear cries of ` my god ' in several languages ,'' official says .

3.3. Results
The results are as below.
Table 2 Results of learning 287226 articles
using 6 methods
rouge-1
rouge-2
rouge-L

normal
0.38200.16400.3514extended
0.39480.17240.3624ri
0.38560.16490.3527sr
0.38690.16830.3551

rouge-1
rouge-2
rouge-L

rs
0.39110.1696
0.3596rd
0.38170.16570.3499Luhn
0.37390.16400.3449

LexRank
0.37770.1616
0.3489rs
marseille prosecutor brice robin says he was not aware ofany video footage from on board .
robin 's comments follow claims by two magazines ,german bild and french paris match .
the video was found by a source close to theinvestigation .

In order of good results, the existing method[5], baseline,
ri, sr, rd, LexRank, rs, and Luhn.
A summary example generated in each model is shown.

rdfrench prosecutor leading an investigation into the crashof germanwings flight 9525 .
robin 's comments follow claims by two magazines ,german daily bild and french paris match , of a cell phone
video showing the harrowing final seconds from on board
the plane .
german airline lufthansa confirmed tuesday that co-pilot
andreas lubitz had a `` previous episode of severe
depression ''referencemarseille prosecutor says `` so far no videos were used in
the crash investigation '' despite media reports .
journalists at bild and paris match are `` very confident ''the video clip is real , an editor says .
andreas lubitz had informed his lufthansa training schoolof an episode of severe depression , airline says .

Luhn
new : `` so far no videos were used in the crashinvestigation , '' official says .

normal©The 2021 International Conference on Artificial Life and Robotics (ICAROB2021),Jan.21-24,B-ConPlaza, Beppu, Oita, Japan22

Tomohito Ouchi, Masayoshi Tabuse

new : `` a person who has such a video needs to
immediately give it to the investigators , '' says a forensic
psychologist .
LexRank
marseille prosecutor brice robin told cnn that `` so far novideos were used in the crash investigation '' french
president francois hollande says it should be possible toidentify all the victims using dna analysis by the end of
the week .
4. Conclusion
In this study, in addition to the existing studies, we
experimented with a data augmentation method using the
Luhn and LexRank methods. The results confirmed thatthe best data augmentation method is to use the topic
model of the existing research. In the future, we would
like to confirm the effectiveness of data augmentation forstate-of-the-art models. When extracting a sentence froman article, I would like to try a method for extracting
multiple sentences instead of one sentence. We also
expect that the number of sentences will change
depending on the length of the article.
References
1. International
Data
Corporation(IDC)
https://www.idc.com/
2. Abigail. See, Peter J. Liu, et al. “Get To The Point:
Summarization with Pointer-Generator Networks”
arXiv:1704.04368(2017)
3. Yang Liu, Mirella Lapata “Text Summarization with
Pretrained Encoders” arXiv:1908.08345v2(2019)
4. Chin-Yew Lin, “ROUGE: A Package for Automatic
Evaluation of Summaries” Workshop on Text
Summarization Branches Out, Post-Conference Workshop
of ACL 2004, Barcelona, Spain (2004)
5. T. Ouchi, M. Tabuse, “Effectiveness of Data
Augmentationin
Pointer-Generator
Model”ICAROB(2020)
6. Luhn, H.P., “The automatic creation of literature of
abstracts” IBM journal of research and development, vol.
2, No. 2, pp.159-165 (1958)
7. Gunes Erkan, Dragomir R Radev, “LexRank : graph-based
lexical centrality as salience as text summarization”
journal of Artificial Intelligence Research, vol. 22, pp.
457-479 (2004)
8. H.Sigematsu, I.Kobayashi “Generation of abstracts
considering importance of potential topics” The
Association for Natural Language Processing (2012) (InJapanese).

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

23

Music Recommendation System Driven by Variations in Fingertip Skin Temperature
Mayuka Wada1, Taro Asada2, Yasunari Yoshitomi2, Masayoshi Tabuse2

1: Shimadzu System Development Corp., 1 Tokudaiji-cho, Nishinokyo, Nakagyo-ku,Kyoto, Japan
2: Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: t_asada@mei.kpu.ac.jp, {yoshitomi, tabuse}@kpu.ac.jp
http://www2.kpu.ac.jp/ningen/infsys/English_index.html

Abstract
In recent years, music therapy has been performed to recover the cognitive function of elderly people. In thepreviously reported system, it was necessary to input the subjective evaluation of the user to determine the next
recommended song. In this study, variations in fingertip skin temperature were used as input instead of subjective
evaluation, focusing on evaluation of emotion expressed as physiological response through variations in fingertip
skin temperature. Our system can be used with no support for users, such as patients who have severe dementia orare bedridden and cannot respond readily.
Keywords: Music recommendation system, Music therapy, Peripheral skin temperature.

Introduction

Related Studies

In recent years, against the backdrop of increasingly
aging societies in developed countries such as Japan,
activities using music for the elderly ("music therapy")
have been used with the aim of improving recognition
ability. We previously developed music recommendation
systems for music therapy for older people.1
However, in the previously reported music
recommendation system,1 it was necessary to input the
subjective evaluation of the user ("I want to listen to it
again" or "I don't want to listen to it again") into therecommended music in order to determine the next
recommended song. Therefore, in the case of a user who
has difficulty in responding by himself/herself, such as adementia patient, the support of a caregiver may berequired, and the problem of securing human resources
hinders the use of our system.
In this study, we focused on the evaluation ofemotions by measuring peripheral skin temperature,2developed a method for determining recommended
songs based on the physiological evaluation of the user,and evaluated our system.

We now briefly explain the systems used in ourprevious study1 and another study,2 which we build
upon to evaluate emotions in this study.
2.1. Previously reported system for music
recommendation
The method for music recommendation used in thisstudy is based on a previously reported methods.1
Similartothepreviously
reportedmusic
1
recommendation system, our system is a hybrid system
for improving recognition ability using collaborative
filtering and impression words.3 See the paper1 for detailon the music recommendation module in our system.
2.2. Evaluation of emotions by measuring
peripheral skin temperature
In the study by Watanabe et al.,2 the subject watched ahorror video and an environmental video alternately, and the
peripheral skin temperatures of the subjects viewing thevideos were measured. Three peripheral parts, the front ofauricle, the nose, and the back of the right index finger were

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

24

Mayuka Wada, Taro Asada, Yasunari Yoshitomi, Masayoshi Tabusemeasured, and temperature changes were compared andexamined. No significant change was observed in the skintemperature on the front of the auricle and on the nose.
However, the skin temperature on the back of the fingers of
the right hand decreased when viewing the horror video
and increased when viewing the environmental video.
From this result, the influence of autonomic nerve activity
appears to be reflected strongly in the peripheral skintemperature of the fingertip as a change in blood flow due
to vasoconstriction.

the subject had begun listening to music) were set as thethreshold values.
Peripheral skintemperature（℃）

35Construction of a physiological evaluation
method for music

34.5

34

0

100

Time (s)200

Fig. 1. Changes in peripheral skin temperature when listening
to music (Favorite song).
Peripheral skintemperature（℃）

A physiological evaluation method for music, whichmeasures the peripheral skin temperatures of subjects
listening to music, will be described. In this study, we
selected the pad of the left index finger as the peripheral
part for measuring temperature.
3.1. Changes in peripheral skin temperature while
listening to music
In order to evaluate emotions by measuring peripheral
skin temperature in a music recommendation system,
we investigated the changes in peripheral skintemperature of subjects listening to music.
When subject A (a female in her twenties) listened toher favorite song ("Canon," composed by Johann
Pachelbel) and least favorite song ("Mystery III Heterophony for Swirl Butoh -," composed by Akira
Nishimura), the peripheral skin temperature tended to rise
while listening to her favorite song and decreased while
listening to her least favorite song (Figures 1 and 2).3

34
33
32
31
30

0

50

100
150
Time (s)200

Fig. 2 Changes in peripheral skin temperature when listening
to music (Least favorite song).

3.3. Binarization of physiological evaluation
Using the two thresholds described in subsection 3.2, themusic being listened to was judged to be "Physiologicallyfavorable (1)" or "Physiologically unfavorable (0)" basedon the temperature change of the peripheral skin when
listening to the music. Figure 3 shows the judgmentprocessing flow for the physiological evaluation of music.
In Figure 3, the elapsed time from the start ofmeasurement is indicated by "time," and the measured
temperature at that time is indicated by "temp." Themaximum temperature from the start of measurement tothat point is indicated as "max," and the difference
between "max" and "temp" is indicated as "dif."

3.2. Determination of threshold for changes inperipheral skin temperature
Similartothepreviously
reported
music
recommendation system,1 the system used in this study
determines the next recommended song by inputting aevaluation value (typing in "1" if you want to listen to
the song again, or "0" if you don't want to listen to thesong again). In this study, we determined the thresholdvalue for dividing "1" when "physiologically
preferable" and "0" when "physiologically unfavorable"
based on the change in peripheral skin temperature.
Based on the results of the preliminary experiment,
0.4°C (the temperature difference between the
maximum temperature and the subsequent minimum
temperature) and 180 seconds (the elapsed time since

Experimental evaluation
4.1. Conditions
The music recommendation system3 used in this study is
an improved version of the previously reported system.1
The improved system can support multiple music
databases, and is mainly aimed at the elderly. It wasreported that elderly people tend to choose songs that

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

25

Music Recommendation System Driven

concordance rate between subjects (Table 1),
improvements such as reviewing the threshold areconsidered necessary. In addition, depending on thesubject, the concordance rate of either the children’ssong or the popular song is high, and the concordance
rate of the other is low. Therefore, it would appear that
there are some problems that cannot be solved only byadjusting the threshold value.
4.2.2. Correct answer rate of music recommendation
Table 2 shows a comparison of the percentage of correct
answers for the song recommendation results. The
percentage of correct answers for the music
recommendations was relatively low. The probablereason for this was that many of the subjects wererelatively young, not elderly people who were originallytargeted by our system. As an example, a male in his 60s(Subject J) had a relatively high percentage of correct
answers (73.3-83.3%) in both the physiological andsubjective evaluations. Figure 4 shows the change inperipheral skin temperature for the song that Subject J
judged to be his "favorite song" in both the physiologicaland subjective evaluations. The peripheral skintemperature gradually increased over time (Figure 4).

Fig. 3 Flowchart for binarizing physiological evaluation.3

are the subject of elementary school music textbooks asfamiliar songs (favorite songs).5 Therefore, we selected a
CD described as an anthology of older songs enjoyed byolder people with dementia, and selected 52 songs on the
CD that were also included in a music textbook databasefor elementary schools as well as 58 popular songs.6 There
were a total of 10 subjects: 4 females in their 20s (Subjects
A to D), 5 males in their 20s (Subjects E to I), and 1 malein his 60s (Subject J). For programming, we used VisualC++ 6.0 (Microsoft) on a PC (Dell Optiplex790, CPU:Intel Corei7-2600 3.40GHz, main memory: 4.00 GB, and
OS: Windows 7 Professional, Microsoft) for theexperiment. We used a digital temperature and relative
humidity sensor SHT117 to measure temperature.

Peripheral skintemperature（℃）

32

31

30

0

50

100

Time (s)150

Fig. 4. Changes of subject J in peripheral skin temperaturewhen listening to music (Favorite song).

4.2. Results and discussion

Subject J (a male in his 60s) had a high rate ofaccuracy for both children’s songs and popular songs.
From this, the selection of the music database isconsidered to be one of the factors that contributed to
the low recommendation accuracy for one of thesubjects who was in their 20s. In future research, weplan to conduct experiments that focus on the elderly.

4.2.1. Concordance rate
Table 1 shows the results of the concordance rate
between the physiological evaluation and the subjective
evaluation. The concordance rate of all experimental
results was relatively high, with an average of 70.9%.
However, since there is a large difference in the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

26

Mayuka Wada, Taro Asada, Yasunari Yoshitomi, Masayoshi Tabuse

Table 1 Experimental results.3

Children's
songs

Subject

A

B

C

D

E

F

G

H

I

J

Number of recommended songs

15

15

15

15

15

15

15

15

15

15

150

Evaluation value

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

Physiological evaluation

8

7

5

10

6

9

6

9

7

8

9

6

8

7

10

5

8

7

11

4

78

72

Subjective evaluation

9

6

8

7

14

1

9

6

8

7

5

10

7

8

11

4

5

10

13

2

89

61

Number of concordances:
Individual

8

6

3

5

6

1

5

5

4

4

5

6

7

7

8

2

2

4

11

2

59

42

Number of concordances: All

14

8

7

10

8

11

14

10

6

13

101

Concordance rate (%)

93.3

53.3

46.7

66.7

53.3

73.3

93.3

66.7

40

86.7

67.3

Number of recommended songs

Popular
songs

Total

8

11

10

10

9

8

8

8

9

12

93

Evaluation value

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

Physiological evaluation

3

5

3

8

9

1

8

2

6

3

1

7

3

5

6

2

6

3

10

2

55

38

Subjective evaluation
Number of concordances:
Individual
Number of concordances: All

3

5

6

5

10

0

8

2

2

7

2

6

5

3

7

1

5

4

9

3

57

36

3

5

1

3

9

0

7

1

2

3

1

6

2

2

6

1

4

2

9

2

44

25

8

4

9

8

5

7

4

7

6

11

69

Concordance rate (%)

100

36.4

90

80

55.6

87.5

50

87.5

66.7

91.7

74.5

Table 2 Recommendation accuracy.3
Subject

Children's songs
Popular songs

A

B

C

D

E

F

G

H

I

J

53.3

33.3

40

40

46.7

60

53.3

66.7

53.3

73.3

Subjective evaluation (%)

60

53.3

93.3

60

53.3

33.3

46.7

73.3

33.3

86.7

Physiological evaluation (%)

37.5

27.3

90

80

66.7

12.5

37.5

75

66.7

83.3

Subjective evaluation (%)

37.5

54.5

100

80

22.2

25

62.5

87.5

55.6

75

Physiological evaluation (%)

Conclusion
By focusing on the evaluation of emotions by measuring
the peripheral skin temperature, we have developed amusic recommendation system that inputs subjective
evaluations using peripheral skin temperature. Thissystem can be used even when users who cannot respondactively, such as patients with severe dementia or
bedridden patients, are not supported by a caregiver. The
concordance rate between the physiological evaluationand the subjective evaluation varied depending on thesubject and the music genre. One of the factors is thatthe threshold is set for one subject. In addition to music,
there are many factors that affect peripheral skintemperature, such as room temperature and the physicalcondition of the subject.

2.
3.

4.

5.
6.

Acknowledgements

7.

We would also like to thank the subjects of ourexperiments for their cooperation.

recognition ability using collaborative filtering andimpression words, J. Artif. Life and Robotics 18(1-2),
2013, pp. 109–116.
T. Watanabe, M. Okubo, T. Kuroda, Evaluation of emotion
using peripheral skin temperature-effectiveness of the backof finger measurement, JJME 36, 1998, pp. 637.
M. Wada, T. Asada, Y. Yoshitomi, and M. Tabuse,
Music recommendation system using peripheral skintemperature as input data (in Japanese), in Proc. of
Human Interface Symposium 2019, Kyoto, Japan, Sep.,
2019, pp. 736–739.
T. Kumamoto and K. Ohta, Design of scales to represent
user's impressions of a music piece for use in a musicretrieval system (in Japanese), IPSJ SIG Notes 2001-NL147(6), (2002), 35–40.
T. Takahashi, Research report on songs familiar to people
advanced in years (in Japanese), J. Japanese music
therapy associate 15(1) , 1997, pp. 68–75.
T. Akahoshi, Good old anthology enjoyable for people
advanced in years and troubled with dementia (inJapanese) , Kirara shobo, Tokyo, 2009.
Sensirion Digital Humidity Sensor,
https://www.sensirion.com/en/environmentalsensors/humidity-sensors/digital-humidity-sensors-foraccurate-measurements , Accessed 4 December 2020.

References
1. S. Yoshizaki, Y. Yoshitomi, C. Koro, and T. Asada,Music recommendation hybrid system for improving© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

27

Total
52
59.3
57.6
60

Music Recommendation System Driven by Interaction between User and Personified Agent
Using Speech Recognition, Synthesized Voice and Facial Expression
Ayumi Matsui1, Miki Sakurai2, Taro Asada3, Yasunari Yoshitomi3, Masayoshi Tabuse3

1: Sumitomo Mitsui Card Co., Ltd., 4-5-15 Imabashi, Chuo-ku, Osaka 541-0042, Japan
2: TIS Inc.,8-17-1 Nishi-Shinjuku, Shinjuku-ku, Tokyo 160-0023, Japan,
3: Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: t_asada@mei.kpu.ac.jp, {yoshitomi, tabuse}@kpu.ac.jp
http://www2.kpu.ac.jp/ningen/infsys/English_index.html

Abstract
We propose a music recommendation system that is characterized by an interaction between a user and apersonified agent. Speech is recognized using a speech recognition system called Julius, and the facial expressionof the agent is then synthesized using preset parameters that depend on each vowel. We add a new function thatchanges the facial expression of the agent according to the response of the user to music recommended by ourpreviously proposed system. The effectiveness of the proposed system is verified.
Keywords: Music recommendation, MMDAgent, Facial expression synthesis, Speech recognition, Speech synthesis.

1. Introduction

2. Proposed System and Method

In Japan, the average age of the population has been
increasing, and this trend is expected to continue. This
trend is more pronounced in rural areas. Recently, music
therapy has been used to improve the recognition abilityof people, particularly older people. Music therapy may
be more effective if music that is liked by an individual isadopted. We have previously developed a music
recommendation system that aims to improve recognition
ability, including a system through the Internet using avideophone system.1-3 In our previously proposedsystem,1-3 it is necessary to input the subjective evaluationof the user into a computer to determine the next musicthat is to be recommended. User input is inconvenient for
elderly people, especially those with dementia.
In the present study, to overcome this inconvenience,
we propose a system for music recommendation that ischaracterized by an interaction between a user and apersonified agent, which uses speech recognition,
synthesized voice and facial expression generation.

2.1. System overview
Figure 1 shows the processing flow of the proposedsystem, the music recommendation module of which isbased on the previously proposed system2 that usedcollaborative filtering and impression words (see thepaper2 for detail on the music recommendation module).
The proposed system is characterized by an interaction
between a user and a personified agent, and uses speech
recognition, synthesized voice and facial expression
generation for integrative music recommendation.
To make the agent on a personal computer (PC)screen appear more human-like, we developed a system
for agent facial expression generation that uses vowel
recognition when generating synthesized speech.4 We
added a new function to change the facial expression of
the agent according to the response of the user to music
recommended by our system.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

28

Ayumi Matsui, Miki Sakurai, Taro Asada, Yasunari Yoshitomi, Masayoshi Tabuse

Creating facial expression data↓
Recording agent voice↓
Automatic division of WAVE files↓
Speech recognition using Julius↓
Inserting data of agent neutral facial expression↓
Creating motion data for facial expression
Fig. 2. Processing flow of agent generation in our system.4

Fig. 1. Processing flow of the proposed system.

2.2. Personified agent/a/Figure 2 shows the process for agent generation in oursystem, which consists of six steps: creating facial
expression data, recording vocal utterances, automaticWAVE file division, speech recognition by the Julius5,
insertion of expressionless data, and the creation of
facial expression motion.4 The facial expression data

are created in advance.

/i/

/e/

Expressive motions are generated by combining
the expression data of each vowel for each utterance
motion. Then, utterance contents are input as text andused by the MikuMikuDanceAgent (MMDAgent),6
which is a freeware animation program that allows users
to create and animate movies with agents, to output
synthesized voice that is then recorded by a stereo mixer
inside a PC and saved as a WAVE file. Speech isrecognized using a speech recognition system called
Julius,5 followed by facial expression synthesis of theagent using preset parameters depending on each vowel.
Facialexpression
data
were
createdwith
MikuMikuDance.7 In this study, in order to generate
more human-like agent facial expressions, facial
expression data were created for the vowels / a /, / i /, / u/, / e /, and / o / (Fig. 3).4 In order to create more natural
agent facial expressions, processing is then performed toinsert a neutral facial expression when the same vowel,
for example / a /, is continuous.4

/u/

/o/

Fig. 3. Facial expression of the agent in uttering each vowel.4

Figure 4 shows the flow of creating a facial
expression motion (see the paper4 to understand theprocessing described in Fig. 4 in more detail).
2.3. Navigations by improved agent
In the music-recommendation process, all user
navigations are performed by the synthetic voice of theagent, appearing on the PC screen facing the user. All
dialogue spoken by the agent is situationally selected by
the proposed system. The user’s answers to the questions
generated by the agent are recognized using the voicerecognition function of the system, and the agent motions,
including facial expressions, are then generated.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

29

Music Recommendation System Driven

selected a CD9 described as an anthology of older songs
that are enjoyed by older people with dementia, andselected (1) 52 songs on the CD that were also included
in a musical textbook database10 for elementary schools,and (2) 58 other popular songs on the CD.
The experiment was performed on a Dell Inspiron
15 PC, equipped with an Intel Core i7-6700HQ 2.60
GHz central processing unit (CPU) and 8.0 GB of
random access memory (RAM). The Microsoft VisualC++ 2010 Express and Visual C++ 6.0 were used as thedevelopment languages.
To evaluate the proposed method described in
Section 2, 10 males (Subjects A to G in their 20s, I in his
40s, J in his 50s, and L in his 80s) and 2 females (Subject
H in her 20s and K in her 50s) participated in theexperiments. Each of the subjects navigated the musicrecommendation process of the agent. The upper limit for
the number of songs recommended by the proposedsystem was set to be 15. The experiment was performed
separately using the database consisting of the 52
children’s songs, and that consisting of the 58 popularsongs, with two experimental conditions for each
(Condition 1: with the input to the system being
performed by a supporter instead of the agent navigating
the system, and Condition 2: with the agent navigating
the proposed system). Following the experiment, allsubjects were requested to select one of five [evaluation
value] answers ([5] absolutely yes, [4] yes, [3] I can’t sayeither, [2] no, [1] absolutely no) to seven questions (Table
1) in order to evaluate the proposed system. The subjects
were also asked to provide comments on the system.

Reading audio data from Julius out file↓Calculating the total number of bones andwriting it to the file header.
↓
Reading facial expression dataaccording to vowel uttering time↓
Change the vowel uttering time to the number of frames andwriting corresponding data to the vmd fileFig. 4. Flow of facial expression motion creation.4

(a)

(b)

Fig. 5. Snapshots in the respective reactions of the agent after
recognizing: (a) a positive answer, and (b) a negative answer.

Figure 5 shows two snapshots of the reaction of theagent after recognizing: (a) a positive answer, i.e., the user
wishes to listen to the recommended song again in the
future, and (b) a negative answer, i.e., the user does notwish to listen to the recommended song in the future. In thecase of (a), the agent nods twice and raises the corners ofthe mouth slightly, while in the case of (b), the agent also
nods twice, but instead lowers the corners of the mouthslightly. Figure 6 shows a snapshot of music
recommendation being performed by the proposed system.

Table 1. Questionnaire to evaluate the proposed system.
No.
Question
Was music-recommendation on condition 2 smoother1
than that on condition 1?
2
Were explanations by the agent easy to understand?
3
Were dialogues with the agent natural?
4
Were movements of agent mouth natural?
Were agent’s reactions natural after recognizing
5user’s positive answer for listening to the justrecommended music again in the future?
Were agent’s reactions natural after recognizing
6
user’s negative answer of no more he justrecommended music from now on?
7
Did you feel enjoyable in using the proposed system?

Fig. 6. Snapshot in performing song-recommendation by the
proposed system.

3. Experiment3.1. Conditions
Since older people tend to prefer children’s songs,8 we© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

30

Ayumi Matsui, Miki Sakurai, Taro Asada, Yasunari Yoshitomi, Masayoshi Tabuse

3.2. Results and discussion

added a new function that changes the facial expressionof the agent according to the responses of the users tomusic recommended by our previously proposed system.
The effectiveness of the proposed system was verified.

Table 2 shows the average-values for each question
listed in Table 1. The mean value of the averages listed
in Table 2 was 4.1, suggesting a positive overall
evaluation of the proposed system. For questions 5 and7, the evaluation averages were very large, while thoseof questions 1 and 4 were lower.
Table 2. Evaluation of the proposed system.
Question no.
1
2
3
4
5
6
Average
3.6 4.4 4.0 3.6 4.5 4.3

Acknowledgements
The authors would like to thank Professor J. Narumoto,of the Kyoto Prefectural University of Medicine, for his
valuable support and helpful advice during the course ofthis research. We would also like to thank the subjectsof our experiments for their cooperation. This research
was supported by COI STREAM of the Ministry of
Education, Culture, Sports, Science and Technology ofJapan.

7
4.5

Table 3 shows the average values for each gender(M=male, F=female) and age group for each of thequestions listed in Table 1. For Male Subject L, who
was in his 80s, the evaluation values for questions 1 and2 were very low compared with those of other subjects.
Subject L described in his comments that the agent
spoke too rapidly for him to understand its statements,
and that it was difficult to input his voice using the
microphone. Three male subjects (D and E in their 20s
and Subject J in his 50s) also described in theircomments that it was difficult to input their voices using
the microphone. Consequently, speech recognition for
Subjects D, J, and L was not performed smoothly by the
proposed system, resulting in inputting voice again after
training the system. Nearly all of the comments by the
subjects expressed positive evaluation of the proposedsystem, except in relation to voice input to the system.

References
1. C. Koro, Y. Yoshitomi, T. Asada, and S. Yoshizaki,
Music recommendation aimed at improving recognition
ability using collaborative filtering and impression words,
in Proc. 17th Int. Symp. on Artificial Life and Robotics,
ed. M. Sugisaka (Japan, Beppu, 2012), pp. 222–225.
2. S. Yoshizaki, Y. Yoshitomi, C. Koro, and T. Asada,Music recommendation hybrid system for improving
recognition ability using collaborative filtering andimpression words, J. Artif. Life and Robotics 18(1-2)(2013) 109–116.
3. Y.Yoshitomi, T.Asada, R.Kato, Y.Yoshimitsu, M.Tabuse,
N.Kuwahara, and J.Narumoto, Music recommendation
system through Internet for improving recognition ability
using collaborative filtering and impression words, J.
Robotics, Networking and Artif. Life, 2(1), 2015, 54–59.
4. T. Asada, R. Adachi, S. Takada, Y. Yoshitomi, and M.
Tabuse, Facial expression synthesis using vowel
recognition for synthesized speech, in Proc. 2020 Int.
Conf. on Artificial Life and Robotics, ed. M. Sugisaka(Japan, Beppu, 2020), pp.398–402.
5. Julius, http://Julius.osdn.jp/, Accessed 24 November 2020.
6. MMDAgent, http://www.mmdagent.jp/ Accessed 24
November 2020.
7. MikuMikuDance, https://sites.google.com/view/vpvp/,
Accessed 29 November 2020.
8. T. Takahashi, Research report on songs familiar to people
advanced in years (in Japanese), J. Japanese music
therapy associate 15(1) (1997) 68–75.
9. T. Akahoshi, Good old anthology enjoyable for people
advanced in years and troubled with dementia (inJapanese) ( Kirara shobo, Tokyo, 2009).
10. Music textbook database for elementary school by
Kanagawa prefectural education center (in Japanese),
http://kjd.edu-ctr.pref.kanagawa.jp/daizai_music/, Accessed
27 November 2020.

Average

Table 3. Evaluation of the proposed system by each age group.
Question no.
1
2
3
4
5
6
7
M-20s (n=7)
3.7
4.7 4.4 3.6 4.6 4.3 4.1
F-20s (n=1)
4
5
3
4
5
5
4
M-40s (n=1)
3
4
4
3
4
4
4
M-50s (n=1)
4
4
4
4
4
4
5
F-50s (n=1)
5
5
5
5
5
5
5
M-80s (n=1)
2
2
4
4
4
4
5

4. Conclusions
We proposed a system for music recommendation,
characterized by an interaction between a user and apersonified agent that uses speech recognition,
synthesized voice and facial expression generation.
Speech is recognized using a speech recognition systemcalled Julius, and the facial expression of the agent isthen synthesized using preset parameters that depend oneach vowel. We used MMDAgent to create the agent.
To produce the voice of the agent, we used the built-in
speech synthesis function setting in MMDAgent. We© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

31

Wallet Operation Evaluation System Using Deep Learning
Junichiro Yamawaki1, Yasunari Yoshitomi2, Masayoshi Tabuse2, Taro Asada2

1: SKY Co., Ltd.,3-4-30 Miyahara,Yodogawa-ku, Osaka 532-0003, Japan
2: Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: {yoshitomi, tabuse}@kpu.ac.jp, t_asada@mei.kpu.ac.jp
http://www2.kpu.ac.jp/ningen/infsys/English_index.html

Abstract
As the average age of Japan’s population increases, as a method for investigating mild cognitive impairment (MCI),
wallet operation evaluations have been receiving considerable attention recently. Herein, we propose a system for
conducting wallet operation evaluations based on deep learning. In our system, the bills and coins extracted from awallet are automatically scanned and recognized, which makes it possible to evaluate a person’s ability to correctly
select and extract the correct bills and coins from the wallet within a reasonable period.
Keywords: Mild cognitive impairment, Dementia, Wallet operation evaluations, Deep learning

records the results. This process can be broken down
into the following six operational units:

1. Introduction
The average age of Japan’s population is increasing, andthis trend is expected to continue. As a result, thenumber of elderly persons suffering from dementia is
growing rapidly, and it has become necessary to findways to identify persons suffering from mild cognitive
impairment (MCI)1 in order to ensure they have propercare while suppressing the progression of the disease.
As a method for investigating MCI, which is one ofthe pre-stages of dementia, wallet operation evaluations
have been receiving considerable attention recently.2
However, in the normal course of such testing, the test
subject must be observed by an examiner who provides
instructions on how the evaluation is conducted,
observes the testing, and record the results.2
In contrast, this paper reports on an automatic
evaluation system in which deep learning is used to
conduct wallet operation evaluations automatically.

(1) The examiner hands the test subject a walletcontaining five (¥1, ¥5, ¥10, ¥50, and ¥100) Japanesecoins each, five ¥1,000 Japanese bills, and one(¥2,000, ¥5,000, and ¥10,000) Japanese bills each , aspartially shown in Fig. 1. (In the wallet, the total
number of coins is 25, while that of bills is eight.)(2) Next, the test subject is shown a sheet of paper
describing two trials for 10 seconds. In the first trial,
he or she is instructed to extract ¥1,165 from thewallet and place the money on the desk. The second
trial is the same but the amount is ¥123.
(3) The test subject is instructed to begin the trials
described in (2) when the examiner presses the
button on his or her stopwatch.
(4) The test subject performs the two tests and reports
his/her completion as instructed by the examiner.
(5) After visually confirming that both operations havebeen completed and receiving the test subject’s report,
the examiner presses the stopwatch button andinforms the test subject that the evaluation is over.
(6) The time required by the test subject to complete
both the first operation and second operation is thenrecorded by the examiner.

2. Wallet Operation Evaluations
During a conventional wallet operation evaluation,2 a
subject is tested by an examiner who instructs him or
her on how to proceed, observes the process, and then

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

32

Junichiro Yamawaki, Yasunari Yoshitomi, Masayoshi Tabuse, Taro Asada

2 is set in front of the test subject. The processing flowof the second part of the proposed system consists of thefollowing 10 processing units:(1)Japanese explanations of the evaluation process (three
pages) are displayed sequentially on a personal
computer (PC) screen, as shown in Fig. 4. The test
subject indicates when he/she has finished reading eachpage by pressing the button connected to the PC.
(2)The PC then displays a 10-second message showing thetwo monetary amounts (first: ¥1,165, second: ¥123) thetest subject will be requested to extract from the wallet.
(3)The evaluation begins when the test subject presses the
button. At that point, the web camera and the system
begin recording the elapsed time.
(4)The test subject extracts the appropriate bills and coinsfrom the wallet and then presses the button again when
he/she desires to report that an amount of money equalto ¥1,165 has been placed on the desk plate as instructed.
(5)An image showing the present state of the desk plate isthen uploaded into the system.
(6)Next, the test subject proceeds to extract ¥123 fromthe wallet and place it on the desk plate. Note that a
condition of this test portion is that the test subject is
instructed not to use the coins extracted for the¥1,165 test portion.
(7)The test subject then signifies the completion of that
portion of the text by pressing the button again.
(8)An image showing the present state of the desk plate is
uploaded into the system.
(9)Using object recognition, the total amount of money
extracted from the wallet by the test subject, and thetime required for both test segments are calculated by
the proposed system.
(10)Finally, the time test and object recognition results are
displayed on the PC screen, as shown in Fig. 5.

Fig. 1. Some of objects used in wallet operation evaluation.

3. Proposed System and Method
Our proposed system consists of two parts: (1) datapreparation and deep learning and (2) the wallet
operation evaluation module. Figure 2 shows theprocessing flow of the first part of this system, which
consists of four processing units:(1) Obtaining object image data
(2) Data augmentation(3) Annotation(4) Deep learning
Obtaining object image data↓
Data augmentation↓
Annotation↓
Deep learning
Fig. 2. Processing flow of data preparation and deep learning.

As objects to be recognized, we use ¥1, ¥5, ¥10,
¥50, ¥100, and ¥500 Japanese coins and ¥1,000, ¥2,000,¥5,000, and ¥10,000 Japanese bills. For dataset
assembly and recording the bills and coins during actual
testing, we use a web camera set 40 cm above a desk
plate to obtain object images, as shown in Fig. 3.

Fig. 3. Example of object image.

Fig. 4. Japanese language explanations of the evaluation
process as displayed on the PC screen in order of upper left,
upper right, and lower left. The bottom right photos show the
button used for changing the display and recording elapsedtime.

In this study, we use a region-based convolutional
neural network (R-CNN) known as Faster R-CNN3 fordeep learning and object recognition.
At the beginning of the evaluation, the same walletcontaining the same coins and bills described in Section© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

33

Wallet Operation Evaluation System

Fig. 6. Original image (left) and augmented image (right).

The experiments were performed under two
conditions listed below:
Condition 1: with the conventional method2 describedin Section 2.
Condition 2: with the proposed system described in
Section 3.

Fig. 5. Calculated evaluation results (upper) and object
recognition results (lower left: ¥1,165, lower right ¥123).

After the experiments, all of the test subjects wererequested to select one of five answer-choices ([5]definitely yes, [4] yes, [3] no difference, [2] no, [1]definitely no) for each question in Table 1 in order toevaluate the proposed system.

4. Experiment4.1. Conditions
The experiment was performed on a PC equipped withan Intel Core i7-6700K central processing unit (CPU), a
GeForce GTX 1080 graphics processing unit (GPU),and 32.3 GB of random access memory (RAM). The
Ubuntu LTS18.04 64-bit operating system (OS) was
installed on the PC and Python 3.7.3 was used as thedevelopment
language.
Jupyter
Notebook,4
ImageDataGenerator,6
labelImg,7TensorFrow,5
CUDA10.0,8 cuDNN,9 and OpenCV10 were used as
toolkits and libraries.
For moving image input, a stream web camera wasused. ImageDataGenerator was used for data
augmentation, while rotation, vertical shift, horizontal
shift, expansion, shrink, and channel shift were used asprocessing. Next, labelImg was used for annotations.
For deep learning, the TFRcode11 for the
faster_rcnn_resnet_101_coco model in TensorFlow11
was generated using a program obtained from Ref. 12.
The mean average precision (MAP) obtained by thelearning model used in the present study was 0.755.
The resulting dataset contained 1,200 images
showing the wallet and/or the Japanese bills and coinsused in our experiment. Using ImageDataGenerator fordata augmentation, 12,000 images were generated fromthe above 1,200 images. 10,000 of those images wereused as learning data and the remaining 2,000 images
were used for the evaluations. Figure 6 shows anexample of data generated by augmentation.
To evaluate our proposed system, nine males(Subjects A to I) and one female (Subject J), all ofwhom are in their 20s, participated in our experiments.

Table 1. Questionnaire to evaluate the proposed system.
No.
Question
Was the wallet operation test on condition 2 easier1
than that on condition 1?
Did you feel less stress during the wallet operation2
test on condition 2 than on condition 1?

4.2. Results and discussions
Table 2 shows the average time measured for the 10
subjects during these experiments. Here, it can be seenthat the time required for the wallet operation evaluation
using our proposed system was slightly longer than thatof the conventional method. This might be caused by
the additional button-pushing operations required during
the process of our proposed method.
However, in both the conventional and proposed
methods, none of the test subjects exceeded the limit
time (120 s) that was assigned for the wallet operation
evaluation in this experiment. This was thought to bebecause all subjects were in their 20s and in goodmental and physical condition.
Table 2. Comparison of the average time of wallet operation
evaluation between the conventional and proposed methods.
Conventional method
Proposed method

Average time (s)
32.2
34.8

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

34

Junichiro Yamawaki, Yasunari Yoshitomi, Masayoshi Tabuse, Taro Asada

Table 3 shows the average questionnaire score forthe 10 subjects. Here, it can be seen that the proposed
method was better than the conventional method fromthe viewpoints of ease of use and stress reduction.
For all of the objects on the desk plate, the averagecorrect recognition rate by the proposed system was
80% for ¥1,165 and 90% for ¥123. Additionally, ¥100
coins were sometimes misrecognized as ¥500 coins (Fig.
7) and ¥1 coins were sometimes misrecognized as ¥100
coins, possibly due to light reflection. However,
mistakes in both cases were rare.
As our future targets for improving our proposedsystem, we intend to perform the wallet operation
evaluation with the participation of elderly persons who
may have MCI or other forms of dementia. Moreover,
we will reduce potential misrecognitions of the ¥1, ¥100,and ¥500 coins, primarily by making improvements tothe lighting conditions and/or the learning dataset.

extractions from the wallet are made within a reasonable
period.
Acknowledgments
The authors would like to thank Professor J. Narumotoof the Kyoto Prefectural University of Medicine for his
valuable support and helpful advice during this research.
We would also like to thank the test subjects of ourexperiments for their cooperation. This research was
supported by COI STREAM of the Ministry of
Education, Culture, Sports, Science and Technology(MEXT) of Japan.
References
1. B. Winblad, K. Palmer ， M. Kivipelto ， V. Jelic ， L.
Fratiglioni, et al., Mild cognitive impairment – beyond
controversies, towards a consensus: report of theInternational Working Group on Mild Cognitive
Impairment, J Int. Med., 256(3), 2004, pp. 240–246.
2. R. Senba, K. Kamijo, A. Kubo, and S. Murata,
Comparison of wallet operations of community-dwellingelderly people by cognitive function category (inJapanese), Japanese J. Clinical Occupational Therapy,
3(1), 2016, pp. 5–9.
3. S. Ren, K. He, R. Girshick, and J. Sun, Faster R-CNN:
towards real-time object detection with region proposal
networks, in Advances in Neural Information Processing
Systems 28 (NIPS 2015), eds C. Cortes and N. Lawrence
and D. Lee and M. Sugiyama and R. Garnett, 2015.
4. Jupyter Notebook, https://jupyter.org/, Accessed 6
December 2020.
5. TensorFrow, https://github.com/tensorflow/, Accessed 6
December 2020.
6. ImageDataGenerator, http://keras.io/preprocessing/image/,
Accessed 6 December 2020.
7. labelImg, https://github.com/tzutalin/labelImg, Accessed
6 December 2020.
8. CUDA,
https://developer.nvidia.com/cuda-downloads,
Accessed 6 December 2020.
9. cuDNN, https://developer.nvidia.com/cuDNN, Accessed
6 December 2020.
10. OpenCV, https://opencv.org/, Accessed 6 December
2020.
11. TensorFlow model zoo,
https://github.com/tensorflow/models/blob/master/researc
h/object_detection/g3doc/detection_model_zoo.md,
Accessed 27 February, 2020.
12. https://github.com/EdjeElectronics/TensorFlow-ObjectDetection-API-Tutorial-Train-Multiple-ObjectsWindows-10, Accessed 27 February, 2020.

Table 3. Average questionnaire scores for 10 subjects.
Question No.
Average score
1
4.2
2
4.1

Fig. 7. Misrecognition of a ¥100 coin that was identified as a¥500 coin.

5. Conclusion
As the number of elderly persons in Japan’s super-aging
society continues to increase, it is becoming
increasingly important to find ways to identify persons
suffering from MCI, which is one of the pre-stages ofdementia, in order to ensure they receive proper careand make efforts to suppress the progress of the disease.
As a method for investigating MCI, wallet operation
evaluations have been receiving considerable attention
recently.
Herein, we proposed a wallet operation evaluation
system based on deep learning. In our system, a walletcontaining bills and coins is supplied to the test subject,and his or her ability to extract bills and coins in thecorrect amounts is automatically recognized. The
proposed system also determines whether the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

35

Architecture of a student training computer program for preparing professional outpatient
consulting skills within an electronic medical records system during COVID-19 alertness
situation
Sergey Bulatov1, Evgeni Magid2, Enzhe Kharisova1, Roman Lavrenov2, Vitaly Dudin1, Artur Khazetdinov2

Center of a practical skills, Kazan State Medical University, 49 Butlerov street, Kazan, Russian Federation
Laboratory of Intelligent Robotic Systems (LIRS), Intelligent Robotics Department, Institute for Information
Technology and Intelligent Systems, Kazan Federal University, 35 Kremlyovskaya street, Kazan, Russian Federation
1

2

Abstract
One of the important goals of a medical education in the context of the COVID-19 pandemic is a broad introductionof simulated-based teaching methods, which might include elements of robotics and artificial intelligence. We
analyzed computer programs that are currently used for maintaining medical records of patients by various polyclinics
of Kazan city. Based on these results, we summarized requirements for a training computer program that could provide
students with medical records maintaining skills and developed program’s preliminary architecture. Our main
attention focused on modeling situations that are associated with pre-hospital stage processing of medical records forpatients with suspected or confirmed COVID-19.
Keywords: medical education, computer program, COVID-19 infection

1. Introduction
The current epidemiological situation led to a transitionof a higher medical education to distance learning1.
While a main pedagogical emphasis moved to
information technologies applications for educational
process, including computer programs and artificial
intelligence2,3, specifics of medical information systems(MIS) do not allow all students to master them
independently. A preliminary training of students towork with computer MIS used in practical health carehas become acute.
The COVID-19 pandemic, which already had
infected almost 60 million people all around the world
and took over 1.4 million lives just within less than a
year, has been a contributing factor in this process as itchanged a traditional doctor-patient relationship andreplaced face-to-face communication with a remotemonitoring of patients4. In local polyclinics, computer
programs create electronic medical records (EMR) of
patients, draw up sick lists, monitor laboratory tests, andensure preservation of information about patients'
treatment. In literature sources, the authors had foundonly isolated reports about special training of students towork EMRs and MIS5.

2. Related work
In medical universities of Russia, a special training of
graduate students to work with MIS of a regional
polyclinic is still experimental. Programs that are usedin multi-specialty hospitals for inpatient patients might
be employed as educational materials in universities,e.g., MIS "Aurora" for multi-specialty hospitals6
provides a training interface "UMS University" forindividual tasks, with a separate entrance for students
and teachers, and an access to a server via the Internet.
The authors6 reported a positive impact of the training
course on preparing students to work with a MIS.
In regional polyclinics of Kazan city various
commercial MISs are used, including BARS.MED,
ERP-system, 1C: Medicine (Registry), and VITACORE.
These systems have centralized databases with a secureaccess for each user through standard operating
environments (Microsoft, Linux) and a Web browser.
Yet, application of training versions of these programsfor student training was not reported.
In an international literature, an equivalent of a MIS
is an EMR, which combines all types of MISs. A
positive effect of using EMR in training of higher
medical schools’ students was established in7,8,9 as an© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

36

Sergey Bulatov, Evgeni Magid, Enzhe Kharisova, Roman Lavrenov, Vitaly Dudin1, Artur Khazetdinov

important part of an education, while students, in turn,reported that mastering EMR had a positive role inpreparing them for a future profession10,11.
Today, a unified pedagogical approach of teaching
students to work with EMR/MIS does not exist9. We
believe, the reasons are related to a commercial natureof such products, which prevents their free distribution,
as well as an absence of a single EMR/MIS that isrequired for an obligatory use by a local government.
Therefore, in order to facilitate medical education, local
teams of educators create unique training computer
programs that simulate real MIS behavior to support thenecessary training. Main characteristics of such training
programs (in order to provide an acceptable level ofsimulation) are their similarity to existing commercial
products, a capability to evaluate student's actions andto help him/her during a task execution. This paper
presents a preliminary architecture and interface for the
MIS training program for students of higher medical
schools. As an example, we discuss a consultation of an
outpatient patient in a polyclinic with signs of COVID19 infection.

patient information is stored in the database and can bereused later by the student at the next stages.
3.2 Analysis
After making a preliminary diagnosis, in accordancewith a treatment process algorithm, the patient should be
assigned the necessary laboratory and instrumental
assessment methods. It is necessary to select an optimal(strictly regulated by current standards) set of methods
among proposed ones, which is performed by clicking
interactive buttons on the right side of the screen. Next,
the student can see all the results in a form of an actual
data from a laboratory: a blood test, a CT scan of lungs,
an electrocardiogram (Fig. 2).

3. Program architecture and interfaces
3.1 Situation modelingCollecting information from a patient and conducting an
objective examination of the patient comprise apreliminary stage of work. The stage begins fromcreating am instance for a patient database and fillingout a corresponding EMR. A student manually enters
his/her name, date and time of a consultation, and apatient's name to open active fields, which consist ofcomplaints at the time of examination, medical history,
patient's life history, and objective examination data
(Fig. 1).

Fig. 2. Laboratory assessment methods.

Fig. 3. Consultation and clinical diagnosis.

Fig. 1. Collecting information from a patient.

The stage ends with establishing a preliminary
diagnosis and entering it in the corresponding field. The

The student should identify existing deviations fromnormal ranges and mark them. The next step is to consult
the patient with narrow profile specialist(s), which could
be selected from a predefined list. In accordance withcurrent medical standards, for certain diseases, these
could be a cardiologist, an endocrinologist, a surgeon orother specialists (Fig. 3). The stage ends with creating aclinical diagnosis and recording it into a speciallydesignated field.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

37

Architecture of a student

3.3 Creating a treatment program
On the next stage the student determines the patient's
treatment plan. This is a creative process associated witha specialist qualification. An interface of this page has
special fields where you the student can enter all detailsof the treatment. There is a field for prescribing
medications, a recommended regimen, a diet, and othertypes of treatment (Fig. 4). At this stage, the student has
an opportunity to refresh the patient informationcollected earlier and store it in the database.

Fig. 4. Treatment strategy.

3.4. Assessment
The student's actions could be assessed by two methods:
automatic (conducted by the program) and expert(conducted manually by a teacher). Based on theprescribed algorithm, the program evaluates acompleteness of the patient's examination. If the patient
information contains empty spaces, the program will
require to complete this work.

of the student's work is carried out by the teacher during
an individual discussion.
3.5 Example
Today, the problem of treating patients with COVID-19
symptoms at home is of a great social significance.
Every graduate medical student must study not only themain symptoms of this disease and know the treatment
program, but also take into account nuances ofmonitoring a potential COVID-19 carrier in an
outpatient setting. One of the goals of the program is tocombine disparate information (patient data, proposed
treatment and additional literature) within a single block
to allow transforming theoretical knowledge into
practical skills.
We constructed the following example case. The
student plays a role of a clinic’s entrance department
doctor. Patient K., 51 years old, came with complaintsof a high fever, a cough, and a general weakness. The
student’s task is to create an EMR for the patient anddetermine a medical treatment plan. The program
focuses the student's attention on the most typicalsymptoms of COVID-19. In the "patient’s complaints"
field main signs of the disease appear in a form of a list.
The student should ask the patient more carefully foreach symptom (Fig. 6). The preliminary diagnosis of
COVID-19 should be confirmed by appropriate
laboratory and instrumental assessment (Fig. 7, 8).

Fig. 6. Patient’s complaints.

Fig. 5. Mistakes of a student’s work.

The program uses color indicators (Fig. 5) as follows.
Red color indicates an error, yellow indicates deviations
from accepted standards, and green indicates the correct
actions. The program allows students to improveknowledge using the interactive buttons that appear near
the yellow and red fields. After pressing the button, a
student can visit a library in the database and read
regulatory documents and standards. A final assessment

Fig. 7. Laboratory assessment results.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

38

Sergey Bulatov, Evgeni Magid, Enzhe Kharisova, Roman Lavrenov, Vitaly Dudin1, Artur Khazetdinov

The program generates a list of diagnostic tests thatshould be used for patients with suspected COVID-19.
Next, the student should identify deviations from thenormal ranges in laboratory assessment results of thepatient. A correct decision is illustrated on the screenwith a green background. Errors are marked in red, andincomplete solutions are marked in yellow. In thissituation, a special symbol, which appears on the screen,
recommends visiting the "library" in the database andstudying the standards for laboratory and instrumental
assessment of patients with suspected COVID-19. After
successfully completing the first two stages and making
a clinical diagnosis, the student is asked to make acomprehensive treatment plan for a suspected COVID19 carrier at home (Fig. 8).

selecting its treatment, including a special case of
potentially infected with COVID-19 patients.
References
1. L. S. Schneiderand, M. L. Council (eds.), Distancelearning in the era of COVID-19, Arch. Dermatol. Res., 2020,
pp. 1-2.
2. S. Shah, S.Diwan, L. Kohan, D. Rosenblum, C. Gharibo,
A. Soin, A. Sulindro, Q. Nguyen and D. A Provenzano, The
technological impact of COVID-19 on the future of educationand health care delivery, Pain Physician, 23(4S), 2020, pp.
367-380.
3. A. Sagitov, H. Li, N. Schiefermeier-Mach and E. Magid,
Extending Gazebo simulator for surgical robotics: tissue andsuture modeling. In International Conference on Artificial Lifeand Robotics, 2019, pp. 124-127.
4. H. Lal, D. K. Sharma, M. K. Patralekh, V. K. Jain and L.
Maini, Outpatient department practices in orthopaedics amidst
COVID-19: The evolving model, J. Clin. Orthop. Trauma,
11(4), 2020, pp.700-712.
5. M. Samadbeik, F.Fatehi, M.Braunstein, B. Barry,
M.Saremian, F.Kalhor and S. Edirippulige, Education andTraining on Electronic Medical Records (EMRs) for healthcare professionals and students: A Scoping Review, Int. J. Med.
Inform., 142:104238, 2020, pp. 1-10.
6. S. I. Karas, I. O. Korneva, M. B. Arzhanik, O. L.
Semenova, E. V. Chernikova, O. V. Urneva and A. Y.
Grechishnikova, The role and prospects of information andcommunication technologies in developing of physician
competencies, Doctor and information technology (IDMZ),№4, 2018, pp.46-58.

Fig. 8. Final steps to create a clinical diagnosis.

4. Conclusions
The paper presented a preliminary architecture and aninterface for the MIS training program for students ofhigher medical schools. The program simulates mainstages of a polyclinic doctor work in monitoring andtreating patients at a pre-hospital stage, which opens up
opportunities for independent and distance education. It
allows creating EMRs and using a database withregulatory documents for outpatients’ treatment. The
proposed interface is similar to commercial software
products currently used in polyclinics of Kazan city,
Russia. The program develops practical skills of medical
students in modern approach of EMR processing using
artificial intelligence-based support for evaluation of
outpatient’s conditions, diagnosing a disease and7. N. M. Pageler, C. P. Friedman and C. A. Longhurst,
Refocusing medical education in the EMR era, JAMA, 310(21),pp. 2249-2250.
8. D. R. Cheng, T. Scodellaro, W. Uahwatanasaku and M.
South, An Electronic medical record in pediatric medical
education: survey of medical students’ expectations andexperiences, Appl. Clin. Inform, 9(4), 2018, pp. 809–816.
9. A. Rajaram, Z. Hickey, N. Patel, J. Newbigging and B.
Wolfrom, Training medical students and residents in the useof electronic health records: a systematic review of theliterature, J. Am. Med. Inform. Assoc., 27(1), 2020, pp. 175180.
10. C. Mountain, R. Redd, C. O'Leary-Kelly and K. Giles,
Electronic medical record in the simulation hospital: does it
improve accuracy in charting vital signs, intake, and output?,
Comput. Inform. Nurs., 33(4), 2015, pp. 166-71.

11. C. M. Yuan, D. J Little, E. S Marks, M. A Watson, R.
Raghavan and R. Nee, The electronic medical record andnephrology fellowship education in the united states: an
opinion survey, Clin. J. Am. Soc. Nephrol., 15(7), pp. 949-956.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

39

Satisfaction Assessment on the Counseling Service System for
Full-Time Teacher-Counselor in Tainan Elementary School
Hsiu-Hao Liu

The Ph.D. Program in Business and Operations Management, College of Management,
Chang Jung Christian University
No.1,Changda Rd.,Gueiren District, Tainan City, 711301, Taiwan
Yun-Syuan Jhang

Department Of Counseling and Guidance, National University of Tainan
No.33, Sec. 2, Shu-Lin St., Tainan City, 700301, Taiwan
E-mail: harry2356969@gmail.com, s16784@hotmail.com

Abstract
As the most important case management system in the school counseling work, the counseling service system mustmeet the needs and expectations of the school teacher-counselors (the main users). This investigation uses thequestionnaire survey method aimed at teacher-counselors in elementary schools in Tainan City to understand thedegree of teacher-counselors’ satisfaction when they use the counseling service system. The results mainly found that
counselor-teachers were most satisfied with “The items provided by the system website are completed and various.”
followed by “The items provided by the system website are necessary items meeting the needs for counseling work.”
while the least satisfactory item was “Using the system website helps me communicate and interact with others incounseling work.” In the five satisfaction aspects of the system, the perceived ease of use and the usage attitude were
found to be effective in explaining the system usage intention. Some practical implications of this research as well as
topics worthy of further research were discussed at the conclusion of the research.
Keywords: Perceived Usefulness, Perceived Ease of Use, Counseling Service System, Teacher-Counselor

1. Introduction
and other operations of the case. In actual use, theschool teacher-counselors find that they haveencountered difficulties and doubts in the operation of
the system, filling in items, and review procedures,and even ethical concerns about the leakage of
personal data, which may endanger the relevant rights
and protection of the case, and therefore also affect thesystem usage intention of the school teachercounselors.
This study is to explore the reasons why the school
teacher-counselors accept or reject this system. It is

The counseling service system is the most important
tool for case management in counseling work in the
junior high and elementary schools in Tainan City.
The purpose of the system design is to assist school
teacher-counselors to care for students and track their
psychological development, and hope that students
will have good adaptive growth. The school teachercounselors are the main users of using the system. In
addition to daily counseling affairs, the school teachercounselors also use the system to record thebackground information, counseling records, referrals© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

40

Hsiu-Hao Liu, Yun-Syuan Jhang

Counseling Service System" to conduct a survey,
including demographic characteristics (4 questions),
system satisfaction (18 questions) and system usage
Intention (2 questions) consists of three parts, which arein the form of a Likert four-point scale. The higher thescore, the higher the degree of satisfaction or system
usage intention.
This questionnaire has a very good internalconsistency reliability (α= .92). In terms of validity, 7senior experts and scholars in the relevant fields,
including information, counseling and education, andwith seniority between 6 and 20 years are invited to
conduct the survey. The content is appropriate for the
discussion, so the research tools used in the researchshould have content validity.

based on the Technology Acceptance Model (TAM)(Davis, 1989) and refers to Ahmad and Khalid (2017),
Elena and Detmar (1999), Luarn and Lin (2005),
Moore and McElroy (2012), etc., used the fivedimensions, these are system quality (SQ), system
response time (SRT), perceived usefulness (PU),perceived ease of use (PEOU) and usage attitudes(UA), as the main factors to explore the system usage
intention (SUI) of the school teacher-counselors.
The purpose of this study is to explore the current
situation of the school teacher-counselors using thecounseling service system, and to understand whether
the school teacher-counselors are satisfied with thesystem and the system usage intention due todemographic characteristics (seniority, system using
frequency, the average using time), there aredifferences depending on the difference, and then we
will further explore whether the school teachercounselors satisfaction with the system can effectively
explain the system usage intention, and follow-up
discussion based on the results.

Table 1. Experts and Scholars BackgroundA

University
Assistant
Professor
University
lecturer
Junior HighSchool TeacherCounselor
Junior HighSchool TeacherCounselor
Elementary
School TeacherCounselor
Elementary
School TeacherCounselor
Elementary
School TeacherCounselorC

The conceptual framework of this study is as follows:
System Satisfaction

Teacher-Counselors’ Seniority
System Using Frequency
The Average Using Time

Present Job

B

2. Research Design

Demographic Characteristics

No.

D

System Quality
System Response Time
Perceived Usefulness
Perceived Ease of Use
Usage Attitudes

E
F

System Usage Intention

G

Fig. 1 Conceptual Framework

2-1. Participant
The main objects of this research are full-time teachercounselors in Tainan Elementary Schools, with a totalof 90 teachers who voluntarily participated inanswering questionnaires. Finally, 20 teachers
submitted responses.

Areas ofexpertise
Information
Management

Seniority

Information
EngineeringCounseling& Education

8 years

Counseling& Education

11 years

Counseling& Education

19 years

Counseling

6 years

Counseling

6 years

10 years

20 years

3. Result
3.1

System Satisfaction Score

The results of the elementary school teachercounselors satisfaction items are as follows: "Theitems provided by the system website are completed
and various." (M=2.90), was the highest of all items.
"The items provided by the system website arenecessary items meeting the needs for counselingwork." (M=2.75) was secondly. The least satisfactory

2-2. Data Collection Instrument
This research was based on the self-compiled"Questionnaire on the Status of ProfessionalCounselors in Tainan Elementary Schools Using the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

41

Satisfaction Assessment on the

Table 4. Summary table of single factor varianceanalysis (System Satisfaction Dimensions)

item was "Using the system website helps mecommunicate and interact with others in counselingwork" (M=1.85). The second lowest is "I think thesystem helps me in the word processing of counselingwork is more efficient and convenient" (M=1.90).
From the perspective of the five aspects, except for thesystem quality, the other aspects have the lowest
scores, but none of the aspects have the highest scores.
Observing the average score of the aspect again, it canbe found that the school teacher-counselors are
somewhat dissatisfied with the degree of satisfaction
with the system

3.2

M

SD

MIN

MAX

15.65
4.55
4.30
8.20
8.60

1.79
1.43
1.42
2.48
2.42

11
2
2
4
4

18
7
6
12
12

3.3

MIN

MAX

4.15

1.50

2

6

Demographic Characteristics and System
Usage Intention

Dimension

Seniority

SUI

1.48

3.5

Usage
Frequency
2.48

Average
Use Time.00

System Satisfaction and System Usage
Intention

System satisfaction and system usage intention. are
related to the Pearson product difference. It is foundthat there is a high positive correlation between thetwo (r= .89, p< .01), and four of the five dimensions
are significantly correlated (r = .71~ .87, p<.01).
Only the system reaction time and the willingness to
use the system are not significantly correlated (r=.33,
p> .05). It shows that in the usage intention of theschool teacher-counselors to use the system,compared with other aspects, the system responsetime is not the main consideration.

Table 3. Descriptive Statistics of System Usage
Intention
SD

.61
1.13
1.61
2.03
.59

Average
Use Time
.50
.02
.04
.42
.02

Table 5. Summary table of single factor varianceanalysis (System Usage Intention)

In terms of the performance of the school teachercounselors in elementary schools, from the highest,
lowest and average scores, it can be found that school
teacher-counselors have a little low system usage
intention.

M

SQ
SRT
PU
PEOU
UA

Usage
Frequency
2.85
1.22
1.89
2.16
1.51

A single-factor analysis of variance found that thesystem usage intention of the elementary school
teacher-counselors does not depend on seniority(F=1.48, p> .05), usage frequency (F= 2.48, p> .05)
and average use time (F= .00, p> .05).

System Usage Intention Score

Dimension /Full marks
SUI/8

Seniority

3.4

Table 2. Descriptive Statistics of System Usage
Satisfaction
Dimension /Full marks
SQ/24
SRT/8
PU/8
PEOF/16
UA/16

Dimension

Demographic Characteristics and System
Satisfaction

Table 6. Correlations between the Variables of theStudy

After performing a single-factor variance analysis, it
is found that the system satisfaction, system quality,
system response time, perceived usefulness, perceived
ease of use and usage attitudes of the school teachercounselors.
There will be no difference due to age, frequency ofuse and average use time.

SQ
SRT
PU
PEOF
UA
SUI

SQ
-

SRT
.55*
-

PU
.61**
.43
-

PEOF
..52*
.38
.89**
-

UA
.66**
.20
.76**
.81**
-

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

42

SUI
.71**
.33
.87**
.83**
.85**
-

Hsiu-Hao Liu, Yun-Syuan Jhang

guidance and communication when the system waslaunched, and no good consultation channels were
provided. What is more worthy of discussion is the
management of information security. Since school
counseling work has a fairly strict principle ofconfidentiality, the use of any information is regulated
by law.
From this result, it can be found that the school
teacher-counselors have great concerns about data
preservation or management.
In order to explore the critical factors of the school
teacher-counselors system usage intention, thisresearch will conduct qualitative research, integrate
the results, and provide the results to relevant agencies
to improve the counseling service system.

**p< .01，*p< .05
Carry out multiple regression analysis based on thefive dimensions of system satisfaction and system
usage intention, and use stepwise regression as themethod of regression model to screen predictive
variables. It is found that the two dimensions ofperceived usefulness and usage attitude in the degreeof system satisfaction can effectively explain thesystem usage intention (multiple R²= .84, adjusted
r²= .82, p< .01). It shows that there may be key factors
in the system usage intention of the school teachercounselors in the two aspects.
Table 7. Summary table of multiple regression
analysis coefficient
Selected forecast
variables and
order
PU
UA

Multiple
R²

ΔMultiple
R²

ΔFvalue

.76
.84

.76
.08

56.37**
8.74**

1.

Based on the above results, it can be found thatdifferent seniority, usage frequency, average use time,the elementary school teacher-counselors are moredissatisfied with the degree of system satisfaction.
Similarly, elementary school teacher-counselors withdifferent background variables tend to be less system
usage intention. In terms of this result, it can be seenthat the current counseling service system is nottrusted by the elementary school teacher-counselors,and it cannot help the school teacher-counselors to
improve their work efficiency or improve the timeconsuming trouble of paper processing. It is difficult
to obtain the expected benefits in practice.
The main factors influencing the system usage
intention are related to the two dimensions ofperceived ease of use and usage attitudes. From theitems of the two dimensions, it can be found that there
may be difficulties in learning to operate thecounseling service system, and the security of thesystem cannot be trusted by the school teachercounselors. These results show that there was a lack of2.

References**p< .01，*p< .05
4. Conclusion

3.

4.

5.

6.

Ahmad, S. Z., & Khalid, K. (2017). The
adoption of m-government services from theuser’s perspectives: Empirical evidence fromthe United Arab Emirates. International
Journal of Information Management, 37(5),367-379.
Davis, F. D. (1989). Perceived usefulness,
perceived ease of use, and user acceptance of
information technology. MIS Quarterly, 13(3),319-340.
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R.
(1989). User acceptance of computer
technology: A comparison of two theoreticalmodels. Management Science, 35(8), 982-1003.
Elena, K., & Detmar, W.S. (1999).The
psychological origins of perceived usefulness
and ease-of-use. Information ＆ Management,
35, 237-250.
Luarn, P., & Lin, H. H., (2005), Toward an
Understanding of the Behavioral Intention to
Use Mobile Banking. Computers in Human
Behavior, 21(6), 873-891.
Moore, K., & McElroy, J. C. (2012). The
influence of personality on Facebook usage,
wall postings, and regret. Computers in Human
Behavior, 28(1), 267- 274.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

43

Assess the Critical Factors for the Counseling Service System Usage Intention
Hsiu-Hao Liu

The Ph.D. Program in Business and Operations Management, College of Management,
Chang Jung Christian University
No.1,Changda Rd.,Gueiren District, Tainan City, 711301, Taiwan
E-mail: harry2356969@gmail.com

Abstract
As the most important case management system in the school counseling works, the counseling service system mustmeet the needs and expectations of the school teacher-counselor (the main users). Semi-structured interviews wereused to interview teacher-counselor who have experience in using system in Tainan. The data are recorded andsemantic analyzed after interviewing, summarize the critical factors that affect the usage intention of the teachercounselor. The conclusion was to discuss the implications of this research and suggest several future research issues.
Keywords: Usage Intention, Qualitative Research, Counseling Service System, Teacher-Counselor

1. Introduction
Researchers have used self-made questionnaires inprevious study. Obtained the result that the school
teacher-counselors have low system satisfaction andsystem usage intention under different backgroundchanges. After further analysis, it is found that the two
aspects of perceptual ease of use and use attitude willaffect the system use willingness. Based on the above
research background and qualitative research methods,
this study hopes to explore the critical factors that
affect the willingness of school teacher-counselors to
use the system, and provide suggestions to relevant
agencies for improving the system.

The counseling service system is the most important
tool for case management in counseling work in the
junior high and elementary schools in Tainan City.
The purpose of the system design is to assist school
teacher-counselors to care for students and track their
psychological development, and hope that students
will have good adaptive growth. The school teachercounselors are the main users of using the system. In
addition to daily counseling affairs, the school teachercounselors also use the system to record thebackground information, counseling records, referrals
and other operations of the case. In actual use, theschool teacher-counselors find that they haveencountered difficulties and doubts in the operation of
the system, filling in items, and review procedures,and even ethical concerns about the leakage of
personal data, which may endanger the relevant rights
and protection of the case, and therefore also affect thesystem usage intention of the school teachercounselors.

2. Research Methods
2-1. Research Design
The structure of qualitative interview questions can bedivided into: 1. Structured interviews, which refer tothe interview content and questions prepared inadvance, also known as standardized interviews; 2.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

44

Hsiu-Hao Liuview of the critical factors of the system usage
intention of the teacher-counselors:
Explore the main motivations and functions ofusing system.

Explore the attraction and resistance factors ofusing system.

Unstructured interviews, refer to the undeveloped
interview topics, from the focus of the research
participants, and ask questions according to the needsof the interview situation, also known as nonstandardized interviews 3. Semi-structured interviews,
between structured and unstructured, the interview
content will be prepared in advance, but the questions
will also be adjusted according to the subsequentsituation (Lincoln & Guba, 1985).
This research adopts semi-structured face-to-faceformal interviews. Through one-to-one in-depth
interviews with research participants, the content anddata of the interviews are analyzed, reviewed,
aggregated, and summarized to determine thedimensions of critical factors that affect the system
usage intention.

2-4. Data analysis
Before the interview, an interview outline should be
drawn up according to the research question andresearch purpose. After recruiting the research
participants, first send the interview outline to theresearch participants to discuss the direction andcontent of the interview. During the interview, thewhole process was recorded after seeking the consent
of the research participants.
After the interview, the recording files of eachresearch participant were compiled into a verbatim
manuscript, then the key points were sorted out andanalyzed, and the elements and core themes weresummarized. Finally, the interview interpretation was
written, and the interview interpretation was sent to theresearch participants In addition, the "ResearchParticipant Interview Interpretation Confirmation
Reply Form" is attached. The research participants are
asked to check whether the researcher's interpretationof the interview content is incorrect, and correct any
doubtful points for further discussion and clarification.
In order to interpret the experience and information of
the research participants, the original interviews wereinterspersed with the description of the thesis whenwriting the paper. When quoting the content of theinterview, if the entire conversation is quoted, it will
be presented in italicized block text; if it is a few
sentences, it will be framed with " " and changed toitalics. The source of the interview content is marked
after the quoted paragraph, such as A001: the first
English letter is the respondent code, and the last threenumbers are the paragraph number of the interviewed
content.
When quoting the entire conversation, in order tomake the theme more clear, when deleting someinterview sentences, "..." will be used. In order tofacilitate the smooth reading of the quotation, it will
be indicated by [ ] as the text added by the researcher.

2-2. Participant
In this study, purposive sampling was adopted, and"information-rich cases" were selected as research
participants, and a total of four "information-richcases" were selected as research participants.
The selection criteria for participants in this study are
as follows:
The current position is as a full-time teachercounselors in Tainan Elementary Schools(including public and private).

These teachers come from different schools andstill use the counseling service system.

These teachers are willing to share personal
experiences related to research topics.
Through the researcher's own workplace, directcontact with potential participants, a total of four
current full-time teacher-counselors who meet the
selection criteria and are willing to accept interviews
were invited. The basic information of the participants
is shown in Table 1.
Table 1. The basic information of the participants
No.
A
B
C
D

AnonymTea Bag
Little Buck Teeth
Agatsuma Zenitsu
Chibi Maruko Chan

Seniority
4 years
5 years
5 years
3 years

Interview Date
2020/11/3
2020/11/4
2020/11/6
2020/11/9

2-3. Research Tools
Based on the researcher’s previous questionnaire, thisstudy divided the interview content into two parts in© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

45

Assess the Critical Factors

3. Discussions and Suggestions
3.1

Suppose I fill in [the system], or if our senior director
or team leader fills in it, they may be less familiar withcomputer use and it will take longer to fill in. (D002)
Finally, we will discuss the positive and negative
opinions of participants on the system. As for theattraction factor, two participants thought that the useof systems would help their work. This is a Perceived
Usefulness dimension.
Because before you upload [the Case Record], inorder to write more detailed information, you must
spend some time to do some case review. I think this is
helpful for self-organizing and helping, that is, forself-organizing the case. (B005)
Another participant thought that the system responded
smoothly, which is the system response time
dimension.
For the resistance factor, the three participants said
that the review mechanism of the system is suspicious.
One participant reflected the inconvenience of thecountdown design and the lack of pre-teaching andpost-assistance in the system. All of the above can besummarized in the system quality dimension.
…These team leaders, directors, and principals haveseen those cases [records] in everything they
reviewed, …, we would not write them in such detail,because we have the principle of confidentiality andthere is a problem of funding. I just think why the
director, group leader, and principal would also see it?
If it's just a review, then what is the purpose of seeing
these things, and what is the purpose of these caserecords and case conceptualizations? (D010)
I think the countdown timer is very unfriendly. I think,I don’t know the purpose of its setting? Then again,
for the teacher who uses this system for the first time,
I think he is also relatively unfriendly, because he only
introduced to the teacher-counselors how to use thesystem at the beginning, and then I reviewed it again.
He actually put a teaching manual on it, but thecontent is trivial and complicated. (B007)
The four participants agreed that the system cannot
protect the safety of case data and that the uploading
behavior may cause personal information leakage.
This is the usage attitudes dimension.
As far as I know, this system is no longer specificallymanaged, especially his database for storing data. I
heard that there is no special person to manage it.
Then how do we know that after we fill in the

Discussions

The main purpose of this research is to explore thecritical factors of the system usage intention ofteacher-counselors. First of all, it can be known fromthe interview content that the main reasons for theparticipants to use the system are because of therequirements of the system or regulations, and there is
no choice. It is only used under circumstances; and thecase record is also because the information is passively
registered when a case needs to be referred.
It was because the Student Counseling Center asked
us to go to a meeting, and then announced that wewould all use the online record to fill in and use it.
(A001)…Because we were actually a crisis incident at thattime, we had to ask for the tertiary prevention
psychiatrist, which is the part of emergency
intervention in school services. Then, [the psychiatrist
in the school] he reminded us that we must go to thesystem, and then make such an application. (B001)
The tertiary prevention part seems to be the Student
Counseling Center. If you want to transfer thesecondary prevention case to the tertiary prevention.
They stipulate that there must be a record that is
uploaded. (C002)
Secondly, the background variables of the teachercounselors are also one of the factors that affect theusage intention. Three participants thought that thefrequency of use and the average time of each use
would affect the usage intention, because they may
become more proficient or unsatisfied as they learn
more about the functions of the system.
The reason for frequency of use is closer to the reason
for using average time. It refers to the people who use
it frequently. He should know more about his
advantages and disadvantages. If there are many
disadvantages, then of course the less satisfied. If
there are many advantages, of course the moresatisfaction. (A005)
In addition, two participants believed that age or
seniority is also one of the factors that affect the usage
intention. Senior teachers with older age or seniority
may be less familiar with operating information
systems.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

46

Hsiu-Hao Liu

be themselves, and they must be ethically confidential
and protected. Since they have designed this [system],
there should be a mechanism. (D012)

information of these cases, Will this information beleaked? Or will it be used? (A009)
As for the confidentiality of this [case records], itshould be the unit that requires us to report. They must
Dimensions

Demographic
Characteristics

System Quality

SystemResponse Time

Perceived
Usefulness

Usage Attitudes

Criterias

1.

1.

1.

1.

1.

2.

3.

Age and
Seniority
SystemUsing
Frequency
The Average
Using Time

Ensure thatthe audit
mechanism isreasonableandconfidential
2. Provide
sufficient time
to fill in
3. Provide
adequate pretraining andpost-event
assistance

Smoothsystemresponse2.

Provide aunified fileformatContributeto caseanalysisHave a goodmanagement
authority design
2. Ensure theconfidentiality ofcase informationand counsellingrecords

Table 2 The structure of the critical factors for the counseling service system usage intention
3.2

Suggestions

description and confidentiality when conducting
closed group interviews, so that participants can
express their own feelings without scruples.

Based on the above, the researcher sorted out thecritical factors of the system usage intention of theteacher-counselors. There are five main dimensions
and eleven criteria (see Table 2).
Compared with the results of previous questionnaire
surveys, the usage attitudes has been mentioned all thetime, which shows that this dimension requires special
attention. Past research can also confirm that there is apositive correlation between usage attitudes and usage
intention (Ahn et al., 2007; Juarez Collazo et al., 2014;
Wilson, 2019).
Finally, during the interview, many participants
mentioned their concerns about participating in thisresearch. Compared with other teachers, the teachercounselors are a closed group. If the upper-levelauthorities use this research to investigate the identityof the participants, it may affect the rights of manyteachers. Therefore, it is recommended that
researchers pay more attention to the research

References
1.

2.

3.
4.

Ahn, T., Ryn, S., & Han, T. (2007). The impact
of web quality and playfulness on user
acceptance of online retailing. Information andManagement, 44(3), 263-275.
Juarez Collazo, N. A., Wu, X., Elen, J., &Clarebout, G. (2014). Tool use in computerbased learning environments: Adopting andextending the technology acceptance model.
ISRN Otolaryngology, 2014, 1-11.
Lincoln, Y. S., & Guba, E. G. (1985).
Naturalistic inquiry. Beverly Hills, CA: Sage.
Wilson, N. (2019). The Impact Of Perceived
Usefulness And Perceived Ease-Of-Use Toward
Repurchase Intention In The Indonesian ECommerce Industry. Jurnal Manajemen
Indonesia, 19(3), 241-249.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

47

Design and implementation of EMI suppression filter
for electronic commutation fan systems
Ching-Chun Chuang 1, Chih-Chian Hua 2, Chung-Wen Huang2, Chun-Jen Yao3*

Department of Computer Science and Information Engineering, National Formosa University,
No. 63, Wunhua Rd., Huwei Township, Yunlin 632, Taiwan (R.O.C),
2
Department of Electrical Engineering, National Yunlin University of Science and Technology, No. 123, University Rd.,
Section 3, Douliu, Yunlin 640, Taiwan (R.O.C),
3
Department of Electronic and Computer Engineering, National Taiwan University of Scienceand Technology, No. 43, Section 3, Keelung Rd., Taipei 106, Taiwan (R.O.C),
1

E-mail: *austincc@nfu.edu.tw, huacc@yuntech.edu.tw, wenhung@yuntech.edu.tw, alanyao@mail.ntust.edu.tw

Abstract

An electronically commutated fan has a complex digital control mode compare to the conventional alternating current
fans. Since a serious electromagnetic compatibility situation was produced in the electronic commutation fan system, thefilter circuit of a fan system and its mechanical design was significantly important. Because the digital control strategy for
an electronically commutated fan was used to reduce the component of the EMI filter so the suppression of conducted andradiated electromagnetic interference will be discussed in this fan system. Moreover, the fan system can be combined withthe mechanical design for getting over a series of issues of radiated electromagnetic interference. Thus, fig.1 reveals thata single-phase BLDC motor driver schematic diagram. Eventually, a cause that electromagnetic interference was produced
from the electronically commutated fan will be analyzed and its suppression strategy also provided in this paper. The
experiment verification for the electronically commutated fan will be made available to the public.
Keywords：Electronically commutated fan, Electromagnetic compatibility, Digital control mode
1. Introduction
The electronic communication fan that combines thefrequency conversion function, and the power transistors
for frequency modulation to achieve speed control.
Because of the higher speed, the motor output efficiency
is improved. The Single-Phase BLDC motor system isdriven by AC power supply. A DC motor is used insideof this system, so the friction loss is reduced. Thus, theadvantages of AC and DC fans are combined to improvethe efficiency and life of the overall electronic
commutation fan. Fig. 1 show a commercial electronic
commutation fan product architecture [1]. A number ofswitch devices were switched at high speed, and theFig. 1. Single-Phase BLDC motor driver schematic diagram

voltage and current are changes at the same time. The
result not only cause more serious electromagnetic
interference problems [2] [3] [4], but also affected theelectromagnetic tolerance of the internal controller.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

48

Ching-Chun Chuang, Chih-Chian Hua, Chung-Wen Huang, Chun-Jen Yao

Therefore, an attention has to consider in the production
design for passing relevant electromagnetic compatibility
certification.
2. EMI Standard for Electronic Communication
SystemGenerally speaking, Electromagnetic Compatibility(EMC) is divided into two types of tests: Electromagnetic
Interference (EMI) and Electromagnetic Susceptibility(EMS) [5][6]. Electromagnetic interference testing refers
to measure of the intensity of conductive interference andradiant interference while the electronics device isnormal operated under testing. The purpose ofelectromagnetic interference suppression design is to
reduce the intensity of noise generated by the electronics
device under testing, and it was not to affect otherelectronic products in the test environment. On thecontrary, the purpose of electromagnetic tolerance is toprevent the DUT from being interfered with by otherelectronic products in the environment during operations
There are international standards for theelectromagnetic interference law of electronic
communication fans, which can refer to the European
regulations EN 55011, the United States FCC PART 18
and the domestic CNS 13803. Among them, there are twodifferent limit values of class A and class B based on thisproduct are used in commercial or residential households.
However, the products analysis in this paper will be used
in information products since Europe is the main market.
So, the information products for European standards will
be adopted. Class B electromagnetic interference
standard in EN55022, EN55022[7] and EN55011[8]were the same Class B electromagnetic interference test
standard.

Fig. 2 EMI testing of L Phase conductivity

Fig. 3 EMI testing of N Phase conductivity

3. EMI Suppression and Analysis of Electronic
Communication System

Fig. 4 The vertical polarity test of radiation EMI

3.1 Conduction EMI measurement
To measure the noise values of the live wire (Line,L) and neutral wire (Neutral, N) after a connection of the
LISN. The measure results are shown in Fig. 2 and Fig.
3. Observing the two figures, it can be seen that the test
value without any filter and other countermeasures far
exceeds the standard limit value.

Fig. 5 The horizontal polarity test of radiation EMI© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

49

Design and implementation of

3.2 Radiation EMI measurement
According to section, the noise measurement of the
EUT was received through the antenna under two
different polarities, vertical and polar. The actual test
diagrams are shown in Figure 4 and Figure 5. It can also
be seen that the noise radiated through space is very
serious.
3.3 The Suppression Strategy for Conduction EMI and
Radiation EMI
3.3.1 A Strategy of Conduction EMI
Conducted EMI is mainly used to measure the noisetransmitted by the power line through LISN, so the main
suppression method is to add an EMI filter to the powerline [13]. However, it can be seen that a multi-stage filter
with grounding is required to have a wider suppression
range from Fig. 2. In this paper, the space of the DUT islimited, and there is no ground wire architecture. The
input EMI filter architecture is adopted in the electronic
commutation fan systems. The differential mode
inductance is 100uH, the X capacitor is 0.1uF, and thecommon mode inductance is 20mH. Fig. 6 and Fig. 7
show the actual test patterns after adding the input EMI
filter and other countermeasures to the EUT internal
circuit.
 N × δ × fc Attenuation = 10 log 

fm



Fig.7 EMI testing of N Phase conductivity after suppression

3.3.2 A Strategy of Radiation EMI
In order to improve the radiation problem between
the circuit output and the motor, an output EMI filter
design is need in the electronic commutation fan systems.
The common mode inductance is 0.95mH and the X
capacitor is 4700pF.
The EUT in this paper has two specifications,
namely plastic fan blades and metal fan blades. Amongthem, the series of metal fan blades have the most serious
radiated EMI problem. The noise generated by the drive
motor will provide a path to the metal fan blade throughthe metal shaft, which will cause a serious antenna effect.
Therefore, an insulating part will be added to themechanism part, as shown in Fig. 8, which not only saves
the cost of motor manufacturing but also greatly
improves the problem of radiated EMI. Fig. 9 and Fig. 10
show the actual test patterns after EUT suppression.

(1)

where N is harmonic number𝛿𝛿 is spread spectrum
𝑓𝑓𝑐𝑐 is switching frequency
𝑓𝑓𝑚𝑚 is frequency modulation

Fig.6 EMI testing of L Phase conductivity after suppression

Fig. 8 A strategy for motor shaft and mechanism
insulation design© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

50

Ching-Chun Chuang, Chih-Chian Hua, Chung-Wen Huang, Chun-Jen Yao

References
1. S. Chen, T. A. Lipo and D. Fitzgerald,"Measurement and Analysis of Induction Motor
Bearing Currents in PWM Inverter Drives", IEEEPES Summer Meeting, July 1995.
2. J. Erdman, R. J. Kerkman, D. Schegel and G.
Skibinski, "Effect of PWM Inverters on AC Motor
Bearing Currents and Shaft Voltages", APEC '95,
vol. I, pp. 24-33, Mar. 1995.
3. Henry W. Ott, “Noise Reduction Techniques inElectronic Systems,” John Wiley & Sons, 1988.
4. C. R. Paul, “Introduction to Electromagnetic
Compatibility,” John Wiley & Sons, 1992
5. EN 55022, Information Technology Equipment Radio Disturbance Characteristics - Limits andMethods of Measurement, 2010.
6. EN 55011, Industrial, Scientific and Medical Radiofrequency Equipment – Radio DisturbanceCharacteristics – Limits and Methods of
Measurement, 1998.
7. S. Ogasawara, H. Akagi, “Analysis and Reductionof EMI Conducted by a PWM Inverter-fed AC
Motor Drive System Having Long Power Cables,”IEEE Power Electronics Specialists Conference,Vol. 2, pp. 18-23, 2000.
8. M. L. Heldwein, T. Nussbaumer, F. Beck and J. W.
Kolar, “Novel Three-phase CM/DM Conducted
Emissions Separator,” IEEE Appl. Power Electron.
Conf. Expo., pp. 797-802, March 2005.
9. R. L. Ozenbaugh, EMI Filter Design, Marcel
Dekker, New York, 1996.
10. J. Kim, P. Jun, J. Kim, “Dithered Timing Spread
Generation for Reduction ofSpectrum Clock
Electromagnetic Radiated Emission from Highspeed Digital System,” In Proc., IEEE Int. Symp.,
Electromagn. Compat., pp 413-418, 2002.
11. Hardin, Fessler and Bush, “A Study of theInterference Potential of Spread Spectrum Clock
Generation Techniques,” IEEE International
Symposium on EMC, August 1995.
12. A. Santolaria, J. Balcells, D. Gonzalez, J. Gago, andS. D. Gil,“EMI Reduction in Switched Power
Converters by Means of Spread Spectrum
Modulation Techniques,” Power Electronics
Specialists Conference, 2004. PESC 04. 2004 IEEE
35th Annual, vol. 1, pp. 292 – 296, June 2004.

Fig.9 The vertical polarity test of radiation EMI after
suppression

Fig.10 The horizontal polarity test of radiation EMI after
suppression

4. Conclusion
This paper proposed an electronic commutation fanthat is to perform the actual electromagnetic interference
certification process for the EUT. The suppression of
conductive EMI and radiate EMI was discussed, and adigital control system was to reduce the EMI filter
components in this research. Meanwhile, it combines themechanism to improve the design to overcome an issue
that the system without grounding was operated thatcauses a serious radiated EMI problem. This method can
not only be applied to this product, but also produced thesimilar effect for the other relative products. The goal thatsaves time and cost will be implemented after asuccessful certification of the subsequent relative
products© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

51

Application of IOT to Forest Management Taking Fushan Botanical Garden as An Example
Shuo-Tsung Chen1, Chih-Chiang Hua2, and Ching-Chun Chuang3*

Bachelor Program in Interdisciplinary, National Yunlin University of Science and Technology;123 University Road,
Section 3, Douliou, Yunlin 64002, Taiwan, R.O.C.
2
Department of Electrical Engineering, National Yunlin University of Science and Technology;123 University Road,
Section 3, Douliou, Yunlin 64002, Taiwan, R.O.C.
3
Department of Computer Science and Information Engineering, National Formosa University; No. 63, Wunhua Rd.,
Huwei Toshnship, Yunlin 632, Taiwan, R.O.C.

1

E-mail: shough34@yahoo.com.tw, huacc@yuntech.edu.tw, austincc@nfu.edu.tw

Abstract
In recent years, the Internet of Things (IoT) technology has developed rapidly and has been successfully applied indifferent fields, and has begun to expand the application context of the Internet of Things. This work aims to study
how to apply the IoT technology to forestry management, including: 1. Forestry management using wireless network
communication technology of Low Power Wide Area Network (LPWAN) such as LoRa and NB-IoT; 2. Apply
different sensing technologies to survey resource of forest and monitor the microclimate changes in forest. In orderto actually verify the proposed feasible communication technology, sensors, and sensor arrangement, we chose theFushan Botanical Garden with the most diverse and complex terrain in Taiwan as the experimental site. We actuallybuilt LoRa and NB-IoT communication equipment (including repeat equipment) and various sensors to transmit thereal-time sensing data. The returned data also verifies the successful operation of various communication devices andsensors.
Keywords: Internet of Things (IoT), Low Power Wide Area Network (LPWAN), LoRa, NB-IoT

sensor (PIR), which are all included in the platform'splanning and application. Suciu [2] proposed a wirelesssensor network architecture with a solution that
effectively uses available resources; that is, it tries tooptimize the duty cycle of a single sensor (but a singlenode) or maximize the service life of the network.
Kelvin Hirschet al. [3] proposed an ecosystem
based on Canadian forest fire protection, as shown in
Figure 7, which is a practical method of sustainable forest
management. The goal is to use forest management
practices in an active and planned way to reduce the total
area burned by wildfires and the risks associated with theuse of designated fires. The authors [4,5] proposed aforest fire detection system, which consists of sensor

1. Introduction
The application of IoT in forestry can be divided into
three major aspects: forest environment and resource
monitoring planning, forest fire intelligent management
and illegal logging prevention. Environmental andresource monitoring should be a more successful part of
the application of wireless sensing network (WSN) in theforestry Internet of Things. Zhang et al. [1] proposed aforest environmental factor collection platform based on
the ZIGBEE protocol. This platform includes various
terminal monitoring equipment such as temperature,humidity, water level, gas, micro-electromechanicalsystem (mems), photoresistor (LDR) and human infrared© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

52

Shuo-Tsung Chen, Chih-Chian Hua, Ching-Chun Chuang

nodes randomly deployed in the forest area. Each node isequipped with a temperature sensor. Nodes periodicallycheck the environment to determine whether there is an
emergency. When some sensor nodes detect a significant
change in temperature, they will broadcast a data packet
containing their measured value and present the data on
the computer web page and mobile phone page.
Suguvanam et al. [6] proposed a model to prevent thesmuggling of valuable trees such as sandalwood and redsandalwood in forest areas. As shown in Figure 10, theirmodel has three units: 1. Tree unit 2. Sub Server Unit 3.
Forest officer.
In this work, we implement the IoT technology
for forestry management in Fushan Botanical Garden in
Taiwan. In the areas with 3G/4G, we install NB-IoTsupport sensors and continuously send out their sensing
data by using NB-IoT communication technology. Dueto the fact that Fushan Botanical Garden has the mostdiverse and complex terrain without 3G/4G, the real-time
sensing data including illuminance, atmospheric pressure,ultraviolet light, and carbon dioxide are repeated to two
repeat stations which is installed on hillside in order.
Thus, these sensing data are transmitted to LoRa getway
connected with 3G/4G. Finally, the real-time sensing
data are transmitted to web server.
The rest of this work are as follows. Section 2 proposes
Forest Sensors and Communication Technology. Section
3 presents the implementation and experiment results.
Section 4 lists conclusion.

(a) NB-IoT communication technology and its
respective sensors(b) LoRa communication technology and its
respective sensors
Fig. 1. Two communication technologies, NB-IoT and
LoRa, combined with their respective sensors.
2.1. NB-IoT communication technology and its
respective sensors
In a few areas with 3G/4G, we use NB-IoT
communication technology to continuously return thesensing data from NB-IoT-support sensors.

2. Proposed Forest Sensors and Communication
Technology
This section presents proposed forest sensors andcommunication technology for the Fukuyama Botanical
Garden in Yilan County, New Taipei City, Taiwan. A
few areas in the Fukuyama Botanical Garden have 3G/4G
signals, but most areas do not have 3G/4G signals.
Therefore, as shown in Figure 1, we use two wirelessnetwork communication technologies of Low Power
Wide Area Network (LPWAN), NB-IoT and LoRa,
combined with their respective sensors to send thesensing data back to the Forestry Bureau of the Councilof Agriculture of the Executive Yuan (Forestry Bureau).
The detail is presented in subsections 2.1 and 2.2.

NB-IoTsensor

3G/4G

monitoring data
in web

Fig. 2. Flowchart of NB-IoT method for returning thesensing data.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

53

Application of IOT to2.2. LoRa communication technology and its
respective sensors
Due to the complex and diverse terrain of the Fushan siteand most areas have no 3G/4G signal, the LoRa signal
repeat method of this work will continuously return thesensing data of the LoRa-support solar micro-weather
station.

solar
microweather
stationLoRa
signalrepeat1

LoRa
signalrepeat2

LoRa
getway
+
3G/4G

monit
oring
data inweb

Fig. 3. Flowchart of LoRa signal repeat method for
returning the sensing data of the LoRa-support solar
micro-weather station.
3.

Fig. 5. NB-IoT-support temperature and humidity
sensors.

Implementation and Experiment results

This section shows the implementation of the proposed
IoT planning applied to forest management and its
experimental results.
3.1. Implementation and Experimental Results of
NB-IoT Planning
In the areas with 3G/4G, as shown in Figure 4, we first
install NB-IoT-support sensors as shown in Figure 5 andcontinuously send out their sensing data by using NB-IoT
communication technology. Next, the real-time sensing
data including temperature and humidity are connectedwith 3G/4G and finally transmitted to web server asshown in Figure 6.

Fig. 6. Real-time data of temperature and humidity.
3.2. Implementation and Experimental Results of
LoRa Repeat Planning
In the large areas without 3G/4G, we first install LoRasupport solar micro-weather station in the top of somemountain as shown in Figure 7 and continuously send out
sensing data by using LoRa communication technology.
Next, the real-time sensing data including illuminance,
atmospheric pressure, ultraviolet light, and carbon
dioxide are repeated to two repeat stations which isinstalled on hillside in order. Respectively, these sensing
data are then transmitted to LoRa getway which isconnected with 3G/4G. Finally, these real-time sensing
data are transmitted to web server as shown in Figure 8.

Fig. 4. Implementation of NB-IoT-support sensors andcommunication technology.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

54

Shuo-Tsung Chen, Chih-Chian Hua, Ching-Chun Chuang

3G/4G. Finally, these real-time sensing data aretransmitted to web server.
References
1. Zhang Yu, Liu Xugang, Geng Xue, Li Dan, “IoT Forest
Environmental Factors Collection Platform Based on
ZIGBEE,” CYBERNETICS AND INFORMATIONTECHNOLOGIES, Volume 14, Special Issue, 2014,
pp.51-62.
2. George Suciu, Ramona Ciuciuc, Adrian Pasat, and Andrei
Scheianu, “Remote Sensing for Forest Environment
Preservation,” Advances in Intelligent Systems andComputing, 2017, DOI 10.1007/978-3-319-56538-5_23
3. Kelvin Hirsch, Victor Kafka, Cordy TJlmstra, Rob
McAlpine', Brad Hawkes, Herman Stegehuis, Sherra
Quintilio, Sylvie Gauthier, and Karl Peck, “Fire-smart
forest management: A pragmatic approach to sustainable
forest management in fire-dominated ecosystems,” THE
FORESTR Y CHRONICLE MARCWAPRlL, VOL. 77,
NO.2, pp. 357-363, 2001.
4. Pouya Bolourchi and Sener Uysa, “Forest Fire Detection
with Wireless Sensor Networks,” 2013 Fifth International
ConferenceonComputational
Intelligence,
Communication Systems and Networks.
5. Murat Dener, Yunus Özkök, Cevat Bostancioglu, “Fire
Detection Systems in Wireless Sensor Networks,” World
ConferenceonTechnology,
Innovation
andEntrepreneurship, Social and Behavioral Sciences, 195( 2015 ) pp.1846-1850.
6. Suguvanam K, Senthil Kumar, Partha Sarathy, Karthick,
Raj Kumar, “Innovative Protection of Valuable Trees from
Smuggling Using RFID
and Sensors,” International
Journal of Innovative Research in Science, Engineering,and Technology, Vol. 6, Issue 3, March 2017, pp.38363845.

Fig. 7. Implementation of LoRa-support solar micro
weather station, two repeat stations, and communication
technology.

Fig. 8. Real-time data of illuminance, atmosphericpressure, ultraviolet light, carbon dioxide.
4. Conclusions
In this work, we study how to apply the IoT technology
to forestry management in Fushan Botanical Garden in
Taiwan. In the areas with 3G/4G, we install NB-IoTsupport sensors and continuously send out their sensing
data by using NB-IoT communication technology. Dueto the fact that Fushan Botanical Garden has the mostdiverse and complex terrain without 3G/4G, the real-time
sensing data including illuminance, atmospheric pressure,ultraviolet light, and carbon dioxide are repeated to two
repeat stations which is installed on hillside in order.
Respectively, these sensing data are then successfullytransmitted to LoRa getway which is connected with© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

55

Development of the IoT module using MQTT Protocol and AES

Jr-Hung Guo *Department of Electrical Engineering, National Yunlin University of Science & Technology
123 University Road, Section 3,Douliou, Yunlin 64002 , Taiwan, R.O.C †Lin Tzu Yuan

Department of Electrical Engineering, National Yunlin University of Science & Technology
123 University Road, Section 3,Douliou, Yunlin 64002 , Taiwan, R.O.C
Kuo-Hsien Hsia

College of Future National Yunlin University of Science and Technology
123 University Road, Section 3,Douliou, Yunlin 64002 , Taiwan, R.O.C
E-mail: jrhung@yuntech.edu.tw, m10812021@yuntech.edu.tw, khhsia@yuntech.edu.tw(leave space here should be 4 lines between www.university_name.edu and Abstract)
Abstract
The efficiency and safety of the Internet of Things (IoT) have always been the focus of the development of IoT
devices. Because the chips used in IoT devices generally have poor computing power, they cannot transmit data
quickly and in large amounts, and use more complex security algorithms. Therefore, this thesis is to develop an IoT
module with STM32 chip as the main controller. This module uses MQ Telemetry Transport (MQTT)[1] Protocol
and AES encryption technology, and this IoT module can be operated directly with a browser. MQTT is acommunication protocol for the Internet of Things, which was developed by IBM and Eurotech, and officially became
an OASIS international standard in 2014. The purpose of development is to send and receive processing messages
under narrow bandwidth and low energy consumption conditions. To ensure the security of IoT communications, we
use AES encryption technology. Through this design, the communication of the entire IoT module is more efficientand safe. Finally, we applied this IoT module to the home security system, and the overall efficiency and safety havebeen verified. In the future, we will continue to improve related software and hardware so that this IoT module canbe used in different fields.
Keywords: Internet of Things (IoT), MQ Telemetry Transport (MQTT), AES, Home Security System.

[2], the efficiency and safety of Smart House Systems
have been studied. It concluded that the safety of SmartHouse Systems still needs to be strengthened. AHMAD,
Farhan, et al. [3] researched the safety of VANET(Vehicular Ad-hoc NET) Because VANET has a great
relationship with people's life and property safety, how toensure the safety of communication and information is

1. Introduction
The devices of the IoT have been widely used in our
living environment, and these devices have brought many
conveniences to our lives. However, the efficiency andsafety of these devices are still very hot topics. In
previous studies such as Parkhomenko, Anzhelika, et al.
.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

56

Jr-Hung Guo, Lin Tzu Yuan, Kuo-Hsien Hsia

protocol between the oil pipeline sensor and the artificial
satellite under the premise of the narrow network
bandwidth and small power loss. MQTT originally stoodfor Message Queueing Telemetry Transport. This kind of
expression is no longer used. MQTT is MQTT, not an
abbreviation of other words.
Because the message content of the MQTT protocol
is very streamlined, it is very suitable for IoT devices
with limited processor resources and network bandwidth.
Many MQTT libraries have been developed for Arduino
control boards (C/C++), JavaScript (Node.js, Espruino
control boards), Python, etc. There is also an open source
MQTT server, which makes it very easy to develop the
MQTT IoT and the communication between machines(Machine-to-Machine, M2M).
The advantage of MQTT is that it is a lightweight
protocol. Since it is a protocol designed for the Internetof Things, the network bandwidth it requires is very low.
And the required hardware resources are also low. It isvery suitable for IoT environments with low power
consumption and limited network bandwidth. Such as
smart appliances or medical devices.
MQTT uses the Publish/Subscribe mechanism formessaging, which contains 4 main elements, Publisher,
Subscriber, Topic, and Broker. Among them, the IoT
module is the Publisher, and the information that the IoT
module needs to send out is the Topic. And theseinformation Topics will not be sent directly to thedemand-side Subscriber, but sent through the Broker. So
the Publisher and the Subscriber are not directly
connected. But because of this, the information andcommunication security of the entire IoT system must be
controlled through a Broker.
At present, most Brokers have SSL (Secure Sockets
Layer) encryption mechanism when transmitting data,
but SSL is too complicated for IoT devices. Therefore,
this paper uses AES (Advanced Encryption Standard)
technology to ensure the security of all communicationand data in the communication of the IoT module and thecommunication of the Broker.
Finally, we also modified the MQTT architecture.
Because the original MQTT architecture Subscriber can
only simply respond to whether it has received a Topic
from the Publisher. But in the application of the IoTs
system, in many cases, the Subscriber needs to send
messages or control the Publisher. Therefore, the revised
MQTT architecture of this paper is shown in Fig. 1.

very important. MENEGHELLO, Francesca, et al. [4]researched the security issues of the Internet of Thingsand proposed related solutions. NAIK, Nitin. [5]compares and analyzes the efficiency and security ofseveral commonly used IoT communication protocols
such as MQTT, CoAP, AMQP and HTTP.
From previous research, we can find that thecommunication efficiency and security of IoT are still
important research topics. Therefore, this article uses the
STM32 chip to develop an IoT module and uses the
MQTT protocol as the entire system communication
protocol. MQTT is a communication protocol invented in
1999 by Dr. Andy Stanford-Clark of IBM and Dr. Arlen
Nipper of Arcom (renamed Eurotech) [6]. This protocol
was originally designed to allow IoT devices toeffectively transmit messages even with limited
bandwidth and computing power. And this agreement
also became an international standard of OASIS(Organization Advancement Structured Information
Standards) in 2014. Although this standard is designed
for communication between IoT devices, large network
service providers such as Facebook Messenger[7] andAmazon Iot[8] also use the MQTT protocol. It can beseen that the MQTT protocol is still one of the important
protocols for network communication and IoT systems.
Therefore, we chose MQTT as the main communication
protocol of the Internet of Things system in this paper.
In this paper, we have also modified the MQTT
protocol, so that the original MQTT protocol, which can
only be transmitted in one direction and cannot be
operated, becomes a system that allows users to operate.
Punctuation marks are used at the end of equations as if
they appeared directly in the text. In order to ensure thesecurity of the entire system, we have also adopted AES
to ensure the security of the communication and data ofthe entire system.
Through the integration of MQTT and AES, and theIoT module developed in this paper using STM32 chips,
the relevant software and hardware of this paper can
provide a safe and efficient system. And used in different
fields.
2. System Architecture
MQTT is a communication protocol invented in 1999 by
Dr. Andy Stanford-Clark of IBM and Dr. Arlen Nipperof Arcom (renamed Eurotech). They were in order toprovide a lightweight and reliable binary communication© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

57

Development of the IoT

Fig. 1. The Revised MQTT Architecture Diagram of This
Paper.

Fig. 2. Base on STM32 Chip IoT Module Block Diagram.

2.1. Hardware Architecture

3. Experimental Results

The IoT module developed in this paper uses a 32-bit
STM32 series microcontroller. This series of
microcontrollers are designed based on ARM Cortex™M. Mainly to provide MCU users with new development
freedom. It integrates high performance, real-timefunctions, digital signal processing, low power
consumption and low voltage operation. At the same time,
it is highly integrated and easy to develop. For the IoT
module developed in this paper, the microcontroller wechose is STM32F103VET6. It is an integration CPU,
RAM, ROM, a variety of I/O ports (including display
drive circuit, PWM, analog multiplexer, A/D converter,etc.) and interrupt system, timer/counter and otherfunctions constitute a microcomputer system. Because ofits better stability and low price, this paper chooses thischip as the main controller of the IoT module.
In the communication interface part, the IoT module
developed in this paper can connect to Wi-Fi, Ethernet,
and USART (Universal synchronous/asynchronousreceiver transmitter), LoRa (Long Range) etc. for thecommunication interface. Moreover, this IoT module
adopts a modular design, so you can choose to use
different communication modules according to different
needs. And in order to ensure the security of data andcommunication, we use AES encryption technology inthis IoT module. Let this IoT module provide secure data
communication and transmission. The block diagram ofthis IoT module is shown in Fig. 2.

We use STM32 chip as the main controller of the IoT
module developed in this paper. This module adopts
modular design, so most of the communication interface
can be replaced according to needs. And we also add atouch LCD to this IoT module. When necessary, we don't
need to connect with the host, and directly operate
through the touch LCD. In addition, we also added thefunction of Web Server to this IoT module. Allow users
to directly use the browser to control this IoT module.
But for the sake of security, the MQTT setting still has tobe set through the Broker. Through this design, the IoT
module we developed is both convenient and safe. Fig. 3
shows the browser interface of the IoT module developedby us, and the screen of the entity with the touch LCD
and the IoT module connected with the browser.

(a)IoT Module.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

58

Jr-Hung Guo, Lin Tzu Yuan, Kuo-Hsien Hsia

sensor and integrate it with the MQTT system wedeveloped. The result is shown in Fig. 5.

(b)IoT Module Browser and LCD Screen.
Fig. 3. STM32 chip IoT Module and Screen.
Fig. 4. MQTT Broker Software System with MQTT Topic
Names Sample.

3.1. MQTT Broker System and Experiment
Results.
In the MQTT Broker, we use Visual Studio 2017 C# todevelop our system. The picture of this system is shownin Fig. 4. In this system, users can define the map of theenvironment and the related attributes of the IoT module.
And the I/O or A/D related parameters of which IoT
module the Topic corresponds to. In addition, this system
also provides an automatic search function. The system
can automatically search for a Ethernet or Wi-Fi was the
IoT devices having MQTT function. After searching, it
will automatically name the I/O or A/D on the device
based on the IP. This simplifies the user naming
operation, and the system management of MQTT Topics
will be more efficient. The rules for this automatic Topics
naming are as follows:
Topic Rule:
INDEX,I/O Type,I/O Locate,IP, PORT

Fig. 5. MQTT Broker System with IoT Module.

4. Conclusions
This paper uses STM32 chip to develop a multifunctional IoT module, and uses MQTT protocol todevelop an application system. Finally, the software andhardware are integrated and applied to the home security
system. The performance and condition of the entire
operation are very good. At present, we put the MQTT
management interface and the user's operation interface
together. Although this design has its convenience, there
may be doubts in practical application and safety.
Therefore, in the future, we will separate the management
interface from the user's operating interface, and convert
this system to a web. I believe that the overall ease of use
and safety can be improved. In the future, we willcontinue to add artificial intelligence and more sensor
functions. In addition to being used in home security, thissystem can also be used in different fields such as smart
cities and Industry 4.0.

Table 1. Topic Rule Description

Finally, we apply the IoT module developed in thispaper and the MQTT system to home security. We
connect the IoT module to the temperature and humidity© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 202159

Development of the IoT

5. References
1. http://public.dhe.ibm.com/software/dw/webservices/wsmqtt/mqtt-v3r1.html,2020/12/01
2. Parkhomenko, A., Tulenkov, A., Sokolyanskii, A.,
Zalyubovskiy, Y., Parkhomenko, A., & Stepanenko, A.
(2018, March). The application of the remote lab forstudying the issues of Smart House systems power
efficiency, safety and cybersecurity. In International
conference on Remote engineering and virtual
instrumentation (pp. 395-402). Springer, Cham.
3. AHMAD, Farhan, et al. A comparative analysis of trustmodels for safety applications in IoT-enabled vehicular
networks. In: 2019 Wireless Days (WD). IEEE, 2019. p.
1-8.
4. MENEGHELLO, Francesca, et al. IoT: Internet of
Threats? A survey of practical security vulnerabilities inreal IoT devices. IEEE Internet of Things Journal, 2019,
6.5: pp. 8182-8201.
5. NAIK, Nitin. Choice of effective messaging protocols for
IoT systems: MQTT, CoAP, AMQP and HTTP. In: 2017
IEEE international systems engineering symposium(ISSE). IEEE, 2017. p. 1-7.
6. http://www.steves-internet-guide.com/mqtt/,2020/12/01
7. https://www.facebook.com/notes/facebookengineering/building-facebookmessenger/10150259350998920,2020/12/01
8. https://aws.amazon.com/tw/blogs/aws/aws-iot-cloudservices-for-connected-devices/,2020/12/01© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

60

Exploring the Intention to Continuance of Learning Programmingat Elementary School of Rural Area by the mBot Robot
Yung-Hsin Cheng

Department of Information Management, National Yunlin University of Science and Technology,
123 University Road, Section 3, Douliou, Yunlin 64002, Taiwan, R.O.C.
Jia-Ming Hsiao

Bachelor Program in Interdisciplinary Studies, National Yunlin University of Science and Technology,
123 University Road, Section 3, Douliou, Yunlin 64002, Taiwan, R.O.C.
E-mail: cindy6092624@gmail.com, hsiaojm@yuntech.edu.twhttps://www.yuntech.edu.tw

Abstract
Since Curriculum Guidelines of 12-Year Basic Education implemented by the Ministry of Education in 2018, theprogram learning courses have been added to junior high school education. However, there are no programmingcourse in the elementary school. This study is proposed to explore the continuity and intentions of the rural area
students in the programming course with mBot robot and mBlock programming tool through the Post-Acceptance
Model of IS Continuance. It is indicated that enlightenment education of computational thinking should be
implemented during the elementary school by means of graphical programming software and robot practice. Throughthe graphical software and teaching robots, we can cultivate problem solving skills for students' logic, creative
thinking and communication, as well as through the task-guided way to train the students' concentration andperseverance.
Keywords: Rural Education, Robot programming, Makeblock mBot, Post-Acceptance Model of IS Continuance.

practice. Through the graphical software and education
robots, we can cultivate problem solving skills forstudents' logic, creative thinking and communication, as
well as through the task-guided way to train the students'
concentration and perseverance. On the other hand, it isalso observed that there exists digital divide in education
between urban and rural schools. The digital divide canbe reduced by the skill training of software, programmingand problem solving. In the mean time, many volunteers
provide learning activities of programing and robot
education for rural school to assist in teaching andlearning in Taiwan.

1. Introduction
In 2018, the Curriculum Guidelines of 12-Year Basic
Education had been implemented by the Ministry of
Education, Taiwan. One of the major changes is thatcurriculum development in the field of technology injunior and senior high school should be compatible withinformation technology and life technology. The
programing language courses have been added to juniorhigh school education. However, the programmingcourse is not essential but implemented as a flexible
curriculum in the elementary school. It is indicated thatenlightenment education of computational thinking
should be implemented during the elementary school bymeans of graphical programming software and robot

mBot is an education robot for beginners, that wasannounced by Makeblock Co., Ltd in 2015. In Taiwan, it© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

61

Yung-Hsin Cheng, Jia-Ming Hsiao

is often used in elementary and junior high school todevelop the students’ computational thinking andprogramming skills. The programming tool is mblock
which is a block-based and code-based programming
software developed from Scratch. The mBot robot course
is aimed on the Grade 3 to 6 students of elementary
students. There are four topics that include logicalthinking, sensors, differential driven wheel robot andproject practice to cultivate problem solving skills forstudents' logic, creative thinking and communication, as
well as through the task-guided way to train the students'
concentration and perseverance.

(iv) Questionnaire collecting and analysis.

Expectation-Confirmation Theory was proposed by
Oliver1. The sustainability theory in the IS research fielddraws on Expectation-Confirmation Theory in consumer
behavior. However, Bhattacherjee2 believes that the
Expectation-Confirmation Theory is controversial andunreasonable in some aspects. He proposed postacceptance model of IS continuance by the concept that
continuance usage intention of information system will
be influenced by usage satisfaction and perceived ease ofuse. Many scholars conduct related research based on the
post-acceptance model of IS continuance, but there are
not many related researches on education. Based on the
post-acceptance model of IS continuance, the students’
perceived usefulness of the curriculum can be used as areference benchmark for confirmation and judgment,
which will positively affect their satisfaction with thecurriculum3.

There are four rural elementary schools participated inthis study. The course is implemented in the formal
course in two schools, and in the student club in the othertwo schools. There are 85 students participated in thecourse. The students are divided into groups in which
there two members each. Group members are encouraged
to show their team-work spirit and knowledge sharing.

Fig. 1 Post-Acceptance Model of IS Continuance2

2.1. Teaching and learning mBot

2.2. Example exercise
Each lesson is arranged to focus on one theme, e.g.,
motor control, sensor application, etc. A short example isgiven in class, and the students are asked to test andpractice an exercise. Both mechanical and programming
discussion are applied to each group. The subject of
exercise may be from the missions of MakeX robot
competition.
2.3. Group projectThis study is proposed to explore the continuity andintentions of the rural area students in the programmingcourse with mBot robot and mBlock programming tool
through the post-acceptance model of IS continuance.
The purpose is to explore students’ satisfaction with
programming courses, and to understand whether it canstimulate students’ motivation to continue participating
advanced programming courses.

In the last two or three lessons of the semester, a groupproject is assigned to each group. The project assigned to
Grade 3 and 4 students is RoboRAVE a-maze-ing
problem, and Grade 5 and 6 students are assigned to solve
the line-following and object avoiding problems.

2. Research structureThis study is based on the Post-Acceptance Model of IS
Continuance, as shown in Fig. 1. It is implemented by
the following steps:(i) Teaching and learning mBot,(ii) Example exercise,(iii) Group project,© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

62

Exploring the Intention to

2.4. Development
propositions
Perceived
Usefulnessofresearch

model

and

Table 1 Students’ basic personal information
Basic personal information
Male
GenderFemale
<= 10

H3
H2
H5

H1

Satisfaction

Confirmation

continuanceintention

Age

H4
Grade

Fig. 2 research framework

Based on the Post-Acceptance Model of IS Continuance,
the research framework is shown as Fig. 2. There are fivehypotheses made in this study:

How long tolearn mBot?

H1: Users’ confirmation has the significant influence on
satisfaction.
H2: Users’ perceived usefulness has the significant
influence on satisfaction.
H3: Users’ perceived usefulness has the significant
influence on continuance intention of programmingcourse.
H4: Users’ confirmation of programming course has thesignificant influence on satisfaction.
H5: Users’ satisfaction of programming course has thesignificant influence on continuance intention ofprogramming course.

Experiencecoding?
Interested incoding before
learning mBot?
Using computeror tablet after
school?
Average time ofusing computeror tablet everyday

The four factors of research framework are defined asfollowings:• perceived usefulness: Students subjectively feel that
programming courses are helpful for learning.
• confirmation: After students experience theprogramming course, their before and after
psychological expectations of the experience are thesame.
• satisfaction: How the students feel about the
programming course after the experience.
• continuance intention: Students are willing tocontinue learning programming courses in the future.

Count
43

Percentage(%)
66.2

22
37

33.8
56.9

11
12

15
8

23.1
12.3

> 12
3

5
14

7.7
21.5

4
5

30
8

46.2
12.3

6
1 semester

13
57

20.0
87.7

2 semesters
3 semesters

7
0

10.8
0

4 semesters
Yes

1
47

1.5
72.3

No

18

27.7

Yes

55

84.6

No

10

15.4

Yes

60

92.3

No

5

7.7

0

5

7.7

< 1 Hr
1~2 Hr

27
14

41.5
21.5

2~3 Hr> 3 Hr

7
12

10.8
18.5

After the data analysis, the result of five hypotheses isshown as Table 2. It is shown that hypotheses H1, H2,
H4 and H5 have significant influence, and H3 does not.
Table 2 Data analysis result
Hypothesis
H1
H2
H3
H4
H5

Path coefficient(β)
0.691
0.410
-0.0050.518
0.925

t-value
6.335
3.273
0.053
4.769
11.782p-value< 0.001***< 0.001***
0.958< 0.001***< 0.001***

3. Main results

4.

Conclusions

After the programming course, a questionnaire is issued
to students. There received 68 questionnaire samples, and65 samples are valid, the other 3 samples are invalid. The
students’ basic personal information are listed in Table 1.

From Table 1, it is shown that there are 72.3% students
experienced programming course, 84.6% students areinterested in coding before learning mBot, and 87.7%
students have one-semester experience of learning mBot.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

63

Yung-Hsin Cheng, Jia-Ming Hsiao

3. Lee, M.-C., Explaining and predicting users’ continuanceintention toward e-learning: An extension of the
expectation–confirmation model, Computers & Education,
54(2), 2010, pp. 506-516.

This research believes that after experiencing the mBot
programming course, students have more willing andmotivation to learn to code, and are willing to learn aboutother programming after school.
We observed that most of the students are a little afraidof learning the new programming course before thecourse. For the brand-new programming course, theoriginal expectation was low. After experiencing themBot programming course, through the combined
teaching of mBlock and mBot robot, the expectation after
the course was greater than expected. With the mBot
robot programming course, students can learn andunderstand programming applications faster with fun, so
as to be satisfied with the mBot robot programmingcourse. With respect to the degree of confirmation of
students in the mBot program course, it is indicated thatthe higher their degree of confirmation, the higher theirsatisfaction. It is confirmed that the satisfaction of mBot
programming courses will positively affect the students'
intention to continue participating programming courses.
It is observed in this research, students’ satisfaction with
programming courses makes them willing to spend moretime on learning programming, and students will be more
willing to actively learn advanced programming courses.
However, students’ enthusiasm may decay along withtime passed. It could result from that students do notknow why they should learn robot programming.
Teachers should link the robot programming course toreal life applications or examples. With the linkage,
students are guided to build and program useful robots tosolve problems. It is also important to encourage students
to share their ideas of building robots, and then to
maintain students’ enthusiasm for learning programming
and robot building.
References
1. Oliver, R. L. and Richard, L., A cognitive model for theantecedents and consequences of satisfaction decision,
Journal of Marketing Research, 17(4), 1980, pp. 460-469.
2. Bhattacherjee, A., Understanding information systems
continuance: An expectation-confirmation model, MIS
Quarterly, 25(3), 2001, pp. 351−370.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

64

Basic Research on Parameter Tuning Skills Evaluation Based on Sensor Car Behavior Data inTechnology Education
Teruyuki Tamai

Graduate School of Humanities and Social Sciences, Hiroshima University / Faculty of Education, Ehime University
3, Bunkyo-cho, Matsuyama, Ehime 790-8577, Japan
Yoshihiro Ohnishi

Faculty of Education, Ehime University
3, Bunkyo-cho, Matsuyama, Ehime 790-8577, Japan
Kazuo Kawada

Graduate School of Humanities and Social Sciences, Hiroshima University
1-1-1, Kagamiyama, Higashi-hiroshima, Hiroshima 739-8524, Japan
E-mail: tamai.teruyuki.xq@ehime-u.ac.jp, ohnishi@ehime-u.ac.jp, kawada@hiroshima-u.ac.jp
www. ehime-u.ac.jp, www. hiroshima-u.ac.jp

Abstract
The "Future Investment Strategy 2018" presented by the Cabinet Office in 2018 states that there is an urgent need todevelop human resources who can use AI to achieve the goal of Society 5.0. As a part of this strategy, programming
education in primary/secondary level of education is being promoted. However, there are some issues in thequantitative evaluation of learning and teaching methods based on evaluation. In this paper, sensor car teachingmaterials will be used to teach programming. Then, a new lesson is proposed to tune the optimal parameters by
quantitative evaluation. The results of the university students practice of the proposed lesson was explained.
Keywords: Junior High School, Parameter Tuning Skill, Quantitative Evaluation

schools and the development of local environments sothat students can learn more advanced programming arebeing carried out. Thus, there is an urgent need to
improve the education and instruction of programming.
This is not limited to Japan, in recent years, there havebeen many studies on education and lesson inprogramming. For example, it has been shown that there
are many difficulties in learning programming2 and thatbeginners in programming are more likely to learn inclass than in take-home learning3. Thus, there is an urgent
need to improve human resource development, including
programming, both domestically and internationally. In

1. Introduction
The Cabinet Office published "Future Investment
Strategy 2018" in 20181. In this strategy, it states that in
the AI era, in addition to the ability to understand and use
AI and data with high mathematical capabilities, theability to set up and solve problems and to create
heterogeneous Human resources who can create value
with abilities that are difficult to replace by AI, such as
the ability to combine things, are needed. AI education at
the primary/secondary levels is being strengthened. The
development of teaching materials so that programming
education can be effectively implemented in elementary© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

73

Teruyuki Tamai, Yoshihiro Ohnishi, Kazuo Kawada

Japan, the study of programming at theelementary/secondary level school is mainly conductedin junior high school technology and home economics (inthe technology field) (hereinafter referred to as JuniorHigh School Technology). Teaching materials forteaching programming in Junior High School
Technology include teaching materials for programming
the timing of lights and sounds, and robotics4.
Programming skills are generally assessed by taking"Can you use sequential, iterative, and conditional
processing?" and "Are there typing errors?". There areproblems with these types of evaluation, such as the factthat they take a lot of time and depend on the discretionof the teacher. To solve this problem, quantitativeevaluation studies have been conducted. The "effect" ofthe program is visualized by having the learner completetasks of varying difficulty5. This method of evaluation
allows students to feel their programming skills improve.
On the other hand, in the learning process to develop thecompetency of Junior High School Technology, it isimportant to learn how to optimize the solution ofproblems. There is no practice of learning programmingthat incorporates quantitative evaluation into thislearning process. In this paper, A new evaluation method
to quantitatively evaluate a sensor car's linear motion interms of time and left-right swing during the motion areproposed. Then, A lesson for beginners in programming
to develop the skills to tune the optimal parameters areproposed.

500mm

Left-right swing
Fig. 2. Field to run.

Threshold

M1y
M2y
M1n
M2n
Fig. 3. Studuino mini programming.

microcontroller board with input/output interfaces foreducation. An infrared photoreflector (RPR-220, Made
by ROHM) is used as a sensor to identify the line to be
traveled. It was powered by three AA batteries at 4.5V.

2. Teaching Materials

3.

In this paper, A sensor car (Made by Artec) as shown in
Fig. 1 are used. As a microcontroller board, Studuino
mini is used, which is an Arduino-compatible

Learning Task

A learning challenge aimed at the beginning student ofprogramming was devised.
3.1. Evaluation Items

Studuino mini

As shown in Fig. 2, a black line of 10mm width is drawn
on the white surface of the field. The sensor car runs
straight 500mm along the black line in this field. The
running time and left-right swing are measured. Then,
Task 1 is to shorten the running time. Task 2 is to reduce
left-right swing. The learning task was to tune theparameters to solve these two tasks.

Infrared
photoreflector
Fig. 1. Sensor car.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

74

Basic Research on Parameter

3.2. Parameters to be tuningy

As shown in Fig. 3, the Studuino mini software is capableof Scratch-based visual programming. The learner tunes
five parameters. It is the threshold of the sensor(threshold), and the ratio of PWM to adjust the speed ofthe left and right motors (M1y, M2y, M1n, M2n).
4.

(𝑥𝑥1, 𝑦𝑦1)(𝑥𝑥2, 𝑦𝑦2)xParameter Evaluation

The running time 𝑡𝑡 in Task 1 was measured using a
stopwatch. The left-right swing of the sensor car in task
2 was recorded by a digital camera (IXY 92015, Made by
CANON). Then, it was analyzed with the video analysis
software PV Studio 2D ver2 (Made by L.A.B). To
measure the left-right swing, markers were attached at
two points as shown in Fig.4. The x-axis and y-axis were
defined as shown in Fig. 5, and the position of the marker
was read. A left-right swing 𝑦𝑦12 was calculated by Eq.
(1).
𝑦𝑦12 = |𝑦𝑦1 − 𝑦𝑦2|

Fig. 5 Set axis.
Table 1. Lesson plan.
Hours
[min]

(1)

The variance 𝑦𝑦12 was calculated in Eq. (2). However, n
is the number of data extracted from the video from thestart to the goal by the video analysis software. This𝜎𝜎 2 𝑦𝑦12 was used as the left-right swing.
𝜎𝜎 2

𝜎𝜎 2 𝑦𝑦12 =������ 2∑𝑛𝑛
𝑖𝑖=1(𝑦𝑦12𝑖𝑖 −𝑦𝑦12)
𝑛𝑛

5.

(2)

Lesson 1

60

Lesson 230

Lesson 390

Main learning contents
Check the relationship between
the adjustment of parameters
and the movement of thesensor car's wheels.
Know how to program a line
trace using a sensor car.
Understand the learning task.
Tuning the optimal parameters.

Lesson Suggestions

To examine the time required to implement the proposed
learning evaluation method and the learning effects, aclass was practiced by university students in teacher
training. Two people could use one sensor car. eightgroups of 16 people were conducted. Table 1 shows thelearning plan.
In Lesson 1, students experienced how changing thedirection and speed of rotation from left to right changed
the behavior of the sensor car. The reason for this wasthat most of the students had never experienced
programming a sensor car before. In Lesson 2, students
were shown the program shown in Fig. 3 and were taught
the steps of the program to do a line trace. In Lesson 3,
the students first predicted the movement of the sensor
car and designed the parameters to complete the task.
Then, they programmed the sensor car according to thedesign and made it work. In this way, Lesson 3
incorporated the cycle of design, creation, evaluation,and improvement so that the solution to the problem

If the running time 𝑡𝑡 and the degree of left-right swing
𝜎𝜎 2 𝑦𝑦12 become small, the learner is able to choose a goodparameter. In other words, the parameter tuning skill ishigh.

Markers to adjust
the scaleMarker tomeasure swing
Fig. 4. Marker position.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

75

Teruyuki Tamai, Yoshihiro Ohnishi, Kazuo Kawada

Table 2. Sensor car results.
Group 1thresholdGroup 2

4.17

100

100

55

4.01

𝜎𝜎 2 𝑦𝑦12
0.32

46

100

20

20

100

4.12

37.80
75of driving
100
Table 2.60 Results

100

75

4.05

0.31

45

100

30

30

100

4.02

0.24

60

80

100

100

60

3.73

0.25

43

100

80

80

100

3.98

0.18

60

90

100

100

75

3.73

0.16

43

100

90

90

100

t[s]
First
time
Secon
d timeThird
time
Fourthtime

threshold

𝜎𝜎 2 𝑦𝑦12
0.085

80

M1y

M2y

M1n

M2n

t[s]

55

M1y

M2y

M2n

M1n

would be optimal, which is considered important in thelearning process.

Acknowledgements

6. Results and Discussion

The authors are also grateful that this work was supportedby the Ehime University Challenge Support Fund.

Table 2 shows the results of the practice of the university
students. In the practice of this study, university students
carried out the learning process of Lesson 3 four times.
Some of the results of the students' practice are shown in
Table 2. For Group 1, the fourth time was the fastest for
Task 1. And for Task 2, the second time was the worst.
However, it was improved in the third and fourth times.
For Group 2, for the third and fourth times is the same for
Task 1. However, the left-right swing was improved from0.25 to 0.16. For this value, the difference in variance wasdetermined by the F test. Assuming that the left-rightswing of the sensor car is normally distributed, a onetailed test with 114,114 degrees of freedom was
performed and the F value was 0.0070. Therefore, it canbe said that there is a significant difference at 5% level ofsignificance. For the groups shown here, it was
concluded from the quantitative evaluation that theparameters were better regulated.

References
1. Cabinet Office, Future Investment Strategy 2018, 2018.
http://www.kantei.go.jp/jp/singi/keizaisaisei/pdf/miraitou
si2018_en2.pdf[Accessed December 10, 2020]
2. Y. Bosse and M. A. Gerosa, Why is programming so
difficult to learn? Patterns of Difficulties Related toProgramming Learning, ACM SIGSOFT Software
Engineering Notes, 41(6) 2016, pp.1-6
3. O. Karnalim, Comparing the Impact of Programming
Assessmennt Type:In-Class Vs Take-Home, iJEP, 10(4)2020, pp.125-132
4. Y. Ohnishi, K. Honda, R. Nishioka, S. Mori and K.
Kawada, Robotics Programming Learning for Elementary
and Junior High School Student, Journal of Robotics andMechatronics, 29(6) 2017, pp. 992-998
5. Y. Ohnishi, T. Tamai and K. Kawada, Estimation of
Programming Learning Achievement by Line Tracing
Robot, Proceedings of the 2019 International Conferenceon Artificial Life and Robotics, Beppu, Oita, Japan, Feb.
2019, pp. 431-434

7. Conclusion
In this study, a lesson and an evaluation method were
proposed to develop the skills to tune parameters. The
proposed lesson and evaluation method were practiced
by university students. It was shown that the parameter
regulation skills could be evaluated quantitatively. The
evaluated data were compared and statistically processed.
And the goodness of parameter tuning was demonstrated.
In future studies, learning time will be organized andsupplementary materials will be developed so that thisstudy can be implemented in junior high schools.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

76

Mini Windmill Generator Kit for Homework for Hiroshima Univ. Monozukuri Junior Doctor
Special Educational Program
Hiroyuki Y. Suzuki, Kazuo Kawada, Masayasu Nagamatsu

Graduate School of Humanities and Social Sciences, Department of Educational Science, Hiroshima University,
1-1-1-Kagamiyama, Higashi-Hiroshima City, Hiroshima 739-8524, Japan
E-mail: hiro-suzuki@hiroshima-u.ac.jp, kawada@hiroshima-u.ac.jp, nagamatu@hiroshima-u.ac.jp

Abstract
Hiroshima University Monozukuri Junior Doctor is a special educational program for young (11 – 15 in age) students.
It had to be stopped, however, at the midst because of COVID-19 epidemic. We changed it to homework based one.
Modified “Mini Windmill Generator Kit” were sent to them with a manual of “Procedure of quantitative research”
for self-learning. The result shows that several specific tendencies when the data summarized statistically, although
the data collected with different places with different conditions.
Keywords: Technology education, STEM, Quantitative evaluation, Moment of inertia

selections at each stage-up timing. This selection policy
is based on JST’s requirement, that is, only few excellent
students will be selected “individually” and continue tostudy throughout the program. Nonetheless, we introduce
distinctive characteristics of “group activity” to theprogram (approved by JST). We believe that selection of
only few “individual” leaders is somehow oversimplified
vision to compete with today’s science and technology
developments. Many of current problems are resolved by,sometimes very large, projects.
We are going to find and cultivate young people ofdistinctive talents, not only the merely innovative leaders
but also capable supporters of innovators, by the program.
Proper mixing of diverse talents is a key of success formany kinds of today’s projects.

1. Introduction
Hiroshima University Monozukuri Junior Doctor is a
special educational program for young (11-15 in age)volunteered students, aimed for cultivating innovativeand outstanding young people as leaders for next
generation. Original idea of the program is proposed by
Japan Science and Technology Agency (JST).
Our program has started in Jul. 2019, giving 5 seminars
in Pre-Stage to around 300 participants (5 times oforiginally recruited number), and selected to 40 for FirstStage by the end of Sept. The First-Stage also went onsmoothly, giving three seminars to the end of 2019.
However, following seminars had to be cancelled
because of COVID-19 epidemic. We had to change all
the following program to self-learning ones. This reportpresents details of first homework for First-Stage student.

3. Monozukuri Seminar using Mini Windmill
Generator Kit

2. Characteristics of Our Program

We originally introduced Mini Windmill Generator Kit(Artec co., Japan) in First-Stage’s third seminar. The
windmill generator has weathercock appearance, that is,
it always turns to upwind direction automatically. TorqueOur program is composed of three stages of Pre-Stage(recruit of 60 participants), First-Stage (selected to 40)
and Second-Stage (selected to 20), in which there are© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

77

Hiroyuki Y. Suzuki, Kazuo Kawada, Masayasu Nagamatsu

3.2. Making smoothly moving bearing using 3D
printer

generated by the wind at the tail fin makes the body turn.
Proper size of tail fin and the smoothly moved bearing
under the body are the keys for smooth function.
Two themes were given to students (9 groups, 4-5
member each) .
• Finding proper size and shape of tail fin.
• Making smoothly moving bearing using 3D printer.

Students were experienced 3D printing as an
introduction to digital fabrication. They firstly studied
basic commands of 3D CAD (Design spark Mechanical)
and “extrude” concentric shapes with a hole, namely
bearing, in monitor. Then, they made the bearings data toreality, using 3D printer (XYZ printing PRO).
Majority of the students were successful to draw bearing
design but only few were printed, because of limitationof printing speed. Therefore, theme 3.1 was performed
using original bearings in the kit.

3.1. Finding proper size and shape of tail fin
Plastic sheet of 0.8 mm was used to make tail fins. The
sheet was easy to be cut by scissors and set to the rear of
the body with a screw (a screw hole was added). Wind
was given by electric fans and generated voltages were
measured by a tester changing “parameters” of power ofelectric fan, distance between fan and generator, etc.
We modified the kit to be able to measure the generated
voltages by extending thin lead wires from the generator.
Additionally, turning torques of the whole body (withfins) were measured using a jig we prepared (Fig. 1).

3.3. Result of seminar
Each team could take data of tail fins performance bychanging distance between electric fan and the generator
and power needed to turn the fins. Group work made easy
to perform experiments in shorter time.
They were eager to propose new design of tail fins, tocollect data by experiments, to design bearings in 3D
CAD by themselves. Nonetheless, sufficient data were
not collected for the discussion in the seminar, becauseof the time limitation (a half-day seminar). We weregoing to continue the same theme, changing approach
rout, in following seminar.
4.

Homework

4.1. Preparation of homework kit

Fig. 1 Jig for measuring turning torque of thewindmill generator body.

All meetings were banned by Univ. official, after rising
COVID-19 epidemic. In Mar. 2020, we sent a homework
kit of mini windmill generator to every student, as
supplemental contents of previous seminar.
Three color LEDs are added (with a series resistor) as
voltage indicator. A tester is attached to the kit. Manualof “Procedure of quantitative research” in which weroughly explained how to figure the graphs andencouraged them to “find a peak value” in figures. Notethat we did not designate any parameters to be changed.
All the experimental conditions were decided by oneself.
In homework, we gave two themes of,• finding best design of tail fin (same as seminar)• finding best design of windmill.
The windmills could easily made by cutting pet PETbottles. The kit contained a coupler to unite PET-bottle
windmill to generator axle.

Generated torque by wind is in proportion to the size ofthe fin. At the same time, the required torque to turn thewhole body is also in proportion to the moment of inertia.
That is, the larger the fin is, the larger the torque isrequired to turn the body. They are trade-off relationshipeach other. The optimum shape and size of the tail fin
will be found at the balanced point of those two factors.
Students were expected to find appropriate sizes andshapes of fins by themselves. We did not give, however,detailed lecture on above mentioned mechanism
beforehand. We rather expected that students grabbed
overall picture of the mechanisms by themselves throughquantitative data collection and arrangement of the datausing tables and graphs.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

78

Mini Windmill Generator Kit

4.2. Reports returned from students
An example of report returned from a student (female,first grade in junior high school) is shown in Fig. 2.

Number of Rotation

Paper
Card Bord
Felt
Drawing Paper
Plastic Sheet
Tissue Paper

Distance Between Electric Fan and Generator

Fig. 2 Research for finding best material for tail fin.

Fig. 3 Performance of many designs of PET-bottle
windmills were measured by a student.

She tried to evaluate the effect of materials on theperformance of tail fin. She chose fan shape fins (radiusof 100 mm and angle of 120 deg.) made of different
materials of, i) copy paper, ii) card-bord, iii) felt, iv)
drawing paper, v) plastic sheet and vi) tissue paper.
Number of rotation (reacted to the wind) were countedchanging distance between electric fan and the generator.
She concluded that the drawing paper showed the best
performance. She also checked size of the fins in otherpart of the report and concluded about 100cm2 was the
optimum size.
Figure 3 is an example of report on windmill experimentfrom another student (female, first grade in junior highschool), showing various windmills. She checked
windmill performance from two viewpoints of, i) numberof blades (upper table and figure) and ii) length of blade(lower table and figure) and summarized that the
optimum number of blades was three and size of blade
was 3cm, respectively.
Students made plan of experiment by themselves since
we did not give step-by-step procedure in the instruction
manual but gave basic idea of appropriated datacollection, so that somehow unique trials were performedby many students.

contains “complexity” as “real product”. It contains all
STEM factors. Therefore, some research gave up
quantitative evaluations by students1) or evaluated only
by instructors2,3). Some research came into quantitativeevaluations, but they still scarcely used graphs4,5).
Our program is forced to introduce, by chance,experiments planned and performed by students“individually”. We introduced quantitative evaluation
using graphs, but experimental parameters andconditions were all different in each student. Maximum
voltages attained were varied from 0.3 V to 6 V, forexample, among their experiments. Nonetheless, there

Vertical Length [mm]300

Windmill generators are one of popular teachingmaterials for energy conversion since it is easy to makewindmills by hands. However, there are hurdles when wetry to understand “the essence” of the product, since it

Square200

Vertically
Long

100

0
0

5. Discussions

Rectangular fins
Triangle fins
Tapered(narrowing to end)

Laterally Long

100

200

Lateral Length [mm]300

Fig. 4 Optimum sizes and shapes of tail fins foundby each student experiment.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

79

Hiroyuki Y. Suzuki, Kazuo Kawada, Masayasu Nagamatsu

Vertical Length [mm]300

Number of Students

are several noteworthy tendencies can be found when wesummarize their data statistically.
Figure 4 shows the lateral and vertical sizes of tail finsof the best performance. Although argument for selecting
the best fin were merely scattered, there is tendency
between shape and aspect ratio of fins. Rectangular andtapered (narrowing to the end) fins are relatively laterally
long. In contrast, triangle fins (widen to the end,
including fan shape) have vertically long. Higher
moment of inertia of triangular ones must be one reasonof such difference. Interestingly, on the other hand, sizesof fins are almost indifferent to their shapes.
Materials of fins also have no considerable effect tosizes and shapes of them, as shown in Fig. 5, although theresult in Fig. 2 by a student indicates clear difference.
Much higher weight of generator body may mask theweight differences of fin materials.

6
4
2
0

20 40 60 80 100 120
Length of blade [mm]Fig. 6 Optimum blade length of windmills.
6. Summary
Optimum sizes and shapes of tail fins and windmills formini generator kit were determined by quantitative datacollection by students’ (11-15 in age) homework. The
result shows that several specific tendencies when thedata summarized statistically, although the data collectedwith different places with different conditions.

Paper fins
Plastic fins

Acknowledgements
This program was fully supported by JST fund of JuniorDoctor Educational program.

200

References

100

0
0

8

100

200

1. R. Khunyakari, S. Mehrotra, S. Chunawala, C. Natarajan,
Design and technology productions among middle school
students: an Indian experience, Int. J. Technol. Des. Educ.
17, 2007, pp. 5–22.
2. Y. Kawamura, K. Honda, O. Izutsu, Y. Matsumoto,Development of educational materials savonius type
windmill aerogenerator using handmade generator, J. Jpn.
Inst. Energy 92(9), 2013, pp. 913-918.
3. R. Hikiji and Y. Fujimoto, Study on improving
experiment-teaching materials of wind power generator
for junior high school students to get interested in naturalenergy, J. JSEE 65(5), 2017, pp. 48-53.
4. J. Concannon and P. L. Brown, Windmills by design:
Purposful curriculum design to meet next generation
science standards in a 9-12 physics classroom, Science
Activities 54(1), 2017, pp. 1-7.
5. Y. Ohnishi, S. Mori, T. Deyama, J. Ono, H. Muko,
Development of teaching materials featuring wind power
generator for 6th elementary science class based on therelationship with technology class of junior high school,
Ehime Univ. Edu. Faculty Kenkyu Kiyo 61, 2014, pp. 105110.

300

Lateral Length [mm]Fig. 5 Effect of materials on sizes and shapes of tailfins.
As for the windmill design, optimum blade sizes are
summarized in Fig. 6, showing an optimum of around 40
mm. In contrast, blade number (from 4 to 16) is relatively
indifferent among students results (graph is omitted).
These graphs indicate that, the data collection of
scattered experimental conditions was also effective tofind out optimum conditions of the products, if themember took data quantitively and summarized the data
statistically. It is possible that this style of approach israther matched to improve working conditions of many
products since many “real” products essentially containcomplexity in themselves.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

80

Development of Teaching Materials to Learn the Efficient Use of Energy
Yoshihiro Ohnishi*

Faculty of Education, Ehime University, 3, Bunkyocho
Matsuyama, Ehime 790-8577,JAPANE-mail: ohnishi@ehime-u.ac.jp
www. ehime-u.ac.jp
Teruyuki Tamai*

E-mail: tamai.teruyuki.xq@ehime-u.ac.jp
Shinnosuke Mori*

E-mail: mori.shinnosuke.my@ehime-u.ac.jp
Kawada Kazuo**

Faculty of Education, Hiroshima University, 1-1-1, Kagamiyama
Higashihiroshima, Hiroshima 739-8524,JAPANE-mail: kawada@hiroshima-u.ac.jp

Abstract
Control technology using the computer is one of the key technologies for improving energy efficiency. PWM control,
which is a control technology for DC motors, is widely used as the basis of power electronics with the developmentof semiconductor devices. On the other hand, programming techniques are required to effectively use these techniques.
In this paper, we propose teaching materials that can visualize the performance improvement by programming as theoperating time performance index of the wind turbine.
Keywords: teaching materials, electric double layer capacitor, power electronics, control engineering

1. Introduction

Since the original control performance assessment is amethod based on the variance of control error, it isdifficult for elementary and junior high school students
to understand its meaning. Therefore, presenting moreeasy-to-understand index to students as a scale may leadto independent learning.
In this paper, we propose to use the operating time of thewind turbine as a performance index. Using an electric
double layer capacitor (EDLC), which is used as a new
storage device for electrical energy, as a power source,
we provide activities to think about how to use thatenergy efficiently. Teaching materials that can visualize
performance improvement by programming are
explained.

Computer-based control technology is one of the key
technologies for improving energy efficiency. PWM
control, which is a control technology for DC motors, iswidely used as the basis of power electronics with thedevelopment of semiconductor devices. On the other
hand, programming techniques are required to makeeffective use of these techniques. Control performanceassessment[1],[2] which is a measure of how effectively
control technology is working. This method originated inHarris's research[3] and is widely used in the field ofprocess control. The authors of this paper are conducting
research to apply this control performance assessment
method to the achievement evaluation of programming
learning[4],[5].

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

81

Yoshihiro Ohnishi, Teruyuki Tamai, Shinnosuke Mori, Kawada Kazuo

2. The Overview of the developed equipment
The overview of the teaching device is shown in Fig. 1.
The red windmill is rotated by a blue fan. The blue fan isdriven by the Mabuchi DC motor FA-130RA shown in
Fig. 2. The power source for the blue fan is a charged
EDLC. The EDLC used as a power source is shown in
Fig. 3, with a withstand voltage of 2.5V and thecapacitance of 25F. 2.5V constant voltage power supply
was used to charge the capacitance. This is a kit using thevariable regulator NJM2397, which is shown in Fig. 4
and can be purchased at Akizuki Denshi. If the regulator
for charging is turned on, the motor will rotate with theregulator as the power source instead of the EDLC.
Therefore, exclusive control as shown in Table 1 isrequired. A circuit using NAND was constructed toachieve this exclusive control. The blue fan can be driven
by PWM control. The PMW control frequency is 10Hz
and is programmed so that the sum of on-time and offtime is 0.1 seconds. PWM control to the motor is realizedby MOSFET 2SK4017. The circuit diagram is shown in
Fig. 6. A0 in Fig. 5 is the output of the ON / OFF signal
from the computer, and A4 is the AD input formonitoring the voltage of the EDLC. The voltage of the
A4 channel can be monitored with 3.3V as the maximum
value of 100. This relationship is shown as Fig.6.

Fig. 1. The Overview of the developed equipment

Fig. 2. The Drive wind turbine which is driven by FA-130RA.

Fig. 4. Variable regulator.
Table 1. Output signal for FET.
A0
0
0
1
1

Fig. 3. Power supply Electric Double Layer Capacitor withwithstand voltage 2.5V and capacitance 25F.

5V
0
1
0
1

Out
0
0
1
0

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

82

Development of Teaching Materials

Fig. 5. Overall circuit diagram.

Number V

different conditions. First, when the duty ratio was 100%,
the time was the shortest in this experiment. Next, when
the duty ratio is 50% at all times, the time is extended alittle more. However, not all the energy in the EDLC canbe used. Therefore, if the program is switched to a duty
ratio of 50% at first and then to a duty ratio of 100% whenthe voltage drops, the time will be further extended. The
longest time could be obtained by increasing thiscondition. From these results, it can be seen that the timeof the result changes by changing the programming. At
this time, the performance index by programming is theresult time. Students try to improve programming inorder to increase the time of results. Teachers can look attime and evaluate student skills.

voltage

4. Conclusions

Fig. 6. The relationship between the monitoring voltage anddigital number which is loaded by AD input.

In this study, we created teaching materials aimed at
efficiently using the energy stored in EDLC. In order touse energy efficiently, PWM control technology isutilized, and students perform activities to increase thetarget performance index by programming. In actual
products, energy is used efficiently by control technology
such as refrigerators, air conditioners, and even hybrid
vehicles. This teaching material aims to learn that. In the
future, students will be able to carry out activities andtheir effectiveness will be verified.

3. Experimental evaluation
The purpose of the device experiment described in theprevious section is to turn the red windmill as long as
possible with a blue fan. At this time, the number ofrotations of the red windmill does not matter, as long as
it is moving. By adjusting the duty ratio of PWM control
by programming, the speed of the blue fan can bechanged. The control purpose is to give the minimum
energy for the red windmill to move by the blue fan.
Table 2 shows the results of the experiment under© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

83

Yoshihiro Ohnishi, Teruyuki Tamai, Shinnosuke Mori, Kawada Kazuo

Table 2. Experimental result.
Condition(Duty ratio)

Measurement time

D=100%

3m 09s 69

D=50%

3m 31s 99

If V>30, then D=50%
else D=100%
If V>25, then D=50%
else D=100%

4. Y.Ohnishi: Educational Effect Measurement of Control
Technology based on Performance Index, Proceedings of
the SICE Annual Conference 2019, pp. 1323-1327(2019)
5. Y.Ohnishi, T. Nakano, T. Tamai, S. Mori and K. Kawada :Practical Learning of Science for Elementary SchoolStudents via Programming and Control Experimentation,
Journal of Robotics, Networking and Artificial Life, inpress (2021)

3m 56s 83

4m 17s 48

If V>65, then D=20%
else if V>51 then D= 30%
else if V>37 then D=50%

5m49s 51

else if V>27 then D=80%
else D=100%

Acknowledgements
The development of the experimental system was carriedout by Mr. Tomoki Yokohama, Mr. Yuji Mukaigawa andMr. Motoki Nanjyo former students of Ehime University.
And the practical evaluation was done with kindly
support by Mr. Kimitoshi Honda Junior high school
teacher in Ehime prefecture. This work was supported by
JSPS KAKENHI Grant Number JP18K02980.
References
1. B. Huang and S. L. Shah: Performance Assessment ofControl Loops: Theory and Applications, Springer,
London (1999)
2. M. Jelali: An overview of control performance assessment
technology and industrial applications, Control
Engineering Practice, Vol.14, pp. 441-466 (2006)
3. T. J. Harris: Assessment of Closed Loop Performance,
Canadian Journal of Chemical Engineering, Vol.67, pp.
856-861 (1989)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

84

Networking Integration Application of an Intelligent Production Line
for Aerospace Precision Manufacturing
Chau-Chung Songa,b*, Chun-Chi Wangc, Geng-Yi Lina and Chung-Wen Hungdab

Department of Aeronautical Engineering,
National Formosa University, Huwei, Yulin, Taiwan

Smart Machine and Intelligent Manufacturing Research Center,
Nation Formosa University, Huwei, Yunlin, TaiwancGraduate Institute of Aeronautical and Electronic Technology,
National Formosa University, Taiwan, R.O.C.
d
Department of Electrical Engineering,
National Yunlin University of Science &Technology, Taiwan, R.O.C.
Email: ccsong@nfu.edu.tw*

Abstract
In this paper, the networking integration application of an intelligent production line is studied and developed foraerospace precision manufacturing. The technical development of the performance equalization and control for EDM
machine is also introduced with the related technologies of the machine network and EtherCAT network. In addition,
the development of intelligent production lines is integrated to improve the system stability and production efficiency
of production line systems for aerospace component manufacturing. The key technologies of Cyber-Physical System(CPS) and intelligent networking is also integrated to constructs the real-time intelligent system monitoring system(SCADA) for data record and management. The key technologies of the drilling process are developed for intelligent
diagnosis and drilling holes inspection of related production processes. The integrated robot arm is responsible forthe loading and unloading of the material, and the entire production line is connected by the EtherCAT network. The
automatic optical inspection (AOI) is introduced at the rear end of the production line to realize the quality judgmentof the finished product. Furthermore, the virtual cloud network and the intelligent factory are achieved with theinformation collection and intelligent monitoring system.
Keywords: Aerospace Precision Manufacturing, EtherCAT, SCADA, Intelligent Production Linethe Executive Yuan hopes to introduce the manufacturing
line to carry out the aerospace components and thenpromote the domestic tool machine industry to enter theaerospace processing field. The final goal is to promote
domestic machine tools to enter the international
aerospace supply chain. In this paper, the intelligent
networking system is integrated and developed for
production line applications based on the EtherCAT
network, including control of the robotic arm.

1. Introduction
In recent years, as the tendency of the popularity in the
global aerospace industry, the number of airline
passengers has grown year by year. Therefore, there is astrong demand for new civil aircraft development.
However, the machine object equipment mostly relied on
aerospace manufacturing overseas. Because of the weak
confidence level, the domestic aerospace manufacturingindustry goes into a declining phase. Therefore, in 2014,© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), Japan85

Chau-Chung Song, Chun-Chi Wang, Geng-Yi Lin and Chung-Wen Hung

2. Methods

EtherCAT network. The SCADA system is to connect
the EDM, the robotic arm, and the automated opticalinspection system (AOI system) through thecommunication network to establish an aerospace
precision manufacturing, as shown in Figure 2.

2.1 EtherCAT
EtherCAT (Ethernet for Control Automation Technology)
is an Ethernet-based Fieldbus system, invented by
Beckhoff Automation. This technology breaks throughthe system limitations of other Ethernet solutions in thepast and has the characteristics of high availability
requirements, flexible topology, high-performance,
Distributed Clocks for High-Precision Synchronization,and low-cost. The EtherCAT master uses a standard
Ethernet port and network configuration information
stored in the EtherCAT Network Information file (ENI).
The ENI is created based on EtherCAT Slave
Information files (ESI) which are provided by thevendors for each device. Slaves are connected via
Ethernet, any topology type is possible for EtherCAT
networks. EtherCAT architecture is shown in Figure 1.

Fig. 2. Aerospace precision manufacturing

4. System Integration and Testing
The experimental method of Intelligent Production Line
Application is to run the overall electric dischargemachining production line after the EtherCAT local area
network and the SCADA system have been constructed
and connected to monitor. Review the results of drillingat special angles of the workpiece to verify theeffectiveness of the EDM production line automation. To
add the Six-Axis articulated robot to the overallmachining production line network for control, it willbypass its upper controller of the initial factory andconnects to the six motion control cards, which are
regarded as nodes. To control the Six-Axis articulated
robot, use the kinematic calculation method above tocalculate the corresponding value of pulse to the rotation
angle of each axis required to reach the target position.
Then use the CiA402 moving object of the motion
control cards itself for EtherCAT transmission. Then use
the value of pulse which is obtained from the feedback
object to infer the current angle position information.
There are several modes for motion control of the roboticarm, here use interpolated position mode. Also, thecontrol of the electric gripper matched with the robotic
arm uses its USB port to communicate and control withthe computer via UART. Before the SCADA systemstarts to connect and control, it is necessary to use
TwinCAT to create related corresponding objects andsettings through the description of each slave device
XML file to construct an EtherCAT network. After thecompletion of network construction, set the initial
parameters of SDO, and set through the objects in theobject dictionary, then set the PDO Output object that

Fig. 1. EtherCAT Network Architecture

2.2 CANopen over EtherCAT
Using the CoE application layer protocol, EtherCAT
communication has the same standard protocol as
CANopen, Including the same Object Dictionary (OD),
Service Data Object (SDO) settings, Process Data Object(PDO) data transmission, and the objects refer to CAN in
Automation (CiA) 402 for motion control. These objects
can define the state control of the motor, basic motor
parameter settings, and data feedback objects, etc.,
providing objects that can be used for basic motor control
operations.
3. System Design and Architecture
For the development of an intelligent production line
application, the EDM (Electric Discharge Machine) is thecore machine and integrates the robotic arm by the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), Japan86

Networking Integration Application ofwill be added during subsequent operations. Real-Time
numerical control can execute after it into the operational
state. Then, SCADA software can be used to connect tothe network for reading object information and issue
control commands. The SCADA system is to monitors
the EtherCAT network status of the entire production line,
it is mainly connected to the master station TwinCAT via
API. SCADA system can be used to instantly grasp theactions and status of each machine and equipment, andestablish a user interface for users to monitor and manage
the production line status and to provide the inspection
results of the finished workpiece. The SCADA system isshown in Figure 3.

Fig. 4. The interface of TwinCAT master4.2 Six-Axis articulated robot
After connecting its motion control cards via the networkand completing the network communication, the roboticarm can be controlled. It operates on the Cartesian
coordinate system and coordinated with single-axis angle
numerical control of the rear three axes to adjust thedetection angle, and calculated by feedback object toknow the current position information and feedback
information of each axis. Also, a single-axis control isadded for the user to find the grasping point to ease thescheduling design.
4.3 AOI system

Fig. 3. The SCADA system

This AOI system consists of two parts, the automatic
positioning system, and the AOI optical inspection.
Connect the MCU and AOI system to obtain the current
positioning information and display the result of thefinished workpiece inspection on the interface. The
analysis information is shown in Figure 5.

According to the function, C# based SCADA system canbe divided into four parts, device connection, Six-Axis
articulated robot, AOI system, and electric dischargemachine. The functions of each part will be introduced
below.
4.1 Device connection
The main function of the device connection is to connect
with the EtherCAT Master TwinCAT on the production
line and communicate and control transmission with eachnode through this network. In addition to the EtherCAT
network connection, this SCADA system also integrates
other hardware connections, such as the USB connectionof the AOI automatic positioning system, and theconnection control of the electric gripper of the roboticarm. On the network connection, the customized RJ45
interface 100BASE-TX is used as the network physicallayer, which is connected to the main station by serial
connection with each station, and a personal computer isused as the main station. Other devices are connected toa personal computer using USB Port and then controlled
by the SCADA system. The interface of TwinCAT
master is shown in Figure 4.

Fig. 5. Images inspected by AOI system

4.4 Electric discharge machine
Given the issues related to industrial safety, the EDM
manufacturer (Ching Hung Machinery & ElectricIndustrial Co., Ltd.) does not provide control methods
related to EDM computer host and external connection,
like an independent working machine. However, toachieve the goal of automation of the production line, an
EtherCAT network node must be constructed separately.
Through the external relay circuit with the EtherCAT© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), Japan87

Chau-Chung Song, Chun-Chi Wang, Geng-Yi Lin and Chung-Wen Hungnetwork to integrate the Six-Axis articulated robot and
EDM, and design a SCADA system to connect the EDM,
Six-Axis articulated robot, and AOI system via thecommunication network to build an automated network
control station. During the trial run of the overall electric
discharge machining production line, the network
controller issues work instructions, the robotic arm
carries out the reclaiming operation, and the electric
discharge machine is processed. After the machining iscompleted, the robotic arm sends it to the AOI system for
quality inspection. The entire automated production line
operates smoothly. On the network control station, the
SCADA system can be used to grasp the actions andstatus of each machine tool and equipment of the
production line in real-time, and establish a user interface
for users to monitor and manage the production line
status, and to provide the inspection results of thefinished workpiece. This Networking System Integration
and Development of an Intelligent Production Line
Application has been completed, which can implement
intelligent communication between machine tools andequipment, and achieve the goal of intelligent control ofmachining equipment, improve machining quality andproductivity.

communication control board that transmits the signal tothe internal relay of the EDM, then the existing programsin the EDM computer will be activated, the program
including executing, signals of fixed-point feedback, andemergency stop. The control interface is as shown in
Figure 5, the red light is matched with the feedback signal.
While the electric discharge machine is stably ready, thegreen light will be on, and the processing program can bestarted with Start at this time. The workpiece being
drilled by EDM is as shown in Figure 6.

Fig. 6. The workpiece being drilled by EDM

As aerospace consumables workpieces are not available,
it can only be machining through similar workpieces,such as galvanized iron blocks, stainless steel plates,
aluminum plates, and radiator stator scrap for machining
and measurement. During the trial run of the overall
electric discharge machining production line, thenetwork controller issues work instructions, the roboticarm carries out the reclaiming operation, and the electric
discharge machine is processed. After the machining iscompleted, the robotic arm sends it to the AOI system for
quality inspection. The entire automated production line
operates smoothly. The workpieces after machining areshown in Figure 7.

6. References
1. ETG. EtherCAT The Ethernet Fieldbus. (2009).
2. H. Y. Tang, “Key Technologies of Realizing Advanced
Motion Control Design on CNC Machines through the
EtherCAT,” M.S. theses, National Chiao Tung University,
Taiwan, R.O.C., 2015.
3. Y. L. DENG, “Design and Implementation of with
EtherCAT and Low Power Consumption Sensing Systemapplied in Smart machinery,” M.S. theses, National
Formosa University, Taiwan, R.O.C., 2019.
4. ETG. 2200 Slave Implementation Guide V3.1.0. (2018).
5. ETG. 1000 EtherCAT Specification Part 6: Application
layer service definition. (2017).
6. J. J. Craig, “Manipulator kinematics,” in Introduction to
Robotics: Mechanics and Control, 3rd ed., London:
Pearson Education Ltd, pp. 66-76, 2017.
7. J. G. JIANG, “The Development of Chopstick Tube
Feeding Detection System based on the Visual Angle ofRobot Arm,” M.S. theses, National Yunlin University of
Science and Technology. Taiwan, R.O.C., 2019.

Fig. 7. Workpieces after machining

5. Conclusion

Acknowledgements

In this paper, the networking system integration anddevelopment is focused on EtherCAT protocol for an
intelligent production line application, Use the EtherCATThis research was supported by the Ministry of Scienceand Technology, Taiwan, R.O.C. under Grant: MOST
107-2218-E-150-007 and 108-2218-E-150-004.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), Japan88

Low-Cost Indoor Localization Using Sound Spectrum of Light Fingerprints
Chung-Wen Hung

Department of Electrical Engineering, National Yunlin University of Science and Technology. Douliou, Taiwan
Hiroyuki Kobayashi

Osaka Institute of Technology, Osaka, Japan
Jun-Rong Wu

Department of Electrical Engineering. National Yunlin University of Science and Technology. Douliou, Taiwan
Chau-Chung SongDepartment of Electrical Engineering, Department of Aeronautical Engineering, Taiwan
E-mail: wenhung@yuntech.edu.tw, hiroyuki.kobayashi@oit.ac.jp, m10812085@yuntech.edu.tw, ccsong@gs.nfu.edu.tw

Abstract
A low-cost indoor localization system using sound spectrum of light fingerprint is proposed in this paper, an artificial
intelligence, AI, algorithm will be implemented in a low-cost micro-control unit, MCU, to perform the localization.
The unit, complex and tiny differences of the light fingerprints are caused by the different characteristics of thediscrete components used in lighting devices. To reduce the memory size for low-cost MCU, only sound spectrum oflight fingerprint is used to identification the lighting device. Moreover, the grid search also is used to optimize thehyperparameters to compact the AI model. The system architecture and algorithm development are discussed in thispaper, and the experimental results will be present to show the proposed system workable
Keywords: light fingerprint, machine learning, indoor, localization

preprocessing and 1-dimensional convolutional neuralnetwork CNN were calculation on the server, andclassification is performed. [1] and [2] used othercharacteristics for localization, and never mention about
realization. [3] achieved the modularized target, and the
proposed system included an extra high sampling rate
ADC. The method proposed in [4] only needed theacoustic frequency spectrum of the light, it means lower
sampling frequency requirement. Then, all the proposed
localization algorithm was performed in cloud sever. The
proposed method in this paper is similar to [4] withextension and modification, and how to realize the method
into a module will be detailed.

1. Introduction
Indoor localization is an issue in may application,such as automated guided vehicle, smart building.
However, the traditional Positioning method, Global
Positioning System (GPS), is not workable indoors.
Many indoor localization literatures based localization
caught eyes due to infrastructure-less character. However,
how to modularize the localization method is other key
point for using the technology in indoor equipment.
A mobile phone was used to sense the light andconvert it to the received light strength (RLS). This RLS
would be compared with other RLSs on the map, then theposition was located by the similar RLS [1]. Four LEDs
placed on the ceiling were used to transmit the encodedand modulated optical signals with position coordinates,
and the signal received from photodiode receiver was
analyzed to calculated the position [2]. In [3], light signals
were acquired at high frequency, up to 1 megasamples/second, then mapped into the frequency domain
by Fast Fourier Transform (FFT). The k-Nearest Neighbor(KNN) and Convolutional Neural Net (CNN) classifiers
were discussed. The hardware system comprised of ahigh-frequency light sensor, analog-to-digital converter(ADC) scope, and Raspberry-PI processor to theperformance of the classifiers. [4] used the microphonebuild in the mobile phone to sampling the light signal
sensed by a photodiode (PD), then the sampling were sent
to the cloud server through the network. The FFT

2. Data Acquisition
The text is to be typeset in 10 pt Times Roman, single
spaced with interline spacing of 13 pt. Text area
(excluding running title) is 6.75 inches across and 8.8
inches deep. Final pagination and insertion of running
titles will be done by the publisher, so make sure that nopage numbers are given in your paper and only therunning titles provided in this template (authors’ names
and paper title) are used.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

89

Chung-Wen Hung, Chau-Chung Song, Hiroyuki Kobayashi, Jun-Rong Wu

A Data Acquisition and feature spectrum
extraction

The spectrums of these two fluorescent lamps aredifferent, and the most features of the spectrum locate at
low frequency interval. Note, here is a huge spike at
120Hz no matter lamp 1 or 2, because fluorescent lights
is powered by magnetic ballasts with a frequency of 60Hz
(50Hz in some country) and lamps flicker with the double
frequency. The portion less than 150 Hz will not be usedin localization. In the other side, there are also somepeaks around 32K Hz. The portion near Nyquist
frequency is also ignored. Only the spectrum from 150
Hz to 30K Hz is used to machine learning andlocalization. The Fig. 3 and 4 are the spectrums of the twolamps in selected frequency range

The analog to digital converter, ADC, build in MCU
is used to sample the time domain data. First, higher
sampling rate and the larger number of sampling points isselected to confirm the feature frequency, first. Then, thesuitable sampling rate and point will be optimized, means
to minimize these two number but keep the feature. The
preprocessor is discrete Fourier transform, DFT, topresent the frequency feature of the sampling data. The
formula is show in (1), 𝑋𝑋[. ] and x[.] represent respectively
the frequency and time domain data; 𝑘𝑘 and n denotes thek-th point frequency and n-th point sampling data, and N
is the totally sampling points. The frequency resolution,∆𝑓𝑓, of FFT is as (2), here, 𝑓𝑓𝑠𝑠 indicates the sampling rate
(frequency). The typical FFT results of two fluorescentlamps are shown in Fig. 1 and 2, the 16,384 points time
domain data sampled at 65536Hz.
𝑁𝑁−12𝜋𝜋𝜋𝜋𝜋𝜋𝑋𝑋[𝑘𝑘] = � �𝑥𝑥[𝑛𝑛] × 𝑒𝑒 −𝑖𝑖 𝑁𝑁 � ,
𝑛𝑛=0

𝑓𝑓𝑠𝑠
𝑁𝑁

150

For modularization, the artificial intelligencealgorithm will be implemented into an MCU, whichcalculation and memory capabilities are limited. Smaller
node number of input layer means less operation andmemory requirement. The reduction of the input nodes but
without losing frequency feature is important. As shownin Fig. 3 and 4, the low frequency spectrum contains mostinformation, so frequency rescaling equation will be
adopted to cover original FFT data to the characteristic
vector[4]. The frequency axis will be rescaled to power of𝐹𝐹𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 as shown in (3), and the 𝐹𝐹𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 is shown as (4).
Here, 𝑓𝑓𝑚𝑚𝑚𝑚𝑚𝑚 and 𝑓𝑓𝑚𝑚𝑚𝑚𝑚𝑚 indicate the frequency range which
is used in machine learning, and 𝑁𝑁𝐹𝐹_𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 is the target
number of reduction. Finally, the amplitude of rescaling
spectrum, 𝐴𝐴[𝑖𝑖], is the summation of the amplitude which
frequency between two rescaling frequency axis point. (5).

(1)

𝑘𝑘 = 0,1, … , 𝑁𝑁 − 1∆𝑓𝑓 =B Rescaling(2)

Signal-2

X 120
Y 146.4592

Amplitude

100

50

𝑖𝑖

0
0

0.5

1

1.5

2

2.5

3

3.5

Frequency(Hz)104Fig. 1 The spectrum of the fluorescent tubes 1

𝐴𝐴[𝑖𝑖] =120
Signal-2

X 120

(3)

𝐹𝐹(𝑖𝑖) = �𝐹𝐹𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 �
𝑓𝑓𝑚𝑚𝑚𝑚𝑚𝑚 − 𝑓𝑓𝑚𝑚𝑚𝑚𝑚𝑚 𝑁𝑁 1
𝐹𝐹𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 = () 𝐹𝐹_𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡
𝑓𝑓𝑚𝑚𝑚𝑚𝑚𝑚

Y 116.6494

𝐹𝐹𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 𝑖𝑖 ≤𝑗𝑗∗∆𝑓𝑓≤𝐹𝐹𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 𝑖𝑖+1

�
𝑗𝑗

4

100

(4)

|𝑋𝑋[𝑗𝑗]|Lamp-1

3.5
80

2.5

Amplitude

Amplitude

3
60

40

2

1.5
20

1
0
0

0.5

1

1.5

Frequency(Hz)

2

2.5

3

0.5

3.5
10

4

0

Fig. 2 The spectrum of the fluorescent lamp 2

500

1000

1500

2000

2500

3000

3500

4000

Frequency(Hz)Fig. 3 The spectrum of 150 to 4000 Hz for the lamp 1© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

90

(5)

Low-Cost Indoor Localization

Grid search is standard skill to get the hyperparameter
combination of the highest accuracy [7], such as CNN
Filter Kernel number and size. Different from generalgrid search which only focus on accuracy, the memory
size of parameter is also a key factor for modularization.
The grid search items and the range is expressed in tableI, here are totally 240 parameter combination.

3
Lamp-2

2.5

Amplitude

2

1.5

1

Table I. The items and parameter of grid search

0.5

Item
CNN Filter Kernel number

Parameter
1、2
2、4、8、16、32、64、
CNN Filter Size
128、256
Activation Function of Fully Connected layer ReLU、Tanh、Sigmoid
64、32、16、8、4
Node of Fully Connected layer

0
500

1000

1500

2000

2500

3000

3500

4000

Frequency(Hz)Fig. 4 The spectrum of 150 to 30k Hz for the lamp 2

The Fig 5. presents the rescaling results of FFT for
the two lamp. Here, 𝑓𝑓𝑚𝑚𝑚𝑚𝑚𝑚 and 𝑓𝑓𝑚𝑚𝑚𝑚𝑚𝑚 are set to 150 and 30k
Hz, and rescaling points are 400. The rescaling results
keep the original features in low frequency spectrum, andhigh frequency’s is summed into low resolution form.
However, the input data are 75 times smaller, this
rescaling processing will reduce the requirement ofmemory and calculation dramatically. In other words, it
is very helpful for modularization in a low-cost MCU.

Localization is designed for eight lamps position, and
100 averaged rescaling data is collected for each lamp fortraining. To eliminate random errors caused by initial
random weights, the three training results for eachparameter combination are averaged. The iteration
number set to 1000, and early stopping is adopted to
avoid overfitting per training round. The grid search
results are sorted by verified accuracy and parameter filesize. In the 240 training results for all parameter
combination, the top ten results are shown in Table 2.

3
Lamp-1Lamp-2

2.5

Table 2. The Top Ten Results Gid Search

Amplitude

2

1.5

Train. Train.
Acc.
Loss

1

0.5

0
10

2

10 3

10

4

1
1
1
1
1
1
1
1
1
1

Frequency(Hz)Fig. 5 The rescaling result for the lamp 1 and 2

3.

Identification Algorithm and Grid Search forHyperparameters

CNN is a popular algorithm for classification due tothe its ability to learn to extract the features, and lessparameter memory and calculation requirement issuitable for modularization [5]-[6]. 1-D CNN is adopted
in this paper due to the resource limitation in low-cost
MCU, no matter memory or calculation speed. The filter
kernel in convolution layer is used to extract certain
features from rescaling spectrum, and full connected
layers perform the classification, but pooling layer isskipped in this application.

Verified Filter Filter
Acc.
Kernel Size

0.000577 1
0.010547 1
0.051400 1
0.010734 1
0.048613 1
0.010579 1
0.085243 1
0.011359 1
0.002533 1
0.010598 1

1
1
1
1
1
2
2
1
1
1

256
256
256
128
128
256
256
64
256
256

Nodes of
Fully
MemoryConnected Size (kB)
Layer
ReLU 32/16/16 5.96875
Tanh 32/16/16 5.96875
Sigmoid 32/16/16 5.96875
Tanh 32/16/16 9.71875
Sigmoid 32/16/16 9.71875
Tanh 32/16/16 10.4687
Sigmoid 32/16/16 10.4687
Tanh 32/16/16 11.5937
Tanh 64/32/32 12.9375
Sigmoid 64/32/32 12.9375

Act.
Func.

The parameters of CNN model are detailed in Table
3, the some parameters are get from the grid search. The
settings of fully connected layer number and activation
function set to 3 and tanh [4]. The number of output nodeis eight, its form is likelihood by softmax function.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

91

Chung-Wen Hung, Chau-Chung Song, Hiroyuki Kobayashi, Jun-Rong Wu

4. Implementation

The signal processing flow is dedicated in Fig. 7,
moreover, the processing time of every sub processing
block is also shown. The longest duration is dataacquisition. However, it is necessary, because the data issampled three times, and there is 16,384 points sampled
at 65,536 Hz every time. The transfer time from time
domain data to frequency domain may be shorted if fast
MCU. And before localization AI algorithm, therescaling FFT results of three samples are averaged.

The main target in this paper is a low–cost indoor
localization module. The RX65N MCU provided byRenesas Electronic Corporation is used to implemented
the proposed algorithm. The build-in 2 M bytes of read
only memory, ROM, and 640 K Bytes random accessmemory, RAM, are enough to install the optimized AI
model, and the high-speed operation with 32bit floating
point could support the calculation requirement of FFT.
Table 3 The Items and Parameter of Grid Search
Item
CNN Filter Kernel number
CNN Filter Size
Activation Function of CNN
Node of Fully Connected layer 1
Node of Fully Connected layer 2
Node of Fully Connected layer 3
Activation Function of Fully Connected layer
Node number of Output layer
Activation Function of Output layer

Parameter
1
256*1
ReLU
32
16
16
Tanh
8
SoftMAX

Fig. 8 The signal processing flow is dedicated

5. Experiment
A Setup
The experiment was performed in the EN303 roomon third floor of the Engineering sixth hall, National
Yunlin University of Science and Technology, as shownin Fig. 8. The proposed module is located about 1.7
meters below lamps. The numbers denote the lamps. The
floor plan is revealed in Fig. 9, which also includes thelocation of lamps, the distance between each lamps andfrom lamp to hall. Note, the ‘X’ marks indicate thelocation when testing the lamp, and the ‘+’ marks

shows the position when testing not below thelamps.

Fig.6 The architecture diagram of system

The proposed system is shown in Fig. 5, it includes asensor circuit, amplifier block and MCU. The sensor
circuit consists of a photodiode and resister, and theamplifier block is implemented with operational
amplifier to perform the gain and filter functions. The
system hardware is present as Fig. 6. Note, the hardware
is only for experimentation but not size-optimization yet.

Fig. 9 Experiment in the EN303 room

Fig. 7 The photo of proposed hardware© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

92

Low-Cost Indoor Localization

Table 4. (Continued)

B Results
The estimated likelihoods of classification are shownin Table 4, the proposed module is placed below thelamps which are marked by ‘X’ turn by turn. The numberin the table indicates the most likelihood lamp number,and the number in the parentheses shows the estimatedlikelihood. The experiment ran 10 turns, and every ‘X’point is tested every turn. The localization accuracy
performed 100%. The least likelihood in the table locatesat lamp 4 in the turn 8, however the likelihood stillachieved 75.8%, it demonstrated the localization iscorrect.
The Table 5 present an interesting estimation results,
the average likelihood of 10 turns is expressed bypercentage, when the proposed module is placed at mark‘+’ position. The two adjacent likelihoods shown in bold
are expected to be biggest two number, because thetesting position is located between the two lamps
indicated in the first line of table. However, theexpectation was correct such as position 9, 11 and 13, but
incorrect for other points. The reason should be that the
sample located the position 9 to 14 never be adopted for
AI model training and the estimated likelihood is not theinterpolation of any two training points. The future work
is to adopt such position sample to machine learning toevaluate the localization effect if the light sampled not
below the lighting source.

6
7
8
9
10

1(98.9) 2(99.0) 3(98.9) 4(96.4) 5(98.2) 6(98.8) 7(98.5) 8(99.1)
1(98.6) 2(99.0) 3(98.5) 4(94.2) 5(98.2) 6(98.9) 7(98.5) 8(99.1)
1(98.8) 2(98.9) 3(96.4) 4(75.8) 5(98.2) 6(98.9) 7(98.5) 8(99.0)
1(98.7) 2(98.9) 3(98.5) 4(93.9) 5(98.0) 6(98.8) 7(98.5) 8(99.1)
1(98.9) 2(98.9) 3(95.0) 4(90.8) 5(98.1) 6(98.8) 7(98.3) 8(98.8)

Table 5. Likehood Estimation Between Two Lamps
Pos. Lamp1 Lamp2 Lamp3 Lamp4 Lamp5 Lamp6 Lamp7 Lamp8
9
10
11
12
13
14

28.333
0.113
0.138
0.033
0.011
0.080

26.87980.002
0.054
0.652
1.321
0.034

1.589
7.090
38.109
0.017
0.046
0.004

3.464
0.288
60.799
0.123
0.267
0.193

36.366
0.033
0.490
0.509
0.757
0.657

1.612
10.3780.056
97.52122.136
0.976

1.506
1.790
0.300
0.863
74.460
1.443

0.251
0.306
0.055
0.283
1.001
96.6206 Conclusion
In this paper, a low-cost indoor localization systemusing sound spectrum of light fingerprint is proposed. To
use this light fingerprint based localization indoor mobile
equipment, modularization is the key consideration. So,
the memory and calculation of the selected MCU arelimited due to cost. Only the rescaling frequency features
between 100 Hz to 30k Hz are used in classification, anda simplified 1-D CNN is performed to classify the lamps.
Grid search is adopted to optimize the hyperparameters
to find the highest-accuracy and smallest-size model. The
experiment shows the system could be implemented in alost-cost MCU with few components. The on-line tests
present the localization accuracy achieve 100% when the
proposed device located below the lamps. However, notas expected, when testing position locates between lamps,
the likelihood is not an interpolation of the nearest twolamps’. It may be the future works of this topic.
Acknowledgment
Part of this work was financially supported by the“Intelligent Recognition Industry Service ResearchCenter” from The Featured Areas Research Center
Program within the framework of the Higher Education
Sprout Project by the Ministry of Education (MOE) in
Taiwan. And it is also supported by Ministry of Scienceand Technology, Taiwan, under contract MOST 1082221-E-224-045- and 108-2218-E-150-004-.

Fig. 10 Floor Experiment in the EN303 room
Table 4. Like hoods of Classification of the Turns
Turn Lamp1 Lamp2 Lamp3 Lamp4 Lamp5 Lamp6 Lamp7 Lamp8
1
2
3
4
5

1(98.8) 2(98.9) 3(98.9) 4(97.6) 5(96.2) 6(98.9) 7(98.4) 8(99.1)
1(98.6) 2(98.9) 3(97.2) 4(97.6) 5(98.1) 6(98.8) 7(96.5) 8(99.1)
1(98.2) 2(98.9) 3(98.2) 4(95.3) 5(98.2) 6(98.8) 7(98.5) 8(99.0)
1(98.5) 2(98.9) 3(98.6) 4(93.1) 5(96.9) 6(98.9) 7(98.3) 8(99.1)
1(97.4) 2(99.0) 3(91.3) 4(95.1) 5(97.3) 6(98.9) 7(98.5) 8(98.9)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

93

Chung-Wen Hung, Chau-Chung Song, Hiroyuki Kobayashi, Jun-Rong Wu

References
[1]

[2]

[3]
[4]

[5]

[6]

[7]

Y. Hu et al., "Lightitude: Indoor Positioning Using
Ubiquitous Visible Lights and COTS Devices," 2015 IEEE
35th International Conference on Distributed Computing
Systems, Columbus, OH, 2015, pp. 732-733.
C. Guo, J. H. Shao, W. Ke and C. Y. Zhang, "A Visible
Light Indoor Positioning Algorithm Based on
Fingerprint," 2018 4th Annual International Conference on
Network and Information Systems for Computers(ICNISC), Wuhan, China, 2018, pp. 71-77.
S. Hamidi-Rad, K. Lyons and N. Goela, "Infrastructureless indoor localization using light fingerprints," 2017
IEEE International
H. Kobayashi, "CEPHEID: the infrastructure-less indoor
localization using lighting fixtures' acoustic frequency
fingerprints," IECON 2019 - 45th Annual Conference of
the IEEE Industrial Electronics Society, Lisbon, Portugal,
2019, pp. 6842-6847.
T. Ince, S. Kiranyaz, L. Eren, M. Askar, and M. Gabbouj,‘‘Real-time motor fault detection by 1-D convolutional
neural networks,’’ IEEE Trans. Ind. Electron., vol. 63, no.
11, pp. 7067–7075, Nov. 2016.
S. Kiranyaz, T. Ince, and M. Gabbouj, “Real-timepatientspecific ecg classification by 1-d convolutional
neural networks,” IEEE Transactions on Biomedical
Engineering, vol. 63, no. 3, pp. 664–675, 2016
Sun, Y.; Wang, Y.; Guo, L.; Ma, Z.; Jin, S. The
comparison of optimizing SVM by GA and grid search. In
Proceedings of the 2017 13th IEEE International
Conference on Electronic Measurement & Instruments(ICEMI), Yangzhou, China, 20–22 October 2017.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

94

Air Valve Fuzzy Control Combined with Sheet Music Recognition Techniques Applied toAutoplaying Soprano Recorder Machines
Chun-Chieh Wang1, Guang-Ming Jhang2
1

Department of Automation Engineering and Institute of Mechatronoptic Systems, Chienkuo Technology University, Taiwan(Tel: 886-4-7111111ext3350, Fax: 886-4-7111164)
1
jasonccw@ctu.edu.tw
2
Department of Electronic Engineering, Chienkuo Technology University, Taiwan

Abstract

In the past research, there are many disadvantages to score recognition and flute performance. Therefore, this paper would
improve on these two points. For the part of music score recognition, we use the y-axis projection method to detect the staff
position and remove it to replace the erosion and expansion in morphology. It was found that the notes have a specific writing
style on the staff. Therefore, this feature is used to distinguish the notes. For the soprano recorder playing part, the fingershaped electric arm is used to press the blow hole, it will cause the situation that the speed of the score cannot be kept up.
Therefore, the motor is changed to a solenoid valve to facilitate the pneumatic cylinder to smoothly press the blow hole. In
addition, since the difference in pitch of the soprano recorder requires different air pressure, we increase one valve to threevalves. Moreover, the range is divided into bass, midrange, and treble. Not only that, fuzzy control theory is used to control
the air valve to greatly improve the sound distortion caused by the original single air valve. Experiments prove that the air
valve fuzzy control combined with sheet music recognition techniques can fully realize the functions of autoplaying soprano
recorder machines.

Keywords: Autoplaying Soprano Recorder Machines (ASRM), LabVIEW, Arduino, Air Valve Fuzzy Control,
Pneumatic Cylinder
Due to the addition of a variety of staff removal and note
recognition methods, the overall score recognition time was
extended. Lee H.W. [3] and Peng Z.X. [4] use opticalsymbol recognition (OMR). Since it must build a symbol
database, it is possible to recognize errors as long as there
are slight errors. In 2009, Lan S.M. [5] presented themusical notation recognition system for two-wheeled robot.
The disadvantage is that the recognized scores are only
digital scores. In 2014, Wang B.R. [6] proposed a musicalscore recognition system for iOS devices, because its
recognition process requires human assistance, so there is
no way to achieve the effect of automatic recognition.
Based on the shortcomings of the above-mentioned
scholars, this article proposes air valve fuzzy control
combined with sheet music recognition techniques toimprove the recognition of music scores and improve thephenomenon of out of sound. Experiments prove that
presented method can fully realize the functions of ASRM.

1 INTRODUCTION
Nowadays, robots are no longer just used in industrial
production, and they can also be seen in medical and artistic
fields. The International Robotic Art Competition, whichstarted in 2016, has also been held three times. Many works
created by robots have reached a level comparable tohuman artists. As for performance robots, there have beensignificant advances and improvements due to artificial
intelligence. Among them, the research on music scorerecognition technology has been proposed in many
documents. Although the music notation is limited andthere are restrictions on the writing position on the staff.
However, there are still many blind spots to be overcome
and improved in identifying it with artificial intelligence
technology, even if it is only for non-handwritten scores.
In 2018, Yang S.F. [1] proposed sheet music detection
techniques applied to fluted recorder robots. The
disadvantage is that the mechanism is poorly designed,
which often leads to inaccurate beats. In 2004, Tsai Z.W.
[2] proposed the recognition system of printed music score.

2 ASRM
To improve the problem that the finger-shaped robotic© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

95

Chun-Chieh Wang, Guang-Ming Jhang

arm cannot keep up with the rhythm of the music, the motor
is changed to a solenoid valve and combined with theArduino UNO board to control the air cylinder to press theblow hole. The finished ASRM is shown in Fig. 1.

control part has two more solenoid valves. It solves the
problem of broken sound when blowing.
Blow hole and press
control mechanism
Blow hole and pressing mechanism
under soprano recorder

Autoplaying control
mechanism

Fig.1. Finished product of the ASRM

Fig.4. The physical architecture of the ASRM platform

2.1 Control Systems
The Arduino UNO board is used as the control core of
the ASRM. LINX in LabVIEW Graphical Programming
environment is the bridge between LabVIEW program andArduino UNO board. Like LIFA, LINX must first write aset of communicable commands in Arduino. The differenceis that LIFA can provide users with the ability to directly
use Arduino connection operation in LabVIEW without
writing Arduino code. The LINX kit is an open source kit
developed by MakerHub. Its feature is that it cannot only beconnected to Arduino, but also supported by many
development platforms such as myRIO/Digilent/. Moreover,
LINX can directly burn the firmware required for LINX
execution on the Arduino through this LINX Firmware
wizard. The control system architecture diagram is shownin Fig. 2. Fig. 3 is the Arduino UNO control board.

3 MUSIC SCORE RECOGNITION SYSTEMFirst, we remove the parts of the score that are notrelated to performance, as shown in Fig.5. The processedmusic score is input into the system. After that, binarizationand scale recognition are performed through LabVIEW.
Fig.6 is its program flow chart.
Input score

Binarization

Remove staff

Rest

Recognize rests

Distinguis
h notes

Note

Remove note
stems

Is it continuousnotes?

Small servo motor

YES

Continuous
notes

NO
Discontinuous
notes

USB/Serial
Arduino UNO
main control board

Air compressor

Provide air pressure

PC

Identify themusical scale

Performance
arrangement

Solenoid valve

Export to ASRM
Platform

Pneumatic
cylinderFig.2. The control system architecture diagram

Fig.3.
ArduinoUNO

Fig.5. Score preprocessing

Fig.6. Program flow chart

We use the y-axis projection method to detect the staff
position and remove it to replace the erosion and expansion
in morphology. It was found that the notes have a specificwriting style on the staff. Therefore, this feature is used to
distinguish the notes. First, the note stems are distinguishedand removed by the x-axis projection detection method.
Second, the center point of the head of the note and theposition of the staff are identified by the pixel clustering
method. Finally, the encoding of the scale is arranged insequence from bass to treble.

2.2 ASRM Platform
The physical architecture of the ASRM platform isshown in Fig. 4. The main feature is that the part of theoriginal flute that presses the blow hole is pressed by thesolenoid valve controlled pneumatic cylinder. Significantly
improved the original inability to play music faster than the
response speed of the motor. Not only that, the blowing

96

Air Valve Fuzzy Control

3.1 Binarization
Binarization is to divide the grayscale image into black
and white according to the threshold set by the user. Whenthe grayscale value of the pixel is greater than the threshold,it is judged as a white point, otherwise it is a black point.
The grayscale image can be converted into a binary image
through binarization, as shown in Fig.7.

Fig.10. Quarter noteFig.11. Quarter rest

To distinguish between continuous notes anddiscontinuous notes, the image is projected on the x-axis
using the x-projection in the orthogonal projection method.
The note stems exceeding a certain value are removed, as
shown in Fig.12. Moreover, the pixel clustering method
will be used to distinguish the following three types:
discontinuous notes, continuous notes, and discontinuousnotes but with tails, as shown in Fig. 13.

Fig.7. Difference before and after binarization
3.2 Remove staff
We use the y-projection in the orthogonal projection
method to project the music score to be identified onto asingle axis, which will form a graph called the projection
profile, as shown in Fig.8. We can clearly find the positionof the staff and remove it. This method greatly improves theoriginal use of erosion and swelling in typology to cause
unclear symbols, as shown in Fig.9.

(a) The result after removing the staff(b) The result after x-axis projection(c) The result after removing note stems

Fig.12. The process of removing note stems(a) discontinuous notes(a) Original staff(b) continuous notes(c) discontinuous notes but with tails(b) The result after y-axis projection

Fig.13. Comparison chart after removing note stems(c) The result after removing the staff

The pixel clustering method uses the IMAQ Count
Objects 2 VI component in the Vision development modulein LabVIEW. The function of this component is to cluster
the binarized pixels. Let the user set the pixel threshold todistinguish the number of pixel groups. Fig.14 is the resultof pixel cluster identification.

Fig.8. The process of removing the staff(a) Original image (using the previous method)(b) The result after removing the staff (using the previous method)

Fig.9. The process of removing the staff(using theprevious method)
3.3 Symbol distinction
For rests, this article only deals with the commonly usedfull rests, two-quarter rests, four-quarter rests, eighth rests,
and sixteenth rests. General notes are also processed forcommonly used whole notes, half notes, quarter notes,
eighth notes, and sixteenth notes. Generally speaking, theheight of most rests is not greater than the height of thenotes. For example, the height of a quarter note on the staff
is approximately equal to the height of the staff, as shownin Fig. 10. The height of the quarter rest is only 3/4 of theheight of the staff, as shown in Fig.11. Based on this, thenotes are preliminarily divided into rest notes and generalnotes to facilitate subsequent identification.

Fig.14 The result of pixel cluster identification
3.4 Scale recognition
While the scale recognition is performed, the note timingis also recognized. Therefore, the head and tail of the notemust be distinguished first. First, the discontinuous notes
have been separated out after removing the note stem, so

97

Chun-Chieh Wang, Guang-Ming Jhang

there is no need to deal with it. The part with discontinuous

the black pixel ratio in the red frame to make a preliminary
judgment, as shown in Fig.20. Second, make a detailed
distinction for the above three types. (1) The image
coordinates of the y-axis projection of the full rest and thebipartite rest are captured and compared with the staff
position. If the image coordinates are closer to the fourthline of the staff, it is judged as a whole rest. If thecoordinates are closer to the third line, it is a half rest. (2)Compare the height value captured after y-axis projection
with the highest point value after x-axis projection. If thevalues are similar, it is judged as a quarter rest, as shown in
Fig.21. (3) The 8th rest has only one peak in the y-axis
projection image. The 16th rest has two peaks, as shown in
Fig.22 and Fig.23.

notes but with tails is distinguished by the proportion of
black pixels in the image extracted by pixel clustering. The
pixel ratio calculation is based on the range enclosed by thered frame to calculate the image size and the ratio of black
pixels, as shown in Fig.15. Continuous notes cannot be
identified by this method because the proportions of black
pixels of note stems and note heads are too close. Therefore,
we use the aforementioned note stems to account for morethan 2/3 of the overall note width to distinguish. The
position of the beam will change due to the way the music
theory is written. So we divide the continuous note from thecenter point into the upper and lower parts, as shown in Fig.
16. ((a) Original image with Note head at the bottom, (b)Original image with Note head at the top)(a) Note head (80%)(b) Note tail (30-40%)

Fig.15. Distinguish of discontinuous notes but with tails
=

+

(a)

=

+

Fig.19. Scale press fingering table of the soprano recorder

(b)

(a)

Fig.16. The process of cutting continuous notes(b)(a) Whole rest and half rest (99%)(b) Quarter rest (50%~70%)

Fig.20. Distinguish between rests

For judgment of discontinuous notes, the difference
between discontinuous notes is whether the beam and thehead are hollow. We use the ratio of black pixels in the redbox to distinguish half notes from quarter notes, as shownin Fig.17. Because we have already recorded the position of
the staff while removing the staff. Therefore, we only needto extract the center point of the talisman through the pixel
clustering, and then compare it with the previously recorded
staff position to know which line or interval the talisman is
located on. Put it into the scale table of the recorder to get
the scale of the note, as shown in Fig.18 and Fig.19.

(a) Original image(b) y-axis projection(c) x-axis projection

Fig.21. Quarter rest(a) Original
image(b) y-axis
projection(c) x-axis
projection

Fig.22. 8th rest(a) Solid note head (80%)(b) Hollow note head (50%)

Fig.17. The solid and hollow of the note head(a) Original image(b) y-axis projection(c) x-axis projection

Fig.23. 16th rest
4 AIR VALVE FUZZY CONTROL DESIGN

Fig.18. Scale table of the soprano recorder

To provide a smoother air pressure with different soundranges, this article increases the number of solenoid valves
to 3. In addition, we use the fuzzy control law to write thevalve control program. The range (R) is divided into bass,
midrange, and treble, as shown in Fig.24.

3.5 Rest identification
The identification of rests is double identification using
orthogonal projection method and pixel clustering.
According to the appearance of commonly used rests, they
are divided into three types: (1) whole rest and half rest, (2)quarter rest, (3) 8th rest and 16th rest. First, we use thepixel clustering method to capture the rest image, and use

98Air Valve Fuzzy Control[4] Peng Z.X. (2014), Automatic Music Score Recognition
System Using Digital Image Processing, Master Thesis of Chung
Yuan Christian University.
[5] Lan S.M. (2009), Musical Notation Recognition System for
Two-Wheeled Robot, Master Thesis of National Taiwan
University of Science and Technology[6] Wang B.R.(2014), A Musical Score Recognition System for
iOS Devices, Master Thesis of National Kaohsiung First
University of Science and Technology[7] Egg Head Liu
https://www.youtube.com/channel/UCC6UAv477s0YiQIfioIj0Vw

Tone number

bass

midrange

trebleFig. 24. The range
Define the opening degree of air pressure valve be V.
TS1, TS2, and TS3 belong to Takagi-Sugeno type. Fuzzy
IF-THEN rules are expressed as follows.
IF R is bass, THEN V is TS1.
IF R is midrange, THEN V is TS2.
IF R is treble, THEN V is TS3.

(Search date: 12/15/2020)[8] TSAI,JU-HSIN
https://www.youtube.com/channel/UChE9nxA6DKQUvsJHBSgUJQ (Search date: 12/15/2020)

5 EXPERIMENTAL RESULTS
In view of the large differences in the way of notation fortuplets (Fig.25(a)) and dotted notes (Fig.25(b)) bymusicians around the world, it is impossible to effectively
identify all these two symbols with a general rule.
Therefore, this article will ignore it during the experiment.
In addition, the score used for testing is taken from a website made by netizens and provided free of charge [7, 8].
The scores we used to test included 5 Mandarin pop songs,
2 Japanese pop songs, and 3 movie theme songs. Pleaserefer to the following URL directly for the experimental
results. https://www.youtube.com/watch?v=1Z-kPIPQG-U

(a)

(b)

Fig. 25(a). TupletsFig. 25(b). Dotted notes

6 CONCLUSION
This paper uses the pixel clustering method, the x & yaxis projection method to successfully improve the scorerecognition results. Moreover, the finger-shaped electricarms are changed to solenoid valves to facilitate thepneumatic cylinder to smoothly press the blow hole. Thismethod has also successfully improved the phenomenon of
air leakage. Not only that, fuzzy control theory is used tocontrol the three air valves to greatly improve the sounddistortion caused by the original single air valve.
REFERENCES[1] Yang S.F. (2018)，Sheet Music Detection Techniques Applied
to Fluted Recorder Robots, Master Thesis of Chienkuo
Technology University.
[2] Tsai Z.W. (2004), Recognition System of Printed Music Score,
Master Thesis of National Sun Yat-sen University.
[3] Lee H.W. (2016), Automatic Music Score Recognition,
Master Thesis of National Tsing Hua University.

99

Image Inpainting Techniques Combined with Isolated Pixel Filtering Applied to Multifunctional
Drawing Robots
Chun-Chieh Wang1, Zhan-Xian Ye2
1

Department of Automation Engineering and Institute of Mechatronoptic Systems, Chienkuo Technology University, Taiwan(Tel: 886-4-7111111ext3350, Fax: 886-4-7111164)
1
jasonccw@ctu.edu.tw
2
Department of Electronic Engineering, Chienkuo Technology University, Taiwan

Abstract

The purpose of this thesis is to assist street painters to improve their work efficiency. That is, the simpler part of thedrawing is drawn by the robot. Once the initial picture is completed, the artist will take over the painting. In thispaper, HSI color space is used to improve the effect of color simplification such that the recognition of image
processing results is enhanced. Moreover, the isolated pixel filtering is used to replace the less-affected isolated
point color with the surrounding color. Furthermore, image inpainting techniques are utilized to reduce the distortion
caused by the isolated pixel filtering. Besides, we adjusted the path planning as well as reduced isolated points todramatically reduce drawing time. At the same time, LabVIEW can issue commands directly to control the robot byadding the communication function. In addition, to achieve the multifunctional drawing robotic mode, this system
adds the sketch function. Through the image resolution adjustment as well as the shortening of the spacing of thedrawing lines, the robot can draw more detailed pictures in the same size of drawing space. The measured results
confirm that the application of the technology in this paper can shorten the drawing time by about 55% on themultifunctional drawing robot system.

Keywords: Isolated pixel filtering, Image inpainting, Watercolor, Sketch, Multifunctional drawing robots[4] used online PNG to SVG software to simplify the colors
used Linux CNC with a multi-axis robotic arm for drawing.
For the sketch drawing part, Lin L.C. [5] used edge
detection to generate contour images and used mobile
robots to draw on a flat surface in 2017. In 2009, Mac T.T.
[6] and Chuang L.W. [7] used PI and PID controllers toadjust motors to control multi-axis robotic arms to draw
portraits, respectively. The above three documents are all
drawn with the edge of the image. In 2016, Jhang G.M. et
al. [8] used multi-layer binary images to overlap drawing. It
focused on the performance of shadows to achieve theeffect of light and shadow drawing.
In order to improve the above shortcomings, we apply
image inpainting techniques combined with isolated pixel
filtering to multifunctional drawing robots.

1 INTRODUCTION
In view of that many street painters usually takes a longtime to draw a picture such that it is impossible to produce
a large amount in a short time. Therefore, we hope todevelop a drawing robot that can assist painters. With thehelp of robots, the artist can complete more paintings in thesame time and improve drawing efficiency.
In the literature of robot drawing in the past, the image
detection applied in a watercolor painting robot proposedby Hung YY [1] in 2017. It is characterized by using HSVcolor simplification method with a self-made drawing
mechanism, and then using a limited variety of colors tocomplete the drawing. Although the recognition of theimage is acceptable, the long drawing time is a major
disadvantage. In 2016, Hong M.J. [2] proposed robotics
artistic colorful picture drawing and painting using visualfeedback control system. Two different painting styles wereused. 2017 Columbia University Creative Machines Lab /PIX18 [3] uses Gantry multi-axis robotic arm to draw
pictures with artificial intelligence. In the same year,
Massachusetts Institute of Technology Dane Kouttron et al.

2 RESEARCH METHOD
2.1 HSI color model and color simplification
Digital images often use the RGB color space to definecolors. The parameters of the RGB color space are allconnected, which makes image processing difficult. In
contrast, the parameters of the HSI color space are© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

100

Chun-Chieh Wang, Zhan-Xian Ye(Dark, light, primary, black, gray and white) is set.
Moreover, the color spectrum established by the HSI color
space is used for adjustment. Since everyone's perception ofcolor is different, this article is based on the author's
perception of color to simplify the range of adjustments.

independent of each other, namely Hue (H), Saturation (S),Intensity (I). Although the HSI color model is very similar
to the HSV color model, there is a difference in the
representation of the Intensity of HSI and the Value of HSV,as shown in Fig.1. Fig. 1(a) is a chromatogram image
created by using the S and V parameters in HSV. Fig. 1(b)is a chromatographic image created by using the S and I
parameters in HSI. From these two color spectrums, it canbe found that the overall color change of the HSI color
model is relatively smooth. In contrast, the HSV color
model has more compact changes between light colors andprimary colors, and it is more difficult to set the color
reduction range. Therefore, this article uses the HSI color
model with smoother color changes for color simplification.
The advantage is that the parameter range setting is more
flexible.

(a)HSV color model

(1)
(2)
(3)
(4)
In Fig. 4, in the diagonal part, we use Eq.1~Eq4 to import
the S value to get the relative range of I value. The scope of
HSI color simplification is summarized in a rule table.
Through the H, S, and I parameters in the captured target
pixel, we can determine from the rule table which shade ofcolor the pixel belongs to. In order to avoid errors inrecognizing pictures with the naked eye due to too close
shades, the 39 simplified colors are fine-tuned. The color
codes used are shown in reference [9].

(b)HSI color modelFig.1. Comparison of HSV and HSI Green Chromatogram

2.2 Isolated pixel filtering for watercolor system
In the process of image thumbnailing, the intersection ofdifferent colors is prone to color mixing. This change incolor mixing is usually not too large. We want to remove
the mixed color as much as possible and replace it with thesurrounding colors. In addition, filtering out isolated pixels
with low influence in the image can reduce the time it takes
to repeatedly raise and lower the pen. Therefore, this paper
proposes to filter out the isolated pixels and replace themwith surrounding colors to eliminate the isolated pixels, so
as to reduce the time consumed for lifting and lowering asingle pixel. The method of filtering out isolated pixels
starts from the upper left corner of the picture. A 3 × 3
matrix is used to scan from left to right and top to bottom.
The center color of the matrix is used as the target forcomparison. When the neighboring points have the samecolor, the original color of the target is maintained. If there
is no same color, it is judged as an isolated pixel andreplaced with the color with the largest surrounding area.
As shown in Fig. 5(a), the central color is an isolated pixel,
which will be replaced with the color of the largest areaaround it. When the surrounding eight pixels have the samecolor, they are not isolated points, as shown in Fig. 5 (b).

In this paper, the scanning method from left to right and
top to bottom is used to perform HSI color simplificationon the color of each pixel. First, the RGB value of the pixel
color is captured through the system. Since the color
reduction method uses the HSI color space for processing,it is necessary to convert the RGB value to the HSI value
before further processing, as shown in Fig.2. Since the hue
representation in HSV and HSI color models is the same,the hue part continues the setting of reference [1], and its
hue color distribution is shown in Fig. 3.
Detect color RGB
values

HSI color spaceconversion

Output simplified
result

HSI color
simplification

Fig. 2. The process of color simplification
52

yellow
green
黃綠色

95

yellow
黃色

11orange
橘色

綠色
green

red
紅色

grass
green
草綠色

244

116

pink
桃紅色

cyan
青色

137

S

31

74

紫色
purple

royal
blue
寶藍色
blue
藍色

159

I

223

靛色
indigo202
180

Fig. 3. Distribution of 12
hues

Fig. 4. Distribution of Color
simplification range(a)After determining the range of hue, the range of 6 colors

Unconnected target points(b) Connected target points

Fig. 5. Judgment of target point connectivity for watercolor© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

101

Image Inpainting Techniques Combined

When there are two or more colors with the largest pixel
area around the target, the system will scan from left toright and then from top to bottom according to the scanningorder, starting from the top left pixel, as shown in Fig. 6.
After that, the first and second ranking colors are taken out.
The HSI color space is used to convert these two colors to
the target color. At the same time, H is used to determine
which color is closer to the target color. After that, the color
with a closer hue replaces the target color.

matrix around it is extracted. If the color of the remaining 8
cells except the center point in the matrix is the same as thecolor of the pixel before filtering, it is judged to be
inpaintinged. This inpaintinging action is to restore the"pixel in the image after filtering out the isolated point" tothe "color of the pixel before filtering out".
Before孤立點濾除前
filtering out isolated points

After 孤立點濾除後
filtering out isolated points

Compare whether there are the same比對範圍內是否有相同色彩
colors in the 3*3 matrix.

(1)

(2)

(3)

(4)

Target
目標

(5)

(6)

(7)

(8)

Before processing

處理前Fig. 6. Scanorder of colors
around the target

第一順位
First color
色彩

H:21

Target color
目標色彩

H:42

第二順位
Second color
色彩

H:42

Capture the pixel color before filtering.
擷取濾除前像素色彩

Fig. 9. Judgment method of image inpainting
After processing

處理後Fig. 7. Judgment method of the twolargest color pixel areas around thetarget(a) After filtering out isolated points

Taking Fig. 7 as an example, since the number of orange
and yellow pixels is 3, it is impossible to determine whichcolor replaces the target color. Therefore, it is necessary tojudge by hue. First, it is determined by the scanning orderthat orange is the first color and yellow is the second color.
Secondly, convert the color to HSI color space, andcompare the H parameters with each other. The hue of thetarget color in the figure is the closest to the hue of thesecond-ranked color, so the target color is replaced byyellow. Fig. 8 is a comparison diagram before and after theisolated point filtering process.

(a) Color simplified image(b) Results of image inpainting

Fig. 10. Comparison before and after image inpainting
2.4 Optimized path movement for watercolor system
The area of the watercolor painting system consists ofthree parts: drawing, paint dipping, and brush cleaning.
When moving in the drawing area, you only need to movethe pen up and down by 4mm. In the paint picking andbrush cleaning area, the brush needs to be raised to 16mm.
Adjust the pen lift height for different areas to improve
efficiency. Fig. 11 is a schematic diagram of the movementof the brush.
2.5 Grayscale
Most common images are color images in RGB color
space. However, the common sketches are painted in black,white and gray. Therefore, we convert the RGB image intoa grayscale image. The conversion formula is as follows:(5)(b) Image after isolated pixel filtering

Fig. 8. Before and after filtering isolated pixels

2.6 Binarization
We use binarization to divide the grayscale image into
black and white. Black represents the area not to be drawn,
and white represents the area to be drawn. By setting 4
different thresholds, the specific part of the grayscale image
is captured, as shown in Fig. 12.

2.3 Feature inpainting for watercolor system
After the isolated points are filtered out, some features in
the image may be affected and cause distortion. The righthand line features are distorted, as shown in Fig. 8(b).
Therefore, a new feature inpainting method is proposed inthis article. That is, a 3×3 array is used to determine
whether it needs to be inpaintinged. First, the images before
and after the isolated points are filtered out are subtracted
from each other. In the subtraction result, the part where thepixel value is not equal to 0 is the position filtered by theisolated point. The color of the pixels before the filtering of
the processed position points is captured from the image
before the isolated points are filtered. At the same time, in
the image after the isolated points are filtered, theselocation pixels are found and centered on it. Then the 9 × 9© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

102

Chun-Chieh Wang, Zhan-Xian Ye

3 SYSTEM ARCHITECTURE OF DRAWING ROB
OTS

3.1 Watercolor system
The process of watercolor drawing system is shown in
Fig.15. The direct connection between the drawing platform
and the computer replaces the previous steps of manuallyloading data into the drawing platform to complete theautomation of the watercolor painting system.
Fig. 11. Schematic diagram of the movement of the brush

(a)

(b)

(c)

(d)

3.2 Sketch system
The process of the sketch system is shown in Fig.16. By
adjusting the image resolution and the line spacing of thedrawing track, the drawing results are more detailed.
3.3 Communication system
To improve the convenience of use, we have added a
communication part to the program. The user can decide
whether to draw directly with LabVIEW through the
buttons in the HMI. After the program converts theprocessing result and saves it as a G-Code file, if the buttonin the human-machine interface is not pressed, the program
will be directly terminated, allowing the user to transfer
data through the SD Card. If the user presses the button,
LabVIEW will connect with the robot and send the drawing
path data for drawing.

(e)Fig. 12. Results of multiple binarization: (a) grayscale;binarization-(b) the shallowest (c) the second shallow (d)the second deep (e) the deepest
2.7 Isolated pixel filtering and image inpainting forsketch system
The isolated point filtering and image inpainting in thesketch system will process the black and white pixels in thebinary image, as shown in Fig. 13. If there is the same color
around the target point, it is judged as connected.
Conversely, if there is no same color around the target point,
it is judged as an isolated point. The image inpainting
method is the same as in section 2.2.

3.4 Configuration of drawing platform
Fig. 17 (a) is the plane configuration of the drawing area.
The physical drawing of the multifunctional drawing robot
is shown in Fig. 17 (b).
Image input(a) Connected target point(b) Unconnected target point

Fig. 13. Judgment of target point connectivity for sketch
2.8 Drawing method for sketch
We use lines in different directions for multi-level
depiction. Draw the four shades of images mentioned
earlier with lines in different directions, as shown in Fig. 14.
The more the lines are stacked, the darker the color. In thisway, the changes in depth are expressed.

Imagecompression

HSI color
simplification

Feature repair

filtering
isolated pixels

Image cutting

Path planning

Image output

G-CodeencodingUSB
serialDrawing

Fig. 15. The process of watercolor drawing system
Image input
GrayscaleParameter s etting

Threshold s etting

Shallow image
binarization

Sub-shallow
image
binarization

Sub-deep Image
Binarization

Deepes t image
binarization

Filtering isolated
pixelsFiltering isolated
pixelsFiltering isolated
pixelsFiltering isolated
pixelsImage inpainting

Image inpainting

Image inpainting

Image inpainting

Calculate drawing
path (0°)

Calculate drawing
path(45°)

Calculate drawing
path(90°)

Calculate drawing
path(135°)

G-Code encoding

Fig. 14. Drawing trajectory in four directions

USB serial

Drawing

Fig. 16. The process of the sketch system© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

103

Image Inpainting Techniques Combined
20cm

15cm

20cm

15cm

海
綿

繪圖區域

[2]

(a)

顏料盒

(b)

[3]

Fig. 17. (a) The plane configuration of the drawing area (b)The physical drawing of the robot[4]

4 EXPERIMENTAL RESULTS[5]

4.1 Watercolor drawing results
Fig. 18 shows the drawing results of Bill Gates photo.

[6]
[7]

(a) Original image(b) Result of [1](c) Our drawing[8]

Fig. 18. The drawing results of Bill Gates photo[9]

4.2 Sketch drawing results
Fig. 19 shows the drawing results of different line
spacing. Although the drawing time with denser lines will
increase a bit, the recognition of the portrait will besignificantly improved.

(a) Result for line spacing 1mm

Watercolor Painting Robot, Master Thesis of Chienkuo
Technology University.
Hong M.J. (2016)，Robotics Artistic Colorful Picture
Drawing and Painting Using Visual Feedback Control
System，Master Thesis of National Taiwan University.
Columbia University Creative Machines Lab PIX18
https://www.hodlipson.com/robotic-art.html(Searchdate:12/28/2017)
Massachusetts Institute of Technology BABOT
http://transistor-man.com/bluebot_revival.html(Searchdate:12/28/2017)
Lin L.C. (2017)，Trajectory Planning and Tracking
Control of a FPGA-based Mobile Robot in a Picture
Drawing Application, Master Thesis of National Taiwan
University of Science and Technology.
Mac T.T. (2009), Autonomous Drawing Manipulation of
Humanoid Robot Arm, Master Thesis of National Taiwan
University of Science and Technology.
Chuang L.W. (2009), Human Portrait Generation System
for Humanoid Robot Arm Drawing, Master Thesis of
National Taiwan University of Science and Technology.
Jhang G.M., Li C.J. (2016), Multilevel Based Image
Recognition Applied in Automatic Sketch XYZ Platform,
Graduation monograph of Chienkuo Technology
University.
Ye Z.X. (2018)，Isolated Pixel Filtering and Image
Inpainting Techniques Applied in System Improvementof Multifunctional Drawing Robots, Master Thesis of
Chienkuo Technology University.

(b) Result for line spacing 0.5mm

Fig. 19. The drawing results of different line spacing

5 CONCLUSION
In this paper, HSI color space method strengthens therecognition of image processing results. Moreover, we use
the isolated pixel filtering, image inpainting techniques andpath planning optimization to effectively shorten thedrawing time. In addition, through the image resolution
adjustment as well as the shortening of the spacing of thedrawing lines, the robot can draw more detailed pictures.
Experimental results prove that image inpainting techniques
combined with isolated pixel filtering have beensuccessfully applied to multifunctional drawing robots.
REFERENCES[1]

Hung Y.Y. (2017)，Image Detection Applied in a
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

104

Mobile Robot with Image Recognition -- Using LabVIEW and KNRm
Kuo-Hsien Hsia

Bachelor Program in Interdisciplinary Studies
National Yunlin University of Science and Technology
123 University Road, Section 3, Douliou, Yunlin 64002, Taiwan, R.O.C.
Bo-Jung Yang

Graduate School of Engineering Science and Technology
National Yunlin University of Science and Technology
123 University Road, Section 3, Douliou, Yunlin 64002, Taiwan, R.O.C.
Jr-Hung Guo

Bachelor Program in Interdisciplinary Studies
National Yunlin University of Science and Technology
123 University Road, Section 3, Douliou, Yunlin 64002, Taiwan, R.O.C.
Chang-Sheng Xiao

Department of Electrical Engineering
National Yunlin University of Science and Technology
123 University Road, Section 3, Douliou, Yunlin 64002, Taiwan, R.O.C.
E-mail: {khhsia, D10810211, jrhung, M10712003}@yuntech.edu.tw
www.yuntech.edu.tw

Abstract
The main purpose of this paper is to use the image recognition of LabVIEW software to construct mobile robots withvarious functions, and make the robots applicable to the industry and have web monitoring applications. The core of
the robot is mainly the KNRm controller. This controller is suitable for beginners, and can be connected to DC servo
motor, RC servo motor, infrared, ultrasonic and camera to achieve various functions of the robot. The structure of
the robot uses metal parts sold by Studica company, which can be in accordance with the desired function to assemble
the robot. Since the company is a designated equipment sponsor company for World Skills competitions, it can also
be in line with international standards. Finally, PID control and sensors are added to make the robot movement andposition more accurately.
Keywords: Image Recognition, Mobile Robot, Web monitoring, KNRm.

restaurants. Many countries are focused on the robot
industry. And the growth of mobile robots will be fasterthan industrial robots. The design and development ofmobile robots has also received increasing attention in
the field of education and competition [1, 2, 3]. Taking
the high school industrial student skills competition heldby the Ministry of Education, which is the most important

1. Introduction
With the increasing advancement of science andtechnology, the products of robots are becoming more
and more diversified, especially in terms of mobile robots.
Mobile robots are generally closely related to people's
lives, such as sweeping robots and robots used in© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

105

Kuo-Hsien Hsia, Bo-Jung Yang, Jr-Hung Guo, Chang-Sheng Xiaohigh school competition in Taiwan, as an example, thenumber of participants has grown from 86 to 102 in 3
years [4]. For Taiwan, which is facing a declining
birthrate, this is a big growth rate. This shows thebooming development of robot education.
2. Problem Description
This paper takes the project of the 49th National Skills
Competition in Taiwan as the subject of problem solving.
The competitors are asked to design a toy organizing
robot. This robot can help the children put the toys into
categories after the game. The layout of the competition
field is shown in Fig. 1. Zone A in Fig. 1 is the startingarea of the robot, and the size of the robot must becompletely within the range of the starting area. There isa height-limit gate on the passage as shown in Fig. 1
marked as G, and the robot must be below this height topass the gate. Zone B is the golf ball sprinkling area.
Before the start of the evaluation, 5 to 6 golf balls of 5colors are sprinkled via the sprinkler, as shown in Fig. 2,
to Zone B. Zone C is the target ball-collection area. There
are five slots in Zone C. A barcode is hanged on the wallof each grid to indicate balls of what color should beplaced in the grid. At the beginning, the robot starts fromthe starting area, and firstly reads the barcode of each gridin Zone C, and then passes through the passage and gate
to the golf sprinkling area to pick up the golf ball. After
gripping, the robot returns to the ball-collection area andplaces the balls according to the color assigned by thebarcode. The robot must collect the balls as much as
possible and place them to the correct slots within thelimited time duration, and return to the departure area
before the end of the time. In order to test the design ofthe robot mechanism under limited resources, the numberof motors and sensors are limited as mentioned in the test
project.

Fig. 1. Field of the considered problem.

Fig. 2. The field with the sprinkler.

useful pins can be wired out. Most of the pins that need
to be used in KNRm have been designed around thecontroller for user convenience. Like myRIO, KNRm canbe programmed with LabVIEW, developed by National
Instruments (NI) Company. Different from traditional
programming language, LabVIEW uses G language, adata-flow design, that allows users to programming
according to the desired functions in conceptualization.
In addition, G language has the ability of parallel
processing [5].
In terms of mechanism design, because the golf balls
have to be gripped out from the sprinkling area, a liftingmechanism and a ball clamping mechanism are necessary.
In addition, the picked golf balls need to be stored on the
robot. Hence a ball storage mechanism is required. It
requires the use of 3 RC servo motors to meet above
requirements, so an RC expansion box must be used. Dueto the use of the RC expansion box will occupy one DC
motor-port and there are only 4 motor-ports in a KNRm,
only 3 DC motors with omni wheels can be used for the
chassis. With omni wheels, it can achieve the functions

3. Hardware Design
We mainly use the integrated kit, including Tetrix metal
blocks, sensors and motors packed by Studica, to design
and assemble the robot according to the assigned
functions. As for the controller, the KNRm developed by
KKITC is used, and its main core is NI myRIO-1950. The
main difference from KNRm to myRIO is that a single
myRIO needs to be wired by oneself or a suitable
expansion board needs to be made by oneself, and the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

106

Mobile Robot with Imageof forward, backward, left and right motion, rotation anddiagonal movement of the chassis.
The analysis about the robot body design is asfollowing.
1. The ball clamping mechanism: Since the golf balls
have to be moved to the collection area in a limited
time, the design of the clamp mechanism is to clamp
as many golf balls as possible. However, the blocks
provided by Studica are not suitable for use. We
design a customized part by 3D printing.
2. The ball storage mechanism: Due to the need of storing
multiple golf balls on the robot without increasing thenumber of RC motors, a ramp with end-block isdesigned to allow the balls to roll into for storage.
3. The lifting mechanism: In order for the balls rollingsmoothly into the ball storage mechanism, it isnecessary to raise the clamp mechanism to a certain
height. In addition, beside the lifting mechanism, amechanism that can move back and forth must also beconnected.
The overall design of the mechanism of the robot isshown in Fig. 3.
For obstacle avoidance, we use ultrasonic sensors as
the main sensor, and infrared sensors as an auxiliary. The
reason is to improve the accuracy of travel and overcome
the lack of infrared rays that can pass directly throughwhen encountering certain obstacles. The advantage ofultrasound is that the distance is farther than infrared. The
ultrasonic sensor used in this paper is PING))), and theinfrared sensor used is SHARP GP2Y0A21YK0F.
In the motion control part of the robot, we use
trapezoidal acceleration and deceleration and PID control
for control.
4.

Fig. 3. The mechanism of the designed robot.

number, etc. Then use Vision Assistant to process theimage obtained by the camera. In Vision Assistant, one
can set and process the options of Image, Color,
Grayscale, Binary, Machine vision, Identification andother items. Because the images we deal with in thispaper are mainly the interpretation and recognition of bar
codes and golf balls, and bar code interpretation is a builtin function in LabVIEW, we use Find Circular Edge andcolor comparison directly instead of the technique ofimage gray-scale processing in the paper.
5. Experimental Results
We conducted field test with the robots we designed.
After the robot leaves the starting area, it reads thebarcodes first, as shown in Fig. 4, and stores the color
specified for each ball slot. After that, the height of therobot must be lowered so that it can pass through thegates in the passage smoothly. After the robot hascompletely passed the passage, it uses the side wall tocorrect the robot's posture, and turns to make the gripper
face to the golf sprinkling area, and then goes to thesprinkling area. In order to facilitate the gripping of golfballs scattered in the sprinkling area, the robot will sweep
the balls into the same area after reaching the sprinklingarea, and then perform the action of gripping the balls.
After performing multiple clamping, the robot thenreturns according to the path it came. When passing
through the gate, the clamping mechanism should belowered to pass it. After passing through the gates, thegripper returns to the original position, and place the golfball in the corresponding grid when the robot passes

Image Recognition

In the problem to be solved in this paper, barcode reading
and color determination are needed. Therefore, we use
Logitech C310 camera as the image sensing device. In
LabVIEW, users can combine their own image
processing programs with built-in VIs, or use the Vision
Assistant function of LabVIEW to achieve quick start
processing. Vision Assistant is an integrated program for
LabVIEW image processing, which has many image
processing functions. The Vision Acquisition program
can be used to set up the camera, including Acquisition
Source, Acquisition Type, camera resolution and frame© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

107

Kuo-Hsien Hsia, Bo-Jung Yang, Jr-Hung Guo, Chang-Sheng Xiao

through each collection area. Finally, after releasing all
the balls, the robot returns to the starting area to complete
the experiment. After actual testing, the robot wedesigned can be successfully used to achieve the tasks
required by the problem and return to the starting area, as
shown in Fig. 5.
6. Discussions and Conclusions
In this paper, a KNRm is used as the core controller ofthe robot, combined with the Tetrix metal building blocks,
three motors, three RC servo motors and appropriate
sensing devices, to design a solution that can solve theproject of Mobile Robotics of 49th National Skills
Competition in Taiwan. The designed robot has beenverified on the field and found that it can indeed achieve
the specified task.
Because it is composed of metal building blocks, thedesigned robot will be insufficient in terms of mechanicalrigidity. When designing robots to solve other application
problems, metal building blocks similar to this paper canalso be used to design the prototype of the robot to test
whether the function can meet the requirements or not.
When the designed robot can meet the requirements, one
can proceed with the appearance design of the robot to be
used as a commercial product. At the same time, themechanical rigidity of the robot is included in thematerial consideration. This can greatly reduce the time
for product development, and can more accurately
demand for application products.

Fig. 4. The robot starts and reads the barcodes.

Fig. 5. Task completed.
Both Intelligence and Interest, 2nd edn., GOTOP Press,
Taipei, 2014. (in Chinese)

References
1. S.E. Jung and E.S. Won, Systematic Review of Research
Trends in Robotics Education for Young Children,
Sustainability 10(4), 2018, 905.
2. D. Karahoca, A. Karahoca, H. Uzunboylu, Robotics
Teaching in Primary School Education by Project based
Learning for Supporting Science and Technology Courses,
Procedia Computer Science 3, 2011, pp. 1425-1431.
3. D. Bazylev, A. Margun, K. Zimenko, A. Kremlev, E.
Rukujzha, Participation in robotics competition as
motivation for learning, Procedia - Social and Behavioral
Sciences 152, 2014, pp. 835-840.
4. National High Schools Skills Competition. (in Chinese)
Available: http://sci.me.ntnu.edu.tw/Contest/ Retrieved:
11/30/2012.
5. C.H. Tseng, W.H. Guan, W.H.Wu, Advanced Teaching
Manual of Robot with LabVIEW - Building Robots with© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

108

Nonlinear Internal Model Controller based on Local Linear Models, and its Application
Shinichi Imai *Graduate school, Tokyo Gakugei University,
4-1-1, Nukuikita-machi, Koganei, Tokyo, 184-8501, Japan.
E-mail: shimai@u-gakugei.ac.jp
http://www.u-gakugei.ac.jp/ †Abstract
In this paper, nonlinear internal model controller based on local linear models, and its Application. The internal model
control has a simple structure and has a high robustness for system uncertainties. However, there are few studies ofinternal model control schemes for nonlinear systems. On the other hand, many controlled systems have thenonlinearity. The effectiveness of the newly proposed control scheme is numerically evaluated on experimentexamples in comparison with the conventional control methods for nonlinear systems.
Keywords: Control, . Local Linear Models, Internal Model Controller

high robust stability against uncertainty of the controlled
object. However, there are few examples of applying
IMC to nonlinear systems. For example, data-driven IMC
that uses a database to extract data similar to the current
data as a neighborhood and control it according to therequired point. Systems have been proposed, but thesemethods require enormous amounts of time forcomputational processing [4].
By the way, the authors have previously proposed amethod for calculating control parameters using theconcept of the local linear model method [5]. In thismethod, multiple local linear models can be constructed
for a nonlinear system, system parameters corresponding
to each local linear model can be obtained, and weights
can be applied to design a control system. For nonlinear
systems. It is possible to control with fast calculation
processing.
Therefore, in this paper, as a method of designing aninternal model for a nonlinear system, we propose a

1. Introduction
In recent years, with the development of computer
technology, design methods for control systems withhigher-order compensators have been considered withthe aim of improving control performance [1]. In
particular, it is difficult to obtain a desirable control
response with a simple controller using fixed parameters
because the characteristics of the system change
significantly due to changes in the system with nonlinearity, operating conditions, and changes in theenvironment.
On the other hand, a model-driven control method [2] hasbeen proposed in which the controlled object is describedby the most detailed mathematical formula and the modelof the controlled object is incorporated into the control
system. Internal model control (IMC) is one of themodel-driven control methods [2,3]. IMC ischaracterized by a simple control system structure and© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

109

Shinichi Imaimethod of designing a control system by individuallycalculating the system parameters corresponding to each
local linear model and weighting them. In this method,
the system parameters are determined for the local linear
model divided into multiple parts, so it can be expectedthat more appropriate tuning can be performed for the
nonlinear system. Since it is not necessary to build the
database required by the data-driven IMC system, thetime required for construction can be reduced. As a result,the load and processing time can be significantly reduced
in terms of memory capacity and calculation time.
Fig. 1. Block diagram of the proposed method

2. IMC design using a local linear model
In this paper, we construct a local linear model aroundthe equilibrium point that differs depending on thecharacteristics of the static characteristics of thenonlinear model. Calculate the control parameters
corresponding to each local linear model and obtain theestimated value of the system output. Then, the nonlinear control is realized by calculating the distance
between the actual system output and the estimated value
and adjusting the load of the system parameters
according to the distance. Fig.1 shows the block diagramof the proposed control system.

systems represented by chemical processes, it is often
impossible to clearly specify the dead time. Therefore, if
the dead time is known, set 𝑘𝑘𝑚𝑚 to that value, and if therange of dead time is unknown, set 𝑘𝑘𝑚𝑚 = 0.
In addition, A(𝑧𝑧 −1 ) and B(𝑧𝑧 −1 ) are given by thefollowing equations.
A(𝑧𝑧 −1 ) = 1 + 𝑎𝑎1 𝑧𝑧 −1 + ⋯ + 𝑎𝑎𝑛𝑛𝑦𝑦 𝑧𝑧 −𝑛𝑛𝑦𝑦(4)
B(𝑧𝑧 −1 ) = 𝑏𝑏0 + 𝑏𝑏1 𝑧𝑧 −1 + ⋯ + 𝑏𝑏𝑛𝑛𝑢𝑢 𝑧𝑧 −𝑛𝑛𝑢𝑢(5)where, if the dead time is unknown or ambiguous, theinsufficient dead time information is supplemented by
securing the order of B(𝑧𝑧 −1 ) in 𝑛𝑛𝑢𝑢 dimensions.

2.1. System description

2.2. Designing an internal model controller using
a local linear modelFirst, it is assumed that the system dealt with in this paper
is given by the following equation.
y(t) = f(φ(t − 1))(1)where, y(t) represents the system output and f(∙)
represents the nonlinear function. In addition, φ(t − 1)
represents the state (historical data) before the time t − 1of the system, and is called an information vector. The
information vector φ(t − 1) is defined by the following
equation.
φ(t − 1) ≔ [y(t − 1), y(t − 2), ⋯ , y(t − 𝑛𝑛𝑦𝑦 )u(t − 1), u(𝑡𝑡 − 2), ⋯ , u(t − 𝑛𝑛𝑢𝑢 ) ] (2)Furthermore, u(t) is the control input, and 𝑛𝑛𝑦𝑦 and 𝑛𝑛𝑢𝑢 are
the output and input orders, respectively. Now, supposethat the nonlinear system represented by Eq. (1) can belocally represented by a linear model like the following
Eq.
A(𝑧𝑧 −1 )y(t) = 𝑧𝑧 −(𝑘𝑘𝑚𝑚 +1) 𝐵𝐵(𝑧𝑧 −1 )𝑢𝑢(𝑡𝑡)(3)−1
where, 𝑧𝑧
represents a time-delayed operator thatmeans 𝑧𝑧 −1 𝑦𝑦(𝑡𝑡) = 𝑦𝑦(𝑡𝑡 − 1). In addition, 𝑘𝑘𝑚𝑚 represents
the minimum estimate of wasted time. In many process

In this paper, we consider the control law (IMC) of thefollowing Eq.
u(t) = Q(𝑧𝑧 −1 )𝑒𝑒(𝑡𝑡)(6)
Q(𝑧𝑧 −1 ) =𝐴𝐴(𝑧𝑧 −1 )
𝐵𝐵(1)

�

1−𝜆𝜆

1−𝜆𝜆𝑧𝑧 −1

�

𝑛𝑛

(7)

where, λ is the design parameter of the filter used in therange of 0 ≤ λ < 1, and n is the order of the filter. In
addition, by using B(1) , even if B(𝑧𝑧 −1 ) contains an
unstable zero point (non-minimum phase system), it canbe dealt with because pole-zero cancellation is avoided.
In addition, e(t) is a control error signal and can be
defined as follows with r(t) as the target value.
e(t) ≔ r(t) − {y(t) − 𝑦𝑦�(t)}(8)
where, 𝑦𝑦�(𝑡𝑡) is the internal model output and is shown by
the following Eq.
(9)𝑦𝑦�(𝑡𝑡) = −𝐴𝐴(𝑧𝑧 −1 )𝑦𝑦(𝑡𝑡) + 𝑧𝑧 −(𝑘𝑘𝑚𝑚 +1) 𝐵𝐵(𝑧𝑧 −1 )𝑢𝑢(𝑡𝑡)Since many real systems have non-linearity, it is difficult
to always obtain good control results when the systemparameters are fixed. Therefore, in this method, thesystem parameters included in equations (4) and (5) are© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

110

Nonlinear Internal Model Controller

self-adjusted according to the characteristics of thesystem based on the local linear model, so they are
replaced as follows.
� (𝑧𝑧 −1 : 𝑡𝑡) = 1 + 𝑎𝑎�1 (𝑡𝑡)𝑧𝑧 −1 + ⋯ + 𝑎𝑎�𝑛𝑛 (𝑡𝑡)𝑧𝑧 −𝑛𝑛𝑦𝑦
A
(10)
𝑦𝑦
� (𝑧𝑧 −1 : 𝑡𝑡) = 𝑏𝑏�0 (𝑡𝑡) + 𝑏𝑏�1 (𝑡𝑡)𝑧𝑧 −1 + ⋯ + 𝑏𝑏�𝑛𝑛 (𝑡𝑡)𝑧𝑧 −𝑛𝑛𝑢𝑢 (11)B𝑢𝑢
� (𝑧𝑧 −1 : 𝑡𝑡) and B� (𝑧𝑧 −1 : 𝑡𝑡) represent the control
where, Atarget at time t, and the above assumptions are inherited.
Also, 𝑛𝑛𝑦𝑦 is the order of the output. Along with this,equations (7) and (9) are described as the following Eq.
Q(𝑧𝑧 −1 : 𝑡𝑡) =𝑛𝑛
𝐴𝐴�(𝑧𝑧 −1 :𝑡𝑡)
1−𝜆𝜆��𝐵𝐵�(1:𝑡𝑡)
1−𝜆𝜆𝑧𝑧 −1−1(12)𝑦𝑦�(𝑡𝑡) = −𝐴𝐴(𝑧𝑧 : 𝑡𝑡)𝑦𝑦(𝑡𝑡) +(13)
𝑧𝑧 −(𝑘𝑘𝑚𝑚 +1) 𝐵𝐵(𝑧𝑧 −1 : 𝑡𝑡)𝑢𝑢(𝑡𝑡)After the above preparations, design an internal model
controller using a local linear model. The specificalgorithm is summarized below.

Fig. 2. Process system

Furthermore, 𝜔𝜔𝑖𝑖 (𝑡𝑡) is the weight corresponding to theselected i-th information vector. The smaller thedifference between the output value of the actual systemand each linear model, the larger this weight becomes.
Note that the following equation is satisfied when 𝜔𝜔𝑖𝑖 (𝑡𝑡)
is calculated based on equation (19).
(20)∑𝑁𝑁
𝑖𝑖=1 𝜔𝜔𝑖𝑖 (𝑡𝑡) = 1[STEP1] Construction of multiple linear models
For the nonlinear model, multiple linear models areconstructed, the system is identified by the collective
least squares method, and the parameters of A(𝑧𝑧 −1 ) andB(𝑧𝑧 −1 ) included in the linear model of the following
equation are estimated.
A𝑖𝑖 (𝑧𝑧 −1 )y(t) = 𝑧𝑧 −(𝑘𝑘𝑚𝑚 +1) 𝐵𝐵𝑖𝑖 (𝑧𝑧 −1 )𝑢𝑢(𝑡𝑡)(i = 1,2, ⋯ , N) (14)where, N represents the number of divisions of the local
linear model, and i = 1,2, ⋯ , N or less Unless otherwise
specified, i takes these values. In addition, A𝑖𝑖 (𝑧𝑧 −1 ) andB𝑖𝑖 (𝑧𝑧 −1 ) are given by the following equations.
A𝑖𝑖 (𝑧𝑧 −1 ) = 1 + 𝑎𝑎𝑖𝑖,1 𝑧𝑧 −1 + ⋯ + 𝑎𝑎𝑖𝑖,𝑛𝑛𝑦𝑦 𝑧𝑧 −𝑛𝑛𝑦𝑦(15)
B𝑖𝑖 (𝑧𝑧 −1 ) = 𝑏𝑏𝑖𝑖,0, + 𝑏𝑏𝑖𝑖,1 𝑧𝑧 −1 + ⋯ + 𝑏𝑏𝑖𝑖,𝑛𝑛𝑢𝑢 𝑧𝑧 −𝑛𝑛𝑢𝑢(16)[STEP3] Determining system parameters
Using the weights obtained in [STEP2] and𝐴𝐴𝑖𝑖 (𝑧𝑧 −1 ) and𝐵𝐵𝑖𝑖 (𝑧𝑧 −1 ) in Eqs. (15) and (16), the system parameters arecalculated by the following equation.
−1(21)
𝐴𝐴̂(𝑧𝑧 −1 : 𝑡𝑡) = ∑𝑁𝑁
𝑖𝑖=1 𝜔𝜔𝑖𝑖 𝐴𝐴𝑖𝑖 (𝑧𝑧 )𝑁𝑁(22)𝐵𝐵� (𝑧𝑧 −1 : 𝑡𝑡) = ∑𝑖𝑖=1 𝜔𝜔𝑖𝑖 𝐵𝐵𝑖𝑖 (𝑧𝑧 −1 )With this system parameter, equations (10) and (11) are
updated to obtain the output 𝑦𝑦�(𝑡𝑡) of the local linear
model.
3. Experimental example[STEP2] Weight calculation
Next, for each local linear data calculated in [STEP1], theestimation error 𝜖𝜖𝑖𝑖 (𝑡𝑡) is calculated for each model, andthe weight 𝜔𝜔𝑖𝑖 is calculated based on this. 𝜖𝜖𝑖𝑖 (𝑡𝑡) is theerror between the system output value y(t) and theestimated output value 𝑦𝑦�𝑖𝑖 (𝑡𝑡) of each linear model. Here,𝑦𝑦�𝑖𝑖 (𝑡𝑡) is calculated by the following equation based onequation (14).
𝑦𝑦�𝑖𝑖 (t) = −𝐴𝐴𝑖𝑖 (𝑧𝑧 −1 )𝑦𝑦(𝑡𝑡) + 𝑧𝑧 −(𝑘𝑘𝑚𝑚 +1) 𝐵𝐵𝑖𝑖 (𝑧𝑧 −1 )𝑢𝑢(𝑡𝑡) (17)where, 𝐴𝐴𝑖𝑖 (𝑧𝑧 −1 ) and 𝐵𝐵𝑖𝑖 (𝑧𝑧 −1 ) use the system parametersof each linear model estimated in [STEP1].
(18)
𝜖𝜖𝑖𝑖 (𝑡𝑡) = |𝑦𝑦(𝑡𝑡) − 𝑦𝑦�𝑖𝑖 (𝑡𝑡)|1�
𝜖𝜖 (𝑡𝑡)𝜔𝜔𝑖𝑖 (𝑡𝑡) = ∑𝑁𝑁 1𝑖𝑖

𝑖𝑖=1 �𝜖𝜖𝑖𝑖 (𝑡𝑡)

The effectiveness of the proposed method will be
examined through application to the thermal processsystem shown in Fig.2. This system uses an incandescentlight bulb (40W) as the control target, and controls thesurface temperature of the light bulb by changing thevoltage applied to the light bulb by controlling the Jouleheat of the filament. A heat transfer pair (R52-CA10AE)
sensor is attached to the top of the light bulb. Also,
measure the temperature of the light bulb with athermocouple. Furthermore, the temperature of thethermocouple is converted into a voltage by thethermocouple conversion IC, and after A/D conversion,(19)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

111

Shinichi Imaithe data is output to the computer. The control input iscalculated using the output data.
A PWM signal with a duty ratio according to the control
input is output through D/A conversion, and a current
flows through the heater by a solid state relay (SSR).
Therefore, the control input u(t) in this experiment is theduty ratio (0 to 100%) of the PWM signal given to the
SSR, and the control output y(t) is the temperature of thesurface of the light bulb.
First, the target value r(t) is given as follows.
50(0 ≤ 𝑡𝑡 < 50)
70(50 ≤ 𝑡𝑡 < 100)r(t) = �(23)100(100 ≤ 𝑡𝑡 < 150)
130(150 ≤ 𝑡𝑡 < 200)
Next, a local linear model is constructed in the control
input range shown below. The number of divisions was
N=2.
0 ≤ 𝑢𝑢1 < 30
�(24)
20 ≤ 𝑢𝑢2 < 100
Here, the input / output data in the range of u is saved as
the initial database. In Eq. (24), there is a place where thearea of u overlaps, but this avoids that a good response
cannot be obtained by selecting the database when therequest point is selected near the division of eachdatabase. It is provided for this purpose. The values ofvarious design parameters included in the proposed
method are 𝑛𝑛𝑦𝑦 = 2, 𝑛𝑛𝑦𝑦 = 2, and 𝑘𝑘𝑚𝑚 = 0. Furthermore,
the parameters of IMC are λ = 0.5 and n = 1. Here, λwas designed to have the desired rising characteristics.
First, for comparison with the conventional method, thefixed PID control method widely used in the industry isapplied. However, for the PID parameter, the value
calculated based on the CHR method is used. Its PID
parameters are shown below.
𝐾𝐾𝑃𝑃 = 3.66, 𝐾𝐾𝐼𝐼 = 0.38, 𝐾𝐾𝐷𝐷 = 5.98
(25)
First, Fig.3 shows the control results of the fixed PID
method and the control results of the proposed method.
In addition, Fig.4 shows the temporal change of theweight by the proposed method in this case. From theresults in Fig.3 and Fig.4, it can be seen that the weightof the proposed method changes according to thecharacteristics of the system, and the responsiveness isgreatly improved.

140
120

temperature(℃)

10080Reference

60

Fixed method
Proposal method

40
20
0

0

50

100

150

200

150

200

150

200

ｔ

w2

w1

Fig. 3. Experimental result
1
0.8
0.6
0.4
0.2
0

1
0.8
0.6
0.4
0.2
0

0

50

100

t

0

50

100

t

Fig. 4. Change in weightadjusted appropriately according to the characteristics ofthe system and good control results could be obtained. In
the future, we plan to study the method of dividing the
local linear model of this method.
References
1. Y.Ashida, S.Wakitani and T.Yamamoto : Design of a
Data-Driven Multi-loop Self-Tuning PID Controller, IEEJ,
Vol.139-C, No.4, 2019, pp.356-36.
2. H.Kimura: Is the model a good controller?, 39th IEEE
Conference on Decision and Control, Plenary Lecture I ,2000.
3. T.Shigemasa, M.Yukitomo and R.Kuwata: A modeldriven PID control system and its case studies, Proc.CCA,
2002, pp.571-576.
4. K.Tsutsumi and T.Yamamoto: Design of a Data-Driven
Internal Model Controller, IEEJ, Vol.131-C, No.4, 2011,
pp.781-787.
5. S.Imai and T.Yamamoto: “Design of a Self-Tuning
Controller using Multiple Local Linear Models for
Nonlinear Systems,” IEEJ,Vol.132-C, No.6, 2012, pp860865

4. Conclusion
In this paper, we proposed a new design method for an
IMC system that constructs multiple local linear models
for a nonlinear system and adjusts the system parameters
corresponding to them. As a result, it was verified
through experiments that the system parameters were

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

112

Design of a Data-Driven Control System for Reference Model Design using Predicted Signals
Yuki Nakatani *, Takuya Kinoshita, Toru Yamamoto

Graduate School of Advanced Science and Engineering, Hiroshima University, 1-4-1 Kagamiyama, Higashi-Hiroshima
Higashi-Hiroshima-shi, Hiroshima-ken, 739-8527, Japan †E-mail: {nakatani-yuki, kinoshita-takuya, yama}hiroshima-u.ac.jp
www.hiroshima-u.ac.jp

Abstract
In recent years, data-driven control schemes that do not require system identification have been actively studied.
Generally, it is easy to give a reference model focusing only on the output response. In contrast, it is difficult to givea reference trajectory considering the input signals based on the controlled system's characteristics. Furthermore, itis necessary to consider the output signal and the input signal since there is a limit of the actuator performance in thecontrol design of the actual machine. This paper proposes a data-driven control scheme that can predict the
input/output response of an unknown system in offline using operating data. The effectiveness of the proposed scheme
is numerically verified by a simulation example.
Keywords: Data-driven control, extended output, predicted data, off-line tuning.

considering input signals for unknown structure system.
The features of this scheme are as follows.
(i)
1DOF control system design that predicts input
signals in advance, even for unknown controlled
system(ii)
Adjusting the desired input/output response with
one parameter

1. Introduction
In industrial systems, data-driven control methods1, 2
have been actively studied to achieve the desired control
performance for the controlled system with unknown
structure and parameters. These methods use only a setof experimental data to design a controller that realizes
the desired reference output offline. However, the control
system design should consider the input/output response
because there is a limit of the actuator performance of theactual machine.
The ERIT (Estimated Response Iterative Tuning)
method3 has been proposed to predict the input/output
response. However, this method can only be applied to
two-degree-of-freedom (2DOF) control systems4. On theother hand, it is useful to design a controller to consider
input signals in one-degree-of-freedom (1DOF) because
there are also many 1DOF control systems in the industry.
Therefore, in this paper, the new data-driven control
scheme is proposed to design a 1DOF controller

2. Overview of the proposed scheme
Fig. 1 shows an overview of the proposed data-driven
control scheme. In the proposed scheme, the controller isdesigned by the following procedure.
(I) Obtaining the desired predicted data 𝑦𝑦� ∗ (𝑡𝑡) and𝑢𝑢�∗ (𝑡𝑡)(II) Constructing the reference model 𝐺𝐺𝑚𝑚 (𝑧𝑧 −1 ) basedon 𝑦𝑦� ∗ (𝑡𝑡) and 𝑢𝑢�∗ (𝑡𝑡) obtained in (I)(III) Designing a controller using the reference model
𝐺𝐺𝑚𝑚 (𝑧𝑧 −1 ) obtained in (II)The detailed procedure is explained in the next section.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

113

Yuki Nakatani, Takuya Kinoshita, Toru Yamamoto

3. Design of the controller
3.1. Generating predicted the data using the ERIT
methodIn this section, the ERIT method3 is described togenerate the predicted data. This method is theoretically
only applicable to 2DOF control system4. Therefore, inthe proposed scheme, the ERIT method is utilized for
only predicting the input/output data in Fig. 2. Note thatthe controller is not designed by the ERIT method.
First, the initial output 𝑦𝑦0 (𝑡𝑡) is obtained in a 1DOF
control system. In the case of the feed-forward controller
𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 ) = 0 in Fig. 2, 𝑦𝑦0 (𝑡𝑡) is given as follows:
𝑦𝑦0 (𝑡𝑡) =𝐺𝐺(𝑧𝑧 −1 )𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 )𝑟𝑟(𝑡𝑡).
1 + 𝐺𝐺(𝑧𝑧 −1 )𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 )Fig. 1. Overview of the data-driven control systemby the proposed scheme.

(1)Then, extending to a 2DOF control system for predicting
the data, the predicted output 𝑦𝑦�(𝑡𝑡) is defined as:
𝐺𝐺(𝑧𝑧 −1 )𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 ) + 𝐺𝐺(𝑧𝑧 −1 )𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 )𝑦𝑦�(𝑡𝑡) =𝑟𝑟(𝑡𝑡).
1 + 𝐺𝐺(𝑧𝑧 −1 )𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 )Fig. 2. 2DOF control system.
problem of the evaluation norm based on the ITAE
(Integral of Time squared Absolute Error) method5 for
𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 ):(2)However, system identification is required to obtain anexactly predicted output 𝑦𝑦�(𝑡𝑡) because Eq. (2) contains acontrolled system 𝐺𝐺(𝑧𝑧 −1 ). Therefore, substituting Eq. (1)into Eq. (2) yields the following equation:
𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 )𝑦𝑦�(𝑡𝑡) =𝑦𝑦 (𝑡𝑡) + 𝑦𝑦0 (𝑡𝑡).
𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 ) 0(3)

𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 )𝑢𝑢 (𝑡𝑡) + 𝑢𝑢0 (𝑡𝑡).
𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 ) 0

(4)

𝑁𝑁

𝑡𝑡=0

𝑡𝑡=0

𝐽𝐽𝑑𝑑𝑑𝑑𝑑𝑑 = � 𝑡𝑡|𝑟𝑟(𝑡𝑡) − 𝑦𝑦�(𝑡𝑡)| + 𝜆𝜆 � 𝑡𝑡|Δ𝑢𝑢�(𝑡𝑡)| ,(5)where 𝑟𝑟(𝑡𝑡) and Δ(≔ 1 − 𝑧𝑧 −1 ) denote the reference
signal and the difference operator.
Firstly, 𝑟𝑟(𝑡𝑡) − 𝑦𝑦�(𝑡𝑡) in Eq. (5) shows the difference
between the reference signal and the predicted output
signal. The desired predicted response is obtained by
reducing it. Secondly, Δ𝑢𝑢�(𝑡𝑡) = 𝑢𝑢�(𝑡𝑡) − 𝑢𝑢�(𝑡𝑡 − 1) in Eq.
(5) represents the difference of input signal. Therefore, asmall value of 𝜆𝜆 results in a highly responsive response,while a large value of 𝜆𝜆 results in a less responsive
response by depending on the tunable parameter 𝜆𝜆. Thus,
the input/output response can be easily adjusted with 𝜆𝜆.
Additionally, it is easy for the user to select the optimal𝑦𝑦� ∗ (𝑡𝑡) and 𝑢𝑢�∗ (𝑡𝑡) by changing 𝜆𝜆 because predicted data of
unknown system can be calculated in the ERIT method.
The adjustment of 𝜆𝜆 is currently a trial and error process.
However, this process is very easy because it is only
necessary to adjust 𝜆𝜆 offline.

Consequently, the predicted data 𝑦𝑦�(𝑡𝑡) can be derived
offline without 𝐺𝐺(𝑧𝑧 −1 ) from the initial data 𝑦𝑦0 (𝑡𝑡) bychanging 𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 ). 𝑢𝑢�(𝑡𝑡) can also be derived in the sameprocedure using 𝑢𝑢0 (𝑡𝑡) as follows:𝑢𝑢�(𝑡𝑡) =𝑁𝑁

From Eqs. (3) and (4), it is possible to obtain thepredicted data 𝑦𝑦�(𝑡𝑡) and 𝑢𝑢�(𝑡𝑡) offline corresponding to thevarious 𝐶𝐶𝐹𝐹𝐹𝐹 (𝑧𝑧 −1 ) by using the initial data 𝑦𝑦0 (𝑡𝑡) and𝑢𝑢0 (𝑡𝑡) . Note that the 2DOF control system was only
implemented virtually off-line to obtain the predicted
data.

3.3. Constructing 𝑮𝑮𝒎𝒎 (𝒛𝒛−𝟏𝟏 ) based on the desired�∗ (𝒕𝒕)predicted data 𝒚𝒚

3.2. Obtaining the desired predicted input/output
data

In this section, constructing 𝐺𝐺𝑚𝑚 (𝑧𝑧 −1 ) is described basedon the desired predicted data 𝑦𝑦� ∗ (𝑡𝑡) . The evaluation
function for constructing 𝐺𝐺𝑚𝑚 (𝑧𝑧 −1 ) is defined as follows:

In this section, obtaining the desired predicted data𝑦𝑦� ∗ (𝑡𝑡) and 𝑢𝑢�∗ (𝑡𝑡) is described by using the predicted data𝑦𝑦�(𝑡𝑡) and 𝑢𝑢�(𝑡𝑡) . Specifically, 𝑦𝑦� ∗ (𝑡𝑡) and 𝑢𝑢�∗ (𝑡𝑡) arecalculated by solving the following minimization© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

114

Design of a Data-Driven

Table 1. Obtained parameters in the simulation.
𝛼𝛼1

𝛼𝛼2

Fig. 3. Block diagram of a temperature control
device.
𝐽𝐽𝑟𝑟𝑟𝑟𝑟𝑟 =𝑁𝑁

21
� ∗ (𝑡𝑡) − 𝑦𝑦𝑚𝑚 (𝑡𝑡)�
� �𝑦𝑦
𝑁𝑁
𝑡𝑡=0

𝑦𝑦𝑚𝑚 (𝑡𝑡) = 𝐺𝐺𝑚𝑚 (𝑧𝑧 −1 )𝑟𝑟(𝑡𝑡),𝜖𝜖(𝑡𝑡) = 𝐺𝐺𝑚𝑚

𝑁𝑁

1
� 𝜖𝜖(𝑡𝑡)2𝑁𝑁

𝑡𝑡=0(𝑧𝑧 −1 )𝜙𝜙(𝑡𝑡)− 𝑦𝑦(𝑡𝑡).

𝐾𝐾𝐼𝐼−4.07 × 10−2
2.23 × 10−4
1

−1.42

4.20 × 10−1
4.24 × 102

8.16 × 10−1−9.75 × 10−1
1.72 × 10−6
1

−1.99

9.93 × 10−1
4.27

5.60 × 10−3

and-bound welding owned by our laboratory. Fig. 3
shows the model of the system, and the coefficients were
derived by system identification. The details are omitted
due to space limitation.
Then, the proposed scheme is applied to the system inFig. 3. The reference signal is 𝑟𝑟(𝑡𝑡) = 100, the samplingtime is TS = 0.02 s, and the room temperature is 𝑑𝑑(𝑡𝑡) =20 [℃]. In this numerical example, I-P controller is used,
and the proportional gain and integral gain are set as
follows:(8)

𝐾𝐾𝑃𝑃 = 1.5, 𝐾𝐾𝐼𝐼 = 1.0 × 10−3 .

(11)∗ (𝑧𝑧 −1 )was
Next, the feed-forward controller 𝐶𝐶𝐹𝐹𝐹𝐹
calculated by the proposed scheme as follows:

(9)

(10)

∗ (𝑧𝑧 −1 )=𝐶𝐶𝐹𝐹𝐹𝐹𝛼𝛼1.
𝛼𝛼2 + 𝛼𝛼3 𝑧𝑧 −1 + 𝛼𝛼4 𝑧𝑧 −2(12)𝐺𝐺𝑚𝑚 (𝑧𝑧 −1 ) =𝜃𝜃1 + 𝜃𝜃2 𝑧𝑧 −1,𝜃𝜃3 + 𝜃𝜃4 𝑧𝑧 −1 + 𝜃𝜃5 𝑧𝑧 −2(13)

The order of the denominator was set to second orderbecause the order which significantly affects the output
response of controlled system is generally low order.
Furthermore, the value of 𝛼𝛼𝑖𝑖 (𝑖𝑖 = 1, 2, 3, 4) is determined
to minimize 𝐽𝐽𝑑𝑑𝑑𝑑𝑑𝑑 in Eq. (5), for example, using the
Matlab/Simulink
Ver.9.8.0.1396136(R2020a),
Optimization Toolbox ’fminsearch.m ’.
Furthermore, 𝐺𝐺𝑚𝑚 (𝑧𝑧 −1 ) was determined as follows:

where 𝜖𝜖(𝑡𝑡) denotes the difference between the output
𝑦𝑦(𝑡𝑡) of the closed-loop transfer function and the output
𝐺𝐺𝑚𝑚 (𝑧𝑧 −1 )𝜙𝜙(𝑡𝑡) of the reference. 𝐶𝐶(𝑧𝑧−1 ) can be designedby minimizing the evaluation 𝐽𝐽. For reasons of space, thedetails are omitted, refer to the PID control method basedon extended output7.
4.

6.89 × 10−6

𝐾𝐾𝑃𝑃

where 𝐾𝐾𝑃𝑃 , 𝐾𝐾𝐼𝐼 , and 𝐾𝐾𝐷𝐷 are the proportional gain, integral
gain, and derivative gain respectively. In this paper, PID
gains are tuned by using the extended output. In thismethod, 𝐶𝐶(𝑧𝑧−1 ) is designed by minimizing theevaluation norm based on the property that the extended
output 𝜙𝜙(𝑡𝑡) and the reference signal 𝑟𝑟(𝑡𝑡) are equal in Fig.
1. In other words, the evaluation norm is defined as:
𝐽𝐽 =8.92 × 10−4𝜃𝜃5

In this section, a 1DOF controller 𝐶𝐶(𝑧𝑧−1 ) is designed as
an I-PD controller6 by using the reference model
𝐺𝐺𝑚𝑚 (𝑧𝑧 −1 ) that considers the input response obtained in theprevious section.
I-PD controller is given as follows:Δ𝑢𝑢(𝑡𝑡) = 𝐾𝐾𝐼𝐼 𝑒𝑒(𝑡𝑡) − 𝐾𝐾𝑃𝑃 Δ𝑦𝑦(𝑡𝑡) − 𝐾𝐾𝐷𝐷 Δ2 𝑦𝑦(𝑡𝑡),

1

𝜃𝜃2
𝜃𝜃4

3.4. Design of a PID controller based on extended
output1.71 × 10−4

−1.57 × 10−2𝜃𝜃3where 𝑁𝑁 denotes the number of data. 𝐺𝐺𝑚𝑚 (𝑧𝑧 −1 ) isconstructed by minimizing the evaluation 𝐽𝐽𝑟𝑟𝑟𝑟𝑟𝑟 .

1

𝜆𝜆 = 35−2.35 × 10−2

𝜃𝜃1

(7)

2.23 × 10

−2

𝛼𝛼3
𝛼𝛼4

(6)

𝜆𝜆 = 1

Numerical example

4.1. Controlled system
The effectiveness of the proposed scheme is verified
using the numerical simulation. The controlled system isthe thermal experimental apparatus for simulating bag-

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

115

Yuki Nakatani, Takuya Kinoshita, Toru YamamotoFig. 5. Simulation results by applying the proposed
scheme where 𝜆𝜆 = 35.

Fig. 4. Simulation results by applying the proposed
scheme where 𝜆𝜆 = 1.

choose 𝜆𝜆 and the controller design considering the input
saturation will be studied.

where the value of 𝜃𝜃𝑖𝑖 (𝑖𝑖 = 1, 2, 3, 4, 5) is determined byusing the least-squares method to minimize 𝐽𝐽𝑟𝑟𝑟𝑟𝑟𝑟 in Eq.
(6).
In this numerical example, the results are compared bychanging 𝜆𝜆 in Eq. (5), and Table 1 shows the feed∗ (𝑧𝑧 −1 ),forward controller 𝐶𝐶𝐹𝐹𝐹𝐹
the reference model−1
𝐺𝐺𝑚𝑚 (𝑧𝑧 ), the proportional gain, and the integral gain
respectively in the cases of 𝜆𝜆 = 1, 35.

References
1. T. Yamamoto, O. Kaneko, S. Wakitani, T. Kinoshita, Y.
Ohnishi, H. Kugemoto, K. Koiwai, Data-oriented PID
control, Morikita publish, 2020.
2. T. Yamamoto, O. Kaneko, A Direct Design Method forControl Systems Based on Closed-Loop Data, Journal of
the SICE 52(10), 2013, pp. 841-846
3. O, Kaneko, H. Nakamura, T. Ikezaki, A New Approach toUpdating Feedforward Controllers in Two-Degree-ofFreedom Control Systems Estimated Response Iterative
Tuning (ERIT) is proposed, Transactions of the SICE
54(12), 2018, pp. 857-864.
4. Y. Chida, Design Method of Two-Degree-of-Freedom
Control System Using Characteristics of Feedback
Controller, Transactions of the Institute of ElectricalEngineers of Japan 117(4), 1997, pp.439-445
5. M. Masubuchi, Revised Basic Theory of AutomaticControl, Corona Corporation, 2005, pp. 181-184
6. T. Kitamori, Principles and Design Methods of I-PD
Control Systems (Special Issue on PID Control - Theory
and Application), Transactions of SICE 54(12), 1998, pp.
7-17
7. A, Yoichiro, S. Wakitani and T. Yamamoto, Design of an
Augmented Output-Based Multiloop Self-Tuning PID
Control System, Industrial & Engineering Chemistry
Research 58(26), 2019, pp. 11474-11484

4.2. Control result

Figs. 4 and 5 show the initial data 𝑦𝑦0 (𝑡𝑡) and 𝑢𝑢0 (𝑡𝑡), thedesired predicted data 𝑦𝑦� ∗ (𝑡𝑡) and 𝑢𝑢�∗ (𝑡𝑡), and the results ofimplementing a 1DOF controller 𝑦𝑦1 (𝑡𝑡) and 𝑢𝑢1 (𝑡𝑡) in thecases of 𝜆𝜆 = 1, 35 . The input-output response can be
adjusted with one parameter according to 𝜆𝜆 because theresponsiveness is high because 𝜆𝜆 = 1 in Fig. 4, and the
responsiveness is low because 𝜆𝜆 = 35 in Fig. 5.
Furthermore, it is possible to design a controller thatconsiders input signals because the input signals 𝑢𝑢� ∗ (𝑡𝑡)and 𝑢𝑢1 (𝑡𝑡) are almost identical in Figs. 4 and 5.
5. Conclusion

In this paper, the new data-driven control scheme has
been proposed to design a 1DOF to consider input signal
for unknown structure system. The effectiveness has
been verified by the numerical example. It is possible topredict the appropriate amount of input signal tocorrespond the performance of the actuator by predicting
the input in advance. In the future, the method how to© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

116Design of a Database-Driven GMV Controller Using the Nelder-Mead Method*
Liying Shi

Graduate School of Engineering, Hiroshima University, Japan
1-4-1 Kagamiyama HigashiHiroshima ,739-8527, Japan
E-mail: m192383@hiroshima-u.ac.jp
Zhe Guan

KOBELCO Construction Machinery Dream-Driven Co-Creation Research Center, Hiroshima University, Japan
1-4-1 Kagamiyama HigashiHiroshima ,739-8527, Japan
E-mail: guanzhe@hiroshima-u.ac.jp
Toru Yamamoto

Graduate School of Advanced Science and Engineering, Hiroshima, Japan
1-4-1 Kagamiyama HigashiHiroshima ,739-8527, Japan
E-mail: yama@hiroshima-u.ac.jp

Abstract
This paper presents a design scheme that can obtain the optimized generalized minimum variance (GMV) control
parameters by applying the Nelder-Mead (NM) method based on proportional-integral-derivative (PID) controller.
The NM method is used to find the most suitable parameter λ. The application of NM method can optimize thecalculation of λ. Furthermore, the estimation of closed-loop response method is introduced in database-driven
approach. The effectiveness is verified by using a simulation example.
Keywords: PID controller, GMV, Nelder-Mead method, Database-driven approach

Then, the GMV 5 is proposed that takes into accountcontroller error variance as well as the manipulating
variable variance. Furthermore, an adjustable parameter
λ (a weighting factor penalty on the manipulating
variable) is included in the GMV. In the previous GMV
controller, the PID parameters are calculated by simply
changing λ manually. Therefore, it is hard to get desirable
control performance. In order to improve the control
performance, the Nelder-Mead method6 is introduced. It
can optimize the calculation of the most suitableparameter 𝜆𝜆.
In this paper, a new GMV control scheme is proposed inwhich the controller parameters can be calculated
without any model of the process by applying the Nelder-

1. Introduction
A Proportional-Integral-Derivative (PID) controller1,2 isa three-term controller that has a long history in theautomatic control field, and it is certainly the most widelyused control scheme today owing to its intuitiveness andits relative simplicity. Over the last half-century, a great
deal of academic and industrial effort has focused on
improving PID control. In previous controller, using theminimum variance (MV)3,4 as the benchmark can claim
to be successful. The basic idea behind the MV index is
to only consider the controller error variance. However,few of the available techniques in use take the control
effort or the manipulative variable activity into account.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

117

Liying Shi, Zhe Guan, Toru Yamamoto

Mead method6 and method of the estimation of closedloop response7
The rest of this paper is organized as follows: the problem
is formulated in section 2; the proposed scheme is themain topic in section 3 and section 4; the section 5
provides the simulation and analysis, and the section 6
concludes the paper.

𝑢𝑢(𝑡𝑡) =𝐶𝐶(𝑧𝑧 −1 )∆𝑒𝑒(𝑡𝑡)(4)

The GMV control law for the system (1) can be derived
by minimizing the following cost function:(5)
𝐽𝐽 =𝐸𝐸[𝜙𝜙 2 (𝑡𝑡 + 𝑘𝑘 + 1)].
Here, 𝜙𝜙(𝑡𝑡 + 𝑘𝑘 + 1) is the generalized output given byfollowing equation:
𝜙𝜙(𝑡𝑡 + 𝑘𝑘 + 1) = 𝑃𝑃(𝑧𝑧 −1 )𝑦𝑦(𝑡𝑡 + 𝑘𝑘 + 1) + 𝜆𝜆𝜆𝜆𝜆𝜆(𝑡𝑡) −𝑃𝑃(1)𝜔𝜔(𝑡𝑡)(6)where ω(t) denotes the step reference signal, 𝑘𝑘 is theminimum time-lag which is estimated by an operator (𝑘𝑘is set as 0 when the time-lag is unknown). In addition, λis the weight coefficient for the variance of the control
input and it is set by a user arbitrarily. Next, the following
Diophantine equation is introduced.
𝑃𝑃(𝑧𝑧 −1 ) = ∆𝐸𝐸(𝑧𝑧 −1 )𝐴𝐴(𝑧𝑧 −1 ) + 𝑧𝑧 −(1+𝑘𝑘) 𝐹𝐹(𝑧𝑧 −1 ) (7)In Fig. 1, an adjustable parameter λ is included in the
GMV, and PID parameters are calculated by simply
changing λ. Using the method of estimation of closedloop response to predict new output, then the applicationof NM method can optimize the calculation of the mostsuitable λ according to the predict output and get optimal
PID parameters without any model of the process.

𝐸𝐸(𝑧𝑧 −1 ) = 1 + 𝑒𝑒1 𝑧𝑧 −1 +…+𝑒𝑒𝑘𝑘 𝑧𝑧 −𝑘𝑘
𝐹𝐹(𝑧𝑧 −1 ) = 𝑓𝑓0 +𝑓𝑓1 𝑧𝑧 −1 +𝑓𝑓2 𝑧𝑧 −2(8)Moreover, 𝑃𝑃(𝑧𝑧 −1 ) is the design polynomial and it isdesigned based on the following equation. where, theorder of 𝐸𝐸(𝑧𝑧 −1 ) and 𝐹𝐹(𝑧𝑧 −1 ) are set to decide thesecoefficients uniquely from ∆𝐴𝐴(𝑧𝑧 −1 ) and 𝑃𝑃(𝑧𝑧 −1 ).
(9)𝑃𝑃(𝑧𝑧 −1 ) = 1 + 𝑝𝑝1 𝑧𝑧 −1 + 𝑝𝑝2 𝑧𝑧 −2

3. THE GENERALIZED MINIMUMVARIANCE CONTROL OF PARAMETERS

𝑝𝑝1 = −2exp (−

𝜌𝜌

2𝜇𝜇

) cos

�4𝜇𝜇−12𝜇𝜇𝜌𝜌𝑝𝑝2 = exp (− )3.1. System description

𝜇𝜇

(10)

𝜌𝜌

(11)

𝜌𝜌 ∶= 𝑇𝑇𝑠𝑠 /𝜎𝜎(12)𝜇𝜇 ∶= 0,25(1 − 𝛿𝛿) + 0.51𝛿𝛿
(13)
𝜎𝜎 and 𝛿𝛿 are parameters which express the rise time andthe damping property which are set by an operator
respectively. Moreover, (𝜎𝜎 denotes the rise time that thesystem output attains about 60% of the value of the stepreference signal. From (1), (4) and (5), the predictive
value of the generalized output after 𝑘𝑘 + 1 at 𝑡𝑡 can beobtained as following equation:
𝜙𝜙(𝑡𝑡 + 𝑘𝑘 + 1) = 𝐹𝐹(𝑧𝑧 −1 )𝑦𝑦(𝑡𝑡) + {𝐸𝐸(𝑧𝑧 −1 )𝐵𝐵(𝑧𝑧 −1 ) +(14)𝜆𝜆}Δ𝑢𝑢(𝑡𝑡) − 𝑃𝑃(1)𝜔𝜔(𝑡𝑡) + 𝐸𝐸(𝑧𝑧 −1 )𝜉𝜉(𝑡𝑡 + 𝑘𝑘 + 1).
The control law is described as the following equation:

Consider the following process model and design of PID
controller:
ξ(t)𝐴𝐴(𝑧𝑧 −1 )𝑦𝑦(𝑡𝑡) = 𝑧𝑧 −1 𝐵𝐵(𝑧𝑧 −1 )𝑢𝑢(𝑡𝑡) +(1)∆

where 𝐴𝐴(𝑧𝑧 −1 ) and 𝐵𝐵(𝑧𝑧 −1 ) are given by the following
polynomials：
𝐴𝐴(𝑧𝑧 −1 ) = 1 + 𝑎𝑎1 𝑧𝑧 −1 + 𝑎𝑎2 𝑧𝑧 −2
(2)
−1 )
−m
B(z
= b0 + ⋯ + bm z
(3)where 𝑢𝑢(𝑡𝑡) and 𝑦𝑦(𝑡𝑡) are respectively the input and thecorresponding output; ξ(t) is the Gaussian white noise
which has zero mean; 𝑧𝑧 −1 , the backshift operator which
implies 𝑧𝑧 −1 𝑦𝑦(𝑡𝑡) = 𝑦𝑦(𝑡𝑡 − 1) ; Δ, the differencing
operator (∆= 1 − 𝑧𝑧 −1 ); and 𝑚𝑚, the order of 𝐵𝐵(𝑧𝑧 −1 ).
∆{𝑟𝑟(𝑡𝑡) − 𝑦𝑦(𝑡𝑡)} =3.2. Benchmark of the Generalized Minimum
Variance Controller

Fig. 1. The Block diagram of database-driven GMV
controller.

𝐶𝐶(𝑧𝑧 −1 )∆where 𝑟𝑟(𝑡𝑡) is the reference signal; 𝑒𝑒(𝑡𝑡), the control error.

2. Overview of the Data-Driven GMV controller

In this paper, the PID controller is introduced as
the following equation:

𝐶𝐶(𝑧𝑧 −1 )Δ𝑢𝑢(𝑡𝑡) =𝑃𝑃(1)

𝐺𝐺(𝑧𝑧 −1 )+𝜆𝜆𝜔𝜔(𝑡𝑡) −𝐹𝐹�𝑧𝑧 −1 �𝑦𝑦(𝑡𝑡).

𝐺𝐺(𝑧𝑧 −1 )+𝜆𝜆
−1

(15)

By replacing the polynomial 𝐺𝐺(𝑧𝑧 ) in (15) by thesteady-state term 𝐺𝐺(1), the following equation can beobtained:inΔ𝑢𝑢(𝑡𝑡) =𝑃𝑃(1)

𝐺𝐺(1)+𝜆𝜆𝜔𝜔(𝑡𝑡) −𝐹𝐹�𝑧𝑧 −1 �

𝐺𝐺(1)+𝜆𝜆© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

118

𝑦𝑦(𝑡𝑡).

(16)

Design of a Database-Driven

Therefore, PID parameters can be calculated as follows:
𝑓𝑓 +2𝑓𝑓
𝐾𝐾𝑃𝑃 = 1 2(17)
𝐺𝐺(1)+𝜆𝜆𝑓𝑓 +𝑓𝑓2 +𝑓𝑓0

(18)

𝑓𝑓2

(19)

𝐾𝐾𝐼𝐼 = 1

𝐾𝐾𝐷𝐷 =

𝐺𝐺(1)+𝜆𝜆

𝐺𝐺(1)+𝜆𝜆•In this method, the control performance is strongly
depended on 𝜆𝜆. Then the user-specified parameters 𝜆𝜆 is
calculated by using the NM method. The predicted output
adopts the estimation of closed-loop response method
without any model. The detail of this approach will not
be given because of pages limitation. Please refer to theRef. 7.

Fig. 3. Expansion of the Nelder-Mead method
Contraction
𝑆𝑆𝐶𝐶 is determined as 𝑆𝑆𝐶𝐶 = (1 − 𝜂𝜂)𝑆𝑆𝐺𝐺 + 𝜂𝜂𝜂𝜂𝐻𝐻 ,where 𝜂𝜂 is set as1 > 𝜂𝜂 > 0, and it corresponds to
the ratio of distances 𝑆𝑆𝐶𝐶 𝑆𝑆𝐺𝐺 and 𝑆𝑆𝐻𝐻 𝑆𝑆𝐺𝐺 .

4. Evolutionary Computation Based on The
Nelder-Mead method
The Nelder-Mead method6 is briefly explained is tocalculate the user-specified parameter 𝜆𝜆. In this paper,the objective function is determined by
𝑀𝑀

𝐻𝐻(𝑆𝑆𝑖𝑖 ) = �{𝑦𝑦𝑚𝑚 (𝑡𝑡) − 𝑦𝑦𝑑𝑑 (𝑡𝑡)}2
𝑡𝑡=1

Fig. 4. the contraction of the Nelder-Mead method
Reduction
𝑆𝑆𝐻𝐻 and 𝑆𝑆𝑀𝑀 are moved in the direction of 𝑆𝑆𝐿𝐿 .
The proposed control system can be designed by
calculate the most suitable 𝜆𝜆 by the step based on thementioned procedure.

•

(20)

where 𝑦𝑦𝑚𝑚 (𝑡𝑡) denotes the reference model output and M
denotes the evaluation horizon. 𝑦𝑦𝑑𝑑 (𝑡𝑡) is the predicted
output corresponding to 𝑆𝑆𝑖𝑖 . The objective function can be
determined. The top with the largest target function value
𝐻𝐻(𝑆𝑆𝑖𝑖 ) , the second large top and the smallest top are
respectively assumed to be 𝑆𝑆𝐻𝐻 , 𝑆𝑆𝑀𝑀 and 𝑆𝑆𝐿𝐿 . Furthermore,
the center between 𝑆𝑆𝑀𝑀 and 𝑆𝑆𝐿𝐿 is determined as 𝑆𝑆𝐺𝐺 . Four
operations are briefly explained as follows:• Reflection
𝑆𝑆𝑅𝑅 is determined as 𝑆𝑆𝑅𝑅 = (1 + 𝛽𝛽)𝑆𝑆𝐺𝐺 − 𝛽𝛽𝛽𝛽𝐻𝐻 ,where β is set as β > 0, and it corresponds to theratio of distances 𝑆𝑆𝑅𝑅 𝑆𝑆𝐺𝐺 and 𝑆𝑆𝐻𝐻 𝑆𝑆𝐺𝐺 .

•Fig. 5. Reduction of the Nelder-Mead method
The proposed control system can be designed by
calculating the most suitable 𝜆𝜆 by the step based on thementioned procedure.

Fig. 2. Reflection of the Nelder-Mead method
Expansion
𝑆𝑆𝐸𝐸 is determined as 𝑆𝑆𝐸𝐸 = (1 − 𝛾𝛾)𝑆𝑆𝐺𝐺 + 𝛾𝛾𝛾𝛾𝐻𝐻 ,where 𝛾𝛾 is set as 𝛾𝛾 > 1, and it corresponds to theratio of distances 𝑆𝑆𝐸𝐸 𝑆𝑆𝐺𝐺 and 𝑆𝑆𝑅𝑅 𝑆𝑆𝐺𝐺 .

Fig.6. Procedure of the Nelder-Mead method© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

119

Liying Shi, Zhe Guan, Toru Yamamoto

If 𝐻𝐻(𝑆𝑆𝐻𝐻 ) > 𝐻𝐻(𝑆𝑆𝑅𝑅 ) , accept expansion, and then if
𝐻𝐻(𝑆𝑆𝐻𝐻 ) > 𝐻𝐻(𝑆𝑆𝐸𝐸 ), 𝑆𝑆𝐸𝐸 replace 𝑆𝑆𝐻𝐻 . If 𝐻𝐻(𝑆𝑆𝐻𝐻 ) < 𝐻𝐻(𝑆𝑆𝐸𝐸 ), 𝑆𝑆𝑅𝑅
replace 𝑆𝑆𝐻𝐻 . If 𝐻𝐻(𝑆𝑆𝐻𝐻 ) < 𝐻𝐻(𝑆𝑆𝑅𝑅 ), accept contraction, andthen if 𝐻𝐻(𝑆𝑆𝐻𝐻 ) > 𝐻𝐻(𝑆𝑆𝐶𝐶 ) , 𝑆𝑆𝐶𝐶 replace 𝑆𝑆𝐻𝐻 . If not, accept
reduction.
5. Simulation Example
The effectiveness of the newly proposed design scheme
is evaluated on a numerical simulation example.
𝑦𝑦(𝑡𝑡) = −1.575𝑦𝑦(𝑡𝑡 − 1) − 0.654𝑦𝑦(𝑡𝑡 − 2) +ξ(t)0.023𝑢𝑢(𝑡𝑡 − 1) − 0.019𝑢𝑢(𝑡𝑡 − 2) +(21)∆

Firstly, Fig. 7 shows the control result by using manually
adjusted 𝜆𝜆.
when 𝜆𝜆 = 0.09

Fig. 8.the control result by using calculated 𝜆𝜆.

6. Conclusions

In this paper, a design scheme of database-driven GMV
controller has been proposed, which can obtain GMV
control parameters by applying the NM method based on
PID controller. The features of the newly proposed
control scheme are summarized as follows:• PID parameters are adjusted based on the GMV
control system.
• The user-specified parameter included in GMV
control system, is calculated by NM method for liner
system.
• The estimation of closed-loop response is introducedin database-driven approach.
The effectiveness of the proposed scheme has been
evaluated on the simulation example.

Fig. 7. Control result by using manually adjusted 𝜆𝜆.

From the result, using manually adjusted 𝜆𝜆 can not get
desirable control performance. Then Fig.8 shows thecontrol result by using the newly proposed method,where the design parameters of Nelder-Mead method are
set as shown in Table 1.

7. References
1. Michael A. J, Mohamman H. M. PID Control. SpringerVerlag, 2005.
2. Antonio V. Practical PID control. Springer-Verlag, 2006.
3. Harris, T.J. Assessment of closed loop performance. Can.
J. Chem. Eng. 1989, pp.856-861.
4. S. Wakitani, S. Hanata, T. Yamamoto. Design andApplication of a Direct GMV-Based PID Controller.
Proceedings of the 2012 International Conference on
Advanced Mechatronic Systems, 2012.
5. Astrom K. J; Hagglund T. The future of PID control.
Control Engineering Practice 2001, pp.1163-1175.
6. Nelder J. A, Mead R. A Simplex Method for Function
Minimization. Computer J. 1965, pp.100-313.
7. T. Sakatoku, K. Yubai, D. Yashiro, S. Komada. Estimationof Closed-Loop Response and Controller Retuning.
IEEJapan, 2020.

Table 1. Parameters used in Nelder-Mead method.

𝑆𝑆𝑚𝑚𝑚𝑚𝑚𝑚(1.0,1.0)

𝑆𝑆𝑚𝑚𝑚𝑚𝑚𝑚(0.0,0.0)

𝛽𝛽

0.1

𝛾𝛾

1.1

𝜂𝜂

0.1

Using the NM method, 𝜆𝜆 = 0.05 is the most suitableparameter. Comparing with the two kinds of control
results, it is clear that the newly proposed control scheme
works well.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

120

Design of a Database-Driven GPC for Nonlinear Systems
Zhe Guan

0F

KOBELCO Construction Machinery Dream-Driven Co-Creation Research Center, Hiroshima University
Higashi-Hiroshima, 739-8527, Japan
E-mail: guanzhe@hiroshima-u.ac.jp,
Toru Yamamoto

Graduate School of Advanced Science and Engineering, Hiroshima University
Higashi-Hiroshima, 739-8527, Japan
E-mail: yama@hiroshima-u.ac.jp

Abstract
This paper presents a Generalized Predictive Control (GPC) design scheme based on database-driven approach for
nonlinear systems. In industrial systems, lots of controlled plants with unknown time-delay and strong nonlinearity,
are difficult to be handled in terms of control performance. The databased-driven approach has been attracted
attention to tackle non-linearity issue without system information. On the other hand, GPC is considered in cases
where the time-delay is unknown. The proposed scheme is validated through a numerical simulation.
Keywords: GPC, Database-Driven approach, Nonlinear Systems

process model. Unfortunately, as it is well known,modeling for an industrial process using first principles
or identification is a difficult task. Further, even themathematical model of the industrial process isestablished, it only approximates the process but not
involves entire process dynamics. Moreover, it is timeand cost-consuming to build the relative accuratecomplicate model of industrial process. Therefore, somechallenges have been addressed in the model-based
technique.
With the rapid development of the computer
technology, industrial processes have undergone
significant changes. It is possible to store and processhuge amounts of process data with the help of computers.
In recent years, it is a trend to design the controller bydirectly using online or offline Input/Output (I/O) dataobtained from the controlled system or knowledge,
instead of the explicit mathematical model of thecontrolled process3. Several methods4,5 have been

1. Introduction
Industrial process control is a large and diverse field,
and its broad range of applications require a wide rangeof controllers including model predictive control (MPC)
and a variety of nonlinear controllers in order to maintainsystem performance. Among them, the Generalized
Predictive Control (GPC) algorithm1 can adapt to process
time-delay and model order based on the long-range
prediction. As a result, it is shown to be particularly
effective for the self-tuning control of industrial
processes2, in which are widely applied. However,regarding to the GPC, the excellent control behavior canbe achieved, provided that the number of parameters in
the model are required. It is important to include areasonable representation of disturbances and otherunknown dynamics, such that the correct controller canbe designed. In other words, the GPC is model-based
technique, so good performance needs a high-quality© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

121Zhe Guan, Toru Yamamotoproposed by considering the above idea. Moreover, thedata-oriented GPC controller6 has been proposed, where
the initial control parameters are estimated from the I/O
data, and then the corresponding polynomials arecalculated. However, two issues are still not addressed.
The aforementioned method is not able to deal with somesystems with nonlinearity. The other issue is that thetuning performs in off-line manner.
The database-driven approach7 is one of control
strategies applied in systems with non-linearity since it
can tune the control parameters rapidly at eachequilibrium point. Furthermore, the database-driven
approach performs the tuning in on-line manner in orderto deal with the operation condition along the time, suchthat the performance deterioration is able to be avoided.
Successful implementations of the data-driven approach
have been presented in the technical literatures8,9,10.
Based on the above observation, this paper newly
presents a design scheme that the controller is designedwith GPC algorithm based on database-driven approach.
The proposed scheme inherits the advantages of GPC
algorithm in terms of handling delay. Moreover, thecontrol parameters are computed on-line in a local
bounded neighbor data which is extracted from thedatabase.

Fig. 1. Block diagram of the proposed scheme.

where z −1 represents the backward shift operator which
implies z −1 𝑦𝑦(𝑡𝑡) = 𝑦𝑦(𝑡𝑡 − 1), and ∆ denotes the differential
operator defined by ∆≔ 1 − z −1 . 𝐴𝐴(z −1 ) and 𝐵𝐵(𝑧𝑧 −1 ) are
the polynomials which are formulated as follows.
(4)

(5)

Based on the above statement, the objective of thisstudy is to design a GPC controller that generates 𝑢𝑢(𝑡𝑡) tocontrolling the system Eq. (1) based on database-driven
approach such that the output can track the reference
value. As a result, the block diagram of the proposed
scheme is depicted in Fig. 1.
3.

3.1. Generalized Predictive Control

2. Problem Statement
Consider the following nonlinear system:

Proposed Scheme Design

The GPC law is derived based on the minimization of thecriterion in the following equation:(1)

where 𝑦𝑦(𝑡𝑡), 𝑓𝑓(∙) and 𝜉𝜉(𝑡𝑡) denote the system output, thenonlinear function and the white Gaussian noise withzeros mean and variance 𝜎𝜎 2 , respectively. Additionally,
𝜙𝜙(𝑡𝑡 − 1) is called information vector, which is defined
as follows:(6)
where 𝐸𝐸(∙) is the expectation value and [1, N] denotes
the range of control horizon and prediction horizon. 𝑤𝑤(𝑡𝑡)is the reference value and 𝜆𝜆(𝑗𝑗) is a user-specified control
weighting sequence. The control law based on Eq. (6) is
derived as:(2)
where 𝑢𝑢(𝑡𝑡) and 𝑑𝑑 represent the control input and systemdelay time. Also, 𝑛𝑛𝑦𝑦 and 𝑛𝑛𝑢𝑢 are the order of the system
output and control input, respectively.
When considering regulation about a particular operation
point, it is reasonable to assume that the nonlinear system
can be described as a locally-linearized model by thefollowing equation:(7)

with 𝐹𝐹𝑗𝑗 (𝑧𝑧 −1 ) and 𝐺𝐺𝑗𝑗 (𝑧𝑧 −1 ) are derived by the following
Diophantine equations:(3)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

122

Design of a Database-Driven

(8)

where

Moreover, 𝑝𝑝𝑗𝑗 is formulated by

(9)

(10)
Fig. 2. Control results using the conventional PID control.

with the matrix G is composed by(11)
3.2. Derivation of output by control parameters
Considering the Diophantine equations Eq. (8) andsubstitute to system Eq. (3), the system output can beobtained one-step ahead as:(12)Fig. 3. Control results using the data-oriented GPC.

The above equation is written in the form of information
vector as:(13)

4. Numerical Simulation

with 𝐹𝐹1 and 𝐺𝐺1 are the components in 𝜃𝜃(𝑡𝑡).
Once the control parameters 𝐹𝐹1 and 𝐺𝐺1 are identified, andthen by recursive formula, the control signal is able to be
derived. Please refer to the Ref. 6.

In this section, we have conducted the simulation andillustrate the effectiveness of the proposed scheme. The
11
Hammerstein model is considered and is given in thefollowing form:

3.3. Database-driven approach𝑦𝑦(𝑡𝑡) = 0.6𝑦𝑦(𝑡𝑡 − 1) − 0.1𝑦𝑦(𝑡𝑡 − 2)

+1.2𝑥𝑥(𝑡𝑡 − 𝑑𝑑 − 1) − 0.1𝑥𝑥(𝑡𝑡 − 𝑑𝑑 − 2) + 𝜉𝜉(𝑡𝑡)The detail of this approach will not be given because ofpages limitation. Please refer to the Ref. 7. We will givecore part in terms of control parameters update. As
mentioned in Eq. (13), the control parameters will be
updated based on database by adopting the steepest
descent method.

with 𝜂𝜂 learning rate, and 𝐽𝐽(𝑡𝑡 + 1) is defined by:𝑥𝑥(𝑡𝑡) = 𝑢𝑢(𝑡𝑡) − 𝑢𝑢2 (𝑡𝑡) + 0.5𝑢𝑢3 (𝑡𝑡)(17)

where 𝜉𝜉(𝑡𝑡) is the Gaussian noise with zero mean andvariance 0.012, and delay time d is set as 2 in this study.
The reference signals are given by:

(14)
(18)

(15)
(16)

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

123Zhe Guan, Toru Yamamotostrong non-linearity and provides a way for the online
utilization of the proposed scheme. Furthermore, theproposed scheme inherits the advantage of GPC
algorithm such that good performance can be achieved
regardless of time-delay. Finally, the effectiveness has
been verified on a numerical simulation. The
experimental evaluation will be in our future work.
References
1. D. W. Clarke, C. Mohtadi, and P S. Tuffs, Generalized
Predictive Control – Part I. The basis algorithm,
Automatica, 23(2), 1987, pp. 137-148.
2. D. W. Clarke, Application of generalized predictive
control to industrial processes, IEEE Control Systems
Magazine. 8(2), 1988, pp. 49-55.
3. Z. S. Hou and S. S. Xiong, On Model-Free Adaptive
Control and its Stability Analysis, IEEE Transactions on
Automatic Control, 64(11), 2019, pp. 4555-4569.
4. M. C. Campi, A. Lecchini, and S. M. Savaresi, Virtualreference feedback turning: a direct method for the designof feedback controllers, Automatica, 38(8), 2002, pp.
1337-1346.
5. H. Hjalmarsson, M. Gevers, S. Gunnarsson, and O. Lequin,
Iterative feedback tuning: theory and applications, IEEE
Control Systems Magazine, 18(4), 1998, pp. 26-41.
6. Z. Guan, S. Wakitani, and T. Yamamoto, Design andExperimental Evaluation of a Data-Oriented Generalized
Predictive PID Controller, Journal of Robotics andMechatronics, 28(5), 2016, pp. 722-729.
7. T. Yamamoto, K. Takao, and T. Yamada, Design of a
Data-Driven PID control, IEEE Transactions on Control
Systems Technology, 17(1), 2009, pp. 29-39.
8. K. Takao, T. Yamamoto, and T. Hinamoto, A new GPCbased PID controller using memory-based identification,
Proceedings of the 11th IFAC International Workshop on
Adaptation and Learning in Control and Signal
Processing, 2013, pp. 587-592.
9. S. Wakitani, M. C. Deng, and T. Yamamoto, Design of adata-driven controller for a spiral heat exchanger,
Proceedings of 11th IFAC Symposium on Dynamics andControl of Process Systems Including Biosystems, 49(7),2016, pp. 342-346.
10. Z. Guan, I. Mizumoto, and T. Yamamoto, Design of PCAbased Data-Driven Adaptive Output Feedback Control for
Nonlinear Systems, IEEE 28th International Symposiumon Industrial Electronics, 2019, pp. 497-502.
11. Z. Q. L, On identification of the controlled plants
described by the Hammerstein system, IEEE Transactions
on Automatic Control, 39(2), 1994, pp. 569-573.
12. K. L. Chien, J. A. Hrones, and J. B. Reswick, On theautomatic control of generalized passive systems,
Transaction ASME, 74, 1972, pp. 175-185.

Fig. 4. Control results using the proposed scheme.

Additionally, the user-specified parameters involved in
the proposed scheme are determined as shown in Table 1.
Table 1. The user-specified parameters.
The orders of information vector
Numbers of Neighbors
Weighting factor
Initial data-base N(0)Learning rate𝑛𝑛𝑦𝑦 = 2
𝑛𝑛𝑢𝑢 = 5

𝑘𝑘 = 5𝜆𝜆 = 1520
100

The proposed scheme cannot work unless the initial
database is generated. Consequently, the conventional
Proportional-Integral-Derivative (PID) control tuned by
the Chien-Hrones-Reswick method12 is employed. The
related parameters are shown as:
𝐾𝐾𝑃𝑃 = 0.486, 𝐾𝐾𝐼𝐼 = 0.227, 𝐾𝐾𝐷𝐷 = 0.122.

The control results are depicted in Fig. 2, where theoutput is become unstable after 200 [STEP] because of
nonlinearity and delay time. These I/O data form thebasis of the initial database. Fig. 3 shows the control
results where the control parameters 𝐹𝐹1 and 𝐺𝐺1 in GPC
algorithm are identified from the I/O data and are
identical in the control process. The tracking property ispoor, and oscillation occurs after 200 [STEP]. Finally,
the proposed scheme is employed to the system, and Fig.
4 shows that the proposed scheme outperforms the onesin Fig. 3. It should be noted that the control parameters
𝐹𝐹1 and 𝐺𝐺1 are updated based on database along the time.
5. Conclusions

In this paper, the design of a data-driven GPC control for
nonlinear systems has been presented. First, according to
the proposed scheme, the model information is not
required in designing the controller. Then, the databasedriven approach is considered to deal with systems with© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

124Design of a Data-Driven Controller using Open-Loop Data
Yasuteru Nishiya

Graduate School of Advanced Science and Engineering, Hiroshima University
1-4-1, Kagamiyama, Higashihiroshima city, Hiroshima, Japan
Takuya Kinoshita

Graduate School of Advanced Science and Engineering, Hirosihma University,
1-4-1, Kagamiyama, Higashihiroshima city, Hiroshima, Japan
Toru Yamamoto

Graduate School of Advanced Science and Engineering, Hirosihma University,
1-4-1, Kagamiyama, Higashihiroshima city, Hiroshima, Japan
E-mail: {nishiya-yasuteru, kinoshita-takuya, yama}@hiroshima-u.ac.jp
http://www.hiroshima-u.ac.jp

Abstract
In recent years, data-driven control has been proposed and extended to a non-linear system by using database. At thistime, various data are required to obtain good control performance but the cost is required. In this paper, a new schemethat enables various data generation and control system design from a set of open-loop data is proposed. Besides, thefilter is designed to keep the value of the reference signal constant. The effectiveness of the proposed scheme isnumerically verified.
Keywords: data-driven control, PID controller, response prediction, offline, reference signal, filter

estimated data. Moreover, the estimation accuracy isdegraded when constructing a one-degree-of-freedom
control system. Therefore, in this paper, a new control
scheme that can generate I/O data offline using a set ofopen-loop data. The proposed scheme is a simple methodthat requires only the replacement of blocks on the block
diagram to generate data. It is also an effective method
for the unknown high order systems. Since someprediction I/O data can be obtained offline withexperimental data, the time and human costs forexperiments can be reduced. Furthermore, the feature of
the proposed scheme are as follows:(i) The frequency response of the system can be
estimated.
(ii) It can be extended to control system design.

1. Introduction
Recently, data-driven control schemes1,2,3 have been
proposed, and it can calculate control parameters directlyfrom a set of input/output data without system modeling.
In addition, a Database-Driven control methods (DD)4,5
have been proposed for nonlinear systems using
databases. In a DD control system, various data need to
be stored in a database to obtain good control
performance, but collecting enough data requires
multiple experiments, and it takes time and human cost.
In order to avoid the costs associated with dataacquisition, DD control using an Estimated Response
Iterative Tuning (DD-ERIT)6 has been proposed. It
generates estimated input/output data from a closed-loop
data set. However, in the ERIT method, a two-degree-offreedom control system must be used to obtain the

2. Basic idea of the proposed scheme© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

125

Yasuteru Nishiya, Takuya Kinoshita, Toru YamamotoFig. 1. Block diagram of open-loop system.

Fig. 2. Outline of the proposed scheme.

Fig. 3. Generate signal 𝐸𝐸 ∗ (𝑠𝑠) and 𝑅𝑅�(𝑠𝑠) based on the
proposed scheme.

In the proposed scheme, open-loop data 𝑈𝑈0 (𝑠𝑠) and 𝑌𝑌0 (𝑠𝑠)are obtained in Fig. 1. Next, consider applying an
arbitrary input signal 𝑈𝑈(𝑠𝑠) shown in Fig. 3. The
experiment is required to obtain 𝑌𝑌(𝑠𝑠) by applying an
arbitrary 𝑈𝑈(𝑠𝑠) when the system is unknown. Therefore,
in the proposed scheme, the following 𝑈𝑈(𝑠𝑠) is given byusing the initial data 𝑈𝑈0 (𝑠𝑠):𝑈𝑈(𝑠𝑠) = 𝑋𝑋(𝑠𝑠)𝑈𝑈0 (𝑠𝑠),(1)where 𝑋𝑋(𝑠𝑠) denotes the transfer function given by theuser to calculate 𝑈𝑈(𝑠𝑠) using 𝑈𝑈0 (𝑠𝑠). Here, the output 𝑌𝑌(𝑠𝑠)in Fig. 2 is equivalent even if the front and back of theblocks are swapped because of the properties of the linear
block diagram. Therefore, the output 𝑌𝑌(𝑠𝑠) is calculated
by following equation:
𝑌𝑌(𝑠𝑠) = 𝑋𝑋(𝑠𝑠)𝑌𝑌0 (𝑠𝑠).
(2)
From equation (1) and (2), the output 𝑌𝑌(𝑠𝑠) can becalculated offline by using the initial open-loop data
𝑈𝑈0 (𝑠𝑠), 𝑌𝑌0 (𝑠𝑠). Also, in Fig. 2, various input/output data
can be generated offline by changing 𝑋𝑋(𝑠𝑠) . The
following sections describe the control system designusing an open-loop data.

of Fig. 3 and Fig. 4 will be described later. First, theoutput of the reference model 𝑌𝑌𝑚𝑚 (𝑠𝑠) is given by
𝑌𝑌𝑚𝑚 (𝑠𝑠) = 𝐺𝐺𝑚𝑚 (𝑠𝑠)𝑅𝑅(𝑠𝑠)
𝐺𝐺(𝑠𝑠)𝐶𝐶 ∗ (𝑠𝑠)(3)=
𝑅𝑅(𝑠𝑠),
1 + 𝐺𝐺(𝑠𝑠)𝐶𝐶 ∗ (𝑠𝑠)where 𝐺𝐺𝑚𝑚 (𝑠𝑠) denotes the reference model, 𝐺𝐺(𝑠𝑠) denotes
the controlled system, 𝐶𝐶 ∗ (𝑠𝑠) shows the optimal
controller and 𝑅𝑅(𝑠𝑠) represents the reference signal. In
this case, the desired error 𝐸𝐸 ∗ (𝑠𝑠) is defined by
𝐸𝐸 ∗ (𝑠𝑠) = 𝑅𝑅(𝑠𝑠) − 𝑌𝑌𝑚𝑚 (𝑠𝑠)= �1 − 𝐺𝐺𝑚𝑚 (𝑠𝑠)�𝑅𝑅(𝑠𝑠).
(4)
The error 𝐸𝐸(𝑠𝑠) is expressed by
1
𝐸𝐸(𝑠𝑠) =𝑅𝑅(𝑠𝑠).
(5)
1 + 𝐺𝐺(𝑠𝑠)𝐶𝐶(𝑠𝑠)From equation (5), 𝑅𝑅(𝑠𝑠) is expressed by
𝑅𝑅(𝑠𝑠) = �1 + 𝐺𝐺(𝑠𝑠)𝐶𝐶(𝑠𝑠)�𝐸𝐸(𝑠𝑠).
(6)∗�Here, 𝑅𝑅(𝑠𝑠) which achieve the desired error 𝐸𝐸 (𝑠𝑠) is
introduced by following equation:
𝑅𝑅� (𝑠𝑠) ≔ �1 + 𝐺𝐺(𝑠𝑠)𝐶𝐶(𝑠𝑠)�𝐸𝐸 ∗ (𝑠𝑠)= 𝐻𝐻(𝑠𝑠)𝐸𝐸 ∗ (𝑠𝑠)(7)
𝐻𝐻(𝑠𝑠) ≔ 1 + 𝐺𝐺(𝑠𝑠)𝐶𝐶(𝑠𝑠).
(8)
𝑅𝑅� (𝑠𝑠) is a reference signal in a closed-loop system, but it
is not computable because the parameters of the transfer
function 𝐺𝐺(𝑠𝑠) in equation (7) are unknown. Here, 𝑅𝑅� (𝑠𝑠)is generated based on the proposed scheme, and its
overview is shown in Fig. 3. Assuming that the reference
signal 𝑅𝑅(𝑠𝑠) and the initial input 𝑈𝑈0 (𝑠𝑠) are equivalent
step signals, the 𝑅𝑅(𝑠𝑠) in equation (4) can be replaced by

3. Design of a data-driven PID controller using a
set of open-loop I/O data
Fig. 3 and Fig. 4 show an overview of the controller
design using the proposed scheme. First, a set of data is
generated using the proposed scheme according to Fig. 3and then design the controller 𝐶𝐶(𝑠𝑠) to minimize 𝜖𝜖1 (𝑠𝑠)shown in Fig. 4 based on the generated data. The details© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

126Design of a Data-Driven

4.

�(𝑠𝑠) = 𝐶𝐶(𝑠𝑠)𝐸𝐸 ∗ (𝑠𝑠).
𝑈𝑈

(12)

Simulation Examples

4.1. Controlled system
The following controlled system is discussed:
𝑠𝑠 + 6
𝐺𝐺(𝑠𝑠) = 3.
𝑠𝑠 + 6𝑠𝑠 2 + 11𝑠𝑠 + 6

Fig. 4. Block diagram of the minimization problem.
𝑈𝑈0 (𝑠𝑠). 𝐸𝐸 ∗ (𝑠𝑠) in equation (4) can be computed offline
because 𝑈𝑈0 (𝑠𝑠) has already been obtained as initial input
data. However, from equation (7), it is impossible tocompute 𝑅𝑅(𝑠𝑠) offline using the desired error 𝐸𝐸 ∗ (𝑠𝑠)because the controlled system 𝐺𝐺(𝑠𝑠) is unknown.
Therefore, in the proposed scheme, 𝑅𝑅� (𝑠𝑠) can becomputed offline by replacing the order of the blocks, as
shown in Fig. 3. Note that, the signal 𝑂𝑂(𝑠𝑠) in Fig. 3 canalso be calculated according to the proposed scheme.
From the above, 𝐸𝐸 ∗ (𝑠𝑠) and 𝑅𝑅� (𝑠𝑠) can be computed
offline for any 𝐶𝐶(𝑠𝑠) using the initial data 𝑈𝑈0 (𝑠𝑠) and𝑌𝑌0 (𝑠𝑠).
However, the original goal of control is to design acontroller 𝐶𝐶(𝑠𝑠) in which output 𝑌𝑌(𝑠𝑠) follows to thereference output 𝑌𝑌𝑚𝑚 (𝑠𝑠) . The output 𝑌𝑌� (𝑠𝑠) , where thereference signal is 𝑅𝑅� (𝑠𝑠) and the error is 𝐸𝐸 ∗ (𝑠𝑠) in aclosed-loop system, can be expressed by the following
equation using (4) and (7):(9)
𝑌𝑌�(𝑠𝑠) = 𝑅𝑅� (𝑠𝑠) − 𝐸𝐸 ∗ (𝑠𝑠).
�Fig. 4 shows the relationship between 𝑌𝑌(𝑠𝑠) and 𝑌𝑌𝑚𝑚 (𝑠𝑠)using the step signal. At this time, let 𝑌𝑌(𝑠𝑠) and 𝑌𝑌𝑚𝑚 (𝑠𝑠) be
Laplace inverse transformed signals 𝑦𝑦�(𝑡𝑡) and 𝑦𝑦𝑚𝑚 (𝑡𝑡) ,respectively, and define the following equation as anevaluation function using the sum of the squares of theerrors:2

The step signal used in the initial experiments 𝑢𝑢0 (𝑡𝑡) is
equal to 1.0. The reference output 𝑦𝑦𝑚𝑚 (𝑡𝑡) is given by𝑧𝑧 −1 𝑇𝑇(1)𝑦𝑦𝑚𝑚 (𝑡𝑡) =𝑟𝑟(𝑡𝑡).
(14)
𝑇𝑇(𝑧𝑧 −1 )Here, let 𝑇𝑇(𝑧𝑧 −1 ) be the denominator of the discrete timeis defined as follows7:

2

2

(15)

𝑇𝑇(𝑧𝑧 −1 ) = 1 − 1.213𝑧𝑧 −1 + 0.368𝑧𝑧 −2 ,(17)

where 𝑇𝑇(𝑧𝑧 −1 ) was designed by setting 𝜎𝜎 as 0.5 and 𝛿𝛿 as
0.0. Here, 𝑇𝑇𝑆𝑆 is equal to 0.001. In addition, the PID
controller 𝐶𝐶(𝑠𝑠) is designed as follows:1(18)
𝐶𝐶(𝑠𝑠) = 𝐾𝐾𝑃𝑃 + 𝐾𝐾𝐼𝐼 + 𝑠𝑠𝐾𝐾𝐷𝐷 .
𝑠𝑠

The controller in a closed-loop system is designed by
optimizing the control parameters using equation (10).
However, the input response is unknown when designinga control system focusing on only the output signal. In
this case, the system will be overloaded and cause failure
if the input response is excessive. Therefore, thefollowing criterion 𝐽𝐽2 with a weight coefficient 𝜆𝜆 is
introduced to consider input signal:
𝐽𝐽2 = � ��𝑦𝑦𝑚𝑚 (𝑡𝑡) − 𝑦𝑦�(𝑡𝑡)� + 𝜆𝜆�𝑢𝑢(𝑡𝑡) − 𝑢𝑢(𝑡𝑡 − 1)� � 𝑑𝑑𝑑𝑑,𝑇𝑇(𝑧𝑧 −1 ) = 1 + 𝑡𝑡1 𝑧𝑧 −1 + 𝑡𝑡2 𝑧𝑧 −2⎧𝑡𝑡 = −2 exp �− 𝜌𝜌 � cos ��4𝜇𝜇 − 1 𝜌𝜌�
1
2𝜇𝜇
2𝜇𝜇
⎪
⎪
𝜌𝜌
𝑡𝑡2 = exp �− �
,
(16)
𝜇𝜇
⎨
𝑇𝑇𝑠𝑠
⎪𝜌𝜌 =
⎪
𝜎𝜎
⎩𝜇𝜇 = 0.25(1 − 𝛿𝛿) + 0.51𝛿𝛿
where 𝑇𝑇𝑆𝑆 denotes sampling time. 𝜎𝜎 and 𝛿𝛿 show theparameters related to the rise-time and the damping
characteristic, respectively. Here, 𝑇𝑇(𝑧𝑧 −1 ) was set as
follows:(10)

𝐽𝐽1 = ��𝑦𝑦𝑚𝑚 (𝑡𝑡) − 𝑦𝑦�(𝑡𝑡)� 𝑑𝑑𝑑𝑑.

(13)

4.2. Design of data-driven PID controllerFirst, the step response for equation (13) as a set of
initial data 𝑢𝑢0 (𝑡𝑡) and 𝑦𝑦0 (𝑡𝑡) is obtained. Next, Fig. 5
shows the control results of applying the proposed
scheme to minimize the evaluation function of equation(10)(9) using the initial data. In Fig. 5, 𝑦𝑦(𝑡𝑡) and 𝑢𝑢(𝑡𝑡)denote the output and input for a closed-loop systemwith a reference signal 𝑟𝑟̂ (𝑡𝑡). 𝑦𝑦�(𝑡𝑡) and 𝑢𝑢�(𝑡𝑡) represent theprediction output and input. Also, 𝑦𝑦𝑚𝑚 (𝑡𝑡) denote theoutput of the reference model. In this case, the control
parameters were tuned as follows:(11)

where 𝑢𝑢�(𝑡𝑡) is an estimate of the input and the Laplace�(𝑠𝑠) can be calculated as follow
transformed value 𝑈𝑈equation:© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

127

Yasuteru Nishiya, Takuya Kinoshita, Toru YamamotoFig. 5. Control results by the proposed scheme thatminimizing equation (10).

Fig. 6. Control results by the proposed scheme thatminimizing equation (11).

(19)

planned to design a controller for nonlinear systems andverify the effectiveness of the proposed scheme byexperiment.

𝐾𝐾𝑃𝑃 = 0.261, 𝐾𝐾𝐼𝐼 = 0.167, 𝐾𝐾𝐷𝐷 = 0.091.

Fig. 5 shows that the accuracy of 𝑦𝑦�(𝑡𝑡) and 𝑢𝑢�(𝑡𝑡) are high.
However, the input 𝑢𝑢(𝑡𝑡) changes abruptly and theactuator is heavily loaded since evaluation function inequation (9) is considered only 𝑦𝑦(𝑡𝑡).
Next, the controller is designed by minimizing theevaluation function of equation (11). Fig. 6 shows thecontrol results. In this case, the design the desired
characteristic polynomial included in the reference model
was set as equation (17), and the weight coefficient λ isset as 0.016. In addition, the control parameters weretuned as follows:
𝐾𝐾𝑃𝑃 = 0.223, 𝐾𝐾𝐼𝐼 = 0.143, 𝐾𝐾𝐷𝐷 = 1.85 × 10−5 .

References
1. S. Soma, O. Kaneko and T.Fujii, “A New Approach to
Parameter Tuning of Controllers by Using One-Shot
Experimental Data: A Proposal of Fictitious Reference
Iterative Tuning (in Japanese)”, Trans. of the Institute ofsystems, Control and Information Engineers, 17(2) (2004)
pp. 528–536.
2. M.C. Campi, A. Lecchini and S.M. Savaresi, “Virtual
Reference Feedback Tuning: A Direct Method for the
Design of Feedback Controllers”, Automatica, 38(8) 2002,
pp. 1337-1346.
3. T. Nakamura, O. Kaneko, “Estimated Response Iterative
Tuning of Feed-Forward Controller in the Two-Degree ofFreedom Control System (in Japanese)”, The society of
Instrument and Control Engineers, 2017
4. T. Yamamoto, K. Takao and T. Yamada, “Design of a
Data-Driven PID Controller”, IEEE transactions oncontrol systems technology, 17(1), 2009, pp. 23-29.
5. S. Wakitani, T. Yamamoto and B. Gopaluni, “Design andApplication of a Database-Driven PID Controller with
Data-Driven Updating Algorithm”, Industrial &Engineering Chemistry Research, 58(26), 2019, pp.
11419-11429.
6. R. Okada, T, Kinoshita and T. Yamamoto, “Design of a
Database-Driven Controller Using Estimated Data”,
Proceedings of the Society of Instrument and Control
Engineers Annual Conference 2019, WeB02.3, Hiroshima.
7. T. Yamamoto, S. L. Shah, “Design and Experimental
Evaluation of a Multivariable Self-Tuning PID Controller”,
IEEE Proc. of Control Theory and Applications, 151, 2004,
pp. 645-652.

(20)

Comparing Fig. 6 with Fig. 5, it can be confirmed that the
input changes gently and the maximum value becomes
smaller by changing the evaluation function. This
indicates that the load on the actuator becomes smaller.
In addition, Fig. 6 shows the prediction I/O data 𝑦𝑦�(𝑡𝑡) and𝑢𝑢�(𝑡𝑡) are consistent with the actual data 𝑦𝑦(𝑡𝑡) and 𝑢𝑢(𝑡𝑡),respectively. The prediction accuracy is well estimatedwithout degradation.
5.

Conclusion

In this paper, a new scheme that multiple data can begenerated offline from a set of open-loop I/O data has
been proposed. According to the proposed scheme,
controller can be designed by generating data offline
from a set of open-loop I/O data. The effectiveness of the
proposed scheme has been verified by a numericalsimulation. It is possible to design the controller to avoid
overloads in actual machines by setting the evaluation
functions appropriately. The future work for this study is© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

128

Improved Estimation of Sway-Angle for Overhead Crane based on Phase Difference ofAcoustic Signals in Frequency Domain
Hanako Ogawa, Takeshi Yamada, and Masayoshi Nakamoto

Hiroshima University, Kagamiyama 1-4-1
Higashi Hiroshima, 739-8527, Japan
E-mail:oagwa-hanako@hiroshima-u.ac.jp, msy@hiroshima-u.ac.jp
Abstract
During a crane operation, swaying of the payload is an issue that should be controlled. For anti-sway control, it isessential to detect the sway angle. In this paper, we propose a method for obtaining the phase difference of the acoustic
signals within the frequency domain by using the discrete Fourier transform. Based on the phase difference, we
compute the time of arrival and convert it into the sway-angle of deflection. In experiment, we show the effectivenessof the proposed method by comparing with the sway angle observed from camera.
Keywords: Overhead crane, sway angle, two microphones, time difference of the arrival (TDOA)

Another method for measuring the sway angle of acrane was also proposed, which uses the time differenceof the arrival (TDOA) of sound waves [5]. Although withthis method the sway angle is calculated from TDOA of
the sound waves, the accuracy depends on the sampling
frequency. In addition, with such methods, the differencein the arrival time is calculated based on the correlation;
however, the computational burden is high because
multiple signals are applied to find the maximum value.
In addition, the experimental results also show large
errors, and thus there is still room for improvement.
Therefore, we propose a method for determining thephase of the acoustic signal using da discrete Fouriertransformtr (DFT) and calculating the difference in thearrival time and the sway angle from the phase. We alsopresent a frequency selection method for a sound source
generation that improves the accuracy of the DFT phasecalculation. There are multiple candidate values when
determining the difference in the arrival time. When thefrequency of the acoustic signal is low, it is more
susceptible to environmental noise, and at higher
frequencies, even small noise can affect the accuracy ofthe signal because the difference in the interval between
the candidate values of the arrival time narrow despite thehigher accuracy. Such aspects make it difficult to choose
an appropriate difference in the arrival time. To avoid
this problem, we also propose using signal processing forthe correct tracking of the differences in the arrival time.

1. Introduction
Cranes are widely used in industries such as factories,
warehouses, and construction sites, and various types of
cranes available according to their intended use. Amongthem, overhead cranes are frequently applied. An
overhead crane carries a load using a cart on rails
provided in the ceiling of the building. During theoperation of the crane, there is a need for anti-swaying of
the load. Anti-swaying is currently conducted manuallyand requires skilled workers. However, crane operators
are aging, and the number of skilled operators isdecreasing. Therefore, there is a growing need forcomputerized anti-sway control of the lifting load tosupport unskilled personnel, promote efficiency, andimprove the safety of crane operations. With thisbackground, research on anti-sway control is being
conducted.
To conduct anti-sway control using a computer, it isessential to detect the sway angle, and an image sensor[1]or encoder[2]-[4] is mainly used. To estimate the swayangle using an image sensor, it is necessary to install the
sensor at the location where the sway angle can be
captured correctly. In additionFuthermore, a lengthy
processing time is required to process the video signal
obtained from the sensor. Direct measurements of thesway angle are difficult to achieve using an encoderwhen the rope of the load is deflected.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

129

Hanako Ogawa, Takeshi Yamada, Masayoshi Nakamoto

In experiments on measuring the crane sway angle, acomparison was made between the sway angle estimatedby the proposed method and that measured by thecameras. We also confirmed that the difference in thearrival time can be correctly tracked through signal
processing.

between 𝑃𝑃𝑀𝑀1 and 𝑃𝑃𝑀𝑀2 . Now, consider estimating 𝛿𝛿using the TDOA. Let 𝜏𝜏𝑘𝑘 be the number of samples for thetime difference between the two signals.
In [5], 𝜏𝜏𝑘𝑘 can be computed using the correlation of two
signals. The TDOA between 𝑀𝑀1 and 𝑀𝑀2 is 𝜏𝜏𝑘𝑘 𝑇𝑇𝑠𝑠 , where
𝑇𝑇𝑠𝑠 is the sampling period. It follows that 𝛿𝛿 can becomputed as follows:𝛿𝛿 = 𝜏𝜏𝑘𝑘∗ 𝑇𝑇𝑠𝑠 𝑣𝑣𝑎𝑎

(1)

where 𝑣𝑣𝑎𝑎 is the velocity of sound.

The problem here is that the resolution of 𝛿𝛿 depends on
𝑇𝑇𝑠𝑠 because 𝜏𝜏𝑘𝑘 𝑇𝑇𝑠𝑠 are discrete values of interval 𝑇𝑇𝑠𝑠 . Hence,
we present a new approach to obtain 𝛿𝛿 that is
independent of 𝑇𝑇𝑠𝑠 . Let 𝜆𝜆1 (𝑘𝑘�) and 𝜆𝜆2 (𝑘𝑘�) be phases of
𝑠𝑠1 (𝑡𝑡) and 𝑠𝑠2 (𝑡𝑡), respectively. Using a discrete Fouriertransform (DFT), 𝜆𝜆1 (𝑘𝑘� ) and 𝜆𝜆2 (𝑘𝑘� ) can be obtained by
the following:

Fig. 1 Overhead crane system and its parameters

𝑁𝑁−1

2𝜋𝜋 �

�

𝑆𝑆1 �𝑘𝑘� � = � 𝑠𝑠1 (𝑛𝑛)𝑒𝑒 −𝑗𝑗 𝑁𝑁 𝑘𝑘𝑛𝑛 = 𝐴𝐴1 �𝑘𝑘� �𝑒𝑒 𝑗𝑗𝜆𝜆1�𝑘𝑘� ,2. Acoustic signal processing for measurement ofsway angle

𝑛𝑛=0
𝑁𝑁−1

2𝜋𝜋

�
�
𝑆𝑆2 �𝑘𝑘�� = � 𝑠𝑠2 (𝑛𝑛)𝑒𝑒 −𝑗𝑗 𝑁𝑁 𝑘𝑘𝑛𝑛 = 𝐴𝐴2 �𝑘𝑘� �𝑒𝑒 𝑗𝑗𝜆𝜆2�𝑘𝑘�,

In [5], two microphones are used for the contactlesssensing of the angle, given the assumption that the
payload (or hook) includes sound sources, as shown in
Fig. 1. Using the two microphones, we estimated TDOA
between the acoustic signals of the sound source in the
payload.

𝑛𝑛=0

(2)

(3)

where 𝑘𝑘� is a positive integer that corresponds to thefrequency of the sound source, N is the number of data,
and 𝐴𝐴1 �𝑘𝑘�� and 𝐴𝐴2 (𝑘𝑘� ) are the amplitudes of thefrequency spectra.
Define the difference between 𝜆𝜆1 �𝑘𝑘�� and 𝜆𝜆2 (𝑘𝑘� ) as

2.1 Estimation of TDOA
Fig. 1 shows an overhead crane system in which thetrolley has two microphones labeled 𝑀𝑀1 and 𝑀𝑀2 . The
sound source is embedded in the payload (P in Fig.1). We
assume the sway angle is 𝜃𝜃 , and the distance from thetrolley to the sound source is 𝑙𝑙𝑤𝑤 . Using the microphones,
we first observe the sound signal originating from thesound source. It should be noted that the sensitivity to thesound signal from behind the microphones is extremely
low because the microphones have directivity. Now, let
𝑠𝑠1 (𝑡𝑡) be the sound signal observed in 𝑀𝑀1 , and 𝑠𝑠2 (𝑡𝑡) be
the sound signal observed in 𝑀𝑀2 .
Assume that the distance between 𝑀𝑀1 and the trolley (Qin Fig. 1) is 𝑋𝑋1 and the distance between 𝑀𝑀2 and thetrolley is 𝑋𝑋2 .
Hence, the distance between 𝑀𝑀1 and the sound source is𝑙𝑙𝑑𝑑 + 𝛿𝛿 , and the distance between 𝑀𝑀2 and the soundsource is 𝑙𝑙𝑑𝑑 . Assume that 𝑋𝑋1 , 𝑋𝑋2 , and are known in Fig. 1.
It should be noted that 𝛿𝛿 is the difference in distanceΔ𝜆𝜆 = 𝜆𝜆1 �𝑘𝑘� � − 𝜆𝜆2 �𝑘𝑘� �.

Using (4), the TDOA can be computed through thefollowing:

(4)

Δ𝜆𝜆
,
2𝜋𝜋𝑓𝑓𝑘𝑘�(5)𝛿𝛿 = 𝑣𝑣𝑎𝑎 Δ𝑡𝑡.

(6)

Δ𝑡𝑡 =

where 𝑓𝑓𝑘𝑘� is the frequency of the sound source.
Consequently, the following velocity of sound 𝑣𝑣𝑎𝑎 , 𝛿𝛿 canbe obtained :To generate the acoustic signal of the sound source, thediscrete frequency of the DFT is computed as© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

130

Improved Estimation of Sway-Angle(𝑁𝑁 − 1)𝑓𝑓𝑠𝑠
𝑓𝑓𝑠𝑠 2𝑓𝑓𝑠𝑠
(7)
,
,… ,
,
𝑁𝑁 𝑁𝑁
𝑁𝑁
Here 𝑓𝑓𝑠𝑠 is the sampling frequency 𝑓𝑓𝑠𝑠 = 1/𝑇𝑇𝑠𝑠 . Now, it isassumed that N and 𝑓𝑓𝑠𝑠 are given in the specifications of
the hardware. In this study, the frequencies
corresponding to the discrete frequencies, as indicated in(7), are used. Hence, the frequency (or frequencies) of
the sound are selected such thatLet 𝜃𝜃0 be an initial value of the sway angle, and 𝜃𝜃𝑛𝑛 be
the sway angle at the 𝑛𝑛-th step. It follows from (12) that
Newton’s method can be used to solve (11). Thus, we
obtain the sway angle by repeating the following
procedure:0,

𝑓𝑓𝑘𝑘� =𝑘𝑘� 𝑓𝑓𝑠𝑠,𝑁𝑁

𝑓𝑓(𝜃𝜃𝑛𝑛 )(13).
𝑑𝑑𝑑𝑑(𝜃𝜃𝑛𝑛 )�
�𝑑𝑑𝑑𝑑
The terminal condition is |𝜃𝜃𝑛𝑛+1 − 𝜃𝜃𝑛𝑛 | < 𝜀𝜀 wwhere 𝜀𝜀 is asmall positive number.
𝜃𝜃𝑛𝑛+1 = 𝜃𝜃𝑛𝑛 −(8)where 1 ≤ 𝑘𝑘� ≤ 𝑁𝑁 − 1 . That is, to achieve a highperformance of (6), the frequency (or frequencies) of thesound source is decided by N and 𝑓𝑓𝑠𝑠 of the soundreceiving device.
2.2 Sway angle estimation Consider the transformation of𝛿𝛿 to 𝜃𝜃 by the following procedure: Using the cosine
theorem for △ M1 PQ and △ M2 PQ , we have thefollowing:𝜋𝜋2− 2𝑋𝑋1 𝑙𝑙𝑤𝑤 cos � − 𝜃𝜃�𝑙𝑙𝑑𝑑 = 𝑋𝑋12 + 𝑙𝑙𝑤𝑤2�
𝜋𝜋2
(𝑙𝑙𝑑𝑑 + 𝛿𝛿)2 = 𝑋𝑋22 + 𝑙𝑙𝑤𝑤
− 2𝑋𝑋2 𝑙𝑙𝑤𝑤 cos � + 𝜃𝜃� .
2Now, letthe following:2 − 2𝑋𝑋 𝑙𝑙 sin𝜃𝜃𝑓𝑓(𝜃𝜃) = 𝛿𝛿 + �𝑋𝑋12 + 𝑙𝑙𝑤𝑤
1 𝑤𝑤
2 + 2𝑋𝑋 𝑙𝑙 sin𝜃𝜃.
−�𝑋𝑋22 + 𝑙𝑙𝑤𝑤
2 𝑤𝑤

Microphones
PayloadCamera(9)
Fig. 2 Experimental devices (microphones, payload, camera)

3.

(10)

Fig. 2 shows the experimental equipment. In thismeasurement, a sine wave was used as the sound sourcein the payload. Here, the parameter of the terminal
condition is set as 𝜀𝜀 = 0.0001. The parameters are 𝑙𝑙𝑤𝑤 =15.11 [m], 𝑁𝑁 = 102 , 𝑓𝑓𝑠𝑠 = 44100 [Hz] , and 𝑋𝑋1 = 𝑋𝑋2 =30 [cm]. In addition, video clips were captured, and thesway angles were then calculated from the image
processing and compared. The frequency spectra of theobserved signals are shown in Fig. 3. In this experiment,
the frequencies of 𝑓𝑓𝑘𝑘 = 5038.77 [Hz]and 𝑓𝑓𝑘𝑘 =8010.35 [Hz] were used.
The payload (sound source) was individually vibrated.
The estimation of 𝜃𝜃 for 𝑓𝑓𝑘𝑘 = 5038.77 [Hz] is shown in
Fig. 4 where an offset of stationary state is considered.
Similar to Fig. 4, the estimation of 𝜃𝜃 for 𝑓𝑓𝑘𝑘 =8010.35 [Hz] is shown in Fig. 5. The blue line is theThen, 𝜃𝜃 can be obtained by computing the solution of thefollowing equation:
𝜋𝜋 𝜋𝜋𝑓𝑓(𝜃𝜃) = 0, 𝜃𝜃 ∈ �− , � .
2 2Experiment(11)

However, it is difficult to solve (11) explicitly because
1it is a non-linear equation. Hence, we introduce analgorithm to solve (11) based on Newton’s method.
Differentiating 𝑓𝑓(𝜃𝜃) by 𝜃𝜃, we have the following:1
𝑑𝑑𝑑𝑑(𝜃𝜃)
2= {𝑋𝑋12 + 𝑙𝑙𝑤𝑤
− 2𝑋𝑋1 𝑙𝑙𝑤𝑤 sin𝜃𝜃}−2 {−𝑋𝑋1 𝑙𝑙𝑤𝑤 cos𝜃𝜃}
𝑑𝑑𝑑𝑑
1
2+ 2𝑋𝑋2 𝑙𝑙𝑤𝑤 sin𝜃𝜃}−2 {𝑋𝑋2 𝑙𝑙𝑤𝑤 cos𝜃𝜃}. (12)−{𝑋𝑋22 + 𝑙𝑙𝑤𝑤

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

131

Hanako Ogawa, Takeshi Yamada, Masayoshi Nakamoto

3. N. Sun, T. Yang, H. Chen and Y. Fang: Dynamic
Feedback Antiswing Control of Shipboard Cranes Without
Velocity Measurement: Theory and Hardware
Experiments, IEEE Trans. Ind. Inform., vol.15, no.5,
pp.2879-2891, May 2019.
4. N. Sun, Y. Fu, T. Yang, J. Zhang, Y. Fang and X. Xin:
Nonlinear Motion Control of Complicated Dual Rotary
Crane Systems Without Velocity Feedback: Design,
Analysis, and Hardware Experiments, IEEE Trans.
Automation Science and Engineering, vol.17, no.2,
pp.1017-1029, April 2020.
5. M.Matsunaga, M.Nakamoto and T.Yamamoto: A soundbased measurement of sway angle for anti-sway control ofover head crane, Journal of Robotics, Networking andArtificial Life, Vol.4, No.4, pp.322-355, March 2018.

Fig. 3 Frequency spectra of observed signals

estimated sway angle from the sound (proposed method),
and the red line is the sway angle from the image
processing.
Fig. 5 shows that the sway angle of the proposed method
agrees well with the sway angle of the image processing,and it can be confirmed that the quality point tracking canbe achieved even under the influence of disturbances andnoises on the sound source from a blower. Comparing
Figs. 4 and 5, we can see that the result in Fig. 5 has asmaller error. As the reason for this, the frequency bandsof the blower noise do not overlap with the frequency
bands of the sound source. That is, the deflection angle
can be accurately estimated, which confirms theeffectiveness of the proposed method.
4. Conclusion
We considered a soft sensor to measure the sway angle
for the anti-sway control of an overhead crane. We
proposed an approach to estimate the angle based on
acoustic signals to achieve contactless sensing of theangle. The method uses the phase difference of acoustic
signals measured by two microphones. We presented analgorithm to obtain the angle from the phase difference
based on Newton’s method. During the experiment, we
compared the proposed method with measurements takenfrom photographs. As a result, it was confirmed that the
proposed method can dynamically follow the sway angleof the payload . Furthermore, we confirmed that the
proposed method can correctly track the sway angle.

Fig. 4 Results of estimating 𝜃𝜃 for 𝑓𝑓𝑘𝑘 = 5038.77 [Hz]5. References
1. C.-Y. Chang and H. W. Lie: Real-Time Visual Tracking
and Measurement to Control Fast Dynamics of Overhead
Cranes, IEEE Trans. Ind. Electron., vol.59, no.3, pp.16401649, March 2012.
2. B. Lu, Y. Fang, N. Sun and X. Wang: Antiswing Controlof Offshore Boom Cranes With Ship Roll Disturbances,
IEEE Trans. Control Systems Technology, vol.26, no.2,
pp.740-747, Mar. 2018.

Fig. 5 Result of estimating 𝜃𝜃 for 𝑓𝑓𝑘𝑘 = 8010.35 [Hz]© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

132

*

Study on an Optimal Design Method for Control Systems based on Bayesian Optimization
Koichi Hirota †Graduate School of Advanced Science and Engineering, Hiroshima University, 1-4-1, Kagamiyama
Higashi-Hiroshima city, Hiroshima, Japan
Shin Wakitani,

Graduate School of Advanced Science and Engineering, Hiroshima University, 1-4-1, Kagamiyama
Higashi-Hiroshima city, Hiroshima, 739-8527 Japan
E-mail: wakitani@hiroshima-u.ac.jp
Toru Yamamoto,

Graduate School of Advanced Science and Engineering,Hiroshima University, 1-4-1, Kagamiyama
Higashi-Hiroshima city, Hiroshima, 739-8527 Japan
E-mail: yama@hiroshima-u.ac.jp

Abstract
When designing a complex product system, must be considered the optimal function distribution between thecontroller and the plant. This makes it possible to design a plant that is easy to control for fixed PID control, and theperformance of the entire system can achieve the desired performance. Here, the evaluation function of the system iscalculated from the output error and the input difference value. Then, the optimum design parameters are calculated
by using Bayesian optimization.
Keywords: Optimal Allocation of Function, Model-Based Development, Bayesian Optimization, Evaluation
Function,.

user can evaluate the subsystem’s performance and easily
improve their design without any prototyping.
Meanwhile, the actuators of many subsystems are
becoming electrified. Therefore, a feedback control
system including controllers represented by by-wire
technology is used in various situations. However, inmany situations of control system design, plant design iscompleted in advance. In the conditions of this plantcharacteristic, the structure and parameters of thecontroller are often adjusted to satisfy the desired control
performance. In particular, when the control target
includes high-order components, strong non-linearity,and large dead time, it is difficult to achieve high control
performance with a low-order controller such as a PID

1. Introduction
Most product systems consist of a combination ofmultiple subsystems. Appropriate setting and design ofoperating goals (functional goals) of these subsystems
are important to achieve and maintain the target
performance of the system. In addition, the performancerequired by users for products is becoming more diverseand complex year by year. It is necessary to efficiently
design a combination of subsystems to satisfy theserequired performances in a short period of time and at alow cost. In recent years, model-based development(MBD) has been attracting attention in the industrial
world, and its introduction is being active. In the MBD
process, entire subsystems are modeled as simulation
models and are connected to each other on the simulation
environment (such as MATLAB/Simulink). Moreover, a© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

133

Koichi Hirota, Shin Wakitani, Toru Yamamoto

However, when designing a control system, there are
input restrictions and non-linearities in the designed plant,
and these characteristics appear as constraints when
designing the controller. Therefore, it is often the casethat the desired control performance cannot be obtainedwith low-order controllers such as PID controllers thatare usually introduced in systems. However, if the spreadof MBD makes it possible to allocate the optimumfunctions of the plant and controller in advance, theperformance of the entire system will be closer to thedesired performance by designing a plant that is easy tocontrol for fixed PID controller. Therefore, this study,
considers the optimum allocation method of plant andcontroller functions for the system function in the designphase. In the subsystem, to reach a given functional goalis the most important. Where, it is assumed that thefunctional goal can be described as the dynamic
input/output relationship, that is the reference model1.
The error between the output of the subsystemcorresponding to a certain input scenario and the outputof the reference model is evaluated as follows.
𝑁𝑁21
�𝑦𝑦𝑟𝑟 (𝑘𝑘) − 𝑦𝑦(𝑘𝑘)�
(1)�� .
𝐼𝐼𝑒𝑒 = � �1 − exp �−
2𝜎𝜎𝑒𝑒2𝑁𝑁

Fig. 1. V-shaped development process.

controller. On the other hand, if a high-order controller
is introduced, the adjustment of control parameters
becomes complicated, and the uncertainty of the system
may impair the stability of the control loop. Therefore, insubsystem design, a plant and a controller are required to
be optimized at the same time for the functional goals
determined upstream design process. Based on the
background, this study considers the simultaneous
optimum design of the controller and the plant.
2. Simultaneous Optimal Design of Controller
and Plant in MBD
2.1. V-shaped development process

𝑘𝑘=1

However, 𝑁𝑁 represents the number of data acquired in
the simulation evaluation interval. In Eq. (1), theGaussian kernel is used for the output error. As a result,the value of Eq. (1) is normalized between 0 to 1 and thepermissible ratio to the error can be adjusted by 𝜎𝜎𝑒𝑒2 .
Moreover, in the controller design, it is desirable toimpose some restrictions on the controller output fromthe viewpoint of actuator protection and noise robustness.
Here, the following evaluation function is designed forthe input difference value Δ𝑢𝑢(𝑡𝑡) = 𝑢𝑢(𝑡𝑡) − 𝑢𝑢(𝑡𝑡 − 1).

Figure 1 shows the product development process, Vshaped development process, in MBD. The V-shaped
development process (V-model) is divided into themodel-in-the-loop simulation (MILS) which is a designarea using a model and the hardware-in-the-loop
simulation (HILS) which is a verification area that uses
hardware and a model simultaneously. Since thisresearch is mainly related to the design method, only
MILS is be explained. Firstly, the necessary functions fora designed product system and specifications will be
examined based on the results of market research. Next,
the product system is decomposed into several
subsystems, and the functions that these subsystems
should satisfy are examined using 1D simulation
technology. Furthermore, the parts (detailed design), thatcompose a subsystem is designed so that it operates in thesame way as the functional model determined in the
upstream design process. In this process, a detailed
design including the shape of parts is performed using 3D
simulation technology.

𝐼𝐼𝑒𝑒 =

𝑁𝑁(Δ𝑢𝑢(𝑘𝑘))2
1
� �1 − exp �−
�� .
2
𝑁𝑁
2𝜎𝜎Δ𝑢𝑢𝑘𝑘=1(2)

By using the Gaussian kernel as in Eq. (2), the evaluation
value for the input difference value is normalized, and acertain permissible ratio is given as 𝜎𝜎Δu . For Eqs. (1) and(2), the evaluation function for optimizing the entire
subsystem is defined by the following equation.
(3)
𝐼𝐼 = 𝜆𝜆𝐼𝐼𝑒𝑒 + (1 − 𝜆𝜆)𝐼𝐼Δ𝑢𝑢 .
However, 𝜆𝜆(0 ≤ 𝜆𝜆 ≤ 1) is an adjustment parameter.
Based on the above equation, if the design parameter
vector composed of control parameters and plantparameters is 𝜽𝜽 = (𝜃𝜃1 , 𝜃𝜃2 , ⋯ , 𝜃𝜃𝑛𝑛 , )𝑇𝑇 , the optimization
problem is formulated by the following equation.
(4)𝜽𝜽∗ = arg min �𝐼𝐼(𝜽𝜽)� .

2.2. Optimal allocation of plant and controller
functions𝛉𝛉

The next chapter explains the design parameter
optimization method based on Bayesian optimization.

When designing a feedback system composed of a plant
and a controller, the controller is usually designed fromthe state in which the plant has already been designed.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

134

Study on an Optimal

The function that finds the value of the 𝑲𝑲 element 𝐾𝐾𝑖𝑖𝑖𝑖 iscalled a kernel function. Kernel functions represent thesimilarity of inputs. There are several types, and theGaussian kernel is widely used. The Gaussian kernel isshown in Eq. (13).

3. Bayesian Optimization
Bayesian optimization2,3 can be divided into theregression process based on Gaussian process regression
using obtained input/output data and a process fordetermining the next search point by calculating an
acquisition function.

2

�𝒙𝒙𝑖𝑖 − 𝒙𝒙𝑗𝑗 �𝑘𝑘�𝒙𝒙𝑖𝑖 , 𝒙𝒙𝑗𝑗 � = 𝜃𝜃1 exp �−�.
𝜃𝜃2

3.1. Gaussian process regression
In the Gaussian process regression, the mean andvariance of the system output are estimated. First, thiscalculation method is explained. Let the input 𝑥𝑥 and theoutput 𝑦𝑦 are expressed by Eqs. (5) and (6).
(5)
𝒙𝒙 = (𝜽𝜽1 , 𝜽𝜽2 , ⋯ , 𝜽𝜽𝑁𝑁 )𝑇𝑇 .
(6)
𝒚𝒚 = (𝐼𝐼1 , 𝐼𝐼2 , ⋯ , 𝐼𝐼𝑁𝑁 )𝑇𝑇 .
Where, 𝑁𝑁 is the number of data.
The feature vector of 𝒙𝒙 is given as Eq. (7).
(7)
𝝓𝝓 = (𝜙𝜙0 (𝒙𝒙), 𝜙𝜙1 (𝒙𝒙), ⋯ , 𝜙𝜙𝐻𝐻 (𝒙𝒙))𝑇𝑇 .
If the weights vector is described as 𝒘𝒘 =(𝑤𝑤0 , 𝑤𝑤1 , ⋯ , 𝑤𝑤𝐻𝐻 )𝑇𝑇 , the linear regression model becomes
Eq. (8).
𝑤𝑤0
𝑦𝑦�1
𝜙𝜙0 (𝒙𝒙1 ) 𝜙𝜙1 (𝒙𝒙1 ) ⋯ 𝜙𝜙𝐻𝐻 (𝒙𝒙1 )𝑤𝑤1𝑦𝑦�2
𝜙𝜙0 (𝒙𝒙2 ) 𝜙𝜙1 (𝒙𝒙2 ) ⋯ 𝜙𝜙𝐻𝐻 (𝒙𝒙2 )� �=�� � ⋮ � . (8)
⋮
⋮
⋮
⋱
⋮
𝑤𝑤𝐻𝐻
𝑦𝑦�𝑁𝑁
𝜙𝜙0 (𝒙𝒙𝑁𝑁 ) 𝜙𝜙1 (𝒙𝒙𝑁𝑁 ) ⋯ 𝜙𝜙𝐻𝐻 (𝒙𝒙𝑁𝑁 )(9)� = Φ𝒘𝒘 .
𝒚𝒚�However, 𝒚𝒚 is the predicted value of the output y and isshown by Eq. (10).
� = (𝑦𝑦�1 , 𝑦𝑦�2 , ⋯ , 𝑦𝑦�𝑁𝑁 )𝑇𝑇 .
𝒚𝒚

(10)

In addition, Φ is called a design matrix and becomes Eq.
(11).
𝜙𝜙0 (𝒙𝒙1 ) 𝜙𝜙1 (𝒙𝒙1 )
𝜙𝜙0 (𝒙𝒙2 ) 𝜙𝜙1 (𝒙𝒙2 )Φ=�
⋮⋮𝜙𝜙0 (𝒙𝒙𝑁𝑁 ) 𝜙𝜙1 (𝒙𝒙𝑁𝑁 )⋯ 𝜙𝜙𝐻𝐻 (𝒙𝒙1 )⋯ 𝜙𝜙𝐻𝐻 (𝒙𝒙2 )�.
⋱⋮⋯ 𝜙𝜙𝐻𝐻 (𝒙𝒙𝑁𝑁 )(16)‖∙‖ indicates the Euclidean norm of the vector. In
addition, 𝜃𝜃1 and 𝜃𝜃2 are parameters that determine theproperties of the kernel function. Here, consider finding
the prediction point 𝑦𝑦 ∗ for an arbitrary vector 𝑥𝑥 ∗ . First,
using the output data vector in Eq. (6) and the predicted
output 𝑦𝑦 ∗ , a new vector 𝒚𝒚𝑛𝑛𝑛𝑛𝑛𝑛 is defined as follows.
(17)
𝒚𝒚𝑛𝑛𝑛𝑛𝑛𝑛 = (𝒚𝒚𝑇𝑇 , 𝑦𝑦 ∗ )𝑇𝑇 .
If the covariance matrix 𝑲𝑲𝑛𝑛𝑛𝑛𝑛𝑛 is calculated from 𝒚𝒚𝑛𝑛𝑛𝑛𝑛𝑛 , 𝒙𝒙,
and 𝒙𝒙∗ , then the following relationship is hold.
(18)
𝒚𝒚𝑛𝑛𝑛𝑛𝑛𝑛 ~𝑁𝑁(𝟎𝟎, 𝑲𝑲𝑛𝑛𝑛𝑛𝑛𝑛 ).
Here, the covariance matrix 𝑲𝑲𝑛𝑛𝑛𝑛𝑛𝑛 is expressed by thefollowing equation.
𝑲𝑲 𝒌𝒌∗(19)�.
𝑲𝑲𝑛𝑛𝑛𝑛𝑛𝑛 = � 𝑇𝑇
𝒌𝒌∗ 𝑘𝑘∗∗𝑇𝑇(20)
𝒌𝒌∗ = �𝑘𝑘(𝒙𝒙∗ , 𝒙𝒙1 ), 𝑘𝑘(𝒙𝒙∗ , 𝒙𝒙2 ), ⋯ , 𝑘𝑘(𝒙𝒙∗ , 𝒙𝒙𝑁𝑁 )� .
(21)
𝑘𝑘∗∗ = 𝑘𝑘(𝒙𝒙∗ , 𝒙𝒙∗ ).
From Eq. (18), the mean and variance of 𝒚𝒚∗ can becalculated by Eq. (23).
(22)
𝑦𝑦 ∗ ~𝑁𝑁(𝜇𝜇, 𝜎𝜎 2 ).
(23)
~𝑁𝑁(𝒌𝒌𝑇𝑇∗ 𝑲𝑲−1 𝒚𝒚, 𝑘𝑘∗∗ − 𝒌𝒌𝑇𝑇∗ 𝑲𝑲𝒌𝒌∗ ).
From the above results, the mean 𝜇𝜇 and variance 𝜎𝜎 2 ofthe unknown 𝒚𝒚∗ corresponding to arbitrary point 𝒙𝒙∗ canbe derived.
3.2. Acquisition function
The acquisition function is an equation calculated using
the mean 𝜇𝜇 and standard deviation 𝜎𝜎 obtained by
Gaussian process regression. This function is used todetermine the next point to search. There are several
types of acquisition functions, in this study, the following
Lower Confidence Bound (LCB) is introduced.

(11)

�, Eq. (9) can be obtained.
Where, by assuming 𝒚𝒚 = 𝒚𝒚
(12)𝒚𝒚 = Φ𝒘𝒘 .
Here, if a weights vector 𝒘𝒘 is generated from theGaussian distribution 𝒘𝒘~𝑁𝑁(𝟎𝟎, 𝜆𝜆2 𝑰𝑰) , 𝒚𝒚 also follows aGaussian distribution, then the mean E [y] and thecovariance matrix K becomes as follows.
(13)
𝐸𝐸[𝒚𝒚] = 𝐸𝐸[Φ𝒘𝒘] = Φ𝐸𝐸[𝒘𝒘] = 0 .
𝑲𝑲 = 𝐸𝐸[𝒚𝒚𝒚𝒚𝑇𝑇 ] − 𝐸𝐸[𝒚𝒚]𝐸𝐸[𝒚𝒚]= 𝐸𝐸[(Φ𝒘𝒘)(Φ𝒘𝒘)𝑇𝑇 ]= Φ𝐸𝐸[𝒘𝒘𝒘𝒘𝑇𝑇 ]Φ𝑇𝑇(14)
= 𝜆𝜆2 ΦΦ𝑇𝑇 .
Therefore, the distribution of y is a multivariate Gaussian
distribution shown in Eq. (12).
(15)
𝒚𝒚~𝑁𝑁(𝟎𝟎, 𝜆𝜆2 ΦΦ𝑇𝑇 ).

𝑎𝑎𝐿𝐿𝐿𝐿𝐿𝐿 = 𝜇𝜇 − �

log 𝑁𝑁
𝑁𝑁

𝜎𝜎 .

(24)

The LCB searches for a place with a large standard
deviation, that is, an unknown region, as the number ofdata decreases. Also, as the number of data increases, thesearch proceeds based on the data. From the above, it ispossible to search for the optimum value of the function
obtained by Bayesian optimization globally andefficiently.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

135

Koichi Hirota, Shin Wakitani, Toru Yamamoto

4. Numerical Example

Table 2. Evaluation function parametersparameters
𝜆𝜆
𝜎𝜎𝑒𝑒
𝜎𝜎Δu

The effectiveness of the proposed design method is
evaluated by numerical example. Here, consider acontrol system consisting of a controller, actuator, andplant as shown in Fig. 2.

value
0.5
0.1
0.01

Fig. 2 Feedback control system

Here, 𝑓𝑓𝑠𝑠 (𝑢𝑢) and 𝑊𝑊𝐴𝐴 are the characteristics of theactuator, and 𝑊𝑊𝑃𝑃 is the dynamic characteristics of theplants that make up the subsystem. 𝑓𝑓𝑠𝑠 (𝑢𝑢) is a functionthat expresses the input saturation of the actuator and isgiven by the following equation.
(𝑢𝑢1 (𝑡𝑡) < 𝑢𝑢𝑚𝑚𝑚𝑚𝑚𝑚 )𝑢𝑢𝑚𝑚𝑚𝑚𝑚𝑚 ,(𝑢𝑢𝑚𝑚𝑚𝑚𝑚𝑚 ≤ 𝑢𝑢1 (𝑡𝑡) < 𝑢𝑢𝑚𝑚𝑚𝑚𝑚𝑚 ) . (25)
𝑢𝑢2 (𝑡𝑡) = �𝑢𝑢1 (𝑡𝑡),𝑢𝑢𝑚𝑚𝑚𝑚𝑚𝑚 ,(𝑢𝑢1 (𝑡𝑡) ≥ 𝑢𝑢𝑚𝑚𝑚𝑚𝑚𝑚 )The transfer function of plant 1 (𝑊𝑊𝐴𝐴 ) is shown in thefollowing equation.
𝐾𝐾𝐴𝐴
(26)
.
𝑊𝑊𝐴𝐴 (𝑠𝑠) =1 + 𝑇𝑇𝐴𝐴 𝑠𝑠
Here, 𝑇𝑇𝐴𝐴 is the time constant of the actuator, and 𝐾𝐾𝐴𝐴 is the
maximum output of the actuator. The transfer function ofplant 2 (𝑊𝑊𝑃𝑃 ) is shown in the following equation.
𝐾𝐾𝜔𝜔2
(27)
.
𝑊𝑊𝑃𝑃 = 2
𝑠𝑠 + 2𝜁𝜁𝜁𝜁𝜁𝜁 + 𝜔𝜔 2
Where,
1

𝐾𝐾 = 𝑘𝑘 , 𝜁𝜁 =
𝑃𝑃

𝐷𝐷𝑃𝑃

2�𝑘𝑘𝑃𝑃 𝑀𝑀𝑃𝑃𝑘𝑘, 𝜔𝜔 = �𝑀𝑀𝑃𝑃 .
𝑃𝑃

Fig. 3. Control result using parameters (left-hand side: λ =0.5, right-hand side: λ = 1.0).

As a result of Bayesian optimization under the condition
λ = 0.5, the optimum values became 𝐾𝐾𝐴𝐴 = 61, 𝑘𝑘𝑃𝑃 = 50,
𝐷𝐷𝑃𝑃 = 0.1. At this time, the control result is shown in Fig.
3 (left-hand side). From the control result, it can be seenthat the output 𝑦𝑦 generally follows the reference output
𝑦𝑦𝑟𝑟 . If λ = 0.5 in the evaluation function, the optimum
valuesare 𝐾𝐾𝐴𝐴 = 102, 𝑘𝑘𝑃𝑃 = 80, 𝐷𝐷𝑃𝑃 = 35 when
Bayesian optimization is performed. At this time, theresult of the simulation is shown in Fig. 3 (right-handside). Compared with the result obtained by λ = 0.5 , itseems that the output error is smaller. However, since it
can be seen that the input fluctuates drastically, it canconclude that a system in which the fluctuation of the
input is suppressed can be designed by adding 𝐼𝐼Δ𝑢𝑢 to theevaluation function.
5. Conclusion(28)

This study proposes an optimal design method of asubsystem composed of a plant and a controller by using
Bayesian Optimization. This paper defines a criterion for
optimizing the plant parameters so that the plant isdesigned suitable for a controller. The effectiveness ofthe proposed design scheme is verified by a simulation
example.

However, it is assumed that the system parameters 𝑀𝑀𝑃𝑃 ,𝑘𝑘𝑃𝑃 , and 𝐷𝐷𝑝𝑝 are all positive real numbers. In thissimulation, 𝐾𝐾𝐴𝐴 , 𝑘𝑘𝑃𝑃 , and 𝐷𝐷𝑃𝑃 are the design parameters,
and the other parameters are the fixed values shown in
Table 1. In the controller design, the PID gains are
determined based on the pole assignment controller
design only using the model of plant 2. Here, thereference model expressing the desired function of thesubsystem is given by the following equation.
(29)
𝐺𝐺𝑚𝑚 = 1 3.

6. References1.

𝜎𝜎
�1+ 3 𝑠𝑠�The parameter 𝜎𝜎 = 0.5 for the rise time was set in thiscase. The evaluation function uses Eq. (3), and Table 2
shows the parameters used in the function design.

2.

Table 1. Plant parametersparameters𝑢𝑢𝑚𝑚𝑚𝑚𝑚𝑚
𝑢𝑢𝑚𝑚𝑚𝑚𝑚𝑚
𝑇𝑇𝐴𝐴
𝑀𝑀𝑃𝑃

value
0
1
0.01
10

3.

Takeshi Shigemasa, Yasuo Takagi, Yoshinori Ichikawa
and Toshiyuki Kitamura, A Practical Reference Model
for Control System Design, Transactions of the Society of
Instrument and Control Engineers, Vol. 19, Issue. 7, 1983,
pp.592-594.
Daichi Mochihashi and Ooba Shigeyuki, GaussianProcess and Machine Learning ~ Machine Learning
Professional Series, Kodansya, Tokyo,Japan, 2019, pp.
57-95.
C. M. Bishop, Pattern Recognition and Machine Learning,
Springer, New York, 2012

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

136

Unconstrained Measurement of Heart Rate Considering Harmonics of a
Respiratory Signal by Flexible Tactile Sensor Sheet
Kazuya Matsuo

Department of Mechanical and Control Engineering, Kyushu Institute ofTechnology, 1-1, Sensui, Tobata, Kitakyushu, Fukuoka, 804-8550, JAPANToshiharu Mukai

Meijo University, JAPANShijie Guo

Hebei University of Technology, CHINA
E-mail: matsuo@cntl.kyutech.ac.jp

Abstract
This study describes a heart rate measurement method by considering the harmonics of a
respiratory signal. Respiration and heart rates were measured by applying frequency analysis
to the time-series data of body pressure. The harmonics of a respiratory signal serve as noises
in heart rate measurement. Therefore, heart rate measurement was improved by eliminating
the effects of the harmonics. The results of experiments demonstrated that our proposed
method enhances the precision of heart rate measurement.
Keywords: Harmonics of a respiratory signal, Heart rate measurement, Unconstrained
measurement, Sleeping state.

1. Introduction
In Japan, the percentage of the elderly, aged 65 orolder, against the total population will increase to 29% in
2020, 32% in 2030, and surpass 36% in 2040 [1], [2].
The population of the elderly was estimated to be
2,924,000 in 2010 and is projected to be 3,619,000 in2020 [3]. It is also estimated that the number ofindividuals requiring care will increase. However, theincrease in the number of caregivers may not keep pacewith this rise. Therefore, Japan may face an acute lack of
caregivers in the immediate future, and the burden on
nursing staff is expected to increase. The need for the
introduction of a nursing-care apparatus has thus beenheightened, and the watching of the perception sickness

has been decided by the national government as one of thepriority fields.
If the condition of the patient could be assessed using
sensors, it could greatly contribute to the reduction of
burden on the nursing staff. Conditions of the nursed
person to be distinguished include whether they are on the
bed and whether their sleeping posture is supine or prone.
Addition- ally, measurements of the sleeping state, such as
the depth of sleep, would be useful to monitor the healthof the nursed person.
Polysomnography (PSG) [4] is a well-known methodof measuring the sleeping state. PSG can also diagnose
sleep apnea syndrome and periodic limb movement
disorder. To recognize the sleeping state, PSG requires
biophysiological information, such as brain activity© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

137

Kazuya Matsu, Toshiharu Mukai, Shijie Guo(electroencephalograms),
heart
activity(electrocardiograms), eye movements, muscle activity(electromyograms) of the chin, and respiratory air- flow.
The monitored body functions are used to score eachperiod of sleep according to one of the six sleep stages
based on the R&K manual [5]; these include awake,
REM sleep, and four stages of non-REM sleep. As a
medical specialist makes the final diagnosis, PSG iscapable of accurately determining the sleep stage.
However, attaching many sensors to the body to obtain
numerous biological signals is necessary. Thus, PSG
hinders natural sleeping, accompanied by a feeling ofrestrain. Further, there is a need for the judgment of aninspection technician to distinguish the sleeping statefrom the biological signal, and thus, automation of thistask is difficult. Therefore, a technique that canconveniently estimate the sleeping state is required.
The sleeping state can be estimated from biologicalinformation such as the respiration rate, heart rate,
body motion, and lying posture [6]–[13], which can bemeasured relatively easily. For example, there is a strong
correlation between heart rate and sleep state [6].
Many unconstrained and non- invasive sleep estimation
methods have been reported [7]-[11]. Watanabe et al.
developed a noncontact algorithm for the sleeping state
estimation using heartbeat and body- movement signals[7]. Kambayashi et al. estimated the sleeping state cycle
using body-movement density, measured from a passive
infraredtype
motionsensor [8]. Okada et al.
proposed a non-restrictive and non-invasive sleeping
state evaluation technique from body movements
obtained from video images [9]. Deguchi et al. evaluatedheart rate variability and respiratory variability [10].
Harper et al. evaluated the potential of the classificationof sleep states using cardiorespiratory measures [11].
This paper proposes a method to measure sleep posture[12], respiration, and heart rate [13] using a flexible
tactile sensor sheet.
Conventional sleep estimation methods are often
basedon the assumption that respiration and heart rates
can be accurately measured. However, it is not always
possible to accurately measure biological information.
In fact, in the method used in [13], the percentage ofsuccessful heart rate measurement in overnight
experiments was only 40%.
There are several possible reasons for the failure ofac- curate heart rate measurement. For example, if thepressure information corresponding to the type ofsleeping posture is used, and if the identification of thesleeping posture fails, accurate measurements of therespiration and heart rates also fail. Additionally, there isa problem that the harmonics of the respiratory signal
behave like noise signals in the heart rate measurement.
As the heartbeat signal is weaker than the respiratory
signal, the intensity of the frequency spectrum may besmaller than the harmonics of the respiratory signal. In
such a case, the heart rate measurement will fail. In thispaper, we propose two methods to improve themeasurement accuracy of the heart rate by eliminating
the effects of the harmonics of the respiratory signal.
2. Heart Rate Measurement Method Considering
Harmonics of a Respiratory Signal
2.1. Measurement method of respiratory and heart rates
from body pressure
A method for measuring respiration and heart rates
from body pressure information using a flexible planar
tactile sensor has been described (Fig. 1) [13]. The
body pressure of a person lying on a bed is measured
using a tactile sensor sheet. It is observed that the
respiration and heart rates remain nearly constantduring rest. The frequency bands of respiratory andheart rates do not overlap. Frequency analysis is applied
to the time-series data of body pressure, measured by
sensor cells in the appropriate position, such as on thechest. A fast Fourier transform (FFT [14], has been used
for frequency analysis. Finally, the respiratory and heart
rates were acquired by extracting the appropriate
frequency band. The frequency bands of the extracted
respiratory and heart rate data were determined bypreliminary experiments in the following manner: (a)The frequency band of© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

138

Unconstrained Measurement of Heart

Proposed Method 1: The frequencies of the second,
third, and fourth harmonics are calculated from theFig. 1. Measurement method of respiration and heart rates using aflexible planar tactile sensor.

Fig. 3. Measurement of heart rate by eliminating
respiratory harmonics.
frequency of respiration, obtained using theconventional method. The spectral intensity is then set
in the ranges of zero to 0.05 Hz around the harmonics.
The frequency of the heart rate is obtained by finding
the dominant frequency in this modified frequency
spectrum. The eliminated ranges are determined on
the basis of the results of the preliminary experiments.
Proposed Method 2: In the case that a harmonic of
the respiratory signal and the frequency of the heart
rate are close to each other, proposed method 1 also
eliminates the heart rate signal. Thus, some ratio of
the intensity of the fundamental respiratory signal issubtracted from those ranges instead of setting theintensity in the ranges to zero. Specifically, 20% of
the respiratory signal is subtracted from the range ofthe second harmonic and 10% of the respiratory
signal is subtracted from the ranges of the third andthe fourth harmonics. These ratios (20%, 10%) aredetermined using preliminary experiments.
These two methods are compared with theconventional method to investigate the validity ofour proposed methods.

Fig. 2. Appropriate positions for measuring heart rate. The oval
shapes show the appropriate positions [12].

respiration is 0.1 to 0.4 Hz (6 to 24 times/min).
(b) The frequency band of the heart rate is 0.8 to
1.6 Hz (48 to 96 times/min). When obtaining thefrequencies of the respiration and heart rates, their
dominant frequencies can be found.
Measuring body pressure at appropriate positions
such as the chest to estimate the heart rate isnecessary. Fig. 2 shows the appropriate positions forheart rate measurement determined by Mukai [12].
2.2. Heart rate measurement by eliminating
respiratory harmonics
In the conventional method [13], the harmonics ofthe respiratory signal act as noise signals in themeasurement of the heart rate. When the heartbeat
signal is weak, the intensity of the frequency spectrum
may be smaller than the harmonics of the respiratory
signal. Hence, the conventional method sometimes
fails to measure the heart rate. In this paper, we
propose two methods to improve the measurement
accuracy of the heart rate by eliminating theharmonics of the respiratory signal. A sample
measurement response of the proposed methods isshown in Fig. 3.
The specifications of the proposed methods toeliminate the harmonics of the respiration are
described below.

3. Flexible Tactile Sensor Sheet
To conduct experiments according to the proposed
methods, a sensor that can measure the body pressureof a person lying on a bed at a high resolution isrequired. To measure the small pressure fluctuation of
the heartbeat signal, a high- pressure resolution sensor
is required. In addition, to identify the type of lying
posture and to measure the body pressure
atappropriate positions, a high spatial resolution sensor
is required. A Smart Rubber (SR) sensor [15](Sumitomo Riko, Ltd.), which is a flexible andstretchable planar tactile sensor, made of rubber-based© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

139

Kazuya Matsu, Toshiharu Mukai, Shijie Guomaterials, including the wiring, was used. Hence, it
was sufficiently soft and elastic and was able tomeasure body pressure without causing the subject toFig. 4 shows an outline of the structure of the SR
sensor comprising two electrode layers made of
conductive rubber sheets and a thin dielectric layer
made of urethane. The electrode layers have manylong narrow belt-like printed electrodes. The
dielectric layer is sandwiched between the twoelectrode layers. The belt-like electrodes in the twoelectrode layers are orthogonal to each other, and anelectric capacitance sensor cell is formed at every
intersection of the electrodes. When one layer has melectrodes, and the other layer has n electrodes, m x n
sensor cells are formed as a whole. When the widthsof the electrodes were narrowed, high spatial
resolution measurement was realized. In contrast,
when the widths were widened, a high-pressure
resolution measurement was realized.
When a vertical force is exerted on the SR sensor,
the dielectric layer is deformed to be thinner. The
electrostatic capacitance of the sensor cell increases
because the distance between the two electrode layers
becomes shorter. The pressure distribution on the SR
sensor was calculated by measuring the electrostatic
capacitances of all cells. Many pressure sensors
having the same mechanism have been proposed. For
instance, XSENSOR is commercially available [16].
However, this sensor is expensive and is lessstretchable owing to its metal electrodes. It is, thus,
not suitable for use in medical and nursing care fields.
However, the SR sensor is suitable for these fields.
As this sensor is made of rubber printing, it isaffordable and stretchable.

feel any discomfort. The body pressure could bemeasured non-invasively, and without restraint.
structure.
Table 1 summarizes the specifications of the SR
sensor used in this study. The size of a sensor cell
enables sufficient spatial resolution to distinguishsleep postures. In contrast, when the sensor cell issmaller, the pressure resolution deteriorates. Thissensor can measure respiration without any problems
because of large pressure fluctuations. However, theheart rate cannot be measured from the noise because
the pressure fluctuation is small. Therefore, tomeasure the heart rate, increasing the pressure
resolution by suppressing the noise is critical. The
signal-to-noise ratio can be improved by sampling
many times, compared to the signal period at highfrequency and by obtaining the average of thesesignals. This technique is called oversampling. In thisstudy, the noise is suppressed by taking an average of
46 samples that were collected.
Table 1. Specifications of the SR
Sensor800
LengthWidth
Number of sensor
32
25
cells
Size of the whole882
686
sensor[mm][mm]
Size of a sensor cell
14
14
[mm]
[mm]
Size of a gap
14
14
between cells
[mm]
[mm]
Thickness of the
3.5 [mm]sensorMeasuring20
frequency[Hz]
It is possible to measure the heart rate byoversampling. However, it takes a considerable
amount of time to oversample all the cells of thesensor; thus, the sampling rate required formeasuring the respiration and heart rates cannot be
obtained. Therefore, the high-pressure resolution
measurement using oversampling should be limited
to certain cells. The high-pressure-resolution cells
are referred to as the precision cells. In this study, thenumber of precision cells is four, owing to thelimitation of the computational power of the
microcomputer mounted on the sensor. As theFig. 4. Outline of the SR sensor© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

140

Unconstrained Measurement of Heartdeviations of these values. The average errors of all
subjects decreased in the following order: proposed
method 2 (0.144 Hz), proposed method 1 (0.146
Hz), and the conventional method (0.149 Hz). The
calculation of the paired t-test shows that thedifference between the proposed method 2 and theconventional method is statistically significant at thesignificance level, p = 0.05 (significance probability:p = 0.011, t-ratio: t = 2.29). In contrast, thecalculation of the paired t-test shows that thedifference between the proposed method 1 andthe conventional method is not statisticallysignificant at the significance level p = 0.05
(significance probability: p = 0.107, t-ratio: t =1.25). The proposed method 2 significantly
improves the measurement of the heart rate.
However, the errors of subject-B and subject-G
reduce in the following order: the conventional
method, proposed method 2, and proposed method 1,
potentially because the harmonics of the respiratory
signal and the frequency of the heart rate of these two
subjects may be close to each other. In this case, theproposed method 1 eliminates not only the harmonics
of the respiratory signal, but also the heart rate signal.
Furthermore, even with the proposed method 2, thepeak of the heart rate signal is lower, thus increasing
the possibility of measurement failure.
Table 2 summarizes the number of data points
where the harmonics of the respiratory signal andfrequency of the heart rate are close to each other foreach subject. Data, where the absolute value of thefrequency difference is 0.05 Hz or less, are considered
to be close. Of the 120 data points per subject, thenumber of data points where the second, third, andfourth harmonics of the respiratory signal andfrequency of the heart rate are close to each other isshown in the columns titled 2nd, 3rd, and 4th inTable 2. Subject-B and subject-G have more data
where the harmonics of the respiratory signal andfrequency of the heart rate are closer to each otherthan the other subjects. This is believed to be thereason for the measurement error of heart rate, which
was greater with the proposed method than with theconventional method for these two subjects.

numberof precision cells is limited, they have tobe appropriately positioned to measure the heartrate.
4. Experiments
4.1. Experimental setup
To evaluate the effectiveness of the proposed
methods, tests were conducted on subjects
positioned on the SR sensor and laid on a bed. Eightmales aged 21 to 35 years, were selected for thisstudy. This study has been approved by the Research
Ethics Committee of the Kyushu Institute ofTechnology, and an informed consent has beenacquired from all the subjects.
The methods were applied to three types of
sleeping postures: supine position, prone position,and lateral position. For every posture of eachsubject, ten measurements were taken. In eachmeasurement, four precision cells were positioned atsuitable locations. One hundred and twenty data
points per subject were collected. For the FFT,
1,024-point data during 51.2 s were used (51.2 s x
20 Hz = 1,024).
To calculate errors in the frequencies of the heartrate, the true values of the heart rate were collectedusing a biological information sensor (biosignalsplux)[17]. The subjects wore the sensor to enablemeasurements of the true values of the biologicalinformation. The sensor measures the subject’s heart
rate in an electrocardiogram. This sensor alsomeasures respiration by measuring the movement of
the chest.
4.2. Measurement results of heart r ate
Fig. 5 shows the measurement frequency errors ofheart rate. The horizontal axis shows the label
corresponding to every subject, and the right end
indicates the average of all the subjects. The two
proposed methods (described in Section II-B) andthe conventional method [13] were applied. The
vertical axis shows the averages of the absolutevalues of the errors evaluated by the three methods
in the subjects. The error bars show the standard© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

−

−

141

Kazuya Matsu, Toshiharu Mukai, Shijie Guo

6. References

Table 2. Date points where the absolute value of thedifference between the harmonics of the respiratory
signal and the frequency of the heart rate is 0.05 Hz
or less.
Subject
s
(A to
H)
A
B
C
D
E
F
G
H

Harmonics
3r
4t
2nd
d
h
0
0
0
0
4
40
0
0
8
0
0
0
0
0
8
0
0
0
0
12
28
0
0
8

[1] Statistics Bureau, Ministry of Internal Affairs andCommunications, Japan. (2011, Oct.). The 2010
Population Census of Japan. Public announcement
on 26 Oct. 2011. [Online]. Available: <http://www.
stat.go.jp/english/data/kokusei/index.htm>,(accessed on 3rd Dec. 2020).
[2] National Institute of Population and Social Security
Research. (2012, Mar.). Population Projections forJapan (January 2012) - 2011 to 2060-. Publicannouncement on 30 Mar. 2012. [Online].
Available:<http://www.ipss.go.jp/site-ad/index
english/ esuikei/gh2401e.asp>, (accessed on 3rd Dec.
2020).

Numberofdata
points
0
44
8
0
8
0
40
8

5. Conclusion
Two methods were developed to improve themeasurement of heart rate by eliminating the effectsof the harmonics of the respiratory signal. The
experiments showed that the error in heart ratemeasurement could be reduced by using
theproposed method 2 than the conventional method[13]. Our proposed method has enhanced theprecision of the heart rate measurement. This willlead to strengthening of measurements of thesleeping state.
For the subjects in whom harmonics of therespiratory signal and frequency of the heart rate
were close to each other, the accuracy ofmeasurement of heart rate decreased when the
proposed methods were used. The proposed method
cannot enhance the precision of the heart ratemeasurement for such subjects. Identifying such
subjects is a future challenge. Additionally, if aparameter for the percentage of the respiratory signalthat should be subtracted from the harmonic band in
the proposed method 2 can be determined
appropriately, this would enable the heart rate to bemeasured at the same level as the conventional
method, even for such subjects. In future research, amethod to determine the optimal parameter for eachsubject will be developed. The method will be able
to avoid the deterioration of the accuracy of heart
rate measurement.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

142

Unconstrained Measurement of Heart

Fig. 7. Measurement frequency errors of heart rate of the two proposed methods and the conventional method. The proposed method 1 eliminates the
harmonics of a respiratory signal. The proposed method 2 reduces the effects of the harmonics.

Lorenz Plot,” Journal of Sleep and Biological Rhythms, vol.
9, no. 4, p. 366, 2011.
[11] R. M. Harper, V. L. Schechtman, and K. A. Kluge,“Machine Classification of Infant Sleep State Using
Cardiorespiratory Measures,” Electroencephalography andClinical Neurophysiology, vol. 67, no. 4, pp. 379–387, 1987.
[12] T. Mukai, K. Matsuo, Y. Kato, A. Shimizu, and S. Guo,
“Lying Posture Detection for Unconstrained Measurementof Respiration and Heartbeat of a Person on a Bed,” in Proc.
11th IASTED Int’l Conf. on Biomedical Engineering, pp.
46–53, DOI: 10.2316/P.2014.818-014, 2014.
[13] K. Matsuo, T. Mukai, Y. Kato, A. Shimizu, and S. Guo,
“Measurement of Respiration and Heartbeat Using a
Flexible Tactile Sensor Sheet on a Bed,” in Proc. 11th
IASTED Int’l Conf. on Biomedical Engineering, pp. 59–64,
DOI: 10.2316/P.2014.818-031, 2014.
[14] J. W. Cooley and J. W. Tukey, “An Algorithm for the
Machine Calculation of Complex Fourier Series,”
Mathematics of Computation, vol. 19, no. 90, pp. 297–301,
1965.
[15] S. Guo, Y. Kato, H. Ito, and T. Mukai, “Development of
Rubber-Based Flexible Sensor Sheet for Care-Related
Apparatus,” Sumitomo Electric Industries Technical Review,
no. 75, pp. 125–131, 2012.
[16] R. Cork, “XSENSOR Technology: a Pressure Imaging
Overview,” Sensor Review, vol. 27, no. 1, pp. 24–28, 2007.
[17] PLUX Wireless Biosignals S. A., Biosignalsplux. [Online].
Avail- able: <https://plux.info/12-biosignalsplux>, (accessedon 3rd Dec. 2020).

[3] Ministry of Health, Labour and Welfare, Japan. (2011, June).
General Welfare and Labour. Public announcement on 2 June
[Online].
Available:
2011.
other/social<http://www.mhlw.go.jp/english/policy/
security/index.html>, (accessed on 3rd Dec. 2020).
[4] C. A. Kushida, M. R. Littner, T. Morgenthaler, C. A. Alessi, D.
Bailey, J. Coleman Jr, L. Friedman, M. Hirshkowitz, S. Kapen,
M. Kramer, T. Lee-Chiong, D. L. Loube, J. Owens, J. P. Pancer,
and M. Wise, “Practice parameters for the indications forpolysomnography and related procedures: An update for 2005,”
Journal of Sleep, vol. 28, no. 4, pp. 499–521, 2005.
[5] A. Rechtschaffen and A. Kales, A Manual of Standardized
Terminology, Techniques and Scoring System for Sleep Stagesof Human Subjects. Bethesda, MD. : U.S. Dept. of Health,
Education, and Welfare, Public Health Services-National
Institutes of Health, National Institute of Neurological Diseases
and Blindness, Neurological Information Network, 1968.
[6] T. Watanabe and K. Watanabe, “Estimation of the Sleep Stagesby the Non-restrictive Air Mattress Sensor: Relation between
the Change in the Heart Rate and Sleep Stages,” Trans. of theSociety of Instrument and Control Engineers, vol. 37, no. 9,
pp. 821–828, 2001.
[7] Y. Kurihara and K. Watanabe, “Sleep-Stage Decision
Algorithm by Using Heartbeat and Body-Movement Signals,”IEEE Trans. on Systems, Man, and Cybernetics, Part A, vol. 42,
no. 6, pp. 1450–1459, 2012.
[8] Y. Kambayashi and H. Hagiwara, “Estimating Sleep Cycle
Using Body Movement Density,” in Proc. 2012 5th Int’l Conf.
on Biomedical Engineering and Informatics, pp. 1029–1033,
2012.
[9] S. Okada, Y. Ohno, Goyahan, K. Kato-Nishimura, I. Mohri, and
M. Taniike, “Examination of Non-restrictive and Noninvasive Sleep Evaluation Technique for Children Using
Difference Images,” in Proc. 30th Annual Int’l Conf. of theIEEE Engineering in Medicine and Biology Society, pp.
2483–2487, 2008.
[10] A. Deguchi and H. Hagiwara, “Evaluation of Heart Rate
Variability and Respiratory Variability during Sleep Using a© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

143

Image Registration Method for Chest MDCT Images Based on 2-D Finite Element Method
Takuji Ogimoto, Tohru Kamiya

Department of Mechanical and Control Engineering, Kyushu Institute of Technology, 1-1 Sensui-cho, Tobata-ku,
Kitakyushu-shi, Fukuoka, 804-8550, Japan
Takatoshi Aoki

University of Occupational and Environmental Health, 1-1, Iseigaoka, Yahatanishi, Kitakyusyu-shi 807-8555, Japan

Abstract
Multi detector-row computed tomography (MDCT) device has been used for early detection of lung cancer.
However, there is concern that an increase in the burden on doctors will be caused by improvements in CT
performance. Therefore, developing a computer aided diagnosis (CAD) reduces the burden on doctors. A temporalsubtraction (TS) technique is one of the CAD systems and it is the subtraction operation of the current image and theprevious one of the same patients that emphasizes temporal changes. In the TS image, when the position of the current
image and the previous image are misaligned, subtraction artifacts are remained. In this paper, we propose aregistration method based on the 2-D finite element method (FEM). In our proposed method, to improve the highcomputed cost that is the biggest problem of the FEM, we introduce the 2D FEM registration. We applied this method
to 31 series MDCT image sets which was obtained previous and current from the same subjects and evaluated. As aresult, full width at half maximum (FWHM) of 28.0, artifact to lung volume ratio of 5.77% and computational timeof 140 sec were obtained.
Keywords: Computer Aided Diagnosis System, Temporal Subtraction Technique, Non-Rigid Image Registration,
Finite Element Method.

to make a diagnosis and eliminate the variability indiagnosis by doctors. In this paper, we propose aregistration method for the generation of temporalsubtraction (TS) images from CT images of the chest,
which is a CAD system. A TS technique is thesubtraction operation of the current image and theprevious one of the same patients and emphasizes
temporal changes 2. The TS technique can emphasizeinterval changes such as new lesion shadows and shapechanges in the current image by removing the normal
structures (blood vessels, bones, muscles etc.) that appear
in both images that do not change much over time. Sucha technique has been shown to be effective in detecting
nodular shadows by physicians in a clinical evaluation
experiment 3.

1. Introduction
In recent years, multi detector-row computed
tomography (MDCT) of the chest device has been used
for the early detection of lung cancer. Chest MDCT
images can detect lung cancer at an earlier and morecurable stage than routine chest radiography. However,while MDCT images can be obtained in a short period oftime and have a high resolution, there are problems thatthe frequency of CT scans and the number of images
increase. To solve these problems, computer aided
diagnosis (CAD) has been attracting attention. The
results of the analysis by the CAD system are provided
to the radiologists as a "second opinion" 1. By using the
CAD system, it is expected to shorten the time required© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

144

Takuji Ogimoto, Tohru Kamiya, Takatoshi Aoki

In general, subtraction artifact in TS images is a normal
structure that remains in the temporal subtraction image
without being erased by the differencing process. Also,
artifacts can be caused by misalignment between current
and past images. Misalignment is caused by various
factors such as the photographic equipment and thephotographic environment. Therefore, we can obtain asubtraction image in which the normal structure isremoved and only the abnormal structure is emphasized
by deforming the previous image to the current image
and computed the difference between the two images.
Such an image registration technique is necessary to
generate the TS images.
In this paper, we propose a registration method for
reducing the subtraction artifacts on TS image by using
the finite element method (FEM). In a previous study 4,
to reduce the computational cost (processing time andmemory) required for the FEM registration use othersimple alignment methods 5 for generating a temporal
difference image. After that a local matching technique
was performed on to the high-density regions (i.e.,
abnormal regions or subtraction artifacts) on the TS
images. In this paper, to further reduce the computational
cost compared to the previous studies, we applied 2-D
FEM to each slice instead of 3-D FEM to reduce thecomputational cost. This can be expected to reduce theprocessing time and to generate highly accurate TS
images. In our experiments, we apply the proposed
method to chest MDCT images and verify its
effectiveness in terms of both positioning accuracy andcalculation time.

2.1. Initial temporal subtraction image
Itai et al. 5 proposed a method based on voxel matching
technique for creating a temporal subtraction
image. In this study, to reduce the computational time,
we made an initial subtraction image based on voxel
matching technique. In general, when we implement the
FEM technique on the image for registration, it required
computational time. To avoid this problem, we
performed the FEM technique on the initial subtraction
areas.
2.2. Creating the mesh of triangle elements
We set the region of interest (ROI) around the detected
regions on initial temporal subtraction image that meetsthe conditions. The conditions are the subtraction value
is 200 over and volume is over 50mm3. We installed nodepoint at equal intervals and created tetrahedron elements
by using Delaunay triangulation 6.

2. Proposed Method

2.3. Creating the final subtraction image

Our proposed method consists of three major phases:
create an initial temporal subtraction image, mesh oftriangle elements, 2-D FEM registration for final
subtraction image. Flow is shown in Fig 1. In the firstphase, extract the regions to be aligned using the
temporal subtraction technique. In the second phase,create the mesh for FEM for the extracted regions. In the
final phase, 2-D FEM registration is performed.

Ferrant et al. 7 proposed a registration method using
FEM. They expressed potential energy E as:
𝐸𝐸 = � 𝜎𝜎 𝑇𝑇 ∙ 𝜖𝜖𝜖𝜖𝜖𝜖𝛺𝛺− ∫𝛺𝛺 [𝐼𝐼1 �𝑥𝑥 + 𝛿𝛿(𝑥𝑥)� − 𝐼𝐼2 (𝑥𝑥)]2 𝑑𝑑𝑑𝑑 .

(1)
where 𝛺𝛺 is the body on which one is working, 𝜎𝜎 is thestress vector, 𝜖𝜖 is the strain vector, 𝐼𝐼1 is deformation
image, 𝛿𝛿 is displacement vector, and 𝐼𝐼2 is current image,respectively.
From principle of minimum potential energy, they sought𝜕𝜕𝜕𝜕minimum of this function by solving = 0. Eq. (1) then𝜕𝜕𝜕𝜕
becomes:© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

145

Image Registration Method for

4. Discussion and conclusion(𝑲𝑲 + 𝑮𝑮)𝜹𝜹 = 𝑷𝑷.
(2)Matrix 𝑲𝑲 , 𝑮𝑮 , and 𝑷𝑷 are defined as follows:
𝑲𝑲 = ∫𝛺𝛺 𝑩𝑩𝑇𝑇 𝑫𝑫𝑇𝑇 𝑩𝑩 𝑑𝑑𝑑𝑑 , 𝐺𝐺 = ∫𝛺𝛺 𝑵𝑵𝑇𝑇 ∇𝑰𝑰1𝑇𝑇 ∇𝑰𝑰𝟏𝟏 𝑵𝑵𝑑𝑑𝑑𝑑 , and
𝑷𝑷 = ∫𝛺𝛺 (𝐼𝐼1 − 𝐼𝐼2 )∇𝑰𝑰𝟏𝟏 𝑵𝑵𝑑𝑑𝑑𝑑 ; where 𝑩𝑩 is strain
displacement function, 𝑫𝑫 is D matrix, 𝑁𝑁 is shapefunction. By solving Eq. (2), the displacements of allnodes are determined and the displacements inside theelements are determined. Then we obtained warped
images. We performed FEM registration on a slice byslice to reduce the computational cost.

In this study, we proposed a 2-D FEM registration. The
result of the 2-D FEM registration method was obtainedwith FWHM: 28.0, artifact to lung volume ratio: 5.77%,
cost time: 140sec. Table 2 shows the result of theprevious method 4 and the proposed method. From theTable 2, the value of FWHM and artifact to lung volume
ratio for the previous method and the proposed method is
almost the same, and the proposed method reducescomputational time. Therefore, the proposed method
improved cost time without decreasing registration
accuracy.
The previous method performed 3D-FEM registrationand has high registration accuracy. So, we need todevelop a more accurate registration method.
In this paper, we developed the registration method forchest MDCT images based on 2-D FEM. Our proposed
method has three phases which are temporal subtraction
technique, creation of the mesh of triangle elements, andregistration based on 2-D FEM. We evaluated our
method by FWHM and artifact to lung volume ratio, andcost time and obtained. Furthermore, the result

3. Result
In this paper, we evaluated the performance of the image
registration method using the 2-D FEM. Twoexperiments were conducted. Experiment 1 confirmed
the effectiveness of the method using synthetic data. In
Experiment 2, previous and current chest CT of the same
subject were set as a set and registration processing was
performed.
3.1. Quantitative evaluation on synthetic data
We confirmed the effectiveness of the method usingsynthetic data. The previous image is rotated by 5 degrees
to become the current image, and the previous image iswarped and registered to create the temporal subtraction
image.
As the evaluation method, Full Width at Half Maximum(FWHM) and subtraction artifact to lung volume ration
were used. FWHM represents the spread of the histogramand smaller values of FWHM indicate fewer artifacts.
The offset width is 71. An example of a concentration
histogram is shown in Fig. 2.
In the experiment, we applied this method to 31 cases. As
a result of the experiment, FWHM: 12.1, artifact to lungvolume ratio: 2.70%, cost time: 22sec were obtained.
Table 1 shows the result.

Previous
method 4
ProposedmethodFWHM

Artifact tolungvolumeratio [%]

Cost time
[sec]

28.1

5.70

228

28.0

5.77

140

Table 2 Experimental 2 result

FWHM:28.0, artifact to lung volume ratio: 5.77%, andcost time: 140 sec was obtained.

3.2. Quantitative evaluation on real data
The previous and current chest CT of the same subject
was set as one pair and the registration process was
performed. We applied the proposed method to all 31
pairs and found the result. From Table 2, FWHM: 28.0,
artifact to lung volume ratio: 5.77%, cost time: 140 wereobtained.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

146

Takuji Ogimoto, Tohru Kamiya, Takatoshi Aoki

References
1. K. Doi, Current status and future potential of computeraided diagnosis in medical imaging, British Journal ofRadiology, Vol. 78, pp. S3-S19, 2005.
2. A. Kano et al., Digital image subtraction of temporally
sequential chest images for detection of interval change,
Medical Physics, Vol. 21, No.3, pp.453-461, 1994.
3. T. Aoki et al., Temporal Subtraction Method for Lung
Nodule Detection on Successive Thoracic CT Soft-Copy
Images, Radiology, Vol. 27, No. 1, pp. 255-261, 2014.
4. S. Yamada et al., Three-dimensional non-rigid registrationof thoracic CT image based on finite element method,
Joint International Conference on Soft Computing andIntelligent Systems (SCIS) and International Symposium
on Advanced Intelligent Systems (ISIS), pp. 1-6, 2014.
5. Y. Itai et al., Development of A Voxel Matching
Technique For Substantial Reduction of Subtraction
Artifacts in Temporal Subtraction Images Obtained fromThoracic MDCT, Journal of Digital Imaging, Vol. 23, No.
1, pp. 31-38, 2010.
6. L.P. Chew, Constrained Delaunay triangulations,
Algorithmica, Vol. 4, No. 1, pp. 97-108, 1989.
7. M. Ferrant et al., 3D Image Matching Using a Finite
Element Based Elastic Deformation Model, Lecture Notes
in Computer Science, Vol. 1679, pp. 202-209, 1999.

(a)previous image(b)rotated image (c)output image
Fig. 3 Experimental result: rotated image(a)previous image (b)current image (c)output image
Fig.4 Experimental result

Fig. 5 Experimental 2 FWHM result

Fig. 6 Experimental 2 artifact to lung volume ratio
resultTable 1 Experimental 1 results

FWHM

Artifact tolungvolumeratio [%]

Cost time[sec]Previous
method 4

12.1

2.68

45

Proposedmethod

12.1

2.70

25

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

147

Detection of Abnormal Shadows in Low-dose CT Images Using CNN
Hiromu Ikeda, Tohru Kamiya

Department of Mechanical and Control Engineering, Kyushu Institute of Technology, 1-1 Sensui-cho, Tobata-ku,
Kitakyushu-shi, Fukuoka, 804-8550, Japan
Takatoshi Aoki

University of Occupational and Environmental Health, 1-1, Iseigaoka, Yahatanishi, Kitakyusyu-shi 807-8555, Japan

Abstract
CT imaging is a very effective way to detect abnormal shadows in the lungs. Considering thephysical burden on the patient, it should be taken at a low dose. However, there are currently few
studies using low-dose CT as compared to normal-dose CT. Therefore, it is necessary to develop asystem to support the diagnosis of low-dose CT. To detect lung nodules and GGO from a low-dose
CT image, we propose image processing techniques based on CNN. Our approach consists of two
main elements. First, candidate abnormal shadow areas are extracted from low-dose CT images of
the chest by threshold processing and filtering. Second, we implement a CNN model to classify
candidate region on images. To improve identification accuracy in CNN, we devised a method to
add spatial information to the input image. In this paper, the proposed method is applied to 13
abnormal shadow images and 100 normal tissue images, and true positive ate of 84.6%, false
negative rate of 15.4% were achieved respectively.
Keywords: Low-dose CT, Computer aided diagnosis, Dual-ring filter, Convolutional neural network,
input image processing

1.

CT imaging is often used for early detection of thedisease. However, there are some problems in CT
imaging, such as an increase in the burden of the reader
due to the huge number of images, and a difference indiagnostic results due to differences in the experience ofphysicians. To avoid this problem, there are someapproaches in medical imaging fields. The CAD system
digitizes the medical data, quantifies and analyzes the
medical data using a computer. As a result, physician can
use the results of the CAD as a "second opinion". In
addition, the low-dose CT used in this paper, and is lessburdensome on the patient than the one taken at a normaldose, however the image quality is poor.
In recent years, deep learning has attracted a lot of
attention in the field of artificial intelligence. Especiallyin the field of image recognition fields, convolutional

Introduction

According to the world health organization (WHO),
cancer is the second leading cause of death worldwide,
and is responsible for an estimated 9.6 million people in2018. Among them, lung cancer accounting 2.09 million
cases, and is the large number of deaths cause. It is ranked
the most common causes of cancer death [1]. Early
detection is important factor to reduce mortality fromcancer. If the cases are detected and treated in early stage,
the mortality rate from cancer may decrease. When thecancer is detected in early stage, it has high survival rate,
low morbidity, low treatment costs and high potential for
effective treatment. Also, detecting the cancer in earlytime and avoiding delays in treatment may make a huge
difference in the lives of cancer patients.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

148

Hiromu Ikeda, Tohru Kamiya, Takatoshi Aoki

neural networks (CNNs) have become the mostimportant technology to classified a target from an image.
The various CNN architectures are also becoming
increasingly important in the medical imaging field. In
this paper, we describe the development of a CAD system
for the detection of abnormal shadows from low-dose CT
images. In this approach, candidate abnormal shadow
areas are extracted from CT images and the regions are
inputted into the CNN to classify normal or not.
2. Method
In this system, we extract candidate abnormal shadow
areas from CT images based on CNN. The details areshown in following. Figure 1 shows an overview of the
CAD system which is built in this paper. Figure 2 shows
the flow of the initial candidate region extraction.

Fig. 1. The flow of CAD system

2.1. Extraction of initial candidate area
Lung field areas are extracted from CT images. In thispaper, the areas were extracted by binarization, labeling,
and thresholding techniques. Next, emphasis was applied
to make it easier to detect abnormal shadows. The details
are shown in follows.
A. Segmentation of lung area
To segment region of interest (ROI), we perform theGaussian filter for smoothing, and binarization based on
Otsu’s method [2]. Next, the labeling process isperformed, and the largest area detected as lung area.
B.

Enhance abnormal shadows based on
double-ring filter

Most of the abnormal shadows in CT images hascircularshape. In addition, because of the use of low-dose CT, theeffect of noise should be considered. From these pointsof view, we use a double-ring filter [3] for emphasis
processing, which can enhance the difference in pixel
values between the center and the surrounding area toreduce the effect of noise. The output value of the doublering filter is the average value of the inner pixel value
minus the average value of the outer pixel value. In thispaper, the 7×7 pixels of a 9×9 filter are considered.

Fig. 2. initial candidate area extraction

2.2. Input images to CNN
To make it easier to identify the abnormal shadows, we
added processing to the images input to the CNN.
Specifically, the image before one slice of the candidate
region was made into R components, the image of thecandidate region was made into G components, and theimage after one slice of the candidate region was madeinto B components on the CNN model. This allows us toobtain a two-dimensional image with spatial information.
Figure 4 shows an example of the input image.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

149

Detection of Abnormal Shadows

2.3. CNN
3. Results

In this paper, we perform transfer learning based on the
VGG-16 [4] network structure learned using ImageNet.
In addition, we frozen weights up to convolutional layer
10 and retrained after that layers with the generated
images. Also, we add a batch normalization [5] behind
all convolutional layers to suppress the influence of bias
in the data. To adjust the weights of the training, and to
suppress over-learning and to obtain global features, wechange all the coupled layers to global average pooling[6] and reconstruct the model as shown in Fig.5.

3.1. Detail of Dataset
In this paper, we prepare 13 classes of data sets. One classconsists of test and training data. The test data contains 1
abnormal shade and 6 normal tissues. The training datacontains 156 abnormal shades and 936 of normal tissue.
The training data contains augmented data. Data augment
includes vertical and horizontal inversion, mirror image
inversion, contrast adjustment and gamma correction.
3.2. Evaluation metrics
The performance is evaluated using leave-on-out crossvalidation. Classification accuracy is evaluated by thefollowing equations: TPR(True Positive Rate) %, andFPR(False Positive Rate) %, respectively. Here a,b,c,d
are shown in Table.1.
𝑎𝑎
𝑎𝑎 + 𝑏𝑏
𝑐𝑐
𝐹𝐹𝐹𝐹𝐹𝐹 =𝑐𝑐 + 𝑑𝑑

𝑇𝑇𝑇𝑇𝑇𝑇 =Fig.3 Extraction of candidate areas

(1)
(2)

3.3. Results
In this paper, we use the CNN model described in 2.3.
We discuss the case where the inter-slice information isreflected in the input image (proposed method) and thecase where it is not reflected in the input image as
described in 2.2. Table 2 shows classification results. In
Table 2, the case where the inter-slice information isreflected in the input image is shown as the proposed
method, and the case where it is not reflected is shownas denoted as baseline. As a comparison, we also
performed with a method of classifying the SVM [7]with features extracted using CNN, which is effective
when the number of images in the dataset is small. The
proposed method achieved a TPR of 84.6%, FPR of
15.4%. In contrast, when the input image does notreflect the inter-slice information, the TPR of 53.8%,
FPR 15.4%. The SVM is used as another classifier, theTPR of 53.8%, FPR 1.3% was obtained.

Fig.4 Input images to the CNN

conv

global average pooling© Themax
2021
Internationalon Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021
softConferencemax
poolingFig.5 CNN

150

Hiromu Ikeda, Tohru Kamiya, Takatoshi Aoki

4. Discussion

4. K. Simonyan, “Very deep convolutional networks forlarge-scale image recognition”, arXiv preprint arXiv: 1409.
1556, 2014.
5. Sergey Ioffe Christian Szegedy, “Batch Normalization:
Accelerating Deep Network Training by Reducing
Internal Covariate Shift”, arXiv:1502.03167，2015．
6. Min Lin et al., Network In Network ” ， Cornell
University Library, arXiv: 1312.4400，2014.
7. B. E. Boser et al., “A training algorithm for optimal marginclassifiers”, Proc. 5th ACM Workshop on Computational
Learning Theory, pp.144-152, 1992.

We used a CNN model to discriminate the presence orabsence of lung cancer in low-dose CT images. The TPR
Table.1 Confusion matrix

ActualClass

positive
negative

Prediction Class
positive
negative
𝑎𝑎
𝑏𝑏
𝑐𝑐
𝑑𝑑

Table.2 Experimental results
TPR%
FPR%
Proposed method
84.6
15.4
Baseline
53.8
15.4
SVM
53.8
1.3

of the proposed method was 30.8% higher than that of thebaseline. This suggests that reflecting inter-slice
information in the input image may be useful. In somecases, abnormal shading could not be detected. The
reason may be that what could not be detected was very
faint. This can be solved by using an image withenhanced shading in the input image. In addition, we
would like to improve the structure of CNN in the future.
5.Conclusion
We developed a CNN-based method for detecting
abnormal shadows from low-dose CT images. We
achieved a TPR of 84.6% and FPR of 15.4%. In future,
we need to develop a shading enhancement method forthe input images and improve the structure of the CNN
to further improve the classification accuracy.
References
1. World
HealthOrganization,
Cancer,https://www.who.int/en/news-room/factsheets/detail/Cancer
2. Otsu, “A threshold selection method from gray-level
histograms”, IEEE Transactions on Systems, Man andCybernetics, Vol.9, No. 1, pp.62-66, 1979.
3. Y. Hatanaka, et al., “Automated detection algorithm forarteriolar narrowing on fundus images,” Proceedings ofthe 27th Annual International Conference of the IEEE
Engineering in Medicine and Biology Society, pp. 286–
289, 2006.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

151

EEG Signal Extraction Method Based on HHT and CSP
Lei Gong, Chengzhi Liu

Tianjin University of Science and Technology, China;
E-mail: * 1442064639@qq.com
www.tust.edu.cn

Abstract
In this paper, the Hilbert Huang transform is used to extract the marginal spectrum of the four-task motor imagery of
EEG signals. Firstly, a spatial filter constructed by common spatial pattern is used to extract the feature of signal
marginal spectrum. The MI EEG Data2a provided by The University of GRAZ is verified by the experiment, and theresults show that compared with the traditional single CSP feature extraction method, the effect of MSCSP feature
extraction was significantly improved.
Keywords: motor imagery (MI), Hilbert Huang transform (HHT), common spatial pattern (CSP), marginal spectrum(MS)

1. Introduction

hands, feet and tongue is a typical four-task in MI brain
computer interface. EEG signal processing and analysis is
the core part of the whole BCI system. Feature extraction
is mainly through the analysis of EEG signal in time
domain, frequency domain and spatial domain to find thefeatures which are difficult to be found in the original EEG
signal and can best reflect the thinking activity 4. The
existing feature extraction methods mainly include power
spectral density (PSD), independent component analysis(ICA), discrete wavelet transform (DWT), Hilbert Huang(HHT) and common space pattern (CSP)5,6,7,8.
In this paper, the marginal spectrum (MS) of four-task
motor imagery of EEG is extracted by Hilbert Huangtransform (HHT). For the first time, the common spatial
pattern (CSP) is used to construct a spatial filter to extract
the marginal spectral features (MSCSP) of the signals.
Combining with the classification method of linear
discriminant analysis (LDA), the classification accuracy of
EEG signals is improved, which creates conditions for thepractical application of BCI system.

Electroencephalogram (EEG) is to record the spontaneousand rhythmic activity of brain cells through electrodes.
EEG based brain computer interface (BCI) system does
not depend on muscle nerve activity, but on EEG activity
recorded at scalp 1. As a new interdisciplinary subject, BCI
technology has attracted the attention and research ofmany international and domestic researchers, and has
shown broad application prospects in the fields ofrehabilitation medicine engineering, auxiliary control andmilitary 2. With the rapid development of science andtechnology, although the quality of BCI technology has
been improved rapidly, there is still a certain distance
between BCI technology and practical application, among
which the search for excellent feature extraction method is
one of the keys.
The brain computer interface based on MI is acommonly used experimental paradigm of spontaneous
EEG signals. The basic principle of MI is that when people
imagine a certain limb of the body moving, according to
the previous motor memory, the motor cortex of the brain
will have corresponding potential changes 3. Imagining

2. Experimental Data

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

152

Lei Gong, Chengzhi Liu

signal after HHT transformation can obtain the timefrequency information of EEG signal more visually.

The data set selected in this paper is the BCI competition
IV dataset2a motor imagery data set in the fourth brain
computer interface competition in 2008, provided by Graz
University of science and technology, Austria 9. The data
set was collected from 9 subjects. Each subject performedfour-task motor imagery tasks: left hand, right hand,double feet, and tongue. Each subject was given two
experiments. The EEG data collected for the first time isused as training set, and the task tag is known. The EEG
data collected in the second time is used as the test set, andthe task tag is unknown. In each data set, the subjects
conducted 6 rounds of experiments, and 48 groups of data
were collected in each round. Therefore, a total of 288
groups of experimental data were collected for eachsubject 10.
The whole process of the experiment is shown in
Fig.1.The first stage is the preparation stage, i.e., when t =0s, a cross symbol appears on the screen and a soundprompts the subject to prepare to start the experiment; thesecond stage is the motor imagery task prompt stage, thatis, when t = 2s, a task prompt arrow appears on the screen.
At this time, the subjects were reminded to start to prepare
for the motor imagery task; the third stage was the motor
imagery stage, that is, when t = 3s, the motor imagery task
was started, lasting for 6s until the cross symbol on thescreen disappeared. The fourth stage is the rest stage. Aftera short interval, the next round of experiment is carried out.

Fig.2. Three-dimensional spectrum of EEG signal after
HHT transformation

4. Improved CSP Algorithm Based on MarginalSpectrum
The traditional CSP algorithm is operated directly in thetime domain signal. It is difficult to train a good
adaptability projection matrix on account of thecomplexity of EEG signal time-domain waveform change.
When the brain carries out motor imagery, obvious ERD
and ERS phenomena will appear in the C3, C4 and CZ
regions of the cerebral cortex 13. This phenomenon willcause obvious changes in the amplitude of the rhythm (813hz) and β rhythm (14-25hz) of the signals in this region.
This frequency phenomenon has more obvious differences
than the time-domain signals. Therefore, this paper
proposes a method combining CSP algorithm with signal
marginal spectrum (MS) algorithm (MSCSP). The
algorithm combines the good binary classification effect of
CSP algorithm and the good time-frequency
characteristics of the marginal spectrum, so the MSCSP
feature matrix has better adaptability.

Fig.1. Experimental flowchart

3. Hilbert Huang Transform

4.1. Marginal spectrum feature extraction

HHT includes two parts: Empirical Mode Decomposition(EMD) and Hilbert transform11,12. It retains the advantagesof time-frequency local transform in discrete wavelet
transform and overcomes the problem of no adaptive basis
function. This method of adaptive EEG decomposition
does not necessarily require automatic signal processing.
After HHT transformation, the three-dimensional
spectrum of time amplitude frequency is obtained. As
shown in Fig.2, the three-dimensional spectrum of an EEG

In order to obtain the marginal spectrum, EMD
decomposition is used to divide the complex signal into
simple IMF single component signal, so that the concept
of instantaneous frequency can be applied. Then the
instantaneous frequency is analyzed by Hilbert transform,
and the Hilbert spectrum is integrated with time to obtainthe Hilbert marginal spectrum14,15. Therefore, MS isdefined as follows:© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

153

EEG Signal Extraction Method

4.2. Improved algorithm of CSP(1)

In this paper, the "one-to-one" CSP feature extraction
method is used to process four kinds of task motor imagery
EEG signals. In this paper, four-task of motor imagery
signals are divided into two groups (left hand 1vs righthand 2, left hand 1vs feet 3, left hand 1vs tongue 4, righthand 2vs feet 3, right hand 2vs tongue 4, feet 3vs tongue
4). Six sets of CSP features are obtained by constructing
six spatial filters through CSP, which are used as input ofclassifiers and the final output category.
After preprocessing the motor imagery data of the
training set 4S, the marginal spectrum of each channel isobtained by HHT transformation. The CSP projection
matrix and training set features are obtained by CSP
operation of the marginal spectrum of each channel. For
the test set data, after the same preprocessing, the marginal
spectrum of each channel is obtained by HHT
transformation, and the test set characteristics can beobtained by multiplying it with the CSP projection matrix.
The MSCSP characteristics of EEG signals before andafter marginal spectrum processing are shown in Fig.4, in
which Fig.4(a) is the six groups of CSP feature maps
without obtaining the marginal spectrum by HHT
transformation, and Fig.4 (b) is the six groups of MSCSP
feature maps with marginal spectrum obtained by HHT
transformation. From the feature map, it can be found that
the CSP features of left hand and right hand, left hand andtongue, right hand and tongue, and tongue and feet before
transformation have small distance and feature overlap
seriously, so they can not be well distinguished. After themarginal spectrum is obtained by HHT transform, theinterval between the two kinds of MSCSP features
increases and the feature overlap decreases, which candistinguish the two kinds of signals well. Therefore, themarginal spectrum EEG signal processing method basedon HHT transform can improve the discrimination of
MSCSP features and ensure the accuracy of classification.

Where T is the length of the signal. four-task of motor
imagery data, including left hand, right hand, double feet,and tongue, are selected from the standard data set, and themarginal spectra of C3, C4 and CZ channel data of thesefour tasks are extracted by HHT transformation, as shownin Fig.3.
It is obvious from the Fig.3 that when the subjects are
imagining the movement of their left and right hands, feetand tongue, obvious ERD and ERS phenomena appear in
C3, C4 and CZ channels, and the amplitude of the edge
spectrum changes with the change of frequency in thewhole frequency band. Although it is similar to thespectrum, its meaning is different from that of thefrequency spectrum, which represents the value of eachfrequency amplitude in the whole data set. It canaccurately reflect the actual frequency component of thesignal. The amplitude of a point in the spectrum proves that
there is a trigonometric function of the frequency in thewhole signal. The larger the amplitude, the higher thematching degree between the trigonometric function of thefrequency and the original signal. By comparing thedifference between the marginal spectrum and Fouriertransform, it can be found that the marginal spectrum can
not only process the stationary signal, but also the nonstationary signal. We can sensitively find the energy
transformation of a certain frequency in the signal. So themarginal spectrum can reflect the actual frequency
component of the signal more accurately.
Left-hand marginal
spectrum
0.35

0.4

C3
C4

C3
C4

0.25

0.25

0.15

0.1

0.3

Amplitude

0.3

0.2

双脚Cz
舌头Cz

0.35

0.3

Amplitude

Amplitude

Marginal spectrum of feetand tongue

Right-hand marginal
spectrum

0.35

0.2

0.05

0.05

0

0

10

15

20

Frequency（Hz）

25

0.2

0.15

0.1

5

0.25

5

0.15

6

0.1

4
2

0.05

10

15

20

Frequency（Hz）

25

0

5

10

15

20

Frequency（Hz）

5
4.5
4

1
2

3

3.5

4

4.5

5

5.5

3.5

Fig.3. Marginal Spectra of Four Kinds of Motion
Imagination

4

4.5

5

4.5

5

4.5

5

5.5

5

4.5

25

1
3

3.5

4
3.5

4

1
4

4

4.5

5

3

5.5

2
3

3.5

4

5.5

5
4.5

4

4.5

2
4

3.5

4

4.5

5

(a)

4
3.5

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

154

3
4

4

5.5

Lei Gong, Chengzhi Liu
0.5

0

0.5

1
2

0

0.1

0.2

0.3

0.4

0.5

0

0.1

0.2

0.3

0.4

0.5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.5

0.5

0

0

0.6

MSCSP algorithm, the final classification result is still
relatively low. Therefore, compared with the other threealgorithms, the improved MSCSP algorithm can extract
EEG features and has great advantages.

1
3

1
4

0

0.2

0.4

0

0.8

0.6

2
3

6. Conclusion

0.6

0.5

In the BCI system based on motor imagery, effective
feature extraction and classification is the key to thesystem. This paper improves the CSP algorithm, uses HHT
to extract the marginal spectrum of EEG signals, andselects the marginal spectrum as the input to extract thefeatures of CSP. Based on the data provided by GRAZ
University, the paper analyzes the feature extraction effectof the improved MSCSP algorithm, and finds it is better
than the traditional CSP algorithm. The feature extraction
method in this paper is in line, it is with the current
development direction of BCI, and provides anexperimental basis for application of BCI system.

0.2

0

2
4

0

0.2

0.4

0.6

0

0.8

3
4

0

0.2

0.4

0.6

0.8

1

(b)
Fig.4. (a) CSP feature map before marginal spectrum
processing, (b) MSCSP feature map before marginal spectrum
processing

5. Analysis of Experimental Results
After preprocessing the BCI competition IV dataset 2adata set, the EEG signals were divided into four groups
according to the motor imagery tasks of left hand, righthand, double feet and tongue. Each group contained 72
single experiments. In each experiment, the EEG signal
components during motor imagery, i.e., the data from 3s to
6s, were intercepted, and the power spectral density (PSD),
discrete wavelet coefficients (DWT), common spacepattern features (CSP) and the improved algorithm
MSCSP were extracted respectively. SVM classifier wasused for feature classification. The final classification
results are shown in Table 1.

Acknowledgements
References
1.

2.

Table 1. Comparison of classification accuracy of features
extracted by each algorithm
Subjects
A01T
A02T
A03T
A04T
A05T
A06T
A07T
A08T
A09T
average3.

Feature extraction method (accuracy%)
PSD

DWT

CSP

MSCSP

44.83

51.72

71.43

67.86

43.38

48.28

53.57

57.14

41.38

68.97

76.79

83.93

20.69

41.38

50.89

58.93

43.10

20.69

33.57

41.07

25.86

31.03

42.14

55.36

46.55

51.72

68.21

67.86

48.83

55.17

80.18

82.14

44.83

48.28

63.75

66.07

39.93±9.77

46.36±14.01

60.05±15.98

64.48±12.61

4.
5.
6.

From the final classification results in Table 1, under thesame preprocessing conditions, it can be found that themethod of extracting PSD of signal correlation frequency
band has the lowest classification effect. Secondly, DWT
is used to extract low-frequency coefficients of signals,
which has no good classification effect. In time domain,
the effect of CSP feature extraction is much better than thefirst two methods. However, compared with the improved

7.

8.

Blankertz B ， Tomioka R ， Lemm S ， etal． Optimizing spatial filters for robust EEG single-trial
analysis． IEEE Signal Processing Magazine， 2007，
25(1)：41-56．
Emanuel N ， Felix B ， Harald A ， et al． RegularizedLinear Discriminant Analysis of EEG Features in Dementia
Patients． Frontiers in Aging Neuroence， 2016， 8：
273-．
Haas， L F． Hans Berger (1873-1941)， Richard Caton(1842-1926)， and electroencephalography． J Neurol
Neurosurg Psychiatry， 2003， 74(1)：9．
Vidal
Towarddirectbrain-computercommunication ． Annual Re- view of Biophysics andBioengineering， 1973， 2(1)：157．
Tiwari N， Edla D R， Dodia S， et al． Brain computerinterface ： A comprehensive survey ． Biologically
Inspired Cognitive Architectures， 2018， 26： 118-129．
Dong E Z， Li C H， Li L T， et al． Classification ofmulti-class motor imagery with a novel hierarchical SVM
algorithm for brain-computer interfaces ． Medical &Biological Engineering & Computing， 2017， 55(10)：1809-1818．
Jeunet C， Lotte F， Batail J M， et al． Using recent
BCI literature to deepen our understanding of clinicalneurofeedback： A short review． Neuroscience， 2018，
378： 225-233．
Naeem M， Brunner C， Leeb R， et al． Seperabilityof four-class motor imagery data using independent© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

155

EEG Signal Extraction Method

9.

10.

11.

12.
13.

14.
15.

components analysis． Journal of neural engineering，
2006， 3(3)： 208．
Naeem M， Brunner C， Leeb R， et al． Seperabilityof four-class motor imagery data using independent
components analysis． Journal of neural engineering，
2006， 3(3)： 208．
Pfurtscheller G ， Neuper C ，Guger C ，et al． Current
trends in Graz Brain-Computer Interface (BCI)
research． IEEE Trans Rehabil Eng， 2000， 8(2)：216219．
Dong E ， Zhou K ， Tong J ， et al． A novel hybridkernel function relevance vector machine for multi-task
motor imagery EEG classification． Biomedical Signal
Processing and Control， 2020， 60：101991．
Bauschke H H ， Combettes P L ． Convex Analysis andMonotone Operator Theory in Hilbert Spaces． 2011，10．1007/978-1-4419-9467-7．
Liu C ， Zhao H B ， Li C S ， et al． Classificationof ECoG motor imagery tasks based on CSP and SVM
2010 3rd International Conference on Biomedical
Engineering and Informatics． IEEE， 2010．
Yeo M V M ， Li X ， Shen K ， et al． Can SVM beused for automatic EEG detection of drowsiness during car
driving． Safety Science， 2009， 47(1)：115-124．
Vladimir Cherkassky and Yunqian Ma． Practical selectionof SVM parameters and noise estimation for SVM
regression． Neural Networks， 2004．

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

156

Verification of CG character operation by brain wave discrimination
Kenji Sakoma, Kodai Miyamoto, Taketo Kamasaka, Makoto Sakamoto*, and Amane Takei

Faculty of Engineering, University of Miyazaki,
Miyazaki-City, Miyazaki, Japan
Tsutomu Ito

National Institute of Technology, Ube College
Ube-City, Yamaguchi, Japan
Takao Ito

Graduate School of Engineering, Hiroshima University
Higashi-Hiroshima, Hiroshima, Japan
E-mail: hm16011@student.miyazaki-u.ac.jp, hm15037@student.miyazaki-u.ac.jp, hm16043@student.miyazaki-u.ac.jp,
fruits2000jp@yahoo.co.jp*Corresponding Author

Abstract
Recently VR technology is expected to develop in various fields. In this research, we verified whether the VR spacecan be operated just by thinking in the head by using a device that is cheaper than the device used in the existing
research. We used the fast Fourier transform and the support vector machine as the methods of EEG analysis. In the
future, we will try to improve it further by changing the method of EEG analysis and other conditions.
Keywords: EEG, BMI, FFT, SVM, VR

We paid attention to the troublesomeness of wearing
in this. When using VR equipment, it is often necessary to
set up cables and sensors in advance. Also, when using a
VR device, it is necessary to have a certain amount of
space around the user. This is to ensure the safety of thesurroundings and the user himself / herself because thecontroller is always used when operating / moving in theworld inside VR. Therefore, there is a problem that theplace where the VR device can be used is limited. It tends
to be more expensive than models that require cables, but
recently, integrated (standalone) models that do not requirecables have also been released. Therefore, the problem ofsetting up the cable or the like can be solved. However, thespatial problem when using VR equipment has not beensolved.

1. Introduction
Since 2016, which was called the "first year of VR(Virtual Reality)", VR technology has been researched anddeveloped all over the world and has made remarkableprogress. Various companies have begun to develop andsell VR equipment, and although it is a little expensive,
even ordinary people can easily purchase it. VR
technology has become widespread, as PCs and otherdevices that can use VR devices have become affordable
even for individuals. As a result, VR technology is
expected to develop in various fields such as medical care,education, business, and entertainment. However, there aresome major challenges in the development of VR
technology.

@ The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

157

Kenji Sakoma, Makoto Sakamoto, Takahiro Ishizu, Takahiro Shinoda, Amane Takei

Therefore, in order to try to solve the spatial problem,the operation by brain waves --- BMI (Brain-MachineInterface) is used. Since the frequency band of humanbrain waves, such as "α wave" and "β wave", fluctuates
depending on the activity state, the purpose is to use themfor controlling CG characters and robots by reading themwith a device and discriminating them. It is a study. I thinkthat it may be possible to solve the spatial problem whenusing VR equipment by using this method1-4).

Equipment sold by NeuroSky (see Figure 1). In
this study, we will use this device to measure brain waves.

2.

2.4. Library

2.3. Development environmentOS
Programming language
Measuring equipment

Windows 10
C#
Mind Wave Mobile
Visual Studio 2019
Software
Unity 2019.2.18f
Table１．Development environment．

Research Method

libStreamSDKUsed to receive brainwave data from Mind Wave
Mobile in Visual Studio 2019.

MathNet.Numerics
Accord
Accord.MachinerLearning
Accord.Math
Accord.Statistics
Used to process data received from Mind Wave
Mobile in Visual Studio 2019.


2.1. VR
VR (Virtual reality) is a technology and system thatcreates an environment that is not the actual thing or thereal thing but has the same essence as a function bystimulating the senses including the five senses of the user.
Translated as "artificial reality" or "virtual reality" inJapanese. In the olden days, novels, paintings, plays,
television, etc. also had more or less VR functions.
Several problems have been identified with VRtechnology, and these are problems that must be resolved
for future development.
1. Troublesome installation problem
2. Spec problem
3. Social and market issues
4. Problems at the production site
5. Health issues
The above-mentioned "installation troublesome
problem" includes a safe space problem that must be
ensured in order to use the VR product that occurs when
the VR product is used. This problem means that you can't
play anywhere when using a VR product.
This study focused on this problem.

2.5. Brainwave measurement
Use NeuroSky's Mind Wave Mobile6) brain wave
sensor to measure brain waves. In addition, in order toreceive brain wave data in Visual Studio 2019, we used alibrary called libStreamSDK included in the development
tool provided by NeuroSky.
2.6. EEG processing method
2.6.1.

It is a type of DFT (Discrete Fourier Transform) andis an algorithm devised so that DFT can be executed athigh speed on a computer (see Equation 1). When using
this analysis method, the MathNet.Numerics library wasused7).

2.2. Used equipment


FFT (Fast Fourier Transform)

Mind Wave Mobile

𝑁𝑁−1

2𝜋𝜋

𝑋𝑋𝑘𝑘 = � 𝑥𝑥𝑛𝑛 𝑒𝑒 −𝑖𝑖 𝑁𝑁

2.6.2.

𝑛𝑛=0(1)

EEG feature discrimination method

For EEG characterization, SVM (Support Vector
Machine), which is generally used for EEG
characterization, was used. SVM is one of the patternrecognition models that uses supervised learning (seeFig. 1．Mind Wave Mobile．

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

158

Verification of CG charactercharacter in Unity moves accordingly. In this study, there
are only two types of CG character movements: stop
when you are relaxed and keep moving forward while
you are thinking of moving your body.

Equation 2). SVM has high discrimination performancefor untrained data. In this study, a Gaussian kernel (seeEquation 3, 4) was used as the kernel function. In addition,5
libraries
ofMathNet.Numerics,
Accord,Accord.MachinerLearning,
Accord.Math,and
Accord.Statistics were used to use SVM5).
y(x) = sgn � � wn K(xn , x) + b�Visual Studio 2019

5.

Visual Studio 2019 processes the brain wave dataread by Mind Wave Mobile and performs feature detection.
At this time, FFT is used to detect the feature amount of
the electroencephalogram data. Then, using SVM, theprocessed brain wave data is discriminated so that 0 is
output when the brain wave is relaxed and 1 is output when
the brain wave is conscious of moving.
2.6.4.

Evaluation Result

The results of the questionnaire are as follows.
In "Good points", "It is fresh and interesting to move justby thinking" was mentioned.
"Bad points" are "I was worried because there
was a little time from thinking until the operation wasreflected in the CG character", "Mind Wave Mobile isuncomfortable to wear", "The action that the CG
character is thinking There were times when it did the
opposite of what it was. ""Others" included "Can I do anything other thanmove forward?" "Can I use running properly when
walking?""Other" were that "is it not compatible with othermovements?" and "is there no other CG character?",etc.

Unity 2019.2.18f

We created CG characters and programs that act
according to the characteristics of the read brain waves.
The brain wave data was processed by Visual Studio 2019,the result of discrimination by SVM was received, and the
CG character was created to act according to thediscrimination result. The CG character uses Unity-Chan !,which is provided free of charge in Unity's Asset Store (seeFig. 2) .

6.

Consideration

From the "good points" of the evaluation results, it was
found that the control of the CG character by brain waves, whichis the purpose of this study, can obtain a certain result indistinguishing only two patterns of forward and stop.
From the "bad points", it was found that there was a
problem with the accuracy and speed of EEG discrimination, andit was found that it was necessary to improve the EEG featuredetection method and discrimination method, and to improve theperformance of the processing device. It was also found that there
was a problem with the installation of Mind Wave Mobile, but itis thought that the problem with this is that the size of the productused this time cannot be adjusted. From the above "bad points",
it was found that there are points that should be further improvedin this study.
From the opinion of "Other", it was found that actions otherthan advancing and stopping are necessary in this research.

Fig. 2．The CG character which used in Unity．

3.

Evaluation Experiment

In this experiment, three university students willbe the subjects. An evaluation experiment was conducted
and a questionnaire was conducted. The evaluation
contents were "good points", "bad points", and "others" ina free description format.

(2)

0 𝑥𝑥 ∈ 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐(3)
1 𝑥𝑥 ∈ 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐
K(x1 , x2 ) = exp(−‖x1 − x2 ‖2 ∕ 2σ2 ) (4)=�

2.6.3.

n=SV

4.

Experimental Method

Have the subject sit in a chair, wear Mind Wave
Mobile on his head, and then start measuring brain
waves. During the experiment, the subjects are madeaware of relaxing and moving, and evaluate how the CG© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

159

Kenji Sakoma, Makoto Sakamoto, Takahiro Ishizu, Takahiro Shinoda, Amane Takei

7.

a portable terminal”, Information Processing Society ofJapan, 2014．
4. Uchida Yu, Kikuro Fujimura, Yoshihiro Maegaki, “A Studyon Numerical Processing of EEG Data for EEG Analysis”,
Japan Society for Fuzzy Theory and Intelligent Informatics,
2014．
5. Takio Kurita，“Introduction to Support Vector Machine”[Online].https://home.hiroshimau.ac.jp/tkurita/lecture/svm.pdf.
6. “Mind Wave Mobile headset by NeuroSky”,[Online]. https://www.neurosky.jp/mindwave-mobile2/
7. Jun Sakamoto, Yasuhiro Mori, Takayoshi Sekioka,“Probability Analysis Method By Discreate Fast Fourier
Transform”, J.Struct, Constr, Eng.AIJ, No.472, 39−45, Jun,
1995．

Conclusion

In this research, we tried to solve the space problem
around the user, among the "troublesomeness of wearing"which has become an issue in the development of VR
technology. The brain wave data was measured by Mind
Wave Mobile, the brain wave data from Mind Wave
Mobile was received and processed by Visual Studio 2019,
and the characteristics of the brain wave data processed by
SVM were determined. We sent the discrimination result
to Unity and verified the method in the field called BMI,
which manipulates the characters in Unity based on thediscrimination result.
From the results of the evaluation experiment, it was
found that although the operation of the CG character by
brain waves can obtain certain results, the accuracy is not
sufficient and there are many points to be improved. Futuretasks include improving the accuracy of brain wave featuredetection and brain wave feature discrimination,
increasing character movements by discriminating theratio of brain wave frequency bands, and improving theuncomfortable wearing of Mind Wave Mobile. It isthought that there is, and we plan to work on it in the future.
If this research progresses, it is expected that it willplay a major role in the medical field. By using thistechnology for people with physical disabilities who arephysically handicapped or who are using artificial limbs, itwill be possible to operate the body only with brain waves,which will be much more convenient than it is now. In
addition, the field of e-sports is currently developing in theworld, and VR technology is attracting attention in thatfield as well, so it is expected that if this research
progresses, it will be able to play an active role in the fieldof e-sports.
References
1. Takahir Ishizu, Kenji Sakoma, Makoto Sakamoto,“Fundamental Study on Control of CG Characters by
Electroencephalography (EEG) analysis”, MEMORIES OF
FACULTY OF ENGINEERING UNIVERSITY OF
MIYAZAKI, NO.48, 159-162, 2019.
2. “2017 Advanced Content Technologies, etc. Distribution
Promotion Business Report，Part I, guidelines for using
content production technology such as VR”，2018[Online].https://www.vipo.or.jp/u/I1_SenshinContents_Guideline.pdf.
3. Tomoki Hidaka, Yuki Shiraishi, Yukiko Kawai, Jiro Okud,“Information control system by electroencephalogram using© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

160

Approach to tourism support by aerial video using CG
Kenji Sakoma, Noritake Seto, Kodai Miyamoto, Taketo Kamasaka, Makoto Sakamoto*, Amane Takei

Faculty of Engineering, University of Miyazaki,
Miyazaki-City, Miyazaki, Japan
Tsutomu Ito

National Institute of Technology, Ube College
Ube-City, Yamaguchi, Japan
Takao Ito

Graduate School of Engineering, Hiroshima University
Higashi-Hiroshima, Hiroshima, Japan
E-mail: hm16011@student.miyazaki-u.ac.jp, hm15037@student.miyazaki-u.ac.jp, hm16043@student.miyazaki-u.ac.jp,
fruits2000jp@yahoo.co.jp*Corresponding Author

Abstract
Recently, as a first step in the tourism business, we will consider whether it is possible to increase the number oftourists by incorporating IT technologies such as CG in Introductory video. In this paper, we created a CG image ofa hawk flying in the sky combined with a real Takachiho video taken with a drone. As a result, various problems
have been identified, and I hope to improve them in future research to produce even better ones.
Keywords: EEG, BMI, FFT, SVM, VR

introductory videos such as PV and commercials.
This research is part of a joint development with
Takachiho Town, and the beautiful nature of Takachiho iscreated by making the images taken by the drone look likethe mountain hawk-eagle inhabiting Takachiho is swirlingor plunging in the sky. We conducted basic research tocreate a promotional video to introduce people in the area.

1. Introduction
Sightseeing is for people to refresh themselves andfor their hobbies, so it heals the mind in a different
environment than usual. However, from the perspective ofthe local people, it is a big commerce that can be expected
to revitalize the area and increase the number of residents.
In recent years, rural areas have been declining due
to urban concentration of work and declining birthrate andaging population. Meanwhile, the number of local
governments that are trying to regain the vitality of the pastby focusing on the tourism industry is increasing year by
year. As a first step in the tourism industry, consider
whether it is possible to increase the number of tourists byincorporating IT technologies such as CG into2.

Benefits of tourism4)

When many tourists come to visit due to thedevelopment of the tourism industry, economic activities
in various fields such as accommodation, transportation,eating and drinking, and travel industry become active, andthe economic ripple effect is high. For example, by@ The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

161

Kenji Sakoma, Makoto Sakamoto, Takahiro Ishizu, Takahiro Shinoda, Amane Takei

composite of images created by a computer with theoriginal photographs, drawings, and videos. Including
those processed and processed in a manner significantly
different from the original6)
In recent years, AR (Augmented Reality) and VR(Virtual Reality) have made remarkable progress.
However, CG is playing an active role not only inentertainment, but also in invisible parts such as model
production of architectural design and simulation in theevent of a disaster.

utilizing the nature and historic sites that originally existed
in the area, even a small scale can be established as anindustry. Therefore, at least some funds can be developed.
Also, if you can attract tourists from abroad, you can earn
foreign currency. In particular, in Takachiho Town, themythical nature is the centerpiece of tourism, so even asmall amount of assets can demand high productivity.
Initiatives of Takachiho Town5).
・ A tour of mythical nature and shrines (see Fig. 2).
・ PR of traditional Takachiho Kagura based on myth (seeFig. 1).
・ Sale of gourmet foods and souvenirs that make use of
the rich land of Takachiho Town.

4.

CG Tool

In this research, Blender is used for CG creation,and Unity is used for CG operation control.
4.1.

Blender8)

Objects are created from a single cube using the
extrusion and split functions (see Fig. 3). In order tosmoothly reproduce the flapping of the wings, which is
the greatest feature of birds, a total of 12 bones, 6 on eachside from the base to the tip of the wings, are added toplay the role of joints. Bones are normal ones and Bbones that are easier to express curves.

Fig. 1. Actual Kagura.

Fig. 3. The created hawk.
4.2.

·animation
In order to express the flapping and dive of thewings, the animation function is used to assign numericalvalues to the main body of the hawk-eagle and the bones
attached to the wings. The animation function applies
keyframe animation, and by changing the interval of eachnumerical value, smooth movement is reproduced.
·Turning
The movement of the object turning is processed
only by the script without using the animation function.
In this research, since the object is moved according to

Fig. 2. Amanoyasukawara, Kunimigaoka, Takachiho
Gorge.
3.

Unity1-3)

CG

In a narrow sense, it refers to images and videos
generated from scratch through drawing, editing, andprocessing on a computer, and in a broader sense, it is a

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

162

Approach to tourism support

the video actually shot, the rotation is only rotated aroundthe z-axis.
The turning is limited to 30 degrees on each side,and it is set so that it does not turn even if the key input iscontinued beyond that. However, as an important process
here, the turning angle is also different between the valueon the console and the value to be acquired, so conditions
are set so that the value is the same.
・ Image composition
An ICO ball is used to make the image change when the
hawk turns.
This is achieved by installing a hawk object and amain camera inside the ICO sphere (see Fig. 4) so that thesphere rotates when the arrow keys are input.

Blender can export in glTF format, which allows
it to be imported into Unity with almost the same shading
as Blender (see Fig. 6).

Export By FBX.

Color information etc. is lost.

Export By glTF.

Fig. 6. glTF
Since it is necessary to paste the image on the
inner surface of the sphere, download and use the
UniGLTF plug-in10).
・ Screen transition due to a sudden descent of a hawk
The screen transition is processed by the script,
and the screen transition is performed when the space key
is input.
In this state, the screen transition will be
performed before the animation of the sudden descent of
the mountain hawk eagle is performed, so when the space
key is input, the elapsed time will be observed and thevideo of the tourist introduction will be played when theelapsed time reaches the end of the animation. Transition
to the scene.
・ Sound / Video
Free materials are used for sounds, hawk-eagle
calls, aerial images, and sightseeing introduction videos
7).

Fig. 4. Composition of 360-degree image on a sphere
・ ICO sphere

Fig. 5. 3D sphere model.
In this research, since it is necessary to paste thetexture without distortion as much as possible, ICO
spheres are used in the 3D sphere model (see Fig. 5), andICO spheres are treated as distributed9).
・ glTF

5.

Consideration / Future issues

In this research, we used Blender and Unity tocreate a CG animation that promotes Takachiho bysynthesizing the CG of a mountain hawk eagle flying inthe sky with the actual image of Takachiho taken aerial
with a drone. This time, we succeeded in pasting theimage on the sphere, but there is a problem that theresolution of the free and usable image is too low and it
becomes too rough at the time of compositing.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

163

.

Kenji Sakoma, Makoto Sakamoto, Takahiro Ishizu, Takahiro Shinoda, Amane Takei

In addition, although the specification was
originally to combine images, the moving image file must
be in the Equirectangular format in order to combine
images from a 360-degree viewpoint, so in this research,
images are combined.
Since it is possible to transition to only one scene
in the screen transition, I think that it will be possible totransition to multiple scenes by acquiring the rotation
angle of the sphere and assigning it to the scene
corresponding to a specific angle.
References
1. Mikito Yoshito, “Unity5 3D/2D Introduction to gamedevelopment practice”, Soshimu, 2015.
2. “Ministry of Internal Affairs and Communications
Information and Communications White Paper Internet
penetration (2015 version)”,[Online].
http://www.soumu.go.jp/johotsusintokei/whitepaper/ja/h27/html/nc372110.html.
3. “Unity DOCMENT/Unity Manual”,[Online].
https://docs.unity3d.com/ja/current/Manual/index.html.
4. “Tourism Wikipedia”,[Online].
https://ja.wikipedia.org/wiki/%E8%A6%B3%E5%85%89%E6%A5%AD.
5. “Takachiho Kankokyokai Official site”,[Online]. http://takachiho-kanko.info/.
6. “IT Yogo jiten e-Words CG”,
[Online]. http://e-words.jp/w/CG.html.
7. “On-Jin ~Onjin~”,[Online].http://onjin.com/sound/kan.php?kate=%E9%A2%A8.
8. “Blender.jp”,[Online]. https://blender.jp/.
9. “360-dgree eizo no gosei”,[Online]. https://shinrinmusic.com/unity-360-image/.
10. “gITF”,[Online]. https://xr-hub.com/archives/20285.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

164

Method for detecting eye misalignment based on movement near the center of the pupil
Noriyuki Uchida1,2

Kyushu University of Health and Welfare
1714-1 Yoshino-machi, Nobeoka-City, Miyazaki, Japan1

Kayoko Takatuka2, Hisaaki Yamaba2, Masayuki Mukunoki2, Naonobu Okazaki2
2
University of Miyazaki
1-1 Gakuen Kibanadai-nishi, Miyazaki-City, Miyazaki, japan
E-mail: n-uchida@phoenix.ac.jp

Abstract
Only ophthalmologists and orthoptists are eligible to diagnose misalignment of the eyes amid persistent understaffing.
We automated the traditional eye misalignment evaluation (Cover-test) in our earlier studies and developed atechnique to distinguish the misaligned eyes from its behavior. This method, however, had a problem with the verticaleye movements being affected by the eyelids and eyelashes, which prevented thorough detection. Therefore, byobserving only the movement near the center of the pupil, where it is less likely to be disturbed by the eyelids andeyelashes, we arranged an alternative method to recognize anomalies. Based on this process, we achieved aninspection system and carried out a performance experiment. We confirmed that we could observe not only horizontal
motion, but also that of the vertical eye.
Keywords: eye position examination, cover test, strabismus/ heterophoria， compact camera

movements that suggest strabismus and heterophoria.
After the recent development of medical image
diagnostic systems, we explored the possibility ofcreating a machine learning-based detection system
based on continuous pictures of eye movements during
the examination. We found it arduous to collect study
data because the ocular inspection has never been aprocedure for image diagnosis before.
As a result, our research developed a basic inspection
system capable screening strabismus and heterophoria

1. Introduction
As misalignment of the eyes, such as strabismus andheterophoria, causes defective binocular vision andeyestrain, early detection and training are required.
Strabismus in children, in particular, is a risk factor thatcauses amblyopia. One of the research subjects ofpediatric ophthalmology is the diagnosis of strabismus at
an early age and the start of treatment.1 Only qualified
personnel such as ophthalmologists and orthoptists candiagnose eye misalignment, despite the ongoing shortageof skilled personnel. Therefore, it is significant todevelop a method to verify the alignment of the eye
without trained staff. 1,2
There are two types of eye position examination:
qualitative tests and quantitative tests. Specifically, thequalitative test investigates abnormalities, while thequantitative test measures the magnitude of the oculardeviation. While there are research reports on methods
for measuring the amount of ocular deviation,2 thisapproach only works for exotropia, as it is easy to assess
the eye location during the measurement. We also
explored the idea of using the current eye-tracking
system as an alternative tool. With this method, we
considered it necessary to plan various parameters for theapplication site and light to identify the slightest eye

Fig. 1. Cover-test automation and abnormality detection basedon the eye movement© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

165

Noriyuki Uchida, Kayoko Takatuka, Hisaaki Yamaba, Masayuki Mukunoki, Naonobu Okazaki

performance evaluation experiments of the proposed
method, and Section 5 summarizes them.

screening without restrictions. We automated theinspection process of the traditional qualitative
inspection form, Cover-Test (Fig.1-(A)) (Fig.1-(B)-I,II),1 and established and systematized a procedure for thedetection of eye movement abnormalities. (Fig.1—(C)—I
to IV). Cover-test is an inspection method to decidetropia or heterophoria by the subject focusing on an
object. We determine it tropia if the exposed eye moves
when the other eye is covered. If the covered eye moves
as it is exposed, we judge it heterophoria.
The conventional approach is to cover-uncover the eyeswith an occluder and evaluate abnormality by visualinspection (Fig.1-(A)). In the previous analysis, we
automated the cover test as follows. With the method we
originally developed, we extracted the area for anomaly
detection from the moving image around the eye (Fig.1(C)-III). Then, using optical flow technology,3 wedeveloped a series of processes that detect eye movement(Fig. 1-C-IV) and assess abnormality.4,5 During the test,
we took the moving picture around the eye with an ultracompact camera (Fig.1-B-I-(a)) mounted within the left
and right lenses. Eye movements displaying signs of eye
misalignment occur in approximately 7 to 8 seconds
immediately after being covered or exposed. Therefore,
we extracted a series of photos taken at a time interval of
about 0.2 seconds during that span. However, due to the
influence of eyelids and eyelashes, we were unable toobserve the vertical movements of the eye.
Vertical strabismus/heterophoria does not have as manycases compared with horizontal strabismus. Early-stage
diagnosis is crucial, however, as it can be a sign of severe
disorders such as cerebral palsy and heart failure. Also,
the number of cases has increased over the past few years
due to an aging population.6 Therefore, there isconsiderable significance in developing a system fordetecting strabismus and heterophoria in the verticaldirection

3. Proposal method
We use an ultra-compact infrared camera (referred to as
an "infrared camera") to take pictures of the center of thepupil and extract its contour. At the beginning of the test,
we refer to the location of the pupil center as the"reference point"(described later in 3.3). Focusing on thedispersion of the distance from the reference point, wefind that we can detect strabismus and heterophoria inboth directions, both horizontally and vertically, andabnormality.
The following explains the three processes in order.
(1) the extraction of the pupil area (Fig.2-I, II), (2) the
measurement of the pupil center coordinates (Fig.2-III),and (3) the determination of the abnormality.

Fig. 2. process of pupil center detection

3.1. Extraction of the pupil area
First, we tried to remove the pupil contour with theexisting ellipse fitting method.7 But we could not extract
stable enough to detect the slightest twitch of the eye
showing signs of strabismus and oblique location. We,
therefore, use the color identification function based on
the RGB color model given in OpenCV as another tool.
We found that we can extract the pupil contour by setting
the RGB values from (0,0,0) to (15,15,15).
3.2. Calculation of pupil center coordinates
Fig.2-Ⅲ shows the coordinates of the extracted pupilarea. The y coordinate, where the position coordinate is
maximum, is A (𝑎𝑎𝑥𝑥 , 𝑎𝑎 𝑦𝑦 ), and the minimum coordinate isC(𝑐𝑐𝑥𝑥 , 𝑐𝑐𝑦𝑦 ). The position coordinate where the x coordinate
is maximum is B (𝑏𝑏𝑥𝑥 , 𝑏𝑏 𝑦𝑦 ), and the minimum is D (𝑑𝑑𝑥𝑥 ,𝑑𝑑𝑦𝑦 ). We calculate the pupil central coordinate cp asfollows.
𝑐𝑐𝑦𝑦 + 𝑎𝑎𝑦𝑦
𝑏𝑏𝑥𝑥 − 𝑑𝑑𝑥𝑥
) (1)𝑐𝑐𝑐𝑐𝑗𝑗 = (𝑑𝑑𝑥𝑥 +, 𝑎𝑎𝑦𝑦 +22(j: Image number)

2. Research purpose
The goal of this study was to develop an alternative
method for detecting eye misalignment to distinguish not
only horizontal but also vertical eye movements. Basedon the issues of the previous study, we devised atechnique to diagnose disorders by monitoring only theactions near the center of the pupil, where eyelids andeyelashes are less likely to affect the pupil. Section 3
explains the proposed method, Section 4 introduces the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

166

Method for detecting eye

3.3. Anomaly Evaluation

Through the above method, we have achieved thevariances of all the collaborators to produce the study
data and the correct judgment results of each individualwith the Maddox rod: the precision inspection device.
We set the criterion by comparing each variance with the
Maddox rod test results. Table 1 indicates the evaluation
criteria. Forty-four collaborators contributed to thedevelopment of this criterion; the anomalies were 13 in
the right position and 36 in the presumed horizontal
strabismus or oblique position. The result was: 15
individuals had an angle of deviation in strabismus of 1Δ(prism), 15 had 2Δ, three had 3Δ, and three 4Δ. To
generate pseudo-perspective data on minimal cases ofstudy data for the development of a criterion for verticalstrabismus, we followed the specified procedure under
the guidance of an expert with the aid of 8 of the 13
collaborators listed above.
Apart from the main contents of the proposed detection
method that we mentioned, it was necessary todifferentiate between the blinking movement and to
exclude the corresponding system from the assessment
objective to automate the proposed judgment processcompletely. And specify the frame when the shutter
opens and closes. We also automated these processes andmodified the automation of the entire detection process.

In the previous analysis, we used the real automatic
Cover-Test apparatus (Fig.1-(B)) to assess theabnormality of constant eye movements. We found it
possible to evaluate anomaly by checking the eye
movements seen continuously in the same direction, inthe photographs taken at 0.2-second intervals for around
7 to 8 seconds immediately after the cover/uncover:
specifically, the 4th to 10th pictures. Using the pupil
center location at the start of the test as a reference point,
we found that the distance difference between thisreference point and the pupil center obtained at each time
point during the test estimate the degree of abnormality.
Our initial assumption was that the degree of theabnormality would correspond with the absolute distanceof eye movement.
However, the findings of the study did not show any
correlation. The gap between the eyes and the location of
the infrared camera differs between subjects. From theabove, we set the measurement parameter based on thevariance of the distance from the reference point of thepupil center and carried out an abnormality judgment
based on these evaluation criteria.
Setting evaluation criteria
We achieve a series of transitions of the pupil center
coordinates 𝑐𝑐𝑐𝑐 𝑗𝑗 = (x(𝑐𝑐𝑐𝑐 𝑗𝑗 )，y(𝑐𝑐𝑐𝑐 𝑗𝑗 )) from a series ofimage data (referred to as a test image) taken from aroundthe eye. (1≦𝑗𝑗≦n and 𝑗𝑗 is the serial number sequence
assigned to the test image.) Next, we set the central
coordinate 𝑐𝑐𝑐𝑐1 taken as the reference point from the firstsheet of a series of test images. The distance from thereference point of the center coordinates 𝑐𝑐𝑐𝑐 𝑗𝑗 (2 ≤ 𝑗𝑗 ≤ n)on the second and subsequent sheets is measured inhorizontal and vertical directions.We obtain the average
horizontal direction 𝑎𝑎𝑥𝑥 and the average vertical direction
𝑎𝑎 𝑦𝑦 , and the variances: s2x , s2y .
1

s2x = 𝑛𝑛 �
1

𝑛𝑛

𝑛𝑛=1
𝑛𝑛

𝑎𝑎𝑥𝑥 = �1𝑛𝑛s2y = 𝑛𝑛 �1𝑛𝑛𝑎𝑎𝑦𝑦 = �𝑛𝑛2

��|𝑥𝑥�𝑐𝑐𝑐𝑐𝑗𝑗 � − 𝑥𝑥(𝑐𝑐𝑐𝑐1 )|� − 𝑎𝑎𝑥𝑥 � (2)�|𝑥𝑥�𝑐𝑐𝑐𝑐𝑗𝑗 � − 𝑥𝑥(𝑐𝑐𝑐𝑐1 )|�

𝑗𝑗=1

𝑛𝑛=1
𝑛𝑛

Table.1. evaluation criteria of abnormal

4.

Verification experiment4.1. Experiment 1
First, we performed an anomaly judgment using thismethod for all 20 subjects, compared the judgment
results with those of the Maddox rod test, and assessed
the system performance by the coincidence rate. We
evaluated the outcomes in the following three stages.

2

��|𝑦𝑦�𝑐𝑐𝑐𝑐𝑗𝑗 � − 𝑦𝑦(𝑐𝑐𝑐𝑐1 )|� − 𝑎𝑎𝑦𝑦 � (3)�|𝑦𝑦�𝑐𝑐𝑐𝑐𝑗𝑗 � − 𝑦𝑦(𝑐𝑐𝑐𝑐1 )|�

𝑗𝑗=1© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

167

Noriyuki Uchida, Kayoko Takatuka, Hisaaki Yamaba, Masayuki Mukunoki, Naonobu Okazaki

Deviation angles of strabismus larger than 1Δ andsmaller than 3Δ were abnormal, those larger than 3Δ
were highly abnormal, while those less than 1Δ werenormal.
Table 2-(1) displays the results of the analysis. The
notation form is that V and H provide the direction ofeye movement, and the numerical value after V and H
represents the angle of strabismus. For example, V0 of
V0H6 indicates normality in the vertical direction, andH6 shows that there is a horizontal
strabismus/heterophoria of about 6Δ. H0-1 means thatthe horizontal anomalies are less than 1Δ. Table 2-(1)shows a case, subject B, in which the resulting judgment
differs from the correct result of the Maddox rod test
resulting in a concordance rate of 95 percent (19/20).

and the pupil was substantial. We did not accurately
identify the area near the center of the pupil, which
could have led to a wrong decision. We will deal withthis problem by taking photos of the eyes from the front.
As a consequence, the orientation and angle of thecamera is the concern.
Experiment 2
As shown in Table 2, the difference between theoutcomes of all 4 cases {D', J', M', T'} that did not fit
the right device judgment was around 1Δ, which was
subtle. We have grouped the causes into three patterns.
As in Case B in Experiment 1, we will deal with one outof the four cases by changing the angle of the shootingof the camera.Two cases were irrelevant because they
could not construct pseudo strabismus results. The causeof the remaining was a distortion of the pupil contourwith the use of hard contact lenses, which we
considered irrelevant to this system. Based on theabove, we assessed the actual failure to be one out offour.

4.2. Experiment 2
As mentioned at the beginning, this study aimed todevelop a method to detect misalignment in horizontal
and vertical eye movements. However, of the 20
subjects, only one case was abnormal in the verticaldirection, which did not verify whether we achieved the
purpose of this study. Therefore, using pseudo
strabismus data, we performed further experimental
verification.
With the cooperation with 15 {A, C, D, E, E, F, G, H, I,
K, L, N, O, S, T} of the 20 subjects whose system
judgment result in Experiment 1 was Normal,
abnormality degree smaller than 1Δ, we used 20 data
(A’ to T’) randomly selected from the pseudostrabismus data - 8 x 15 data: pseudo-strabismus datafor 15 subjects, four types (1Δ, 2Δ, 3Δ, 4Δ) each in thevertical and horizontal directions - as test data. We
performed an abnormality judgment and compared itwith the True Value expected when creating pseudo
data.
Table 2-(2) shows the judgment results. Four {D', J', M',
T'} out of 20 cases differed from the expected correct
judgment by the system, and the concordance figure
was 75 percent (15/20).

Table 2. Results(1) Experiment 1(2) Experiment 2
True Value（Expected
subject value）

True Value System
subject （Maddox） results

4.3. Discussion
Experiment 1
As shown in Table 2, the test results of subject B fromour system were highly abnormal in both the verticaland horizontal directions. The accurate findings of the
Maddox rod test were consistent with the results of thehorizontal direction, which were abnormally high, while
the results of the vertical direction revealed a significant
difference. What we found from the subject B test
image was that the overlap between the lower eyelidA

normalnormal

A’

V4H0

B

V0H6

VH H.Ab.

B’

V2H0

V Ab.

C

normalnormal

C’

V0H2

H Ab.

V H.Ab.

D

normalnormalD’

V0H2

VH Ab.

E

normalnormalE’

V3H0

V H.Ab.

F

normalnormalF’

V3H0

V H.Ab.

G

V0H0-1

normal

G’

V2H0

V Ab.

H

normalnormal

H’

V0H1

H Ab.

I

normalnormal

I’

V1H0

V Ab.

J

V0H0-1

normalJ’

V0H2

VH Ab.
V H.Ab.

K

normalnormal

K’

V4H0

L

normalnormal

L’

V0H1

H Ab.

M

VH0-1

normalM’

V1H0

normalN

normalnormal

N’

V0H2

H Ab.

O

normalnormalO’

V3H0

V H.Ab.
V Ab.

P

V0H5

H H.Ab.

P’

V1H0

Q

normalnormal

Q’

V0H1

H Ab.

R

V3H12

VH H.Ab.

R’

V1H0

V Ab.

S

normalnormal

S’

V2H0

V Ab.

T

normalnormalT’

V2H0

VH Ab.

V: Vertical , H: Horizontal
Ab.: Abnormal ,H.Ab.: Highly Abnormal© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

168

Systemresults

Method for detecting eye

5. Summary
To automate the strabismus/heterophoria screening, wehave developed an alternative method that can detect not
only horizontal but also vertical eye movements.
It is less susceptible to eyelids and eyelashes that make it
crucial to recognize the vertical flow of the eye.
Specifically, it only senses the motion of the center of theeye where eyelids and eyelashes cannot intervene. With
this system, we measured the distance variance from thereference point of each pupil center position and thenused it as an assessment criterion to devise a method for
detecting anomalies. We have put in place an inspection
framework based on the proposed approach and carriedout a performance assessment test. First, we set 44
standard inspection data, four pseudo-data forms ×
13=52, and 96 research data, and evaluation criteria.
We then carried out an anomaly judgment on 40 test data,
which is 20 inspection data and 20 pseudo data, andcompared the results with the right ones to test theefficiency of the proposed system. As a consequence ofthe experiment, the success rate of judgment on the datais 85 percent (34/40). However, we found four among thesix failures irrelevant to this method because three werefalse pseudo data, and one was for the use of hard contactlenses. Therefore, the number of actual judgment failuresof this system was two out of 40 cases, and the successrate was 95 percent (38/40). From the above, we defined
the problems with the pupil center detection error for the
infrared camera angle of the photograph and explained
the future theme. [Acknowledgment] This work was
supported by JSPS KAKENHI Grant Number
JP19H00505

5.

6.

7.

International Symposium on Artificial Life and Robotics(AROB), 2018.
N. Uchida, K. Takatuka, K. Hinokuma, K. Hirata, H.
Yamaba, N. Okazaki, HMD-based Cover Test System forthe Diagnosis of Ocular Malalignment, Artificial Life andRobotics (AROB), Vol.24-3, pp332-337, 2019.
A. Oohira, Excyclotorsion and Eye Dominancy in Agerelated Small-angle Vertical Strabismus, Journal ofJapanese Ophthalmological Society 123(2) pp.161-166,
2019
K. Yokota, K. Murata, Y. Sugaya, K. Kanatani, Accuracy
Comparison of Ellipse Fitting: From Least Squares toHyper-Renormalization, IPSJ SIG Technical Report111(379) pp.123-130, 2012

6. References
1.
2.

3.

4.

S. Hasebe, Objective Tests of Strabismic Deviation :Basics and Perspective, Journal of the eye 18(9),pp.1105-1110, 2001
Y. Takahashi, Y. Shinomiya, T. Noguchi, T. Niida, Studyof the occlusion time required for stable fusion-free
position by using an eye-tracking system, Folia Japonica
de Ophthalmologica Clinica 9(3), pp.234-237, 2016
Y. Yamauchi,T. Yamashita, H. Fujiyoshi, Human
Detection Based on Statistical Learning from Image,IEICE TRANSACTIONS Vol.J96-D, No.9, pp.20172040,2013
N. Uchida, K. Takatuka, K. Hinokuma, K. Hirata, H.
Yamaba, N. Okazaki , Automated cover-uncover test
system using active LCD shutter glasses, 23rd
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

169

Basic Research on Video Production for Educational Support by Virtual Technology
Kodai Miyamoto, Taketo kamasaka, Kenji Sakoma, Makoto Sakamoto, Amane Takei

Faculty of Engineering, University of Miyazaki,
Miyazaki-City, Miyazaki, Japan *Tsutomu Ito

National Institute of Technology, Ube College,
Ube-City, Yamaguchi, Japan
Takao Ito

Guraduate School of Engineering, Hiroshima University,
Higashi-Hiroshima, Hiroshima, Japan
E-mail: hm16043@student-u.ac.jp, hm16011@student-u.ac.jp, hm15037@student-u.ac.jp,
fruits2000@yahoo.co.jp*Corresponding Author

Abstract
As a result of conducting a questionnaire about science classes to high school students in 2016, the percentage ofhigh school students who answered "I like science" and "Science is important" is lower than other subjects. However,more than 80% of elementary and junior high school students said they like experiments and observations. In addition,
the 2019 smartphone penetration rate survey found that it is popular among about 90% of students. In addition, VR
technology has recently made remarkable progress. From the above, I researched the idea that creating a simulation
application using VR technology using smartphones would change the way high school students think about scienceclasses. In this paper, we have developed a simulation application for science experiments. Subjects were asked toexperience the newly created app and complete a questionnaire. As a result, the average score is 4 out of 5 and it is
not bad. But at the same time, a problem was found. The problem was that this app was a simulation app, so there
wasn't much user operability, so I wanted a little more operability. I want to make apps in other fields while improving
the problem.
Keywords: Education, science, chemistry, physics, experiment / observation, virtual reality, simulation app

elementary and junior high school students who
answered "I like experiments and observations" exceeds
80%.
A survey of smartphone penetration in 2019 found thatabout 90% of students have it. Most schools have PC
classrooms. In addition, each classroom has a personal
computer for teachers. Under these circumstances, it is

1. Introduction
In the materials related to science surveyed by the
Science Working Group in 2016, it was found that thepercentage of high school students who answered "I likestudying science" and "It is important to study science"is lower than other subjects. However, the percentage of© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

170

Kodai Miyamoto, Taketo kamasaka, Kenji Sakoma, Makoto Sakamoto, Amane Takei, Tsutomu Ito, Takeo Itothought that the educational effect can be further
improved in various classes.
Based on the above, the purpose of this research is tocontribute to education by creating a science simulation
application using VR technology.

Fig. 2. Created projectile motion app

2.2.2 Falling exercise
I made a slope that controls the falling speed by rolling the ballwhile changing the angle of the slope. Determine the angle oftilt and press Start to start spinning the sphere. The tilt angle,
ball speed, and ball position are displayed.

Fig. 1. Example of an English conversation learning app that
supports VR

2. Physics experiment app
In this study, we tried to create physics simulations of"projectile motion (projectile motion)" and "falling body
motion". Since these experiments require large
experimental tools and it may be difficult to obtainaccurate numerical values, we thought that they would besuitable as experiments to be performed in a virtualenvironment.
2.1. Development environment
This survey was conducted in the environment shown in
Table 1.
operating system

Windows10

Programming language

C#

Fig. 3. Created Falling exercise app

3. Chemistry experiment app
In this study, we tried to make prototypes of chemical
experiment simulations of "flame reaction" and "silver
mirror reaction". Since these experiments use manysolutions and experimental tools, it may be difficult toprepare and clean up, and since the experiments take time,
we thought that they were suitable for solving theshortage of class time.

software
Unity 2019.2.15f1
Table. 1. Development environment

2.2. Implemented function
In this study, we implemented the following two physics
experiment simulations.
1. Projectile motion
2. Falling exercise

3.1. Development environment
This survey was conducted in the environment shown in
Table 2.

2.2.1. Projectile motion
Projectile motion moves from the initial position.
After deciding the angle and speed and pressing Start
Button, the object will be fired and the distance will be
displayed. If it is difficult to check the current situation,
you can zoom in and out on the ball with the zoom below.

operating system

Windows10

Programming language

C#

software

Unity 2019.2.15f1

Blender 2.81
Table. 2. Development environment© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

171

Basic Research on Video

3.2. Implemented function

 Easy to understand the flow of the experiment
There were opinions such as.

In this study, we implemented the following two
simulations of chemical experiments.
1. Flame reaction
2. Silver mirror reaction

 Improvements“Physics experiment”
 Display of the distance of the object that wasskipped one before
 Change the trajectory of the ball from a point to aline
“Chemical experiment”
 Addition of tutorial Change the touched part to make it easier tounderstand
There were opinions such as.

Fig. 4. Created scientific experiment simulation app

Questionnaire results

4. Evaluation experiment5

An evaluation experiment was conducted to verify
whether the developed science experiment simulation
app was useful. We conducted a questionnaire to 10
people, including students in the laboratory, andevaluated the usefulness and usability of the system. The
contents of the questionnaire are the following four
points.

4.5

4.4

4.4

Evaluation
item 1

Evaluation
item 2

4.8

4
3.5
3

Evaluation item 1: Was it easy to operate?
Evaluation item 2: Was the result of the experiment easy
to understand?
Evaluation item 3: Was it easy to imagine a physicalphenomenon?
Evaluation item 4: Opinions and points to be improved(free description)

Evaluation
item 3Fig. 5. Questionnaire average score

Evaluation items 1 to 3 will be evaluated on a scale of 1
to 5 points, and evaluation item 4 will be freely described.
4.1. Experimental result
The figure below shows the average of the evaluation
results for evaluation items 1 to 3.
The average score for evaluation item 1 and evaluation
item 2 was 4.4 points.
The average score for evaluation item 3 is 4.8 points,
which is higher than that of evaluation item 1 andevaluation item 2.

Fig. 6. Examples of questionnaire improvement points

5. Consideration
Regarding evaluation item 4, there were many opinions
about the expansion of functions. Extension to VR was
also mentioned as an extension of the function. With
regard to expansion to VR, effects such as immersive
feeling and easy image of physical phenomena can be
expected. However, since VR-specific equipment isrequired, I think we must devise ways to make it easy foranyone to use.

For evaluation item 4
 good point“Physics experiment”
 Easy to understand the trajectory of the sphere“Chemical experiment”© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

172

Kodai Miyamoto, Taketo kamasaka, Kenji Sakoma, Makoto Sakamoto, Amane Takei, Tsutomu Ito, Takeo Ito

6. Future tasks
If we can create a simulation app that is similar to theteaching materials that students usually handle, we can
support education not only in science but also in various
subjects. Also, before creating an app, you need to think
about what kind of app can support student education.
This time, it could not be expanded to VR. If it is possible
to expand to VR, it will be possible to realize a more
immersive feeling by actually expressing the sound andscience room during the experiment, so this is a futureissue.
7. Summary
As mentioned at the beginning, VR technology has
grown remarkably in recent years. In this research, we
tried to create an application that can perform science
experiments using virtual reality, but as mentioned infuture tasks, special equipment is required to reproduce
VR. It has become a trial of only CG images. However,from the results of the evaluation experiment, it was
found that a useful application was created. Ultimately, I
think it is to create a system that will help support
education.

Fig. 7. Extension to VR

References

1. Curriculum Subcommittee Science Working Group"Materials on Science" , February 5, 2016.
2. National Academic Achievement / Learning Situation
Survey , 2015.
3. "[Latest version] The smartphone penetration rate in 2019is released to the public by age group, region, and age
group! It's a smartphone-only era! I predicted how
marketing should change in the future." Marketing
Research Camp.
4. Aimi Kitamura "Unity 5 Textbook 2D & 3D Smartphone
Game Introductory Course", SB Creative Co., Ltd., 2016.
5. "AEON provides VR English conversation app ..."
Hospitality "simulated experience improves ability".

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

173

There is a movement towards the development of hula costume CAD
Taketo Kamasaka, Kenji Sakoma, Kodai Miyamoto, Makoto Sakamoto*, Amane Takei

Faculty of Engineering, University of Miyazaki,
Miyazaki-City, Miyazaki, Japan
Tsutomu Ito

National Institute of Technology, Ube College
Ube-City, Yamaguchi, Japan
Takao Ito

Graduate School of Engineering, Hiroshima University
Higashi-Hiroshima, Hiroshima, Japan
E-mail: hm16011@student.miyazaki-u.ac.jp, hm15037@student.miyazaki-u.ac.jp, hm16043@student.miyazaki-u.ac.jp,
fruits2000jp@yahoo.co.jp*Corresponding Author

Abstract
In recent years, three-dimensional computer graphics (3DCG) technology has been applied in various fields. One ofthe problems in this technology is the problem of contact between clothes and other objects (such as the body). In
this paper, we verified each simple animation using the three methods of calculation. As a result of the verification,
the FB Euler method creates a simple but stable animation Was completed. In this paper, we were able to create
simple animations.

The Modeling methods for expressing cloth can beroughly divided into two types: geometric models andphysical models.
In order to perform dynamic simulation, we will use aphysical model that considers the change process thistime.

1. Introduction
In recent years, 3D computer graphics (3DCG)
technology has been applied in various fields such as AR/ VR technology, movies, games, and virtual fitting ofclothes1.
In this research, we will use a physical model to simulate
cloth, which is applied to 3DCG animation.
In this research, we focus on past research and work on asimulation of a simple cloth (for example, a skirt) thatconsiders contact with other objects (the body of a huladancer)2,3,4.
2. Simulation method© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

174

Taketo Kamasaka, Kenji Sakoma, Kodai Miyamoto, Makoto Sakamoto, Amane Takei, Tsutomu Ito, and Takeo Ito

2.1. Mass spring model

2.5. Numerical calculation method

In the mass spring model, in order to express theanisotropy of the cloth, the shape of the cloth is a springthat connects the mass points arranged in a square gridand the mass points adjacent to each other with thedirection along the warp and weft threads of the cloth as
the axis. (See Fig. 1.).

In this study, Euler method, FB Euler method, andRunge-Kutta method were used from the explicit
formulas6.
2.5.1. Euler method
Assuming that the mass point moves linearly withconstant acceleration between time 𝑡𝑡 and time 𝑡𝑡 + ∆𝑡𝑡
the velocity 𝒗𝒗𝒊𝒊 and the coodinates 𝒙𝒙𝑖𝑖 can be calculatedby the following equations using the force acting on theparticles calculated in Section 2.2.
𝒗𝒗𝑖𝑖 (𝑡𝑡 + ∆𝑡𝑡) = 𝒗𝒗𝑖𝑖 (𝑡𝑡) + 𝒂𝒂𝑖𝑖 (𝑡𝑡)∆𝑡𝑡(3)𝒙𝒙𝑖𝑖 (𝑡𝑡 + ∆𝑡𝑡) = 𝒙𝒙𝑖𝑖 (𝑡𝑡) + 𝒂𝒂𝑖𝑖 (𝑡𝑡)∆𝑡𝑡.
(∆𝑡𝑡: time step)

Fig. 1. Mass spring model

2.5.2. FB Euler method

It is a numerical calculation method that uses both the
Euler method and the backward Euler method at thesame time. Velocity applies the Euler method for
current acceleration, and position applies the backward
Euler method for future speed.
𝒗𝒗𝑖𝑖 (𝑡𝑡 + ∆𝑡𝑡) = 𝒗𝒗𝑖𝑖 (𝑡𝑡) + 𝒂𝒂𝑖𝑖 (𝑡𝑡)∆𝑡𝑡𝒙𝒙𝑖𝑖 (𝑡𝑡 + ∆𝑡𝑡) = 𝒙𝒙𝑖𝑖 (𝑡𝑡) + 𝒂𝒂𝑖𝑖 (𝑡𝑡)∆𝑡𝑡.
(4)(∆𝑡𝑡: time step)

2.2. Mechanical properties of cloth
Of the six mechanical properties of cloth listed by KES,
four are considered when modeling the shape of cloth:tensile properties, shear properties, bending properties,
and weight properties (See Fig. 2.)5.

2.5.3. Runge-Kutta method
At first glance, the amount of calculation seems to befour times, but since the calculation accuracy is high,the time step width Δt can be increased, and more stable
simulation is possible.
𝑘𝑘𝒗𝒗𝟏𝟏 = 𝒂𝒂𝟏𝟏 𝛥𝛥𝛥𝛥, 𝑘𝑘𝒙𝒙𝟏𝟏 = 𝒗𝒗(𝑡𝑡)𝛥𝛥𝛥𝛥

Fig. 2. Mechanical propeties of cloth

2.3. Power on mass points
The resultant force 𝑭𝑭𝒊𝒊 (𝒕𝒕) of the force applied to theparticle 𝑖𝑖 at time 𝑡𝑡 is as follows from the external force
such as wind force 𝒇𝒇𝒘𝒘𝒘𝒘𝒘𝒘𝒘𝒘 and gravity 𝒇𝒇𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈 and theforce generated from the internal structure of the clothsuch as the damping force 𝒇𝒇𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅 against the vibrationof the spring and the force 𝒇𝒇𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔 by the spring. Can be
expressed in𝑭𝑭𝒊𝒊 (𝑡𝑡) = 𝒇𝒇𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈 + 𝒇𝒇𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅𝒅 + 𝒇𝒇𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔 + 𝒇𝒇𝒘𝒘𝒘𝒘𝒘𝒘𝒘𝒘

𝒂𝒂𝟐𝟐 =𝐹𝐹 �𝒙𝒙(𝑡𝑡) +𝑘𝑘𝒙𝒙𝟏𝟏�
2�

𝐹𝐹 �𝒙𝒙(𝑡𝑡) +𝑘𝑘𝒙𝒙𝟐𝟐�
2�

𝐹𝐹 �𝒙𝒙(𝑡𝑡) +𝑘𝑘𝒙𝒙𝟑𝟑�
2�

𝑚𝑚− 𝑐𝑐𝒗𝒗(𝑡𝑡) + 𝒈𝒈

𝑘𝑘𝒗𝒗𝟐𝟐 = 𝒂𝒂𝟐𝟐 𝛥𝛥𝛥𝛥, 𝑘𝑘𝒙𝒙𝟐𝟐 = �𝒗𝒗(𝑡𝑡) +𝒂𝒂𝟑𝟑 == 𝑚𝑚𝑖𝑖 𝒈𝒈 + 𝒓𝒓𝒊𝒊 (𝑡𝑡) + 𝒇𝒇𝒊𝒊 (𝑡𝑡) + 𝛼𝛼 cos(𝛽𝛽) sin(𝛾𝛾) + 𝛼𝛼 sin(𝛽𝛽) sin(𝛾𝛾) . (1)2.4. Equation of motion

Based on the equation of motion, the acceleration 𝒂𝒂𝒊𝒊 (𝑡𝑡)is expressed by the following equation.
(2)
𝒂𝒂𝒊𝒊 (𝑡𝑡) = 𝑭𝑭𝑖𝑖 (𝑡𝑡)⁄𝑚𝑚𝑖𝑖 .

𝑚𝑚

𝑘𝑘𝒗𝒗𝟏𝟏�
2� 𝛥𝛥𝛥𝛥− 𝑐𝑐𝒗𝒗(𝑡𝑡) + 𝒈𝒈

𝑘𝑘𝒗𝒗𝟑𝟑 = 𝒂𝒂𝟑𝟑 𝛥𝛥𝛥𝛥, 𝑘𝑘𝒙𝒙𝟑𝟑 = �𝒗𝒗(𝑡𝑡) +𝑘𝑘𝒗𝒗𝟐𝟐�
2� 𝛥𝛥𝛥𝛥− 𝑐𝑐𝒗𝒗(𝑡𝑡) + 𝒈𝒈
𝑚𝑚
𝑘𝑘𝒗𝒗𝟒𝟒 = 𝒂𝒂𝟒𝟒 𝛥𝛥𝛥𝛥, 𝑘𝑘𝒙𝒙𝟒𝟒 = (𝒗𝒗(𝑡𝑡) + 𝑘𝑘𝒗𝒗𝟑𝟑 )𝛥𝛥𝛥𝛥
11
11𝒗𝒗(𝑡𝑡 + 𝛥𝛥𝛥𝛥) = 𝒗𝒗(𝑡𝑡) + � 𝑘𝑘𝒗𝒗𝟏𝟏 + 𝑘𝑘𝒗𝒗𝟐𝟐 + 𝑘𝑘𝒗𝒗𝟑𝟑 + 𝑘𝑘𝒗𝒗𝟒𝟒 �
6
3
3
6
1
1
1
1
𝒙𝒙(𝑡𝑡 + 𝛥𝛥𝛥𝛥) = 𝒙𝒙(𝑡𝑡) + � 𝑘𝑘𝒙𝒙𝟏𝟏 + 𝑘𝑘𝒙𝒙𝟐𝟐 + 𝑘𝑘𝒙𝒙𝟑𝟑 + 𝑘𝑘𝒙𝒙𝟒𝟒 � . (5)6336𝒂𝒂𝟒𝟒 =(∆𝑡𝑡: time step)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

175

There is a movement

The motion of mass points in the virtual world can be
simulated by these equations.

ends. Let the point 𝑥𝑥 for collision determination be theintersection of the perpendicular and the line segment
drawn from the point 𝑥𝑥 to the line segment: the point 𝑦𝑦
is obtained. At this time, the coordinates 𝑎𝑎 and 𝑏𝑏 at bothends of the line segment are set, and since the point 𝑦𝑦 ison the line passing through the coordinates 𝑎𝑎 and 𝑏𝑏, it isexpressed by the following equation.
𝒚𝒚 = 𝒂𝒂 + 𝑠𝑠(𝒃𝒃 − 𝒂𝒂).
(6)(𝒂𝒂, 𝒃𝒃, 𝒚𝒚: 3D vector, 𝑠𝑠: Scalar)
Since the perpendicular line is orthogonal to this line, itis necessary to satisfy the following equation.
(𝒚𝒚 − 𝒙𝒙) ∙ (𝒃𝒃 − 𝒂𝒂) = 0.
(7)
(𝒙𝒙: 3D vector)
The only unknown is 𝑠𝑠, and when 𝑠𝑠 is found, the point 𝑦𝑦
is found, and it is possible to determine the collision
between the cylinder and the foot. At this time, if thepoint 𝑦𝑦 exists on the line segment and the distance 𝑟𝑟 ′between the point 𝑥𝑥 and the point 𝑦𝑦 is smaller than 𝑟𝑟, it
can be seen that the point 𝑦𝑦 collides with the cylinder.
Since 𝑟𝑟 − 𝑟𝑟 ′ is the distance from the surface of thecylinder to the point 𝑥𝑥, this value is calculated using thepenalty method.
If it is found that the cylinder does not collide, then thecollision between the point and the sphere isdetermined. The collision between a point and a spherecan be determined by examining whether the distance
between the points 𝑎𝑎 and 𝑏𝑏 and the point 𝑥𝑥 is within 𝑟𝑟.
Collision determination between a capsule and a sphere(radius 𝑟𝑟 " ) is based on the same principle, and the above
𝑟𝑟 is replaced with 𝑟𝑟 + 𝑟𝑟 " for calculation anddetermination.

2.6. Collision with other objects
In this study, we consider the collision between thecloth and the body. Since a large number of mass points
and the body must be judged, a penalty method thattakes a simple calculation is used. This time, the masspoint is regarded as a point and the bone is regarded as acapsule, and collision judgment is performed.
2.6.1. Expression of the body
A capsule is a cylinder with hemispheres attached toboth ends, defined by a line segment and radius 𝑟𝑟, and is
the part within the distance and radius 𝑟𝑟 from this line
segment.
Calculating the position of the capsule with the back,hips, thighs, and shin bones as the axis of the cylinder
did not perform collision detection correctly(see Fig.
3.). Therefore, this time, in order to increase the numberof capsules and improve the accuracy of collision
detection, the back and shin bones are divided into two,and the thigh bones are divided into three.
2.6.2. Penalty method

3. Simulation environment
This program was created using the DirectX SDK, which
is a standard CG library, in a program development
environment using the C / C ++ language. Microsoft
Visual Studio Community 2019 was used to create theprogram this time. It ran on Intel Core i7 (Windows 10
Pro).

Fig. 3. Execution result by FB
Euler method

The penalty method is one of the rigid body simulation
methods. When objects collide with each other and aslight sunk occurs, a force (penalty force) proportional
to the sunk value can be applied to obtain the behavior
after the collision (See Fig.3.).
From this, it is possible to push back the lattice points
colliding with the cloth to the outside of the body by theforce of the spring so as not to be sunk into the body.

4.

Simulation result

2.6.3. Collision detection with capsule point
The reason why capsules are used for collision detectionis that the judgment only needs to be replaced. The
collision between the capsule and the point is divided
into a columnar part and a hemispherical part at both
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

176

Taketo Kamasaka, Kenji Sakoma, Kodai Miyamoto, Makoto Sakamoto, Amane Takei, Tsutomu Ito, and Takeo Ito•

The FB Euler method is used as the numerical calculation
method. The simulation results are shown in the figure
below (See Fig. 4.).

6. Consideration
The purpose of this study was to dynamically express
the cloth and to try a simple animation considering thecontact between the body and the skirt with reference topast studies.
This study, the completed animation is a simple
animation, the number of quality points is small, andthere are many parts that cannot be implemented such as
texture mapping and texture expression of cloth. It is
still difficult to do a realistic clothing simulation.
Furthermore, we must pay attention to how to put
clothes on 3D characters, how to express the difference
in cloth materials, how to judge collisions with higher
accuracy, and how to calculate less.
In addition, including the improvement measures
mentioned above, we will work on CAD production thatcan simulate various movements of hula and dress
design, etc., and express more realistic cloth, which will
be a practical research. I want to aim for that.

Fig. 4. Execution result by FB Euler method using the
proposed method

Simulations were performed using the Euler method andthe Runge-Kutta method. The Euler method resulted in
the cloth moving violently and moving away from thebody (See Fig. 5.). In the Runge-Kutta method, themovement of the animation became heavy, and the result
was that the contact between the foot and the skirt caused
the animation to dig in (See Fig. 6.).
In addition, all the calculation methods resulted in the
animation moving being heavier than the execution result
when the bones were not divided.

References
1.

2.
3.
Fig. 6. Execution result by
Fig. 5. Execution result by Euler
Runge-Kutta method using themethod using the proposed
proposed method
method4.

5. Consideration

5.

As a result of the simulation, the collision judgment
considering the contact between the skirt and the
body was stable by the FB Euler method as
mentioned in the execution result of Section 4, and asimple animation could be performed. However,there are some issues:•••

Examination of a method with a small amount ofcalculation

6.

Zorah Lahnar, Daniel Cremers and Tony Tung
DeepWrinkles: Accurate and Realistic Clothing
Modeling, 15th European Conference on Computer
Vision, 2018, pp. 1-18.
Ayumi Taniue, Attempt to create cloth animation with
OpenGL, Memoirs of the Faculty of Engineering,
University of Miyazaki, 2007, pp. 299-304．
Seira Nagatomo, Attempt of cross simulation by massspring model, Memoirs of the Faculty of Engineering,
University of Miyazaki, 2009, pp. 327-332．
Minaho Sakumoto, Attempt of cloth animationconsidering contact with an object, Memoirs of theFaculty of Engineering, University of Miyazaki, 2011,
223-227.
Mari Inoue, Research on the Mechanics of Fibrous
Materials, SEN’I GAKKAISHI(Textiles and industry).
Vol. 64. No.8, ,2008, pp.246-251．
Mikihiro Satou, Research on interactive cloth
simulation, Outline of the 37th Academic Lecture,
Faculty of Industrial Science and Technology, Nihon
University. No.20, 2004, pp.23-26．

The parameters must be changed manually.
Examination of numerical calculation method andcontact force calculation method by implicit
solution method.
Examining the method of texture mapping to theskirt.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

177

Analysis and Construction of Elements of
the Stage Performance Structure in a Kabuki-dance
Miku Kawai

Faculty of Software and Information Science, Iwate Prefectural University,
152-52, Sugo, Takizawa, Iwate, 020-0693, Japan
Jumpei Ono

Faculty of Software and Information Technology, Aomori University,
2-10-1, Seishincho, Edogawa, Tokyo, 134-0087, Japan
Takashi Ogata

Faculty of Software and Information Science, Iwate Prefectural University,
152-52, Sugo, Takizawa, Iwate, 020-0693, Japang031p035@s.iwate-pu.ac.jp, j.ono@aomori-u.ac.jp
ogata@iwate-pu.ac.jp

Abstract
We conducted a detailed analysis of the stage performance structure of Kyōganoko Musume Dōjōji and reproduced
it on the animation tool KOSERUBE with the music, lyrics, and images aligned. In this paper, 11 scenes ofKyōganoko Musume Dōjōji are organized. The actual flow is compared with the movement on the system. The goalof this paper is to understand the entire stage structure of Kyōganoko Musume Dōjōji in detail. In addition, thisresearch aims to use the stage performance structural system for an integrated narrative system that the authors aredeveloping. We believe that this system will be the basis for the construction of the Kabuki stage. Further, we want
to apply it not only to Kabuki but also to other genres, such as computer games and commercials.
Keywords: Dōjōji, Stage Performance Structure, Narrative generation, Story

COVID-19, a new performance form called Zūmu
Kabuki was created.
Zūmu Kabuki is a Kabuki stage in which each actor
takes a video on the “Zoom” video calling app, and thevideos are put together to form a single stage. This is also
known as online Kabuki. In fact, from the end of June tothe middle of July, the Zūmu Kabuki Chūshingura was
performed. In this way, Kabuki continues to develop by
utilizing artificial intelligence and digital technology.
For example, last year, Hatsune Miku and Nakamura
Shido co-starred at the Nico Nico Chōkaigi * , where a
Kabuki called “Hana kurabe sen bon zakura” was

1. Introduction
Kabuki is a valuable cultural heritage, one of theelements of Japanese culture with a long history.
Therefore, many studies have been conducted regarding
the preservation and inheritance of such a heritage. In
addition, Kabuki incorporates various performing arts
from the genres of dance and classical performance, suchas Noh and Kyogen. In recent years, we have worked
actively to incorporate video technologies such as
Computer Graphics. Additionally, in the vortex of*
A festival organized by DWANGO Co. Ltd. The festival has thecontent “Chō Kabuki” (https://chokabuki.jp/2019minamiza/english/).
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

178

Miku Kawai, Jumpei Ono, Takashi Ogata

Maiden at the Dōjōji Temple) is one work in many works
based on the “legend of Dōjōji,” told in narratives andpicture scrolls. This is a latter-day description of thelegend of Anchin and Kiyohime. The oldest legend of the
Dōjōji work is Dainihonkoku Hokekyō Genki(Miraculous Tales of the Lotus Sutra from Japan).1 In thestory, Anchin lies and is burned to death by Kiyohime;
however, the power of the Lotus Sutra frees him from
suffering.
We have conducted the following research so far.

performed. This is a stage that combines the world viewof Hatsune Miku’s representative song Senbonzakurawith Kabuki (Minamiza, Kyoto, Aug. 2019). In addition,
this year, the Ningyo Joruri Koimusume Tsumugi no
Labyrince, using Hatsune Miku, is being held at the
Tomioka Silk Mill (Tomioka, Gunma, Japan/Oct. 2020).
In Kabuki research, there have been many studies of
the introduction of video technology. The heritage of
Shosagoto (the body actions of actors) using motion
capture has been analyzed in some studies. However, ourresearch is different.
･ Oda and Genda study the characteristic motion in
Kabuki using a motion capture system. Omoto et al.
create a model of Minamiza in Kyōto (Minamiza is a
traditional Kabuki theatre).1･A study (Kobayashi, 2017) compares Western dance inthe ballet La sylphide, and the Japanese dance
Kyōganoko Musume Dōjōji to investigate the creation ofdances. This research focuses on “Furi <swing>,” mainly
analyzing small movements such as walking and hand
expressions.2･Using a commercially available robot (Desktop robot“Premade AI”), the stage of Ningyō Jōruri Hidakagawa
Iriaizakura is reproduced remotely by the Tokyo
University of Technology.3
We investigated and analyzed several of the elementsthat make up Kabuki and developed a system thatsimulates the interplay of stories. We use a system called
KOSERUBE that partially reproduces the visual imageof the stage performance structure of Kyōganoko
Musume Dōjōji. In addition, we have conducted thefollowing research so far.





1.
2.

3.
4.

As a study of Kabuki-dance: Analysis of
Kyōganoko Musume Dōjōji.4
Analyze the stage performance structure of
Kyōganoko Musume Dōjōji: We have developed aperformance structure reproduction system.5
A combination of Unchiku, an explanation
generation system, and an animation system.
Story generation related to love and sexuality using
a Japanese classic tale.

We extracted the components of the story for use in
our explanation generation system for a Kabuki-dance
representation system.
KOSERUBE7, 8 is an animation tool developed by theauthors. We develop a system that reproduces the stage
performance structure of Kyōganoko Musume Dōjōji
with KOSERUBE. This research aims to automate thereproduction system of the stage performance structure
and connect the system to story generation.9 To that end,
this paper focuses on the dance scenes of Kyōganoko
Musume Dōjōji and analyzes their stage elements.
Kyōganoko Musume Dōjōji focuses on dance, but it
involves scenes with narratives. However, in this study,
we focus only on the dance scenes. Therefore, we havenot analyzed all the scenes in Kyōganoko Musume Dōjōji.

We have surveyed and analyzed Dōjōji stories andKyōganoko Musume Dōjōji in Kabuki.4
We have developed a system that simulates thestage performance structure of Kyōganoko Musume
Dōjōji.5
We have integrated narrative generation systems,
including explanation generation, with the above
simulation system.6

2. Analysis of Kyōganoko Musume Dōjōji
Dōjōji is a temple of the Tendai sect in Hidakachō,
Hidaka, Wakayama, Japan. This temple is the subject of
the legend of Dōjōji, which has been recorded for over a
thousand years. Although this legend has changed its
shape over the long history, we think it is still known tomany people. Of the versions of this legend, we focus on
the Kabuki dance Kyōganoko Musume Dōjōji, which isconsidered the legend of modern Dōjōji.

The current study focuses on the stage structure andaims to display the entire stage. Specifically, the purpose
is to perform a detailed analysis of the stage structure andreproduce the stage performance structure. Therefore,
this study does not necessarily analyze the body actionsof the actors in detail. Kyōganoko Musume Dōjōji (The© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

179

Analysis and Construction of

2.1. Kyōganoko Musume Dōjōji

uses the actual music because there are no scores for“Michiyuki,” “Ranbyōshi,” and “Kaneiri.” In otherscenes, we only reproduced the music for the shamisen.
Since the sound of the shamisen music feels higher thanthat on the actual stage, there is a lingering sense of
discomfort compared with the actual music;(3) [Lyrics] Excerpts from the beginning and end of the
lyrics (only the “Ranbyōshi” is shown in full). The lyrics
are romanized because they are written in old Japanese;(4) [Images] Images of appearing characters, the mainstage, and stage equipment. The props are listed inparentheses in the “Shirabyōshi Hanako”;(5) [Commands] The command is written in the code.
Commands used in the system (input sentences
instructing the reproduction system to create thecharacter animation or the background).

Kyōganoko Musume Dōjōji is a representative work ofKabuki dance, known as a work played by representative
female onnagata such as Nakamura Utaemon 6th andTamasaburo Bando 5th. The ancient legend that led to the
Kabuki dance Kyōganoko Musume Dōjōji wastransmitted through Buddhist narratives, Noh, texts,
picture scrolls, etc.
Kyōganoko Musume Dōjōji is a work that is a later
development of the legend of Anchin and Kiyohime.
Unlike the Kiyohime legend, which depicts a woman’sobsession, she does not bring it to the fore in this version,using various kimonos and props to show her daughter inlove, making it a completed work of Musume Dōjōji as a
Kabuki dance. That is, it basically inherits the form of the
Noh Dōjōji intact, but the legend of Dōjōji is not
specifically shown in the verse. The theme is to show achanging dance.
The foundation of Dōjōji dance had already been laidin Noh’s Dōjōji, and in Kyōganoko Musume Dōjōji it istransformed into a gorgeous and bright work, positioned
as the completion of the modern transformation.

3.

An Animation Mechanism based on the
Structural Analysis of Scenes

This section describes the generation of stage
performance structures based on our analysis of
Kyōganoko Musume Dōjōji. As we mentioned in Section
1, our goal is to create a system that automaticallygenerates a staging structure from certain inputs. This
study is a preliminary phase. To study the system for
automatic generation, we created an animation from thecontent shown in Table 1. First, this section describes ananimation tool. Next, it shows the flow of creating an
animation.
This study uses the user interface of KOSERUBE,
which has two modules. The first module converts thegenerated stories into animation code specific to
KOSERUBE, based on which the second module plays
the animation.
In this study, we used the second module in the userinterface and the date base of images. Moreover,additional music and images were prepared for the
animation of Kyōganoko Musume Dōjōji. We manuallycreated the animation code based on the analysis shownin Table 1. Next, we described how to create theanimation code.
Figure 1 shows an example of an animation. The
system reproduces a performance by combining a
background and a character image on another image that
imitates the play’s stage. In addition, the music in
Kyōganoko Musume Dōjōji, along with the text showing
the story and lyrics of the music, are displayed at thebottom of the screen. Our research is not on computer
graphics. Rather, the goal is to reproduce the elements

2.2. The legend of Dōjōji
The story of the legend of Dōjōji (also known as thelegend of Anchin and Kiyohime) is Kiyohime, a woman
living in Kumano who is betrayed by a Buddhist monk
(Anchin). Her strong anger transforms her into a snake,and she burns the monk together with a bell in Dōjōji.
The legend of Dōjōji took a male point of view of awoman as evil, but since the Muromachi period, it haschanged to the point of view of a woman’s sadness.
2.3. Analysis of Kyōganoko Musume Dōjōji
Table 1 (following the references) shows the elementsof Kyōganoko Musume Dōjōji and their implementation
methods. The table includes the following five items foreach scene:(1) [Actual stages] Flow and characteristics of the danceon stage (the contents in brackets indicate scenecharacteristics);(2) [Music] Music used in the system, based on the
Kineie Shamisen score, and created with a freecomposition
tool(“Wagaku
Hitosuji,”
https://jonkara.net/soft/wagaku/). However, the system© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

180

Miku Kawai, Jumpei Ono, Takashi Ogata

that make up the entire stage. For this reason, the paper
does not pursue reality and the animations that we createdare below par.

layers: text, character, stage, and background. Here, thedancing figures are the characters, and the
instrumentalists, monks, and bells are included in the
background.
The system uses two kinds of music. First, the musicof the actual Kyōganoko Musume Dōjōji and second, theelectronic music created from a musical score. The
musical score used was Nagauta Kyōganoko Musume
Dōjōji: Shamisen Bunka-fu by Kineie.10 To create themusic, we used a free composition tool called “Wagaku
Hitosuji.”
Next, we explain the details of the animation code.
The animation code is a set of instructions to the userinterface describing the combination of images and texts,
detailed instructions for character animation, and thetiming of playing music. The user interface section
dynamically composes images, plays animations, andplays music based on the animation code.
Figure 3 is an example of character animation basedon the animation code. The code represents an animationthat makes the character jump in Figure 3. The “mov”
command in Figure 3 is a primitive command for thecharacter’s image to move. We used multiple “moves” torepresent the dances of the characters. For example,“mov, 11, v, −50, −25” means, from left to right,“Command,” “Character ID,” “Direction of movement,”“movement distance,” and “movement speed.” Forexample, “mov, 11, v, −50, −25” moves “Shirabyōshi
Hanako,” with a character ID of 11. The movement
distance is the −50 coordinate in the vertical direction
relative to the current coordinates. The movement speed
is −25 (every frame, the coordinates of the image move
relative to the current coordinates by −25 pixels).
In Figure 3, the coordinates of the character image are
moved up and down by multiple “moves.” The “Code”
in Figure 3 is a combination of “mov” for moving up and“mov” for moving down repeated three times, so that
“Shirabyōshi Hanako” jumps off three times.

Fig. 1. Example of a visualized stage performance structure(Ranbyōshi).

Fig. 2. Example of a visualized stage performance structure(Michiyuki).

The screen structure shown in Figures 1 and 2 uses
background and human images stored in the image
database of KOSERUBE. The screen consists of four

Code

Animation

mov, 11, v, −50, −25mov, 11, v, 50, 25
mov, 11, v, −50, −25mov, 11, v, 50, 25
mov, 11, v, −50, −25mov, 11, v, 50, 25
Fig. 3. An Example of Code for Visualizing Animation.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

181

4.

4.3. Future Work: Parallel Generation System

Discussion

As mentioned in Section 2, the story of the legend of
Dōjōji has changed during its long history. In addition,
the character of Kiyohime has changed. Therefore, we
reproduced the character of Kiyohime, which changes
with the transformation of the story, with 2D and 3D
animation.
In the legend of Dōjōji, the outline of the story is thesame, but there are turning points in the story, such as thescene where Kiyohime turns into a snake and the way it
ends. Scenes from the legend of Dōjōji, such as theturning point and turning into a snake, were reproducedwith 2D animation and 3D animation. The goal of thissystem is for all scenes to be seamlessly represented
using CG.
In particular, based on branches, many story scenes
are generated in parallel, using the story units of thelegend of Dōjōji. The system can also make one story line
by selecting a possibility at each branch point.
Figure 4 shows an image of the prototype system,11in which the 3D and 2D generation results for 16 scenes
flow. In the first and last scenes, a part of the 2D
animation system is played for about 30 seconds. In the
2D animation system, it represents a scene of Kyōganoko
Musume Dōjōji.
The flow of the system starts from the first scene,
after which the story is divided into multiple scenes. The
scene expressed in 3D is in the Hokkekyō Genki12 andKonjaku Monogatari,13 containing the narratives of thelegend of Dōjōji and the story of Dōjōji Engi. Then,
returning to the first scene, the story ends.
In the scene where only the code is displayed, only
the structure of the generated story is flowing because 3D
has not been created yet.

The future potential of the animation system used in thispaper and how to use it are described below.
4.1. Results
We have analyzed the stage performance structure of
Kyōganoko Musume Dōjōji and reproduced it withanimation. In addition, the story of the legend of Dōjōji
is organized and summarized in a table. We analyzed andorganized the stage performance structure, including
such elements as musical instruments and stage
backgrounds, and created animation using KOSERUBE.
We also analyze the history and story of the legend of
Dōjōji, focusing on the transformation of the story fromthe legend of Dōjōji to Kyōganoko Musume Dōjōji andthe personality of the characters.
The future of the animation system and the prototype
system using the story of Dōjōji are described below.
4.2. Future Work: Combination of INGS and theAnimation System
The authors believe that this system will be the basis forcreating the Kabuki stage. Figure 4 is an image of thesystem. The system outputs the stage performance
structure by automatically generating the stage
components based on the input story. The story enteredin the system is prepared based on two methods, creationby hand and automatic generation. We have developed asystem that automatically generates stories, theIntegrated Narrative Generation System (INGS). We are
studying a system that automatically generates the stage
performance structure by combining the reproduction
system with INGS. Finally, we want to apply the system
not only to Kabuki but also to other genres, such as
computer games and commercials.

Fig. 4. Schematic of the stage structure reproduction system.
Fig. 5. Images of stage structure reproduction system.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

182

Miku Kawai, Jumpei Ono, Takashi Ogata

5.

Conclusion

In this paper, 11 scenes of Kyōganoko Musume Dōjōji areorganized to provide a detailed description of the entire
stage structure of a Kabuki dance performance. In thisstudy, we conduct a detailed analysis of the stage
performance structure of Kyōganoko Musume Dōjōji andreproduce it with the animation tool KOSERUBE byaligning the music, lyrics, and images. The purpose ofthis analysis is to develop a system for reproduction of
the stage performance structure of the dance. In the future,
we will study its generation method based on the resultsof the analysis and develop an approach to build a moredetailed reproduction system.

8.

9.

10.
11.

Acknowledgements
The research for this chapter was supported by the Japan
Society for the Promotion of Science (JSPS KAKENHI),
Grant No. 18K18509.

12.
13.

References

Grammar Transactions, Proc. of the Japanese Society for
Artificial Intelligence, 2012.
S. Imabuchi, T. Akimoto, J. Ono and T. Ogata,
KOSERUBE: An Application System with a Propp-based
Story Grammar and Other Narrative Generation
Techniques, Proc. of the Symposium on Cryptography andInformation Security, 2012.
S. Imabuchi, T. Akimoto, J. Ono and T. Ogata,
KOSERUBE: An Application System with a Propp-based
Story Grammar and Other Narrative Generation
Techniques, Proc. of the Symposium on Cryptography andInformation Security, 2012.
Kineie Yashichi IV, Nagauta Kyōganoko Musume Dōjōji:
Shamisen Bunka Fu Hōgakusha Beppu, Oita, Japan, 1952,
Feb. 2018, pp. 174-177.
M. Kawai, J. Ono and T. Ogata, Dual Story Generation
Based on Love and Extreme Emotions, Proceedings of2020 The 3rd International Conference on Computational
Intelligence and Intelligent Systems, 2020.
M. Inoue and S. Ōsone (Collation Eds.), Dainipponkoku
Hokkekyō Genki, Ōjōden Hokkegenki, Iwanami Shoten,
Tokyo, 1974, pp. 43-219.
K. Mabuchi, F. Kunisaki and Toru Konno,
Konjalkumonogatarisyu vol.1., Shogakukan, 1919.

1. N. Omoto, K. Hasegawa, H. Motojima, S. Nagata and S.
Tanaka, 3D model of Kyoto Minami-za theater and its
application Prof, IPSJ SIG Computers and the Humanities,
2004, pp. 109-112.
2. Kobayashi. S, Approach to choreography “Kiyohime”: acomparative study of Japanese traditional dance andwestern ballet for female (Doctoral dissertation), Osaka
University of Arts, 2018.
3. R. Dong, Y. Chen, D. Cai, S. Nakagawa, T. Higaki and N.
Asai, Robot motion design using bunraku emotional
expressions: focusing on Jo-Ha-Kyū in sounds andmovement, Advanced Robotics, 34(5), 2019, pp.299-312.
4. M. Kawai, J. Ono and T. Ogata, Analysis and Generationof the Narrative Elements in a “Dojoji” Story. Proceedingsof the 3rd International Conference on Computational
Intelligence and Intelligent Systems, 2020, pp.44-45.
5. M. Kawai, J. Ono and T. Ogata, Analyzing the Stage
Performance Structure of a Kabuki-Dance, Kyōganoko
Musume Dōjōji, Using an Animation System, In E. Ishita,
N. L. S Pang & L. Zhou (Eds.) Digital Libraries at Timesof Massive Societal Transition (LNCS 12504), pp. 248254.Switzerland: Springer Nature, 2020.
6. J. Ono, M. Kawai and T. Ogata, Implementation of an
Explanation Generation Mechanism Using Attribute
Frames and a Noun Conceptual Dictionary. Proceedingsof the 3rd International Conference on Computational
Intelligence and Intelligent Systems, 2020, pp.48-50.
7. S. Imabuchi, J. Ono, J. Endo, Y. Kurisawa, R. Michimata,
M. Kamada, T. Akimoto and T. Ogata, KOSERUBE: Folk
Tales of Iwate: An Integrated Generation System of
Language/Music/Image Based on Propp-based Story© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

183

Table 1. Stage performance structure.

Scene nameMichiyuki

Actual stages
Shoke walks from
Hanamichi．
Talk about Kane’s
memorial service onstage. A stage device
imitating the door to thetemple will be preparedon the stage.
The narrator and theshamisen player comeout and sit on a longtable.
Shirabyōshi Hanako
comes out of Hanamichi
with a fan.
Perform a dance alongwith Shirabyōshi.
After the dance, head tothe stage.
Put on the Eboshi given
by Shoke. Move fromthe stage to Hanamichi
according to Noh. Show
off the dance. After that,
return to the stage from
Hanamichi. Perform adance at the center of thestage (Mai has manysmall movements ofwrists, feet, and props.)
Continuing fromRanbyōshi, perform adance at the center of thestage in tune withKanedukushi.

Music
Existingmusic(Jōruri)

Lyrics“Tsuki hahodonaku
hairu Shiono(omission)
Hirari bōshino fuwafuwa
to”

Images・Characters: Shoke (12 people)
・Katarite
・Musicians: Shamisen [Japanese guitar]
・Place on stage: Main stage, Hanamichi・Stage equipment: Dōjōji, Kane, Cherry
blossoms

Existing
music
(Noh)

“Shidokenari
furi ā
hazukashi ya(omission)
Dōjōji ni koso
tsukinikere”

Existing
music
(Noh)

“Hana nohoka ni hamatsu bakari
hana no hoka
ni ha matsu
bakarikuresometekane yahibikuran”

・Characters: Shirabyoushi Hanako,
Shoke (12 people)
・Katarite
・Musicians: Shamisen [Japanese guitar]
・Place on stage: Main stage, Hanamichi・Stage equipment: Dōjōji, Kane, Cherry
blossoms
・Characters: Shirabyōshi Hanako, Shoke(12 people)
・Katarite
・Musicians: Hyōshiita
・Place on stage: Main stage, Hanamichi・Stage equipment: Dōjōji, Kane, Cherry
blossoms

Createfromscore(Nagauta)“Kane niurami hakazukazugozaru (omission)
Shinnyo no
tsuki wonagameakasan”

Teodori

Reappear on the stage
without props. Perform
at the center of the stage
along with Hauta (Shōfu
no Erothishizumu).

Createfromscore(Nagauta)“Iwazukataranu
waga kokoro(omission)
Miyakosodac
hi wa
hasuppanamono ja e”

Mariuta

Perform dance at thecenter of stage inaccordance with Rōdzukushi (Yūjo)(choreography likeplaying with Kakū noMari).

Createfromscore(Nagauta)“Koi no wakesato
(omission)
Omoi
sometaga en
ja e”

Hanakasaodori

Put Hanakasa on thehead and hands at thecenter of the stage.
Perform the dance
according to
Wakitebushi.
Shirabyōshi Hanako anddisappears from thestage.

Createfromscore(Nagauta)“Ume to-sansan sakura wa(omission)
Kawairashi-sa
no hanako”

Ranbyōshi

Chūkeinomai

All Shoke line up onstage with Hanakasa.
Show off the dance.

・Characters: Shirabyōshi Hanako (Have
Chūkei), Shoke (12people)・Katarite
・Musicians: Shamisen [Japanese guitar],
Taiko [Drum]・Place on stage: Main stage
・Stage equipment: Dōjōji, Kane, Cherry
blossoms
・Characters: Shirabyōshi Hanako, Shoke(12 people)
・Katarite
・Musicians: Shamisen [Japanese guitar],
Taiko [Drum], Kodutumi,
Fue [Flute]・Place on stage: Main stage
・Stage equipment: Dōjōji, Kane, Cherry
blossoms
・Characters: Shirabyōshi Hanako, Shoke(12 people)
・Katarite
・Musicians: Shamisen [Japanese guitar],
Taiko [Drum], Kodutumi,
Fue [Flute], Kane
・Place on stage: Main stage
・Stage equipment: Dōjōji, Kane, Cherry
blossoms
・Characters: Shirabyōshi Hanako(Hanakasa), Shoke (12people)・Katarite
・Musicians: Shamisen [Japanese guitar],
Taiko [Drum], Fue [Flute],
Kane・Place on stage: Main stage
・Stage equipment: Dōjōji, Kane, Cherry
blossoms
・Characters: Shoke (12 people)
・Katarite
・Musicians: Shamisen [Japanese guitar],
Taiko [Drum], Fue [Flute],
Kane・Place on stage: Main stage

Commands
Not
reproduced at
the time ofthis paper.

“Hana”
commandmakes it look
like he isgoing backand forthbetween thestage andHanamichi.
Not much
movement,
many small
dances.

Not much
movement,
many small
dances.

By repeating
the verticalmovement, it
is possible toexpress that aball isattached.
The character
does notmove muchbecause themovement of
Hanakasa isthe mainmovement.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

184

Miku Kawai, Jumpei Ono, Takashi Ogata
Scene name

Actual stages

Music

LyricsImages・Stage equipment: Dōjōji, Kane, Cherry
blossoms

Commands

KudokiAppear with Tenugui atthe center of the stage.
Show off Mai along with
Koi to Shitto no Kouta.

Createfromscore(Nagauta)“Koi notenarai tsuiminaraite(omission)
Sawarabaochinfuzeinari”

・Characters: Shirabyōshi Hanako(Tenugui), Shoke (12people)・Katarite
・Musicians: Shamisen [Japanese guitar],
Taiko [Drum], Kodutumi
・Place on stage: Main stage
・Stage equipment: Dōjōji, Kane, Cherry
blossoms

Yamadukushi

Appear with Kakko
close to the center of thestage. Show Mai while
hitting Kakko according
to Yama-Zukushi.

Createfromscore(Nagauta)“Omoshiroshikinonagameya(omission)
Tsuki nokaobaseMikasayama”

Teodori

Reappear on the stage
with nothing. Perform at
the center of the stage
along with Hauta (Shōfu
no Erotishizumu).

Createfromscore(Nagauta)“Tada tanome(omission)
Niku terashi
hodo
itoshirashi”

Shuzudaiko

Around the center of thestage, hit Suzudaiko andperform a dance alongwith taueuta (Saotome
odori).

Createfromscore(Nagauta)“Hana nikokoro ofukamigusa(omission)
Hiki kadezuite zo use
ni keru”

Kaneiri

Shirabyōshi Hanako
defeats all Shoke. Kane
comes down on stage.
Shirabyōshi Hanako
climbs on Kane andgazes around her like asnake.

・Characters: Shirabyōshi Hanako(Kakko), Shoke (12people)・Katarite
・Musicians: Shamisen [Japanese guitar],
Taiko [Drum], Kodutumi,
Hyōshiita, Kakko
・Place on stage: Main stage
・Stage equipment: Dōjōji, Kane, Cherry
blossoms
・Characters: Shirabyōshi Hanako, Shoke(12 people)
・Katarite
・Musicians: Shamisen [Japanese guitar],
Taiko [Drum], Fue [Flute]・Place on stage: Main stage
・Stage equipment: Dōjōji, Kane, Cherry
blossoms
・Characters: Shirabyōshi Hanako(Suzudaiko), Shoke (12people)・Katarite
・Musicians: Shamisen [Japanese guitar],
Taiko [Drum], Kodutumi,
Hyōshiita, Suzudaiko・Place on stage: Main stage
・Stage equipment: Dōjōji, Kane, Cherry
blossoms
・Characters: Shirabyōshi Hanako, Shoke(12 people)
・Katarite
・Musicians: Shamisen [Japanese guitar],Suzudaiko・Place on stage: Main stage
・Stage equipment: Dōjōji, Kane, Cherry
blossoms

Tenugui iswrappedaround the
body or heldin the mouth.
In this way,
the characteris not moving
because thefocus is on
using props.
Rotational
movementsusing Kakko.

Not much
movement,
many small
dances.

Suzudaiko isdifficult toreproduce
because it
turns its wrist
and rings.

Not
implemented
because it is
not a dance
part．

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

185

Unchiku Generation with Moving Illustration Using Kabuki Knowledge
Jumpei Ono

Faculty of Software and Information Technology, Aomori University,
2-10-1, Seishincho, Edogawa, 134-0087, Japan
Miku Kawai

Faculty of Software and Information Science, Iwate Prefectural University,
152-52, Sugo, Takizawa, 020-0693, Japan
Takashi Ogata

Faculty of Software and Information Science, Iwate Prefectural University,
152-52, Sugo, Takizawa, 020-0693, Japan
E-mail: j.ono@aomori-u.ac.jp, g031p035@s.iwate-pu.ac.jp
t-ogata@iwate-pu.ac.jp

Abstract
We developed an explanation-generation system based on narratology. We also developed an unchiku generation
system as an application. After creating a prototype, we then considered a new system that would combine the unchiku
generation system with an animation system. We implemented the animation system in our kabuki study. This paper
considers the possibilities of the unchiku generation system.
Keywords: Kyōganoko Musume Dōjōji, moving illustration, narrative generation, unchiku.

development of the storyline. The unchiku is a form of
explanation that uses deep content to share knowledge orteach a lesson to a disinterested audience.
We also study kabuki narrative generation bycomprehensively examining the structure, methods, andtechniques of narrative in kabuki to refine our narrative
generation system.
In our research system of narrative generation,1,2 thekabuki form is central to the method of narrative
generation. We have placed the “work performed onstage” at the center and presented a list of its multiple
components,7 extracted representative kabuki methods
and techniques, and analyzed them in detail throughprototypes8,9 to further our narratological research in the
kabuki tradition.10
We focused our analysis specifically on the kabuki
dance Kyōganoko Musume Dōjōji,11–14 a later tale of thelegend of Dōjōji that has been performed by many

1. Introduction
Drawing on the foundations of narratology and literary
theory, systematized thought, and philosophy, wedeveloped a program that generates narratives.1,2 We thendesigned a mechanism for unchiku generation3,4 forintegration into the narrative generation system.
We study explanation generation in the context ofnarratology. Narrative means not just the progression of
a plot but also the surplus or excess exposition that caninterfere with its rapid progress. In Gérard Genette’s
theories on narrative discourse,5,6 such techniques—for
example, explanation and description—are seen as
halting the temporal progression of the narrative.
Explanation is the act of justifying various objects orevents in a story (e.g., characters, setting, physicalobjects, abstract ideas, etc.) and their relationships; it
often clarifies but remains separate from the progress and© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

186

Jumpei Ono, Miku Kawai, Takashi Ogata

excellent onnagata (male actors playing female
characters) since ancient times with new content added
beyond the original legend. In our earlier work13,14 weused a system we developed called KOSERUBE,15 an
animation tool for a story generation system, to create
visual representations of the structure of the stage
performance.
Other researchers have also built narrative generation
systems based on the work of Pablo Gervás16 as part ofthe broader computational creativity movement that
focuses on narrative applications in the field of
robotics—for example, developing socially assistive
robots with a narrative approach.17 Our particularresearch proposes a method of combining moving
illustrations on the screen as a tool to inform users ofunchiku. In this paper, story means the combined
storyline and explanation, and the unchiku is the detailed
information within the explanation.
There are many possible ways of using moving
illustrations. We previously described our approach toone of these methods.13,14 In this paper, we examine thatmethod’s opposite: using images as explanations of texts.
The overall concept of using pictures to explain the text
is not new—encyclopedias and dictionaries have done
this for centuries. However, this paper proposes the useof an automated animation tool to explain Kyōganoko
Musume Dōjōji text.
In describing our prototype, we will clarify the concept
and examine the issues and usage for the future. Our final
goal is to integrate the explanation using video and the
explanation of the video and to treat each as a function.

and critic, we conducted a detailed analysis of the “stage
performance structure” (narrative outline) of the play.
The main components were the characters, background(stage set), music (instruments, performers, and genres),lyrics, dialogue, and the core spiritual themes of eachscene.
Musume Dōjōji recounts the legend of Kiyohime andAnchin. In the legend, a young woman named Kiyohim
falls in love with Anchin, a Buddhist monk, who rejects
her. She turns into a snake and kills him. After more thana thousand years of telling this story through oral
histories, picture scrolls, and other reading materials, thestory has transformed into the present-day Kyōganoko
Musume Dōjōji. Our narrative generation system is basedon its Kabuki dance. We summarized the stage
performance structure of Kyōganoko Musume Dōjōji inan analysis table and created a storyboard to develop ouranimation system.
We have previously written about our analysis of thestage performance structure of Kyōganoko Musume
Dōjōji in detail11 and how we developed an animation
system to express it called KOSERUBE, a folktale-style
story generation system.15 With the system, users canselect folktales and characters, and it automaticallygenerates a story based on the selections. The system also
generates an animation for the generated story,
displaying the lyrics at the bottom of the screen; thebackground and characters change from scene to scene.
Our animation system can also reproduce and generatemusic12 based on the score of Syamisen Bunkahu
Nagauta Kyōganoko Musume Dōjōji19 and recorded
music stored in the system.20 The prototype system uses
the analysis of Kyōganoko Musume Dōjōji, the story of
Dōjōji Enki, and the attributes of the characters. Details
are given in Section 3.1.

2. Two Background Systems
This section describes the animation system of the
Kyōganoko Musume Dōjōji and the unchiku generation
system. The modules of the prototype system use these
two systems.

2.2. The Unchiku Generation System
The second system is an unchiku generation system.
Unchiku means deeply preserved knowledge. Unchiku
wo tareru (to tilt one’s unchiku) means putting all yourenergy into that knowledge. In recent years, unchiku isoften used to mean “trivia.” However, unchiku wo tareru(to draw upon one’s unchiku) and unchiku wo hikerakasu(to show off one’s unchiku) mean showing off knowledge
learned from undesirable situations. The story uses
unchiku as one of its rhetorical techniques. A story has arhetorical device that slow or stop the progress of the plot
in the story by, for example, pausing to describe
something in the story. For the characters in the story,“tilting one’s unchiku” is the act of explaining something

2.1. The Animation System of the Kyōganoko
Musume Dōjōji
The first system is an animation system that depicts thestage performance structure. The data on Kyōganoko
Musume Dōjōji used in this study came from our analysis
so far. First, we focused on three items in the stage
structure table—kokoro (core conceptual theme), furi(performance), and lyrics—that are described in Musume
Dōjōji.1 We analyzed the elements in the stage structure
table to build a storyboard.2
Referring to an analysis of Musume Dōjōji18 by
Tamotsu Watanabe, a contemporary kabuki researcher© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

187

Unchiku Generation with Moving(e.g., an object, character, setting, or event, etc.) in detail.
We have developed the unchiku generation system withthe aim of establishing a “starting point and means for
systematically collecting, accumulating, and utilizing
kabuki knowledge.”3,4
As mentioned earlier, unchiku is one rhetorical
technique used in Musume Dōjōji. We have previously
discussed various aspects of narrative discourse,
including the “distance” between the storyteller and thestory,6 based on the structuralist theories proposed by
Genette.5 We position unchiku as an explanation forexcess quantities verbosity. Originally, we extracted
information from Wikipedia to develop our unchiku
generation module,3,4 but we have refined our unchiku
generation in the prototype system. Details are given in
Section 3.2.

(乱拍子 [Ranbyōshi])(人 [Character] 白拍子花子 [Shirabyōshi Hanako])(心 [Core mental theme] 娘 [Young girl])(振り[Performance] 白拍子の舞 [Shirabyōshi no mai])(歌詞 [Music genre] 能[Noh])(楽器 [Instrument] 拍子板 [Hyōshi])(場面 [Scene] 桜の木（道成寺）[Cherry blossoms (Dōjōji)])(衣装 [Costume] 赤 [Red])(小道具 [Prop] 中啓 [Chūkei])(肉体のポイント[Focus point in the dance] 足 [Foot]))

Fig. 2. The unchiku knowledge of the Kyōganoko Musume
Dōjōji
年老いた僧と安珍が熊野に参詣へ行く。年老いた僧と安珍がある家に到着する。清姫が安珍に会う。清姫が安珍に惚れる。安珍は清姫を恐れる。安珍は「安珍と清姫が再会する」という嘘をつく。安珍はある家を過ぎる。清姫は嘘を知る。清姫は激怒する。清姫は蛇体に変身する。清姫は安珍を追う。安珍は道成寺に到着する。年老いた僧と安珍が熊野に参詣へ行く。年老いた僧と安珍がある家に到着する。清姫が安珍に会う。清姫が安珍に惚れる。安珍は清姫を恐れる。安珍は「安珍と清姫が再会する」という嘘をつく。安珍はある家を過ぎる。清姫は嘘を知る。清姫は激怒する。清姫は蛇体に変身する。清姫は安珍を追う。安珍は道成寺に到着する。[An old monk and a young monk, Anchin, go
to visit Kumano. They arrive at a house. Kiyohime meets Anchin.
Kiyohime falls in love with Anchin. Anchin is afraid of Kiyohime.
Anchin tells a lie that “Anchin and Kiyohime will meet again.”
Anchin avoids the house. Kiyohime knows a lie. Kiyohime gets
angry. Kiyohime transforms into a snake. Kiyohime chases Anchin.
Anchin arrives at Dōjōji Temple.]

3. Unchiku Generation with Moving Illustration
Combining the unchiku generation with the insertionof moving illustrations tells the story as animation. Our
goals for the prototype system were (1) to clarify thesystem image and (2) to analyze plans and issues for
future full-scale development. This section describes thedata structure and the system.
3.1. Knowledge Structure for the Proposed System
The prototype system limits the text and animationmaterial of the unchiku to Kyōganoko Musume Dōjōji.
For this reason, we have provided English supplements
to the actual data and generated examples for this paper.
Figures 1 and 2 show the knowledge for unchiku
generation: Figure 1 relates to elements in the story;
Figure 2 relates to the scene of Kyōganoko Musume
Dōjōji associated with the knowledge listed in Figure 1.
Figure 3 shows the story used in this paper.

Fig. 3. The Musume Dōjōji story

3.2. Prototype System
In the prototype system, the unchiku generation is
supplemented by inserting moving illustrations to
produce animation: the story and unchiku appear as text
with animation in the middle of the sentences associatedwith the words. Figure 4 shows the process of the system
and its two modules. The first module creates a story withthe unchiku. The other generates animations keyed toparts of the generated story. The system also contains atextual knowledge base and an image knowledge base.
The system receives a story, then animates the story andthe knowledge contained in it.

(道成寺『京鹿子娘道成寺』である。『京鹿子娘道成寺』とは歌舞伎舞踊の代表的な作品である。昭和の六代目中村歌右衛門
や五代目坂東玉三郎など、代表的な女形が演じる作品として知られている。古くからある伝説が、仏教説話や能、絵巻等の読み物を経て、歌舞伎舞踊の『京鹿子娘道成寺』に至る 。 [Kyōganoko Musume Dōjōji is a representative work of
Shosagoto. It is a work played by representative onnagata such as
Nakamura Utaemon VI and Bandō Tamasaburō V from the Showa
period. An old legend leads to the kabuki dance Kyōganoko Musume
Dōjōji through reading materials such as Buddhist narratives, Noh,and picture scrolls.]

Fig. 1. The unchiku knowledge of the Dōjōji
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

188

Jumpei Ono, Miku Kawai, Takashi Ogata

In the text representation, events are shown in blackand insights in red. An animation is inserted at the end of
the inscription.
Fig. 4. The flow of a unchiku generation system with moving
illustrations

4.

Discussion

Figure 4 depicts the unchiku with moving illustration
generated based on Figure 3. The unchiku is related toMusume Dōjōji. The content is as follows:

The knowledge generation module takes a story andadds to it from the knowledge database. The unchiku
generation module, using a story as its input, compareswords in the story with information in the knowledge
base, then generates an unchiku based on what it finds.
Figure 2 is an example of knowledge retrieval forunchiku generation from Musume Dōjōji. Figure 3 is an
input example showing how the module inserts theretrieved knowledge from Figure 2 into the Musume
Dōjōji story in Figure 3. We made two changes to thesystem presented in Section 2.1: one, the knowledge
generation module now adds a symbol to the output
sentence to clearly distinguish between events andknowledge, and two, it explicitly states that an animation
will be inserted immediately after the pronouncement.
The following is the flow of unchiku generation:

The story about Dōjōji is Kyōganoko Musume
Dōjōji. Kyōganoko musume dōjōji. It is a
representative work of Shosagoto. It is a work
played by representative onnagata such as
Nakamura Utaemon VI and Bandō Tamasaburō Vfrom the Showa period. An old legend leads to thekabuki dance Kyōganoko Musume Dōjōji throughreading materials such as Buddhist narratives, Noh,and picture scrolls. For example, the scene ofRanbyōshi is as follows: hito: Shirabyōshi Hanako;
kokoro: daughter; huri: Shirabyōshi no mai; lyrics:
Noh, gakki: hyousiita; scene: cherry tree (Dōjōji);ishō: red; kodougu: Chūkei, nikutai no; point: The
legs. Figure 5 represents Ranbyōshi.

1. The module receives a story from the user or story
generator.
2. The module searches the knowledge for elements
and keywords from the story to find relevant
knowledge.
3. The system generates text presenting the retrievedknowledge.
4. The system inserts the generated knowledge into
the story.
5. The system generates knowledge related to thegenerated knowledge.
6. The system inserts the generated knowledge into
the story.
7. The system specifies the animation associated withthe generated knowledge.
The knowledge retrieval and generation are repeateduntil there is no more relevant knowledge available. After
that, the inserted animation will be related to the lastgenerated knowledge.
The animation module receives the story from theunchiku generation module and generates a code foranimation appropriate to the story. The module expresses
the generated code and story as an animation as describedin Section 2.

Fig. 5. Unchiku generation with moving illustration

In the future, we have two challenges for the system.
First, we need to expand the Yunnan information we havein the knowledge base from our analysis of Kyōganoko© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

189

Unchiku Generation with Moving

Musume Dōjōji. 11,12,13,14 Second, we need to expand theanimations. We have created animations for Dōjōji,13,14
and our goal is to use those animations.
5.

8. T., Ogata, Kabuki ni mukete (2): Tajū monogatari kōzō noshosō [Toward kabuki (2): Aspects of multiple narrative
structures]. in Jōhō monogatariron: Jinkōchinō, ninchi,
shakai katei to monogatari seisei [Informational
narratology:
Artificial
intelligence/cognition/social
process and narrative generation], eds. T. Ogata, Y.
Kawamura and A. Kanai, Hakutō Shobō, Tokyo, Japan,
2018, pp. 209–244.
9. T. Ogata, Kabuki as Multiple Narrative Structures andNarrative Generation, in Post-Narratology ThroughComputational and Cognitive Approaches, eds. T. Ogata
and T. Akimoto, IGI Global, PA, USA, 2019, pp. 192–275.
10. T. Ogata, Kabuki as a synthetic narrative: Synthesis andexpansion. in Internal and external narrative generation
based on post-narratology: Emerging research andopportunities, T. Ogata, IGI Global, PA, USA, 2019, pp.
315-403.
11. M. Kawai and T. Ogata, An analysis of Kyōganoko
Musume Dōjōji for the generation of the staging structuresof Kabuki from narrative plots, SIG-LSE Proc., 63, 2019,
pp. 25–61.
12. M. Kawai and T. Ogata, Analyzing the stage performance
structure of Kabuki-dance “Kyōganoko Musume Dōjōji”,
Proc. of the 34th Annual Conference of the JapaneseSociety for Artificial Intelligence, 2020, 3D1-OS-22a-04.
13. M. Kawai, J. Ono and T Ogata, The Stage Structure of a
Kabuki-dance: Survey, Analysis, and Representation of
Kyoganoko Musume Dojoji, Proc. of the 37th Japanese
Cognitive Science Society, 2020, pp. 776–785.
14. M. Kawai, J. Ono and T. Ogata, Dual Story Generation
Based on Love and Extreme Emotions, Proceedings of2020 The 3rd International Conference on Computational
Intelligence and Intelligent Systems, 2020, pp. 9–10.
15. T. Akimoto, S. Imabuchi, J. Endo, J. Ono, Y. Kurisawa,
M. Kamada and T. Ogata, Development of a generation /expression system of narratives in the style of a folk tale,
KOSERUBE version 1, Transactions of the JapaneseSociety for Artificial Intelligence, 28(5), 2013, pp. 442–
456.
16. P. Gervás, Propp' s Morphology of the Folk Tale as a
Grammar for Generation, Workshop on Computational
Models of Narrative, 2013, pp. 106–122.
17. K. L. Koay, D. S. Syrdal, K. Dautenhahn, and M. L.
Walters, A narrative approach to human-robot interaction
prototyping for companion robots, Journal of Behavioral
Robotics, 11(1), 2020, pp. 66–85.
18. T. Watanabe, Musume Dōjōji, Shinshindo, Tokyo,1986.
19. Y. Kineie, Syamisenbunkahu nagauta Kyōganoko
Musume Dōjōji, Hougakusya, Tokyo, 1952.
20. Kyōganoko Musume Dōjōji, Bandō Tamasaburō butō shū
Vol.1 (DVD), Syotsiku Home Video, Tokyo, 2003.

Conclusion

We have developed an advanced explanationgeneration system with two functions using Musume
Dōjōji as our test model. First, the system inserts anunchiku explanation into a story. Second, it generates ananimation to explain the unchiku. These two functions
combine to deliver a more fully fleshed out explanationof the narrative.
Acknowledgement
This study was supported by JSPS KAKENHI,
18K18509.
References
1. T. Ogata, Toward an Integrated Approach to Narrative
Generation: Emerging Research and Opportunities, IGI
Global, PA, USA, 2020．
2. T. Ogata, Internal and External Narrative Generation
Based on Post-Narratology: Emerging Research andOpportunities, IGI Global, PA, USA, 2020．
3. J. Ono, K. Fukuda, M. Kawai, and T. Ogata., Considering
unchiku generation mechanism through system
implementation, Proc. of the 37th Annual Meeting ofJapanese Cognitive Science Society, Japan, Tokyo, 2020,
pp. 776–786.
4. J. Ono and T. Ogata, Unchiku Generation Using a
Narrative Explanation Mechanism. In E. Ishita, N. L. S
Pang & L. Zhou (Eds.) Digital Libraries at Times ofMassive Societal Transition (LNCS 12504), pp. 240–247.
Switzerland: Springer Nature.
5. G. Genette, Narrative discourse: An essay in method (J. E.
Lewin, Trans.), Cornell University Press, NY, 1972.
6. T. Akimoto, and T. Ogata, Towards a discourse
mechanism in narrative generation system: Proposal of a
System Introducing Narrative Discourse Theory andReception Theory, Cognitive Studies, 20(4), 2013,
pp. 396–420.
7. T. Ogata, Kabuki ni mukete (1): Shiisei to henshūsei nomonogatari kara tajūsei to jitsuzon no monogatari he[Toward kabuki (1): From narratives of arbitrariness andediting to narratives of multiplicity and real existence], in
Jōhō monogatariron: Jinkōchinō, ninchi, shakai katei tomonogatari seisei [Informational narratology: Artificial
intelligence/cognition/social process and narrative
generation], eds. T. Ogata, Y. Kawamura and A. Kanai,
Hakutō Shobō, Tokyo, Japan, 2018, pp. 187–208.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

190

Implementing Story Units of Japanese Folktales with Conceptual Dictionaries
Takuya Ito

Graduate School of Software and Information Science, Iwate Prefectural University, Sugo 152-52, Takizawa, Iwate,Japan
Jumpei Ono

Faculty of Software and Information Technology, Aomori University, Seishincho 2-10-1, Edogawaku, Tokyo, Japan
Takashi Ogata

Faculty of Software and Information Science, Iwate Prefectural University, Sugo 152-52, Takizawa, Iwate, Japan
E-mail: g231r003@s.iwate-pu.ac.jp, j.ono@aomori-u.ac.jp, t-ogata@iwate-pu.ac.jp

Abstract
This paper reports the development of story units that are knowledge representing story structure for story generation.
A story structure in our narrative generation study is formed by a tree that consists of events connected based on therelationships, such as the causal and temporal relations of events. We have been implemented story units based on
the “Type of Folktale” in the Nihon Mukashi-banashi Taisei (The Concluded Compilation of Japanese Folktales).
The current main theme is to use our conceptual dictionaries, including verb and noun conceptual dictionaries, tocombine the current version of story units to our “integrated narrative generation system,” which is a generalframework of our narrative generation study. In this paper, we explain the above mechanism.
Keywords: List four to six keywords which characterize the article.

Story units are in the prototype stage, and the verbs
representing events in the Story Unit were createdaccording to the descriptions of the folktales to beanalyzed, and do not correspond to the descriptions in theverb concept dictionary that INGS refers to whengenerating stories.
In this paper, we aim to map the verbs in the story
unit to the verb concept lexicon of INGS in order togenerate a story using the story unit. In this paper, we aim
to map the verbs in story unit to the verb concept
dictionary of INGS. Specifically, we examine thedescriptions of the verbs in story unit, and if the verbs instory unit can be mapped to the existing verb concepts inthe verb concept dictionary, we register the verbs in story
unit as the verb concepts of INGS. This correspondence
enables us to use the rich sentence generation mechanism

1. Introduction
In a study of folk tales and folktales from around theworld, Propp collected a genre of Russian folk tales
called magical folktales, and analyzed the tales
structurally in terms of commonalities among thecollected stories[1]. Propp called the structuralcommonalities "functions," which are related to theclassification of characters, and in this respect are like themain action of the main characters in the definition of
motifs.
This research is underpinned by narratology. We
analyze existing stories and propose story units forgenerating a story using the results of the analysis in the
Integrated Narrative Generation System (INGS), which
is being developed by the authors[2,3].

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

191

Takuya Ito, Jumpei Ono, Takashi Ogata

addition, these three classifications have “divisions” thatgroup folktales according to their motifs, with 11
divisions for “animal folktales,” 16 for “authentic
folktales,” and 12 for “funny stories” (a total of 39
divisions). Under each category, 827 types of folktales
are assigned. The three major categories are further
divided into 39 divisions. 827 types of folktale belong to
one of the divisions, and each division contains aminimum of 2 and a maximum of 66 types of folktale.

and about 110,000 noun concepts in INGS, and to
generate a variety of stories from a story unit.
2. Motif and Form in the study of folktales

Class
・・・

Animal
Tales

Class

Class

Class

Type offolktale

FolktaleType offolktale

FolktaleType offolktale・・・

・・・

Jokes and
Anecdotes

Folktale・・・

Ordinary
Folktales

・・・

Nihon
Mukashibanasi Taisei

Type offolktale・・・

A motif is a unit that contains one major action of amajor character in the composition of a story, as well as
actions that directly correspond to that action [3].
In addition to the research on motifs mentioned in
section 1, Aarne and Thompson collected folk tales
mainly from various parts of Europe and produced theInternational Folk Tale Abstracts [4,5] (compiled by
Aarne and expanded and revised by Thompson, so it iscalled the Aarne-Thompson type index (AT index)).
To analyze Aarne-Thompson's International Folktale
Abstracts, it was noted that there were differences in thecharacters and actions of the different categories offolktales in the International Folktale Abstracts.
There are several efforts by Seki. The folktales usedin this study are the efforts of Keigo Seki. The details aredescribed below.

FolktaleType offolktale

FolktaleFig. 1. The hierarchy of folktales in the “Nihon Mukashibanashi Taisei”

3. Seki’s Study of Folktales
In Nihon Mukashi-banashi Shusei (Collection ofJapanese Folktales), Seki classified about 8,700 folktales
collected in various parts of Japan. Later, Seki and hiscolleagues expanded the number of tales to about 35,000
and updated the classification in Nihon Mukashi-bahashi
Taisei (The Complete Japanese Folktales)[7-18]. Sekisummarized the contents of 827 “type of folktale,” whichwere the result of the classification. The 827 types offolktales based on the AT index[5,6]. The types offolktale is a classification of Japanese folktales as definedby Seki and a summary of their contents.
[Seki, 78a-e, 79a-d, 80a] has collected and classified
about 35,000 folktales from various parts of Japan in
Nihon Mukashi-banashi Taisei. There are 12 volumes of
Nihon mukashi-banashi Taisei in total, and volumes 1 to10 [7-16] contain the collected folktale themselves,volume 11 [17] contains materials on the type of folktale,
an index, and a bibliography, and volume 12 contains
contributed articles on folktales [18].
Each type has a summary of the contents of one ormore folktales corresponding to the type. These types offolktales are systematized, with the largest categories
being “animal folktales,” “authentic folktales,” and“funny stories,” which are based on the AT index. In

Figure 2 shows an example of a type of folktale. typeof folktale has four items.
 Serial number: The serial number for type offolktale (e.g., 一四二 (One hundred forty-two)) Title: Motifs representing the characteristics offolktales (e.g., 蛇息子 (The snake son)) AT Classification Number: These are the numbers
corresponding to the motifs in the AT index. Some
types of folktales do not have corresponding
numbers. (e.g., AT 四三三 B)
 Content: The structure of a type of folktale. It
consists of one or more sentences. The sentences arenumbered roughly by scene. It may also contain
branching patterns of events and patterns ofelements. For example, the structure is as shown in
Figure 3.
一四二 A 蛇息子 (AT 四三三 B)
1、子どものない夫婦が(a)神に祈願して蛇の子を生む。(b)卵を拾い、それから蛇の子が生まれる。2、あまりに大きくなりすぎて村人に恐れられるので山に捨てる。3、この蛇の援助によって(a)雨を降らせる。(b)田に雨がかかるようにする。(c)蛇が禍をなして爺婆に仇を討たれる、4、そのために爺婆は富を得る。

Fig. 2. An example of a type of a folktale© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

192

Implementing Story Units ofWolf and fox
1

Three little
fox become
independent from theirparents.

2

3

He
3rd foxsurvived.
makeshouse by
tails

Wolf
invadesfrom theroof.

Wolf falls
to hearth.
1st foxmakeshouse bystraw.

Wolfeats
the fox.

2nd foxmakeshouse bystones.

story units [19]. The created story units can be used tocreate folktale-like story. The story units can be used tocreate folktale-like story, and by combining the story
units, it is possible to create a story that is different fromthe original story.
Section 4.2 and 4.3 show the procedure for creatingstory units.
4.2 The Structure of Type of Folktales

Wolf isdead.

We first reconstructed the folktale type into a form
suitable for the program. We split the content of the typeof folktale so that an item contains a verb. Table 1 shows
a form of splitting.

Wolf
eats
the fox.

Fig. 3. An example of structure of a type of a folktale
The type of folktale is a structure that integrates the actual
collected folktales. Figure 4 shows an example of collected
folktales.



「昔、三匹の子狐がおった。一番兄狐が、もう一匹前になったからといって、親狐に藁から家をつくってもらった。
ところが、ある日、・・・」[Once upon a time, There are threelittle foxes. An eldest brother fox is presented a house that makes bystraw from their parent, because the eldest brother fox becomes anadult. But, one day, …]Serial number: Referring Figure.2 in the section3.
Title: Referring Figure.2 in the section3.
Content number: The specific content of the folktale
type is summarized by scene, with the events
contained therein.
Sentence number: Indicates a sentence break in typeof folktale.
Event: Events segmented by verb units

Fig. 4. An example of motif variation

4.3 Making Story Units

Each type of folktale has a serial number called ataisei number and a title that represents the mostcharacteristic motif of the content (the bold Chinese
numerals in the figure are taisei numbers).

To make a story unit, we create a case structure foreach of the “event” descriptions in Table 1.
This story unit consists of three scenes, each of whichis represented by A, B, and C. The events in each scene
are numbered, and there are six events in this story unit.
Each event is composed of a verb and its case information
based on the way events are described in INGS. The
descriptions of verbs and case elements in the Story Unit
are based on the descriptions in type of folktale, and do
not correspond to the verb and noun concepts in INGS.
However, in the case of verbs, a temporary number 1 is

4. First Step for Making Story Units
4.1 Story Units
A story unit is the knowledge that makes up a story.
There are many kinds of story units. For example, there
is a story unit that shows the structure of a large story. On
the other hand, there is a story unit that shows the local
structure of a story. We used "Type of folk tale" to create

Table 1. The example of a structure form of type of folktale
大成番号 モチーフ（昔話）

474

内容番号

鬼の面
(a)
(b)

文番号
事象（内容）①
ある女(男)が山(化け物屋敷)で鬼の面をかぶっている。
②
化け物がそれを見て逃げる。
③
女は宝物をとって
①
帰る、
&または&
①
化け物屋敷の主人になる。

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

193

Takuya Ito, Jumpei Ono, Takashi Ogata

5. Combining Story Units and a Verb
Conceptual Dictionary

written after the verb in order to correspond to the verb
concept.
The branching structure of the Type of folktale isrepresented by a description starting with "or", as shownin Figures 5.4a and 5.4b. In addition, there are branching
structures that affect the choice of subsequent branching
structures. In order to preserve the pairing relationship,
there are examples where several different story units
were created from the same type of folktale. Therefore,
there are 912 story units that we have created.
The Story Unit is created based on the description of
the type of Folk. In the case structure of verbs involving
a dialogue such as “say,” the content of the dialogue isinserted directly into the object case. In this case, theelements of the object case cannot be changed or
expanded without changing the content of the description.
Therefore, we consider a more flexible use of the story
unit by creating a nested structure of events.
Among the case elements of the story units, there are
761 cases that can be nested. Table 2 shows theclassification of nested structures.

We combine the story unit shown in section 4 with a verb
concept dictionary.
5.1 Goal of Combining Story Units and a Verb
Conceptual Dictionary
The combination of verb concept dictionaries provides
two advantages to story units. (1). Through the sentence
pattern of the verb concept, single sentence generationand sentence generation with conjunction in INGS will
be available, and the following functions will beavailable--“Ordering words,” “Transforming inflected
form,” “Transforming notation,” “Multi-sentence.” (2).
Story units can use the system structure of the verb
concept dictionary. For example, the story units canmanipulate events using the systematic structure of theverb concept dictionary, and to use the co-occurrence
information of the verb concepts associated with the verb
concept dictionary.
5.2 Verb Conceptual Dictionary

Table 2. The type of nested structure in the story units

Category
An action

Amount217

Actions

50

Avoid

1

Prohibition
ProposeDesireSpeak
DenialImpossibility
Command

15
10
33
4
29
3
68

Rewording

3

合計

433

A concept here is data that shows the meaning andclassification of a word. A concept dictionary is a kind of
ontology of words belonging to a certain part of speech.
In this section, two types of concept dictionaries are
explained. One is the verb concept dictionary, and theother is the noun concept dictionary. The verb concept
dictionary is a dictionary that systematically registers
verb concepts. The verb concept dictionary has about
10,000 verb concepts. A noun concept dictionary is asystematic dictionary of noun concepts. The Noun
Concept Dictionary has about 110,000 noun concepts.
Figure 6 shows an example of a verb concept. A verb
concept has a sentence pattern, a case frame, and aconstraint. Constraint describes the constraints for

Definition
Contents consisting of a single action
Contents consisting of multiple
actions
Contents that attempt to avoid acertain action.
Contents that prohibit certain actions
Contents of a proposal for an action
Contents that want to an action
Contents that characters speakContents that negate an action.
The impossibility of an action.
To command an action.
The action is difficult to understand,so it is replaced by a different verb
expression.

Fig. 5. An example of a story unit
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

194

Implementing Story Units ofsentence generation mechanism of INGS is used to
expand the sentence into a natural sentence, and if anatural sentence that matches the content of the Story
Unit is expanded, it is used as is. In case of no case, we
selected an appropriate verb concept number from theverb concept dictionary and changed the number of the
Story Unit verb to that number.

selecting a case frame from the noun concept dictionary,and is-a information for systematically classifying verbs.
5.3 The Method for Combining Story Units and a
Verb Conceptual Dictionary
This paper describes the binding of verbs contained instory units to a verb concept dictionary. We organized thetypes of verbs contained in story units and implemented
methods according to the classification.

5.4.3 (B). New registration
For the verbs in (B), we first checked whether there wereany verb concepts with the same meaning but different
notation in the verb concept dictionary, and if so, we
processed them as the task in (C). If there was no verb
with different notation, we registered it as a new verb
concept in the verb concept dictionary.
For each new verb, we created a sentence-pattern anda case-frame based on the case structure of the target verb
in the story unit. case-frame refers to the case structure of
the verb in the story unit, and case-frame refers to thecase of the target verb in the story unit. The sentencepattern was created by specifying the particle
corresponding to each case used in the case-frame, andcombining the "case number" and the "particle
corresponding to the case". In the tentative data, we did
not give any constraints on the cases to be used, and all
is-a to classify verb types were set to "physical action".
In this example, the case registered in the original story
unit is the agent case, which represents the subject, andthe object case, which represents the object.

5.4.1 Counting kinds of verbs in the story units
The 912 story units we created have 1107 different
verbs. These verbs can be divided into those that can beeasily combined with the verb concept dictionary andthose that require some work to be done in order to be
combined with the verb concept dictionary. First, weclassified the verbs in the story unit in terms of thechallenges in binding. Table 3 shows the breakdown of
the classification, the number of units corresponding to
the classification, and an example.

Table 3. The category of task in the story units
Task

AmountExample(A). Direct connect

615

食べる

(B). New registration

363

あさる

(C). Notation

121

あきらめる

(D). Passive form

13

追われる

(E). Negative form

5

承知しない

Total

1117

5.4.4 (C). Notation
This problem stems from the linguistic nature of Japanese.
The Japanese language can express homophonic words
with different notations. For example, “食べる (taberu)”
which means “eat,” can be expressed as “たべる,” “タベル ,” etc. In story units, “ 気 づ く (notice)” can be
expressed as "notice," "notice," "notice," etc.
In the story unit, there is a verb “気づく(kiduku)”
which means “notice.” However, the verb concept

5.4.2 (A). Direct connect(A) is a verb that can be easily combined with the verb
concept dictionary. For verbs that have the same notation,
once the description of the Story Unit is used as is, the(set '通行する 1'((name 通行する 1)(sentence-pattern "N1 が N2 を 通行する")(case-cons-set((case-frame ((agent N1) (location N2)))(constraint ((人 -死人 -人間〈人称〉 -準人間)(場所 -世界 -景 -宇宙 建造物 -屋根 -柱・梁〔はしら・はり〕 -壁 -家屋{部分〈要素{その他}〉}[アンテナ] -家屋{部分〈要素{その他}〉}[避雷針] -建具 -塀)))))(is-a (v 物理的移動))))© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

Fig. 6. An example of a verb concept

195

Takuya Ito, Jumpei Ono, Takashi Ogata

dictionary registers it as “ 気 づ く .” Therefore, wechanged the notation of the verb in the story unit side tothat of the verb concept dictionary.

9.

5.5.5 (D). Passive form and (E). Negative form

10.

(D) and (E) are issues that are addressed by revising thestructure of the story unit. (D) is dealt with by using theoriginal verb concept of the verb and placing the person
who is the target of the action in the counter-agent case.
(E) is expressed by adding a not structure to the casestructure of the verb.

11.

12.

6. Conclusion
In this paper, we combined story units and the verb
conceptual dictionary. Therefore, the story units can use
sentence generation in the INGS. And the system
structure of the verb concept dictionary.
In the future works, we have two tasks. At first, wehave to combine story units with the noun conceptual
dictionary. By combining with the noun conceptual
dictionary, story units can manipulate the case of story
units based on the systematic structure of noun concepts.
At second, we revise the new verb concepts. The newly
registered verb concepts have been created according to
the structure of the folktale. In particular, we have torevise the constraints of the verbal concepts in order tomake them more generic.

13.

14.

15.

16.
17.

References
1. Propp
2. T. Ogata, Toward an Integrated Approach to Narrative
Generation: Emerging Research and Opportunities, IGI
Global, PA, USA, 2020．
3. T. Ogata, Internal and External Narrative Generation
Based on Post-Narratology: Emerging Research andOpportunities, IGI Global, PA, USA, 2020．
4. [小澤 97] 小澤俊夫：昔話入門，ぎょうせい，1997．
5. [Aarne 61]Aarne, A.: The types of the folktale: aclassification and bibliography, "FF communications,"
184, Helsinki: Academia Scientarum Fennica, 1961.
6. [Aarne 69]Aarne, A.: 関啓吾訳. 『昔話の比較研究』.
東 京 : 岩 崎 美 術 社 , 1969. (Antti A.: Leitfaden dervergleichenden. Marchenforshung, 1913.)
7. [Seki 1979] Seki, K., Nomura, J., & Ōshima, H. (Eds.):
Nihon mukashibanashi taisei: vol. 12 kenkyū hen[Complete collection of Japanese folktales: Vol. 12,
researches]. Tokyo, Japan: Kadokawa Shoten, 1979.
8. [Seki 1980] Seki, K., Nomura, J., & Ōshima, H (Eds.):
Nihon mukashibanashi taisei: Vol. 11 shiryō hen

18.
19.

[Complete collection of Japanese folktales: Vol. 11, data].
Tokyo, Japan: Kadokawa Shoten. 1980.
[Seki 1979a] Seki, K.: Nihon mukashibanashi taisei vol.1:
dobutsu monogatari [The Concluded Compilation of
Folktales]. Tokyo, Japan: Kadokawa Shoten, 1979.
[Seki 1978a] Seki, K.: Nihon mukashibanashi taisei vol.2:
honkaku mukashibanashi (1) [The Concluded
Compilation of Folktales]. Tokyo, Japan: Kadokawa
Shoten, 1978.
[Seki 1978b] Seki, K.: Nihon mukashibanashi taisei vol.3:
honkaku mukashibanashi (2) [The Concluded
Compilation of Folktales]. Tokyo, Japan: Kadokawa
Shoten, 1978.
[Seki 1978c] Seki, K.: Nihon mukashibanashi taisei vol.4:
honkaku mukashibanashi (3) [The Concluded
Compilation of Folktales]. Tokyo, Japan: Kadokawa
Shoten, 1978.
[Seki 1978d] Seki, K.: Nihon mukashibanashi taisei vol.5:
honkaku mukashibanashi (4) [The Concluded
Compilation of Folktales]. Tokyo, Japan: Kadokawa
Shoten, 1978.
[Seki 1978e] Seki, K.: Nihon mukashibanashi taisei vol.6:
honkaku mukashibanashi (5) [The Concluded
Compilation of Folktales]. Tokyo, Japan: Kadokawa
Shoten, 1978.
[Seki 1979b] Seki, K.: Nihon mukashibanashi taisei vol.7:
honkaku mukashibanashi (6) [The Concluded
Compilation of Folktales]. Tokyo, Japan: Kadokawa
Shoten, 1979.
[Seki 1979c] Seki, K.: Nihon mukashibanashi taisei vol.8:
warai banashi (1) [The Concluded Compilation of
Folktales]. Tokyo, Japan: Kadokawa Shoten, 1979.
[Seki 1979d] Seki, K.: Nihon mukashibanashi taisei vol.9:
warai banashi (2) [The Concluded Compilation of
Folktales]. Tokyo, Japan: Kadokawa Shoten, 1979.
[Seki 1980] Seki, K.: Nihon mukashibanashi taisei vol.10:
warai banashi (3) [The Concluded Compilation of
Folktales]. Tokyo, Japan: Kadokawa Shoten, 1980.
[Ono, 2019] Ono, J., A., Ito, T., & Ogata, T.: Mukahi
banashi no motif no program ka to motif kozo no hikaku.
In Proceedings of the 61th Special Interest Group on
Language Sense Processing Engineering, pp. 51-62, 2019.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

196

Story creation and cognitive biases
Jun Nakamura

Faculty of Global Management, Chuo University, Higashinakano 742-1
Hachioji-shi, Tokyo 192-0393, Japan *E-mail: jyulis.77f@g.chuo-u.ac.jp
www. Global.chuo-u.ac.jp/English/

Abstract
A wide variety of stories can be created from the same material. The author asked several teams to create stories,
focusing on cognitive biases. As expected, a variety of stories were created by each team, and cognitive biases wereidentified in the process of creating them. In other words, if it is assumed that cognitive bias is a driving force behindthe creator's intentions, then it can be concluded that the creator must have a certain strong intention when creating astory.
Keywords: Story, cognitive bias, analogy, and creativity

paper, such as the selective use of memory. I explore theprocess by which different teams given the same data set
create different stories.
The second chapter touches on related research andexperimental tools. Next, Chapter 3 describes stories'
impression points and items for assessing cognitive
biases. This is followed by Chapter 4, which introduces
an experiment based on participants’ self-assessment.
The results of the experiment are then presented in
Chapter 5. This is followed by a discussion aimed atanalyzing and interpreting the results of the experiment,
and the final chapter offers a conclusion.

1. Introduction
Studies of "creativity" are difficult to replicate. This
research has sometimes been said to be inherently
dubious. However, human prosperity is nothing but theaccumulation of creativity derived not only fromdestructive innovation but also from gradual creative
innovation that makes it sustainable.
The word “creativity” can be used to describe a wide
range of products, depending on what one is creating.
The product may be an object, a service, or a newtechnology. Whatever the case, to attract people, it isimportant to tell a "narrative story", and I believe that an
attractive story makes a deep impression on people andinspires them to remember it.
I have been studying the process of designing andgenerating stories. In doing so, I have focused on how thecharacteristics of stories change depending on thepresence or absence of external stimuli (Nakamura,2020). This paper is focused, first, on the effects ofavailability heuristics, termed cognitive biases in this

2. Related research and experimental tool
As an area of research in support of activities to create
stories, the aim was to develop games that enhancecomposition and creativity to reconstruct combinationsof words and thereby create new concepts. The
importance of the meaning and role of words in thedesign process, which is a creative activity, has been
noted (Nakakoji and Yamamoto, 2010). The game© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

197

Jun Nakamura

developed is aimed at connecting words by analogy(Holyoak and Thagard, 1996) and discovering new
expressions in various combinations. It is called theAnalogy Game (Nakamura and Ohsawa, 2007).
This type of game falls into the category of “serious
games” in the broadest sense of the word. Serious games
are defined as "games used to solve problems in
education and other areas of society" (Sawyer 2001).
Examples would include training in firefighting activities,
emergency and medical training, and language learning.
Overseas, the use of serious games to address the issue ofthe educational use of games has been attracting attention(Fujimoto, 2006). Since this analogy game was aimed at
fostering composition and creativity, it is of a type that
players explore and structure by themselves (Quinn,
Beattie, McNaught, and Wills, 1994).
The analogy game is introduced as an experimental
tool in the following manner. At the beginning, players
are given 20 word cards that have been pre-classified(panel A in Figure 2). The word cards are clustered into
several categories that are given relevant concept names
commonly used in daily life, such as "fruits", which
would cover bananas, strawberries, and melons. In thispaper, such concepts are regarded as ready-madeconcepts.
After the game begins, players attempt to dismantle
the ready-made concepts and reconstruct new
combinations by moving the word cards around (Figure
2 B). Words that are grouped together at this stage arehighlighted in the same color. This is because the system
recognizes groups by their color code. Players thengenerate new concept names for the reconstructed groups.
If one word card is left out of these groups, thecomposition must be reviewed again. The game ends
when all of the word cards are classified into one of the
new groups (Figure 2 C). For example, suppose that
services and medical care were categorized as"businesses", sushi and baseball as "Japan", and cars andbarbecue as "family travel”. Players could recombine
these ready-made concepts by combining service,
medical, and barbecue into "compassion" and by
bundling sushi, baseball, and cars into "overseas
expansion", thereby creating concepts that arecompletely different from the original concept.

Fig. 1. Flow of the Analogy Game
3. Story impression points and assessment
The aim of the experimental tool mentioned in theprevious chapter was to support creative activities (Hori,
2007). The present chapter explains impression points instory creation as a deliverable self-assessment throughdebriefing after the game.
3.1. Story impression points
It has been suggested that creativity can be evaluatedfrom the viewpoints of originality and novelty as well aspracticality and appropriateness (Paletz and Peng, 2008).
Therefore, the author concluded that "Innovative" and"Feasible" are appropriate as impression points forcreated stories. In addition, three impression points,
Usefulness, Scale expansion, and Barrier to rival, wereadded, with reference to the value proposition design
proposed by Imazu † (2016), which excluded quantitativeevaluation points. Therefore, the following five items
were identified as the impression points for createdstories in this paper: Innovation, Feasibility, Usefulness,
Scale expansion, and Barrier to rival.
3.2. Assessment
Next, the assessment items for cognitive biases are
introduced. In relation to cognitive biases, self-awarenessis metacognition (Brown, 1978). This paper aimed to use
metacognition to explore the source of story creation. For
this purpose, as described in the first chapter, theparticipants were asked to self-assess relevantcharacteristics, considering the availability heuristics
such as people's selective use of memory, as forms of
cognitive bias. In this case, what are the types ofcognitive bias? Bazerman and Moore identified 11 typesof bias that can occur during decision making (Baserman© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

198

Story creation and cognitive

cognitive bias, and assessment item 2-Y asked whether
they recognized the expansion of their ideas without their
being anchored to a given card.
Assessment item 3 was not a straightforward questionof cognitive bias, but it asked for "bounded awareness"in conducting teamwork. In this experiment, theparticipants spent more than 2 hours in teamwork, in
effect, and the amount of information in the conversation
was considerable. In sorting this information during thediscussion, the participants had to process unconsciously.
In the post-experimental review, assessment item 3-X
asked participants whether they felt that the informationthat they considered useful (i.e., the cognitive bias of theconfirmation trap was also present here) was ignored oroverlooked. In contrast to 3-X, assessment item 3-Y
asked whether the respondents felt that the informationthat they found useful was sufficiently shared and agreed
upon within the team.

and Moore, 2013). In this paper, four of these wereselected for consideration in creative activities, excluding
quantitative items.
-

Ease of Recall: cognitive bias that occurs towardeasily recalled material based on the clarity ofmemoryRetrievability: cognitive bias toward an object to beremembered that is structurally easy to retrieve
Confirmation trap: cognitive bias based on theassumption that one's own hypothesis is correct
Anchoring effect: cognitive bias toward available
information as a starting point and tendency not tomove far beyond that anchor

Based on the above, the assessment items for cognitive
bias were organized in this experiment as follows.
Assessment item 1 (memory versus hypothesis)
1-X = Recalling someone’s memories of past eventsaffected your ideas
1-Y = Not your memories but your team’s own tentative
hypothesis affected your ideas

4.

Experiment

In this paper, we conducted an experiment using theanalogy game introduced in Chapter 2 in the following
manner:Experiment date: Friday, November 27, 2020
Participants: Twenty-one second-year students of theFaculty of Global Management, Chuo University
Experimental method: Twenty-one people were divided
into six teams and presented with a set of word cards on
the screen as follows. Capital letters in the following
indicate the names of the clusters, and lower-case letters
indicate individual words on the cards.
TRIP ADVISOR: discount ticket, foreign tour,
backpacker, guide, word of mouth
FACEBOOK: search, friends, like!, share,
network
IKEA LIFE: living room, do it yourself, easy tostore, Europe, wardrobe
STARBACKS: yen400, third place, extra job,
coffee, steamer

Assessment item 2 (stuck to the cards versus took offfrom cards)
2-X = You were affected by the meaning of a given setof cards and continued to stick with that until the end
2-Y = You were influenced by a given set of cards, but
the idea took off from that
Assessment item 3 (not shared versus shared)
3-X = During the team discussion, a unique idea arosethat was not shared by other individuals
3-Y = The ideas were sufficient using only the
information shared among team members
Assessment item 1-X was combined with Ease ofrecall and Retrievability because both were types ofcognitive bias related to ones’ memory of the past. On theother hand, assessment item 1-Y asked the participants
whether they felt a cognitive bias suggesting that they
believe their hypothesis is correct as a confirmation trap
or as a memory of the past.
Assessment item 2-X asked whether the participants
recognized the anchoring effect, which is a type of

The above set of word cards is the same as that usedin our previous experiment (Nakamura, 2020). The
difference between the previous experiment and thepresent one is that this was an experiment not with one
team but with six teams. Participants were instructed todevise a story connecting several newly reconstructed© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

199

Jun Nakamura

Story. Since they are working in foreign countries to earnmoney, study a language, and learn the culture at thesame time, it is better to save clothing money and travel
around if they have time. To save money on their clothing,
it would be better for them to purchase or borrowseasonal clothes at a sales market or garage sale. Also,
since it will be difficult to get information in foreigncountries, where they do not have friends or family, theymight need to use an online community to get various
kinds of information during their working holidays.
Finally, it is important to relax after work every day.

clusters (a set of word cards) using the experimental tool.
The story was to be expressed in PowerPoint using a copyof the screen created with the experimental tool. The
story was then presented to the participants. After thepresentation of the story, participants were asked to do
the following two tasks:Give a score to stories created by teams other thanyour own, taking into account the impression points
mentioned in Chapter 3. A total of 20 points was theupper limit of the score. However, the same numberof evaluation points should not be given to morethan one team.
Assess what cognitive biases were present inresponse to your own team using the assessment
items described in Chapter 3. Participants were
instructed to answer questions X and Y for each of
the assessment items (1, 2, and 3), for a total of 10
points, and determine the ratio of X and Y that they
think applies to them.

An interesting slide from the team 4 presentation isshown in Figure 3.

5. Results
Even though the word cards provided in the experimentwere the same for all teams, different teams createddifferent stories. The impression point scores for the six
teams are shown in Figure 2, with team 4 receiving thehighest rating.

Fig. 3. A slide representing the thinking processThis is a picture of a note on the thinking process
underlying creating a story. The note was written down
and projected onto one of the presentation slides. The
explanation given for this slide was as follows:(1) The first four words that stood out to me were, “Do
it yourself”, “coffee”, “extra-job”, and “Europe”. I
decided to connect the four.
(2) Think of it this way, it's a working holiday! Thatis said, a part-time job in a foreign country is
indeed a working holiday!
(3) On a working holiday, you have to do it all by
yourself.
(4) As a mini-story derived from the above, on aworking holiday we need to save money, we mightwant to go on a tour in our free time, we might needto relax, and we might need information.

Fig. 2. Impression points rating by team
The team 4 story that received the best ratings was thefollowing:

The results of the self-assessment from all of participants
are shown in Table 1.

Story theme. Four things that the students on Working
Holiday need might be Cheap clothes, Information, Relax
and Tour.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

200

Story creation and cognitive

Table 1. Self-assessment of cognitive biases.

uniquely to the question, it is difficult to figure out thestory-generating algorithm. However, at least from theabove results, we can say that the cognitive biases ofassessment items 1 and 2, whether X or Y, arecharacteristically different to those of the other teams (i.e.,
representing highest or lowest awareness).
Now, we discuss assessment item 3. For both team 3
and team 4, assessment item Y was ranked higher thanitem X. This implies that both teams seemed to be in agood environment for the members to share theiropinions with each other. On the other hand, for team 5,
which had the lowest score, X was higher than Y,
indicating that individual opinions were not reflected.
That means that no chemical reaction yielding new ideas
occurred in the mix of different opinions from the teammembers.

6. Discussion
Focusing on teams 4 and 3, which had the highest ratings,
what kind of cognitive biases did the team members feel?
In team 4, the most remarkable assessment item that theteam member reported was item 2-Y, which was uniquecompared to the other teams. This indicates that they did
not stick to the given key words until the end, and they
perceived themselves to have made a leap in the meaningof the words. This may be due to the so-called polysemyeffect, in which a given key word has multiple meanings(Costello and Keane, 2000). It could be said that blendinga meaning that is different from the original meaning(Fauconnier, 1994) produced a creative story. In fact, if
we look at the thought process of team 4, as depicted in
Figure 3, the approach to the mini-story that connects thefirst four key words (Europe, Extra-Job, Coffee, and Do
it yourself) was noteworthy. From this mini-story, theteam expanded the idea and connected the remaining key
words to form the whole story.
The second most highly rated team was team 3, which,like team 4, showed characteristic cognitive biases thatwere different from the other teams. Team 3 had a higher
rating on item 1-X than the other teams, i.e., they felt that
their memories were more effective in generating stories
than were their own free ideas and hypotheses. The
possible reasons for this are discussed below.
In the case of team 3, it might be presumed thatmemory was given priority because the consensus wasthat teamwork must take precedence over mutuallyagreed-upon ideas, resulting in the hypotheses of various
participants cancelling each other out. However, team 4,
the most highly rated team, gave the opposite answer.
That is, team 4 responded with a free hypothesis that was
not anchored in their own memory. This contrast is quite
interesting. Given that teams 4 and 3 did not respond

7. Conclusions
Cognitive biases were evident throughout the experimentin the process of creating stories. In other words, if we
assume that cognitive biases are the driving force behindthe creator's intentions, then it is essential that the creator
have a certain strong intention to yield a memorable story.
The experiment in this paper, however, is just a casestudy of a limited number of people, and there are still
many works to be addressed before the findings can begeneralized. In the future, the author intends to focus
more on the process of creating stories.
Acknowledgement
This study was carried out with the cooperation ofundergraduate students of Chuo University, Japan. Also,
the Analogy Game was upgraded and developed further
with the support of Mr. Tano, President of Expert Gig Co.,
Ltd. This research was supported by Kakenhi (JSPS
KAKENHI Grants Numbers 17K03872 and 19K03062).
The English in this document has been checked by atleast two professional editors, both native speakers ofEnglish. For a certificate, please see:
http://www.
textcheck. com/certificate/rTtVeD.
References
1. J. Nakamura, Effect of external activate factors serving as
clue: creating new products or services ideas withstorytelling, T. Ogata and J. Ono (eds.), Bridging the Gap
Between AI, Cognitive Science, and Narratology With
Narrative Generation, IGI Global, Hershey, PA, 2020.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

201

Jun Nakamura

2. K. Nakakoji and Y. Yamamoto, Wording as external
representations to interact with: its role and nature
throughout a design process, Cognitive Studies: Bulletin of
the Japanese Cognitive Science Society, 17(3), 2010, pp.
474-490.
3. K.J. Holyoak and P. Thagard, Mental Leaps, MIT Press,
MA, 1996.
4. J. Nakamura and Y. Ohsawa, The visualization of concept
creation throughout the analogy game, Proceedings of the
2nd Oukan Conference, Transdisciplinary Federation of
Science and Technology, 2007, pp. 157-160.
5. B. Sawyer, Serious Games: Improving Public Policy
through Game based Learning and Simulation, in
Foresight and Governance Project, Woodrow Wilson
International Center for Scholars, Publication 2002-1,
retrievedonDec.5,
2020,
athttps://www.scribd.com/document/38259791/SeriousGames-Improving-Public-Policy-through-GamebasedLearning-and-Simulation.
6. T. Fujimoto (2006) Kaigai niokeru serious game no
kanousei (in Japanese), in Proceedings of the Conferenceon Simulation & Gaming, Autumn, 2006.
7. C. N. Quinn, K. Beattie, C. McNaught, S. Wills, Designingeducational computer games, Proceedings of Interactive
Multimedia in University Education: Designing for
Change in Teaching and Leadership IFIP Transactions,
1994, pp. 45-47.
8. K. Hori, Creativity Support, Theory and Application of AI
Approach toward Creativity, Ohmusha, Tokyo, 2007.
9. S. B. Paletz and K. Peng, Implicit theories of creativity
across cultures, Journal of Cross-Cultural Psychology,
39(3), 2008, pp. 286-302.
10. M. Imazu, Hashirinagara kangaeru sinkijigyou no
kyoukasho (in Japanese), Kanki Pub., Tokyo, 2016.
11. A. L. Brown, Knowing when, where, and how toremember: a problem of metacognition, in R. Glaser, (Ed.),
Advances in Instructional Psychology, Lawrence Erlbaum
Associates, Hillsdale, NJ, 1978.
12. M. H. Bazerman and D.A. Moore, Judgement inManagerial Decision Making, John Wiley & Sons, Inc.,MA, 2013.
13. F. J. Costello and M. T. Keane, Efficient creativity:
constraint-guided conceptual combination, Cognitive
Science, 24(2), 2000, pp. 299–349.
14. G. Fauconnier, Mental Spaces, Cambridge University
Press, Cambridge, UK, 2008.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

202

Time in an Aesthetic Experience of a cup of Sake
Hiroki Fukushima

Kyushu Women’s University, 1-1, Jiyugaoka, Yahata-hnishi Ku
Kitakyushu City, Fukuoka, Japan

Abstract

There are two broad genres of artworks depending on the nature of their appreciation: arts of space andarts of time. In the aesthetic appreciation of taste, which is the focus of the experience of a cup of Sake?
Is there the dimension of time in the experience of taste, and if so, what is it? Because taste has neverbeen considered a formal subject of aesthetics, there is little discussion of describing and appreciating theaesthetic experience. This study deals with the concept of time in the appreciation of Sake, with somereference to the concept in narrative studies. As with wine, the time associated with Sake is multifaceted,and each is intrinsic to the aesthetic experience. Consider some examples: Time from the Sake is brewed
to the time it is opened on the table (so-called "vintage"; for some years). The time from opening thebottle to pour into the glass (for minutes to tens of minutes). The time between pouring into the glass andentering the mouth (for a few minutes). Time to swallow in the mouth (a few seconds). The flavor felt
after swallowing ("after flavor"; tens of seconds). This study focuses on the period from the Sake is placedin the mouth to the time it is swallowed. The purpose of this study is to reveal how time is narrated(explicitly or metaphorically) in the tens of seconds of a sake tasting.
Keywords: Sake, Taste, Temporal Arts, Plastic Arts.

discussions on how to classify them systematically. Max
Schasler (1819 -1903), a German philosopher, sets two
opposing principles for the artworks: das Simultane andSukzessive. The former corresponds to the plastic arts andthe latter to temporal art. Architecture, sculpture, andpainting are classified into plastic art, and music, mimic,
and poetry are classified into temporal art. This
classification is based on the conflict between material
elements and ideological connotations.
The opposition between temporal art and spatial
art was made clear in Lessing's (1729-1781) Laocoon: An
Essay on the Limits of Painting and Poetry. In this debate,
Lessing criticizes the tendency to equate poetry andpainting, as typified by Horatius' proverb, “ut pictura
poesis”. The opposition between poetry and painting, heargues, is that painting and sculpture depict the decisive
moment of a subject in a spatial extension, whileliterature depicts a story in a temporal extension. If wefollow Lessing's argument, spatial art can be further
divided into two categories: three-dimensional (e.g.,

1. Introduction
Taste has been largely neglected in traditional aesthetics.
Since Sibley discussed the aesthetics of taste and smell in
1959, several philosophers have tried to theorize theaesthetics of taste and smell and have introduced artistic
activities of smell, but not sufficient.
The few researchers who deal with theaesthetics of taste and smell must devote the early part oftheir works to a masochistic discussion of how taste andsmell have been neglected in aesthetic research.
In this study, we temporarily put aside the questionthat whether taste and smell are the subjects of aesthetics.
Then, if the art works of taste and smell are established,
what kind of character they can have is discussed.
2. The Temporal Art and the Spatial Art
Consider the position of taste art. There are many genresin art such as painting, music, architecture, drama,
literature, and so on. There is a long history of© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

203

Hiroki Fukushimas

The time between when the liquor is brewed andwhen it is opened on the table. (so-called "vintage"for some years).
•
The time from opening the bottle to pour into theglass (for minutes to tens of minutes).
•
The time between pouring into the glass andentering the mouth (for a few minutes).
•
Time to swallow in the mouth (a few seconds).
•
The flavor felt after swallowing ("after flavor"; tensof seconds).
Among these time aspects, this study focuses on the
period from the Sake is placed in the mouth to the time itis swallowed.

architecture and sculpture) and two-dimensional (e.g.,
painting).
The classification of temporal art and spatial art
may seem unassailable, but E. Souriau argued against it.
Souriau criticizes the disconnection between temporaland spatial art. In his main work, La Correspondance desarts (1943), he compares music and painting, and points
out the similarities between their various elements.

•

3. Taste and The Genre of the art
Expressions of taste can be broadly classified into verbal
and non-verbal expressions. Verbal expressions are, as
typified by sommeliers, the descriptions of the charactersand impressions of a taste experiences. Linguistic data
can be collected from publications such as books andmagazines, or through verbalization experiments andinterviews.
There are few practical attempts of non-verbal expressionof taste. In the narrower sense, non-verbal expression canbe defined as a work of art in which a person draws apicture or plays music to describe the flavors of all along
the time a wine enters the mouth to the time it isswallowed. Such artworks will express a single,
subjective, individual experience of eating and drinking.
On the other hand, in a broader sense, non-verbal
expression can include labels, posters, and commercial
images of luxury goods such as wine, sake, cigarettes, etc.
These can be thought of as depictions of the character ofthe flavors of the product, but they do not represent a
specific eating or drinking experience. The fact that abear is depicted on a chocolate package does not meanthat the chocolate tastes like bear meat.

3.2. Describing the Time of the Taste
Case 1: Dominio IV in Oregon

Figure 1. Riesling 2014, Dominio IV

3.1. Multiple Time aspect in the Appreciation of
Taste

Oregon winery Dominio IV's imagination series features
taste drawings on wine labels. In this drawing, called"shape tasting," the impression of taste is represented bya two-dimensional picture along a time axis. The shapes
and colors of the drawing synaesthetically represent thetaste elements of the wine. The time axis and the numberof seconds are at the bottom of the painting. The timegoes from left to right, symbolically depicting the flavors

In this section, I will analyze how time is discussed in the
representation of taste.
In the appreciation of taste, the most crucial time is the
few seconds between when the food enters the mouth andwhen it is swallowed into the back of the throat. However,those few seconds are only a small part of the many
aspects of the concept of time relating to the appreciationof taste.
As with wine, the time associated with Sake ismultifaceted, and each is intrinsic to the aesthetic
experience:© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

204

Time in an Aesthetic

as they appear, spread, move, unfold, change, and linger
in the mouth.
At first glance, it appears as if the artist is freely drawing
the flavor images with paint, but in this series, the verticaland horizontal axes of the canvas are defined first. The
wine label is oriented horizontally, and the horizontal
axis direction is a time axis with seconds. In the pictureof Riesling shown in Figure 4, the leftmost point is 0
seconds, and the changes in taste over 60 seconds are
drawn in 15-second increments. The vertical axis isshown as the extent of the palate breadth. The middle ofthe vertical axis represents the center of the oral cavity.
The 1/4 width range from the center is depicted as the"narrow" area. The outer drawing area shows how wide
the flavor spreads. Thus, in the example of fig. 1, a time
variation can be seen as follows; the flavor first spreads
widely in the mouth -> gradually converges narrowly
from 15 seconds to 30 seconds -> leading to the final taste.
In the imagination series, the drawings are describedvertically symmetrically, with some exceptions.
In addition to the time axis and the spatial axis of the oralcavity, another feature is that what the shapes (i.e., circles,
lines, and arrows in the drawings) represent are defined.
Again, in figure 1, the example of Riesling 2014, we cansee a note at the bottom of the figure. According to thenotation, the red circles represent "fruit" flavors, and theyellow curve represents "fruit-based acidity." Then, thebrownish arrows indicate "linear acidity."In this way, the definitions of the drawing elements
enable us to decode the drawings. The 2014 Riesling is
expected to taste as follows; fruit-based acidity and sweet
fruit flavors that fill the mouth as soon as you put it in ->and then gradually fade after about 15 seconds -> After
30 seconds, a linear acidity emerges -> a lingering acidity
can be enjoyed from 45 to 60 seconds."
The expression of wine flavor is often given ina language-based manner, as typified by sommeliers.
And such descriptions are often printed on the back labelsof wines. In the Imagination Series of Dominio IV,
captions are provided just below the illustrations.
Additionally, verbal explarnations of the flavors are also
provided. This series is an eloquent example of thepossibility of the non-verbal description of the taste.

Figure 2, Sake Art Label project
The attempt to present the image of taste in pictorial form
has also been practiced in sake.
In the past, sake labels were generally designedby selecting a background from a template and placing
the brush-written brand name. In recent years, however,with the development of distribution and the boom in
local sake and ginjo sake, stylish and unique designs havebeen seen on sake labels. This shift represents a turning
point. Up until now, it was enough to know the name of
the brand in order to select the "usual" sake from the local
brewery, such as Tsukasa-Botan or Tosatsuru in Kochi
Prefecture. But now, with the development oftransportation, consumers tend to seek unique local sakeall over Japan.
Against such a background, it has becomecommon for sake breweries to express their thoughts andthe characters of their brand into their label design.
One of the notable efforts to express the image of sakeflavor through labels is the "Sake Art Label Project,"which was realized in 2014 through a collaboration
between the Obata Shuzo Brewery (Sado City, Niigata
Prefecture, the representative brand "Manotsuru") andthe Isetan Mitsukoshi Group (Figure 6).
The project is an effort to break away from the former
sake labels style, as a list of technical terms and express
the image of flavor as an art label. Sake labels generally
contain not only the name of the brand but also the variety
of rice, such as "Yamadanishiki," the standard, such as

Case 2: Sake Art Label Project© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

205

Hiroki Fukushimas

Acknowledgements"Junmai" or "Ginjo," the method of production, such aspolishing ratio or "Namagenshu," and numerical values,
such as amino acid content or alcohol content. While thisinformation is useful for predicting the flavor, it also
requires a certain level of knowledge to decipher, which
can put off beginners.
In the sake art label project, the opinion
exchange meeting which invited cooking party,
copywriter, and general consumers was carried out, and8 kinds of sake were tasted blindly (concealed brandnames). About 40 participants answered a questionnaire
to answer images, colors, and objects associated witheach brand according to their preferences. At the tasting
site, Takeshi Hirashima, an illustrator, was present anddrew illustrations based on the comments of theparticipants. Finally, the brewery, the buyer of Isetan
Mitsukoshi, and the illustrator selected the two highly
evaluated brands, and the illustrator designed the imagesof each flavor on the labels.

This work was supported by JSPS KAKENHI Grant
Number 20K20127.
References
1. Lessing, G. E. (1853). Laocoon: an Essay on the Limits ofPainting and Poetry. Longman, Brown, Green, andLongmans.
2. Souriau, E. (1949). Time in the Plastic Arts. The Journalof Aesthetics and Art Criticism, 7(4), 294-307.
doi:10.2307/426722

4. Conclusion
The characteristic of painting is that it has no temporality.
Of course, some philosophers such as Souriau insist on
the concept of time in paintings. And in the plastic arts,
aesthetic experiences can occur chronologically.
However, unlike music, the object itself does not change
its form in time.
There are many efforts to express time on
canvas. These efforts are not avant-garde. Manga uses amethod of developing time through the combination ofmultiple panels of pictures. Or the illustrated handscrolls"Emaki-mono" (literally 'picture-scrolls'), traditionallyproduced in Japan and China, are pictorial artworks inwhich the story unfolds chronologically. The oldest
picture scrolls in Japan date back to the Nara period(Einga-kyo, 7C-8C). From the 10th century onward,
picture scrolls have been actively produced in Japan invarious genres, including war stories, diaries, fairy tales,
and comic stories. Anna Willmann points out that;
Reading a handscroll can become an almost cinematic
Anna Willmann points out that; Reading a handscroll canbecome an almost cinematic experience as the viewer
scrolls through a narrative from right to left, rolling out
one segment with his left hand as he re-rolls the righthand portion.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

206

Extension of Clinical/Psychological Approach Using Post-Narratology: Possibility of
application on Artificial Intelligence and Robot
Kai Seino

Department of Social Rehabilitation of Research Institute, National Rehabilitation Center for Persons with Disabilities,
1, Namiki 4, Tokorozawa City, Saitama Prefecture, 359-8555, Japan
E-mail: seino-kai @rehab.go.jp
www.rehab.go.jp/english/index.html

Abstract
A purpose of this research is to examine the psychological approach from a narrative viewpoint and discuss expanding
collaboration with AI and Robot. Methods are a literature review and theoretical examination. As a result,
psychological narratology was confirmed to have expanded beyond the so-called narrative therapy. In addition, AIor robot research in the clinical fields have been identified. This will give suggestions to the extension of
psychological narratology.
Keywords: Narratology, Computational Narratology, Counseling, Clinical Psychology, Disability, Psychiatry

In the clinical domain, the approach is key to any
narrative, and is called the narrative approach. For
example, representation is a narrative therapy in
psychotherapy. However, the conceptualization of anarrative in recent years has not been limited to thenarrative approach and is considered to have a wider
extent and possibility. One key to understanding thewider possibility of a narrative is the term “narrative
generative system,” which is advocated by Ogata,. The
narrative generative system conceived by Ogata not only
considers the structure or form of a story, but also
incorporates the process by which the story is constituted
and received. In addition, “computational narratology,”
one of the underlying theories of the narrative generative
system, is a concept resulting from the fusion ofcomputational technology and story. Authors havedivided narratology into literary narratology andpsychological narratology from the viewpoint of anarrative generation system (Seino 2016). A literary

1. Introduction
This research examine the potential of combining thenarratology as a problem-solving method withtechniques such as artificial intelligence (A.I.) or Robot,
and the applicability of clinical areas. As a general term,“narrative” is understood as the “act of discussing andbeing discussed using language (Spoken Language andwritten language, etc)” (Yamada, 2007). “Narrative” canalso be defined more narrowly as in the term
“narratology,” which refers to the reproduction of real or
fictional events and is always accompanied by time
constraints (Prince, 2003).
Bruner (1986) suggested that the narrative provides
individuals with a method for understanding and thinking
about their everyday lives and interactions with others. In
recent years, the narrative approach has been utilized inclinical domains such as medicine, nursing, psychology.
The narrative may be thought of as an evidence basecontaining conventional, objective data.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

207

Kai Seino

narratology.” Moreover, a narrative may be used outsideof the field of literature, as in a clinical domain such as
psychology, medicine, nursing, and social work thatprovides services to support persons. The combination of
narratology and the use of a narrative in a clinical domain
is called “psychological narratology.

narratology considers the contents and form of a story,
whereas psychological narratology is called a story
treatment in the clinical domain and is a narrative
approach.
Against the backdrop of Ogata’s theory, Seino et al (2016,2018) proposed a new technique for data mining
designed to provide support for persons with disabilities.
The technique provides a method for the analysis andgeneration of a story, through a combination of textmining technology and the viewpoint of literary
narratology. The results demonstrated that a theory could
be established based on a narrative generative system.
However, to investigate the practical implications of thesystem for the theory and method of a psychologicalnarratology, a discussion of technology (e.g., computer,software, coding) is omitted.
Therefore, the psychological approach is presented in thisresearch from the viewpoint of a narrative, and aims at
clarifying the possibility of a psychological narratology.
disabilities in order to investigate the possibility of using
AI and robot for narrative research into the clinical field.
Method in this research is literature review. Medical
treatment and support for any illness that exists in theclinical domain is defined as a psychological approach.
In this research, the possibility of extending thepsychological approach through collaboration with AI or
robot has been examined. Specifically, we review the
research and practice of psychology, psychiatry, and aproximity domain in recent years and their use ofnarrative. Our discussion is broadly focused on theapproach and practice relevant not only to narrative
therapy but also to a narrative, or to narratology. We will
examine the potential of AI and robot using
psychological narratology from applications of AI androbotics in research on actual diseases and support forpeople with disabilities.

2.2. Literary narratology and “computational
narratology” as an application
The narrowest definition of narratology is regarded as aliterary theory, which is the study of the nature, form, andfunction of narrative (Prince, 2003). Ogata (2010)
organized narratology into five categories that establish
the theoretical underpinnings of narratology: (1) "ThePoetics" of Aristotle; (2) structuralism, which developed
based on the ideas of Saussure; (3) the literary art
movement of Russian Formalism ; (4) British andAmerican literary theory; and (5) the literary theory of
structuralism.
Narrative comprises "story" and "discourse." According
to narratology (Prince, 2003), the story is the content of
the narrative while the discourse concerns the expressive
side. Narratology employs an analysis of both story anddiscourse. Propp (1968) described the fundamental
components of any Russian folktale. In particular, hedefined the narratological notions of “function” andcharacters’ “roles.” “Function” is a character’s act
defined from the viewpoint of its significance in the story.
Furthermore, Genette (1980) divided discourse into three
categories: (1) “tense,” which is the temporal relation
between discourse and story; (2) "mood,” which is the
reproduction of the story through the discourse; and (3)“voice,” which is the relationship between the narrating
and the narrated and the relationship between the
narrating and text.
In applied studies of narratology, some researchers
analyze and generate stories based on the theories
constructed by Propp and Genette (Akimoto & Ogata,2013; Ogata, 2011). Regarding these applied studies,
Ogata (1999) suggested a study framework called"computational narratology." This research offered aliterary study frame that fused an understanding of
narratology with the generation of narrative through theuse of AI and cognitive science, thereby broadening theconcept of computational narratology.

2. A Definition and Theory of a Narrative
2.1. Definition of “Narrative”
A narrative has been defined in multiple ways, and there
are also many theoretical positions on the topic(Riessman, 1991). For example, one definition of
narrative is that it is an explanation of the experience ofa continuous story or people. A narrative may also be
communicated using several methods (Clandinin, 2007).
Narratology, which is the theory of a narrative, is definedin the narrowest sense in the literary theory of “literary

2.3. Interdisciplinary narrative generative system
In this section, “interdisciplinary narrative generative
system,” which is derived from Ogata’s narrative
generation system, is outlined. Considerable research on
story generation has been conducted in the domains of© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

208

Extension of Clinical/Psychological Approach

3. The Narrative in Clinical / PsychologicalApproachcognitive science and AI. Ogata and Kanai (2010) havecompleted advanced research on the interdisciplinary
approach for the narrative generative system, which
introduced narratology and literary theory. In addition,
Ogata (2011) proposed the concept for the narrative
generative system, which continues to be developed at
two levels: at a broad level, the narrative generative
system constitutes a method for symbolicallyunderstanding a human being and society in aggregate;
and at a narrow level, a narrative generative system
consists of a computer program. Recently, research in theclinical domain, which is based on Ogata's concepts, hasbeen progressing. Specifically, the research is on disease
and disability, and the results will be used for clinicalsupport (Aoki et al, 2018; Seino et al, 2016, 2018).

In this section, we will examined psychologicalnarratology from the viewpoint of a narrative, anddiscussed its extension and the possibility of applying it
to other areas of research. The conventional clinicaldomain attaches importance to evidence is objective. The
concept of an evidence-based approach originated fromevidence-based medicine (EBM) in the 1990s (Guyatt,
1991; Sackett, Richardson, Straus, Rosenberg, & Haynes,
1997),. Sackett et al. (1997) define EBM as the “theconscientious, explicit, and judicious use of current best
evidence in making decisions about the care of theindividual patient.” Conversely, the idea of “evidencebased” has been reasonably criticized (Gergen, 1994) .
Based on the limits of objective facts, social
constructivism, which provides the relativist conceptionof reality, has come to be recognized (Gergen, 1999).
Against the backdrop of social constructivism, a trendthat allows a reconsideration of EBM from theperspective of the narrative is Narrative Based Medicine(NBM) (Greenhalgh & Hurwits, 1998). Since the 1990s,
the narrative approach has attracted greater attention ininterpersonal help domains, such as medical care, nursing,and psychology. The development of epistemology andthe methodology that attached importance to narrative isknown as the “Narrative Turn.” (Denzin & Lincoln,2000).

2.4. The Research Framework for narrative
Several studies have suggested how to treat a narrative.
In a macroscopic position, the framework of a narrative
includes all life activities. In a micro position, a narrative(story) expresses the causalities of an event (Riessman,1993). In addition, the narrative approach is defined as“the method of making the concept of a narrative key andpresenting a certain phenomenon” (Noguchi, 2005).
Numerous disciplines study narratives. Ogata (2018)classified the research in the entire discipline with thenarrative into three more categories (Table 1).
Table 1. The division between a discipline and narrative1

2

3

Classification
Symbol level[Literature, art, andan entertainments
domain]A brain, a nerve,
and a psychologicallevel[Psychiatry and a
psychologicaldomain]Social level [Social
domain]4. Review of a Psychological Narratology

The contained discipline
Narratology, poetics, rhetoric,
literary criticism, linguistics,
folklore, folktale study, and an old
tale, entertainment research, Nohplay theory, kabuki theory etc.,
cultural anthropology and folklore,
philosophy and thought, movie
theory and cinematic review
Psychology, psychiatry andpsychopathology, psychoanalysis,
tale treatment, neuropsychoanalysis, brain science (Theauthor added a postscript.)

The system of a psychological narratology is proposed inthis section, which shows that the clinical / psychologicalapproach is extensible through post-narratology. Postnarratology is a narrative generative system that includes
story generation by A.I. In order to actuate story
generation, a literature review is performed with aclinical/ mental approach considering that the extension
to post-narratology is possible. The system thenconstructs the story through AI and authors argue about
the ability of a bridge to be carried out and whether byre-extending the clinical/mental approaches throughstory generation, a clinical contribution is possible. These
approaches are concretely aimed at the approach(research and practice) in clinical domains, such as
psychology. A psychological narratology is the theory of
the clinical/ psychological approach that is relevant to theSociology, history, businessadministration, economics,
jurisprudence, marketing theory,
and advertising theory© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

209

Kai Seino

for individuals addicted to alcohol, drugs/psychotropic
substances, or gambling, or who are dealing with thesame life obstacle or chronic malady. In recent years,
self-help group have come to be viewed as a form of anarrative approach (Noguchi 2005). “Alcoholics
Anonymous (AA)” was the first self-help group, which
was developed in the United States in 1935, and is forrecovery from alcohol addiction. AA was effective fortreating alcohol addiction, and over time, many othergroups adopted the method used by AA to deal not onlywith addiction or dependence but also for other problems.
The typical definition of a self-help group is that of Kats
&E. I. Bender (1976), who defined a self-help group as asmall group of companions (peers) who provide mutual
aid for the achievement of a specific purpose. AA
meetings are regularly scheduled and the members whogather at each meeting talk in turn about various topics.
The talk may be of experiences from one’s past or theutterance of a single word may constitute a speaking turn
(Noguchi 1995). The self-group does not have a technicaldefinition (Ito, 2000).

narrative in the broadest sense that the author has
proposed.
4.1.1. Narrative approach in the counselingIn the counseling field, a “narrative” has come to be
associated with the narrative therapy since the 1980s. In
a narrow sense, psychological narratology is narrative
therapy in psychotherapy and family therapy. Narrative
therapy was advocated by White and Epston (1990) andit developed through the context of family therapy in the
1980s. White and Epston (1990) applied a classificationof Foucault (1980) to narrative therapy, aiming to
excavate the new “alternative story” from the “dominantstory“ that restricts life. This excavation is conducted by“externalizing” and seeking a “unique resolution” for the
problem. In addition, Spence (1983) described theoccurrence in which a narrator is in treatment, and ahearer hears the talk of a narrator as a tale. In analytic
psychology, the role of a “tale,” like the myth, is touched
upon by Kawai (2001), who presupposes that the processof psychotherapy is a tale. In addition, in Gestalt therapy,
Polster (1987) presupposed that the recovery of harmonyof the heart requires a dialog, and proposed the methodof making achieving this harmony is by having a real
person sit on an “empty chair” that is assumed to be there
and engage in a dialog.

4.2.2. Consideration of a proposal
The first-line specialty of support of a self-help group is“experiential knowledge.” Experiential knowledge is theknowledge based on a member's experience, and serves
as a source of support for a self-help group. The
knowledge of members is more practical and morecomprehensive than the knowledge of professionals(Borkman 1976). Each member’s “experiential
knowledge” is discussed and the collective knowledge if
the group is accumulated and utilized as one taleknowledge from a group. We can interpret sharing of
knowledge as the process of making one big narrativefrom two or more members’ narratives. The process may
be adopted into theory as a narrative generative system
and may be realized as a system. For example, studies
have proposed collect many texts that were spoken andgenerating the tale based on some typical elements. In
addition, Ito (2000) examined a self-help group from theperspective of an individual tale. The results of the study
indicated that the tale should have been regarded as an
object with the effect of showing not the collective
knowledge system of a group, but of a participant’s self.
To determine the degree of change for each participant,it was presupposed that it was important to determine thestandard by which a participant made a tale “the goodtalk.” In other words, it was important for a tale orsuggestion to achieve a set basis for valuation. An

4.1.2. Consideration of a proposal
Narrative therapy creates a narrative that provides theclient with a tool for problem solving. A typical narrative
approach heads toward the solution to a question as theclient completes a tale. We propose the following
possibility from the viewpoint of post narratology: what
is treated is the discovery or description of a "dominantstory" and an “alternative story,” and the technique of
externalization. There are two implications of thissuggestion. First, the “dominant story” and “alternative
story” may be identified in order to construct the theoryof a narrative generative system. Second, computational
technology may be able to realize two stories.
4.1. Self-help group
4.2.1. The group for mutual aid
A self-help group is practiced among those who sufferfrom the same illness and offers support through peer
relations. For example, self-help groups may be formed© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

210

Extension of Clinical/Psychological Approach

4.3. Tohjisha-Kenkyu (Self-directed studies)

evaluation of the tale could be conducted based on how
the group absorbed the tale and how it affected the
production and consumption of future tales.

4.4.1. Empowerment approach by oneself and acompanion

4.2. Open Dialogue

In recent years, the “Tohjisha-Kenkyu” approach thatoriginated in Japan has attracted international attention.
Tohjisha-Kenkyu may be introduced as an open dialog inJapan (Saito 2016) and is an approach commonly usedwith patients with schizophrenia. In recent years, it has
also been developed as a treatment for persons with
developmental disabilities. Tohjisha-Kenkyu is anempowerment approach, and it happened among theactivity and liveing of the person with disabilities(Mukaiyachi, 2013).

4.3.1. Philosophy of discussing
In recent years, Open Dialogue (OD) is anotherpsychological approach that has attracted attention in
Japanese psychiatry. OD started in Finland in the 1980s
and is a method of therapeutic intervention for an acuteterm mental disease. In OD, two or more staff conduct acrisis intervention within 24 hours of a request. Patientsand families practice OD every day by sitting in a circle
and performing “OD.” OD seldom uses a drug but makes
a patient's critical situation cease (Saio 2014). All matters
are discussed and determined as a characteristic portionof OD at the members’ meeting place. In addition, an
expert team discusses a patient and a family (exchange)in their presence, as they reflect on what is being sharedwith them. Olson, Seikkula, and Ziedonis (2014)
summarized the following key OD items:

4.4.2. Consideration of a proposal
Tohjisha-Kenkyu provides suggestions for examining anarrative generative system and how to connect narrative
generation for problem solving. In Tohjisha-Kenkyu, it isassumed that the party concerned will themselves
become a hero of difficulties. In addition, TohjishaKenkyu allows the narrative to pass and has suggested
that it is a narrative approach tied to the problem solvingof the party concerned. In addition, Noguchi (2005)
reported that Tohjisha-Kenkyu is “a community ofnarration” that makes it “the community of the tale.”
Many audiences are in a “narrative community,” and new
narration becomes more certain with an existing audience.
A narrative community is the space wherein new
narration is shared and established. In addition, theaspects of changing attitudes, and how to identify andposition attitudes Tohjisha-Kenkyuare are also common
in narrative therapy. Moreover, the conceptualization of
one’s own experience and that of a friend is similar to thenotion of “experiential knowledge” in a self-help group.
In addition, when considering tale generation, an
improvisatorial, unique understanding or an idea isimportant and shows how to utilize the result, which
creates a “relation“ as a “behavior“ in a scene received
by the narrative generative system.

4.3.2. Consideration of a proposal
OD is considered to be a narrative approach given that a
conversation called a dialog becomes a key medical
treatment. OD contains narrative therapy and a common
feature of OD is that it has an ideological background. In
addition, it is also a common feature with narrative
therapy to use reflecting. The 12 elements of OD consider
tale generation and its effect, and the process is thoughtprovoking. A dialogue is performed by two or moremembers of various positions. In addition, open-ended
questions are used as an interlocutory trigger. OD may beable to be realized as an automatic dialogue system or anarrative system by speech recognition. By pulling out
two or more viewpoints and observing relationships, anautomatic dialogue system may be able to return utilize
an extraction method to return an element as the narrative
is being generated.

5. Psychological Approach for Application of
A.I.
In this section, two studies of the clinical/ psychologicalapproach and the possibility of narrative generation by
A.I. are introduced.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

211

Kai Seino

5.1. Narrative generation for employment supportof persons with disabilities

and clarified social stories of disability based on the
results of multivariate analysis involving text mining.
This is an example from our research entitled“Employment Support and Story Generation for Students
with Developmental Disabilities.” The results indicatedthat story generation tends to materialize through the
combining of existing elements with those from previous
research (Seino, Enomoto, & Miyazawa, 2018). The
figure below shows a story generative system.
The analysis of job descriptions analysis results wereused to create the image. For cases in which this system
was created, it was important to consider how elements
were combined. The probability of a story being
generated is attained by calculating the probability that it
may happen and the strength of the relationship between
the text elements. In addition, in order to utilize the resultof the support generation simulation, it is important tounderstand what kind of elements have been collected.
For example, the elements of support needs may becollected based on responses to a specific problem, and it
may be possible to examine what kind of problem andsupport needs occur at what time, thus enabling the
preparation of appropriate support.

In this section, the authors' research is introduced, inwhich we extracted text elements and employed a systemof story generation using A.I. Specifically, we analyzed
a disabled person’s free descriptive answer to aninterview question using the method of "informatics tale
analysis" to extract the elements of the story. This method
combines the technology of text mining, and sometheories of a literary narratology. Text mining involves
the quantitative analysis of text data based on natural
language processing or data mining. After we extracted
the element of the story from the individual’s text
response, the system proposed creating a collectivestory.
An example of the narrative is shown below. Please notethat to protect respondents’ privacy, some answers arealtered, but the basic meanings remain unchanged. In theauthors’ study, both the wider and narrow senses ofnarrative were used depending on the particular contextsof the analysis.
[Concerning work description] In response to thequestion: "Describe your job in detail."Example 1: Since my legs were injured, I have been
assigned paperwork. I am now engaged in computer
work. (Physical disability)[Concerning care and support at the workplace] In
response to the question: "Specify the care and support
provided in your workplace as a means of assisting yourcareer."Example 2: At my workplace, employees often get
acquainted with coworkers at informal meetings, such as
drinks parties. I did not attend these parties because of
my disease, so I had no opportunity to get to know them.
The other day, I took the opportunity of participating in
one, and it worked much more effectively than I had
expected. (Mental disorder)
Example 3: I first wanted to work without mentioning my
disease. The other day, a relative of a person who had
supported me asked me to work in their office. When I
began working there, I soon found that this relative did
not care about my condition, so it was a disastrous year.
From this experience, I realized that I should conceal mydisease because people consider me to be inferior tohealthy people. (Intractable disease)
The authors’ study analyzed the descriptive texts thatreflected the personal stories of persons with disabilities,

Fig. 1. The image of the story generative system

5.2. Narrative generation for paper creation
support of a student with a developmental
disease tendency
Joint research by Ogata and Aoki et al. (Aoki, et al. 2018)
proposed students’ paper creation support and narrative
generative system based on the cognitive tendencies andaction patterns of individuals with autism spectrum© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

212

Extension of Clinical/Psychological Approach

6.2. Use of machine learning in AI

disorders. Aoki et al (2018) suggested that a paper
creation might address the problem of a student with
autism spectrum disorder. they explained why the
problem occurs based on the cognitive tendencies and theaction patterns of people with autism spectrum disorder,
who may be sensitive to "a surprise and a gap" in the"discontinuous nature" of a story. Therefore, a story(paper creation), after making it continuous, stops thereaction of a student with autism spectrum disorder. The
evaluation mechanism was set up and support was
proposed for advancing a story by issuing directions on amacro and a micro level.

AI is being used in research to predict disease andidentify contributing factors.
Rahman et al (2020) developed a method for predicting
autism in newborns by machine learning the parents' age,socioeconomic status, medication, etc. from their
medical records.
This algorithm clarifies, autism and the use of substancessuch as caffeine and certain antidepressants by parents
are relevant. The results may also help to identify factorsthat may induce autism.
AI has also been used in research to differentiate between
diseases and disease subtypes. Stevens et al (2019)analyzed behavioral data in autism and found twocomprehensive behavioral profiles. Each profile had its
own subgroups based on the severity of different traits.
Chand et al (2020) used machine learning to analyze
brain scans of schizophrenia and clinical informationsuch as age of onset and medication use. As a result, they
identified two subgroups of schizophrenia based on
different patterns of brain structure. One of the subtypes
showed brain volume increasing almost normally in two
regions, rejecting previous ideas that schizophrenia isassociated with reduced brain volume.

6. AI and ROBOT for interpersonal support andclinical practiceIn this section, we review some of the practicalapplications of AI and robot in assisting patients andpeople with disabilities as a reference for considering AI
and robotics applications in narratology.
6.1. AI applications in psychiatric research
Ikeda (2019) reviewed the use of AI in the psychiatricfield.
According to this study, 393 articles were extracted fromthe article search database Pubmed using keywords
related to psychiatry and AI. In the 382 abstracts obtained,
the most frequently occurring psychiatric diagnoses weredepression, schizophrenia, and Alzheimer's_disease. In
addition, the research objectives of these articles weremainly the following four. (1) to improve the accuracy of
medical treatment (e.g., diagnostic support, prediction oftreatment effects and side effects, etc.), (2) to elucidate
the etiology and pathogenesis of diseases (e.g., genome
analysis, image analysis, etc.), (3) to develop newtreatment methods (e.g., drugs and medical devices), and(4) to reconstruct diagnostic concepts and systems. The
AI-related terms in those papers are, in descending orderof frequency, support_vector_machine, random_forest,
logistic_regression,
deep_learning,
natural_language_processing, decision_tree, convolution
decision_tree,
convolutional_neural_network,andnaive_bayes.
The following is an overview of the specific studies.

Koike and his research group developed a machinelearner that determines schizophrenia and developmental
disorders through image analysis by machine learning ofmagnetic resonance imaging (MRI) brain structure
images (Yassin et al, 2020).
Thus, machine learning research is expected to be applied
to differential diagnosis and treatment prediction inclinical practice. It may also be useful for discovering
subtypes of the same disease and understanding thereasons for differences in nature and severity.
6.3. Robots for Supporting Children with Autism
Spectrum Disorder (ASD)
Hirokawa (2014a, 2014b) et al. have proposed a robot
intervention activity for children with autism spectrum
disorder (ASD), a developmental disability, to improvetheir social skills and promote their communication
abilities. They have developed a robot manipulationinterface that can be used by supporters for this purpose,as well as a method for quantitatively measuring face-to-© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

213

Kai Seinoof the worth of the tale of the party concerned and theparty concerned. The following specific suggestions are
based on such a system. (1) Consider the community
itself which generates a tale and utilizes it to be a system
and as well as its construction and practical use. (2) The
idea of a system and the realization that this idea places
more value on the process rather than the contents of tale
generation. (3) Get participation and collaboration of theconcerned parties towards realization and practical use ofa system.

face behavior and facial expression during intervention
activities, and have verified its effectiveness.
Jain et al (2020) developed a machine learning algorithm
that automatically tells a robot when to encourage Autism
Spectrum Disorder (ASD), as a therapeutic intervention.
The reason for the use of robots in these proposals is thathumanoid robots, with mechanical features that arehighly compatible with ASD children and physical andsocial characteristics similar to those of humans, areexpected to play an important role in interventions for
ASD children.

7.2. Potential Applications of PsychologicalNarratology to AI or Robot

7. Discussion

Research had been accumulated on the use of AI androbot for clinical support and it was confirmed thatalgorithms had already been developed or started to be
implemented on robots. In addition, the previous studies
had the following four main objectives (Ikeda, 2019). (1)Improvement of medical treatment system, (2)Elucidation of etiology and pathogenesis, (3)Development of new treatment methods, (4)Reconstruction of diagnostic concepts and systems
Firstly, by machine learning of existing data, the use of
AI in clinical support has enabled to predict and identify
diseases, discover subtypes, and identify factors in thedevelopment of diseases (Stevens et al, 2019; Rahman etal, 2020; Chand et al, 2020; Yassin et al, 2020).
If this method is applied to psychological narratology,
applications to detect subtype, and to predict and identify
tasks are considered throughout machine learning ofspeech, dialogue, and written text data.
Secondly, the use of robots for clinical support had beenstudied to improve the convenience and effectiveness ofrobots for supporting children with autism spectrum
disorder (Horikawa, 2014, 2014b; Jain et al, 2020). If theapplication to psychological narratology is consideredfrom this point, incorporating narrative elements into
robot for autism spectrum disorders is possible. Also, it
may be necessary to examine the effectiveness of robot
support for non-autism spectrum disorders.

7.1. Extension and possibility of a psychologicalnarratology
In this paper, the psychological narratology was divided
into some items and reviewed. The author showed thatthe narrative approach in clinical domain, defined as
psychological narratology, went beyond the narrative
therapy in psychotherapy and was expanded to various
experiments and research. Moreover, is thought that three
features are among the psychological narratologies
reviewed in this paper. As the first, many practice is not
simple technology and method but stance, and a way of
thinking. Since it has narrative approach against thebackground of social constructionism, fixation is avoided
and relativized. As the second, the narrative can point out
being made communally. The third point is all the tales
are built through a dialog. They were not monologues andwere constructed through colloquial expressions.
Based on these points, the narrative generative system isconsidered in both a narrow sense and a broad sense. The
following possibilities exist regarding the automatic tale
generation machine realized as a narrative generative
system in a narrow sense, i.e., a computer program etc.
(1) It is the construction of a fluid system. For example,
a random element is taken in accepted. This system
carries out narrative generation by combining elements.
(2)Two or more persons are the systems that generate atale communally. (3) It is a system of the tale generation
through simple dialogue but not an input.
In addition, the following is mentioned as a suggestion
for the practice of this research. A psychologicalnarratology is not merely a methodology but is also themechanism for converting of a relative sense of values. It
is given significance with conversion (or improvement)

8. Conclusion
This research defines the narrative approach of a clinicaldomain as a psychological narratology, and examines its
extension and possibility for application on AI or robot.
As a result of the review, psychological narratology wasconfirmed to have widely expanded beyond the so-called© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

214

Extension of Clinical/Psychological Approachnarrative therapy of psychotherapy and family therapy.
In addition, suggestions from these practices are thoughtprovoking and also utilize expansion, realization, and thesystem of the narrative generative system in a narrow
sense and a broad sense for actual problem solving. In
addition, AI or robot research in the clinical domain has
been identified. This gives suggestions to the extensionof psychological narratology. It is meaningful to consider
the practice of a clinical domain from the viewpoint of anarrative, as well as in respect to clinical reduction. We
expect that expansion of new discovery and practice will
be performed because a supporter, and a researcher inconnection with a theme, and the researcher of related
domains, such as A.I., computing science, cognitive
science, will argue and collaborate in the future.

9.
10.
11.

12.

13.

References

14.

1. Y. Yamada, Shitsutekishinrigaku no houhou: Katari o kiku
[Method of qualitative psychology: Listening to narrative].
Shinyousha, 2007.
2. G. Prince, A dictionary of narratology (Revised ed.).
University of Nebraska Press, 2003.
3. J. Bruner, Actual minds, possible worlds, Harvard
University Press, 1986.
4. K. Seino, K, Y. Haruna and S. Ishizaki, Employment status
and support needs of persons with disabilities in Japan: An
analysis of narrative using narratology and text mining ona national survey. in T. Ogata, and T. Akimoto (eds.),
Computational and cognitive approaches to narratology(pp .245-275). Information Science Reference , IGI Global,
2016.
5. K. Seino, Y. Enomoto and S. Miyazawa, Narrative
analysis of employment support for students with
developmental disabilities: Using an objective analysis of
free-expression answers. In T. Ogata and S. Asakawa(eds.),
Content
generation
throughnarrative
communication and simulation (pp. 341-357). Information
Science Reference , IGI Global, 2018.
6. C. D. Riessman, Beyond reductionism: Narrative genres indivorce accounts, in Journal of Narrative and Life History,
1(1), 1991, pp.41-68.
7. D. Clandinin, D. (ed.), Handbook of narrative inquiry:
Mapping a methodology. Sage Publications, 2007.
8. T. Ogata (2010). “Monogatari seisei sisutemu” no haikei
oyobi monogatari to bungaku no aida [Backgrounds of the“narrative generation system” and “relations between
narrative and literature”]. In T. Ogata & A. Kanai,
Monogatariron no jouhougaku josetsu: monogatari seisei
no shisou to gijutsu wo megutte [An introduction to theinformatics of narratology: about the thought and

15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.

technologies of narrative generation] (pp. 186-258).
Tokyo, Japan: Gakubunsha.
V. Y. Propp, Morphology of the folktale (L. Scott, Trans.).
University of Texas Press, 1968 (Original work published1928).
G. Genette, Narrative discourse: An essay in method (J. E.
Lewin, Trans.). Cornell University Press, 1980. (Original
work published 1972).
T. Akimoto, T and T. Ogata, Towards a discourse
mechanism in narrative generation system: Proposal of asystem introducing narrative discourse theory andreception theory. Cognitive Studies, 20(4), 2013, pp.396420.
T. Ogata, Narrative generation system as the practice of“informatics of narratology.” Journal of Japan Society for
Fuzzy Theory and Intelligent Informatics, 23(5), 2011,
pp.676-685.
T. Ogata, Proposal of computational narratology. In
Proceedings of IPSJ SIG Computers and the Humanities(pp. 53-60), 1999.
S. Aoki, T. Ogata and J. Ono, Narrative generation related
to cognitive patterns seen in ASD -From the perspective of“Surprise”-, In Proceedings of the 35th Annual Meeting of
the Japanese Cognitive Science Society (pp. 652-660).
(2018).
C. K. Riessman, Narrative analysis. Newbury Park, CA:
Sage, 1993.
Y. Noguchi (ed.), Naratibu no rinshoshakaigaku [Clinicalsociology of narrative]. Keiso shobo, 2005.
T. Ogata, Narratology of Narrative Generation or PostNarratology, in Cognitive Studies, 25(2), 2018, pp.200217.
G. H. Guyatt, Evidence-based medicine. American
College of Physicians Journal Club, 1991. 114, A-16.
D. L. Sackett, W. S. Richardson, W. Rosenberg and R. B.
Haynes, Evidence-based medicine: How to practice andteach EBM. New York, NY: Churchill Livingstone, 1997.
K. J. Gergen, Toward transformation in social knowledge.
Sage Publications, 1994.
K. J. Gergen, An invitation to social construction. Sage
Publications, 1999.
T. Greenhalgh and B. Hurwitz (eds.), Narrative based
medicine, BMJ Books, 1998.
N. K. Denzin and Y. S. Lincoln (eds.). Handbook ofqualitative research, Sage Publications, 2000.
M. White and D. Epston, Narrative means to therapeuticends. New York: Norton. in Transformative Learning
Conference, Oct. 1990, pp. 23–25, 2003 184/492
M. Foucault, Power/knowledge. (C. Gordon, L. Marshall,
J. Mepham and K. Soper, Trans.). Pantheon Books ,1980.
(Original work published 1972, 1975, 1976, 1977).
C. D. Spence , Narrative persuasion, in Psychoanalysis
and Contemporary Thought, 6, 1983, pp. 457-468.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

215

Kai Seino

27. H. Kawai, Shinriryoho niokeru Monogatari no Igi[Significance of the "narrative" in psychotherapy], inJapanese Journal of Psychotherapy , 27, 2001, pp. 3-7.
28. E. Polster, Every person’s life is worth a novel. W. E.
Norton & Company Inc, 1987.
29. A. H. Katz and E. I. Bender (eds.), The Strength in Us:
Self-Help Groups in the Modern World. New York: New
Viewpoints, 1976).
30. T. Borkman, Experiential knowledge: A new concept for
the analysis of self-help groups. in Social Service Review,
50 (3), 1976, pp.445-456.
31. T. Ito, Serufu herupu gurupu to kojin no monogatari [Selfhelp groups and personal stories]. in JapaneseSociological Review, 51(1), 200, pp. 88-103.
32. T. Saio, Open dialogue approach to acute psychosis: Is its
effectiveness well-established? in Clinical Evaluation, 42（2）, 2014, pp. 531-537.
33. I. Mukaiyachi, Tojisha-kenkyu toha: Tojisha-kenkyu norinen to kousei [What is a Tojisha-Kenkyu? The concept
and structure of a Tojisha-Kenkyu]. Tojisya-Kenkyu
network (https://toukennet.jp/?page_id=56) , 2013.
34. S. Ikeda, Current Trends, Issues and Potentiality of AI
Technology in Psychiatry. in The 33rd Annual Conferenceof the Japanese Society for Artificial Intelligence, 2N5-J13-03, 2019, pp. 1-4.
35. R. Rahman, A. Kodesh, S. Z. Levine, S. Sandin, A.
Reichenberg and A. Schlessinger, Identification of
newborns at risk for autism using electronic medical
records and machine learning, in Eur Psychiatry, 63(1),2020, e22.
36. E. Stevens, D. R. Dixon, M. N. Novack, D. Granpeesheh,
T. Smith and E. Linstead, Identification and analysis ofbehavioral phenotypes in autism spectrum disorder via
unsupervised machine learning. in Int J Med Inform, 129,2019, pp. 29-36.
37. G. B. Chand, D. B. Dwyer, G. Erus, A. Sotiras, E. Varol
and D. Srinivasan,et al. , Two distinct neuroanatomical
subtypes of schizophrenia revealed using machinelearning. in Brain, 143(3), 2020, pp. 1027-1038.
38. W. Yassin, H. Nakatani, Y.Zhu, M.Kojima and Owada K.,
et al., achine learning classification using neuroimaging
data in schizophrenia, autism, in Translational Psychiatry.
10(1), 2020, pp.278.
39. M. Hirokawa, A. Funahashi, and K. Suzuki, A Doll-type
Interface for Real-Time Humanoid Teleoperation in
Robot-Assisted Activity: A Case Study, in Proc.
ACM/IEEE Intl. Conf. on Human-Robot Interaction, 2014,
pp.174-175.
M. Hirokawa, A. Funahashi, Y. Itoh and K. Suzuki, Designof affective robot-assisted activity for children with autism
spectrum disorders, in Proc. of The 23rd IEEE
International Symposium on Robot and Human Interactive
Communication, 2014, pp. 365--370.
40. S. Jain, B.Thiagarajan, A.Shi, C. Clabaughand and M.J.
Matarić, Modeling engagement in long-term, in-homesocially assistive robot interventions for children with

autism spectrum disorders, in Sci. Robot, 5(39), 2020,
eaaz3791.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

216

A Relationship between Narratology and Marketing
Akinori Abe

Faculty of Letters, Chiba University, 1-33 Yayoi-cho, Inage-ku,
Chiba 263-8522, Japan

Abstract
We have been studying an expression of the taste of Japanese sake. Actually, the descriptions of the taste can be
regarded as stories. Accordingly, we conducted an experiment, in which we asked participants to draw descriptionsof the taste of the Japanese sakes and to design labels of them. From the results of the experiment, we will analyse
the relationship between narratology and marketing of Japanese sake.
Keywords: Narratology, Taste of sake, Marketing.

We could collect several results. From the results ofthe experiment, we will analyse the relationship between
narratology and marketing of Japanese sake.

1. Introduction
We have been studying an expression of the taste ofJapanese sake (alcoholic drink) 1,2,3,4. Since we do nothave a dictionary of the taste of Japanese sake, we tried
to create words or phrases for the taste of Japanese sake.
Generally we do not describe the taste of sake as a still
situation, but as a changing situation. During describing
the taste, we will deal with time. We will describe agradual change of the taste. Thus the descriptions of thetaste can be regarded as stories. The descriptions areinteresting as a set of corpus. However more interesting
matter is that we may use the descriptions for themarketing. Since the descriptions are generated from thetaste of sake, it should contain the taste. And if labels aredesigned based on the description, it should contain theinformation of the taste.
Usually labels of the Japanese sake are not attractive.
At least, what labels should inform us are name of thesake. No attractive matter is necessary for selling the sake.
However, according to the Jensen's theory5, it isimportant to prepare a story for selling products.
Accordingly we think it will better to include a story in
the labels of sakes. At least labels of sakes should be
attractive for a certain generation.
Therefore, we conducted an experiment, in which we
asked participants to draw descriptions of the taste of theJapanese sakes and to design labels of them. The design
should include the taste information of the sake.

2. Description of Sake as a Story
2.1. Descriptions of the taste of sakeIn several books, we can ﬁnd the descriptions of taste ofsake. For instance, for the taste of Japanese sake, in“LOVE ♡ Japanese sake!, 2014, 2 Gakken,” we can ﬁnd
the following descriptions:• Kamoshi-bito Kuheiji (醸し人九平次) Eau Du Desir2012
At the ﬁrst moment in the mouth, strong fruit ﬂavour
can be felt. After a proper maturing time, the mouthis fragrant with the smell of honey and vanilla. A
good ﬂavour is spread and in the center of it splendid
sour can be found... (口にした瞬間は果実のフレーバーが強く感じるものの、適度な熟成を経ると蜂蜜やバニラの香を覚えます。旨味は華やかな酸を軸に広がりますが....)• Isojiman Junmai ginjou (磯自慢純米吟醸)
Refreshing scent such as the scent of muskmelon andpear is elegant... (マスクメロンや洋ナシを思わせる爽快とした香りは気品すら感じさせるもの....)
The above descriptions do not include stories. Perhaps,for the evaluation, subjective descriptions will not be© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

217

Akinori Abe

preferable. On the other hands, descriptions in several
book contain the stories of production such as “Mr.
Fernandez who was 79 years old had spent all his life forthe wine production (El Vinclo 2010 by Real Wine Guide
Vol. 52 (2016, Winter))” and “This taste was born by themalted rice which the president Kimizuka in the winterputs his heart and soul into producing and thefermentation technique that pulls out the potential tasteof Fusakogane (an eating rice only in Chiba prefecture)at the best (Naruka special junmai direct packing raw[white] (鳴海特別純米直詰め生【白】) the homepageof Nishiura liquors shop (酒舗にしうら)).”
Previously, we described the taste of Japanese sake1. I
will show one of the descriptions; The followings arerecords of two person's expressions for the taste ofJapanese sake. We tasted the same sake (Shizengou
seven (自然郷 セブン)) at the same time in the sameroom. We used different cup but the environmental
condition will be almost the same.

Fig. 1. A label of Japanese sake (華姫桜し ずく 媛無濾過純
米吟醸原酒)

our expectation. According to the home page, her nameis Sakurashizuku ( 桜 雫 ) and she is 138 years old.
Accordingly the sake might be matured sake. However, I
do not think actual taste of the sake is coincide with thelady in the illustration. At least, the taste is not the similar
to what I expected when I saw the label. This illustration
might represent the image of the sake as the sake brewery
imaged, but does not represent the taste of the sake. Ifthey want to sell this sake to those like idols, it might besuccessful. Because the illustration contains the story
which will be liked by idol lovers. This label includes astory with this lady. We would like to buy the sake withregard to the story read in this label. Thus a story in the
product will play an important role in marketing.

• A: The first impression in my mouth was oh good
taste. However, after that the taste disappeared as if
water escaped from the side of the tongue. The firsttaste of sake did not spread and taste like water went
through tongue then escape from my mouth between
teeth. Very “interesting” taste.
• F: The first half of tongue can feel good taste. I felt
oh good taste. However gradually the taste
disappeared as if the taste was veiled in mist. At thelast, I had an aftertaste such that sugar is diluted inwater. I felt quite a few satisfaction. If I dare todescribe the last half taste as a refreshing taste, it canbe described that the taste is tightened.

3. Experiment

In fact, we used the phrases for the flow of time.
Though they do not contain the story of the producer nor
process of production, the flow of time during the tasting
can be regarded as a story.

We conducted an experiment as follows;
3.1. Objectives
Sub-headings should be typeset in boldface italic andcapitalize the first letter of the first word only. Section
number to be in boldface Roman. We reviewed the roleof narratology in the previous section. We concluded in
the previous section that a story in the product will play
an important role in marketing. Accordingly, we would
like to see how can we generate stories from the productand how can we express the stories for the marketing.

2.2. A label of sake
Certain labels of Japanese sake contains certain
illustrations. “The Japanese sake stories (日本酒ものがたり)” (http://sakemono.com/) by the Japanese sake ×
produce production project (日本酒×作家創作プロジェクト) is conducted as an intention to draw a story in
the Japanese sake by artists. For instance, the illustrationon the label of Hanahimesakura Shizuku-hime nonfiltered junmai ginjou no-water-added sake (華姫桜 しずく媛 無濾過純米吟醸原酒) is shown in Figure 1.
This is an anime-like illustration.
Though this sake is a non-filtered junmai ginjou nowater-added sake, its taste was fully matured opposite to

3.2. Participants
Seven persons from 21 to 29 years old took part in theexperiments. Some of them frequently drink Japanesesakes and describe the taste of them.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

218

A Relationalship between Narratology

3.3. Method
We used the following six kinds of sakes: Ichijima
junmai ginjou raw no-water-added sake (市島 純米吟醸
生原酒), Azumaichi junmai daiginjou (東一 純米大吟
醸 ), Azumaichi Yamadanishiki junmai-shu 64\%
polishing (東一 山田錦純米酒 64\%磨き), Azumaichi
Nero, Azumaichi junmai daiginjou drip squeezing (東一
純米大吟醸 雫搾り), Inabazuru junmai ginjou gouriki
freshly squeezed non-filtered raw no-water-added sake
Fig. 3. One of results in this experiment(taste) The taste comes again and again. I feelsomewhat an image of rain and shadow.

4) (fragrance) I feel a straight alcoholic fragrance.

Fig. 2. Japanese sakes used in this experiment(taste) Strong sour and slightly sweet. After taste
includes spicy. Bitter taste is weak.

(いなば鶴 純米吟醸 強力 しぼりたて 無濾過生原酒)
We asked the participants to taste Japanese sake todescribe the taste. The description could include texts andﬁgures. Then we asked them to create catch copies for the
sake. In addition, we asked them to design the label of thesake. The target person was young person in 20’s. Those
descriptions and drawings were drawn on the papers in
A3 size. They used several pens such as coloured pensand pencils.

5) Spicy bananamine) I feel fragrance of cloth. First sweet taste suchas pears comes to mouth and spread slightly.
Unfavourable taste flows out of mouth but it is nothate taste. With a duck, I can taste without
conflictions. Smoked taste becomes weak.
• Azumaichi junmai daiginjou
1) I feel sweet and floral feeling. When I open thesake, I feel a fragrance from the 20cm distance, thetaste is soft and I feel umami of rice. The more I drink,
I become aware of biting taste. If I drink a lot, spreadsof the sake will change? It is easy to drink, so I candrink more and more. I think I can get drunk
comfortably. It will be good for drinking when tired?
But it is not good to drink before holiday. I might not
be aware of having a hangover.

3.4. Results
We collected such results as shown in Figure 3.
First, we show some descriptions of the taste of sake.
All descriptions were written in Japanese. We translated
them in English.
• Ichijima junmai ginjou raw no-water-added sake1) Delicious. I like this because I can smell and taste
rice. Since it contains spicy taste, the taste comes
gradually. At first, I feel gentle taste, but in latter half
I feel spicy taste.

2) Refreshing fragrance. The first attack is pliant
sweet. It is rather bitter. The balance of the taste isgood. After a while, I feel a bitter taste more. The
target generation of the sake will be elder persons.
Younger generation will not like this. The taste is
bitter, but after taste does not remain long. So it will
be good for with a meal. After a long time, the tastechanges flat and I do not feel a bitter taste.

2) Jyowa jyowa--- The fragrance disappears suddenly
and clearly. A refreshing fragrance which is going out
from nose. For the summer season? I can not say it isa refreshing type. With snack it is not good. With aCamembert cheese, the taste becomes soft. For aparty? It is difficult to drink without any foods.
3) (fragrance) I like the fragrance.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

219

Akinori Abe3) (fragrance) Fresh and young type. (taste) strongsour. At (1), a prickly taste comes and at (2) also a
prickly taste comes. Then the taste spread to (3)rapidly. The taste remains more at (1) and (2) than at(3). It has a peculiar taste but is easy to drink. It is afeminine type sake.
4)(fragrance) Rather strong fragrance.
(taste) Just after putting the sake in my tongue, I feela spicy taste. I feel sweetness little and the after taste
has a bitterness. But the taste is more refreshing thanthat of Azumaichi junmai ginjou. I feel a spicy taste
at the top of my tongue.

4) (fragrance) Sweet fragrance of rice which comes
my nose straightly and a straight alcoholic fragrance.

5) Middle: clear, a clear stream, acid, forest, summer,
late summer, September, sun light through the trees
Later: Autumn rice field with swaying inaho,refreshed taste. A fine autumn day. A rim of the taste
is clear.

(taste) First, I feel a sweetness of rice. The taste
becomes bitter as the sake enters the back of tongue.
At the throat I feel a biting taste. It is easy to drink(because a bitter taste dose not come suddenly). After
taste is sweetness of a rice and I feel a biting taste in
the throat. At the back of throat, I feel a little bitter
taste.

mine) A little fragrance of an alcohol exists bottom of
the sake. The taste is matured and at the bottom of thesake, an alcoholic taste sinks. At the upper side of thesake exists thin sweetness. I feel a spicy taste at the
top of my tongue. I feel an astringency spreading
around my cheek.

mine) Fragrance like a dust. A taste like a fruit (apple)
comes in the shape of round and thickly. The taste
does not spread and comes heavily. However, At the
top of the tongue, spicy taste. In the mouth, aremaining taste is faintly sweet and clear. If it does
not have spicy taste, it is clear taste.

• Azumaichi Nero
1) Its taste is similar to that of Azumaichi junmai
daiginjou? If it is said as sweets, it will belong to thesweets. The taste is rather sweet and it comes
gradually. The taste is not spreading wider than thatof Azumaichi junmai daiginjou. I feel like a drinking
sake more than Azumaichi junmai daiginjou drip
squeezing.

• Azumaichi Yamadanishiki junmai-shu 64% polishing
1) A sharp feeling. Cool. Good for summer. Coolfeeling → I cannot feel the fragrance. The taste issour? like a caramel. I feel bitterness rather thansweetness. But sweet. It’s difficult to deal withexpressing the taste. The taste spreads a little. The
taste stops at the place where I feel like to go more. It
the type of sake that tantalizes after appealing to me.
It is like a quiet and cool pretty girl such as Rei
Ayamaru (綾〇レイ).

2) I can smell nothing. Its taste is calm and gentle. I
feel carbonic acid. I think it is better to drink withyakitori. Perhaps with sashimi. After a little while, it
becomes melting. And more time after, it becomessticky.

2) It is a normally delicious sake. It is neither goodnor bad. It is better than Ichjima and Azumaichi
Junmai daiginjou. If I do not have any sake, I will
drink. I can drink with every foods.

3) (taste) Its taste is strong. (4) remains ..., but is rathersmooth. At the last, taste is clean.

3) (fragrance) astringent. (taste) A prickly taste comesand spreads with numbing. After drinking the sake, at(2) and (4) the taste remains. In addition, at (5), thetaste remains. The taste is like a little bitter chocolate
remains in the mouth (A cacao degree is high.). The
taste in (3) moves back and forth according to the wayof putting it in my mouth. The taste in (3) spreads
overall slowly. The after taste is a chocolate.

4) (fragrance) I feel a straight fragrance. No sweet
fragrance I feel.
(taste) The taste of sweetness of a rice and sour
strongly comes. The after taste is slightly (gentle)
spicy and bitter.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

220

A Relationalship between Narratologymine) I feel a closed fragrance. A fruit type of taste
comes smoothly. It spreads very little and on my
tongue sweetness remains. I feel little unfavourable
taste. If I inflate the sake in my mouth, a little sourcomes and the taste of the sake becomes very wellbalanced. At the last, I feel a spicy taste on my tongue,
but I mind a little.
• Azumaichi junmai daiginjou drip squeezing1) The spreading of the taste is a little. I have aunremarkable impression. Spicy? I feel a prickly taste.
I think I can recommend to everybody?

mine) At first, the taste is closed. A sour taste stands
out, but is not strong. It is easy to drink, but does nothave a special feature.
Next, we show some catch copies for sakes.
• Ichijima junmai ginjou raw no-water-added sake2) Best for the summer party... with cheese and ham.
3) It's raining... today I walk alone.
4) Spreading taste.
5) If we get drunk, everything is tasteful.
• Azumaichi junmai daiginjou
2) For the sake lovers!! The best in Azumaichi.
3) Scared 怖い lady (gentle lady)
4) Seven changes
5) Flower blooms four times.

2) I feel a rich fragrance. This is a hoping sake. The
taste is not bad. I will drink later. A cheap taste. I candare to drink it in a train alone. For a party. Afterseveral minutes, a bitter taste comes a little.

• Azumaichi Yamadanishiki junmai-shu
polishing1) A friend with an evening drink.
2) The entry item of Azumaichi!!
3) The first step to adult.

3) (fragrance) I smell a little. pretty.
(taste) like a melon. pretty. At (3) the taste comes witha heavy impact. “It is not just a pretty face.” I feel it
is like a pearl.

64\%• Azumaichi Nero2) It's very clear taste. So you can taste it with bothmeats and fishes.
5) 25 years old, week end, girls party.
• Azumaichi junmai daiginjou drip squeezing2) Please taste slowly.
4) Happiness of a drop.
5) Metamon transforms Japanese sake. Drinking riceball sake.

4) (fragrance) Fragrance of sweetness of rice andalcohol.
(taste) I feel a taste of strong flavour of a rice. In
addition, I feel sweetness and slightly acidity. On thethroat, I feel relatively strong spicy taste. Even if I
drink it with air, a spicy taste comes.

Several catch copies were collected. These copies arefor younger generation. An interesting copy is that for
Azumaichi junmai daiginjou. It might have changing
taste, accordingly they generate phrases with words
meaning change.
Then according to the catch copies, participants
created labels for the sakes as shown below:5) Mowa--n, unformel, it has an outline.
mine) I smell fragrance of cloth a little. On the
periphery, clear sweetness exists. The taste enters inmouth cleanly. I feel a spicy taste over the sweetness.
Its clear sweetness is like a delicious water andbeautiful. However, I want a more spreading taste.
Without the last spicy taste, it will become a nonstimulative sake? It it had umami or rich flavor, it willbe the best.
• Inabazuru junmai ginjou gouriki freshly squeezed
non-filtered raw no-water-added sake2) I drink it with sweets. The taste is not bad. (It might
because I drink it with sweets.) It smells like fruits(green apple).

• Ichijima junmai ginjou raw no-water-added sake

• Azumaichi junmai daiginjou© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

221

Akinori Abe• Azumaichi
polishing

Yamadanishiki

junmai-shu

A sommeliere sometimes describes a taste of a winewith a scene such as “walking in a forest with stepping
firmly dead leaves.'” This expression comes from thechanging taste. We think for the description of the tasteof Japanese sake, this type of expression can be made.
Reviewing the labels, we think a story of the taste is
described in certain label. And the story (narratology)functions well in the labels. We think illustration in thelabel expresses the story of the taste of the Japanese sake.
We think it will be better to express the story of the tasteof the sake. It has a lot of information to select a product.
We did not conduct an experiment to evaluate thedesign of labels by somebody. This is a unsolved
question for the future.

64\%

5. Conclusions•

In this paper, we conducted series of experiments. First,
we asked the participants to taste Japanese sake todescribe the taste. The description could include texts andfigures. Then we asked them to create catch copies of thesake. In addition, we asked them to design the label of thesake.
We could collect several expressions from thisexperiment. The expressions are very interesting. The
more interesting results are labels designed by theparticipants. We think some labels can be used for themarketing.
We have not evaluate the designs. In the future, it isnecessary to evaluate them by outsiders.

Azumaichi Nero

Acknowledgements• Azumaichi junmai daiginjou drip squeezing

We thank Mr. Ippei Setou for offering us Japanese sakes(Azumaichi).
References
1. Akinori Abe: About the better expression of taste, Proc.of
the 2nd. Int’l Workshop on Language Sense on Computerin IJCAI2017, pp. 28–34 (2017)
2. Akinori Abe: Is the expression of taste abstract orrepresentational?, Proc. of SMC2018 Workshop on
Narratology and Cognitive Content Generation, pp. 18–23(2018)
3. Akinori Abe: Expression of the taste of Japanese sake andmetaphor, Proc. of ICAROB2019, pp. 589–592 (2019)
4. Akinori Abe: Taste of Japanese Sake as Metaphor andShape, Proc. of IJCAI2019 The 3rd. International
Workshop on Language Sense on Computer, pp.29–34(2019)
5. Rolf Jensen: The Dream Society: How the Coming Shiftfrom Information to Imagination Will Transform Your
Business, Mcgraw Hill (1999)

Some labels have stories.
4.

Discussions

In the experiment, we collected several results. They aredescriptions of the taste of the sake, catch copies for the
selling the sakes, and labels of sakes. We expected stories
in the descriptions of the taste. Some participants
described changing tastes. We think these are stories forthe sakes.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

222

Layout decision system for multiple production lines
using work-flow-line and GA
Masato Noda

Department of Intelligence Science and Engineering, Gifu University, Yanagido 1-1,
Gifu, Gifu 501-1194, JAPAN
Hidehiko Yamamoto

Department of Mechanical Engineering, Gifu University, Yanagido 1-1,
Gifu, Gifu 501-1194, JAPAN
Hirohumi Tsuji, Yasuhisa Terawa, Yoshinori Nakamura, Masayuki Tsuchida

InfoFarm co.,Ltd, Yanaizu-cho Distribution Center 1-8-4
Gifu, Gifu 501-6123, JAPANKatsuaki Yamada, Yukiyasu Kuriyama

Kai industries co., ltd, Oyana 1110
Seki, 501-3992, JAPANE-mail: y4525062@edu.gifu-u.ac.jp, yam-h@gifu-u.ac.jp

Abstract
We develop the system to decide the efficient layout of assembly production line by using Genetic Algorithm (GA).
We call the system as System of Production-line-layout Decision by Chameleon-code and GA (PDCG). PDCG
decides the efficient layout of production line by using GA, work-flow-line acquired by Chameleon code and themachine breakdown data. PDCG evaluates the layout efficiency by calculating the operator’s walking time in order
to fix the machine breakdown occurred on the production line. By the evaluation.
Keywords: Genetic Algorithm, Chameleon-code, Assembly production, Work-flow-line, Machine breakdown

line to verify whether it is effective in improving work
efficiency.

1. Introduction
In order to improve the production efficiency, many
Japanese companies adopt Kaizen. The line manually
performed by several operators is needed to be also
improved. In this study, we develop the system of
Production-line-layout Decision by Chameleon-code andGenetic Algorithm (PDCG). PDCG determines thelayout of multiple production lines that some operators
work. The determination is given by using Genetic
Algorithm (GA), and the work flow lines obtained from
Chameleon-code. PDCG is applied to the razor assembly

2. Work-flow-line with Chameleon code
Chameleon code is the new color barcode whose function
has a high speed and precision recognitions as shown in
Figure1.This study puts Chameleon code on the top of an
operator’s hat and obtains the operator’s work flow line
data by overhead cameras. The acquired work flow line
data includes the name of the operator, the camera names,
the dates and times, and the X and Y coordinates of thechameleon code.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

223

Masato Noda, Hidehiko Yamamoto, Hirohumi Tsuji, Yasuhisa Terawa, Yoshinori Nakamura, Masayuki Tsuchida, Katsuaki Yamada andYukiyasu Kuriyama

4.

Individual representation

The individual representation of GA used in PDCG
adopts the structural gene, as shown in Figure 3. We
express the x-coordinate, the y-coordinate and the angle
indicating the position of one production line as the threeconsecutive loci. When representing multiple productionlines, these three loci are connected in series. Figure 3
shows the individuals in the case of three production lines.
The x-coordinates, y-coordinates and angles are shown in
Figure 4. In order to secure the movement path of theoperator, we set the constraint that the width between
each production line is an arbitrary value Z mm.
Individuals that don’t meet the constraint are deleted aslethal genes.

Fig. 1. Chameleon-code

3. PDCG outline
PDCG is the system to determine the layout of
production lines using GA, based on the work flow line
database acquired from the chameleon code and themachine breakdown database. Specifically, PDCG
calculates the walking time of operator moves to fix
machine breakdowns, evaluates the operator’s walking
time and determines the better layout where one operator
manages some production lines. As shown in Figure 2,
PDCG consists of the condition module and the GA
module. Condition module reads the acquired work flowline database and sets various parameters for GA. GA
module determines the better production line layout by
using GA.

Fig. 3. Individuals expression

Input data

・work flow line data・machine breakdown data
Condition






Read input data
Production line modelingIndividual expression
Fitness evaluation values
Setting each parameterOperator’s walking time
Genetic algorithm

Fig. 4.

GA

How to judge lethal chromosome

5. Fitness evaluation values
PDCG adopts the walking time of the operator during theoperating time as the evaluation values of fitness. If thewalking time of individual 𝑖𝑖 is 𝑡𝑡𝑖𝑖 and the slowest
walking time among the population of a generation is 𝑡𝑡𝑤𝑤 ,the fitness 𝑓𝑓𝑖𝑖 of individual 𝑖𝑖 is expressed by thefollowing equation.

Output data

・Coordinates・Operator’s walking
time
i l i
i

𝑓𝑓𝑖𝑖 = (𝑡𝑡𝑤𝑤 − 𝑡𝑡𝑖𝑖 + 1)2

Fig. 2. PDCG outline(𝑖𝑖 = 1,2,3, … )© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

224

(1)

Layout decision system forproduction line and the contours of other production lines.
⑶ If the calculated distance is less than Z mm, theindividuals are deleted as a lethal gene. ⑷Return to ⑴and regenerate the individual.

If the time difference between each individual andthe slowest individual is taken as the fitness, thedifference in fitness between an excellent individual andanother individual becomes small. To make large
difference, we give the difference a square. (1)．

Step2: Calculate the machine breakdown coordinates foreach production line.
The coordinates whose machine gets breakdown in thefactory layout change according to the coordinates of the
production line layout. That is, since the coordinates of amachine breakdown occurrence for every generated
individual are different, it is necessary to calculate eachwalking time. In PDCG, the coordinates of a machine
breakdown occurrence of all individuals are calculatedby the following operations.
(1) Search the work flow line data corresponding to themachine breakdown occurrence time in the machine
breakdown data and find the coordinates of the machine
breakdown that occurred in one production line. (2)Based on the found coordinates, calculate the machine
breakdown occurrence coordinates for other all
individuals. For example, if the coordinates and angles of
one production line A of an individual generated in Step
1 are (𝑥𝑥𝑎𝑎 , 𝑦𝑦𝑎𝑎 , 𝜃𝜃𝑎𝑎 ) and the machine breakdown
coordinates found in (1) are (x_e,y_e),the machine
breakdown coordinates (𝑋𝑋𝑒𝑒 , 𝑌𝑌𝑒𝑒 ) in production line A are
expressed as the following equations.

6. GA module algorithm
The GA module is the module to improve the efficiency
of production line layout by GA. The algorithm of the GA
module is shown below.
Step1: Generate the initial population.
Step2: Calculate the machine breakdown coordinates
for each production line.
Step3: Calculate the walking time of the operator.
Step4: Calculate fitness.
Step5: Judge whether the end condition is satisfied ornot, and if it is satisfied, go to Step 8. If not, go backto Step6.
Step6: Carry out selection, crossover and mutation
operations.
Step7: Carry out Step 2 on the new generation’s
individuals.
Step8: The production line layout of the individualwith the shortest walking time is adopted as thesatisfying layout and is output as a solution.
We describe the details of the above algorithm.
Step1: Generate the initial population.
The individuals of the initial population are randomly
selected from a range of coordinates that can be placed in
the factory layout where the x- and y-coordinates andangles of the production line are applied. When there are
multiple target production lines, the layout is randomly
generated. There is a possibility that the production line
layout may overlap or be too close. In this case, theselayouts correspond to lethal genes. Not to include thelethal genes in the population, the following operations
are carried out.
⑴ Generate individuals.
⑵ As shown in Fig. 4,
calculate the distance between the contour of a
Time
411
565
890
1,156

X coordinate
3,513
3,858
4,128
2,651

𝑋𝑋𝑒𝑒 = 𝑥𝑥𝑎𝑎 + 𝑥𝑥𝑒𝑒 ∗ 𝑠𝑠𝑠𝑠𝑠𝑠𝜃𝜃𝑎𝑎
𝑌𝑌𝑒𝑒 = 𝑦𝑦𝑎𝑎 + 𝑦𝑦𝑒𝑒 ∗ 𝑐𝑐𝑐𝑐𝑐𝑐𝜃𝜃𝑎𝑎(𝑒𝑒 = 1,2,3, … )(𝑒𝑒 = 1,2,3, … )

(2)
(3)

Step3: Calculate the walking time of the operator.
When a machine breakdown occurs, the operator
moves from the waiting place to the place where themachine breakdown occurred to fix it. PDCG calculates
the total walking time to get there and back.
Step4: Calculate fitness by using the equation (1).

Table.1 Mechanical breakdown example
Y coordinate
Breakdown name A
4,046
Whether the blade is abnormal
2,046
Whether the blade is abnormal
881
On standby
3,468
Abnormal presence or absence of frame© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

225

Masato Noda, Hidehiko Yamamoto, Hirohumi Tsuji, Yasuhisa Terawa, Yoshinori Nakamura, Masayuki Tsuchida, Katsuaki Yamada andYukiyasu Kuriyama

The simulation results are shown below. The
simulations were performed 20 times, and the layout
obtained by PDCG is the solution. The layout is shownin Figure 6. The solution corresponds to the individual

Step5: Judge whether the end condition is satisfied or not.
If it is satisfied, go to Step 8. If not, go back to Step6.
After the walking time of the operator is updated, if nobetter individual appears after certain generations, GA isfinished.
Step6: Carry out selection, crossover and mutation
operations.
Using the fitness calculated in Step 4 and using thefitness proportional method, select a set of the individuals
as a crossover target. As the crossover, we adopt the twopoint crossover. Mutations occur with a certain
probability within the population.

Fig. 6. Parallel layout and PDCG layout

with the minimum walking time. As a comparison, thewalking time of the current layout of Kai Industries Co.,
Ltd. which is arranged in parallel was also calculated.
The layout determined by PDCG shortened the operator
walking time by about 63 minutes compared to theparallel layout. The layout determined by PDCG was
effective in improving work efficiency.

Step7: Carry out Step 2 on the new generation’s
individuals.
Repeat steps 2 to 6 for the next generation population.
Step8: The production line layout of the individual withthe shortest walking time is adopted as the satisfying
layout and PDCG outputs it as a solution.
The coordinate data of the satisfying individual is
output, and the layout is represented as an image basedon it.

Table. 2.

Comparison of simulation results
Walking time[sec]

PDCG layout

2,405

Parallel layout

6,164

8. Conclusions

7. Simulations applications and results

After application simulations, it is ascertained that PDCG
using the chameleon code of Infofarm Co., Ltd. iseffective for the decision to find better layouts withmultiple lines that one operator works.

PDCG was applied to the razor assembly line in the Kai
Industries Co., Ltd. Since the shape of the razor assemblyline is extremely complicated, we simplified it to thegraphic model as shown in Fig. 5 and applied it to PDCG.

9. Reference(1) K. Honda, H. Yamamoto and T. Yamada, “Design
System of Cell Type Assembly Machine with Dual
Arms Robot by GA”, The 2018 International
Conference on Artificial Life and Robotics(ICAROB2018), Feb. 1-4, Beppu International
Convention Center, Oita, Japan

Fig. 5. Production line model

In this case, we acquired the work flow line data for 8hours per one shift for 3 days with the chameleon code
and also used the machine breakdown data of the sametime. We adopt the GA parameters as follows. A
mutation rate is 10%, a population size is 200 individuals,
one individual is preserved as an elite. Selection andcrossover methods were carried out by the fitness
proportional method and two-point crossover. We also
adopted b = 500mm and Z = 800mm.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

226

Autonomous decentralized FMS’s AGVs moving control
by mind change with deep learning
Ryunosuke Yamane

Department of Human Information Systems, Gifu University, Yanagido 1-1,
Gifu, Gifu 501-1194, JAPAN
Hidehiko Yamamoto

Department of Mechanical Engineering, Gifu University, Yanagido 1-1
Gifu, Gifu 501-1194, JAPANE-mail: y4525092@edu.gifu-u.ac.jp, yam-h@gifu-u.ac.jp, yamat@gifu-u.ac.jp
Abstract
This study describes the control method of Automated Guided Vehicles (AGV) movement by using a mind modelin order to avoid AGVs interferences by using two types of mind , the arrogant mind and the modest mind model,
the interferences can be avoided by repeating the two types of mind changes. This study develops the new mind
model. The model includes deep learning. By the mind of the model of this new method, we can improve thedecrease of the route interference time.
Keywords: Autonomous decentralized FMS, AGV with a mind, Deep learning

1. Introduction
We have developed the control of movements ofAutomated Guided Vehicles (AGVs) in an autonomous
decentralized flexible manufacturing system (AD-FMS).
The AGVs have a minimum model of mind (MUM) [1].
However, in the past research, if the number of AGVs
operating in the factory increases, it is difficult to deal
with efficient mind changes and it takes time to avoid
route interferences. As a result, production efficiency
decreases. In order to solve the above mentioned
problems, we develop the new mind model including
deep learning system. We call the model as Minimum
Unit of Mind with Deep learning (MUMD). We compare
the AGVs efficiencies of MUMD with the AGVs
efficiencies of our developed two mind models, MUM,
and MUMN by applying the three types of the mind
models to FMS constructed in a computer. MUMN is
including m a neural network into the MUM.

2. Autonomous Decentralized FMS
The example of AD-FMS is shown in Fig.1. The factory
floor is divided into a grid pattern where an AGV moves.
The system of AD-FMS is carrying the parts to thewarehouse or machining centers (MCs). The AD-FMS
does not have a management mechanism that controls thewhole system. The agents (MCs, AGVs and Warehouses)
which are the components of the AD-FMS exchange
knowledge and determine the behavior of each agent.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

227

Ryunosuke Yamane, Hidehiko Yamamoto

Fig.2 MUM model

The problem with MUM is that there are various patternsof path interferences in a large number of AGVs thatmove in AD-FMS. Because of it, it takes time to solveroute interferences.

Fig.1 Example of AD-FMS

3. AGV behaviors and mind
AGVs autonomously decide their actions in the AD-FMS
of this study. AGVs sometimes meat path interference in
AD-FMS. To avoid the interferences, we adopted if-thenrules in our past study. However, it was not possible
flexibly to cope in the case of increasing the number of
AGVs and changing the layouts of the factory. In orderto solve this, we insert the MUM to AGV and realize
autonomous movements without AGVs interferences.

3.2. MUMN
MUMN is a model of the heart that was developed inorder to achieve more efficient heart changes by using aneural network for the MUM and to change the heart
according to the situation. The MUMN model is shownin Fig. 3.

3.1. Mind model and its problem
The AGVs mind expresses two kinds of mind, thearrogant mind that takes action forcebly to approach thedestination and the modest mind that takes action to give
ways to other AGVs. To express the mind, we use MUM
in Fig.2. A1 and A2 are units, X are the weights andarrows are called stimulus vectors. Units have thresholds
and internal values. A unit whose internal value reaches
the threshold is called excited. A unit whose internal
value does not reach the threshold is called normal. By
the amount of the signal input to the unit, MUM changes
the mind to arrogant or modest frequently. We describe
the internal functions of MUM. When the arrogant AGV
has a path interference, A1 is increased by 1. Keeping thesituation of the interference and being increased by 1, theA1 value becomes the threshold, and the AGV is changed
to a modest one. When AGVs with a modest mind keep
giving a way, the value of A2 is increased by 1. When thesituation is repeated at optional times, A2 becomes
excited and a signal is sent to a load. The received loaddecreases the values of units A1 and A2 by optional
values. Owing to this, A1 and A2 are returned to normal
and a modest mind is changed to an arrogant mind. In thisway, AGV avoids the path interference by the change ofmind.

Fig.3 Model of MUMN

3.3. MUMD
We explain how MUMD affects the production in thefactory by increasing the number of intermediate layersof MUMN explained in 3.2 to four layers. Fig.4 shows
the MUMD that adjusts the amount of stimulus sent to
A1 and A2 by a deep learning system after receiving an
information such as how close or how far they are fromthe destinations and how many AGVs are in thesurrounding areas. The model is shown in the diagram
below.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

228

Autonomous decentralized FMS’s AGVs

x_4: The number of other AGVs which moves from thedestinations. The AGVs take the bold line of Fig. 6 andwe call the AGVs as not neighbor AGVs.

Fig.4 MUMD model

4. Inside information of MUMN
Fig.6 Not neighbor AGV

4.1. Input information of deep learning

4.3. Output information of deep learning

The input information of the deep learning is thefollowing five.
x_0: bias (always 1)
x_1: Destination
x_2: Distance from the destination
x_3: Number of other AGVs which are moving aroundthe destination
x_4: Number of other AGVs numbers which go awayfrom the destination

The output information of the deep learning has thefollowing three patterns. The input and output
relationships are shown in Fig. 7.
z_0: Pattern to move to the destination with an arrogant
mind ①
z_1: Pattern of giving way with a modest mind ②
z_2: neutral pattern ③

4.2. How to create teacher signals
x_1: The destination is the sum of the X and Y
coordinates of the AGV destination.
x_2: The distance to the destination is the absolute valueof the difference between the coordinates of thedestination (X, Y) of the AGV and into current position(X ', Y'), as shown in equation (1).
Distance to destination = | X - X'| + | Y - Y' | (1)Fig.7 Internal information of MUMD

x_3: The number of other AGVs within the range of boldline as shown in Fig. 5. We call the AGVs as neighbor
AGVs.

4.4. Effect of MUMD
The MUMD receives the output patterns and sends thestimulations, as shown in Table 1, to the units A1 andA2.
Table.1 Stimulation of each patternFig.5 Other neighbor AGVs

A1

A2

arrogant pattern【1】1

4

modest pattern【2】41

neutral patter【3】22© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

229

Ryunosuke Yamane, Hidehiko Yamamoto

First, we explain the actions of each pattern in Table.1.
When the pattern 【1】 that moves with an arrogant
mind is output, the stimulus to A1 gets reduced and thestimulus to A2 gets increased. Because of this, MUMD
can keep an arrogant mind long.
Second, when the pattern 【 2 】 that moves with amodest mind is output, the stimulus to A1 gets increased
and the stimulus to A2 gets reduced. Because of this,
MUMD can keep a modest mind long.
Third, when pattern 【 3 】 is output, the amount ofstimulus sent to A1 and A2 keeps equal. Because of this,
MUMD can keep a neutral pattern between pattern 【1】and pattern 【2】.

Fig.8 AD-FMS

5. Application Simulations and their results

6. Conclusions

By using the AD-FMS layout as shown in Fig. 8, thevirtual production simulations with the three minds types,
the conventional MUM, MUMN and MUMD, werecarried out. Every simulation includes ten AGVs. The
simulations for 8 hours AD-FMS operations were carriedout 10 times each. As a result, Table 2 was acquired. Itshows the comparisons of production outputs, theaverage operation rates of AGVs, the time of routeinterference for each AGV and the MC average operating
rates.
From the table, the followings are found, MUMD
improved the production outputs by 2.1%, AGV
operation rate by 1.3% and MC operation rate by 1.9%,
compared with MUM. MUMD decreased the production
outputs by 0.9%, the AGV operation rate by 0.5% and theMC operation rate by 0.9%, compared with MUMN.
MUMD reduced route interference time by 5.6%,
compared with MUM. MUMD reduced routeinterference time by 3.0%, compared with MUMN. The
reason MUMD is excellent is that MUMD can send
appropriate stimulations to the AGVs minds and can givemind changes that strongly adapt the production
situations. The production outputs and MC operating rate
were not high. This is why the modest mind AGVs thatneed to avoid interferences increased and they had tochoose detours for destinations.

From the simulation results, it was found that the routeinterference times of our proposed MUMD was shorter
than those of MUMN. The study of our deep learning is
to classify mind patterns by using 3 outputs and 4 inputs
for each mind model. As our future study, we willdevelop AGVs that change their minds more flexibly bychanging hyper parameters such as the numbers of input
information, output information and the units of the
middle layer.
References
1.

Hidehiko YAMAMOTO, Takayoshi YAMADA and
Shinsuke KATO, AGV Mind Model and its Usage for
Decentralized Autonomous FMS by Change of Mind,
Proceeding of Third KES International Symposium,
KES-AMSTA 2009, Agent and Multi- Agent Systems :Technologies and Applications, Lecture Notes in
Artificial Intelligence, Edited by Anne Hakansson, Ngoc
Thanh Nguyen etc. , ISSN 0302-9743, Uppsala, Sweden,
pp.744-753, June(2009), Springer.

Table.2 Simulation results© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

230

Secondary School Robotics Education in Camarin High School:
Developments and Challenges for Improvement
Jeffrey Rivera Galino

Camarin High School, Address Cadena de Amor Street, Area B, Camarin Caloocan City, Philippines
Graduate School of Humanity and Social Science, Hiroshima University, Address
1-1-1 Kagamiyama, Higashi-Hiroshima city, Hiroshima, 739-8524 Japan
Hideyuki Tanaka

Graduate School of Humanity and Social Science, Hiroshima University, Address
1-1-1 Kagamiyama, Higashi-Hiroshima city, Hiroshima, 739-8524 Japan
E-mail: tanakalpha@hiroshima-u.ac.jp

Abstract
Camarin High School is one of the schools in the Philippines that is making an initiative in improving
STEM education through robotics education. In this paper, the authors report the development of a robotics
program for students in junior high school. Through the Teacher Training Program of the Japanese
Government for international teachers, localized challenges were identified and advances in technology
education were examined for consideration. This report is hoped to provide useful insights
to both practitioners of robotics education and initiators in a similar situation.
Keywords: Robotics Education, STEM Education, Robotics Club,

Various robotics competitions reported positive impacts
on students. Participants of FIRST LEGO League,
FIRST Tech Challenge, and FIRST Robo-tics
Competition show significantly greater gains in STEM
related measures such as interest, careers, identity,
involvement and knowledge.6 RoboCup Junior haspresented success on enhancing learning of STEM
contents and skills for innovation and creativity among
students
partakingthecompetition.7 Using Test of Science Related Attitudes(TOSRA), it has been shown that students who
participated in a robotics competition had positive
attitude toward science and areas such as social
implication of science, normality of scientists, attitude
toward scientific inquiry and adoption of scientificattitudes.8

1. Introduction
Various research has substantiated positive effects ofrobotics in Science, Technology Engineering andMathematics (STEM) education. According to Mosley,
et al.1 robotic cooperative learning metho-dologies
promote STEM interest and enhance critical thinking ofstudents. Kanlhofer and Steinbauer revealed thesignificant impact of educational robotics on pupil’s
mathematics and scientific investigation, teamwork andsocial skills. 2 A systematic review of Bareto and Benitti
showed that the most common result on the topic is thatthe use of robotics helps the understanding of concepts
related to the STEM areas.3
Robotics activities can be beneficial to students invarious ways. Participation of the youth in robotics
summer camps, academic year clubs and compe-titions
increase their STEM content knowledge, their perceived
problem solving skills, and their interest in engineering
careers.4 Robotics clubs positively change students
perception of robots, humans and society while they
increase skills in scientific creativity and science processskills.5

Robotics programs are seen to yield positive outcomes.
First LEGO league robotics program conveyed prospects
for learning 21st century skills such as systems thinking,decision making, problem solving, teamwork, conflict
resolution, flexibility, perseverance and selfmanagement.9 Students acknowledged that learning
skills in cooperation and communication as well as© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

231

Jeffrey Rivera Galino, Hideyuki Tanaka

collaboration as one of the best results of an educational
robotics course.10

In 2018, a robotics team was formed to satisfy the needof producing entries for science and technology faircompetitions. Robotics team members were selectedfrom grade 10 students based on their Science andMathematics performance as well as their dedication to
give time to the endeavor. Members received basic
training on micro-controllers and proceeded to develop
their own projects. Figure 1 shows the first robotics team.
Fig. 1. Training of robotics team members

Robotics programs can be done with various approaches.
Instructional strategies such as modeling, coaching,scaffolding, examples and case studies provide fruitfulexperience to children.9 A project-based robotics
program can be beneficial to students in implementing
informal instruction in science, technology and problem
solving.11 Challenge-based learning approaches inrobotics are able to enhance student’s course
achievement and motivation.12
This paper focuses on the initiative and development ofa school robotics program for a public general highschool primarily enthused by the participation in scienceand technology fairs. Localized challenges and advancesin technology education are explored for consideration.

Robotics club was also offered as an after-school activity
to students interested in robotics. Since training was only
given to members of the robotics team, a 3-day workshop
was conducted to teach the students the fundamentals ofrobotics and micro-controllers.

2. Robotics in Camarin High School
Camarin High School is the largest public junior highschool in Caloocan City and one of the largest schools in
the Philippines in terms of population size. The school
caters four grade levels – grade seven, grade eight, grade
nine and grade ten with ages 13, 14, 15, and 16
respectively. As of 2019, it has 347 staff members and9923 students divided into 224 sections.

In 2019, robotics class was offered as a non-credit
elective subject for grade 7 and grade 8 students ofSpecial class and an option for research for grade 9
special class. Figure 2 shows the robotics class.

Camarin High School has Special class sections
comprised of students selected based on their elementary
school grade and screening examination scores. The
program started in 2017 with the first section in grade 7
rolling out as the years pass. As of 2019, there are 3
sections of special class, one for grade 7, grade 8 andgrade 9. Aside from the regular core-subjects, additional
Mathematics, English and Science classes are taken by
the students of Special class. Students of the program arealso required to do research projects during their grades
9 and 10.

Fig. 2. Robotics Class

National Science and Technology Fair (NSTF) is anannual science fair organized by the Department of
Education of the Philippines that aims to promote
Science and Technology and a culture of innovation
among the youth. It also aims to identify the mostcreative and innovative student researchers from juniorand senior high school who shall represent the country in
the international science fairs. There are four categories,
one of which is Robotics and Intelligent Machines
category.13

The teacher-in-charge of the robotics team, robotics classand robotics club was selected among the existing faculty
members based on exposure to related seminars andtraining. The teacher-in-charge (the first author) also
attended available training in robotics prior to formationof the robotics team and start of the robotics class.
3. Developments in Robotics Education in
Camarin High School
This section explains initial developments in robo-tics
education at Camarin High school as well as the
problems encountered during the development.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

232

Secondary School Robotics Education

4. Challenges for Improvement

3.1 Use of Arduino.
Arduino was used to introduce the students to microcontrollers. Arduino is an open-source electronics
platform based on easy-to-use hardware and software.14
It is a low-cost alternative to popular robotics kits. An
Arduino set was purchased for the robotics team. Sensors
and other materials were bought according to need.

With the participation of the teacher-in-charge on
Teacher Training Program of the Japanese Govern-ment
for foreign teachers, several activities have been
executed and expected to bring improvement in the
robotics program of Camarin High school.
4.1 Seminars
Seminars on microcontrollers and mechanisms for theteacher-in-charge provided technical training andpedagogical examples in teaching robotics as well as theoutline of topics for a classroom-based robotics program.
Since the teacher-in-charge had no formal training ontechnology education, the second author provided
several seminars in Hiroshima University. Topics arelisted as follows:(i)
Software and Programming:(a) C coding(b) state transition diagrams.
(ii) Electronics:(a) pull-up and pull-down registers(b) transistors(c) oscilloscope(d) relay logic circuit(iii) Hardware and microcomputer:(a) digital input and output(b) AD converter(c) PWM(d) real time interrupt(e) DA converter(f) motor driver(g) bypass capacitor(h) sensors(iv) Mechanism(a) linkages(b) pulleys, gears and cams(v) Teaching:(a) connecting the electrical devices on thebreadboard(b) wiring layout diagram
The topics cover programming, microcomputers,
electronics, mechanics and robotics and are selectedbased on practicality.

3.2 Use of Simulations
To cater more students in the workshop, Tinkercad wasused. It is a free and easy to use application for 3D design,
electronics and coding.15 Students who can borrow andbring laptops to school were strategically grouped toform teams. Simulation is a way to reduce the need formore Arduino kits and to prevent unnecessary breakage
of devices.
3.3 Flipped Learning
With a huge amount of resources available in the internet
and video streaming platforms such as YouTube,robotics team was advised to utilize differentiatedresources. Flipped learning is an approach in which
students watch recorded videos for homework and tasks
are completed when the teacher is present.16 The use offlipped learning with the available online resources is away for the teacher-in-charge to save time on instruction
and focus more on solving problems, monitoring
progress and giving feedback.
3.4 Project-Based Learning
Since the initial purpose of the robotics initiative is toprovide entries to science and technology fairs, students
studied robotics based on the needs in the project thegroups are working on. Project-based learning is ateaching method in which students are engaged incomplex real-world tasks that result in a product or apresentation (e.g. 17). The use of project-based learning
enabled the students to learn tools and techniques
required for developing their projects in a short amountof time.
3.5 Problems EncounteredWith the establishment of a local robotics program,several problems were encountered such as (1) topics andprojects to be included in robotics program (class, teamand club) and effective practices in teaching to students(2) further training of teacher-in-charge project
development and supervision and (3) low-cost devices
and instructional materials.

4.2 Robotics Projects
The first author has built various low-cost robotics
projects during the teacher training to be used as common
projects for project-based-learning in Camarin HighSchool. The building process was also a coachingexample in guiding students in creating their projects.
The following projects were created:(i) Tank type robot(ii) Remote controlled robot(iii) Edge avoidance and obstacle avoidance robot© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

233

Jeffrey Rivera Galino, Hideyuki Tanaka

Creating robotics projects integrate connected robotics
topics. The tank-type robot project involves the basics ofmaking robot bodies and mechanics. The remotecontrolled robot project deals robotic control practices
without using microcontrollers. The edge avoidance andobstacle avoidance robot project includes topics on
sensors, actuators and micro-controllers.

Human and Society, Proceedings of 3rd International
Workshop Teaching Robotics, Teaching with Robotics
Integrating Robotics in School Curriculum, 2012
6. A. Melchior, C. Burack, M. Hoover and Z. Haque,
FIRST® Longitudinal Study: Findings at 72 Month
Follow-Up., Heller School for Social Policy andManagement Brandeis University, 2020
7. A. Eguchi, RoboCupJunior for promoting STEM
education, 21st century skills, and technologicaladvancement through robotics competition, Robo-tics
and Autonomous Systems, 75 (Part B), 2016 pp. 692-699
8. A. G. Welch, Using the TOSRA to Assess High SchoolStudents’ Attitudes toward Science after Competing In
the FIRST Robotics Competition: An Exploratory Study,
Eurasia Journal of Mathematics, Science & Technology
Education, 6(3), 2010, pp. 187-197
9. Y. Ma and D. C. Williams, The Potential of a First LEGO
League Robotics Program in Teaching 21st Century
Skills: An Exploratory Study, Journal of Educational
Technology Development and Ex-change, 6(2), 2013, pp
13-28
10. A. Eguchi, Educational Robotics for Promoting 21stCentury Skills, Journal of Automation, Mobile Robotics& Intelligent Systems, 8(1), 2014, pp. 5-11
11. M. Barak and Y. Zadok, Robotics projects and learning
concepts in science, technology and problem solving,
International Journal of Techno-logy and Design
Education 19, 2009, pp. 289-307.
12. M. Jou, C.-K. Hung, & S.-H. Lai, Application of
Challenge Based Learning Approaches in Robotics
Education, International Journal of Technology andEngineering Education, 7(2), 2010, pp.17-20
13. DepEd Memorandum No. 113, s. 2019, retrieved fromhttps://www.deped.gov.ph/wp-content/
uploads/2019/09/DM_s2019_113.pdf
14. Arduino, https://www.arduino.cc/
15. Autodesk, https://www.tinkercad.com/
16. J. Bergmann and S. Sams, Flip Your Classroom: Reach
Every Student in Every Class Every Day, ISTE:
Washington DC, USA, 2012
17. C.-H. Chena, Y.-C. Yangb, Revisiting the effects ofacademic
project-based learning on students’achievement: A meta-analysis investigating mode-rators,
Educational Research Review, 26, 2019, pp. 71-81.

4.3 Instruction Materials
Guidance on the use of Raspberry Pi is valuable to theteacher-in-charge in utilizing it as a low-cost computerfor programming and project development. The authors
explored the use of Raspberry Pi as an alternative tolaptop computers in teaching Arduino. The teacher-incharge is going to prepare teaching materials for robotics
class.
5. Conclusions
This paper reported that the robotics program of CamarinHigh School consists of the robotics team, robotics classand robotics club, primarily enthused by the participationin science and technology fairs. In its initial phase,flipped learning and project-based learning was utilized
while using an Arduino kit. Simulation was used to cater
many students. It also reported that the teacher-in-charge(the first author) is in the teacher training program of theJapanese Government and that he has received seminars
in Hiroshima University and created robotics projects forenhancing teaching skills in robot education. The
teacher-in-charge is going to prepare instruction
materials for robotics class.
Acknowledgements
The teacher training program of the first author wassponsored by the Japanese Government through the
Ministry of Education, Culture, Sports, Science, andTechnology.
References

1. P. Mosley, G. Ardito, and L. Scollins, Robotic
Cooperative Learning Promotes Student STEM Interest.
American Journal of Engineering Edu-cation, 7(2), 2016,
pp. 117-128.
2. M. Kandlhofer, and G. Steinbauer, Evaluating the impactof educational robotics on pupils’ technical and socialskills and science related attitudes, Robo-tics andAutonomous Systems, 75, 2016, pp. 679-685.
3. F. Bareto and V. Benitti, Exploring the educational
potential of robotics in schools: A systematic review,
Computers and Education, 58(3), 2012, pp. 978-988
4. G. Nugent, B. Barker, N. Grandgennett and G. Welch,
Robotics camps, clubs and competitions: Results from a
US robotics project. Robotics and Autonomous
Systems,75 (Part B), 2016, pp. 686-691
5. B. Cavas, T. Kesercioglu, J. Holbrook, M. Rannikmae, E.
Ozdogru1 and F. Gokler, The Effects of Robotics Clubon the Students’ Per-formance on Science Process &Scientific Creativity Skills and Perceptions on Robots,© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

234

Deep-Learning Based Segmentation Algorithm for Defect Detectionin Magnetic Particle Testing Images *Akira Ueda, Huimin Lu, Tohru Kamiya †Department of Mechanical and Control Engineering, Kyushu Institute of Technology, 1-1 Sensui-Cho, Tobata-ku,
Kitakyushu-shi, Fukuoka, 804-8550, Japan ‡Abstract
Magnetic Particle Testing (MPT), also referred to as magnetic particle inspection, is a nondestructive examination(NDE) technique used to detect surface and slightly subsurface flaws in most ferromagnetic materials such as iron,nickel, and cobalt, and some of their alloys. In a bad environment, the procedure is complicated, and automation of
MPT is strongly desired. To find defects in the formed magnetic powder pattern, it is required to be highly skilled
and automation has been considered difficult. In recent years, many defect detection methods based on deep learning
have been proposed, and the effectiveness of deep learning has been shown in the task of automatically detecting
various types of defects having different shapes and sizes. In this paper, we describe the development of deep learning
based segmentation algorithm for defect detection in MPT images. We have achieved a F2 score of 84.04% by using
U-Net as the segmentation model and by utilizing a strong backbone network and an optimal loss function.
Keywords: Magnetic Particle Testing, Nondestructive Examination, Defect Detection, Segmentation.

Regarding the identification of fatigue cracks of thetarget item, it is easy to check whether it is normal or
abnormal, and it is possible to inspect it on the line.
Therefore, automation has been realized by highresolution cameras and image processing technologies
such as binarization and thinning. However, in the fieldof welding processing of magnetic materials such asbuildings and facilities, the surface condition is poor, andtherefore many pseudo patterns occur. Moreover, since
the shooting environment cannot be completely fixed, thedifference in images is large. For the above reasons,image analysis has been difficult and automation has notbeen realized.
In recent years, research on anomaly detection areas
by deep learning has made remarkable progress. There
are many approaches such as binary classification ofabnormal and normal, classification of abnormal data,

1. Introduction
Magnetic Particle Testing (MPT) is used the leakage
flux generated in cracks, holes, and internal defects whenmagnetizing a ferromagnetic material. By scattering thefluorescent magnetic powder while the test target ismagnetized, the magnetic powder is attracted to theleakage magnetic flux generated in the defect. In that way,
defects are highlighted to help visual confirmation.
Defects in MPT have various shapes such as circular,
linear, etc. In addition, magnetic particles are also
adsorbed on the irregularities other than defects, theresidual magnetism, and the boundaries of the metal
structure, and they have the property of forming "pseudo
patterns". MPT provides very good defect resolution andis used extensively on: welded fabrications in magnetic
material, castings, locating fatigue cracks in items subject
tocyclicalstress.
However,cleaning
anddemagnetization are required when performing
inspection, and development of automation technology isrequired to reduce work load and improve efficiency.
.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

235

Akira Ueda, Huimin Lu, Tohru Kamiya

GAN[1] and VAE[2] that learns only normal data anddetects abnormal data, detection of abnormal areas byobject detection method and segmentation. They arehighly accurate and have already been put to practical use
in many fields. As described above, anomaly detection
by deep learning is highly reliable. We believe that theeffective use of deep learning can realize the automationof MPT, which has been considered difficult to detect
defects.
In this paper, we discuss a deep learning based
segmentation algorithm for the MPT automation. Basedon U-Net, we apply an effective backbone network andloss function, and describe its effectiveness.

(a)

(b)

Fig. 1. Examples of annotation of defect image. (a) An image
including defects and (b) a label image are shown.

(a)

2. Datasets

(b)

(c)

Fig. 2. Examples of defect types. (a) Circular and near circular
defects, (b) linear defects, and (c) large linear defects.

2.1. Dataset
In this article, we use images taken in the field
inspection of the welding fabrications of magnetic
materials. The image size is 4608 x 3456 pixels (138
images) and 4000 x 2256 pixels (35 images). However,the defect area is very small for the image, which is toolarge to apply the deep learning model. Therefore, thearea including the defect is cropped, and the training andtesting are executed. The procedure for creating a data set
is as follows.

3. Methods
We adopted U-Net[3] as the segmentation model. The
detection score is improved by changing the encoder andapplying the loss function considering the balance
between classes. This section gives an overview of ourapproach.
3.1. Models

1. The original image is cropped at the size of 512
x 512 pixels including defects (total 173 images). An
example of the generated images is shown in Fig.1 (a).
When cropping, we consider not to have all the defects inthe same position.
2. In the cropped image, the label images were created byannotating the defect area in detail using labelingtechnique. An example of annotation of the defect images
is shown in Fig.1.

U-Net is a state-of-the-art semantic segmentation
method with an encoder/decoder architecture. These
encoder/decoder architectures use skip connections topreserve the location information lost due to
convolutions, allowing for more precise output. Also,
instead of using the original U-Net encoder, we use
EfficientNet[4], which claims to be balanced between
network depth, width, and resolution.
EfficientNet is available with different versions, starts
from B0 at 5.3 million parameters to B7 at 66 million.
Subsequent results show that a deeper encoder is notneeded, so the subsequent experiments will be performed
with a smaller encoder (EfficientNet B1).
We speed up the training process with pre-trained
weights. Although it was trained on ImageNet, which isa database of natural images, the pre-trained weights do
improve the training and local validation scores.

2.2. Type of defects
The dataset includes defects of various sizes and shapes,
from 5 mm to 150 mm with circular, linear, and so on.
Circular and near-circular defects with the size of 5 mm
to 10 mm are the most common, followed by linear
defects with the size of 20 mm to 50 mm. It also contains
a small number of linear defects of about 150 mm anddefects that do not fit any of them. Fig.2 shows anexample of defects.

3.2. The loss function
Since the defects in MPT images are very small, training
is greatly affected by the balance between the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

236

Deep-Learning Based Segmentation Algorithm

Table. 1. Quantitative result of using different backbone network for U-Net encoder. In addition to the original encoder and EfficientNet
B1 to B4, these show the results of experiments with ResNeXt50[5].
Models
U-Net

Precision
0.7147

Recall0.7390F1
0.6980F2
0.7156

U-Net + EfficientNetB1

0.8166

0.79290.79330.7904

U-Net + EfficientNetB2

0.80850.77180.77770.7716

U-Net + EfficientNetB3

0.7886

0.78980.77120.7784

U-Net + EfficientNetB4

0.81430.7817

0.78650.7811

U-Net + ResNeXt50

0.74900.75000.72270.7362

background and defect classes. Therefore, it is desirable
to use a loss function suitable for the target.
In the task of segmentation, Dice score coefficient(DSC) is commonly used. The 2-class DSC variant forclass c is expressed in Equation 1, where g_ic∈{0,1} and
p_ic ∈ {0,1} represent the ground truth label and thepredicted label, respectively. The total number of pixels
in an image is denoted by N. The ϵ provides numericalstability to prevent division by zero.
∑𝑁𝑁
𝑖𝑖=1 𝑝𝑝𝑖𝑖𝑖𝑖 𝑔𝑔𝑖𝑖𝑖𝑖 + 𝜖𝜖
𝐷𝐷𝐷𝐷𝐷𝐷𝑐𝑐 = 𝑁𝑁∑𝑖𝑖=1 𝑝𝑝𝑖𝑖𝑖𝑖 + 𝑔𝑔𝑖𝑖𝑖𝑖 + 𝜖𝜖

(1)

𝐷𝐷𝐷𝐷𝑐𝑐 = � 1 − 𝐷𝐷𝐷𝐷𝐷𝐷𝑐𝑐(2)

where, 𝑝𝑝𝑖𝑖𝑖𝑖 is the probability that pixel 𝑖𝑖 is of the lesion
class 𝑐𝑐 and 𝑝𝑝𝑖𝑖𝑐𝑐̅ is the probability pixel 𝑖𝑖 is of the nonlesion class, c¯. The same is true for gic and gic¯,
respectively. Hyperparameters 𝛼𝛼 and 𝛽𝛽 can be tuned toshift the emphasis to improve recall in the case of large
class imbalance. The Tversky index is adapted to a loss
function (TL) in [6] by minimizing ∑𝑐𝑐 1 − 𝑇𝑇𝐼𝐼𝑐𝑐 .
4.

To detect a defect region on a sequential image, wemainly conducted two experiments.
1. Experiment to verify the effectiveness when using
EfficientNet as U-Net encoder and to determine theoptimal EfficientNet version.
2. Experiment to verify the effectiveness of Tversky loss
compared to Dice loss. For Tversky loss, we search for
optimal 𝛼𝛼 and 𝛽𝛽 values.

The linear Dice loss (DL) is therefore defined as aminimization of the overlap between the prediction andground truth:

𝑐𝑐

We used 80% of the dataset for training and 20% for
testing. When partitioning the dataset, we took care toensure that the defect types (circular, linear, large defects,etc.) are even. We trained for 500 epochs with batch sizeof 2 and evaluated using the weight that gave the highest
score. Also, we used Adam as an optimization algorithm
and trained with an initial learning rate of 0.001. We usedprecision and recall, F1 score, which is the harmonicmean of them, and F2 score, which emphasizes recall, as
evaluation functions.
The results of each experiment are shown in Table.1 andTable.2. From the Table.1, by using EfficientNet as theencoder of U-Net, both the precision and recall improved.
In addition, among the models used, EfficientNet B1
achieved the highest score in both precision and recall,
and we did not see any improvement by increasing the

One of the limitations of the Dice loss function is that it
equally weighs false positive (FP) and false negative(FN) detections. In practice, this results in segmentation
maps with high precision but low recall. With highly
imbalanced data and small ROIs such as small defects,
FN detections need to be weighted higher than FPs to
improve recall rate. The Tversky similarity index is a
generalization of the Dice score which allows forflexibility in balancing FP and FNs:
𝑇𝑇𝑇𝑇𝑐𝑐 =Experiments and results∑𝑁𝑁
𝑖𝑖=1 𝑝𝑝𝑖𝑖𝑖𝑖 𝑔𝑔𝑖𝑖𝑖𝑖 + 𝜖𝜖𝑁𝑁𝑁𝑁
∑𝑖𝑖=1 𝑝𝑝𝑖𝑖𝑖𝑖 𝑔𝑔𝑖𝑖𝑖𝑖 + 𝛼𝛼 ∑𝑖𝑖=1 𝑝𝑝𝑖𝑖𝑐𝑐̅𝑔𝑔𝑖𝑖𝑖𝑖 + 𝛽𝛽 ∑𝑁𝑁
𝑖𝑖=1 𝑝𝑝𝑖𝑖𝑖𝑖 𝑔𝑔𝑖𝑖𝑐𝑐̅ + 𝜖𝜖(3)

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

237

Akira Ueda, Huimin Lu, Tohru Kamiya

Table. 2. Quantitative results of using different loss functions (Dice loss, Tversky loss). We use EfficientNet B1 for Unet encoder(Efficient U-Net)
Models
Efficient U-Net + DL
Efficient U-Net + TL
Efficient U-Net + TL
Efficient U-Net + TL
Efficient U-Net + TLParameters

Precision

RecallF1F2𝛼𝛼 = 0.5, 𝛽𝛽 = 0.5

0.8166

0.79290.79330.7904𝛼𝛼 = 0.6, 𝛽𝛽 = 0.4

0.70990.87600.77250.8281

0.71120.89220.78120.8408𝛼𝛼 = 0.8, 𝛽𝛽 = 0.2

0.70860.87650.77200.8276𝛼𝛼 = 0.9, 𝛽𝛽 = 0.1

0.61120.92580.72750.8305𝛼𝛼 = 0.7, 𝛽𝛽 = 0.3

accuracy. Therefore, we are mainly considering thefollowing approaches.
i) Include normal images with remarkable pseudo
patterns in the training dataset.
ii) Build a model to classify normal and abnormal andensemble with our segmentation model. The approach to
the normal area is important, and we would like to workon it in the future.
We developed a deep learning based segmentation
algorithm for MPT defects. We achieved an F2 score of
84.08% by utilizing the backbone network and the loss
function that facilitates optimization. In the future, wewould like to consider the approach to the normal area.

version. It shows that deeper encoders are not needed in
the segmentation of MPT defects. We also found thatusing EfficientNet as the backbone is more effective than
ResNeXt50.
From Table.2, it was found that using Tversky loss
instead of Dice loss improves Recall and F2 score. It isconsidered that the weighting of FN detection higher thanthat of FP led to improvement in recall. The highest F2score was obtained when 𝛼𝛼 = 0.7 and 𝛽𝛽 = 0.3 . Fig.2
shows a comparison of output images when using Diceloss and Tversky loss. We obtained better output results
when using Tversky loss.
5. Discussion and conclusions

References

We examined the defect segmentation method using the
MPT defect dataset. It was verified that the EfficienNet
used as the backbone of U-Net is effective and does not
require a large model holding a large number ofparameters. It was also shown that Tversky loss iseffective in advancing the training of MPT defect dataset.
In this paper, we focused only on the areas containing
defects, but we need to consider an approach to normal
areas. Therefore, we created normal area images dataset
from the original images described in Section 2. We
prepared about 3000 normal images of 512 x 512 pixels.
For those normal images, we tested using thesegmentation model built in Section 3. Images that could
be recognized as normal were not over-detected, andthere was no false detection of pseudo-patterns or othernoise that did not resemble defects. However, there wereseveral cases in which the pseudo patterns similar todefects were erroneously detected. We need to improvethem.
As the countermeasure, it is possible to include normal
images in the training dataset. However, it is consideredthat the inclusion of normal images that has not been
erroneously detected causes a decrease in detection

1. Houssam Zenati et al., “Efficient GAN-Based Anomaly
Detection”, arXiv preprint arXiv: 1802.06222, 2018.
2. Yuchen Lu et al., “Anomaly Detection for Skin Disease
Images Using Variational Autoencoder”, arXiv preprint
arXiv: 1807.01349, 2018.
3. Ronneberger et al., “U-Net: Convolutional networks forbiomedical
image
segmentation”,
International
Conference on Medical Image Computing and Computerassisted Intervention, pp.234-241, 2015.
4. Mingxing Tan et al., “EfficientNet: Rethinking Model
Scaling for Convolutional Neural Networks”, arXiv
preprint arXiv: 1905.11946, 2019.
5. Saining Xie et al., “Residual Transformations for Deep
Neural Networks”, arXiv preprint arXiv: 1611.05431,
2016.
6. Seyed Sadegh Mohseri Salehi et al., “Tversky loss
function for image segmentation using 3D fully
convolutional deep networks”, arXiv preprint arXiv:
1706.05721, 2017.
7. Nabila Abraham et al., “A Novel Focal Tversky loss
function with improved Attention U-Net for lesion
segmentation”, arXiv preprint arXiv: 1810.07842, 2018.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

238

A Method for Improving Recognition of Lying Postures Using a Measured
Signal Intensity of Respiration and Heartbeat by Flexible Tactile Sensor Sheet
Kazuya Matsuo

Department of Mechanical and Control Engineering, Kyushu Institute ofTechnology, 1-1, Sensui, Tobata, Kitakyushu, Fukuoka, 804-8550, Japan
Toshiharu Mukai

Meijo University, Japan
Shijie Guo

Hebei University of Technology, China
E-mail: matsuo@cntl.kyutech.ac.jp

Abstract
We describe a method for improving recognition of lying postures using a measured signal
ntensity of respiration and heartbeat. We can obtain respiration and heartbeat by means ofusing the body pressure measured at the suitable location determined by lying posture in theconventional method. Therefore, a recognition rate of lying postures and a measured signal
intensity of respiration and heartbeat have positive correlation. In the experiments, we showthat recognition of lying postures is improved by our method.
Keywords: Unconstrained measurement, Lying posture, Respiration, Heartbeat,ubber-based flexible tactile sensor sheet.

1. Introduction
Japan’s population of people aged 65 and above
increased by 7 million, that is, from 29 million to 36
million in the ten years between 2010 and 2020. Further,
the aging rate is expected to increase from 23.0% to 29.1%[1], [2]. Seemingly, the number of people that requirelong-term care will substantially increase. However, theburden on individual care staff is expected to increase as
there will be no corresponding increase in the number ofcare staff [3]. There is therefore an increasing need for the
introduction of care robots in order to reduce the burden
on care staff and further solve this problem.
The main areas of focus that have been chosen for theuse of robot technology in care are transfer assistance,
mobility support, toilet support, watching over dementia
patients and bathing assistance [4]. Out of these, the ability
to use sensors to watch over dementia patients would

significantly help to reduce the burden on care staff.
Moreover, there are many benefits of the ability to identify
a patient’s position in bed that are not limited to dementia
patients. The patient positions
that must be identified include the patient being in or outof bed, sitting up or lying down, any danger of the patient
falling out of bed or a patient lying on their back or on
their front. Additionally, the ability to measure the depth
or state of sleep is useful too. Increasing the quality ofsleep by observing the state of sleep is effective formaintaining the patient’s healthy sleep state.
The standard and reliable method for measuring sleep
states is polysomnography (PSG) [5]. This techniquemeasures brain waves, ECG, eye movement, chin muscle
activity and pneumogram. Further, based on the R&K
method [6], the technique classifies sleep state into six
categories namely wake, REM sleep and non-REM sleep
stages 14. The technique is beneficial as it measures sleep
states with high accuracy. However, since measuring© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

239

Kazuya Matsuo, Toshiharu Mukai, Shijie Guobiosignals requires the attachment of large numbers of
sensors to the body, the technique brings a sense of
constriction and may impede natural sleep. Moreover,apart from being difficult to automate, recognition of sleep
states from biosignals requires judgment by a laboratory
technician. Consequently, there is need to develop amethod that allows for easy estimation of sleep state.
Evidently, sleep state can be estimated from biosignals
such as respiratory rate, heart rate, number of body
movements and lying posture. These biosignals can bemeasured relatively easily. For instance, a strong
correlation has been found between heart rate and sleep
state [7]. The relationship between the heartbeat and thenumber of body movements is also strong [8], [9].
Additionally, respiratory disorders such as sleep apnea
syndrome are known to be unlikely to manifest whensleeping in a lateral position [10]. Watanabe et al. used airpressure mattress-sensors [11] to measure heart rate,
respiration, body motion, snoring and coughing andfurther developed a method for estimating sleep statebased on rhythm irregularities such as increased
respiration and heartbeat during REM sleep [12]. Other
studies use infrared motion sensors [13] or cameras [14]and focus on the extent and frequency of the measured
body motion to assess the depth of sleep.
We propose a method that uses a flexible tactile sensor
sheet to measure lying posture [15], respiration andheartbeat [16]. This involves the use of rubber, flexible
capacitive tactile sensors that we developed (Smart Rubber(SR) sensors [17]) to measure lying posture, respirationand the heartbeat of people lying on their side in bed andin a non-restrictive manner. First, we identify lying posture
based on the tactile image formed by the body pressure
distribution pattern. We then obtain the rate respirationand heartbeat by measuring the pressure of the body
locations in line with the recognition results. Further, lying
posture, respiration and heartbeat can be measured
simultaneously by using high spatial and pressure
resolution SR sensors and the measured pressure
information appropriately. The method has additional
benefits in that, at a few mm, it is a great deal thinner thanan air mattress. Further, measurements can be performed
while the subject’s privacy is taken into consideration,
which is different from when cameras are used. SR sensors
are easy to use on a daily basis as they are cheaper thanexisting measuring devices and require no effort to install.
We want to estimate sleep state based on lying posture,
respiration and heartbeat that have been measured using

SR sensors. However, the main challenge is that thesebiosignals can occasionally be inaccurately measured. For
instance, heart rate causes minimal pressure fluctuations.
Therefore, it is likely to be hidden by respiratory and body
motion fluctuations and noise, and simultaneously difficult
to measure. Moreover, errors in identifying a lying posture
may also occur following the differences in physique andways of sleeping.
Errors are highly likely in recognizing lying postures aswell as in measuring the respiration and heart rates our
method uses information on pressure in the locations that
conform to the type of lying posture. On the other hand, if
the signal intensity for the measured respiration andheartbeat is high, then it can be assumed that therecognition of the lying posture is successful. In thecurrent paper, we propose a method that can help improve
the recognition of lying posture using measured respirationand heartbeat signal intensity. First, we confirm that thesuccess or failure of the recognition of a lying posture in
the method we previously developed are correlated withthe success or failure of the measurement of a heartbeat. In
addition to the probability obtained through machinelearning from tactile images, we use signal intensity for
respiration and heartbeat to propose a method to improve
the performance of the recognition of a lying posture. We
demonstrate the usefulness of the proposed method byconducting an experiment using real data. In conclusion,
we summarize our findings.
2. Lying Posture Recognition Method Using Tactile
Images and Respiration and Heartbeat Signal
Intensity
2.1. Respiration and heartbeat measuring method
First, we describe the method to measure a person’s
respiration and heartbeat based on the pressure data thathave been measured using a flexible tactile sensor sheet(Fig. 1) [16]. We first measure the body pressure of aperson lying on their side using tactile sensors laid out on
the bed. We then conduct a frequency analysis for the timeseries data for the body pressure measured by the sensor
cells in appropriate locations such as the chest etc. We
used a fast Fourier transform (FFT) for the frequency
analysis [18]. We obtained the person’s respiration andheartbeat by extracting the appropriate frequency band.
The person whose measurements were taken was resting
on a bed. While resting, respiration and heartbeat are kept
virtually constant. The frequency bands for respiration andheartbeat can be separated using a band pass filter as they
do not overlap in rest. In a preliminary experiment, theextracted respiration and heartbeat frequency band are© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

240

A Method for Improving

determined as follows. (1) The respiration frequency band
was set at 0.1[Hz] - 0.4[Hz]. (2) The heart rate frequency
band was set at 0.8[Hz] 1.6[Hz]. Respiration and heartbeat
wave forms can be obtained by returning the frequency
band to the signal of the time band. Moreover, thedominant frequency in the corresponding band after FFT
should be obtained in order to obtain the respiration andheart rates alone.

Pressure fluctuations are higher for respiration than forheartbeat. Therefore, respiration can often be measured
anywhere on the body and without any challenges.
However, if lying posture recognition fails substantiallyand the pressure measuring location is placed outside thebody, then it is patent that respiration cannot be measured.
Fig. 3 illustrates such a case. The actual left side lying
posture was erroneously detected as a supine posture.
Therefore, the pressure measuring location is placedoutside the body and the respiration measurement failed.
The time series data shown in the top right corner of Fig. 3
reveal that the fluctuation range is extremely small andrepresents noise. Therefore, no peak respiration can beobtained by applying frequency analysis.

Fig. 1. Measurement method of respiration and heartbeat
using a flexible tactile sensor sheet.
Furthermore, body pressure at the appropriate locations
such as the chest must be measured in order to accurately
measure respiration and heartbeat. A lying posture is
detected using machine learning AdaBoost [19] while
body pressure is measured in locations in accordance tothe results (Fig. 2). This achieved robust measurements of
respiration and heart rate in relation to changes in lying
posture [15]. Fig. 2 shows the appropriate measuringlocations where heartbeat peaks were seen in all of the 11
subjects that we took actual measurements for. For thecurrent study, we used body pressure data of the locations
where pressure was the highest, in the range shown in theellipse in Fig. 2.

Fig. 2. Suitable locations for measurement of heartbeat.
The circles show the suitable locations for different body
postures. The suitable locations depend on not only thelocation but also the lying posture of the person [17].

Fig. 3. Failure example of respiration measurement. We cannot obtain
respiration by means of using the time series data measured at cells without
body pressure.

2.2. Improving Lying Posture Recognition Using
Respiration and Heartbeat Signal Intensity
Measuring respiration and heartbeat is more likely tofail if lying posture recognition fails. On the other hand,lying posture recognition is highly likely to be successful
if signal intensity for the measured respiration andheartbeat is high. We therefore propose a method for the
performance of improving lying posture recognition byusing respiration and heartbeat signal intensity in addition
to probability obtained from the tactile images. In specificterms, lying posture is obtained based on probability qi(x)in addition to probability pi(x) for each lying posture thatis obtained through machinelearning from time series data x for the tactile images
qi(x) = α · ri(x) + β · hi(x) + γ · pi(x)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

241

(1)

Kazuya Matsuo, Toshiharu Mukai, Shijie Guousing respiration signal intensity ri(x) and heartbeat signal
intensity hi(x) measured for each assumed lying posture.
However, α, β, γ are weights. Eventually, they shall be α +β + γ = 1 and 0 ≤ α,β,γ ≤ 1. Moreover, i is the number oflying posture. Further, when the lying posture is categoryN, probability from q1(x) to qN(x) can be obtained. In thecurrent study, these were the maximum lying posturerecognition results. Seemingly, ri(x) and hi(x), which
represent signal intensity for respiration and heartbeat, are
the SN ratio. This study’s experiment used, SN ratios forrespiration and heartbeat, respectively.
SN ratio for respiration S Nri and SN ratio for heartbeat S
Nhi are obtained based on the frequency analysis of timeseries data for body pressure locations in line with each of
the assumed lying postures. These SN ratios were defined
as:

S Nri

=

S Nhi

=

S ri
20log10,Ni
S hi
20log10
.
Ni

(2)

(3)

S ri, S hi, and Ni represent the respiration, heartbeat, andnoise signals respectively. The maximum spectra in thefrequency bands for respiration (0.1[Hz]-0.4[Hz]) andheartbeat (0.8[Hz]-1.6[Hz]) were set to S ri and S hi,
respectively. Moreover, Ni was set to the RMS of thesignal in the part remaining after deducting the respiration,
heartbeat and body movement signal bands from theoverall frequency band. The respiration signal band is therange within 0.05[Hz] from frequency fri to S ri and thesecond harmonic 2fri and the third harmonic 3fri for
respiration. The heartbeat signal band is the range within
0.05 [Hz] from frequency fhi to S hi and the second
harmonic 2fhi for heartbeat. Further, we used a range of ±0.05[Hz] from the peak as the signal band as the peakwidth of the signals for respiration and heartbeat in the
preliminary experiment was 0.1 [Hz]. Moreover, we set
the body movement band as 0[Hz] - 0.1[Hz].
3. Flexible Tactile Sensors
Sensors that are capable of high-resolution
measurements of the body pressure of a person lying on abed, in spatial terms as well as in terms of pressure are
required in order to implement the proposed method. We
used SR sensors [19], which are tactile sheet sensors. SR

sensors, including the wiring, are entirely made of rubber.
This makes them flexible and pliable and allow formeasurements that avoid any sense of discomfort. Further,covering them with a sheet for instance does not result in
any issues as it is possible to take measurements without
the need to directly touch them.
This means that using SR sensors that are laid out on a bed
allow for measurements that are non-restrictive, noninvasive and that can be conducted in circumstances thatare similar to natural sleep.
SR sensors consist of a thin dielectric layer between two
conductive rubber sheets of electrodes (Fig. 4). The blackpart of the rubber sheet are electrodes made fromconductive rubber that was made using carbon filler, andare further laid out perpendicular to each other and on twolayers. The total number consists of m × n cells when thenumber of electrodes on both sides is m and n. This isbecause capacitive sensors occur where sensors on bothsides intersect. Spatially high-resolution measurements areachieved by narrowing the electrodes. Further, highresolution measurements in terms of pressure can beachieved by increasing the cell surface area.
When pressure is applied, the dielectric layer deforms
hence reducing the distance between the electrodes. This
increases the capacitance of the said area. Scanning thecapacitance of the respective cells using the sensor
controller allows for the recognition of the 2D pressure
pattern. In specific terms, a sinusoid voltage is applied toeach cell. Impedance is further obtained from the current
going through thus obtaining capacitance. The effect ofdifferent wire resistance values for each cell is eliminatedbecause for equivalent circuits at the time of scanning,
wire resistance is connected in a series of cell condensers.
This is a sensor composition that has been proposed in thepast. Further, sensors with metal electrodes arecommercially available. However, for applications in care
aids such as sleep monitoring systems, issues such as lackof pliability and high cost remained. We therefore madeuse of rubber printing technology to develop flexible andlow cost SR sensors. These sensors are sold by Sumitomo
Riko Co., Ltd under the name SR Soft Vision.
Table 1 outlines the specification of the SR sensors usedin this study. This electrode width allows for themeasurement of body pressure with sufficient spatial
resolution to enable the recognition of different lying
postures. On the other hand, reducing the cell area would
reduce the resolution in terms of body pressure. Thesesensors can measure respiration without any challenges aschanges in body pressure are substantial. However,© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

242

A Method for Improving

because these changes are small in relation to heartbeat,
respiration cannot be measured as is, due to noise.

Fig. 4. Schematic structure of SR sensor.
Noise must therefore be suppressed and resolution interms of pressure increased so as to enable themeasurement of heartbeat.
Table 1. Specifications of the SR sensor.
Number of cells
Size of sensor sheet
Size of a sensor cell
Size of a gap between cells
Thickness of sensor sheet
Sampling rate of all the 800
cells
Sampling rate of the 4
precision cells

800

Length
Width
32
25
882
686
[mm]
[mm]
14
14
[mm]
[mm]
14
14
[mm]
[mm]
3.5 [mm]
5 [Hz]
20 [Hz]obtained for a total of 40 times. For each of these 40 data
sets, probability qi(x), as defined in Section II-B, wasobtained and lying postures detected.
First, we used the method by Mukai et al. [17] to obtainprobability pi(x) for each lying posture based on the bodypressure data. The length of one body pressure data set
was set to 51.2 seconds (51.2 x 20 [Hz] = 1024) because
conducting FFT requires that the number of data sets be a
power of 2. The subject maintains the same lying posturefor the full 51.2 seconds. Therefore, the obtained
probability is virtually constant. The average for the 51.2
seconds was therefore set as pi(x). The 4 high-accuracy
cells were then placed in the respective suitable locations
for measuring the heartbeat for all lying postures, as
shown in Fig. 2. Further, the signal intensity for respirationand heartbeat (ri(x) and hi(x): i = 1,2,3,4) was obtained
from the time series data for body pressure. We used the
SN ratio as the signal intensity for this study. Finally, qi(x)is calculated from these and lying posture recognition is
performed. The present study was approved by the
Research Ethics Committee of the Kyushu Institute ofTechnology. Further, Informed Consent was obtained in a
writing from each subject taking part in the experiment.
If the signal intensity for respiration and heartbeat
during the true lying posture is stronger than others, thefinal recognition result can be accurately corrected basedon qi(x) by setting the appropriate weight. This can bedone despite an error existing in the posture recognition
using probability pi(x) from the tactile image only. We
further conducted the following experiment to evaluate theperformance of the proposed method.
Forty data sets were obtained for one subject, a 34yearold male. These data sets were labeled A1.
Recognition rates were examined for various weights(α,β,γ) for data set A1. A confusion matrix of the 4 lying
postures was also studied.
4.2. Experimental results and discussion

4. Lying Posture Recognition Experiment4.1. Experimental method
To verify the usefulness of the proposed method, we
conducted an experiment to detect the lying posture of aperson lying on their side on top of SR sensors laid out on
a bed. The subject’s positions were divided into 4 namely
facing up, facing down, lying on their right side and lying
on their left side. Each of these lying postures were taken
10 times. Further, time series data for body pressure were

Fig. 6 shows the recognition rates with weights varying
between 0 and 1 at 0.1 intervals. The horizontal andvertical axes represent weight α and β respectively. The
colors represent the recognition rates corresponding to theweight indicated by the bottom left coordinates. In
conditions where α + β + γ = 1 and 0 ≤ α, β, γ ≤ 1, values
with the respective weights are α = 0, 0.1, ··· , 1, β = 0, 0.
1,··· ,1−α, γ = 1 − α − β. Fig. 6 does not display weight γ
as γ is determined when α and β are decided upon.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

243

Kazuya Matsuo, Toshiharu Mukai, Shijie Guo

recognition can be improved by the appropriate use of
signal intensity for respiration and heartbeat.
Table 2. Parameters of dataset A1 with (α,β,γ) =(0.6,0.1,0.3): pi(x), qi(x), pT(x)/pF(x), and qT(x)/qF(x).
Recognition performance using qi(x) is improved than thatusing pi(x).
LyingpostureTrue
Fig. 6. Recognition rates of the lying postures (Dataset
A1). Recognition of lying postures is improved by meansof using a measured signal intensity of respiration andheartbeat. For example, the recognition rate of (α,β,γ) =(0.6,0.1,0.3) is higher than that of (α,β,γ) = (0,0,1).
Fig. 6 shows the recognition rate is the highest at 82.5%
when (α,β,γ) = (0.6,0.1,0.3) or close to that. In detail,weights are the highest for (α, β, γ) = (0.5,0.1,0.4),(0.5,0.2,0.3), (0.6,0,0.4), (0.6,0.1,0.3), (0.7,0,0.3).
The recognition rate is 72.5% when (α, β, γ) = (0,0,1). The
recognition rate for the proposed method with weight set
at e.g. (α,β,γ) = (0.6,0.1,0.3), is 10% higher than forconventional methods that use only tactile images.
Moreover, Table 2 shows the averages and standard
errors for the parameters obtained from the 40 data sets
rather referred to as A1. These are pi(x), ri(x), hi(x), qi(x)when weight is (α,β,γ) = (0.6,0.1,0.3), with the top andbottom rows showing the parameters when lying posture istrue and false respectively. A false lying posture is theposture with the highest value out of the 3 false postures.
All the parameters are higher for the true lying positions.
Further, it is clear that they are appropriate to use forrecognition. Furthermore, the ratios for pi(x) and qi(x) fortrue and false lying postures are shown in rows 7 and 8,
respectively. The higher the ratio, the higher the
probability for the true lying posture in relation to theprobability for other lying postures. Furthermore, stable
recognition is possible. We further performed a paired ttest with a significance level of 0.05 in order to examineand confirm whether there is a difference between these
two ratios. The null hypothesis states that the ratio
between pi(x) and qi(x) does not differ. Furthermore, thedegree of freedom is 39. This revealed that t-value t =−3.27, p = 2.3 × 10−3, and that there was no significant
difference. This confirmed that in addition to theprobability obtained from the tactile image, lying posture

False

pi(x)

qi(x)

pT(x)/pF(x) qT(x)/qF(x)50.1 ± 40.2 ±2.09 ±
15.7
5.8
1.30
29.7 19.019.8 11.6

2.42 ±
1.04

5. Conclusion
We proposed a method to improve the performance oflying posture recognition by using measured respirationand heartbeat signal intensity. We conducted experiments
using actual data and demonstrated the fact thatrecognition performance can be improved by using
respiration and heartbeat signal intensity along with theprobability obtained from the tactile images. Moreover,
we used data from different trials with the same subject
and data for different subjects to confirm the versatility oftheproposedmethod.
6. References
Statistics Bureau, Ministry of Internal Affairs andCommunications, Japan. (2011, Oct.). The 2010 Population
Census of Japan. Public announcement on 26 Oct. 2011.
[Online].
Available:<http://www.
stat.go.jp/english/data/kokusei/index.htm>, (accessed on 3rd
Dec. 2020).
[2] National Institute of Population and Social Security
Research. (2012, Mar.). Population Projections for Japan(January 2012) 2011 to 2060-. Public announcement on 30
Mar. 2012. [Online]. Available: <http://www.ipss.go.jp/sitead/index english/ esuikei/gh2401e.asp>, (accessed on 3rd
Dec. 2020).
[3]
Ministry of Health, Labour and Welfare, Japan. (2011,
June). General Welfare and Labour. Public announcement
on
2
June
2011.
[Online].
Available:<http://www.mhlw.go.jp/english/policy/
other/socialsecurity/index.html>, (accessed on 3rd Dec.2020).
[4] Ministry of Economy, Trade and Industry and Ministry of
Health, Labor and Welfare, Japan. (2012, Nov.). Priority
Areas to Which Robot
Technology Is to Be Introduced in Nursing Care of theElderly. Public announcement on 22 Nov. 2012. [Online].
Available:<http://robotcare.jp/en/home/index.php?\&lang=en\ &PHPS[1]© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

244

A Method for Improving

ESSID=q34fa48pe9plr3go0oo9vndmon>, (accessed on 3rd
Dec. 2020).
[5] C. A. Kushida, M. R. Littner, T. Morgenthaler, C. A. Alessi,
D. Bailey,
J. Coleman Jr, L. Friedman, M. Hirshkowitz, S. Kapen, M.
Kramer,T. Lee-Chiong, D. L. Loube, J. Owens, J. P. Pancer, and M.
Wise, “Practice parameters for the indications forpolysomnography and related procedures: An update for
2005,” Journal of Sleep, vol. 28, no. 4, pp. 499–521, 2005.
[6] A. Rechtschaffen and A. Kales, A Manual of Standardized
Terminology, Techniques and Scoring System for Sleep
Stages of Human Subjects. Bethesda, MD. : U.S. Dept. of
Health, Education, and Welfare, Public Health ServicesNational Institutes of Health, National Institute of
Neurological Diseases and Blindness, Neurological
Information Network, 1968.
[7] T. Watanabe and K. Watanabe, “Estimation of the Sleep
Stages by the Non-restrictive Air Mattress Sensor: Relation
bitween the Change in the Heart Rate and Sleep Stages,”
Trans. of the Society of Instrument and Control Engineers,
vol. 37, no. 9, pp. 821–828, 2001.
[8] A. Deguchi and H. Hagiwara, “Evaluation of Heart Rate
Variability and Respiratory Variability during Sleep Using a
Lorenz Plot,” Journal of Sleep and Biological Rhythms, vol.
9, no. 4, p. 366, 2011.
[9] K. Otsuka, Y. Ichimaru, T. Yanaga, and Y. Sato, “Studies of
Arrhythmias by 24-hour Polygraphic Recordings:
Relationship between Atrioventricular Block and Sleep
States”, American Heart Journal, vol. 105, no. 6, pp. 934–
940, 1983.
[10] Z. Beattie, C. Hagen, and T. Hayes, “Classification of Lying
Position Using Load Cells under the Bed”, in Proc. 33rd
Annual Int’l Conf. of the IEEE Engineering in Medicine andBiology Society, pp. 474–477, 2011.
[11] Y. Kurihara and K. Watanabe, “Sleep-Stage Decision
Algorithm by Using Heartbeat and Body-Movement
Signals,” IEEE Trans. on Systems, Man, and Cybernetics,
Part A, vol. 42, no. 6, pp. 1450–1459, 2012.
[12] T. Watanabe and K. Watanabe, “Noncontact Method forSleep Stage Estimation,” IEEE Trans. on Biomedical
Engineering, vol. 51, no. 10, pp. 1735–1748, 2004.
[13] Y. Kambayashi and H. Hagiwara, “Estimating Sleep Cycle
Using Body Movement Density,” in Proc. 2012 5th Int’l
Conf. on Biomedical Engineering and Informatics, pp.
1029–1033, 2012.
[14] S. Okada, Y. Ohno, Goyahan, K. Kato-Nishimura, I. Mohri,
and M. Taniike, “Examination of Non-restrictive and Noninvasive Sleep Evaluation Technique for Children Using
Difference Images,” in Proc. 30th Annual Int’l Conf. of theIEEE Engineering in Medicine and Biology Society, pp.
2483–2487, 2008.
[15] T. Mukai, K. Matsuo, Y. Kato, A. Shimizu, and S. Guo,
“Lying Posture Detection for Unconstrained Measurementof Respiration and Heartbeat of a Person on a Bed,” in Proc.
11th IASTED Int’l Conf. on Biomedical Engineering, pp.
46–53, DOI: 10.2316/P.2014.818-014, 2014.

[16] K. Matsuo, T. Mukai, Y. Kato, A. Shimizu, and S. Guo,

“Measurement of Respiration and Heartbeat Using a
Flexible Tactile Sensor Sheet on a Bed,” in Proc. 11th
IASTED Int’l Conf. on Biomedical Engineering, pp. 59–64,
DOI: 10.2316/P.2014.818-031, 2014.
[17] S. Guo, Y. Kato, H. Ito, and T. Mukai, “Development of
Rubber-Based Flexible Sensor Sheet for Care-Related
Apparatus,” Sumitomo Electric Industries Technical Review,
no. 75, pp. 125–131, 2012.
[18] J. W. Cooley and J. W. Tukey, “An Algorithm for the
Machine Calculation of Complex Fourier Series,”
Mathematics of Computation, vol. 19, no. 90, pp. 297–301,
1965.
[19] Y. Freund and R. Schapire, “A Decision-Theoretic
Generalization of On-Line Learning and an Application to
Boosting”, Journal of Computer and System Science, vol.
55, pp. 119–139, 1997.
[20]© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

245

Basic research for the realization of online MEG using SSD
Kazuhiro Yagi

Interdisciplinary Graduate School of Agriculture and Engineering, University of Miyazaki,
1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Yuta Shibahara

Graduate School of Engineering, University of Miyazaki,
1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Lindsey Tate

Faculty of Engineering, University of Miyazaki,
1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Keiko Sakurai

Faculty of Engineering, University of Miyazaki,
1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Hiroki Tamura

Faculty of Engineering, University of Miyazaki,
1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
E-mail: kazuhiro_yagi@junwakai.com, hi16019@student.miyazaki-u.ac.jp, teitorinzeirini.c4@cc.miyazaki-u.ac.jp,
sakurai.keiko.u6@cc.miyazaki-u.ac.jp, htamura@cc.miyazaki-u.ac.jp

Abstract
Neurofeedback systems have been found to be effective in the clinical rehabilitation of paralysis. However, most
systems exist only for use with EEG, which is cumbersome to apply to patients and has lower spatial resolution than
MEG. Furthermore, the best practices for neural data feature extraction and feature selection are not well established.
The inclusion of the best performing feature extraction algorithms is critical to the development of clinicalneurofeedback systems. Using simultaneously collected MEG and accelerometer data before and during 10
spontaneous finger movements, we performed an in-depth comparison of independent components analysis (ICA)
and spatio-spectral decomposition (SSD) algorithms for their individual abilities to isolate movement-relevant
features in brain activity. Having restricted raw data to that from sensorimotor rhythm (SMR) frequencies in selectMEG sensors over sensorimotor cortex, we compared ICA and SSD components using: (1) 2D topographies, (2)activations over time, (3) and correlations with accelerometer data at both 0ms and 60ms time delays. SSD performed
more quickly and produced components that were more highly correlated with the behavioral data than ICA. We will
discuss these results and suggestions for application to neurofeedback systems. In particular, we will present detailed
visualizations of SSD results and discuss potential strategies and pitfalls for feature selection.
Keywords: Magnetoencephalography, Spatio-spectral decomposition, Morlet wavelet transform, Neurofeedback

1.

important, but post-onset rehabilitation also plays acritical role in the wellbeing of patients.
One of the techniques used for non-invasive brain
function evaluation is magnetoencephalography (MEG)1-

Introduction

For diseases that affect brain function, such as strokes,
immediate treatment with medication and surgery is© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

246

Kazuhiro Yagi, Yuta Shibahara, Lindsey Tate, Keiko Sakurai, Hiroki Tamura. MEG has high temporal resolution as well as highspatial resolution, and it is commonly used clinically for
epilepsy diagnosis and rehabilitation10. Without theability to monitor a patient’s relevant brain activity in realtime during rehabilitative exercises, efficientrehabilitation cannot occur.
The purpose of this study was to evaluate the use of
spatio-spectral decomposition (SSD) of real-time MEG
data during spontaneous movement3-6. Performance was
evaluated by comparing SSD results to the results from astandard analysis technique, independent components
analysis (ICA). SSD completed decomposition fasterthan did ICA, with the SSD analysis completed about 270
times faster in the preliminary experiment. In addition, as
shown in Table 1, the correlation with the accelerometerdata (shifted by 60[ms] to account for the time difference
between neural motor planning and actual motor
execution) was also stronger for the most highly
correlated SSD component as compared to the mosthighly correlated ICA component. Our results indicatedthat SSD outperforms ICA in the context of feature
extraction for online, real-time MEG analysis; therefore,
we present our investigation of the SSD components andconclude with suggestions for feature selection.
2

interference, measurements were taken inside a magnetic
field shield room (1 [kHz] shielding rate 55.2 [dB]). The
participant was a healthy person who attached an
acceleration sensor to the middle finger of his left handand performed ten spontaneous flexion (i.e., bending)and extension (i.e., relaxing) movements of the indicatedfinger as shown in Fig.1 (hereinafter referred to as “thetask”). The accelerometer data indicated the start andduration of flexion and extension.

Fig.1: The spontaneous flexion (bending) and extension(relaxing) movements.

Table 1. The correlation with the accelerometer data andICA or SSD.

3.

2.

Analysis Method

After collecting the raw MEG data from all 306 channels,
we sub-selected 26 gradiometers corresponding to theright sensorimotor cortex (SMC) and performed ICA and
SSD analyses7-8, which each identified components
within the largest cited frequency band for sensorimotor
rhythm (SMR)9, 8-30 [Hz]. As presented in Table 1, we
calculated Pearson correlations between the timeadjusted accelerometer data and each of the top fivecomponents from ICA and SSD. We calculated the SSD
topographies about 20 seconds and then performed a
Morlet wavelet transform to examine frequency powerover time (Fig.2). In addition, in order to increase thesignal-to-noise ratio (SNR), the wavelet analysis was
performed on task-locked averaged components toindicate average brain activity in the final 2 seconds
before flexion start. There were 10 tasks completed over
20 seconds at irregular intervals.

Experiment

The data were collected using a full-head 306-channel
magnetoencephalograph (Vectorview, Elekta-Neuromag,
Helsinki, Finland) at a sampling frequency of 1000 [Hz].
In order to reduce power supply noise and other© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

247

Basic research for the

3rd SSD
Fig.2: The flow of analysis method

4th SSD

Fig.3: The results of Topography using SSD

5th SSD
Fig.4: The results of the across-task averaged second,
third, fourth and fifth SSD components
4.

Results

From the results of topography (Fig. 3), the first eight
SSD components showed activity localized around theright SMC. As shown in Fig. 4, the across-task averaged
third and fourth SSD components had higher SMR band
power immediately preceding the task. The pattern ofactivity observed in the SMR frequency band across task
instances indicated that the first four SSD components

2nd SSD© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

248

Kazuhiro Yagi, Yuta Shibahara, Lindsey Tate, Keiko Sakurai, Hiroki Tamura

captured artifactual activity (e.g., movement activity). In
the averaged component as well as across task instances,
steady activity during the two seconds preceding flexion
was observed at 12-13 [Hz] in the fifth SSD component.
5.

3.

Conclusion

4.

In this paper, we performed offline MEG data analysis
during flexion and extension of the middle finger of theleft hand in order to investigate the use of SSD in online,
real-time MEG analysis.
There are several key points to take away regarding theautomation of feature selection in this context. First, theSSD component with the highest correlation toaccelerometer data (comp. 2, r = -0.89) was, in this case,indicative of muscular noise reaching the MEG sensors
rather than indicating neural activity as desired. Second,the component most likely to indicate relevant neuralactivity (comp. 5) was, in addition to not being the mosthighly correlated with accelerometer data (r = -0.45), also
not the first component (i.e., it was not the component
with the strongest eigenvalue or largest SNR). Finally,
when only considering a few trials (i.e., at the beginningof neurofeedback rehabilitation) or particularly noisy
data, using averaging techniques can lead to incorrectfeature selection due to the extreme amplitude of artifacts
and the components that capture them. Therefore, acomponent to use for patient feedback cannot be selectedbased solely on having the highest average SMR power,
the highest correlation with behavioral data, or thehighest eigenvalue.
Future study will include the same analysis for SSD
components 6-8, which show promising topographies.
Furthermore, we will investigate the usefulness ofbaseline correction in component selection and oflogarithmically scaled components as features in
unsupervised learning.

5.
6.
7.

8.

9.

10.

Felice, Stefano Masiero and Francesco Piccione,
Magnetoencephalography in Stroke Recovery andRehabilitation, Frontiers in Neurology March 2016
Volume7 Article 35.
Vadim V. Nikulin, Guido Nolte, Gabriel Curio, A novel
method for reliable and fast extraction of neuronal
EEG/MEG oscillations on the basis of spatio-spectral
decomposition, NeuroImage 55 (2011) 1528–1535.
Natalie Schaworonkow, Pedro Caldana Gordon, Paolo
Belardinelli, Ulf Ziemann, Til Ole Bergmann
andChristoph Zrenner, μ-Rhythm Extracted With
Personalized EEG Filters Correlates With Corticospinal
Excitability in Real-Time Phase-Triggered EEG-TMS,
Frontiers in Neuroscience December 2019 Volume12
Article954.
Stefan Haufe, Sven Dähne, Vadim V. Nikulin,
Dimensionality reduction for the analysis of brain
oscillations, NeuroImage 101 (2014) 583–597
Hanna-Leena Halme, Lauri Parkkonen, Comparing
Features for Classification of MEG Responses to Motor
Imagery, PLOS ONE December 16, 2016.
Franc¸ois Tadel,1 Sylvain Baillet,2 John C.Mosher,3
Dimitrios Pantazis,4 and Richard M. Leahy1, Brainstorm:
A User-Friendly Application for MEG/EEG Analysis,
HindawiPublishing
Corporation
Computational
Intelligence and Neuroscience Volume 2011, Article ID
879716, 13 pages.
Maren Stropahl, Anna-Katharina R. Bauer, Stefan
Debener and Martin G. Bleichner, Source-Modeling
Auditory Processes of EEG Data Using EEGLAB andBrainstorm, Frontiers in Neuroscience May 2018 Volume
12 Article 309.
Benjamin Blankertz, Claudia Sannelli, Sebastian Halder,Eva M. Hammer, Andrea Kübler, Klaus-Robert Müller,
Gabriel Curio, Thorsten Dickhaus, Neurophysiologicalpredictor of SMR-based BCI performance, NeuroImage
51 (2010) 1303–1309.
Stephen T. Foldes, Douglas J. Weber and Jennifer L.
Collinger, MEG-based neurofeedback for handrehabilitation, Journal of NeuroEngineering andRehabilitation (2015) 12:85.

References
1. Franca Tecchio, Filippo Zappasodi, Mario Tombini,
Antomio Oliviero, Patrizio Pasqualetti, Fabrizio Vernieri,
Matilde Ercolani, Vittorio Pizzella, and Paolo Maria
Rossini, Brain plasticity in recovery from stroke: An MEG
assessment, NeuroImage 32 (2006) 1326 – 1334.
2. Andrea Paggiaro , Niels Birbaumer , Marianna Cavinato ,Cristina Turco, Emanuela Formaggio, Alessandra Del© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

249

Motion control of a cable-restricted underwater vehicle
for long-term spot observation
Yoshiki Tanaka

Dept. of Life Science and Systems Engineering, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku,
Kitakyushu, Fukuoka, 808-0196, Japan
Yuya Nishida

Dept. of Human Intelligence Systems, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu,
Fukuoka, 808-0196, Japan
Kazuo Ishii

Dept. of Human Intelligence Systems, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu,
Fukuoka, 808-0196, Japan
E-mail: tanaka.yoshiki732@mail.kyutech.jp, y-nishida@brain.kyutech.ac.jp, ishii@brain.kyutech.ac.jp

Abstract
In order to acquire a time fluctuation data of the resources which needs for a marine resource development, wedeveloped an observation device with low operational risk and a wide observable area. The observation device
consists of an underwater station and an underwater vehicle, and underwater vehicle is tethered with a cable. By using
the restraint condition of the cable, our device was able to navigate the trajectory planned only by thrust control withan error of up to 0.14m.
Keywords: Underwater vehicle, Long-term observation, Cable-tethered, Self-localization

paper, we explain the proposed long-term observation
device and show the experimental results for evaluating
the position accuracy using the underwater vehicle.

1. Introduction
In Japan’s exclusive economic zone, there are many
marine resources including mineral resources and energy
resources [1]. These are very important basic resources
for the development of an industrial society, but they
have not yet reached the economic use of marine
resources. The reason is that these resources are widly
distributed on the seabed at a depth of 50m or more wheregeneral divers cannot dive, it is difficult for divers tosearch and observation. Un-tethered underwater vehicles,
AUVs (automonous underwater vehicles) are used as thepractical tools to investigate these marine resources[2][3]. However, the AUVs have the risk of their losing
itself on a hardware or software failures, the AUVs
cannot be used often for the resource investigation. Thus,
a time fluctuation data of the resources which needs for amarine resource development cannot be obtained by anexisting survey. This research aims to develop a longterm observation device with low operation risk and wide
observable area to acquire a time fluctuation data. In this

2. Trajectory model and Motion control
2.1. Overview
As shown in Fig.1, the proposed system consists of anunderwater station for supplying electric power and anunderwater vehicle, the underwater vehicle is tethered tothe underwater station with a cable. Underwater vehicle
can observe around the underwater station for a longterm by keeping the altitude constant and navigating
while applying tension to the cable. By using the restraint
condition by tethering, an underwater vehicle cannavigate the same trajectory with only simple motion
control without using self-localization based on DVL or
INS or using waypoints. Therefore, it is possible toobserve a wide area with lower risk than existingunderwater vehicles.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

253

Yoshiki Tanaka, Yuya Nishida and Kazuo Ishii

Fig. 1 Overview of proposal method

2.2. Trajectory model
To derive the relationship between the structure of theunderwater station and the trajectory of the underwater
vehicle, we constructed a theoritical formula based on the
involute curve. The involute curve is a curved line by theend point 𝑝𝑝𝑖𝑖 = [𝑥𝑥𝑖𝑖 𝑦𝑦𝑖𝑖 ]𝑇𝑇 of the cable when the cablewound around the basic circular radius 𝑎𝑎 centered on theorigin us always pulled and unwound, 𝑥𝑥𝑖𝑖 and 𝑦𝑦𝑖𝑖 areexpressed by the following equations:𝑥𝑥𝑖𝑖 = 𝑎𝑎(cos 𝜃𝜃 + 𝜃𝜃 sin 𝜃𝜃)𝑦𝑦𝑖𝑖 = 𝑎𝑎(sin 𝜃𝜃 − 𝜃𝜃 cos 𝜃𝜃)(1)Here, 𝜃𝜃 express the angle from the initial position. In thisresearch, we consider that the basic circular radius 𝑎𝑎changes by the cable thickness 𝑑𝑑𝑐𝑐𝑐𝑐 depending on thenumber of times 𝑛𝑛 of winding and unwinding of the cable.
Thus, the basic circular radius 𝑎𝑎𝑖𝑖 is defined by thefollowing equations:(2)
𝑎𝑎𝑤𝑤𝑤𝑤 = 𝑛𝑛 ∙ 𝑑𝑑𝑐𝑐𝑐𝑐 + 𝑟𝑟𝑏𝑏
(3)
𝑎𝑎𝑢𝑢𝑢𝑢𝑢𝑢 = (𝑁𝑁 − 𝑛𝑛)𝑑𝑑𝑐𝑐𝑐𝑐 + 𝑟𝑟𝑏𝑏

Fig. 2 Cable model

1(4)
𝐿𝐿𝑤𝑤𝑤𝑤 = 𝐿𝐿𝑚𝑚 − 2𝜋𝜋 �𝑛𝑛 ∙ 𝑟𝑟𝑏𝑏 + 𝑛𝑛(𝑛𝑛 − 1)𝑑𝑑𝑐𝑐𝑐𝑐 �
2
1(5)
𝐿𝐿𝑢𝑢𝑢𝑢𝑢𝑢 = 2𝜋𝜋𝜋𝜋 �𝑟𝑟𝑏𝑏 + �𝑁𝑁 − 𝑛𝑛(𝑛𝑛 − 1)� 𝑑𝑑𝑐𝑐𝑐𝑐 �2Here, 𝐿𝐿𝑤𝑤𝑤𝑤 is the cable length when winding, 𝐿𝐿𝑢𝑢𝑢𝑢𝑢𝑢 is thecable length when unwinding, and 𝐿𝐿𝑚𝑚 is the maximum
length of the cable. When applied to Eq. (1) with thechanges in the radius of the device and cable length

Here, 𝑎𝑎𝑤𝑤𝑤𝑤 is the basic radius when winding the cable,
𝑎𝑎𝑢𝑢𝑢𝑢𝑢𝑢 is the basic radius when unwinding the cable, and 𝑛𝑛
is the maximum number of times the cable can bewinding and unwinding. 𝑟𝑟𝑏𝑏 is the radius of the device,
and we adopted a parameter that considers the allowablebending radius of the cable used. Considering that thelength of the cable when winding or unwinding changes
depending on the number of times 𝑛𝑛 and the angle 𝜃𝜃, it isdefined by the following equations:© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

254

Motion control of acalculated from Eq. (2) and Eq. (4) as constraints, thetrajectory 𝑝𝑝𝑤𝑤𝑤𝑤 = [𝑥𝑥𝑤𝑤𝑤𝑤 , 𝑦𝑦𝑤𝑤𝑤𝑤 ]𝑇𝑇 at the time of winding is
expressed by the following equation:

𝑥𝑥𝑤𝑤𝑤𝑤 = 𝑎𝑎𝑤𝑤𝑤𝑤 cos 𝜃𝜃 − (𝐿𝐿𝑤𝑤𝑤𝑤 − 𝑎𝑎𝑤𝑤𝑤𝑤 𝜃𝜃) sin 𝜃𝜃
𝑦𝑦𝑤𝑤𝑤𝑤 = 𝑎𝑎𝑤𝑤𝑤𝑤 sin 𝜃𝜃 + (𝐿𝐿𝑤𝑤𝑤𝑤 − 𝑎𝑎𝑤𝑤𝑤𝑤 𝜃𝜃) cos 𝜃𝜃(6)Similarly, the trajectory 𝑝𝑝𝑢𝑢𝑢𝑢𝑢𝑢 = [𝑥𝑥𝑢𝑢𝑢𝑢𝑢𝑢 , 𝑦𝑦𝑢𝑢𝑢𝑢𝑢𝑢 ]𝑇𝑇 at the timeof unwinding is expressed by the following equation:

𝑥𝑥𝑢𝑢𝑢𝑢𝑢𝑢 = 𝑎𝑎𝑢𝑢𝑢𝑢𝑢𝑢 cos 𝜃𝜃 − (𝐿𝐿𝑢𝑢𝑢𝑢𝑢𝑢 + 𝑎𝑎𝑢𝑢𝑢𝑢𝑢𝑢 𝜃𝜃) sin 𝜃𝜃
𝑦𝑦𝑢𝑢𝑢𝑢𝑢𝑢 = 𝑎𝑎𝑢𝑢𝑢𝑢𝑢𝑢 sin 𝜃𝜃 + (𝐿𝐿𝑢𝑢𝑢𝑢𝑢𝑢 + 𝑎𝑎𝑢𝑢𝑢𝑢𝑢𝑢 𝜃𝜃) cos 𝜃𝜃

Fig.3 Experimental setup(7)

When the cable length calculated from Eq. (4) or Eq. (5)reaches 𝐿𝐿𝑚𝑚 , the trajectory of the underwater vehicle
becomes a circle centered on the device. Therefore, it isexpressed by the following equation.

𝑥𝑥𝑖𝑖 = 𝐿𝐿𝑚𝑚 cos 𝜃𝜃𝑦𝑦𝑖𝑖 = 𝐿𝐿𝑚𝑚 sin 𝜃𝜃Table1 Thrust value 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 and 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 [N]𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 [N]
4.0
5.0
10.0
15.0
20.0
9.0
5.0
10.0
15.0
20.0
16.0
5.0
10.0
15.0
20.0

(8)

𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 = 𝐷𝐷𝑥𝑥 |𝑉𝑉𝑥𝑥 |𝑉𝑉𝑥𝑥(10)
Here, 𝐷𝐷𝑥𝑥 is the fluid drag force. This parameter wasapproximated by experiment. Next, we explain the thrustforce 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 in the direction of extension. Fig.2 shows thecable model of the underwater vehicle when it is tethered
to the underwater station. If the cable has ideal sag
without considering the stiffness, and the underwater
vehicle moves slowly, the cable can be considered tohave a catenary shape [4]. Therefore, the sag 𝑧𝑧𝑐𝑐𝑐𝑐
𝑦𝑦𝑦𝑦
generated by the cable’s own weight and the tension𝑇𝑇𝑖𝑖
applied to the cable can be expressed by the following
eqations:
𝐻𝐻 𝑦𝑦𝑦𝑦
𝑤𝑤𝐷𝐷𝑚𝑚
𝑤𝑤 𝐷𝐷𝑚𝑚𝑧𝑧𝑐𝑐𝑐𝑐 =�cosh 𝑦𝑦𝑦𝑦 − cosh 𝑦𝑦𝑦𝑦 �− 𝐷𝐷𝑖𝑖 �� (11)
𝑤𝑤
2𝐻𝐻
𝐻𝐻
2
𝑤𝑤 𝐷𝐷𝑚𝑚𝑦𝑦𝑦𝑦
𝑇𝑇𝑖𝑖 ≥ 𝐻𝐻 𝑦𝑦𝑦𝑦 cosh 𝑦𝑦𝑦𝑦 �− 𝐷𝐷𝑖𝑖 �(12)
𝐻𝐻
2
Here, 𝐻𝐻 𝑦𝑦𝑦𝑦 is the horizontal tension at the lowest point of
the cable, 𝑤𝑤 is the weight in water per unit cable length,
𝐷𝐷𝑖𝑖 is the distance between center of the device andunderwater vehicle, 𝐷𝐷𝑚𝑚 is the maximum distance
between center of the device and underwater vehicle.
Therefore, the thrust force in the direction of extension
required for the underwater vehicle to suppress the sag
caused by the weight of the cable is expressed by thefollowing equation:
𝑦𝑦𝑦𝑦
𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 ≥ 𝑇𝑇𝑖𝑖 cos 𝜑𝜑(13)
In this paper, we calculated the minimum thrust 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 in
the direction of extension by assuming that the trajectoryof the underwater vehicle obtained from Eq. (6) to Eq. (9)contains an error of at most 1% due to cable’s sag.

Here, the cable length in the trajectory calculated fromEq.
(8) is symmetrical around the device. Therefore, therange of the angle 𝜃𝜃 is 0 ≤ 𝜃𝜃 ≤ 180 [deg].
2.3. Motion control

In the water, the weight and fluid resistance of the cableincrease depending on the length of the unwound cable,
so the sag of the cable increases. As the sag increases, it
becomes more difficult for underwater vehicle to draw
the planned trajectory and the positioning accuracy
decreases. Therefore, we calculated the thrust required in
the direction of travel and the thrust required in the
direction of extension. At first, we explain the thrustforce 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 required in the direction of travel. In the future,
we plan to create a map from seabed images by a camera
mounted on an underwater vehicle. We are considering
the creation of a seafloor map for long-term observationof marine resources. In order to create a map using
images taken by the equipped camera in underwater
vehicle, it is necessary to input velocity parameters to theunderwater vehicle so that the captured images overlap.
The velocity𝑉𝑉𝑥𝑥 required to satisfy a specific overlap rateis expressed by the following equation:
𝑑𝑑𝑆𝑆 (1 − 𝛼𝛼𝑂𝑂𝑂𝑂 )(9)𝑉𝑉𝑥𝑥 ≤
𝑇𝑇𝑆𝑆

Here, 𝑑𝑑𝑆𝑆 is the shooting range of the camera, 𝛼𝛼𝑂𝑂𝑂𝑂 is theoverlap rate, and 𝑇𝑇𝑆𝑆 is the shooting cycle. By substituting
the velocity calculated from Eq. (9) into the following
equation, the thrust𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 in the direction of travel can beobtained.

3. Wet test and evaluation of trajectory
To evaluate the trajectory of the cable-tethered
underwater vehicle described in section 2.2, we© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

255

Yoshiki Tanaka, Yuya Nishida and Kazuo Ishii

Table2 Position error when 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 = 4[N]MSE
𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆X
YDistance[N]
position[m] position[m]
[m]
5.0
0.06
0.03
0.07
10.0
0.03
0.03
0.03
15.0
0.02
0.04
0.03
20.0
0.01
0.10
0.06
Table3 Position error when 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 = 9[N]MSE
𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆X
YDistance[N]
position[m] position[m]
[m]
5.0
0.06
0.05
0.07
10.0
0.05
0.03
0.04
15.0
0.03
0.04
0.04
20.0
0.03
0.11
0.07
Table4 Position error when 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 = 16[N]MSE
𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆X
YDistance[N]
position[m] position[m]
[m]
5.0
0.12
0.14
0.14
10.0
0.05
0.07
0.05
15.0
0.03
0.02
0.03
20.0
0.03
0.09
0.06

Fig.4 Trajectory of AUV KYUBIC(Theoretical value vs. Experimental value)experimented using the hovering type AUV KYUBIC.
This AUV is equipped with IMU, DVL and depth sensors
as navigation sensors. We used the velocity data from
DVL and the heading data from IMU to evaluate the selflocalization. Fig.3 shows an overview of this experiment.
The AUV is tethered to the device by a cable (thickness
is 12mm and length is 2.2m). The device resembling an
underwater station is set in the center of the water tank(diameter is 6m and depth is 1.2m). In this experiment,
we used the parameters shown in Table1 as the thrustforce applied to the AUV and its self-localization wasmeasured when winding the cable. Fig.4 shows the
trajectory calculated using Eq. (6) and the selflocalization of the AUV at 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 = 16[N]. The positional
accuracy is improved by increasing the thrust force 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 .
However, a different trajectory from the theoretical value
can be confirmed immediately after the AUV starts
moving. This cause is considered that the azimuth during
navigation contained an error with the teoretical value
because the AUV moved only by thrust control. Table2
to Table4 show the mean square error (MSE) of the selflocalization when the thrust applied to the AUV ischanged. It can be confirmed that the distance error
between the center of the device and AUV is reduced byincreasing the thrust force 𝐹𝐹𝑆𝑆𝑊𝑊𝑊𝑊 . However, the MSE of
distance at 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 = 20[N] is larger than at 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 = 15[N].
This cause is considered that the velocity of the AUV is
reduced by increasing the thrust force 𝐹𝐹𝑆𝑆𝑆𝑆𝑆𝑆 , and theposition data calculated by the DVL and IMU containaccumulated error.

4.

Conclusions

We proposed a new method that can observe a wide areawith lower risk than exisiting method. Our instrument
was able to move only by thrust control, with an average
error of up to 0.14m relative to the theoreticel value. In
future work, we plan to evaluate the trajectory when
unwinding the cable and add the heading control.
References
1. C.E. Fitzgerald, et al., Hydrothrmal Manganese Oxide
Deposits from Baby Bare Seamount in the Northeast
Pacific Ocean, Marine Geology, Vol.225, Issue 1-4,
pp.145-156,2006
2. Y. Nishida, et al., Development of an autonomous
underwater vehicle for survey of cobalt-rich manganesecrust, OCEANS2015 MTS/IEEE Washington, pp.1-5,
2015
3. Toshihiro Maki, et al., Docking Method for HoveringType AUVs Based on Acoustic and Optical Landmarks,
Journal of Robotics and Mechatronics, Vol.30, No.1,
pp.55-64, 2018
4. Atsushi Imadu, et al., Control Method for Helicopters
Tethered to Ground Station with Compensation forDisturbance Caused by Cable Tension, Jornal of Robotics
and Mechatronics, Vol.29, No.4, pp.737-745, 2017

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

256

Development of current sensors for digitizing expert knowledge in fish feeding towards
sustainable aquaculture
Dominic B. Solpico

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0135, Japan
Yuya Nishida

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0135, Japan
Kazuo Ishii

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0135, Japan
E-mail: solpico.dominic-bautista806@mail.kyutech.jp, y-nishida@brain.kyutech.ac.jp, ishii@brain.kyutech.ac.jp
www.lsse.kyutech.ac.jp

Abstract
Improving the efficiency of fish feeding contributes to achieving sustainable expansion of the aquaculture industry.
However, expert knowledge on feeding remains reliant on experience. This paper presents a new approach ofdigitizing such knowledge by measuring underwater currents induced by fishes as indicator of their behavior andappetite. A prototype current sensor suite was constructed to measure the current around the fish cage, especiallyduring feeding.
Keywords: aquaculture, fish feeding, underwater currents, current sensors

threat leads to slower growth rates, poorer quality of
harvest, or at worst, massive fish kills.6,7,8 Fish farmers
incur losses in their operations as a result.3,5,6
Efficient feeding has usually been achieved with adecision made by an expert farmer. There is a significant
difference in the quality of the harvested fish fed basedon expert and non-expert decisions. Such decision
making remains to be an “art,” where prediction is still
intuitive, subject to the expert’s experience, andunquantifiable by a unified standard.9,10
Digitization of experts’ knowledge may help non-experts
improve their feeding decisions, which will not only
improve the amount and quality of their harvest, which
will not only consequently increase their income, but will
also increase the supply of high-quality seafood, lower
their prices, and at the same time minimize pollution in
the farm environment.

1. Introduction
While seafood consumption has been increasing every
year, capture fisheries production, facing issues ofoverfishing, has remained static with slight fluctuationsin the past decades. Since early 1990s, the aquacultureindustry has been expanding consistently to meet therising demand worldwide, making up 46% of the global
production in 2018.1
The problem arises in the industry’s sustainability.
Unsustainable management of fish farms not only posesa threat to their surrounding aquatic environment but also
to the health of the fish stocks.2,3,4 One crucial issue is theneed for efficient feeding decision-making. Poorly timed
and excessive feeding leads to poor cost-efficiency inraising fish.5 Worse, it produces uneaten feeds that
decompose at the bottom, contributing to the decline ofwater quality in the surrounding environment.6 Such

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

257

Dominic B. Solpico, Yuya Nishida, Kazuo Ishii

Current research efforts on this problem focused on themeasurement of fish behavior through machine vision
and water quality changes using sensors as input
parameters into machine learning models to make afeeding decision.5,10,11 So far, experiments in these works
were performed in intensive farms where environmental
factors were controlled. No research work performed inopen environments, i.e. extensive farms, have been foundso far.
This paper explores a new approach on digitizing expert
feeding knowledge in tuna farms in aquatic environments.
In this approach, velocities of underwater currents
induced by the movement of fishes inside a cage will bemeasured and their relationship to the behavior andappetite of the fish inside will be investigated. While
presenting the sensor system’s overall architecture, thispaper focuses on the design of a prototype sensor for
initial current measurements and on its ongoingdevelopment.

the sensor readings to obtain the measurements,
timestamps them, and stores them internally.
These sensor nodes communicate in a star network,where one node is designated as the master and the rest
as slaves. Slaves send their measurement data to themaster, which also collects its own measurements. Data
aggregated by the master node may either be collected by
the farmer onsite or be transmitted directly to a data
center. Communication technology to be used – WiFi, 3G,
4G, LoRa, etc. – will be configured accordingly to
implement this network architecture.
Initial data collection is to be made in a fish farm, where
underwater currents are to be measured near cages at
depths with fish presence, especially during feedingtimes. This is to provide insight for validating or in
improving the proposed design.

2. Sensor System Architecture

For the initial measurements, two custom-built current
speed loggers are mounted on a metal frame at two
different depths at which fishes are observed before,
during, and after feeding. These sensors measure only

3. Sensor Suite for Initial Measurements
3.1. Sensor suite system design

In this system, a network of sensor nodes is placedaround a 40-meter diameter fish cage with depth of 20
meters. Each node consists of multiple current sensors
measuring current velocities for every 1-meter depth. In

Fig. 2. Design of the initial farm experiment using theprototype sensor suite consisting of flow and IMU sensors

Fig. 1. System architecture for monitoring fish appetite andwater quality in a fish cage

one flow axis given their fixed mounting. An inertial
measurement unit (IMU) sensor is mounted closest to thecurrent sensor of interest. This will measure movementof the frame caused by fish-induced currents. Thesesensors are remotely triggered by Bluetooth to startlogging measurements before they are placed underwater
for hours.
Ideally, at least two sets of sensor units should be
deployed so that the other will measure the current goingtowards the fish cage, which will be cancelled from theaddition to measuring currents, it also has water quality
sensors (dissolved oxygen, temperature, conductivity,
pH, etc.) at multiple depths. An underwater camera isplaced to observe the fish movements, especially during
feeding. Each node is designed to have a capacity forenergy-harvesting– solar, wind, tidal, or other sources –so that it can operate continuously off-grid. A computer
above surface performs corresponding calculations on© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

258

Development of current sensors

permanently sealing its electronics enclosure and by
replacing its original cable with a waterproof rubber
molded cable. Other unit components are housed in an
IP68 enclosure.
The mounting is a 12-meter aluminum structure with fourlegs to which the sensors are attached to. The sensors
depth can be adjusted by sliding them through the legs.
Each leg is made of three four-meter frames. Cross-like
reinforcements are attached at the leg joints to minimize

outgoing current. Due to time constraints, however, one
set will be used for the initial measurement. Two or more
sets will be deployed for the next measurement
campaigns.
3.2. Custom-built current sensor
3.2.1. Components and operation
Since initial measurements will be performed, it was
decided to use a low-cost current sensor. The sensor
developed for this experiment is a modified propellertype flow sensor intended for measuring water flow
through water pipes. Its Hall-effect sensor generates

Fig. 3. Aluminum frame mount mounting for the sensors
without reinforcement at the segments which explains theFig. 3. IMU sensor enclosed in its waterproof box with its
bendingaluminum attachment to the framebending and to enable the mounting to withstand theunderwater currents.

Fig. 3. Calibration test setup of the current sensor where it isconnected
to a digitalcurrent
flow sensor
theyinitialboth measure
water
Fig. 3. Constructed
sensorasfor
measurements
flow
from
the
container
with the reinforced flow sensor and the electronics box on its

3.2.2. Flow sensor cross calibration
For accurate measurements, the developed flow sensor
was cross-calibrated with a digital clamp-on type flow
sensor (Keyence FD-Q32C). Both sensors wereconnected to an elevated water source where flow waspartially controlled, as velocity and flow rate weredependent on the height, and subsequently the volume, of
the water in the container.
Calibration is done by correlating the pulse frequency tothe speed of water through the custom sensor, which is
calculated by measuring the flow rate through the digitalsensor. This is calculated using the relationship between
the flow rate and speed of a fluid through a pipe and thecontinuity equation, which given in the following
equation:

aluminum mounting

pulses proportional to the magnetic propeller’s rotation.
An Arduino microcontroller counts these pulses for agiven period, obtains the average frequency throughout
the period, and then calculates the current speed using acalibration coefficient. It then adds timestamping to thereading and stores it in a microSD card with its
datalogging shield with SD and RTC capability. A
Bluetooth module is also connected so it can be triggered
remotely by a computer to start and stop measurements.
The current sensor is powered by a 9-volt supply (six AA
batteries in series) with a capacity of 2700 mAh. A power
endurance test was made by allowing it to operatecontinuously while being powered by the batteries.
Result showed that it can collect measurements reliably
for around 22 hours.
The flow sensor was not originally designed to bewaterproof and reinforcements were therefore made by

𝑣𝑣𝐹𝐹 =

𝑣𝑣𝐷𝐷𝐷𝐷 𝐴𝐴𝐷𝐷𝐷𝐷
𝑞𝑞𝐷𝐷𝐷𝐷 1000
��
�
=�
𝐴𝐴𝐹𝐹
𝐴𝐴𝐹𝐹
60

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

259

(1)

Dominic B. Solpico, Yuya Nishida, Kazuo Ishii

The core component of this sensor is the Sparkfun 9DoF
Razor IMU M0, a very compact microcontroller with an
MPU-9250 IMU and a µSD card slot onboard. Its IMU
consists of accelerometer, gyroscope, and magnetometer
sensors, and is therefore capable of measuring linear
acceleration, angular rotation velocity, and magnetic
field vectors. With a Real-Time Clock (RTC) attached,this board can timestamp its measurements before
writing them to an µSD card. Powered by a 1000-mAh
lithium-ion polymer (Li-Po) battery, it is enclosed in an
IP68 enclosure. Power endurance test result showed that
this sensor can collect data for around 22 hours as well.

respectively. The second measurement was along Nishi
River (西川), a small river connected to the northwesternend of Onga River, where the width and depth at themeasurement point is around 75 and 1.3 meters,
respectively. Due to the shallow depths, only one 4-meter
frame segment was used. Currents at the bottom and nearsurface of both rivers were measured, with the heights of
the upper sensor were adjusted accordingly. The IMU
sensor was placed in proximity to the bottom current
sensor. Measurements at each river were taken for around30 minutes.
Save for two data points for each depth ranging from 0.75
to 2.26 cm/s, the currents sensor readings along Onga
River were at 0 cm/s. More non-zero sensor readings
were collected at Nishi River, with the measurement at
the surface peaking to 18.8 cm/s. On the other hand,almost all readings at the bottom of this river is at zero,
except for a few instances peaking at 3.76 cm/s. No
significant changes were observed in the accelerometerand gyroscope readings since the frame was settled at theriverbed. Any changes observed was attributed to lack of
calibration as well as human intervention.
It was expected for Onga river’s current to be slow
because of its large width. However, visual observations
suggest that there still were small movements by thewater, as seen from the waves at the surface. This may
indicate that the flow sensor could not detect those small
currents, reading them as 0 cm/s.
For better results in future measurements, we willcalibrate the sensors’ pulse readings for eachmeasurement to the readings of a calibrated digital sensor.
Doing this can verify whether the custom sensors candetect small currents or not. In addition, we areconsidering attaching a funnel at the inlet of the current
sensors. This will amplify small currents entering thesensor, which will then use its measured current tocalculate for the current at the funnel inlet.

4.

5.

In this equation, v is the water speed, A is the crosssection area of the sensor pipe, and q is the flow rate.
Subscripts F and DF denote the custom and the digitalsensors, respectively. Regardless of the difference of thecross-section areas of the two sensors, that of the digitalsensor is cancelled out in the equation. The constant at
the right converts the units from flow rate (L/min) to
speed (cm/s).
To select the best averaging period to be used inmeasurements, three periods were selected for calibration– 1, 5, and 10 seconds. Readings at 1 second were foundto be discrete as the microcontroller count discrete
number of pulses per second. Range of readings from thedigital sensor are classified as the discrete readings fromthe custom sensor. Readings using the 10-second
configuration were more continuous. However, there arehigher chances of averaging high and low sample values,
which may not properly represent the actual
measurement. Using the 5-second period seems to be afavorable configuration as there are smaller chances ofsamples with large differences, while its readings are still
continuous. This is therefore the selected configuration
for the upcoming experiments.
3.3. IMU water movement sensor

Trial river measurements

Conclusion and future work

While waiting for the plans of the initial fish farmexperiment to be finalized, we decided to perform current
measurements with the constructed sensors along tworivers in the northern part of Fukuoka. The firstmeasurement was along Onga River (遠賀川), one of thelongest rivers in Kyushu island. Its width and depth at the
point of measurement is around 290 and 1.6 meters,© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

260

Development of current sensors

research tasks towards digitizing expert knowledge infish feeding in aquaculture farms.

This paper presents the ongoing development of a
prototype sensor suite, consisting of modified flow formeasuring water currents induced by fish movement.
Flow sensors were modified and recalibrated to measure
underwater currents. The IMU sensor was also developed
for measuring movement of the mounting frame. Boththe current sensors and the IMU sensor can record current
measurements reliably for at least 22 hours.
As the mounting frame has been completed, the sensor
suite is ready to be deployed in the fish farm for the initial
measurements, which will be performed within the next
few weeks. Trial measurements have been made along
two rivers in Fukuoka, showing A second sensor suite
will also be constructed after the first experiment. The

References
1. FAO: The State of World Fisheries and Aquaculture 2020.
In brief. Sustainability in action, FAO, 2020.
2. R. S. S. Wu, “The environmental impact of marine fishculture: towards a sustainable future,” Marine Pollution
Bulletin, vol. 31, no. 4–12, pp. 159–166, 1995.
3. A.R. Brown et al., “Assessing risks and mitigating impactsof harmful algal blooms on mariculture and marine
fisheries,” Review in Aquaculture, vol. 12, no. 3, pp. 16631688, 2020.
4. M. Martinez-Porchas, L. R. Martinez-Cordova, "World
Aquaculture: Environmental Impacts and Troubleshooting
Alternatives", The Scientific World Journal, vol. 2012,
Article ID 389623, 9 pages, 2012.
5. T. Wu, Y. Huang, J. Chen, “Development of an adaptive
neural-based fuzzy inference system for feeding decisionmaking assessment in silver perch (Bidyanus bidyanus)
culture,” Aquacultural Engineering, vol. 66, pp. 41-51,
2015.
6. M. L. San Diego-McGlone, R. V. Azanza, C. L. Villanoy
and G. S. Jacinto, “Eutrophic waters, algal bloom and fish
kill in fish farming areas in Bolinao, Pangasinan,
Philippines,” Marine Pollution Bulletin, vol. 57, no. 6-12,
pp. 295-301, 2008.
7. G. A. Santos, J. W. Scharma, R. E. P. Mamauag, J. H. W.
M. Rombout and J. A. J. Verreth, “Chronic stress impairs
performance, energy metabolism and welfare indicators in
European seabass (Dicentrarchus labrax): The combined
effects of fish crowding and water quality deterioration,”Aquaculture, vol. 299, no. 1-4, pp. 73-80, 2010.
8. H. Sun, J. Li, L. Tang and Z. Yang, “Responses of crucian
carp Carassius auratus to long-term exposure to nitrite andlow dissolved oxygen levels,” Biochemical Systematics
and Ecology, vol. 44, no. 1, pp. 224-232, 2012.
9. M. Sun, S. G. Hassan, D. Li, “Models for estimating feedintake in aquaculture: A review,” Computers andElectronics in Agriculture, vol. 127, pp. 425-438, 2016.
10. C. Zhou et al., “Near infrared computer vision and neurofuzzy model-based feeding decision system for fish inaquaculture,” Computers and Electronics in Agriculture,
vol. 146, pp. 114-124, 2018.
11. C. Zhou, et al., “Evaluation of fish feeding intensity inaquaculture using a convolutional neural network andmachine vision,” Aquaculture, vol. 507, pp. 457-465, 2019.

Fig. 3. Current sensor readings at both depths at Onga River
were very low, while larger readings were measured at the nearsurface of Nishi River.

relation of current measurements from the experimentswith the fish behavior during feeding will be analyzed.
Future work also includes development of a networks of
sensor nodes as described in the system architectureusing more robust current sensors. These are future© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

261

Underwater image reconstruction using convolutional auto-encoder
Shinsuke Yasukawa

Dept. of Human Intelligence Systems, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: s-yasukawa@edu.brain.kyutech.ac.jp
Sreeraman Srinivasa Raghura

Dept. of Human Intelligence Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: srinivasa.raghuraman-sreeraman344@mail.kyutech.jp
Yuya Nishida

Dept. of Human Intelligence Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: y-nishida@brain.kyutech.ac.jp.
Kazuo Ishii

Dept. of Human Intelligence Systems, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka, 808-0196, Japan
E-mail: ishii@brain.kyutech.ac.jp.

Abstract
One of the main tasks of AUVs is to capture deep-sea images like fishes, crabs, other living organisms and resources
for information leading to research on deep-sea ecosystems. Acoustic transmission are used to establish wirelessunderwater communications between the AUV and the ship. However, there are some limitations in thecommunication channels due to limited bandwidth, multi-path, temperature distribution and change in the directionof transmitting source and receiving sensor which results in losses in data being transmitted. Initially, the captured
images are enhanced to reduce the effect of light attenuation and then compressed for transmission through acoustic
modems. Only an important part of image is being transmitted through set of data packets. The received data packets
in the ship will be reconstructed to predict the presence of living organisms. The loss in data during transmission
creates a difficulty for the operators to predict the exact information. In this research, to compensate this transmission
loss, an efficient compression and reconstruction technique using convolutional autoencoder with minimal distortionis proposed. Finally, for evaluation of the proposed image compression technique, the quality of reconstruction ofimages with and without data loss will be compared using the quality metrics signal to noise ratio (PSNR), structuralsimilarity index(SSIM) and perceptual quality of image.
Keywords: sampling-AUV, acoustic communication, image compression, convolutional auto-encoder1.

in ocean engineering. Research studies, especially in thearea of autonomous underwater vehicles (AUV) which
are being used as tools for deep sea-floor surveying andobservation. AUVs are mainly employed for performing

Introduction

Presently numerous research and development activities
have been carried out in the field of underwater robotics© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

262

Shinsuke Yasukawa, Sreeraman Srinivasa Raghura, Yuya Nishida, Kazuo Ishii

tasks like underwater imaging, sea-floor mapping,
exploiting resources and minerals which are of greatinterest to geologists and marine biologists to understandthe sea flora and fauna.
One of the main tasks of AUV is optical imaging ofparts of the sea-floor that contains marine habitat. These
kinds of images contribute to research studies on deepsea biology. The further task involves collecting samples
using end-effectors. It is difficult for the AUV to performthe above task autonomously as it does not have any priorknowledge on the targets of interest. To overcome thisdifficulty, it operates semi-autonomously by acquiring
assistance from biologists and geologists. Fig. 1 shows
the flow of the mission to sampling objects on sea-floor.
The acquired image is then transmitted to the vessel.
Images are transmitted underwater using acoustic waves
through hydrophones. The received image in the ship is
examined by the operator and if it shows any existence ofmarine habitat then behavior commands are sent to the
AUV. Finally, AUV is sent back to the specified location
for collecting samples. Our group developed thesampling-AUV for the sampling mission1,2,3 andsucceeded three times in a row in sampling experiments
in Suruga Bay in March 20183.
The process of image transmission involves thefollowing steps:(i) First, the AUV acquires raw images of seafloor thatare of interest.
(ii) These images are then enhanced to reduce the effectof light attenuation and unbalanced illumination.
(iii) Next, images compressed using the requiredcompression technique for transmission through theacoustic medium.
(iv) Finally, the data received in the ship reconstructed
and the operator examines the reconstructed image.
During the process of image transmission, there is a lossof data(information). The data loss is mainly due to thefollowing factors: multipath, temperature distribution,
Signal attenuation over long ranges, and directionalchanges with the transmitting source and the receiving
sensor. Underwater communication has low data rates
when compared to terrestrial communication since it uses
acoustic waves instead of electromagnetic waves.
It is known that some loss occurs during thetransmission of the compressed data through the acoustic
communication. This data loss has to be minimized toobtain a better-quality image reconstruction at thereceiving end. Therefore, we propose a lossy based image
compression method that utilizes the merits ofconvolutional autoencoder network. The purpose of thisresearch is developing an efficient image compression
model that can reconstruct the image with minimal data
loss. To achieve the above, the deep-learning approach isused to simulate the compression/ reconstruction modelwith the generation of data loss in the hidden codes thatwould result in improved Peak Signal to Noise Ratio(PSNR) and better Structural Similarity Index (SSIM).

Fig. 1. Underwater Image Transmission.

2. Related works
Ahn et. al.2 have developed an image enhancement and acompression technique for underwater acoustic
communication with limited data information density.
Their proposed method enhances the brightness of the sea
floor images and reduces the image depth from 24-bit to
4-bit. They have used a data compression method that isbased on indexed color technique which expresses
images with a fixed number of color indices using asuitable color palette. The color palette consists of set ofindexed colors to express the enhanced images
depending on the targeted creatures. As shown in Fig. 2,
the problem of data loss that occurred duringtransmission could not be minimized which affected thequality of image reconstruction at the receiving end. The
image is transmitted in the form of packets of
information.
Hoag et.al.4 have developed an underwater image
compression technique using wavelet transforms (WT).
They have employed a wavelet decomposition method in
combination with vector quantization for compressing
still underwater images. Compared to JPEG compression
their proposed method tends to perform well at low bit
rates. However, over a wide range of directionality the
wavelet transforms are generally inefficient inrepresenting geometric structures. The texture and detailsof reconstructed images are not efficiently retained using
wavelet transforms.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

263

Underwater image reconstruction using

compressed data (i.e. from a dimension of 14×14×16 to
8×8×16, where 16 refers to the number of feature maps)
there by creating some loss of data randomly. Therefore,
the decoder attempts to reconstruct the image from thisrepresentation. Depending on the size of data loss, thedecoder needs to be modified to match the size of theoriginal image. Due to the above sperate decoder
networks are required for reconstruction.

Fig. 2. Selected and received images with data loss duringtransmission[2].

In the research work carried out by Cheng et.al.5 a
lossy image compression method using deep
convolutional autoencoders (CAE) was proposed. They
have designed a convolutional autoencoder network that
replaces conventional transforms and performed training
using rate-distortion loss function. Compared to
JPEG2000 and other traditional algorithms their method
performs well on the images from kodak database.
3. Proposed Method

Fig. 3. Overview of the proposed image compression method.

The Fig. 3 shows an example of overview of the proposed
image compression method utilized for underwater
image transmission process. Initially the acquired image
data is compressed to codes (assuming the encoder in the
AUV). In the compressed representation we generate adata loss by discarding a limited amount data from thecodes (i.e. eliminating few rows or columns). This
corrupted data will be sent to the decoder forreconstruction. The decoder extracts relevant
information from this data and attempts to reconstruct theimage similar to that of the original. The reconstruction
depends on the robustness of the decoder.
Fig. 4 shows CAE Network with data loss. As an
initial step, the dataset consisting of interesting images
was created for training and testing from a databaseconsisting of enhanced underwater images. The selected
images from the database were resized from a dimensionof 320×240×3(width, height) to 224×224×3(width,height) to match the overall receptive field of thestandard convolutional layers. As a preprocessing step,
the input images of dimension 224×224×3 are initiallynormalized to [0,1]. The dataset consists of a total 100
images with 70 images for training and 30 images fortesting. Using convolution and deconvolution filters asymmetric convolutional auto-encoder network is
designed. Then, training is performed using meansquared error loss function and Adam optimizer.
Here, based on the standard CAE, an additional
pseudo layer was introduced which takes the input datafrom the encoded layer. This layer attempts to slice theFig. 4. CAE Network with data loss.

4. Experiments and Result
4.1 Experimental setup
A dataset consisting of 100 images was used for training
and testing the proposed model. These images wereobtained from previously conducted seafloor exploration
experiments using Sampling-AUV: TUNA-SAND2. The
number of filters in the convolutional layers were set as{128,64,64,32,16} and the decoder part mirrors theencoder. Mean square error loss function was used during
training process to compute the distortion between
original and reconstructed images. The parametric ReLU
activation was employed in the network to improve thequality of reconstruction at high bit rate. For measuring© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

264

Shinsuke Yasukawa, Sreeraman Srinivasa Raghura, Yuya Nishida, Kazuo Ishii

the efficiency of the proposed image compression
technique, the quality of reconstruction of images withand without data loss were compared using the quality
metrics signal to noise ratio (PSNR) and structuralsimilarity index(SSIM).
The simulations were performed on a PC with thefollowing specifications Intel Core i7-8750H CPU at
2.40 GHz, 8GB RAM and GeForce GTX 1060Ti GPU.
The CAE model was trained for 150 epochs i.e. 150
iterations over all the samples in mini-batches of 2. The
running time i.e. the time taken for one complete
encoding and decoding process for RGB images withresolution of 224 × 224 was approximately 3.60s/step.

Fig. 6. Comparison of SSIM with data loss.

4.2 Results

accuracy of 55 to 70%.

The Fig. 5 shows the comparison of PSNR at various
compression ratios of original and reconstructed images.
The proposed model was able to achieve a high signal tonoise ratio which results in better quality of image. At
higher compression ratios there is slight reduction inquality along with decrease in PSNR value.
Since the PSNR evaluation considers only numericalcomparison of images, SSIM is also taken as a measurefor evaluating the quality of the image.
From the graph shown in Fig. 6 we can see that amaximum SSIM value of 0.78 for sample image 1 at acompression ratio of 18:1, the decoder is able toreconstruct the image with satisfactory image quality
even with data loss in the compressed codes. The qualityof reconstruction decreases at higher compression ratios.
This is because when trying to reconstruct the discarded
portion the autoencoder tries to add black and white color
pixel intensities in place of lost pixels.
While evaluating the quality metrics in terms ofluminance, contrast and structure of the reconstructed
image with the original image considering the data loss,
the results indicate that the proposed CAE model is able
to provide better reconstruction quality and achieve an

5. Conclusion
In this research, an efficient lossy image
compression/reconstruction
architectureusing
convolutional autoencoder was developed. The proposed
CAE model tends to achieve better image reconstruction
and perceptual quality with minimal distortion. The
performance of the image compression algorithm wasevaluated using quality metrics like PSNR and SSIM. It
was observed from the results that the proposed
autoencoder model was able to achieve fairly high signal
to noise ratio with data loss providing a goodreconstruction quality image.
References
1. Nishida, Y., Sonoda, T., Yasukawa, S., Nagano, K.,
Minami, M., Ishii, K., & Ura, T., Underwater Platform forIntelligent Robotics and its Application in Two Visual
Tracking Systems, Journal of Robotics and Mechatronics,
30(2), 238-247, 2018.
2. Ahn, J., Yasukawa, S., Sonoda, T., Nishida, Y., Ishii, K.,& Ura, T. An optical image transmission system for deepsea creature sampling missions using autonomous
underwater vehicle, IEEE Journal of Oceanic Engineering,2018.
3. Yasukawa, S., Nishida, Y., Ahn, J., Sonoda, T., Watanabe,K., & Ishii, K., Development and Sea Trials of VisionBased Control for Sampling-AUV, In 2019 IEEE
Underwater Technology, pp. 1-4, 2019.
4. Hoag, D. F., & Ingle, V. K., Underwater image
compression using the wavelet transform. In Proceedingsof OCEANS'94, Vol. 2, pp. II-533, 1994.
5. Cheng, Z., Sun, H., Takeuchi, M., & Katto, J., Deep
convolutional
autoencoder-based
lossy
image
compression. In 2018 Picture Coding Symposium (PCS) ,pp. 253-257, 2018.

Fig. 5. Comparison of PSNR at different compression ratioswith data loss.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

265

The Spherical Robot Transfer Problem with Minimal Total Kinetic Energy *Kenji Kimura

Engineering course, Fukuoka Daiichi High School
22-1 Tamagawamati, Minamiku, 815-0037, Fukuoka, Japan
E-mail: kimuken1977_2058@yahoo.co.jp †Kazuo Ishii

Graduate School of Life Science and engineering, Kyusyu Institute of Technology
2-4 Hibikino,Wakamatsu-ku, Kitakyushu 808-0196, Fukuoka, Japan
E-mail: ishii@brain.kyutech.ac.jp

Abstract
Previous spherical mobile robots were driven by two rollers with a fixed rotational axis, which restricts the angularvelocity vector of the sphere to two dimensions. Three-dimensional freedom is expected to improve the rotational
diversity of the sphere. This study proposes a spherical mobile robot with a variable roller-rotational axis that allows
three-dimensional freedom of movement. Furthermore, the kinetic energy of transporting the sphere by the rollers
is minimized by an optimization procedure.
Keywords: Angular velocity vector of the sphere, Angle of sphere rotational axis, Total kinetic energy

spheres (Figure 1 (b)). The rollers are arranged on theequator, generate an angular velocity vector on thehorizontal plane, and can move in all directions. The
angular velocity vector of the sphere has twodimensional freedom. This situation is theoreticallyconsidered in [3]. In the ball dribbling mechanism [4], therollers are arranged in the upper hemisphere, where theyhold the balls by friction (See Figure 1 (c)).
Among the three-roller cases (𝑁𝑁𝑤𝑤 = 3), OWMPs [5]deployed in highway maintenance move the sphereswithin three constrained rollers (See Figure 1 (d)).The
rollers are arranged on the equator parallel to thehorizontal plane, and the sphere can be rotated in all
directions by generating an angular velocity vector on the

1. Introduction
Many types of robot, such as omnidirectional mobile
robots and sphere-transported robots, are based on
spherical
motions.
Therefore,variousrollerarrangements have been proposed for mobile robot
applications.
Table 1 shows the usage statuses of single sphericalrobots operated by different mechanisms, and thedimensions of the existence spaces of their angularvelocity vectors. Figure 1 shows the numbers of rollers
𝑁𝑁𝑤𝑤 arranged per wheel in each mechanism, and theircontact types.
In the ACROBAT-S [1] mechanism with 𝑁𝑁𝑤𝑤 = 2, thecaster of each sphere is driven by two roller drives (seeFigure 1 (a)). The wheel chair mechanism [2] has three.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

266

Kenji Kimura, Kazuo Ishii

plane (note that the existence space of the angularvelocity vector is two-dimensional).
A ball balanced robot [6] has three unconstrainedrollers (omni-rollers) arranged in a regular triangularconfiguration (See Figure 1 (e)). CPU-Ball Bot [7] hasfour unconstrained rollers fixed in a square configuration(See Figure 1 (f)). Such three-dimensional freedom ofenergy, and Section 4 summarizes the results and suggests
ideas for future work.

2. Total kinetic energy
2.1 Angular velocity vector of the sphere
The center 𝑶𝑶 of a sphere with radius r is fixed as theorigin of the coordinate system 𝛴𝛴 − 𝑥𝑥𝑥𝑥𝑥𝑥 . The 𝑖𝑖 thconstraint roller (i = 1 or 2) is in point contact with thesphere at a position vector 𝑷𝑷𝒊𝒊 and is arranged such thatthe center of mass of the roller 𝑷𝑷𝒊𝒊 and 𝑶𝑶 are on the sameline. 𝝎𝝎 denotes the angular velocity vector of thesphere. 𝜼𝜼𝒊𝒊 denotes the unit vector along the rotational
axis of constraint roller. sphere direction 𝜑𝜑 (0° ≤ 𝜑𝜑 <360°) is the angle from 𝑥𝑥-axis.
Now, given the sphere mobile velocity 𝑽𝑽 (the center
velocity of sphere)

Table 1
Existence space of angular velocity vector for
sphere mobile robot
statuses

Rotational
Dimension

ACROBAT-S[1]Caster2

Wheel chair [2]

Wheel

2

Mechanism

OWMPs [5]Sphereconveyance
Wheel

Ball Balanced robot [6]

Wheel

3

CPU-Ball Bot [7]

Wheel

3

Ball dribbling mechanism [4]22

𝑽𝑽 = ‖𝑽𝑽‖[cos 𝜑𝜑

sin 𝜑𝜑0]𝑇𝑇(1)

𝝎𝝎́ which is perpendicular to 𝑽𝑽 is determined asfollow.
𝝎𝝎́ =‖𝑽𝑽‖[−sin 𝜑𝜑
𝑟𝑟

cos 𝜑𝜑0]𝑇𝑇(2)tan 𝜌𝜌]𝑇𝑇(3)

And. 𝝎𝝎́ is orthogonal projection of 𝝎𝝎 with respect to𝑥𝑥𝑥𝑥 − plane.The angle of sphere rotational axis 𝜌𝜌( −90° ≤ 𝜌𝜌 ≤ 90° ) is the angle between 𝝎𝝎 and 𝝎𝝎́ .
𝑇𝑇
Therefore, 𝝎𝝎 = �𝜔𝜔𝑥𝑥 , 𝜔𝜔𝑦𝑦 , 𝜔𝜔𝑧𝑧 � has one-dimensional of
freedom with parameter 𝜌𝜌 (𝜔𝜔𝑧𝑧 = ‖𝑽𝑽‖ tan 𝜌𝜌 /𝑟𝑟).
𝝎𝝎 =

‖𝑽𝑽‖
[−sin 𝜑𝜑
𝑟𝑟

cos 𝜑𝜑2.2 Kinetic energy of the two rollers

Figure 1 Type of roller arrangement for sphere mobile robot

Consider two rollers with radius 𝑅𝑅 , moment 𝐼𝐼, androller’s angular velocity 𝜔𝜔𝑖𝑖 at 𝑷𝑷𝒊𝒊 . The summed kinetic
energy of the rollers is given by Eqs.(4).

the angular velocity vector of a sphere will improve themotion diversity of the sphere. However, a mechanism
adapting two constrained rollers is suitable in a sphericalobject conveyance. It is desired to transport with highkinetic energy efficiency.
In this study, we optimize the total kinetic energy of
two rollers. Section 2 calculates the angle of therotational axis of the sphere, and theoretical formula ofminimizes the sum of the kinetic energies of the tworollers. Section 3 presents a simulation of total kinetic

𝐸𝐸 = 𝐼𝐼(𝜔𝜔12 + 𝜔𝜔22 )=𝐼𝐼(‖𝝎𝝎 × 𝑷𝑷1 ‖2 + ‖𝝎𝝎 × 𝑷𝑷2 ‖2 )
𝑅𝑅2

(4)

Using the angular velocity vector of a sphere, we now
derive the kinetic energy of Type-I (with both rotational© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

267

The Spherical Robot Transfer

axes fixed on the same plane) and Type-II (with(a)

As shown in Figure 2 (b), we determine 𝜌𝜌 = 𝜌𝜌1 , 𝝎𝝎 =(b)Figure. 2 Existence of the sphere angular velocity vector with respect to sphere movile velocity.(a) 𝝎𝝎𝟏𝟏 is determined(𝝎𝝎𝟏𝟏 is lay on intersection 𝐬𝐬𝐬𝐬𝐬𝐬𝐬𝐬{𝑷𝑷𝟏𝟏 , 𝑷𝑷𝟐𝟐 } and line 𝒍𝒍 . (b) 𝝎𝝎𝟐𝟐 is determined as minimizes the summed kinetic energiesof the two rollers.
variable rotational axes) configurations(indicated in
Figure 2 (a)(b)).

𝝎𝝎𝟐𝟐 that minimizes the summed kinetic energies of thetwo rollers, and calculate the minimum energy (𝐸𝐸 = 𝐸𝐸2 ).
To this end, we first express 𝝎𝝎 as the sum of 𝝎𝝎́ and𝜔𝜔𝑧𝑧 𝒆𝒆𝟑𝟑 .

(A) Case of Type- I
As shown in Figure 2 (a), the end point of 𝝎𝝎𝟏𝟏 canbe determined as the intersection of 𝑙𝑙 (the line
perpendicular to the horizontal plane passing throughend point 𝝎𝝎́) and 𝐬𝐬𝐬𝐬𝐬𝐬𝐬𝐬{𝑷𝑷𝟏𝟏 , 𝑷𝑷𝟐𝟐 }. Therefore, The angleof sphere rotational axis is determined as 𝜌𝜌 = 𝜌𝜌1 .
(𝑷𝑷𝟏𝟏 × 𝑷𝑷𝟐𝟐 )𝑥𝑥 sin 𝜑𝜑 − (𝑷𝑷𝟏𝟏 × 𝑷𝑷𝟐𝟐 )𝑦𝑦 cos 𝜑𝜑𝜌𝜌1 = tan−1 ��(𝑷𝑷𝟏𝟏 × 𝑷𝑷𝟐𝟐 )𝑧𝑧

Substituting Eqs.(5) into Eqs.(3), we obtain 𝝎𝝎 = 𝝎𝝎𝟏𝟏
as Eqs.(6).

𝐼𝐼(‖𝝎𝝎𝟏𝟏 × 𝑷𝑷1 ‖2 + ‖𝝎𝝎𝟏𝟏 × 𝑷𝑷2 ‖2 )
𝑅𝑅2

(10)

‖𝝎𝝎 × 𝑷𝑷𝑖𝑖 ‖2(11)= 𝜔𝜔𝑧𝑧2 ‖𝒆𝒆𝟑𝟑 × 𝑷𝑷𝑖𝑖 ‖2 + 2𝜔𝜔𝑧𝑧 〈𝝎𝝎́ × 𝑷𝑷𝑖𝑖 , 𝒆𝒆𝟑𝟑 × 𝑷𝑷𝑖𝑖 〉 + ‖𝝎𝝎́ × 𝑷𝑷𝑖𝑖 ‖2

𝑇𝑇

Using, Eqs.(11), it is represented as quadratic function
with respect to 𝜔𝜔𝑧𝑧 as Eqs.(12)Substituting Eqs.(6) into Eqs.(4)， we obtain 𝐸𝐸 = 𝐸𝐸1
as Eqs.(7).
𝐸𝐸1 =𝝎𝝎 × 𝑷𝑷𝑖𝑖 = (𝝎𝝎́ + 𝜔𝜔𝑧𝑧 𝒆𝒆𝟑𝟑 ) × 𝑷𝑷𝑖𝑖

= 〈𝝎𝝎́ × 𝑷𝑷𝑖𝑖 + 𝜔𝜔𝑧𝑧 (𝒆𝒆𝟑𝟑 × 𝑷𝑷𝑖𝑖 ) , 𝝎𝝎́ × 𝑷𝑷𝑖𝑖 + 𝜔𝜔𝑧𝑧 (𝒆𝒆𝟑𝟑 × 𝑷𝑷𝑖𝑖 )〉(6)(𝑷𝑷𝟏𝟏 × 𝑷𝑷𝟐𝟐 )𝑥𝑥 sin 𝜑𝜑 − (𝑷𝑷𝟏𝟏 × 𝑷𝑷𝟐𝟐 )𝑦𝑦 cos 𝜑𝜑‖𝑽𝑽‖�−sin 𝜑𝜑 , cos 𝜑𝜑 ,�(𝑷𝑷𝟏𝟏 × 𝑷𝑷𝟐𝟐 )𝑧𝑧
𝑟𝑟

(9)

And.

𝒆𝒆𝟑𝟑 = [0, 0, 1]𝑇𝑇 , 𝝎𝝎́ = [𝜔𝜔𝑥𝑥 , 𝜔𝜔𝑥𝑥 , 0]𝑇𝑇= 𝝎𝝎́ × 𝑷𝑷𝑖𝑖 + 𝜔𝜔𝑧𝑧 (𝒆𝒆𝟑𝟑 × 𝑷𝑷𝑖𝑖 )(5)𝝎𝝎𝟏𝟏 =(8)where𝝎𝝎 = 𝝎𝝎́ + 𝜔𝜔𝑧𝑧 𝒆𝒆𝟑𝟑

‖𝝎𝝎 × 𝑷𝑷𝑖𝑖 ‖2 + ‖𝝎𝝎 × 𝑷𝑷𝑖𝑖 ‖2 =(12)(‖𝒆𝒆𝟑𝟑 × 𝑷𝑷1 ‖2 + ‖𝒆𝒆𝟑𝟑 × 𝑷𝑷2 ‖2 )𝜔𝜔𝑧𝑧2(7)+ 2(〈𝝎𝝎́ × 𝑷𝑷1 , 𝒆𝒆𝟑𝟑 × 𝑷𝑷1 〉 + 〈𝝎𝝎́ × 𝑷𝑷2 , 𝒆𝒆𝟑𝟑 × 𝑷𝑷2 〉)𝜔𝜔𝑧𝑧

(B) Case of Type-Ⅱ+ ‖𝝎𝝎́ × 𝑷𝑷1 ‖2 + ‖𝝎𝝎́ × 𝑷𝑷2 ‖2© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

268

Kenji Kimura, Kazuo Ishii

Hence, focusing the coefficients of Eqs.(12), 𝐸𝐸 takes
minimal value 𝐸𝐸2 as Eqs.(13).
𝐸𝐸2 = (‖𝝎𝝎́ × 𝑷𝑷1 ‖2 +‖𝝎𝝎́ × 𝑷𝑷2 ‖2 )
−

where

𝐼𝐼

𝑅𝑅 2

(13)

(〈𝝎𝝎́ × 𝑷𝑷1 , 𝒆𝒆𝟑𝟑 × 𝑷𝑷1 〉 + 〈𝝎𝝎́ × 𝑷𝑷2 , 𝒆𝒆𝟑𝟑 × 𝑷𝑷2 〉)2 𝐼𝐼‖𝒆𝒆𝟑𝟑 × 𝑷𝑷1 ‖2 + ‖𝒆𝒆𝟑𝟑 × 𝑷𝑷2 ‖2
𝑅𝑅2𝜔𝜔𝑧𝑧 = −〈𝝎𝝎́ × 𝑷𝑷1 , 𝒆𝒆𝟑𝟑 × 𝑷𝑷1 〉 + 〈𝝎𝝎́ × 𝑷𝑷2 , 𝒆𝒆𝟑𝟑 × 𝑷𝑷2 〉(14)‖𝒆𝒆𝟑𝟑 × 𝑷𝑷1 ‖2 + ‖𝒆𝒆𝟑𝟑 × 𝑷𝑷2 ‖2𝜌𝜌2 = tan−1 �−(a)〈𝝎𝝎́ × 𝑷𝑷1 , 𝒆𝒆𝟑𝟑 × 𝑷𝑷1 〉 + 〈𝝎𝝎́ × 𝑷𝑷2 , 𝒆𝒆𝟑𝟑 × 𝑷𝑷2 〉�‖𝒆𝒆𝟑𝟑 × 𝑷𝑷1 ‖2 + ‖𝒆𝒆𝟑𝟑 × 𝑷𝑷2 ‖2(15)

Substituting Eqs.(15) into Eqs.(3), 𝝎𝝎𝟐𝟐 as Eqs.(16).
𝝎𝝎𝟐𝟐 =(16)𝑇𝑇〈𝝎𝝎́ × 𝑷𝑷1 , 𝒆𝒆𝟑𝟑 × 𝑷𝑷1 〉 + 〈𝝎𝝎́ × 𝑷𝑷2 , 𝒆𝒆𝟑𝟑 × 𝑷𝑷2 〉‖𝑽𝑽‖
�−sin 𝜑𝜑 , cos 𝜑𝜑 , −�‖𝒆𝒆𝟑𝟑 × 𝑷𝑷1 ‖2 + ‖𝒆𝒆𝟑𝟑 × 𝑷𝑷2 ‖2
𝑟𝑟

3.

(b)
Figure 3 Simulation result. (a) Trajectory of angular velocity
vector of sphere. (b) Totally kinetic energy

Simulation

This section presents the simulation results 𝝎𝝎1 (SeeEqs.(6)), 𝝎𝝎2 (See Eqs.(16)), 𝐸𝐸1 (See Eqs.(7)) and𝐸𝐸2 (See Eqs.(13)) with parameter 𝜑𝜑 (0° ≤ 𝜑𝜑 < 360°) inthe given sphere mobile speed: ‖𝑽𝑽‖ = 1 [m/s]. The
conditions are as follows: 𝐼𝐼 = 1 , 𝑅𝑅 = 00.1[m] , ‖𝑽𝑽‖ =1 [m/s] , 𝑟𝑟 = 0.1[m], 𝜃𝜃1,1 = 215°, 𝜃𝜃1,2 = 60°, 𝜃𝜃2,1 =
325°, 𝜃𝜃2,2 = 60 °.

As shown in Figure 3 (b), 𝐸𝐸1 is minimized at spheredirection
angles
𝜑𝜑 =
90°
and
270°,
and maximized at 𝜑𝜑 = 0° and 180°. Meanwhile, 𝐸𝐸2 isminimized at 𝜑𝜑 = 0° and 180°, and maximized at 𝜑𝜑 =90° and 270° . And. 𝐸𝐸1 and 𝐸𝐸2 are same value at 𝜑𝜑 =90° and 270°.

4. Conclusion

3.1 Trajectory of the end point of the angler velocity
vector and Totally kinetic energy
As shown in Figure 3 (a), the ellipsoid trajectory of𝝎𝝎2 lies nearer to the 𝑥𝑥𝑥𝑥-plane than that of 𝝎𝝎1, andboth trajectories cross a common line parallel to the
𝑦𝑦-axis.

We optimized the total kinetic energy of the two rollers’
movement and calculated the angle of the rotational axisof the sphere, and theoretical formula and we present asimulation of total kinetic energy.
In future work, we plan to evaluate the kinetic energyof the roller arrangement at an arbitrary contact point on
the upper hemisphere as an evaluation function.

References[1]

M.Wada, K.Kato, "Kinematic modeling and simulationof active-caster robotic drive with a ball transmission(ACROBAT-S)".
2016
IEEE/RSJInternational© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

269

The Spherical Robot Transfer

[2]

[3]

[4]
[5]

[6]

[7]

Conference on Intelligent Robots and Systems. Daejeon,
2016-12-9/25, IEEE Robotics and Automation Society,pp. 4455-4460, 2016.
S.Ishida，H.Miyamoto,“Holonomic Omnidirectional
Vehicle with Ball Wheel Drive Mechanism, 2012The
JapanSocietyofMechanicalEngineers.Vol.78,No.790,
pp.2162-2170, 2012.
K. Kimura, K. Ishii, Y. Takemura, M. Yamamoto,
Mathematical Modeling and motion analysis of the wheel
based ball retaining mechanism, SCIS & ISIS, pp.41064111, 2016.
S. Chikushi, M. Kuwada, et al., Development of NextGeneration Soccer Robot”Musashi150”for RoboCup
MSL,30th Fussy System Symposium, pp. 624-627,2014
Lee,Y.C,Danny,Lee,D.V.,Chung,J.,andVelinky,S.A.,"Co
ntrol of a redundant, reconfigurable ball wheel drive
mechanism for an omunidirectional platform",Robotica,
Cambridge University Press,Vol.25, pp.385-395, 2007.
M. Kumagai, T. Ochiai: “Development of a robot
balanced on a ball – Application of passive motion totransport”,Proc. ICRA IEEE(2009), pp. 4106-4111,
2009.
Endo, Tatsuro. et al. "An omnidirectional vehicle on abasketball". 12th International Conference on Advanced
Robotics. Seattle, 2005-07-18/20, IEEE Robotics andAutomation Society, pp. 573-578, 2005.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

270

Acoustic Impedance Measurement through the Modelling of Ultrasonic Wave Transmission *Ryuugo Mochizuki

Center for Socio-Robotic Synthesis, Kyushu Institute of Technology, 2-4 Hibikino Wakamatsuku
Kitakyushu, 808-0196/Fukuoka, Japan †Yuya Nishida, Kazuo Ishii

Department of Human Intelligence Systems, Kyushu Institute of Technology, 2-4 Hibikino Wakamatsuku
Kitakyushu,808-0196/Fukuoka, Japan
E-mail: mochizuki.ryuugo126@mail.kyutech.jp, y-nishida@brain.kyutech.ac.jp,ishii@brain.kyutech.ac.jp
www.lsse.kyutech.ac.jp

Abstract
In food industry, shortage of workers is a serious problem. Automation of food handling is a critical nowadays. To
alleviate the damage during food picking by robotic hand, we propose non-contact acoustic impedance estimation
with ultrasonic wave. We have the assumption of the correlation between hardness and acoustic impedance, and, built
up ultrasonic transmission model considering attenuation by reflection and absorption, then, made an experiment toestimate the impedance. As the result, we succeeded in detecting acoustic impedance without contact.
Keywords: Acoustic Impedance, Ultrasonic Transmission, Hardness, Reflection, Absorptiondifficulty of acoustic transmission in a material, and canbe estimated from reflection coefficient4. Confirming therelation of acoustic impedance and hardness realizes noncontact hardness measurement system.

1. Introduction
In Japan, the demand for prepared food has been
increasing recently. The survey1 says the expenditure forprepared food has been increasing since 2004 for all
human ages. The rate of the increasing expenditure is283.9[%] for the internet shopping, 28.0[%] forconvenience stores. In domestic food industry, a few
millions of lunch boxes are produced and consumed aday. Food packing stage requires human resources
generally2. For alleviation of labor burden by human,
automation of food packing is effective. however,picking food by solid hand causes of damages3. Hardnessof food should be measured without contact, then,picking force should be optimized for the automation.
We assumed relation between acoustic impedance andhardness, then estimated the impedance through analysis
of ultrasonic reflection. The impedance shows the

2. Related Works
J. Machando et.al introduced ultrasonic wave into NonDestructive Analysis for evaluation of wood strength5.
B. Cho and J. Irudayaraj measured cheese depth withultrasonic wave and accuracy of the measurement was
99.98[%]. Moreover, they recorded more than 0.9 ofcorrelation between cheese mechanic property and resultof sonic velocity measurement6. S. Srivastava et.al
introduced ultrasonic measurement for hardness
estimation of tomatoes, and succeeded in detecting
hardness changes during the growth7.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

271

Ryuugo Mochizuki, Yuya Nishida, Kazuo Ishii

3.2.2. Attenuation by Reflection

However, to our knowledge, no report has been foundas measuring hardness unevenness. The unevenness
should be considered because uniqueness of heating foodis not guaranteed under cooking.

As Fig. 2 shows, samples with uneven hardness is
assumed to have multiple media of different ζk, thus,
reflection occurs at a boundary where difference of ζk isobserved. As Eq. (2) shows4, small difference of ζkcauses of the attenuation by reflection. The acousticpressure is attenuated to rk times of Pi,k(zk) after thereflection.

3. Proposed Method
3.1. Outline
Fig.1 shows the process of our acoustic impedance
estimation. In the system, receiver and transmitter are
attached parallel, which enables observation of reflected
wave. Here, the sample is assumed as a composition ofmultiple media with different acoustic impedances ζk[Pas/m] (k shows index of media), therefore, reflection
occurs at zk where ζk changes. After the transmission, we
calculate amplitudes of sound pressure Pr,k(z0) in thereflected wave by maxima detection (Here, z0 denotes
observation point.), to easily evaluate the attenuation
between reflections. Then, reflection coefficient rk, andattenuation coefficient αk, finally ζk is estimated from theresult of rk estimation. Evaluation of the correlation
between ζk and hardness is our future work.

𝜁𝜁

𝑟𝑟𝑘𝑘 = 𝑘𝑘−1−𝜁𝜁𝑘𝑘𝜁𝜁𝑘𝑘−1 +𝜁𝜁𝑘𝑘=

𝑃𝑃𝑟𝑟,𝑘𝑘 (𝑧𝑧𝑘𝑘 )𝑃𝑃𝑖𝑖,𝑘𝑘 (𝑧𝑧𝑘𝑘 ).

(2)Fig. 2 Acoustic Transmission and Reflection in MediaThe reflection coefficient of k-1 from k should beconsidered by changing the numerator of Eq. (2) into ζk– ζk-1. As the result, the coefficient is expressed in –rk.
3.2.3. Attenuation by Absorption
The attenuation is caused by viscosity of media and heattransmission in the event of longitudinal wave
penetration8, whose magnitude is equal to attenuation
coefficient α [Np/m]8-9. Supposing P0[Pa] is original
acoustic pressure, and, the wave moves z [m], thepressure P diminishes as Eq. (3) shows.
𝑃𝑃 = 𝑃𝑃0 𝑒𝑒 −𝛼𝛼𝛼𝛼 .

Fig. 1 Process of Acoustic Impedance Calculation

3.2. Acoustic Transmission Model

We disregard wave diffusion10 because the angle ofwave expansion is enough small as our instrument.

3.2.1. Condition at media boundary

3.2.4. Integration of Reflection and Absorption

In view of acoustic pressure sequence at media boundary
z=zk, the relation among incident wave (Amplitude:
Pi,k(zk)[Pa]), reflected wave (Amplitude: Pr,k(zk)[Pa]),
transmitted wave (Amplitude: Pt,k(zk)[Pa]) is shown as
Eq. (1). Here, wave frequency is ω[rad/s], sonic velocity
is ck-1[m/s], ck[m/s] each other in media k-1 and k.
𝑃𝑃𝑖𝑖,𝑘𝑘 (𝑧𝑧𝑘𝑘 )𝑒𝑒𝑧𝑧−𝑧𝑧𝑘𝑘�
𝑘𝑘−1

𝑗𝑗𝑗𝑗�𝑡𝑡−𝑐𝑐+ 𝑃𝑃𝑟𝑟,𝑘𝑘 (𝑧𝑧𝑘𝑘 )𝑒𝑒𝑧𝑧−𝑧𝑧𝑘𝑘�
𝑘𝑘−1

𝑗𝑗�𝑡𝑡+𝑐𝑐= 𝑃𝑃𝑡𝑡,𝑘𝑘 (𝑧𝑧𝑘𝑘 )𝑒𝑒

(3)

𝑧𝑧−𝑧𝑧

𝑗𝑗𝑗𝑗�𝑡𝑡− 𝑐𝑐 𝑘𝑘 �
𝑘𝑘As Fig. 2 shows, when P0[Pa] is emitted at z=z0[m] of
media 0, penetration and reflection occurs at eachboundary zk. During the transmission in media k,absorption attenuation occurs in αk[Np/m]. By the arrival
at media k (z=zk), totally, k absorptions and k-1
reflections occur. The state of incident wave (Amplitude:
Pi,k(zk)) into media k is expressed by Eq. (4).

. (1)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

272

Acoustic Impedance Measurement through−𝛼𝛼𝑘𝑘′−1 �𝑧𝑧𝑘𝑘′ −𝑧𝑧𝑘𝑘′−1 � (𝑟𝑟𝑗𝑗𝑗𝑗𝑗𝑗𝑃𝑃𝑖𝑖,𝑘𝑘 (𝑧𝑧𝑘𝑘 )𝑒𝑒 𝑗𝑗𝑗𝑗𝑗𝑗 = �𝑃𝑃0𝑒𝑒 −𝛼𝛼𝑘𝑘−1 (𝑧𝑧𝑘𝑘 −𝑧𝑧𝑘𝑘−1 ) ∏𝑘𝑘−1
𝑘𝑘′ + 1)� 𝑒𝑒
𝑘𝑘 ′ =1 𝑒𝑒(4)

𝑃𝑃𝑟𝑟,𝑘𝑘+1 (𝑧𝑧0 )Immediately after the reflection, the acoustic pressureof reflected wave is Pr,k(zk) [Pa]. By the arrival at thereceiver (z=z0), totally, k absorptions and k-1 reflections
occur again. Pr,k(zk) is diminished to Pr,k(z0). This state isexpressed in Eq. (5). All signs of all rk s are opposite toEq. (4). Eq. (4)(5) is summarized to Eq. (6)(7).
−𝛼𝛼𝑘𝑘′−1 �𝑧𝑧𝑘𝑘′ −𝑧𝑧𝑘𝑘′−1 � (−𝑟𝑟𝑗𝑗𝑗𝑗𝑗𝑗𝑃𝑃𝑟𝑟,𝑘𝑘 (𝑧𝑧0 )𝑒𝑒 𝑗𝑗𝑗𝑗𝑗𝑗 = �𝑃𝑃𝑟𝑟,𝑘𝑘 (𝑧𝑧𝑘𝑘 )𝑒𝑒 −𝛼𝛼𝑘𝑘−1 (𝑧𝑧𝑘𝑘 −𝑧𝑧𝑘𝑘−1 ) ∏𝑘𝑘−1.
𝑘𝑘′ + 1)�𝑒𝑒
𝑘𝑘 ′ =1 𝑒𝑒
2 −2𝛼𝛼𝑘𝑘′−1 �𝑧𝑧𝑘𝑘′ −𝑧𝑧𝑘𝑘′−1 � 𝑗𝑗𝑗𝑗𝑗𝑗
𝑃𝑃𝑟𝑟,𝑘𝑘 (𝑧𝑧0 )𝑒𝑒 𝑗𝑗𝑗𝑗𝑗𝑗 = �𝑟𝑟𝑘𝑘 𝑃𝑃0 𝑒𝑒 −2𝛼𝛼𝑘𝑘−1 (𝑧𝑧𝑘𝑘 −𝑧𝑧𝑘𝑘−1 ) ∏𝑘𝑘−1
�𝑒𝑒 .
𝑘𝑘 ′ =1(1 − 𝑟𝑟𝑘𝑘′ )𝑒𝑒

𝑃𝑃𝑟𝑟,𝑘𝑘 (𝑧𝑧0 )𝑃𝑃𝑟𝑟,𝑘𝑘−1 (𝑧𝑧0 )3.3. Solution= (1 − 𝑟𝑟𝑘𝑘−1 2 )𝑟𝑟𝑘𝑘

𝑟𝑟𝑘𝑘−1𝑒𝑒 −2𝛼𝛼𝑘𝑘−1(𝑧𝑧𝑘𝑘−𝑧𝑧𝑘𝑘−1) .

0 < 𝑟𝑟𝑘𝑘 <

(5)
(6)
(7)

In this experiment, we confirmed possibility of finding(Fig. 3), transmitter and receiver were attached parallel.
Sponges were prepared to remove echoes. The distance
between transmitter and sample surface is Lss[mm]. As
the setting, Lss=100[mm], frequency ω=800π[krad],sample width Ws = 100[mm], depth Hs=10[mm] (Weselected ABS). We emitted ultrasonic wave and analyzed
reflected wave by Monte Carlo Method generating
random values of rk within the range Eq. (9), (11), andobtained αk-1 following Eq. (8). Then we averaged rk andαk-1 for the same k. Acoustic impedance ζ0 is known to4.1x102[Pa s/m]11, and, true ζk of ABS is 2.4x106[Pas/m]12. Nk , Nrd (=500) is the number of amplitudes, rk
subsets (Each subset includes r1, r2, .., rNk-1) each other.
Because Hs is sufficiently thin, we ignored the differenceof sonic velocity c in each layer. We calculated distancefrom z0 to zk by Eq. (12). tk is the time to receive Pr,k(z0).

� . (8)Here, αk-1 ≥ 0 should be satisfied. αk-1<0 means
amplification, which leads a contradiction against
attenuation. We treat only positive for rk, then, necessary
requirement of rk is stated in Eq. (9) for αk-1≥ 0.
𝑃𝑃𝑟𝑟,𝑘𝑘 (𝑧𝑧0 )𝑟𝑟𝑘𝑘−12 �
𝑃𝑃𝑟𝑟,𝑘𝑘−1 (𝑧𝑧0 ) �1−𝑟𝑟𝑘𝑘−1≤ 𝑟𝑟𝑘𝑘 < 1 .

(11)

ζk through reflected wave analysis. In the environment

𝑟𝑟𝑘𝑘 𝑃𝑃𝑟𝑟,𝑘𝑘−1 (𝑧𝑧0 )𝑟𝑟𝑘𝑘−1 𝑃𝑃𝑟𝑟,𝑘𝑘 (𝑧𝑧0 ).

4.1. Method

Though all Pr,k(z0) are solved, unknown αk-1, rk-1, rk areincluded in one equation (Eq. (7)). We introduce Monte
Carlo Method to find the solutions. The relationshipbetween αk-1, rk-1, rk is expressed in Eq. (8)From the Natural Logarithm of Eq. (7).
2 )ln �(1 − 𝑟𝑟𝑘𝑘−1

2𝑃𝑃𝑟𝑟,𝑘𝑘 (𝑧𝑧𝑜𝑜)

4. Experiment3.3.2. Coefficient αk and rk

1−𝑃𝑃𝑟𝑟,𝑘𝑘+1 (𝑧𝑧0 )+�𝑃𝑃𝑟𝑟,𝑘𝑘+1 (𝑧𝑧0 )2 +4𝑃𝑃𝑟𝑟,𝑘𝑘 (𝑧𝑧0 )2

rk=0, 1 should be excluded so that antilogarithm of Eq.
(8) is one or more. The upper limit in Eq. (11) is
considered so as not to make contradiction for rk+1.
Finally, necessary requirement of rk is defined as theoverlapped range of Eq. (9) and (11).

To obtain the Amplitudes, we introduce Gradientmethod. From intersections with z-axis, the reflection
climbs up then descends. In our condition, Large gradient(more than Tε ) should be observed at the intersections
where maxima searching starts. the derivative ofreflection should be Tε or less, and the reflection should
be Tp or more where candidates of Pr,k(z0) locate.

2(𝑧𝑧𝑘𝑘 −𝑧𝑧𝑘𝑘−1 )

(10)

< 1.

As the result of Eq. (10), (11) is obtained as a solution.

3.3.1. Amplitude of Reflected Wave Pr,k(z0)𝛼𝛼𝑘𝑘−1 =𝑟𝑟𝑘𝑘

𝑃𝑃𝑟𝑟,𝑘𝑘 (𝑧𝑧0 ) �1−𝑟𝑟𝑘𝑘2 �1𝑧𝑧𝑘𝑘 − 𝑧𝑧0 = 𝑐𝑐(𝑡𝑡𝑘𝑘 − 𝑡𝑡0 ) .

4.2. Result and Discussion

2

(12)

Fig. 4 shows the results of measurement. (a) shows
vr(z0), that is correspondent of reflected wave in voltage,
where amplitude is shown in orange. (b)(c)(d) shows
average of rk, αk-1 ,ζk, each other. In Fig. 5, the same
averages ((a)rk, (b)αk-1 , (c)ζk,) are expressed in blue bar
and the standard deviations are red. ζk = 2.5x106[Pa s/m]
(at zk=107.5[mm]) was the closest from the truth.

(9)

According to Eq. (9), left side of the inequality exceeds
one depending on the combination of Pr,k(z0) and rk. To
solve the contradiction, Eq. (10) should be satisfied.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

273

Ryuugo Mochizuki, Yuya Nishida, Kazuo Ishii

5. Conclusion

According to Fig. 5, 6, vr(z), rk and ζk rose, while αk-1
decreased in deeper layer of the sample. From the point,
rk also naturally increases as vr(z) rises. However, αk-1 isparticularly large near the surface. The deviation of αk-1
is outstanding in Fig. 5(b). We estimated r1 and α1
without Po, then, the accuracy of ζ1 cannot be guaranteed.
rk and ζk may change if truly observed value of P0 is used
for the solution. Eq. (2) can be transformed into Eq. (13)𝜁𝜁𝑘𝑘 =

1+𝑟𝑟𝑘𝑘𝜁𝜁
.
1−𝑟𝑟𝑘𝑘 𝑘𝑘−1

In this research, we proposed acoustic impedance
measurement method with ultrasonic wave. As the result,reasonable solutions of reflection coefficients wereobtained in deeper layer. then accuracy of acoustic
impedance estimation was 0.1[Pa s/m] error for
2.4x106[Pa s/m] of true ABS acoustic impedance.
However, as near the surface, the solution may change
according to setting of emitted acoustic pressure.
Enhancing reliability in reflection coefficient estimationis necessary for better accuracy of acoustic impedance.

(13)

Thus, ζk dramatically changes if rk is near 1.0. Enhancing
the accuracy of rk is essential for higher accuracy of ζk.

References

1. YanoResearch
Institute
HP(In
Japanese)https://www.yano.co.jp/press-release/show/press_id/2245
2. Ministry of Agriculture, Forestry and Fisheries, Summaryof the Annual Report on Food, Agriculture and Rural
Areas in Japan, MAFF, FY2017, 2017, pp. 17-18.
3. Z. Wang, M. Zhu, S. Kawamura and S. Hirai, Comparisonof Different Soft Grippers for Lunch Box Packaging,
Robio, 4:10, 2017, pp, 1-9.
4. Marine Acoustic Society of Japan, Kaiyo Onkyo no Kiso
Fig. 3 Environment for the Experiment
to Oyo, (in Japanese), Seizando-Shoten Publishing, 2004.
vr(z)r
5. J. Machado, P. Palma, S. Simoes, Ultrasonic Indirect
Method for Evaluating Clear Wood Strength and Stiffness,
Non-Destructive Testing in Civil Engineering, 2009, pp.
1-6.
z[mm] 6. B. Cho and J. Irudayaraj, A Noncontact Ultrasoundz[mm](a)Reflected Wave(b)Reflection CoefficientApproach for Mechanical Property Determination ofζkαk-1
Cheeses, J. Food Sci., 68(7), 2003, pp. 2243-2247.
7. S. Srivastava, S. Vaddadi and S. Sadistap, Non-Contact
Ultrasonic Based Stiffness Evaluation System forTomatoes during Shelf-Life Storage, Int. J. Food Sci. Nutr.,
z[mm]
z[mm]
4(3), 2014, pp 1-6.
(d)Acoustic Impedance(c)Absorption CoefficientFig. 4 Result of Experiment
8. The Institute of Electronics, Information and Communication Engineers, Chishiki Base (In Japanese), 1(10),Chapter 2, 2014.
9. J. Carlson, J. Deventer, A. Scolan and C. Carlander,
Frequency and Temperature Dependence of Acousticz[mm]Properties of Polymers Used in Pulse-Echo Systems,(a)Reflection Coefficient
Proceedings in IEEE Symposium on Ultrasonics, 2003, pp.
885-888.
10.
Y. Yokono, Measurement of Ultrasonic Attenuation inz[mm]Materials, (In Japanese) Keisoku Series, Japan Welding(b)Absorption Coefficient
Society, 62(7), 1993, pp. 522-527.
11. S. Takeuchi, Hajimete no Suichu Cho-onpa Tranceducer,(In Japanese) J. Acoust. Soc. of Japan, 72(5), 2016, pp.
z[mm]264-272.
(c)Acoustic Impedance
12. D. Serrano, A. Uris, S, Ibáñez, C. Rubio, MRI Compativle
Fig. 5 Averages(Blue) and Standard Deviation (Red)Planar Material Acoustic Lenses, Appl. Sci., 8(12), 2018,
pp. 1-9.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

274

Human-Computer Communication Using Facial Expression
Yasunari Yoshitomi

Graduate School of Life and Environmental Sciences, Kyoto Prefectural University,
1-5 Nakaragi-cho, Shimogamo, Sakyo-ku, Kyoto 606-8522, Japan
E-mail: yoshitomi@kpu.ac.jp
http://www2.kpu.ac.jp/ningen/infsys/English_index.html

Abstract
To develop a complex computer system such as a robot that can communicate smoothly with humans, it isnecessary to equip the system with a function for both understanding human emotions and expressing emotionalsignals. From both perspectives, facial expression is a promising research area. In our research, we have exploredboth aspects of facial expression using infrared-ray images and visible-ray images and have developed a personified
agent for expressing emotional signals to humans.
Keywords: Emotion, Facial expression recognition, Infrared-ray image, Facial expression synthesis, Personified agent.

1. Introduction

2. Facial Expression Recognition

The goal of our study is to present a paradigm whereby
a complex computer system such as a robot cancooperate smoothly with humans. To do this, thecomputer system must have the ability to communicate
with humans using some form(s) of informationtransmission. Such a system must be equipped with afunction for both understanding human emotions andexpressing emotional signals to its human counterparts.
In this regard, facial expression is a promising target for
research. Accordingly, we have been investigating bothaspects of facial expression.
In this paper, we describe the challenges ofreaching our goal. The remainder of the paper isorganized as follows: Section 2 summarizes our studies
on facial expression recognition; Section 3 briefly
describes our studies on human-computer-humancommunication via the Internet; Section 4 outlines ourstudies on human-computer communication; Section 5
discusses our work on integration with speech; Section
6 concludes the paper.

2.1. Infrared-ray image utilization
We have developed a method for recognizing facial
expressions using thermal image processing.1 In thisstudy, infrared-ray was used. Figure 1 shows theinfluence of lighting at night on a facial image using
both visible-ray ((a),(c)) and infrared-ray ((b),(d)). As isevident in the figure, the visible-ray image is strongly
influenced by lighting conditions, while the thermal
image is unaffected. With our method, neutral, happy,
surprised, and sad facial expressions were recognizedwith 90% accuracy.1 Figure 2 shows examples.
2.2. Sensor fusion
Sensor fusion is a promising way to improve therecognition accuracy of facial expression or emotion
recognition. Several studies2,3 to improve accuracy
using sensor fusion have produced promising results.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

275

Yasunari Yoshitomi

3. Human-computer-human Communication(a)Social network services (SNSs) have become extremely
popular as communication tools on the Internet.
However, while it is possible to post a message, a static
image, or a moving image on a platform such as Twitter,
it is difficult to communicate the actual emotions felt
when writing a message or posting an image. We
believe that a support system is needed to facilitatesmoother communication between humans in their useof SNSs. Not having immediate and direct contact with
one another risks misunderstanding, especially from anemotional point of view.
One of our studies is aimed at expressing the realemotions of individuals writing messages for posting on
an SNS site by analyzing their facial expressions andvisualizing them as pictographs. To this end, we havedeveloped a real-time system for expressing emotion asa pictograph selected according to the writer’s facial
expression while writing a message.8,9 We applied thesystem to the posting on Twitter of both a message anda pictograph.8,9The lower panels in Figure 3 show the output of
our system in a situation where the subject was asked tointentionally show two types of emotions—neutral orsmiling—when writing the message, '明日は情報伝達システム学サブゼミに参加します。時間は 5 時限
目、場所は先生の部屋です。' (in Japanese), whichmeans, “I will attend the discussion section held at the
professor’s room in the information communication
system lab in fifth period tomorrow.” 8

(b)

(c)
(d)
Fig. 1. Examples of face-image at night; (a) visible-ray image
with lighting, (b) infrared-ray image with lighting, (c) visibleray image without lighting, (b) infrared-ray image without
lighting.1

(a)

(b)

(c)
(d)
Fig.2. Examples of facial expressions; (a) neutral, (b) happy,(c) surprised, (d) sad.1

2.3. Analysis for medical use
Facial expression analysis using infrared-ray images4 orvisible-images5-7 has been successfully conducted withthe goal of identifying subjects suffering from pre-stage
dementia or other medical problems.
Fig. 3. Snap-shots (upper) of posting on Twitter; messages
and pictographs (lower) posted on Twitter (left: neutral; right:
smiling).8© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

276

Human-computer Communication Using Facial

4. Human-computer Communication

4.2. Human-computer communication in music
recommendation

4.1. Personified agent

The music recommendation module of the proposedsystem14 is based on a previously proposed system15 thatuses collaborative filtering and impression words (see thepaper15 for details of the music recommendation module).
In the music-recommendation process, all user
navigations are performed by the synthetic voice of theagent appearing on the PC screen facing the user. All
dialogue spoken by the agent is situationally selected by
the proposed system14. The user’s answers to thequestions generated by the agent are recognized using thevoice recognition function of the system, and the agent
motions, including facial expressions, are then generated.
Figure 5 shows two snapshots of the reaction of theagent after recognizing (a) a positive answer, i.e., the user
wishes to listen to the recommended song again in the
future, and (b) a negative answer, i.e., the user does notwish to listen to the recommended song in the future. In thecase of (a), the agent nods twice and raises the corners ofthe mouth slightly, while in the case of (b), the agent also
nods twice, but lowers the corners of the mouth slightly.
Figure 6 shows a snapshot of the music recommendation
being performed by the proposed system14.

The process of agent generation in our system10 consistsof six steps: (1) creating facial expression data, (2)recording vocal utterances, (3) automatic WAVE filedivision, (4) speech recognition by Julius11, (5) insertionof expressionless data, and (6) the creation of facial
expression motion.
Expressive motions are generated by combining
the expression data of each vowel for each utterance
motion. Then, the utterance contents are input as text
and used by the MikuMikuDanceAgent (MMDAgent),12
which is a freeware animation program that allows users
to create and animate movies with agents, to output
synthesized voice that is then recorded by a stereo mixer
inside a PC and saved as a WAVE file. Speech isrecognized using a speech recognition system called
Julius,11 followed by facial expression synthesis of theagent using preset parameters depending on each vowel.
Facialexpression
data
were
createdwith
MikuMikuDance13. In this study, in order to generate
more human-like agent facial expressions, facial
expression data were created for the vowels / a /, / i /, / u/, / e /, and / o / (Fig. 4).10 In order to create more natural
agent facial expressions, processing is then performed toinsert a neutral facial expression when the same vowel,
for example / a /, is continuous.10

(a)
/a

/i/

/e

(b)

Fig. 5. Snapshots of the reaction of the agent after recognizing(a) a positive answer, and (b) a negative answer. 14

/u/

/o/

Fig. 4. Facial expression of the agent when uttering eachvowel.10

Fig. 6. Snapshot of performing song-recommendation by the
proposed system.14© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

277

Yasunari Yoshitomi

5. Integration with Speech

4. Y. Yoshitomi, T. Asada, R. Kato, and M.T abuse, Methodof facial expression analysis using video phone and thermal
image, J. Robotics, Networking and Artif. Life 1(1) (2014) 7-11.
5. T. Asada, Y. Yoshitomi, R. Kato, M. Tabuse, and J. Narumoto,
Quantitative evaluation of facial expressions andmovements of persons while using video phone, J.
Robotics, Networking and Artif. Life 2(2) (2015) 111-114.
6. T. Asada, Y. Yoshitomi, A. Tsuji, R. Kato, M. Tabuse, N.
Kuwahara, and J.Narumoto, Facial expression analysis
while using video phone, J. Robotics, Networking andArtif. Life 2(4) (2016) 258-262.
7. R. Shimada, T. Asada, Y. Yoshitomi, and M. Tabuse,
Real-time system for horizontal asymmetry analysis on
facial expression and its visualization, J. Robotics,
Networking and Artif. Life 6(1) (2019) 7-11.
8. Y. Yoshitomi, T. Asada, K. Mori, R. Shimada, Y. Yano,and M. Tabuse, Facial expression analysis and its
visualization while writing messages, J. Robotics,
Networking and Artif. Life 5(1) (2018) 37-40.
9. T. Asada, Y. Yano, Y. Yoshitomi, and M. Tabuse,
A system for posting on SNS portrait selected using facial
expression analysis while writing message, J. Robotics,
Networking and Artif. Life 6(3) (2019) 199-202.
10. T. Asada, R. Adachi, S. Takada, Y. Yoshitomi, and M.
Tabuse, Facial expression synthesis using vowel
recognition for synthesized speech, in Proc. 2020 Int.
Conf. on Artificial Life and Robotics, ed. M. Sugisaka(Japan, Beppu, 2020), pp.398–402.
11. Julius, http://Julius.osdn.jp/, Accessed 24 November 2020.
12. MMDAgent, http://www.mmdagent.jp/ Accessed 24
November 2020.
13. MikuMikuDance, https://sites.google.com/view/vpvp/,
Accessed 29 November 2020.
14. A. Matsui, M. Sakurai, T. Asada, and M. Tabuse, Music
recommendation system driven by interaction between
user and personified agent using speech recognition,
synthesized voice and facial expression, in Proc. 2021
Int. Conf. on Artificial Life and Robotics, ed. M.
Sugisaka (Japan, 2021), in press.
15. S. Yoshizaki, Y. Yoshitomi, C. Koro, and T. Asada,Music recommendation hybrid system for improving
recognition ability using collaborative filtering andimpression words, J. Artif. Life and Robotics 18(1-2)(2013) 109–116.
16. Y. Yoshitomi, T. Asada, K. Shimada, and M. Tabuse, Facial
expression recognition of a speaker using vowel
judgment and thermal image processing, J. Artif. Lifeand Robotics 16(3) (2011) 318–323.
17. T. Fujimura, Y. Yoshitomi, T. Asada, and M. Tabuse, Facial
expression recognition of a speaker using front-view facejudgment, vowel judgment and thermal image processing,J. Artif. Life and Robotics 16(3) (2011) 411-417.
18. R. Makino, Y. Yoshitomi, T. Asada, and M. Tabuse, Speechsynthesis of emotions in a sentence using vowel features, J.
Robotics, Networking and Artif. Life 7(2) (2020) 107-110.

Utterance judgment is necessary for deciding the timingof facial expression recognition. Moreover, the mouth
shape with or without an utterance influences facial
expression. In our studies, the first and last vowels in an
utterance such as a name were recognized for deciding
the timing of the facial expression recognition.16,17
Speech recognition and synthesis are indispensable forhuman-computercommunication.
In
particular,
developing the function of emotional speech synthesis18
offers a way to create a paradigm whereby a computer
system such as a robot can work seamlessly with humans.
6. Conclusions
To develop a complex computer system such as a robotthat can communicate smoothly with humans, it isnecessary to equip the system with the ability to bothunderstand human emotion and express emotional signals
to humans. From both points of view, facial expression is apromising research field. In developing a method forrecognizing facial expressions, we have used infrared-ray
images as well as visible-ray images. For expressing
emotional signals to humans, we have developed apersonified agent. Developing the function of emotional
speech synthesis is the next target of our studies.
Acknowledgements
We would like to thank Dr. J. Narumoto, Professor of
the Kyoto Prefectural University of Medicine, Dr. M.
Tabuse, Professor of the Kyoto Prefectural University,and Dr. T. Asada, Associate Professor of the Kyoto
Prefectural University, for their valuable cooperation
during the course of this research.
References
1. Y. Yoshitomi, N. Miyawaki, S. Tomita and S. Kimura,
Facial expression recognition using thermal image processing
and neural network, in Proc. of 6th IEEE Int. Workshop on Robotand Human Communication, (Japan, Sendai, 1997), pp. 380-385.
2. Y. Yoshitomi, S. Kim, T .Kawano and T. Kitazoe, Effectof sensor fusion for recognition of emotional states using
voice, face Image and thermal Image of face, in Proc. of of
9th IEEE Int. Workshop on Robot and Human Interactive
Communication, (Japan, Takamatsu, 2000), pp.178-183.
3. Y. Oka, Y. Yoshitomi, T. Asada, and M. Tabuse, Emotion
recognition of a speaker using facial expression intensityof thermal image and utterance time, J. Robotics,
Networking and Artif. Life 3(3) (2016) 148-151.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

278

Detection of a Fallen Person and Estimation of the Head Position from UAV Images
Haruka Egawa, Seiji Ishiakwa, Joo Kooi Tan

Department of Mechanical and Control Engineering, Kyushu Institute of Technology, Sensui-cho Tobata-ku
Kitakyushu, 804-8550, Japan
E-mail: {egawa-haruka , etheltan }@ss10.cntl.kyutech.ac.jp
http://lab.cntl.kyutech.ac.jp/~etheltan/

Abstract
In order to search for victims in the event of a disaster, we propose a method of detecting a fallen person from UAV
images and estimating its head position. Rotation-invariant HOG and Rotation-invariant LBP are used, so that they
may detected a fallen person regardless of its body orientation. In addition, the position of the head of a fallen person
is estimated using the peak of the gradient histogram. Experimental results show satisfactory performance of the
proposed method.
Keywords: Aerial photographs, Fallen persons, Rotation-invariant HOG, Rotation-invariant LBP, Random Forest,
Gradient histogram.

The proposed method uses Rotation-invariant Histogramof Oriented Gradients (Ri-HOG) features [6] andRotation-invariant Local Binary Pattern (Ri-LBP) as
features that are robust to rotation in order to represent a
fallen person. ) Use the feature quantity [7]. In addition,
Random Forest [8] employed as a classifier. In addition,
the head position is estimated using the peak value of thegradient histogram in the circular region.
The proposed method is examined its performance by theexperiment using bird-eye view videos.

1. Introduction
Japan is one of the world's leading countries ofearthquakes [1] and has been hit by disasters caused byearthquakes. In the event of such a disaster, it is oftenvery difficult to search for the victims manually.
Therefore, recently, attention has been paid on thedevelopment of an automatic search method for thosevictims from aerial images taken by drones.
There are various human detection methods forautomatic search [2,3,4], but most of them are based on
the premise that the target person's body orientation is
unified and the subject is upright [2,3]. Exists. However,there is no uniformity in the orientation of the person's
body taken with the drone. Therefore, in this paper, we
propose an automatic detection method for a fallen
person that does not depend on the body orientation of
the detection target. In this method, after detecting afallen person [5], his/her head position is estimated anddetected. This enables quick support of the fallen person
for communication, medical care, food, etc.

2. Method
2.1. Ri-HOG features
The Ri-HOG feature is a feature of HOG [9] withrotational invariance. Unlike the original HOG, whichuses rectangular cells, the cell arrangement obtained by
dividing concentric circles is used in Ri-HOG. Thissection describes the Ri-HOG feature.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

279

Haruka Egawa, Seiji Ishiakwa, Joo Kooi Tan

2.1.3. Normalization

2.1.1. Calculation of luminance gradient
information

In order to reduce the influence of changes in local
brightness and contrast, normalization is performed foreach 2-D histogram obtained by Eq. (2).
Since the values of luminance gradient direction θ of theentire circular region in the Cartesian coordinate
representation range from 0 ° to 360 °, it is quantized in
36 directions every 10 ° interval. A gradient histogram iscreated by adding the luminance gradient intensities in
the circular region based on the luminance gradientdirection in the quantized Cartesian coordinate
representation. After that, the cells are rearranged in theclockwise order from the reference direction based on theangle at which the gradient histogram takes a peak, andthe vector in which the 2-D histograms of each cell areconnected is used as the Ri-HOG feature quantity. Here,the cell numbers of the smallest concentric circle areexpressed as follows;

The input color image is gray-scaled and the brightness
gradient at each pixel is calculated. From the obtained
luminance gradient, the luminance gradient intensity andthe luminance gradient direction in the Cartesian
coordinate representation are obtained.
Assuming that the origin O is at the center of an input
image, the difference between the declination 𝜑𝜑(𝑥𝑥, 𝑦𝑦) ofa pixel position (𝑥𝑥, 𝑦𝑦) provided by the polar coordinate
representation and the obtained brightness gradientdirection 𝜃𝜃(𝑥𝑥, 𝑦𝑦) of the pixel defines the brightness
gradient direction 𝜃𝜃 ′ in the polar coordinate
representation . 𝜃𝜃 ′ is defined by𝜃𝜃 ′ (𝑥𝑥, 𝑦𝑦) = 𝜃𝜃(𝑥𝑥, 𝑦𝑦) − 𝜑𝜑(𝑥𝑥, 𝑦𝑦)(1)

2.1.2. Creating a 2-D histogram

In the proposed method, a concentric circle consisting of
three large, medium and small circles is divided into 36
in the angular direction, and each small area is defined asa cell. The cell numbers of the small concentric circle are
given as 1,4,7⋯,106 clockwise from the start point,
starting from the area in the 0 ° direction. Similarly, thecell numbers of the middle circle are 2,5,8, ⋯, 107, andthe cell numbers of the largest circle are 3,6,9, ⋯, 108.
The offset regions W1 (𝑝𝑝), W2 (𝑝𝑝), ⋯ , W6 (𝑝𝑝) arefurther set by dividing the semicircular region existing in
the radial direction of the pixel of interest 𝑝𝑝 = (𝑥𝑥, 𝑦𝑦) into
six parts. Let 𝑞𝑞 be the pixel in the offset region, and, frompixels p and q, a 2-D histogram is calculated to use theco-occurrence information of their luminance gradientdirections 𝜃𝜃 ′ . Six 2-D histograms are created with eachcell as it has 6 offsets. The 2-D histogram is calculatedusing the following formula;𝑖𝑖𝑢𝑢 = 1,4,7, ⋯ , 𝑁𝑁 (𝑢𝑢 = 1,2,3, ⋯ 36)

Using Eq.(3), the cells are connected as i1 , i1 + 1, i1 +2, i2 , i2 + 1, i2 + 2, i3 ⋯ , i36 + 2 , if the peak of thegradient histogram is 0 °, and they are connected as
i36 , i36 + 1, i36 + 2, i1 , ⋯ , i34 + 2, i35 , i35 + 1, i35 + 2, if
the peak of the gradient histogram is 350 °.
2.2. Ri-LBP features
The Ri-LBP feature is a feature obtained by adding
rotational invariance to the Local Binary Pattern (LBP)[10]. This section describes the Ri-LBP feature.
2.2.1. Local binarization
The input image is gray-scaled, and, as shown in Fig.
1, the brightness values of the pixel of interest and its 8
adjacent pixels are compared. If the gray value of an
adjacent pixel is larger than or equal to that of the center
pixel, the adjacent pixel is given 1, otherwise given 0. Byaligning the obtained 0 or 1 in the direction of the arrow
shown in Fig. 1 (b), an 8-digit binary number is obtained,
which represents the center pixel.
In Ri-LBP, the starting point is changed in turn to acquire
eight 8-digit binary numbers, and the minimum value outof the 8 numbers represents the center pixel.
For example, in Fig. 1, the normal LBP value is0100 0011𝑏𝑏 = 67𝑑𝑑 , but the Ri-LBP value is0000 1101𝑏𝑏 = 13𝑑𝑑 .

H𝑖𝑖𝑖𝑖 (𝑜𝑜1 , 𝑜𝑜2 )=�� �𝑚𝑚(𝒑𝒑) + 𝑚𝑚(𝒒𝒒)�𝐾𝐾(𝜃𝜃 ′ (𝒑𝒑) (2)𝒑𝒑∈S𝑖𝑖 𝒒𝒒∈𝑊𝑊𝑗𝑗 (𝒑𝒑)(3)

∈ 𝑜𝑜1 )𝐾𝐾(𝜃𝜃 ′ (𝒒𝒒) ∈ 𝑜𝑜2 )Here, 𝑖𝑖 is the cell number, 𝑗𝑗 is the offset number, 𝑜𝑜1 is theluminance gradient direction of pixel 𝑝𝑝 after quantization,and 𝑜𝑜2 is that of the offset pixel 𝑞𝑞 after quantization. 𝑆𝑆𝑖𝑖 isthe cell of present concern, and 𝐾𝐾() is the function whose
value is 1, if the argument is true, and 0, otherwise.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

280

Detection of a Fallen

3. Head position estimation(a)Fig. 1. LBP

The head position of a fallen person is estimated using
the angle at which the gradient histogram in the circularregion obtained in Section 2.1.3 has a peak. When a fallen
person exists in the detection window in the orientation
shown in Fig. 2, the horizontal gradient becomes large
and the peak angle of the gradient histogram is 0 ° or
180 °. Therefore, the areas shown by the red frame in Fig.
2 are the estimated head position.

(b)

2.2.2. Histogram creation
The cell of the Ri-LBP feature is the same as the cell of
Ri-HOG feature.
Since LBP is given as an 8-digit binary number, there are
256 values from 0 to 255, but in Ri-LBP, there is an 8digit binary number that becomes the same value bychanging the start point. For example, 1000 0000𝑏𝑏 =128𝑑𝑑 in LBP is expressed as 0000 0001𝑏𝑏 = 1𝑑𝑑 when
the start point is changed. Hence 1000 0000𝑏𝑏 and0000 0001𝑏𝑏 are equivalent in Ri-LBP. After all, the total
number of Ri-LBP values is 36.
Since there are cases where the number of pixels in one
cell is less than 36, the proposed method divides 36
patterns into 9 patterns and creates a histogram with eachcell. This histogram is normalized. The cells are
rearranged based on the reference angle that gives thepeak in the brightness gradient direction obtained in
Section 2.1.3, and the histograms of all the cells areconcatenated. This provides the Ri-LBP feature.

4. Experiment
In the experiment, an image of a fallen person is used as
an image of the Positive class of the image database.
Images other than people are used for Negative classimages. The INRIA Person Dataset [11] was also used as
an image of the Negative class. A classifier is constructedusing these training images. The size of the training
image is all 61 pixels vertically and 61 pixels horizontally.
4.1. Experiment 1
The first experiment is the detection of a fallen personfrom an image. Four types of moving images taken witha multicopter are used in the experiment. In all of thesevideos, there is only one fallen person to be detected. In
Experimental Video 1, there are some obstacles otherthan a fallen person. In the experimental video 2 and theexperimental video 3, the amount of movement of thecamera is large, and there are some frames in which there
is no fallen person in the image. In addition, there is an
upright person who is not fallen in the experimental video
4.
The detection is evaluated with each detection windowusing Intersection over Union (IoU). The threshold of the
IoU is experimentally set to 0.5. As a result of theexperiment, the recall was 0.990 for movie 1, 0.855 formovie 2, 0.565 for movie 3, and 0.875 for movie 4.
An example of the experimental results is shown in Fig.
3. The green rectangle is the manually set Ground Truth,
whereas the red rectangle shows a detected fallen person

2.3. Detection of a fallen person
The proposed method uses Random Forest as adiscriminator. The features of Random Forest include
that it is possible to learn efficiently even with highdimensional features by random learning, and that the
influence of noise in the teacher signal can be suppressedby random selection of learning data.

4.2. Experiment 2
In the second experiment, head of a fallen person issearched on the image of a detected fallen person. For theexperiment, 137 images of the fallen person area cut out
from the video taken by the multicopter are used.

Fig. 2. Example of presumed head positions in the fallen
person detection window.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

281

Haruka Egawa, Seiji Ishiakwa, Joo Kooi Tan

Fig. 3. Example of detection result of Experiment 1 ．

References
1. 平成 25 年版防災白書 付属資料, 内閣府, 2013, p.1．
2. P. Viola, M. Jones: “Rapid object detection using aboosted cascade of simple features”, Proc. IEEE Conf. on
Computer Vision and Pattern Recognition, 2001, pp.511518.
3. R. Sato, S. Koshimura: “Development of a method forsearching survivors in disaster-affected areas with UAV
and image analysis”, J. JSCE, Ser.B2, Coastal engineering
Vol. 69, No. 2,2013, pp.I_1461 - I_1465.
4. A. Gaszczak，T. P. Breckona，J. Hana： “Real-time
people and vehicle detection from UAV imagery”, Proc.
SPIE Conference on International Robots and Computer
Vision XXVIII:Algorithms and Techniques, Vol.7878,
2011, pp.78780B-78780B-13.
5. H. Egawa, S. Isihkawa, J. K. Tan: “Detection of a fallen
person from UAV images using rotation invariant
features”, Proc. of Biomedical Fuzzy Systems Association,
2020, pp.1-4(in Japanese).
6. T. Fujimoto, Y. Kameda, I. Matsuda, S. Itoh: “Humandetection from top view images using rotation invariant
HOG features”, Proc. of The Institute of Image
Information and Television Engineers, 2014, pp.17_10_117_10_2．
7. M. Pietikaine,T. Ojala,Z. Xu: “Rotation-invariant texture
classification using feature distribution”, Pattern
Recognition, Vol.33, 2000, pp. 43-52．
8. L. Breiman: “Random forests”, Machine Learning, Vol.45,
No.1, 2001, pp.5-32．
9. N. Dalal, B. Triggs: “Histograms of oriented gradients forhuman detection”, Proc. IEEE Conference on Computer
Vision and Pattern Recognition, 2005, pp.886-893．
10. T. Ojala, M. Pietikainen, D. Harwood: “A comparative
study of texture measures with classification based onfeatured distributions”, Pattern Recognition, Vol. 29, No.
1, 1996, pp. 51–59.
11. INRIA Person Dataset:（ http://pascal.inrialpes.fr/data/human/( Accessed:10
Jan,2019）

Fig. 4. Example of the estimated head position inExperiment 2

Fig. 5. The result of head detection with a fallen person

In order to visualize the result, the estimated head
position is painted black. The case where the estimatedhead position overlaps with the head of the actual lying
person is regarded as true. As the result of the experiment,
the correct answer rate was 73.7%. An example of theexperimental results is shown in Fig. 4.
Fig. 5 shows the results of head detection. It is indicatedby a small light-blue square.
5. Conclusion
In this paper, we proposed a method of acquiring aerial
images with a camera mounted on a multicopter anddetecting a fallen person from the images.
In the experiment, the recall rate of the fallen person
detection was about 82% when there was one fallen
person in the image frame. The correct answer rate forhead position estimation using the peak angle of thegradient histogram was 73.7%. In the future, we aim at
further improvement of the detection rate by increasing
the number of learning samples and detecting a fallen
person with occlusion.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

282

Development of a Pedestrian Crossing Navigation System
for a Visually Impaired Person Using MY VISION
Kohei Kitagawa

Department of Mechanical and Control Engineering, Kyushu Institute of Technology, Sensui-cho Tobata-ku
Kitakyushu, 804-8550, Japan
Seiji Ishikawa, Joo Kooi Tan

Department of Mechanical and Control Engineering, Kyushu Institute of Technology, Sensui-cho Tobata-ku
Kitakyushu, 804-8550, Japan
E-mail: {kitagawa-kohei ,etheltan}@ss10.cntl.kyutech.ac.jp
http://lab.cntl.kyutech.ac.jp/~etheltan/

Abstract
In this paper, we propose a system for a visually impaired person to cross a pedestrian crossing safely by theemployment of the state of a traffic light and the remaining distance to the other side obtained from the analysis of
MY VISION images. The traffic light at a pedestrian crossing is detected by a discriminator using HOG (Histogramsof Oriented Gradients) features and Random Forest. The effectiveness of the proposed method was verified byexperiments.
Keywords: MY VISION, Visually Impaired, HOG, Random Forest, Traffic light, Pedestrian crossing behavior.

navigation system that recognizes and analyzes(pedestrian) traffic lights, so that a visually impaired
person may cross the intersection safely.
As a traffic light detection method, a method using
template matching and a method utilizing position
information by GPS or a map have been proposed3-5.
However, all the studies are limited to the detection of
traffic lights and they do not analyze possible timing tocross. However, in practice, it is vital for a visually
impaired person to receive the information on the timing
to cross. It should be realized by the analysis of traffic
light images, once it is detected.
In this paper, we propose a method of detecting apedestrian traffic light at a pedestrian crossing by MY
VISION and analyzing it if one can walk into the crossing
now. MY VISION is a self-virtual viewpoint employing
a body (or a head) mounted camera. It can be a virtual
eye of a visually impaired person or the third eye of aperson who is careless ahead. In the proposed method,

1. Introduction
The 2011 Ministry of Health, Labor and Welfare report
writes that there are approximately 310,000 people inJapan who have visual impairment. Looking at the world,
it is predicted that the number of visually impaired people
worldwide will triple within the next 40 years 1.
Accidents of visually impaired people are notuncommon, and there is a survey result that about 42%of the subjects experienced accidents with injuries while
walking outdoors within the past 5 years, especiallyagainst automobiles 2. It is easy to imagine that accidents
with cars may often happen at an intersection, which may
be one of the difficult points for a visually impaired
person to cross.
It is based on Braille blocks and acoustic signs that a
visually impaired person finds an intersection and judges
the timing of crossing it. However, neither all
intersections are installed Braille blocks, nor they haveacoustic instruments. Therefore, it is necessary to have a© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

283

Kohei Kitagawa, Seiji Ishikawa, Joo Kooi Tan

the traffic light is detected employing the images
captured by a camera attached to the head of a person.
The HOG feature is used for describing a traffic light.
The traffic light is recognized by a Random Forestclassifier, which is a multi-class classifier. Then patternanalysis is performed with the traffic light in the image
sequence, and the possibility is judged if one may crossthe frontal pedestrian crossing.

improve generalization ability for unknown data inpreparation for future multiclass classification.．

2. Traffic Light Detection

3.1. The hypothesis on pedestrian crossing
behavior principle

3. Crossing Analysis
In this section, the method is explained for analyzing thesituation of a traffic light and determining the timing ofcrossing. The analysis is performed based on thepedestrian crossing behavior principle hypothesis9.

In this section, we describe detection of traffic lights,
extraction of target object candidate regions, feature
extraction using HOG features, and learning andidentification by Random Forest.

The hypothesis on pedestrian crossing behavior principle
has been proposed, which gives insight into therelationship between pedestrian signals and pedestrian
behavior.
The following four empirical patterns are consideredin the hypothesis based on the situation of the traffic light.
1. Experience on the change of the light from red toblue: The case where the length of the blue light isperceptible and it is considered to be the safest to
cross.
2. Experience on only blue light display: The casewhere the length of the blue light display cannot beperceived and it is unknown when it will turn to a
blinking signal. Depending on the walking speed ofa person, it may be dangerous.
3. Experience on the change from blue to blinking
blue: This is the case where the length of the blue
blinking time is perceptible, and a person on cross
can finish crossing.
4. Experience on only blinking blue light. The lengthof the blue blinking time cannot be perceived, and
it is not possible to cross.)
Based on this crossing behavior principle hypothesis,
pattern identification and analysis of a traffic light are
performed.

2.1. Extraction of a signal candidate area
In the proposed method, instead of searching for a traffic
light in a whole input image, it is searched in the upper
half of the input image after converting it to a HSV image,which is close to human color recognition6. A candidate
area of a traffic signal is extracted by searching for thesignal color (red or green) in the upper half image. The
reason the upper half of an image is considered is that thetraffic light normally exists only in the upper half area ofa self-virtual viewpoint image.
Once the candidate areas of a traffic signal are
extracted, the areas receive expansion processing, since
the color area of the traffic light may be extracted smaller
than it actually is. After the expansion, labeling is
performed to the expanded areas and those noisy areas
are removed using a certain threshold value. Thus acandidate area of a traffic light is obtained.
2.2. Histogram of Oriented Gradients (HOG)
The HOG feature7 is used in the proposed method as afeature for detecting a traffic light, as it is robust to
lighting changes, misalignment, and rotation. The
gradient direction is quantized in four directions toaccommodate angular changes.

3.2. Input of walking status
In the proposed method, crossing analysis takes thewalking condition of a user into account. The information
on what kind of walking situation a user is in is collectedin advance.
The followings are two patterns of walking situation
with a user.
A) Walks normally.
B) Cannot walk normally.

2.3. Learning and identification by Random Forest
In the method, Random forest8, which is capable of multiclass discrimination, is employed for a classifier.
Random forest is possible to learn and identify multiple
target objects at the same time. By using it, we can© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

284

Development of a Pedestriancrossing. This process is faster than normal Houghtransform.
Finally, based on the obtained straight lines (Fig. 2 (5)),
the number of lines is counted and the length of thecrossing is calculated (Fig.2(6)).

Once either of the above walking conditions is entered
before using the system, it is taken into consideration
when the pattern 2 in 3.2 is identified.
3.3. Discrimination of traffic lights
This section describes how to discriminate the color andblinking of a traffic light.

3.6. Judgment on crossing possibility
Whether to cross or not is judged based on the hypothesis
on crossing behavior principle. As shown in Fig.2, thesignal pattern is classified into 6 patterns and pattern
identification is performed.
The output is done by voice. (For example, “The traffic
signal will change soon.”, “About 3 meters remaining.”)

3.3.1. Color discrimination
The color of the traffic light is determined based on the
HSV model. The color of the signal (red/blue) is
identified by setting certain thresholds on H, S and V
values and counting the number of pixels of each color in
the search window.

4.

3.3.2. Identification of blinking signal

Experimental Results

An experiment was conducted to examine theperformance of the proposed method. The used camera isa Sony FDR-X3000. It is mounted on the head of a
subject. Five videos were evaluated for the situation
analysis including traffic light detection. The traffic
signal situation was analyzed including stationary signal
and blinking signal discrimination, the judgment on
crossing possibility. The pedestrian crossing distance
was also estimated.
The results are shown by F-values. The average Fvalue of traffic light detection was 0.89. The average Fvalues of red, blue and blinking signals detection were0.87, 0.89 and 0.94, respectively. Finally, the correct
answer rate for crossing possibility judgment was 0.91,
and the relative error of the distance estimation on a

The blinking cycle of the traffic light is used to determine
whether the signal is blinking or it is the frame a traffic
light cannot be detected. The blinking cycle of the traffic
light is 0.5 seconds in Japan. If this cycle is satisfied, it isdetermined as a blinking signal, and if it is any othercycle, it is not detected.
3.4. Pattern discrimination on traffic signals
Based on the hypothesis on pedestrian crossing behavior
principle explained in Section 3.1, pattern discriminationis performed in the sequence of video frames. There are
four signal conditions: (a) red to blue, (b) blue, (c) blue
to blue blinking, and (d) blue blinking.
3.5. Cross judgment
The proposed method determines whether a person iscrossing a pedestrian crossing or before crossing it, andcalculates approximate crossing distance. Consideringthat a pedestrian crossing exists in the lower half area ofan image, the input image is processed only the lower half.
The pedestrian crossing image is processed in thefollowing way. Image noise is removed by preprocessing
using a Gaussian filter (Fig. 1(1)). Then it is binarized by
the discriminant analysis10 to make it less insensitive tooutdoor weather change.
After performing edge extraction with a Sobel filter, aprobabilistic Hough transform11 is applied to the edge
image to detect straight lines composing a pedestrian

Fig. 1. Flow of the pedestrian crossing distance calculation: (1)Input image, (2) noise reduction, (3) binarization, (4) edge
detection, (5) line detection, and (6) calculation of the distance.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

285

Kohei Kitagawa, Seiji Ishikawa, Joo Kooi Tan

Fig. 3. Part of the experimental video.
2. S. Abe, N. Hashimoto, Survey on walking for the visually
impaired, Bulletin of Hachinohe Institute of Technology 24,
2015, pp. 81-92. (In Japanese)
3. M. Imano，T. Hosokawa, et al., Research on autonomous
movement of guide robots that guide visually impaired
people, 2018 Tohoku chiku wakatekenkyusha happyoukai,
2018. (In Japanese)
4. Y. Baba ， Y. Sakai et al., Real-time recognition ofpedestrian signals from moving images, Proc.77th
National Convention of IPSJ 1, 2015, pp.543-544. (InJapanese)
5. N. Fairfield, C. Urmson, Traffic light mapping anddetection, 2011 IEEE International Conference on
Robotics and Automation (ICRA), 2011, pp.5421-5426.
6. T. Kumano, J. K. Tan, H. Kim, S. Ishikawa, Traffic signs
and signals detection employing the My Vision system fora visually impaired person, International Journal of
Research and Surveys, ICIC Express Letters, partB:
Applications 7(2), 2016, pp.385-391.
7. N. Dalal, B. Triggs, Histograms of oriented gradients forhuman detection, Proc. of IEEE Conference on Computer
Vision and Pattern Recognition 1, 2005, pp.886-893.
8. L. Breman, Random forests, Machine Learning 45(1),2001, pp.5-32.
9. K. Hatoyama, S. Sugimori, Analysis on Pedestrian
Behavior Considering Spaces before the Crosswalk,
Proceedings of infrastructure planning 30, 2004, p.171.
(In Japanese)
10. N. Otsu, A threshold selection method from gray-level
histograms, IEEE Transactions on systems, man, andcybernetics 9(1), 1979, pp.62-66.
11. N. Kiryati, Y. Eldar, A. M. Bruckstein, A probabilistic
Hough transform, Pattern Recognition 24(4), 1991,
pp.303-316.

Fig. 2. Pattern identification: Patterns are defined using
Pedestrian Crossing Behavior Principle Hypothesis, user status
and judgment during crossing.

pedestrian crossing was 25.4%. Fig. 3 shows a part of theexperimental video.
5.

Discussion

In the traffic light detection experiment using HOG
features and Random Forest, the average F-value was
0.89, which was an acceptable detection result. The
reason of this good result may be the selection of 4 bins,
which could absorb varying inclination of the line
segments composing a traffic light.
Regarding the analysis of the traffic light, the signal
detection including the blinking signal showedsatisfactory results.
6.

Conclusion

We proposed a method of detecting and analyzing
pedestrian traffic lights from MY VIOSIN images. The
effectiveness of the proposed method was verified byexperiments. We plan to analyze not only traffic lights
but also other kinds of information for a visually
impaired person to walk safe in an outdoor environment,
and consider developing a navigation system that is more
appropriate to the actual environment.
References
1. Bourne, R. R. A., et al., Magnitude, temporal trends, andprojections of the global prevalence of blindness anddistance and near vision impairment: a systematic review
and meta-analysis, The Lancet Glob Health 5(9), 2017,
e888–e897.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

286

Abnormal Human Action Detection Based on GANTomoya Sano, Seiji Ishikawa, Joo Kooi Tan

Department of Mechanical and Control Engineering, Kyushu Institute of Technology,
Sensui-cho, Tobata-ku, Kitakyushu 804-8550, Japan *E-mail:{sano-tomoya, etheltan}@ss10.cntl.kyutech.ac.jp
http://lab.cntl.kyutech.ac.jp/~etheltan/

Abstract

One of the important roles of a camera surveillance system is to detect abnormal human actions or events.
In this study, we propose a method of abnormal human actions/events detection method using Generative
Adversarial Nets (GAN). In anomaly action detection, the main problem is that the image data ofabnormal human actions is more difficult to obtain than normal human actions. To solve this difficulty,
we use only normal human action data in the employed training network and those actions not recognized
as normal are judged as abnormal. Experimental results show effectiveness of the proposed method.
Keywords: anomaly detection, camera surveillance system, CNN, GAN, optical flow.

neural network. As mentioned above, abnormal events
rarely occur compared to normal events. Therefore, it isvery difficult to collect data on abnormal events. We
solve this problem by using only the data on normal
events when training a model. Then an action notrecognized as a normal action is judged as an abnormal
action.

1. Introduction
Recently, surveillance systems using cameras have been
widely used according to frequent outbreak of crimes.
The number of surveillance cameras installed is on therise. It is used in various places such as convenience
stores, supermarkets, banks, stations, roads, and homes,
and not only law enforcement agencies but also privatecompanies and individuals exist as users. However, due
to the shortage of manpower, it has not been possible tosecure human resources to monitor surveillance camera
images. Generally, the frequency of abnormal events ismuch lower than the frequency of normal events. As aresult, most of the time and money is wasted on the work
done by the observer. Therefore, there is an urgent needto develop a technology that automatically detects
abnormal events from the images of surveillance cameras.
In this paper, we propose a detection method of ananomaly action on a video image using a convolutional

2. Acquisition of motion information
We estimate optical flows to capture motion features ona video. We use FlowNet [1] for this purpose. FlowNet isa deep convolutional neural network that provides a highdensity optical flow for each estimated pixel, given two
consecutive video frames. The estimated optical flow isprovided in the form of a 2-channel (horizontal andvertical) map. Optical flow expresses the magnitude anddirection of movement by a vector.
In the proposed method, we use the 2-channel map thattakes the absolute value of the optical flow as motion© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

287

Tomoya Sano, Seiji Ishikawa, Joo Kooi Tan,the probability that a given sample is AOF obtained by
FlowNet. 𝐷𝐷 discriminates whether the given input is
AOF obtained by FlowNet or AOF estimated from one
frame by 𝐺𝐺. Figure 1 shows the structure of the proposed
framework.
Through this training, we aim at enabling the 𝐺𝐺 topredict the appropriate motion information of the training
data (normal event). We use the trained 𝐺𝐺 during test.
When the data of a normal action is given to 𝐺𝐺 , themotioninformation
predictionisperformed
appropriately. On the contrary, when the data including
an abnormal action is given to 𝐺𝐺, the motion information
is not estimated properly. As a result, the prediction result
generated by 𝐺𝐺 differs from the result obtained by
FlowNet. In this way, anomaly detection is executed.

information. This prevents the loss calculated by the loss
function during training (discussed later in Section 3.3)
from becoming large due to the inversion of the positive
and negative signs and promotes early convergence of theparameters. We call this feature expression Absolute
Optical Flow (AOF) for convenience.
3. Framework
3.1. GAN
Generative Adversarial Nets (GAN) [2] is a framework
for training generative models. It aims at finding agenerative data distribution 𝑝𝑝𝑔𝑔 (𝒙𝒙) that matches the true
data distribution 𝑝𝑝𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 (𝒙𝒙) obtained from the training
data 𝒙𝒙 . Both the generator 𝐺𝐺 , that captures the data
distribution, and the discriminator 𝐷𝐷, that estimates theprobability that the input data is obtained from 𝑝𝑝𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 (𝒙𝒙),are trained while competing with each other.
The generator 𝐺𝐺 generates the generated data 𝐺𝐺(𝒛𝒛)when random noise 𝒛𝒛 according to the distribution 𝑝𝑝𝒛𝒛 (𝒛𝒛)is input to it. The input data 𝒖𝒖 to the discriminator 𝐷𝐷 is
the training data 𝒙𝒙 or the generated data 𝐺𝐺(𝒛𝒛). The output
𝐷𝐷(𝒖𝒖) of the discriminator 𝐷𝐷 is the probability that the
input data 𝒖𝒖 belongs to the distribution of training data
𝑝𝑝𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 (𝒙𝒙). In GAN, 𝐺𝐺 and 𝐷𝐷 are optimized by the valuefunction 𝑉𝑉(𝐺𝐺, 𝐷𝐷) of the following equation;
min max 𝑉𝑉(𝐺𝐺, 𝐷𝐷) = 𝔼𝔼𝒙𝒙~𝑝𝑝𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑(𝒙𝒙) [log 𝐷𝐷(𝒙𝒙)]𝐺𝐺

𝐷𝐷+𝔼𝔼𝒛𝒛~𝑝𝑝𝒛𝒛 (𝒛𝒛) �log �1 − 𝐷𝐷�𝐺𝐺(𝒛𝒛)���

3.3. Loss function
The two models 𝐺𝐺 and 𝐷𝐷 minimize the following losses
𝐿𝐿𝐺𝐺 and 𝐿𝐿𝐷𝐷 by learning.
𝐿𝐿𝐺𝐺 = softplus �−𝐷𝐷�𝒙𝒙𝑡𝑡−1 , 𝒙𝒙𝑡𝑡 , 𝐺𝐺(𝒙𝒙𝑡𝑡 )��

𝐿𝐿𝐷𝐷 = softplus�−𝐷𝐷(𝒙𝒙𝑡𝑡−1 , 𝒙𝒙𝑡𝑡 , |𝐹𝐹(𝒙𝒙𝑡𝑡−1 , 𝒙𝒙𝑡𝑡 )|)�+softplus �𝐷𝐷�𝒙𝒙𝑡𝑡−1 , 𝒙𝒙𝑡𝑡 , 𝐺𝐺(𝒙𝒙𝑡𝑡 )��

softplus(𝑛𝑛) = log(1 − exp 𝑛𝑛)(1)

In the above equation, 𝐺𝐺 minimizes the objectivefunction to trick 𝐷𝐷 into generating data that is
indistinguishable from true data. On the other hand, 𝐷𝐷
finds the distinguishing boundary between the true data
and the generated data by maximizing the objectivefunction and avoids being fooled by 𝐺𝐺. In this way, 𝐺𝐺 and
𝐷𝐷 have a hostile relationship and are optimized while
competing in GAN.
3.2. Proposed framework
Our framework learns only normal event data insurveillance camera images. We train two deep
convolutional neural networks based on GAN. The firstis generator 𝐺𝐺, which captures the data distribution of
normal events. 𝐺𝐺 gives one raw frame obtained from thecamera and predicts the estimated motion information(AOF). The second is discriminator 𝐷𝐷, which estimatesFig. 1. The structure of the proposed framework.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

288

(2)

(3)
(4)

Abnormal Human Action Detection

fully connected layer. It should be noted that noactivation function is applied to the output layer.

where 𝐺𝐺(𝒙𝒙𝑡𝑡 ) is the output of the generator 𝐺𝐺 ,𝐹𝐹(𝒙𝒙𝑡𝑡−1 , 𝒙𝒙𝑡𝑡 ) is the output of FlowNet, 𝐷𝐷�𝒙𝒙𝑡𝑡−1 , 𝒙𝒙𝑡𝑡 , 𝐺𝐺(𝒙𝒙𝑡𝑡 )�
is the output of discriminator 𝐷𝐷 that estimates theprobability that the data generated by 𝐺𝐺 belongs to the
distribution of the estimation results by FlowNet.
3.4. Anomaly detection
Anomaly detection is executed every frame except thefirst frame. To detect anomalies, the squared error of theoutput of generator 𝐺𝐺 and AOF obtained from FlowNet
is calculated. The abnormal score at time 𝑡𝑡 is calculatedby the following equation.
2

𝐸𝐸𝑡𝑡 = ��𝐺𝐺(𝒙𝒙𝑡𝑡 ) − 𝐹𝐹(𝒙𝒙𝑡𝑡−1 , 𝒙𝒙𝑡𝑡 )�

(5)When 𝐸𝐸𝑡𝑡 exceeds a threshold value, the video frame attime 𝑡𝑡 is detected as a frame containing an abnormal
event.
3.5. Network architecture

Fig. 2. Details of generator 𝐺𝐺. The blue block shows the feature
map. The orange arrow indicates 3×3 convolution (padding=1,
stride=1) with batch normalization, followed by a ReLU. The
red arrow indicates 2×2 max-pooling with stride 2. The green
arrow indicates 2×2 deconvolution with stride 2. The purple
arrow indicates a 1×1 convolution with the sigmoid function
applied. The black branched arrow indicates that the two featuremaps are connected.

3.5.1. Generator 𝐺𝐺

Generator 𝐺𝐺 is a Deep Encoder-Decoder that has a skip
coupling between the intermediate layer of thecontracting part and the expansion part like U-net [3].
The input is one raw video frame (128×128×3). The
output is AOF (128×128×2). Details are shown in Fig. 2.
In the 3×3 convolution layer, batch normalization is
performed, followed by a rectified linear unit (ReLU). In
the contracting part (upper side of Fig. 2), downsampling is performed by 2×2 max-pooling, which
doubles the number of channels in the feature map. After
that, in the expansion part, on the contrary, up-samplingis performed by 2×2 deconvolution that halves thenumber of channels in the feature map.
3.5.2. Discriminator 𝐷𝐷

Discriminator 𝐷𝐷 is a deep convolutional neural networkwith 10 convolutional layers. The input is a feature map(128×128×8) that connects two consecutive video frames
and AOF. The output is the probability (scalar) that the
AOF contained in the input is obtained by FlowNet.
Details are shown in Fig. 3.
Discriminator 𝐷𝐷 has the same structure as the encoder
part of the generator 𝐺𝐺. Finally, it is output through a

Fig. 3. Details of discriminator 𝐷𝐷 . The dark green arrow
indicates a fully connected layer. For other arrows, see theexplanation in Fig. 2.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

289

Tomoya Sano, Seiji Ishikawa, Joo Kooi Tan,videos for 20 actors are used for model training. Six
motion videos of walking, jogging, running, boxing,
hand waving, and hand clapping by the remaining 5
people are used for the test.

True Positive Rate (TPR)

1
0.8
0.6

4.2. Result
Figure 4 shows the ROC curve of a single-class motion
classification experiment using a KTH dataset. The AUC
calculated from Fig. 4 was 0.94.

0.4
0.2
0

5. Conclusion
0

0.2
0.4
0.6
0.8
False Positive Rate (FPR)

We proposed a GAN-based method for detecting
abnormal human action using only normal event data fortraining. Our experiments show that the proposed method
is effective in detecting abnormal human actions. Future
work includes application to more realistic surveillance
video images and improvement of accuracy.

1

Fig. 4. ROC curve.

4. Experiment

References

Using the proposed framework, we perform a singleclass motion classification experiment. For evaluation, a
receiver operating characteristic (ROC) curve and an area
under the curve (AUC) are employed. The ROC curve isobtained by calculating the true positive rate (TPR) andthe false positive rate (FPR) for each of the normal andabnormal cutoff values for abnormality detection, andplotted on the coordinate plane with TPR on the verticalaxis and FPR on the horizontal axis. The area under the
curve is calculated to evaluate the effectiveness of the
proposed method.

1.

2.

3.

4.1. Dataset
The dataset used for the experiment is KTH dataset [4, 5].
KTH dataset is a human motion recognition dataset by 25
actors, including six motion classes: walking, jogging,running, boxing, hand waving, and hand clapping. Thisdataset contains videos shot under four different
conditions (d1, d2, d3, d4), but in this experiment only
the data under condition d1 is used. When inputting animage to the network, both ends are cut so that the image
becomes square, and it is resized to a 128×128 pixel
image. In walking, jogging, and running, there are frameswith backgrounds that do not include people. In thisexperiment, such a frame is excluded in advance.
The walking class in this data set is treated as a normalclass, and the other five classes of jogging, running,boxing, hand waving, and hand clapping are tested asabnormal classes. Out of the data of 25 actors, walking

4.

5.
6.

7.

A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazırbas,
V. Golkov, FlowNet: Learning Optical Flow with
Convolutional Networks, Proceedings of the IEEE
International Conference on Computer Vision (ICCV),
2015, pp. 2758–2766.
I. J. Goodfellow, J. P. Abadie, M. Mirza, B. Xu, D. W.
Farley, S. Ozair, A. Courville, Y. Bengio, Generative
Adversarial Nets, Advances in neural information
processing systems, 2014, pp. 2672–2680.
O. Ronneberger, P. Fischer, T. Brox, U-Net:
Convolutional Networks for Biomedical Image
Segmentation, International Conference on Medical
Image Computing and Computer-Assisted Intervention –MICCAI, 2015, pp. 234–241.
C. Schuldt, I. Laptev, Barbara Caputo, Recognizing
Human Actions: A Local SVM Approach, Proceedings ofthe 17th International Conference on Pattern Recognition(ICPR’04), 2004, Vol. 3, pp. 32–36.
KTH dataset: https://www.csc.kth.se/cvap/actions/M. Ravanbakhsh, E. Sangineto, M. Nabi, N. Sebe,
Training Adversarial Discriminators for Cross-Channel
Abnormal Event Detection in Crowds, Proceedings of2019 IEEE Winter Conference on Applications ofComputer Vision, 2019, pp. 1896–1904.
P. Isola, J. Y. Zhu, T. Zhou, A. A. Efros, Image-to-Image
Translation with Conditional Adversarial Networks,
Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2017, pp. 1125–1134.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

290

Bus Line Number Detection Employing MY VISIONYe Zhou, Yosiki Hamasaki, Joo Kooi Tan

Department of Mechanical and Control Engineering,
Kyushu Institute of Technology,
Sensui-cho Tobata-ku
Kitakyushu, 804-8550, Japan
E-mail: {zhou-ye, etheltan }@ss10.cntl.kyutech.ac.jp
http://lab.cntl.kyutech.ac.jp/~etheltan/

Abstract
In response to the requests from visually impaired people that they wish better assistance tools in their daily activities,
and to the facts that they find difficulty in using public transportation, we propose a bus boarding support systemusing MY VISION. For the support, we detect a bus approaching a bus stop from MY VISION images employing
Haar-like filters, and acquire the moving object area with the pyramidal Lucas-Kanade method. Then, we extract thefrontal upper area displaying bus line number and the destination by random forest. Having obtained the area, amethod of connecting area marking is used to obtain the bus line number area. The effectiveness of the proposed
method is shown by experiments.
Keywords: Bus line number, Moving area detection, Haar-like filters, Random forest, Visually impaired.

method, and corrects the tracking bounding box tocomplete the entire tracking and recognition process.
The system in this paper contains two structures:
learning unit and recognition unit. (1) Learning unit.
According to the input image, the Haar-like [3] feature
quantity is calculated and extracted, which is based on
the brightness difference between the two regions and isrobust to noise and brightness changes. After that, therandom forest classification method is used to effectively
classify the feature quantities. (2) Identification unit. The
bus LED area recognition is performed using Harris
angle detector to extract the feature points of the input
image, and the pyramidal Lucas-Kanade optical flow
algorithm[4] to obtain the moving object area [5]. After that,
Random Sample Consensus (RANSAC) [6] is used tofilter out the interference optical flow information
generated by the camera movement, and then use the kmeans clustering method to obtain the target area. In theclustering area, the learning-based recognition method isused to complete the task of bus LED area recognition.
After a certain target area is obtained, the tracking task is
performed.

1. Introduction
According to the statistics survey from the Ministry ofLabor in the past 23 years, there are about 320,000 people
with visual impairment in Japan [1]. Meanwhile, about
69% of these people are over 65 years old. It can be seenthat with the development of social aging, the number ofpeople with visual impairment is further increasing. For
the visually impaired, considering practicality andeconomic cost, visual impairment usually chooses public
transportation. However, public transportation needs a lotof visual information and there are many interference
problems, such as a bus. In addition, a bus also contains
critical information of the line number and entrance. If
the lack of the above information, the visually impaired
cannot successfully complete the ride behavior.
Therefore, it is necessary to develop a support system thatenables visually impaired people to easily use public
transportation facilities. In this paper, we propose a visualdisability support system that can detect the frontal LED
area of a bus and recognize its route number.
Although the computational cost of using learning
results to detect specific moving objects in a complex
image background is high, the detection results areaccurate. This paper proposes a method of detecting a
specific moving object. It extracts a moving object areaby using the optical flow method, updates the color
histogram of the area based on the Camshift [2] tracking

2. Related work
In the literature, there are many automatic bus number
recognition systems. However, most of them use active
sensors, such as GPS tracking systems, RFID, etc.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

291

Ye Zhou, Yosiki Hamasaki, Joo Kooi Tanbrightness distribution of the image area effectively.
Figure 1 shows an example diagram of the Haar-likefilter and calculation is done by the following formula;

For vision-based methods, Guida[7] proposed thedetection and identification of bus line numbers [8]. Theirmethod uses Adaboost-based cascaded classifiers tolocate the main geometric entities of the bus facade, andthe matching is improved by robust geometric matching.
Then the bus façade image is converted to a HSV color
space and threshold operations are applied to segmenteach number.
Pan [9] proposed another vision-based method that
detects information from the front of the bus. Theirsystem can be divided into 2 subsystems: bus detection
and bus line number detection. In the bus detection, theyuse HOG and SVM to detect the bus position. In the bus
line number detection, they use adjacent charactergrouping and sophisticated edge detection to findcandidate regions. The Haar-like feature is extracted
from edge distribution. The acquired features are input
into Adaboost to classify each component. Finally, they
combine an optical character recognition software with atext-to-speech synthesizer to produce audio.
In short, compared with the aforementioned tracking
and recognition method, computational complexity of
their methods is high, and the recognition accuracy andthe effect of tracking moving objects still need to be
improved. Therefore, in the feature area selection, theproposed method uses the optical flow method toeffectively filter out the moving object area. At the sametime, the Camshift tracking method used in this article
has low computational complexity and has certain
practicability.

𝐻𝐻(𝐴𝐴, 𝐵𝐵) = 𝐹𝐹(𝐴𝐴) − 𝐹𝐹(𝐵𝐵)(1)Here, F(A) and F(B) are the average brightness of area A
and B.
AB

As shown in Fig. 2, the proposed method uses 6
different Haar-like filters. The 6 Haar-like features

Fig. 1 Haar feature

Fig. 2 Six types of filters

include two edge features, three linear features and thecenter-around feature. After obtaining the Haar-likefeatures of the LED area, it is necessary to classify andlearn the obtained features. Random forest is used as aclassification method in the proposed method.
3.2. Random forest

3. Learning unit

The system proposed in this paper, it needs to complete
the task of identifying multiple types of targets.
Therefore, random forest learning [10] meets the needs ofthis method. It creates a feature tree with low correlation
by selecting feature quantities at the non-terminal nodesof each decision tree. It calculates the Haar-like feature
value of each input image to create a sample. A subset is
randomly generated from the sample, and a random
decision tree is created using the branch function of eachsubset to perform learning. The random forest isdetermined by inputting the feature vector of theunknown sample into the decision tree. It calculates thecategory distribution of the terminal node that finallyreaches the unknown sample, and uses this result toestimate the category of the unknown sample through thefollowing equation;

3.1. Haar-like feature
For the detection of a bus LED area, this article is divided
into two types of images for feature learning: the bus
LED area and the background area. In order to unify the
input image size, it is standardized to 100×30 pixels. In
the previous methods based on feature learning, the mostcommonly used method for vehicle feature learning is
HOG feature representation. However, when the featureshapes of two regions are similar, the HOG feature isprone to representation errors, which will affect thesubsequent recognition tasks. Therefore, in this article, itis proposed to use Haar-like features for regional
characterization. Its main advantage is that it
characterizes not only the shape of the object, but also the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

292

Bus Line Number Detection

1

𝑃𝑃(𝑐𝑐/𝑆𝑆) = 𝑇𝑇 ∑𝑇𝑇𝑡𝑡=1 𝑃𝑃𝑡𝑡 (𝑐𝑐/𝑆𝑆)

According to the relatively fixed position distributionof the LED display area of Japanese buses, a moreaccurate line number area can be obtained. Then use HSVcolor processing to obtain the binary image of the digitalpart, and use the connected area labeling method tosegment the digital area.

(2)

In the above formula, T is the number of trees in the
random forest, and P(c/S) is the class distribution.
4.

Recognition

5. Experimental results

The target recognition task is divided into two subtasks:
bus LED area recognition and bus line number
recognition.

5.1. Setup
In this section, we show the results of the method
proposed in this paper to learn the characteristics of thebus LED area, as well as the recognition results of thespecific moving target area and the line number
recognition results obtained from object tracking.
The learning results of the random forests areevaluated by test data. At the same time, the accuracy ofthe bus LED area detection is also evaluated.
Table 1 shows detailed parameter values for thelearning of the random forest. Among them, thedimension of Haar-like features is 5102.

4.1. Bus LED area recognition
In order to reduce the amount of data processing, theoptical flow method is used to obtain a moving object
area [10] to achieve a faster target recognition task. The
calculation process is as follows:
1) Use Harris angle detector to acquire the feature pointsof the target and establish a moving target area. After thefeature points are detected, take r frames as the interval.
The moving object area is obtained by using the LK
pyramid optical flow and k-means clustering method.
2) Since the moving object area obtained by the opticalflow method usually includes the noise caused by thecamera movement, it is necessary to use the projection
transformation model to estimate the camera movement
and remove the noise area generated by it. The
calculation formula of this model is as follows;ℎ 𝑥𝑥 +ℎ 𝑦𝑦 +ℎ𝑥𝑥2 = ℎ0 𝑥𝑥1 +ℎ1 𝑦𝑦1 +12

6 1
7 1
�
ℎ3 𝑥𝑥1 +ℎ4 𝑦𝑦1 +ℎ5
𝑦𝑦2 = ℎ 𝑥𝑥 +ℎ 𝑦𝑦 +1
6 1

Table 1 Learning parameters of the random forest.
Learning images
Test images
Size
Number of subsets
Tree depth
Feature selection times
Threshold selection times

Positive image:
Negative image:
Each class:
100 x 30
30
5
30
30

1200
800
500

(3)

7 1

5.2. Results

Use RANSAC to determine the location parametersℎ𝑖𝑖 (𝑖𝑖 = 0,1 … 7)of Eq. (3). The RANSAC can effectively
divide the internal optical flow area of moving objects
and the external optical flow of noise generated bycamera movement. Then use the random forest tocomplete the recognition of the bus LED area.

For the learning results of random forest, we use theclassification accuracy to evaluate the performance forthe plate of bus and the background.
Table 2 shows the result of the classification. It provesthat the proposed method can achieve a good
performance for the task of classification. Figure 3
shows the detection performance of the method.

4.2. Bus line number recognition
After obtaining the LED area, it needs to be further
processed to obtain the digital area.

Table 2 Random forest learning results© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

293

Ye Zhou, Yosiki Hamasaki, Joo Kooi Tan

Plate
Background

Plate
90.4%
4.0%

Therefore, in future work, it is necessary to expand theexisting data set and adjust the learning parameters to
obtain higher recognition accuracy.

Background
9.6%
96.0%

7. References
1. Ministry of Health, Labor and Welfare, Social AssistanceBureau, Disability Health and Welfare Department: 2011
Survey on Difficulty in Living (Fact-finding Survey onChildren at Home, etc.) Results, Ministry of Health, Laborand Welfare, Social Assistance Bureau, Disability Healthand Welfare Department, 2013, pp.13.
2. O.-D. Nouar, G. Ali, C. Raphael Canals. Improved object
tracking with camshift algorithm, IEEE International
Conference on Acoustics Speech and Signal Processing,
2006, pp.36-42.
3. Gong Haifan, Chen Limin, Li Changhao, Wang Yue.
Online tracking and relocation based on a new rotationinvariant Haar-Like statistical descriptor in endoscopic
examination, IEEE Access, 99, 2020, pp.69-83.
4. Dowson Nicholas, Bowden Richard. Mutual information
for Lucas-Kanade tracking, An inverse compositional
formulation, IEEE Transactions on Pattern Analysis andMachine Intelligence, 30(1), 2008, pp.180-185.
5. Ishitobi K. Tan J K. Kim H. Detection of a specific moving
object from head-mounted camera images. IEEE
International Symposium on System Integration (SII),
2017, pp. 6pages.
6. M.A. Fischler, R.C. Bolles. Random Sample Consensus: aparadigm for model fitting with applications to image
analysis and automated cartography. The ACM
Communication, 1981, pp.381-395.
7. C. Guida, D. Comanducci, C. Colombo. Automatic bus
line number localization and recognition on mobile phones.
A computer vision aid for the visually impaired, Image
Analysis and Processing (ICIAP), 2011, pp. 323–332.
8. Pitchakorn Wongta. An automatic bus route number
recognition. International Joint Conference on Computer
Science and Software Engineering (JCSSE), 2016, pp.1924.
9. H. Pan, C. Yi, Y. Tian. A primary travelling assistantsystem of bus detection and recognition for visually
impaired people, IEEE International Conference on
Multimedia and Expo Workshops (ICMEW), 2013, pp. 1–
6.
10. F.Breiman. Random forest. Machine Learning, Vol, 45,
No. 1, 2001, pp. 5-32.

(a)

(b)
Figure 3 Examples of detection result for the bus LED andthe line number

The average processing time of each picture is 1.56 [s],and the success rate of detecting the LED panel area inthe window of the moving object is 91.3%.
6. Discussion and conclusion
This paper proposes a bus travel assistance system forthe visually impaired. The system includes bus LED area
recognition and line number recognition tasks. Amongthem, the overall system includes learning method design,area tracking method design and recognition method
design. The final experimental results prove theeffectiveness of the proposed method and thepracticability of the system.
In the course of the experiment, it was found that whenidentifying the LED area, there was misrecognition due
to lighting conditions, and interference from moving cars.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

294

A Method for Patterns of Cell-Like Images Based on Distance Transformation
Toru Hiraoka

Department of Information Systems
University of Nagasaki
1-1-1, Manabino, Nagayo-chou, Nishisonogi-gun, Nagasaki-ken, 851-2195, Japan
E-mail: hiraoka@sun.ac.jp
Kohei Maeda

Department of Information Systems
University of Nagasaki
1-1-1, Manabino, Nagayo-chou, Nishisonogi-gun, Nagasaki-ken, 851-2195, Japan
E-mail: bs118031@sun.ac.jp

Abstract
A non-photorealistic rendering method has been proposed to generate cell-like images in which cell-like patterns are
represented in photographic images. Cell-like patterns are automatically generated by the change of density ofphotographic images. However, cell-like patterns are irregularly arranged. In this paper, we propose a method toarrange cell-like patterns along the edges of photographic images. We improves the conventional method by using
Euclidean distance from the edges. We show that appealing cell-like images can be generated by our method throughexperiments using various photographic images.
Keywords: Non-photorealistic rendering, Cell, Euclidean distance, Edge, Automatic generation

The conventional method9 is implemented bysynthesizing sine and cosine waves into photographic
images. However, in the conventional method 9, cell-likepatterns are mainly arranged in a grid pattern, andtherefore the edge preservation of cell-like images is nothigh. Also, the actual cells are not arranged in the grid
pattern.
In this paper, we propose a method to arrange celllike patterns along the edges of photographic images. We
improve the conventional method 8 by using Euclidean
distance from the edges. Cell-like images of our method
can give an impression different from cell-like images of
the conventional method 8,9. By conducting experiments
using various photographic images, it is visuallyconfirmed that cell-like patterns can be generated along
the edges. In addition, it is visually confirmed that thesize of cell-like patterns can be changed by changing thevalues of the parameters in our method.

1. Introduction
Non-photorealistic rendering is a field of computer
graphics that can generate effective illustrations andattractive artistic images. Some researchers haveproposed non-photorealistic rendering methods tosimulate art expression techniques and to develop
unprecedented artistic approaches 1,2,3,4,5,6,7. One of theunprecedented artistic approaches is a method togenerating cell-like images by an iterative calculation
using inverse iris filter from photographic images 8. Celllike patterns are composed of cell membrane and cell
nucleus. Cell-like images are overlaid with cell-likepatterns in photographic images. Cell patterns areautomatically generated by the change of density ofphotographic images. However, cell-like patterns are
irregularly arranged. Therefore, a method for aligningcell-like patterns in cell-like images has been proposed 9.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

295

Toru Hiraoka, Kohei Maeda

pixel.
If the Euclidean distances 𝑑𝑑1,𝑖𝑖,𝑗𝑗 are within mD +D/2 ± 0.5(m = 0, 1, 2, …), the Euclidean distances
𝑑𝑑2,𝑖𝑖,𝑗𝑗 are set to 0, where 𝐷𝐷 is a positive constant.
Otherwise, the Euclidean distances 𝑑𝑑2,𝑖𝑖,𝑗𝑗 are set
to ∞. When the positions where the Euclidean
distances 𝑑𝑑2,𝑖𝑖,𝑗𝑗 are 0 are expressed on the image,it becomes lines of the interval 𝐷𝐷 along theedges.
Scan from the upper left to the lower right of theimage, and in the case that the Euclidean distance
𝑑𝑑2,𝑖𝑖,𝑗𝑗 are 0 at the target pixel (𝑖𝑖, 𝑗𝑗), the Euclidean
distances 𝑑𝑑2,𝑘𝑘,𝑙𝑙 in the range where the Euclidean
distance from the target pixel (𝑖𝑖, 𝑗𝑗) is smaller than𝐷𝐷 are updated to ∞. Where the pixels (𝑘𝑘, 𝑙𝑙) havethe Euclidean distances from the target pixel (𝑖𝑖, 𝑗𝑗)
smaller than 𝐷𝐷, and the Euclidean distance 𝑑𝑑2,𝑖𝑖,𝑗𝑗
of the target pixel (𝑖𝑖, 𝑗𝑗) is not updated. When thepositions where the Euclidean distances 𝑑𝑑2,𝑖𝑖,𝑗𝑗 are
0 are expressed on the image, it becomes dottedlines with the spacing 𝐷𝐷.
Scan from the upper left to the lower right of theimage, and in the case that the Euclidean distance
𝑑𝑑2,𝑖𝑖,𝑗𝑗 are ∞ at the target pixel (𝑖𝑖, 𝑗𝑗), the Euclidean
distance 𝑑𝑑2,𝑖𝑖,𝑗𝑗 of the target pixel (𝑖𝑖, 𝑗𝑗) is updated
to 0 if there is no pixel where the Euclidean
distances 𝑑𝑑2,𝑘𝑘,𝑙𝑙 are 0 in the range where theEuclidean distance from the target pixel (𝑖𝑖, 𝑗𝑗) issmaller than 𝐷𝐷. In all pixels (𝑖𝑖, 𝑗𝑗), there are pixels
that the Euclidean distances 𝑑𝑑2,𝑘𝑘,𝑙𝑙 are 0 within theradius 𝐷𝐷. Shortest Euclidean distances 𝑑𝑑𝑖𝑖,𝑗𝑗 fromthe pixels that the Euclidean distances 𝑑𝑑2,𝑘𝑘,𝑙𝑙 are 0
are calculated at each pixel. An image composedof the shortest Euclidean distances 𝑑𝑑𝑖𝑖,𝑗𝑗 is called a
distance-transformed image.
Step 2: Let output pixel values after processing with iris
filter on 𝑑𝑑𝑖𝑖,𝑗𝑗 be IF(𝑑𝑑𝑖𝑖,𝑗𝑗 ). Iris filter is executed withthe 2r+1 peripheral pixels (𝑘𝑘, 𝑙𝑙) in the window ofsize r. Angles 𝜃𝜃𝑖𝑖,𝑗𝑗,𝑘𝑘,𝑙𝑙 between a vector (𝑖𝑖 − 𝑘𝑘, 𝑗𝑗 −𝑙𝑙) from the peripheral pixels (𝑘𝑘, 𝑙𝑙) to the target
pixel (𝑖𝑖, 𝑗𝑗) and a vector ��𝑑𝑑𝑘𝑘+2,𝑙𝑙+2 + 𝑑𝑑𝑘𝑘+2,𝑙𝑙+1 +𝑑𝑑𝑘𝑘+2,𝑙𝑙 + 𝑑𝑑𝑘𝑘+2,𝑙𝑙−1 + 𝑑𝑑𝑘𝑘+2,𝑙𝑙−2 � − �𝑑𝑑𝑘𝑘−2,𝑙𝑙+2 +𝑑𝑑𝑘𝑘−2,𝑙𝑙+1 + 𝑑𝑑𝑘𝑘−2,𝑙𝑙 + 𝑑𝑑𝑘𝑘−2,𝑙𝑙−1 + 𝑑𝑑𝑘𝑘−2,𝑙𝑙−2 � ,�𝑑𝑑𝑘𝑘+2,𝑙𝑙+2 + 𝑑𝑑𝑘𝑘+1,𝑙𝑙+2 + 𝑑𝑑𝑘𝑘,𝑙𝑙+2 + 𝑑𝑑𝑘𝑘−1,𝑙𝑙+2 +𝑑𝑑𝑘𝑘−2,𝑙𝑙+2 � − �𝑑𝑑𝑘𝑘+2,𝑙𝑙−2 + 𝑑𝑑𝑘𝑘+1,𝑙𝑙−2 + 𝑑𝑑𝑘𝑘,𝑙𝑙−2 +𝑑𝑑𝑘𝑘−1,𝑙𝑙−2 + 𝑑𝑑𝑘𝑘−2,𝑙𝑙−2 �� are computed. Let
convergence indices of the target pixel (𝑖𝑖, 𝑗𝑗) beThis paper is organized as follows: the second
section describes our method to arrange cell-like patterns
along the edges, the third section shows experimental
results and reveals the effectiveness of our method, andthe conclusion of this paper is given in the fourth section.
2. Our method
Our method is implemented in two steps. In the first step,
distance-transformed images are created by calculating
Euclidean distance from the edges of photographic
images. In the second step, cell-like images are generatedusing distance-transformed images and inverse iris filter.
Inverse iris filter is calculated by the procedure that
restores the transformed image using iris filter 10 to theoriginal image using inverse filter 11. The flow chart ofour method is shown in Fig. 1.
We explain details of the procedure in Fig. 1 below.
Step 0: Let input pixel values of RGB on coordinates(𝑖𝑖, 𝑗𝑗) in a color photographic image be 𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗 ,𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗 and 𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗 (𝑖𝑖 = 1, 2, 3, … , 𝐼𝐼; 𝑗𝑗 = 1, 2, 3, … ,𝐽𝐽). The pixel values 𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗 , 𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗 and 𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗 havevalue of 𝑈𝑈 gradation from 0 to 𝑈𝑈 − 1.
Step 1: Gray-scale pixel values 𝑓𝑓𝑖𝑖,𝑗𝑗 are calculated asfollows.
𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗 +𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗 +𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗

𝑓𝑓𝑖𝑖,𝑗𝑗 ＝3(1)

Edges are extracted from the gray-scale image
using EDISON 12 that is a feature extraction toolthat integrates edge detection and image
segmentation. Shortest Euclidean distances
𝑑𝑑1,𝑖𝑖,𝑗𝑗 from the edge pixels are calculated at each
Photographic images

Step 1: Calculation of Euclidean distance from theedges of photographic images
Step 2: Calculation from distance-transformed
images using inverse iris filter
Cell-like images of our method
Fig. 1. Flow chart of our method© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

296

A Method for Patterns

𝑐𝑐𝑖𝑖,𝑗𝑗 . The convergence indices 𝑐𝑐𝑖𝑖,𝑗𝑗 are calculated
as follows.
1

𝑗𝑗+𝑟𝑟

𝑖𝑖+𝑟𝑟
∑𝑙𝑙=𝑗𝑗−𝑟𝑟 cos 𝜃𝜃𝑖𝑖,𝑗𝑗,𝑘𝑘,𝑙𝑙 |𝑐𝑐𝑖𝑖,𝑗𝑗 = (2𝑟𝑟+1)2 | ∑𝑘𝑘=𝑖𝑖−𝑟𝑟(2)

Let minimum and maximum values of 𝑐𝑐𝑖𝑖,𝑗𝑗 in all
pixels be 𝑐𝑐𝑚𝑚𝑚𝑚𝑚𝑚 and 𝑐𝑐𝑚𝑚𝑚𝑚𝑚𝑚 , respectively. The
convergence indices 𝑐𝑐𝑖𝑖,𝑗𝑗 are converted to 𝐶𝐶𝑖𝑖,𝑗𝑗 asfollows.
𝐶𝐶𝑖𝑖,𝑗𝑗 = 255 �

𝑐𝑐𝑖𝑖,𝑗𝑗 – 𝑐𝑐𝑚𝑚𝑚𝑚𝑚𝑚

𝑐𝑐𝑚𝑚𝑚𝑚𝑚𝑚 – 𝑐𝑐𝑚𝑚𝑚𝑚𝑚𝑚

�(3)
Figure 3. Various photographic images.

The values IF(𝑑𝑑𝑖𝑖,𝑗𝑗 ) and 𝐶𝐶𝑖𝑖,𝑗𝑗 are the same value.
Pixel values 𝑔𝑔𝑅𝑅,𝑖𝑖,𝑗𝑗 , 𝑔𝑔𝐺𝐺,𝑖𝑖,𝑗𝑗 and 𝑔𝑔𝐵𝐵,𝑖𝑖,𝑗𝑗 arecomputed by using inverse iris filter as follows.
𝑔𝑔𝑅𝑅,𝑖𝑖,𝑗𝑗 = 𝑎𝑎� 𝑓𝑓,𝑖𝑖,𝑗𝑗 − 𝐶𝐶𝑖𝑖,𝑗𝑗 � + 𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗
𝑔𝑔𝐺𝐺,𝑖𝑖,𝑗𝑗 = 𝑎𝑎� 𝑓𝑓,𝑖𝑖,𝑗𝑗 − 𝐶𝐶𝑖𝑖,𝑗𝑗 � + 𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗
𝑔𝑔𝐵𝐵,𝑖𝑖,𝑗𝑗 = 𝑎𝑎� 𝑓𝑓,𝑖𝑖,𝑗𝑗 − 𝐶𝐶𝑖𝑖,𝑗𝑗 � + 𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗(4)where a is positive constant. If 𝑔𝑔𝑅𝑅,𝑖𝑖,𝑗𝑗 , 𝑔𝑔𝐺𝐺,𝑖𝑖,𝑗𝑗 and
𝑔𝑔𝐵𝐵,𝑖𝑖,𝑗𝑗 are less than 0, then these values must be set
to 0, respectively. If 𝑔𝑔𝑅𝑅,𝑖𝑖,𝑗𝑗 , 𝑔𝑔𝐺𝐺,𝑖𝑖,𝑗𝑗 and 𝑔𝑔𝐵𝐵,𝑖𝑖,𝑗𝑗 are
greater than U − 1, then these values must be set
to U − 1, respectively. The image composed ofthe pixel values 𝑔𝑔𝑅𝑅,𝑖𝑖,𝑗𝑗 , 𝑔𝑔𝐺𝐺,𝑖𝑖,𝑗𝑗 and 𝑔𝑔𝐵𝐵,𝑖𝑖,𝑗𝑗 is a celllike image of our method.

(a) Edge-extracted image (b) Distance-transformed image
Figure 4. Edge-extracted image and distance-transformed
image.

3.1. Experiments with different parametervalues
We visually assessed the change in appearance of celllike images as the value of the parameter D was changed
using Lenna image. The value of the parameter D was set
to 10, 15 and 20, and the value of the parameters r and awere set to 3 and 0.4, respectively. Cell-like images in thecase are shown in Fig. 5. The larger the value of theparameter D, the larger cell-like patterns were expressed.
We visually assessed the change in appearance ofcell-like images as the value of the parameter r waschanged using Lenna image. The value of the parameterr was set to 2, 3 and 4, and the value of the parameters D
and a were set to 15 and 0.4, respectively. Cell-like
images in the case are shown in Fig. 6. The larger thevalue of the parameter r, the larger the cell nuclei of celllike patterns were expressed.
We visually assessed the change in appearance ofcell-like images as the value of the parameter a waschanged using Lenna image. The value of the parameter

3. Experiments
We conducted two experiments. In the first experiment,
we visually confirmed the appearance of cell-like images
generated from Lenna image shown in Fig. 2 by changing
the values of the parameters. In the second experiment,
used in the experiments were 512 * 512 pixels and 256
gradation. For reference, the edge extracted image andthe distance-transformed image (D = 15) of Lenna image
are show in Fig. 4 (a) and (b), respectively.

Figure 2. Lenna image.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

297

Toru Hiraoka, Kohei Maeda(a) D = 10(b) D = 15(c) D = 20

Figure 5. Cell-like images for D = 10, 15 and 20.

Figure 8. Cell-like images.

4. Conclusions(a) r = 2(b) r = 3

We proposed a method to arrange cell-like patterns along
the edges of photographic images. We improved theconventional method 8 by using Euclidean distance fromthe edges of photographic images. We demonstrated theeffectiveness of our method through experiments using
various photographic images. The experimental results
showed that our method can express cell-like patterns
along the edges. In addition, the experimental results
showed that the size of cell-like patterns can be changed
by changing the values of the parameters in our method.
However, cell-like patterns were hard to occur in the finearea and in the black and white areas.
In future work, we will try to make it possible togenerate cell-like patterns in the areas where cell-likepatterns do not easily occur. And, we will try to apply our
method to videos and three-dimensional data.

(c) r = 4

Figure 6. Cell-like images for r = 2, 3 and 4.

(a) a = 0.2(b) a = 0.4(c) a = 0.6

Figure 7. Cell-like images for a = 0.2, 0.4 and 0.6.

a was set to 0.2, 0.4 and 0.6, and the value of theparameters D and r were set to 15 and 3, respectively.
Cell-like images in the case are shown in Fig. 7. The
larger the value of the parameter a, the more clearly celllike patterns were emphasized.

Acknowledgment. This work was supported by JSPS
KAKENHI Grant Number JP19-K12664.
References
1. P. Haeberli, Paint by numbers: abstract image
representations, ACM SIGGRAPH Computer Graphics,
24(4), 1990, pp.207-214.
2. D. D. Seligmann and S. Feiner, Automated generation of
intent-based 3D illustrations, ACM SIGGRAPH Computer
Graphics, 25(4), 1991, pp.123-132.
3. J. Lansdown and S. Schofield, Expressive rendering: areview of nonphotorealistic techniques, IEEE Computer
Graphics and Applications, 15(3), 1995, pp.29-37.
4. W. Qian, D. Xu, K. Yue, Z. Guan, Y. Pu and Y. Shi, Gourd
pyrography art simulating based on non-photorealistic
rendering, Multimedia Tools and Applications, 76(13),2017, pp.14559-14579.
5. D. Martin, G. Arroyo, A. Rodriguez and T. Isenberg, A
survey of digital stippling, Computers & Graphics, 67,
2017, pp.24-44.

3.2. Experiments with various photographic
images
We applied our method to six photographic images
shown in Fig. 3. Referring to the results of theexperiments in the previous section, the values of theparameters D, r and a were set to 15, 3 and 0.4,
respectively. Cell-like images generated from the six
photographic images are show in Fig. 8. In all cell-like
images, our method could express cell-like patterns along
the edges of photographic images. In addition, the celllike patterns generated by our method were expressed thecell membrane and cell nucleus more clearly than thoseof the conventional methods 8,9.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

298

A Method for Patterns

6. W. Qian, D. Xu, J. Cao, Z. Guan and Y. Pu, Aesthetic art
simulation for embroidery style, Multimedia Tools andApplications, 78(1), 2019, pp.995-1016.
7. T. Hiraoka, Generation of edge-enhancing labyrinth
images using inverse filter and two improved Laplacian
filters, ICIC Express Letters, 14(2), 2020, pp.121-127.
8. T. Hiraoka, M. Hirota, K. Inoue and K. Urahama,
Generating cell-like color images by inverse iris filter,
ICIC Express Letters, 11(2), 2017, pp.399-404.
9. T. Hiraoka, A method for emphasizing and aligning
patterns in cell-like images, ICIC Express Letters, 13(11),2019, pp.1031-1037.
10. H. Kobatake and S. Hashimoto, Convergence index filter
for vector fields, IEEE Transactions on Image Processing,
8(8), 1999, pp.1029-1038.
11. Z. Yu and K. Urahama, Iterative method for inverse
nonlinear image processing, IEICE Transactions on
Fundamentals, E97-A(2), 2014, pp.719-721.
12. The Robust Image Understanding Laboratory at Rutgers
University, Edge detection and image segmentation(EDISON) system, http://rci.rutgers.edu/˜meer/RIUL/
research/code/EDISON/doc/help.html, 2002.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

299

Generating Striped Animations by Inverse Line Convergence Index Filter
Toru Hiraoka

Department of Information Systems, University of Nagasaki
1-1-1, Manabino Nagayo-chou, Nishisonogi-gun, Nagasaki-ken, 815—2195, Japan
E-mail: hiraoka@sun.ac.jp
Ryosuke Takaki

Division of Computer Science, Graduate School of Regional Design and Creation, University of Nagasaki
1-1-1, Manabino, Nagayo-chou, Nishisonogi-gun, Nagasaki-ken, 815-2195, Japan
E-mail: mc220002@sun.ac.jp

Abstract
A non-photorealistic rendering method has been proposed for generating a striped image which is overlaid striped
patterns in a photograph. The conventional method generates the striped image by an iterative process using an
inverse line convergence index filter. When a striped animation is generated by converting each frame of a video by
the conventional method, flickering occurs in the generated striped animation. In this paper, we propose a method forgenerating a striped animation that has characteristic with less flicker from the video. The effectiveness of the
proposed method is investigated experimentally. As a result of the experiments, the proposed striped animation had
less flicker than the conventional striped animation.
Keywords: Non-photorealistic rendering, animation, striped pattern, inverse line convergence index filter

1. Introduction
Various non-photorealistic rendering methods for a
photograph have been proposed1,2,3,4,5,6. These methods
are used in a wide range of applications, for examples,
applications embedded in a personal computer and aportable terminal. Applying these methods to a video isto improve the visual appearance. A Study on nonphotorealistic rendering of the video has also been
conducted7, but the conventional method was been
known that flickering occurs in the non-photorealistic
rendering animation.
In this paper, we focus on the non-photorealistic
rendering method for generating a striped image6 fromthe photograph. Also we apply the conventional method
to the video. The striped image is overlaid the striped
patterns in the photograph as shown in Fig. 1. The
conventional method generates the striped image by an
iterative process using an inverse line convergence index(a) Lena image(b) Striped image
Fig. 1. Lena image and striped image

filter, and is characterized by the ability to automaticallygenerate the striped patterns in accordance with theshading and the edge of the photograph. When a stripedanimation is generated by converting each frames of thevideo by the conventional method, flickering occurs in
the generated striped animation. Therefore, we suppress
flicker by using the forward and backward frames of the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

300

Toru Hiraoka, Ryosuke Takaki

video. The effectiveness of the proposed method isinvestigated experimentally. By visually andquantitatively comparing the proposed striped animation
to the conventional striped animation, we show that the
proposed method can suppress flicker better than theconventional method.
2. Method
Let input pixel values of RGB on coordinates (𝑖𝑖, 𝑗𝑗) in 𝑜𝑜th frame of the video be 𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 , 𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗,𝑜𝑜 and 𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗,𝑜𝑜 (𝑖𝑖 =
1,2, … , 𝐼𝐼; 𝑗𝑗 = 1,2, … , 𝐽𝐽; 𝑜𝑜 = 1,2, … , 𝑂𝑂) , and let output
pixel values after processing with a line convergenceindex filter8 be 𝐿𝐿𝐿𝐿�𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 �, 𝐿𝐿𝐿𝐿�𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗,𝑜𝑜 � and 𝐿𝐿𝐿𝐿(𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗,𝑜𝑜 ).
The line convergence index filter is executed in thefollowing manner. Consider the straight line 𝑙𝑙𝑚𝑚,𝑜𝑜 (𝑚𝑚 =1,2, … , 𝑀𝑀) inclined 𝜃𝜃𝑚𝑚,𝑜𝑜 (= 0, 𝜋𝜋⁄𝑀𝑀 , 2𝜋𝜋⁄𝑀𝑀 , … , (𝑀𝑀 −1)𝜋𝜋/𝑀𝑀) degrees from the 𝑥𝑥-direction of the target pixel(𝑖𝑖, 𝑗𝑗) . Let the length of the straight line 𝑙𝑙𝑚𝑚,𝑜𝑜 in eachdirection around the target pixel (𝑖𝑖, 𝑗𝑗) be 𝑊𝑊1 pixels.
Consider the inclined rectangle which has the center at(𝑖𝑖, 𝑗𝑗) and the length of sides are 𝑊𝑊1 , 𝑊𝑊2 . Let 𝑁𝑁 be thenumber of pixels (𝑘𝑘, 𝑙𝑙) inside the rectangle (Conceptual
diagram for 𝑊𝑊1 , 𝑊𝑊2 and 𝑁𝑁 are shown in Fig. 2).
Compute a cosine of the angle between a vector
perpendicular to a straight line 𝑙𝑙𝑚𝑚,𝑜𝑜 from a neighboring
pixel (𝑘𝑘, 𝑙𝑙) and a vector (( 𝑠𝑠𝑘𝑘+2,𝑙𝑙+2,𝑜𝑜 + 𝑠𝑠𝑘𝑘+2,𝑙𝑙+1,𝑜𝑜 +𝑠𝑠𝑘𝑘+2,𝑙𝑙,𝑜𝑜 +𝑠𝑠𝑘𝑘+2,𝑙𝑙−1,𝑜𝑜 +𝑠𝑠𝑘𝑘+2,𝑙𝑙−2,𝑜𝑜 )−(𝑠𝑠𝑘𝑘−2,𝑙𝑙+2,𝑜𝑜 +𝑠𝑠𝑘𝑘−2,𝑙𝑙+1,𝑜𝑜 +𝑠𝑠𝑘𝑘−2,𝑙𝑙,𝑜𝑜 +𝑠𝑠𝑘𝑘−2,𝑙𝑙−1,𝑜𝑜 +𝑠𝑠𝑘𝑘−2,𝑙𝑙−2,𝑜𝑜 ),( 𝑠𝑠𝑘𝑘+2,𝑙𝑙+2,𝑜𝑜 +𝑠𝑠𝑘𝑘+1,𝑙𝑙+2,𝑜𝑜 +𝑠𝑠𝑘𝑘,𝑙𝑙+2,𝑜𝑜 +𝑠𝑠𝑘𝑘−1,𝑙𝑙+2,𝑜𝑜 +𝑠𝑠𝑘𝑘−2,𝑙𝑙+2,𝑜𝑜 ) −(𝑠𝑠𝑘𝑘+2,𝑙𝑙−2,𝑜𝑜 +𝑠𝑠𝑘𝑘+1,𝑙𝑙−2,𝑜𝑜 +𝑠𝑠𝑘𝑘,𝑙𝑙−2,𝑜𝑜 + 𝑠𝑠𝑘𝑘−1,𝑙𝑙−2,𝑜𝑜 + 𝑠𝑠𝑘𝑘−2,𝑙𝑙−2,𝑜𝑜 )) calculated from thedensity variation. Let the cosine of the angle be𝑐𝑐𝑚𝑚,𝑛𝑛,𝑜𝑜 (𝑛𝑛 = 1,2, … , 𝑁𝑁). The term 𝑠𝑠𝑖𝑖,𝑗𝑗,𝑜𝑜 is calculated as thefollowing equation.
𝑑𝑑𝑖𝑖,𝑗𝑗,𝑜𝑜 =
𝑠𝑠𝑖𝑖,𝑗𝑗,𝑜𝑜 =𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 +𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗,𝑜𝑜 +𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗,𝑜𝑜
3

1
𝑑𝑑
1+|𝑝𝑝| 𝑖𝑖,𝑗𝑗,𝑜𝑜+𝑝𝑝1𝑃𝑃∑𝑝𝑝=−𝑃𝑃
1+|𝑝𝑝|∑𝑃𝑃
𝑝𝑝=−𝑃𝑃.

.

Fig. 2. Conceptual diagram for 𝑊𝑊1 , 𝑊𝑊2 and 𝑁𝑁

With the maximum value of 𝐶𝐶𝑚𝑚,𝑜𝑜 in all straight lines 𝑙𝑙𝑚𝑚,𝑜𝑜in each pixel (𝑖𝑖, 𝑗𝑗), denote as 𝑔𝑔𝑖𝑖,𝑗𝑗,𝑜𝑜 . With the minimum
and maximum values of 𝑔𝑔𝑖𝑖,𝑗𝑗 in all pixels, denoted as
𝑔𝑔𝑚𝑚𝑚𝑚𝑚𝑚,𝑜𝑜 and 𝑔𝑔𝑚𝑚𝑚𝑚𝑚𝑚,𝑜𝑜 , respectively. Transform 𝑔𝑔𝑖𝑖,𝑗𝑗,𝑜𝑜 to ℎ𝑖𝑖,𝑗𝑗,𝑜𝑜and thus to the pixel values had from 0 to 255 by
thefollowing equation.
ℎ𝑖𝑖,𝑗𝑗,𝑜𝑜 = 255

1

𝑁𝑁

.

(4)

𝐿𝐿𝐿𝐿�𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 � , 𝐿𝐿𝐿𝐿�𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 � and 𝐿𝐿𝐿𝐿(𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 ) are the samevalue as ℎ𝑖𝑖,𝑗𝑗,𝑜𝑜 .
(𝑡𝑡)

(𝑡𝑡)

(𝑡𝑡)

Compute the pixel value 𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 , 𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗,𝑜𝑜 and 𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗,𝑜𝑜 byusing the inverse line convergence index filter as(𝑡𝑡)(𝑡𝑡−1)(𝑡𝑡−1)(𝑡𝑡)(𝑡𝑡−1)(𝑡𝑡−1)

𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 = 𝑎𝑎(𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 − 𝐿𝐿𝐿𝐿(𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 )) + 𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 . (5)(𝑡𝑡)(𝑡𝑡−1)(𝑡𝑡−1)

𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗,𝑜𝑜 = 𝑎𝑎(𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗,𝑜𝑜 − 𝐿𝐿𝐿𝐿(𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗,𝑜𝑜 )) + 𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗,𝑜𝑜 . (6)𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗,𝑜𝑜 = 𝑎𝑎(𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗,𝑜𝑜 − 𝐿𝐿𝐿𝐿(𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗,𝑜𝑜 )) + 𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗,𝑜𝑜 . (7)(1)

Where 𝑎𝑎 is a positive constant and 𝑡𝑡 is the number of
(0)
(0)
(0)
iterations. Let the initial value 𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 , 𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗,𝑜𝑜 and 𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗,𝑜𝑜

(2)

(𝑡𝑡)

(𝑡𝑡)

be 𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 , 𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗,𝑜𝑜 and 𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗,𝑜𝑜 , respectively. 𝑓𝑓𝑅𝑅,𝑖𝑖,𝑗𝑗,𝑜𝑜 , 𝑓𝑓𝐺𝐺,𝑖𝑖,𝑗𝑗,𝑜𝑜

Where 𝑃𝑃 is a positive constant. Next, compute 𝐶𝐶𝑚𝑚,𝑜𝑜 thatis the average of the absolute value of 𝑐𝑐𝑚𝑚,𝑛𝑛,𝑜𝑜 of 𝑁𝑁
neighboring pixels (𝑘𝑘, 𝑙𝑙) in each straight line 𝑙𝑙𝑚𝑚,𝑜𝑜 as thefollowing equation.
𝐶𝐶𝑚𝑚,𝑜𝑜 = ∑𝑁𝑁
𝑛𝑛=1�𝑐𝑐𝑚𝑚,𝑛𝑛,𝑜𝑜 �.

𝑔𝑔𝑖𝑖,𝑗𝑗,𝑜𝑜 −𝑔𝑔𝑚𝑚𝑚𝑚𝑚𝑚,𝑜𝑜

𝑔𝑔𝑚𝑚𝑚𝑚𝑚𝑚,𝑜𝑜 −𝑔𝑔𝑚𝑚𝑚𝑚𝑚𝑚,𝑜𝑜(𝑡𝑡)

and 𝑓𝑓𝐵𝐵,𝑖𝑖,𝑗𝑗,𝑜𝑜 set to 0 if their values are less than 0, and set
to 255 if their values are greater than 255.
Finally, the 𝑁𝑁 striped images are obtained after
processing of the inverse line convergence index filter of𝑇𝑇 generated from these striped images.

(3)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

301

Generating Striped Animations by

3. Experiments
We applied the proposed method to the Yuzenzome
video9 which consists of 703 frames, 30 frames / second,352 ∗ 240 pixels and 256 tone. The 100, 101, 387 and388th frames of the Yuzenzome video are shown in Fig.
3. The 387 and 388 th ftames are the scene change
frames. In the following experiments, referring to theliterature 6), we set the values of the parameters 𝑀𝑀, 𝑊𝑊1 ,𝑊𝑊2 , 𝑎𝑎 and 𝑇𝑇 to 8, 4, 4, 0.4 and 30, respectively.
First, we visually compared the proposed stripedanimation to the conventional striped animation. The
100, 101, 387 and 388th frames of the conventional
striped animation and the proposed striped animation are(a) 100th(a) 100th(c) 387th(d) 388th
Fig. 5. The frames of the proposed animation
shown in Fig. 4 and Fig. 5, respectively. We set the valueof the parameter 𝑃𝑃 to 3. Observing these animations, theproposed striped animation were suppressed flicker morethan the conventional striped animation. Observing Fig.
4 and Fig. 5, the proposed striped animation had lesschanges in the striped patterns than the conventional
striped animation.
Next, we quantitatively compared the proposed
striped animation to the conventional striped animation.
We calculated the average of the absolute difference
value of pixel values between front and rear frames ofthese animations. As the average become smaller,
flickering is less. The average of the conventional stripedanimation is shown in Table 1. On the other hand, theaverage of the proposed striped animation is shown in
Table 2, when we changed the values of parameter 𝑃𝑃
from 1 to 10 . Observing Table 2, as the values ofparameter 𝑃𝑃 became bigger, the average became small.
Observing Table 1 and Table 2, the average of the
proposed striped animation were smaller than the averageof the conventional striped animation. Thus, the proposed
method can suppress flicker better than the conventional
method.
Lastly, we compared the calculation time of the
proposed method to that of the conventional method. The
calculation time of the conventional method is shown in
Table 3. On the other hand, the calculation time of the
proposed method is shown in Table 4, when we changed
the values of parameter 𝑃𝑃 from 1 to 10. The calculation
time in Table 3 and Table 4 is one iteration calculation(b) 101th(c) 387th(d) 388th
Fig. 3. The frames of the Yuzenzome video(a) 100th(b) 101th(b) 101th(c) 387th(d) 388th
Fig. 4. The frames of the conventional striped animation© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

302

Toru Hiraoka, Ryosuke Takakiwith the conventional method on the calculation time.
However, considering real-time use on portable terminals,
it is necessary to speed up the proposed method.
Thecomputing environment of the experiment is
Windows 7 Professional for OS, 3.40 GHz for CPU,
8.00 GB for RAM and C language.

Table 1. The average of the conventional striped animation
Average23.856
Table 2. The average of the proposed striped animation whenthe value of the parameter 𝑃𝑃 is changed
𝑃𝑃 Average
1

20.4032
3

20.26420.1904
520.151
20.1186
7

20.10020.0878
920.07520.06710

20.062

4.

We proposed a method for generating a stripedanimation that has characteristic with less flicker fromthe video. The effectiveness of the proposed method wasinvestigated experimentally. As a result of theexperiments, the proposed striped animation had lessflicker than the conventional striped animation. The
future tasks are to speed up the proposed method and to
apply the proposed method to other videos.
Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP19K12664.

Table 3. The calculation time of the conventional method[second]
Average

References
1. D. Nakagawa, K. Yamaguchi, K. Muraoka, and N. Chiba,
Generating Pen and Ink Illustrations from Gray-scale
Images : Stippling and Line Drawing Style, Journal of
IIEEJ. 30(4), 2001, pp. 350-361.
2. J. Sugita and T. Takahashi, A Generation Method forColored-Paper-Mosaic Images Based on Hierarchial
Poisson Disk Sampling, Journal of IIEEJ. 36(4), 2007, pp.
407-416.
3. Y. Takada and K. Urahama, Dendritic Line Images
Generated from Distance Transform and its Application to
NPR, Journal of IIEEJ. 38(5), 2009, pp. 791-794.
4. T. Saito, Non-photorealistic Rendering, Journal of IIEEJ.
39(6), 2010, pp. 837-839.
5. R. Ando and R. Tsuruno, An Interactive Brush Stroke
Synthesis Using Exemplar Images, Journal of IIEEJ.
40(4), 2011, pp. 578-586.
6. T. Hiraoka and K. Urahama, Generating Striped Color
Images by Inverse Line Convergence Index Filter, IIEEJTransactions on Image Electronics and Visual Computing.
2(2), 2014, pp. 190-194.
7. B. J. Meier, Painterly Rendering for Animation,
Proceedings of ACM SIGGRAPH 1996, 1996, pp. 477-484.
8. H. Kobatake and S. Hashimoto, Convergence Index Filter
for Vector Fields, IEEE Transactions Image Processing. 8,
1999.
9. The Video-Database-Working Group of PRMU, Video
Database for Evaluating Video Processing, http://www.
nii.ac.jp/dsc/idr/video/VDB/documentary.html, Yuzenzome, 2002.

4.377
Table 4. The calculation time of the proposed method when thevalue of the parameter 𝑃𝑃 is changed [second]

𝑃𝑃

Average

1
2

4.381
4.390

3
4

4.394
4.399

5
6

4.401
4.405

7
8

4.410
4.413

9
10

4.420
4.426

Conclusion[second]. Observing Table 4, the calculation time became
big as the values of parameter 𝑃𝑃 became bigger, but there
are no big differences in these calculation times.
Observing Table 3 and Table 4, there was no big
difference in the calculation time between the proposed
method and the conventional method. Thus, the proposed
method is considered to be an effective method compared© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

303

Estimating Home Location of Foreigners in Japan
Using Photograph Location
Masaharu Hirota

Department of Information Science, Okayama University of Science
1-1 Ridaicho, Kita-ku, Okayama-shi, 700-0005, Japan
Tetsuya Oda

Department of Information and Computer Engineering, Okayama University of Science
1-1 Ridaicho, Kita-ku, Okayama-shi, 700-0005, Japan
E-mail: hirota@mis.ous.ac.jp, oda@ice.ous.ac.jp

Abstract
The attributes of travelers such as home location and age could be useful for many applications such as information
recommendation and targeted advertisement. However, this information is not accessible in most Web services
because users do not reveal them. We propose a method to estimate a foreign tourist's home location based on eachregion's tendency, in which tourists from a region tend to visit certain places when traveling abroad. The feature forthe estimation uses the frequency of photograph location in a geohash. In this paper, we use foreigners in Japan as acase study. We evaluate the performance of our proposed method by using photographs obtained from their user
accounts on Flickr.
Keywords: home location estimation, user trajectory, machine learning, Flickr.

To estimate the user's home location, we can use thefrequency of geo-tags annotated to the user's content. For
example, a simple way is to regard the area with the most
geo-tags indicating a particular area as the user's homelocation. However, this method's limitation is that it
cannot be applied when the analyst only has data for a
specific region. For example, it is impossible to estimate
the country of residence outside of Japan if the analyst
only has data posted in Japan. Therefore, a method toestimate the home location from a dataset for a particularregion only is necessary.
This paper proposes a method to estimate the user's
home location based on geo-tags visiting from abroadusing contents obtained from a specific area. Our
hypothesis is that travelers in different home locations
visit different areas. Some places are widely popular
tourist spots, like the Eiffel Tower in France and the

1. Introduction
With the rapid growth of social media websites such as
Flickr or Twitter, it has become common for tourists to
upload geo-tagged content from their trips. The contents
are recommended to the other users based on user
information and its friendship network. Therefore, thecontent strongly influences tourists' decisions, such as
their travel destinations and routes, because tourists may
refer to this information when they plan their trips.
The user attribute of users on social media websites(e.g., home location, age, and gender) is important for
many applications. The user's home location is especiallyused for targeting advertisement, information
recommendation, and user behavior analysis. However,many users on social media websites do not provide
information on the home location. It is also possible thatthe user provides false information.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

304

Masaharu Hirota, Tetsuya Oda

3. Proposed Method

Leaning Tower of Pisa in Italy. However, when travelers
go abroad, tourists from a particular country may beinterested in specific places, not necessarily conventional
tourist spots. These are places associated with a specificregion, such as a famous movie in their origin region, arestaurant whose owners are from their region, or a place
featured on a local TV show. If we can extract thetendency of the user's visitation by home location, we
assume that it is possible to estimate the user's homelocation from geo-tags.
In this paper, we estimate the region of residence ofusers who come to Japan from abroad. We used geotagged photographs obtained from Flickr. We evaluate
the performance of our proposed method by estimating
the home location of users in Flickr. Also, we discussed
the effect of area size of geohash on the estimation
performance.
This paper's remainder is organized as follows: Section
2 describes the work related to this topic. Section 3
presents our proposed method for estimating the user's
home location. Section 4 presents the experimentconducted to evaluate the performance of our proposed
method. Finally, Section 5 concludes our work anddiscuss results and areas of future work.

Our proposed method consists of two steps as follows.
(i). Extracting features based on occurrence of users'
visits from user's contents.
(ii). Estimating the home location of a tourist from theirtrajectory using a machine learning algorithm.
3.1. Feature extraction
We describe the generation of features for machinelearning.
In this study, we use geohash generated from a
photographic location as a user's feature. A geohash is astring representation of a geographical area, created byencoding latitude and longitude. Geohash converts apoint expressed in latitude and longitude into a
geographical area, which reduces the accuracy ofrepresenting a geographical point. In addition, dependingon the string's length of geohash, it is possible to control
the degree of accuracy loss. For example, the latitude andlongitude of Okayama Castle (latitude, longitude) =(34.66568196, 133.93593695) is converted to 'wypjpy9'
in 7-characters geohash.
To create features, we count the number of photographs
taken in each geohash for each user. Then, we delete
geohashs with fewer than ten users who took thephotographs. Finally, we apply L2 normalization to thefeatures.

2. Related WorkA wide range of home location methods estimating fromsocial media websites content has been proposed inrecent years.
Hironaka et al. used data from Twitter to analyze
users’ home location based on friend relationships1. Hu
et al. proposed a method to estimate home location fromsparse and noisy Twitter data within 100 by 100-metersquares at high accuracy using users’ trajectories in theirhome country2. Jurgens et al. evaluated several methods
for geo-location prediction using data from Twitter3.
The above methods focus on estimating users' homelocation of social media websites by using content posted
in their home country. These researches estimate thehome location of users within the posted range of the data.
Therefore, it is difficult to estimate the home location ofusers visiting from other regions. Our previous research4
proposed a new method for home location inference byanalyzing foreign tourists' tendencies in a different region
from their home country. In this research, we proposed amethod for estimating the user's home location andevaluating the method's performance.

3.2. Estimating Home Location
In this research, we use Random Forest5 (RF) to estimate
the user's home location based on feature vectors
obtained from Section 3.1. This algorithm is a multi-classclassifier algorithm in supervised learning. In thisresearch, we regard the predicted class from the feature
vector obtained from a user's photostream as the user's
estimated home location.
4.

Experiment

In this paper, we evaluate the performance of our
proposed method through an evaluation experimentbased on classification. We describe the experimentconditions of the dataset and the evaluation criteria.
As described in Section 3.1, the geohash allows
controlling the degree to which latitude and longitude arequantized depending on the string's length to beconverted. Therefore, our experiment evaluates the effectof geohash string length on classification performance.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

305

Estimating Home Location of

4.1. Dataset

Table 1. Evaluation result.

Geohash
3
4
5
6
7
8

In this section, we describe the dataset used forevaluating our proposed method. We used Japan as aparticular country for this experiment. We estimate thehome location of users who visit from other regions andusing the photographs taken in Japan.
The evaluation experiments used photographs obtained
from Flickr. The photographs include the metadata:
latitude, longitude, and home location. The homelocation is the text data where the user in Flickr describes
their home location.
Because Flickr's home location field is an open text field,
the data in it varies widely. Therefore, we mapped all theabbreviations and significant states/cities to the same
country name (i.e., “usa”, “u.s.a” and “Dallas, Texas”
were mapped to “usa”).
We excluded photographs where either the latitude or
the longitude does not have a value with a precision lessthan or equal to the third decimal place. We also exclude
users who describe more than one home location in theirphotographs and users who have less than ten
photographs.
In this research, we used three home location: Taiwan,
America, and United Kingdom. This reason is that the
number of users in which we obtained photographs from
Flickr is Taiwan, USA, and UK. Those regions are the
top three regions in the number of users in our
photographs obtained from Flickr. Therefore, our method
tackles three-class classification.
Consequently, the number of photographs used in thisexperiment was 461,454 and the number of users is 1,683.
Also, we randomly split into a ratio of 9:1 for train andtest data.

Precision
0.499
0.543
0.716
0.537
0.597
0.570

Recall
0.494
0.461
0.529
0.527
0.522
0.513

F1-score
0.463
0.426
0.455
0.479
0.491
0.515

Accuracy
0.552
0.554
0.581
0.625
0.680
0.649

4.3. Experimental conditions
This section describes the procedure used for theclassifiers to estimate home locations.
This experiment used entropy and Gini impurity for a
split of nodes in RF. In addition, the hyperparameterssuch as max nodes and the number of trees were searchedusing Optuna6 with five cross-validations, which is a
software for automatically optimizing hyperparameters.
We used the parameters with the highest accuracy
measured through this experiment. In addition, we used
the Python software scikit-learn7 for the implementationof RF.
4.4. Evaluation results
This section describes and discusses the evaluation
results of classifying home locations and the effects on
string length of geohash.
Table 1 shows the evaluation results of Accuracy andmacro average of Recall, Precision, and F-score. The firstcolumn represents the string length of geohash. The bestvalue of the length in Precision and Recall is 5. In thiscase, the UK's Precision value was less than 0.5 whenother parameters were used, but it was 1.0 when thisparameter was used. The Recall values for the other two
regions were also high. Therefore, the result of thisparameter may overfit to the UK. Also, in Table 1, thelarger the value of the length of geohash, the larger thevalue of Accuracy. The proposed method may have thebest classification performance when the length of thegeohash string is 7.
Table 1 shows that the proposed method works wellbecause some performance is obtained for each
evaluation criterion.

4.2. Evaluation criteria
We used the following widely used performancemeasures for classification: Accuracy, Recall, Precision,and F-score. In this research, our classification is multiclass classification. Therefore, we used the macro
average of them.

5. Conclusion
In this paper, we proposed a method to estimate the homelocation using the user’s trajectory. Our approach uses
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

306

Masaharu Hirota, Tetsuya Oda

the tendency of users from the same location to travel tothe same places. Also, our approach uses the geohashobtained from the photograph location. Experimental
results showed that our classifier could estimate thecandidates of home location.
Our future work will include a more detailed experiment.
This paper is not enough in the analysis of the homelocation predicted by our proposed method. Also, theexperiment in this paper uses a few photographs. As aresult, the experiment result has the limitation of thevalidity of our proposed method's effectiveness.
Therefore, another further study may make the data
which includes more number of a dataset.
Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP19K20418 and Grant for Promotion of OUS
Research Project (OUS-RP-20-3).
References
1. Shiori Hironaka, Mitsuo Yoshida, Kyoji Umemura, User’s
Centrality Analysis for Home Location Estimation.
Proceedings of IEEE/WIC/ACM International Conferenceon Web Intelligence, 14–17, 2019.
2. Tianran Hu, Jiebo Luo, Henry Kautz and Adam Sadilek,
Home Location Inference from Sparse and Noisy Data:
Models and Applications. Proceedings of IEEE 15th
International Conference on Data Mining Workshops,
Atlantic City, 14-17, 2015.
3. David Jurgens, Tyler Finethy, James McCorriston, Yi Tian
Xu, Derek Ruths, Geolocation Prediction in Twitter Using
Social Networks: A Critical Analysis and Review ofCurrent Practice. Proceedings of the Ninth International
AAAI Conference on Web and Social Media, 26-29, 2015.
4. K. Lugasi, M.Hirota, Inferring Home Location of Foreign
Tourists Based on Travel Routes Extracted from Social
Media Sites, The 2020 International Conference on
Artificial ALife and Robotics, 2020.
5. L. Breiman, “Random forests,” Machine learning, vol. 45,
no. 1, pp. 5–32, 2001.
6. T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama,
Optuna: A next-generation hyperparameter optimization
framework, 2019.
7. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B.
Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss,
V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M.
Brucher, M. Perrot, and E. Duchesnay, Scikit-learn:
Machine learning in Python, Journal of Machine Learning
Research, vol. 12, pp. 2825–2830, 2011.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

307

A Proposal of Online Map-matching Based
Trajectory Compression Algorithm Using Road Networks
Shota Iiyama, Masaharu Hirota

Department of Information Science, Okayama University of Science
1-1 Ridaicho, Kita-ku, Okayama-shi, 700-0005, Japan
Tetsuya Oda

Department of Information and Computer Engineering, Okayama University of Science
1-1 Ridaicho, Kita-ku, Okayama-shi, 700-0005, Japan
E-mail: i17i005is@ous.jp, oda@ice.ous.ac.jp, hirota@mis.ous.ac.jp

Abstract
As the data size of GPS logs increases, the amount of data transferred from mobile devices to the server, and thecomputational cost of analysis of GPS logs increase. One of the solutions to these problems is to compress the GPS
logs. However, it is difficult to compress a sparse GPS log while preserving the feature points in the GPS log, suchas the user's movement speed, the shape of the GPS log trajectory, and the direction of movement of the GPS log.
This study proposes an online compression algorithm for GPS logs that maintains the compression rate while
preserving GPS logs' feature points. Our proposed method compresses GPS logs by using information from a road
network to identify roads traveled by the user. We evaluate the performance of this method using the GPS data of thebus.
Keywords: GPS, compression algorithm, road network, characteristics of user movement.

algorithm for compressing GPS logs moving on the road
network while preserving the trajectory's feature points.

1. Introduction
The number of GPS logs that record users' activities
increases. Therefore, the data transferred from mobile
devices to the server is a critical problem. Anotherproblem is that the processing speed of analysis of GPS
logs decreases when analyzing the GPS logs.
One solution to these issues is to compress the GPS logs.
Compressing the GPS logs reduces the amount of themtransmitted from the mobile device to the server. Also,
adequately compressed GPS logs reduce the amount ofdata processing for analysis.
However, if the GPS log compression lost feature points,
the analysis's performance may be worse than theuncompressed. Therefore, it is important to preserve as
much of the features as possible. This paper proposes an

2. Related work
The method for GPS log compression without using theroad network are Douglas-Peucker Algorithm (DPalgorithm)1 and top-down time-ratio algorithm (TD-TR
algorithm)2. These methods focus on the shape of the
GPS log and compress the GPS log.
Also, the methods for GPS log compression using theroad network include Schmid et al.'s method3. Thismethod is a compression method for GPS logs of users
moving on roads in urban areas. In this method, mapmatching is applied to the GPS logs to identify the roads
moved by the user. Next, the method uses compression© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

308

Shota Iiyama, Tetsuya Oda, Masaharu Hirota

Figure 1 Map-matching.

Figure 1 Map-matching.

Figure 2 Synchronized Euclidean Distance.

points the GPS logs obtained at intersections and bus
stops on the roads moved by the user.
Our method compresses GPS logs not only by focusing
on the shape, but the features of movement.
3. Method
Our compression method employs three methods topreserve the feature points of GPS logs: Opening
Window Compression Algorithm (OW)2, CriticalPoint
Algorithm (CP)4, and StayPoint (SP)5. OW is a shapeoriented compression algorithm. CP is a compression
algorithm that focuses on changes in velocity anddirection of motion. SP is the point at which the user wasstaying. Our method makes a compression result by
combining each of the OW, CP, and SP extracted fromthe GPS logs.

Figure 3 Opening Window Compression Algorithm.

3.2. Opening Window Compression Algorithms
First, we describe the SED that we uses in OW. The
SED is the value of the distance between point 𝑃𝑃′ 𝑖𝑖 of the
GPS log after compression and point 𝑃𝑃 of the GPS log
before compression, considering the time information.
We use the method of Meratnia et al.2 to obtain the SED.
Figure 2 shows an example of SED. The 𝑃𝑃𝑖𝑖 shows one of
the GPS points in the GPS log. 𝑃𝑃′𝑖𝑖 is the point at which
𝑃𝑃𝑖𝑖 is approximated on the line 𝑃𝑃𝑠𝑠 to 𝑃𝑃𝑒𝑒 . 𝑥𝑥′𝑖𝑖 and 𝑦𝑦′𝑖𝑖 arefound by the following equations.
(2)∆𝑒𝑒 = 𝑡𝑡𝑒𝑒 − 𝑡𝑡𝑠𝑠 ,∆𝑖𝑖 = 𝑡𝑡𝑖𝑖 − 𝑡𝑡𝑠𝑠
∆𝑖𝑖
(𝑥𝑥 − 𝑥𝑥𝑠𝑠 )(3)𝑥𝑥 ′ 𝑖𝑖 = 𝑥𝑥𝑒𝑒 +∆𝑒𝑒 𝑒𝑒∆𝑖𝑖(𝑦𝑦 − 𝑦𝑦𝑠𝑠 )(4)
𝑦𝑦 ′ 𝑖𝑖 = 𝑦𝑦𝑒𝑒 +∆𝑒𝑒 𝑒𝑒OW defines the 𝑎𝑎𝑎𝑎𝑎𝑎ℎ𝑜𝑜𝑜𝑜 and 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓. The 𝑎𝑎𝑎𝑎𝑎𝑎ℎ𝑜𝑜𝑜𝑜 𝑝𝑝𝑠𝑠 isthe first point in the GPS log. The 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 𝑝𝑝𝑒𝑒 is the third
point from the beginning of the GPS log. The SED is
calculated for the GPS point between the 𝑎𝑎𝑎𝑎𝑎𝑎ℎ𝑜𝑜𝑜𝑜 and the𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓, respectively. If there are no points at which theSEDs are above the threshold, the 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 is changed to thenext GPS point. Otherwise, the point with the highest
SED among the points whose SED are above thethreshold is the compressed point and the next 𝑎𝑎𝑎𝑎𝑎𝑎ℎ𝑜𝑜𝑜𝑜.

3.1. Map-matching
Our method uses the offline map-matching method
proposed by Brakatsoulas et al.6. In our method, we
extend this method to apply to online data.
Figure 1 shows the example of a network. 𝐶𝐶1 , 𝐶𝐶2 , and𝐶𝐶3 in Figure 1 represent the vectors from the start to theend node of the road. A node represents an intersection.
This method finds the candidate roads 𝐶𝐶1 and 𝐶𝐶2 andselects one of them using equation (1) for the point 𝑃𝑃𝑖𝑖 .
𝑛𝑛𝛼𝛼(1)𝑆𝑆𝑝𝑝𝑖𝑖 = 𝜇𝜇𝑑𝑑 − 𝑎𝑎 ⋅ 𝐷𝐷(𝑃𝑃𝑖𝑖 , 𝐶𝐶𝑖𝑖 )𝑛𝑛𝑑𝑑 + 𝜇𝜇𝛼𝛼 − cos�𝛼𝛼𝑖𝑖,𝑗𝑗 �𝜇𝜇𝑑𝑑 , 𝜇𝜇𝛼𝛼 , 𝑛𝑛𝑑𝑑 , and 𝑛𝑛𝛼𝛼 are constants. 𝐶𝐶𝑗𝑗 is a candidate roadin 𝑃𝑃𝑖𝑖 . 𝐷𝐷�𝑃𝑃𝑖𝑖 , 𝐶𝐶𝑗𝑗 � is the geographical distance between 𝑃𝑃𝑖𝑖and 𝐶𝐶𝑗𝑗 . 𝛼𝛼𝑖𝑖,𝑗𝑗 is the difference between the angles of 𝑙𝑙𝑖𝑖 and𝐶𝐶𝑗𝑗 . 𝑙𝑙𝑖𝑖 is the vector from 𝑃𝑃𝑖𝑖−1 to 𝑃𝑃𝑖𝑖 . cos�𝛼𝛼𝑖𝑖,𝑗𝑗 � is the cosine
value of 𝛼𝛼𝑖𝑖,𝑗𝑗 . Among the candidate roads, the road withthe highest value of 𝑆𝑆𝑝𝑝 𝑖𝑖 is the destination of 𝑃𝑃𝑖𝑖 .

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

309

A Proposal of Online

𝑆𝑆𝑆𝑆(𝑃𝑃𝑖𝑖 , 𝑃𝑃𝑖𝑖−1 ) calculates the difference in speed between
𝑃𝑃𝑖𝑖 and 𝑃𝑃𝑖𝑖−1 .𝐴𝐴𝐴𝐴(𝑃𝑃𝑖𝑖 , 𝑃𝑃𝑖𝑖−1 ) calculates the angle difference
between 𝑃𝑃𝑖𝑖 and 𝑃𝑃𝑖𝑖−1 . 𝑆𝑆𝑆𝑆 is the threshold for thedifference in velocity between 𝑃𝑃𝑖𝑖 and 𝑃𝑃𝑖𝑖−1 . Ar is thethreshold for the angle difference between 𝑃𝑃𝑖𝑖 and 𝑃𝑃𝑖𝑖−1 .

The process of computing the SED of a GPS point
between each 𝑎𝑎𝑎𝑎𝑎𝑎ℎ𝑜𝑜𝑜𝑜 and 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 is repeated. This
process is repeated until the OW terminates when GPS
positioning is finished. Figure 3 shows an example of acompressed GPS log with OW applied.
The difference between our proposed OW and theoriginal OW is how to choose 𝑎𝑎𝑎𝑎𝑎𝑎ℎ𝑜𝑜𝑜𝑜 and 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 . In
original OW, where an 𝑎𝑎𝑎𝑎𝑎𝑎ℎ𝑜𝑜𝑜𝑜 is the first point in the
GPS log, and 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 is the third point from the GPS log's
beginning. On the other hand, our method uses the resultsof map-matching to select 𝑎𝑎𝑎𝑎𝑎𝑎ℎ𝑜𝑜𝑜𝑜 and 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓.

3.5. Integrated processing of compression points

If there is SP or CP compression point within a certain
distance from the OW compression points, our method
deletes the OW compression points. This reason iscompression points by OW preserve only the shapefeatures, but the SP or CP preserve velocity or state ofstaying in addition to the shape features. The proposed
method regards the points obtained as a result of thisprocess as compressed points.

3.3. StayPoint

The StayPoint represents a point where the user was in arange for a amount of time. A GPS log that becomes a
StayPoint satisfies the following conditions.
(5)𝑚𝑚 < 𝑖𝑖 ≤ 𝑛𝑛, 𝐼𝐼𝐼𝐼𝐼𝐼(𝑃𝑃𝑚𝑚 , 𝑃𝑃𝑛𝑛 ) ≥ 𝑇𝑇𝑟𝑟
(6)
𝐷𝐷(𝑃𝑃𝑚𝑚 , 𝑃𝑃𝑖𝑖 ) ≤ 𝐷𝐷𝑟𝑟 , 𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷(𝑃𝑃𝑚𝑚 , 𝑃𝑃𝑛𝑛+1 ) > 𝐷𝐷𝑟𝑟
𝑃𝑃𝑖𝑖 is a point of trajectory in the GPS log. 𝐷𝐷(𝑃𝑃𝑚𝑚 , 𝑃𝑃𝑖𝑖 ) is
the distance between the two points 𝑃𝑃m and 𝑃𝑃𝑖𝑖 based on
the Hubeny formula. 𝐷𝐷𝑟𝑟 is the threshold for the distance
between 𝑃𝑃𝑚𝑚 and 𝑃𝑃𝑖𝑖 . 𝐼𝐼𝐼𝐼𝐼𝐼(𝑃𝑃𝑚𝑚 , 𝑃𝑃𝑛𝑛 ) is the difference in time
between 𝑃𝑃𝑚𝑚 and 𝑃𝑃𝑛𝑛 . 𝑇𝑇𝑟𝑟 is the threshold for the differencein time between 𝑃𝑃𝑚𝑚 and 𝑃𝑃𝑛𝑛 . From the GPS log thatsatisfies these equations, we calculate the StayPoint. Let
StayPoint be 𝑠𝑠 , then 𝑠𝑠 = (𝑥𝑥, 𝑦𝑦, 𝑡𝑡𝑎𝑎 , 𝑡𝑡𝑙𝑙 ) . 𝑠𝑠. 𝑥𝑥 and 𝑠𝑠. 𝑦𝑦
represent the coordinates of 𝑠𝑠. 𝑠𝑠. 𝑡𝑡𝑎𝑎 represents the start
time of the user's stay at the StayPoint. 𝑠𝑠. 𝑡𝑡𝑙𝑙 represents thetime when the user ends his stay at the StayPoint. 𝑠𝑠. 𝑥𝑥,
𝑠𝑠. 𝑦𝑦, 𝑠𝑠. 𝑡𝑡𝑎𝑎 , and 𝑠𝑠. 𝑡𝑡𝑙𝑙 are computed by following equations.
𝑛𝑛
𝑛𝑛
𝑃𝑃𝑖𝑖 . 𝑦𝑦
𝑃𝑃𝑖𝑖 . 𝑥𝑥(7), 𝑠𝑠. 𝑦𝑦 = �
𝑠𝑠. 𝑥𝑥 = �
|𝑃𝑃|
|𝑃𝑃|
𝑖𝑖=𝑚𝑚

4.

Experiment

We evaluate the performance of our method for the GPS
logs compression preserving the features of the GPS logs
while maintaining the Compression Rate. We used the
GPS log of buses in Okayama Prefecture, Japan. The
number of GPS logs is 297. The average number of GPS
points in the GPS log is about 185.9. The average
positioning interval of the GPS logs is about 18.3 seconds.
Also, map-matching uses the road network of Okayama
Prefecture obtained from OpenStreetMap.
This experiment uses three evaluation criteria: SED
Error, Speed Error, and Compression Rate. The SED
Error is the difference between the shape of the GPS log
before and after compression. This value is the medianand average of the SED calculated from the GPS logs
before and after compression. Speed Error is thedifference between the speed of the GPS log before andafter compression. Compression Rate is the compression
rate of the GPS logs
The proposed method parameter is the threshold of theSED, which is used to find the compression point in the
OW. We changed the threshold of the SED by ten from30 to 200. For the GPS logs compressed by the proposed
method, we evaluate them by each evaluation criterion.

𝑖𝑖=𝑚𝑚

𝑠𝑠. 𝑡𝑡𝑎𝑎 = 𝑃𝑃𝑚𝑚 . 𝑡𝑡𝑚𝑚 , 𝑠𝑠. 𝑡𝑡𝑎𝑎 = 𝑃𝑃𝑛𝑛 . 𝑡𝑡𝑛𝑛
(8)
Our method extracts the StayPoint from the GPS log
and preserves the state of staying at one point in the GPSlog before compression. In our method, 𝑠𝑠 is represented
by 𝑠𝑠𝑠𝑠 = (𝑥𝑥, 𝑦𝑦, 𝑡𝑡𝑎𝑎 ) and 𝑠𝑠𝑠𝑠 = (𝑥𝑥, 𝑦𝑦, 𝑡𝑡𝑙𝑙 ) to describe thesame format as the traditional GPS log.

3.4. CriticalPoint Algorithm
CP is a compression method that focuses on the speed
and direction of the user's movement. At two consecutive
points 𝑃𝑃𝑖𝑖 , 𝑃𝑃𝑖𝑖−1 in the GPS log, the compression point
satisfies either of the following conditions.
𝑆𝑆𝑆𝑆(𝑃𝑃𝑖𝑖 , 𝑃𝑃𝑖𝑖−1 ) > 𝑆𝑆𝑆𝑆, 𝐴𝐴𝐴𝐴(𝑃𝑃𝑖𝑖 , 𝑃𝑃𝑖𝑖−1 ) > 𝐴𝐴𝐴𝐴
(9)4.1. Experiment Results and Discussion
Figure 4, Figure 5, and Figure 6 show the evaluation
results of SED Error, Speed Error, and Compression Rateof the GPS logs compressed by our method. Their© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

310

Shota Iiyama, Tetsuya Oda, Masaharu Hirota

Figure 4 SED Error.

Figure 6 Compression Rate

increased. Therefore, our method can compress GPS logs
while considering the trade-off between the shape andvelocity information and the Compression Rate.
5. Conclusion and future work
This paper proposes an online compression algorithm for
GPS logs that uses three methods to preserve the featurepoints. Our experiments show our method can compress
GPS logs preserving velocity and shape features.
Future work will improve map-matching performance
because the compression of our proposed method
depends on the performance of the map-matching.

Figure 5 Speed Error.

horizontal axis is the threshold value of the SED. The
vertical axis is the value of each evaluation criterion.
In Figure 4, the average value of SED Error is large morethan the median. This result shows that our method cancompress almost GPS logs preserving the feature, but themethod when compressing some GPS logs failed.
The reason why the proposed method has tremendous
SED Error values in some GPS logs is due to calculating
the pseudo arrival time of the 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 used in the OW of
the proposed method. The pseudo arrival time assumes
that the user moves straight from the GPS point to the𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓. However, because roads are not always straight
but have curves, there is a difference between the
expected and the actual arrival time.
The Speed Error in Figure 5 shows that the Speed Errors
are about 3m/s in all SED thresholds. When the thresholdof SED was changed, the value of Speed Error did notchange much. This result indicates that the proposed
method is not much affected by the threshold of SED and
can preserve velocity features.
Figure 6 shows the Compression Rate results of the
proposed method. As the threshold of SED increases, theCompression Rate also increases. This result shows thatthe proposed method can change the Compression Rateby varying SED's threshold value. Figure 4 and Figure 5
show that the proposed method can preserve velocity andshape features, even when the Compression Rate is

Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP19K20418 and Grant for Promotion of OUS
Research Project (OUS-RP-20-3).
References
1. Douglas, D. H. and Peucker, T. K.: Algorithms for the
reduction of the number of points required to represent aline of its caricature, IJGIG, Vol. 10, No 2, pp. 112–122,
1973.
2. Meratnia, N. and de By, R. A.: Spatiotemporalcompression techniques for moving point objects, EDBT,
pp. 765–782, 2004.
3. Schmid, F., Richter, K. F. and Laube. P.: Semantic
trajectory compression, SSTD, Vol.5644, pp. 411–416,
2009.
4. Barbeau, S., Labrador, M. A., Perez, A., Winters, P.
Georggi, N., Aguilar, D. and Perez, R.: Dynamic
management of real-time location data on GPS-enabledmobile phones, UBICOMM, pp. 343– 348, 2008.
5. Zheng, V. W., Zheng, Yu., Xie, X. and Yang, Q.:
Collaborative location and activity recommendations withGPS history data, WWW, pp. 1029–1038, 2010.
6. Brakatsoulas, S., Pfoser, D., Salas, R. and Wenk, C.: On
map-matching vehicle tracking data, VLDB, pp. 853–864,
2005© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

311

Case Study and Direction of Bicycle-Sharing Systems in Japanese Cities
Minoru Kumano

Faculty of Regional Innovation, University of Miyazaki, Japan
E-mail: kumano@cc.miyazaki-u.ac.jp
Toru Hiraoka

Department of Information Systems, University of Nagasaki, Japan
E-mail: hiraoka@sun.ac.jp

Abstract
Bicycle-sharing systems allow people to share bicycles with others, borrow and return them using multiple cycle
ports, and register and pay for them with an IC card or smartphone. Because these systems are still new in Japan,
few papers can be found on them, even in the National Institute of Informatics. The purpose of this article is to clarify
the business purpose, issues, effects, features, numerical values, and direction of bicycle-sharing systems. Therefore,
we selected 16 advanced cases nationwide and conducted a survey in 2019. As a result, it was clarified that the main
issues facing bicycle-sharing systems are improving the ease of migration and supplementing public transportation,
improving profitability, and so on.
Keywords: Bicycle-Sharing System，Case Study，Direction

purpose of this article is to clarify the business purpose,issues, effects, features, numerical values (number ofports, number of bicycles, etc.), and direction of the 16
cases and to secure useful materials for the introductionof bicycle-sharing systems in the future. For the selectionof the precedent cases in April 2019, local governmentswith a proven track record were selected from online
searches, literature, and interviews with the Ministry ofLand, Infrastructure, Transport and Tourism.
A survey of local governments was conducted from
May to October 2019. In addition, we visited with the
majority of the selected local governments to conduct
interviews and collect materials.

1. Introduction (Background and purpose of
research, method)
In Japan, the Bicycle Utilization Promotion Law was
passed in 2017, and town development and bicyclesharing businesses have been progressing since. Bicyclesharing is a system that allows users to share bicycles andfreely get on and off at multiple ports.
In addition, bicycle-sharing systems alleviate traffic
congestion in cities, save energy, prevent air pollution,
reduce travel costs, and improve public health. Therefore,
they have recently become widespread in urban
municipalities nationwide. However, previous research is
not sufficient, and no paper has considered the
management trends, issues, and directions of bicyclesharing systems by conducting a survey of advanced
local governments all over Japan.
To fill this gap, 16 advanced cases nationwide wereselected and a survey was conducted in 2019. The

2. Research results① The most common purpose of the bicycle-sharing
businesses is to improve the mobility of the town (10cases). In particular, the effects of revitalizing the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

312

Minoru Kumano, Toru Hiraoka

issue. In addition, there is a problem of selecting
candidate locations for port installation (3 cases). It isnecessary to take into consideration the assumption of
possible harmful effects such as new construction,
location conditions that are more effective in improving
the utilization rate, and an increase in the number ofillegally parked bicycles. The problem of manners whenusers use bicycles and the risk of accidents (3 cases) werealso issues. Problems with maintenance of management
systems and equipment (2 cases), simplification ofmembership registration (1 case), improvement of theenvironment for inbound tourists (1 case), and low use by
citizens (1 case) were also mentioned.
④ Relationship between population size, number ofbicycles, and number of ports
See the correlation between the population size of eachmunicipality, the number of bicycles installed, and thenumber of ports in the bike-share example.
It is expected that the bicycle-sharing system of a citywith a large population will have a large number ofbicycles and ports. Figures 1 and 2 are correlation
diagrams of population size, the number of bicycles
installed, and the number of ports; they were created fromthe data in Table 1. Regarding population size and thenumber of ports, the larger the population, the larger thenumber of ports and the correlation. On the other hand,regarding the relationship between population size andthe number of bicycles, there are some data with large
outliers, and there is little correlation in the overall value.
The data with particularly large outliers is that of"Shimanami Rental Cycle (Imabari City / OnomichiCity)." Unlike other local governments, Shimanami
Rental Cycle emphasizes external effects such as widearea sightseeing tours. As a result of arranging bicycles
for tourists, the proportion of the local population ishigher than in other cases, i.e., the number of bicycles
was large. However, Imabari and Onomichi on the
Shimanami route; Takamatsu, a city that frequently uses
bicycles; and Koto Ward, which is located in
Megalopolis in the metropolitan area, are considered to
be special, and there is a correlation in cities other thanthese.

central city area and enhancing the convenience oftransportation for tourists have increased.
Next, bicycle-sharing systems aim to help society
break away from its dependence on automobiles (4 cases)
and reduce the environmental burden (4 cases) by
switching to using bicycles. The third most common
purpose is to supplement public transportation (3 cases)
and focus on bike-sharing as a means of promoting theuse of public transportation.
Interregional exchange or the promotion of recreation (1case) is also the goal of a wide-area bicycle-sharing
system that spans multiple prefectures.
② In terms of actual effects or advantages, bicyclesharing systems contribute most to tourism (7 cases). In
particular, they are highly effective as a means oftransportation that could increase the number of users forsightseeing purposes and help tourists plan sightseeing
excursions at their own pace.
Next, we have achieved about 50% of the purpose of
improving the ease of migration (5 cases). In addition, thesupplementation of public transportation (4 cases) has
also been effective, including the development ofregional transportation and the establishment of regionaltransportation networks. The introduction of electricbicycles (4 cases) has been well received by users, andpeople are able to travel smoothly. PR (6 cases)
concerning these efforts has also been effective.
There were three PR projects that improved therecognition of this project throughout the region.
According to survey participants, the development ofguided bicycle tours, the suppression of the inflow of
automobiles into the city (2 cases), and the conversion
from automobiles to bicycle use (1 case) have beenachieved.
③ The most common problems/issues are theimprovement of expenses such as balance of income andexpenditures and costs (8 cases). Regarding this, survey
participants expressed the opinion that a review of the fee
structure should be considered in consideration of the
profitability of the business.
The next most common issue is securing more users
and creating PR for that purpose (5 cases). The third issue
is the relocation of bicycles (4 cases), which is based ateach port. Since it is possible to return a bicycle to a port
different from the one from which you rented, the numberof bicycles at the ports will be uneven. The high laborcosts necessary for the relocation of bicycles is still an© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

313

Case Study and Direction

Table 1.

Contents of advanced cases of bicycle-sharing systems in each local government in Japan

Municipality

Okayama City

Miyazaki City

Name

Momochari

Miyako PiPPA

Start date

Imabari City

Takamatsu City
Bicycle Rental

Onomichi City

Shimanami Rental Cycle

May-01

Apr-99332 units(as of H27)

200 units

1,207 units(as of R1.8.30)

1,250 units(as of R1)

1,800 units

Number of ports

35 locations(1 of which is
closed as of
R1.8)

42 places

117 locations (asof R1.8.30)

7 places

13 locations(9 locations in
Imabari City)

May-991,025(Onomichi City
side as ofAugust 2019)13 locations (5locations on theOnomichi side)

Registration
number

91,714 people(as of H30
degrees)

2,700 (as of
H31.6.30)

128,804 people(as of December
31, H30)

---

---

---

Operational
trendsFujieda City

Fukuoka City

Fujieda Trip (bikeshare in the city)

Melchary (currently
in the demonstration
experiment stage)
Mar-15Mar-1845 units (withelectric assist)

About 1,000

16 places

More than 180
locations(as of H30)

---

---

A

B

C

D

D

B

B

A, B, D, F

B, C, D, E, F

A, B, C, D,

A, D

A

B, C

B, F

[Number ofuses] Average
1,453 times aday, total
number of uses
530,324 times,
turnover rate
3.85 times / day/ unit (H29
degrees), H30
degrees
cumulative
number of uses
about 1.63
million times

The breakdown of
the occupancy
rate of the port
utilization rate is
62.1% for public
land (17locations), 37.7%
for private land
(25 locations),
and 0.2%
unknown.

Bicycles may beparkedexcessively, andcollection andrelocation work
to level thenumber ofbicycles isincreasing.

Operated bybicycles withabandoned
bicycles---High utilization
rate of ports on
public land under
demonstration
experiment

Mostly used forcommuting on
weekdaymornings andevenings. In
particular, there
are many uses ofports near tower
apartments andstations.

JR Takamatsu
station square
undergroundport and
Kotoden
Kawaramachi
stationunderground
port

---

Environmentallyfriendly and ecofriendly,
contributing tohealth promotion

Reduction of
CO2 by reducing
the movement of
automobiles,
improvement of
attractiveness of
the introductionarea,
improvement of
convenience of
movement,
creation of
liveliness byimprovement ofmobility in thearea, etc.

Control the total
amount ofbicycles throughmutual use andeffectively
utilize illegallyparked bicycles

Promotion of
stay-typetourism,
promotion ofhealth,promotion of
purpose of lifeand friendship,
realization of asociety with
reducedenvironmentalload, promotionof inboundtourism due tointernational
popularity

Port with many
users

Pros

･Change oftraffic behavior,revitalization of
town･Reduction ofenvironmental
load

Problems/issues･Simplificationof membershipregistration
method･Disseminationof first-timeusers･PR forincreasing users
for sightseeing
purposes･Elimination of
bicycle bias ateach port

Direction

Jul-17

Takamatsu City

Number ofbicyclesUser registration[A: port, B:
smartphone app,
C: web, D:
window]
Payment method[A: cash, B: credit
card, C: mobile
phone payment,
D: IC card, E:
electronic money,
F: other]

7/27/2013

Koto-ku, Tokyo
Koto Ward
SeasideCommunity
Cycle
Nov-12---

There is concernthat the risk ofaccidents due tocrossing with
automobiles andpedestrians isincreasing due tothe lack offollowing traffic
rules such as
driving on the left
side of the road.

---

Annual usage:
Approximately
310,000･Improved
income andexpenditure bystreamlining thecollection andrelocation of
expensive
bicycles･Optimization of
the number ofbicycles at thecycle port

･Improvementof balance ofincome andexpenditure･Update ofmanagement
system due todeterioration
over time

Examining andimplementing
problems andsolutions toproblems indemonstration
experiments, andshifting to fullscale
implementation

With the entry ofprivatebusinesses, wewill consider the
future of therental cycle
business whileconsidering acertain divisionof livingaccording toeach role and incooperation withprivatebusinesses.

Approximately
150,000 annual
users

Reiwa first year
budget amount:
10,000,000 yen

H25: 81,851
H26: 116,303
H27: 135,229
H28: 141,205
H29: 149,740
H30: 132,075* 2017 budget
amount:2,000,000 yen,2018 budget
amount: 4,500,000
yen

---

---

---

You can enjoy
the beauty of theislands of
Setouchi,interact with thelocals, and enjoy
gourmet food atyour own pace.

① Cheaper than a
bus and convenientwith no waitingtime. You caneasily climb aslope with anelectric bicycle. ②
There is a station at
Fujiedastation. Available
24 hours a day. ③
There is no
problem even if
you use it
frequently. It'scheaper thanparking in theparking lot.

･Use unused space
as a port (individual
application required)･Can travel shortdistances faster thanusing public
transportation

Inboundenvironment
maintenance(Wi-Fienvironment,
communicationmeans, cashless
payment)

As the numberof cyclists
increased, we
began to hear
complaints
about driving
manners･Implementationof effective PR toincrease thenumber of users･Selection of
station installation
locations that are
expected to be
used･Reduction ofrunning costs

Expanding thebase of cyclists
by attracting
non-cyclists
such as theelderly andwomen

We will
continue todisseminate
information on
the appeal of
Shimanami
Kaido Cycling
bothdomesticallyandinternationally,and strive toimprove thedrivingenvironment andmanners.

While taking in theopinions of users
through effective
PR and surveys,
we will improvethe utilization rateand ease of
migration. Since it
is an initiative thatutilizes IoT, wewill consider how
to utilize usage
data.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

314

Mercari dismantled
its consolidated
subsidiary Sozo in
June 2019 andstarted a newcompany "neuet" inAugust. Operated in
the form of a joint
system between"Co., Ltd." and"Clara Online Co.,
Ltd."

[Number ofloans (fiscal
year)]

・ There is apossibility that thecorrect position of
the bicycle will not
be displayed on theapp due to GPS
accuracy･There are users withbad manners such as
personalization･Because there is noreservation system,other people will use
the bicycle first(possibility of not
being able to ride
when you want to)
Business succession
was passed to "neuet
Co., Ltd." and the
Mercari business as a
new system started incollaboration with"Clara Online".
"The purpose of the
new system is tofocus on Mercari andMercari. Results ofbusiness selection
and further businessexpansion"

Minoru Kumano, Toru Hiraoka

Table 1. (Continued)

Kitakyushu

Oita City

Hiroshima City

Start date
Number of bicycles

Kagoshima
City
Kagoshima City
Community
Cycle "Kagorin"
Mar-15
192 units

Kitakyushu City
Bike
Mar-10246 units

Oita Cycle
Share
Oct-18300 units

Hiroshima City
Peasukuru
Feb-15
About 500

Number of ports

25 locations (asof R1 / 7)

21 places

35 places

62 locations---

535 general
members
Shear type
(corporate) 45(as of R1 / 5)

Approximately
1,000 people (asof R1 / 7)

A, D

C, D

A, B, E

A, B, C

Municipality
Name

Registration number
User registration [A: port,
B: smartphone app, C:
web, D: window]
Payment method [A:cash, B: credit card, C:
mobile phone payment,
D: IC card, E: electronicmoney, F: other]

Operational trends

The number ofuses increases
year by year (thetotal number ofuses until theend of July of
the first year of
Reiwa is about
670,000)

Number ofusers: 147 units /day (2018)

Fukui City

Toyama City

Himeji City

Kanazawa

Fukui City2013
44 units[Number ofports] 16
locations (as of
R1 / 7)

Toyama City
Avile
Mar-10255 units[Number ofports] 23
locations (as of
R1 / 9)Himeji City
HimejiJul-16150 units[Number ofports] 20
locations (as of
R1 / 7)

Kanazawa City
Machinori
Mar-12155 units[Number ofports] 21
locations

---

---

---

180 units (as of
R1 / 7)

296,616 people(H24 / 3 / 21H31 / 3/31)

B, C

C, D

D

D

A, C

A, D

B, F

A, B, C, D

A

A, B, F

A, B

A, B

The monthly
membership freecampaign washeld fromFebruary 12,2019 to June 30,
2019, and theusage increased
significantly.

Consignment
contract amount2014 ...
20,000,000 yen
2015 ...
18,000,000 yen
2016 ...
14,000,000 yen
2017... 0 yen 2018 ...
4,300,000 yen

The number ofuses isincreasing year
by year, and thedemand isincreasing,
especiallyin. We are able
to operate thebusiness on a
low budget
while
coordinatingwith the private
sector.

---

The share cycle
has become
established as asecondary meansof transportation,and the numberof users and thenumber of timesof use are
increasing year
by year.

Number of uses
178,619 times,
number of users
63,284 (2018)

Three stations at
the south exit ofToyama Station,
Nishicho Station(located in frontof Toyama City
Hall), and in thecity center

Ports around
Himeji Station,
often used forcommuting andsightseeing

Most often used
at Kanazawa
Station, thestarting point,
and ports aroundmajor tourist
destinations---

Port around
Fukui Station

Because it is acity of deltas,
you cansmoothly tour
the city withmany undulating
bridges byelectric bicycle

All of them are
equipped withelectric power
assist, andrentals are
accepted faceto-face, so we
can provide
tourists
information as
well as rental
services.

Cyclocity Co.,
Ltd., theoperator, isoperatingsmoothly andsafely by making
the best use of
the know-how
cultivated so far.

It can be usedwith cash, and it
can be used
immediately
without the needfor preregistration orother operations.

･It costs a lot of“relocation” tocorrect the biasof bicycle
placement･It takes acertain period oftime to increase
the number ofusers

As a candidate
for port
installation, it isa highly
convenient
place, but there
are times when
it is necessary toabandon the
installation due
to concerns
about problems
caused byinstalling theport, such as thenumber ofillegally parked
bicyclesincreasing･If you have acredit card, you
can easily
register on thespot･There are many
narrow streets inthe city ofKanazawa, andthe main tourist
spots areconcentrated ina narrow area,
so it is suitable
as a means oftransportation
for tourists

Businessmanagementthat is consciousof the difficultyof establishing a
new lending
port andprofitability

Stable securingof financial
resources suchas operation andmaintenancecosts forproperly
conducting thebicycle-sharing
business･Efforts toincrease thenumber of users･Naming rights
and otheradvertising
revenues need to
be expanded, fee
structurereviewed andcosts reduced･Low citizenuse (about 90%of the total is fortourism)･Increase inrepair costs overtime

We would liketo set up aservice area sothat therelocation work
will not be
excessive,
balance theincome andexpenditures,and verify that it
will beestablished as asustainable newtransportation
system.

High density ofcycle ports sothat domestic
and foreign
tourists andother visitors
can comfortably
visit tourist
facilities in thecity area andcitizens canconveniently use
the ports as ameans of daily
use that
complements
publictransportation.
Promote tourism
and regional
revitalization
using bicycles

Port with many users

---

---

---

Pros

Reducing
environmental
impact,improvingmobility andconvenience,
responding to awide range of
tourism needs(attractiveness of
town tours
unique tobicycles)By shifting to
the use of
bicycles, it ispossible toreduce theenvironmentalload by means
such asreduction ofcarbon dioxide
emissions,electronic
assistance, and"drop offs"
betweenstations.

･You can dropoff･You don't haveto wait like for abus･You can use it
24 hours a day･ Contribute totheestablishment ofa regionaltransportation
network by
combining withthe public
transportation
network

Problems/issues

Direction

Continue to
strive for public
relations andmaintenance ofequipment, etc.

Bycommunicating
the significanceof riding abicycle in aneasy-tounderstand andpolite manner,
we will further
increase interest
andunderstanding ofbicycles･Ensuringbusiness
profitability･A large ratio oflabor costs tocorrect the biasof bicycles
between stations

Aiming tocontinuebusiness while
selectingstations that
better meetneeds

We increased
the operating
rate of bicycles
by driving up
demand, andbased on thatsituation, we
consideredestablishing new
rental ports andincreasing thenumber ofturntables.

The expansionof stations andservices has justbeen
implemented by
the operator inconsideration of
profitability,
etc., and there
are no plans to
expand at this
time.

While
considering thenumber ofbicycles and thearrangement ofstations thatmeet the needs,
we will proceedwith theintroduction ofan "in-vehicle
share cycle" thatcan reduce
operating costs
compared to thecurrent system asa "next
generation
Hime-chari."© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

315

A new systemstarted fromMarch 2nd, yearof Reiwa. Basic
direction: (1)Encourage
citizens to use(currently, users
are mainly
tourists); (2)Respond to new
tourism demand
based on thecentral city area;(3) Maintainpublic-private
management
and otherelectric assists inorder to comply
with the policyof towndevelopment
Equipped withfunctions andGPS functions

Case Study and Direction

3. https://www.city.toyama.toyama.jp/kankyobu/kankyoseis
akuka/ondankataisakukikaku/jitenshakyodoriyo.html
4. https://www.machi-nori.jp/
5. https://docomo-cycle.jp/koto/whatiscs/
6. https://www.keifuku-hotels.com/news/?p=65
7. https://www.momochari.jp/howto/index.html
8. http://www.city.hiroshima.lg.jp/www/contents/15815526
07625/index.html
9. http://www.city.kagoshima.lg.jp/faqkankyomachizukuri/kanseisaku/q9.html
10. https://hachimansama.jp/blog/9305#
11. https://www.itmedia.co.jp/news/articles/1802/13/news11
5.html12. https://www.kobe-np.co.jp/news/odekakeplus/news/detail.shtml?news/odekakeplus/news/pickup/201809/11648577
13. https://www.city.miyazaki.miyazaki.jp/city/policy/urban_
planning/164580.html
14. https://www.city.fujieda.shizuoka.jp/trip/oshirase/152118
4175837.html
15. https://rembrandt-group.com/oita/7842.html

Figure1. Relationship between population size and number of bicycles

Figure2. Correlation diagram between population size and number of port

3. Conclusion
According to a nationwide survey, the number of cycle
ports and the number of users are correlated in thebicycle-sharing businesses located in urban areas, and thecycle ports located at railway stations representing thecities have the highest utilization rates. Cycle ports withheavy traffic at transportation hubs have many users,
complementing public transportation; shortening travel
times for sightseeing, commuting to school, business,
etc.; and increasing mobility. However, it is difficult tomake a profit in the bicycle-sharing business, so most ofthese businesses are subsidized by the local government.
Large-scale bicycle-sharing businesses with a relatively
large number of cycle ports and bicycles have a higher
turnover rate and are more likely to improve profitability
than small bicycle-sharing businesses.
Therefore, for the sustainable development of bicyclesharing businesses in the future, it is important to secure
the number of bike-share ports and the number ofbicycles in a convenient position commensurate with thesize of the population and to devise ways to improve
profitability, for example, in Toyama City. As you cansee, new ideas and implementations are needed, such asattaching electronic public notices to cycle ports to earnsponsorship income.

References
1. https://shimanami-cycle.or.jp/rental/course
2. http://takamatsuparking.com/for_use_onetime_rider_2017/© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

316

The IoT Solution to Learn English Words
Shogo Aizawa, Motohide Yoshimura

Department of Information Systems, Faculty of Information Systems, University of Nagasaki
Japan
1-1-1 Manabino, Nagayo-cho, Nishi-Sonogi-gun, Nagasaki 851-2195, JAPANE-mail: yxsimura@sun.ac.jp
http://sun.ac.jp/e

Abstract
We innovate an IoT solution to play the English word learning system. It uses a Raspberry Pi and two NFC readers.
You can learn English words in three steps by using our system. First, prepare NFC tags corresponding to English
words and illustrations, such as a cat, dog, chicken etc. Next, select a pair of the tag corresponding to an English word
and illustration. Finally, hold them over the each NFC reader, then the answer that is correct or incorrect are displayedon a touch panel. Our solution has two aspects. The one is an educational system for children and the other is an IoT
toy playing with children. As to the former, the children can learn English words by intuitive operation such as
operating the touch panel and holding the NFC tags. As to the latter, they can experience a part of an IoT technology.
In this paper, we report findings through the construction of the English word learning system..
Keywords: English word learning system, IoT, NFC, educational system, IoT toy

third is the class improvement for the proactive,
interactive and deep learning.[3]" In addition to ICT
utilization such as efficient and effective presentation
using the large presentation screen, the goal is to preparea terminal for each student and promote learning
independently and interactively. By using ICT equipment,
it is possible to study efficiently during the limited
learning time. Aggressive utilization of ICT is an
important issue for enhancing future educational
activities.
The utilization of ICT equipment is underway in all
subjects. Especially, the use of ICT in English education
is being actively promoted, and 99.1% of schools utilize
ICT equipment in English education in elementary
schools [4]. However, according to a survey of Japanesechildren's use of digital devices in schools at OECD PISA
2018, the percentage of students who answered that they
did not use ICT devices in their school's "foreign
language" class was 67.0%, it is the worst among
participating countries [4]. It can be said that theenvironment in which teachers utilize ICT in English
education is in place, but the environment in which
students themselves use ICT to learn English is not
sufficiently developed.

1. Introduction
Information and communication technology is now anintegral part of our lives. In 2019, 96.1% of households
own mobile terminals, and 83.4% of households own
smartphones, there is more than 80% of households [1].
The development and spread of ICT has brought aboutmajor changes outside the ICT field. The role of ICT in
the field of education is also important. In 2020,
programming education became compulsory inelementary schools, and the new teaching guidelines forelementary schools announced in March 2017 stated asfollows, "In order to cultivate the ability of information
utilization, ICT equipment such as computers andnetworks will be provided at every school. We will equip
the appropriate environment to utilize these tools andenhance learning activities [2]." The basic idea of ICT
utilization by the Ministry of Education is as follows,"Based on the new teaching guidelines, there are threeimportant aspects. The one is the balanced developmentof the three pillars of qualities and abilities. The second
is the aggressive utilization of ICT as the teachingmaterials, the teaching and learning tools according toboth of the actual conditions of children and schools, andcharacteristics and learning steps of each subject. The© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

317

Shogo Aizawa, Motohide Yoshimura

Familiarity with English in active English learning inelementary school is very important in learning English
after junior high school. The liking and proficiency ofEnglish in junior high school has a great influence on thefrequency and amount of learning of English [6]. It iseffective to "like" English in order to effectively advancelearning English. For that purpose, it is necessary forstudents to "like" English from the English education ofelementary schools and to maintain their interest in
English. A game that utilizes English is used as a way toinvite the interest. It has been found that learning English
in combination with games is effective in familiarizing
yourself with English and practicing and establishing the
vocabulary that is the basis of foreign language activities[7].

In this research, we will develop an English learning
system by using ICT, especially IoT. The IoT is atechnology that connects things via the Internet, astranslated as the "Internet of Things". Simply speaking, itis the immediate reflection of happenings in the realworld to the digital world. With this technology, both
analog and digital advantages can be leveraged. An IoT
toy is the one that incorporates the elements of IoT. By
utilizing this element, not only will it attract the interestof children, but it will also be possible for them toexperience ICT with simple operations. By using oursystem, users can learn English while playing and cancome into contact with IoT, which is a part of ICT. In thispaper, we report the findings obtained in the constructionof the system. We will show examples of using English
learning tools in Section 2, and describe examples of IoT
toys for children in Section 3. In Section 4, we will
describes our system using NFC. In Section 5, we willsummarize the paper.

2.2. Flashcard caseFlashcards are used as teaching materials when you startlearning English. A flash card is a set of a card with anillustration and a card with words written on it. It is usedin school lessons and English conversation classes, andthe teacher can show the illustration card and the English
word card at the same time to teach pronunciation, orshow only the illustration card and let them answer the
English word. You can also incorporate game elements
such as playing with listening to English and taking
illustration cards like playing cards. Since it is a simple
structure, it can be used in various ways by the ingenuityof the teaching side. Currently, various types of flash
cards are on sale, so you can choose a flash card that suits
your purpose, such as age and type of English words.
There are also flashcards that incorporate ICT elements[8]. A flash card can be displayed on an ICT terminal andused in the same way as a conventional flash card. In
addition, since the native speaker's voice is attached, you
can hear and acquire the correct pronunciation of thenative speaker. By using an ICT terminal, students can bemade to hear the correct pronunciation of native speakers
without the teacher knowing the correct pronunciation ofEnglish.

2. Examples of using English learning tools
2.1. Current status of English learning
As in the case of compulsory English education inelementary schools in 2020, the timing to start learning
English is getting earlier year by year. Specifically, the"foreign language activities" that were conducted in the
5th and 6th grades in the teaching guidelines before 2020
will be advanced to the 3rd and 4th grades, and English
will be started as a subject of "foreign language" from the
5th grade. The goal of "foreign language activities",
which will be held in three or four years in the elementary
school, is to familiarize yourself with English throughactivities such as quizzes, dance, and singing. By starting
to learn English from an early stage, children can takeadvantage of their flexible adaptability, develop a
positive attitude toward communication, and become
familiar with English expressions. In addition, more than
70% of junior high school students wanted to read
English words and sentences in elementary school, andfelt that they wanted to learn English words among
elementary school [5]. By becoming accustomed toEnglish in elementary school, children will be able toacquire a foundation for English education after juniorhigh school, which will lead to improvement of theirEnglish proficiency.

3.

Examples of IoT toys for children

3.1. Overview of IoT toys
With the development of ICT in recent years, IoT is also
being incorporated into toys for children. Products so-© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

318

The IoT Solution to

English words and illustration cards at which the NFC
tags attached. Python (version 3.7.3) is used to construct
the system. NFC Py (version 1.0.3) is used to operate theNFC reader.

called IoT toys are on the market. You can experience theworld of the Internet with more intuitive operations thanusing a smartphone or PC, expanding the range of playfrom the analog world to the digital world. It is also
possible to get interested in the ICT field by playing
while touching a part of IoT technology. In addition, IoT
toys are more attractive than their toys by connecting
smartphones, game consoles, and toys via the Internet.
IoT toys also have an aspect as educational toys, and are
attracting attention as toys that can be learned while
playing. It can be said that IoT toys are also very usefulas a means to improve their abilities in the present age
when programming education is emphasized. Here, wewill introduce specific examples of IoT toys in Japan.

4.3. System overview
We will explain the outline of our system. Figure 1
shows the system configuration diagram.

3.2. CORO-PUTER-Ja.
"CORO-PUTER-Ja." is a wooden IoT educational toy
developed by TDS Co., Ltd. [9]. When a wooden
hiragana block with an embedded RFID tag is rolled onto
a wooden frame with an embedded reader, the reader
reads the tag and the hiragana is displayed on asmartphone connected via Bluetooth. This allows you tolearn letters and words and get in touch with technology
through the play of rolling blocks. Even infants can easily
pick up blocks, carry them to the frame, and roll them, sothey can learn while playing. It can be said that thestrength of IoT toys is that a lot of information can beobtained with this simple operation.

Fig1. System configuration diagram
This tool processes the content read by the NFC reader
with Raspberry Pi and reflects it on the touch panel.
Sony's PaSoRi RC-S380 / S is used to read NFC tags,
Kuman's 7-inch touch panel is used to display the screen,
ZoeeTree S1 portable Bluetooth speaker is used to playaudio, and Raspberry Pi 4 is used to process and executecodes. The purpose of this system is to learn the English
names of animals. There are two kinds of cards, "Animal
Card" with animal illustrations and "Animal Name Card"with animal English names as shown in Fig. 2.

4. English learning system using NFC
4.1. Features of our English learning system
In this research, we innovate an IoT solution to play the
English word learning tool as a framework of English
learning by IoT toys and prototype an English learning
system for children that aims to integrate English
learning and ICT. By using this tool, you can learn
English while playing with IoT toys, and you can also
experience some of the IoT technologies.

Fig2. Card with NFC tag attached
We attach the NFC tag to the back of each card. In thissystem, unique IDm of each NFC tag is associated witheach of the "Animal Card" and "Animal Name Card", andthe correct answer or incorrect answer is determined bypairing two IDm in the code. The NFC tag used for eachcard is Sanwa Supply's NFC tag (product part number:
MM-NFCT). Two NFC readers are prepared so that the"Animal Card" and "Animal Name Card" can be read at
the same time. They are decorate as shown in Fig. 3 toclarify the reader on which each card should be placed.

4.2. Development environment
In this system, the users proceed the English wordlearning, while the contents of the NFC tag read by the
NFC reader are displayed on the touch panel. The system
consists of Raspberry Pi 4, and the OS Raspbian (version
10 (buster)) as the Platform, and two NFC readers and© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

319

Shogo Aizawa, Motohide Yoshimura

reason for playing the correct animal bark that matches
the "Animal Name Card" is to make it easier to imagine
an animal illustration from the bark.
5. Conclusions
In this research, we prototyped an English wordlearning system that integrates English learning and IoT
by using NFC tags. Our system can memorize English
words and illustrations as a word meaning in combination
by reading NFC tags with two NFC readers. In the future,
we have a plan to enhance the mechanism so that children
can learn with more interest.

Fig3. Decorated NFC readers
As shown in Fig. 4, place the "Animal Card" and"Animal Name Card" on each reader, and press the"Play" button on the touch panel to execute the code anddisplay the correct or incorrect answers. When reading acard, the IDm of the NFC tag is read, and the illustrationand name of the animal on the card corresponding to that
IDm are displayed on the touch panel.

References
1. Ministry of Internal Affairs and Communications, 2020
WHITE PAPER Information and Communications in Japan,
2020, in Japanese.
2. Ministry of Education, Elementary School Programming
Education Guide (3rd Edition),
https://www.mext.go.jp/content/20200218-mxt_jogai02100003171_002.pdf(accessed 2020-12-8), 2020, in Japanese.
3. Ministry of Education, Effective use of ICT in teaching eachsubject,
https://www.mext.go.jp/content/20200911-mxt_jogai01000009772_19.pdf(accessed 2020-12-8), 2020, in Japanese.
4. Ministry of Education, Effective use of ICT in teachingforeign languages,
https://www.mext.go.jp/content/20200911-mxt_jogai01000009772_13.pdf (accessed 2020-12-8), 2020, in Japanese.
5. Ministry of Education, Current status, achievements, andchallenges of foreign language activities in elementary
schools,
https://www.mext.go.jp/b_menu/shingi/chousa/shotou/102/s
hiryo/attach/1347444.htm (accessed 2020-12-8), 2020, inJapanese.
6. KANEKO Yoshitaka, Educational Implications to Help
Students to Enjoy Learning English, Bulletin of Meikai
Teacher-Training Support Center, vol.1, pp.1-15, 2018, inJapanese.
7. Chihiro Tachibana, Game activity functions used in theteaching process of elementary school foreign language
activities: Focusing on the functions of game activities andthe effects of learning, Himeji University Academic
Institutional Repository, no.12, pp.9-22, 2019, in Japanese.
8. CHIeru Co, Elementary school flash English words,
https://www.chieru.co.jp/lp_english/(accessed 2020-12-8), in
Japanese.
9. TDS Co., LTD., CORO-PUTER-Ja.,
https://www.replug.jp/coro-puter/(accessed 2020-12-8), in
Japanese.

Fig4. Operating environment
4.4. System behavior
The user places the "Animal Card" and "Animal NameCard" with the NFC tag attached on the corresponding
NFC reader, and presses the "Play" button to confirm thecorrect or incorrect. In case that the answer is correct, forexample, the dog's "Animal Card" and the dog's "Animal
Name Card", a sound indicating the correct answer andan illustration ( ○ ) indicating the correct answer are
displayed on the touch panel. In case that the answer isincorrect, for example, a cat's "Animal Card" and a dog's"Animal Name Card", a sound indicating an incorrect
answer and an illustration (×) indicating an incorrect
answer are displayed on the touch panel. In addition, only
if the answer is incorrect, the correct animal bark thatmatches the "Animal Card" is played through the speaker,
and the frame of the "Animal Name Card" on the screen
is blinked to suggest another "Animal Name Card". The© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

320

Using Various Evaluation Standards to Determine an Error Recovery Process in an
Automation Plant
Akira Nakamura*1, Natsuki Yamanobe*1, Ixchel Ramirez Alpizar*1, Kensuke Harada*1, *2 and Yukiyasu Domae*1*1

Industrial Cyber Phsical System Research Center
National Institute of Advanced Industrial Science and Technology (AIST)
Second Annex, AIST Tokyo Waterfront, 2-4-7 Aomi, Koto-ku, Tokyo 135-0064 Japan*2

Robotic Manipulation Research Group
Systems Innovation Department
Graduate School of Engineering Science, Osaka University
1-3 Machikaneyama, Toyonaka 560-8531, Japan
E-mail: a-nakamura@aist.go.jp, n-yamanobe@aist.go.jp, ixchel-ramirezalpizar@aist.go.jp,
harada@sys.es.osaka-u.ac.jp, domae.yukiyasu@aist.go.jp
www.aist.go.jp

Abstract
In an automated plant, an error is more likely to occur in difficult tasks, which are complicated in nature. Such atask is often re-executed after returning to the previous step when a large-scale error occurs. Therefore, deciding
both the past step that the task should return to and the recovery planning following this return becomes important.
In this study, error recovery planning considering these two factors using various evaluation standards is realized.
Keywords: error recovery, task stratification, error classification, automation plantof fundamental elements with sequences of sensing,modeling, planning, and execution (Fig. 1). If an error
occurs here, the process advances to the recovery part.
In this part, the error cause is estimated, error isclassified, system is corrected, and process is reexecuted using the corrected system with an improved
reliability.
Currently, deciding both the past step that theprocess should return to and the recovery planning after
returning has become problematic. For this, we
proposed a planning of the error recovery by deriving
these two factors in consideration of cost.8 In this study,
we proposed a planning method for error recovery
derived using various evaluation standards.

1. Introduction
Automated plants have been used for production invarious industries due to which the efficiency of
production increases. On the contrary, system
automation also leads to errors. Therefore, the progressof the error recovery method is crucial, which has
demanded the increased study of error recovery inrecent years.1-5 However, the recovery technique used inthese studies is ad hoc, and it is difficult to utilize themin various actual plants.
For several years, we performed a study on thesystematization of the error recovery theory. We
proposed a novel error recovery method based on theconcepts of both task stratification and error
classifications.6-8 The main part of this method consists© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

321

Akira Nakamura, Natsuki Yamanobe, Ixchel Ramirez Alpizar, Kensuke Harada and Yukiyasu Domae

2. Concept of Skill

Main partPlant tasks

Sensing
Modeling

In this study, it is assumed that “skill” represents theunit of motion. In this section, we describe theindispensable aspects of skills, which are components ofhuman behavior and automated plant operations.9-11

Re-executeusing the system withimproved reliability

Recovery part

Planning
Execution

Corrected System

2.1. Skill primitives

Cause estimation anderror classification

Failureoccurred!!

Class 1 (Execution error)
Class 2 (Planning error)
Class 3 (Modeling error)
Class 4 (Sensing error)

We derived the motion primitives constituting tasks
such as assembly and carrying by analyzing a person's
behavior and called this motion unit "skill".9-11 Thus,
three basic skills of "move-to-touch", "rotate-to-level"and "rotate-to-insert" are important (Fig. 2). The
representative person’s behavior can be composed ofthese three basic skills and other resembled skills. The
unit of machine motion can be considered to be similar
to the motion primitives of human behavior.

Fig. 1 Robot task system with an error recovery function

2.2. Stratification of tasks……

(a) Move-to-touch skill (b) Rotate-to-level skill

In general, it is possible to regard a task performed byan equipment in an automated plant as a hierarchicalstructure (Fig. 3).9-11 The layer “task(i+1)” occurs one tierabove the layer “task(i)”, and the layer “skill primitive”
is represented by the lowest layer “task(0).”task (i)……

task (2)task (1)(c) Rotate-to-insert skill

task (0)(Skill-primitive)

Fig. 2 Three fundamental skills

Fig. 3 Hierarchyof tasks

SensingChange ofplanning parameters

Modeling

Change ofmodeling parameters

Planning
Class 4error

Class 3task 2

Class 2error

In an actual environment unlike the ideal, errors inequipment performance often occur due to various
causes. In this section, we describe the concept of error
classification and our proposed technique of error
recovery.6-8

Change ofsensing parameters
Class 1

task 1

3. Error Recovery

3.1. Error classification
Errors can be classified into four groups: execution,planning, modeling, and sensing errors on the basis ofpossible causes (Fig. 4).6-8

Classification of error
task kerror

Class 1： Execution error
Class 2： Planning error
Class 3： Modeling error
Class 4： Sensing error

3.2. Error recovery based on classification
When an error occurs in automated plants, its cause is
first estimated, and a suitable correction derived
according to this tentative cause is performed at theequipment system. Then, the executive process returns
to the previous step, and the task is restarted from thestep (Fig. 4).6-8 The same error has a lower probabilityof occurrence due to the corrected operation of theequipment.

Fig. 4 Fundamental process flow with error recovery

The concept of skill, which is motion primitive, isdescribed in Section 2. The error recovery technique isexplained in Section 3. A method of recovery planning
using plural evaluation standards is proposed in Section
4, and finally, a simple sample is presented in Section 5.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

322

Using Various Evaluation Standards

If a small-scale error arises, the process returns to
the previous step in the lowest layer of the hierarchy(Fig. 4, Fig. 5). If an error is large-scale, the process
goes back to the previous step in the high-ranking
hierarchy layer. In both cases, it is restarted from theprevious step (Fig. 5).

task (M)task (2)task (1)task (0)3.3. Candidate processes for recovery

Task layer

Let us consider candidates for error recovery processes.
In the previous subsection, it was shown that the step inwhich the process returns to after the error occurs varies
according to its scale. However, it is possible to back up
bigger than a retreat step of necessity minimum.
Figure 6 shows various possible recovery processes
when an error occurs at subtaskm in task T with start S
and goal G. Then, task T consists of a sequence ofsubtaskk (k = 1, 2, 3, …, n), which comprises a skill
primitive sequence [ skill k1, skill k2, …, skill kn_k(max) ],where subtaskm indicates the minimum traceable unit
explained in the study,6 that is, the smallest unit,
wherein it is necessary to return to the first node of askill primitive sequence if an error occurs. In theabsence of any problem, there is a possibility that theexecution of several skills will be continued within
subtaskm without returning to the previous step
immediately after the error occurrence.
After the process returns to the j-th node before
subtaskj (j = 1, 2, 3, … , m), the process [ subtaskj
subtaskj+1, …, subtaskｎ ] proceeds along the center boldarrows in the corrected plant (Fig. 6). However, reexecution using the same sequence of subtasks is often
impossible for various reasons such as the displacementof the target object and the change in the arrangement ofobjects. In such a case, an equivalent task of the original
sequence, which is derived by large-scale re-planning ofsubtasks, is carried out as shown in the right-side arrowsof Fig. 6.

Modification Modification Modification
Class N1
Class N2Class NM

The error recovery which
returns to the previous step

Task T

Start S

subtask1

An equivalent task

subtask2

An equivalent task

subtaskm-1 An equivalent task

subtaskm
Failure occurs!

An equivalent task

subtaskn

An equivalent task

Performing equivalent sub-tasks

Minimum traceable unit

Returning to a previous step

Fig. 5 The expression of task stratification andthe process flow of the error recovery

Goal G

Fig. 6 Various processes of error recovery considered
for a failure occurred in subtaskm

We proposed a suitable recovery planning methodby considering the practical cost as an evaluation
standard as in the study.8 In this study, we proposed amethod to derive the most suitable recovery processusing various evaluation standards.
4.1. Evaluation standards
Let us consider the following eight evaluation standards
to select a recovery process.
(i) Cost
We considered only the cost as the evaluation
standard,8 where costs include material costs, part
costs, electricity bills, and planning expenses. A
process with minimum practical costs is selected.

4. Evaluation Standards for Selection of a
Recovery Process
The previous section shows that there is a possibility
that many recovery processes are generated. Therefore,
it becomes important to restrict routes by deciding boththe prior step that should be returned to and the recovery
process after return.

(ii) TimeTime is considered as the evaluation standard. The
process where it is expected that the recovery
sequence takes the shortest time is chosen.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

323

Akira Nakamura, Natsuki Yamanobe, Ixchel Ramirez Alpizar, Kensuke Harada and Yukiyasu Domae

5.1. Several types of recovery process(iii) Reliability
Reliability is considered as the evaluation standard.
The process that emphasizes most on the operationalreliability is chosen.

Let us consider an assembly task in which a hook is to
be stuck to the surface of a product using four precision
screws, which resembles the exercise illustrated in the
study.8 Figure 7 shows a sequence of this task: thetacking task is performed in the first phase (Fig. 7(a)–(d)) so that these four screws are fastened temporarily tothe plate part of the hook placed levelly. The erecting
task is performed in the second phase (Fig. 7(e)–(f)) sothat the plate part of hook with four screws stoodvertically. The touching task is performed in the third
phase (Fig. 7(g)–(h)) so that the hook is moved in themounting location. The installation task is performed in
the final phase (Fig. 7(i)–(m)) so that the hook is fixed
on the product plane.
In this section, it is supposed that the drop of ascrew occurring at the step of Fig. 7(h) represents theerror as shown in Fig. 8, similar to the study.8 Let us
consider the following three types of recovery
sequences chosen from a lot. Please refer to this study8
for details.

(iv) Safety
Safety is considered as the evaluation standard. A
safety problem to a person and singular point
problem are included in this standard. The processthat ensures maximum safety of the operation ischosen.
(v) Finishing
The finishing of operation is considered as theevaluation standard. In the example of production
and repair, a process whose outward appearance andfeel are the cleanest is chosen.
(vi) Recovery data
Data on the recovery process are considered as theevaluation standard. A recovery process with asufficient amount of data is chosen.
(vii) Tool
The tool used in the recovery process is consideredas the evaluation standard. The recovery process withthe special tool necessary for the practice of the work
is chosen.

•[ER-I] Error Recovery I
The first sequence is a recovery process based onrerun from start S, using a new hook and four newscrews, where the tacking task (Fig. 7(a)–(d)) is reexecuted using new parts (Fig. 9).

(viii) Operator skill
The skill of an operator in the recovery process is
considered as the evaluation standard. When an
expert is present, the necessary process of the skill iscarried out with precedence.

•[ER-II] Error Recovery II
The second sequence is a recovery process carried out
by returning to the step of Fig. 7(d) in the tacking task,
as shown in Fig. 10. Two types of methods exist for
the use of a substitute of the dropped screw. One is
the use of a new screw in the parts box, while theother is the reuse of the dropped screw by searchingand picking up. The former is transcribed with [ERII(N)] and the latter with [ER-II(F)].

4.2. Simultaneous use of evaluation standards
Eight types of evaluation standards were used as
mentioned above, which are assumed to be used for theevaluation of the process independently. However, it isnecessary to consider using several evaluation standards
simultaneously considering the practical use. Of course,
in this case, it is possible that the calculation of theevaluation function may become complicated.

•[ER-III] Error Recovery III
The final sequence is a recovery process carried out
by returning to a skill primitive in subtaskm of the
failure occurrence as shown in Fig. 11. A
disadvantage of this method is that the task to fasten ascrew temporarily to the hole in the plate part of thehook stuck perpendicularly is more difficult thanwhen it is placed levelly. The method uses a newscrew [ER-III(N)], and the method reuses the dropped
screw [ER-III(F)].

5. Precedence of Recovery Processes Based on
Various Evaluation Standards
In this section, the change in the priority of the recovery
process based on one evaluation standard chosen among
various standards discussed in Section 4 is observed via
a simple sample exercise.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

324

Using Various Evaluation Standards

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(i)

(j)

(k)

(h)

(l)

Fig. 7 An assembly task in which a hook is stuck
to a plate by four precision screws

5.2. Suitable process in each evaluation standard

(m)

(h)’

Fig. 8 An error in which a screw isdropped at (h) in Fig. 7[ER-I] with a restart operation becomes the mostsuitable process. [ER-II(N)] and [ER-III(N)] withplanned operation and without search operation
become the next suitable processes, whereas [ERII(F)] and [ER-III(F)] with search operation become
unsuitable processes. [ER-II(N)] and [ER-II(F)]without difficult operations are suitable for eachorder.

In the case of each evaluation standard shown in Section
4, let us consider a priority order of five recovery
processes [ER-I], [ER-II(N)], [ER-II(F)], [ER-III(N)],and [ER-III(F)]. The explanation is carried out
qualitatively, not quantitatively, for simplicity.
(i) Cost
Priority order: [ER-II(N)], [ER-III(N)], [ER-II(F)],[ER-III(F)], [ER-I].
When considering cost as the only evaluation
standard, the selection method of a suitable process
has already been explained.8 The costs include
material costs, part costs, electricity bills, andplanning expenses. [ER-II(N)] without search
operation and difficulty becomes the most suitableprocess, and [ER-I] where many parts are exchanged
become the most unsuitable one.

(iv) Safety
Priority order: [ER-I], [ER-II(N)], [ER-II(F)], [ERIII(N)], [ER-III(F)].
[ER-I] is the most suitable process. [ER-II(N)]and [ER-II(F)] without difficult operation andunstable posture become the next most suitableprocesses, whereas [ER-III(N)] and [ER-III(F)] withdifficult operation becomes a process that is notsuitable. [ER-II(N)] and [ER-III(N)] without a search
operation are suitable for each order.

(ii) Time
Priority order: [ER-III(N)], [ER-II(N)], [ER-I], [ERIII(F)], [ER-II(F)].
Here, [ER-III(N)] with a small return operation
and without any search operation becomes the mostsuitable process, while [ER-II(F)] with return andsearch operation becomes the most unsuitable.

(v) Finishing
Priority order: Same for all.
There was no difference in finishing with respect
to this task. However, the operation that may
adversely affect a target item or the surroundings,
such as scratch or scatter, should be avoided.
(vi) Recovery data
Priority order: A process with a large amount of data
is given priority.

(iii) Reliability
Priority order: [ER-I], [ER-II(N)], [ER-III(N)], [ERII(F)], [ER-III(F)].

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

325

Akira Nakamura, Natsuki Yamanobe, Ixchel Ramirez Alpizar, Kensuke Harada and Yukiyasu Domae
Using a new hook and screws

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(i)

(j)

(k)

(h)’

Discarded

(h)

(l)

(m)

Fig. 9 [ER-I] Error Recovery I

(p)

(h)’

(q)

(r)

(e)

(f)

(g)

(i)

(j)

(k)

(d)

(h)

(l)

(m)

Fig. 10 [ER-II] Error Recovery II

be fastened temporarily as for any posture of a target,[ER-III(N)] (or [ER-III(F)]) becomes a top-priority
process.

The amount of data set is important when using adata science method. Even if the execution ofoperation is extremely difficult, a considerable
amount of data enables the work to be performed
skillfully.

(viii) Operator skill
Priority order: A process of forte of teaching operator
is given priority.
The recovery process in which the teaching
operator is used is the most suitable. There is apossibility that a suitable process for an expert
operator is different from that of an unskilled
operator.

(vii) Tool
Priority order: A process with an exclusive tool isgiven priority.
If a special tool for recovery is available, there isa high possibility that the process surely returns in ashort time. If there is a tool by which four screws can© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

326

Using Various Evaluation Standards

(h)’

(i)

(j)

(k)

(u)

(v)

(l)

(m)

Fig. 11 [ER-III] Error Recovery III
3. E. D. Lello, M. Klotzbucher, T. D. Laet and H.
Bruyninckx, Bayesian Time-Series Models forContinuous Fault Detection and Recognition in Industrial
Robotics Tasks, in Proc. IEEE/RSJ Int. Conf. on Intell.
Robots Syst., (Tokyo, Japan, 2013), 5827-5833.
4. E. Krabbe, E. Kristiansen, L. Hansen and D. Bourne,
Autonomous Optimization of Fine Motions for Robotic
Assembly, in Proc. IEEE Int. Conf. Robot. Autom.,(Hong Kong, China, 2014), 4168-4175.
5. A. S. Wang and O. Kroemer, Learning Robust
Manipulation Strategies with Multimodal State Transition
Models and Recovery Heuristics, in Proc. IEEE Int. Conf.
Robot. Autom., (Montreal, Canada, 2019), 1309-1315.
6. A. Nakamura, K. Nagata, K. Harada, N. Yamanobe, T.
Tsuji, T. Foissotte and Y. Kawai, Error recovery using
task stratification and error classification for manipulation robots in various fields, in Proc IEEE/RSJ Int. Conf.
on Intell. Robots Syst., (Tokyo, Japan, 2013), 3535-3542.
7. A. Nakamura, K. Nagata, K. Harada and N. Yamanobe,
Technique of Recovery Process and Application of AI in
Error Recovery Using Task Stratification and Error
Classification, J. Robotics, Networking and Artificial
Life, 5(1), 2018, pp. 56-62.
8. A. Nakamura, N. Yamanobe, I. R. Alpizar, K. Harada
and Y. Domae, Cost-oriented Planning for Error
Recovery in an Automation Plant, J. Robotics,
Networking and Artificial Life, 6(4), 2020, pp. 225-230.
9. T. Hasegawa, T. Suehiro and K, Takase, A robot system
for unstructured environments based on an environment
model and manipulation skills, in Proc. IEEE Int. Conf.
Robot. Autom., (Sacramento, USA, 1991), 916-923.
10. A. Nakamura, T. Ogasawara, T. Suehiro and H. Tsukune,
Fine motion strategy for skill-based manipulation,
Artificial Life and Robotics, Springer, 1(3), 1997, pp.
147-150.
11. A. Nakamura, K. Kitagaki and T. Suehiro, Using
simplified geometric models in skill-based manipulation,
Advanced Robotics, 18(8), 2004, pp. 835-858.

We considered a suitable process derived by selecting
one evaluation standard at a time from eight. However,in actuality, there is a possibility that the optimum
process is derived using multiple evaluation standards.
6. Conclusion
When an error occurs in automated plants, the processadvances to the recovery part. In general, many types ofrecovery processes can be considered. In this study, we
have shown a method to derive a suitable process using
only one evaluation standard at a time selected fromvarious standards. The preferential order of recovery
processes can be obtained considering each evaluation
standard.
However, we have used only one standard, although
we did consider eight types of evaluation standards.
Therefore, in future studies, we will include techniques
to derive a suitable recovery process using several
evaluation standards simultaneously and determine theapplicability of our method to actual systems.
Acknowledgements
This work was supported by New Energy and Industrial
Technology Development Organization (NEDO) in
Japan.
References
1. B. R. Donald, Planning multi-step error detection andrecovery strategies, Int. J. Robot. Res., 9(1) (1990) 3-60.
2. T. Niemueller, G. Lakemeyer and S. S. Srinivasa, A
Generic Robot Database and its Application in Fault
Analysis and Performance Evaluation, in Proc. IEEE/RSJ
Int. Conf. on Intell. Robots Syst., (Vilamoura, Portugal,2012), 364-369.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

327

Robotic Picking for Piled Sushi Topping

Kenta Matsuura

Graduate School of Engineering Science, Osaka University
1-3 Machikaneyama, Toyonaka 560-8531, Japan
Currently with Yaskawa Electric Co. Ltd., Japan
Keisuke Koyama, Weiwei Wan, Kensuke Harada

Graduate School of Engineering Science, Osaka University
1-3 Machikaneyama, Toyonaka 560-8531, Japan
E-mail: {koyama, wan, harada}@sys.es.osaka-u.ac.jp

Abstract
This paper proposes a method for picking piled sushi topping. By observing a human motion picking a sushi topping,
we propose two picking strategies where one is to insert a finger into the separation among two toppings with shaking
the finger, and the other is to insert a finger into the separation between a topping and a table. Along with twosegmentation method, i.e., plane segmentation and LCCP segmentation methods, we experimentally verify theeffectiveness of the proposed approach.
Keywords: Robot, Picking, Food Elastic Object.

methods[2], and machine learning-based methods[3]1. Introduction
In recent years, labor shortage has become a serious
problem in the food manufacturing industry, and thedemand for automation of food manufacturing has been
increasing. For example, taking food out of containers is
a monotonous and heavy work for workers, so
automation using robots is expected. However, foodproducts are often flexible and irregularly shaped that do
not retain a specific shape, such as fish fillets and noodles.
This makes difficult to recognize their pose with vision
sensors or to grasp them with a robot hand. In this study,
we propose a recognition and grasping method focusing
on the characteristics of flexible irregular objects, andgrasp a sushi topping as an example (Fig.1).
In research on picking industrial parts without
deformation, model-based methods[1], visibility-based

Figure 1 Overview of robotic picking for piled sushi
toppings
have been proposed. However, none of them assume
flexible food as the object to be grasped. In contrast, in
this study, we assume a situation where sushi toppings
are stacked, and we consider grasping one of them. Whena human grasps a sushi topping, he or she moves his or
her fingers from side to side and puts his or her fingers in
between the sushi seeds, and grasps one of them by© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

328

Kenta Matsuura, Keisuke Koyama, Weiwei Wan, Kensuke Harada

the grasping pose. The density of the object was assumed
to be uniform, and the center of gravity was calculated by
dividing the sum of the x, y, and z coordinates of thepoints belonging to the cluster by the number of points inthe cluster for each cluster. The grasped object is selectedbased on the obtained center of gravity.
2.2 Grasping method
In this study, we propose two grasping methods based on
the human behavior of grasping sushi toppings. The first
method is to insert the finger into the trough between two
sushi toppings while shaking the finger to separate them(Fig. 3). The fingertip of the hand is first placed verticallyabove the separation between the two sushi pieces, andthe hand is tilted slightly about the x-axis shown in Fig.
3, and then moves in the vertically downward direction
until the hand hits the sushi piece. Then, the finger isinserted into the trough between the two sushi toppings
with shaking the finger. While inserting the finger, thehand is gradually tilted back about the x-axis. The handgrasps the sushi topping after sliding the finger into thesushi sopping.
The second method is to place a finger between the sushi
topping and the cutting board (Fig.4). In the coordinate
system shown in Fig. 4, the fingertip is placed verticallyabove the point with the largest x-coordinate of thecluster of the largest z-coordinate of the center of gravity.
Then, the hand is tilted about the y-axis where its rotation
center is located at the fingertip. Then, the hand is moved
vertically downward direction until it touches the cuttingboard. Then, the finger is inserted into the troughbetween a sushi topping and the cutting board. Whileinserting the finger, the hand is tilted back about the yaxis. The hand grasps the sushi topping after sliding thefinger into the sushi sopping.

Figure 2 Human strategy for picking piled sushi
topping
separating it (Fig. 2). This research examines thepossibility of using a robot to perform this kind ofgrasping task for a flexible object. We consider twosegmentation methods of point clouds and two grasping
methods based on the human behavior. We evaluate thesuccess rate of each segmentation method and eachgrasping method by conducting experiments using realsushi toppings.
Section 2 of this paper describes the proposed method,
Section 3 describes the experiments and their results, andSection 4 summarizes and discusses future plans.
2. Proposed Method
In this study, we use a depth sensor to obtain a point cloudand segment it. To pick a sushi topping, we use a twofingered parallel gripper where each finger is made ofsoft material. Section 2.1 describes the segmentation
method for the point cloud of each sushi species and thecalculation of the grasping pose, and Section 2.2
describes the method to grasp one sushi species based on
the center of gravity of each segment.
2.1 Segmentation
We consider two segmentation methods of captured point
cloud. The first method is to detect planes in a 3D point
cloud dataset using the RANSAC algorithm and tosegment the point cloud into planes [5] (Plane Model
Segmentation). The second method is to divide similar
voxels into groups of small convex regions, which areseparatedbyconcaveboundaries[6](LCCP
Segmentation). In this study, after segmentation using
Plane Model Segmentation[5] or LCCP Segmentation[6],the center of gravity of a cluster whose size is greater thana threshold is calculated for the purpose of determining

Figure 3 A finger is inserted into the separation
between two sushi toppings (Picking strategy 1)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

329

Robotic Picking for Piled3.1. Experimental Results and Discussions
In Figs. 6 and 7, (a) shows the sushi toppings arranged on
the cutting board, (b) shows the result of segmenting (a)into clusters by using the Plane Model Segmentation
where points with the same color belong to the samecluster, and (c) shows the result of segmenting (a) into
clusters by using the LCCP Segmentation. Fig. 8 showsthe experimental result using the grasping method 1, andFig. 9 shows the result of using the grasping method 2.
The number of successful grasps out of 10 trials with two
different segmentation methods is shown in Table 1.
In Plane Model Segmentation, as the distance between
two bumps became small, it often becomes impossible tosegment each sushi piece accurately. This is due to theunevenness and curvature of the sushi species since thismethod divides the point cloud into planes. Therefore, we
could not successfully grasp the object when theseparation between two pieces was 10 mm or 15 mm. On
the other hand, LCCP Segmentation succeeded insegmenting the region with high accuracy. In the Plane
Model Segmentation, we adjusted the thickness of eachcluster. On the other hand, in LCCP Segmentation, we
adjusted the angle at which the object was considered tobe concave. The grasping method 1 was able to grasp all
the sushi species at all distances between two pieces ifthey were divided into appropriate clusters. Even if there
was almost no gap in the trough between two sushi
species, it was sometimes possible to grasp them byinserting the fingers while moving it back and forth.
Grasping method 2 was successful when the distance
between two pieces was 15mm and 20mm. When the
distance was 10mm, the finger interfered with theneighboring sushi topping and could not insert the finger
between the sushi topping and the cutting board. In bothgrasping methods 1 and 2, the weight of the scallop waslarger than that of the salmon, so the scallop wassometimes dropped even after grasping it once if it was
grasped too shallowly.

Figure 4 A finger is inserted between a sushi topping
and the chopping board (Picking strategy 2)

3. Experiment
In our experiment, we use an industrial manipulator to
grasp salmon and scallops, which positions and sizes are
unknown as shown in Figure 5(a).
The robot grasps the object based on the information
about the position and shape of the object calculated fromthe obtained point cloud. After acquiring the point cloud,we segment it by using the two methods described in
Section 2.1. After segmentation, we calculate the centerof gravity of the cluster with more than 100 points. The
robot picks the object for 10 times by using two
segmentation methods and two grasping methods.
The hand used in the experiment is an electric handmanufactured by THK Corporation where its fingers are
made of silicon that allows the fingertips to maintain amoderate stiffness for grasping food (Fig. 5). The
dimensions of a finger are 45 mm in length, 20 mm inwidth, 30 mm in opening and closing width, and 6 mm inthickness of the fingertip. We used salmon and scallops,
and the scallops were opened and laid out on the table infront of the robot.

Figure 5 Electric hand used in our experiment© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

330

Kenta Matsuura, Keisuke Koyama, Weiwei Wan, Kensuke Harada

information and to find the center of gravity of clusters.
From the experiments, we confirmed that when theobjects are lined up with flexible irregular shapes, suchas the sushi species used in this study, LCCP
Segmentation can provide robust recognition, and thegrasping method1 is relatively flexible regardless of thespacing of the objects.
In the future, we will develop the segmentation method
used in this study to further improve the accuracy ofsegmentation and to estimate not only the center ofgravity but also the approximate size and angle of theobjects from the point cloud. In addition, we areinvestigating the construction of a system that can
flexibly grasp various types and arrangements of objects
instead of a fixed placement method.

Figure 6 Result of segmentation (Salmon)

Figure 7 Result of segmentation (Scallop)

References
1. K. Harada, W. Wan, T. Tsuji, K. Kikuchi,K. Nagata, and
H. Onda. Iterative visual recognition for learning based
randomized bin-picking. Experimental Robotics, pp. 646–
655. Springer, 2016.
2. Y. Domae, H. Okuda, Y. Taguchi, K. Sumi, and T. Hirai.
Fast graspability evaluation on single depth maps for bin
picking with general grippers. IEEE Int. Conf. on Robotics
and Automation (ICRA), pp. 1997–2004. 2014.
3. R. Matsumura, Y. Domae, W. Wan, and K. Harada,
Learning Based Robotic Bin-picking for Potentially
Tangled Objects, IEEE/RSJ Int. Conf. on Intelligent
Robots and Systems, pp. 7984-7991, 2019.
4. Sushi-master Toshi, https://youtu.be/QgcxLLZ7KJs, Jan.,2019.
5. A. Hast, J. Nysjo, and A. Marchetti. Optimal ransactowards a repeatable algorithm for finding the optimal set.
2013.
6. S. Christoph Stein, M. Schoeler, J. Papon, and F.
Worgotter. Object partitioning using local convexity,
IEEE Conference on Computer Vision and Pattern
Recognition, pp. 304–311, 2014.
7.

Figure 8 Result of picking strategy 1

Figure 9 Result of picking strategy 2

Table 1 Number of successful picking

4.

Conclusions

In this study, we proposed a unique grasping method forgrasping flexible irregular objects by using two different
methods to recognize objects from point cloud© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

331

Motion Generation by Learning Relationshipbetween Object Shapes and Human Motions
Tokuo Tsuji*1, Sho Tajima2, Yosuke Suzuki1, Tetsuyou Watanabe1,
Shoko Miyauchi3, Ken’ichi Morooka3, Kensuke Harada4, and Hiroaki Seki1

Institute of Science and Engineering, Kanazawa University,
Kakuma-machi, Kanazawa-shi, Ishikawa 920-1192, Japan.

1

Graduate School of Natural Science and Technology, Kanazawa University,
Kakuma-machi, Kanazawa-shi, Ishikawa, 920-1192, Japan.

2

Faculty of Informatqion Science and Electrical Engineering, Kyushu University,744, Motooka, Nishi-ku, Fukuoka-shi, Fukuoka 819-0395, Japan.

3

Department of Systems Innovation, Graduate School of Engineering Science, Osaka University,
Machikaneyama-chou, Toyonaka-shi , Osaka 560-8531, Japan4

E-mail: {tokuo-tsuji, suzuki, twata, hseki}@se.kanazawa-u.ac.jp

Abstract
This paper presents a method for planning a robot motion of daily tasks by learning the relationship between object
shapes and human motions. Robots are required to be able to deal with multifarious objects in various categories.
However, it is difficult for robots to plan motions automatically for performing a task because objects even in thesame category have different shapes. In our method, the motions are estimated by learning the relationship between
object shapes and human motions using linear regression analysis.
Keywords: Motion Planning, Learning by demonstration, Machine Learning, Linear Regression Analysis, Robothand

the method for planning motions automatically according
to object shapes is needed. Most motion planners ofprevious works are used for pick-and-place task andassembly task. However, it is difficult to plan acomplicated motion like pouring water by using them.
We propose a method for planning motions bylearning the relationship between shapes and human

1. Introduction
Robots are expected to handle various objects in the fieldsuch as variable volume production, sorting of logistics,
and daily tasks of life support. Since industrial objects
and daily-life objects have different shapes even in thesame category, robots have to adjust motions according
to the difference in object shapes. For example, it isconsidered that the motion for pouring water with a bottle
is generated by human teaching. Even though the motion
can be used for the bottles which have the same shape,
the motion may not be used for the bottles which havedifferent shapes. The trajectory and execution timeshould be different according to object shapes. Since it is
impossible to teach motions of every shapes manually,

Fig. 1. Overview.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

332

Tokuo Tsuji, Sho Tajima, Yosuke Suzuki, Tetsuyou Watanabe, Shoko Miyauchi, Ken’ichi Morooka, Kensuke Harada, Hiroaki Sekimotions. We applied the proposed method to the task ofpouring water from plastic bottles (Fig.1).
Our main contributions in this paper are as follows.
(1) Estimating shapes from measurement data using an
RGB-D camera with one viewpoint(2) Learning the relationship between human motions
and object shapes for the motion generation in advance(3) Acquisition object shapes and human motions byusing an RGB-D camera and motion capture system(4) Planning a motion automatically according to shapeparameters acquired automatically(5) Generating a motion for novel objects which hasdifferent shapes in the same category without a trial
Motions of pouring water from bottles into a cup arecorrelated with shapes of plastic bottles and cups.
However, we assume the following things to verify theeffectiveness of our proposed method. The object shapes
except the manipulated object are the same. The initialwater mass in the plastic bottles are known and constant.

Fig. 2. Overview of motion generation.

It is necessary to handle the different shapes of objects
so that motions can be planned with consideration fortasks, objects, and environments simultaneously.
However, the previous researches uses simple rotation
for pouring motions and is not adapted to arbitrary
trajectory which depends on the shape of bottles. In thisstudy, we propose a motion trajectory generation tohandle different shapes of manipulated objects.

2. Related works
In daily tasks, a task of folding cloth has been learned byrobots based on working images [1]. Ochi et al. [2]proposed for scooping powder with a specific spoon byusing deep learning. Rozo et al. [3] performed thepouring motion by learning time series data of joint
angles and force sensor from pouring motion by a robot
which human manipulated. Schenck et al. [4] poured a
specific amount of liquids for different containers byusing real-time visual feedback in a PID controller anddetermining the amount of liquid in a container fromvision sensor during a pouring action. Lopez et al. [5]optimize pouring motion for novel containers as areceiver efficiently by learning the motion to mitigate
spillage from the simulation of the pour. Kroemer et al.
[6] learned from the information of object shapes andhuman demonstrations and generalize the pouring motion
for novel containers by using a trial-and-error approachquickly. Tamosiunaite et al. [7] learned motion
primitives by reinforcement learning. The pouringmotion was performed efficiently because the learned
motion was adapted to novel objects and the motion was
relearned. Although the shapes of containers as a receiver
were considered, the shapes of pouring containers were
not considered in these researches or they needed sometrials to perform pouring motions.

3. Overview of motion generation
A flow chart of the overview is shown in Fig.2. The
proposed method consists of two phases: learning andexecution.
In the learning phase, two types of learning are
performed. The first type constructs a statistical shapemodel (SSM) [8] from learning data collected fromobjects in the same category with different shapes. The
second type derives relational equations between shapes
and time series of human motions via linear regression
analysis. The learning phase is completed prior to the
execution phase.
In the execution phase, the shape of an object isestimated using an SSM. Motions are estimated by thelinear regression equations based on the parameters of the
shape obtained by the SSM.
4.

Modeling and estimation of shapes by the
SSM

SSMs of main objects are prepared for each category in
the learning phase. In the execution phase, after
recognizing the category of an object, its shape isestimated by using an RGB-D camera and expressed byvarious parameters. Shape estimation is performed by© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

333

Motion Generation by Learning

using point clouds from depth images and contours from
RGB images. An SSM describes an average shape andvariations of multiple shapes that vary between different
objects. A three-dimensional point cloud is resampled by
mapping it onto an arbitrary target surface [8].
The shape vector 𝑋𝑋𝑖𝑖 (𝑖𝑖 = 1, … , 𝑛𝑛) of an SSM isrepresented as follows:
𝑋𝑋𝑖𝑖 = [x𝑖𝑖0 , yi0 , zi0 , … , xim , yim , zim ](1)
where the number of points is 𝑚𝑚 and the number of shapemodels is 𝑛𝑛. A deformed shape vector 𝑋𝑋𝑖𝑖 for the SSM is
obtained by summing the calculated average shape vector𝑋𝑋� and vectors 𝑈𝑈𝑖𝑖 as follows:𝑛𝑛
𝑥𝑥 = 𝑋𝑋� + Σ𝑖𝑖=1
𝑈𝑈𝑖𝑖 𝑏𝑏𝑖𝑖 (2)where the values of the shape parameters 𝛽𝛽 =[𝑏𝑏1 , 𝑏𝑏2 , … , 𝑏𝑏𝑛𝑛 ] represent the weight vector for each vector.
The deformed shapes for each category are expressed bychanging 𝛽𝛽.
Main objects are measured from one viewpoint by an
RGB-D camera for shape estimation. A point cloud of thesurface of a main object is extracted from a depth image.
The contours of a main object are extracted from an RGB
image. The iterative closest point algorithm (ICP) is used
for registration between the point cloud of a depth image
and that of an SSM. 50 SSMs were generated via random
deformation for finding the parameters which have theminimum error. This procedure repeats until the shapeparameters 𝛽𝛽 are constant for ten iterations. Some
examples of shape estimation via SSM are presented inFig. 3.

Fig. 3. Shape estimation (a) RGB Image, (b) Measurement
point cloud, (c) SSM.

5.2. Linear regression
Linear regression is applied to the data for the shapeparameters and motions and a regression coefficientmatrix is calculated. Let 𝑛𝑛 be the number of learningmodels for the construction of an SSM.
Let 𝑁𝑁 be the number of learning data for linear regression.
Only motion in a two-dimensional plane is considered.
The position in the horizontal direction is defined as
𝑦𝑦 and the position in the vertical direction is defined as 𝑧𝑧.
The rotation angle in the 𝑦𝑦𝑦𝑦 plane is defined as 𝜃𝜃.
A linear equation is formulated from the shapeparameters B = [β1 , β2 , … βN ]𝑇𝑇 and the position in thehorizontal direction at a particular time 𝑌𝑌 =[y1 , y2 , … , yN ]𝑇𝑇 . The normal equation for the linear
equation 𝑌𝑌 = 𝐵𝐵𝐵𝐵 is solved as follows:
𝑃𝑃 = (𝐵𝐵𝑇𝑇 𝐵𝐵)−1 𝐵𝐵𝑇𝑇 𝑌𝑌(3)
A numerical solution (𝐵𝐵𝑇𝑇 𝐵𝐵)−1 is derived via singularvalue decomposition. The posture and the position in thevertical direction at a particular time are also calculated.
The regression coefficient matrices for the execution
time are obtained in the same manner for motion1,
motion2, …, and motion𝑁𝑁𝑡𝑡 . The products of the shapeparameters 𝛽𝛽 from novel objects and regression
coefficient matrices 𝑃𝑃 are calculated and the relative
position and posture of the main object for each section(time series), as well as the execution time $t$ for eachmotion are estimated.

5. Motion generation
The shape of a main object is represented by shapeparameters 𝛽𝛽 . The relationships between the shapeparameters 𝛽𝛽 and a human motion are learned andmodeled by using linear regression analysis during thelearning phase. The motion of a main object is estimatedby using linear regression analysis during the execution
phase.

6. Evaluation

5.1. Learning data acquisition

We evaluated the proposed method for pouring water into
a cup from a plastic bottle. The generated motions were
evaluated experimentally using an actual robot. Shapemodels of nine plastic bottles were used for constructing
an SSM with n=9. Ten real plastic bottles that weredifferent from the nine shape models for the SSM wereused in our experiments. Their point cloud data were

The speeds of human motions are different in each
iteration. Therefore, we divide a motion into several
smaller motions at various key frames and acquirelearning data for an execution time 𝑡𝑡 . The divided
motions are defined as motion1, motion2, … , andmotion𝑁𝑁𝑡𝑡 .

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

334

Tokuo Tsuji, Sho Tajima, Yosuke Suzuki, Tetsuyou Watanabe, Shoko Miyauchi, Ken’ichi Morooka, Kensuke Harada, Hiroaki Sekimeasured and shape parameters 𝛽𝛽 were estimated 10
times for each bottle. Human motions of pouring water
into the cup were performed 10 times for each bottle. The
initial water mass in each bottle was 0.2 kg and 0.1 kg ofwater was poured into the cup. The number of the dataset
is 100 sets in total.
The data from nine out of the ten plastic bottles wereused as learning data ($N$=90) and the data from theremaining bottle were used as testing data. Robotmotions were generated by estimating the positions andpostures of main objects for each section, as well as theexecution times for each motion based on shapeestimation of the testing data. We used five sequences per
object and performed 50 experiments using an actual
robot.
In these experiments, the motions generated based on
the prepared dataset were verified by using an actual
robot. The experimental conditions were as follows. The
initial position and posture of the cup and plastic bottles
were determined manually. The grasping points of theplastic bottles were also determined manually. The initialwater mass in the plastic bottles was 0.2 kg. The
experiment is shown in Fig. 4. The success rates of
avoiding spilling water is 92% (46/50).

Fig. 4. Experiment.
3. L. Rozo, P. Jim´enez, and C. Torras, Force-based robot
learning of pouring skills using parametric hidden Markov
models, Robot Motion and Control, 2013 9th Workshop
on. IEEE, 2013, pp. 227-232.
4. C. Schenck and D. Fox,Visual closed-loop control forpouring liquids.,In 2017 IEEE International Conference on
Robotics and Automation, pp. 2629-2636. IEEE, 2017.
5. T. Lopez-Guevara, N. K. Taylor, M. U. Gutmann,
S.Ramamoorthy, and K. Subr,Adaptable Pouring,
Teaching Robots Not to Spill using Fast but Approximate
Fluid Simulation, Proceedings of the 1st Annual
Conference on Robot Learning, vol.78, pp. 77-86, 13-15
Nov 2017.
6. O. Kroemer, E. Ugur, E. Oztop, and J. Peters, A kernelbased approach to direct action perception, in 2012 IEEE
International Conference on Robotics and Automation,
may 2012, pp. 2605-2610.
7. M. Tamosiunaite, B.Nemec, A. Ude, and F. Worgotter,
Learning to pour with a robot arm combining goal andshape learning for dynamic movement primitives,
Robotics and Autonomous Systems, vol. 59, no. 11, pp.
910-922, 2011.
8. S. Miyauchi, K. Morooka, Y. Miyagi, T. Fukuda, T. Tsuji
and R. Kurazume, Tissue Surface Model Mapping onto
Arbitrary Target Surface Based on Self-organizing
Deformable Model, Workshop on Computer Vision at the
4th International Conference on Emerging Security
Technologies，pp.79-82，2013.

7. Conclusions
We proposed a method for generating motions by
estimating the positions and postures of main objects andthe execution time for each motion based on shapeestimation. We performed position and posture
estimations and obtained results very close to thevariation of positions and postures in human motions. In
experiments using an actual robot, the success rate forpouring water into a cup was 92% in terms of avoiding
spilling water. In the future, we plan to generate motions
based on shape changes in target objects, the initial water
masses in plastic bottles for generalization of the
proposed method.
References
1. P.-C. Yang, K. Sasaki, K. Kase, S. Sugano and T. Ogata,
Repetable Folding Task by Humanoid Robot Worker
using Deep Learning, IEEE Robotics and Automation
Letters(RA-L), pp. 397-403, 2017.
2. H. Ochi, W. Wan, Y. Yang, N. Yamanobe, J. Pan and K.
Harada, Deep Learning Scooping Motion using Bilateral
Teleoperations, Proc. of IEEE International Conference on
Advanced Robotics and Mechatronics, 188, 2018.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

335

Simulation and Regression Testing for Behavior of Software Models
Based on Extended Place/Transition Net with Attributed Tokens
Tomohiko Takagi

Faculty of Engineering and Design, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, JapanRyo Kurozumi

Graduate School of Engineering, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
E-mail: takagi@eng.kagawa-u.ac.jp, s19g457@stu.kagawa-u.ac.jp

Abstract
We propose a technique of simulation and regression testing for Extended Place/transition Net with Attributed Tokens(EPNAT) models, and then show an early prototype tool to partially support it. In the technique, the information
about a current marking (a current distribution of attributed tokens, including current values of attributes), current
values of global variables, and current fireable transitions is indicated for the simulation, and also the good execution
traces in the simulation are recorded as test cases for the regression testing. When an EPNAT model is modified, thetest cases can be applied to it in order to reveal regression failures.
Keywords: software modeling, place/transition net, VDM, simulation, regression testingunderstand such complex aspects of the EPNAT model
when constructing, validating and refining it.
In order to address this problem, we propose atechnique of simulation and regression testing for
EPNAT models, and then show an early prototype tool topartially support it. In the technique, the information
about a current marking (that is, a current distribution of
attributed tokens, including current values of attributes),
current values of global variables, and current fireable
transitions is indicated for the simulation, and also thegood execution traces in the simulation are recorded as
test cases for the regression testing. When an EPNAT
model is modified, the test cases can be applied to it inorder to reveal regression failures.
This paper is organized as follows. Section 2
describes our technique of simulation and regression
testing for EPNAT models, and then section 3 illustrates

1. Introduction
Formal specifications that represent abstracted software
requirements in unambiguous and executable forms play
an important role in the development of high-quality
software. Extended Place/transition Net with Attributed
Tokens (EPNAT)1 is a formal specification description
language for modeling the expected behavior of statetransition-based software that consists of multiple objects,
such as modules and subsystems. In an EPNAT model,
each attributed token corresponds to an object and hasvariables to characterize the object, which are called
attributes. The firing of transitions leads to the increase,decrease, and move of attributed tokens, the change ofvalues of attributes and global variables, and so on. The
behavior is constrained by invariants, pre-conditions,post-conditions, and type constraints. Engineers need to© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

336

Tomohiko Takagi, Ryo Kurozumi

our early prototype tool. Finally, we show discussion andfuture work in section 4.

start
Step 1. Specify a starting state.

2. Technique Overview

yes

In this section, we propose the technique of simulation
and regression testing for EPNAT models.

violated?
no

Step 2. Initialize the current state to the starting state.
Step 3. Evaluate pre-conditions of all the transitions.

2.1. Simulation of EPNAT models

no fireable
transitions?
no

The simulation is intended to help engineers to construct,validate, refine, and understand EPNAT models, and it isapplicable to both completed EPNAT models and
EPNAT models under construction. The simulation of agiven EPNAT model is performed according to thefollowing procedure. The overview of the procedure isshown in Fig. 1. Step 2, 3, 5, and 7 can be automatically
executed, and should be supported by a tool.
Step 1. An engineer specifies a starting state (that is, astate to start the simulation) according to the aim of
the simulation. In this study, a state is characterized
by a marking and values of all the global variables in
an EPNAT model. Note that a marking in an EPNAT
model includes values of attributes. The starting state
does not necessarily need to be an initial state (that is,a state established just after software is invoked) or
reachable state (that is, a state that can be reachedfrom the initial state).
Step 2. If the starting state violates any type constraints
and invariants, the procedure returns to Step 1.
Otherwise, the current state of the given EPNAT
model is initialized to the starting state.
Step 3. The pre-conditions of all the transitions areevaluated in order to determine fireable transitions in
the current state. If there are no fireable transitions,the simulation is terminated.
Step 4. The engineer confirms the determined fireable
transitions and acceptable values of their arguments.
If he/she finds any faults, the simulation will be
stopped. Otherwise, the engineer selects one arbitrary
fireable transition and specifies values of its
arguments according to the aim of the simulation.
Step 5. If the specified values of the arguments violate
any type constraints and invariants, the procedure
returns to Step 4. Otherwise, the selected transition isfired, and then its actions are executed by using thespecified values of arguments. As a result of them, thecurrent state is changed. If any invariants, and theyes

Step 4 (1). Confirm the fireable transitions andacceptable values of their arguments.
faults?
no

yes

Step 4 (2). Select one arbitrary fireable transition.
Specify values of its arguments.
yes

violated?
no

Step 5. Fire the selected transition, and execute its actions.
violated?
noyes

Step 6. Confirm the current state.
no (Step 7)

faults? orthe aim achieved?
yes

end* The shaded parts can be automatically executed.

Fig. 1. Procedure of simulation (overview).

post-condition of the fired transition are violated due
to some sort of failures included in the given EPNAT
model, the simulation will be stopped.
Step 6. The engineer confirms the current state. If he/she
finds any faults, or if he/she achieves the aim of thesimulation, the simulation will be terminated.
Step 7. This procedure returns to Step 3.
2.2. Regression Testing of EPNAT models
A good execution trace in the simulation is recorded as atest case for the regression testing. In this context, theword "good" means that (a) the execution trace does notinclude the occurrence of faults, (b) the execution trace isuseful for the growth of test coverage, (c) the execution
trace corresponds to the typical use of software, and (d)the execution trace is useful to cover fault-prone parts2.
The item (a) is particularly essential for a test case.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

337

Simulation and Regression Testing<test-case> ::= <state> (<fireable-transitions> <state-transitions>)<state-transitions> ::= <transition> <state> (<fireable-transitions> <state-transitions>)<fireable-transitions> ::= <event> (<fireable-transitions>)<transition> ::= <event> <values-of-arguments><state> ::= <marking> <values-of-global-variables> | <violation-state><marking> ::= <place> <attributed-tokens-in-the-place> (<marking>)<attributed-tokens-in-the-place> ::= <attributed-token> (<attributed-tokens-in-the-place>)<attributed-token> ::= <object-type> <values-of-attributes>

Fig. 2. Abstracted structure of a test case for an EPNAT model (written in BNF).

differences between expected output and test output
through the execution of a given test case, it is concluded
that the test case has successfully passed. Otherwise, anengineer needs to find and fix a regression failure, or
update the test case so as to reflect the latest true software
specification. After that, the engineer should performconfirmation testing, that is, apply the failed test case to
the fixed EPNAT model, or apply the updated test caseto the EPNAT model.

When an EPNAT model is modified, the test cases
are applied to it in order to reveal regression failures.
Some model-based coverage criteria3,4 can be introduced
to evaluate the effectiveness of the regression testing.
Additionally, EPNAT models can be converted into
VDM++ specifications1,5, and thus common code-based
coverage criteria6 also can be introduced. In general,
multiple test cases need to be executed to satisfy
coverage criteria. Also, regression testing is repeatedly
performed, and thus it should be automated by a tool inorder to save time and effort.
Fig. 2 shows the abstracted structure of a test case for
an EPNAT model. It is written in BNF, and parentheses
are used to represent an optional element. A test case is asequence of successive state transitions of arbitrary
length. Each transition is identified by an event, and canhave values of arguments. Each transition should follow
fireable transitions that include its event. Fireable
transitions are determined by a state, and thus should
follow a state in a test case. Each state is characterized bya marking and values of all the global variables. If aninvalid value is given to an argument of a transition inorder to test invariants and a post-condition, thefollowing state should be a violation state, that is, a statein which some invariant or the post-condition has been
violated. A marking is expressed as a sequence of
attributed tokens in each place, and an attributed token ischaracterized by object type and values of attributes.
Each place can hold one or more attributed tokens, but
the place and its attributed tokens need to have the same
object type. Therefore, the object type of each attributed
token is important information in regression testing of
EPNAT models.
Transitions and a starting state correspond to test data(also called test input). On the other hand, fireable
transitions and states excepting the starting statecorrespond to expected output. When there are no

3. Early Prototype Tool
This section shows our early prototype tool to partially
support the simulation and regression testing of EPNAT
models. The tool includes two functions. One is an
EPNAT editor to support the construction of EPNAT
models, which has been developed in our previous study1.
Another is an EPNAT simulator we are developing in thisstudy, and it can be invoked from the EPNAT editor. The
EPNAT simulator interacts with an existing tool called
VDMJ5 in order to execute a given EPNAT model.
Fig. 3 shows a screen shot of the EPNAT simulator
that executes the simulation of the EPNAT model that
represents the behavior of a simple load balancer1. The
EPNAT model under simulation is visualized on the rightpane of the EPNAT simulator. Fireable transitions in thecurrent state are highlighted in green. Therefore, anengineer as a user of the EPNAT simulator will be able
to easily confirm them, and select a next transition inorder to proceed with his/her simulation. Also, thecurrent state and the simplified execution trace(described as a sequence of fired transitions) areindicated on the left pane. When an engineer selects a
fireable transition and specifies its values of argumentson the right pane, the EPNAT simulator automatically
executes the firing of the selected transition, and thenupdates the current state, the execution trace, and thegraphical image of the EPNAT model. An engineer can© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

338

Tomohiko Takagi, Ryo Kurozumi

stop and reset the current simulation at any time. Whenan engineer thinks that the execution trace is good for atest case in regression testing, he/she can save it.
An engineer can start automated regression testing byloading a test case and an EPNAT model to be tested.
4. Discussion and Future WorkIn this paper, we have proposed a technique of simulation
and regression testing for EPNAT models, and then haveshown an early prototype tool to partially support it. In
the technique, the information about a current marking(including current values of attributes), current values ofglobal variables, and current fireable transitions isindicated for the simulation, and also the good execution
traces in the simulation are recorded as test cases for the
regression testing. When an EPNAT model is modified,
the test cases can be applied to it in order to reveal
regression failures.
When a failure that was caused in the earlier stage ofsoftware development is found in the later stage(typically, the processes of system testing and acceptancetesting), the cost to fix it generally tends to become higher.
Therefore, it is important to find and fix failures in theearlier stage, and it is expected that our technique can beapplied to address this problem. However, our tool isunder development, and the functions to support our
technique are not completely implemented at present.
Also, the following matters need to be tackled in future
work in order to improve the technique:• An engineer will often need to confirm that the setof reachable states in an EPNAT model underconstruction includes all the indispensable states tosatisfy given software requirements and also the set
does not include any invalid states. However, test
cases are manually created in our technique and tool,and thus it will be difficult for most engineers to dosuch task systematically at small cost. Modelchecking7 may be useful to address this problem.
• After an engineer has made a modification on an
EPNAT model, he/she will often need to maintainsome existing test cases, that is, to update someexisting test cases so as to reflect the latest true
software specification. A technique and tool tosystematically support it should be constructed.
• Coverage criteria are useful to create good execution
traces in simulation and to select good test cases inregression testing. Engineers will need guidelines forthe effective use of coverage criteria, and will needFig. 3. Screen shot of an early prototype tool (EPNAT simulator).

a tool to automatically suggest execution traces andtest cases according to coverage criteria.
Based on the above, we will develop the tool tosupport our extended technique, and apply it to pilot
projects in order to evaluate its effectiveness.
Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP17K00103.
References
1. T. Takagi and R. Kurozumi, Software ModelingTechnique and its Prototype Tool for Behavior of Multiple
Objects Using Extended Place/Transition Nets with
Attributed Tokens, Journal of Robotics, Networking andArtificial Life, Vol.7, No.3, pp.194-198, Dec. 2020.
2. J.A. Whittaker, J. Arbon and J. Carollo, How Google Tests
Software, Addison-Wesley Professional, 2012.
3. T. Takagi, R. Kurozumi and T. Katayama, State Transition
Tuple Coverage Criterion for Extended Place/Transition
Net-Based Testing, Proceedings of Pacific Rim
International Symposium on Dependable Computing,pp.29-30, Dec. 2019.
4. T. Takagi, N. Oyaizu and Z. Furukawa, Concurrent Nswitch Coverage Criterion for Generating Test Cases from
Place/Transition Nets, Proceedings of 9th International
Conference on Computer and Information Science,
pp.782-787, Aug. 2010.
5. J. Fitzgerald, P.G. Larsen, P. Mukherjee, N. Plat and M.
Verhoef, Validated Designs for Object-Oriented Systems,
Springer-Verlag London, 2005.
6. B. Beizer, Software Testing Techniques, Van Nostrand
Reinhold, 2nd edition, 1990.
7. E.M. Clarke Jr., O. Grumberg and D. Peled, Model
Checking, MIT Press, 1999.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

339

Development of an Early Prototype Tool for Learning Software Modeling
Using Extended Place/Transition Net
Tomohiko Takagi

Faculty of Engineering and Design, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
Akio Usuda

Faculty of Engineering, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
E-mail: takagi@eng.kagawa-u.ac.jp, s17t213@stu.kagawa-u.ac.jp

Abstract
This paper shows an early prototype tool for learning software modeling using Extended Place/transition Net (EPN),
and then gives the discussion about its effectiveness and challenges. A user of the tool, that is, an engineer tries toconstruct his/her EPN model based on given software requirements by selecting and putting the given components
of EPN. The EPN model is converted into a VDM++ specification for a user who is familiar with Vienna
Development Method (VDM). Also, the behavior of software based on the EPN model is partially visualized by using
animated graphics for a user who is a learner at the first stage. In the end, the correctness of the user's EPN model is
automatically checked.
Keywords: software modeling, place/transition net, VDM, personal on-demand learning

the given components of EPN. The EPN model isconverted into a VDM++ specification3 for a user who isfamiliar with Vienna Development Method (VDM). Also,
the behavior of software based on the EPN model ispartially visualized by using animated graphics4,5 for auser who is a learner at the first stage. In the end, thecorrectness of the user's EPN model is automaticallychecked. This study is intended to support personal ondemand learning, and the key ideas of this study are (1)the construction using components, (2) the conversion
into VDM++ specifications, and (3) the visualization
using animated graphics.
This paper is organized as follows. Section 2 shows
the steps to learn software modeling using EPN in thisstudy. Section 3 illustrates the early prototype tool we aredeveloping in this study, and then section 4 gives thediscussion about its effectiveness and challenges.

1. Introduction
Extended Place/transition Net (EPN)1 is Place/transition
Net (PN) that includes some additional elements written
in VDM++2 to enhance its representation power, and canbe used to formally model the state transition-based
behavior of software in development processes. An EPN
model, that is, a software model drawn by using EPN, canbe executed on interpreters to understand and validate
software specifications, and also can be converted toanother formal software specification, source codes, andtest cases. However, the use of EPN is based on technicalknowledge and skills, and therefore engineers will needto learn them.
This paper shows an early prototype tool for learning
software modeling using EPN. A user of the tool, that is,an engineer tries to construct his/her EPN model basedon given software requirements by selecting and putting© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

340

Tomohiko Takagi, Akio Usuda

2. Learning Software Modeling Using EPN...

This section shows the three steps to learn softwaremodeling using EPN in this study, that is, (1) creating
exercises, (2) working on exercises, and (3) checking
learner's answers. They have been developed based on
our previous study5.

......
......
....

...

...

...
...

......
......
....

...

......
......
....

...

......
......
....

......
......
....

...

......
......
....

Legend...

placetransition

...

...

...

......
......
....

token
arc
......
......
....

...

extension(a) Completed EPN model

2.1. Creating exercises
In the first step, instructors, that is, skilled engineers
create exercises for learners. Each exercise consists of (i)software requirements, (ii) a completed EPN model, (iii)component candidates, (iv) animated graphics, and (v)hints to construct EPN models. They are generally
created in this order.

...

......
......
....

...

...

2.1.1. Software requirements

...
...

......
......
....

...

......
......
....

...

......
......
....

...

......
......
....

...

...

......
......
....

...

...

......
......
....

...

(b) Correct components for learners at the first stage

The software requirements are written in natural
languages. They should include enough information to
construct a correct EPN model in the next step, such as
the use case scenarios, state transitions, data processing,and constraints of the software.

......

...
......

......
...

....

......

......

...

......
...

2.1.2. Completed EPN model

...

....
......

...

......

......

The completed EPN model is a correct answer in theexercise, and will be used to check learner's answers in
the last step. It should be strictly based on the software
requirements.

......
...

...

....

......

......

....

......
...

2.1.3. Component candidates

...

....

......
...

...

....
...

....

(c) Correct components for advanced learners

The component candidates that will be used by learners
for constructing their EPN models in the next step areclassified into the following two sets.
One is the set of correct components, and they are
obtained by disassembling the completed EPN model, as
shown in Fig. 1. When the exercise is intended forlearners at the first stage, the size of each component may
be made bigger in order to reduce the level of itsdifficulty. Another is the set of incorrect components, andthey are created by mutating the correct components.
Model-based mutation operators6 can be applied to theelements of PN (that is, places, transitions, arcs, andtokens), and traditional mutation operators can be applied
to the extension from PN to EPN (that is, actions andguards written in VDM++). When the exercise isintended for learners at the first stage, the set of incorrect

Fig. 1. Creation of the set of correct components (overview).

components may be made smaller or empty in order toreduce the level of its difficulty.
The instructors should confirm whether thecomponent candidates lead to any other correct answers,that is, EPN models that do not have the same structure
as the completed EPN model but satisfy the software
requirements.
2.1.4. Animated Graphics
The animated graphics consist of several graphical parts,and visualize the behavior of software based on a given
EPN model. Some of the graphical parts are programmed
to move by trigger, such as the fire of specific transitions© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

341

Development of an Early

and the satisfaction of specific conditions in a given EPN
model.
2.1.5. Hints to construct EPN models
The hints are written in natural languages, and will be
used by learners as clues about how their EPN models
can be correctly constructed. They are not always needed
when the exercise is for advanced learners.

(A)

2.2. Working on exercises(B)

In the second step, learners work on the exercises the
instructors have created. The learners are given all thematerials excepting the completed EPN model. They will
firstly try to understand the given software requirements
and hints to construct EPN models. After that, they will
select arbitrary components from the given component
candidates in order to add onto their EPN models. The
learners' EPN models under construction areautomatically converted into VDM++ specifications3,
and learners who are familiar with VDM will review the
VDM++ specifications in order to confirm their EPN
models from another viewpoint. Also, learners at the firststage will watch animated graphics in order to confirm
their EPN models intuitively4,5. When learners finish
constructing their EPN models, they move to the last step.

(C)

(E)

(D)

(F)

Fig. 2. Screen shot of our early prototype tool.

2.3. Checking learner's answers

Fig. 2 is the screen shot of the tool that shows asample exercise on the subject of a simple elevator
control system. The tool is used on a Web browser. Its
GUI chiefly consists of (A) the description of software
requirements, (B) the description of hints to construct
EPN models, (C) the pane to construct a learner's EPN
model, (D) the pane to select component candidates, (E)the pane to show a VDM++ specification, and (F) the
pane to show animated graphics.
In (D), component candidates are classified by thekinds of elements of EPN, that is, places, transitions,tokens, guards and actions. A learner can select anarbitrary one from (D), and can move it to (C).
Components that have been moved from (D) can be
returned to (D) by the learner. When a learner's EPN
model is changed, its VDM++ specification shown in (E)is automatically updated in order to help a learner who isfamiliar with VDM to confirm his/her EPN model fromanother viewpoint. Also, in order to help a learner at thefirst stage to confirm his/her EPN model intuitively, (F)In the third (that is, the last) step, an EPN model that has
been constructed by a learner in the second step iscompared to the completed EPN model that has been
constructed by the instructors in the first step. If they areexactly the same, it is concluded that the learner has
successfully constructed the correct EPN model. If they
are not the same, the learner shall try to correct all the
mistakes on his/her EPN model. If needed, the learner isgiven additional hints to construct the correct EPN model,such as the information about the mistaken parts in
his/her EPN model and the unsatisfied items in the given
software requirements.
3. Early Prototype Tool
In this study, we are developing an early prototype tool
for learning software modeling using EPN. The tool does
not fully support the steps that have been discussed in theprevious section, but includes some essential functions.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

342

Tomohiko Takagi, Akio Usuda

study. Ref. 4 shows a training support method and tool
for bug fixing of EPN models. The characteristics of thisprevious study are to introduce animated graphics and to
focus on bug fixing. The idea of the animated graphics isused also in this study.

is intended to show the animated graphics of the elevatorof which behavior is based on his/her EPN model. A
learner can check the correctness of his/her EPN model,that is, can perform the automatic comparison between
his/her EPN model and the correct EPN model at any
time. The result of checking the correctness is indicatedas "O" or "X" that are the symbols for a correct/incorrect
answer, respectively. When a learner's EPN model isincorrect, the messages about mistakes may be given to
the learner in order that he/she retries to the exercise.
When the learner gives up constructing his/her EPN
model, the completed EPN model is shown in (C).

5. Conclusion
In this paper, we showed an early prototype tool forlearning software modeling using EPN, and then gave thediscussion about its effectiveness and challenges. The
key ideas of this study are (1) the construction using
components, (2) the conversion into VDM++
specifications, and (3) the visualization using animatedgraphics. In a future study, we plan to develop the
prototype tool and conduct preliminary experiments toimprove our method and tool.

4. Discussion
As shown in the previous sections, the key ideas of thisstudy are (1) the construction using components, (2) the
conversion into VDM++ specifications, and (3) thevisualization using animated graphics. (1) will be usefulto adjust the level of the difficulty of exercises. (2) will
help learners who are familiar with VDM to confirm their
EPN models from another viewpoint. (3) will help
learners at the first stage to confirm their EPN models
intuitively. However, through the development of theearly prototype tool, we found the following challenges
to be addressed in future:• As with the method of Ref. 4, the creation of
exercises, particularly the creation of animatedgraphics will require hard effort, and authoring tools
should be introduced to support it.
• The animated graphics will not be so easy to quickly
reveal learner's mistakes included in a large andcomplex EPN model. Optimized test case generation
techniques may need to be introduced to solve it.
• Component candidates often lead to other correct
answers. Instructors will require some techniques
and tools to avoid the other correct answers or toconfirm whether the set of their completed EPN
models covers all the possible correct answers.
There are some closely related works. For example, alearning support technique for software modeling using
PN was discussed in our previous study5. The steps
discussed in section 2 have been developed based on it.
However, unlike the previous study, this study isintended to support personal on-demand learning, andtherefore does not include the steps of advising, review,and demonstration by instructors and other learners. Also,
the previous study does not include the discussion about
tools, and does not take EPN and VDM as objects of

Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP17K00103.
References
1. T. Takagi, R. Kurozumi and T. Katayama, State Transition
Tuple Coverage Criterion for Extended Place/Transition
Net-Based Testing, Proceedings of Pacific Rim
International Symposium on Dependable Computing,pp.29-30, Dec. 2019.
2. J. Fitzgerald, P.G. Larsen, P. Mukherjee, N. Plat and M.
Verhoef, Validated Designs for Object-Oriented Systems,
Springer-Verlag London, 2005.
3. T. Takagi and R. Kurozumi, Prototype of a Modeling Tool
to Convert between Extended Place/Transition Nets and
VDM++ Specifications, Proceedings of International
Conference on Artificial Life and Robotics, pp.157-160,
Jan. 2019.
4. T. Takagi, S. Morimoto, Y. Ue and Y. Imai, AnimatedGraphics-based Training Support Method and Prototype
Tool for Bug Fixing of Extended Place/Transition Nets,
Journal of Robotics, Networking and Artificial Life, Vol.5,
No.4, pp.278-282, Mar. 2019.
5. Y. Ue and T. Takagi, Learning Support Technique ofSoftware Visual Modeling Using Place/Transition Nets,
Proceedings of International Conference on Artificial Lifeand Robotics, pp.751-754, Jan. 2020.
6. T. Takagi, R. Takata, Z. Furukawa, F. Belli and M. Beyazıt,
Metrics for Model-Based Mutation Testing Based on
Place/Transition Nets, Proceedings of Joint Conference ofInternational Workshop on Software Measurement andInternational Conference on Software Process andProduct Measurement, pp.7-10, Nov. 2011.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

343

Test Suite Reusability Measurement Based on Uses Frequently and Coverage of Reused Test
Cases
Mochamad Chandra Saputra*, Tetsuro Katayama*, Yoshihiro Kita†, Hisaaki Yamaba*, Kentaro Aburada*, Naonobu
Okazaki*Interdisciplinary Graduate School of Agriculture and Engineering, University of Miyazaki,
1-1 Gakuen-Kibanadai Nishi, Miyazaki, 889-2192 Japan†
Department of Information Security, Faculty of Information Systems, Siebold Campus, University of Nagasaki,
1-1-1 Manabino, Nagayo-Cho, Nishi-Sonogi-gun, Nagasaki, 851-2195, Japan
Email: chandra@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kita@sun.ac.jp,
yamaba@cs.miyazaki-u.ac.jp, aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp*

Abstract
Reusability of a test suite is one of important factors to decrease the cost in software testing. This research definesand measures the test suite reusability score expressed with the degree of reusability of the test suite. It is calculated
by considering both frequency and code coverage of successful reused test cases in the test suite. Test suite reusability
measurement provides useful information to improve the efficiency of software testing, especially in regression
testing and automated testing.
Keywords: Test suite reusability, reused frequency, code coverage, reused test cases,

The research on test cases reusability measurement
considers to code coverage is reported by modifying the
program with reducing branches, the test suite is highly
reusable but, modifying the program such as splitting,and loop, the test suite is less reusable4. The research has
shown test suite reusability measurement is needed toconsider the code coverage.
The use of code clones is increased day by day due tothe growth of the use of open-source software andvariants5. Code clones are efficient in reducing the cost
and time on software development that have similar
requirements. Testing code clones is needed a strategy toachieve efficiency on it. Test suite reusability is one of
the best strategies to test the code clones.
This research proposes the formula for the test suite
reusability score to measure the degree of test suite
reusability based on reused test cases on another program.
The formula is considered to test suite reusability
frequency and distinct of code coverage. The test suite
reusability score measurement is including good

1. Introduction
White box testing knows as structural testing is one of thesoftware testing approaches to test the program based on
the internal structures of the program1. It is used toanalyze internal logic and behavior, data structure, andcode coverage. Test cases in the test suite is an important
asset for software testing. The test cases are important on
white box testing method to guarantee all independent
paths or statements within a program executed at least
once. Test cases should execute all true or false logicaldecisions and all loops at their or within the boundaries
and ensure the validity of internal data structures2.
Currently, reusing assets in the software development
process have a great purpose. The background of reusing
assets in software development is to reduce cost andeffort3. A reusable test suite at the testing phase isdecreasing testing time and cost. Generally, the test suite
effectiveness is depended on the capability of the test
cases on error detection related to coverage of the test
cases on the program.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

344

Mochamad Chandra Saputra, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

information to evaluate the efficiency of the test suite on
reusability.
2. Principles of Reusability and Code Clones
The principle of software reusability is defined as thecapability of an attribute to be reused in various objects6.
The terminology of test suite reusability in this researchis related to software reusability as the capability of test
cases in the test suite to examine several or all paths ofmethod should be tested on diverse objects. The test suite
reusability additionally needs to consider code coverageon the reusability measurement. The case of test suite
reusability measurement on white box testing, thecapability of the test suite to examine different objects is
not good enough. Consider the test suite can achieve
100% code coverage are one of criteria of a good test
suite.
This research uses several types of code clone7,8 suchas code clone type 1 (exact clones) are identical clones
except in white space and comments. The second type iscode clone type 2, the differences from the original codeare renamed identifiers, literals, types, layout, andcomments but the structurally and syntactically is similar.
The code clones type 3 are modified the statement suchas statement insertions/deletions in addition to changes in
identifiers, literals, types, and layouts. Code clone type 4
has been modified on code fragments to perform thesame objective but different syntactic variants. The test
suite reusability measurement is used on code clones
because the test cases in the test suite need to use thesame characters and by using the code clones we canhave an identical or near identical program for uses of thetest suite9.

Figure 1. Test Suite Reusability Measurement Activity

The similar lines of code executed then count as one lineof code from redundant lines of code or called distinct
code coverage.
To simplify the formula, the research uses thefollowing notation.
• SRTC: Successful reused test cases• DCC: Distinct code coverage• OT: Objects tested• OLOC: Original line of code
• TC: Test Cases
By using the notation, the formula for test suite
reusability score as follows.
𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 = (∑∑ 𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆+∑ 𝐷𝐷𝐷𝐷𝐷𝐷

𝑂𝑂𝑂𝑂×∑ 𝑇𝑇𝑇𝑇)+∑ 𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂(1)

The proposed formula for test suite reusability scorehas considering frequency and code coverage ofsuccessful reused test cases in the test suite.
4. Methodology
The test suite reusability measurement in this research isused on white box testing approach considered to thenumber of reused test cases in test suite to test the
programs and code coverage. The test suite reusability
measurement activity is shown in Figure 1. The activityof this experiment is using a Java program and given test
suite to measure its reusability. Test suite reusability
measurement based on code clones approach is used thenumber of successful reused test cases, and distinct code
coverage from the test suite on calculation. The researchuses code clones type 1, type 2, type 3, and type 4. Test
suite reusability measurement results are informing thedegree of test suite reusability.

3. Test Suite Reusability Measurement
Reusing assets on software testing will help to reducethe cost of software testing. The asset is defined as any
kind of product from any part of the software process10.
The test suite is not very valuable if is not reusable intesting another program. The test suite reusability
measurement is important to know how reusable the test
suite.
The test suite reusability measurement is conducted byinvestigating the test case execution on the Java programs.
First information used on this measurement is the numberof successful reused test cases in the test suite. The
successful test cases are the examination result of the test
cases on Java program that could achieve the objective ofthe testing without any error. The next information is
distinct code coverage. The test cases in the test suite arepossible to execute similar lines of code.

5. Experiment
The research uses the parallelogram Java program asshown in Figure 2 with two given test suites. On theoriginal parallelogram Java program, the test suites are© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

345

Test Suite Reusability Measurement

Table 1. The result from code coverage information on code
clonespackage parallelogram;
import java.io.BufferedReader;import java.io.InputStreamReader;public class Parallelogram {public static void main(String[] args) {double base,height;
BufferedReader br = new BufferedReader(newInputStreamReader(System.in));
System.out.println("This program calculate value thearea of a trapezium");
System.out.println("Enter the base and the height ofthe parallelogram");try{
base=Double.parseDouble(br.readLine());
height=Double.parseDouble(br.readLine());}catch (Exception e){
System.out.println("An error occurred");return;}if(base<=0 || height<=0){
System.out.println("Wrong Input");
return;}System.out.println("Area = " + base*height );}}Code Clone Type-1
Test S uite-1
Test S uite-2
Disticnt CodeDisticnt Code
No. LOC TC-1 TC-2 TC-3
TC-1 TC-2 TC-3 TC-4
Coverage
Coverage
1

1

1

1

1

1

1

1

1

1

2

1

1

1

1

1

1

1

1

1

3

1

1

1

1

1

1

1

1

1

4

1

1

1

1

1

1

1

1

1

5

1

1

0

1

1

1

1

1

1

6

1

1

0

1

1

1

1

1

1

7

0

0

1

1

0

0

0

0

0

8

0

0

1

1

0

0

0

0

0

9

0

0

1

1

0

0

0

0

0

10

1

1

1

1

1

1

1

1

1

11

1

1

0

1

1

1

1

1

1

12

0

1

0

1

1

1

1

0

1

13

0

1

0

1

1

1

1

0

1

14

1

0

0

1

0

0

0

1

1

15

1

0

0

1

0

0

0

1

1

Code Clone Type-2
Test S uite-1
Test S uite-2
Disticnt CodeDisticnt CodeTC-1 TC-2 TC-3 TC-4
No. LOC TC-1 TC-2 TC-3
Coverage
Coverage

Figure 2. Parallelogram Source Codeexamining for collecting the coverage information. The
test suite reusability measurement on this experiment is
excluding the result of original test suite examination
data. The experiment is focused on the test suite
reusability score on another Java program that reuses thetest suite to testing the program. The original Java
program is cloning by code clone type 1, type 2, type 3,
and type 4 approach.
This research has two given test suites. The first test
suite contains three test cases and the second test suite
contains four test cases are examined on code clones. The
information of successful reused test cases and distinct
code coverage from test cases examination on code
clones are used for test suite reusability measurement.

1

1

1

1

1

1

1

1

1

1

2

1

1

1

1

1

1

1

1

1

3

1

1

1

1

1

1

1

1

1

4

1

1

1

1

1

1

1

1

1

5

1

1

0

1

1

1

1

1

1

6

1

1

0

1

1

1

1

1

1

7

0

0

1

1

0

0

0

0

0

8

0

0

1

1

0

0

0

0

0

9

0

0

1

1

0

0

0

0

0

10

1

1

1

1

1

1

1

1

1

11

1

1

0

1

1

1

1

1

1

12

0

1

0

1

1

1

1

0

1

13

0

1

0

1

1

1

1

0

1

14

1

0

0

1

0

0

0

1

1

15

1

0

0

1

0

0

0

1

1

Code Clone Type-3
Test S uite-1
Test S uite-2
Disticnt CodeDisticnt CodeTC-1 TC-2 TC-3 TC-4
Coverage
Coverage

No. LOC TC-1 TC-2 TC-3
1

1

1

1

1

1

1

1

1

1

2

1

1

1

1

1

1

1

1

1

3

6. Result and Discussion
The research is examined the given test suites on code
clones type 1, type 2, type 3, and type 4 to collectinformation number of reused test cases in the test suiteand code coverage of the test suite. Table 1 shows thatthe given test cases on the test suite successfully examine
the code clones. The distinct code coverage on code
clones shows that several lines of code on code clones
type 1, 2, and 3 are not executed by test cases on the test
suites during the testing.
The result of code coverage information shows inTable 1. The value 1 means this line is executed and 0 is
not executed. Table 1 also shows the result of distinct
code coverage in another column. The value 1 means thisline is executed and 0 is not executed during the testingby using one or more test cases. The result of the numberof successful reused test cases shows in Table 2. The
result obtains from test cases examination on code clones.
Table 3 shows the result of test suite reusability

1

1

1

1

1

1

1

1

1

4

1

1

1

1

1

1

1

1

1

5

1

1

0

1

1

1

1

1

1

6

1

1

0

1

1

1

1

1

1

7

0

0

1

1

0

0

0

0

0

8

0

0

1

1

0

0

0

0

0

9

0

0

1

1

0

0

0

0

0

10

0

0

1

1

0

0

0

0

0

11

1

1

1

1

1

1

1

1

1

12

1

1

0

1

1

1

1

1

1

13

0

1

0

1

1

1

1

0

1

14

0

1

0

1

1

1

1

0

1

15

1

0

0

1

0

0

0

1

1

16

1

0

0

1

0

0

0

1

1

Code Clone Type-4
Test S uite-1
No. LOC1

TC-1 TC-2 TC-3
1

1

1

Test S uite-2

Disticnt CodeDisticnt CodeTC-1 TC-2 TC-3 TC-4
Coverage
Coverage
1

1

1

1

1

1

2

1

1

1

1

1

1

1

1

1

3

1

1

0

1

1

1

1

1

1

4

1

1

0

1

1

1

1

1

1

5

1

1

0

1

1

1

1

1

1

6

1

1

0

1

1

1

1

1

1

7

1

1

0

1

1

1

1

1

1

measurement uses the formula(1). The test suite
reusability measurement is very common information to
reduce the cost of software testing. The different numberof lines of code is possible for code clones type 3 and 4,
for code clone type 1 and 2 usually the number lines of© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

346

Mochamad Chandra Saputra, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

result shows the number of successful reused test casesin the test suite did not guarantee 100% code coverage on
the test suite reusability measurement. The result of test
suite reusability measurement is valuable information to
reduce the cost of software testing, especially in
regression testing and automated testing.
Our future works will focus on other measurement fortest suite quality measurement.

Table 2. Result of the number of successful reused test cases
Code Clones

Number of
Test Cases

Number of S uccessful
Test Cases Tested

Type-1
Type-2

Test S uite-1

Type-33
3Type-43Type-1

4

Type-2

Test S uite-2

3
3Type-34

4

4Type-4

References
1. Sommerville, I. Software Engineering. AddisonWesley Publishing Company, 2010, pp. 133–170.
2.
Roger S. Pressman, P. D. Software Engineering: A
Practitioner’s Approach. McGraw-Hill, 2009.
3.
Sutcliffe, A. The Domain Theory Patterns forKnowledge and Software Reuse. Lawrence
Erlbaum Associates, Inc.Publishers, 2002.
4.
Dong, Y., Wang, Y., Lau, M. F. & Lin, S.
Experiments on Test Case Reuse of Test Coverage
Criteria. 7th International Conference on
Ubiquitous Intelligence & Computing and 7thInternational Conference on Autonomic & Trusted
Computing, 2010, pp. 277–281.
5.
Saini, N., Singh, S. & Suman. Code Clones:
Detection and Management. Procedia Computer.
Science, 2018, pp. 718–727.
6.
Younoussi, S. & Roudies, O. All About Software
Reusability: A Systematic Literature Review.
Journal of Theoretical and Applied Information.
Technology, 2015, pp. 64–75.
7.
Bellon, S., Koschke, R., Antoniol, G., Krinke, J. &Merlo, E. Comparison and Evaluation of Clone
Detection Tools. IEEE Transactions on Software
Engineering, 2007, pp. 577–591.
8.
Roy, C. K. & Cordy, J. R. Survey on Software
Clone Detection Research. in Technical Report
No. 2007-541, 2007.
9.
Asaithambi, S. P. R. & Jarzabek, S. Towards Test
Case Reuse: A Study of Redundancies in Android
Platform Test Libraries. in Lecture Notes in
Computer Science vol. 7925, 2013, pp. 49–64.
10. Ezran, M., Morisio, M. & Tully, C. Practical
Software Reuse. Springer London, 2002.

4

Table 3. Result of test suite reusability measurement
Code Clones Test S uite Reusability S core Average

Test S uite-1

Test S uite-2

Type-1100%

Type-2100%

Type-3100%

Type-4100%

Type-1

84%

Type-284%

Type-380%

Type-4

100%

100%

87%

code are the same. The distinct code coverage is used to
selecting the redundant line of code executed by the test
cases on the test suite. By using the distinct code
coverage, the calculation of percentage code coverage isrepresented the actual condition. The test cases in the test
suite are possible to execute similar lines of code. The
number of successful reused test cases on the test suite as
shown in Table 2 is important for test suite reusability
measurement. The number of successfully reused test
cases is one of the important parameters for the test suite
reusability measurement.
The information of code coverages and number ofsuccessful reused test cases in the test suite is used fortest suite reusability measurement. The result of test suite
reusability measurement uses the formula (1) shown in
Table 3. The result of test suite reusability score shows
the different scores for several test suites because they
have the different number distinct code coverage by thetest cases as shown in Table 1. The number of distinct
code coverage is important to ensure the capability of thetest suite to achieve 100% code coverage. The score oftest suite reusability 100% means the test suite has theperfect capability to achieve 100% code coverage andreused on another program. The test suite reusability
measurement provides useful information to reduce thecost of software testing, especially in regression testing
and automated testing.
7. Conclusion
This research confirms the test suite reusability
measurement is calculated by combining the number ofsuccessful reused test cases and code coverage. The© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

347

The Seven Information Features of Class for Blob and Feature Envy
Smell Detection in a Class Diagram
Bayu Priyambadha*, Tetsuro Katayama*, Yoshihiro Kita†, Hisaaki Yamaba*, Kentaro Aburada*, Naonobu Okazaki**University of Miyazaki, 1-1 Gakuen-kibanadai nishi, Miyazaki, 889-2192, Japan†Department of Information Security, Faculty of Information Systems, Siebold Campus, University of Nagasaki,

1-1-1 Manabino, Nagayo-cho, Nishi-Sonogi-gun, Nagasaki, 851-2195, Japan
E-mail: bayu@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kita@sun.ac.jp, yamaba@cs.miyazaki-u.ac.jp,
aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp

Abstract
Measuring the quality of software design artifacts is difficult due to the limitation of information in the design phase.
The class diagram is one of the design artifacts produced during the design phase. The syntactic and semantic
information in the class is important to consider in the measurement process. The class-related information is used todetect the smell as an indicator of a lack of quality. All information related to the class is used by several classifiers
to prove how informative it to be used to detect the smell. The smell types that are a concern in this research are Bloband Feature Envy. The experiment using three classifiers (j48, Multi-Layer Perceptron, and Naïve Bayes) confirmsthat the information can be used to detect Blob smell, on the other hand, Feature Envy, still needs more research. The
average true positive rate of each classifier is about 80.67%.
Keywords: Smell Detection, Class Diagram Smell, Design Quality, Software Design1.

important to inform us about the compactness of the class.
The level of compactness of the class is related toexternal quality attributes named maintainability andunderstandability. The more level of compactness ofclass then the class is easier to change and understand.
Nowadays, several researchers are extensively
discussed and evaluate software design problems named
smell, in the other term is “code smell”4, or “designflaw”5. The existence of smell may affect the quality
attributes of the software.
The metric and the existence of smells indicate thequality of software artifacts. The metric can be used as atool to measure quality. And, the existence of smell
indicates lack of quality. The relationship between
software metrics and smells is an interesting thing tolearn. Bigonha et. al., explain the usefulness of thesoftware metric threshold for detection of bad smells andfault prediction6. All matrices used by Bigonha are theIntroduction

Good quality of the software artifacts affects the final
result of software products. It is important to measure thelevel of its quality. The measurement is not only to makea judgment about the best result of software product1 but
also for the process improvement in softwaredevelopment2. The metric quality is often used tomeasuring the quality of software artifacts.
The quality measurement is often done to the quality
indicators of the software artifacts. For example,
complexity, cohesion, and coupling3. Those three
indicators are used to the quality measurement of theclass. The software consists of a lot of classes as the
template or blueprint of the objects. The object can work
together with other objects to accomplished the specificfunctionality of the software. The inner classmeasurement means measure relationships between
elements inner the class (cohesion). The cohesion is© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

348

Bayu Priyambadha, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

class level’s metric. The measurement is done on thesource code level.
The position of smell becomes a guide for thedeveloper to improve the software artifacts. The smell
can be overcome by refactoring activity. Fowler defines
the term refactoring as the process of changing internal
structure that is not changing the external behavior ofsoftware4. The code is modified over time, and thestructure of the code will gradually disappear. The badsmell is emerging and the quality of code is decay asstated in the Lehman’s Laws that said as the software isevolving then the structure is degraded1. Refactoring is
the effort that we have to pay to manage the structure ofcode.
Mostly smell detection and refactoring process is doneat the level of source code. There is the researcher thatconcern about the importance of shift the work to thedesign phase based on design artifacts. Start from thesmell detection in design7 then continue to the refactoring
in design8,9. The challenge to do the smell detection on
the design artifacts is the limitation of information. At thedesign level, there is only the model of the software thatwants to build. One of the models as the product of thedesign is class diagram. Class diagram informs the
structure of the software based on classes and theirrelationship. The possibility to collect the informationfrom the class diagram is by understanding the syntactic
and semantic information that may exist10.
This research is aimed to collect the information basedon the syntactic and semantic information that exists
inner the class diagram. It uses the information to detect
the existence of the smell in the class diagram. And, it
uses the inner information of class to detect the smell.
Two types of smell that are considered related tocohesion are Blob and Feature Envy. There isinformation that will be extracted from the class diagram
and then combine with the smell dataset that is takenfrom the Landfill dataset11. That information will beclassified using three classifiers, j48, Multi-Layer
Perceptron (MLP), and Naïve Bayes, and the result will
be compared. The focus of this research is how theusefulness of the information can be used to detect thesmell.
The rest of the paper is organized as follows. In section
2, we present every data that will be used in this researchand how to get those data. Section 3 describes how tolabel every data. Section 4 describes the whole process ofFig. 1. Process of Semantic Similarity Analysis of the LabelNameclassification. Section 5 describes the result anddiscussion. Then the last is the conclusion and future
work in section 6.
2. Seven Information
There is seven information as the candidate data to use insmell detection. The data is related to one of the internal
quality attributes called cohesion. The data are number of
attributes, number of methods, number of the relation
between method and attribute, number relation between
method and method, number relation between attributeand attribute, the capacity of relation inner the class andcohesion.
In collecting the data it is important to consider not
only the syntactic information but also the semantic
information. Syntactic information is the information that
we can directly read or extract from the class diagram.
And, the semantic information is the information thatneeds a little processing (interpret the meaning) to extract.
The object of study in this research is the class diagramthat resulted by using the Visual Paradigm tools. Then, it
converts to the XML file then called XML based class
diagram (Fig. 1).
2.1. Number of Attributes
The first data is the attribute. The collection of theattributes in the class diagram is easy. The notation for
the attributes name and the type of attribute is clearly
described in the class diagram.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

349

The Seven Information Features
No.
1.
2.
3.
4.
5.
6.
7.
8.

FieldsMAR
MMR
AAR
Number of Attributes
Number of Methods
Capacity
Cohesion
Label

Type
Numeric
Numeric
Numeric
Numeric
Numeric
Numeric
Numeric
Blob, Feature Envy, No Smell

relation between attributes if there are two attributes thatare related to the same method. This type of relation is
included in the category of semantic information. Fig. 2
shows the analogy of both relations.

Fig. 2. The Description of Transitive Relation

The information about attributes is extracted from XML
based class diagram.

2.5. Relation Capacity
The maximum relation is the maximum number ofrelations that possible to exist in the class. The class is
assumed as the area of the relation, that's has a capacityof relation. The maximum relation is calculated by using(1).

2.2. Number of Methods
The second is the number of methods. The name, return
value type and the parameters of the method are also
clearly described in the class diagram. Collecting all
information about the method is including in the syntactic
information.

𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 =(𝑚𝑚+𝑎𝑎)((𝑚𝑚+𝑎𝑎)−1)2(1)

Where 𝑚𝑚 is the number of methods and 𝑎𝑎 is the numberof attributes.

2.3. Number of Relation between MethodAttribute

2.6. Cohesion Value

The number of relations between Method and Attribute(MAR) can be count by considering not only syntactic
but also semantic information. The relation between
method and attribute is determined based on thesimilarity of type (syntactic) and the similarity of
meaning (semantic) between method and attribute. Fig.1
describes the process to measure the closes meaning
between the label name of method and attribute. Alllabels name are extracted from the XML’s based class
diagram. Then the labels are split into words. After thatthe similarity of meaning between the set of words is
calculated12. If the label name between attribute andmethod is semantically similar then it will be considered
as a relation. Matrix relation is generated to make it easier.

The cohesion is calculated by considering MAR, MMR,
and AAR. The relation is direct relation and transitive
relation. All of the relations between method and attributeis divided by the maximum relation (1) that possibly
exists in one class. The cohesion calculation is expressedas (2).

𝐶𝐶𝐶𝐶ℎ𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒 =3. Dataset

𝑀𝑀𝑀𝑀𝑀𝑀+𝑀𝑀𝑀𝑀𝑀𝑀+𝐴𝐴𝐴𝐴𝐴𝐴
𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀(2)

The Landfill dataset is consists of 243 instances of five
types of code smells identified from 20 open-source
software projects11. In the Landfill dataset, all classes that
contain bad smell are labeled based on the type of smell.
In this research the dataset used to experiment iscombine of the seven information and the label of smell
provided by the Landfill dataset. The representation of
the dataset is described in Table 1. The main dataset is
extracted from six projects: jEdit, jHotDraw, FreeMind,
HSQLDB, aTunes, and ArgoUML. The dataset consistsof 300 data based on the smell class listed by the Landfill

2.4. Number of Relation between Method-Method
and Attribute-Attribute
Both the type of relation is called transitive relation orindirect relation. That relation is determined based on the
direct relation between attribute and method. There are
two definitions to determine method-method (MMR) andattribute-attribute relation (AAR). There will be a
relation between methods if there are two methods thatare related to the same attribute. Then, there will be a
Table 1. Representation of Dataset

Table 2. Recap of The Testing Result© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

350

Bayu Priyambadha, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki
No.

Classifier

Blob

Feature
Envy

No
Smell

1.
2.
3.

J48
MLP
Naïve Bayes

94.7%
94.7%
52.6%

25.5%
0%
16.7%

22.2%
38.9%
55.6%

identify the Feature Envy smell. The dataset does not
express the big differential for Blob and Feature Envy. It
makes the classifier hard to identify each type of smell.
Only j48 and Naïve Bayes can identify the Feature Envy
but in the low of TPR.
To continue the research, it needs more exploration toincrease the differentiation between both smells. The
additional information of class maybe would be worth
finding to increase the richness of data. The other,weighing every variable (MAR, MMR, AAR) is also
considered important to find to sharpen the differences inthe dataset.

True
Positive
Rate
(TPR)
51%
51%
44.9%

dataset and the smell free classes, then used as training
data. And, the other set of 49 data is generated for testingdata.
4. Classification
In this experiment, the process of detecting the smell will
use the classification method. This research uses threeclassifiers: j48, Multi-Layer Perceptron, and Naïve Bayes.
The tool used in this experiment is Weka as a Machinelearning software to solve data mining problems13. Allclassifiers are run by using a basic configuration. The useof a classifier is only the way to prove how the dataset
can be distinctive to the bad smell (Blob and Feature
Envy).

References
1.
2.
3.
4.

5. Result and Discussion

5.

This section explains the result of the experiment byusing the dataset and three classifiers. The recapitulationof the result is described in Table 2.
j48 has a TPR of about 51%. The correct classify forevery label of data is, for Blob data is 94.7%, Feature
Envy data is 25% and No data is 22.2%. MLP has a TPR
value of about 51% of the data. Based on the label data,
for Blob data is 94.7%, for Feature Envy data is 0% andfor No data is 38.9%. Then, Naïve Bayes has a TPR valueof about 44.9% where Blob data is 52.6%, Feature Envy
data is 16.7% and No data is 55.6%.
Based on the result, two classifiers have a TPR value
above 50%. The classifiers are j48 and MLP. But, theMLP classifier cannot identify the Feature Envy smell
because the TPR for Feature Envy of MLP is 0%. The j48
and Naïve Bayes can detect the Feature Envy smell eventhough with the low rate. All classifiers can detect Blob
smell with the TPR above 50%. The average TPR for
Blob smell is 80.67%. The characteristics expressed in
the dataset led to the identification of Blob smell. The
existing information is not enough used to find both typesof smells, especially for the Feature Envy smell. Moredetailed data is needed to improve the differentiation
between the two types of smells.

6.
7.

8.

9.

10.
11.

12.

6. Conclusion and Future WorkThe use of seven information as a dataset of class to
identify the Blob smell in the class diagram is very
effective. The effectiveness is proven by the TPR above
50%. On the other hand, it is not very good to use to13.

Sommerville, I. Software Engineering. Software
Engineering (Addison-Wesley Professional, 2010).
Chidamber, S. R. & Kemerer, C. F. A Metrics Suite for
Object Oriented Design. IEEE Trans. Softw. Eng. 20,
1994, pp. 476–493.
Chowdhury, I. & Zulkernine, M. Using complexity,
coupling, and cohesion metrics as early indicators ofvulnerabilities. J. Syst. Archit. 57, 2011, pp. 294–313.
Fowler, M. Refactoring Improving the Design of Existing
Code. (Pearson Education - Wesley, 2019).
Riel, A. J. Object-oriented Design Heuristics. (AddisonWesley Publishing Company, 1996).
Bigonha, M. A. S. et al. The usefulness of software metricthresholds for detection of bad smells and fault prediction.
Inf. Softw. Technol. 115, 2019, pp. 79–92.
Sidhu, B. K., Singh, K. & Sharma, N. A Catalogue ofModel Smells and Refactoring Operations for ObjectOriented Software. Proc. Int. Conf. Inven. Commun.
Comput. Technol. ICICCT 2018, 2018, pp. 313–319
doi:10.1109/ICICCT.2018.8473027.
Misbhauddin, M. & Alshayeb, M. Model-driven
refactoring approaches: A comparison criteria. Proc.
African Conf. Softw. Eng. Appl. Comput. ACSEAC 2012,2012, pp. 34–39 doi:10.1109/ACSEAC.2012.20.
Dharmawan, T. & Rochimah, S. Systematic literature
review: Model refactoring. Proc. 2017 4th Int. Conf.
Comput. Appl. Inf. Process. Technol. CAIPT 2017. 2018,
pp. 1–5.
Priyambadha, B. et al. The Measurement of Class
Cohesion using Semantic Approach. Proc. Int. Conf. Artif.
Life Robot. 25, 2020, pp. 759–762.
Palomba, F. et al. Landfill: An open dataset of code smells
with public evaluation. IEEE International Working
Conference on Mining Software Repositories vols 2015Augus. 2015, pp. 482–485.
Dijkman, R., Dumas, M., van Dongen, B., Käärik, R. &Mendling, J. Similarity of business process models:
Metrics and evaluation. Inf. Syst. 36, 2011, pp. 498–516.
Hall, M. et al. The WEKA Data Mining Software : An
Update. SIGKDD Explor. 11, 2009, pp. 10–18.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

351

Improvement of RETUSS to Ensure Traceability between Sequence Diagram in UML andJava Source Code in Real Time
Kaoru Arima*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, Naonobu Okazaki**Department of Computer Science and Systems Engineering, Faculty of Engineering, University of Miyazaki,
1-1 Gakuen-kibanadai nishi, Miyazaki, 889-2192, Japan†Department of Information Security, Faculty of Information Systems, Siebold Campus, University of Nagasaki,
1-1-1 Manabino, Nagayo-cho, Nishi-Sonogi-gun, Nagasaki, 851-2195, Japan
arima@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kita@sun.ac.jp,
yamaba@cs.miyazaki-u.ac.jp, aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp

Abstract
Ensuring traceability of software deliverables is one of the methods to ensure software quality. RETUSS (Real-time
Ensure Traceability between UML and Source-code System) is a tool that saves labor and time, and eliminates
mistakes by human handling in ensuring traceability between UML and source codes. However, RETUSS is notuseful due to its limited scope of application. This paper improves the usefulness of RETUSS by extending the scopeof application of sequence diagram in UML and Java source code.
Keywords: software quality, traceability, UML, sequence diagram, Java

Language)4 and source codes by transforming them toeach other in real time. Therefore, RETUSS can savelabor and time, and eliminate mistakes by humanhandling in ensuring traceability between UML andsource codes. RETUSS has the following functions.

1. Introduction
The importance of software in society is increasing, andsystem failures and software bugs cause significant
economic and social impact. Therefore, ensuring thequality of systems and software has become more
important. Ensuring traceability of software deliverables
is one of the methods to ensure software quality.1 It can
specify the scope of the impact due to the modification in
the requirements and remove the gap between the
documents and the source codes. However, it has thefollowing two problems.










Taking much labor and time to modify similarly
other related deliverables in modifying a part ofdeliverables
Having a risk that you cannot ensure traceability
because of causing mistakes to ensure traceability
by human handlingDescription of class diagram
Description of sequence diagram
Description of Java source codes
Description of C++ source codes
Bidirectional transformation between class diagramand Java source codes
Bidirectional transformation between class diagram
and C++ source codes
Bidirectional transformation between sequence
diagram and Java source codes

However, RETUSS is not useful in ensuring traceability
between sequence diagram and Java source codes due to
its limited scope of application.
This paper improves the usefulness of RETUSS byextending the scope of application of sequence diagramand Java source codes.

In order to solve them, our laboratory developed
RETUSS (Real-time Ensure Traceability between UML
and Source-code System).2,3 RETUSS ensures
traceability between UML (Unified Modeling© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

352

Kaoru Arima, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

2. The extended RETUSSThis paper extends the following two functions of theexisting RETUSS.
Description of sequence diagram
Bidirectional transformation between sequence
diagram and Java source codes

Fig. 1 shows the interface of the extended RETUSS.
The extended RETUSS has the main window and thesource code window. The main window is a window for
describing UML. The source code window is a window
for writing source codes.
Fig. 2 shows the structure of the extended RETUSS.
The extended RETUSS consists of five parts: display part,
correspondence part, description part, transformation
part, and storage part. To extend the function of thedescription of sequence diagram, we mainly extend thedisplay part, correspondence part, and description part.
To extend the function of the bidirectional transformation
between sequence diagram and Java source codes, we
mainly extend the description part, transformation part,and storage part.

Fig. 1. The interface of the extended RETUSS

2.1. Extending the function of the description ofsequence diagram
We add the following three functions to the function of
the description of the sequence diagram.
Adding a message
Adding a combined fragment
Deleting elements

By adding these functions, a user can edit the sequence
diagram directly on RETUSS.
To add these three functions, we extend the display
part, the correspondence part, and the description part.
The extended display part displays three buttons in themain window: Message button, Combined Fragment
button, and Delete button. The extended correspondence
part has three new processes: displaying the dialog to add
a message, displaying the dialog to add a combined
fragment, and displaying the dialog to delete elements.
The three processes are called from the event handlers of
the three buttons displayed by the extended display part.
The extended description part has four new processes:
adding a message, adding a combined fragment, deletinga message, and deleting a combined fragment. These four
processes are called by correspondence part and theychange the sequence diagram information in the storage
part.

Fig. 2. The structure of the extended RETUSS

2.2. Extending the function of the bidirectionaltransformation between sequence diagramand Java source codes
We define four new transformation rules to extend thescope of the bidirectional transformation between
sequence diagram and Java source codes. Table 1 shows© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

353

Improvement of RETUSS to

Table 1. The four new transformation rules for the bidirectional transformation between sequence diagram and Java source codes.
Name in sequence
diagram

Notation in sequence diagram

Name in Java
Method invocation
expression

Message

Method declaration

Combined fragment opt

if-then statement

if-then-else
statement

Combined fragment alt











accessModifier returnType
methodName(parameterTypeparameterName, …) { … }if (expression1) {…
}if ( expression1 ) {…
} else if ( expression2 ) {…
}…
while ( expression ) {…
}for statementfor(int i=X; i<Y; i++) { … }for(int i=X; i<=Y; i++) { … }for(int i=X; i>Y; i--) { … }for(int i=X; i>=Y; i--) { … }process: transformation from Java source codes to Java
information. The extended transformation part has a newprocess: transformation between sequence diagram
information and Java information based on thetransformation rules in Table 1. The extended storage
part has three new classes in the sequence diagram
information: interaction fragment class, combined
fragment class, and interaction operand class. In addition,
the extended storage part has three new classes in theJava information: If class, For class, and While class.

Message of operation invocation with parameters
Combined fragment opt
Combined fragment alt
Combined fragment loop

In addition, the extended RETUSS can transform thefollowing Java source code syntaxes into sequence
diagram elements.



methodName(parameter, …);

while statement
Combined fragment loop

the four new transformation rules for the bidirectionaltransformation between sequence diagram and Javasource codes. By the transformation rules in Table 1, theextended RETUSS can transform the following sequence
diagram elements into Java source code syntaxes.

Syntax in Java

3. Application example
Fig. 3 shows the screenshot when the Java source codes
are written in the extended RETUSS. It shows that theextended RETUSS can ensure traceability between
sequence diagram and Java source code in writing ifthen-else statement, while statement, and for statement ofJava. In addition, we confirmed that the extended
RETUSS can ensure traceability between sequence
diagram and Java source codes in describing sequence
diagram.

Method invocation expression with parameters
Method declaration
if-then statement
if-then-else statement
while statement
for statement

Here, the extended RETUSS does not support nested
structure of the above syntaxes.
To implement the transformation rules in Table 1, we
extend the description part, transformation part, andstorage part. The extended description part has a new

4. Evaluation
To evaluate the usefulness of the extended RETUSS, we
experiment with four students of University of Miyazaki.
The steps of the experiment are shown below.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

354

Kaoru Arima, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

(i)
(ii)
(iii)

The experimenter prepares traceable sequence
diagrams and Java source codes. We call thesedeliverables.
The experimenter instructs the participants tochange the deliverables.
The participants change the deliverables as
instructed

There are two types in the changes: sequence diagrams
changes, Java source codes changes. There are two cases
below for participants to change the deliverables.
Fig. 3. The screenshot when the Java source codes are written
in the extended RETUSS

Case A: using the extended RETUSS.
Case B: using EA (Enterprise Architect)5 and a text
editor.

Table 2. The times it took the participants to change (seconds)

Table 2 shows the times it took the participants tochange in the two cases and the two changes. From Table
2, the time in case A was about 76.5% shorter than thetime in case B, when the sequence diagrams changes was
instructed. In addition, the time in case A was about
69.0% shorter than the time in case B, when the Javasource code changes were instructed.
In summary, the extended RETUSS can save laborand time in ensuring traceability between sequence
diagram and Java source codes. Therefore, the usefulness
of RETUSS has improved by extending its scope of
application while retaining the benefits of the existing
RETUSS.

Java source codeschangesCase A

Case B

Case A

Case B

1

59

205

134

369

2

62

221

95

346

3

38

216

102

361

4

60

289

126

398

Average54.75

232.75114.25368.50Participants

References

5. Conclusion

1.

This paper has improved the usefulness of RETUSS byextending the scope of application of sequence diagramand Java source codes. The extended RETUSS allows
you to edit sequence diagram directly on RETUSS, andalso supports four new transformation rules for sequence
diagram and Java source codes.
The experimental results showed that the extended
RETUSS can save the time to ensure traceability between
sequence diagram and Java source code by about 76.5%
for sequence diagram changes, and about 69.0% for Javasource code changes. Therefore, the usefulness of
RETUSS has improved by extending its scope of
application while retaining the benefits of the existing
RETUSS.
The future works are as follows.
Sequence
diagrams changes

2.

3.

4.
5.

SQuBOK Sakutei Bukai, Guide to the Software Quality
Body of Knowledge, 2nd edn. Ohmsha, 2014 (in Japanese).
Tetsuro Katayama, Keisuke Mori, Yoshihiro Kita, Hisaaki
Yamaba, Kentaro Aburada, Naonobu Okazaki: RETUSS:
Ensuring Traceability System between Class Diagram in
UML and Java Source Code in Real Time, Journal of
Robotics, Networking and Artificial Life, Vol. 5(2), pp.
114–117, 2018.
GitHub, RETUSS: Real-time Ensure Traceability between
UML
and
Source-code
System,https://github.com/Morichan/Retuss (Accessed 2020-1209)
The Object Management Group, Welcome to UML Web
Site!, https://www.uml.org/ (Accessed 2020-12-09)
SPARX SYSTEMS, UML modeling tools for Business,
Software,
SystemsandArchitecture,https://www.sparxsystems.com/ (Accessed 2020-12-09)

Corresponding to other sequence diagram elements
Corresponding to other Java source code syntaxes
Corresponding to other UML diagrams
Corresponding to other programming languages© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

355

Faster R-CNN Based Defect Detection of Micro-precision Glass Insulated Terminals
Qunpo Liu, Mengke Wang, Zonghui Liu, Bo Su

School of Electrical Engineering and Automation, Henan Polytechnic University, 2001 Century Avenue
Jiaozuo, (454003), Henan, P.R.China,
Hanajima NaohikoCollege of Information and Systems, Muroran Institute of Technology, 27-1 Mizumoto-cho,Hokkaido
Muroran-shi, Hokkaido (050-8585), Japan
E-mail: lqpny@hpu.edu.cn, , hana@mondo.mech.muroran-it.ac.jp
www.hpu.edu.cn, www.muroran-it.ac.jp(leave space here should be 4 lines between www.university_name.edu and Abstract)
Abstract
Micro-precision glass insulated terminals (referred to as glass terminals) are the core components used in precisionelectronic equipment. As glass terminal, its quality has a huge impact on the performance of precision electronicequipment. Due to limitations in materials and production processes, some of the glass terminals produced havedefects such as missing blocks, bubbles, and cracks. At present, it is difficult to ensure product quality and production
efficiency with manual inspection methods. However, the defect characteristics of glass terminals are quite different,
and it is difficult for traditional defect detection technology to design an ideal feature extractor for detection.
Therefore, this paper proposes to use deep learning technology to detect missing blocks. First, preprocess the sample
pictures of missing block defects of glass terminals, and then train the deep learning network based on Faster RCNN.
According to the test results, the algorithm has an accuracy of over 90% in detecting missing defects in glass terminals.
Keywords: Micro-precision Glass Insulated Terminal; Faster R-CNN; Missing Block Detection

The equipment used for image acquisition andphotography is the CCD 540TVL (High Resolution B /W Black and White Camera) carried by the
MVP400CNC automatic image measuring instrument;
the working distance is 86mm; The glass terminal image
shooting method adopts the vertical shooting mode of theeight-zone light source; the collection condition is strong
light; the magnification during shooting is 1.0*30.77.
This article uses labeling software for labeling. Afterlabeling, an xml file will be generated, which can be used
for Faster RCNN4 training.

1. Introduction
Micro-precision glass insulated terminals are the corecomponents used in precision electronic equipment. The
glass terminal has a small size and requires high accuracy.
Defects in the glass terminals would bringing serious
losses and consequences1. The difficulties in defectdetection are mainly three points2: (1) The complex
imaging background of the defect contains a variety ofinterference noise; (2) The shape, size, and location of
defects are diverse; (3) The difference in thecharacteristics of defects of the same type is large, andthe difference in the characteristics of defects of different
types is small. Therefore, this paper proposes to use deeplearning technology to detect missing blocks3.

3. Faster RCNN algorithm

2. Image acquisition and annotation© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

356

Qunpo Liu, Mengke Wang, Zonghui Liu, Naohiko Hanajima, Bo Su,

3.1. Feature extraction networkand classification (cls) operations of the prior box
respectively. Assuming that there are k anchors for eachpixel on the shared feature map, it is necessary to predictthese k pixels, and the classification operation can get 2k
outputs , Indicates whether the k area contains the target.
The regression operation refers to the calculation ofcontinuously approaching the label frame with k anchors.
This operation can get 4k outputs. The k anchors get thecenter point coordinates and width and height
information of the suggested frame (x,y,w,h), the processof getting the suggestion box from anchor is also calledanchor parameterization. Taking a pixel of the current
shared feature map as the center, k anchors with different
aspect ratios and sizes are generated, and their aspectratios and sizes are represented by ratio and scale,respectively. For a feature vector of size w*h, there are atotal of w*h*k anchors. The structure of the RPN
network is shown in Figure 2.

The feature extraction network used in this article isResNet50, and its structure is mainly composed of Conv
Block and Identity Block5,6. Among them, the Conv
Block cannot be directly connected to the network. Its
main function is to change the dimension of the network,and the dimensions of the input and output results aredifferent. The identity block can be connected to the
network, and its main function is to deepen the network
depth, and the input and output results have the same
dimensions.
ResNet50
7×7 64,Conv layer,stride 2
3×3 Max pool, stride 2
1 × 1 64
3 × 3 64
1 × 1 256
1 × 1 1283 × 3 1281 × 1 512

1 × 1 2563 × 3 256
1 × 1 1024
1 × 1 5123 × 3 512
1 × 1 2048

×3

×4

×6

×3

Average poolFig.1 ResNet50 network structure
Fig.2 RPN network structure

ResNet50 is mainly composed of residual blocks.
Its structure is as shown in Figure 1. First, it performs a
convolution operation with 64 convolution kernels of
size 7*7 and step size of 2, and then performs a
maximum pooling with a pooling kernel size of 3*3 Then,after going through 3, 4, 6, and 3 residual blocks,
respectively, 1*1 and 3*3 represent the size of the
convolution kernel of the residual block, and 64, 256 andso on represent the number of convolution kernels,
respectively.

3.3. RPN loss function
The overall loss function of RPN is
𝐿𝐿({𝑝𝑝𝑖𝑖 }, {𝑝𝑝𝑖𝑖 ∗ }) =1
1
� 𝐿𝐿𝑐𝑐𝑐𝑐𝑐𝑐 (𝑝𝑝𝑖𝑖 , 𝑝𝑝𝑖𝑖 ∗ ) + 𝜆𝜆
� 𝑝𝑝𝑖𝑖 ∗ 𝐿𝐿𝑟𝑟𝑟𝑟𝑟𝑟 (𝑡𝑡𝑖𝑖 , 𝑡𝑡𝑖𝑖∗ ) (1)
𝑁𝑁𝑐𝑐𝑐𝑐𝑐𝑐
𝑁𝑁𝑟𝑟𝑟𝑟𝑟𝑟
𝑖𝑖

𝑖𝑖

where, i represents the index of the anchor, pi and pi*
respectively indicate that the i-th anchor contains thetarget predicted value and the anchor tag value. When thei-th anchor contains the target, pi*=1, and pi*=0 in othercases. ti and ti* respectively represent the position andscale information of the suggestion box and the label box,

3.2. RPN
RPN is the region generation network. By using d n*nwindow sliding on the shared feature map extracted by
the feature extraction network7, it is mapped into a ddimensional feature vector, and k Anchors are generated
on each pixel. These anchors perform the regression (reg)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

357

Faster R-CNN Based DefectLcls represents the classification loss, that is, the log lossof the anchor corresponding to the target category, andLreg is the regression loss, that is, the loss of thesuggestion box obtained from the candidate box,
definition for
𝐿𝐿reg (𝑡𝑡𝑖𝑖 , 𝑡𝑡𝑖𝑖 ∗ ) = 𝑅𝑅(𝑡𝑡𝑖𝑖 − 𝑡𝑡𝑖𝑖 ∗ )Proposal(2)

where, R is the robust loss function (SL1), that is, using
Smooth_L1 to calculate the loss value.
It can be seen from pi*Lreg that the regression loss
can be calculated when pi*=1, and there is no regression
loss in other cases. Among them, Ncls represents theselection of Ncls anchors for RPN training, Nreg represents
the shape of the shared feature map, and λ is equivalent
to a scale factor, so that the weights of the classification
loss and the regression loss are basically the same(generally λ=Nreg/Ncls≈10).
The regression process uses 4 coordinates:
𝑡𝑡𝑋𝑋 = (𝑥𝑥 − 𝑥𝑥𝑎𝑎 )/𝑤𝑤𝑎𝑎 , 𝑡𝑡𝑦𝑦 = (𝑦𝑦 − 𝑦𝑦𝑎𝑎 )/ℎ𝑎𝑎(3)t ∗X = (x ∗ − xa )/wa , t y∗ = (y ∗ − ya )/ha(5)

𝑡𝑡𝑤𝑤 = 𝑙𝑙𝑙𝑙𝑙𝑙( 𝑤𝑤/𝑤𝑤𝑎𝑎 ), 𝑡𝑡ℎ = 𝑙𝑙𝑙𝑙𝑙𝑙( ℎ/ℎ𝑎𝑎 )(4)𝑡𝑡𝑤𝑤∗ = 𝑙𝑙𝑙𝑙𝑙𝑙( 𝑤𝑤 ∗ /𝑤𝑤𝑎𝑎 ), 𝑡𝑡ℎ∗ = 𝑙𝑙𝑙𝑙𝑙𝑙( ℎ∗ /ℎ𝑎𝑎 )(6)

Fig.3 ROI Pooling working diagram

dividing it into 7*7 grids, that is, to achieve a fixed length
output .
3.5. Classification network
The fully connected layer and the activation function are
mainly used to judge and return the object category in theprediction frame to obtain a more accurate prediction
frame, and its structure is shown in Figure 4.

where, x, y, w, and h respectively refer to the center
coordinates, width, and height of the a priori box. The
variables x, xa, and x* respectively refer to the xcoordinates of the suggestion box, anchor a priori box,
and label box (the same applies to y, w, and h).
In the RPN network, the obtained prediction framesneed to be screened. This process uses IOU and nonmaximum suppression methods. The following is ananalysis of these two methods.

Fig.4 Classification network structure diagram

3.6. Loss function of Faster RCNN
Faster RCNN combines the loss functions of theclassification model and the regression model, and acalculation formula can be used to calculate the overallloss of the network. The loss function expression after the
network is merged is shown in equation (7).

3.4. ROI PoolingIts main function is to merge the shared feature map andthe ROI (region of interest) extracted through RPN, andthen divide the prediction frame into a set number ofgrids, and pool each grid, which is equivalent to The
prediction boxes of different sizes in the shared feature
map are pooled into feature maps of the same size, whichis convenient for inputting subsequent networks forclassification and regression.
As shown in Figure 3, the size of the prediction box
output by the RPN is fixed to pool_w and pool_h (where
pool_w and pool_h are both 7), which is equivalent to

𝐿𝐿(𝑝𝑝, 𝑢𝑢, 𝑡𝑡 𝑢𝑢 , 𝑣𝑣) = 𝐿𝐿𝑐𝑐𝑐𝑐𝑐𝑐 (𝑝𝑝, 𝑢𝑢) +𝜆𝜆(𝑢𝑢 ≥ 1)𝐿𝐿𝑙𝑙𝑙𝑙𝑙𝑙 (𝑡𝑡 𝑢𝑢 , 𝑣𝑣)(7)

where, p represents the probability that the corresponding
anchor output is a certain category, u is the actual
category label corresponding to the anchor, tu represents
the regression of the candidate frame to the scale factor
corresponding to the predicted frame, and v is the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

358

Qunpo Liu, Mengke Wang, Zonghui Liu, Naohiko Hanajima, Bo Su,

Acknowledgements

transformation parameter vector of the candidate frame
regression.
𝐿𝐿𝑐𝑐𝑐𝑐𝑐𝑐 (𝑝𝑝, 𝑢𝑢) = − 𝑙𝑙𝑙𝑙𝑙𝑙 𝑝𝑝𝑢𝑢

(8)

𝐿𝐿𝑙𝑙𝑙𝑙𝑙𝑙 (𝑡𝑡 𝑢𝑢 , 𝑣𝑣) = � 𝑖𝑖 ∈ {𝑥𝑥, 𝑦𝑦, 𝑤𝑤, ℎ}𝑠𝑠𝐿𝐿1 (𝑡𝑡𝑖𝑖𝑢𝑢 − 𝑣𝑣𝑖𝑖 )(9)This work is partially supported by National Key
Research and Development Project (2016YFC0600906),
Innovation Scientists and Technicians Troop
Construction
Projectsof
Henan
Province(CXTD2016054) and also supported by Innovative
Scientists and Technicians Team of Henan Provincial
High Education (20IRTSTHN019).

where, Lcls(p, u) represents the classification loss, which
is defined as the Equation (8), and the log base is e.

where, Lloc(tu, v) represents the loss function of theprediction box obtained by regression of the candidatebox, and SL1 represents the Smooth L1 loss function.
4.

References

1.

Experimental results
2.

Using Faster RCNN to detect missing blocks and defects
requires only 2 hours of training, and the number oftraining rounds is set to 50. It can detect whether morethan 50 samples contain defects in 1 minute, with anaccuracy rate of 98.03%, and with training as the numberof rounds increases, there is still room for improvementin accuracy. Some results of this paper are shown in
Figure 5.

3.

4.

5.

6.

7.

8.

Fig.5 Example of Faster RCNN's Missing Block DefectDetection Results

5. Conclusion

9.

The design of this article is supported by the working
mechanism of CNN, and is designed for the
implementation of the subsequent Faster RCNN and thedetection of missing block defects. The detection process
does not require manual feature extraction, but learns
features based on the sample of the glass terminal toidentify missing block defects and Perform calibration.
With the expansion of the data set and the improvementof the model, there is still much room for improvementin the accuracy of missing block defect detection.

Q.Liu, M.Wang, G.Wang, R.Gao, N.Hanajima,Detection
Algorithm of Porosity Defect on Surface of Microprecision Glass,Encapsulated Electrical Connectors,
Journal of Robotics, Networking and Artificial Life, 3(2020) 212-216
Q.Liu, M.Wang, and N.Hanajima, Defect Sample
Generation System Based On DCGAN for Glass Package
Electrical Connectors, Proceedings of 2019 ChineseIntelligent Systems Conference, (2020)
WANG Z, ZHU D. An accurate detection method forsurface defects of complex components based on support
vector machine and spreading algorithm[J]. Measurement,
2019,147:106886.
Ren S, He K, Girshick R, et al. Faster R-CNN: Towards
Real-Time Object Detection with Region Proposal
Networks[J],
NeuralInformation
Processing
Systems(NIPS). 2015.
Wang T, Chen Y, Qiao M, et al. A fast and robust
convolutional neural network-based defect detection
model in product quality control[J]. The International
Journal of Advanced Manufacturing Technology, 2017.
He K , Zhang X , Ren S , et al. Deep Residual Learning
for Image Recognition[C],IEEE Conference on
Computer Vision and Pattern Recognition (CVPR).
IEEE Computer Society, 2016.
Ren S , He K , Girshick R , et al. Object Detection
Networks on Convolutional Feature Maps[J]. IEEE
Transactions on Pattern Analysis & MachineIntelligence, 2015, 39(7):1476-1481.
Y.Jia, Alternative proofs for improved LMI
representations for the analysis and the design ofcontinuous-time systems with polytopic type uncertainty:
a predictive approach, IEEE Transactions on AutomaticControl, 48(2003) 1413-1416
Y.Jia, General solution to diagonal model matching
control of multi-output-delay systems and its applicationsin adaptive scheme, Progress in Natural Science,
19( 2009), 79-90.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

359

Adaptive Sliding Mode Control for a Constant Tension
Suspension System
Yuxin Jia

The Seventh Research Division and the Center for Information and Control, School of Automation Science andElectrical Engineering, Beihang University (BUAA),
Beijing, 100191, China
Yingmin Jia *The Seventh Research Division and the Center for Information and Control, School of Automation Science andElectrical Engineering, Beihang University (BUAA),
Beijing, 100191, China
Kai Gong

The Seventh Research Division and the Center for Information and Control, School of Automation Science andElectrical Engineering, Beihang University (BUAA),
Beijing, 100191, China
E-mail: yuxinjia@buaa.edu.cn, ymjia@buaa.edu.cn, buaa_gk@buaa.edu.cn
www.buaa.edu.cn

Abstract
The constant tension suspension system (CTSS) can be used to counteract the gravity of experimental spacecraft, andis widely used in the ground test of spacecraft control schemes. In this paper, the mechanical model of a CTSS isestablished, and an adaptive sliding mode controller (ASMC) is designed for the CTSS. The simulation results showthat the transient time and steady-state error of the ASMC system are small, which satisfies the requirements of
microgravity simulation.
Keywords: constant tension suspension system, microgravity simulation, sliding mode control, adaptive law

difficult for humans to reach, space technology is theresearch field that relies on ground experiments. The
ground test of spacecraft control scheme can not only
reduce the risk of actual operation in space, but also
shorten the development cycle. Therefore, how toestablish microgravity environment on the ground tosimulate the real motion of spacecraft is a majordemand.5 Many research institutions have developed
different kinds of devices to simulate the microgravity
environment. The methods mainly include free fall,parabolic flight, suspension, liquid and air floatation. The

1. Introduction
Space technology is a competitive focus of the cuttingedge technology in the world today. Many countries havesuccessively carried out a series of major projects, suchas space station construction, deep space exploration, andsatellite navigation. These achievements indicate thatmankind has entered a new stage of space exploration.
On the other hand, higher requirements are also put
forward for the reliability of the spacecraft control
scheme.1-4 Because spacecraft work in an area which is*Corresponding author.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

360

Yuxin Jia, Yingmin Jia, Kai Gong

joint and unconstrained structure can ensure that the
freedom of experimental spacecraft is not restricted, sothat the experimental spacecraft can perform approximate unconstrained pitching, yawing and rolling motion
under a small frictional torque disturbance. The verticalmovement of lifting unit ensures that the output tensionof the CTSS is always equal to the gravity of experimenttal spacecraft, realizing the long-term simulation of
microgravity environment. The physical meanings of themain symbols are as follows: g, Gravitational acceleration.
 R, Radius of the gear in lifting unit.
 k, Elastic coefficient of low stiffness structure.
 ω, Swing angle between lifting unit and low stiffness
structure.
 L0, Initial length of low stiffness structure.
 L, Variation length of low stiffness structure.
 M, The sum of the masses of unconstrained structureand experimental spacecraft.
 Tα, Driving torque of the motor in lifting unit.
 M1, M2, The loads of the motors in X-direction and Ydirection units.
 Ma, Mb, The masses of the gear and rack in lifting unit.
 Fx, Fy, Driving forces of the motors in X-direction andY-direction units.
 δx, δy, δz, Disturbing forces.
 ωx, ωy, Orthogonal decomposition values of ω.
According to the operation principle of CTSS, themodel can be obtained by mechanical analysis.

suspension method balances the gravity of experimental
spacecraft by adjusting the tension of hanging wire. The
system has a simple structure and low cost. It not only
overcomes the disturbance of air and liquid floatation on
the motion of experimental spacecraft, but also can beused for long-time microgravity simulation.6 Japan, theUnited States, Canada, China and other countries haveapplied the suspension method to microgravity simulation for the ground experiments of spacecraft. In 2019,
China Academy of Space Technology (CAST) used thesuspension method to counteract part of the gravity of theMars probe, verifying the performance of the probe
control scheme in the Martian gravity environment. The
system consists of a 140-meter-high test tower, a followup system and a Mars surface simulation area, which
satisfies the requirements of the Mars environment simulation experiment. The constant tension suspension
system (CTSS) is the key component of suspended
microgravity experimental device. It is of great significance to research how to guarantee that the output forceof CTSS is constant under disturbance.
An adaptive sliding mode controller (ASMC) is
proposed for the CTSS in this paper. Compared with thecontrol method in Ref. 5, the stability of the system isimproved.
2. System Modeling
The structure of the CTSS studied in this paper is shownin Fig. 1, including a micro delay servo module, a low
stiffness structure, an unconstrained structure, a universal
joint, a lifting unit, and an experimental spacecraft.5(1)
Hq + Jq + N = F + HδWhere F is the control vector of the CTSS, q is the statevector. δ is the external disturbance, and it is assumed
that δ is bounded. The matrices defined in the model is
as follows: -ω x (η1 − 1)
H =  -ω y (η 2 − 1) − R ( M − M c ) M

Fig. 1. Structure of the CTSS-d (η1 − 1)
0
0

0



-d (η 2 − 1) 

0



0
0
−η1 d



0
0
H=
−η 2 d RM c M -ω x RM c ( Md ) -ω y RM c ( Md ) δ z 
 -2ω x (η1 − 1) 0 0  Fx 
1  
1  
 

Fy , δ =δ x J=-2ω y (η 2 − 1) 0 0  , F =M  Mδ y 
Tα 
00 0  

The micro delay servo module is composed of Xdirection unit and Y-direction unit. It can measure theinclination of low stiffness structure by tilt sensor, anddrive the lifting unit to track the movement of experimental spacecraft quickly, reducing the disturbance ofthe horizontal component of output tension on the
movement of experimental spacecraft. The universal

η1 =( M + M 1 ) M , η2 =(M + M2 ) M© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

361

Adaptive Sliding Mode Control

N



0


0
=
, q

2 
 R ( g + ( MM b g + LM c k ) M ) 

F = Fe + Fd + FsL ω x 
ω y 
 



1
1
1
−1
 De
Fe = − B ( q ) diag 
,
,
z −1
z −1
z −1
 zv e



zv
zv
e
e
2 2
3 3
(6)
 1 1


( − A ( q, q ) + q )F = − B ( q ) ( diag ( φ ) sign ( s ) )Fd = B ( q )d =L + L0 + Mg k , M c =M a 2 + M b + M

−1

d

−1

3. ASMC DesignsWhen e p = 0 , define

In order to suppress the uncertain disturbance of experimental spacecraft during the operation of CTSS, a new
ASMC is designed in this section. Define variables:(A ( q, q ) =− H ( Jq + N )−1

 =  δ , δ , δ ==, δ H Hδ
B (q) H 1 2 3
−1

−1

z −1

)

1 zv p e p = 0,=
p 1, 2, 3

Proof. The error of the adaptive parameter φ is defined asφ =φ − φˆ , and the positive definite scalar function is
selected as:

T

The system model (1) is transformed intoq=A ( q, q ) + B ( q ) F + δ

=
VL

(2)

(

diag (θ1 , θ 2 , θ3 ) = zVdiag e

The simulation of total weightlessness environment isstudied in this paper, so the expected value of q is defined

z −1

)

The differentiation of Eq. (4) and Eq. (7) is

as qd = [ 0, 0, 0] . The system error is e= q − qd , and theT(=s De + zVdiag eupper bound of δ is β = [ β1 , β 2 , β 3 ] . Thus, Eq. (2) is
T

z −1

) e

 s T s + φ T diag ( f , f , f ) diag (θ , θ , θ ) φ
V=
L
1
2
3
1
2
3

transformed intode=A ( q, q ) + B ( q ) F + δ − q

1 T 1
s s + φ diag ( f1 , f 2 , f 3 ) diag (θ1 , θ 2 , θ3 ) φ
2
2
(7)

(3)

Substitute the ASMC Eq. (6) into Eq. (8)(The switching function is defined as: =VL s T diag (θ1 , θ 2 , θ3 ) -diag ( φ ) sign ( s ) + δT=s De + Vsgn(e) z+ φ T diag (θ1 , θ 2 , θ3 )  s 1 , s 2 , s 3 
(4)
T
z
z
z
sgn(e) z =  e1 sign(e1 ), e2 sign(e2 ), e3 sign(e3 ) = θ1 δ1 s 1 −ϕˆ1 s 1 + θ 2 δ2 s 2 −ϕˆ2 s 2(

Where, D = diag(d1 , d 2 , d3 ) and V = diag(v1 , v2 , v3 ) are

(

)

)

(

)

)
(9)

+ θ3 δ3 s 3 −ϕˆ3 s 3 ≤ 0

positive definite matrices, s = [ s1 , s2 , s3 ] , z = diag ( z, z, z )T

According to the analysis of Eq. (9), the controller Eq.
(6) can make the system stable. Further, substituting thecontroller Eq. (6) into the system error model Eq. (3), it
can be found that the solution ( e, e ) of the system error

and satisfies 1 < z < 2 .
Define φ = [ϕ1 , ϕ 2 , ϕ3 ] is the adaptive parameter ofT

the ASMC, and the adaptive law is selected as:

s s
s 
φ =  1 , 2 , 3  f1 f 2 f 3 (8)model will not make VL ≡ 0 except for the origin ( 0, 0 ) .

T

Therefore, the system error e will converge to 0. The
proof is completed.

(5)

Where, f1 , f 2 , f 3 are all positive constants.

4. Experimental Analysis

Assumption 1. The stable value φˆ = [ϕˆ1 , ϕˆ2 , ϕˆ3 ] of theT

In this section, the ASMC is applied to the CTSS for
simulation experiments. The adaptive parameter of the
ASMC is dynamically adjusted according to the value ofEq. (4), and finally converge to a constant. In order to

1, 2, 3 .
adaptive parameter satisfies ϕˆ p > β p , p =Theorem 1. Under Assumption 1, the system error e will
converge to 0, if the ASMC F is designed as:© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

362

Yuxin Jia, Yingmin Jia, Kai Gong

reduce the oscillation caused by the Sign(s), Eq. (10) is
used to replace it in the simulation.

s ( s + 0.05 )(10)

The parameters of the CTSS model Eq. (1) are R=0.02m,
M=15.5kg, Ma=0.52kg, Mb=4kg, M1=16kg, M2=41.5kg，
k=690N/m, L0=0.45m, g=10m/s2. The pulse disturbanceshown in Fig. 2 is applied to the system.

0.0022°, the maximum error of ω y is 1.254×10-5 rad,which is 0.0007°, and the maximum error of L is
5.151×10-5 m. Therefore, the output tension error of thesystem is 0.036 N, which satisfies the requirements of
microgravity simulation.
The SMC in Ref. 5 is also applied to the CTSS, andits state response is shown in Fig. 4. The results show thatthe convergence speed of the ASMC system is faster andthe steady-state error is smaller. When the simulation
time t >1.6 s, the state error of the ASMC system satisfies
e < 4 × 10−6 , while the SMC system needs 1.7 s.
5. Conclusions
A new ASMC is designed for the model of CTSS in thispaper. The adaptive parameter can be dynamicallyadjusted with the value of the switching function. The
asymptotic stability of the ASMC system is proved, andthe simulation experiment is carried out to compare withthe controller in Ref. 5. Experiments show that the
ASMC system has smaller steady-state error and fasterconvergence speed, which satisfies the requirements of
microgravity simulation.

Fig. 2. External disturbance

The parameters of the ASMC are as follows:

f=
f=
f=
1, φ
=
1
2
3

[1, 1, 1] , z =1.1
T

D =diag(280, 510, 450), V =diag(7.5, 8, 9)

Acknowledgements

Where φ is the initial value of the adaptive parameter.

This work was supported by the NSFC (61520106010,
61327807, 61134005, 61521091) and the National Basic
Research Program of China (973 Program: 2012CB
821200, 2012CB821201).
References
1. Z. Jiang, J. Xu, H. Li, et al, Stable Parking Control of a
Robot Astronaut in a Space Station Based on Human
Dynamics, IEEE Trans. Robot., 36(2), 2020, pp. 399-413.
2. C. Wang, H. Nie, J. Chen, et al, The design and dynamic
analysis of a lunar lander with semiactive control, Acta
Astronaut., 157, 2019, pp. 145-156.
3. X. Liu, G. Cai, M. Wang, et al. Contact control for
grasping a non-cooperative satellite by a space robot,
Multibody Syst. Dyn., (5), 2020, pp. 119-141.
4. A. Witze, NASA rethinks approach to Mars exploration,
Nature, 538(7624), 2016, pp. 149-150.
5. J. Jia, Y. Jia, S. Sun, Continuous Non-singular Fast
Terminal Sliding Mode Control for an Active Gravity
Field Simulator, Journal of Robotics Networking andArtificial Life, 3(4), 2017, pp. 253-256.
6. V. Arakelian, Gravity compensation in robotics, Adv.
Robot., 30(2), 2016, pp. 79-96.

Fig. 3. State response of the ASMC system

Fig. 4. State response of the SMC system in Ref. 5

The state response is shown in Fig. 3. It can be found thatthe maximum error of ω x is 3.779×10-5 rad, which is© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

363

Weighted Multiple Model ADRC for Uncertain Linear Systems
Weicun Zhang1*，Liuqiao Song2, Jing Ge1

School of Automation and Electrical Engineering, University of Science and Technology Beijing,

1

30 Xueyuan Road Haidian District, Beijing, China
E-mail: weicunzhang@ustb.edu.cn
www.ustb.edu.cn
School of electrical engineering and computer science, Oregon State University,

2

Corvallis, Oregon OR 97331-5501, USA

Abstract
For uncertain linear systems such as parameter jumping, this paper presents a weighted multiple model adaptive
control frame work, which uses fixed model set to cover the uncertainties of the real plant to be controlled, and foreach local model with minor uncertainties, the corresponding local controller is designed based on active disturbancerejection controller (ADRC) approach. Some simulations have been conducted based on MATLAB to verify theeffectiveness of the proposed weighted multiple model ADRC adaptive control algorithm.
Keywords: Multiple model adaptive control, ADRC, weighting algorithm.

strategy, which was originally proposed by Professor
Jingqing Han [13-20].
Aiming at a class of uncertain linear controlled plants
with nonlinear disturbances, this paper combines theactive disturbance rejection controller and the weighted
multiple model adaptive control system to design aweighted multiple model ADRC control system for thesecond-order time-delay system and a weighted multiple
model ADRC control system for the third-order uncertain
system, so that to completed the control of these two
uncertain linear systems with nonlinear disturbances. The
simulation results show that compared with a single
ADRC control system, the weighted multiple model
ADRC control system can better meet the control

1. Introduction
For systems with large uncertainties, such as, model
structure change or parameter jumping, conventional
adaptive control can no longer meet the performanceneeds of industrial control systems. Therefore, thecontrol research of uncertain systems has important
theoretical significance and practical significance.
Weighted multiple model adaptive control (WMMAC)[1-12] can effectively control complex and uncertain
systems with prescribed performance indexes. Aiming at
the uncertain system, this paper proposes a weighted
multiple model adaptive control approach with active
disturbance rejection control (ADRC) as local control*

Corresponding author：W. Zhang, weicunzhang@ustb.edu.cn© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

364

Weicun Zhang，Liuqiao Song, Jing Ge

performance requirements and has stronger tracking
ability and robustness when the parameters are uncertain
in a large range or with multiple jumps.

Second, we need to design the control law based on the
ESO

The block diagram of the proposed system is shown in
Fig. 1.

where e1==
u0 k p e1 − kd z2(4)

v1 − z1 , v is the reference input of thesystem, k p and kd are adjustable parameters.
Based on (4), the local control law is given by𝑢𝑢 =𝑢𝑢0 −𝑧𝑧3𝑏𝑏0(5)

3. Multiple model ADRC

Consider time-varying uncertain plant

Figure 1 The block diagram of the proposed system

𝑥𝑥̇1 = 𝑥𝑥2
𝑥𝑥̇ 2 = 𝑥𝑥3 + 𝑏𝑏(𝑡𝑡)𝑢𝑢(𝑡𝑡)̇(=𝑓𝑓
𝑥𝑥1 , 𝑥𝑥2 , 𝑑𝑑(𝑡𝑡), 𝜔𝜔(𝑡𝑡), 𝑡𝑡) = ℎ(𝑡𝑡)
𝑥𝑥̇
⎨ 3
⎩
𝑦𝑦 = 𝑥𝑥1⎧

2. Local controller design based on ADRC
Consider a plant with small uncertainties
𝑥𝑥̇1 = 𝑥𝑥2⎧
𝑥𝑥̇ 2 = 𝑥𝑥3 + 𝑏𝑏0𝑢𝑢(𝑡𝑡)(1)⎨𝑥𝑥̇ 3 = 𝑓𝑓̇ (𝑥𝑥1 , 𝑥𝑥2 , 𝜔𝜔(𝑡𝑡), 𝑡𝑡) = ℎ(𝑡𝑡)⎩𝑦𝑦 = 𝑥𝑥1
where 𝑓𝑓(. ) represents unknown unmodeled dynamics or()Then we need to design multiple model ADRC, whichuses multiple models (model set) to cover theuncertainties of 𝑏𝑏(𝑡𝑡) , for each (local) model, a local
controller (ADRC controller) is designed off-line, and theglobal control law is weighted sum (on-line) of all the
local controllers. That is()‘total disturbance’, u t control input, ω t external()

disturbance, 𝑏𝑏0 control gain, y t system output.

First, we need to design the extended state observer(ESO)
𝜀𝜀1 = 𝑧𝑧1 − 𝑦𝑦𝑧𝑧̇1 = 𝑧𝑧2 − 𝛽𝛽01 𝜀𝜀1�𝑧𝑧̇ = 𝑧𝑧 − 𝛽𝛽 𝜀𝜀 + 𝑏𝑏 𝑢𝑢
2
3
02 1
0
𝑧𝑧̇3 = −𝛽𝛽03 𝜀𝜀1(6)𝑢𝑢(𝑡𝑡) = � 𝑝𝑝𝑖𝑖 (𝑡𝑡)𝑢𝑢𝑖𝑖 (𝑡𝑡)where 𝑢𝑢𝑖𝑖 (t) is calculated according to (5).

The weighting algorithm is described as follows(Suppose the number of local controllers is N)(2)

Its discrete-time version has the following form

z1 ( k ) − y ( k )ε=1) z1 (k ) + h ( z2 ( k ) − β 01ε1 ) z1 ( k + 1=

) z2 (k ) + h ( z3 ( k ) − β 02ε1 + b0u ) z2 ( k + 1=
 z ( k + 1=
) z3 (k ) − hβ 03ε1
 3

(3)

Where ε1 is observation error， β 01 , β 02 , and β 03 are
the gains of the observer ，

z1 , z2 , and z3 are thewhere 𝑒𝑒𝑖𝑖 is the output error of each local model, i.e.,𝑒𝑒𝑖𝑖 (𝑘𝑘) = 𝑦𝑦(𝑘𝑘) − 𝑦𝑦𝑖𝑖 (𝑘𝑘).

estimations of y , y , and 𝑓𝑓(. ), respectively.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

365

Weighted Multiple Model ADRC

The calculation of weights values is carried out at eachtime instant 𝑘𝑘, and through a ZOH (Zero Order Hold),
𝑝𝑝𝑖𝑖 (𝑡𝑡)can be obtained from 𝑝𝑝𝑖𝑖 (𝑘𝑘) .
4.

5.

This paper presented a multiple model ADRC framework
for uncertain dynamic time-varying system. Simulation
results verified the effectiveness of the proposed methods
including weighting algorithm and local controller
scheme. In the future research work, we will focus onmodel set selection for time-varying system with morethan one variable.

Simulation results

To verify the effectiveness of the proposed multiple
model ADRC framework, many simulations have been
conducted with ω ( t ) = 0.1sin t in equation (6) ，basedon MATLAB/ Simulink.
Suppose the plant will change from Model 1(corresponding to 𝑏𝑏(𝑡𝑡) = 0.6 in equation (6)) to Model 2(corresponding to 𝑏𝑏(𝑡𝑡) = 1 in equation (6)) at time 30s,
and finally change to Model 3 (corresponding to 𝑏𝑏(𝑡𝑡) =1.5 in equation (6)) at time 60s. The model set include 4
models, i.e., Model1, Model 2, Model 3, and Model 4(corresponding to 𝑏𝑏(𝑡𝑡) = 2 in equation (6)). The step
responses of multiple model ADRC system and single
ADRC system are shown in Fig. 2.

system output

2

Acknowledgements
This work was partly supported by National Natural
Science Foundation of China (No. 61520106010;
61741302).
Referencesreference input
multiple ADRC
single ADRC

1.5
1
0.5
0

0

10

20

40
t / sec

30

50

60

70

80

Figure 2. Step response comparison between multiple model
ADRC and single ADRC

p3

p2

p1

Weight values of multiple model ADRC are shown in
Figure 3.

p4

Conclusion and future work

1
0.5
0
0

10

20

30

40

50

60

70

80

1
0.5
0
0

10

20

30

40

50

60

70

80

1
0.5
0
0

10

20

30

40

50

60

70

80

1
0.5
0
0

10

20

30

40
t / sec

50

60

70

80

Figure 3. Weight values of multiple model ADRC

1. Fekri, M. Athans and A. Pascoal, Issues, progress andnew results in robust adaptive control, International
Journal of Adaptive Control and Signal Processing, 2006;
20(10): 519-579
2. D. T. Magill, Optimal adaptive estimation of sampled
stochastic processes, IEEE Trans. Automat. Contr., 1965;
10: 434-439
3. D. G. Lainiotis, Partitioning: A unifying framework foradaptive systems-I: Estimation; II: Control, Proc. IEEE,
1976; 64:1126-1143 and 1182-1197
4. M. Athans et al., The stochastic control of the F-8C
aircraft using a multiple model adaptive control (MMAC)
method-Part I: Equilibrium flight, IEEE Trans. Automat.
Contr., 1977, 22: 768-780
5. A. Badr, Z. Binder, D. Rey, Weighted multi-model control,International journal of systems science, 1992; 23(1): 145149
6. M. Kuipers, P. Ioannou, Practical Robust Adaptive
Control: Benchmark Example, Proc. of American Control
Conference, Seattle, Washington, USA, 2008:5168-5173
7. Kuipers and P. Ioannou, Multiple model adaptive control
with mixing, IEEE Transactions on Automatic Control,
2010; 55(8), 1822-1836
8. S. Baldi, P. A. Ioannou, and E. Mosca, Multiple Model
Adaptive Mixing Control: the Discrete-time case, IEEE
Trans. on Automatic Control, 2012; 57(4):1040-1045
9. N. Sadati, G. A. Dumont, and H. R. Feyz Mahdavian,
Robust Multiple Model Adaptive Control Using Fuzzy
Fusion, 42nd South Eastern Symposium on System
Theory, Tyler, TX, USA, March 7-9, 2010
10. Z. Han, and K. S. Narendra, New Concepts in Adaptive
Control Using Multiple Models, IEEE Transactions on
Automatic Control, 2012; 57(1): 78-89
11. W. Zhang, Stable Weighted Multiple Model Adaptive
Control: Discrete-Time Stochastic Plant, International
Journal of Adaptive Control and Signal Processing, 2013;27(7): 562-581© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

366

Weicun Zhang，Liuqiao Song, Jing Ge

12. W. Zhang, Further Results on Stable Weighted Multiple
Model Adaptive Control: Discrete-Time Stochastic Plant,
International Journal of Adaptive Control and Signal
Processing, 2015; 29(12): 1497-1514
13. J. Han. Active disturbance rejection control technique-thetechnique for estimating and compensating theuncertainties. Beijing: National Defense Industry Press;
2008.
14. J. Han. From PID to active disturbance rejection control.
IEEE Transactions on Industrial Electronics 2009; 56(3):900-906.
15. Aaron Radke, and Z. Gao. A survey of state anddisturbance observers for practitioners, Porc. American
Control Conference. 2006 5183-5188.
16. Y. Huang, W.C. Xue. Active disturbance rejection control:
Methodology and theoretical analysis. ISA Transactions
2014; 53: 963-976.

17. Wenchao Xue, Wenyan Bai, Shen Yang, Kang Song, Yi
Huang, and Hui Xie. ADRC with adaptive extended state
observer and its application to air-fuel ratio control ingasoline engines. IEEE Transactions on Industrial
Electronics 2015; 62: 5847-5857.
18. Z. Gao. Active disturbance rejection control: a paradigm
shift in feedback control system design. Proc. American
Control Conference. 2006 2399-2405.
19. Qing Zheng, Linda Q. Gao, & Zhiqiang Gao. On stability
analysis of active disturbance rejection control for
nonlinear time-varying plants with unknown dynamics.
Proc. IEEE Conference on Decision and Control. 2007
3501-3506.
20. Qing Zheng, Zhongzhou Chen, & Zhiqiang Gao. A
practical approach to disturbance decoupling control.
Control Engineering Practice 2009; 17: 1016-1025.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

367

Trajectory Tracking Control of Differential Wheeled Mobile Robots
Based on Rhombic Input Constraints
Kai Gong

The Seventh Research Division and the Center for Information and Control, School of Automation Science andElectrical Engineering, Beihang University (BUAA)
Beijing, 100191, China
Yingmin Jia *The Seventh Research Division and the Center for Information and Control, School of Automation Science andElectrical Engineering, Beihang University (BUAA)
Beijing, 100191, China
Yuxin Jia

The Seventh Research Division and the Center for Information and Control, School of Automation Science andElectrical Engineering, Beihang University (BUAA)
Beijing, 100191, China
E-mail: buaa_gk@buaa.edu.cn, ymjia@buaa.edu.cn, yuxinjia@buaa.edu.cn,
www.buaa.edu.cn

Abstract
This paper focuses on the trajectory tracking control algorithm for differential wheeled mobile robots (DWMRs)
based on rhombic input constraints. The kinematics and dynamics model of DWMRs are Established, and vector
analysis method is used to design the controller when the linear velocity and angular velocity of DWMRs were notindependent of each other. Through the trajectory tracking simulation of the 8-shaped curve, a good control
performance is obtained.
Keywords: differential wheeled mobile robots, rhombic input constraints, trajectory tracking, vector analysis

existing studies are assume that the input constraints of
the robots’ linear velocity v and angular velocity ω areindependent of each other, that is, v ≤ m1 , ω ≤ m2 ,1. Introduction
The tracking control of DWMRs has a very broad
application background. There are many methods havebeen used in controller design for trajectory tracking.
Sliding mode control,1 backstepping control,2 robust
control,3 fuzzy control,4 active disturbance rejection
control5 etc. are used to solve this problem. From apractical perspective, the input constraints must be
considered when designing controller, however mostwhere m1 and m2 are positive constants. (Fig.1). A proof
will be given later, the actual situation is that the input
field of DWMRs is the rhombic area defined byv / m + ω l / m ≤ 1 , where m is the maximum velocity ofthe two driving wheels and l is half of the distance*Corresponding author.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

368

Kai Gong, Yingmin Jia, Yuxin Jia

2.2. Tracking Control Based on Rhombic Input
Constraints

between them. If the controller in Ref.6 is applied to adifferential drive robots, the rectangle estimation needs
to be extracted in the rhombic input field. The largest
rectangle estimate that can be calculated is determined byv ≤ m / 2 and ω ≤ m / 2l , it is only half of the rhombic

The kinematics and dynamics equations of two-wheel
differential mobile robots isx =v=cos θ , y v=
sin θ , θ ω
(5)input field, which leads to the robots’ mobility cannot be
fully utilized. Rhombic input constraints are considered
first time in Ref.7, it proposed a geometric analysis
method to design time-varying feedback parameters.

( x, y ) is the center point coordinates of DWMRs and θ
is used to indicate its azimuth angle (see Fig.2).

ω

θe

m/l

θr

yr
vl

v

m/2l

ω

y
m/2

m

θ

2l

v

vr
x

xr

Fig. 2. Trajectory tracking of DWMRs
Assumption 1. The input constraint of DWMRs is (4),and its reference trajectory satisfies:x =v=cos θ , y v=
sin θ , θ ω
(6)Fig. 1. Rectangular and diamond constraints

r

r

r

r

r

r

r

r

2. Problem Statement

andvr / m + ω r l / m ≤ 1 − lε / m

2.1. Rhombic Input Constraints

Among them, ( xr , yr , θ r , vr , ωr ) is the target values of( x, y, θ , v, ω ) , where ε is a constant satisfies 0 <ε < m / l .

Consider DWMRs shown in Fig.2, the driving wheels’velocities of the robots are vl and vr respectively.
Assuming that the two driving wheels have the same
performance, there maximum velocities are both m , thatis vl ≤ m and v r ≤ m . Usually v and ω of DWMRs areused as control inputs, and their relationship with the
speed of the driving wheel is(1)=
v (vl + vr ) / 2=
ω (vr − vl ) / 2l
Thus v and ω are constrained by−(m + v) / l ≤ ω ≤ (m + v) / l , v ∈ [−m, 0)−(m − v) / l ≤ ω ≤ (m − v) / l , v ∈ [0, m]
The above is collated into one expression:
Equation (3) can be sorted into one expression:v / m + ω l / m ≤1

(7)

Remark 1. We introduce a constant ε in equation (7) toensure that the reference trajectory can be tracked by
DWMRs under the input constraints (4)Fig.2 shows that system errors of DWMRs are definedas: xe   cos θ sin θ 0   xr − x y  =(8) e   − sin θ cos θ 0   yr − y θ e   0

0 1 θ r − θ 

(2)
(3)

The tracking errors system can be obtained by deriving
the two sides of the above formula (8)=xe vr cos θ e − v + ω ye

(4)=y e vr sin θ e − ω xe
(9)=θe vr ωr − ω
Now our goal is to design the controller to make thesystem errors xe , ye and θ e tend to zero, and the control

This constraints is shown in the Fig.1 as a rhombus withblack solid line.

variables v and ω must meet the constraints (4).

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

369

Trajectory Tracking Control of

3. Controller Design Based on Rhombic Input
Constraints

then by defining other vectors as:0 vr cos θ e   

, AB
=
OA =
Before our controller design work begins, we first k y vr ye sin θ e ωr understand two important lemmasθ e 
(12)
Lemma 1.8 f :[0, ∞) → R is first-order continuous  k x xe    0 BC =
 , CD  k θ differentiable and lim f (t ) is a finite value, if =
t →∞
 0 
 θ e
f (t ), t ∈ [0, ∞) is uniformly continuous, then lim f (t )=0 .
Then we can get the vector representation of thet →∞controller
7    Lemma 2. There is a scalar function ρ ( x), x ∈ [0, ∞) ,OD = OA + AB + BC + CD
(13)
which satisfies the following properties:
In order to make the controller meet the rhombic input
1. ρ ( x) is a continuous and non-decreasing function;
constraints, we need to analyze each vector in turn.
2. ρ (0)=0 , and 0 < ρ ( x) ≤ 1 for x ∈ (0, ∞) ;vr cos θ e ωr lv
ωllε
3. lim+ ρ ′( x) = ρ0 , which ρ0 is a positive constant.
≤ r + r ≤ 1−
+
(14)
x →0
m
m
m
m
m
Define ψ ( x) asF2ω ρ ( x) / x x ∈ (0, ∞)ψ ( x)= 
(10)
x=0
 ρ0
Then, for ∀σ ∈ (0, ∞) , there always exist α and β , suchm/lthat α ≤ ϕ ( x) ≤ β holds for x ∈ [0, σ ] , where both α andβ are positive constants.

②

ρ ( x) = tanh( x) is a function that satisfies the above

E3

conditions.
There are many results about the design of the tracking
controller of the differential drive robots in the existing
literature. In this paper we choose the controller in Ref.9.
=v vr cos θ e + k x xeω=
ωr + k y vr yesin θ eθe+ kθ θ e

F
F1
①

D

E2

E1

B

C
O

E4
m

v

A
③

④
F4

(11)
F3

where k x , k y and kθ are positive constants. ThroughFig. 3. Vector method design controller
Equation (14) shows that OA satisfy the rhombic input

constraints, without loss of generality, we represent OA
as shown in Fig.3, and because the length of AB is
proportional to k y , we can definitely find a k y to make
AB within the rhombic input constraints. Similarly, we


can also find suitable k x and kθ , so BC and CD can meetthe rhombic constraints respectively. Obviously, thecontroller OD will definitely meet the rhombic input
constraints.
Since the requirement for k y is ky ≥ 0 , we intuitively

formula (11), we can easily see that too large errors willcause the control variables v and ω to be too large, andthen the control variables will exceed the constraints. In
this way, the control commands cannot be executed well.
Lemma 3.7 For controller (11), if following conditions
are met:1) k x ≤ k x ≤ k x , k y ≤ k y ≤ k y , kθ ≤ kθ ≤ kθ2) k y is differentiable and ky ≥ 0 .
where k x , k x , k y , k y , kθ , kθ are positive constant values,
Then, trajectory tracking errors of DWMRs will
converge to zero, that is xe , ye , θ e will converge to zero.
Controller (11) can be designed by using the vector
analysis method, define the controller v, ω as a vector  v 
OD =  ω thought of designing if from Lyapunov function V (t )V (t )=

θ2
1 2( xe + ye2 + e )2ky© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

370

(15)

Kai Gong, Yingmin Jia, Yuxin Jia

sgn(θ e )(m − sgn( xe )vr cos θ e ))lWhere sgn(⋅) is sign function

Let k y be

ky =

λε

(16)

m 2V (t ) + µ 2

Where λ and µ are constants, 0 < λ < 1 and µ > 0 .
According to (15) (16), we can get

ky
ky =

−mθ e2 + m 2θ e4 + 4λ 2ε 2 ( xe2 + ye2 + µ 2 )2m( xe2 + ye2 + µ 2 )2k x k y2 xe2 + 2kθ k yθ e2
2k y ( x + y + µ ) + θ
2
e

2
e

2

2
e

>0

F : (vr cos θ e ,(22)

| x | / x x ≠ 0sgn( x) 
x=0
0

(23)



Further we can get the expressions of BE and BF as  BE
= OE − OB(17)= (sgn( xe )(m − sgn(θ e )(ωr + k y vr ye(18)

sin θ e

θe

)l )

(24)

−vr cos θ e , 0)T  BF
= OF − OBysgn(θ e )(m − sgn( xe )vr cos θ e )
get
= (0,
− ωr
(25)
l
λε
λε
=
=
≤ ky ≤kyky(19)
sin θ e Tmµm 2V (0) + µ 2−k y vr ye
)

θe
In this way, the vector OB can be expressed as:
To design k x and kθ , let  sin θ e T
OB = OA + AB = (vr cos θ e , ωr + k y vr ye)(20) ρ ( xe ) 
θe
BC =
BE

2
(26)
Similar to OA satisfy the rhombic input constraints, we ρ ( θ e ) can easily verify that OB satisfies the rhombic input
CD =
BF
2
constraints through formulas (14), (15) and (16). Since


Then, we get from (12)(24)(26), thatk x , kθ > 0 , so the directions of the vectors BC and CD
ψ ( xe )sin θ e(m − sgn(θ e )(ωr + k y vr ye)lkx =are determined by the signs of xe and θ e , In order to2θeexpand the input field as much as possible while meeting− sgn( xe )vr cos θ e )the rhombic input constraints. First, we need to determine(27)ψ ( θ e ) m − sgn( xe )vr cos θ e
the triangle area △BEF where the points of C and D are(kθ=− sgn(θ e )located, as shown in Fig.3, when xe < 0 and θ e > 0 , we2lsin θ etake the constraint segment ② to determine the reference(ωr + k y vr ye))θetriangle △BE 2 F2 , similarly, when xe > 0 and θ e > 0 , we
By formula (15)(16)(20)and Lemma2 we can easily gettake the constraint segment ① to determine the reference
α (1 − λ )ε ltriangle △BE1F1 , when xe < 0 and θ e < 0 , we get thekx ≤≤ kx ≤ β m ≤ kx
2
(28)
reference triangle △BE 3 F3 , and when xe < 0 and θ e < 0 ,α (1 − λ )ε
βm
k
k
k
≤
≤
≤
≤
we get the reference triangle △BE 4 F4 . Through the
θ
θ
θ
2
2
equations of the four constraint lines and the coordinates
At this point, the k x , k y and kθ meet the two conditions inof point B , we can easily obtain the coordinates of point
Lemma 3, so the system error will converge to zero. And
E as:because of our vector method design the parameterssin θ eensure that the parameter control variables v and ω meetE : (sgn( xe )(m − sgn(θ e )(ωr + k y vr ye)l ),θe
the rhombic input constraints too.
(21)
sin θ ewr + k y vr ye)
4. Simulation Results
θeIn this section, we simulate and verify the effect of the
Similarly, we can get the coordinates of F as:
controller. The maximum speed of the drive wheels is set
to m = 0.4m / s , the wheel spacing is set to l = 0.16m ,If k x > 0 and kθ > 0 , then according to equations (17) and(18), k < 0 can be derived, and further from (15) we can© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

371

Trajectory Tracking Control of

5. Conclusion

For setting some parameters of the controller, we choosex), ε 0.1,=
ρ ( x) tanh(==
λ 0.99,=µ 0.01 .
Fig.4(a) shows that DWMRs gradually tracks on thereference trajectory. The tracking errors xe , ye , θ e are
gradually converge to zero as shown in Fig.4(b), also we
can guarantee the control variables v, ω satisfy therhombic input constraints through Fig.4(c), andsometimes v can basically reach the bounds of rhombic
input constraints.

The trajectory tracking problem of DWMRs withrhombic input constraints is solved in this paper,compared with the controller based on rectangular input
constraints. It can better play the robots' mobility and thecontroller not only solves the tracking problem, but also
solves the stability problem, the simulation shows thecontroller is effective. Future work will focus on thecontroller design with uncertainty based on a more
complex application environment.

1.5
Reference
Trajectory
Tracking

1

Acknowledgements

TrajectoryThis work was supported by the NSFC (61520106010,
61327807, 61134005, 61521091) and the National Basic
Research Program of China (973 Program: 2012CB
821200, 2012CB821201).

y(m)

0.5

0

-0.5

-1

-1.5

References
-1

-0.5

0

0.5

1. Jinhua. Ye , Y. Suzhen , and J. Xiao . Trajectory Tracking
Control of WMR Based on Sliding Mode DisturbanceObserver with Unknown Skidding and Slipping. 2017 2ndInternational Conference on Cybernetics, Robotics andControl (CRC) IEEE, 2018.
2. Hadi, Nabil H , and K. K. Younus . Path tracking andbackstepping control for a wheeled mobile robots (WMR)
in a slipping environment. IOP Conference Series
Materials ence and Engineering 671(2020):012005.
3. Inoue, Roberto S. , et al. Robust recursive linear quadratic
regulator for wheeled mobile robots based on opticalmotion capture cameras. Asian Journal of Control (2019).
4. Chen, Yung Hsiang . A Novel Fuzzy Trajectory Tracking
Control Design for Wheeled Mobile Robots. Advances In
Robotics & Mechanical Engineering 2(2020).
5. J. Han, From PID to Active Disturbance Rejection Control,
IEEE Transactions on Industrial Electronics, vol. 56, no.
3, pp. 900-906, March 2009.
6. Y. X. Su and C. H. Zheng, Global asymptotic stabilizationand tracking of wheeled mobile robots with actuatorsaturation, in Proc. IEEE Int. Conf. Robots. Biomim., Dec.
2010, pp. 345–350..
7. X. Chen, Y. Jia and F. Matsuno, Tracking Control for
Differential-Drive Mobile Robots With Diamond-Shaped
Input Constraints, IEEE Transactions on Control Systems
Technology, vol. 22, no. 5, pp. 1999-2006, Sept. 2014, doi:
10.1109/TCST.2013.2296900.
8. H. K. Khalil, Nonlinear Systems, 3rd ed. Upper Saddle
River, NJ, USA: Prentice-Hall, 2002.
9. Blazic S.. A Novel Trajectory-Tracking Control Law for
Wheeled Mobile Robots[J]. Robotics and Autonomous
Systems, 2011, 59(11): 1001-1007

1

x(m)

Tracking reference trajectory

(a)

x (m)
e

0.2
0.1
0
0

5

10

15

20

25

30

35

40

45

0

5

10

15

20

25

30

35

40

45

0

5

10

15

20

25

30

35

40

45

y (m)
e

0
-0.1
-0.2
-0.3

e (rad)

1
0.5
0

t(s)

(b)

(c)

Tracking errors

Input and constraints

Fig.4. Simulation results.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

372

Maneuvering target tracking with improved interactive multiple model algorithm
Meiyu Zhu1, Guanyu Chen2, Weicun Zhang1*

School of Automation and Electrical Engineering, University of Science and Technology Beijing,

1

30 Xueyuan Road Haidian District, Beijing, China
E-mail: weicunzhang@ustb.edu.cn
www.ustb.edu.cn

University of Nottingham, NG7 2RD, UK

2E-mail: eexgc10@nottingham.ac.uk

Abstract
With the rapid development of artificial intelligence, maneuvering target tracking technology has become a major
aspect of scientific and technological research with increased performance requirements. Maneuvering target tracking
technology is widely used in military and civilian fields. This paper presents a new maneuvering target tracking
framework, which adopts an improved interactive multiple model (IMM) with new weighting algorithm. Some target
tracking simulations have been conducted based on MATLAB to verify the effectiveness of the proposed
maneuvering target tracking approach.
Keywords: Maneuvering target tracking, interactive multiple model, weighting algorithm.

Introduction

maneuverability of various maneuvering targets has
become more complicated, the motion state is moredifficult to estimate, and the target maneuvering methods
are complex and diversified, making it more and moredifficult to track maneuvering targets. The original
maneuvering target tracking technology can no longer
meet the needs of modern maneuvering target tracking
requirements.
The two key parts of maneuvering target tracking are themaneuvering target tracking model and the adaptive
filtering algorithm. Considering that the motion model ofIn recent decades, with the rapid development of artificial
intelligence, target tracking technology has become amajor aspect of scientific and technological research.
Maneuvering target tracking technology is widely usedin military and civilian fields. For example, in air defense
and air traffic control, reliable and accurate tracking oftargets is always the main purpose of target tracking
system design. However, with the rapid development ofmodern military weapons and the continuous
advancementofaerospacetechnology,
the*Corresponding author：W. Zhang, weicunzhang@ustb.edu.cn© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

373

Meiyu Zhu, Guanyu Chen, Weicun Zhang

Step 1: Interaction

𝑢𝑢𝑖𝑖|𝑗𝑗 (𝑘𝑘 − 1|𝑘𝑘 − 1) =𝑝𝑝𝑖𝑖𝑖𝑖 𝑢𝑢𝑖𝑖 (𝑘𝑘−1)

𝑐𝑐𝑗𝑗 = ∑𝑟𝑟𝑖𝑖=1 𝑝𝑝𝑖𝑖𝑖𝑖 𝑢𝑢𝑖𝑖 (𝑘𝑘 − 1)

𝑐𝑐𝑗𝑗

(4)
(5)

𝑋𝑋�0𝑗𝑗 (𝑘𝑘 − 1|𝑘𝑘 − 1) = ∑𝑟𝑟𝑖𝑖=1 𝑋𝑋�𝑖𝑖 (𝑘𝑘 − 1|𝑘𝑘 − 1)𝑢𝑢𝑖𝑖|𝑗𝑗 (𝑘𝑘 −1|𝑘𝑘 − 1)(6)

1|𝑘𝑘 − 1)(7)𝑋𝑋�𝑗𝑗 (𝑘𝑘|𝑘𝑘 − 1) = 𝐹𝐹𝑗𝑗 (𝑘𝑘 − 1)𝑋𝑋�0𝑗𝑗 (𝑘𝑘 − 1|𝑘𝑘 − 1)

(8)

𝑃𝑃0𝑗𝑗 (𝑘𝑘 − 1|𝑘𝑘 − 1) = ∑𝑟𝑟𝑖𝑖=1{𝑃𝑃𝑖𝑖 (𝑘𝑘 − 1|𝑘𝑘 − 1) + [𝑋𝑋�𝑖𝑖 (𝑘𝑘 −1|𝑘𝑘 − 1) − 𝑋𝑋�0𝑖𝑖 (𝑘𝑘 − 1|𝑘𝑘 − 1)][𝑋𝑋�𝑖𝑖 (𝑘𝑘 − 1|𝑘𝑘 − 1) − 𝑋𝑋�0𝑖𝑖 (𝑘𝑘 − 1|𝑘𝑘 − 1)]𝑇𝑇 ]}𝑢𝑢𝑖𝑖|𝑗𝑗 (𝑘𝑘 −Step 2: Filtering

𝑃𝑃𝑗𝑗 (𝑘𝑘|𝑘𝑘 − 1) = 𝐹𝐹𝑗𝑗 (𝑘𝑘 − 1)𝑃𝑃0𝑗𝑗 (𝑘𝑘 − 1|𝑘𝑘 − 1)𝐹𝐹𝑗𝑗𝑇𝑇 (𝑘𝑘 − 1) +𝛤𝛤𝑗𝑗 (𝑘𝑘 − 1)𝑄𝑄𝑗𝑗 (𝑘𝑘 − 1)𝛤𝛤𝑗𝑗 𝑇𝑇 (𝑘𝑘 − 1)(9)𝜀𝜀𝑗𝑗 = 𝑍𝑍(𝑘𝑘) − 𝐻𝐻𝑗𝑗 𝑋𝑋�𝑗𝑗 (𝑘𝑘 − 1|𝑘𝑘 − 1)

(10)

𝐾𝐾𝑗𝑗 (𝑘𝑘) = 𝑃𝑃𝑗𝑗 (𝑘𝑘|𝑘𝑘 − 1)𝐻𝐻𝑗𝑗 (𝑘𝑘)𝑆𝑆𝑗𝑗−1 (𝑘𝑘)(12)

𝑆𝑆𝑗𝑗 (𝑘𝑘) = 𝐻𝐻𝑗𝑗 (𝑘𝑘)𝑃𝑃𝑗𝑗 (𝑘𝑘|𝑘𝑘 − 1)𝐻𝐻𝑗𝑗𝑇𝑇 (𝑘𝑘) + 𝑅𝑅𝑗𝑗 (𝑘𝑘)(11)𝑋𝑋�𝑗𝑗 (𝑘𝑘|𝑘𝑘) = 𝑋𝑋�𝑗𝑗 (𝑘𝑘|𝑘𝑘 − 1) + 𝐾𝐾𝑗𝑗 (𝑘𝑘)𝜀𝜀𝑗𝑗 (𝑘𝑘)(13)

𝑃𝑃𝑗𝑗 (𝑘𝑘|𝑘𝑘) = [𝐼𝐼 − 𝐾𝐾𝑗𝑗 (𝑘𝑘)𝐻𝐻𝑗𝑗 (𝑘𝑘)]𝑃𝑃𝑗𝑗 (𝑘𝑘|𝑘𝑘 − 1)

Suppose target motion can be described by 𝑟𝑟 model，
model set 𝛭𝛭 = {𝑀𝑀1 , 𝑀𝑀2 , ⋯ , 𝑀𝑀𝑟𝑟 }𝑀𝑀𝑀𝑀:
𝑍𝑍(𝑘𝑘) = 𝐹𝐹𝑗𝑗 (𝑘𝑘)𝑋𝑋(𝑘𝑘) + 𝑣𝑣𝑗𝑗 (𝑘𝑘)(3)

IMM filtering has the following 4 steps:

1. Interactive multiple model estimation

𝑋𝑋(𝑘𝑘 + 1) = 𝐹𝐹𝑗𝑗 (𝑘𝑘)𝑋𝑋(𝑘𝑘) + 𝐺𝐺𝑗𝑗 (𝑘𝑘)𝑤𝑤𝑗𝑗 (𝑘𝑘)⋯ 𝑝𝑝1𝑟𝑟⋱⋮ �⋯ 𝑝𝑝𝑟𝑟𝑟𝑟

𝑝𝑝11
𝑃𝑃 = � ⋮𝑝𝑝𝑟𝑟1

the target will also change when the target is maneuvered,any single model will produce larger errors. Also, thetracking process is often accompanied by externalinterference, system parameter jumps and other problems,
these problems will cause the tracking performances todeteriorate. Kalman filtering or multiple model adaptive
estimation (MMAE) or [1-6] method is a good idea tosolve such kind of problem. This article will focus onweighted multiple model adaptive filtering withapplications in maneuvering target tracking.
This article first introduces several commonly usedmaneuvering target tracking models, and analyzes theiradvantages, disadvantages, and scope of application
through simulation, and proposes directions for
improvement. Then introduce and study the filtering
algorithm of target tracking, mainly including Kalmanfiltering algorithm, extended Kalman filtering algorithm,
unscented Kalman filtering algorithm, and analysis andsummary through simulation. Then this paper proposes a
new weighting algorithm based on the traditional MMAE
algorithm, and proves its convergence. The simulation
proves that the new weighting algorithm has thecharacteristics of fast convergence speed and small
calculation amount. Finally, the new weighting algorithm
is combined with the interactive multi-model filtering
algorithm to track the maneuvering target. The
experimental results show that the new algorithm cantrack the target well, and compared with the traditionalalgorithm, the tracking speed is faster and the error issmaller.

(14)

Step 3: Model probability updates(1)

𝑙𝑙𝑗𝑗 (0) =

(2)

1

𝑚𝑚

; 𝑢𝑢𝑗𝑗 (0) = 𝑙𝑙𝑗𝑗 (0)1𝑙𝑙𝑗𝑗′ (𝑘𝑘) = 1 + ∑𝑟𝑟𝑞𝑞=1�𝜀𝜀𝑗𝑗 (𝑞𝑞)�

𝑤𝑤𝑗𝑗 (𝑘𝑘) and 𝑣𝑣𝑗𝑗 (𝑘𝑘) are independent Gaussian noises. The

𝑟𝑟′𝑙𝑙𝑚𝑚𝑚𝑚𝑚𝑚= min{𝑙𝑙𝑗𝑗′ (𝑘𝑘)}covariance matrixes of 𝑤𝑤𝑗𝑗 (𝑘𝑘 and 𝑣𝑣𝑗𝑗 (𝑘𝑘) are 𝑅𝑅𝑗𝑗 and 𝑄𝑄𝑗𝑗 ,𝑖𝑖respectively. 𝑢𝑢𝑗𝑗 (𝑘𝑘 − 1) represents the probability of

(15)
2

(18)

1

(19)

𝑙𝑙𝑗𝑗

described by

𝑢𝑢𝑗𝑗 (𝑘𝑘) = 𝑙𝑙𝑗𝑗 (𝑘𝑘)𝑐𝑐𝑗𝑗
𝑐𝑐

𝑐𝑐 = ∑𝑟𝑟𝑗𝑗=1 𝑙𝑙𝑗𝑗 (𝑘𝑘)𝑐𝑐𝑗𝑗© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

374

(17)

𝑙𝑙′

𝑙𝑙𝑗𝑗 (𝑘𝑘) = ′𝑚𝑚𝑚𝑚𝑚𝑚
𝑙𝑙 (𝑘𝑘 − 1)(𝑘𝑘) 𝑗𝑗model 𝑀𝑀𝑀𝑀 , the model transition probabilities can be

(16)

(20)

Maneuvering target tracking with

Step 4: Combination𝑋𝑋�(𝑘𝑘|𝑘𝑘) = ∑𝑟𝑟𝑗𝑗=1 𝑋𝑋�𝑗𝑗 (𝑘𝑘|𝑘𝑘)𝑢𝑢𝑗𝑗 (𝑘𝑘)(21)

𝑋𝑋�(𝑘𝑘|𝑘𝑘)] [𝑋𝑋�𝑗𝑗 (𝑘𝑘|𝑘𝑘) − 𝑋𝑋�(𝑘𝑘|𝑘𝑘)]𝑇𝑇 ]}𝑢𝑢𝑗𝑗 (𝑘𝑘)(22)

𝑃𝑃(𝑘𝑘|𝑘𝑘) = ∑𝑟𝑟𝑗𝑗=1{𝑃𝑃𝑗𝑗 (𝑘𝑘|𝑘𝑘) + [𝑋𝑋�𝑗𝑗 (𝑘𝑘|𝑘𝑘) −2. Simulation results
To verify the effectiveness of the proposed maneuveringtarget tracking based on improved IMM algorithm, many
simulations have been conducted with MATLAB
platform. Simulation results are satisfactory, details areomitted here due to space limit, just to illustrate somepictures of simulation results. Figure 1 shows plane
tracking performance comparisons among Current
Statistical (CS) model, IMM algorithm, and improved
IMM algorithm; Figure 2 and Figure 3 show tracking
performance comparisons in X-direction and Ydirection; Figure 4 shows the weights values of IMM
algorithm; Figure 5 shows the weights values ofimproved IMM algorithm.

Figure 2 Positions: X direction

Figure 3 Positions: Y direction
Figure 4 Weight values of traditional IMM algorithm

0.7
Model 1Model 2

0.6

Figure 1 The filtering results of three algoritms

Model 3

0.5

0.4

0.3

0.2

0.1

0
0

50

100

150

200

250

300

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021n

375

350

400

450

Meiyu Zhu, Guanyu Chen, Weicun Zhang

Acknowledgements
This work was partly supported by National Natural
Science Foundation of China (No. 61520106010;
61741302).
References
1. D. T. Magill, Optimal adaptive estimation of sampled
stochastic processes, IEEE Trans. Automat. Contr., 1965;
10: 434-439
2. D. G. Lainiotis, Partitioning: A unifying framework foradaptive systems-I: Estimation; II: Control, Proc. IEEE,
1976; 64:1126-1143 and 1182-1197
3. R. E. Kalman, “A new approach to linear filtering andprediction problems,” Journal of Basic Engineering, vol.
82, no. 1, pp. 35–45, 1960.
4. P. Zarchan and H. Musoff, Fundamentals of Kalman
Filtering: a Practical Approach, American Institute of
Aeronautics and Astronautics, 4th edition, 2015.
5. X.-R. Li and Y. Bar-Shalom, “Multiple-model estimation
with variable structure,” IEEE Transactions on AutomaticControl, vol. 41, no. 4, pp. 478–493, 1996.
6. W. Zhang, S. Wang, and Y. Zhang, Multiple-model
adaptive estimation with a new weighting algorithm,
Complexity, 2018:1-11,2018

Figure 5 Weight values of improved IMM algorithm

3. Conclusion and future work
This paper presented an improved IMM maneuveringtarget tracking approach. Simulation results verified theeffectiveness of the proposed method including new
weighting algorithm. In the future research work, we will
focus on model set selection with prescribed performancerequirements.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

376

Encapsulated Agents of Hybrid Order Discrete Dynamics
Yunzhong Song †School of Electrical Engineering and Automation, Henan Polytechnic University, 2001 Century Avenue
Jiaozuo, 454003, P.R.China
Weicun Zhang

School of Automation and Electrical Engineering, University of Science and Technology Beijing, 30Xueyuan Road
Beijing, 100083, P.R. China
Fengzhi Dai

School of Electronic Information and Automation, Tianjin University of Science and Technology, 1038 Dagu Nanlu
Tianjin, 300222, P.R. China
Huimin Xiao

School of Computer and Information Engineering, Henan University of Economics and Law, 180 Jinshui Donglu
Zhengzhou, 450046, P.R.China
Shumin Fei

School of Automation, South East University, 2 Sipai Lou
Nanjing, 210096, P.R.China
E-mail:songhpu@126.com,weichunzhang@263.net, daifz@tust.edu.cn, xiaohm@huel.edu.cn, smfei@seu.edu.cn

Abstract
This paper is targeting to investigate the encapsulated cell realization of the hybrid order agents in discrete time,
where agents are composed by two different dynamic order ones, one is the first order and the other is the second
order, the work is the counter of its continuous ones of the encapsulated agents. To be further, the first order agentsare assigned at low speed, and the second order agents are selected at comparatively higher speed. To overcome the
obstacle brought by difference of the sampling rates, the lifting technology is taken into use to analyze theencapsulated system. Theoretical analysis and simulation results are made available for the further reference.
Keywords: encapsulated agents, hybrid order, discrete dynamics, lifting techniques, consensus

attention, partially because of their widely distributed inindustries, social activities and biological groups; one of
the striking point of heterogeneities of the multi-agents istheir difference in dynamics, especially of dynamicorders. The hybrid order dynamics of multi-agents werefirstly put forward and solved by match principle or

1. Introduction
Multi-agent systems have been extensively exploredthese days for their theoretical values and potential
prospects in real engineering applications 1-9. Amongthem, the heterogeneous agents were paid much more†

Corresponding author© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

377

Yunzhong Song, Weicun Zhan, Fengzhi Dai, Huimin Xiao, Shumin Fei

match equivalence, i.e. protocols should be different foragents with different dynamic orders 3,6.
However, it is not so wise for us just to stop at the lowlevel of the multi-agents, something new should be added
to push this research further. Is it possible to pack theagents with hybrid order together? Does that bring us
much more promising solutions to the real engineering
problems? These problems are full of the meaning. Oneof the work of this direction has been done, it is for hybrid
order agents packed in an encapsulated cell. The
dynamics for them is described in continues time 3. In thisnote, discrete time agents of the hybrid order packed in
an encapsulated cell will be touched upon here.
The following of this note will be arranged as follows:
Problem description will be provided at first, thenfollowed by theoretical analysis and simulations, somecomments will also be available in this part; the last partwill be conclusions. Acknowledgements will beacknowledged for the fund support.

 x1i (k + 1) − x1i (k )= u1i (k );n0T0 x i (k + 1) − x i (k )2 2
= v2i (k );T0 v i (k + 1) − v i (k )2 2
= u2i (k );

T0
 i
i
 y (k ) = x1 (k );Corresponding to Equation (1),

 X 1i (k ) = x1i (k ); x2i (k )  i; X 2 (k ) =  i
 v2 (k ) (3)As to the controlled inputs, both for the first order agentsand the second agents, are enforced by three different
parts, one is from the individual agent itself, the anotheris from the same layer of the different order agents, andthe third is from the other layer agents. And in this way,
we have system level of the whole agents, and sub-levelof the layer, followed by the atom-level agents withdifferent dynamic orders.
Notation 1: Parameters of n0 and T0 is assigned torepresent the times of the sampling period and standard
sampling period, respectively; since the first order agent
system only has one integrator, and the second orderagent system has two integrators, the information
updating rate for the second is much more slower than
the first agent system; and in this kind of sense, we chose
T0 as 1 and no as 4.

2. Problem Description
The dynamics of the encapsulated agents are describedin the following equations:i X=
f ( X ij (k ), P) + U ij (k );j ( k + 1)
 ii
Y (k ) = x1 (k );

(2)

(1)

Here, in Equation (1), index i is used to mark the layer of
the agents, and index j is used to mark the order of theagents of the layer i; u(k) is the controlled input for theagents, it comes from three different places, one is fromthe agent itself, the other is from the other agents of thesame layer, and the third is from the other layer agents;
the output signal is extracted from the first order agents;
the capital x and u is employed to express the packed statevariables, i.e. to the agent systems combined by the firstand second order dynamics, both position sub-states andvelocity sub-states are covered by the capital x, thecapital letter p is used to denote the parameters of theagent systems; and the controlled input can be inferredcorrespondingly.
In case of simplicity and without loss of generality, we
assume the multi-agents system is composed by twolayers; each layer includes two different agents, among
them one is the first order and the other is the second
order. So, the Equation (1) can be refined as follows:

3. Theoretical Analysis and Simulation Results
This part will contribute for theoretical analysis andsimulation results, some comments are also offered totrigger the prospective topics.
3.1. Theoretical analysis
Take two homogeneous encapsulated cell agent systems
as an example, each one of them is composed of one of
the first order agent and one of the second order agent, tobe specific, for the first cell encapsulated agent system, it
can be expressed as:© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

378

Encapsulated Agents of Hybrid

 x11 (k + 1)
= x11 (k ) + 4u11 (k ); 1= x12 (k ) + v12 (k ); x2 (k + 1)
 1
= v12 (k ) + u12 (k );v2 (k + 1)
 y1 (k ) = x1 (k );1And the second cell encapsulated agent system, thecontrolled input can be denoted as follows: x12 (k + 1)
= x12 (k ) + 4u12 (k ); 2u112 (k ) + u122 (k );u=
1 (k )
 22
u11 (k ) = −α1 x1 (k );u 2=(k ) β1[ x22 (k − 4) − x12 (k )]; 122= x22 (k ) + v22 (k ); x2 (k + 1)
 2
= v22 (k ) + u22 (k );v2 (k + 1)
 2
2
2
2
u2 (k ) = u21 (k ) + u22 (k ) + u23 (k );u 2 (k ) =−α 2 x22 (k ) − α 3 v22 (k );
 21
2
u22
(=
k ) β 2 [ x12 (k ) − x22 (k − 4)];
 2
1
2
=
u23 (k ) β 3 [ x1 (k ) − x1 (k )]; y 2 (k ) = x 2 (k );
1


(4)

Correspondingly, the second cell encapsulated agent
system can be expressed as: x12 (k + 1)
= x12 (k ) + 4u12 (k ); 2= x22 (k ) + v22 (k ); x2 (k + 1)
 2
= v22 (k ) + u22 (k );v2 (k + 1)
 y 2 (k ) = x 2 (k );

1

(5)

Notation 2: The dynamic of the first order agent in cell
encapsulated agent system is only influenced by its own
dynamics and the dynamics of the second order agent of
the same cell. To be different, the dynamic of the second
order agent in cell encapsulated agent system can bealtered by its own dynamic and also the dynamic of thefirst order agent, which is assigned to the same cell,
besides that, the output of the other cell agent system canalso contribute to the dynamics of the second order agent.

Parameters of the whole system can be listed as follows:


=
α1

β
=
 1

Notation 3: Consider the different sampling speed, thesignal of the second order agent feed to the first orderagent is the 4 times period delayed signal, that embodies
the lifting technique.

1
;α 2
=
8
1
; β2
=
8

1
5
;α 3
;
=
2
3
1
1
; β3
;
=
2
16

(8)

Notation 4: The parameters are selected to guarantee thestability of the whole system.

For the first cell encapsulated agent system, thecontrolled input can be denoted as follows: x11 (k + 1)
= x11 (k ) + 4u11 (k );
 1
1
1
u11
(k ) + u12
(k );
u=
1 (k )
u1 (k ) = −α x1 (k );
1 1
 11
u1 =
(k ) β1[ x12 (k − 4) − x11 (k )]; 12= x12 (k ) + v12 (k ); x12 (k + 1)
 1
= v12 (k ) + u12 (k );v2 (k + 1)
 1
1
1
1
u2 (k ) = u21 (k ) + u22 (k ) + u23 (k );u1 (k ) =−α 2 x12 (k ) − α 3 v12 (k ); 21u122 (=k ) β 2 [ x11 (k ) − x12 (k − 4)];
 1
2
1
=
u23 (k ) β 3 [ x1 (k ) − x1 (k )]; y1 (k ) = x1 (k );
1


(7)

3.2. Simulation results
When the initial values are provided, under thesimulation environment like SIMULINK, the Simulink
results are easily to be made possible. Figure 1 is thesimulation results of the encapsulated system:(6)

Fig.1 State curves of the system© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

379

Yunzhong Song, Weicun Zhan, Fengzhi Dai, Huimin Xiao, Shumin Fei

From the simulation results, it could be seen clearly that
all of the states of the system are driven to the stableequilibrium.
Notation 5: The different equilibrium values can bemade possible if we assign different working modes ofthe sub-system.

SRF for ROCS, SEM to Yunzhong Song as well as
NaturalScience
Fund
of
Henan
Province(182300410112).
References
1.

R. Olfati, R. Murray, Consensus problems in networks ofagents with switching topology and time delays, IEEE
Transactions on Automatic Control, 49(2004) 1520-1533.
2. Y.Song, F. Dai, H. Xiao and S. Fei, No free lunch principle
in agent swarm systems: one Case Study, Journal ofAdvances in Artificial Life Robotics, 1 (2020) 33-37.
3. Y.Song, F. Dai, H. Xiao, On the complexity of encapsulated
agent cell, Proceedings of 2019 Chinese Intelligent
Systems Conference, (2019) 24-30
4. Y. Song, Z. Fu, F. Wang, Socialized multi-agent systemrendezvous via networks of networks, Journal of Robotics,
Networking and Artificial Life, 2 (2016) 136-139.
5. Y. Song, Z. Fu, F. Wang, Flock guiding of hybrid agents via
root block, Journal of Robotics, Networking and Artificial
Life, 5 (2019) 245-248
6. Y. Song, Consensus of agents with mixed linear discrete
dynamics, International Journal of Control, Automationand Systems, 14 (2016) 1139-1143.
7. Y. Song, W. Zhao, Multi-agent system rendezvous via
refined social system and individual roles, WSEAS
Transactions on Systems and Control, 9(2014),526-532
8. Y. Hong, J. Hu, L. Gao, Tracking control for multi-agent
consensus with an active leader and variable topology,
Automatica, 42(2006) 1177-1182.
9. Y. Hong, G. Chen, L. Bushenell, Distributed observers
design for leader-following control of multi-agent networks,
Automatica, 44(2008) 846-850.
10. Y. Jia, Robust control with decoupling performance forsteering and traction of 4WS vehicles under velocityvarying motion, IEEE Transactions on Control Systems
Technology, 8(2000) 554-569.
11. Y. Jia, Alternative proofs for improved LMI representations
for the analysis and the design of continuous-time systems
with polytopic type uncertainty: a predictive approach,IEEE Transactions on Automatic Control, 48(2003) 14131416
12. Y. Jia, General solution to diagonal model matching control
of multi-output-delay systems and its applications in
adaptive scheme, Progress in Natural Science, 19( 2009),
79-90.

3.3. Some comments
In order to make the work tight and complete, twocomments are offered here.
Comment 1: Agent system is kicked off by studying thesame order ones, especially of the first order agents andthe second order agents, after then the hybrid order agent
systems are initiated 3,6 and references there in followedby colleagues of the control field. Is it possible for us tocome back to the same agents but with much more higher
viewpoints?
Comment 2: Encapsulated cell agent systems are theright way for us to dig deep on the road of the same at
low level, the different to deepen of the same, and cometo the same in a higher level.
4. Conclusion
This note contributes to the background introduction,
theoretical analysis and simulation research of theencapsulated cell agents with discrete dynamics. The
value of this note is obvious in several aspects:(1) It reiterates the importance of the layer of the system,
now it is clear that the encapsulated cell agents are
divided into three different levels, from the top to thedown, they are the system level, the cell level, and theagent level;(2) According to our knowledge, it is first discussion
about the sampling speed of the hybrid order agents,
lifting techniques are borrowed to design the controller;(3) For the complex engineering system, the sub-systems
are always intertwined with each other, hierarchy should
be considered of course, and in this sense, our work ismeaningful;Besides, the complexity is the essential characteristics,
time-delay, coupling, fragility, robust property et al, allof them should be taken into count 10-12, it is a long way
for us to explore.
Acknowledgements
This work is partially supported by NSFC Grant(61340041 and 61374079) and the Project-sponsored by© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

380

Kinematic modeling and Simulation of humanoid dual-arm robot
Jiwu Wang

School of Mechanical and Electronic Engineering, Beijing jiaotong University
Beijing, Haidian District, China
Junxiang Xu

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
E-mail: jwwang@bjtu.edu.cn, 19121251@bjtu.edu.cn
www.bjtu.edu.cn

Abstract
In recent years, dual-arm robots have attracted more and more attention due to their advantages such as strong
cooperation ability and high flexibility. With the improvement of real-time requirement of dual-arm cooperation, theinverse kinematics solution of robot becomes a key problem to be solved urgently. In order to solve the timeconsuming problem of inverse kinematics of robot arm, a closed inverse kinematics solution algorithm for humanoiddual-arm robot was proposed. The effectiveness of the algorithm was verified by simulation.
Keywords: dual-arm robots ; real-time ; inverse kinematics；simulation.

Because it takes a long time to meet the accuracy
requirement and solve the problem of manipulator
inverse kinematics solution, the analytical solution isadopted. A closed inverse kinematics algorithm is
proposed for the humanoid robot arms. The algorithm isused for simulation analysis. Simulation results show thatthe algorithm is effective.

1. Introduction
Humanoid dual-arm robot is a kind of bionic robot
designed by imitating the shape, structure and function ofhuman body. It can work with both hands cooperatively
just like human. The inverse kinematics of manipulator
mainly includes geometric method, analytical method
and numerical method. The geometric method is a special
case of analytic method in some cases, and its
applicability is weak1,2. Analytical inverse kinematics ofthe manipulator can efficiently obtain all the inverse
solutions of the manipulator in the desired position, but
the manipulator must satisfy the Piper criterion3. The
numerical method has no special requirements for thejoint number and structure of the manipulator, but itneeds to be solved through continuous iteration, which
not only takes a long time, but the average calculation
error is also 10 times that of the analytical method4,5.

2. Kinematic modeling of dual-arm robot
2.1. Forward kinematics
The robot model in this paper is shown in Figure 1. Eacharm of the robot has six degrees of freedom. The threeaxes of the shoulder joint intersect at one point. Because
this case conforms to the Piper criterion, there are
analytic solutions. Taking the shoulder joint as the basic
coordinate frame6, figure 1 also shows the link coordinate© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

381

Jiwu Wang, Junxiang Xu

frames of the right arm and its D-H parameters. The left
arm and the right arm are identical.

5

'

A6T =

5

4

A4

A3

3

A2

2

1

A1 A 0(3)

The left-hand side of E2 isEL 2 = 0  C6 ( px' + lL 4 ) − S6 p 'y   S6 ( px' + lL 4 ) + C6 p 'y 

 
pz'


0 0
1


(4)

And the right-hand side of E2 is   S4C5lL 2    −C l − l 
4 L2
L3 
ER 2 =    S4 S5lL 2 
1
0 0 0Fig. 1. Link coordinate frames of the right arm and its D-H
parameters.

The position and orientation of the end-effector can beobtained by chain-multiplying the 6 link-transformation
matrices together to obtain the spatial displacement of the
6th coordinate frame with respect to the base coordinate
frame:
0=
T6

where Si ≡ sin θi , Ci ≡ cos θi , Sij ≡ sin (θi + θ j ) , Ci ≡ cos (θi + θ j ) , andlLi are geometric link parameters in Fig. 1. By comparing
the elements (1,4), (2,4) and (3,4) of the EL 2 and ER 2 , we
can obtain C4 and then S4 from C4 , and from which we
can find the joint solution for θ 4 . we can solve θ5 when
we have solved θ 4 .Similarly, we can solve θ 6 when we
have solved θ 4 and θ5 .

6

i −1
Ai 0 A1 1 A 2 2 A3 3 A 4 4 A5 5 A 6∏=
i =1

 x 6 y6 z 6 p6   n s a p 
= =
 

 0 0 0 1   0 0 0 1

C4 =

(1)

S5 =

6

'2
z

2l A2l A3

pz'

2
L2

2
L3

)
(6)

( S 4 lL 2 ))θ 6 = a tan 2 ( − ( C4lL 2 + lL 3 ) , S4C5lL 2 ) − a tan 2 ( p 'y , px' + lL 4 )To solve for the remaining joint angles, we can move thelink transformation matrix A A to the left-hand side ofEq.(3). And we can obtain an equation that we label as
E4 equation.
3

4

4

3

A4

4

A5

5

5

'

A6T =

3

A2

2

1

A1 A 0(7)

The left-hand side of E4 is

 ' '
'
'
n s a p  0 0 0 1 =
A5 5 A 4 4 A3 3 A 2 2 A1 1 A 0 6 A 0

'2
y

(

We take the inverse of both sides of Eq. (1). The new
matrix 𝑇𝑇 ′ is

=
T

2

L4

θ5 a tan 2 S5 , ± 1 − S52=

2.2. Inverse kinematics

−1

'
x

(

principal axes of the coordinate frame i, i−1 A i is a generallink transformation matrix, relating the ith coordinate
frame to the (i−1)th coordinate frame, and [n, s, a, p]
represents the normal vector, the sliding vector, theapproach vector and the position vector of the hand
respectively.

n s a p = 0 0 0 1 

( p +l ) + p + p −l −lθ 4 a tan 2 ± 1 − C42 , C4=

where xi , yi and zi represent the unit vectors along the'(5) a11 a12 a13 a14  a 21 a 22 a 23 a 24 
EL 4 =  a31 a32 a33 a34 


0
0
1 
 0

(2)

And the right-hand side of E4 is

We can obtain an equation that we label as G2 equation
by moving the link transformation matrix 5 A 6 to the lefthand side of Eq.(2).

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

382

(8)

Kinematic modeling and Simulation

C1C2C3 − S1S3 C1S3 + S1C2 S3 S2C3
 −C S− S1S2C2
1 2
ER 4 =  S1C3 + C1C2 S3 S1C2 S3 − C1C3 S2 S3

0
0
0


0
lL 2 
0
1

obtain the angles of each joint. Then we output theseangles to the various joints of the arm. The joint angle
input model for trajectory planning is shown in Figure 3.

(9)

By comparing the element (2,3) of EL 4 and ER 4 ,we get
C2 and then S 2 , and the joint solution θ 2 from them,C2 =a 23 =
az' S4 S5 − a 'y ( C4C6 + S4C5 S6 ) − ax' ( C4 S6 + S4C5C6 )(=θ 2 a tan 2 ± 1 − C22 , C2

)

(10)

By comparing the elements [(1,3) (3,3)] and [(2,1) (2,2)],
we can find the joint solution θ3 and θ1 ,Fig. 3. The joint angle input model for trajectory.

3.3. Establishment of robot ontology module=
a13 ax' ( C4 C5 C6 + S 4 S6 ) + C4 S5 az' + ( S 4 C6 − C4 C5 S6 ) a y'

Figure 4 shows the robot ontology module. It includes
world coordinate frame and robot ontology. Each arm has
six degrees of freedom, including three degrees of
freedom for shoulder joint, one degree of freedom forelbow joint and two degrees of freedom for wrist joint.

a33= S5 C6 ax' − C5 az' − S5 S6 a y'

θ3 = a tan 2 ( a33, a13)

( S4 C5C6 − C4 S6 ) nx' + S4 S5 nz' − ( S4 C5 S6 + C4 C6 ) ny'
a 22 = ( S 4 C5 C6 − C4 S6 ) sx' + S 4 S5 sz' − ( S 4 C5 S6 + C4 C6 ) s y'θ=a tan 2 ( −a 22, −a 21)
1
a 21 =(11)

The solution of the six joints is obtained by the above
procedure. The left arm is exactly the same as the rightarm for the solution.
3. Dual-arm robot modeling based on Simscape
3.1. Model building

Fig. 4. The robot ontology module.

In this paper, the dual-arm robot simulation model
mainly includes three parts, namely the trajectory
planning joint Angle input module, the robot bodymodule and the output measurement module. Its
Simscape Multibody model is shown in Figure 2.

3.4. Establishment of output measurement module
The output measurement module shown in Figure 5 canmeasure the rotation Angle of each degree of freedom.
Through this module, we can also obtain thedisplacement of the terminal coordinate frame of themanipulator relative to the base coordinate frame.

Fig. 2. Dual-arm robot simulation model

3.2. Establishment of joint Angle input module fortrajectory planning

Fig. 5. The output measurement module.

We plan the trajectory of space in Cartesian coordinates.
After solving the inverse kinematics function, we can© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

383

Jiwu Wang, Junxiang Xu

4. Simulation verification and analysis

Acknowledgements

4.1. Simulation processThis work was supported by the Key Research andDevelopment Program (No.M18GY300021).

The robot first raised their arms and then approachedeach other. The entire process is shown in Figure 6.

(a)t=5s(b)t=10s(c)t=15s(d)t=20s

References
1. Ian S. Fischer. A geometric method for determining joint
rotations in the inverse kinematics of robotic manipulators.
2000, 17(2):107-117.
2. Samer Tahya, M. Moghavvemi, Haider A.F.Mohamed.
Geometrical approach of planar hyper-redundant
manipulators: Inverse kinematics, path planning andworkspace. Simulation Modelling Practice and Theory.
2010(1).
3. Buss S R. Introduction to inverse kinematics with Jacobiantranspose, pseudoinverse and damped least squares
methods, J. IEEE Transactions on Robotics andAutomation, 2004.
4. Momani S,Abo-Hammour Z S,Alsmadi O M K. Solutionof inverse kinematics problem using genetic algorithms.
Applied Mathematics&Informaion Scienced . 2016
5. Mostafa Bayati. Using cuckoo optimization algorithm andimperialist competitive algorithm to solve inverse
kinematics problem for numerical control of robotic
manipulators. 2015, 229(5):375-387.
6. Ali MA, Park HA, Lee CSG. Closed-form inverse
kinematic joint solution for humanoid robots. In:
IEEE/RSJ 2010 international conference intelligent
robotics system IROS 2010-coference proceeding.;2010.
pp. 704-9

Fig. 6. Simulation diagram of each moment.

4.2. The motion angles of each joint
Figure 7 shows the motion angles of each joint of themanipulator. At the initial moment, the Angle of eachjoint is 0 radian, and at about 10 seconds, the robotic arms
begin to approach each other. The Angle of each joint is
continuous in the whole process, which proves that theinverse kinematics solution algorithm in this paper cansolve the analytic solution effectively.
The joint angles of the left arm

2

Authors Introduction

The joint angles of the right arm

2

1.5

1.5

joint2

joint2
joint3radjoint4

0.5

1

）

）

joint4（

joint5
joint6

joint5angle

radjoint1

joint3angle（

Dr. Jiwu Wang

joint11

0.5

joint6

0

0

-0.5

-1

He is an associate professor, Beijing Jiaotong
University. His research interests are Intelligent
Robot, Machine Vision, and Image Processing.

-0.5
0

2

4

6

8

10

time

12

14

16

（ s）

(a) Left arm’s angles

18

20

0

2

4

6

8

10

time

12

14

16

18

20

（ s）

(b) Right arm’s angles

Fig. 7. The motion angles of each joint of the left and rightarms.

Mr. Junxiang xu
He is a postgraduate in Beijing Jiaotong University.
His research interest is robot modeling andsimulation.

5. Conclusion
In this paper, the kinematics modeling of dual-arm robot
is carried out. We present an analytical solution method
for 6 DOF dual-arm robot for each arm. Simulation
results show that the proposed method is effective. In the
future, it will be applied to the arm trajectory planning of
the robot.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

384

Kinematics analysis and simulation of 6R Robot
Based on Matlab/Simulink
Jiwu Wang

School of Mechanical and Electronic Engineering, Beijing jiaotong University
Beijing, Haidian District, China
Shuo Han

School of Mechanical and Electronic Engineering, Beijing jiaotong University
Beijing, Haidian District, China
E-mail: jwwang@bjtu.edu.cn, 19140566@bjtu.edu.cn
www.bjtu.edu.cn

Abstract
The 6R robot can imitate the human arm to complete some target grabbing tasks, so the kinematics analysis of therobot is significant in scientific research and practical application. In this paper, a kinematics solution method of 6R
robot based on analytic method is introduced, which is faster and more accurate in solution than the numerical method.
Then the trajectory of the end effector is planned by using the quintic polynomial method,in this way, there are no
sudden changes in the speed of the end effector of the robot, and the operation is more stable.Furthermore, theaccuracy of the kinematics solution method is verified and the motion trajectory of the manipulator is simulated byMatlab. At last ,the visualization of the robot kinematics model was realized based on the Simulink, and thekinematics simulation control system was established.
Keywords: 6-DOF robot ; forward kinematics ; inverse kinematics ; simulation.

completed,so that the angle of each joint can be found out
when the robot grabs the target. Then the motion
trajectory of the robot was simulated in the joint space,
and the smooth curves of angular displacement,angularvelocity were obtained.
Furthermore, the rationality of the D-H parametersand Inverse kinematics results of the robot were verified
by MATLAB, and the visualization of the robot
kinematics model was realized based on the Simulink,
and the kinematics simulation control system wasestablished.

1. Introduction
The industrial robot has played an important role in
improving working conditions and production efficiency,
and some algorithms and programs can control the robot
to complete some specified tasks in a certain way.So
that,the 6R robot is widely used in machining, electronic
welding, industrial handling and other industries becauseof its flexibility and maneuver ability.
The basic characteristic of the 6R robot is the
kinematics characteristic, so the kinematics analysis of
the robot is significant in scientific research and practicalapplication. In this paper,the mathematical model of the
6R robot is established by D-H algorithm, and theforward and inverse kinematics calculation is

2. The Kinematics Analysis of the 6R Robot© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

385

Jiwu Wang, Shuo Han

2.1. Establish D-H coordinate system

between adjacent coordinate systems are represented by
4×4 homogeneous transformation matrix as follows[1]:In this paper,the ABB-irb120 robot that shown in figure
1 is used as an example of 6R robot in our discussion.

I-1

cosθi
Ti = sinθi-sinθicosαi
cosθicosαi

00sinθisinαi aicosθi-cosθisinαi aisinθi

sinαi
0

cosαi
0

di
1

(1)

Ti represent the transformation matrix from ‘the i-1
coordinate system to the i coordinate system. Throughsubstituting the D-H parameters in Table 1 into formula(1), the homogeneous transformation matrix between
each connecting rod can be obtained as follows:
i-1

Fig. 1. The model of the 6R robot0

We used the Denavit-Hartenberg (D-H) matrix
representation for each link to describe the rotational andtranslational relationship between adjacent links. The
coordinate system of each connected link is established ,as shown in figure 2,and the D-H parameters wereobtained, as shown in the table 1.
The coordinate system {0} in figure 2 is a definedreference coordinate system, which is fixed on the baseand usually coincides with the origin of the coordinate
system {1}.

24cosθ1 0 -sinθ1
T1 = sinθ1 0 cosθ1
0 -1 0
0
0 0

0
0
d1
1

cosθ2 -sinθ2 0 a2cosθ2
T2 = sinθ2 cosθ2 0 a2sinθ2
0
0 1
0
0
0 0
1

1

cosθ4
cosθ3 0 -sinθ3 a3cosθ3
T3 = sinθ3 0 cosθ3 a3sinθ3 3T4 = sinθ4
0 -1 0 0
0
0
0 0 1
0
cosθ5 0 -sinθ5 0
T5 = sinθ5 0 cosθ5 0
0 -1 0000 0
10 sinθ30 -cosθ3
1
0
0
0

0
0
d4
1

cosθ6 -sinθ6 0 0
T6 = sinθ6 cosθ6 0 0
0
0 1 0
0
0 0 1

5

So that,we can get the the spatial displacement andposture transformation of the 6th coordinate frame
relative to the base/reference coordinate frame{0}:0

If the parameters of each joint and link of the robot andthe relative position of the end effector relative to the
fixed reference coordinate system are known, how to get
the angleθi between the connected link of the robot ,which is the inverse kinematics analysis solution.

Tab.1. D-H Parameter Table

θi
θ1
θ2
θ3
θ4
θ5
θ6

αi
-90°
0
-90°
90°
-90°
0

ai
0
270
70
0
0
0

(2)

2.3. Inverse Kinematics Solution

Fig. 2. Coordinate system of each link of the robot

Joint i
1
2
3
4
5
6

T6 = 0T1 1T2 2T3 3T4 4T5 5T6

di
290
0
0
168
0
0

Solution of θ1, According to the analysis of forward
kinematics，we can multiply 0T1-1 form the left on bothside of the Eq.（2）:02.2. Forward Kinematics Solution

T1-1 0T6= 1T2 2T3 3T4 4T5 5T6(3)

By comparing the elements (3,4)of the Eq.(3),we canknow that :The purpose of the forward kinematics is to find out theposition and posture of the end effector that relative tothe coordinate system {0}. The position and posture-py c1+pxs1=0© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

386

(4)

Kinematics analysis and simulation

s5=-ax(s1s4+c1c4c23)-ay(c1s4-s1c4c23)-azc4s23

where si ≡ sinθi, ci ≡ cosθi, sij ≡ sin(θi+θj), cij≡cos(θi+θj ) ,so that :θ1=atan2(py-d6ay,px-d6ax)

c5=-axc1s23-ays1s23-azc23(5)θ5=atan2(s5,c5)

Solution of θ2. We can multiply 5T6-1 form the left on
both side of the Eq.（2）:5

T6-1 0T6= 0T1 1T2 2T3 3T4 4T5(12)Solution of θ6. It is similar to the solution of θ5,we justneed to multiply 0T5-1 form the left on both side of the Eq.
（2）

(6)

By comparing the elements (1,4),(2,4),(3,4) of theEq.(6),we can know that :0

T5-1 0T6= 5T6(13)p5x=px-d6ax=c1(a2c2+a3c23-s23d4)

By comparing the elements of the Eq.(13),we can knowthat :

p5y=py-d6ay=s1(a2c2+a3c23-s23d4)

s6=-nx(s1c4+c1s4c23)-ny(c1c4+s1s4c23)+nzs4s23

p5z=pz-d6az=d1-a2s2-a3s23-d4c23

c6=ox(c4s1-c1s4c23)-oy(c1c4+s1s4c23)+ozs4s23

p5x2+p5y2, L=pz-d6*az-d1,
In there ,we let K=
2
2
2
2
2
M=(L +K +a2 -a3 -d4 )/2a2,
accordingtothetrigonometric theorem, θ2 can be obtained that[2]:θ6=atan2(s6,c6)

θ2=atan2(K,L)-atan2(-M,(K2+L2-M2)0.5)

2.4. Trajectory planning of the 6R robot
In this paper, we use the method of quintic polynomialinterpolation to plan the trajectory of the 6R robot.
θ(t)=a0+a1t+a2t2+a3t3+a4t4+a5t5
So that,the function expression of the velocity is:
θ(t)’=a1+2a2t+3a3t2+4a4t3+5a5t4
And the function expression of the acceleration is:
θ(t)’’=2a2+6a3t+12a4t2+20a5t3
By making constraints about the position, angularvelocity and angular acceleration of each joint in space atthe starting and ending positions of the path point:(7)

Solution of θ3. We can know that:
K-a2s2=a3s23+d4c23;
L-a2c2=a3c23+d4s23;
according to the trigonometric theorem, θ23 can beobtained that:θ23=atan2(a3,d4)-atan2(E,(a32+d42-E2)0.5)
E=K-a2*cosθ2
θ3=θ23 -θ2

(14)

(8)

Solution of θ4. Multiply T3 form the left on both sideof the Eq.（2）:0-1

T3-1 0T6= 0T1 1T2 2T30

So this is obtained from the above formula:

W=-s4s5=axs1-ayc1
Z=-c4s5=c1c23ax-s23az+s1c23ay
So that,we can get the angle θ4:θ4=atan2(W,Z);(10)

Solution of θ5. Multiply 0T4-1 form the left on both side of
the Eq.（2）0T4-1 0T6 = 4T5 5T6

By substituting the above coefficients into the basic
expression, the function expression using quintic
polynomial interpolation method can be obtained.

(11)

By comparing the elements of the Eq.(11),we canknow that :© The 2020 International Conference on Artificial Life and Robotics (ICAROB2020), Jan. 9-12, B-Con Plaza, Beppu, Oita, Japan

387

Jiwu Wang, Shuo Han

3. The Kinematics Simulation Verification basedon MATLAB
We use the Matlab Robotics Toolbox to verify theaccuracy of kinematics solution. By given a joint anglegroup ‘q=[pi/3, pi/4, 3*pi/4, -pi/5, pi/5 ,pi/6]’, we use
the forward and inverse kinematics algorithm finishedin chapter 2.2 and 2.3 to get the pose and posture matrixof end effector, and the solution of each joint angle in
Matlab , as shown in figure 3.
Then by comparing with the result that obtained byusing the ‘Fkine’ and ‘Ikine’ function of Matlab,wefound that the two results are consistent,so that thekinematics solution is verified.

Fig. 6. Kinematics visual simulation by Simulink

5. Conclusion
In this paper,we take the IRB120 robot as the research
object, using the D-H method to establish the connecting
rod coordinate system, finish the solution of the forward
and inverse kinematics analysis, get the 6R robot
kinematics algorithm. Then the Matlab is used to modeland simulate the robot. It can be seen from the simulation
results that the model of the robot established in thisstudy is correct, and the trajectory planning of the end
effector is carried out, and the change of the angle of eachjoint with time is recorded, which verified theeffectiveness of the kinematics algorithm.

Fig. 3. Kinematics solution result finished by Matlab

The trajectory planning of the robot is also simulatedby Matlab ,the curve of the velocity and acceleration ofeach joint angle with time ,shown in the figure 4.

Acknowledgements
This work was supported by the Key Research andDevelopment Program (No.M18GY300021).
References
1. Guan Wei, “Research on obstacle avoidance pathand trajectory planning foe six-dof manipulator,” D.
Shanghai Jiao Tong University, Jan.2016.
2. Hu Yaqing, Yu Jinpeng, Kinematics Analysis andModeling Simulation Based on ABB-IRB120
Robot.J.Journal of Qingdao University(E&T) ,Aug.2017,Vol.32.

Fig. 4. Trajectory path and joint motion characteristic curve

4. The Visual simulation based on Simulink
The robot is modeled by SolidWorks 2016. First, themodels of the robot is built and assembled. Then theassembly model is converted into the format of .urdf
and imported into the Simulink. Then by using themodular simulation interface of the Simulink andreferring with the previous solving function, the robotic
visual simulation framework is finished,as shown infigure 5 and figure 6.

Fig. 5. The model of robot imported in Simulink© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

388

Distance measurement system based on binocular stereo vision
Jiwu Wang

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
Xin Pei

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
E-mail: jwwang@bjtu.edu.cn, 19125998@bjtu.edu.cn
www.bjtu.edu.cn

Abstract
The distance measurement system based on binocular stereo vision is proposed. The binocular vision model isestablished，Matlab toolbox is used to calibrate and obtain the internal and external parameters of the binocularcamera. Then stereo rectification and stereo matching are carried out with OpenCV. Finally, the depth information is
obtained by the parallax measurement of image pairs, and the distance between any two points can be calculated. The
simulation experiments have been done and the results demonstrated that the measuring method is feasible.
Keywords: stereo vision;camera calibration; stereo matching; distance measurement.

The designed binocular stereo vision system consists offive modules: image acquisition, camera calibration,
image rectification, stereo matching, and distancemeasurement. This system can be described by the flow
diagram shown in Figure 1.

1. Introduction
Distance measurement is one of the most important tasks
of a computer vision system, and it is also continues tobe a popular research topic in computer and robot vision
nowadays. Distance measurement plays an importantrole in robot navigation, obstacle detection, navigation ofautonomous vehicles, surveillance monitoring, person
localization and tracking, and many more. The distancemeasurement system based on binocular stereo vision is
proposed in this paper. Firstly, the binocular vision modelis established, a GUI interface is designed in MATLAB
for calibration and the display of parameters. Then stereo
rectification and stereo matching are carried out with
OpenCV. Finally, the depth information is obtained by
the parallax measurement of image pairs, and the
distance between any two points can be calculated. The
simulation experiments have been done and the results
demonstrated that the measurement method is feasible.

Fig. 1. System flow chart

2. Principle of binocular stereo vision ranging© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

389

Jiwu Wang, Xin Pei

Binocular stereo vision distance measurement is modeledon the distance perception methods of human eyes. In thispaper, the stereo vision system consists of two cameras
with the same parameters that is parallel mounted andlooking at the same objects, as shown in Figure 2. The
projection model is created based on the imaging
principle of pinhole camera. The object points P are given
in a world coordinate system. Pl and Pr are imaging points
on left and right images. Assuming that the image planeof the left and right cameras is located on the same plane
and the two images are aligned, it can be known from theprinciple of similar triangle that:B − ( xl − xr ) B=
Z−fZ

are introduced. The relationship between the points
TT
P = [ X Y ] in the imaging plane and P = [ X Y Z ] in thespace can be expressed as: P = sHP
Where H is represented by two matrices: H = MW fx
M =  0 0fB
xl − xr0cx c y 
1 

M is the intrinsic parameter matrix of the camera.Where,cx and cy represent coordinates in pixel of the image
center respectively. Since pixels on a normal camera are
not guaranteed to be square, use two different focal
lengths fx and fy.
𝑊𝑊 = [𝑅𝑅 𝑇𝑇],where, R , T are the extrinsic parameters of
the cameras, denoting the rotation matrix and thetranslation vector from the left camera coordinate system
to the right camera coordinate system, respectively.
Considering the distortion of lens, suppose 𝑞𝑞𝑝𝑝 ( 𝑥𝑥𝑝𝑝 , 𝑦𝑦𝑝𝑝 ) is
the corrected point, and 𝑞𝑞𝑝𝑝 ( 𝑥𝑥𝑝𝑝 , 𝑦𝑦𝑝𝑝 ) is the distorted point,
then:

(1)

Therefore, it can be derived from equation (1):
Z=

0
fy

(2)

Where, B is the base distance between the two cameras,
and f is the focal length of the camera lens. Again, xl andxr respectively represent the pixel distance of Pl and Pr
in the X-axis direction in their respective coordinate
system. The difference in the coordinates of thecorresponding pixels is known as disparity d. Z is thedepth distance of the object point from the camera
position.

 2 p1 xd yd + p2 ( r 2 + 2 xd2 ) x xp 
2
4
6  d 

1
k
r
k
r
k
r
=
+
+
+
+
(
)
y 
1
2
3
y  
 d   p1 ( r 2 + 2 yd2 ) + 2 p2 xd yd 
 p

(3)

Where (K1, K2, P1, P2, K3) constitutes a 5 × 1 matrix,
which is the distortion matrix of the camera.
The method of camera calibration used in this paper isbased on the checkerboard template calibration method
proposed by Zhang, which has the advantages of simple
operation and high precision. In this paper, a GUI
interface is designed in MATLAB for calibration anddisplay of results. The specific steps are as follows:(i) Self-made calibration plate; The number of c corners
is 7×5 and the size of the calibration target is

20mm×20mm(ii) Image acquisition; Images from different angles of
the calibration target are collected simultaneouslywith a binocular camera placed in parallel, and theimage resolution is 640×480. A total of 13 sets ofimage pairs were collected, as shown in Figure 3 andFigure 4.

Fig. 2. Binocular vision model

3. Camera calibration
The process of calibration is actually the process of solving the internal and external parameters of the camera.
Calibration is the basic and necessary process ofbinocular stereo vision. The result of calibration will
determine the accuracy of distance measurement of thetarget object. In the process of camera calibration, threecoordinate systems are needed: image coordinate system,
camera coordinate system and world coordinate system.
The camera is approximated to a pinhole model, and theparameter S (scale factor) and the homography matrix H© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

390

Distance measurement system based

4.

Stereo rectification

The purposes of image rectification are as follows: thedistortions of images are removed by the internalparameters in the calibration results, and the two imagesaligned in non-coplanar lines are rectified to coplanar linealignment such that the polar lines of the two images lie
exactly on the same horizontal line. Any point on such animage must have the same line number as its
corresponding point on another image. Therefore, in caseof searching for corresponding points in two images, it is
only necessary to search in the same epipolar line,
reducing a 2D search space to 1D to reduce the searchtime effectively.
The stereo rectification can be realized by the function of
cvStereoRectify in OpenCV. Figure 6 shows the left andright image pairs before stereo rectification. Figure 7
shows the image pairs after stereo rectification.

Fig. 3. Left images of calibration target images

Fig. 4. Right images of calibration target images(iii) Calibration. Click the calibration button on the GUI
interface to import the binocular image pair collected.
Then, corner detection and stereo calibration are
carried out to obtain the internal and externalparameters of the binocular camera and display themon the GUI interface, as shown in Figure 5.

Fig. 6. image pairs before stereo rectification

Fig. 7. image pairs after stereo rectification

5. Stereo matching
The stereo matching algorithm is an important step ingenerating disparity maps. Whether stereo matching isaccurate directly affects the accuracy of binocularranging results. The basic principle of stereo matching is
to find the corresponding relationship between the pixelsof the same scene in the projected image under different
viewpoints, and then to obtain the distance informationof object by calculating the parallax between the pixels.

Fig. 5. GUI interface© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

391

Jiwu Wang, Xin Pei

References

In this paper, CV Find Stereo Correspondence BM, a
block-matching Stereo Correspondence algorithm
provided by Open CV, is used to find the matching points
between the left and right images after stereo rectification
through a small window (SAD) called " Sum of Absolute
Difference ".
Through matching the left and right rectified image pairs
in OpenCV, the disparity map can be obtained. In the
disparity map, any two target points can be selected withthe mouse to obtain their depth information, and the
distance between the two points can be calculated, as
shown in Figure 8.

1. Gary Bradski, Adrian Kaebler. Learning OpenCV[M].
Translated by Yu Shiqi and Liu Ruizhen. Beijing:
Tsinghua University Press, 2009.
2. Hu Hanping. Research on key Technologies of Binocularstereoscopic ranging [D]. Graduate University of The
Chinese Academy of Sciences (Changchun Institute of
Optics, Fine Mechanics and Physics),2014.
3. Zhao Xunjie, Li Chengjin. Research on key Technologies
of binocular Stereo Real-time ranging System [J]. Laser &Infrared,2006(09):874-877.
4. Wang Zhenzhen, Zhao Lianyu, Liu Zhenzhong. Binocularstereo ranging system based on MATLAB and OpenCV[J].
Journalof
tianjinuniversityoftechnology,2013,29(01):45-48.
5. Shen Tong, Liu Wenbo, WANG Jing. Target ranging
system based on binocular stereo vision [J]. Electronic
measurement technology,2015,38(04):52-54.
6. Wang Hao, Xu Zhiwen, Xie Kun, Li Jie, Song Chenglu.
Binocular ranging system based on OpenCV [J]. Journalof jilin university (information science),2014,32(02):188194.

Authors Introduction
Fig. 5. Disparity map
The actual distance between the corners of the calibration
plate is 20mm. From the results of experiments, it can beseen that the measurement method basically meets therequirement of precision, the measuring method is
feasible.

Dr. Jiwu Wang

6. Conclusion

He is an associate professor, Beijing Jiaotong
University. His research interests are Intelligent
Robot, Machine Vision, and Image Processing.

Ms. Pei Xin

A procedure for distance measurement using a binocularstereo vision system was presented in this article. The
experimental results demonstrated that the system could
meet basically the needs of measuring. The measurement
method is feasible for applications where accurate
distance measurement is not required and could offer
better technical support for machine vision applications.
However, due to the influence of matching and otherfactors, the accuracy should be further improved in the
future research work.

She is a postgraduate in Beijing Jiaotong University.

Her research interest is stereo vision and image
processing.

Acknowledgements
This work was supported by the Key Research andDevelopment Program (No.M18GY300021).

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

392

Robustness Verification Against Noise of Self-localization Method
Using Omni-directional Camera for Soccer Robot
Yuehang Ma, Kaori Watanabe, Hidekazu Suzuki

Graduate School of Engineering, Tokyo Polytechnic University, 1583 Iiyama
Atsugi, Kanagawa 243-0297, Japan
E-mail: xmaancn@gmail.com, watanabe.kougei.karakuri@gmail.com, hsuzuki@em.t-kougei.ac.jp
www.t-kougei.ac.jp

Abstract
The main focus of the RoboCup competitions is the game of football/soccer, where the research goals concern
cooperative multi-robot and multi-agent systems in dynamic adversarial environments1. In the field of RoboCup, selflocalization technique is important to estimate own position including goal and other robot positions and to decidestrategy. This paper presents a self-localization technique using an omni-directional camera for an autonomous soccer
robot. We propose the self-localization method with white line information of soccer field, and recognize the robot
position by optimizing the fitness function using Genetic Algorithm. Moreover, we also verify the robustness of the
proposed method against noise through experiments.
Keywords: Robustness Against Noise, Self-Localization, RoboCup Middle Size League, Soccer Robot, Genetic
Algorithm

1. Introduction
In the field of RoboCup, self-localization technique isimportant to estimate own position including goal andother robot positions and to decide strategy. Basically, we
estimate the self-position with the image information, theenvironment information and the field information. In
this paper, we describe a real-time self-localization
method that applies a genetic algorithm (GA) for the
RoboCup middle size league, and verify the robustness
detection of this method.

Fig. 1. Hardware of vision system

2. Hardware of vision system

2(a), and the image size and frame rate are 512 × 512[pixels] and 30 [fps] respectively.

About the omni-directional vision system of our robot is
consisted of the camera (FLIR, Flea32), a varifocal lens(Vstone) and a hyperboloidal mirror (Vstone). We
developed vision system shown in Fig. 1 for RoboCup
MSL robot by combining with above elements3. The
image captured by this vision system is shown in Fig.

3. Self-localization
We use a white line of MSL field for self-localization.
We have proposed the self-localization method, which
generates the searching space based on a model based© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

393

Yuehang Ma, Kaori Watanabe, Hidekazu Suzuki

matching using white line information4. And this method
recognizes the robot position by optimizing the fitness
function, which has the maximum value at correct robot
position. Moreover, this proposed self-localization
method employs Genetic Algorithm (GA)5 for
optimization of the fitness function.
3.1. Searching model
Figure 2 shows the process of making the searchingmodel of the proposed method. At first, we need thedetection image of the white line for making thesearching model. We obtain the white detection image by
employing the converting method of color space from
RGB to HSV and to YUV like Fig. 2b. Then we generate
the field information by orthogonalizing the white line
information like Fig. 2c. Moreover, we determine thesearching model by thinning down the field information
based on white line like Fig. 3d. Therefore, we use
thinned model as searching model for the selflocalization.

Fig. 2. Process of making search model

3.2. Model-based matching
The proposed self-localization method generates thesearching space by model based matching between
geometric information of the white line in the MSL field
and above-mentioned searching model. We use thismethod to calculate the evaluation function 𝐹𝐹�𝜙𝜙��, 𝜙𝜙� =�𝑥𝑥�, 𝑦𝑦�, 𝜃𝜃�� . The fitness function 𝐹𝐹�𝜙𝜙�� obtains the
maximum value when the position of the searchingmodel corresponds to the correct position that robot exist
in the MSL field. Then, the problem of detection of robot
position is converted to the searching problem of 𝜙𝜙� suchthat 𝐹𝐹�𝜙𝜙�� is maximized. Due to the revolution symmetryof the white line of the MSL field, we may get two
maximum value exist in the fitness function. Here, we
select only one depending on an electric compass.
3.3. Genetic Algorithm
In the proposed self-localization method, we employ
Genetic Algorithm (GA) for searching the maximum
value of the fitness function 𝐹𝐹�𝜙𝜙��. A GA is an example
of an artificial intelligence program and is well known asa parallel search and optimization process that mimics
natural selection and evolution. In the proposed method,an elitist model of a GA that preserves the best individualin the population at every generation is utilized andFig. 3. Error of the self-localization
genetic coding using gray code, roulette selection andone-point crossover are employed. And, the parametersof the GA process are determined by previous
experiments.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

394

Robustness Verification Against Noise

3.4. Verification experiment
We performed the self-localization experiment to verify
the effectiveness of the proposed method. Figure 3 shows
the result of the verification experiment that checked theself-localization error between correct position anddetected position at the quarter area of the MSL field atinterval of one meter. In this figure, each box represents
the error as the brightness of gray scale. Average error ofthis experiment was 12.7[cm], and the accuracy of theself-localization by the proposed method is enough toplay soccer.

Fig.5. Experiment location

4. Robustness verification
The accuracy measured so far is the result of an ideal
environment when there are no other robots on the soccer
field. However, in a real game environment, up to ten
robots may exist in the same soccer field. In addition to
the robots, there are humans such as the referee and theline referee at the same time on the field. Then the robot
may not be able to recognize correct own position,
because these become an occlusions. Therefore, to verify
the robustness of proposed method against various noise,
we conducted experiments.
4.1. Experiment
In a real environment, there will be a variety of noises
appearing on the image, and we cannot perform
qualitative experiments. Here we perform verification
experiments by adding artificial noise.
We conducted the experiments at the seven locationsin the MSL field indicated in Fig. 5. These sevenlocations are especially characteristics point in MSL field.
Taking the center of the panoramic image as the center ofthe circle, we set the fan-shaped area with a center angleof 30 degrees as a noise like Fig. 6. By changing theposition and number of the noise, we can simulatedifferent noise conditions, and verify the results of selflocalization.
Figure 6 shows the change in the number of noise at
the position Fig. 5(d), and the change in angle is shownin Fig. 7. The image in the top left of Fig. 6 is original
image without any noise. We verify the error of selflocalization results measured at the angle and number ofvarious noise at each coordinate location in the MSL field.
And the results of each measurement position are

Fig.6. Noise at different amounts

Fig.7. Noise at different angles© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

395

Yuehang Ma, Kaori Watanabe, Hidekazu Suzuki

Table 1. Error comparison resultsummarized in Table 1. For the position deviation causedby the noise of different angles, we take the average value
combined with the deviation value caused by the amountof noise. The red grid in the table indicates that the error
value of the self-localization exceeds the width of a robot(50cm). It means that the robot has lost its accurate
position.

5. Conclusion
In this paper, we have proposed the self-localization
method that generates the searching space based on amodel based matching with white line information of
RoboCup MSL soccer field, and which recognizes therobot position by optimizing the fitness function using
Genetic Algorithm. Moreover, we have verified theeffectiveness and accuracy of the proposed selflocalization method using GA. Furthermore, we
confirmed that the robustness against noise of the selflocalization by the proposed method is enough to playsoccer through experiments.

4.2. Experiment results
According to the results in Table 1, when the number ofnoises increases, the error of self-localization also
increase. Overall, when the number of noises is greaterthan 7, it will cause large errors except for special
locations. Here, the special locations is when there are
more than two features in the image obtained by the robot(Location B and F) or when the robot is in the centercircle (Location C). Location E and G also have errors
when the amount of noise is not large. The reason may
be that the robot is on the white line crossed vertically,
and the 4 noises hide all the white line information.
Overall, from the results in Table 1, when the number ofnoises is less than 3, the error of self- localization is verysmall. And when the number of noises increases to 4, themaximum error of self- localization is 120cm. In a realgame environment, the probability of more than 4 noises
existing at the same time is very small. Therefore, thismethod has robustness against noise enough to playsoccer.

References
1. “RoboCup
Federation
official
website,”,https://www.robocup.org/
2. FLIR,“Flea3(USB3
VisionCamera)”,https://www.ptgrey.com/flea3-usb3-vision-cameras
3. “RV-Infinity
Discription
paper”,
http://www.seit.t-kougei.ac.jp/vision/rv-infinity/
4. K. Watanabe, Y. Ma and H. Suzuki, Real-time Selflocalization using Model-based Matching for Autonomous
Robot of RoboCup MSL, Journal of Robotics, Networking
and Artificial Life, Vol. 7(2); September (2020), pp. 98–
102
5. D.E. Goldberg, Genetic algorithms in search, optimizationand machine learning, Addison-Wesley, Reading, MA,
USA, 1989.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

396

Tomato-Harvesting Robot Competition: Aims and Developed Robot of 6th Competitions
Takayuki Matsuo

National Institute Technology, Kitakyushu College
2-20-1, Shii, Kokuraminami-ku, kitakyushu-shi, Fukuoka, Japan
Yasunori Takemura, Takashi Sonoda

Department of Engineering, Nishinippon Institute of Technology, 2-11, Aratsu, Kanda-town,
Miyako-gun, Fukuoka 800-0396, Japan
Yuya Nishida, Shinsuke Yasukawa, Kazuo Ishii

Department of Life Science and System Engineering, Graduate School of Kyushu Institute of Technology,
2-4, Hibikino, Wakamatsu, Kitakyushu-city, Fukuoka 808-0196, Japan
E-mail: matsuo@kct.ac.jp, takemura@nishitech.ac.jp, sonoda@nishitech.ac.jp,
s-yasukawa@brain.kyutech.ac.jp, y-nishida@brain.kyutech.ac.jp, ishii@brain.kyutech.ac.jp
http://www.lsse.kyutech.ac.jp/~sociorobo/ja/tomato-robot2019

Abstract
In general, farm work is often hard work. In recent years, the number of farm laborers has been decreasing due to theaging of the workforce. Therefore, there is a need to automate agricultural work, and we are organizing a tomato
harvesting robot competition to automate tomato harvesting. The competition includes a Junior Division for highschool students and a Senior Division for college students. In this paper, we report on the results of 6th tomato
harvesting robot competition.
Keywords: robots for socio synthesis, robot competition and agriculture robot

agricultural work. Among them, we have focused on
tomatoes and have been holding tomato harvesting robot
competitions since 2014. As the research of tomato
harvesting robots in Japan, Kawamura et. al. developed amobile robot with manipulator [2] and proposed thetomato harvesting method using image processing andvisual feedback [3]. Kondo et. al. proposed the method toimprove the success ratio of tomato harvesting andspeed-up technique [4].
In this paper, we will report on the result of the 6th
tomato harvesting robot competition.

1. Introduction
In recent years, the decline and aging of the agricultural
workforce has become an issue. According to data[1]released by Japan's Ministry of Agriculture, Forestry and
Fisheries, the number of agricultural workers in 2020 was
1,361,000, a decrease of 22.5% or 396,000 compared tothe previous survey in 2015. In addition, the percentageof people aged 65 and above rose to 69.8%, up 4.9% fromthe previous survey. This is due to the fact that a large
number of people quit farming due to old age, andsecuring the number of farmers is an issue. In recent years,
in order to cope with the decreasing number ofagricultural workers, there has been a lot of research anddevelopment of agricultural robots aimed at automating

2. Competition regulations© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

397

Takayuki Matsuo, Yasunori Takemura, Takashi Sonoda, Yuya Nishida, Shinsuke Yasukawa, Kazuo Ishii

2.1. Senior League
In Senior League, two kind of competition field aredesigned, the one is rail-style area and the another one isfee-style area as shown in Fig.1 and Fig.2. The rail-stylearea is designed to have the similar environment with thetomato factory. Free-style area is for the robots of general
tomato fields in outdoor environment. In first stage, atomato is suspended and the team is able to advance tothe second stage when an end-effecter of the robot is able
to touch the tomato. In second stage, a cluster of tomato
is suspended. Teams compete against each other to see
how many tomatoes they can harvest. The score is
calculated using Eq. 1 below.
𝑷𝑷 = 𝜼𝜼𝜼𝜼(𝟐𝟐𝟐𝟐 + 𝜷𝜷) − 𝟐𝟐(𝜸𝜸 + 𝜹𝜹) − 𝜺𝜺
𝜼𝜼 =

𝜶𝜶

𝜶𝜶+𝜷𝜷+𝜸𝜸+𝜹𝜹Fig. 3 A slope set up in the freestyle area.
Table 1 The class number and coefficient at choosing
remote control
Method of View
Area
Number of Category
Coefficient C

(1)
(2)

Indirectly
Rail
Free
T3
T4
2
4

Table 2 The class number and coefficient at choosing
autonomous control, where P is score, C is coefficient of magnification
depending on selecting class as shown in Table1, 2, 𝛼𝛼isthe number of tomatoes which is no damaged and correctcolor, 𝛽𝛽 is the number of damaged tomatoes, 𝛾𝛾 is thenumber of drop tomatoes and 𝛾𝛾 is the number ofdamaged tomatoes which are not harvested. 𝜀𝜀 is adeducted point when robots damage stalks of tomato
plants and the point is deducted in Final Stage. Eq. 2
represents the harvest rate 𝜂𝜂. If multiple tomatoes are
harvested but dropped or damaged, the final score will be
less. In final stage, the robots harvest tomato from plant
body. In addition, some farms do not actually have rails,
in which case the road surface in the vicinity of thetomato plants has a slope. Starting from the 5thcompetition, we have installed a slope in the freestylearea to try to replicate the harvesting conditions of an
actual farm.

Fig.1 Free-style area

Directly
Rail
Free
T1
T2
1
2

Area
Number of Category
Coefficient C

Rail
T5
8

Free
T6
16

2.2. Junior League
In Junior League, the subject is to carry small size
tomatoes to assigned positions. Students should develop
the robot using LEGO Mindstorm with the functions suchas line trace, color recognition, end-effector withmechanism design and motors with control, and theirprogramming [5]. The basic specification for robot is thatthe size of the robot is within 300mm x 300mm on ground[5].Height is no limited. Students should develop therobot using LEGO Mindstorm. Competition subjects
include Line Trace Challenge, Color Identify Challenge,
Mechanism Design and Control Challenge and Object
Detection Challenge. In Line Trace Challenge, robots
should detect black line in the competition area and movealong the line using a color sensor. The robot starts fromthe starting point. In the middle of the course, the tomato
harvesting field (harvest field) exists, where tomatoes arearranged. The robot must move to the harvest field inorder to get the 6 tomatoes. Until the 4th Tomato
Harvesting Robot Competition, the tomatoes were stored
in a transparent box called "Tomato Box" beforehand, butfrom the 5th competition, the tomatoes were placed in a
fresh state. Therefore, more careful handling of tomatoes
is needed when the robot manipulates and transports

Fig.2 Rail-style area

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

398

Tomato-Harvesting Robot Competition

them.
In Color Identifying Challenge, robots should explore
and recognize color signs in the middle of the course andthe same color of tomatoes. As guidance to harvest field,
red, yellow and blue lines are drawn in the field as shown
Fig.4. It is necessary for the robots to detect guidance line
by color sensor. The robot gets the tomatoes of the samecolor. Along the black line for line trace, each color issigned. The robot can move to the harvest field from thelone detecting the color. In Mechanism Design andControl Challenge, robot should manipulate tomatoes
using manipulator and carry to the storage location. To
pick up the tomato box by using a manipulator equipment
which is made by each team, participants are expected todesign and make a device to get tomatoes on their idea.
The robot is required to store, transport and relocation
depending on tomatoes in each color. After picking up
the tomatoes, the robot should return to the course. Then,the robot carries the tomatoes to the specific storage
location. In Object Detection Challenge, robots should
detect a battery charging station and stop there.
In addition to the above rules, we have added a new ruleto handle tomatoes with care. Tomatoes are a food.
Tomatoes are a food product, and if they are handled
improperly in this competition, they will not be scoredfor storage points. The procedure for checking tomatoes
in the competition and the criteria for judging whentomatoes are not handled properly are as follows:

Fig.4 Robot Jr. League competition Field
3. Result of 6th tomato harvesting robot
competition
The results of the competition of Senior League areshown in Table 3. The Senior League has revised its rules
since the 5th competition, including changes to thescoring equation and the addition of slopes to thefreestyle area. The equation is shown in Eq. 1, which
introduces the concept of harvest rate, where dropping
tomatoes and scratching tomatoes have a significant
impact on the score. For practical use, the robot needs
to harvest more tomatoes within a certain time frame.
Therefore, it is desirable for the robot to work for theentire 10 minutes of the competition, but many teams quit
the competition halfway through for fear of losing theharvest rate. As for the slope, many people said that it
would be difficult to control the robot because it would
slide down the slope without control.
The results of the competition of Junior League areshown in Table 4. The 6th competition was the secondcompetition after the rules were revised to require theteams to harvest and transport fresh tomatoes. In theprevious year, some teams damaged the tomatoes, but inthis year's competition, there was some ingenuity inhandling the tomatoes with care, and the students wereable to instill an awareness of the importance of handlingraw plants through their crafts.
Overall, the competition scores improved from theprevious year, and one team was able to harvest andtransport all the tomatoes in one week. Therefore, itwould be better to make some rule changes for the nextcompetition based on the competition held this time(since it is a high school competition, we will not change

1. A representative of each team member must checkthe
tomatoesforblemishes
beforetrials.
2. After the trials, all tomatoes must be checked for
scratches by the referee and one team member. If there is
any objection, one team member and the organizing
committee chairman and vice chairman will discuss thematter.
Criteria for improper handling of tomatoes
1. if the liquid inside the tomato has been released to theoutside2. if the tomato is damaged in such a way that the insideofthetomatoisvisible3. tomatoes are clearly dented from their pre-race
condition
4. tomatoes are being transported in contact with the floor
5. if it is determined that the tomatoes have not beenhandled in a manner other than the above criteria
By adding the above rules, the goal was to instill in thecompetitors the awareness of handling tomatoes withcare when carrying the actual product.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

399

Takayuki Matsuo, Yasunori Takemura, Takashi Sonoda, Yuya Nishida, Shinsuke Yasukawa, Kazuo Ishii

4.

the basic base, but will consider some additional
elements).
Finally, some teams commented that it was difficult tomake tomatoes of a consistent size. Lastly, some teams
commented that it was difficult because the size of thetomatoes was not consistent. This may be a result of thefact that the purpose of the competition, which is toencourage ingenuity in the devices and algorithms due tothe inconsistent environment of handling natural objects,
was not well understood. I felt that it was necessary tocreate a mechanism in the regulations and website toappeal the purpose of this competition, that it is not only
about winning and losing as a competition, but also aboutdeveloping the ability to cope with nature

Ranking

In this paper, Tomato robot competition for robot sociosynthesis was introduced. The tomato harvesting robot
competition has two leagues which are Senior League
and Junior League. In Junior League, number of theparticipated teams were 24 teams. In Senior League,
number of participated team 9 team. In this competition,
concept of carefully dealing with tomatoes will be
employed.
Acknowledgements
This works was supported by Kyushu Institute ofTechnology, Center for Socio-Robotics Synthesis, The
University of Kitakyushu, Nippon Bunri University,
Nagasaki Prefecture University, Kyushu Polytechnic
Collage , Nishinippon Institute of Technology(Thissupport was including The Ministry of Education,
Culture, Sports, Science and Technology(MEXT),
Center of Community), National College of Technology,
Kitakyushu College, Hibikinada-Saien, Afrel Co., Ltd.,City of Kitakyushu, The Kitakyushu Chamber ofCommerce and Industry, Institute of Systems,
Information Technologies and Nanotechnologies (ISIT),
Japan Chapter of IEEE Society on Systems, Man, andCybernetics, The Japan Society of MechanicalEngineers(JSME), The Robotics Society of Japan(RSJ),
Kitakyushu Foundation for the Advancement of Industry
Science and Technology (FAIS), IT and Mechatoronics
Chapter and Kyushu Chapter of Japanese Society ofAgricultural Machinery and Food Engineers, The
MathWorks Co. Ltd., and Ministry of Agriculture,
Forestry and Fisheries(MAFF).

Table 3 Result of Senior League
Team

Overall winner and 1st place ofrail-style division

Hibikino-Toms(Kyushu Inst. Of Tech.)Overall runner-up and 2nd placeof rail-style division

Syuga-Lab(Nagasaki Inst.
Applied Science

Of

Table 4 Result of Junior League
RankinTeam
1st Place
Tangokinasu(Fukuoka Joto High School)

2nd Place

Double Lycopin(Fukuoka Joto High School)

3rd Place

NiASience

Ver.2(Nagasaki Sogo
High School)

FuzokuSpecial Award
Best Presentation Award G-Advance

References(Fukuoka Joto Highschool)

Perfect Award

1. 2020 CENSUS of Agriculture and Forestry in Japan,
https://www.maff.go.jp/j/tokei/census/afc/, 2020
2. Kawamura et. al. Study on agricultural robot (1st report),
The journal of JASM, vol. 46 No.3,pp.353-358, 1984 in
Japanese
3. Kawamura et. al., Fruit harvesting robot, The journal of
JASM vol.47 No.2, pp. 237-241, 1985 in Japanese
4. Kondo, et.al., A Machine Vision System for Tomato
Cluster Harvesting Robot, Engineering in Agriculture,
Environment and Food, Vol. 2, No. 2, pp.60-64, 2009
5. The 6th Tomato robot competition junior rule,“http://www.lsse.kyutech.ac.jp/~sociorobo/wpcontent/uploads/2019/07/fb3520b51d93fdb42e034fa83b2
37857.pdf”, 2019, 7

Tangokinasu(Fukuoka Joto High School)

Special Judges Award

Kako Tomato Curry(Kashii
school)Idea Award

Technical

High

Revolutionary(Fukuoka Joto High School)

Challenge Award

Conclutions

SyokugyoMochi(National institute ofTechnology, Kitakyushu
college)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

400

Smart Agriculture IoT Education Course in enPiT-everi (Education Network for Practical
Information Technologies - Evolving and Empowering Regional Industries)*
Yasunori Takemura, Keiji Kamei, Atsushi Sanada

Department of Life Science and System Engineering, Kyushu Institute of Technology, 2-4, Hibkinino, Wakamatsu
Kitakyushu, Fukuoka 808-0196, Japan
Department of Engineering, Nishinippon Institute of Technology, 1-11 Aratsu, Kanda,
Miyakogun, Fukuoka 800-0394, Japan
Kazuo Ishii

Department of Life Science and System Engineering, Kyushu Institute of Technology, 2-4, Hibkinino, Wakamatsu
Kitakyushu, Fukuoka 808-0196, Japan
E-mail: takemura@nishitech.ac.jp, kamei@nishitech.ac.jp,
sanada@nishitech.ac.jp, ishii@brain.kyutech.ac.jp

Abstract
In Japan, expectations for the utilization of IT such as AI (Artificial Intelligence) and IoT (Internet of Things) arebecoming greater and greater to address the shortage of working population due to the declining birthrate and aging
population, and the depopulation of rural areas. There is a serious shortage of IT human resources, especially in rural
areas, and I believe that IT human resource development is an urgent task to promote the super-smart society, Society
5.0, which is being promoted by the government. Thus, in order to promote business efficiency and productivity
improvement, the utilization of IT in companies will become even more important[1].
Then, five universities, mainly in Kyushu and Chugoku regions, have been selected by the Ministry of Education,
Culture, Sports, Science and Technology (MEXT) for the enPiT-Pro project, and will collaborate to provide
systematic and practical IT education programs to working people from local companies, with the aim of fostering
advanced IT human resources. In this paper, we report on a class of the enPIT-everi program, the Agricultural IoT
Education course.
Keywords: IoT, EnPIT-everi, Agriculture, Image processing, Deep Learning, PBL education.

There is a serious shortage of IT human resources,
especially in rural areas, and we believe that IT humanresource development is an urgent task to promote thesuper-smart society, Society 5.0, which is being
promoted by the government. Thus, in order to promotebusiness efficiency and productivity improvement, the

1. Introduction
In Japan, expectations for the utilization of IT such as
AI (Artificial Intelligence) and IoT (Internet of Things)
are becoming greater and greater to address the shortageof working population due to the declining birthrate andaging population, and the depopulation of rural areas.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

401

Yasunori Takemura , Keiji Kamei, Atsushi Sanada, Kazuo Ishii

industrial fields, although they are somewhat moreinterested in manufacturing. Therefore, this program has
set up courses that take advantage of the characteristicsof the five courses. The five courses are Smart Factory,
Intelligent Car, Smart Life Care, Smart Agriculture,
Forestry and Livestock, and Hospitality IoT.
In all courses, a case study lecture is provided as apreliminary lecture, and after that, the common lectures
can be accessed via VOD (Video on Demand). Finally,
there are practical laboratory exercises in each course. In
this paper, we report on the "Practical laboratory
exercises on agriculture" conducted as an exercise of theSmart Agriculture Course.

utilization of IT in companies will become even more
important [1].
In the midst of the global trend of the fourth industrial
revolution, the Kyushu-Chugoku region has theopportunity to realize the development of new key
industries that it has been seeking for many years. In
order to realize this, it is necessary for the various people
already working in society to adapt to the newtechnological innovations and grow. In order to achieve
this aim, we will develop an advanced human resource
development program for working people, targeting
distinctive industries in the region, complementing eachother in a wide range of fields through university
collaboration, and incorporating the latest artificial
intelligence and robotics technologies into social
implementation. Then, five universities, mainly in
Kyushu and Chugoku regions (the University of
Kitakyushu, Kyushu Institute of Technology, Hiroshima
City University, Kumamoto University and Miyazaki
University), have been selected by the Ministry of
Education, Culture, Sports, Science and Technology(MEXT) for the enPiT-Pro project, and will collaborate
to provide systematic and practical IT education
programs to working people from local companies, withthe aim of fostering advanced IT human resources. In thispaper, we report on one of the education courses in
enPIT-everi (Education Network for Practical
Information Technologies - Evolving and Empowering
Regional Industries) program, the Agricultural IoT
Education course.

3. Practical laboratory exercises on Agriculture
In Practical laboratory exercises on agriculture, we
conducted PBL-style problem-solving classes andpractical assignments necessary for problem-solving.
The classes are held on weekends and consist of 24
sessions (36 hours) in total. In the curriculum of theclass, the following lessons were given in each lesson.
The exercises are mainly organized into four major
themes: artificial intelligence exercises, image
processing exercises, mechatronics (microcontroller
programming) and PBL (Project Based Learning)
exercises.
1.
2.
3.

Introduction
Mechatronics practice 1:
IoT Exercise using Microcontroller 1
15. Mechatronics practice 2:
4. IoT Exercise using Microcontroller 2
Mechatronics Exercise 3:
5. IoT Exercise using Microcontroller 3
6. Artificial Intelligence Exercise 1:
Introduction to Python
7. Artificial Intelligence Exercise 2:
Preprocessing of training data (Numpy)
8. Artificial Intelligence Exercise 3:
Deep Learning Recognizer
9. Artificial Intelligence Exercise 4:
Time Series Predictor
10. Artificial Intelligence Exercise 5:
Class determination using self-organizing maps [3]11. Image Processing 1:
Development Environment for Image Processing
Programs, Basics of Image Processing

2. enPIT-everi project
The enPIT-evri program aims to provide working adults
who are already based in some industrial field with theopportunity to acquire the knowledge, skills, thinking,judgment, and organizational abilities described in thefollowing policies by taking a systematic education
program at the graduate school level, despite its shortduration of 3 to 6 months. However, in consideration of
the various motivations and course-taking methods of theparticipants, two policies for the completion of the
program, "IoT Architect" and "IoT Engineer," will beestablished [2].
This educational program will support the development
of distinctive industrial fields in the Kyushu andChugoku regions. Companies operating in the northernpart of Kyushu are interested in IoT in a wide range of© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

402

Smart Agriculture IoT Education

12. Image Processing 2:
Color Detection Processing (Color, Grayscale, Binary)
13. Image Processing 3:
Color Space (RGB, HSV)
14. Image Processing 4:Counting Objects (Labeling, Center of Gravity)
15. Image processing 5:
Object detection (template matching)
16. Image processing 6:
Object detection (Hough transform)
17. Problems in agriculture, forestry and livestockindustry
18. Brainstorming on agriculture, forestry and livestockindustry issues
Fig.1 Experiment green house field

19. PBL exercise 1
20. PBL exercise 2
21. PBL exercise 3
22. PBL exercise 4
23. PBL exercise 5
24. Presentation of results and evaluation
In order to practice in the real field, we prepared anexperimental field (Fig.1, 2). In the experimental field,
we prepared a hydroponic field and an open-air field for
IoT practice. In addition, the power supply for the
exercises is secured by using solar power generation.
3.1. IoT Data acquisition experiment using
Microcontroller

Fig.2 Scene of Experiment (Installation of IoT devices)

We practiced how to use a microcontroller to obtaindata such as temperature, humidity, and water content ina plastic greenhouse from the cloud via the Internet. The
microcontroller is an Arduino microcontroller, and thestudents practiced how to build a system that can obtain
information from the cloud using an MQTT server.
Figure 3 shows the temperature in the greenhouse forone month, obtained by using IoT devices. The data is
acquired once every 30 minutes, and can be checked ona web browser.

Fig.3 Temperature data acquired by IoT devices CNN(Convolutional Neural Network) as a method of
application in a farm. CNN is capable of recognizing
objects in images. Therefore, in this exercise, we createdan artificial intelligence architecture to recognize
caterpillars in a farm. In this exercise, we used google
collaboratory.
3.3. Image Processing exercise
For image processing, we aimed to practice using a
miniaturized computer for use in agriculture. In thisexercise, we used a small computer, Raspberry Pi, tobuild a system that can be equipped with CNN inference

3.2. Artificial Intelligent Exercise
The students have already learned the outline ofprogramming techniques and artificial intelligence in thecommon courses. Therefore, in this exercise, we used

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

403

Yasunori Takemura , Keiji Kamei, Atsushi Sanada, Kazuo Ishii

for a long period of time and to use the data for inferenceby artificial intelligence.
Acknowledgements
This research was supported in part by enPIT-everi[1].
And, we are thankful to all participants of “Practicallylaboratory exercises on Agriculture” and helping students
in Kyushu Institute of Technology and Nishinippon
Institute of Technology
References
1. Enpit-everi homepage : “https://www.enpit-everi.jp”, inJapanese
2. Enpit-everi
education
courseintroduction:“https://www.enpit-everi.jp/course/ “in japanese
3. Kohonen, Teuvo (1982). "Self-Organized Formation ofTopologically Correct Feature Maps". Biological
Cybernetics. Vol.43 (1): 59–69.
4. Matusugu, Masakazu; Katsuhiko Mori; Yusuke

Fig.3 Temperature data in green house fieldat the same time. We mainly learned about classicalobject recognition and filtering methods in image
processing, and then performed CNN inference using the
Neural Computing Stick.

Mitari; Yuji Kaneda (2003). “Subject independent
facial expression recognition with robust facedetectionusingaconvolutional
neuralnetwork”, Neural Networks vol.16 (5): 555–559.

3.4. PBL exercise
In the PBL exercise, the students brought the equipment
they had used in the past and discussed how they could
use it in the company with graduate students and advisors.
One of the students, who is in charge of town
revitalization in the region, proposed a project that uses
IoT devices to educate young people. There were manyother ideas such as branding with IoT, and we were able
to discuss guidelines for applying the technology in theclass.
4. Summary
In this paper, we report on an exercise of agricultural
IoT in the field of smart agriculture, which is one of thecourses of enPIT-everi. In this exercise, we aimed at thedevelopment procedure of IoT devices using
microcontrollers and how to apply artificial intelligence
using image processing and CNN[4] to agriculture. In
addition, we conducted a PBL exercise in which theparticipants, who are working people, think about how toapply the knowledge obtained in this exercise to theirown work.
In the future, we would like to establish a system thatenables us to carry out the exercise using an actual plastic
greenhouse for a long period of time, and to obtain data

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

404

Development of a Handy Autonomous Underwater Vehicle“Kyubic”
Toshimune Matsumura

Dept. of Human Intelligent System, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
E-mail: matsumura.toshimune399@mail.kyutech.jp
Yuichiro Uemura

Dept. of Human Intelligent System, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
Kentaro Yanagise

Dept. of Human Intelligent System, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
Yoshiki Tanaka

Graduate School of Life Science Systems Engineering, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
Yuya Nishida

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
Kazuo Ishii

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
Abstract
Ocean is one of big challenging and extreme environments, and hard for human to access directly. As the tool for
ocean survey, Autonomous Underwater Vehicles: AUVs are expected and developed from ‘80s. The recent rapid
progress of computer and information technologies makes the development of AUVs easier and more practical. We
had developed a handy AUV “Kyubic” for the observation of shallow water and artificial structures. In this paper,
we describe the system architecture of Kyubic and the experimental results in Underwater Robotics Competition in
Okinawa 2020.
Keywords: Autonomous Underwater Vehicle, System design, Robot competition.

water are still often performed by divers or small
Remotely Operated Vehicles (ROV). Working at a depthof several tens of meters is a heavy workload for divers.
In addition, ROV are limited by their tethers. Therefore,
there is a need for a compact and lightweight AUV thatcan be operated by a few people. Since it is easy todevelop a small and lightweight AUV, underwater robot
competitions are being held with the aim of activating
research and education in the underwater field. In
particular, since the Underwater Robotics Competition in
Okinawa is a competition held in the actual sea area, it isrequired to operate an AUV suitable for the actual

1. Introduction
The ocean is one of the most difficult environments forhumans to access directly. Autonomous Underwater
Vehicle (AUV) are expected to be a tool for marine
research [1]. Currently, there is a growing need forunderwater robots for surveys and work in shallow water,
as well as for surveys and work on artificial structures
such as coastal areas, dams, and bridge piers. However,
AUV used for marine research require a large support
vessel equipped with a crane, which makes it difficult fora few people to operate. In addition, works at shallow© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021405

Toshimune Matsumura, Yuichiro Uemura, Kentaro Yanagise, Yoshiki Tanaka, Yuya Nishida, Kazuo Ishii

Fig.1 The 3D design of Kyubic
Table 1 Specifications of Kyubic

Fig.2 System architecture of Kyubic(iii) Modular structure for easy maintenance(iv) AUV and ROV modes can be selected depending on
the mission
Table 1 shows the specifications of the AUV “Kyubic”.
The AUV has a boxed structure and is compact andlightweight.
2.2. Hardware
Fig. 1 shows the layout of Kyubic. The AUV is framedof four pressure vessels. The power circuit, motor driver,microcomputer, and control PC are installed in an 8 inchpressure vessel (a). Wi-Fi and IMU are installed in the 2inch pressure vessel (b). The control and drive batteries
are installed in a 3 inch pressure vessel (c). In addition,
this AUV is installed with six thrusters (d), fourhydrophones (e), two Web cameras (f), an RGB sensor(g), a pressure sensor (h), and a DVL (i). Fig. 2 shows thesystem architecture of Kyubic. This AUV is intended to
be a versatile testbed and software development. For thispurpose, a small computer NUC with high processing
capacity is installed in a pressure vessel. The AUV uses
information from sensors such as the DVL, IMU, andpressure sensors to control the autonomous mode. The
depth sensor, RGB sensor, and ESC are controlled by
Arduino, and the hydrophone is controlled by Rasberry
Pi4. These systems are modularized, and modules can beeasily added or removed. In the previous DaryaBird, it
was not possible to turn off the power of the sensor byitself. Therefore, it was necessary to turn off all thesystems when a problem occurred. To solve this problem,
Kyubic has a system configuration that allows the power

environment. The paper explains the configuration of the
newly developed AUV “Kyubic” as one of the small
underwater robots and reports on its operation in
Underwater Robotics Competition in Okinawa 2020.
2. The hovering type AUV Kyubic
2.1. Kyubic specifications
In our laboratory have developed “AquaBox” and“DaryaBird” as small and light AUV systems [2][3]. The
AUV “Kyubic” aims to be smaller and lighter thanconventional robots, and its development concept is asfollows.
(i) Small size and lightweight, enabling operation by afew people(ii) Boxed structure with additional options dependingon the mission© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

406

Development of a Handy

supply of the sensor to be controlled by itself. In addition,
DaryaBird control with control PC directly the motor
driver. Therefore, the user could not control the robot if
the control PC system terminated improperly. On theother hand, Kyubic has an Arudino between the control
PC and the motor driver. Therefore, when the command
value from the control PC is not received for a certain
period of time, the power supply of the motor is stopped.
The thruster control unit is modularized, and the modules
can be easily used individually or installed on a robot,
resulting in a highly versatile system configuration. The
same applies to the sensor unit and hydrophone unit.
2.3. Software

Fig.3 Software architecture of Kyubic

Kyubic’s control system uses Mathworks’ MATLAB/ Simulink and ROS control methods [4]. In particular,
the Robotics System Toolbox supports ROS, whichenables highly functional and fast development using
MATLAB and ROS programs. The Parallel Computing
Toolbox is a script-based and Simulink model-based
program which can build multiprocess and parallel
computations. Stateflow is a tool for computing statetransition control methods. Fig. 3 shows a architectureof the software. Simulink is divided into 1-5 models(Sim1-Sim5), each of which exchanges data through a
ROS network. Sim1A gets the attitude data of the AUV
from IMU and the positioning data from GNSS. Sim1B
gets the ground speed from the DVL and the depth datafrom the depth sensor. Sim2 handles the PID control of
the thrusters, and Sim3 handles the image processing.
Sim4 records data and does action strategies. Stateflow
is used for the action strategy, making it easy to visually
understand the values and transitions of the process
being executed. Kyubic uses positional control toperform waypoint tracking. Waypoint tracking is amethod in which a specific position on a defined path isspecified and the robot moves to pass through that
position. For the path, the CSV data includes the
waypoint number, target position (x, y, z), target attitudeof the AUV (roll, pitch, yaw), arrival judgment
threshold, timeout, and mission number. Hence, load the
CSV as external data. The previous DaryaBird needed
to write the target value directly to stateflow. However,Kyubic only updates the csv data and does not need tochange the stateflow. In addition, this AUV can control
the operation values of Sim4 by GUI. Fig.4 shows theFig.4 Sim4 operation GUI

GUI of Kyubic. (a) in the GUI is an item that the user
inputs before the robot starts its action, such as theorigin setting and the command to start the robot’s
action. (b) shows the set waypoint number, distance tothe waypoint, and mode number. (c) shows the target
value of the robot, and (d) shows the state value of therobot. This GUI can be used to avoid human errors
during robot operation and to monitor the robot status
for efficient operation.
3. Sea trial in shallow water
3.1. Underwater Robotics Competition in Okinawa
Kyubic was operated at the Underwater Robotics
Competition in Okinawa , and the results are described.
The Underwater Robotics Competition in Okinawa is the
only competition in Japan that is held in actual marine© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

407

Toshimune Matsumura, Yuichiro Uemura, Kentaro Yanagise, Yoshiki Tanaka, Yuya Nishida, Kazuo Ishii

Fig.5 Waypoint and Tracking

environments. The rule of the AUV department is that it
navigates there and back in a straight line of about 40
meters in shallow water in five minutes. In the middle
section, there is a diving area, and the AUV needs to diveand navigate 20 meters. For details of the rules, pleaserefer to [5].

Fig.6 Target value and Estimated value

4.

3.2. Navigation results of the competition

Conclusion

We developed a small underwater robot system“Kyubic” for observation in shallow water. In addition,
we were able to confirm its basic performance in acompetition in shallow water. In the future, we will
improve the control performance and implement acoustic
communication and image processing using Kyubic.

Kyubic navigated in the surge direction throughout theentire area. This AUV dived to a target depth of 0.5 m inthe diving area and turned for the turnaround in themaritime navigation area. At the turnaround point, theAUV navigated 1[m] in the x direction to prevent the
AUV from involving the safety rope. Fig.5 shows the
waypoints and the trajectory of Kyubic. In thiscompetition, Kyubic was able to navigate to theturnaround dive area in five minutes. For the trajectory,
this AUV achieved path following with a maximum errorof 1.5 [m] in the y direction. The cause of the error in the
trajectory is considered to effect of tidal currents andcumulative error due to dead reckoning. Fig. 6 shows thecontrol performance of velocity of surge direction, depth,
and heading. The graph of the velocity in the surge
direction show an estimated value of about 0.2 [m/s]against the target velocity of 0.6 [m/s]. It is assumed that
this result was due to inadequate adjustment of PID andinferior to control performance. In the depth direction,
the tracking is slow and overshoots the target value, showthat convergence is taking time. It is assumed that the PID
was not adjusted sufficiently for depth. In the heading,there is a slight overshoot, however, it is generally incorrespond with the target value.

References
1.

2.

3.

4.

5.

Yuya Nishida, et al., Underwater Platform for Intelligent
Robotics and Its Application into Two Visual Tracking
Systems, Journal of Robotics and Mechatronics, Vol.30,
No.2, pp. 238-247, 2018.
Satomi Ohata, et al., Small Underwater Robot Systems“AquaBox” for Shallow Water Observation, The Japan
Society of Naval Architects and Ocean Engineers, Vol.4,
pp.513-516, 2007.
Wataru Hori, et al., An AUV Platform for Robot Behavior
Development: Twin-Robot “DaryaBird” and “OCTA”,
and Their’ Mission Strategy, AUV&ONR’s 17th Annual
RoboSub Competition Journal Paper, 2014.
Takashi Sonoda, et al., System Development of AUV’s
Sampling Device Controller
Employing MATLAB/Simulink Toolboxes, ICAROB,
2019.
Underwater Robotics Competition in Okinawa，
http://www.robo-underwater.jp/2020/rchp/JPN/index.php ，(2020.12.25).

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

408

A Greenhouse Project toward Smart Agriculture
Kazuo Ishii1, Yuya Nishida1, Shinsuke Yasukawa1, Kanako Shirahashi1,
Yasunori Takemura2, Takayuki Matsuo3
1

Center for Socio-Robotic Synthesis, Kyushu Institute of Technology, 2-4, Hibkinino, Wakamatsu
Kitakyushu, Fukuoka 808-0196, Japan
Department of Engineering, Nishinippon Institute of Technology, 1-11 Aratsu, Kanda,
Miyakogun, Fukuoka 800-0394, Japan

2

National Institute Technology, Kitakyushu College
2-20-1, Shii, Kokuraminami-ku, kitakyushu-shi, Fukuoka, Japan3

E-mail: ishii@brain.kyutechg.ac.jp, matsuo@kct.ac.jp, takemura@nishitech.ac.jp, sonoda@nishitech.ac.jp,
y-nishida@brain.kyutech.ac.jp, s-yasukawa@brain.kyutech.ac.j *Abstract
In the future estimation of 2050, the food demand increases 70% and the production decreases 15% caused by global
warming, and farmer population be 1/5 of current workers. Until 2050, we need an agricultural system of twice
production with the same farmland area and 5 times effective operations. To realize the sustainable society, smart
agriculture including robot technology, AI, IoT is one of the solutions for food issues. We have been working for agreenhouse project under enPiT-everi for implementation of robotic, AI and IoT technologies to agriculture. In thispaper, we introduce the project and robotic applications.
Keywords: tomato harvesting, agriculture robot, smart agricultu

addition to the intense heat and heavy rain limiting
people's outdoor activities, it is estimated that the numberof farmers in Japan will decrease sharply to one-fifth dueto the declining birthrate and aging population [3],resulting in a serious labor shortage. It is expected thatthe production will not be able to meet global fooddemand. To realize the sustainable society, smart
agriculture including robot technology, AI, IoT is one of
the solutions for food issues.
On the other hand, the Ministry of Education, Culture,
Sports, Science and Technology of Japan (MEXT)
planned a program to form practical education networks
for industry-academia collaboration in order tostrengthen the function of developing human resources

re1.

Introduction

Due to the frequent occurrence of intense heat andheavy rain, the harsh global environment caused by
global warming, world food production will decrease by
15% to 2.9 billion tons in 2050[1]. Meanwhile, the world
population has increased to her 8.6 billion and fooddemand is expected to be a serious food shortage that willreach 5.8 billion tones [2]. Because of the progress ofdesertification and global warming, rapid expansion of
excellent agricultural land cannot be expected. Therefore,
in order to eliminate food shortages, it is necessary toobtain twice the yield of the same area as before. In© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

409

Kazuo Ishii, Yuya Nishida, Shinsuke Yasukawa, Kanako Shirahashi, Yasunori Takemura, Takayuki Matsuo,

who can solve specific problems in society by making
advanced use of information technology, and practiceproblem-based learning “enPiT" [4] with the aim ofpromoting various education and spreading it nationwide.
Our university, Kyushu Institute of Technology,
proposed an educational program named “enPiT-everi”(Evolving and Empowering Regional Industries) [5] togrow up small and middle size companies in Kyushu
region together with Kitakyushu-city university and soon. We have been working for a greenhouse project forevaluation of robotic, AI and IoT technologies as a partof enPiT-everi. In this paper, we introduce the project androbotic applications for agriculture.
2.

robots is indispensable for reliably growing agricultural
products with a small number of workers.
3. enPiT-Pro and enPiT-everi
The Ministry of Education, Culture, Sports, Scienceand Technology of Japan (MEXT) planned a program toform practical education networks for industry-academia
collaboration in order to strengthen the function ofdeveloping human resources who can solve specificproblems in society by making advanced use of
information technology, and practice problem-basedlearning “enPiT" with the aim of promoting various
education and spreading it nationwide. From 2017, as
enPiT-Pro, MEXT supports the universities to develop
and implement a systematic, advanced and short-term
practical education program for working adults, mainlyin the field of information science and technology, incollaboration with industry and multiple universities. By
disseminating the program widely throughout Japan,
MEXT also supports graduate school to reform efforts
aimed at contributing to strengthening the re-learning
function of working adults in the entire field in Japan.
Our university, Kyushu Institute of Technology,
proposed an educational program named “enPiTeveri” (Evolving and Empowering Regional Industries)
to grow up small and middle size companies in Kyushu
region together with Kitakyushu-city univ., Kumamoto
univ., Miyazaki univ. and Hiroshima-city univ.. In theenPiT-everi program, we provide practical educational
programs for working people in the Kyushu and Chugoku
regions to acquire new technologies such as artificial
intelligence and robot technology. The educational
themes are specialized for manufacturing industry,
automobile industry, nursing care industry, agriculture,
forestry and livestock industry, tourism industry. The
enPiT-everi develops human resources who can promote
the introduction to companies such as IoT, AI and robots.
One of the educational themes is the implementation of
IoT for agriculture, and we have been developing theexperimental greenhouse with IoT and robotic
technologies.

Smart Agriculture

Smart agriculture is a new style of agriculture thatutilizes robot technology and information andcommunication technology (ICT) to promote laborsaving, precision, and high-quality production. In
Japanese agriculture fields, there are still many tasks that
depend on manpower and can only be done by skilledworkers. The labor saving, securing manpower, andreducing the burden in Japanese agriculture are important
issues. By utilizing smart agriculture that makes full useof advanced technology, it is possible to overcome theissues in agricultural work, secure new farmers, andimprove cultivation technology.
(1) Labor saving and labor reduction of agricultural work
Japanese agriculture is facing a serious labor shortagedue to the aging of individual farmers. It is required tosupport such hardships in Japanese agriculture byutilizing ICT.
2. Succession of agricultural technology
The second issue is the transfer of cultivation technology
to new farmers. In the old system, the knowledge istransferred in families or small groups. The shortage ofhuman resources makes difficult to inherit expert
knowledge in agriculture in the old fashion.
3. Improvement of food self-sufficiency
Japan's food self-sufficiency rate (calorie basis) was
about 40% in FY2018, and imports far exceed domestic
production. In order to increase the yield and increase theself-sufficiency rate in the face of the above-mentioned
shortage of human resources, automation by IoT, AI and

4.

Greenhouse project in enPiT-everi

The greenhouse developed for smart agriculture isshown in Fig.1. The house size is 10m x 20m and the half

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

410

A Greenhouse Project toward

area is designed for cultivation with soil and the other isfor hydroponics.

(a) Overview of the house(a) green worms imitating the leaf vein(b) Inside the house
Fig.1 The experimental greenhouse for agriculture IoT
exercise.
(b) Disease detection

Currently, as agricultural IoT Sensors, temperature,moisture data, pH regular measurement of soil, regularmeasurement of temperature, humidity and illuminancein the greenhouse, photography of plants, have been
implemented. As actuators for control air condition andsoil condition, opening and closing the greenhouse walls,
spraying mist, water supply control by solenoid valves,water circular system in hydroponics, the system for
purification of rainwater and control of nutrients areunder implementation.
Using the experimental house, we will encounter tovarious problems in agriculture. For example,
recognition of green worms imitating the leaf vein,disease of plants, harvesting and so on. Transmitting data
to cloud data base is also one research topics with limited
communication like Sigfox, Lora, etc. The detail of
exercise for Agricultural IoT course is presented in thepaper [5] in this conference. Also the TomatoHarvesting-Robot Competition was held in thegreenhouse from 7th competition in 2020 [6].

(c) Insect counting
Fig. 2 Research issues related to the greenhouse.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

411

Kazuo Ishii, Yuya Nishida, Shinsuke Yasukawa, Kanako Shirahashi, Yasunori Takemura, Takayuki Matsuo,

Fig. 3 The future concept of the greenhouse project.
5.

Summary

We have been working for experimental greenhouse
under enPiT-everi educational program. The agricultural
IoT devices are implemented toward smart agriculture.
Future vision of the greenhouse is shown in Fig. 3. The
house system will include IoT sensors and actuators tocontrol environment, robots for monitoring, harvesting,and transportation. Using the greenhouse, the newresearch topics with actual demands will come up toward
smart agriculture.
References
1. https://www.maff.go.jp/j/tokei/sihyo/index.html
2. https://www.maff.go.jp/primaff/seika/attach/pdf/200403_
2019_01.pdf
3. https://www.maff.go.jp/primaff/seika/fsc/faccess/a_map.
html
4. http://www.mext.go.jp/a_menu/koutou/kaikaku/enpit/ind
ex.htm
5. https://www.enpit-everi.jp
6. Y. Takemura et al., Smart Agriculture IoT Education
Course in enPiT-everi (Education Network for Practical
Information Technologies - Evolving and Empowering
Regional Industries), Proc of ICAROB2021, to be appear,2021.
7. K. Ishii et al., Tomato-Harvesting-Robot Competition
Towards Smart Agriculture, Proc of ICAROB2021, to beappear, 2021.
8. T. Matsuo, et al., Tomato-Harvesting Robot Competition:
Aims and Developed Robot of 6th Competition, Proc of
ICAROB2021, to be appear, 2021

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

412

The Security Challenges with The Widespread Use of IT Infrastructure in ICS
Kuan-Ming Su

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 70101, Taiwan
I-Hsien Liu

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 70101, Taiwan
Jung-Shian Li*

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 70101, Taiwan
E-mail: kmsu@cans.ee.ncku.edu.tw, ihliu@cans.ee.ncku.edu.tw, jsli@mail.ncku.edu.tw*

Abstract
The communication established by Ethernet is becoming more and more common in the industrial control systems(ICS), and it brings not only pros but also cons like vulnerabilities from information technology. We generalized aprocedure of attacking an Ethernet-enabled ICS and implemented it to the real industrial system we obtained. The
procedure gets the information and access of the devices in the ICS, like identifying the manufacturer ofprogrammable logic controllers (PLCs) and overwriting the configuration of PLCs.
Keywords: Industrial Control Systems (ICS), Programmable Logic Controller (PLC), Information Technology (IT),Network Security, Ethernet

allows hundreds of devices to communicate with eachother. In addition, there have been many industrial
protocols supporting Ethernet and Internet Protocol.
Although Ethernet meets the demands of IT and OT,
it also brings vulnerabilities to ICS. Considering the costof building information security, most ICS defense
mechanisms only have an external firewall, isolationfrom the office network, or complete independence fromother networks. Moreover, for the purpose of operation
stability, many running operation systems are not
updated to the latest version including the security
patches. Therefore, those devices are very vulnerable tomalware. Let us take TSMC for example. TSMC is themost advanced integrated circuit manufacturer in the

1. Introduction
In the modern industrial control system (ICS),
information technology (IT) is getting more and morewidespread in the operation technology (OT) field
because of Industry 4.0 [1], and serial communication is
becoming incompetent to meet the demands. Therefore,in terms of communication of IT in ICS, Ethernet iscoming to the most popular one. Ethernet is commonlyused in ICS with IT. Comparing to serial connection,
Ethernet has much more flexibility and scalability. For
example, to access and manage multiple devices, all you
need to do is connecting your engineering workstation(EWS) to the network where devices are. Also, Ethernet© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

413

Kuan-Ming Su, I-Hsien Liu, Jung-Shian Li

In addition to Ethernet interfaces on the ICS devices
like PLC and HMI, Ethernet switches and cables are alsoneeded to build the network, and there are somedifferences between common and industrial Ethernet
products. Depending on the different environments, theindustrial cables may have high-quality foil and braid toprotect data transmission from EMI [4], use cable jackets
with different materials like FEP and TPE for durability[5], or have M12 and M8 connectors (see Fig. 1 ) insteadof common 8P8C (often miscalled RJ45) in order to bewaterproof. Besides the difference of the connectors, theindustrial Ethernet switches have other featurescomparing to common switches. Let us take Cisco
industrial Ethernet 4000 series switches (IE-4000) [6] forexample. IE-4000 can work in extreme environments andtemperature range (-40 to 70 Celsius), has a durable
design, support power over Ethernet up to 240W, and soon.
To sum up, the main difference between industrial
and common Ethernet products is the durability indifferent environments and the features in use andmanagement. There is no change to the data transmission
standards.

world, and they have the top of cyber security standards
to protect the intellectual property and factory operation.
However, an accident occurred in 2018. Because of theoperational errors during the software installation on the
new equipment, after the new equipment hooked up to
the internal network, the ransomware infected othercomputer systems and fab tools and caused about $170
million US dollars losses [2].
In order to show how fragile information security
inside the ICS network is, we generalized a procedure toattack ICS networks, and we carried it out on a real ICS
which was used before. Eventually, we compromised the
PLC in the ICS network with MODBUS TCP packets.
We successfully read and wrote the registers in the PLC
and stopped the running PLC.
2. Background
There are devices controlled in ICS, and there must
be some methods for the controller to communicate witheach other to make those devices work together. We candivide it into the physical connection part and thecommunication protocol part. For the physicalconnection in ICS, there are several standards like RS232, RS-422, RS-485, and Ethernet. Among thosestandards, Ethernet is the focus of this discussion due tothe trend of IT. For the communication protocols in ICS,
there are various protocols based on different standards.
Many relatively modern protocol versions are based on
Ethernet. Some of them are based on Internet Protocol,
such as MODBUS TCP and Ethernet/IP, and some ofthem are not, such as EtherCAT and PROFINET. For thefollowing discussion, we will focus on MODBUS TCP.

2.2. MODBUS Messaging on TCP/IP
MODBUS is a popular communication protocol inindustrial environments because it is opened and does notneed a license fee. The protocol data unit (PDU) of

2.1. Industrial Ethernet

Fig. 2. MODBUS on TCP/IP

Fig. 1. M12 connector [3]Fig. 3. MODBUS TCP Data Frame© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

414

The Security Challenges with

Fig. 4. The Scenario and Procedure of Attack in an Ethernet-enabled ICS Network

is able to access the ICS network with some method likebackdoor, we generalized a procedure to get the
information of the ICS and attack it.

MODBUS is simple. It only consists of function code anddata. Depending on the function codes and request or
response, the following data structure is different. For
example, function code 03 is reading multiple holding
registers, the request data structure is composed of 2
bytes starting address and 2 bytes quantity of registers,
and the response data structure is composed of 1 byte
following data size and the value of registers [7].
MODBUS Messaging on TCP/IP, or MODBUS TCP
for short, as the name suggests, is MODBUS
implemented on the TCP/IP (see Fig. 1), and the port
number of it is 502. The data frame of MODBUS TCP isin Fig. 2. Transaction identifier is for pairing the request
and response. Protocol identifier is used for internal
system multiplexing, and value 0 stands for MODBUS
protocol. The length field is the byte count of the rest part
including unit identifier, function code, and data field.
Unit identifier is used for the internal system routing
purpose. Function code and data field are MODBUSPDU [8].

3.2. Procedure
The first thing to do is scanning the internal network.
In most internal ICS networks, due to the weakness of IT
security in ICS, there is no protection method like IDS
and IPS. Therefore, by scanning the network, the attacker
can observe the information in the ICS network such as
enabled services, subnets’ range, the number of devices,manufacturer of devices, and so on. After obtaining theinformation of the ICS network, according to theinformation, the attacker can formulate detailed attack
methods such as man-in-the-middle attack with ARP
spoofing to compromise the information security andoperation safety in the ICS.
4. Case Study
The ICS that we are going to demonstrate theprocedure on is the same as our previous study [9]. The
ICS is composed of tens of PLCs to control the fieldequipment and one computer as the HMI to gather dataand control PLCs. All of them are connected to an
Ethernet switch.
First, we used a tool called Nmap to scan the ICS
network. In the result, we can see the information of
online devices, such as IP address, enabled services, and
MAC address, and we can identify that the manufactureof PLC is Telemecanique Electrique which is Schneider
Electric from the MAC address. The communication

3. Attacking an Ethernet-enabled ICS Network
3.1. Scenario
Since ICS with IT brings vulnerabilities, the attacker
could use those vulnerabilities to inject a backdoor into
the computers in the ICS, bypass the information security
protection measures, and perform malicious operations.
For example, an attacker could implant a Trojan horse
into the EWS in the ICS via social engineering, thenbypass the firewall and invade into the ICS network to do
whatever he wants. Therefore, assuming that the attacker© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

415

Kuan-Ming Su, I-Hsien Liu, Jung-Shian Li

2.
3.

4.

Fig. 5. The MODBUS TCP packets with function code 90

protocol of Schneider PLC is MODBUS, so we tried toread the holding registers with standard MODBUS
function code 03 with the MODBUS TCP testing tool,and it works. Furthermore, the writing holding registers’
function code 16 also works.
In our previous study, we used the IDE called
TwidoSuite to perform some malicious operation on the
PLC. This time, we use a tool called Wireshark to record
the MODBUS TCP packets. Unlike the standard
MODBUS TCP, the function code of those packets is 90(see Fig. 5.), which is manufacturer defined. It is not able
to just do the replay attack because of the authentication
mechanism in the data bytes of MODBUS. However, wefound some rules of it. We successfully established theconnection and commanded the online PLC to stop.

5.

6.
7.
8.
9.

Networks in the Era of the Internet of Things and Industry
4.0, IEEE Industrial Electronics Magazine, vol. 11, pp.
17-27, 2017.
TSMC, TSMC Details Impact of Computer Virus Incident,
TSMC, Hsinchu, 2018.
eicos, M12 Plug Connector - M12 Male Straight
Connector without Cable, eicos, Online. Available:
http://eu.eicos.com/accessories/connectors/m12-malestraight-connector/. Accessed 14 12 2020.
Lapp Tannehill, LAPP ETHERLINE® 2 Pair: CAT5
Flexible - 22 AWG - Green, Lapp Tannehill, Online.
Available:
https://www.lapptannehill.com/etherline2pair-cat5e-flexible-22awg-green. Accessed 15 12 2020.
BELDEN, Category 6 Cable - 7931A, BELDEN, Online.
Available:
https://www.belden.com/products/cable/ethernetcable/industrial-ethernet-cable/7931a. Accessed 15 12
2020.
Cisco, Cisco Industrial Ethernet 4000 Series Switches
Data Sheet, Cisco, 2020.
Modbus Organization, MODBUS APPLICATIONPROTOCOL SPECIFICATION V1.1b3, Modbus
Organization, Hopkinton, 2012.
Modbus Organization, MODBUS MESSAGING ON
TCP/IP IMPLEMENTATION GUIDE v1.0b, Modbus
Organization, Hopkinton, 2006.
Kuan-Ming Su, I-Hsien Liu, Jung-Shian Li, The Risk ofIndustrial Control System: Programmable Logic
Controller Default Configurations, in International
Computer Symposium 2020, Tainan, 2020.

5. Conclusion
In this paper, we discussed the weakness of IT
security in the ICS network, generalized an attacking
procedure for the ICS network, and implemented it to thereal industrial system to support the argument. It turns outthat we can easily obtain the information of the ICS withthe procedure in the certain scenario without theknowledge of the ICS. As the widespread use of IT
infrastructure in ICS, we must pay more attention to thecyber security in ICS.
Acknowledgements
This work was supported by the Ministry of Scienceand Technology (MOST) in Taiwan under contract
numbers MOST 109-2218-E-006-014- and MOST 1092218-E-006-510-.
References
1. Martin Wollschlaeger, Thilo Sauter, Juergen Jasperneite,
The Future of Industrial Communication: Automation© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

416

A Communication System with Equipment's Characteristics
Chia-Chun Lai, I-Hsien Liu, Chi-Che Wu, Jung-Shian Li

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 70101, Taiwan
Chuan-Gang Liu*

Department of Applied Informatics and Multimedia, Chia Nan University of Pharmacy ＆ Science
No.60, Sec. 1, Erren Rd., Rende Dist., Tainan City 71710, Taiwan
E-mail: {cclai, ihliu, ccwu1988}@cans.ee.ncku.edu.tw, jsli@mail.ncku.edu.tw, chgliu@mail.cnu.edu.tw*www.cans.ee.ncku.edu.tw, www.cnu.edu.tw

Abstract
Over the past few years, applications on the internet have grown rapidly. In order to identify a specific device, we
usually use the information which attains from network packets such as IP address, MAC address and communication
port, etc. However, using this kind of information is not enough to identify precisely. Therefore, our research focuses
on characteristics of communication devices that can identify precisely and also create a communication system
which is able to imitate these characteristics.
Keywords: ICS, Communication, Network, Device Identification, Cyber Security

the analyzed device and provide a better effect on device
emulation.

1. Introduction
In the last few decades, communication technology has
grown rapidly and the internet is widely used in our daily
lives. The progress result in devices that can connect tothe internet have an explosive increase. Compare to thepast, some information such as IP address, MAC address,and communication port is commonly used to identify a
specific device on the internet. However, using this kindof information is not enough to identify comprehensively
because the information can simply be imitated by anyother devices. In our research, we focus on thecharacteristics of communication device. And we also
provide a mechanism that can analyze thesecharacteristics and generate configuration references
which other devices can apply. With the customized
communication module and the references mentioned
above, we can make other devices imitate the behavior of

2. BackgroundIn this section, we are going to discuss some researches
and methods which are commonly applied to perform
device identification.
2.1. Packet Header
When devices communicate with each other, they must
use the same protocol. Thus, allow them to understandwhat the other side transmitted. Nowadays, the mostcommonly used protocol is the TCP/IP protocol suite.
This protocol suite includes a great deal of widely used
protocols. Some well-known protocols in the TCP/IP
suite are IP, TCP, UDP, etc., which are performed indifferent layers. These protocols have their own specificpacket header when they are used.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

417

Chia-Chun Lai, I-Hsien Liu, Chi-Che Wu, Jung-Shian Li, Chuan-Gang Liu

Some information in these headers or the combination ofthis information in the different header can be used to
identify which device sends this packet. In the next few
paragraphs, we are going to explain the detail of how
information in headers are used for device identification.

KAWAI has shown that they can analyze traffic patterns
and identify devices in different categories1. Also, there
are some researches use machine learning to identify IoT
devices which can classify all devices that connect to the
network into some specific types2,3,4.

2.1.1. IP header

3. Characteristics of Devices

IP is the principal protocol in network communication. It
can simply be divided into two versions: IPv4 and IPv6.
Although the packet header of IPv4 and IPv6 is different,
they still have some similarities. In the IP header, themajor fields are source address and destination address,
both of them exist in IPv4 and IPv6 header. The mainfunction of these headers is to identify which device
sends out this packet and which device should this packet
send to. With these two pieces of information, we could
simply identify a specific device, but it’s not enough for
precise identification. Some networks may use NAT(Network Address Translation) and many devices would
share a public IP address. In this situation, using an IP
address as the only identifier is not enough for device
identification.

Each device has its special characteristics when they arecommunicating. In our research, we focus on the
behavior of devices when they respond or transmit
packets. In our previous research, we obtain that thelatency of the response packet between each device isunique. This discrepancy can cause by various factors in
different layers.
3.1. Physical Layer
The devices produced by the same factory with different
models will have their unique characteristics due to thevariation of assembly lines. If we focus on the samemodel devices, they still exist unique characteristics
because of the standard error of the machine which isallowed by its supplier. Even more, the manufacturing
tolerance which is set by the manufacturer will also
increase the diversity of characteristics in devices.

2.1.2. TCP header
TCP often uses on reliable transmissions such as a
webpage, video stream, and file transfer. In the TCP
header, there are two fields that can be used for device
identification. These two fields are the source port anddestination field, but we can’t use them independently.
We need to combine port information with IP
information mentioned in the previous paragraph to carryout a better performance for device identification.

3.2. Transport Layer
When devices communicate on the internet they have todetermine which protocol to be used in the transport layer.
Using different protocols will result in respective
behavior therefore generate a distinctive characteristic ofdevices. For instance, in a general WAN internet
environment, if we use UDP protocol for communication,
it might provide lower latency than using TCP protocol5.
This situation shows that which protocol is used will havea significant impact on the characteristics of devices.

2.1.3. Ethernet frame
Ethernet protocol is a protocol that belongs to the datalink layer. Recently, it is the principle protocol in ourdaily internet environment. Each Ethernet packet has an
Ethernet frame that contains a header. An Ethernet
header has four major fields: destination MAC addresses,
source MAC addresses, Ethertype and IEEE 802.1Q tag
or IEEE 802.1ad tag.

3.3. Application Layer
Every device has its own operating systems. Also, theexecutive application on the device would also affect the
behavior. Even the same function programmed in adifferent language would lead to various characteristics
of devices. Furthermore, regardless of having the samemodel device, if they are running in a different version of
firmware, they may also increase the diversity ofcharacteristics. In conclusion, the application layer has
many factors that will affect the behavior of the device.

2.2. Traffic Patterns
Various devices will generate different traffic patterns
depending on their specific needs. Some researches use
traffic patterns to identify devices. Research from Hiroki© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

418

A Communication System with

4. Experiment and Result

program library. Both of them can load the characteristic
model generated by the analysis side. Furthermore, we
add in a specific parameter called the loading parameter.
Using the loading parameter, we can imitate thecharacteristics of the device. Meanwhile, the device is atworking status.

In our research, we propose a communication system that
can analyze the characteristics of devices and then use
these data to perform characteristics emulation of devices.
We use a Schneider programmable logic controller(PLC) to be the device where we want to emulate and use
a Windows 10 PC responsible for analyzing other devices.
Owing to the device we used, the experiment will use a
specific protocol which is commonly used in industrial
control system called Modbus. Also, our communication
system is programmed in C++ and run on Linux which
can perform a better latency control of communication.
Fig. 1. is the overview of our system. We can separate
our system into two parts. In Fig. 2 and Fig. 3, we can see
it have two major chains. The first chain is responsiblefor device analyzation and the second chain is
responsible for emulation of the device.

4.3. Analysis result
In order to analyze the result of our emulation, our result
focuses on the latency of response. In the result analysis,
we use the Kolmogorov–Smirnov test for analyzation.
Table 1 is the statistical table of our experiment, we let
our analysis computer sending 10000 read register
Modbus packet to both Schneider PLC and our emulator.
In the Kolmogorov-Smirnov test, we assume that two
samples come from the same distribution. Table 2 is our
Kolmogorov-Smirnov test result. In the KolmogorovSmirnov test result, we can obtain a field that is
asymptotic significance. In general, if asymptotic
significance is greater than 0.05 or 0.1 then we accept theassumption of the Kolmogorov-Smirnov test, whichmeans the two samples come from the same distribution.
In table 2 we can see that our asymptotic significance is0.997 which is greater than 0.01. As a result, we can say
our emulation can respond to the same characteristics as
the true Schneider PLC.

4.1. Analysis SideOn the analysis side, we use a computer to send the
response request to the device being analyzed. When thedevice response to our computer, we will record its
round-trip time. Then we sorted out these data into acharacteristic model and arrange a configuration file thatcan be used for the emulation side.

Fig. 1. Overview of our specific communication system

4.2. Emulation Side

Fig. 2. Flow chart ofcommunication system

On the emulation side, a device can either run our specificcommunication program on it or import our customize

theanalysisside© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

419

of

specific

Chia-Chun Lai, I-Hsien Liu, Chi-Che Wu, Jung-Shian Li, Chuan-Gang Liu

Table 1. The statistical table of experiment device

which is at work. Also, using the statistical method we
can show the result that using our system, we can let
normal computer imitate the same behavior as theworking PLC.
Acknowledgements
This work was supported by the Ministry of Science andTechnology (MOST) in Taiwan under contract numbers
MOST 109-2218-E-006-014-, MOST 109-2218-E-006510- and MOST 109-2221-E-041-001-.
References
1.

2.

3.
Fig. 3. Flow chart of the emulation side our specificcommunication system
DeviceSample Size

Average

Schneider
PLC

Standard
Deviation

10000

5.92580.378033

Emulator

10000

5.9304

0.391371

4.

5.

Table 2. The Kolmogorov-Smirnov test result
Most Extreme
Differences

Absolute.006

Positive.001

Negative-.006

Kolmogorov-Smirnov Z.403

Asymp. Sig. (2tailed).997

H. Kawai, S. Ata, N. Nakamura and I. Oka, Identificationof communication devices from analysis of traffic patterns,
2017 13th International Conference on Network andService Management (CNSM), Tokyo, Japan, Nov. 2017,
pp. 1-5
Ola Salman, Imad H. Elhajj, Ali Chehab, Ayman Kayssi,
A machine learning based framework for IoT device
identification and abnormal traffic detection, Trans
Emerging Tel Tech., 2019
Antônio J. Pinheiro, Jeandro de M. Bezerra, Caio A.P.
Burgardt, Divanilson R. Campelo, Identifying IoT devices
and events based on packet length from encrypted traffic,
Computer Communications, vol. 144, 2019, pp. 8-17
Yair Meidan, Michael Bohadana, Asaf Shabtai, JuanDavid Guarnizo, Martín Ochoa, Nils Ole Tippenhauer, andYuval Elovici, ProfilIoT: a machine learning approach for
IoT device identification based on network traffic analysis,
SAC '17: Proceedings of the Symposium on Applied
Computing, New York, United States, Apr. 2017
I. Coonjah, P. C. Catherine and K. M. S. Soyjaudah,
Experimental performance comparison between TCP vs
UDP tunnel using OpenVPN, 2015 International
Conference on Computing, Communication and Security(ICCCS), Pamplemousses, Mauritius, Dec. 2015, pp. 1-5

5. Conclusion
In this paper, we propose a mechanism to improve theemulation of device as well as focus on the difference of
behavior on devices. Thus, we analyze the response
latency of device, then create a characteristics model
which can be used in our specific commutation system.
Our specific communication system can load the model
file and provide a load parameter that can imitate theloading of device and imitate the behavior of device© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

420

Application of the Self-Organizing Map (SOM) to
Analyze the Multiple Perspectives on Cross-National Culture
Li-Ming Chuang

The Department of International Business,
Chang Jung Christian University
No.1, Changda Rd., Gueiren District, Tainan City, 711301, Taiwan
Yu-Po Lee

The Ph.D. Program in Business and Operations Management, College of Management,
Chang Jung Christian University
No.1, Changda Rd., Gueiren District, Tainan City, 711301, Taiwan
Shu-Tsung Chao

Graduate School of Business and operation Management,
Chang Jung Christian University
No.1, Changda Rd., Gueiren District, Tainan City, 711301, Taiwan
Email: liming@mail.cjcu.edu.tw, davidlitw@yahoo.com, james@cvig.org,

Abstract
Organizational behavior in different countries and cultures has been the focus of studies in recent years.
Following a literature review, we find that there are many different perspectives and features in the related
cross-cultural studies. However, whether the analysis methods of these different cultural dimensions can
fit into the increasingly complex and diverse topics of cross-cultural studies have not been determined,
which is also a topic of great interest to scholars. Therefore, this study integrates the previous
cross-cultural literature and aims to construct an analysis model of cross-national culture with multiple
dimensions from three important cultural dimension theoretical models commonly used in cross-cultural
studies: Hofstede, Global Leadership and Organizational Effectiveness (GLOBE) and World Values
Survey (WVS). Traditional statistical analysis seems to be unable to solve the problem of the integrationof relevant scales and units in different dimensions of cultural analysis. Therefore, this study uses a
self-organizing map (SOM) as an analysis method to integrate 17 cultural variables from this multiculturaldimension for cluster analysis and explains the cultural types in 26 countries based on the results. This
study explores the differences and similarities of different countries in different cultural dimension
analyses and provides a comparative model of multicultural analysis. This study takes samples from threecross-cultural analysis databases as data sources and employs the self-organizing map for analysis basedon a neural network algorithm that can be used for type discrimination, map analysis, process monitoring,and error analysis. The results identify the cross-cultural groups of 26 countries and reveal their key
cultural similarities and differences. We also elaborate upon the findings of these cultural characteristics
and multi-cultural dimensions. The signification of this study is presented as a reference for subsequentstudies of transnational and cross-cultural analysis and its applications.
Keywords: Cross-Culture, Self-Organizing Map (SOM), Hofstede GLOBE WVS

1.

Although Hofstede's theory is the mostrepresentative, there are still some bottlenecks. For
example, the samples are from the employees of asingle company (IBM), the dimensions of culturaldifferences are insufficient, the sampling is limited,
and the cultural dimensions are not dynamic anddevelopmental. Many studies on national culture
have emerged successively, such as the GLOBE(GlobalLeadershipandOrganizational

Introduction

At the end of the 20th century, many scholars
adopted large sample empirical methods, which havebecome the mainstream of current research on
cultural differences. Representative scholars include
Hofstede(1980,
1991),
Trompenaars&
Hampden-Turner (1993, 1998), Schwartz (1997),
and House, Hanges, & Ruiz-Quintanilla (1997).

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

421

Li-Ming Chuang, Li-Ming Chuang, Shu-Tsung Chao

important cultural dimension theoretical models
commonly used in cross-cultural studies: Hofstede,
Global Leadership and Organizational Effectiveness(GLOBE) and World Values Survey (WVS). This
study focuses on the application of theself-organizing
map
toexplorethemulti-dimensional cross-cultural analysis model. A
self-organizing map neural network can gather alarge amount of information with similar
characteristics through the self-organizing map andthen compare and analyze multiple models based on
the cluster data. Therefore, this study explores thedifferences and similarities of various countries
under different cultural dimension analyses andprovides a comparative model of multicultural
analysis. Samples from three cross-cultural analysis
databases are used as data sources.

Effectiveness) project conducted by House, Hanges,& Ruiz-Quintanilla (1997). GLOBE expanded
Hofstede's five dimensions into nine dimensions,retaining "power distance" and "uncertainty
avoidance".
Hofstede's"individualism
andcollectivism" were divided into "group collectivism"and "public collectivism", while " masculine andfeminine culture" was divided into "gender equalityand decisiveness". The "short- and long-termorientation" was changed to "future orientation".
"Humanistic orientation" is consistent with
Kluckhohn's dimension of "views on human nature",
and the dimension of "performance-orientation" wasadded.
In addition to the above two analysis models ofcross-national culture, the World Values Survey(WVS) has also gained increasing attention in recent
years. WVS originated from the European Values
Survey (EVS) conducted in 1981 for 10 countries ofWestern Europe. The findings are instructive interms of cultural change and can be extended
globally. Generally speaking, this transnational
survey covers a wide range of topics, including
social values, social norms, social issues, social
distance, work issues, labor organization,
employment issues, political attitudes, national
democracy, gender issues, environmental issues,
marriage, and family and child rearing issues. The
literature of the past decade shows thatcross-national culture is an important topic in thefield of international enterprise research (Breuer,
Ghufran, & Salzmann, 2018; Chand & Ghorbani,2011). Relevant contextual factors such as cultural
distance, cultural value, long-term orientation,individualism and physical distance can all predict
different national cultures (Beugelsdijk, Maseland,
Onrust, Van Hoorn, & Slangen, 2015; Malik & Zhao,2013).
After reviewing the past studies, we find that there
are many different perspectives and features in therelated cross-cultural studies. However, whether the
analysis methods of these different culturaldimensions can fit into the increasingly complex anddiverse topics of cross-cultural studies have not been
determined, which is also a topic of great interest to
scholars. Therefore, this study integrates theprevious cross-cultural literature and aims toconstruct an analysis model of cross-national culture
with multiple integration dimensions from three

2.
2.1.

Research Design
Research Method: Self-Organizing
Map

A self-organizing map neural network can gather alarge amount of information with similar
characteristics through the self-organizing map.
Since SOM is a neural network for unsupervised
learning, the target output value of web-basedlearning does not have to be defined in advance.
Cluster rules can be derived according to data
similarity in order to distinguish the differencesamong data groups. It is an effective analysis toolfor Data Mining. Self-Organizing Map (SOM) is anunsupervised artificial neural network model,
proposed by Kohonen (1982). SOM is especiallysuitable for representing the distribution ofhigh-dimensional data vectors in a multidimensional
space. The high-dimensional data vectors can bemapped into two-dimensional space, so that a user
can understand the relationship between the original
data structures, and the number of data groups canbe reduced.
2.2.

Research Subjects and Data Sources

The data sources for this study are from threeimportant cultural dimension theoretical documents
and databases commonly used in cross-cultural
studies: Hofstede (Geert Hofstede's Websites),© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

422

Application of the Self-Organizing

Global Leadership and Organizational Effectiveness(GLOBE), and World Values Survey (WVS). Table
1 lists the data of the cultural dimensions of 26
countries.

3.

This study used a self-organizing map (SOM) as ananalysis method to integrate 17 cultural variablesfrom this multicultural dimension for cluster
analysis and explains the cultural types in 26
countries based on the results. Moreover, this study
explored the differences and similarities of different
countries under various cultural dimension analyses,
and provided a comparative model of multicultural
analysis. It sourced samples from threecross-cultural
analysisdatabases.
The
self-organizing map is for analysis based on a neuralnetwork algorithm that can be employed for typediscrimination, map analysis, process monitoring,and error analysis. The results identify thecross-cultural groups of 26 countries, reveal their
key cultural similarities and differences, and help
elaborate upon these cultural characteristics andmulti-cultural dimensions. The significance of thisstudy is presented as a reference for subsequentstudies of transnational and cross-cultural analysis
and its applications.
According to the results in Figures 2 to 5, Table 2
summarizes the comparisons of cross-cultural
analysis patterns in multiple dimensions. Table 2
shows that there are Eastern cultural group andWestern cultural group in Hofstede 6 analysis. There
are three groups in the analysis of GLOBE 9: high,medium, and low GLOBE cultural groups. WVS 2
analysis shows four groups of country clustering:W1 (High T/R & LOW S/S) culture group, W2(High T/R & High S/S) culture group, W3 (Low T/R& Low S/S) culture group, and W4 (Low T/R &High S/S) culture group. Among them, most east
Asian regions or countries such as Taiwan, Japan,
China, Hong Kong, and South Korea are in the W1(High T/R & LOW S/S) culture group.
The results in Table 2 help us analyze the
distribution of 26 countries after the analysis of fourcross-national cultural analysis modes by SOM. It isinteresting to find that there are two groups in
Hofstede 6 analysis: H1 and H2; and the countries of
H2 are the same as the countries of G1 and G3 after
GLOBE 9 analysis; i.e., Hofstede's Western culturegroup is equal to the high and low cultural groups of
GLOBE, and Taiwan belongs to G2 (i.e., medium
GLOBE culture group) in GLOBE 9 analysis.
Among the 26 regions or countries, only Taiwan
belongs to this group. The cultural attribute andTable 1 List Of Country Information
Country Information
Culture Cluster
Anglo Cultures

Latin Europe

Middle East Cultures

Confucian Asia

Latin America

Country

Ctry. Code

USA

US

CanadaCA

EnglandUK

IrelandIE

New Zealand

NZ

South Africa

ZA

Australia

AU

France

FR

Italy

IT

Portugal

PT

Spain

ES

Swiss

CH

MoroccoMA

Turkey

TR

China

CN

Hong Kong

HK

Japan

JP

Singapore

SG

South Korea

KP

TaiwanTW

BrazilBR

Argentina

AR

ColombiaCO

El Salvador

SV

MexicoMX

Venezuela

VE

Conclusion

Date source：World Value Survey© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

423

Li-Ming Chuang, Li-Ming Chuang, Shu-Tsung Chaogroup, and many advanced countries belong to thisgroup. The W2 (High T/R and High S/S) culturegroup has both tradition and self-expression ability,
represented by two countries: New Zealand andSwitzerland. Countries in the W3 (Low T/R & Low
S/S) culture group are Australia, Morocco, Turkey,
and Singapore. The analysis results of the above
three cultural dimensions are close to the clusteringresults of GLOBE 9 cultural dimension analysis.

classification of Taiwan are worth discussing, andsubsequent research should further analyze its
causes.
WVS 2 cultural dimension clustering analysis
presents four groups of country clustering. Most
countries fall into two of these categories. One partis in the W1 (High T/R and LOW S/S) culturalgroup, and Taiwan belongs to this group. The otherpart is in the W4 (Low T/R and High S/S) cultural
Table 2(www.worldvaluessurvey.org).
Aggregate
Producer: ASEP/JDS, Madrid., 2009.

References
1.

2.
3.
4.

House, R. J., Hanges, P., Ruiz-Quintanilla, S. A., &Dickson, M. W. The development and validation ofscales to measure societal and organizational culture.
Under review, 1997.
Inglehart, R., & Welzel, C. Modernization, culturalchange, and democracy: The human development
sequence. Cambridge Univ Pr, 2005.
Trompenaars A., Hampden-Turner C.. Riding thewaves of culture: Understanding cultural diversity inbusiness[M]. London: Nicholas Brealey, 1997.
WORLD
VALUESSURVEY
1981-2008
OFFICIAL AGGREGATE v.20090901, World
ValuesSurveyAssociation© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

424

File

Threats Hidden in Employee Workstation through Office Files
Tung-Lin Lee

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 70101, Taiwan
I-Hsien Liu

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 70101, Taiwan
Jung-Shian Li*

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 70101, Taiwan
E-mail: tllee@cans.ee.ncku.edu.tw, ihliu@cans.ee.ncku.edu.tw, jsli@mail.ncku.edu.tw*

Abstract
With the advent of the Internet of Things era, IoT devices participate in our work environment. From printers, sharedfiles to temperature control, they make our working environment more convenient. Although these IoT devices
communicate through different communication protocols, the same identity verification method is often used.,
hackers can also use this authentication method to steal user identity verification information, This study combines
past research results to present our new findings, and further, organize them into a complete attack architecture.
Keywords: Active Directory, IoT, Credentials, Spear-fishing

1. Introduction

Due to the single sign-on implementation of windows
operating systems, users could save a lot of time without
having to enter their account and password again andagain. However, this setting also allows hackers to Use
some dedicated constructed malicious files to steal theuser's identity verification information (NTLM hash),and then launch attacks like passing the hash, and thenlog in to the user's computer remotely, which becomes away to gain access to the organization's intranet.

In an environment with Active Directory (AD), allresource access authorization authentication will behandled by Active Directory, so that every Server (EX:Exchange server, NAS, printer) does not need to handle
every user's authentication credential. Although theserver authentication work is shared by Active dictionary,
for users, to access a certain function, they still need toenter the password repeatedly once they want to accessdifferent resource. To deal with this situation single signon implementation was invented.

Although there are many ways to cause an NTLM hashleak, fortunately, most of them cannot be exploited byhackers. In this study, we would discuss several NTLM
hash leak methods, and focus on some of the mostcommonly used method of hacking: Office files, and dive

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

425

Tung-Lin Lee, I-Hsien Liu, Jung-Shian Li

deeper to the possibility of extended application andunder what circumstances will the existing protection
mechanism be bypassed.
In the second section introduced background knowledge
related to authentication, as well as traditional attack
methods. In the third section, we will introduce our new
discoveries and ways to bypass existing protections andcombines past research and our findings to try toconstruct a method that can be used to automate largescale attacks in real scenarios. Finally, the forth section
will be our conclusion.

Fig. 1. The NTLM Challenge-Response Mechanism.

2.2. Typical attack
The security problems of the NTLM mechanism arereflected in two aspects. First, the NTLM hash can be
cracked by rainbow tables or brute force. Common attack
methods used Responder as a malicious server whichanswers to specific queries to receive the credentials sentby the victim to obtain the NTLM hash and then start thecracking process.

2. BackgroundIn this section, we will introduce a common
authentication method "NTLM". Besides, we will discuss
the protocol and application using NTLM, and the pastattacks on this authentication.
2.1. NTLM Overview

However, it is difficult to crack in a limited time, so there
is another attack method called NTLM relay or so callpass the hash. The attack method is another variant of a
man-in-the-middle attack as shown in Figure 2. Although
there are some mitigations such as EPA (Enhanced
Protection for Authentication), there are still ways to
bypass those mechanisms such as “drop the mic”, andNTLM relay is still an indispensable step while
implementing these attack. Through NTLM relay, an
attacker can steal the victim’s credentials, and thus obtainconfidential data that the victim could access or call the
RPC service that is authorized to control the server,
thereby affecting the entire operation of the organizationor enterprise.

NTLM is a challenge-Response authentication
mechanism. First, the client informs the server, theusername and informs the server that it wants to log in.
Then the server sends out a set of random number as achallenge and requires the client to hash it with its own
password. Third, the client sent the hash result to theserver, and then Server will verify whether the password
is correct or not.
The NTLM protocol suite was first implemented in a
Security Support Provider of Microsoft security
protocols and been adopted in other protocols. Because
NTLM has undergone many changes, We are not goingto talk about its cryptography details here. If you areinterested in the detailed, Microsoft official document
could provide more information.
NTLM is mainly used for SMB, LDAP, MSSQL, HTTP,
and other protocols, not only that, although it is not thepreferred authentication method, in fact, NTLM is also
used for wi-fi connection and remote desktop
authentication. Although NTLM has been around for
decades and has been updated many times, its security
mechanism has often been challenged. MITRE
ATT&CK has suggested that enterprises or institutions
should not use NTLM for authentication but instead
using Kerberos. However, Kerberos is more complicatedto set up and there is downward compatibility and thethird-party software requirement which does not support
Kerberos as an authentication option, using NTLM forauthentication is usually a necessary option.

Fig. 2. The NTLM Relay Attack with Active Dictionary

2.3. NTLM leak
For the NTLM relay attack, the key relies on how to makethe victims actively send a login request to the malicious
server. through malicious file is one of the way.
In the article "Living on land: NETNTLM HASHES "introduced those penetrations such as HTML files, PDF,

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

426

Threats Hidden in Employee

directly to mark files from the Internet. If this file
happens to be an Office document while opening this
Office file, we will see it open in Protected View, Similar
to a sandbox to prevent malicious cities from being
executed, noticed that the Enable all feature in ProtectedView is because the files is from the internet. It isdifferent from the Enable Content seen by Macro in the
file. In our study refers to the former, which is "Enable
all feature".

Window media player, Office files, and others to cause
NTLM leakage.
The principle is that when we click on these malicious
files, the files actively request resources embedded in the
UNC path through the SMB protocol. This sounds
reasonable, but when the server located on this path asks
for a certificate, the client will actively send the NTLM
hash, which is the credential of the victim! Because of
Windows integrated authentication, Windows integrated
authentication is mainly suitable for users not torepeatedly enter account and password when accessing
any resources. This is a great function to save users' timeand reduce the possibility of Side-channel attack attacks,
but when files need to access resources on a certain UNC
path, the Windows operating system. It will consider thatthe security of these UNC paths has been verified when
the file is made, so when the manufactured file is opened
again, the file would automatically send the resource
query and user credentials if needed.

3.2. Bypassing Office mitigation
Although there used to be browsers or Mail Desktop
failed to check specific file types, and different
determination on dragging downloadable files or double
clicking to downloadable files. In other words, if the user
dragged a file from Mail Desktop to a folder,the protectmode would be bypassed. However, this type of bypass
technique has been patched by most software developers,
therefore hackers started to focus on other problems:
decompression.
Whencompressed
files
are
decompressing, the unzipped file should inherit thecharacteristics of Zone:identifer. Although mostcompression software has noticed this, different
compression methods in the same decompress software
will have different behaviors. Taking the most common
7zip as an example, it will propagate only when thecompressed file is open in archive and double click the
file. the article "Downloads and the Mark-of-the-Web"
discuss it before, we re-experimented and organized, andfound a little different result (Table 2)

Usually, those malicious files are sent throughattachments in phishing emails. Fortunately, most people,
rarely click on the extensions called .m3u .url .jnlp andother rare seen extensions. The most vulnerable types are
PDF and office documents because these two file types
are also the most common types in daily office work.
Table 1. File Type that Could Cause NTLM Leak.
File Type

File extension

Internet shortcut

urlWindows media player

m3u, asx, wax

Java external jar

jnlp

Microsoft Word

docx, xml

Chrome & IE & Edge

htm

Adobe Acrobat Reader

pdf

Table 2. Popular Extractor List

3. Our findings
3.1. Office mitigation
The NTFS file format supports using the named Zone:
Identifier in the alternate data stream to mark files fromthe Internet. In order to avoid attacks similar to NTLM
relay and other malicious code being executed while thefile is opened, browsers and other internet clients such as
email and chat programs utilize IAttachmentExecuteinterface’s methods or write the Alternate Data Stream

Extractor

Double-clickin Archive

Extract all

Windows
Explorer

Not vulnerable

Not vulnerable

WinRar

Not vulnerable

Not vulnerable

7zip

vulnerable

Not vulnerable

WinZipNot vulnerable

vulnerable

IZArc

Not vulnerable

Not vulnerable© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

427

Tung-Lin Lee, I-Hsien Liu, Jung-Shian Li

3.3. Is stealing NTLM hash over other protocol
possible?

findings, and further, organize them into a complete
attack architecture.

The malicious files in table I steal the NTLM hash only
through the SMB protocol. However, in the real world,
experienced network administrators will prohibit SMB
from connecting to the internet. Not only there is usuallyno related requirement, but to avoid NTLM hash throughSMB. While hackers install Responder on the infected
employee’s computer is not possible to open rogue SMB
servers, because that TCP port 445 has been bound by the
Network Neighborhood or samba. The responder cannot
bind the same port with an existed application. Becauseof those reasons, we try to insert the resource access pathof the HTTP URL path in the Office document.
Unfortunately, HTTP Request will be sent out
successfully, but when the server responds with an accessdenied and requested, it uses NTLM for authentication
The client-side usually does not respond in the default
settings, but when the client-side computer is set to“Automatic logon with current username and password”
In an intranet or trusted site, HTTP will take the initiative
to issue an NTLM hash and complete the entire
authentication process.
This means that once the hacker hijack a host, and theinternal network environment is set to Automatic logon
with the current username and password in the intranet,
Hackers can steal the credential of the entire intranet by
sending spearing fishing e-mail attachments with an
HTTP URL path pointing to the infected server throughthe investigation of the group personnel in theintranet.(Figure 3)

Fig. 3. The Attack Simulation Scenario

Acknowledgements
This work was supported by the Ministry of Science andTechnology (MOST) in Taiwan under contract numbers
MOST 109-2218-E-006-014- and MOST 109-2218-E006-510-.
References
1. Jianing Wang and Junyu Zhou, NTLM Relay is dead,Long Live the NTLM Relay, HITBSecConf, 2018.
2. William Mattin, One-Click to OWA, DEF CON 26, 2018.
3. Marina Simakov and Yaron Zinar, Relaying Credentials
Has Never Been Easier: How To Easily Bypass The Latest
NTLM Mitigation, DEF CON 27, 2019.
4. Downloads and the Mark-of-the-Web, 2016
5. I. Ghafir and V. Prenosil and M. Hammoudeh, Dsguised
executable files in spear-phishing emails: detecting thepoint of entry in advanced persistent threat, ICFNDS ’18,
2018
6. Mohammed M. Alani, IoT Lotto: Utilizing IoT Devices inBrute-Force Attacks, ICIT 2018: Proceedings of the 6thInternational Conference on Information Technology: IoTand Smart, pp 140–144, 2018

4. Conclusion
IoT devices not only appear in our lives but also enter ourwork environment. They all make the working
environment more convenient and improve the work
efficiency of employees. These IoT devices often support
NTLM as an identity verification method. Furthermore,
to prevent employees from constantly entering accountpasswords, the device automatically sends the user's
identity verification information. Although this saves theuser's time to enter the account and password, hackers
can also use this authentication method to steal user
identity verification information, and use the identity
verification information to log in to other devices, This
article combines past research results to present our new© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

428

Microsatellite attitude control approach: combined with Generation Adversarial Networks
fault-detection and Cerebellar Model Articulation Controller fault-tolerant control
Ho-Nien Shou

Dep. of Aviation & Communication Electronics, Air Force Institute of Technology
Gangshan, Kaohsiung, Taiwan (R.O.C.)
E-mail: honien.shou@gmail.com

Abstract
In this paper, a new attitude control architecture for microsatellite is proposed. Based on the deep learning faultdetection method, the Cerebellar Model Articulation Controller (CMAC) is used as the fault-tolerant control. With
the function of Generation Adversarial Networks (GAN) to recognize images, the microsatellite attitude fault wavelet
spectrum is used as a guide for the training of generators and discriminators, and for system real-time fault diagnosis
and classification. When the system fault diagnosis determines that the fault occurs, the cerebellar neural network is
involved in fault-tolerant control. The GAN learning ability of the generative confrontation network is used to solve
the problem of insufficient sample data and sample labeling respectively. The CMAC is used as a local learning
network, which has generalization ability, strong convergence speed, and easy software and hardware implementation.
The simulation results show that compared with other methods, the GAN method of fault-detection combined with
CMAC can achieve higher accuracy and robustness.
Keywords: deep learning, fault-detection, fault-tolerant control, Cerebellar Model Articulation Controller,
Generation Adversarial Networks.

layer is added to achieve the multi classification function[2]. Since the SSL was proposed, it has been applied tospeech generation and recognition, image recognition,
sample classification and other fields [3]. The SSL isapplied to human action recognition, and the images arecaptured by the video frame as the network input foraction recognition [4]. In the above application, the SSL
achieves the recognition effect.
The Cerebellar Model Articulation Controller(CMAC) was developed by Dr. J.S. Albus in 1975, basedon the biological model of cerebellar cortex proposed byMarr in Medical Research Institute. CMAC are widelyused because of characters of fast convergence speed,strong generation ability, it can overcome the local
optimization problem of back propagation neuralnetwork (BP), simple structure, easy implementation by

1. Introduction
The Generative Adversarial Network (GAN) was
proposed in 2014 [1]. The GAN continuously optimizes
the network to achieve better results by using the mutual
confrontation between generator and discriminator.
There are many hidden deep learning network models in
GAN, and it has good feature learning ability. At first, it
is mainly used in data generation and image generation
to solve the problem of insufficient sample data. Basedon different research purposes, derivative models for
GANs are constantly being proposed. A SemiSupervised
Learning
withContext-Conditional
Generative Adversarial Networks (SSL) is proposed, inwhich the generator structure remains unchanged and thediscriminator is classified in two categories an output© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

429

Ho-Nien ShouThe SSL routing generator and discriminator arecomposed of the generator and discriminator. The
generator belongs to the generative model. The
difference between the generator and other models is thatthe generator does not need to be modeled in advance.
However, due to the lack of constraints, the generated
data is too free. This paper proposes a kind of
improvement to solve the above problems Based on thesemi supervised generative countermeasure network
algorithm, conditional variable is added to the model toguide the training process of generator and discriminatorby using additional information. Compared with theprototype generated countermeasure network, the maindifference is that the discriminator outputs a class signal(the generated sample is No For the discriminator, theloss function consists of two parts, one is supervised
learning loss (only need to judge whether the sample istrue or false), and the other is unsupervised learning loss
(judging sample category). The generator only needs togenerate realistic samples as far as possible. After
training, the discriminator can be used as a classification
model to classify.
The generator of the improved SSL consists ofseveral deconvolution layers, excitation layers andextracted over fitting Fitting) and speeding up thetraining speed method, for the input n-dimensional noise
signal connecting condition variables, the final output isalmost the same as the real signal "false" signal, which
can achieve the state of "false" with the real signal, whichis conducive to expand the sample to achieve better
training effect. The discriminator consists of several
convolution layers, excitation layers, fully connected
layers and classified output layers, as well as thesoftware and hardware. CMAC has been successfully
applied in many application fields, such as fault detection
and nonlinear control. In the classification task, Lin et al.
[5], a parameter fuzzy CMAC neural network with mixed
parameters is proposed. Learning algorithm composed ofself-clustering genetic algorithm (GA) and improved GA
for facial detection and breast cancer diagnosis.
In recent years, the research on motor bearing fault
diagnosis has been widely concerned by scholars andengineers. The vibration analysis method is commonlyused for bearing fault diagnosis, which analyzes andclassifies the collected bearing vibration signals.
However, bearing vibration signal has the characteristics
of non-stationary, so it is very difficult to extract thecharacteristics of the bearing.
As a time-frequency analysis method, continuous
wavelet transform can effectively extract the timefrequency characteristics of non-stationary signals. The
function of wavelet transform is to take one-dimensionalsignal as input. The result of time-frequency analysis
essentially reflects the two-dimensional spectrum ofenergy intensity of signal at different time and frequency,
and the output is displayed in the form of time-frequency
diagrams. At present, continuous wavelet transform has
been widely used in the field of mechanical fault
diagnosis [6]. In reference, a combination of wavelet
transform branch support vector machine (SVM) andconvolutional neural networks (CNN) was proposed. In
reference [6], wavelet transform and back propagationneural network (BP) were used to realize fault diagnosis
of different motors. In the application of the above
literature, the feature vector is constructed artificiallybased on the time-frequency map, and the learning andexpression ability of the selected neural network islimited [6].
The content structure of this paper is a brief
introduction to Section 1 and a description of SSL in
Section 2 in Section 3, the CMAC is explained, in Section
4 wavelet transform of fault model, in Section 5 attitudefault diagnosis and fault-tolerant control simulation of
microsatellite in Section 6. Finally, the conclusion isdrawn.

Fig. 1. SSL algorithm process and structure
corresponding methods to prevent over fitting and speed
up the training. The discriminator takes the real signal,that is, the signal to be detected and the "false" signal
generated by the generator, as the input at the same time.

2. Semi-Supervised Learning with ContextConditional Generative Adversarial Networks(SSL)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

430

Microsatellite attitude control approach

introduced. Based on the SSL, the model is introduced
and the loss function is optimized and optimized. In orderto guide the training of generator and discriminator, thegenerator and semi supervised learning ability are used tosolve the problem of sample data shortage and sample
marking respectively.

The flow chart of the improved generative
countermeasure network algorithm is shown in Fig. 1.
3. Cerebellar Model Articulation Controller(CMAC)CMAC is a kind of artificial learning network which
imitates the hierarchical storage of information in humancerebellar cortex, and its operation process is basicallyachieved by a series of mapping methods. As shown in
Figure 2, the basic structure of MIMO CMAC with three
inputs and three outputs is shown Fig. 2.

Case1.

0.5, 40 ≤ t < 70 ( sec )fa = 

0

1

(1)

0
f=
a

f=
a
2

3

1
0.8
0.6

Magnitude (rad/sec)

0.4
0.2
0
-0.2
-0.4
-0.6
-0.8
-1
0

Fig. 2. The MIMO CMAC block diagram

20

40

60

80

100

120

140

160

180

Time(sec)

Fig. 3 Fault case 1

The five units of CMAC neural network consist of (i)input space (X), (ii) sensory cell (s), (iii) Association unit
(A), (iv) physical storage unit (P) and (v) output storage
cell (Y) using a series of images. Fig. 2 shows the
network topology vector of three inputs and three output
vectors of CMAC neural network; the input isrepresented by [ x1 , x2 , x3 ] , and the actual output isT[ y , y , y ] . The figure shows an icon for the input
T

d1

d2

d3

mode 1 image, including three input elements[ x , x , x ] and one desired output yd 1 .
T

1

4.

2

3

Fault Model Wavelet Transform

Fig. 4 wavelet transform coefficient diagram of case 1
attitude fault
The fault signal is transformed into frequency time
wavelet coefficient graph (time-frequency diagram) toachieve time-frequency analysis of vibration signal andobtain more comprehensive and detailed information.

The attitude fault diagnosis method of microsatellite
is based on the fault signal generated by deep convolution
and combined with continuous wavelet transform. The
ability of wavelet transform for nonstationary signal andthe function of SSL processing and image recognition are© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

431

Ho-Nien ShouThe feature extraction and image recognition ability of
deep learning GAN are used, and the compressed timefrequency diagram is used as input. The wavelet
transform coefficients of case 1is shown in Fig. 4.

1.5

1

Magnitude (rad)0.5

5. Fault Estimated and Fault Tolerant Control
Simulation of Microsatellite Attitude
The microsatellite dynamic equation is expressed as anonlinear equation−ω× Jω + τ w + τ uJω ={where J = diag J x , J y , J z

0

-0.5

-1

(2)

-1.5
0

} are inertia moments of the

20

40

60

80

100

120

140

160

180

Time(sec)

Fig. 5 Time response of case 1 attitude angle failure andrecoverysatellite along principal axes; ω are the angular velocityof the body-fixed reference frame. τ w are space
disturbance torques and τ u are the control torques along

6. Conclusion

principal axes.
For small attitude angles, the derivation of eq. (2) is
as follows
 x ( t ) = Ax ( t ) + g ( t , x ) + Bu u ( t ) + Bw w ( t ) + Fa f a ( t )According to the numerical simulation results in theprevious section, the feasibility of applying SSL tomicrosatellite fault estimated. Through SSL of GAN
samples and training parameters, the CMAC has good
generalization ability and stability under the brake fault
model.
References y ( t ) = Cx ( t ) + Fs f s

Where state vector x = [ψnonlinear

term

θ

φ

ψ

θ

(3)
T

φ] ,

0




ω0


0


g (t, x ) = ( J z − J y ) ω yω z J x  ( J x − J z ) ω xω z J y ( J y − J x ) ω x ω y J x 

1. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza,
Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron
Courville and Yoshua Bengio, “Generative adversarial
nets,” In Advances in Neural Information Processing
Systems(NIPS), 2014, pp.2672-2680.
2. Salimans T., Goodfellow I., and Zaremba W., “Improved
techniques for training GANs,” In NIPS, 2016, pp. 2234–
2242.
3. Salimans T, Goodfellow I, Zaremba W, et al., “ Improved
techniques for training GANs,” Advances in Neural
Information Processing Systems, 2016, pp. 2234-2242.
4. Lantao Yu, Weinan Zhang, Jun Wang, Yong Yu,“SeqGAN: Sequence Generative Adversarial Nets with
Policy Gradient,” 31th AAAI Conference on Artificial
Intelligence, , 2017, pp.2852-2858.
5. C.J. Lin, J.H. Lee, C.Y. Lee, A novel hybrid learningalgorithm for parametric fuzzy CMAC networks and its
classification applications, Expert Systems with
Applications 35, 2008, pp.1711–1720.
6. Yuan Jian-hu, Han Tao, Tang Jian, et al. “An approach tointelligent fault diagnosis of rolling bearing using wavelet
time-frequency representations and CNN,” Machine
Design and Research 33(2), 2017, pp.93-97.

,

T0 0 0 1 0 0 
actuator fault matrix Fa =  0 0 0 0 1 0  and 0 0 0 0 0 1 actuator fault vector f a =  f a

1

fa

T

2

f a  .
3© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

432

Contribution to the Theory of Periodic Reaction of Three Bodies Systems

Yasuhiro Suzuki

Graduate School of Informatics, Nagoya University, Furocho Chikusa
Nagoya City, Aichi Prefecture 464/0814, Japan *E-mail: ysuzuki@nagoya-u.jp.com
www.ysuzuki.info

Abstract
The chemical ecosystem is an ecosystem in which chemical substances mediate interactions. When herbivores feed,
the plant analyzes herbivore saliva and produces volatile chemicals that attract herbivore natural enemies. Natural
enemies are attracted to volatile chemicals and eliminate herbivores. This system is a tripartite system consisting of
herbivore, carnivore and volatile chemicals. The basic equation of mathematical ecology is the Lotoka-Volterra
equation. This equation is a two-way system of herbivores and carnivores. This paper proposes the basic equationsof three chemical ecosystems and shows their mathematical features.
Keywords: Chemical Ecology, Lotoka-Volterra equation, Tritrophic system, mathematicalbiology

1. Introduction

The equilibrium points of this simultaneous differential
equation are obtained from 3) and 4)

Mathematical ecology has used the Lotka-Volterra
equation as the basic equation1). This equation is asimultaneous differential equation with the predator as yand the prey as x.
𝑑𝑑𝑑𝑑= 𝑘𝑘1 𝑥𝑥 − 𝑘𝑘2 𝑥𝑥𝑥𝑥,
𝑑𝑑𝑑𝑑
𝑑𝑑𝑑𝑑
= 𝑘𝑘1 𝑥𝑥 − 𝑘𝑘2 𝑥𝑥𝑥𝑥,
𝑑𝑑𝑑𝑑

𝑑𝑑𝑑𝑑
= 𝑥𝑥(𝑘𝑘1 − 𝑘𝑘2 𝑦𝑦),
𝑑𝑑𝑑𝑑
𝑑𝑑𝑑𝑑
0=
= 𝑦𝑦(𝑘𝑘2 𝑥𝑥 − 𝑘𝑘3 ),
𝑑𝑑𝑑𝑑
0=

(1)
as
(2)

(3)
(4)

..

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

433

Yasuhiro Suzuki

exterminated using pesticides, the side effect is toexterminate the pests to be exterminated and the natural
enemies existing in the space. In that case, if pests invade
the place where the natural enemy was exterminatedagain, the natural enemy cannot control it. As a result,plants suffer great feeding damage. To avoid this, it isnecessary to spray pesticides over a wider area, and theecosystem will be seriously damaged.
On the other hand, if a plant that has been damaged byfeeding produces HIPV and provides indirect defense bynatural enemies, the ecosystem will not be seriously
damaged. Plants have their natural enemies exterminate
pests in return for the cost of producing scents. Natural
enemies can increase the efficiency of finding pests inreturn for the cost of learning scents. However, pests arenatural enemies if they eat plants. It is unilaterally
disadvantageous because it is eaten by.
In reality, pests are not unilaterally disadvantaged.
Plants do not produce HIPV immediately after being
damaged by feeding. Plants do not produce HIPV until
they are about to die. Therefore, there is time to sparefrom the start of feeding to the invasion of natural
enemies. Adult females spawn until the invasion ofnatural enemies and move to other safe strains before
invading natural enemies. Therefore, pests get a space
where natural enemies do not exist in return for the costof attracting natural enemies and it can lay eggs safely.

Therefore, if y is larger than k1 / k2, x decreases, and if y
is smaller, it increases. On the other hand, if x is larger
than k3 / k2, y increases, and if it is smaller, y decreases.
Therefore, x and y oscillate around the equilibrium point.
The solution to this equation is periodic1) (Fig. 1). The
Lotoka Volterra equation is a basic equation ofmathematical ecology, and various variants have been
proposed (such as 2)).

Fig. 1. The isocline analysis of LV equation

1.2. Chemical Ecology, rule based expression
1.1. Chemical Ecology
Plants are the foundation and resources that support theentire ecosystem, but on the other hand, indirect defense
using pests and communication between plants have been
revealed. [3]Since the late 1980s, it has become clear that plants
attract predators' natural enemies by releasing volatile
substances in response to feeding damage [3]. In this case,there is an indirect interrelationship through odor
information between plants and natural enemies that are
not in an “eat-eat” relationship that goes beyond thenutritional stage. Plant-predators-predators span threenutritional stages and are linked not only in the foodchain but also in exchanging information.
Volatility produced by plants is called Herbivore
Induced Plant Volatiles (HIPV). When pests are directly

Fig. 2. Schematic rule based model of the tritrophic systemcomposed of Plant-Herbivores and Carnivore© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

434

Contribution to the Theory

interaction between prey and predator by HIPV is theessence of the chemical ecosystem. Therefore, thechemical ecosystem is an interaction of three bodies.
The Lotka-Volterra equation cannot describe thechemical ecosystem in essence.
Lotka-Volterra
equations with multi-bodies have been proposed [4]. Then species Lotka-Volterra equation has two or more
predators or prey but they are essentially two-bodyinteraction, not an n (n> 2) bodies interactions. Therefore,
the generalized Lotka-Volterra equation cannot
essentially describe the chemical ecosystem.
Let x be the number of prey, y be the predator, and i be
the information (HIPV). x, y, i are all positive numbers
greater than or equal to 0.
Prey increases by eating plants and decreases by
predation by predators. Unlike the LV equation, when aprey eats a plant, the plant produces HIPV, which isrepresented by i. Predators cannot find any prey if thedensity of HIPV is low. In addition, the number of preyis reduced by being preyed on by predators. HIPV plays
the role of stabilizer, predators cannot have prey as muchas possible. Because, if predators get many preys, theconcentration of HIPV becomes low and predators
cannot find preys. The basic equation of the chemical
ecosystem based on the LV equation can be obtained
from the above.

Describe the chemical ecosystem on a rule based (Fig.2).
The rule base is an abstract chemical reaction system. An
abstract chemical reaction system is a method of
describing a phenomenon using the notation of achemical formula. Reaction kinetics transforms chemical
equations into differential equations.
Chemical species used in chemical ecosystems are plants,
prey, predators and HIPV.
Plant + Predator-> Prey, Predator, HIPVThis reaction expresses that the prey (herbivore) eats theplant to increase the population and the plant produces
HIPV.
HIPV + prey (carnivorous) + predator-> predator,
predator
This reaction expresses that HIPV-induced predators eat
prey and increase populations. Plants cease to produce
HIPV as there are no more herbivores. Therefore, there
is no HIPV on the right side of this equation.
Predator->
This reaction represents the natural death of a predator.
The reason for not including the natural death of prey isthat the number of prey is decreasing too much. Counting
the increase and decrease of predators with the above
three chemical formulas, it is 0. The increase / decreasein predators is 0. Therefore, the increase / decrease inprey and predator is balanced. If the natural death of theprey is added, the prey will decrease and become extinct
even if it is not predated. Its extinction is not essential,
but the balance of stoichiometry. On the other hand,
without the natural death of predators, the number of
predators increases exponentially. This increase is also a
problem of stoichiometric balance and is not essential.
Therefore, there was no natural death of the predator, andonly the predator died of natural causes.

𝑑𝑑𝑑𝑑= 𝑘𝑘1 𝑥𝑥 − 𝑘𝑘2 𝑥𝑥𝑥𝑥𝑥𝑥,
𝑑𝑑𝑑𝑑
𝑑𝑑𝑑𝑑
= 𝑘𝑘2 𝑥𝑥𝑥𝑥𝑥𝑥 − 𝑘𝑘3 𝑦𝑦,
𝑑𝑑𝑑𝑑
𝑑𝑑𝑑𝑑
= 𝑘𝑘1 𝑥𝑥 − 𝑘𝑘2 𝑥𝑥𝑥𝑥𝑥𝑥.
𝑑𝑑𝑑𝑑

(5)
(6)
(7)

What makes this equation essentially different fromthe LV equation is that it includes a term for three-bodyinteractions.
3.

2. Basic equation of chemical ecosystem

Behavior of basic equations of chemical
ecosystems

The equilibrium points of this equation are

Chemical ecosystems are not just related to prey andpredators. An information chemical called HIPVintervenes between prey and predators. The indirect

0=

𝑑𝑑𝑑𝑑
= 𝑥𝑥(𝑘𝑘1 − 𝑘𝑘2 𝑦𝑦𝑦𝑦),
𝑑𝑑𝑑𝑑© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

435

(8)

Yasuhiro Suzuki

0=

0=

𝑑𝑑𝑑𝑑
= 𝑦𝑦(𝑘𝑘2 𝑥𝑥𝑥𝑥 − 𝑘𝑘3 ),
𝑑𝑑𝑑𝑑
𝑑𝑑𝑑𝑑
= 𝑥𝑥(𝑘𝑘1 − 𝑦𝑦𝑦𝑦2 ),
𝑑𝑑𝑑𝑑

(9)
(10)

and
.
The LV equation has two equilibrium points, but in thisequation the equilibrium points are all points on c / i andare infinite. And when i = 1, the equilibrium point
coincides with the LV equation. When i = 0, x and y haveno equilibrium point. When i = 0, dx / dt> 0 and dy / dt
<0. If k1, k2, and k3 are positive constants, k3 / k2 is c3,
and k1 / k2 is c1, the equilibrium point is (c3 / i, c2 / i,c2).

Fig. 4. Vector space of basic equation of tritrophic system.,
The diagonal is the set of equilibrium points. In reality, theequilibrium points on this diagonal are k / i of the equilibrium
point curve.

sum of these two vectors is exactly the opposite, thevector field shows that the tritrophic basic equation has astable manifold structure. (Fig.4).
From this result, it is foreseen that the basic equationof tritrophic is significantly different from the LV
equation and has an equilibrium solution. In the future, I
would like to proceed with a more rigorous mathematicalanalysis.
Fig. 3. Vector space of LV equation

References
1. Alfred J.Lotoka, Elements of Physical Biology, Williams
and Wilkins Company, 1925.
2. Dimas Martins Afonso, Gjini Erida, ModelingCompetitive Mixtures With the Lotka-Volterra
Framework for More Complex Fitness Assessment
Between Strains, Frontiers in Microbiology, 11, 2132,
2020.
3. Takabayashi J., Dicke M. (1997) Herbivore-Induced Plant
Volatiles with Multifunctional Effects in Ecosystems: A
Complex Pattern of Biotic Interactions. In: Abe T., Levin
S.A., Higashi M. (eds) Biodiversity. Springer, New York,NY. https://doi.org/10.1007/978-1-4612-1906-4_9
4. Kerner, Edward H, Comment on Hamiltonian structures
for the n-dimensional Lotka–Volterra equations, JournalofMathematical
Physics
38,
1218(1997);
https://doi.org/10.1063/1.531875l, Published Online: 04
June 1998

3.1. Vector space of equations
We compared the behaviour of the trytrophic equations
with the LV equation. So we investigated both vector
fields. The LV equation's vector field is a periodic
solution centred on the origin and another equilibrium
point, and this result has been known for a long time (Fig.
3).
The diagonal is the set of equilibrium points. In reality,
the equilibrium points on this diagonal are k / i of theequilibrium point curve. Below this diagonal, theconcentration of HIPV is low, so Y decreases and X
increases. Above this diagonal, the concentration of
HIPV is high, so Y increases and X decreases. Since the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

436

Deep Micro Vibrotactile, DMV and its Applications

Yasuhiro Suzuki

Graduate School of Informatics, Nagoya University, Furocho Chikusa
Nagoya City, Aichi Prefecture 464/0814, Japan *E-mail: ysuzuki@nagoya-u.jp.com
www.ysuzuki.info

Abstract
Deep Micro Vibrotactile, DMV is infrasound. DMVs produce sound effects when mixed with audio. Since the DMV
has a transcendental low frequency, mixing it with sound such as music does not affect the frequency spectrum in the
audible range. However, through demonstration experiments at concerts, it was confirmed that DMV changes thetactile qualities of the sound. This paper introduces the acoustic features and case studies of DMVs
Keywords: Deep Micro Vibrotactile, Infrasonic, very low frequency sound, sound pressure analysis

such as elephants, etc. Artificial objects arise fromoutdoor units of air conditioners, wind turbines for wind
power generation, turbines of thermal power plants,
machine tools, blasts, pipe organs, etc 1).
Infrasound is regarded as noise, and its effects on thehuman body and shielding methods have been
investigated 1). Kuhn, et al. confirmed that infrasoundnear the hearing threshold might induce changes in neuralactivity.
Some of these brain regions are known to be involvedin auditory processing, while others are regarded as key
players in emotional and autonomic control 2).

1. Introduction
Infrasound is a sound generated by vibrations of 20 times
or less per second. In general, the sound loses energy due
to vibration as it passes through the medium. Therefore,high frequencies quickly lose energy and are attenuated,while low frequencies are less attenuated. Therefore, it
can travel up to thousands of kilometres. Low
frequencies are less directional than high frequencies.
Proceed in concentric circles from the source. Even if
there are obstacles, go around 1).
In the natural world, low-frequency sounds are
generated from wind, air vibrations, self, volcanic
activity, avalanches, waterfalls, signals from animals© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

437

Yasuhiro Suzuki

building from becoming structurally stable. Why did it
generate inaudible low-frequency sound at such a cost?
Infrasound is inaudible. Besides, there are no known
musical instruments that generate infrasound other thanpipe organs. If there is an instrument that produces
infrasound, it cannot produce audible sounds. Therefore,
it is natural that music that actively uses infrasound has
not been developed.
In contemporary music, some works use infrasound.
They use electronically generated infrasound. Franco
Schoeman focused on the low frequencies that elephants
use for communication, and composed music using low
frequencies. He has independently developed a notation
for low-frequency composition 5).
In sound sources such as records, infrasound of 20 Hz
or less has been removed. This is because if the soundsource contains infrasound, the housing for music
playback resonates, causing sound deterioration. The
tonearm is a part that picks up sound on a turntable thatplays records. The Tonearm has a record needle andmakes physical contact with the record, so the record
physically vibrates the Tonearm. The resonance between
the vinyl record and the tonearm distorts the sound. It
causes Tonearm resonance when included in a soundsource of 20 Hz or less on a vinyl record 6). Therefore,
frequencies below 20Hz have been removed.

Low-frequency sounds have been used primarily in
medicine. One of the most widely used techniques is
Whole-body vibration (WBV). WBV is a therapeutic
method that exposes the entire body to mechanicaloscillations while the patient stands or sits on a vibrating
platform. WBV appears to affect balance, muscle
function positively, and physical performance, which
would benefit the skeletal muscle's treatment, decline
seen in sarcopenia 3).
In music compose, the lowest note on a pipe organ is
16Hz. The d-Moll Toccata and Fugue BWV 565,
composed by Johann Sebastian Bach (1685-1750),
contains a 16Hz bass.

2. Add a super constant frequency to the soundsourceFourier has shown that a wave with periodicity is the sumof sine waves of any complexity. The Fourier series ofthe periodic function x (t) with the interval (-T / 2, T / 2
as one cycle can be described as follows because theFig. 1. St. Thomas Church, Leipzig and the Pipe-organ. J.S.
Bach worked as a music director from 1723 until his death in
1750. From Wikipedia and this photograph (by S-Kay) is
public domain𝑥𝑥(𝑡𝑡) =Inaudible low frequencies below 20Hz distort thesound in the audible range. Therefore, infrasound is
carefully removed in the production of music software.
On the other hand, as mentioned earlier, in the Middle
Ages, infrasonic was used in music as Bach used.
Acoustic studies have shown that pipe organs cangenerate from 10Hz to 10,000Hz and above 4).
Pipe organs require 7-meter-long pipes to generate
infrasound. Because the low-frequency wavelength islong. Also, because the weight is 150 Kg, the building
sinks after the pipe organ is delivered. It will take a longtime for the pipe organ to stop subsidence and the𝑎𝑎0+ 𝑎𝑎1 cos(𝑡𝑡) + 𝑏𝑏1 sin(𝑡𝑡) + 𝑎𝑎2 cos(2𝑡𝑡)
2+ 𝑏𝑏2 cos(2𝑡𝑡) + ⋯ = 𝑓𝑓(𝑥𝑥)= 𝑎𝑎0∞+ � �𝑎𝑎𝑛𝑛 cos
𝑛𝑛=1

2𝑛𝑛𝑛𝑛𝑛𝑛
2𝑛𝑛𝑛𝑛𝑛𝑛+ 𝑏𝑏𝑛𝑛 sin�𝑇𝑇𝑇𝑇waves to be added have orthogonality (Fig. 3). We
observe the spectrograph of audible part (higher than
20Hz) does not change compared with the case when

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

438

Deep Micro Vibrotactile, DMV

3. Method and Material

15Hz has added (Fig. 3). This is obvious from the Fourier
analysis and pattern of spectrographs do not change.

We analyzed Schumann's second-movement -Symphony No 3 in E-flat major, Op 97, We analyzed thesound intensity by adding a 15 Hz sine wave to this soundsource 8). Besides, frequency spectrum analysis was
performed using Audacity 2.4.2.
3.1. Acoustic sound pressure analysis
We have proposed analyzing the time change of soundFig. 2. Example of frequency spectrograph when infrasonicsound added to a sound. Upper: Frequency Spectrograph of No
infrasound added, Lower: 15Hz added to the sound. A steep
peak of 15Hz shows the added 15Hz. And the frequency
spectrographs higher than 20Hz show no differentiations. We
used the female sample voice in Mexican, which is given as asample of the Audition, which is software produced by Adobe
Co. Ltd.

As mentioned above, low-frequency sound may induce
physical vibration of the playback equipment. Therefore,
when a sound source such as a record or Compact Disc,
CD is expanded by Fourier series, waves of 20 Hz or lessare not included. Therefore, adding infrasound of 20Hzor less to a commercial sound source does not affect eachwave of 20Hz or more.
Add sounds below 20Hz to sounds that do not include
sounds below 20Hz and analyze the frequency spectrum.
The part of the frequency spectrum above 20Hz does notchange before and after adding sound below 20Hz.

Fig. 3. Upper) Autocorrelation of Schumann's symphony,
Bottom) Autocorrelation when 15Hz is put in the same soundsource. Autocorrelation is cube root autocorrelation.

pressure of voice data [8]. The method is described below.
First, the sound pressure is discretized as follows.
𝐹𝐹(𝑝𝑝) = 𝑝𝑝𝑖𝑖 , 𝑖𝑖𝑖𝑖 (𝑏𝑏𝑖𝑖 ≤ 𝑝𝑝 < 𝑏𝑏𝑖𝑖+1 ),© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

439

Yasuhiro Suzuki

a structure when 15 Hz was mixed. Also, acoustic soundpressure analysis showed that mixing 15 Hz made a big
difference.
So far, there has been little scientific evidence about
the use of infrasonic. However, there are relatively many
proven effects. For example, Dave of hard rock band
Metallica5), Dave Pickerel focused on the 16Hz used in
Bach's "Toccata and Fugue" mentioned earlier. And hecreated a subwoofer that was about the size of a guitarcase. Then, I selected a song that contained a lot of deep
bass from their songs and outputs it from this subwoofer.
The super-heavy bass was exposed to whiskey being
brewed in a barrel. A whiskey appraiser has tasted thewhiskey and confirmed that it is a good quality whiskey.
Such examples are used in processes involving
fermentation such as winemaking. However, little
scientific research has been done.
This study showed that adding super-heavy bass tomusic causes a large change in the temporal change of
acoustic sound pressure. This result opens up possibilities
for the use of infrasonic, which was previously unknown.
References
1. Alfred J.Lotoka, Elements of Physical Biology, Williams
and Wilkins Company, 1925.
2. Dimas Martins Afonso, Gjini Erida, ModelingCompetitive Mixtures With the Lotka-Volterra
Framework for More Complex Fitness Assessment
Between Strains, Frontiers in Microbiology, 11, 2132,
2020.
3. Takabayashi J., Dicke M. (1997) Herbivore-Induced Plant
Volatiles with Multifunctional Effects in Ecosystems: A
Complex Pattern of Biotic Interactions. In: Abe T., Levin
S.A., Higashi M. (eds) Biodiversity. Springer, New York,NY. https://doi.org/10.1007/978-1-4612-1906-4_9
4. Kerner, Edward H, Comment on Hamiltonian structures
for the n-dimensional Lotka–Volterra equations, JournalofMathematical
Physics
38,
1218(1997);
https://doi.org/10.1063/1.531875l, Published Online: 04
June 1998
5. Metallica Talk Top-Secret Distilling Process Behind New‘Blackened’ Whiskey, Rolling Stone, SEPTEMBER 15,
2018

Fig. 4. Upper) Acoustic sound pressure analysis of
Schumann's symphony, Bottom) Autocorrelation when 15Hz is
put in the same sound source.

where
𝛽𝛽 =

𝑝𝑝𝑀𝑀𝑀𝑀𝑀𝑀 −𝑝𝑝𝑚𝑚𝑚𝑚𝑚𝑚
𝑛𝑛, 𝑏𝑏𝑖𝑖+1 = 𝑏𝑏𝑖𝑖 + 𝛽𝛽 .

Next, the sound source is divided into t seconds andaveraged. Then, assign bi to pi of each t. From the above,the sound source is converted into a sequence of pi.
Calculate the mean and standard deviation of thissequence of points. Then, plot the mean on the verticalaxis and the standard deviation on the horizontal axis.
Then, the time change of the intensity of the sound sourceis converted into a two-dimensional pattern.
Even if infrasonic was mixed with ordinary music,
there was no difference in the audible part's frequency
spectrum. However, when the autocorrelation wasmeasured, it was confirmed that the autocorrelation had© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

440

Inter-Induce computation and its Philosophical Foundation

Yasuhiro Suzuki

Graduate School of Informatics, Nagoya University, Furocho Chikusa
Nagoya City, Aichi Prefecture 464/0814, Japan *E-mail: ysuzuki@nagoya-u.jp.com
www.ysuzuki.info

Abstract
Set theory is based on the distinguishability of elements. How to recognize and identify the world is the essence ofset theory. If each element cannot be identified, all the elements are one set. So the set does not make sense. The
Heart Sutra is highly rational and can be interpreted mathematically. The mathematical interpretation of the Heart
Sutra shows the divergence of how to discriminate. Based on this world view of Heart Sutra, we propose Inter-Induce
computation, IIC as a novel calculation paradigm that does not depend on set theory. This paper gives an overview
and philosophical foundation of IIC.
Keywords: Inter-Induced computation, Natural computing, Foundation of Computer Science, Heart sutra

Through my research on the origin of life, I finally
understood Kajita's "What is life?"
There are several types of the caligraphy of circles(Enso 円相 in Japanese), for example, the calligraphy ofthe circle of Ikkyu Zen Master is a perfect circle (asshown to the left of the Fig.1). Systematically, this"circle" is in a steady state. Also, since it is a perfect circle,
I do not know where it started. That is, there is no time.
On the other hand, many calligraphies of circles have"beginning" and "end" from the ink marks, and there is aflow of time (Fig.1 right).

1. Introduction
When I started to consider natural computing, when I was
studying "the origin of life" using abstract chemistry, the5th Workshop on Life and Physicality, SOMA 5th~
Theme inter-relationship between "phenomenon" and"things", February 22-23, 2002, I met a priest, Shinsho
Kajita, at (at Honen-in temple, Kyoto).
And he suggested to me that "life is unrelenting." I felt
something related to the "calligraphy of circle" in every
part of the life system and the "circle" of the Zen priest,
but at that time, I could not understand the suggestion.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

441

Yasuhiro Suzuki

meaningless when the theory of element reduction ispursued.
The concept of "colour" is used in the following, but in
Buddhism, "colour" corresponds to vision among the fivesenses. In Chinese, "not unmatch" means "same", buthere "not unmatched" is regarded as inclusion. Because,
if you want to show "colour, 色 and Kuh, 空 are the same",
you can set "colour" = "Kuh". However, in this case,"Kuh" is not specifically shown and becomes undefined.
However, it is possible to define "Kuh" different from"colour" by regarding "Color" ⊆ "Kuh" $ and "Kuh" ⊆"Color" , "Kuh" is truly included in "Color", and "Color"
is truly included in "Kuh". From this equivalent relation,
it is mathematically shown that "color" = "Kuh".
"Colorless Kuh, Kuhless colour" are paired. Still,
regardless of their religious and philological meanings,
mathematically, no matter which one is missing, theidentity cannot be proved, so it is inevitable. This is thedefinition.
"Colorless Kuh" is the definition of the existence of"colour" and "Kuh". Although the existence of "colour"and "Kuh" is defined, this definition claims that "colour"and "Kuh" are the same, and time and dynamics do notoccur from this definition. In terms of Enso, it shows theexistence of Ikkyu's "completely closed perfect circle." It
is the definition of the specific relationship between the
two;

Even if it is a perfect circle, time will flow from one
piece to the other if you cut it somewhere. "Dissipationof information" is equivalent to "breaking" a perfectcircle by "breaking" a structure that is "steady-state" andcreating time.
What if a highly non-linear system with unsteadyFig. 1. Example of calligraphy of circles, left) Ikkyu Zen
Master (Edo period) created a complete circle calligraphy,
right) many calligraphy of circles are not complete circle, which
has a “start point” as breaking point; and also density of a stroke
illustrates the start-end point(unsteady) rows (all) does not take a circular structure? If
not, all systems have a circular structure in infinitedimensional state space, it isn't easy to know what thestate space looks like unless you can see the entire state
space. But if the system has a circular structure, and if it
is low-dimensional (two-dimensional or threedimensional), we can recognize the system.
The calligraphy of a circle that is not closed is
topologically a straight line. Then, the straight line
disappears at the break of the calligraphy of a circle andregenerates forever. We can recognize the calligraphy ofa circle because it is "circular". If it breaks out of the"hanging scroll" as a myriad of straight lines, it is difficult
for us to recognize it. The calligraphy of a circle is an
example of a case where a system of invariant behaviors
can be recognized, such as an attractor of a chaotic
dynamical system.

Color, that is (即是) Kuh,
Kuh, that is (即是) color.
In Chinese, "即是, immediate correction" means "that is".
Therefore, "immediate correction" can be regarded as
implies, ⟹. A ⟹ B by this symbol means "A, that is,
if B". Mathematically, A ⟹ B and B ⟹ A provethat A and B are equivalent. Again, they are opposites,
but they must be mathematically paired because theycannot be shown to be equivalent if either one is missing(this ⟹ can be regarded as a map, but in category theory.
May be regarded as morphism).
In the above, "colour (vision)" of the five senses isdiscussed, but other senses are omitted because the same
discussion is repeated. From this, all "functions of themind" are equal to "Kuh".
The construction of this concept of "Kuh" is logicallyclever (aside from religious discussions), and it does notdefine what "Kuh" is, and what is called "the function of

2. Philosophy of Hear Sutra
The Heart sutra (般若心経)1 is a well-known "sutra".
However, its composition is rational and precise. And it
is suggested that element reductionism causes
bankruptcy. The latter half of the Heart Sutra is aphilosophical suggestion. Still, the first half shows thatthe classification method diverges and becomes© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

442

Inter-Induce Computation and its

list" is a cellular automaton. In cellular automata,components and all interactions are perfectly grasped asa finite number of state transition rules. However, thatdynamics go beyond our perception and creates a new
universe-like world. The universe of cellular automata isvast, and we can only observe it.

the mind" is individual. It is shown that it cannot be
divided into.
If all the "functions of the mind" are equal to "Kuh", theconcepts that conflict with the "functions of the mind"
such as life-destruction, dirt-purification, and increasedecrease are life = destruction = dirt = purification =increase = decrease. = "Kuh", so everything is the same.
In other words, "it cannot be distinguished".
For example, the increase/decrease of "Your juice ismore than mine!" Does not hold if you accept theexistence of "Kuh". Conversely, if we accept theexistence of "Kuh", time-related confrontations such as
life-destruction, dirt-purification, and increase-decrease
do not make sense (immortality and immortality).
Unclean and unclean). Also, as discussed in "colour" and"Kuh", all five senses are the same as "Kuh", so the fivesenses are all the same from visual = auditory = olfactory= taste = tactile = "Kuh" and cannot be distinguished.
The dynamics are "unsteady", that is, unsteady, but inreality, they are all the same "Kuh", and our perception("the function of the mind") is "Kuh" and the visual
image ("colour") is projected. For example, as Jakob
Johann Baron von Uexku2) suggests, recognition differs
depending on the species even in the same field, but inreality, they are the same field.
Attractor-like mechanical structures can sometimes beseen in (low-dimensional) chaos, but their trajectories are
all "solutions" of the same equation. A function does notalways have a unique solution if it does not meet the
Lipschitz condition. In that case, the solution can beinnumerable, and the behaviors of various dynamicalsystems can be seen depending on the parameters.
However, although they are all "unsteady" (unsteady) interms of dynamics, they are really just "solutions" to thesame equation.

3. Framework of Inter-Induced Computation
Based on the Heart Sutra concept, the components of thenatural system are subdivided as much as possible. In
contrast to searching for the ultimate components, in thefollowing, the components of the natural system are notsubdivided but treated as they are. We propose acalculation system.
In formulating the actual nature into a computational
system, Heart sutra suggested it is complicated tounderstand the natural system in element reductionism. It
is premised that classification-analysis is impossible.
Must be. In that case, it does not become a meaningful
set theory. This is because all the elements are the same(one representative source). Therefore, it is necessary tochange the basis for constructing a computational systemfundamentally.
On the other hand, we do not fully understand thenatural system, but we interact and control it to someextent. For example, when communicating with a person,
it is impossible to subdivide the other person intoelements and fully understand them before
communicating.
However, we communicate and compose a society
without completely understanding the other person. Then,as I go out with the other person, I gradually feel that I
can understand the other person. However, neither knows
whether the belief is correct or incorrect (because theycannot know each other's internal state accurately).
Therefore, a framework that guides the target system inthe desired direction while interacting with the target,
such as communication with humans, is called "interinduce computing".
In the following, the guiding side is referred to as the"inducing subject", and the guided side is referred to as
the "inducing target". For the sake of simplicity, in thefollowing, the guiding subject and the guiding target are
independent, autonomous systems. It is possible to argueeven if they are subordinate to each other. Still, in thatOn the other hand, the Heart Sutra suggests thatrecognizing the unchanging natural system leads to the"Kuh." This suggestion is accurate and analytical, andelement-reducing science continues to make "lists" of
how natural systems work, but this lengthy list, as
Yuxcur suggests, can be perceived differently.
Even if a "perfect list" is obtained in the future, it isunlikely that we can understand the natural system that isunrelenting by combining it. An example with a "perfect© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

443

Yasuhiro Suzuki

If there is no object of interaction, the concept of
induction/delusion does not occur. Self-reflection anddelusions are also multiple systems consisting of "selfreflection / delusional subject" and "self-reflection /delusional system" created by that subject. It is a mutual
induction. It is not known in principle (unless it is atranscendental existence) which system is mutual subject
or mutual object, and even how many of them are
unknown.
The essence of induction is that there is an interaction
partner. It doesn't matter if the "other party" actuallyexists. To clarify this, it is called mutual induction.

case, specifically dependent interactions and independentinteractions Since it is necessary to divide into actions
and there are innumerable cases, only independent cases
are dealt with.
And here is the most important point, "It is impossible
to know the internal state of the other party for bothmutual subjects and mutual objects by any method." Theguiding subject has a "direction/state of being guided" asa solution set. Interactions that bring about change arecalled "effective actions", and interactions that do notchange are called "ineffective actions". In the following,
unless otherwise specified, effective action is called an"action".
The inducer "acts" on the induction target. The
induction target changes depending on the "action" andleads to a steady-state (including a dynamic steady-state)
and a normal set. The guiding subject observes thisnormal typeset, and if it matches the guided direction andstate, it is a "correct answer", otherwise it is a "wrong
answer". And from this result, a new "action" may be
performed. The inducer decides whether or not to
perform a new "action".

Isn't mutual induction a feedback control?
The same argument applies to any control system, butwhen controlling, the control subject and the control
target are generally clearly defined, and the internal stateof the control target can be known as a state quantity suchas temperature or velocity, for example. ， In autopoiesis,
it is not possible to clearly separate the control subject
and the object, but it is part of the definition of
autopoiesis, and the discussion, in that case, is similar tomutual guidance. Therefore, in control, the actionresponse of the controlled object is formulated. It can behandled mathematically.
On the other hand, in mutual induction, it is premisedthat the guiding subject and the guiding target cannot be
separated in principle. Other systems can't know theinternal state of the system. It is possible to regard a partof mutual induction as feedback control, etc.. Still, fromthat point alone, it is an over-generalization and an error
to assume that the concept of mutual induction is thesame as feedback control.

4. Remarks
Why is it mutual induction rather than induction?
It is impossible to know the internal state of the otherparty's system with which any natural / life systeminteracts, by any method. Therefore, for convenience, wehave named the guiding subject and the guiding target.
However, the "name" has no essential meaning.
Therefore, the mutual induction system is essentiallycomposed of two or more systems.
It is impossible to determine whether any natural or
biological system is "inducing or being guided". For
example, if I think that someone is "inducing", whether
the object to be guided is "induced by me" or "in reverse,
I am induced", the "action" is established in the first place.
I don't know.
It is impossible to "really know (know the other party's
internal state from a transcendental standpoint)".
For example, a virus needs a "host" because it cannot
propagate on its own. Emerging infectious diseases suchas coronavirus mutate while interacting with the "host".
When the "host" is human, humans make vaccines andnew drugs and "respond" to the "action" of the virus. In
this case, both are the guiding subjects and the guiding
targets.

References
1. Heart Sutra
2. Jakob Johann Baron von Uexküll, Streifzüge durch die
Umwelten von Tieren und Menschen: Ein Bilderbuch
unsichtbarer Welten, 1934

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

444

Digital Testing Device for Active Range of Motion of Finger Joints
Utilizing Artificial Neural Network
Huu-Hieu Quang

Department of Electrical and Mechanical Engineering, Graduate School of Engineering
Nagoya Institute of Technology, Gokiso-cho, Showa-ku, Nagoya, Aichi 466-8555, Japan
Yoshifumi Morita

Department of Electrical and Mechanical Engineering, Graduate School of Engineering
Nagoya Institute of Technology, Gokiso-cho, Showa-ku, Nagoya, Aichi 466-8555, Japan
Makoto Takekawa

everfine, LLC, Japan
Email: 31513006@stn.nitech.ac.jp, morita@nitech.ac.jp,
makoto.takekawa@gmail.com
www.nitech.ac.jp

Abstract
This paper proposed a digital goniometer device for measuring the active range of motion (A-ROM) of finger joints
by artificial neural network (ANN). A therapist typically measures the A-ROM by a conventional goniometer.
However, it consumes a significant amount of time, and imposes a burden on patients. Therefore, we develop a digitalgoniometer using ANN to overcome these problems. The effectiveness of the proposed device is verified based on
the evaluation error on the third joint of the middle finger.
Keywords: Digital goniometer device, Active range of motion, Artificial network, Conventional goniometer.

1. Introduction

measuring methods were proposed by extant studies.3,4 A
VICON system using markers placed on each joint of
the fingers were proposed in order to compute the joint
angles. In addition, a non-invasive automatic testing
device was developed to boost the measurement time andaccuracy. Nevertheless, the developed devices cannot
measure the ROM precisely in a short time, and the
response of the systems during operation is poor. In
addition, a non-invasive automatic testing device wasdeveloped to boost the measurement time and accuracy.
Nevertheless, the developed devices cannot measure theROM precisely in a short time, and the response of thesystems during operation is poor.
In this study, a digital testing goniometer isdeveloped to overcome these problems. A data collection
model is proposed to collect training dataset of the finger
joints. Subsequently, an ANN model is utilized tovalidate the dataset. Hence, it can facilely compute thefinger joint angles. From the viewpoint of improving
measurement accuracy, the effectiveness of the proposed
Digital testing goniometer is evaluated on the third jointof the middle finger in comparison with the conventionalgoniometer.

The range of motion (ROM) is an essential part in clinicalpractices when assessing the effectiveness of therapeutics.
A therapist typically measures the ROM by conventional,
as shown in Fig. 1, or digital goniometers.1,2Fig. 1. A-ROM test with a conventional goniometer.

However, he/she cannot carry out the measurement ina short time by these goniometers. Accordingly, several© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

445

Huu-Hieu Quang, Yoshifumi Morita, Makoto Takekawa

2. A-ROM Testing Method and Device
2.1. Outline of digital testing device
The active range of motion (A-ROM) is the angle whena patient bends or extends his/her own joints as much as
possible. Meanwhile the passive range of motion (PROM) is the joint angles when the therapist bends orextends the patient’s joints as much as possible. Figure 2
shows an overview of digital goniometer device where
the depth camera (Intel RealSense Depth Camera
SR300) was connected to the PC via the USB gate. The
depth image of depth camera has 640 × 480pixel
resolution, and a trigger frequency of 30Hz.

(a) RGB image(b) Depth image
Fig. 3. Interface of system.

3. A-ROM Measurement by ANN Method

Fig. 2. Digital testing goniometer.

3.1. Principle
The points of the proximal interphalangeal,
metacarpophalangeal joints of the middle finger and thecenter point of wrist bones are denoted by PIP3, MP3,
and W, respectively, as shown in Fig. 4. The XYZ-axesof the depth camera coordinate system are shown in Fig.
4. The three-dimensional position coordinates of threepoints are measured with the depth camera. The unit ofX- and Y-axes is pixel. The unit of Z-axis is meter. The
coordinate of each joint is determined by RGB image andDepth image data. It should be noted that the units aredifferent. Subsequently, the joint angle is predicted byusing an ANN model since there is a linear regression
relevance between the joint’s position coordinate andjoint angle.

2.2. A-ROM testing method
In this study, the metacarpophalangeal joint (MP) of the
middle finger is considered. The MP joint is the third
joint and the joint between two bones, namely themetacarpal and the wrist bones. The range of motion of
MP joints in healthy people is 0 to 90 deg (flexion) and 0
to 45 deg (extension). Let the fully extended position be
the neutral position of 0 deg. During A-ROM testing, thetherapist moves the depth camera towards to the patient’s
hand so that the middle finger is into the three-green
boxes, as shown in Fig. 3(a). The three green boxes areoverlaid on the image of RGB camera. The interface
screen shows the taken images on both RGB and depth
modes simultaneously, as shown in Fig. 3.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

446

Digital Testing Device formeasuring the joints of the fingers, the joints were fixedby sandwiching clay between the joints so that the joint
angle of the fingers did not change. Table 1 shows the
MP joint angle measured with the conventionalgoniometer, which are used for constructing a ANN
model and evaluating the accuracy.

Fig. 4. Positions of the phalangeal joints of the middle
finger.

3.2. Data collection
We constructed an ANN model from three-dimensional
position coordinates of W, MP3 and PIP3 joints to the
MP joint angle. We created data collection model for
finger joints position detection by using Hand Keypoints,
which was a well-known open-source model in the fieldof image processing for high predictive accuracy offingers joints recognition.5 Using the image data acquiredby the RGB camera and the position of three points
determined by the Hand Keypoints, the two-dimension
position coordinates of three joints are obtained in pixel
units. Moreover, using the depth data acquired by thedepth camera, the depth coordinates corresponding thetwo-dimensional position coordinates of three points are
obtained in meters.
Moreover, three green boxes are set so that thetherapist can easily locate the patient’s hand, as shown
Fig. 5. The overall size of three boxes is 40 x 120 pixel,
the size of each green box is 40 x 30 pixel, 40 x 60 pixel,
and 40 x 30 pixel from the top to the bottom, respectively.
The position of PIP3, MP3, and W joints are marked byred, blue, and green points, and are denoted by I, J, andK points in Fig. 5, respectively. The MP joint angle wasdetermined by the three points. The input dataset consistsof the coordinates of these points. In addition, thesepoints are features of ANN model.
Ii = (xIi , yIi , zIi ), Ji = �xJi , yJi , zJi �, K i = (xKi , yKi , zKi )(i = 1, 2, . . . , n), n is the number of data contained in thedataset. When the data is acquired for constructing andevaluating a ANN model, the subject moved the hand
until I, J and K points are inside the corresponding green
boxes.
The output of the dataset is the MP joint angle, which
was measured with a conventional goniometer. WhenFig. 5. Processing image to collect data.

Table 1. Measured values of MP joint angle withthe conventional goniometer.

Pose No.

1

2

3

4

5

6

Angle[deg]

38

50

64

68

73

80

3.3. Artificial Neural Network

Fig. 6. Artificial neural network architecture.

The principle of the method is creating a model by ANN.6
The model is determined based on the linear relationshipbetween the joint position and joint angle of sample© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

447

Huu-Hieu Quang, Yoshifumi Morita, Makoto Takekawa

dataset. Subsequently the model is used to predict thejoint angle based on the joint position. We used ANN
model for regression to predict angle value. The ANN
model used Rectified Linear Unit (ReLu) activation
function in both layers. The loss function utilized for
regression was Mean Squared Error (MSE). In addition,
Adam was chosen as the optimizer function. The
architecture of the artificial neural network is shown in
Fig. 6.
ANN model was configured of three middle layers of
250 neurons represented by ReLu activation function. In
addition, a dropout layer was added as a regularization
technique in regression network. The dropout layer
randomly drops some of the inputs to a layer is 0.2.

2.
3.

4.

5.

4. Evaluation of Measurement Predicted
Accuracy
6.

The evaluation of measurement accuracy in A-ROM
testing was conducted to investigate the effectiveness ofthe proposed digital goniometer device. For this purpose,
75% of the dataset was randomly selected forconstructing the ANN model and the remaining 25% was
assigned for validation. The effectiveness of the proposed
ANN model was evaluated base on the 25% dataset.
Figure 7 shows the error between the measured angle
with the conventional goniometer and the predicted angle
by the proposed testing device. The horizontal axis
denotes the trials of 1447. Figure 8 shows the cumulative
histogram of the errors. It was found that the errors of
98.69% and 99.52% trials were less than 2 deg and 5 deg,
respectively.

https://www.sakaimed.co.jp/measurement_analysis/huma
n-body-measurement/hand-goniometer/ (In Japanese).
easyangle, Ito Co., Ltd.,
http://www.medical.itolator.co.jp/product/easyangle/ (InJapanese).
J. R. Cook, N. A. Baker, R. Cham, E. Hale and M. S.
Redfern, Measurements of Wrist and Finger Postures: A
Comparison of Goniometric and Motion CaptureTechniques, Journal of Applied Biomechanics, 23, 2007,
pp.70-78.
Reissner L, Fischer G, List R, Taylor WR, Giovanoli P,
Calcagni M, Minimal detectable difference of the finger
and wrist range of motion: comparison of goniometry and
3D motion analysis, J. Orthop Surg Res, 14(1), 2019.
Simon, Tomas & Joo, Hanbyul & Matthews, Iain &Sheikh, Yaser. Hand Keypoint Detection in Single Images
using Multiview Bootstrapping, Cornell University, USA,2017.
Lau, E.T., Sun, L. & Yang, Q. Modelling, prediction andclassification of student academic performance using
artificial neural networks. SN Appl. Sci. 1, 982, 2019.

5. Conclusion
In order to solve the problems faced when using a
conventional goniometer, we proposed a digitalgoniometer device for measuring the active range of
motion of the MP joint of the middle finger. We
constructed an estimation model of the finger joint byusing RGB and depth data of the finger joint, the Hand
Keypoints, and the ANN model. The estimation accuracyof the MP joint of the middle finger was less than 2 deg.
The effectiveness of the proposed device wasdemonstrated.
In our further study, we plan to extend the proposed
method so that it can be applied to 12 finger joints.
Moreover, we plan to estimate the data collection model
to remove outlier for increase high performance of the
ANN model as well as reduce error value.

Fig. 7. Distribution of the errors of predicted angles.

References
Fig. 8. Histogram of the errors of predicted angles.

1. Goniometer for fingers, SAKAI Medical Co.,Ltd.,© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

448

Position and Force Teaching Method for 6 DoF Manipulator
Using Contact Teaching Tool and Teaching Data Editor
Duy-Do Bui

Department of Electrical and Mechanical Engineering, Graduate School of Engineering,
Nagoya Institute of Technology, Gokiso-cho, Showa-ku, Nagoya, Aichi 466-8555, Japan
Hiroki Tanaka, Quang-Trung Chu, Hideki Inuzuka, Yoshifumi Morita

Department of Electrical and Mechanical Engineering, Graduate School of Engineering,
Nagoya Institute of Technology, Gokiso-cho, Showa-ku, Nagoya, Aichi 466-8555, Japan
Masao Sakai

Industrial Research Center, Aichi Center for Industry and Science Technology
Onda-cho 1-157-1, Kariya-shi Aichi 448-0013, Japan
E-mail: cmc13126@nitech.jp, h.tanaka.683@nitech.jp,
h.inuzuka.776@nitech.jp, morita@nitech.ac.jp, sakai@aichi-inst.jp
www.nitech.ac.jp

Abstract
This paper proposes a direct teaching method for teaching tasks performed by a robot manipulator while the robot’s
tool makes contact with a workpiece. A contact teaching tool designed for accurately grasping the shape of theworkpiece, and a teaching data editor for generating the target trajectory to realize the desired press angle and force,are developed on the assumption that the compliance characteristics of the robot hand are known. The effectivenessof the developed method is confirmed experimentally.
Keywords: Direct teaching method, Robot manipulator, Contact teaching tool, Teaching data editor.

teaching points precisely by the PAWTED since theoperator has to visually checks and adjusts by hand.
Therefore, in this study, a contact teaching tool and ateaching data editor were proposed on the assumptionthat the compliance characteristics of the robot hand are
known. The tool allows us to accurately grasp the shapeof the workpiece just by pressing the teaching tool
against the edges of the workpiece. Subsequently, theeditor will generate the target trajectory to realize thedesired pressing angle and force based on the obtainedteaching data and the robot hand characteristics. The
effectiveness of the proposed method was confirmed
experimentally from the viewpoint of improving theteaching accuracy and reducing the teaching time.

1. Introduction
In the manufacturing industry, efficient robot teaching
technologies are essential to bring out the fullest potentialof robot manipulators. Currently, for the teaching task ofrobots, the teaching pendant is widely employed.
However, it is difficult to teach complex trajectories by
the teaching pendant. Moreover, this method consumes asignificant amount of time to teach robots. Therefore,
new direct teaching methods have been investigated.1,2
To solve these problems, a parallel wire-type teaching
device (PAWTED) was developed to directly teach robot
manipulators using human hands instead of teachingpendants.3 Nonetheless, it is not possible to determine© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

449

Duy-Do Bui, Hiroki Tanaka, Quang-Trung Chu, Hideki Inuzuka, Yoshifumi Morita, Masao Sakai

2. PAWTED with a contact teaching tool
The PAWTED consists of a moving platform, six wires,and a base. The base, which is attached to an industrial
six-degree-of-freedom (6-DoF) manipulator, has six
rotary encoders and six flat spiral springs. The moving
platform is connected to the base by six metal wires. As
a result, the moving platform can move freely in threedimensional space within the working area.
The operation of the PAWTED comprises teachingand playback modes. In the teaching mode, when theoperator moves the moving platform, the system willcompute the position of the end-effector based on thedrawn length of the six wires. Then, the robot will becontrolled to track and maintain a constant distance withthe moving platform. In the playback mode, the moving
platform is fixed to the base. Subsequently, the teaching
trajectory is reproduced by the robot.
In this research, we aimed to develop a contact
teaching tool that allows the teaching position to beobtained easily just by touching the contact teaching tool
to the edge of the workpiece. Therefore, a contact
teaching tool with a design as in Fig. 1 (a) is proposed.
As shown in Fig. 1 (b), 𝑝𝑝𝑎𝑎 and 𝑝𝑝𝑏𝑏 are the two teaching
points of the teaching tools. The midpoint of the line
segment connecting 𝑝𝑝𝑎𝑎 and 𝑝𝑝𝑏𝑏 , which is the rotation
center point of these teaching points is denoted as 𝑝𝑝0 . To
improve the flexibility of the teaching tool, it is designed
so that it can rotate freely around its central axis as
depicted in Fig. 1 (a). Besides, due to the empty v-shaped
structure, the contact teaching tool could be used to makecontact with both straight edge and arc-shaped edge types
workpieces.
In the teaching mode, the contact teaching tool is
attached to a moving platform, as illustrated in Fig. 2.
When the operator touches the contact teaching tool tothe edge of the workpiece, 𝑝𝑝𝑎𝑎 and 𝑝𝑝𝑏𝑏 will make contact
with the edge of the workpiece and the position andorientation of the teaching tool are determined. Afterteaching the trajectory, the editor calculates the trajectoryof the teaching points, and desired trajectory on the
playback mode. In this study, a robot hand withcompliance characteristics and a robot’s tool attached to
the robot hand were simulated as a joystick in the
playback mode. The compliance characteristics of therobot hand are considered to be the compliancecharacteristics of the joystick.

Fig. 1. Contact teaching tool structure

Fig. 2. PAWTED with a contact teaching tool

3. Teaching data editor
A teaching data editor generates the target trajectory torealize the desired pressing angle and force based on theteaching trajectory and the characteristics of the contact
teaching tool and playback tool.
A model diagram of the computing process in orderto obtain the circular arc-shaped trajectory for the robot
is illustrated in Fig. 3. 𝑝𝑝𝑎𝑎𝑎𝑎 and 𝑝𝑝𝑏𝑏𝑏𝑏 are the teachingpositions obtained when the contact teaching tool touches
the edge of the workpiece (𝑖𝑖 = 1, 2, ⋯ , 𝑛𝑛). The rotation
center point of 𝑝𝑝𝑎𝑎𝑎𝑎 and 𝑝𝑝𝑏𝑏𝑏𝑏 is denoted as 𝑝𝑝0𝑖𝑖 . S is theauxiliary circular trajectory created by multiple 𝑝𝑝0𝑖𝑖
points which can be calculated based on the PAWTED
moving platform. 𝐶𝐶0 is the center of S and the hole of theworkpiece. The radii of S and the hole are 𝑟𝑟1 and 𝑟𝑟2 ,respectively. After at least three rotation center points 𝑝𝑝0𝑖𝑖
are obtained, the least squares fitting method is applied
to precisely determine 𝐶𝐶0 and 𝑟𝑟1 from multiple 𝑝𝑝0𝑖𝑖 points.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

450

Position and Force Teachingwhich is attached to 𝑝𝑝𝑎𝑎 , the position and orientation of
the teaching point 𝑝𝑝𝑎𝑎 is calculated to be the desired
contact position and orientation of the robot’s tool. The
position and orientation of 𝐶𝐶𝑅𝑅 as follows:
𝑂𝑂

𝑝𝑝𝐶𝐶
𝑂𝑂
𝑂𝑂
𝐶𝐶𝑅𝑅𝑅𝑅
𝐶𝐶𝑅𝑅 𝑇𝑇 = 𝑝𝑝𝐶𝐶 𝑇𝑇 �𝑝𝑝𝐶𝐶
𝐶𝐶𝑅𝑅 𝑋𝑋 �

01

in which 𝐶𝐶𝑅𝑅𝑇𝑇 is the simultaneous transformation matrix
𝑂𝑂
from frame 𝛴𝛴𝑂𝑂 to 𝛴𝛴𝐶𝐶𝑅𝑅 . 𝑝𝑝𝐶𝐶𝑇𝑇 is the simultaneous
𝑝𝑝𝐶𝐶
transformation matrix from frame 𝛴𝛴𝑂𝑂 to 𝛴𝛴𝑝𝑝𝐶𝐶 . 𝐶𝐶𝑅𝑅 𝑋𝑋
denotes the translation of −𝑙𝑙 in the X axis of frame
𝑝𝑝𝐶𝐶
Σ𝑃𝑃 and 𝐶𝐶𝑅𝑅 𝑅𝑅 denotes the rotation of −𝜑𝜑 about the Y axis
of frame Σ𝑃𝑃 .

Fig. 3. Getting the hole trajectory

𝑝𝑝𝐶𝐶
𝐶𝐶𝑅𝑅 𝑅𝑅 = 𝑅𝑅𝑌𝑌 (𝜑𝜑)𝑝𝑝𝐶𝐶
𝐶𝐶𝑅𝑅𝑋𝑋 = [−𝑙𝑙

0

0]𝑇𝑇

(3)

(4)

Finally, since the simultaneous transformation
matrix from the center of rotation frame 𝛴𝛴𝐶𝐶𝑅𝑅 to the robot
end-effector frame 𝛴𝛴𝑅𝑅 is known as 𝐶𝐶𝑅𝑅𝑅𝑅𝑇𝑇, the target
position and orientation R of the robot at the teaching
points is obtained as:

Fig. 4. Hole-type workpiece and playback tool.

Afterwards, the position of an arbitrary number of theteaching points 𝑝𝑝𝑎𝑎𝑎𝑎 and 𝑝𝑝𝑏𝑏𝑏𝑏 can be determined from S.
Subsequently, the least squares fitting method is adopted
once again to acquire the radius 𝑟𝑟2 of the circular
trajectory that we need to teach the robot. From the newly
obtained teaching trajectory, the tool orientation at thetime of teaching can be changed to the requiredorientation.
In Fig. 4, 𝑝𝑝𝐶𝐶 is denoted as the contact position
between the playback tool and the edge of the workpiece.
𝐶𝐶𝑅𝑅 is the center pivot point of the playback tool and 𝑙𝑙 isthe distance between 𝐶𝐶𝑅𝑅 and 𝑝𝑝𝐶𝐶 . The angle between theline that connect 𝑝𝑝𝐶𝐶 and 𝐶𝐶0 and the playback tool isdenoted as 𝜓𝜓, as shown in Fig. 4 (a). In Fig. 4 (b), 𝐹𝐹 is
the force that the playback tool presses against theworkpiece. 𝜂𝜂 is the angle between the workpiece surface
and the playback tool when viewed from the side of theworkpiece. The inclination angle between thesymmetrical axis of the playback tool and its joystickwhen it is pressed against the workpiece is denoted as 𝜑𝜑.
The pressing force 𝐹𝐹 is adjusted based on 𝜑𝜑. Based on
the compliance characteristics between 𝜑𝜑 and 𝐹𝐹, 𝜑𝜑 canbe approximated as a function of 𝐹𝐹 as follows:
𝜑𝜑 = 𝑓𝑓(𝐹𝐹)(2)

𝑂𝑂 𝐶𝐶𝑅𝑅𝑂𝑂
𝑅𝑅 𝑇𝑇 = 𝐶𝐶𝑅𝑅 𝑇𝑇 𝑅𝑅 𝑇𝑇(5)where 𝑂𝑂𝑅𝑅𝑇𝑇 is the simultaneous transformation matrix fromframe 𝛴𝛴𝑂𝑂 to 𝛴𝛴𝑅𝑅 whose position and orientation are target
coordination of the robot end-effector to realize thedesired trajectory.
4. Experiment
In order to verify the effectiveness of the new directteaching method with the PAWTED, a contact teaching
tool, and a data editor (PAWTED + ce), experiments
were carried out on the 6-DoF industrial robot (DENSO
WAVE, VS-060). Specifically, we have performed theexperiments with the task of teaching a circular arcshaped trajectory for the robot, as shown in Fig. 5 (a).
The experimental results are compared with the other two
methods that used a teaching pendant and PAWTED. For
evaluation purposes, each of the three teaching methods
is performed three times, and their average value is used.
In the teaching mode, to create the same teachingcondition, the operator taught the trajectory that makes
the desired values of 𝑙𝑙 = 45 mm, 𝜓𝜓 = 0°, 𝜂𝜂 = 45°, and𝐹𝐹 = 125 gf for the robot, as illustrated by the red lines inFig. 6-9, respectively. Afterwards, in the playback mode,(1)

In this study, for simplicity of calculation, we chose𝑝𝑝𝑎𝑎 as our teaching point. In the coordinate frame Σ𝑃𝑃© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

451

Duy-Do Bui, Hiroki Tanaka, Quang-Trung Chu, Hideki Inuzuka, Yoshifumi Morita, Masao Sakai

1, the average teaching-editing time in case of using
PAWTED + ce, PAWTED, and the teaching pendant are
11 sec, 29 sec and 104 sec, respectively. PAWTED + ce
method reduced the teaching-editing time significantly.
Table 1. Teaching-editing time for a circular arc trajectory.
Teaching pendant
PAWTEDPAWTED + ce

Average
104 sec
29 sec11 secStandard deviation
46.0 sec
13.0 sec
2.4 sec

5. Conclusion
This study presented a new method for teaching tasks
performed by a robot manipulator while the robot’s tool
makes contacts with the workpiece. We investigated theperformance of the proposed teaching method in the taskof teaching a circular arc-shaped trajectory for theindustrial robot. From the experimental results comparedwith the conventional methods, it has been shown that the
proposed teaching method has the highest teachingposition accuracy and the shortest teaching time. A future
plan is to verify the usefulness of the proposed method in
the actual field. This research was supported by Aichi
Prefectural Government through “Knowledge Hub
Aichi,” Priority Research Project (II) and in theexperiments by Kondo Seisakusho Co., Ltd. and FINETECHNO Co., Ltd.

Fig. 5. Teaching trajectory and conditions.

a joystick is used as a playback tool to perform theevaluation experiments, as illustrated in Fig. 5 (b). Basedon the compliance characteristics of the joystick, thetarget position of the manipulator on the playback modecan be determined according to Eq. (5).
In this study, our objective is optimizing the playback
trajectory so that the robot can make a contact with theworkpiece precisely. The parameters 𝑙𝑙 , 𝜓𝜓, 𝜂𝜂 and 𝐹𝐹 forthe desired positions and orientations of the robot’s tool
are evaluated. The experiment results of 𝑙𝑙, 𝜓𝜓, 𝜂𝜂 and 𝐹𝐹,
which are analyzed with the unequal variances t-test, areshown in Figs. 6-9, respectively. * shows statisticallysignificant at p < 0.05. Regarding the maximum average
errors of each parameter, among the three teaching
methods, the PAWTED + ce has standout performances.
The teaching-editing time of the three teaching
methods for a circular arc trajectory with 11 teaching
points is shown in Table 1. Since the position andorientation of the PAWTED + ce method are determinedjust by touching the teaching tool to the edge of theworkpiece, it is possible to quickly teach any point on theedge of the workpiece for the robot. As shown in Table

Fig. 6. Teaching distance 𝑙𝑙 on
the teaching points.

References
1. S. Masaru, Direct Teaching for Robot, J. Robotics Societyof Japan. 13(5), 1995, pp. 627-628.
2. Y. Mochizuki and T. Matsui, Direct Teaching Method for
Hybrid Control Applying Impedance Control, J. Robotics
Society of Japan. 12(2), 1994, pp. 160-166.
3. S. Sakai, S. Noritaka, M. Yoshifumi, Development of
Parallel Wire Mechanism for Robot Teaching, J. Japan
Society of Mechanical Engineers Series C. 79(807), 2013,
pp. 4321-4329.

Fig. 7. Teaching angle 𝜓𝜓 on
the teaching points.

Fig. 8. Teaching angle 𝜂𝜂 on
the teaching points.

Fig. 9. Teaching force 𝐹𝐹 on
the teaching points.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

452

Haptic Device that Presents Sensation Corresponding to Palm on Back of Hand for
Teleoperation of Robot Hand
Report2: Consideration on Decided Specification
Kyosuke Ushimaru, Noritaka Sato, Yoshifumi Morita

Department of Electrical and Mechanical Engineering, Graduate School of Engineering,
Nagoya Institute of Technology, Gokiso-cho, Showa-ku, Nagoya, Aichi 466-8555, Japan
E-mail: k.ushimaru.884@nitech.jp , sato.noritaka@nitech.ac.jp

Abstract

Recently, teleoperated rescue robots are required. However, it is known that the teleoperation of a robot
hand mounted on a rescue robot is difficult. Therefore, we propose a new haptic device that presents
haptic sensation for teleoperation of a robot hand. The device stimulates the back of the hand instead of
the palm of the operator. Determination of required specifications by an experiment with subjects iswritten in this paper. To design the device, the interval of the stimulation points (i), the diameter of thestimulation point (d), and the force of the stimulation (f) should be optimized. As a result of the experiment,
we found that the accuracy rate is highest, when (i, d, f) = (30mm, 6mm, 0.9kgf). Moreover, we deeply
considered on the decided specification with an additional experiment.
Keywords: Rescue robot, Haptic Device, Teleoperation, Robot Hand, Palm

In this study, we propose a method of substituting thecontact state obtained from the palm of the robot hand tothe back of the operator's hand. Okano et al. [9] haveproposed a method of substituting the tactile presentation
to other parts of the hand. However, Okano et al.
substituted the sole of the foot for the tactile presentationof the whole hand. In contrast, this study substitutes theback of the hand for the tactile presentation of the hand.
In the previous paper[11], we clarified the characteristicof the tactile sensation of the back of the hand in order toformulate the specifications required for the device that
substitutes the back of the hand, as the first step of theresearch.
We prepared three candidates for each of the three typesof stimulus points, i.e., the distance between the stimulus
points, the diameter of the stimulus points, and themagnitude of the stimulus force.

1. Introduction
Since the Great Hanshin-Awaji Earthquake in 1995,
research and development of disaster response robots
have been conducted [1]. In recent years, there has beena demand for disaster response robots that can not only
gather information but also perform tasks [2]. For thisreason, disaster response robots equipped with robotic
hands have begun to be developed so that they can workat disaster sites in the same manner as humans [3].
Since disaster response robots were operated in an
unknown and extreme environment, they are often
controlled by teleoperation [4]. In order to improve theefficiency of teleoperation, it is necessary to provide
feedback to the operator on the state of contact between
the fingers and palms of the robot hand. However, in thefield of robot teleoperation, there are some systems thatprovide feedback on the contact state of the fingers [5][6],but we cannot find systems that provide feedback on thecontact state of the palm.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

453

Kyosuke Ushimaru, Noritaka Sato, Yoshifumi Morita

Fig. 1. Ideal method.

Fig. 4. Experimental scene.

Fig. 2. Proposed method.

Fig. 5. Device for experiment.(Top view).

Fig. 3. Designed device with parameters which are obtainedby experiment.

Fig. 6. Rod of experiment device• The force of the stimulus (force: f)
It is known that human skin sensation has a characteristic
such as a two-point discrimination threshold.
Therefore, in the previous paper [11], we clarified thecharacteristics of the tactile sensation on the back of thehuman hand by subject experiments described in the next
section and determined these specifications. Fig. 3 showsa conceptual view of the device designed by using thespecifications determined by the experiments.

In this paper, we report the overview of the previous
paper and the results of an additional experiment where
the number of the stimulus points are one or two and thesubjects wear gloves to verify whether the tactile
sensation can be presented as a surface instead of a point,
without changing the specifications.
2. Proposed Device
2.1. Overview

2.2. Subject Experiment for Specification

As shown in Fig. 1, it is ideal to present the tactile
sensation on the palm of the hand in order to correctly
convey the state of contact between the robot hand andthe object to the operator. However, this method wouldinterfere with the teleoperation of the robot hand, so we
propose a method of presenting tactile sensations on theback of the hand as shown in Fig. 2. The following
parameters are necessary to design the device:• The interval between the stimulation points(interval: i)• The diameter of the stimulus point (diameter: d)

In this section, we report the optimal combination of theabove parameters determined through a subject
experiment. The procedure of the experiment is asfollows: 10 subjects are prepared, and 10 points on theback of their hands are pressed one by one. The subjects
were asked to answer where on the palm the pressed point
corresponded to on the back of the hand. The interval of
the stimulation points (i), the diameter of the stimulation
point (d), and the force of the stimulation (f) were theparameters of the combination. The combination with thehighest ratio of correct answers is adopted as the required© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

454

Haptic Device that Presents

Fig. 7. Reference image for subject
Fig. 9 Device for experiment.(Top view).

(1) Bare hand (2) Thin grove (3) Thick grove

Fig. 8. Experimental result

Fig.10 Groves for experiment

specification for the device. The objective of theexperiment is to find the combination with the highest
correct ratio among 27 combinations, where thecandidate values of i are 10 mm, 20 mm, and 30 mm, d is
4 mm, 6 mm, and 8 mm, and f is 0.1 kgf, 0.5 kgf, and 0.9
kgf. Fig. 4 shows the experimental scene. Figs. 5 and 6
show the devices for the experiment. Fig. 7 shows theimage for the subject to answer the pressed place, which
is displayed on the monitor.

3.

Experiment with Multiple Stimulation

3.1. Experimental Method
In this section, we report a result of an experiment tocheck whether subjects misidentify the number of points
by pressing the back of their hands simultaneously. In theexperiment, we prepared three subjects, and asked themto press one or two points on the back of their hands andasked them to answer where the pressed points
corresponded to in their palms. We use the combination(i, d, f) = (30 mm, 6 mm, 0.9 kg), which is the required
specification for the device.
The subjects were three males (twenties), and in eachcondition, stimuli were applied to ten points on the backof the hand in a random order. The scene of the subject
experiment is like the situation as shown in Fig. 4. We set
up the experimental device which can press at two points
simultaneously as shown in Fig. 9. The intervals of thepoints are fixed to 30 mm. In this experiment, the subjects
wear gloves to imagine the practical use as shown in Fig.
10. Fig. 11 shows the image for the subject to answer thepressed places.

2.3. Experimental Results
We collected a total of 2,700 data points from 10
subjects, each of whom pressed 10 points. A graph of theresults for i = 30 mm is shown in Fig. 8. Note that otherresults are shown in our previous paper [11]. The verticalaxis shows the average ratio of correct responses for eachcombination, and the error bars show the standard
deviation of the ratio of correct responses. In the graph,three bars are grouped together when d is the same. The
blue, orange, and gray bars represent the cases where f is
0.1 kgf, 0.5 kgf, and 0.9 kgf, respectively.
The experimental results show that the correct response
rate is highest, and the standard deviation is smallest
when (i, d, f) is (30 mm, 6 mm, 0.9 kgf). In this case, theratio of correct answers was 93 % and the standard
deviation was 7 %. Therefore, we decided to design thedevice using this combination since the mean value washighest and the variation was smallest.

3.2. Results and Discussion
We collected data for a total of 135 points, since 15
points were pressed by three subjects in three conditions.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

455

Kyosuke Ushimaru, Noritaka Sato, Yoshifumi Morita

Fig.11 Reference image for subject

We calculated the average ratio of correct answers if thenumber and position of the points pressed corresponded
to the points answered by the subjects. A graph of theresults is shown in Fig. 12.
As a result of the experiment, subjects collectively
recognized the positions of the pressed points similar tothe previous experiment as shown in Fig. 8. Therefore,
we think that the device which is developed with thespecification obtained in the previous experiment may beeffective in the case of multiple stimulations. On theother hand, we think that it is necessary to consider how
to stimulate when one large object touches the robot hand.
If we want the operator to recognize the tactile sensation
as a surface, we may need to set the distance between thestimulus points to less than 30mm, which is thediscrimination threshold of the back of the hand.
Moreover, the accuracy rate was decreased when thesubjects wore thin gloves. Therefore, it is necessary to
pay attention to the selection of the thickness andmaterials of the gloves. We think that it is interesting thatthe accuracy rates of some subjects are increased when
the subjects wore thick gloves. We will verify thisphenomenon with more subjects in the future.

Fig.12 Experiment result
2. S Tadokoro: "Middle-Long Term Issues of Disaster
Robots in Japan", Journal of the Robotics Society of Japan,
Volume 32, Issue 2, Pages 154-161, 2014. (In Japanese)
3. S Tadokoro, et al.: "Overview and Outcome of WRS
Disaster Robotics Category", Journal of the Robotics
Society of Japan, Volume 37, Issue 3, Pages 224-234,
2019. (In Japanese)
4. N Sato, F Matsuno: "User Interface Technology for
Rescue Robot Teleoperation", Journal of the Robotics
Society of Japan, Volume 28, Issue 2, Pages 156-159,
2010. (In Japanese)
5. K Mima, et al.: "Tele-Manipulation with Humanoid Robot
Hand/Arm via Interne", Proceedings of the Japan Joint
Automatic Control Conference, Volume 54, 2K303, 2011.
(In Japanese)
6. K Negishi, et al.: "i-MarioNET: Remote Operated
Humanoid Robot System Controlled by Telexistence
FST(Robotics for Hazardous Fields)", The Proceedings of
JSME annual Conference on Robotics and Mechatronics(Robomec) 1A1-R14, 2013. (In Japanese)
7. K. Minamizawa, et al.：“A PalmWorn Haptic Display for
Bimanual Operations in Virtual Environments”,
Proceedings of International Conference on Human Haptic
Sensing and Touch Enabled Computer Applications, pp458-463, 2008.
8. T Taniguchi, et al.: "Multi-Point Pressure Sensation
Display using Pneumatic Actuators"; Proc. EuroHaptics
2019, LNCS 10894, 58-67, 2018.
9. T Okano, et al.: "Haptic Feedback for Foot Sole Using
Pneumatic Pressure Device", Proc.ASIAGRAPH2016, 36, 2016.3..
10. Wilska, A：“On the vibrational sensitivity in different
regions of the body surface”, Acta Physiologica
Scandinavica, vol. 31, pp. 285-289, 1954.
11. N Sato，K Ushimaru，Y Yoshifumi：” Haptic Device
that Presents Sensation Corresponding to Palm on Back ofHand for Teleoperation of Robot Hand Report 1:
Clarification of Required Specifications by Subject
Experiment”, The 38th Annual Conference of the RSJ,
2I3-03, 2020. (In Japanese)

4. Conclusion
In this study, we propose a method to present the contact
state of the robot hand to the back of the operator's hand
as a substitute for the palm in teleoperation of a disaster
response robot. In this paper, we conducted an additional
experiment with multiple stimulation.
In the future, we will conduct further verification of theexperimental results. After that the device will be
developed and the efficiency of teleoperation of the robot
hand using the fabricated device will be verified.
References
1. S Tadokoro, et al.: "Rescue Robotics", Journal of the
Robotics Society of Japan, Volume 19, Issue 6, Pages 685688, 2001. (In Japanese)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

456

Robot Assisting Water Serving to Disabilities by Voice Control

Yang Chunxin

MIST, Kyushu Institute of Technology, 680-4 Kawazu
Iizuka-shi, Fukuoka 820-8502, Japan
Sakmongkon Chumkamon

MIST, Kyushu Institute of Technology, 680-4 Kawazu
Iizuka-shi, Fukuoka 820-8502, Japan
Eiji Hayashi

MIST, Hayashi Lab, 680-4 Kawazu
Iizuka-shi, Fukuoka 820-8502, Japan
E-mail: chunxin.yang215@mail.kyutech.jp, m-san@mmcs.mse.kyutech.ac.jp
haya@mse.kyutech.ac.jp
www.kyutech.ac.jp

Abstract
ROS is an open-source robot operating system. In this paper, we use ROS to control Conbe robot arm. By
introducing the YOLACT real-time instance segmentation, we trained our own model for Object Detection.
Secondly, the Speech-Recognition system is established through Deep speech and Mozilla Text-To-Speech with
Tacotron2 DDC model. Deep speech is an end-to-end speech system, where deep learning supersedes theseprocessing stages. Combined with a language model, this approach achieves higher performance than traditional
methods on hard speech recognition tasks while also being much simpler. In this way, we create an artificial
intelligence, which accomplished a simple conversation with people. And the voice control system is establishedbased on Speech-Recognition system. In the experiment, we successfully control the robot arm move positions anddo water serving for disabilities by voice command. With this research, voice control robot arm can be apply in thelife support area, it will be more convenient for disabilities in daily life.
Keywords: ROS, Water Serving, Disabilities, YOLACT, Speech-Recognition, voice control, Deepspeec

human body equipment, and the simple service ispossible for the disabled people. By grasping objects
using voice control, such as grasping cups to serve
water,it will be more convenient for these people in dailylife.

1 Introduction
1.1 Background
Robot arm has become an important role inindustrial production and in people's life using theinterplay of robot technology and information
technology.
In this research, the robot arm is able tocommunicate with people accomplished by using
Deepspeech and Mozilla Text-To-Speech with Tacotron2
DDC model. Communication with the arm and serving
water for those who can't drink water on their own by
voice control become possible.

2 System Configuration
The system consists of four parts:（a）Machine Vision
Using YOLACT Real-time Instance Segmentation to
train dataset and detect the object that we need.
（b）Speech Recognition
By using Deepspeech and Mozilla Text-To-Speech,
we create our speech dataset and trained the speechmodel, which successfully created an artificial
intelligence accomplished a simple conversation withpeople and control robot arm to move.
（c）ROS And MoveIt
Using ROS and MoveIt to set robot's posture(position and angle). Using Moveit trac-ik to solve
inverse-kinematics problem for robot.

1.2 Purpose of Research（a）Multipurpose
Voice control robot arms can be applied to home
services, commercial services and industrial production.
（b）Can be used in the life support area
The miniaturized robot arm can be used for the457

Yang Chunxin, Sakmongkon Chumkamon, Eiji Hayashi（d）Conbe Robot Arm

Fig.1 System configuration
3 Machine Vision

Fig.3 Convert Pixel Coordinates To Depth Value

3.1 Object Detection（a）Data collection
In order to detect an object accurately, a large
amount of data are necessary. Therefore, we need tocollect data by photographing each object and eachobject needs to take at least 100 pictures from different
background, colors, angles and distance.
Then, the object will be labeled with themeasurement range and the object name.

In the figure3 ,(0,0) is the point position of object
target in RGB image, and get the parameter of
ppx,ppy,fx,fy from camera internal parameters. In thisway, we can acquire 3-dimensional coordinates of thetarget. Then, we publish markers on this 3-dimensional
coordinates using ROS.
3.2 Test
In the test,it can accurately identify the shortdistance and far distance(2 meter) for cup and straw with
30FPS.The pixel coordinates are displayed in the middleof the object.We can get the real world 3D coordinates
based on this pixel coordinates.

Fig.2 Label The Object（b）Training with YOLACT
YOLACT is a simple, fully-convolutional model
for real-time instance segmentation that achieves 29.8mAP on MS COCO at 33.5 fps evaluated on a single
Titan Xp, which is significantly faster than any
previous competitive approach1).
We trained date using the Resnet101-FPN model,
and complete training until the loss is minimized.

Fig.4 Object Detection Test
4 Speech Recognition
4.1 Deepspeech
Deepspeech is an end-to-end speech system, where
deep learning supersedes these processing stages.
Combined with a language model, this approachachieves higher performance than traditional methods on
hard speech recognition tasks while also being much
simpler3).
We using Deepspeech pre-trained model and add
our own database to fine-tune the model, then trainedwith tensor flow. Currently, we use the Deepspeech 0.9.1
pre-trained model with English alphabet.
(a) Database
For this research, we trained 20 words andsentences that we need. For audio file, we need the audio
in Mono 16K with .Wav format4).

（c）Program
The object detection program uses a case programon YOLACT which is eval.py with trained data2).In orderto do water serving, we need to let robot know theposition of object, therefore, we add coordinates on thetarget using opencv centroid, calculate the coordinates of
the center point from the coordinates of the boundingbox. But in this way we just acquire 2-dimensional
coordinates on the target. The real coordinates of thetarget point are obtained by recognizing the pixel
coordinates and depth values of the target.

458

Robot Assisting Water Serving

5.1 Simulation With MoveIt
Moveit is a Motion Planning Framework for
manipulators. The main use is calculate the path to thespecified position and angle of the gripper. We use voicecontrol such as saying“please give me water”to call
Python scripts and then control robot to serve water.

Fig.5 Database of Deepspeech
4.2 Mozilla TTS
Mozilla TTS is a deep learning based Text To
Speech project.We use the TTS pre-trainedmodel,Tacotron2 Double Decoder Consistency for oursystem to get response from our speech.Tacotron2 is aneural network architecture for speech synthesis directlyfrom text and it faster than real-time inference
performance5).

Fig.8 Setting Robot Position Using Voice Control
5.2 Water Serving With Real Robot
The Water Serving program uses MoveIt Command
Interface. By setting the joint position and get target
position from YOLACT, Moveit will calculate the pathand inverse-kinematics for robot planning group. Robot
will first pick straw into the cup and then bring cup to
people.

4.3 Speech Recognition Program
To recognize people's language correctly :（a）The language need to be record and saved as anaudio file.
（b）Identify text from the audio files.
（c）Then select the corresponding answer text based on
the recognized text.
（d）Convert selected answers to voice.

Fig.9 Water Serving

Fig.6 Construction Of Speech Conversation Program
4.4 Speech Recognition Test
In the test, we can have a simple conversation withthis AI, it can recognize our words correctly andresponse based on our words. For example, If we say“pick and place”, it will print the response and answer”
right away”.For trained words and sentences, the correct
rate is over 90%.

Fig.10 Pick Cup With Real Robot
6 Future WorkIn the future, we would like to create more task forthe robot, such as making coffee and tea. And we want to
use Soft-Actor-Critic Reinforcement Learning method
for our robot in order to make robot do pick and place
object faster and more accurate.
For speech recognition part, we will create a interface
with webserver, and a virtual character, people cancommunicate with robot through this interface and using
voice control easier.

4.5 Voice control
Voice control for robot use this program:“os.system("gnome-terminal -e 'rosrun
conbe_moveit_config moveit_fk_demo.py'")”This program will run the python scripts, in thispython file the pose and movement of robot is defined.
When we speak the command ， it will run thecorresponding python scripts.

References
1.https://arxiv.org/abs/1904.02689

5 Experiment2.https://github.com/dbolya/yolact459

Yang Chunxin, Sakmongkon Chumkamon, Eiji Hayashi

3.https://arxiv.org/abs/1412.5567
4.https://medium.com/visionwizard/train-your-own-speech-recognition-model-in-5-simple-steps-512d5ac348a5
5.https://arxiv.org/abs/1712.05884
6.https://github.com/Eruvae/yolact_ros

460

A Method of Role Differentiation Using a State Space Filter
with a Waveform Changing Parameter in Multi-agent Reinforcement Learning
Masato Nagayoshi, Simon Elderton

Niigata College of Nursing, 240 shinnan-cho
Joetsu, Niigata 943-0147, Japan
E-mail: nagayosi@niigata-cn.ac.jp, elderton@niigata-cn.ac.jp
Hisashi Tamaki

Kobe University, 1-1 Rokkodai-cho, Nada-ku,
Kobe, Hyogo 657-8501, Japan
E-mail: tamaki@al.cs.kobe-u.ac.jp

Abstract
Recently, there have been many studies on the multi-agent reinforcement learning (MARL) in which eachautonomous agent obtains its own control rule by RL. Here, it is considered that different agents having individuality
is more effective than uniform agents in terms of role differentiation in MARL. Then, we have proposed a promoting
method of role differentiation using a waveform changing parameter in MARL. In this paper, we confirm theeffectiveness of role differentiation by introducing the waveform changing parameter into a state space filter throughcomputational experiments using "Pursuit Game" as one of multi-agent tasks.
Keywords: reinforcement learning, role differentiation, meta-parameter, waveform changing, state space filter

agents. In order to produce differences in interpretationsof the agents’ experiences, we utilized Beck’s “Cognitive
distortions” [2], which is a cognitive therapy.
Then we have proposed a “fluctuation parameter”
which is a wave-form changing meta-parameter in orderto realize “Disqualifying the positive” * which is one of
the “Cognitive distortions”, and a promoting method ofrole differentiation using the fluctuation parameter in
MARL [3].
In this paper, we introduce the “fluctuation parameter”
into a state space filter [4] in order to realize“Overgeneralizing” † which is one of the “Cognitive
distortions”, and confirm the effectiveness of roledifferentiation by introducing the fluctuation parameter
into the state space filter through computational
experiments using "Pursuit Game" as one of multi-agent
tasks.

1. Introduction
Engineers and researchers are paying more attention toreinforcement learning (RL) [1] as a key technique forrealizing computational intelligence such as adaptive andautonomous decentralized systems. Recently, there havebeen many studies on multi-agent reinforcement learning(MARL) in which each autonomous agent obtains its
own control rule by RL. Then, we hypothesize thatdifferent agents having individuality is more effective
than uniform agents in terms of role differentiation in
MARL. Here, we define “individuality” in this paper as
being able to be externally observed, but not a difference
that we are incapable observing, such as a difference ofinternal construction.
We consider that differences in interpretations ofexperiences in the early stages of learning have a great
effect on the creation of individuality of autonomous© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

461

Masato Nagayoshi, Simon Elderton, Hisashi Tamaki

2. Q-learning

The state space filter is adjusted by treating this entropyH(s) as an index of a correctness of state aggregation in
the state s. In particular, in case of mapping from theinner state space roughly digitized to the inner state space,a perceptual aliasing problem is happened. That is, theaction which an agent should select cannot be identified
clearly. Thus, the entropy may not be small in the state
space should be divided. In this paper, sufficiency of thenumber of learning opportunities is judged using athreshold value 𝜃𝜃L .
Therefore, if the entropy does not get smaller than athreshold value 𝜃𝜃H despite the number of learning
opportunities is sufficient, the state space filter isadjusted by dividing the state due to that the perceptual
aliasing problem is happened.
Similarly, if the entropy is smaller than 𝜃𝜃H in a state s
and a different state mapping from a transited input states’, and representative actions in each other’s states are
same, the state space filter is adjusted by integrating thestates due to that the states is too divided.

In this section, we introduce Q-learning (QL) [5] which
is one of the most popular RL methods. QL works by
calculating the quality of a state-action combination,
namely the Q-value, that gives the expected utility of
performing a given action in a given state. By performing
an action 𝑎𝑎 ∈ 𝑨𝑨Q , where 𝑨𝑨Q ⊂ 𝑨𝑨 is the set of available
actions in QL and 𝑨𝑨 is the action space of the RL agent,
the agent can move from state to state. Each stateprovides the agent with a reward r. The goal of the agent
is to maximize its total reward.
The Q-value is updated according to the following
formula, when the agent is provided with the reward:
𝑄𝑄�𝑠𝑠(𝑡𝑡 − 1), 𝑎𝑎(𝑡𝑡 − 1)�

← 𝑄𝑄�𝑠𝑠(𝑡𝑡 − 1), 𝑎𝑎(𝑡𝑡 − 1)� + 𝛼𝛼Q {𝑟𝑟(𝑡𝑡 − 1)+𝛾𝛾 max 𝑄𝑄(𝑠𝑠(𝑡𝑡), 𝑏𝑏) − 𝑄𝑄(𝑠𝑠(𝑡𝑡 − 1), 𝑎𝑎(𝑡𝑡 − 1))} (2)𝑏𝑏∈𝑨𝑨Q

where 𝑄𝑄�𝑠𝑠(𝑡𝑡 − 1), 𝑎𝑎(𝑡𝑡 − 1)� is the Q-value for the state
and the action at the time step 𝑡𝑡 − 1, 𝛼𝛼Q ∈ [0,1] is thelearning rate of QL, 𝛾𝛾 ∈ [0,1] is the discount factor.

4.

RL has meta-parameters 𝜅𝜅 to determine how RL agents
learn control rules. The meta-parameters 𝜅𝜅 include thelearning rate 𝛼𝛼 , the discount factor 𝛽𝛽 , 𝜀𝜀 of 𝜀𝜀 -greedy
which is one of the action selection methods, and thetemperature 𝜏𝜏 of Boltzmann action selection method.

The agent selects an action according to the stochastic
policy 𝜋𝜋(𝑎𝑎|𝑠𝑠), which is based on the Q-value. 𝜋𝜋(𝑎𝑎|𝑠𝑠 )specifies the probabilities of taking each action a in eachstate s. Boltzmann selection, which is one of the typicalaction selection methods, is used in this research.
Therefore, the policy 𝜋𝜋(𝑎𝑎|𝑠𝑠) is calculated as

In this paper, the following fluctuation parameter using
damped vibration function is introduced into this 𝜅𝜅.

exp (𝑄𝑄(𝑠𝑠, 𝑎𝑎)⁄𝜏𝜏)
𝜋𝜋(𝑎𝑎|𝑠𝑠) =(3)∑𝑏𝑏∈𝐴𝐴Q exp (𝑄𝑄(𝑠𝑠, 𝑏𝑏)/ 𝜏𝜏)𝜅𝜅�𝑡𝑡p � =where 𝜏𝜏 is a positive parameter labeled temperature.

�

3. Reinforcement Learning with a State Space
Filter(𝑡𝑡pa < 𝑡𝑡ps )𝜅𝜅 + 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴(2𝜋𝜋(𝑡𝑡p / 𝜆𝜆) + 𝜙𝜙)(4)𝜅𝜅 + 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴(2𝜋𝜋(𝑡𝑡p / 𝜆𝜆) + 𝜙𝜙) × 𝑡𝑡ps /𝑡𝑡pa (𝑜𝑜𝑜𝑜ℎ𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒)

where 𝐴𝐴, 𝑡𝑡p , 𝑡𝑡pa , 𝑡𝑡ps , 𝜆𝜆 and 𝜙𝜙 is the amplitude, the phase,the damped phase, the initial phase of damping, thewavelength, and the initial phase parameter of thefluctuation, respectively. The phase 𝑡𝑡p , the damped
phase 𝑡𝑡pa , the initial phase of damping 𝑡𝑡ps , and thewavelength 𝜆𝜆 are needed to set proper units.

We have proposed a state space filter based on theentropy which is defined by action selection probability
distributions in a state5.
The entropy of action selection probability distributions
using Boltzmann selection in a state H(s) is defined by
H(s) = -(1/log|A|) ∑𝑎𝑎∈𝑨𝑨 𝜋𝜋(a|s)log𝜋𝜋(a|s)

Fluctuation Parameter(2)

5. Computational Experiments

where 𝜋𝜋(a|s) specifies probabilities of taking each action
a in each state s, A is the action space and |A| is thenumber of available actions.

5.1. Pursuit GameThe effectiveness of the proposed approach isinvestigated in this section. It is applied to the so-called© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

462

A Method of Role

5.3. Experiment (A): Learning Rate

Table 1. Parameters for Q-learning with a state
space filter
Parameter
Value
0.1
𝛼𝛼Q
0.9
𝛾𝛾
0.1
𝜏𝜏
0.3
𝜃𝜃H
1,000
𝜃𝜃L

The effectiveness of role differentiation by introducing 3
the fluctuation parameters, in which the initial phase 𝜙𝜙 =0 , the amplitude 𝐴𝐴 = 0.09 , and the wavelength 𝜆𝜆 ={50, 100, 500} [episode] , into the learning rate of QL
with the state space filter (hereafter called “50”, “100”,and “500”, respectively) are investigated in comparison
with an ordinary QL with the state space filter without
fluctuation parameter (hereafter called “constant”). Here,the fluctuation parameters of all agents take the samevalue. The unit of the phase 𝑡𝑡p is set [episode] which isthe same as the wavelength 𝜆𝜆, the unit of the damped
phase 𝑡𝑡pa is set [episode], and the initial phase of damping
is set at 𝑡𝑡ps = 250 [episode]. The range of values which
the fluctuation parameter for 𝛼𝛼Q = 0.1 can take e.g. [0.01,
0.19] on the condition of 𝐴𝐴 = 0.09.

“Pursuit Game” where three RL agents move to capturea randomly moving target object in a discrete 10 ×10
globular grid space. Two or more agents or an agent andthe target object cannot be located at the same cell. At
each step, all agents simultaneously take one of the 5possible actions: moving north, south, east, west or
standing still. A target object is captured when all agentsare located in cells adjacent to the target object andsurrounding the target object in three directions.
The agent has a field of view, and the depth of view set
at 3. Therefore, the agent can observe the surrounding(3 × 2 + 1)2 − 1cells. The agent determines the state by
information within the field of view.
The positive reinforcement signal 𝑟𝑟𝑡𝑡 = 10 (reward) is
given to all agents only when the target object is captured,and the positive reinforcement signal 𝑟𝑟𝑡𝑡 = 1 (subreward) is given to the agent only when the agent islocated in the cell adjacent to the target object and thereinforcement signal 𝑟𝑟𝑡𝑡 = 0 at any other steps. The
period from when all agents and the target object are
randomly located at the start point to when the target
object is captured and all agents are given a reward, orwhen 100, 000 steps have passed is labeled 1 episode.
The period is then repeated.

The average numbers of steps and the average size ofthe state space required to capture the target object wereobserved during learning over 20 simulations withvarious wavelength parameters in the learning rate, as
described in Figs. 3 and 4, respectively.

It can be seen from Figs. 3, 4 that, (1) “50”, “100” and“500” show a better performance than “constant” withregard to the obtained control rule, (2) “50”, “100” and“500” are smaller than “constant” with regard to the sizeof the state space.

5.2. RL Agents
Fig. 3. Required steps of various wavelength parametersin the learning rate (𝜙𝜙 = 0 [rad]).

All agents observe the only target object in order toconfirm the effectiveness of role differentiation, e.g.
moving east of the target object. Therefore, the state
space is constructed with a 1 dimensional space.
Computational experiments have been done withparameters as shown in Table 1. In addition, all initial Qvalues are set at 5.0 as the optimistic initial values, andθH was set referring to about 0.288: the maximal value ofthe entropy when the highest selection probability for one
action is 0.9,

Fig. 4. Required size of the state space filter of various
wavelength parameters in the learning rate (𝜙𝜙 = 0 [rad]).

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

463

Masato Nagayoshi, Simon Elderton, Hisashi Tamaki

It can be seen from Figs. 5, 6 that, (1) “50”, “100” and“500” show a better performance than “constant” withregard to the obtained control rule, (2) “50”, “100” and“500” are smaller than “constant” with regard to the sizeof the state space.
Thus, it could be confirmed that the effectiveness of roledifferentiation by introducing the fluctuation parameter
into the state space filter. It could be considered that thisis the result of overgeneralizing.
Fig. 5. Required steps of various wavelength parametersin the temperature (𝜙𝜙 = 0 [rad]).

6. Conclusion
In this paper, we introduced a “fluctuation parameter”
into a state space filter in order to realize“Overgeneralizing” which is one of the “Cognitive
distortions”. Through computational experiments, we
confirmed the effectiveness of role differentiation byintroducing the fluctuation parameter into the state space
filter. It could be considered that this is the result ofovergeneralizing.
Our future projects include to apply real world problems,e.g. improving a schedule for each nurse.

Fig. 6. Required size of the state space filter of various
wavelength parameters in the temperature (𝜙𝜙 = 0 [rad]).

Acknowledgment

5.4. Example (B): TemperatureThis work was supported by JSPS KAKENHI Grant
Number JP19K04906.

The effectiveness of role differentiation by introducing 3
the fluctuation parameters, in which the initial phase 𝜙𝜙 =0 , the amplitude 𝐴𝐴 = 0.09 , and the wavelength 𝜆𝜆 ={50, 100, 500} [episode] , into the temperature of QL
with the state space filter (hereafter called “50”, “100”,and “500”, respectively) are investigated in comparison
with an ordinary QL with the state space fliter without
fluctuation parameter (hereafter called “constant”). Here,the fluctuation parameters of all agents take the samevalue. The unit of the phase 𝑡𝑡p is set [episode] which isthe same as the wavelength 𝜆𝜆, the unit of the damped
phase 𝑡𝑡pa is set [episode], and the initial phase of damping
is set at 𝑡𝑡ps = 250 [episode]. The range of values which
the fluctuation parameter for τ = 0.1 can take e.g. [0.01,
0.19] on the condition of 𝐴𝐴 = 0.09. If the temperature iszero, then action selection of the agent is greedy and thesituations where agents cannot capture the target object
occur. Therefore, 𝐴𝐴 is set at 0.09

References
1. R. S. Sutton and A. G. Barto, Reinforcement Learning (ABradford Book, MIT Press, Cambridge, 1998).
2. A. T. Beck, Cognitive Therapy and the Emotional
Disorders (International University Press, New York,
1976).
3. M. Nagayoshi, S. J. H. Elderton and H. Tamaki, A
Promoting Method of Role Differentiation using a
Learning Rate that has a Periodically Negative Value inMulti-agent Reinforcement Learning, Journal of Robotics,
Networking and Artificial Life, 6(4), 2020, pp.221–224.
4. M. Nagayoshi, H. Murao and H. Tamaki, A State Space
Filter for Reinforcement Learning, Proc. of AROB 11th'06,
2006, pp.615-618.
5. C. J. C. H. Watkins and P. Dayan, Technical note: QLearning, Machine Learning 8 (1992), 279-292.

The average numbers of steps and the average size ofthe state space required to capture the target object wereobserved during learning over 20 simulations withvarious wavelength parameters in the temperature, as
described in Figs. 5 and 6, respectively.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

464

The research about editing system of performance information for player piano- Make inferences about both handed musical composition by using DP matching system Ryo Kinoshita

Department of Intelligent and Control Systems, Kyushu Institute of Technology,
680-4, kawazu, Iizuka-City, Fukuoka, 820-8502, Japan *Eiji Hayashi

Department of Intelligent and Control Systems, Kyushu Institute of Technology, Hayashi Lab,
680-4, kawazu, Iizuka-City, Fukuoka, 820-8502, Japan
E-mail: kinoshita@mmcs.mse.kyutech.jp, haya@mse.kyutech.ac.jp
http://www.kyutech.ac.jp/

Abstract
We have developed a system that allows a piano to perform automatically. In order to play music in the manner of alive pianist, we must add expression to the piano’s performance. Therefore, we have developed an interactive musicalediting system that utilizes a database to edit music more efficiently.
Keywords: Automatic Piano, Knowledge Database, Computer Music, Music Interface

volume, length, and timing of music. We aimed todevelop a system that, like a skilled pianist, can perform
even the first musical score based on information related
to previous skills and experience. So, we developed asystem that automatically estimates the performance
expression of unedited music using edited performancedata and score data. In this paper, we described a phrase
search using Dynamic Programming (DP) matching anda method for selecting an optimal phrase, how to inferparameters of notes, evaluation of an inferred entire songplayed by both-handed.

1. Introduction
We have developed a system that allows a piano toperform automatically. In this system, 90 actuators havebeen installed on the keys and pedals of a grand piano.
These actuators execute key strokes and pedal
movements to govern the piano’s performance, e.g., “Fig.
1.(Continued)” [1,2].

2. Editing Support System
Fig. 1. The automatic piano

2.1. Performance Information
The automatic piano that we have developed uses a music
data structure that is similar to Musical Instrument
Digital Interface (MIDI). This system edits four

Our research focuses on giving player piano performancewith human-like skill and expression. Playing the piano
expressive by player piano, it is necessary to adjust the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

465

Ryo Kinoshita, Eiji Hayashi

2.4. Select Systemparameters involved in producing a tone: “Velo”(velocity), “Gate”, “Step” and “Time”. “Velo” is the
dynamics, given by the value of 1-127. “Gate” is theduration of the note in milliseconds. “Step” is the intervalof time between notes, and it also exhibits tempo. “Time”
is the time since the sound started.

After done the Search system, there are a lot of similarphrases which has the same DP matching points. Similarphrases are phrases we found by doing DP matching frommusic data. Objective phrase is phrase we want toinference in this time. Then, we need to select the optimalphrases for objective phrases by doing a select system. In
this system, five indicators were used from the viewpointof music theory: Dynamic symbols, Beats and Steps,
Similarity in change of interval, Dynamic symbols andVelo, and Musical forms.

2.2. How to Make the Data for Player Piano
We show the structure of the edit system how to make thedata for player piano in “fig. 2. (Continued)”. The system
extract features of a pianist from the music data that the
pianist has played. Next, the system infers performancedata from the features and score information on a musicthat the pianist has not played.

2.4.1. Dynamic symbols
If the Dynamic symbols are different between similarphrase and objective phrase, the performance is affectedeven if the phrase is the same. Then, the search phrase
and the similar phrase that matches the dynamics on thescore are preferentially selected.
2.4.2. Beat and step
Based on the musical grammar, it is known that strong
beats are closely related to Step [3]. In places considered
to be strong beats, similar phrases are selected using theproperty that the Step value is larger than in other places.
The position of the strong beat depends on the rhythms.

Fig. 2. The structure of the edit system
2.3. Search System
As a result of the analysis, it was found that phrases ofthe same pattern existing in the same tune are performed
in a similar expression (Fig. 3.). This time, we use DP
matching to search for similar phrases for arbitrarily
determined search phrase.

2.4.3. Similarity in change of interval
If the phrases have similar change of interval, it isassumed that the performance expression is also similar.
Therefore, we select a phrase with more similarity inchange of interval.
2.4.4. Dynamic symbols and Velo
There is likely to be a trend in Velo value depending on
dynamic symbols. Therefore, we examine the range of
Velo for each dynamic symbol from performance data,and select a phrase which Velo value is correspond to
search phrase’s dynamic symbol.
2.4.5. Musical forms
If the Select system in 2.4.1.-2.4.4. does not narrow down
the number of similar phrases, the selection is performed
according to the Musical Forms. Musical Forms is amusic format such as the Rond Forms or the Sonata
Forms. For example, if different songs were in the sonata

Fig. 3. The discovery about the relation of same tunes andsimilar expression.

DP matching has a feature that can calculate thesimilarity between two words that are different in a
number of characters from each other.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

466

The research about editing

format, it was predicted that the exposition part had apresentation part feature, the development part had adevelopment part feature, and each part had a uniquefeature. We decide same part has similar performance.

The inference equation for Step is “Eq. (1)”. “PStep”
represent provisional Step value which calculate by
optimal phrase and “NF” represent normalization factor.
Step = PStep × NF × All Tempo Avg.

2.4.5. Case of Left Hand

In the case of left-hand, adjust the timing using theresults of right-had inference when the note is sametiming in musical score.

Left hand phrases can divide four categories: Main theme,
Broken chord, Single, Chord. When select left hand’sphrase, select from same category. If similar phrase does
not exist in same category, the system selects optimalphrase from other categories.

2.5.2. Inference of Velo
The investigation of the ranges and average value about
Velo are shown in “Table 3” by using music data in“Table 1”.

2.5. Infer System
If we use the optimal phrase’s expression to searchphrase, it will be unnatural expression. Hence, thissystem infers appropriate Step, Velo, Gate value for the
search phrase from the optimal phrase.

Table 3. Velo’s average value for each Dynamic symbol

2.5.1. Investigation Required for Inference
For appropriate inference, it need investigation fromperformance data. This year, we use four songs, and its
title is shown in “Table 1”. All of them are made by
W.A.Mozart and Pianist is Maria Joao Pires.

②③④

Title of musical compositions.
Piano Sonata No.11 in A major, K.311 3rd
movement “Turkish March”
Piano Sonata No.15 in C major, K.545 1st
movement “Allegro”
Piano Sonata No.15 in C major, K.545 2nd
movement “Andante”
Piano Sonata No.15 in C major, K.545 3rd
movement “Rondo”

2.5.2. Inference of Step
The investigation of tempo and normalization factor are
shown in “Table 2” by using music data in “Table 1”.
Normalization Factor is s number that “All Tempo
Average” divide “Tempo Average”.

0.80

Normalization Factor

1.11

0.82

1.00

0.92

0.89

0.96

Avg.
(left-hand)

p
mf

64
62

44
36

f

64

45

2.5.2. Inference of Gate

(2)
(3)
(4)

The investigation of factors for each musical symbol thatgives a change in the length of a note are shown in “Table4” by using music data in “Table 1”.

0.89
1.07

Avg.
(right-hand)Velo(n) = average in dynamics symbol
Velo(n) = Velo(n − 1)
Velo(n) = Velo(n − 1) − SVelo′ s value

Table 2. Investigation about tempo and Normalization Factor
Music Number①②③④
Tempo Avg.
All Tempo Avg.

Dynamic
symbolWhen the system infers Velo’s value, it uses similarinterval phrase. Similar interval phrase is the phrase withthe highest similarity of interval changes among similarphrases.
The inference equations for Velo are “Eq. (2)” and “Eq.
(3)”, “Eq. (4)”. “n” represents the number of notes inobjective phrase and “SVelo” represents similar intervalphrase’s value of Velo. Eq. (2) is used when the first notein the phrase and the dynamic symbol is different fromthe previous note. Depending on the results of DP
matching, the search phrase may have to be split. In thiscase, the first note of split phrase is calculated by Eq. (3)except case of Eq (2). Other notes are calculated by Eq(4).

Table 1. The studied title of musical compositions.

①(1)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

467

Ryo Kinoshita, Eiji Hayashi

Table 4. Factor for each musical symbolMusical symbols
Factor

staccato
0.4

No symbol
0.9

slur
1

The inference equation for Gate is “Eq. (5)”.
Gate = inferred Step × Factor

3. Inference Experiment(5)

An experiment was conducted to compare the music
reproduced using the editing support system with theperformance of the pianist. The target song is Mozart
Piano Sonata No.11 in A major, K.311 1st movement“Theme”. Some of the inference result are shown asgraphs. “Fig. 4” is about right-handed and “Fig. 5”is lefthanded Step. “Fig. 6” is about right-handed Gate. “Fig. 7”
is about right-handed Velo.

Fig. 7. Inference result about right-handed Velo
4. Consideration
We can confirm that inferred Step value is closer to the
pianist’s performance than unedited data. It is common
to both hands and the same could be seen in Gate. Thus,
the results suggest that infer system about Step and Gate
are effective. However, inferred Velo’s value is notsimilar to performance data. Therefore, infer system
about Velo is need to improve.
5. Conclusion
This year, we introduced a left-handed inference system
and Gate’s inference system in the editing system of
performance information. The inference experimentshowed that we were able to infer a performance
expression in an unperformed song from data ofperformance.
Editing system of performance information is infer aphrase from similar phrase in other songs. And, when thesame phrase is repeated, the infer result is exactly same.
However, pianists should subtly change in inflection andtiming for the same phrase. Furthermore, the current
system can’t infer performance information when thescore doesn’t have dynamic symbols. In order to solvethese problems, we need to develop a new versatilesystem that adds other methods to current system.

Fig. 4 Inference result about right-handed Step

Fig. 5. Inference result about left-handed Step

References
1. Y. Isomichi and N. Ogawa, Pattern Matching by Using
Dynamic Progaming, 1975, Vol 16, No.1, pp.15-22.
2. 2. E.Hayashi, M.Yamane and H.Mori, Development ofMoving Coil Actuator for an Automatic Piano,
Inrwenational Journal of Japan Society for Precision
Engineering, Vol.28 No.2, 1994, pp.164-169.
3. Shimaoka Jo, Analize about Harmony and Musical style,
ONGAKU NO TOMO SHA CORP, 1964.

Fig. 6. Inference result about right-handed Gate

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

468

Development of LiDAR Based Navigation System for Automation of Tree Harvesting Process
Ayumu Tominaga *Graduate School of Computer Science and System Engineering, Kyushu Institute of Technology, 680-4 Kawazu,
Iizuka, 820-0044, Fukuoka, Japan †Akihiro Koubara

Graduate School of Computer Science and System Engineering, Kyushu Institute of Technology, 680-4 Kawazu,
Iizuka, 820-0044, Fukuoka, Japan
Ryusuke Fujisawa

Graduate School of Computer Science and System Engineering, Kyushu Institute of Technology, 680-4 Kawazu,
Iizuka, 820-0044, Fukuoka, Japan
Eiji Hayashi

Graduate School of Computer Science and System Engineering, Kyushu Institute of Technology, 680-4 Kawazu,
Iizuka, 820-0044, Fukuoka, Japan
Abbe Mowshowitz

Iizuka, 820-0044, Fukuoka, Japan
E-mail: tominaga@mmcs.mse.kyutech.ac.jp, koubara.akihiro880@mail.kyutech.ac.jp, fujisawa@ces.kyutech.ac.jp,
haya@mse.kyutech.ac.jp

Abstract

This study focuses on an autonomous moving system for the automation of the harvesting process byhigh-performance machines in the forestry. Many fatal accidents occur due to the harvesting process. In
this research, a navigation system has been developed to enable autonomous travel between accumulationsites and trees to be harvested to improve productivity and safety. A 3D map is generated by LiDAR
observation, and harvester moves autonomously towards the tree as specified by the operator. A test of
the harvesting process was performed in an experimental environment. The evaluation focused on therequired time of the autonomous movement in the process. The effectiveness of the system was confirmed
in operations such as row thinning by the results.
Keywords: Field Robot, Forest Industry, Harvesting, Autonomous Moving

1. Introduction
Forestry is an industry that nurtures and economicallyutilizes forest. The safety of forestry work has become anissue, and the development and proliferation of highperformance forestry machinery has been increasing. In
the future, it is necessary to introduce robotics
technology to further improve safety and reduce laborburdens by utilizing the mobility applied to the forest
environment that has been developed so far.

In this study, we developed a navigation system in theforest environment that is aimed to be operational for the
harvesting process. The management style of Japanese
forestry is wide-area, small-scale afforestation.

.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

469

Ayumu Tominaga, Akihiro Koubara, Ryusuke Fujisawa, Eiji Hayashi

Fig. 1. The scenario of automatic harvesting
Conventional manned machines require long-term, andcontinuously tree felling and transporting of felled tree,
which places a large burden on the operator and leads toserious accidents. The proposed system would navigatethat to approach to the target tree selected by the operator
autonomously. This navigation was constructed using
LiDAR and LiDAR based SLAM which could generate
the 3D map and self-localization. We demonstrated in theexperimental field and verification of the navigation
system.

Fig. 2. Forestry robot "SOMA"
select the target tree, it takes for the felled tree to belowered to an unobstructed position and for the operator
to carried-out the operation in 𝜏𝜏 [sec]. The total time 𝑇𝑇𝑁𝑁
[sec] required to complete the harvesting of the wholetrees can be estimated by Eq. (2). Here, 𝑇𝑇𝐹𝐹 and 𝜏𝜏 areassumed to be the equal for all trees.

2. Tree harvesting scenario
Fig. 1 shows the scenario of harvesting process assumed
in this study. The details of the scenario are as follows.
1) Selection of a target trees to be felled by theoperator (Fig. 1(A)).
2) Approaching the tree by autonomous movement(Fig. 1(B)).
3) Felling of the tree (Fig. 1(B)).
4) Skidding of the tree by autonomous movement (Fig.
1(C)).
In this scenario, the approach and skidding movements
are automated by the autonomous moving system, thepreparation for the felling and the execution of thesawing are subject to the operator’s operation.
Considering a situation where N trees are lined up in astraight line, and the machine fells and skids them fromthe vicinity of the roadside in order. 𝑇𝑇𝐴𝐴𝐴𝐴 [sec] is a time ittakes to approach the n-th tree, 𝑇𝑇𝑆𝑆𝑆𝑆 [sec] is a time it takes
to skid the felled tree to the roadside, define the time 𝑇𝑇𝑛𝑛[sec] for autonomous movement as 𝑇𝑇𝑛𝑛 = 𝑇𝑇𝐴𝐴𝐴𝐴 + 𝑇𝑇𝑆𝑆𝑆𝑆 . Let
L [m] be the distance of neighborhood tree form theroadside, l [m] be the interval of trees, and v [m/s] is theaverage movement velocity of the machine. 𝑇𝑇𝐴𝐴𝐴𝐴 and 𝑇𝑇𝑆𝑆𝑆𝑆
is approximately equal and can be estimated by Eq. (1).
𝑇𝑇𝐴𝐴𝐴𝐴 = 𝑇𝑇𝑆𝑆𝑆𝑆 = �

𝐿𝐿 + (𝑛𝑛 − 1)𝑙𝑙
�
𝑣𝑣

𝑁𝑁

𝑇𝑇𝑁𝑁 = � 𝑇𝑇𝑇𝑇 + 𝑁𝑁𝑇𝑇𝐹𝐹 + 𝑁𝑁𝑁𝑁
𝑛𝑛=1
𝑁𝑁

𝐿𝐿 + (𝑛𝑛 − 1)𝑙𝑙
� + 𝑁𝑁(𝑇𝑇𝐹𝐹 + 𝜏𝜏)
= 2��
𝑣𝑣

(2)

𝑛𝑛=1

The first term in Eq. (2) is the total time involved inmoving, with the progression of the work, the ratio to thetotal work time ∑𝑁𝑁
𝑛𝑛=1 𝑇𝑇𝑛𝑛 ⁄𝑇𝑇𝑁𝑁 will increase.
3. System structure

In this study, we developed autonomous ground vehicle“SOMA” shown in Fig. 2. SOMA is based on an AllTerrain Vehicle (ATV) that is robust designed to
movement on rough terrain. Fig. 3 shows the system
structure that also includes the operation station. The
surrounding trees are observed using LiDAR (Velodyne,
VLP-16). The robot pose is measured by IMU (Xsense,
MTi-30). In this system, the robot was navigated by
LiDAR based SLAM which combined IMU’s output.

(1)

3.1. HDL Graph SLAM

The sum of the time to prepare for felling and to executeof felling is defined as 𝑇𝑇𝐹𝐹 [sec], the sum of the time to

The HDL Graph SLAM was developed for thegeneration of 3D map and self-localization using© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

470

Development of LiDAR Based

Table 1. Experimental R
𝑛𝑛
𝑛𝑛1
𝑛𝑛2
𝑛𝑛3

𝑇𝑇𝐴𝐴𝐴𝐴 （s）
3.87
8.37
10.96

𝑇𝑇𝑆𝑆𝑆𝑆 （s）

5.42
8.65
11.14

���
𝑇𝑇
𝐹𝐹 （s）

20.26

Fig. 4. System structure

LiDAR[1]. It is the SLAM that integrates NDT scanmatching[2] using point clouds obtained from LiDAR
and the graph optimization.

and the ratio of autonomous moving time in working timeincreased.
5. Conclusions

3.2. DWA
The motion plans to approach the target tree isperformed by Dynamic Window Approach (DWA)
algorithms[3]. The operator could select the target tree of
harvesting, the position (𝑥𝑥, 𝑦𝑦) of it’s the point will be
global target position. The control input (liner velocity,
and angular velocity) would be determined by DWA.

In this study, the navigation system based on LiDAR
and SLAM was developed to reduce the burden of
workers in the harvesting process. The future work
includes evaluation of the approaching accuracy andmotion planning to achieve a stable and accurate
approach and return to the roadside.

4. Verification experiment

References

We demonstrated the automatic harvesting process toverify the effectiveness of the proposed system in theexperimental field. Fig. 4 shows the setting of the initialsituation of experiment. The 9 imitation trees wereplacement equally interval. Approaching and skidding
performed for the second row. In this experiment, therobot approached the target autonomously andtransitioned to the waiting state for the felling command.
After an appropriate amount of time, the robot returned
to its initial position by issuing a discharge command.
Table 1. Experimental Results the time spent on eachelement work. 250.8 seconds to complete the process of
felling in three targets. The ratio of autonomous moving
time to total work time was 9.3, 14.6 and 19.2% as thework progressed. This suggests that this system is
effective in operations such as row thinning, where thetrees are felled and skidded in sequence. It was confirmedthat the round-trip movement was achieved as expected,

1. K. Koide, J. Miura, and E. Menegatti, A portable threedimensional LIDAR-based system for long-term andwide-area people behavior measurement, Int. J. Adv.
Robot. Syst., 16(2), 2019.
2. P. Biber and W. Strasser, The normal distributionstransform: a new approach to laser scan matching,
Proceedings 2003 IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS 2003), 2013, 3, pp.
2743–2748.
3. D. Fox, W. Burgard, and S. Thrun, The dynamic window
approach to collision avoidance, IEEE Robot. Autom.
Mag., 4(1), 1997, pp. 23–33.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021471

𝜏𝜏n （s）
70.
43.
28.

Dynamic Characteristics Analysis of a Multi-scroll Conservative Chaotic System with
Sinusoidal Nonlinearity
Zhonggao Chen, Minghan SongDepartment of Automation, Tianjin University of Science and Technology,
1038 Dagunanlu Road, Hexi District, Tianjin 300222, PR China
E-mail: * 313263747@qq.com

Abstract
In this paper, a multi-scroll conservative chaotic system with sinusoidal nonlinearity is studied. Based on theoreticalanalysis and numerical analysis, such as equilibrium point and its stability analysis, Lyapunov exponent spectrum
and bifurcation diagram, the system is found to show some complex dynamics. In addition, NIST test also verify thatthe pseudo randomness of the proposed system is satisfactory.
Keywords: multi-scroll, conservative, NIST test

1. Introduction

increasing the number of equilibrium points. Due to theperiodicity of sinusoidal functions, many researchers
design multi-scroll chaotic systems by modifying theexpression of sinusoidal functions or adding control
functions to chaotic systems with sinusoidal
functions15,16,18,20-22.
However, there are few studies on the multi-scroll
attractors of conservative chaotic systems. In this paper, a
new conservative chaotic system is constructed by using
sinusoidal function. Through Lyapunov exponent diagram,bifurcation diagram and phase diagram, it is found that
there are different attractors with different scroll numbersin the case of different initial values.

In 1993, Suykens1 first proposed a method to generate ndouble scroll attractors by adding breakpoints. Comparedwith single scroll chaotic system or double scroll chaotic
system, multi-scroll chaotic system has more control
parameters and corresponding key parameters. In addition,
it can present complex multi-directional grid like vortices
in phase space. The number and shape of vortices can also
be controlled and adjusted by the parameters of the system.
It has more complex dynamic behavior and is more
conducive to image encryption. Therefore, multi-scroll
chaotic systems have attracted many attentions, anddifferent types of multi-scroll chaotic systems have beenreported, such as unidirectional (1D) multi-scroll chaotic
systems2,3, bidirectional (2D) multi-scroll chaotic
systems4-8, three-dimensional (3D) multi scroll chaotic
systems9, and multi-directional multi-scroll chaotic
systems10-12. In addition, the realization of chaotic system
based on FPGA is also studied13,14. In three-dimensional
differential equations, the general jerk system has fewer
terms, so chaotic systems based on general jerk systemhave been widely studied15-19. In practical application, theincrease of vortex number in chaotic system is realized by

2. Construction of Four-dimension Conservative
SystemFirstly, a new four-dimension conservative system isconstructed. It can be expressed as:0−𝑎𝑎
𝐱𝐱 = 𝐽𝐽(𝐱𝐱)∇𝐻𝐻(𝐱𝐱) �
0
−𝑏𝑏𝑏𝑏
.

𝑎𝑎
0
0
0

0
0
0
−𝑐𝑐

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

472

𝑏𝑏𝑏𝑏 𝑥𝑥
0 𝑦𝑦
�� �
𝑐𝑐 𝑧𝑧
0 𝑤𝑤

(1)

Zhonggao Chen, Minghan Song0−𝑎𝑎
Where, 𝐽𝐽(𝐱𝐱) = �0−𝑏𝑏𝑏𝑏

center point; when the equilibrium point is ((2𝑘𝑘 −1)𝜋𝜋, 0, 0, 0) , the corresponding eigenvalue is(4, −4, 4𝑖𝑖, −4𝑖𝑖), and the type of the equilibrium point is an
unstable saddle point.

𝑎𝑎 0 𝑏𝑏𝑏𝑏
𝑥𝑥
𝑦𝑦
0 0
0
� , ∇𝐻𝐻(𝐱𝐱) = � �.
𝑧𝑧
0 0
𝑐𝑐
𝑤𝑤
0 −𝑐𝑐 0

Secondly, set (𝑎𝑎, 𝑏𝑏, 𝑐𝑐) = (4, 4, 4), equation (1) can be
described as:.

𝑥𝑥 = 4𝑦𝑦 + 4𝑦𝑦𝑦𝑦⎧.
𝑦𝑦 = −4𝑥𝑥.
⎨𝑧𝑧. = 4𝑤𝑤⎩𝑤𝑤 = −4𝑥𝑥𝑥𝑥 − 4𝑧𝑧

3. The Dynamic Characters of the System (5)In this part, the dynamic characteristics of the system (5)are analyzed through Lyapunov exponent spectrum andphase diagram .

(2)

3.1. Lyapunov exponent spectrum of the system (5)Set the initial value (𝑥𝑥, 𝑦𝑦, 𝑧𝑧, 𝑤𝑤) = (𝑥𝑥(0), 1, 1, 1) ,Lyapunov exponent diagram of the system (5) with thechange of the initial 𝑥𝑥(0) is shown in Fig. 1.

Where x, y, z and w is the state variable, and a, b, c issystem parameter, respectively.
2.1. Conservative characteristics of system (2)2The divergence of the system (2) isMoreover, the time domain derivative of the energyfunction is.

𝐻𝐻 = ∇𝐻𝐻(𝐱𝐱)𝑇𝑇 𝐽𝐽(𝐱𝐱)∇H(𝐱𝐱) = 0

According to the above analysis, the system satisfies
both Hamiltonian energy conservation and volumeconservation.

指数

𝜕𝜕𝑥𝑥̇ 𝜕𝜕𝑦𝑦̇ 𝜕𝜕𝑧𝑧̇ 𝜕𝜕𝑤𝑤̇
+++=0𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

1

(3)

-1

-2
0

2

4

6

8

10

12

14

16

18

20

Fig. 1. Lyapunov exponent diagram(4)

It can be seen from Fig. 1 that the Lyapunov exponent of
the system (5) is symmetric about the x-axis, and the
maximum exponent is always greater than 0, so the system
is always in a chaotic state. Secondly, the Lyapunov
exponent of the system shows a periodic distribution of
2𝜋𝜋.

The equilibrium point of the system (2) is obtained as(0, 0, 0, 0) , and the corresponding eigenvalue is(6𝑖𝑖, −6𝑖𝑖, 6𝑖𝑖, −6𝑖𝑖). Set ∇𝐻𝐻 = (sin(𝑥𝑥) , 𝑦𝑦, 𝑧𝑧, 𝑤𝑤), an infinite
number of equilibrium points are obtained. And equation(2) can be express as:
.

-0.5

-1.5

2.2. Equilibrium point of the system (2)𝑥𝑥 = 4𝑦𝑦 + 4𝑦𝑦𝑦𝑦⎧.
𝑦𝑦 = −4sin (𝑥𝑥).
⎨𝑧𝑧. = 4𝑤𝑤⎩𝑤𝑤 = −4𝑠𝑠𝑠𝑠𝑠𝑠(𝑥𝑥)𝑦𝑦 − 4𝑧𝑧

0.50Lyapunov

𝛻𝛻𝑓𝑓 =

1.5

3.2. Multi-stability of the system (5)(5)

It can be found that the equilibrium point of the system (5)is (𝑘𝑘𝑘𝑘, 0, 0, 0), 𝑘𝑘 ∈ 𝑍𝑍 ∗ . When the equilibrium point meets(2𝑘𝑘𝑘𝑘, 0, 0, 0) , the corresponding eigenvalue is(4𝑖𝑖, −4𝑖𝑖, 4𝑖𝑖, −4𝑖𝑖) , and the equilibrium point type is a

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

473

Dynamic Characteristics Analysis of
𝜋𝜋in Fig. 2 and Fig. 3. When the initial value is ( +32𝑘𝑘𝑘𝑘, 1, 1, 1)(𝑘𝑘 = 0, 1, 2) , there are three independent
scroll flows in the system (5), which are represented byblue, pink and purple lines in Fig. 2. When the initial value𝜋𝜋is ( + 2𝑘𝑘𝑘𝑘, 1, 1, 1)(𝑘𝑘 = 0, 1, 2) , there are 11-scrolls

2

1.5

1

0.5

0

2

-0.5

flow, 13-scrolls flow and 16-scrolls flow in the system (5),which are represented by blue, pink and purple lines in Fig.
2. According to the above analysis, it can be seen that thecoexistence of different multi-scrolls flows with thechange of different initial values illustrates the multistability of the system (5).

-1

-1.5

-2
-4

-2

0

2

4

6

8

10

12

14

16

Fig. 2. Coexistence of three independent scrolls

3.3. Grid type multi-scroll flows

2.5
2

In the above, multi-scroll flows are obtained in the system(5) in the x-axis direction by expanding the equilibrium
point in the x-axis direction. Similarly, the construction of
the grid type multi-scroll flows needs to expand theequilibrium point in the x-axis and Y-axis directions. Set∇𝐻𝐻 = (𝑓𝑓(𝑥𝑥), 𝑓𝑓(𝑦𝑦), 𝑧𝑧, 𝑤𝑤), and equation (2) can be express
as:

1.5
1
0.5
0
-0.5
-1
-1.5
-2
-2.5
-10

0

10

20

30

40

50

60

(a) 11-scrolls.

𝑥𝑥 = 4𝑓𝑓(𝑦𝑦) + 4𝑓𝑓(𝑦𝑦)𝑤𝑤⎧.
𝑦𝑦 = −4𝑓𝑓(𝑥𝑥).
⎨𝑧𝑧. = 4𝑤𝑤⎩𝑤𝑤 = −4𝑓𝑓(𝑥𝑥)𝑓𝑓(𝑦𝑦) − 4𝑧𝑧

2.5
2
1.5
1
0.5
0

Where 𝑓𝑓(𝑥𝑥) and 𝑓𝑓(𝑦𝑦) satisfy mapping:

-0.5
-1

𝑢𝑢 + 𝑁𝑁, 𝑢𝑢 < −𝑁𝑁
𝑓𝑓(𝑢𝑢) = �sin(𝑢𝑢) , −𝑁𝑁 ≤ 𝑢𝑢 ≤ 𝑁𝑁 , 𝑁𝑁 = 2𝑛𝑛𝑛𝑛, 𝑛𝑛 ∈ 𝑍𝑍 ∗𝑢𝑢 − 𝑁𝑁, 𝑢𝑢 > 𝑁𝑁

-1.5
-2
-2.5
-40

-30

-20

-10

0

10

20

30

40

50

(b) 13-scrolls
2
1.5
1
0.5
0
-0.5
-1
-1.5
-2
-2.5
-20

0

20

40

60

(7)

It can be found that the equilibrium point of the systemis (𝑘𝑘𝑘𝑘, 𝑘𝑘𝑘𝑘, 0, 0), 𝑘𝑘 ∈ 𝑍𝑍 ∗ . When the equilibrium point is(2𝑘𝑘𝑘𝑘, 2𝑘𝑘𝑘𝑘, 0, 0) and ((2𝑘𝑘 − 1)𝜋𝜋, (2𝑘𝑘 − 1)𝜋𝜋, 0, 0) , thecorresponding eigenvalue is (4𝑖𝑖, −4𝑖𝑖, 4𝑖𝑖, −4𝑖𝑖) and theequilibrium point type is a center point. When theequilibrium point is ((2𝑘𝑘 − 1)𝜋𝜋, 2𝑘𝑘𝑘𝑘, 0, 0) and(2𝑘𝑘𝑘𝑘, ((2𝑘𝑘 − 1)𝜋𝜋, 0, 0), the corresponding eigenvalue is(4, −4, 4𝑖𝑖, −4𝑖𝑖) and the type of the equilibrium point is an
unstable saddle point.
In addition, the scroll of the system (6) will be generated
at the center point, and the bond band will be generated at
the unstable saddle point. When 𝑁𝑁 = 2𝑛𝑛𝑛𝑛 , theequilibrium point of the system (6) has (2𝑛𝑛)2 + (2𝑛𝑛 +1)2 central points and 4𝑛𝑛 × (2𝑛𝑛 + 1) unstable saddle
points. Then, a multi-scroll flow with (4𝑛𝑛 + 1) × (4𝑛𝑛 +

2.5

-40

(6)

80

(c) 16-scrolls
Fig. 2. Different numbers of multi-scroll flows with thechange of initial value
It can be found that different flows of the system (5) willappear with the change of initial value, which are shown© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

474

Zhonggao Chen, Minghan Song1) grid type is formed, it is composed of (2𝑛𝑛)2 +(2𝑛𝑛 + 1)2 vortices and 4𝑛𝑛 × (2𝑛𝑛 + 1) bond bands.
Set 𝑛𝑛 = 1, 2, 5 × 5 and 9 × 9 grid type multi-scroll
flows are shown in Fig. 3.

The test conditions used in this paper are: significancelevel a = 0.01, test sequence B = 100, length of each groupis 10bit, and the confidence interval is 1. The test results
are shown in Fig. 4.

(𝑎𝑎) 5 × 5 grid type multi-scroll flow

Fig. 4. The results of NIST test
The performance of pseudo-random sequence iscomprehensively analyzed in the experiment. The
experiment can pass only when the following conditions
are met:• All P-values must be greater than significance level 𝛼𝛼.
• The pass rate must be in the confidence interval.
• P-values must be evenly distributed.
It can be seen from Fig. 4 that the system (6) has passed
the test and proved that the system has good pseudo
randomness.
In this paper, taking Non-Overlapping Template as an
example, the system (6) can be found to satisfy the uniform
distribution of P-value. P-value distribution of NonOverlapping Template is shown in Fig.5.

(b) 9 × 9 grid type multi-scroll flow
Fig. 3. grid type multi-scroll flows
3.4. NIST test of system (6)At present, NIST SP800-22 standard is the most widelyused and authoritative standard used to detect the pseudorandom sequence generated by the system (6). It has 15
test indexes. The ideal random sequence is taken as areference to test the deviation degree of pseudo-random
sequence from different angles in statistical characteristics.
It is generally believed that the sequence that can pass thedetection has a good pseudo-random performance. Eachtest of SP800-22 standard will provide the pass rate andthe uniformity of P value distribution. If the significancelevel 𝛼𝛼 = 0.01 and the test sequence is group 𝛽𝛽 , theconfidence interval of the passing rate can be defined as:
�1 − 𝛼𝛼 − 3�𝛼𝛼(1 − 𝛼𝛼)𝛼𝛼(1 − 𝛼𝛼), 1 + 𝛼𝛼 − 3�
�
𝛽𝛽
𝛽𝛽

(8)

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

475

Dynamic Characteristics Analysis of

6. Luo, X.H.; Tu, Z.W.; Liu, X.R.; Cai, C.; Liang, Y .L.; Gong, P .
Implementation of a novel two-attractor grid multi-scroll
chaotic system. Chin. Phys. B 2010, 19, 070510.
7. Chen, Z.; Wen, G.L.; Zhou, H.A.; Chen, J.Y . A new M x Ngrid double-scroll chaotic attractors from Rucklidge
chaotic system. Optik 2017, 136, 27–35.
8. Chen, Z.; Wen, G.L.; Zhou, H.A.; Chen, J.Y . A new M x Ngrid double-scroll chaotic attractors from Rucklidge
chaotic system. Optik 2017, 136, 27–35.
9. Lü, J.H.; Han, F.L.; Yu, X.H.; Chen, G.R. Generating 3-D multiscroll chaotic attractors: A hysteresis series switching
method. Automatica 2004, 40, 1677–1687.
10. Peng, Z.P .; Wang, C.H.; Luo, X.W. A novel multi-directional
multi-scroll chaotic system and its CCII+ circuit
implementation. Optik 2014, 125, 6665–6671.
11. Wang, C.H.; Xu, H.; Yu, F. A novel approach for constructing
high-order Chua’s circuit with multi-directional multi-scroll
chaotic attractors. Int. J. Bifurc. Chaos 2013, 23, 13500223.
12. Deng, W.H.; Lü, J.H. Generating multi-directional multiscroll chaotic attractors via a fractional differential
hysteresis system. Phys. Lett. A 2007, 369, 438–443.
13. Zhang, Y .; Yu, S.M.; Liu, M.H. Generating multi-scroll
hyperchaotic attractors based on FPGA technology. Circuits
Syst. 2007, 12, 39–43.
14. Tlelo-Cuautle, E.; Rangel-Magdaleno, J.J.; Pano-Azucena,
A.D.; Obeso-Rodelo, P .J.; Nunez-Perez, J.C. FPGA
realization of multi-scroll chaotic oscillators. Commun.
Nonlinear Sci. Numer. Simul. 2015, 27, 66–80.
15. Yu, S.M.; Lü, J.H.; Leung, H.; Chen, G.R. Design andimplementation of n-scroll chaotic attractors from ageneral jerk circuit. IEEE T rans. Circuits Syst. I Regul. Pap.
2005, 52, 1459–1476.
16. Yalcin, M.E. Multi-scroll and hypercube attractors from ageneral jerk circuit using Josephson junctions. Chaos
Solitons Fractals 2007, 34, 1659–1666.
17. He, S.B.; Sun, K.H.; Wang, H.H.; Ai, X.X.; Xu, Y .X. Design ofn-dimensional multi-scroll Jerk chaotic system and its
performance. J. Appl. Anal. Comput. 2016, 6, 1180–1194.
18. Ma, J.; Wu, X.Y .; Chu, R.T.; Zhang, L.P . Selection of multiscroll attractors in Jerk circuits and their verification using
Pspice. Nonlinear Dyn. 2014, 76, 1951–1962.
19. T chitnga, R.; Nguazon, T.; Fotso, P .H.L.; Gallas, J.A.C. Chaos
in a Single Op-Amp-Based Jerk Circuit: Experiments andSimulations. IEEE T rans. Circuits Syst. II Express Briefs 2016,
63, 239–243.
20. Tang, W.K.S.; Zhong, G.Q.; Chen, G.; Man, K.F. Generationof n-scroll attractors via sine function. IEEE T rans. Circuits
Syst. I Fundam. Theory Appl. 2001, 48, 1369–1372.
21. Hu, X.; Liu, C.; Liu, L.; Ni, J.K.; Li, S.L. Multi-scroll hidden
attractors in improved Sprott A system. Nonlinear Dyn.
2016, 86, 1725–1734.
22. Luo, X.H. Circuitry implementation of a novel
nonautonomous hyperchaotic Liu system based on sine
input. Chin. Phys. B 2009, 18, 3304–3308.

Fig. 5. P-value distribution of Non-Overlapping Template
4. Conclusion
In this paper, a four-dimensional conservative system witha unique equilibrium point is constructed. Firstly, the sinefunction is introduced to extend the equilibrium point inthe x-axis direction so that it has a linear equilibrium pointwith infinite extension in the x-axis direction. Secondly, it
is found that the system (5) has different numbers of multiscroll flows under different initial values, which provesthat the system has multi-stability. Thirdly, by introducing
a sinusoidal function to extend the equilibrium points inthe x-axis and Y-axis directions, a controllable grid typemulti-scroll flow is found. Finally, the system is tested by
NIST, and the test results show that the system has good
pseudo randomness.
Acknowledgements

References
1. 1. Suykens, J.A.K., Vandewalle, J., Generation of n-double
scrolls (n= 1, 2, 3, 4, . . . ), IEEE T rans. Circuits Syst.
IFundam. Theory Appl, 1993, 40, 861–867.

2.

Zhang, C.X.; Yu, S.M. Design and implementationof a novel multi-scroll chaotic system. Chin. Phys. B
2009,18, 119–129.

3. Wang, F.Q.; Liu, C.X. A new multi-scroll chaotic generator.
Chin. Phys. 2007, 16, 942–945.
4. Liu, X.Z.; Shen, X.M.; Zhang, H.T. Multi-scroll chaotic andhyperchaotic attractors generated from Chen system. Int. J.
Bifurc. Chaos 2012, 22, 1250033.
5. Chen, D.Y; Sun, Z.T.; Ma, X.Y .; Chen, L. Circuit
implementation and model of a new multi-scroll chaotic
system. Int. J. Circuit Theory Appl. 2014, 42, 407–424.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

476

Development of in-Home Wireless Continuous Temperature Data Logging and Alarming
System for Fever Monitoring in Pediatrics
Ali S. Al-Mejrad

Department of Clinical Laboratory Sciences, Faculty of Applied Medical Sciences University of Hail, Hail,
Kingdom of Saudi Arabia.
E-Mail: a.almejrad@uoh.edu.sa
Abstract
Fever is one of the most common concern to be taken care by parents that sometimes force them to bring
their child to the hospital emergency for continuous follow up especially at night when sleeping. Fever isdefined when a rectal temperature reaches over 38.0 C (100.4 F). Temperatures measured from other sitesof body are usually less. There is no defined threshold for fever due to variation of body temperatures
between different individuals as much as 1 F. Fever itself is not life threatening unless the rectal
temperature is persistent over than 41.6 C (107 F). Fever may be due to a serious illness and usually iscaused by common infections that are not serious. However, a fever can be caused by other causes otherthan infections. Since continuous temperature monitoring is inapplicable at the rectal site, the paper aims
at developing a prototype system using Wireless Sensor Network (WSN) for wireless continuous fever
monitoring system from the head skin or wrist that are the most applicable area even so one must add 2
degrees for the read temperature correction. The system can record the temperature at the patient terminal
and transmit the fever status and critical situations to an alarming system in the side of taking-care people
that will help in in-home care for pediatrics suffering from fever.
Keywords: Wireless; Temperature monitoring; data logging; alarming system.

1.

Introduction(AAP) recommends against using mercury
thermometers because the glass can break, andmercury is poisonous [3]”. In addition, this kind ofthermometer can work with adult and cannot be usedproperly with infant even at the rectal site. The mostcommonly used and recommended are Electronic
thermometers with a digital temperature readout
display. Another type of thermometers is the plastic
strip that change color to indicate the temperature.
This type is the least accurate. The strip is used byplacing it on the forehead and after one minute, thetemperature can be read while the strip is in its place.
Another type of plastic strip thermometer is used forthe mouth. Hence, there are many kinds of temperature
measurement methods but most of it is inconvenient toinfants and child monitoring, because monitoringmeans continuous reading of the temperature [4].

Fever is one of the most important symptom of many
diseases. Fever may be due to a serious illness andusually is caused by common infections that are notserious. However, a fever can be caused by othercauses other than infections. The body’s temperature
increase is controlled by the part of the brain called the
hypothalamus to fight the infection that may reach thefever level. Fever is considered one of the mostcommon concern to be taken care by parents thatsometimes force them to bring their child to thehospital emergency for continuous follow up
especially at night when sleeping. Fever is definedwhen a rectal temperature reaches over 38.0 C (100.4
F). Temperatures measured from other sites of bodyare usually less. There is no defined threshold for fever
due to variation of body temperatures between
different individuals as much as 1 F. Fever itself is notlife threatening unless the rectal temperature is
persistent over than 41.6 C (107 F) [1][2].

Therefore, a method for continuous temperaturemonitoring has to be developed to be used in home andstill be safe and convenient [5]. This method will
enable a continuous read out of the temperature datautilizing the wireless technology from multiple
transmitter node that enable the person on charge tomonitor the temperature of many patients (infant, andIn the past, body temperature monitoring was using athermometer constructed of glass filled with mercury.
“However, the American Academy of Pediatrics© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

477

Ali S. Al-Mejrad

child). This paper aims at developing a prototype
system using embedded technology and WirelessSensor Network (WSN) technology for wirelesscontinuous fever monitoring system from the headskin or wrist that are the most applicable area taking inconsideration adding 2 degree for the read temperaturecorrection [6]. The wireless temperature monitoring
system (WTMS) provides 24/7 continuous
temperature monitoring of critical patient (Infant, andChild). Continuous monitoring provides the means todetect abnormalities (fever) at an early stage. Utilizing
wireless technology eliminates the need for cabling.
WTMS allows for easy field data collection.
Temperature data is transmitted from the sensor at thetransmitter node to a nearby receiver node via wirelessconnection. The receiver node can be connected to acomputer via USB. In addition, the receiver node canwork standalone without the PC. A universal receiver
node can be utilized as a data logger that can collectdata from any transmitter node. This universal
receiver and the attached PC can work to trigger visual
and/or audible alarm for preset temperature values forfever detection even more it can send SMS to the smart
phone belonging to the person in charge such as
parents. In addition to that, the data can be analyzed
later for diagnosis and reported by a dedicatedsoftware package.
2.

Fig. 1 shows a general block diagram of a wirelesstemperature monitoring system (WTMS). The system
consists of three major parts: transmitter node,
receiver node, and optional data access module.
A. Transmitter NodeThe system is comprised of Wireless transmitter nodethat consists of a power subsystem (battery),temperature sensor, and data acquisition. This moduleis based on wireless Wi-Fi technology. This module isdesigned to operate in moisture environment and isprotected by adding moisture resist material that willprotect and isolate the patient from being connected tothe circuit via moisture. The transmitter node has alead off detector to trigger an alarm on the receiver
node when there is no or bad contact on the patientskin. The transmitter is attached to the patient skin all
the time by the mean of head bandage or wrist strapthat will give the patient a convenient way of every
day practice. Figure 2 shows the transmitter node,
which consists of temperature sensor; lead off detector,
microcontroller, battery, and the transmitting antenna.

Patient Forehead

Temperature
sensorMicrocontroller

Sensor off
detector

Battery

System Development

2.1 Hardware and System Block Diagram
The temperature wireless system designed consists of:

Figure 2. Transmitter Node

A. Transmitter Node
B. Receiver Node
C. Data Access Module

Patient

Data
Acquisition

Temperature
Sensor

B. Receiver Node
It consists of a power subsystem (battery and battery
management; recharge circuit operate from USB port),
and readout device (LCD). This module is based on
wireless Wi-Fi technology that can be directly
connected to the PC via USB for data collection andfurther analysis. The receiver node contains audible
and visual alarms, and data storage unit (SD card).
Any receiver node can be programmed to collect datafrom many transmitters, as the case when there aremany children or infants to be monitored fortemperature rise. The receiver contains SD card to
store the data for further analysis by physicians. The
collected data can be sent to physician over the internet
under user control when the receiver node is connected
to a PC. Figure 3 shows the receiver node, which

Link from thebounded transmitter
node

Transmitter Node

Optional
Data Access Module

Link from thean y transmitternode

USB Interface

USB Interface

Data Logger

Web pageWeb ServerTransmitter
Antenna

SD Card Reader

Alarm Notification SMS

Figure 1. System Block Diagram© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

478

Development of in-Home Wirelessconsists of receiving antenna, microcontroller which
control: LCD as a reading out device for thetemperature and alarm values, buzzer for audible
alarm, mass storage (I2C EEPROM, and SD card).
The controller has a USB interfacing capability to the
PC in order to collect the data for further analysis. In
addition, the receiver node battery could be charged
from the PC USB port.
USB
Interfacing

ReceiverAntenna
Microcontroller

LCD

Buzzer

SD cardFigure 4. Front Panel of the User Interface ofTemperature Data Collection and Fever Alarms
EventsI2C
EEPROM

RTC

Battery3.

Results and Discussion

Analysis of Body Temperature Measurements
Comparison Test

Battery
charger

Testing the sensor DS18B20 as body temperature
sensor is done with comparison to the measurement
results with mercury thermometer. The selection ofthis type of thermometer is due to its accuracy
compared to other types of thermometers. Table 1
below shows the comparison results, the measurement
difference and the average measurement error.

PC USB power

Figure 3. Receiver Node
C. Data access module
It consists of data logger module which can beconnected to the PC or a web server for data
distribution over the web, sending alarm notification
SMS to smart phones. SD card
reader can be connected to the server for reading SD
card from any receiver. The data logger can connect toany transmitter node(s) for data collection and analysis.
The data logger is mainly powered from the PC USB
port.

No. Temperature
Sensor
(0C)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

2.2 Software and User Interface
A user interface is developed using Laboratory Virtual
Instrument Engineering Workbench to control thewireless module and collect the temperature data ofpatient events of high temperature threshold of fever
as shown in Figure 4 [7].

Mercury
Thermometer(0C)

Measurement
Difference
(0C)

36.75
37.60
36.63
37.25
36.25
36.97
36.35
36.94
36.56
37.25
36.06
36.70
35.95
36.49
36.85
37.28
36.75
37.57
36.70
37.53
36.50
37.30
36.53
37.16
36.54
36.96
36.26
36.71
36.23
36.75
Average - Error

0.85
0.62
0.72
0.59
0.69
0.64
0.54
0.43
0.82
0.83
0.80
0.63
0.42
0.45
0.52
0.64 %

Table 1. Body temperature test results© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

479

Ali S. Al-Mejrad

From the test results, obtained results showed thetemperature has an average error around 0.64% using
the Mercury Thermometer compared to the developed
system. Therefore, it can be stated that the sensor
reading s are very good.

be concluded that the website is inaccessible in therange above 50 meters. One of the most importantfactors that affect the process of access and delivery ofthe data is indoor so possible transmitted signals will
be hindered and cannot be well reached.

Analysis of Events of High Temperature Thresholdof Fever Test

4.

The designed system goal is to provide in home, selfdependent, continuous monitoring for critical events,
and alarming system to prevent deterioration of thepatient condition. The system has many features withwireless (Wi-Fi) connectivity in range of up to 50
meter (the distance between the transmitter andreceiver nodes), typical signal reception of once every
1-10 minutes under user control, frequency of datatransmission increases with fast temperature rise,temperature range 25°C to 50 °C with 0.1 °C
resolution, preset alarm values, moisture resistant forhumid environments.

From the test results of some of collected temperature
data shown in Table 2 which gives continuous
monitoring for critical events of fever (as shown in
RED colors ) , and alarming system to prevent
deterioration of the patient condition

No
.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
20

Time
Data
5.7
5.9
6.1
6.3
6.5
6.7
6.9
7.1
8.4
8.6
8.8
9
9.2
9.4
9.6

Temp
Data
35.8
35.8
35.8
35.8
35.2
35.2
34.6
34.6
38.9
39.6
39.6
39.6
38.9
38.3
37.7

Time
Temp
High
Limit
5.7
5.9
6.1
6.3
6.5
6.7
6.9
7.1
8.4
8.6
8.8
9
9.2
9.4
9.6

Temp
High
Limit
38
38
38
38
38
38
38
38
38
38
38
38
38
38
38

Conclusion and Future WorkEvents of
Alarms(in RED)Acknowledgement
I like to thank the Deanship of Scientific Research of
University of Hail, for their support of my research.
References

FEVER
FEVER
FEVER
FEVER
FEVER
FEVER

1-

2-

3-

Table 2. Events of High Temperature Threshold of
Fever Test Results
Analysis of Data Transmission Performance Test

4-

From the test results of this communication between
the master module and the slave, the slave module
used can accept data from master module according todata delivered.

5-

6-

The performance test of Nodemcu ESP8266 works tofind out if the sensor is accessible by other devices
with a certain distance. Testing is done by sending dataon Nodemcu ESP8266 to display the results on thewebsite and then test access with a certain range.
Distance parameter of the maximal use range is about
50 meters. From the tests that have been done, it can

7-

Herzog LW and Coyne LJ., “What is fever?
Normal temperature in infants less than 3 months old”,
Clin Pediatr, 32, pp.142-6, 1993.
Teach SJ and Fleisher GR., “Duration of fever and its
relationship to bacteremia in febrile outpatients three
to 36 months old”, The Occult Bacteremia Study
Group. Pediatr Emerg Care, 13, pp. 317-9, 1997.
Lynn R. Goldman, Michael W. Shannon and theCommittee on Environmental Health, “Technical
Report: Mercury in the Environment: Implications for
Pediatricians, Pediatrics”, Vol. 108 No. 1, pp. 197205, 2001.
Kyriacou P. A., “Temperature sensor technology”,
Biomedical Sensors, pp. 1-38, 2010.
IEC 60601-1, Medical Electrical Equipment- Part I:
General Requirements for Safety and Essential
Performance,
International
ElectrotechnicalCommission, Geneva, Clause 8.7,1995.
A. Tzavaras, and B. Spyropoulos, “ Development of alow-cost wireless monitoring System supporting theContinuity of Medical Care of the Patient at home”,
35th Annual International Conference of the IEEE
EMBS Osaka, Japan, 3 - 7 July, 2013.
Mehta B. et al. "Real Time Patient Tele­ monitoring
System Using LabVIEW", International Journal of
Scientific & Engineering Research (IJSER), Volume 3,
Issue 4, April-2012.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

480

Design and Development of in-home Wireless Crucial Events Logging and Alarming
System for Elderly and Disabled People Care
Ali S. Al-Mejrad

Department of Clinical Laboratory Sciences, Faculty of Applied Medical Sciences University of Hail, Hail,
Kingdom of Saudi Arabia.
E-Mail: a.almejrad@uoh.edu.sa

Abstract
This paper aims at developing a system that will help in in-home care for elderly or disabled people. The system
can record the patient vital parameters such as temperature, SpO2, ECG, heart rate, bed moisture and fall off
sensor which could be installed at the bathroom or shower area at the patient terminal and transmit the criticalsituations such as high or low values to an alarming system. The system will be developed using the National
Instrument Wireless Sensor Network (NI-WSN) to control the crucial events transmission and LabVIEW
software to design the user interface.
Keywords: Elderly; Disabled; In-home care; NI-WSN; PSoC; LabVIEW.

1.

Introduction

Arabia are suffering from chronic health conditions
such as chronic cardio-respiratory diseases, which isone of the major causes of death and admission tohospital in addition to many other concurrent problems[9]. Due to frequent occurrences of such diseases tothese people who in need for continuous care will leadto costly hospitalization of such patients. In this papera system will be developed that will help in in-homecare for elderly or disabled people by recording thepatient vital parameters such as SPO2, heart rate, ECG,
temperature, bed moisture, and fall off sensor on thechair or the bed. In addition, these sensors could beinstalled at the bathroom or shower area at the patient
terminal and transmit the critical situations such as
high or low values to an alarming system. The system
will be developed using the National Instrument
Wireless Sensor Network (NI-WSN) to control thecurial events transmission and LabVIEW software to
design the user interface.

The growth in the number and proportion of elderly
and disable people are significantly increase whichlead to the need of a special care system to give thesepeople a chance to live depending on themselves [1].
In addition, the resources in in-patient facilities arelimited and expensive. Elderly people are suffering
many health problems, mobility limitation, andchronic physical problems and, other disorders
compared to younger age people. All of which arerequiring special continuous attention and care. The
family and/or other help provider often shares the taskof caring for these people. However, the elderly
usually remain independent and alone at home and thisbecomes a critical issue due to non-availability of their
help provider. While at home, smart sensors such as
body temperature and ECG sensors can be used towirelessly connect through NI-WSN to a Computer,which will record and analyze different parameterssuch as, body temperature, blood pressure, cardiac
function and many others [2]-[4]. A connection to theInternet keeps the Elderly and disabled people withinreach of their help providers and family. This methodof transmitting selected data only when required rather
than streaming all data continuously, is perfect method
for monitoring critical situations and activating alarms
for these critical cases [5]-[8]. Elderly people in Saudi

2.

System Development

The designed system in the block diagram consists ofdifferent components as shown in Figure.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

481

Ali S. Al-Mejrad

Figure 2. NI Wireless Sensor Network
Microcontroller PSoC (CY8C29466-24PVXI)
Figure 1. Block Diagram of the Designed System

Programmable System on Chip (PSoC) is a
programmable embedded system on chip which
integrating on a single chip configurable digital andanalog peripheral functions, memory and a
microcontroller [13]. PSoC is designed to replacecomponents in traditional MCU-based system with asingle-chip programmable device. PSoC device has
configurable blocks of digital and analog, as well as
programmable interconnections. This architecture
enables designer for creating customized peripheral
configurations. This microcontroller is responsible fordata collecting from all the analog modules connectedto the patient and preparing the data to be transferred
to/from the wireless module. Also to keep a complete
trend in the MMC module as a data logger.

The system has two major components; the consolethat is fixed at the patient chair, bed, or even held withthe patient with the NI-WSN-3202 wireless module[10] and the other component is the NI-WSN-9791
Ethernet gateway [11] connected to the PC that is also
connected to an alarming system to trigger alarm on
critical situations. A user interface program using
LabVIEW is developed to control the wireless moduleand collect the data of patient events such as high heartrate, high temperature, and fall off the chair or bed [12].
Measurement node NI-WSN-3202:
The measurement nodes connect directly with sensor
and are able to transmit wirelessly a 2.4 GHz radio
data to the WSN Ethernet gateway. Each measurement
node has four analog input channels and four digital
Input/output channels.

3.

Results and Discussion

First, a prototype test system of three main signals
including pulse oximeter, temperature and ECG using
PSoC is developed to ensure the success of the datacollection. The block diagram of the prototype test
system with LCD shown in Figure 3 consisting of
three main traces are described below:

The measurement nodes and the NI WSN-9791
Ethernet gateway:
The measurement nodes and the Ethernet gateway
communicate wirelessly using a radio frequency of 2.4
GHz based on the reliable IEEE 802.15.4 NI WSN
protocol. The network can accommodates up to 36
nodes per gateway and provide an outdoor range of up
to 300 m.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

482

Design and Development of

ADC Converter
The analog to digital converter used to acquire thesignal in digital form in order to display it on the LCD;
here we use the PSoC 12 bits ADC converter withsampling rate of 50.
Now the signal will be handled by to the PSoC internal
controller in order to display it on the LCD.

Thermistor
Bridge Circuit

3.2 Temperature
PGA
The PGA here amplifies the sensing signal from thebridge circuit and pass it to the ADC converter.

Figure 3. Prototype test system of three main signals
including pulse oximeter, temperature and ECG using
PSoC

ADC Converter

3.1 Pulse Oximeter:

Converts the thermistor voltage into digital form to the
PSoC internal controller to check the calibration tables
and display the temperature on the LCD by using thegraphic LCD routine.

First module is Pulse Oximeter, the frequency of thissignal in the range of 1Hz to 40Hz the sensor used inthis module is photo sensor sender and receiver, thefinger of the patient located between the them, theinfrared opt coupler will detect the volume change infinger’s capillaries and to obtain blood flow profile.

3.3 ECG
Instrumentation Amplifier
In our system, some external circuits (outside the
PSoC controller) were made such as Instrumentation
amplifier and isolator circuit because of some
limitation in the ranges of the PSoC.

Programmable Gain Amplifier (PGA)
The received light is converted into an electric signal
by a photodiode and then enters the first block inside
the PSoC controlled is PGA here which act as a buffer
to match the circuit input output impedance, and also
amplify with gain of 4.

Filters
The second block is filter. Two types of filter wereused the first is second order Band Pass filter, thisfilter is used to cut the high and low noise and pass
only the ECG signal (1-150 Hz), the other filter is
Notch filter is used to cut mains noise 60Hz. The
response of the amplifier circuit has been verified
using Orcad Pspice simulation program and the
response is illustrated in Figure 4 bellow

Filters
The second block is filter we use two types of filter the
first is second order Low pass filter, this filter is used
to cut the high frequency component, the cornerfrequency of the filter is 100Hz, the other filter is Band
pass filter is used to cut plethysmograph signal band
(1-40Hz)
Pulse Peak detector and pulse counter
The function of this block is to detect the peaks of thesignal and pass it to the pulse counter, which is adigital counter in order to count the heart rate.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

483

Ali S. Al-Mejrad

The complete system components is shown in Figure
6 and hardware test setup is shown in Figure 7.

Figure 4. The total ECG filter response
PGA
Act here as the main amplifier of the ECG signal, anduses a gain of 16.
ADC Converter
The analog to digital converter used to acquire thesignal in digital form in order to display it on the LCD;
here we use the PSoC 12 bits ADC converter withsampling rate of 50.
Now the signal will be handled by to the PSoC internal
controller in order to display it on the LCD.

Figure 6. The complete system components

3.4 Designed PSoC Circuit Diagram

Figure 5. Designed PSoC circuit diagram

Figure 7. Hardware test setup with user interface

The analog signals of the three signals sampled in
PSoC micro-controller designed as shown in Figure 5
are then stored in the system EEPROM and/or the
MMC card on the system shown in Figure 1 for later
analysis. The digital data is wired to the measurement
node NI-WSN-3202 that is responsible for the wirelesstransfer of the data to the NI WSN-9791 Ethernet
gateway receiver. The receiver node is connected to
the PC through the Ethernet port for data collection
and alarm triggering. Experimentation utilizing theprototype based on wireless technology with the PSoC
system has shown that the communication range varies
from 30 to 50 meters depends on the home walls thatis sufficient.

The system includes main components including auser interface program as shown in Figure 8. The
system is developed to control the whole system bylogging in the system and then configuring the eventswith minimum and maximum limits and configuring
wireless module. When running, the system starts
patient data acquisition, and display of the data andevents of the patient such as high temperature, highheart rate, low oxygen saturation, and fall off the chairor bed. Finally, patient’s data and events can beanalyzed for follow-up and diagnosis purposes.

3.5 Designed System and its User Interface© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

484

Design and Development ofReferences
1-

United Nations, Department of Economic and Social
Affairs, Population Division. World Population Ageing
2015 (ST/ESA/SER.A/390), 2015.
2- Eduardo Freitas and Amandio Azevedo, “WirelessBiomedical Sensor Networks: The Technology”,
Proceedings of the 2nd World Congress on ElectricalEngineering and Computer Systems and Science(EECSS'16) Budapest, Hungary – August 16 – 17, 2016.
3- Panicker N. and A. Sukeshkumar, “Wireless sensor
based systems for biomedical monitoring: A review”,
AMSE JOURNALS –2014-Series: Modelling C; Vol.
75; No. 1; pp 65-78, 2014.
4- Demongeot, J. et al.,”Multi-sensors acquisition, data
fusion, knowledge mining and alarm triggering in
health smart homes for elderly people”, C. R. Biologies
325:673–682, 2002.
5- Clark, A. and K. EMMA, “A comprehensive alarmsystem for the elderly and inﬁrm and disabled who live
at home”, Res. Soc. Health Care, 15:205–27, 1979.
6- Chan, M. et al.,”Smart house automation system for theelderly and the disabled”, IEEE Int. Conf. Syst. Man
Cybern, 2:1586–1589, 1995
7- Peeters, P. H., “Design criteria for an automatic safetyalarm system for elderly”, Technol. Health Care, 8:81–
91,2000.
8- A. Tzavaras, and B. Spyropoulos, “ Development of alow-cost wireless monitoring System supporting theContinuity of Medical Care of the Patient at home”,
35th Annual International Conference of the IEEE
EMBS Osaka, Japan, 3 - 7 July, 2013
9- Khan J. et al., “The Saudi Guidelines for the Diagnosis
and Management of COPD”, Ann Thorac Med. 9(2): pp.
55–76, 2014.
10- NI-WSN-3202 wireless module – User guide andSpecifications, 2010.
11- NI WSN-9791 Ethernet Gateway – User guide andSpecifications, 2010.
12- National Instruments Corporation, “LabVIEW User
Manual”, Edition 2014.
13- Cypress Semiconductors, PSoC 5LP Architecture
Technical Reference Manual, Document No. 00178426 Rev. *F, 2016.

Figure 8. User Interface System
4.

Conclusion and Future WorkThe designed system goal is to provide in home, selfdependent, monitoring for critical events, andalarming system. As the majority of elderly people
prefer to stay in their own homes and may require help
and supervision. The designed system is monitoring
the vital signs of the patient such as SpO2, heart rate,
temperature, ECG, and the critical status (wet clothes
and fallen off the chair) which are added easily as
ON/OFF in order to trigger alarms for such events.
The system is connected to a PC that can easilytransfer the collected data on the web for long distancefurther data analysis and/or call for help. This
continuous remote monitoring and support will
increase the elderly people freedom and safety; andprevent patient condition deterioration. This in turnwill improve the elderly life quality. The designed
system was developed as a prototype test system
including three signals including pulse oximeter,
temperature and ECG using PSoC with LCD. The
future work is all about reducing the component using
the Analog Front End (AFE) chips to improve power
consumption and reduce the cost.
Acknowledgment
I like to thank the Deanship of Scientific Research of
University of Hail, for their support of my research.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

485

Virtual bird’s-eye for remote operationof unmanned construction machinery
Noritaka Sato, Akihiro Fukuda*

Morita and Sato Laboratory, Department of Electrical and Mechanical Engineering, Nagoya Institute of Technology,
Gokiso-cho, Showa-ku, Nagoya, Aichi, 466-8555 Japan
Email: a.fukuda.975@nitech.ac.jp
http://watt.web.nitech.ac.jp/index_2.html

Abstract
Recently, remote operation of construction machinery is required to release humans from dangerous places. However,
the risk of collisions and operation time increases because of lacking a sense of distance. To solve those problems,
we propose a method to display a virtual bird’s-eye view. Experiments are carried out to verify the effectiveness ofthe proposed method which can reduce the number of collisions and operation time comparing to the conventional
method.
Keywords: virtual reality, remote operation, construction machinery, UI

1. Introduction
Recently, remote operation of construction
machinery is required to release humans from dangerous
places. One of the situations where the remote operationof construction machinery is required is “the sorting
operation of slags” in iron factories shown in Fig. 1. “Thesorting operation of slags” is the work of sieving slags
with a power shovel equipped with a bucket and consistsof the following three processes: shoveling up slags,sieving slags, and releasing the remains. The workplaceis harsh for workers because of flying dust, the hightemperature, and the high humidity.
The operation by the remote control of a construction
machine was carried out by using the active robot SAM1
in the past. However, problems such as a significant
increase in working time and the collision of the power
shovel with obstacles have occurred. It is considered thatthe reason for them is lacking the sense of distancebecause of only using a monocular camera.

Fig. 1 The sorting operation of slags
The research by Shiroma et al. has proved that a
bird’s-eye view of the operation target improves thesense of distance2. However, it is difficult to installcameras in the workplace and take bird’s-eye view
images of the construction machine due to problems suchas the enormous amount of data communication, limited
installation position, and poor visibility due to dust. The
researchers also proposed a system that creates a virtual© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

486

Noritaka Sato, Akihiro Fukuda

2.3. Sensing position and posturebird’s-eye view by superimposing a CG model of therobot on an image obtained while the robot is moving3.
In the previous paper, we proposed a new method to
support the remote operation of construction machinery
based on the system which creates virtual bird’s-eye view
images and verified the effectiveness of it on the sorting
operation of slags in the real environment with aprofessional operator4. However, there was a problem
which was an insufficient number of trials as it was hard
to have the professional operator to make time. In thispaper, we did an additional experiment to get
confirmation of the effectiveness of the proposed system.

To sense the position and the posture of the power
shovel, we mounted some sensors on it. Fig. 3 shows
mounted sensors. The angle of each link is calculated
from the acceleration measured by the IMU attached tothem. The position and the posture of the power shovel
are measured by taking a picture of the AR markers andusing ARToolKit3 to get the relative position and posturefrom the external camera.
In the experiment in this study, the angle of each linkis measured by an encoder on the servo motor, and theposition and the posture of the body are fixed, therefore
there is no sensor to measure them.

2. Proposed system
2.1. System overview
In the proposed system, virtual bird’s-eye view
images are created by superimposing a CG of the power
shovel on images. Since the position of the power shovel
did not change during the operation, the position and theposture of the CG without the posture of the body of thepower shovel are changed based on measured values
obtained from sensors attached to the power shovel.

Fig. 3 Installed sensors

2.2. Real-time video

2.4. Creating virtual bird’s-eye view images

In addition to virtual bird’s-eye view images, realtime videos are displayed. In the experiment using theactual power shovel, cameras are mounted as shown in
Fig. 2 for displaying slags and the inside of the bucket
during sieving.
In the experiment in this study, a camera for slags
shown in Fig. 2 is used, because the motion of the sieving
is not required in the experiment.

Virtual bird’s-eye view images were created using agame engine, Unity by the following steps in shown Fig.
4:(i) Putting the images taken in advance inside thevirtual space.
(ii) Putting a 3D CG model of the power shovel inside
the virtual space based on its measured position andposture.
(iii) Getting a view from the position where the image
was taken. (The CG model is virtually
superimposed)

Fig. 4 Generating virtual bird’s-eye viewFig. 2 Cameras on the power shovel© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

487

Virtual bird’s-eye for remote

3.2. Result
Figs. 7, 8 show each subject’s results: average
operation time of 10 times and standard deviation. In
subject 1, an average operation time was 53.7 sec whenusing the only real-time video and 48.4 sec when using
the proposed system, and it was reduced by 9.87%. The
standard deviation was 9.58sec when using the real-timevideo and 7.39 sec when using the proposed system, andit was reduced by 22.8%. In subject 2, the average
operation time was 46.7 sec when using the real-timevideo and 39.1 sec when using the proposed system, andit was reduced by 16.3%. The standard deviation was
12.0 sec when using the real-time video and 6.10 sec
when using the proposed system, and it was reduced by
49.1%.
On the other hand, an average of the number ofcollisions was 0.36 times per one cycle when using thereal-time video and 0.16 times per one cycle when using
the proposed system in subject 1. 0.23 times per one cycle
when using the real-time video and 0 times per one cycle
when using the proposed system in subject 2.
In the questionnaire, the usability of the virtual
bird’s-eye view images was surveyed. Subject 1
answered that the side view and the top view were usefulto recognize the sense of distance between the bucket andwalls and the back view was not useful. On the other hand,subject 2 answered that only the side view was useful.

Fig. 5 The image shown to operators
Fig. 5 shows the video displayed to the operator. The
left-side video (A) is a real-time video and the right-side
videos (B) are virtual bird’s-eye view images.
3. Experiments
3.1. Experimental details
The simulation was conducted in the environment
and procedure shown in Fig. 6. Two male subjects in their
20s have no operation experience in the robot and were
given 10minutes to practice. The sorting operations wereconducted 10 times each using the real-time video only(“A” shown in Fig. 5) and the proposed system (“A” and“B” shown in Fig. 5). They were conducted in analternating cycle of five times to remove the effect of theorder.
The data recorded were operation time, number ofcollisions, and subjective evaluation by questionnaire.

Fig. 7 Operation time (Subject1)

Fig. 6 The environment of the simulation© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

488

Noritaka Sato, Akihiro Fukuda

References
1. T. Toyota, Development of the Rubber Artificial Muscle
Drive Type Remote Operation Robot for Disaster
Restoration, Journal of the Robotics Society ofJapan.Vol.33, 2015, pp. 600-606.
2. N. Shiroma et al. 13th IEEE International Workshop on
Robot and Human Interactive Communication. 2004, pp.
107-112.
3. N. Shiroma et al. Transaction of the Society of Instrument
and Control Engineers. Vol. 41, No. 12, 2005, pp. 10361043
4. A. Fukuda et al. Virtual bird’s-eye view for remote
operation of unmanned construction machinery, Reiwa 2ndyear Tokai-Section Joint Conference on Electrical,
Electronics, Information, and Related Engineering, 2020.

Fig. 8 Operation time (Subject2)
3.3. Discussion
Fig.7 and Fig. 8 show that the proposed system is
effective for the reduction of the operation time. The
results of the t-test showed that the proposed system isstatistically significant against the real-time video systemfor each subject. The reason for the difference in thesignificant trend between the subjects is lack of practice.
On the other hand, the average number of collisions
was reduced. Therefore, it is considered that the proposedsystem shows effective views to help the operation.
Moreover, there are some feedbacks that they are
effective in the questionnaire. However, the images thatare found to be effective or ineffective are different foreach subject.
4.

Conclusion

The purpose of this study is to reduce the operation
time and risk of collisions when the power shovel was
remotely operated to sort slags. In the previous paper, theefficacy of the proposed system was verified in the realenvironment. In this paper, we carried out an additional
experiment to get confirmation of the effectiveness of the
proposed system.
Experiments showed that the proposed system waseffective for the operation and it was statisticallysignificant for subjects. Furthermore, the superiority ofthe proposed system is also confirmed by subjective
evaluation.
In the future, it is necessary to increase the number ofsubjects and conduct a between-subjects design.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

489

Domain Transformation of Chest CT Images Using Semi-Supervised Cycle GAN for Opacity
Classification of Diffuse Lung Diseases
Masashi Miyake

Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai2-16-1
Ube, Yamaguchi 755-8611, Japan
Shingo Mabu
Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai2-16-1
Ube, Yamaguchi 755-8611, Japan
Shoji Kido
Graduate School of Medicine, Osaka University, Yamadaoka2-15
Suita, Osaka 565-0871, Japan
Takashi Kuremoto
Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai2-16-1
Ube, Yamaguchi 755-8611, Japan

Abstract
The aim of this research is to perform domain translation of chest CT images so that medical institutions caneffectively use a computer-aided diagnosis (CAD) system trained at a different institution. We propose a semisupervised Cycle GAN for domain transformation by combining the standard Cycle GAN and the trained CAD. In
the experiment, we classified opacities of diffuse lung diseases in CT images and clarified the effectiveness of the
proposed method for domain transformation.
Keywords: Deep Learning, Domain transformation, GAN, chest CT, Classification, Semi-supervisedannotated medical image data. Also, there arepossibilities that the accuracy of diagnosis may fluctuate
when the photography conditions of the images aredifferent depending on the medical institutions. For
example, different CT systems and their settings canchange the pixel values. Therefore, a CAD systemshowing good diagnostic performance in a certain
medical institution does not always show the same
performance in other medical institutions. In that case,the classifier needs to be retrained, and a large amount oftraining data for each medical institution needs to beprepared for training. The current system is not easy to
be used widely in various medical institutions, and it isnecessary to solve this problem.

1. Introduction
Recently, with the development of digital medical
imaging devices, it has become possible to acquire anenormous amount of medical images. Therefore, studies
on computer-aided diagnosis (CAD) have been actively
conducted, where machine learning is applied to build
CAD systems using a large number of medical images.
Image processing and image recognition techniques areused to construct CAD systems. Recently, the research of
convolutional neural networks has been rapidly
developed in the research field of image recognition andhas been applied to classifiers used in CAD systems1.
However, training a classifier requires a large amount of© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

490

Masashi Miyake, Shingo Mabu, Shoji Kido, Takashi KuremotoFig. 1. The flow of the validation of the proposed method. We compare the classification performance of several domain transformation
methods including the proposed method and without domain transformation. Refer to Table 1 for the details of domain transformation
methods.

The semi-supervised Cycle GAN used for YtoO
transformation is a structure that combines a standard
Cycle GAN with a trained classifier. The structure of theclassifier is based on a Residual Network4 (ResNet), andthe ResNet is trained with domain O image data. ResNetclassifies the opacity of a small number of domain Y
images after the domain transformation YtoO by Cycle
GAN has performed. Based on the classification result,
Cycle GAN is re-trained.
The flow of the validation of the classification
methods is shown in Fig. 1 and the methods for thevalidation are listed in Table 1. We measureclassification accuracy averaged over 20 independent
trials and apply the t-test between the proposed method
and the other methods. After the validation in the case of
YtoO, OtoY transformation is also verified.

A domain transform method to classify opacities inchest CT images using Cycle GAN2 has been proposed
as a method to solve the above problem. However, since
the training of domain transformation does not use
labeled data, the transform is not always suitable for theclassification of the lung opacities. Hence, we propose asemi-supervised Cycle GAN that combines the standard
Cycle GAN with a trained classifier that is trained inother medical institutions. In detail, 1) the trainedclassifier classifies a small number of images of anothermedical institution after the images are domaintransformed by Cycle GAN. 2) We examine whether thedomain transformation by Cycle GAN is appropriately
executed or not. 3) Cycle GAN re-train based on the
results. The above 1), 2) and 3) are repeated to make thedomain transformation suitable for the opacity
classification.

2.1. Creating ROI Images

2. Proposed Method

In this study, 503 chest CT images taken at Yamaguchi
University Hospital and 636 chest CT images taken at
Osaka University Hospital are used. Examples of thechest CT images taken at each hospital are shown in Fig.
2. The chest CT images of domains Y and O contain six
opacities: consolidation (CON), diffuse nodular (DN),
emphysema (EMP), ground-glass opacity (GGO),
honeycombing (HCM), and normal (NOR). The CT
images of domains Y and O are divided into 32×32[pixels] regions of interest (ROIs), which are used fortraining the Cycle GAN. Each CT image has a mask
image that shows the location of each opacity. The mask
images were created by three radiologists. 32×32 [pixels]
regions are scanned while striding from the upper left

This study aims to classify opacities of diffuse lungdiseases in chest CT images, where we also aim todiverse a classifier based on the domain transformationof semi-supervised Cycle GAN. In detail, we regard theconditions of the CT images taken at Osaka University
Hospital (hereafter referred to as Domain O) as thestandard domain and convert the CT images taken at
Yamaguchi University Hospital (hereafter referred to as
Domain Y) to the conditions of Domain O using semisupervised Cycle GAN. This domain transformationfrom domain Y to domain O is defined as YtoO
transformation.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

491

Domain Transformation of Chest

Table 1. Condition of each method. We compare four different methods including the proposed method. The architecture of
the proposed model is a combination of a standard Cycle GAN and a ResNet trained on the target domain: classifier trained at
the other medical institution. "ResNet trained with source domain" represents ResNet trained with only a few teacher labels
for semi-supervised learning.
Cycle GAN
Domain transformation method

With ResNet
Trained withsource domain

No domain transformation
Cycle GAN
Method for comparison
Semi-supervised Cycle GAN (Proposed method)Trained withtarget domain

Fig. 4. Structure of semi-supervised Cycle GAN. Semisupervised Cycle GAN contains a standard Cycle GAN
consisting of two generators and two discriminators, and also
contains a classifier Dcf_o that classifies the opacity.
Discriminator DO, which discriminates real data {o} andtransformed data G(y), learns to achieve high discrimination
accuracy. In other words, DO learns to maximize 𝑙𝑙𝑙𝑙𝑙𝑙 �1 −𝐷𝐷𝑂𝑂 �𝐺𝐺(𝑦𝑦)�� and 𝑙𝑙𝑙𝑙𝑙𝑙𝐷𝐷𝑂𝑂 (𝑜𝑜). While generator G, which converts
data from domain Y to domain O, learns to minimize
𝑙𝑙𝑙𝑙𝑙𝑙 �1 − 𝐷𝐷𝑂𝑂 �𝐺𝐺(𝑦𝑦)��. F and DY are trained in the same way as
G and DO. Dcf_o is a classifier that classifies the opacity G(y)where Dcf_o is trained to achieve high classification accuracy.

Fig. 2. Examples of chest CT images. The left is a chest CT
image taken at Yamaguchi University Hospital, and the right isa one taken at Osaka University Hospital.

2.2. Domain Transformation by Semi-supervised
Cycle GAN

Fig. 3. Training data and testing data of domain Y and O. The
created ROI images are split into a set of training data and test
data. 𝑌𝑌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 and 𝑂𝑂𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 are used to train Cycle GAN, while 𝑌𝑌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡
and 𝑂𝑂𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 are used to verify the effectiveness of the domaintransformation. 𝑌𝑌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡_𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 ⊂ 𝑌𝑌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 is a dataset of a small
number of 𝑌𝑌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 with class labels. Specifically, CT images offive patients are annotated for each opacity.

We apply domain transformation to the ROI images
created in Section 2.1 using semi-supervised Cycle GAN.
As shown in Fig. 3, 𝑌𝑌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 and 𝑂𝑂𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 are used to train
Cycle GAN, and the domain transformation is performed
on 𝑌𝑌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 and 𝑂𝑂𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 .
As shown in Fig. 4, the structure of the semi-supervised
Cycle GAN consists of a combination of the standard
Cycle GAN and a trained classifier. The purpose of thismodel is to learn domain transformation between two
domains Y and O with given training samples {𝑦𝑦𝑖𝑖 }𝑁𝑁
𝑖𝑖=1

corner of the mask image to the lower right corner. If theregions contains more than 50% of the masked area
indicating each opacity, the same areas are extracted
from the corresponding CT images and used as ROI
images.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

492

Masashi Miyake, Shingo Mabu, Shoji Kido, Takashi Kuremoto𝑀𝑀where 𝑦𝑦𝑖𝑖 ∈ 𝑌𝑌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 and �𝑜𝑜𝑗𝑗 �𝑗𝑗=1 where 𝑜𝑜𝑗𝑗 ∈ 𝑂𝑂𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 . N and

M are the numbers of samples in set 𝑌𝑌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 and 𝑂𝑂𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 ,respectively.
In the standard Cycle GAN, the loss functions Eq. (1)through (3) are used for training G, F, DY, DO.

+𝐸𝐸𝑜𝑜~𝑃𝑃𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑(𝑜𝑜) �𝑙𝑙𝑙𝑙𝑙𝑙 �1 − 𝐷𝐷𝑌𝑌 �𝐹𝐹(𝑜𝑜)���
𝐿𝐿𝑐𝑐𝑐𝑐𝑐𝑐 (𝐺𝐺, 𝐹𝐹)= 𝐸𝐸𝑦𝑦~𝑃𝑃𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑(𝑦𝑦) ��𝐹𝐹�𝐺𝐺(𝑦𝑦)� − 𝑦𝑦��

+𝐸𝐸𝑜𝑜~𝑃𝑃𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑(𝑜𝑜) ��𝐺𝐺�𝐹𝐹(𝑜𝑜)� − 𝑜𝑜��
𝐿𝐿𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 (𝐺𝐺, 𝐹𝐹)= 𝐸𝐸𝑜𝑜~𝑃𝑃𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 (𝑜𝑜) [‖𝐺𝐺(𝑜𝑜) − 𝑜𝑜‖]+𝐸𝐸𝑦𝑦~𝑃𝑃𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑(𝑦𝑦) [‖𝐹𝐹(𝑦𝑦) − 𝑦𝑦‖]

Opacity
CON
DN
EMP
GGO
HCM
NOR

𝐿𝐿𝐺𝐺𝐺𝐺𝐺𝐺 (𝐺𝐺, 𝐷𝐷𝑂𝑂 , 𝑌𝑌, 𝑂𝑂)= 𝐸𝐸𝑜𝑜~𝑃𝑃𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 (𝑜𝑜) [𝑙𝑙𝑙𝑙𝑙𝑙𝐷𝐷𝑂𝑂 (𝑜𝑜)]+𝐸𝐸𝑦𝑦~𝑃𝑃𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑(𝑦𝑦) �𝑙𝑙𝑙𝑙𝑙𝑙 �1 − 𝐷𝐷𝑂𝑂 �𝐺𝐺(𝑦𝑦)���
𝐿𝐿𝐺𝐺𝐺𝐺𝐺𝐺 (𝐹𝐹, 𝐷𝐷𝑌𝑌 , 𝑂𝑂, 𝑌𝑌)= 𝐸𝐸𝑦𝑦~𝑃𝑃𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 (𝑦𝑦) [𝑙𝑙𝑙𝑙𝑙𝑙𝐷𝐷𝑌𝑌 (𝑦𝑦)]Table 2. Stride width and the number of ROI images set
for each opacity.

(1)(𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎)

𝐿𝐿

�

3071
3023
3122
3460
3236
3117

3447
3311
3021
3273
3434
3035

Width of stride
Domain Y

Domain O

8
16
24
12
13
29

11
14
27
18
13
32

𝐿𝐿�𝐺𝐺, 𝐹𝐹, 𝐷𝐷𝐴𝐴 , 𝐷𝐷𝐵𝐵 , 𝐷𝐷𝑐𝑐𝑐𝑐_𝑜𝑜 � = 𝐿𝐿𝐺𝐺𝐺𝐺𝐺𝐺 (𝐺𝐺, 𝐷𝐷𝐵𝐵 , 𝐴𝐴, 𝐵𝐵)+𝐿𝐿𝐺𝐺𝐺𝐺𝐺𝐺 (𝐹𝐹, 𝐷𝐷𝐴𝐴 , 𝐵𝐵, 𝐴𝐴)+𝜆𝜆1 𝐿𝐿𝑐𝑐𝑐𝑐𝑐𝑐 (𝐺𝐺, 𝐹𝐹)+𝜆𝜆2 𝐿𝐿𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 (𝐺𝐺, 𝐹𝐹)(2)+𝜆𝜆3 𝐿𝐿𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜 �𝐺𝐺, 𝐷𝐷𝑐𝑐𝑐𝑐_𝑜𝑜 �

(3)

(5)

where λ1, λ2, and λ3 are bias terms, which are set to 40, 5,
and 0.2, respectively. G and F are trained to satisfy theobjective function expressed by Eq. (6).

We denote the data distribution as 𝑦𝑦~𝑝𝑝𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 (𝑦𝑦) and𝑜𝑜~𝑝𝑝𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 (𝑜𝑜). The domain transformation is learned byminimizing the loss function of Eq. (1), but since this loss
function alone will learn to map the same output pattern
to any input images, the loss functions Eq. (2) and (3) areintroduced. Eq. (2) is called Cycle Consistency Loss,
which constrains the original data 𝑦𝑦 and 𝑜𝑜 to match thegenerated data 𝐹𝐹�𝐺𝐺(𝑦𝑦)� and 𝐺𝐺�𝐹𝐹(𝑜𝑜)�, respectively. Eq.
(3) is called Identity Mapping Loss, which constrains thegenerator not to convert any data that have belonged tothe target domain.
In the semi-supervised Cycle GAN, we introduce a
new constraint to re-train Cycle GAN using ResNet. First,
we generate (fake) domain O data 𝐺𝐺(𝑦𝑦) from domain Y
by Cycle GAN. ResNet Dcf_o trained on domain O is used
to identify the data 𝐺𝐺(𝑦𝑦). The classification results are
fed back to the Cycle GAN for re-train. In the re-training,

only the ROI images �𝑦𝑦𝑘𝑘

The number of
ROI images
Domain Y Domain O

𝐺𝐺 ∗ , 𝐹𝐹 ∗ = 𝑎𝑎𝑎𝑎𝑎𝑎 𝑚𝑚𝑚𝑚𝑚𝑚 𝑚𝑚𝑚𝑚𝑚𝑚 𝐿𝐿(𝐺𝐺, 𝐹𝐹, 𝐷𝐷𝑌𝑌 , 𝐷𝐷𝑂𝑂 , 𝐷𝐷𝑐𝑐𝑐𝑐_𝑜𝑜 )𝐺𝐺,𝐹𝐹 𝐷𝐷𝑌𝑌 𝐷𝐷𝑂𝑂

(6)

YtoO transformation is realized by the above procedure.
3.

Experimental Results

The effectiveness of the domain transformation YtoO
and OtoY is verified using the trained ResNet. In YtoO
transformation, after domain Y data are transformed fake
O by YtoO transformation, they are input to ResNet
trained on domain O, and six-class classification is
performed. Then, we measure classification performance.
The classification performance of the other methods
shown in Table 1 is also examined to compare. We
perform t-test on the F-measure of the proposed method
and other methods to verify whether there are significant
differences in terms of p-values.

are used, where𝑘𝑘=1(𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎)
𝑦𝑦𝑘𝑘∈ 𝑌𝑌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡_𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 and 𝑌𝑌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡_𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛 contains annotated

3.1. Creating Dataset

CT images of five patients per opacity class. L is thenumber of samples in dataset 𝑌𝑌𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡_𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 . For learning G
in the semi-supervised Cycle GAN, the loss function
shown in Eq. (4) is used in addition to Eqs. (1) through(3).

If the stride width of each class is kept constant whengenerating ROI images, the number of ROI images ofeach class will be highly biased. Therefore, in this study,
we adjusted the stride width for each opacity to keep thenumber of ROI images as around 3000 for each class.
Table 2 shows the stride width and the number of ROI
images for each opacity. The generated ROI images have
opacity (class) labels based on the annotation by theradiologists.

𝐿𝐿𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜 �𝐺𝐺, 𝐷𝐷𝑐𝑐𝑐𝑐_𝑂𝑂 �= 𝐸𝐸𝑦𝑦 (𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎) ~𝑃𝑃𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑�𝑦𝑦(𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎)� [𝑙𝑙𝑙𝑙𝑙𝑙𝐷𝐷𝑐𝑐𝑐𝑐_𝑜𝑜 (𝐺𝐺(𝑦𝑦 (𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎) ))] (4)Our full loss function is:© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

493

Domain Transformation of Chest

Table 3. The number of ROI images used for training and testing. The number of training data for each opacity is about
1000, and that of testing data is about 2000. The same training data and testing data are used for both YtoO and OtoY
transformation.
Opacity

Training data
Domain Y

Domain O

Testing data
Domain Y

Domain O

CON
1022 (95)*1
1027 (104)*2
2049
2420
DN
1018 (107)1020 (98)
2005
2291
EMP
1020 (105)962 (105)
2102
2059
GGO
989 (108)996 (108)
2471
2277
HCM
1003 (96)1021 (96)
2233
2413
NOR
1003 (105)1024 (101)
2114
2011
*1 The values in parentheses are the number of annotated data used in the training of YtoO transformation.
*2 The values in parentheses are the number of annotated data used in the training of OtoY transformation.

the training of ResNet. The parameters for training were
20 epochs and mini-batch size 16, and we used Adam5 as
the optimization method. As a result of the training, theaccuracy for the training data in domain Y was 98.8%,
and that for domain O was 98.1%. The semi-supervised
learning of the domain transformations is performed by amodel that combines the trained ResNet with a standard
Cycle GAN. In the case of YtoO transformation, theResNet trained on domain O is incorporated into the
Cycle GAN.
3.3. Results of Domain Transformation by Semisupervised Cycle GAN
In this study, the number of residual blocks in thegenerator was set to 3 because the ROI image size input
to the Cycle GAN is small, i.e., 32 x 32 [pixels]. Since
the data are grayscale images, the number of input
channel is one. Cycle GAN was trained using the training
data shown in Table 3. The obtained images before andafter the domain transformation using the semisupervised Cycle GAN are shown in Fig. 5.

Fig. 5. Example of domain transformation using semi-supervised
cycle GAN. The left side shows ROI images after YtoO
transformation, and the right side show the one after OtoY
transformation.

3.4. Performance Evaluation
We measure the classification performance of ResNet on
the domain transformed ROI images. In the case of YtoO
transformation, ResNet trained with domain O was used
to classify the opacity of the transformed data fromdomain Y. For the performance evaluation, we usedprecision, recall, and F-measure. The results averaged
over 20 independent trials for YtoO and OtoY
transformation are shown in Table 4 and Table 5,
respectively. The proposed method shows the best

3.2. Results of ResNet Training
We trained ResNet using the generated ROI images.
ResNet learned the ROI-based opacity classification fordomains Y and O, respectively. We adopted ResNet34,
where the original filter size 7 x 7 of the first convolution
layer was changed to 3 x 3. Then we normalized the value
range to [-1,1] as a preprocessing of ROI images used for© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

494

Masashi Miyake, Shingo Mabu, Shoji Kido, Takashi Kuremoto

Table 4. Precision, recall and F-measure obtained by each method and the results of t-test in the case of YtoO transformation.
The values are the average over 20 trials. P-value is calculated by t-test on F-measure between the proposed method and othermethods.
MethodNo domain transformation
Cycle GAN
Method of comparison
Semi-supervised Cycle GAN

Precision (%)

Recall (%)

F-measurep-value36.5±8.6
67.9±3.6
71.3±2.074.0±1.0

28.6±6.4
65.8±4.2
68.4±2.672.7±0.9

0.228±7.80.655±4.50.682±2.60.727±0.9

1.83×10-17
4.73×10-7
1.13×10-7-

Table 5. Precision, recall and F-measure obtained by each method and the results of t-test in the case of OtoY transformation.
The values are the average over 20 trials. P-value is calculated by t-test on F-measure between the proposed method and othermethods.
MethodNo domain transformation
Cycle GAN
Method of comparison
Semi-supervised Cycle GAN

Precision (%)

Recall (%)

F-measurep-value35.0±4.575.2±1.673.4±2.776.4±1.4

37.0±2.273.4±1.6
70.1±5.974.7±1.4

0.319±1.90.730±2.00.692±6.40.745±1.45.80×10-42
6.00×10-3
1.13×10-3-

performance for both YtoO and OtoY transformation.
Besides, the standard deviation of the results obtained by
the proposed method is the smallest, confirming that theclassification performance is stable when the ResNet isdiverted.

3. Masashi Miyake, Shingo Mabu, Shoji Kido, Takashi
Kuremoto, Yasushi Hirano, Domain Transformation of
Chest CT Images Using Cycle GAN and Its Application toClassification Systems, The 38th JAMIT Annual Meeting,2019, pp. 108-115
4. Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun,
Deep Residual Learning for Image Recognition, IEEE
Conference on Computer Vision and Pattern Recognition，
2016
5. Diederik P. Kingma, Jimmy Lei Ba, ADAM: A method forstochastic optimization, Proceedings of the 3rdInternational Conference on Learning Representations,2015

4. Conclusion
In this study, we proposed a semi-supervised Cycle
GAN that combines a standard Cycle GAN with a trainedclassifier to perform domain transformation of chest CT
images. In the experiment, we classified the normal
opacity and five abnormal opacities of diffuse lungdiseases in the CT images after the domaintransformation and clarified the effectiveness of the
proposed method.
Acknowledgments
This work was supported by JSPS KAKENHI Grant
Number JP19K12120.
References
1. Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Geert
Litjens, et al., Pulmonary Nodule Detection in CT Images:
False
Positive
Reduction
Using
Multi-View
Convolutional Networks, IEEE Transactions on Medical
Imaging, 2016, pp. 1160-1169
2. Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros,
Unpaired Image-to-Image Translation using CycleConsistent
Adversarial
Networks,
International
Conference on Computer Vision, 2017

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

495

Anomaly detection of lung sounds using DAGMM
Ryosuke Wakamoto

Graduate School of Sciences and Technology for Innovation, Yamaguchi University,
2-16-1 Tokiwadai, Ube, Yamaguchi 755-8611, Japan
Shingo Mabu

Graduate School of Sciences and Technology for Innovation, Yamaguchi University,
2-16-1 Tokiwadai, Ube, Yamaguchi 755-8611, Japan
Shoji Kido

Graduate School of Medicine, Osaka University,
2-2 Yamadaoka, Suita, Osaka 565-0871, Japan
Takashi Kuremoto

Graduate School of Sciences and Technology for Innovation, Yamaguchi University,
2-16-1 Tokiwadai, Ube, Yamaguchi 755-8611, Japan
E-mail: b093vg@yamaguchi-u.ac.jp, mabu@yamaguchi-u.ac.jp
kido@radiol.med.osaka-u.ac.jp, wu@yamaguchi-u.ac.jp

Abstract
There are only a few small-scale benchmark datasets of lung sounds that are annotated for the training of machinelearning. Therefore, we aim to build an anomaly detection system that only uses normal data that can be obtained
more than abnormal data. We propose an algorithm that improves the Deep Autoencoding Gaussian Mixture Model(DAGMM), where various types of neural networks are applied to DAGMM as the compression networks.
Experimental results show that the proposed methods obtain effective classification performance.
Keywords: lung sounds, deep learning, anomaly detection, DAGMM

on computer-aided diagnosis using deep learning has
been attempted for image diagnosis, which has greatly
contributed to improve the accuracy of medical image
diagnosis such as lung diseases2 which are dealt with in
this study. However, diagnosis with only images is not
sufficient because, in addition to the limitations ofqualitative diagnosis in imaging, there are multiple ways
to diagnose lung diseases. In the diagnosis of lungdiseases, the patient is interviewed and physicallyexamined, and the condition of the disease is estimatedto some extent, and respiratory function tests, image
diagnosis, electrocardiography, and gait assessment are

1. Introduction
In recent years, deep learning algorithms based on large
amounts of data have been proposed and applied invarious fields1. In the field of image recognition, inparticular, various applications have been made using
convolutional neural networks (CNNs). Because of its
availability, CNNs have been applied to the medical field
where high accuracy is required.
For example, various approaches have been developed
for medical images, such as feature extraction specific toa lesion or 3D modeling of an organ. Most of the research© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

496

Ryosuke Wakamoto, Shingo Mabu, Shoji Kido, and Takashi Kuremoto

peripheral and time series information. These three typesof networks have been shown to be useful indiscriminating lung sounds in previous studies, and since
the standard Multi-Layer Perceptron (MLP) is used as acompression network in the conventional DAGMM, theabove three networks are expected to show better
performance in DAGMM. In our experiments, we
compare the conventional DAGMM and the proposed
methods, and better classification performance isobtained by improving the compression networks in
DAGMM.
The paper is organized as follows. Section 2 describes therelated work and Section 3 presents the proposed method.
Section 4 presents the experimental conditions andresults, and Section 5 presents conclusions and future
work.

performed3. Because of this, the burden on the physicians
making the diagnosis is heavy, requiring precise
treatment and enough experiences.
As mentioned above, because doctors need to conductvarious tests for diagnosis, it is necessary to develop
diagnostic support techniques for not only visual
information such as images, but also for other senses suchas hearing. Therefore, we consider applying deeplearning-based speech recognition to auscultation which
is one of the diagnostic methods for lung diseases. If we
can construct a system for discriminating sounds, we can
improve the accuracy of diagnosis by combining it withother examination results such as imaging.
Although deep learning requires a large amount of sounddata to learn the features of lung sounds, there are few
small benchmark datasets with annotation (classlabeling), which is necessary for machine learning. In
addition, it takes a great deal of effort and time for each
medical facility to acquire a sufficient amount of sounddata. When training is performed with a small amount of
data, it may not obtain generalized performance because
it is difficult to capture essential features that areimportant for the diagnosis due to the large individual
differences of lung sounds. There have been someresearch related to lung sounds classification, such as theanalysis using histogram statistics4 and the lung sounds
classification using deep learning5, however, the numberof data to generate classification models is limited in bothmethods. Therefore, this study focuses on normal data,
which is easier to obtain than abnormal data, andconstructs a system for detecting abnormalities of
pulmonary auscultation sounds using deep learning withthe aim of capturing generalized features even with smalldata.
In this paper, we propose a deep learning algorithm forfeature extraction of lung sounds that improves the Deep
Autoencoding Gaussian Mixture Model (DAGMM)6,
which is an anomaly detection algorithm capable oflearning feature extraction and clustering at the sametime. Specifically, it uses Mel-Frequency Cepstral
Coefficients (MFCC)7 for the feature extraction and three
types of networks (auto-encoders) based on: 1) CNN, 2)
Long short term memory (LSTM), 3) Convolutional
LSTM (C-LSTM) are applied to DAGMM ascompression networks to design an algorithm for
efficient feature extraction. C-LSTM has features of both
CNN and LSTM, so it can consider the features of both

2. Related work
There are two types of research related to sound andspeech: speech analysis and speech recognition. Speech
analysis is a process of extracting features from thesampled sound data for speech recognition. Speech
recognition is the process of identifying the target soundor speech based on the features obtained by the speech
analysis. In addition to these processes, this section also
describes the anomaly detection.
2.1. Speech Analysis
Speech analysis improves recognition accuracy byeliminating unnecessary information as noise as well as
extracting only the features necessary for speech
recognition. In this study, we use MFCC, a feature
extraction method to extract the cepstrum by applying afilter based on the human hearing (Mel-filter bank),
which separates the formant frequencies, which isnecessary for speech recognition, from the pitch
frequencies, which contains individual differences. Since
Formant frequencies of MFCC appear in the lowfrequency regions, important information for soundclassification can be obtained by extracting the lower
dimensional components of MFCC. Fig. 1 shows anexample of MFCC that shows the 20-dimensionalfeatures of MFCC for 5-second of lung sounds. The
horizontal axis of Fig. 1 represents the 20 segments of the
5-second data and the vertical axis represents thedimensions of MFCC.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

497

Anomaly detection of lungIn addition, image recognition methods such as CNNs areoften utilized in speech recognition10. CNNs are widelyused in image recognition, but they also have excellent
performance as a local feature extractor in speech
recognition. Both CNNs and LSTMs are useful forfeature extraction of lung sounds, and in the previous
study, classification models using C-LSTM thatpossesses both CNN and LSTM features have shownhigher accuracy11.
2.3. Anomaly Detection
Fig. 1. MFCC with 20 dimensions

Anomaly detection is a method that creates a distributionof normal data by learning only normal data, then,regards the data that do not belong to the distribution of
normal as abnormal data. In this section, we describe theoutlier (anomaly) detection methods using machinelearning. In general, unsupervised outlier detection
methods execute feature extraction and distribution
creation separately. For example, auto-encoder is firstused to extract features from the input data, thenclustering methods such as k-means and gaussian
mixture model (GMM) are applied to create distributions.
One of the problems in such learning structures is that thelearning of feature extraction and clustering are executed
separately. The features extracted independently from theclustering process may not be useful. In order toaccurately detect outliers, a coherent feature extraction isrequired for clustering.
The structure of DAGMM is shown in Fig. 3. DAGMM
is an outlier detection method that performs feature
extraction and clustering simultaneously. The feature
extraction is realized by a compression network using
auto-encoder and the clustering is realized by combining
an estimation network and GMM. The compression
network outputs the encoded feature and thereconstructed data of the inputs, and the estimation
network outputs the probability that the input 𝑥𝑥 belongs
to each cluster. In the compression network, input 𝑥𝑥 is
encoded to feature 𝑍𝑍𝑐𝑐 . Furthermore, the feature 𝑍𝑍𝑐𝑐 is
decoded to obtain the reconstructed image 𝑥𝑥 ′ . In thecompression network, auto-encoder learns that 𝑥𝑥 and 𝑥𝑥 ′become the same, where the reconstruction error 𝑍𝑍𝑟𝑟 isrepresented as follows.

2.2. Speech Recognition
Before deep learning is actively studied, dynamic
programming (DP) matching8 and hidden Markov
models (HMM)9 were used for speech recognition,
however, once deep learning was applied to speech
recognition, Recurrent Neural Network (RNN) wereapplied and false recognition rate of speech wassignificantly decreased. However, it was difficult to learn
long data stably due to the problem of gradient loss. Then,
LSTM shown in Fig. 2 were used. LSTM has three gates:
an input gate (i. in Fig. 2), a forget gate (ii. in Fig. 2), an
output gate (iii. in Fig. 2), and a cell that holds past
information. The problem of gradient loss is solved byusing gates to select the information in the cells.

𝜎𝜎: Sigmoid function
𝐶𝐶: Cell memoryℎ: Cell output
𝑓𝑓: Forget gate𝑖𝑖: Input gate
𝑜𝑜: Output gate�𝑥𝑥−𝑥𝑥 ′ �𝑥𝑥∙𝑥𝑥 ′

𝑍𝑍𝑟𝑟 = (𝑑𝑑1 , 𝑑𝑑2 ) = � ‖𝑥𝑥‖ 2 , ‖𝑥𝑥‖ ‖𝑥𝑥 ′‖ �.

Fig. 2. Structure of LSTM

2

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

498

2

2

(1)

Ryosuke Wakamoto, Shingo Mabu, Shoji Kido, and Takashi KuremotoFig. 3. Structure of DAGMM

By using the two measures 𝑑𝑑1 and 𝑑𝑑2 , we can measure
the reconstruction error from different perspectives. Then,the concatenated feature of 𝑍𝑍𝑐𝑐 and error 𝑍𝑍𝑟𝑟 generated by
the compression network forms a new feature 𝑍𝑍. With
this as the input to the estimation network, the estimation
network then outputs the affiliation probability 𝛾𝛾� of how
well 𝑍𝑍 is matched to each cluster. Then, GMM is
generated by the feature 𝑍𝑍 and the affiliation probability
𝛾𝛾�. GMM requires three parameters: the mixture ratio 𝑝𝑝ℎ𝑖𝑖,
the mean matrix 𝑚𝑚𝑚𝑚, and the covariance matrix 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠.
The formula for each is as follows.
�
𝛾𝛾

𝑖𝑖𝑖𝑖
𝑝𝑝ℎ𝑖𝑖𝑘𝑘 = ∑𝑁𝑁
.
𝑖𝑖=1

(2)

𝑁𝑁

∑𝑁𝑁 𝛾𝛾� 𝑍𝑍𝑖𝑖𝑖𝑖 𝑖𝑖
𝑚𝑚𝑚𝑚𝑘𝑘 = ∑𝑖𝑖=1.
𝑁𝑁 �

𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑘𝑘 =𝑖𝑖=1 𝛾𝛾𝑖𝑖𝑖𝑖∑𝑁𝑁
�𝑖𝑖𝑖𝑖 (𝑍𝑍𝑖𝑖 −𝑚𝑚𝑚𝑚𝑘𝑘 )(𝑍𝑍𝑖𝑖 −𝑚𝑚𝑚𝑚𝑘𝑘 )𝑇𝑇
𝑖𝑖=1 𝛾𝛾∑𝑁𝑁
�𝑖𝑖𝑖𝑖
𝑖𝑖=1 𝛾𝛾In this study, we deal with an anomaly detection
algorithm for lung sounds. Since previous studies in lungsounds have been concerned with the lack of data, ananomaly detection algorithm is suitable because normal
data is easier to obtain than abnormal data and obtains
generalized features even with a small number of data. In
this study, the conventional DAGMM is regarded as theconventional method that realizes anomaly detection
performing both clustering and feature extraction
simultaneously.
From the next subsection, we propose a deep learningalgorithm that improves the feature extraction of
DAGMM for lung sounds. Specifically, we replace thecompression network in DAGMM in Fig. 3 with various
types of auto-encoders based on CNN, LSTM and CLSTM. Since the three auto-encoders based on CNN,
LSTM and C-LSTM have different structures andcharacteristics, the details of each structure are described
one by one.

(3)
(4)

.

DAGMM calculates the energy of the feature 𝑍𝑍 after thegeneration of the GMM: when a data is located in thecenter of the distribution of the GMM, the energy of 𝑍𝑍 issmall, and the energy of the off-centered 𝑍𝑍 is large. Theenergy function is represented as follows.
𝐸𝐸(𝑍𝑍) = −𝑙𝑙𝑙𝑙𝑙𝑙 �∑𝐾𝐾
𝑘𝑘=1 𝑝𝑝ℎ𝑖𝑖𝑘𝑘

12𝑒𝑒𝑒𝑒𝑒𝑒�− (𝑧𝑧−𝑚𝑚𝑚𝑚𝑘𝑘 )𝑇𝑇 ∑𝑘𝑘−1(𝑧𝑧−𝑚𝑚𝑚𝑚𝑘𝑘 )�
�|2𝜋𝜋𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑘𝑘 |�.

3.1. Convolutional auto-encoder (CAE)The structure of a CAE is shown in Fig. 4. The input datais converted to an image as shown in Fig. 1 by applying
MFCC to 5-second unlabeled sound data. Therefore, it ispossible to perform convolutional processing on sounddata as the same way as on images. CAE encodes the
input by an Encoder consisting of two convolutional andtwo pooling layers, and then reconstructs the input by a
Decoder consisting of two deconvolutional and two
upsampling layers. The structure of the compression
network with convolution and pooling captures the local
features of MFCC making it easier to ignore the noisethat is likely to have an influence on speech recognition.

(5)

DAGMM can learn feature extraction suitable forclustering by the above learning structure. In addition,
when outlier detection is performed, the energy ofabnormal data is expected to be larger than that of normal
data.
3. Proposed methods© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

499

Anomaly detection of lungFig. 4. Structure of CAE

Fig. 5. Structure of LSTM-AE

the effect of giving information on the inverse time axis.
The combined features described above (20×100) are
input to the Decoder LSTM row by row in time order(1×100), and the Decoder LSTM outputs the features(80) corresponding to each time. The output of onedimensional features (80) is converted to a size (20)equivalent to the number of columns, i.e., 20 time
segments of the input data, where the conversion is
implemented by MLP. Here, the features (20) generated
at all the time (20) are joined in sequence to obtain datawith the same size as the input data (20×20). The purposeof an auto-encoder using LSTM is to make the output be
the same as the input.

3.2. LSTM-based auto-encoder (LSTM-AE)
The structure of a LSTM-AE is shown in Fig. 5 The
LSTM-AE dealt with in this study is based on the
structure proposed in the literature12. As in the case of
CNN, we use unlabeled data to train Encoder LSTM and
Decoder LSTM. In this case, the details of the network
structure are as follows. First, Encoder LSTM outputs 80
features from the hidden layer of LSTM. Then, the input
data to the Encoder LSTM (20×20) is mirrored and added
to the generated features. This process is carried out by
duplicating the generated one-dimensional features (80)
20 times to form a two-dimensional features (20×80) as
shown in Fig. 5, and the generated two-dimensionalfeatures is combined with the mirroring input (20×20).
This yields a new feature (20×100). Here, we explain themeaning of combining the generated features andmirroring input. Since the features (80) obtained by the
Encoder LSTM has lost the information on the time axis,
it is difficult to reconstruct the input image at the output
layer. Therefore, the mirroring input serves as a flag forreconstructing the input from the features, and also has

3.3. C-LSTM-based auto-encoder (C-LSTM-AE)
C-LSTM is a neural network that uses convolution layers
when inputs to the sigmoid function of each gate of
LSTM (ⅰ., ⅱ., and ⅲ. in Fig. 2) are calculated, and when
the input to the tanh layer (ⅳ. in Fig. 2) is calculated.
Thus, LSTM can implement data processing considering
peripheral information. The output shape is the same as
the CNN, which is (the size of the feature map after© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

500

Ryosuke Wakamoto, Shingo Mabu, Shoji Kido, and Takashi KuremotoFig. 6. Structure of C-LSTM-AE with convolutional decoder

Fig. 7. Structure of C-LSTM-AE with LSTM decoder

obtained by the C-LSTM's encoder into a onedimensional features (320). In addition, the mirroring
input is generated for combining the one-dimensionalfeatures and the mirroring input. Then, Decoder LSTM
outputs a one-dimensional features (80) every time step,and it is converted by MLP into the array with the size(20) which is the same as the number of columns, i.e., 20
time segments of the input data. Therefore, the shape ofthe final output becomes the same as the input. Since we
use C-LSTM and pooling layers in the Encoder, it isexpected to learn the features considering time series
information while mitigating the effects of noises in the
input.

convolution) x (the number of feature maps). We propose
two structures of C-LSTM-AEs: the first is the C-LSTMAE with convolutional decoder and the second is the CLSTM-AE with LSTM decoder. The former structure isshown in Fig. 6 and the latter structure is shown in Fig. 7.
The difference between the two is the structures of thedecoders, which is described in detail in the next subsubsection.
3.3.1. Convolutional decoder
In Fig. 6, the decoder consists of two deconvolutional
layers and two upsampling layers, which is similar to thedecoder used in the CAE. As the feature map of C-LSTM
is two-dimension like CNN, the network structure issimilar to that of CAE using deconvolution layers in the
decoder. This structure is expected to capture local
features while taking into account time-series
information.

4.

Experiments and results

4.1. Data overview
In this experiment, we used lung sounds data provided by
Yamaguchi University Hospital, Japan to discriminate
between discontinuous rale3 (abnormal sounds) andnormal sounds. The data used in this study are 5-second
data, which were cut out from the data recorded during
auscultation in a private room with the subjects in a

3.3.2. LSTM-Decoder
In Fig. 7, the decoder has the same structure as that usedin the LSTM-AE, which converts the features (2×2×80)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

501

Anomaly detection of lungseated position, and judged to be normal, fine crackle andcoase crackle by the doctor. Here, the 12 areas were used
for auscultation, that is, the chest area was firstly divided
into four areas: upper and lower portions of both left andright lungs, respectively, and furthermore, the threeareas : anterior, lateral, and posterior portions of the lungs
are considered. Therefore, auscultation is done at totally
12 areas of lung, and the lung sounds of each area wererecorded for more than three respiratory phases(inhalation plus exhalation) and more than about 15
seconds. Although the data were acquired from the threeareas in the front, side, and back, the number of channels
was 1 because the diagnosis was made independently
from each area. The sampling rate was set to 11 kHz andthe digital stethoscope (Power stethoscope, Starkey
Japan) was connected to a voice recorder (ICD-MS1,
Sony) and the sounds were recorded in a flash memoryin 16-bit WAV file format. It is noted that noises caused
during data acquisition depend on the way of breathing,subtle shifts in the stethoscope's position, and differencesin the loudness of the sound. If an user is not skilled inhandling the stethoscope, a lot of noises are picked up. In
this study, the doctor who has enough experiences ofauscultation recoded the lung sounds. The discontinuous
rale in this study are discontinuous rales of short duration,and can be classified into two types: fine crackle andcoarse crackle. Table 1 shows the characteristics of each
auscultation sound including the normal sound. The
characteristics of the abnormal intermittent sounds show
sudden appearance, high sound pressure level, and very
short duration. In addition to the paucity of data, thefrequency bands of normal and abnormal sounds areoverlapped, and the individual differences are large, thus,
frequency analysis is difficult for the classification. In
this study, to deal with the anomaly detection algorithm,
two types of discontinuous rale are grouped together as
one abnormal class.

Table 1. Characteristics of lung sounds4
Class
Frequency[Hz]
Duration[ms]
Estimateddiseases

Abnormal

NormalCoarseFine

250-500

200-500
700-1000

150-600

10-15

Less than 5-

Bronchitis, Interstitial
Pneumonia, pneumonia,
Pulmonary Pulmonary
tuberculosis
fibrosis-Table 2. Overview of data of each class
Class

Abnormal

NormalNumber of data
Number of patients(male, female)

79
24(16, 6)140
12(12, 0)*Abnormal includes two of unknown sex patients.

study, we conducted an experiment with 14-fold crossvalidation. The evaluation index is Area under the Curve(AUC) which is area under the ROC curve, and thethresholds of the classification boundary of normal andabnormal when calculating ROC is determined by
dividing the energy range of normal data into 11 ranges.
In order to achieve high performance with this evaluation
method, it is necessary to give lower energy to normal
data and higher energy to abnormal data. In other words,
it is necessary to obtain generalized features of normal
data in training.
For the input data, pre-processing of data sampling(sampling frequency of 2000 Hz), calculation of MFCC,
and normalization (mean 0, variance 1) were appliedprior to the training of each method. Because lung sounds
are diagnosed mostly at frequencies below 2000 Hz13
clinically, the sampling rate was set at 2000 Hz.

4.2. Experimental conditions
The anomaly detection method was applied to twoclasses of auscultation sounds: abnormal and normal. The
number of data of each class, the number of patients
including their sex are shown in Table 2. 130 of thenormal data were extracted as training data, and 10
normal data and 79 abnormal data that were not includedin the training data were extracted as test data. In this

4.3. Results
Table 3 shows the AUC scores obtained by theconventional and proposed methods after 14-fold crossvalidation. From Table 3, we can see that the mean AUC© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

502

Ryosuke Wakamoto, Shingo Mabu, Shoji Kido, and Takashi Kuremoto

Table 3. AUC scores of each method obtained by 14-fold cross validation
Methods
Folds

DAGMM

DAGMM with
CAE

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Mean

0.9316
0.86650.9386
0.86330.8506
0.79490.85190.89620.96650.91460.88420.87090.85060.74110.87300.97910.87220.96330.83990.91010.83100.8696
0.95570.98040.98670.9006
0.95950.91770.90440.9193

DAGMM with
DAGMM with
C-LSTM-AE
C-LSTM-AE(conv)(LSTM)
0.89490.9911
0.99240.89300.89620.90380.9032
0.97410.97660.72410.84370.89180.69870.93800.93230.76330.87150.88920.8886
0.89300.89490.95700.98990.98610.98730.98800.99300.96080.96840.96330.9386
0.93420.94620.8842
0.97280.96460.8842
0.94490.95320.88230.93990.92720.87570.93900.9439(conv): convolutional decoder, (LSTM): LSTM decoder

DAGMM with
LSTM-AE

Table 4. Standard deviation of each methods by 14-folds cross validation
MethodDAGMM

DAGMM withCAE

DAGMM with
LSTM-AE

Standard deviation

0.05600.05080.0841DAGMM with
C-LSTM-AE(conv)
0.0452DAGMM with
C-LSTM-AE(LSTM)
0.0365significant for LSTM but significant for the otherproposed methods, it can be said that the peripheral
information considered by convolution layers and themitigation of the influence of noises by the pooling layers
are related to the superiority of the performance in thefeature extraction of lung sounds. When comparing CLSTM-AE(conv) and LSTM-AE, C-LSTM-AE shows
the better result with a significant difference, thus, thecombination of both the peripheral information obtainedby convolution and the time series information obtainedby LSTM is useful in the feature extraction of lungsounds. Although time series information includes theimportant features in the lung sounds, it is susceptible tonoises, thus it is necessary to use convolution and pooling
together. When comparing C-LSTM-AE(LSTM) and CLSTM-AE(conv), LSTM decoder is better because it
recovers time series information more accurately.

score (0.9439) obtained by C-LSTM-AE with LSTM
decoder is the best. The standard deviation obtained byeach method is shown in Table 4, where C-LSTM-AE
with LSTM decoder is also show the lowest standard
deviation, therefore, the AUC scores are not deviated bya wide margin compared to other models in eachvalidation.
Here, we discuss the usefulness of the proposed method.
While the average AUC score of the conventional
DAGMM without the proposed method is 0.8730, theaverage AUC scores of all the proposed methods arehigher than that of the conventional DAGMM.
The results of the t-test of all methods are shown in Table
5. We can see from Table 5 that there is significant
differences between the proposed methods and theconventional DAGMM, except for LSTM-AE, where thesignificance level is 5% (<0.05). It can be said that theperformance of DAGMM can be improved by improving
the compression network. Since the difference is not© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

503

Anomaly detection of lung

Table 5. P-values of the results of t-test obtained by each method
DAGMM
DAGMM
DAGMM withCAE
DAGMM with
LSTM-AEDAGMM with
C-LSTM-AE(conv)DAGMM with
C-LSTM-AE(LSTM)DAGMM with DAGMM withCAE
LSTM-AE-

1.06 × 10−3-

4.48 × 10−1--

1.43 × 10−2
-

-

-

-

-

-

-

-

DAGMM with
C-LSTM-AE(conv)
1.58 × 10−4

DAGMM with
C-LSTM-AE(LSTM)
1.43 × 10−5

1.76 × 10−3

1.41 × 10−3

2.29 × 10−4

3.39 × 10−4-

1.20 × 10−1--

6. B. Zong, Q. Song, M. R. Min, W. Cheng, C. Lumezanu, D.
Cho, H. Chen, Deep autoencoding gaussian mixture model
for unsupervised anomaly detection, International
Conference on Learning Representations, 2018
7. B. Logan, Mel Frequency Cepstral Coefficients for Music
Modeling, Proceedings of the International Symposium onMusic Information Retrieval (ISMIR) , 2000
8. H. Sakoe, Two-level DP-matching-A dynamic
programming-based pattern matching algorithm forconnected word recognition, IEEE Transactions on
Acoustics, Speech, and Signal Processing 27(6), 1979, pp.
588-595
9. A. P. Varga and R. K. Moore, Hidden Markov model
decomposition of speech and noise, Proceedings of the1990 International Conference on Acoustics, Speech, andSignal Processing 2, 1990, pp. 845-848
10. R. Narasimhan, X. Z. Fern, and R. Raich, Simultaneous
segmentation and classification of bird song using CNN,
Proceedings of the 2017 International Conference on
Acoustics, Speech, and Signal Processing, 2017, pp. 146150
11. R. Wakamoto, S. Mabu, S. Kido, and T. Kuremoto, LungSound Classification Using Deep Neural Networks with
Pre-training - Comparison of the Performance between
CNN, LSTM and Convolutional LSTM -, IEEJ
Transactions on Electronics, Information and Systems
140(12), 2020, pp. 1402-1409 (in Japanese)
12. N. Srivastava, E. Mansimov, and R. Salakhutdinov,
Unsupervised learning of video representations using
LSTMs, Proceedings of the 32nd International
Conference on Machine Learning 37, 2015, pp. 843-852
13. V. Gross, A. Dittmar, T. Penzel, F. Schüttler and P. V.
Wichert, The Relationship between Normal Lung Sounds,
American Journal of Respiratory and Critical Care
Medicine 162(3), 2000, pp. 905-909

5. Conclusions
In this paper, various types of compression networks areproposed and evaluated by the AUC scores. The
experimental results showed that utilizing convolution
and pooling was effective in learning lung sounds, andutilizing time-series information together improved theperformance of the model. In the future, we would like tocompare these methods with other anomaly detection
methods and improve the generalization abilities.
Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP19K12120.
References
1. Y. LeCun, Y. Bengio, and G. Hinton, Deep learning,
Nature 521(7553), 2015, pp. 436-444
2. M. Anthimopoulos, S. Christodoulidis, L. Ebner, A.
Christe, and S. Mougiakakou, Lung Pattern Classification
for Interstitial Lung Diseases Using a Deep Convolutional
Neural Network, IEEE Transactions on Medical Imaging
35(5), 2016, pp. 1207-1216
3. T. Ishihara, T. Kawashiro, T. Abe, K. Kikumaru, and M.
Yonemaru, Auditory training by CD, Nankodo, 1993 (inJapanese)
4. Y. Igami, H. Shouno, and S. Kido, Identification of
pulmonary auscultation data using statistics of adjacent
waveform intervals, The 9th IEEE Hiroshima Student
Symposium (HISS2007) B53, 2007 (in Japanese)
5. D. Bardou, K. Zhang, and S. M. Ahmad, Lung sounds
classification using convolutional neural networks,
Artificial Intelligence in Medicine 88, 2018, pp. 58-69© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

504

A ground reaction force analysis in walking and running gaits in horse leg model on
viscoelastic hoof-ground contact
Dondogjamts Batbaatar

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku,
Kitakyushu 808-0196, Japan
Hiroaki Wagatsuma

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku,
Kitakyushu 808-0196, JapanRIKEN Center for Brain Science, Saitama, Japan
E-mail: batbaatar.dondogjamts581@mail.kyutech.jp, waga@brain.kyutech.ac.jp
www.kyutech.ac.jp

Abstract
In this paper, we focused on the hoof-ground interaction in the simplified horse leg model because walking andrunning gaits are known to be different in trajectories; however, the force analysis still remains as unsolved issues.
The computational experiments in Matlab, elastic and inelastic impact with the ground was resolved by using thedissipative contact force model and the ground reaction force was clearly examined in four different conditions fromthe combination of walking/running and elastic/inelastic contact.
Keywords: Closed-loop linkage; Multibody dynamics; Dissipative contact force model; hoof-ground interaction.

compliant human leg dynamics has been studied with aseries of contact points on the rolling surface [4-6] andwith the effective roll-over geometry of leg motion [7-9].
The fact suggests that the utilization of the simple
skeleton model provides a large benefit to compensate
the limitation of experimental measurements in the gain
analysis to realize the actual phenomenon of the groundreaction force, because there are evidences in the walking
condition [10,11], such as demonstrated two peaks in thetemporal evolution of the ground reaction force during
the stance phase, while there is a lack of evidences inother gait patterns. In robotic applications inspired from
biological mechanisms, an articulated leg with hoof was
inspired from the horse [12] and a linkage mechanism to
mimic the horse leg trajectory by Batbaatar &Wagatsuma (2019) [13] and similar approaches were
found in closed-loop linkage mechanisms [14,15]. In the
animal locomotion, there are various gait patterns known

1. Introduction
In the human gait analysis, three different foot-rocker
mechanisms are known as heel rocker, ankle rocker andforefoot rocker, which are generated from pendulum
dynamics with three fulcrum points of heel, ankle and toe[1]. In a gait cycle, there are two phases as swing stancephases and particularly in the human gait analysis, thestance phase is decomposed to the initial contact for
touching with the ground from the heel, the loading
response for shifting of the center of mass from back tofront, the mid-stance for lifting the heel up, the terminal
stance for transition to the swing phase, and then preswing [2]. Such as bipedal gait cycle can be simplified toa mechanical system based on a coupled pendulum as
demonstrated by McGeer (1990) [3], called a passive
dynamic walking robot without any electromechanicalactuator to mimic three rockers. In this case, the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

505

Dondogjamts Batbaatar, Hiroaki Wagatsuma

and orientation of each body in leg mechanism expressed
as generalized coordinates can be written

as walk, trot, gallop and so on [16]. Therefore, theestablishment of the detail analytical method for the
ground reaction force is highly important to elucidate theprinciple to differ gaits in the sense of the energy
consumption. Focusing on the ground contact
phenomenon, the hoof part of the horse leg mechanism
can be modelled in the form of the triangle structure toconnect three representative points as the toe, heel andankle, which reproduces the rocker mechanisms. If it ispossible to provide kinematic/kinetics of the hoof-groundcontact, the viscoelastic effect is also considerable toanalyse how much the leg absorbs the impact force whentouching with the ground. It realizes the detail analysis of
the amount of the energy storage discussed in the distal
leg movement and landing motions [17,18]. The purposeof the present study is to establish the basement analyticalmethod for the ground reaction forces in gaits anddemonstrate the efficacy of the method in the simplified
leg model to focus on the hoof-ground interaction. This
paper is divided into following sections. Section 2
introduces the general Multibody dynamics (MBD)
formulation applied to the proposed leg mechanism aswell as the viscoelastic contact force introduced in order
to simulate realistic hoof-ground contact. Section 3
contains the kinematic and kinetic analysis focusing on
the hoof-ground interaction, while Section 4 summarizes
the main results and discusses the potential
improvements and limitation of the study.

TT 
q =  q1T , q2T , q3T ,..., q14
 .

(1)

The vector of q contains 42 (= 14 × 3) elements whichare 14 rigid links and their center of position 𝑥𝑥𝑖𝑖 , 𝑦𝑦𝑖𝑖 andorientation 𝜙𝜙𝑖𝑖 are obtained in the generalized coordinatesin the present analysis. A set of kinematic algebraic
constraint equation according to the given initialconfiguration can be written Φ K (q) 0.
=Φ ( q, t ) =(2)Φ D (q, t )  42×1
where the first 41 elements of the column matrix ofkinematic constraint equation Φ K (q ) are derived fromthe absolute constraints between body and fixed groundnode. The last element Φ D (q, t ) defines the driver
constraint (Crankshaft) of the proposed leg mechanism.
The partial derivative of kinematic constraint equation
respect to the generalized absolute Cartesian coordinates
q is Jacobian matrix Φq is obtained as ∂Φ ( q, t ) Φq = , ∂q  42×42(3)

where it allows us to investigate placement, velocity andacceleration analyses kinematically. The forward
dynamics analysis introduces the mass matrixM= ( 42 × 42 ) , and the generalized external force vector(a)h=( 42 ×1) , as follows:

2. Methods

M = diag ( M1 , M 2 ,..., M14 ) ,2.1. Multibody system formulation for horse leg
mechanism={M i

Kinematics and dynamic analysis of horse leg
mechanism with multibody dynamics was presented in
the article by Batbaatar & Wagatsuma (2019) [13] inwhich simplified model of leg mechanism generated a
flexible coordinated movement of hoof with constraint
closed-loop linkage structure. Furthermore, in applying
appropriate driver constraint in system as principle
movement strategy in running animal functional andanimal-like end trajectories were generated and evaluatedwith duty factor which defines the specific locomotor
behavior with respect to the stance and swing timing in
one locomotive cycle. Mathematical model for horse leg
mechanism can be stated by using multibody dynamics(MBD), the vector q with 42 elements including positionTmi , mi , J i ] i 1, 2,...,14},[=TT( a )T h(a) =  h1( a ) , h2( a ) ,..., h14



T

T

{h(a)
1, 2,...,14},[0, −mi g , 0] i =
i =

(4)
(5)
(6)
(7)

where mi is the mass of rigid link to point i, J i = 2li 3
is the polar moment of inertia of rigid link to point i, andg is the gravitational acceleration. The equation motionof the system for the computer system analysis can be isexpressed in general matrix form as

M

Φq


ΦqT   q h( a )
  = .
2 − β Φ 
0  λ  γ − 2αΦ© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

506

(8)

A ground reaction force

where q is the acceleration vector, h( a ) denotes thegeneralized external force vector, λ is the Lagrange
multipliers
withvectorΦq q =− Φq q q − 2Φqt q − Φtt =γ . The parameter αqand β are the feedback compensator in Baumgarte
stabilization method [19] used to guarantee accuracy ofthe numerical solution. In the forward dynamic analysis,
new coordinates and velocities require two arrays for q
and q for the time step t + ∆t as(from the lowest point of the locomotive trajectory in thesense of the relative indentation.

 δ
−δ

)

δ =

0)
(δ > 0 FN =
δ
F
0
0)
≤
≥
(
N

(11)

In Eq. 11, the positive value of δ is that distance
represent a separation, while negative values denoterelative penetration of the contacting bodies. Therefore,
the sign of penetration indicates the phase transition fromswing to stance and vice versa for the case of hoofground contact. By using Eq. 10, it is clear todiscriminate walking phases and contact forces as
continuous function of deformation. Based on updated
position and velocity of potential contact points on thehoof K =  K x , K y  or H =  H x , H y  from themultibody kinematic analysis, relative normal andtangential velocity among the contact points are
estimated as

q 
q 
yields(9), u   , u =u =→ u ( t + ∆t ) ,



q
q
 
 
At the starting point of the numerical simulation, initialconfigurations of target mechanisms are given according
to the primary operation in forward dynamics analysis.
2.2. Formulation of the contact-impact process inplanar multibody system
When describing the contact impact event between thehoof and ground, we need to determine the state of thecontact and accurately acquire the relative deformationand speed at the instant of contact.

vn= δ=( rj − ri ) n,T(12)similarly,=vt( rj − ri ) t ,T(13)

where t is the unit vector in tangential direction obtainedby rotating unit normal vector n counter-clockwise by
90 degrees. When bodies come into contact, the normal
contact force can be estimated with contact force model
introduced by Lankarani & Nikravesh (1990) [20]. In thedamping factor model, the crucial parameter known as
coefficient of restitution was introduced which defines
the whether contact is fully elastic cr = 1 (or plastic)
cr = 0 , resulting impact force model can be written
Fig. 1. Representations of foot-groundkinematic model of hoof-ground contact.

interactions

by

As shown in Fig. 1, only non-contact scenarios is
considered in the kinematic model. When the leg contacts
with the ground, the deformation or penetration isestimated asδ yG − Pn ,=()

 3 1 − cr 2

(14)
δ 
=
.
FN K δ 1 +
(−) 


4
δ


where cr denotes the coefficient of restitution which isdefined as the ratio of relative approach velocity δ ( + )and relative departing velocity of δ ( − ) . The tangential
friction force in contact is calculated using modified
Coulomb friction law [21]. The dynamic friction forces
F f in the presence of sliding [22] can be written as
n 

(10)

where the Pn is the normal component of the position
vector or the potential contact point as K =  K x , K y 
(toe-first) or H =  H x , H y  for the case of heel firstcontact. The ground plane yG is consistently levelled

F f = − µk FN c f cd .

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

507

(15)

Dondogjamts Batbaatar, Hiroaki Wagatsuma

where µk the kinetic friction coefficient. In order to takethe friction force effect into account, the direction of thetangential velocity vector c f = vT with the target body
velocity vT , which is opposite to direction of motion. cd
is the dynamic correction coefficient given by [21]. If thenormal force FN is obtained from a contact with a forcedeformation (Eq. 14.) of contact force model, a logicalpoint-to-point spring-damper element [23] is only active
during the period of contact in normal and tangential
direction as shown in kinematic model in Fig. 1. Finally,
absolute values of the force vector in normal andtangential defines the resultant ground reaction forces
GRF=
FN + F f exerted when the hoof press against
the ground.

resultant motion of the stiffness-damping system under
the compression. Two trajectories were selected as
typical trajectories to reproduce walking and runninggaits according to the mechanics of the simple model,
which are generated by changing the control parameters
associated with driver constraint in the horse leg
mechanism.
As shown in Fig.2, the 1st trajectory considered to be awalking in term of duty factor and step length, therelation was evaluated in previous study. The 2nd

3. Results
According to the MBD formulation above, theconstitutive contact force model is calculated
numerically with kinematic and dynamic analyses, whichallow to visualize temporal evolutions of the locomotive
trajectory, resultant ground reaction force of the leg
mechanism.

Table 1. Parameters used in the numerical simulation.
Kinematic/Dynamic analysis
Gravitational accelerationg
9.81[m/s2]The velocity of the driving𝜔𝜔
2𝜋𝜋
crank [rad/s]
Total simulation time [s]t0≤t≤1Baumgarte parameter
α15
Baumgarte parameterβ2αTime step [s]dt
1.0×10-3
Contact force analysis
Stiffness [N/m]K
cr
Coefficient of restitutionµk
Coefficient of kinetic friction

2×104

Fig. 2. Gait trajectories generated from the horse leg motion.
Walking (a), Running (b).

0 ≤ cr ≤ 1
0.4

trajectory which has longer step length compared to the
1st trajectory considered to be running gait in which leg
orientation controlled by swinging motion with respect tothe body, which may potentially generate the propulsive
ground reaction force due the intrinsic high rate ofangular oscillation at the hoof.
In the temporal evaluation of the hoof angle as shown in
Fig. 3, walking and running gaits were generated inaccordance with the first and second trajectories in Fig.

Table 1 displays the parameters used in the numericalsimulation for contact force analyses of horse leg
mechanisms. Matlab-based numerical simulation was
performed with a combination of the Euler method withthe time step of 1× 10−3 s. For the contact analysis toprevent an unnatural rise of ground reaction force, thestiffness is no smaller than 2 × 104 N/m for the contacting
bodies is considered according to simulation in [24]. It’s
simply that the ground deformation is equivalent to the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

508

A ground reaction force

gradual change of the hoof angle (Fig.3 (b). top), whilein the running gait, the transition started from the toestrike that causes the roll-over sub-phase quickly andthen reached to the toe-off sub-phase with a large gap
from the ground rather than that in the walking gait. The
result was numerically examined as the first contact
occurs at instant time t1 = 0.267 s and α1 = −6.83 leaves
the ground at t3 = 0.569s and α1 = −20.9 for the 1st(walking) trajectory. In the case, the hoof rolling over the
ground and during the mid-stance multiple contact points
were examined at α1 = −0.23 degree in which hoof wasalmost parallel with the ground. When the hoof leaves theground, angular variation during the contact phase wasreached 26 degrees. For the 2nd (running) trajectory, thecontact phenomenon occurred at the instant time
t1 = 0.208s and α1 = 6.20 firstly, shifted to leavingfrom the ground at t3 = 0.478s and α1 = 71.08 . Finally,
the hoof angle was reached 64.8 degrees, which isapproximately 2.5 times larger than the result in thewalking gait. Interestingly, the hoof-ground interaction
was significantly influenced by the initial impact
phenomenon, which differentiate the grounding parteither heel or toe and it reflects to successive sub-phases
too. Even in the simple model, the differentiation was
clearly observed not only in the trajectory level as a
posture and kinematic analysis but also in the kinetic
level as the dynamics analysis, which is easily extended
to the energy analysis. All these sub-phases arecharacterized as representative sub-phase to reconstruct
the target gait and it can provide the parametric analysis
with stiffness and damping, which is associated with anactual parameter from soft tissues in the distal limb. It isbecause that the compliant contact force model wastheoretically described as shown in the method section,
which allow to change those parameters related toreaction forces easily.
In the Matlab based numerical simulation, groundreaction forces were evaluated by using Lankarani &Nikravesh [20] model by changing restitution
coefficients as cr = 1 for elastic and cr = 0 for inelastic
contact cases. According to the analysis, the maximum
contact force was obtained as 453.65N for the 1st(walking) trajectory in the inelastic contact condition and175.34N in the elastic case. Average normal force was
80.83N and 34.88N respectively. For the 2nd (running)
trajectory, the maximum GRF was 8.694kN for the

Fig. 3. Hoof angle variations with respect to the locomotive
trajectory. Angular rotation of hoof in one locomotive cycle ofwalking gait (a) and running gait (b).

2. In the waling gait, the transition of sub-phases
commonly occurred in the rolling motion of the triangle
structure in both cases. In the walking gait, those subphases from heel-strike, flat and toe-off were gradually
generated (Fig. 3(a). top), which is consistent with the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

509

Dondogjamts Batbaatar, Hiroaki Wagatsuma

For the sake of the establishment of the generalized
analysis of the viscoelastic effect with respect to various
gait patterns, we introduced a novel theoretical method
as the integrative framework of the multibody dynamics
and contact force model and then the computational
analysis clearly demonstrated results in four different
conditions from the combination of walking/running andelastic/inelastic contact. In the simplification of the hoofmechanism by using a triangle structure to focus on threenodes as heel, ankle and toe, the result revealed not only
the two peak generation in the temporal evolution of theground reaction force during the stance phase of thewalking gait, but also a single peak observation in thecase of the running gait. The phenomenon was generated
from that wheel-like rolling motion of the hoof supportand stabilizes the body in mid-stance, and partiallyconsistent with the observation in the human foot-groundinteraction [1,10]. In spite of the existence of thecomplexity in the horse leg structure, our computational
result suggested a possible way to analyze the springdamper effect embedded in principle to the biologicalmechanism for absorbing the ground reaction force
flexibly as a common tendency of the walking andrunning gaits. The analysis with damping factor modelby Lankarani & Nikravesh (1990) realized the fact that asoft grounding effectively reduces the ground reaction
force. In the actual phenomenon due to the interaction in
the musculoskeletal systems such as a vibration from thesensory feedback loop [27,28], the further detailmodeling is crucial. Phenomenologically in the level ofthe force generation, we successfully observed a timedelay in a peak in the force time profile specifically inrunning gaits, which indicates an intrinsic behavior due
to the damping factor is highly important in the kicking
motion of the hoof. The phenomenon is also need to beverified with more detail leg models and evidences fromthe biological system [29,30]. Further considerations are
possible to compare in the energy consumption due to theelasticity [18] and robotic application [31]. Geometriccharacteristics of the hoof is of interest to researchers
studying on the biological nature mechanical in the hoofshape and its influence in joint angles and contact timing[32,33].

Fig. 4. Resultant values of ground reaction forces under thedifferent contact condition. Force pattern of walking gait (a)and running gait (b).

inelastic contact condition and 5.719kN in the elastic
case. Average normal force was 978N and 800N
respectively. Result showed that the significant reductionof ground reaction force with respect to the viscoelastic
contact was estimated as 61.35% in 1st trajectory and34.22% in 2nd trajectory. In the walking trajectory,
multiple point support occurring in which hoof loaded the
body weight at mid-stance as shown in Fig. 4(a), themaximum pressure or force peak indicated in only when
the leg first impact and leaves the ground which wasconsistent with the force pattern observed in the humanwalking gait [25,26].

Acknowledgements
This work was supported in part by Mongolia-Japanhigher Engineering Education Development (MJEED-

4. Conclusion and Discussion© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

510

A ground reaction force

13. D. Batbaatar and H. Wagatsuma. A proposal of thekinematic model of the horse leg musculoskeletal system
by using closed linkages. In IEEE International
Conference on Robotics and Biomimetics, pp. 869–874,
2019.
14. T. Jansen. The Great Pretender. Netherlands:010, 2007.
15. K. Komoda and H. Wagatsuma. Energy-efficacy
comparisons and multibody dynamics analyses of legged
robots with different closed-loop mechanisms. Multibody
System Dynamics, 40:123–153, 2017.
16. Sandra D. Starke, Justine J. Robilliard, Renate Weller,
Alan M. Wilson, and Thilo Pfau. Walk-run classificationof symmetrical gaits in the horse: A multidimensional
approach. Journal of the Royal Society Interface, 6:335–
342, 2009.
17. Jeffrey J. Thomason and Michael L Peterson.
Biomechanical and mechanical investigations of the hooftrack interface in racing horses. Veterinary Clinics: Equine
Practice, 24(12):53–77, 2008.
18. R. Mcneill Alexander. Elastic energy stores in running
vertebrates. Integrative and Comparative Biology,
24(1):85–94, 1984.
19. J. Baumgarte. Stabilization of constraints and integrals of
motion in dynamical systems. Computer Methods in
Applied Mechanics and Engineering, 1:1–16, 1972.
20. H. M. Lankarani and P. E. Nikravesh. A contact force
model with hysteresis damping for impact analysis ofmultibody systems. Journal of Mechanical Design,
Transactions of the ASME, 112:369–376, 1990.
21. J A C Ambrósio. Impact of Rigid and Flexible Multibody
Systems: Deformation Description and Contact Models,
pp. 57–81. Springer Netherlands, Dordrecht, 2003.
22. V. P. Chervinskii. Dissipative function of coulomb friction
on elliptical contact area. Journal of Friction and Wear,
32:386–393, 2011.
23. Parviz E Nikravesh. Planar multibody dynamics:
formulation, programming with Matlab, and applications.
CRC Press, second edition, 2019.
24. Gerald Ristow. Simulating granular ow with molecular
dynamics. Journal de Physique I, EDP Sciences, 2:649–
662, 1992.
25. R. McNeill Alexander. Energetics and optimization ofhuman walking and running: The 2000 Raymond Pearl
Memorial Lecture. American Journal of Human Biology,
14(5):641–648, 2002.
26. R. McNeill Alexander and A. S. Jayes. Fourier analysis offorces exerted in walking and running. Journal of
Biomechanics, 13(4):383–390, 1980.
27. Daniel F. B. Haeufle, Katrin Stollenmaier, Isabelle
Heinrich, Syn Schmitt, and Keyan Ghazi-Zahedi.
Morphological Computation Increases From Lower- toHigher-Level of Biological Motor Control Hierarchy.
Frontiers in Robotics and AI, 7:1–13, 2020.

JICA) through the joint research project “Developmentof industrial, service and intelligent systems based on
advanced technologies” (J23A16), JSPS KAKENHI(16H01616, 17H06383) and the New Energy andIndustrial Technology Development Organization(NEDO), and Project on Regional Revitalization
Through Advanced Robotics (Kyushu Institute ofTechnology/Kitakyushu city, Japan).
References
1. D. Joshua Mayich, Alison Novak, Daniel Vena, Timothy
R. Daniels, and James W. Brodsky. Gait analysis inorthopedic foot and ankle surgery - Topical review, part 1:
Principles and uses of gait analysis. Foot and Ankle
International, 35:80–90, 2014.
2. Logan J. Dockery. Analysis of Gait Propulsion During theAnkle and Forefoot Rocker Sub-phases. PhD thesis, 2019.
3. T. Mcgeer. Passive Dynamic Walking. The International
Journal of Robotics Research, 9(2):62–82, 1990.
4. P. Mahmoodi, R.S. Ransing, and M.I. Friswell. Modellingthe effect of ‘heel to toe’ rollover contact on the walking
dynamics of passive biped robots. Applied Mathematical
Modelling, 37(12):7352–7373, 2013.
5. D. Torricelli, J. Gonzalez, M. Weckx, R. Jiménez-Fabián,
B. Vanderborght, M. Sartori, S. Dosen, D. Farina, D.
Lefeber, and JL. Pons. Human-like compliant locomotion:state of the art of robotic implementations. Bioinspiration
and Biomimetics, 11(5), 2016.
6. Eduardo Corral, M. J. Gómez García, Cristina Castejon,
Jesús Meneses, and Raúl Gismeros. Dynamic modeling of
the dissipative contact and friction forces of a passive
biped-walking robot. Applied Sciences (Switzerland), 10,2020.
7. A. H. Hansen, D. S. Childress, and E. H. Knox. Roll-overshapes of human locomotor systems: effects of walking
speed. Clinical biomechanics (Bristol, Avon), 19(4):407–
414, 2004.
8. A. H. Hansen and D. S. Childress. Investigations of rollover shape: implications for design, alignment, andevaluation of ankle-foot prostheses and orthoses.
Disability and Rehabilitation, 32(26):2201–2209, 2010.
9. Bellusci G. Schepers H. M. de Zee M. Andersen M. S.
Karatsidis, A. and P. H. Veltink. Estimation of Ground
Reaction Forces and Moments During Gait Using Only
Inertial Motion Capture. Sensors, 17(1) (75), 2016.
10. Bonnefoy-Mazure A. and Armand S. Normal gait, pages
200–211. Amsterdam: Elsevier, 2015.
11. Alberto Leardini, John J. O'Connor, and Sandro Giannini.
Biomechanics of the natural, arthritic, and replaced humanankle joint. Journal of Foot and Ankle Research, 7, 2014.
12. W. Lee and M. Raibert. Control of hoof rolling in anarticulated leg. In Proceedings. 1991 IEEE International
Conference on Robotics and Automation, pp. 1386–1391.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

511

Dondogjamts Batbaatar, Hiroaki Wagatsuma

28. Z. H. Shen and J. E. Seipel. A fundamental mechanism oflegged locomotion with hip torque and leg damping.
Bioinspiration and Biomimetics, 7, 2012.
29. Ian Abraham, Zhuohua Shen, and Justin Seipel. A
Nonlinear Leg Damping Model for the Prediction ofRunning Forces and Stability. Journal of Computationaland Nonlinear Dynamics, 10:1–8, 2015.
30. Alan M. Wilson, M. Polly McGuigan, Anne Su, and Anton
J. Van den Bogert. Horses damp the spring in their step.
Nature, 414(6866):895–899, 2001.
31. E. Garcia, J. C. Arevalo, G. Múnoz, and P. Gonzalez-deSantos. On the biomimetic design of agile-robot legs.
Sensors, 11:11305–11334, 2011.
32. C. Johnston and W. Back. Hoof ground interaction: Whenbiomechanical stimuli challenge the tissues of the distal
limb. Equine Veterinary Journal, 38:634–641, 2006.
33. Angelos Karatsidis, Giovanni Bellusci, H. Martin
Schepers, Mark de Zee, Michael S. Andersen, and Peter H.
Veltink. Estimation of ground reaction forces andmoments during gait using only inertial motion capture.
Sensors, 17:1–22, 2017.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

512

Satellite Image-based UAV Localization using Siamese Neural Network
Seong-Ha Ahn

Department of Electronic Engineering, Pusan National University,
2, Busandaehak-ro 63beon-gil Geumjeong-gu
Busan, 46241, Republic of Korea
Ho-Sun Kang

Department of Electronic Engineering, Pusan National University,
2, Busandaehak-ro 63beon-gil Geumjeong-gu
Busan, 46241, Republic of Korea
Jang-Myung Lee

Department of Electronic Engineering, Pusan National University,
2, Busandaehak-ro 63beon-gil Geumjeong-gu
Busan, 46241, Republic of Korea
E-mail: seongha7379@pusan.ac.kr, hosun7379@pusan.ac.kr, jmlee@pusan.ac.kr
www.pusan.ac.kr

Abstract
We present a method for UAV localization using pre-existing satellite images. The use of Unmanned Aerial Vehicles(UAVs) has rapidly increased in several applications such as surveillance, search, and defense. When in GPS-denied
situations, however, the onboard GPS signal may be noisy or inaccurate. The proposed method is based on a Siamese
Neural Network that contains two instances of the same neural architecture and weights. Siamese Neural Network
learns the similarity metric so that can recognize the same place from two raw images. Convolutional Neural Network
is used as a backbone in Siamese Neural Network to overcome variation due to differences such as perspective,
shadow angle, and presence of vehicles. We describe UAV localization pipeline and a dataset for training and testing
our networks. Finally, the performance of the proposed method was shown in accuracy.
Keywords: visual localization, uav, satellite image, siamese neural network, image retrieval

Positioning System (GPS). When in GPS-denied
situations, however, the onboard GPS signal may benoisy or inaccurate. Visual localization, which is the
problem of estimating a camera pose, can be a usefulalternative in GPS-denied environments. Image retrievalbased approach is one of the common visual localization
methods [2]. In image retrieval tasks for UAVs visual
localization, satellite imagery can be utilized as ageodatabase. In recent year, deep neural networks have

1. Introduction
Unmanned Aerial Vehicles (UAVs) are aircraft that
remotely controlled or fly autonomously with an onboard
manipulation system. Nowadays, the use of UAVs has
rapidly increased in several applications such as
surveillance, search, and defense [1]. Localization is an
essential task to utilize UAVs in such tasks. Currently,
the onboard navigation system is mostly relied on Global© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

513

Seong-Ha Ahn, Ho-Sun Kang, Jang-Myung Lee

Fig. 1. Image retrieval pipeline of visual localization for UAVs. georeferenced satellite imagery is divided into patches for searchand UAVs image is used as query image. The closest patch image will be retrieved by comparing similarity of every patch and query
image pairs.

UAVs image. Input images are projected onto the low
dimensional space which is called embedding space. By
training siamese neural networks with UAVs and satellite
patch imagery, the networks are trained to extract
discriminative features to measure similarity. It helps the
networks to recognize if two input images are taken fromthe same place or not. Every patch images and UAVs
image are compared by measuring the distance in theembedding space.

been successfully applied in various image processing
and analysis tasks. We apply deep neural architecture in
the image retrieval system to analyze images efficiently.
In this paper, we propose a satellite image-based UAVs
visual localization method with image retrieval approach.
In section 2, we describe the deep neural architecture,
called Siamese Neural Networks that learns metric forsimilarity. Dataset preparation for training neuralarchitecture will be covered in section 3. We train andevaluate the proposed method in section 4. Finally, wesummarize our study in section 5.
2. Image Retrieval
Networks

using

Siamese2.2. Triplet loss
One of the options for training a siamese neural networks
to learn similarity metric is to use triplet loss. The input
data are projected into the embedding space via the
networks. When Anchor, Positive and Negative images
are input, triplet loss is calculated as the euclidean
distance between three embedded points. (see Figure 2)
The loss function can be described as follows:

Neural2.1. Siamese neural networks
Siamese neural networks (SNN) is a neural architecturethat has two networks which share its weights. Siameseneural networks employ a unique structure to learn asimilarity metric between two input [3]. To extract
features effectively from input images, Convolutional
Neural Networks (CNN) is used in feature extraction
layers of siamese neural networks. Figure 1 shows theimage retrieval pipeline using CNN-based siamese neuralnetworks. The proposed method uses semi-supervision of
GPS system. The georeferenced satellite image isselected according to the noisy GPS signal from database.
Satellite image is divided into few numbers of patches.
The network takes two input: patch of satellite image and2

𝐿𝐿(𝐴𝐴, 𝑃𝑃, 𝑁𝑁) = max ��|𝑓𝑓(𝐴𝐴) − 𝑓𝑓(𝑃𝑃)|� −2
�|𝑓𝑓(𝐴𝐴) − 𝑓𝑓(𝑁𝑁)|� + 𝛼𝛼, 0�(1)Where 𝑎𝑎 is a margin between positive and negative
pairs, and 𝑓𝑓 is an embedding. In our implementation, we
set 𝑎𝑎 as 1.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

514

Satellite Image-based UAV Localization

Fig. 2. The distance from the Anchor to Positive is minimized and the distance from the Anchor to Negative is maximized.

3. Dataset
To simulate UAVs and satellite imagery, we used two
satellite imagery sources. As shown in Figure 3, thedifference in characteristics of satellite imagery sources
causes various visual representations.
3.1. Database properties
We build two databases: UAVs image database andsatellite imagery database. These databases are consist of
total 88 correspondence images. Database image pairs
from 3 different cities in New York and Florida. Each
image covers a 0.16km2 area of land. UAVs database
image size with 0.125m resolution is 3200*3200 pixels,
satellite image size with 0.5m resolution is 800*800
pixels. In our experiments, we divide database images
into 16 patches per each. For training and validating, we
generate approximately 30k and 0.5k, 0.5k image pairs
respectively with the augmented images generated byrotating and adding a random value to channels.
3.2. Anchor, Positive and Negative image
As described section 2, triplet loss is calculated bymeasuring euclidian distance between Anchor, Positive
and Negative images embedded in embedding space. For
positive images, we create 4 shifted images from theoriginal patch. The pixel range for the shift is limited to±25 percent of total pixels in the image. The Anchor andPositive image pairs are created by combining 8 images.
The Negative images are sampled in the same database
image except the same patch and the other images.

Fig 3. Various visual representation of the same area causedby different characteristic of satellite imagery sources.

4.

Implementation

We implement the proposed system in Python and
PyTorch. Experiments have been done using 2
RTX2080ti. Stochastic Gradient Descent (SGD)
optimizer is employed to train the networks. Since the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

515

Seong-Ha Ahn, Ho-Sun Kang, Jang-Myung Lee

only image with the shortest Euclidean distance isassumed to be a Positive, we evaluate our methods by
calculating top- 𝑘𝑘 precision score. The proposed method
achieves as Table 1. It retrieves correct images for mostqueries, however, it performed poorly in specific cases,
such as when the area has too many structures. (seeFigure 4) It shows that the dataset requires more images
that contain various visual features for generalization.

Table 1. The experimental results on our test set.
Precision score
Top-𝑘𝑘 precision

score

𝑘𝑘=3

0.74

𝑘𝑘=1

5. Conclusion

0.65

Fig 4. Example image pairs with good retrieval results (leftcolumn) and bad retrieval results(right column).

We proposed and implemented an image retrieval
system that queries UAV images and retrieves the mostsimilar image from pre-existing satellite images using
CNN-based Siamese neural networks. To simulate
UAVs and satellite imagery, we created a dataset using
two different satellite imagery sources. The proposedsystem performed decently on our test set, however, theperformance should be improved in some specific cases.
In order to deploy the system for practical usage, themore diverse dataset is necessary for generalization.
Moreover, further research of the additional post-process
is required to acquire global camera pose with highaccuracy.

3. F. Schroff et al., "FaceNet: A unified embedding for facerecognition and clustering," 2015 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR),
Boston,
MA,
2015,
pp.
815-823,doi:
10.1109/CVPR.2015.7298682.

Acknowledgements
This research was supported by the National Research
Foundation of Korea(NRF) grant funded by the Korea
government(MSIT) (No.2019R1A2C2088859)
References
1. B. H. Y. Alsalam et al., "Autonomous UAV with vision
based on-board decision making for remote sensing andprecision agriculture," 2017 IEEE Aerospace Conference,
Big
Sky,
MT,
2017,
pp.
1-12,
doi:
10.1109/AERO.2017.7943593.
2. Pion, Noé et al., (2020). Benchmarking Image Retrieval
for Visual Localization.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

516

Analysis based on CNN for Automated Vehicle Parking Occupancy
Muksimova Shakhnoza

Department of Computer Engineering, Gachon University, Gyeonggi-do 461-701
Seongnam, Korea
Young Im Cho

Department of Computer Engineering, Gachon University, Gyeonggi-do 461-701
Seongnam, Korea
E-mail: muksimova1986@gmail.com, yicho@gachon.ac.kr
www.gachon.ac.kr
Abstract
The remarkable growth of the automobile industry, along with a lack of urban planning, exacerbated problems
such as traffic congestion, air pollution and driving difficulties. The majority of car manufacturers have a preinstalled or aftermarket navigation device in vehicles to make it easier for customers. This helps drivers to travel totheir destination with ease. However, a rough estimate of the remaining distance to the destination is given bynavigation systems. With the latest advancement in technology, companies are interested to find out whether theseadvancements can assist in reducing the time spent to search for empty parking space. Locating a vacant parking
spot is a big predicament that drivers face because looking for it is a tiring operation. Our investigations haveshown that, in contrast with previous methods, for the purpose of classifying given parking spaces as empty or
occupied, the current solution is more robust, stable and well-generalized for unseen images taken from entirely
separate camera points of view, which has clear signs that it can easily generalize to other parking lots.
Keywords: Convolutional neural network, parking, sensors, lightweight network

1. Introduction

in a parking lot, provided only the picture of a single
parking spot as data.
In addition, the computer vision for traffic
applications is an evolving field of study, with major
developments over the last few years. For example,
intelligent transport management systems, such as thoseexamined in [1], aim to accomplish tasks such as
monitoring, detecting and identifying vehicles, as wellas higher-level tasks such as recognizing vehicle actions
and revealing irregularities. In a vision-based device,
any visual node that contains a camera and a transmitter
will simultaneously observe several cars, minimizing
the cost per parking stand. As it can be used for otheractivities, such as security, the device base is normallyalready configured and can be handled by otherapplications after implementation. However, theliterature review [3] has demonstrated that image-based
parking reservation tracking policies can be enforced
using existing security cameras that are alreadyconnected to the main monitoring system. On the
opposite, those who support sensor-based approaches,

The value of unoccupied parking space detection
systems is growing dramatically, as preventing traffic
congestion and the time-consuming task of locating
empty parking space is a big challenge for urban center
drivers. However, current parking space occupancy
monitoring technologies are either costly hardware or
not well-generalized for differing images taken frommultiple camera views. As a solution, we takeadvantage of an effective visual identification approachthat is made possible by the fact that camera tracking isalready present in most parking spaces. The current
challenge, on the other hand, is a difficult vision job due
to outdoor conditions such as lighting variance,
occlusions, distortion of focus, multiple camera views,
and adjustments due to the different seasons of the year.
In order to address these limitations, we suggest an
approach based on the Convolutional Neural Network
specially constructed to detect parking space occupancy© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

517

Muksimova Shakhnoza, Young Im Chosuch as the authors of [2], contend that video cameras
are significantly more costly than sensors and generate asignificant quantity of data that cannot be readily
disseminated over a wireless network.

Very strong results were obtained, but only for the same
subset of images.

In this paper, to resolve the above problems, we are presenting
a CNN framework that uses dilated convolution neuralnetworks to show the status of parking space occupancy. The
PKLot dataset [4], which is freely available, was used to test
the efficiency of our model. Using pictures of various
situations and parking lots, we were able to test thegeneralizability of our strategy. We learned our model on asubset of images and checked it on a totally different one.
Several methods already exist, such as the one explored in theanalysis mentioned in [5], AlexNet, where this sort ofgeneralization property has been experimentally tested;
however, we have shown that our method outperforms them.
Most of the related studies contrasted their methodology toAlex Net [6], which showed a high degree of success in thismission. Thus, in conjunction with Alex Net, as well as withother well-known deep learning structures, we tested our
method

2. Related Work

Figure 1. Example for a car parking image from PKLot
dataset [4].

To address the problem mentioned above, there are currently
different schemes, such as those suggested in [2, 7]. In several
studies, the issue with the parking management scheme wasresolved by the use of smart devices. Sensor-based solutions
include sensors for identification, such as ultrasonic sensors,which are mounted in each parking room. Huang and Wang
[9] held that image-based approaches can be divided into twoclasses: space-driven and car-driven. Space-driven approaches
are targeted at finding empty parking spaces rather thanvehicles. In the case of images taken from static (such as
surveillance) cameras, the technique most commonly used isbackground subtraction, which implies that the backgrounddeviation is statistically immobile over a small time period.
Considering that this presumption does not hold outdoor oroutdoor scenes, the limitations of this method are instantly
apparent. The authors used Gabor filters to train a classifier
with images of unoccupied parking spaces under different
lighting conditions. In the category of car-driven process,algorithms are learned to identify vehicles that are objects ofinterest. Due to the distortion of perspective found in most of
the parking lot images, as seen in Figure 1, cars positioned at a
substantial distance from the camera occupy a small area ofthe scene, and thus less information can be observed, which
greatly decreases the efficiency of the object detection
algorithms. The technique presented in [5] is based on a deep
Convolutional Neural Network specially developed for smart
cameras. They have obtained reasonably high results, and wewill equate our results with that. The method, named
mAlexNet, is a variant of AlexNet and provides adecentralised approach for visual space occupancy detection.

(a)

(b)
Figure 2. Segmented images of dataset for occupied andvacant space.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

518

Analysis based on CNN

3. Proposed Approach

Figure 3. Comparison of our model with respect to othermodelsThere are three subsets of this PKLot dataset [4] andduring this experiment, we only utilized two of them in

4. CONCLUSION
An efficient solution-Our model, for visual
detection of a parking status, was presented which uses
Convolutional Neural Networks. Our novel model isprovided as a robust design for employees to classify
images of parking spaces taken from a camera as
occupied or vacant. In this proposed approach, we used
three contributions: a little number of layers, small
window sizes. We showed by experiments that thecurrent task can be tackled effective by using thesecontributions. To assess the performance of the modeland to be able to compare it with respect to otherarchitectures, we made our experiments on PKLot
dataset since prior attends were evaluated on this dataset
as well.

our work. As we mentioned above on this task, AlexNet[6] model was achieved highest performance and so far
a number of models were compared to this model in thisreason we also compared our work to AlexNet [6]model. Our model is five layers deep and output layer.
However, AlexNet [6] model is eight layers deep thatfive convolutional and three fully connected layers andlearning of. parameters are also fairly many than our
network. While increasing the number of learnable
parameters makes the model bigger and increases
training time. Below, in Table 1 and Figure 3 is shown
experimental results of the models.

Acknowledgements
This research was supported the National Research
Foundation (NRF)—2018R1D1A1A09084151.
References
1. Tian, B.; Morris, B.T.; Tang, M.; Liu, Y.; Yao, Y.; Gou,
C.; Shen, D.; Tang, S. Hierarchical and Networked
Vehicle Surveillance in Its: A Survey. IEEE Trans. Intell.
Transp. Syst. 2015, 16, 557–580.
2. Tang, V.W.; Zheng, Y.; Cao, J. An Intelligent Car Park
Management System Based on Wireless Sensor Networks.
In Proceedings of the 1st International Symposium on
Pervasive Computing and Applications, Urumchi,
Xinjiang, China, 3–5 August 2006; pp. 65–70.
3. Ichihashi, H.; Notsu, A.; Honda, K.; Katada, T.;
Fujiyoshi, M. Vacant Parking Space Detector for
Outdoor Parking Lot by Using Surveillance Camera andFCM classifier. In Proceedings of the IEEE International

Table 1. Comparison of training and validation scores for
AlexNet, ResNet and Our method on UFPR04.

Name of
MethodOur Method
Alex Net[6]Res Net[5]Training
Accuracy
97.92%
96.99%
96.51%

Validation
Accuracy
97.87%
97.91%
97.80%© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

519

Muksimova Shakhnoza, Young Im Cho

4.

5.

6.

7.

8.

9.

Conference on Fuzzy Systems, Jeju Island, Korea, 20–24
August 2009; pp. 127–134.
Amato, G.; Carrara, F.; Falchi, F.; Gennaro, C.; Meghini,C.; Vairo, C. Deep Learning for Decentralized Parking
Lot Occupancy Detection. Expert Syst. Appl. 2017, 72,
327–334. [CrossRef
De Almeida, P.R.; Oliveira, L.S.; Britto, A.S., Jr.; Silva,
E.J., Jr.; Koerich, A.L. PKLot-A Robust Dataset for
Parking Lot Classification. Expert Syst. Appl. 2015, 42,4937–4949. [CrossRef].
Amato, G.; Carrara, F.; Falchi, F.; Gennaro, C.; Meghini,C.; Vairo, C. Deep Learning for Decentralized Parking
Lot Occupancy Detection. Expert Syst. Appl. 2017, 72,
327–334. [CrossRef].
Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet
Classification with Deep Convolutional Neural Networks.
In Advances in Neural Information Processing Systems
25: 26th Annual Conference on Neural Information
Processing Systems 2012; Curran Associates, Inc.: Red
Hook, NY, USA, 2012.
Ashok, V.G.; Gupta, A.; Shiva, S.; Iyer, H.; Gowda, D.;
Srinivas, A. A Novel Parking Solution for Metropolitan
Parking Garages. In Proceedings of the 3rd WSEAS
International Conference on Urban Planning andTransportation, Sozopol, Bulgaria, 22–24 July 2010; pp.
153–159.
Huang, C.C.; Wang, S.J. A Hierarchical Bayesian
Generation Framework for Vacant Parking Space
Detection. IEEE Trans. Circuits Syst. Video Technol.
2010, 20, 1770–1785. [CrossRef]© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

520

On the Boundary Layer in Singularly Perturbed Problem with An Initial Jump
Assiya Zhumanazarova

Department of Computer Engineering, Gachon University, Gyeonggi-do 461-701
Seongnam, Korea
Young Im Cho

Department of Computer Engineering, Gachon University, Gyeonggi-do 461-701
Seongnam, Korea
E-mail: assiya_mukhamaddin@mail.ru, yicho@gachon.ac.kr
www.gachon.ac.kr

Abstract

In this study, we consider the foundations of the boundary layer theory for studying the boundary layerpart of the asymptotics of a singularly perturbed integro-differential boundary value problem with an
initial jump. A part of the boundary layer is defined as an expansion in powers of a small parameter, andthe coefficients of this expansion are solutions to differential initial value problems. The purpose of thisstudy is to describe in detail the properties of the boundary layer part of the asymptotics based on theprinciples of boundary layer theory.
Keywords: boundary layer, singular perturbation, initial jump, exponential estimates.

near the wall, where friction forces must be considered,and the region outside the boundary layer, where thefriction forces can be ignored due to their smallness.1 The
approximate character of the considered theory is that it
is based on the assumption of a small thickness of theboundary layer.
The boundary layer theory can be considered from aphysical and mathematical point of view. The theory ofdifferential equations containing small parameters at thehighest derivatives is important in mathematicalmodeling of physical processes. In the theory of
singularly perturbed boundary value problems, aboundary layer phenomenon is observed when thesolution to the perturbed problem differs significantly
from the solution of a degenerate problem for arbitrarily
small parameters. The behavior of the solution of asingularly perturbed system in the boundary layer is
described by exponentially decaying functions.2 The
system describing a viscous fluid is an example of asingularly perturbed system. In hydrodynamics, the

1. Introduction
The boundary layer theory studied by Prandtl has
been applied to characterize the problems of external
flow around a body at high flow degree of velocity or low
viscosity. Various fundamental problems of mechanics
and physics can be formulated using a significantly
simplified model using the approximate theory of theboundary layer. The results of analytical solutions tothese problems are provided with a high degree of
accuracy. The practical significance of the study ofboundary layers lies in the fact that they take place inmany important processes, where the effect of fluid
viscosity is significant in the neighborhood of theboundary surface. One of the specific directions of theboundary layer theory is the friction layer inaerodynamics, where it is characterized by a strong
gradient of flow velocity. To the theoretical study of theflow of low-viscosity fluids, the entire field can bedivided into two parts: the region of a thin boundary layer© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

521

Assiya Zhumanazarova, Young Im Cho

equation of an ideal fluid even at low viscosity is notsuitable for describing a process near the boundary,called the boundary layer.2 In perturbation theory themethod of matched asymptotic expansions wasdeveloped to find an approximate solution by
constructing different asymptotic solutions inside andoutside the boundary layer of rapid changes and then,matching them together to obtain a unique approximate
solution.3
In present section, we considered the basic conceptsof the boundary layer theory to find association between
their practical and theoretical importance. In thefollowing section, we consider a singularly perturbed
problem with an initial jump where the foundations of thetheory of boundary layers are applied.4
The remainder of this paper is organized as follows.
Section 2 is devoted to describe the main points based on
the results of previous studies on singularly perturbed
boundary value problem with an initial jump and to theformation of boundary layer part of asymptotics of thesolution. Finally, Section 3 provides some concluding
remarks.

Here, ym ( t ) and ρ m (τ ) denote the regualar andboundary layer terms, respectively, and rn ( t , ε ) denotes
the remainder term of the asymptotics.
Consider the differential initial value problems
defined for finding the boundary layer termsρ m (τ ) + A ( 0 ) ρm (τ ) + B ( 0 ) ρ m (τ ) =ψ m (τ ) , m ≥ 0,

ψ 0 (τ ) ≡ 0,

ρ0 ( 0 ) = −∆ 0 , ρ 0 ( 0 ) = 0, ρ0 ( 0 ) = B ( 0 ) ∆ 0 ,ρ1 ( 0 ) = −∆1 , ρ1 ( 0 ) = β − y0′ ( 0 ) ,∞=ρm ( 0 ) A ( 0 ) ym′ −1 ( 0 ) + B ( 0 ) ∆ m − ∫ψ m ( p )dp, m ≥ 2.
0

By direct calculations, we obtain the solution for the
0-th order approximation in the formB (0)∆

110e µ (0)τ −e µ (0)τ  +−ρ0 (τ ) =µ2 (0) − µ1 (0)  µ1 (0)µ2 (0)

1

2( B (0) − µ (0) µ (0)) ∆ .
+
2

(4)

0

µ1 (0) µ2 (0)It follows immediately from the solution that

Consider a singularly perturbed integro-differential
boundary value problem with an initial jump.5,6 In thesestudies, singularly perturbed and unperturbed problems,
as well as the convergence of solutions using asymptotic
methods are investigated. Since the presence of an
integral term in the original equation significantly affects
the solution, the phenomenon of an initial jump wasobserved in the problem at the left point of the consideredsegment. To form a uniform asymptotic expansion
considering the asymptotic properties of solutions, themethod of boundary functions was used. Thus, analgorithm was formulated for constructing asymptotic
expansion of solution, which was characterized by a
rapid change within the boundary layer. The asymptotics
was sought as the sum of two parts - regular and boundary.
A uniform asymptotic expansion of the solution is
obtained by solving a sequence of simpler problems forfinding the composite parts of the asymptotics. That is, a
uniform asymptotic expansion has the form6
n+20ρ m ( 0 ) = −∆ m , ρ m ( 0 ) = − ym′ −1 ( 0 ) ,2. The Boundary Layer Part of Asymptotics

n

(3)

∞

ρ1 ( 0 ) =− A ( 0 ) ( β − y0′ ( 0 ) ) + B ( 0 ) ∆1 − ∫ψ 1 ( p )dp,1y ( t , ε ) = ∑ ε m ym ( t ) + ∑ ε m ρ m (τ ) + rn ( t , ε ) .

(2)

( j)

δτ
ρ 0 (τ ) ≤ Μe−=
, j 0, 2, τ ≥ 0.

(5)Here, Μ , δ are positive constants.
To find the solutions to problems (2), (3), m ≥ 1 that is,for the non-homogeneous differential equations (2),m ≥ 1 we have used the Lagrange's method of variation of
constants. After finding the solutions ρ m (τ ) , m ≥ 1 andtheir derivatives of first and second order, we haveobtained their exponential estimates. Using the methodof mathematical induction, that is, supposing that theestimates are valid up to the m-1-th order approximation,
we prove them for the m-th order approximation.
The boundary layer phenomenon can occur whensolving a boundary value problem for an integrodifferential equation with a small parameter. The
degenerate equation obtained by equating a small
parameter has an initial jump in the integral term, as wellas in the boundary conditions, modification is required,in consequence of which a boundary layer occurs.

(1)

m 0=
m 0
=

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

522

On the Boundary Layer

3. Conclusion
In this study, we considered the main concepts of theboundary layer theory for studying a singularly perturbed
integro-differential boundary value problem with an
initial jump. To describe their properties, the boundary
layer part of the asymptotic expansion of the solution wasconsidered. The study revealed that the terms of theboundary layer part of the asymptotics are characterized
by exponentially decreasing functions. In conclusion,having studied the foundations of the theory of theboundary layer, it was possible to characterize theproperties of the boundary layer part of the asymptotic
expansion of the solution. Further work is required to
expand this study, including fully proving boundary layerterm estimates and providing detailed solutions to nonhomogeneous differential equations.
Acknowledgements
This research was supported the National Research
Foundation (NRF)—2018R1D1A1A09084151.
References
1. H. Schlichting, Boundary-Layer Theory, Nauka, Moscow,1974.
2. A.B. Vasilyeva and V.F. Butuzov, Asymptotic Expansionof Solutions of Singularly Perturbed Equations, Nauka,
Moscow, 1973.
3. E.J. Hinch, Perturbation Methods, Cambridge University
Press, Cambridge, 1991.
4. K.A. Kasymov, Singularly Perturbed Boundary Value
Problems with Initial Jumps, Sanat, Almaty, 1997.
5. A. Zhumanazarova and Y.I. Cho, Asymptotic convergenceof the solution of a singularly perturbed integrodifferential boundary value problem, J. Mathematics,
8(213), 2020, pp. 1–17.
6. A. Zhumanazarova and Y.I. Cho, Uniform Approximation
to the Solution of a Singularly Perturbed Boundary Value
Problem with an Initial Jump, J. Mathematics, 8(2216),
2020, pp. 1–14.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

523

A Design and Implementation Intelligent flowerpot
Songyun Shi , Yizhun Peng*,

College of Electronic Information and Automation,
Tianjin University of Science and Technology, Tianjin, 300222, China;
E-mail: * pengyizhun@tust.edu.cn
www.tust.edu.cn

Abstract
The original intention of the device is that people often neglect the care of potted plants at home, resulting in plants
unable to grow and survive unattended. This intelligent flowerpot is a device that can make potted plants survive
and grow better without supervision. Smart home products based on Internet of things technology. The data of
temperature sensor, humidity sensor, soil humidity sensor, harmful gas sensor, photosensitive sensor and othersensors are collected by STM32 single chip microcomputer, and the data are cooperated with intelligent tracking
system and automatic irrigation system. To achieve the purpose of potted cultivation, beautify and improve theliving environment. In view of the disadvantages of traditional family life of artificial cultivation and potted plants,
the maintenance of scientific intelligence is realized, and the intelligent flowerpot system is designed.
Keywords: Internet of things technology, singlechip, Smart home, ecology, WIFI, flowerpot.

1. Introduction

things technology, multiple intelligent flowerpots areconnected to form a family ecosystem6.

With the accelerating pace of people's life, people usuallyignore the care of potted plants, and potted plants can notplay their due role. Gradually, people have lost the habit
and interest of planting potted plants at home1. There aremany advantages of planting potted plants at home, suchas raising flowers, which can convey feelings andcultivate sentiment; green leaf potted plants can purify
the air and make people happy. So potted plants havebeen loved by many people2. However, due to the fast
pace of people's life and lack of time management, taking
good care of potted plants has gradually become a luxury.
Many people want to take some time to take good care of
their flowers and plants, but they can't find the right time,so they stay at the stage of "just living". Therefore, thispaper designed a kind of intelligent flowerpot which canbe placed at home and take care of potted plants
independently3,4. Users can not only not take care of thepotted plants, but also know the situation of potted plants
at any time5. Due to the rapid development of Internet of

2. Embedded hardware system design
2.1. Overall structure of flowerpot hardware
When the signal of each sensor changes, the MCU will
give instructions to the motion system

Fig.1. Hardware block diagram© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

524

Songyun Shi, Yizhun Peng

2.2. Main control chip
2.3.2 Photosensitive module
The photosensitive resistance module is the mostsensitive to the ambient light, which is generally used todetect the brightness of the ambient light and trigger the
microcontroller or relay module. When the brightness of
the ambient light fails to reach the set threshold, the doterminal outputs the high level. When the brightness ofthe external environment light exceeds the set thresholdvalue, the do terminal outputs the low-level do The
output end can be directly connected with the single-chip
microcomputer, through which the high and low-level
level can be detected, so as to detect the change of thelight brightness of the environment; the analog output Aoof the small board can be connected with the ad module,
and the more accurate value of the ambient light intensity
can be obtained through AD conversion.

The core microcontroller of the embedded hardware
system is stm32f767igt6 chip, and the core is arm
cortex-m7, which is introduced by STMicroelectronics.
The operating frequency is up to 216mhz. On chip,
512KB flash memory and 256Kb SRAM memory areintegrated; there are up to 21 communication interfaces,
including 4 UARTS, 4 USART interfaces running at 12.5
Mbit / s, 5 SPI interfaces running at 50 Mbit / s, 3 I 2C
interfaces, one can, two SDIO and one USB with PHY on
chip 2.0 full speed device / host / OTG controller; 1 USB
2.0 High Speed / full speed device / host / OTG controller
with built-in dedicated DMA controller, up to 18
synchronous 16 bit timers, and up to 140 I / O ports withinterrupt capability. Compared with other MCU, it has
high performance, low cost and low power consumption,
which can fully meet the system requirements.

Fig.2. STM32F767IGT6 chipFig.4.The motor drive module

2.3.3 Soil moisture sensor
The soil moisture sensor consists of two detectors tomeasure the amount of water in the soil. The two probes
allow current to pass through the soil and measure themoisture content of the soil based on its resistance.
When there is more water, the soil conducts more current,
which means that the resistance will be smaller. So themoisture content will be higher. Dry soil reduces
electrical conductivity. Therefore, when there is lesswater, the soil conducts less electricity, which means thatit has a greater resistance. Therefore, the moisture content
will decrease.

2.3. Main components and sensors
2.3.1
Bluetooth serial communication module
The Bluetooth hc05 is a master-slave Bluetooth serial
port module. In short, when the Bluetooth device andBluetooth device are paired and connected successfully,
we can ignore the internal communication protocol ofBluetooth and directly use Bluetooth as a serial port.
When a connection is established, two devices share achannel, that is, the same serial port. One device sends
data to the channel, and the other device can receive thedata in the channel.

Fig.5. Soil moisture sensor
2.3.4 Integrated temperature and humidity sensor

Fig.3. Hc05 Bluetooth serial communication module© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

525

A Design and Implement

4. Testing and conclusion

The integrated temperature and humidity sensor uses
digital integrated sensor as probe and digital processing
circuit to convert the temperature and relative humidity inthe environment into corresponding standard analog
signal, 4-20mA, 0-5V or 0-10V. The temperature andhumidity integrated analog sensor can transform thechange of temperature and humidity value into thechange of current / voltage value at the same time, andcan be directly connected with various standard analog
input secondary instruments.

4.1. Test plan
We put the seeds of Yushu in the flowerpot for
cultivation, add a certain amount of water and fertilizer,
put them on the windowsill with suitable light intensity,and let the flowerpot carry out self-cultivation for one
month

Fig.8. Autonomic growth test
4.2. Test results
A month later, when we observed the flowerpot
again, it had already sprouted. We extracted and analyzed
the nutrient and water of the pot, and found that it met thenormal standard of plant growthFig.6. Integrated temperature and humidity sensor
3. Software design
STM32 microcontroller judges the growthenvironment of potted plants by reading the changeof the level value of each sensor connecting pin.
Each sensor detects the external environment in realtime and transmits the collected signals to the single
chip microcomputer. The Bluetooth module isconnected with the mobile phone Bluetooth, andreceives the information sent by the mobile phone
Bluetooth for corresponding operation. The user's
operation information can be transmitted to theflowerpot through WiFi.

Fig.9. Self growth test results

References
1. Zhang Liya; informatization and intelligent application ofresidential quarters [a]; science and education first andsustainable development in Western Development
2. Yang Weimin; discussion on the influence ofenvironmental factors on flower growth and development[J]; horticulture and seedlings; issue 12, 2019
3. Tian Jie. Research on detection device of traffic sensing
system based on Bluetooth [J]. Heilongjiang transportation
science and technology, 2008,41 (05): 216-217.
4. Ni Tianle; Cheng Ying; Li Linfei; Xu Xinyi; Huang lifeI;
research on indoor ecological and practical developmentof intelligent flowerpots [J]; rural economy and scienceand technology; issue 16, 2019

Fig.7. Software design flow chart© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

526

Songyun Shi, Yizhun Peng

5. Yuan Zhenghong; Wang sixu; Li Jipeng; Huo Fengcai; anunattended intelligent home flowerpot design; science andtechnology wind, issue 05, 2017
6. Cen Hua; Zhang Na; modular design: an innovative
thinking of green design; ecological economy; issue 05,
2008© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

527

Research of Attention-LSTM Model for Baby Cry Detection Robot
Tianye Jian1,2, Yizhun Peng 1,2*, Wanlong Peng1,2, Zhou Yang1,2

College of Electronic Information and Automation, Tianjin University of Science and Technology,
Tianjin, 300222, China
2
Advanced Structural Integrity International Joint Research Centre, Tianjin University of Science andTechnology, Tianjin, 300222, China* pengyizhun@tust.edu.cn
1

Abstract
In order to achieve the effective acquisition of frame-level speech features under different
emotional needs of baby, a speech emotion recognition model for baby based on improved
long-term and short-term memory (LSTM) network is established. The frame-level speechfeatures are used instead of the traditional statistical features to preserve the temporal
relationships in the original speech, and the traditional forgetting and input gates aretransformed into attention gates by introducing an attention mechanism, in order to improve theperformance of speech emotion recognition, the depth attention gate is calculated according to
the self-defined depth strategy. The results show that, in Fau Aibo Children’s emotional data
corpus and baby crying emotional needs database, compared with the traditional LSTM basedmodel, the recall rate and F1 score of this model are 3.14% , 5.50% , 1.84% and 5.49% higher,
respectively, compared with the traditional model based on Lstm and GRU, the training time isshorter and the speech emotion recognition rate of baby is higher.
Key words: children's emotional; time sequence relationship; frame-level speech feature; deep
attention gate; Long Short-Term Memory(LSTM)

1. Introduction
Children’s emotion recognition is an important part ofaffective computing1. Children are far less able thanadults to act rationally in emotional outbursts and in
response to different emotions, which can lead toemotional disorders if children are not able to act
rationally and are directed in a timely manner, which canlead to anxiety disorders and other mental healthproblems. Therefore, it is of great significance to use
appropriate algorithms or models to judge and guidechildren’s emotions.
The researchers conducted in-depth research on
children’s emotion recognition from acoustic features,
machine learning and deep learning2 it is proposed to use
Support vector machine and convolutional neural network
to construct a system for detecting children’s secondary
emotional states3 real-time emotional state of children isdefined by multi-agent based interaction system4 toestablish the children’s dual-modal emotion database anduse the dual-modal emotion recognition method tomeasure the proportion of children’s emotion contribution,and to point out that infants’(or young children’s)

emotion is more difficult to judge than older children’s,babies usually cry to express their needs to their parents
or guardians5 the Mel-Frequency Cepstrum Coefficients(MFCC) of infant cries were extracted and classified
based on Hidden Markov Model (Hmm) to identify
whether the infant cries were healthy or not6. The
Spectrogram was used as the feature vector, and theConvolutional Neural Network (CNN) was selected as theclassification model to classify and recognize the cryingof infants in pain, hunger and sleepiness7. The Support
Vector Machine (SVM) was used as the classifier toclassify the crying sounds of infants in the condition ofhunger, pain and sleepiness, and the recognition effectwas better than that of the Support Vector Machine(SVM).
Although the above algorithms have been successfully
applied to children’s emotion recognition, traditional
machine learning algorithms, as well as self encoders andconvolutional neural network in deep learning, can only
accept data with fixed dimensions as input, this is incontradiction with the fact that the effective length ofspeech is constantly changing. To solve this problem,
reference extracts emotion-related features (hereinafter© The 2021 International Conference on Artificial Life
528and Robotics (ICAROB2021), January 21 to 24, 2021

Tianye Jian, Yizhun Peng, Wanlong Peng, Zhou Yang

referred to as frame-level features) from short-term
speech frames, and applies static statistical functions(such as mean, variance, maximum, linear regression
coefficient) to frame-level features, finally, the feature
vectors with fixed dimensions are formed to represent thefeatures of the frame speech8-10. Although this method
solves the problem of model input, the time sequence
information of the original speech is lost through thestatistical analysis of the speech features.
2. Related work
2.1 LSTM network
The LSTM Network is a variant of Recurrent Neural
Network (RNN), which is mainly used to process
sequence information with long time difference. The
LSTM network can solve the problem that thelong-term information is difficult to store because of thegradient disappearance of RNN in the reverse
propagation11-13. The LSTM network has beensuccessfully applied in natural language processing(NLP) 14-16. In order to enhance the ability of LSTM
network to process data in specific tasks, theresearchers further optimized the internal structure of
LSTM network17. The fusion of the input gate and theforgetting gate of the LSTM network by the Gated
Recurrent Unit (GRU) reduces the model parameters,
the performance of LSTM network is better than that of
GRU18 in all machine translation tasks19. By using
CONVLSTM network structure, the computing methodof gate structure of LSTM is improved from Matrix
multiplication to convolution20. Infinite Impulse
Response Filter (IIR) Memory block of RNN isimproved to Finite Impulse Response Filter (Fir)Memory block by Feedforward Sequential Memory
Network (FSMN)21, but FSMN usually needs to stackvery deep layers, so FSMN has delay compared with
one-way LSTM network22. Advanced short-term
memory (Advanced LSTM) network, which uses
attention mechanism to weight multiple cell states, canbe effectively used for emotion recognition. However,reference23 indicates that this method does not change
the gate structure in the LSTM network and requires
more training time. In addition, researchers are
exploring how to stack LSTM structures to achieve
more reliable emotion recognition24. End-to-end
emotion recognition is achieved by extracting
multi-channel speech features from a 6s-long speech
waveform via a Convolutional Neural Network (CNN)
as input to the LSTM Network25. In this paper, 1280

In this paper, we propose an improved Long Short-Term
Memory (LSTM) based model for children’s speechemotion recognition. Based on the LSTM network
structure, the frame-level speech features are substituted
for the traditional statistical features, in order to obtainbetter recognition performance, attention gates were used
to replace the traditional forgetting gates and input gates,
and to construct the deep attention gates by weighting
attention in multiple cellular states.
kinds of abstract features were extracted from 6s-longspeech waveform by CNN, and then fused with facial
features as input of LSTM network.
The formula used in the traditional LSTM network is asfollows:
(1)
(2)
(3)
(4)
(5)
(6)
Is the sigmoid activation function,is the hidden
layer output at,
is the input at ,is thecell state at, is the candidate value of the cellstate at t,
、 、 .
and
are respectively
forgetting gate, input gate and output gate.
2.2 Attention Mechanisms
The attention mechanism is based on the human visual
attention mechanism. Attention makes people pay more
attention to the important part of the information
captured by vision, get as much detail information as
possible, and reduce the attention to the irrelevant
information around the target, that is, to suppress theirrelevant information26. In order to effectively utilize theinformation output from the LSTM network, the soft
attention mechanism is introduced into the LSTM
Network Model (hereinafter referred to as the LSTM
model), and it is successfully applied to the field ofmachine translation, by weighting the attention of the
LSTM model at different time, the relevance degree ofthe word to be translated and other words can be© The 2021 International Conference on Artificial Life
529and Robotics (ICAROB2021), January 21 to 24, 2021

Research of Attention-LSTM ModelIn this paper, the attention mechanism is introduced into
the internal structure of LSTM model, and the attention
gate-based LSTM model is proposed, which greatly
reduces the number of parameters of LSTM By
introducing the concept of depth into the attention gate,
the LSTM model can learn the input features better andavoid information redundancy. The attention-gate-based
LSTM structure proposed in this paper enables the cellstate at the last moment to determine the characteristics
to be noticed when calculating each time step, at thesame time, the attention gate is used to modify the
traditional forgetting gate and the input gate is used toweight the features that need attention.

expressed. The encoding-decoder structure based on
attention mechanism is proposed in reference27, and its
application in speech recognition is superior to HMM
decoding system. Based on the encoder-decoder
structure, a local attention model is proposed inreference 28, in which an alignment position is firstpredicted, and then a probability distribution similar tothe soft attention model is obtained over the l window at
the alignment position29. The single-head attention
mechanism is improved to the multi-head attention
mechanism, and the quality of machine translation is
improved significantly by Transformer model.
In recent years, the mechanism of self-attention has
become a hot topic30. Calculates the self-attention of theoutput of the LSTM model, calculates several fractions
for different time steps, and then proposes a new LSTM
model:

Because traditional input gates and forgetting gates are
only implemented by a single fully connected layer, themodel needs to be trained enough times to notice thecellular state information that needs to be left and the
new input information that needs to be added, whichcauses it to converge and slow down33. On this basis,the peephole connection is added, the cell state is also
taken as input, the cell state information is added into
the three gates, and the increase of the parameters
results in the increase of the training time and spacecomplexity. In this paper, self-attention is paid to every
cell state, and input candidate information is added tothe parts of the cell state that do not need attention, theself-attention Algorithm replaces the three matrices
needed for forgetting, input, and peeping connections
with attention gates.

(7)
Where a is the attention fraction matrix, hi is the output
of hidden layer cells at time I, and h is the stack resultof hi in each time of LSTM MODEL:(8)
The weighted output is expressed as:(9)

The attention gate at is defined as follows:

The introduction of the attention mechanism reduces
the computational burden of processing Galway input
data, and makes the task processing system more
focused on finding information in the input data that issignificantly related to the current output, thus
improving the output quality. In recent years,
researchers have applied attention mechanisms to theimprovement of affective effects in speech31. The
attention mechanism was used to screen the featuresand cross-link between the multilayer LSTM networks,
and good effect of emotion recognition was obtained. In
the output of RNN, a Local Attention mechanism is
proposed to improve the effect of emotion recognitionin multiple data sets32.

(10)
The activation is an activation function (the activation
function can be selected according to the need, but its
value range should be less than 1). The calculation
formula for updating the Cell State is:(11)
Attention Gate can improve the recognition rate of themodel while reducing the number of parameters andtraining time. Model distillation34, 8-bit quantization35,
and shared parameter36-37 are usually used in theexisting reports. In this paper, an attention gate basedon attention mechanism is proposed, whichsignificantly reduces the parameters of LSTM model. In
addition, the attention-gate-based model can reduce

3. Improved Deep Attention Gate
3.1 Attention Gate

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

530

Tianye Jian, Yizhun Peng, Wanlong Peng, Zhou Yang

more training time for longer input sequences due to themodification within the LSTM model. For example, fora layer LSTM model with an input dimension of 512
and an output dimension of 256, if the bias is ignored,
the commonly required parameters are: 1) the
dimension weight of [512 + 256,256 × 4] for the threegate structures and the candidate values; 2) if the cellstate at the last moment is taken into account incalculating the gate structure, a vector of 3 × [1,256] isadded as the peephole38. In this paper, we do not need to
introduce the peephole structure because we calculate
the self-attention as the attention gate directly to the cellstate. At the same time, the number of parameters
required is reduced to [512 + 256,256 × 2] and theweight of 2 × [256,256] for calculating attention due tothe combination of forgetting and input gates asattention gates. For this layer, the number of parameters
was reduced from the original 787200 to 524288,
reducing the number of parameters by 33.4%. For the
LSTM model with deeper layers, more complex modeland more data, the parameter quantity is reduced
effectively.

Fig.1 Deep-Attention-Lstm Cell internal
structure sketch

about the state of the cell at the time t-2, t-3, ... , t-n(depth length = n) , which constructs the
Deep-Attention-Lstm Cell structure, as shown in Fig.1
It is worth noting that the introduction of “depth” willlead to an increase in training time. This is because inaddition to forward increasing the number of attention
gates in the loop for computing the state of multiple
cells, reverse propagation also increases the number ofchain derivations. From the point of view of theparameters of the model, although the depth will cause
the increase of the training time, it will not cause theincrease of the parameters of the model because theattention gate weights of each layer are shared by V andW.

3.2 Deep Attention Gate
The LSTM model is usually used to process time series
information, but this information will increase with time,
therefore, the calculation of the LSTM model at a
certain time (that is, updating the cell state C andhidden layer output h) is only based on the external
input and the cell state and hidden layer output at theprevious time. Before the attention mechanism is put
forward, if each moment is considered several times
before, it will lead to too much information and the lossof important information, and increase the calculation
amount and lead to the gradient explosion. However,
the information of cell state at t-time is not only related
to the information at t-1, but also closely related to theinformation at t-2, which is selectively forgotten at t-1.
In this paper, the concept of deep forgetting gate is
proposed and the corresponding input gate is designed.

In this paper, the aim of depth is to improve theperformance of speech emotion recognition. In order tostudy the performance improvement, the following
experiments were carried out:

The deep forgetting gate looks not only at the
information about the state of the cell at the last
moment (depth length = 1) , but also at the information

3.3 Training Framework

In experiment 1 investigated the effect of depth
performance on children’s emotion recognition rate.
The attention-gate-based LSTM model (hereafterreferred to as the attention-gate LSTM model) at Depths
1, 2, and 3 is used for comparison.
In experiment 2, the effects of decreasing the number ofparameters and training time on speech emotion
recognition performance were investigated. The
attention gate LSTM model with depth 1 was comparedwith the traditional GRU model and LSTM model.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

531

Research of Attention-LSTM Modelthe game, with the most emotionally salient dataretained. Natural language contains 48,401 words. In
order to ensure the accuracy of the tagging, each sampleof the Corpus is auditioned by 5 language majors andthen marked by voting. This paper selects fivecategories of tags defined in the INTERSPEECH 2009
emotional
challenge:
A(Angry,Touchy,Reprimanding) , E(Emphatic) , N (Neutral) ,P(Motherese、Joyful) and R (Rest) .

Fig. 2 Training framework of depth ofattention gate LSTM model

2) Infant crying emotional needs corpus: Since there isno universal crying emotional needs Corpus in theworld, the author collaborated with a hospital in China
to record and annotate the audio files of infant crying infive states: Angry, Hungry, Pain, Sad and Tired. Inorder to improve the quality of the Corpus, the author
screened the emotional speech data of infant crying byartificial method, and got rid of the speech-related
frames of parents comforting children, and the
speech-related frames of two or more babies crying at
the same time. The Corpus consists of 10 infants (5boys and 5 girls respectively), each infant has 20 items
in each state, which is 5 × 10 × 20 = 1000 items.

The training framework for the deep attention gate
LSTM model is shown in Fig.2. Among them, LSTM0
represents the LSTM model of the first-level deep
attentional gate, and LSTM1 represents the LSTM
model of the second-level deep attentional gate.
extracts speech features [8-10] from INTERSPEECH
in frame t after framing and windowing,andarethe hidden layer output and cell state of its
corresponding LSTM model output. As can be seenfrom Fig. 2, the input state of the traditional LSTM
model at time t is ( h t -1 , C t -1 ) , while in the training of
this paper, the input state at each time is expanded to( h t -1 , { C t -1 , C t -2 , ... , C t -L }) , where L is thedepth of the attention gate. The last state of the lastlayer of LSTM, which contains all the temporal
information of the preorder, is input into the subsequentclassification network for the identification ofchildren’s emotions.

3) CASIA Chinese affective Corpus was recorded by
Chinese Academy of Sciences, and the pronunciationsof six emotions, namely angry, happy, fear, sad,surprise and neutral, were carried out by four related
professionals. There are 9600 Corpora in this corpus.
4.2 Frame Level Feature Selection

4. Experimental Setup and Analysis
4.1 Experimental Setup

The experiment selects part of the frame-level features
based on the speech emotion features of
INTERSPEECH. In reference [8], 16 low-leveldescriptors (LLD, zero-crossing rate, root-mean-square
Frame Energy, pitch frequency and Mel Frequency
Cepstrum Coefficients 1-12) and their difference
coefficients are extracted. For each descriptor, 12
statistical functions are calculated, so the total
eigenvector has 16 × 2 × 12 = 384 features. On thisbasis, the speech emotion characteristics of
INTERSPEECH 2010(IS2010) have been increased to
38 kinds of LLD, so the total feature dimension has
been expanded to 1582 dimensions. The feature
dimension of the speech feature set is increased to 6373
dimensions.

In the experiment, two databases with different
emotional representation forms are used to verify thevalidity of the algorithm in this paper. In order to study
the performance of this algorithm in dealing with othertypes of emotion recognition problems, and whether
reducing the number of parameters can optimize thetime or reduce the performance, Fau Aibo Children’s
affective Corpus, infant crying affective needs Corpus
and CASIA Chinese affective Corpus were used toverify the results39.
FAU AIBO EMOTIONAL Corpus for children: Fau
Aibo used high performance wireless earphones tocollect and record the vocalization of 51 children aged
around 10 years old and their electronic pet Aibo during

4.3 Setting of Experimental Parameters© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

532

Tianye Jian, Yizhun Peng, Wanlong Peng, Zhou Yang

The original data is divided into two parts, the training
set and the test set, which are separated from each other,and the ratio of the training set and the test set is 4:1.
The experiments all adopt unidirectional two-layer
LSTM stack structure and use one full connection layer
and onelayer as the training model. During
the training, a small batch of gradient descent is used
andis used as the activation function, as shownin Table 1. In order to ensure the validity ofexperimental comparison, the same Corpus and modelexperimental parameters are identical.

Fig. 3The performance of infant crying
emotional needs corpus in different LSTM
models

LSTM model with the attention door proposed by using

5. Algorithm Performance Analysis
The traditional LSTM model gets rid of the redundant
information through the forgotten gate and gets the new
information through the input gate. In this paper, we use
the self-attention and the basic structure of LSTM to do
self-attention to the cell state, so as to compare theforgetting gate and the input gate of LSTM. At the sametime, considering the correlation of time series
information, the depth-based self-attention Gates areproposed and compared at the depth of 1, 2 and 3. Fourkinds of models were compared: Traditional model,
LSTM + deepf_1 model, LSTM + deepf_2 model, andLSTM + deepf_3 model. The depth distributions ofthese models are 0, 1, 2 and 3, as shown in Fig.3. As
can be seen from Fig.3 and Fig.4 , after replacing theforgetting door and the output door of the traditional

Fig. 4 Performance of different LSTM
models using Fau Aibo children's
emotion corpus database

the infant crying affective needs corpus and Fau Aibo
children affective Corpus, the convergence rate of theattention gate LSTM model on the training set and thetest set is much higher than that of the traditional LSTM
model, when FAU AIBO is used in children’s affective
Corpus, the traditional LSTM model starts to converge
at about 30000, while the attention gate model starts toconverge at about 17000 When the model converges,
the attention-gate LSTM model outperforms the
traditional LSTM model in the average recognition rate

Table 1. Experimental parameters.
Name of Parameter

Baby crying needs corpus parametervaluesFau Aibo

CASIA

Eta

1e-3
beta2= 0.7

1e-4
beta2= 0.9

1e-4
beta2= 0.9

Batch size

64

128

128

Epochs

1 500

1 200

1 200

Lstm cells[512, 256]

[512, 256]

[512, 256]

Dense layers[256, 128]

[256, 128]

[256, 128]

Softmax layers[128, 5]

[128, 5]

[128, 6]

L2

1e-4

1e-4

1e-4

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

533

Research of Attention-LSTM Modelof children’s emotion.

From Table 2, it can be seen that the recall rate ofattention-gate LSTM model is better than that of
traditional LSTM model except that the items ofdrowsiness and sleepiness are close to each other The
F1 scores of attention gate LSTM model were better
than those of traditional LSTM model in five kinds ofemotion. In the aspect of depth, the performance of theattention-gate LSTM model of depth 3 and depth 2 isclose to that of the attention-gate LSTM model of depth
1, except “sad”, the recall rate and F1 score of the other4 items of the model are better than those of depth 1.

According to the analysis above, the performance ofattention gate LSTM model is improved because it
modifies the forgetting gate and input gate of traditional
LSTM model, it makes the LSTM model leave
important information by self-attention to the cell stateat the last moment, and supplement the unimportant
information as the new input in the corresponding
position, thus improving the performance of the LSTM
model. The attention gate LSTM model introduces theconcept of depth so that each forgetting operation isdetermined by multiple cell states rather than one ofthem.

From Table 3, it can be seen that when FAU AIBO isused in children’s affective corpus, the recall rate and F
1 score of attention gate LSTM model are lower thanthose of traditional LSTM model except that the e classis lower than that of traditional LSTM model, and theother four items are better than traditional LSTM model.
In the aspect of depth, the performance of theattentional gate LSTM model of depth 3 and depth 2 isclose to that of the attentional gate LSTM model ofdepth 1, except for class R, the recall rate and F1 scoreof the other four items of the attentional gate LSTM
model are better than those of depth 1.

In order to compare the performance of different
models in the test set for each kind of emotion, theperformance index of the model with the highest
recognition rate from the beginning of training to theend of the test set was Quantitative analysis, theperformance indicators
obtained from the infant crying affective needs Corpus
and Fau Aibo children affective corpus are shown intables 2 and 3. It can be seen that for the test set, theTable 2. Performance indicators of different LSTM models using emotional needs corpus of infant crying
database.
model
MeasureAngry
Hungry
Pain
Sad
Tired
AVG
sample size
—
40
45
34
35
46
200
Recall
0.875
0.911
0.824
0.829
0.957
0.885
LSTM
F1 SCORE
0.875
0.901
0.848
0.906
0.889
0.885
LSTM+deepf_1
LSTM+deepf_2
LSTM+deepf_3

Recall

0.875

0.889

0.882

1.000

0.935

0.915

F1 SCORE

0.909

0.941

0.822

0.946

0.945

0.916

Recall

0.925

0.978

0.882

0.943

0.957

0.940

F1 SCORE

0.949

0.957

0.923

0.930

0.936

0.940

Recall

0.925

0.978

0.824

0.943

0.957

0.930

F1 SCORE

0.949

0.926

0.875

0.943

0.946

0.930

performance index of the attention gate LSTM model isbetter than the traditional LSTM model.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

534

Tianye Jian, Yizhun Peng, Wanlong Peng, Zhou Yang

Table 3. Performance indicators of different LSTM models using Fau Aibo children's emotion corpus
database.
model
Measure
A
E
N
P
R
AVG
sample size—
611
Recall0.352
LSTM
F1 SCORE
0.339
Recall0.326
LSTM+deep
f_1
F1 SCORE
0.358
Recall0.339
LSTM+deep
f_2
F1 SCORE
0.342
Recall0.360
LSTM+deep
f_3
F1 SCORE
0.371
It should be noted that the sample size of Fau Aibo
Children’s affective corpus is uneven, with a maximumof 5,376 samples in n category and only 215 samples in
P category. From the above analysis, with the increaseof depth, the model can enhance the learning of a small
number of samples. Compared with the traditional
LSTM model, the recall rate of LSTM + deepf_1model
was increased by 5.50% , and the F 1 score wasincreased by 5.49% , and the recall rate of LSTM +deepf_2 model was increased by 3.14% when Fau Aibo
was used, the F1 score of LSTM + deepf_3 model wasincreased by 1.84% .

1 508
0.373
0.373
0.300
0.338
0.289
0.340
0.327
0.363

5 376
0.755
0.743
0.814
0.770
0.826
0.772
0.800
0.765
References
1. LU Fang,

215
0.088
0.110
0.153
0.173
0.158
0.173
0.191
0.192

546
0.081
0.093
0.119
0.133
0.081
0.104
0.095
0.112

8 256
0.594
0.586
0.621
0.603
0.625
0.601
0.619
0.604

CHEN Guopeng. An overview on thedevelopment of children's emotion regulation[J].
Psychological Science, 2003, 26(5): 928-929. (inChinese)2.

GONG Y, POELLABAUER C.Continuous assessment ofchildren's
emotional
statesusing
acoustic
analysis[C]//Proceedings of 2017 IEEE International
Conference on Healthcare Informatics.Park City, USA:IEEE Press, 2017: 171-178.

3.

DE SILVA P R, MADURAPPERUMA A P,
MARASINGHE A, et al.A multi-agent based interactive
system towards Child ’s emotion performances quantified
through affective body gestures[C]//Proceedings of the
18th
International
ConferenceonPattern
Recognition.Hong Kong, China: IEEE Press, 2006:
1236-1239.

4.

DAI Weijia.Research on expression and speech bimodal
emotion recognition of children[D].Nanjing: Southeast
University, 2016.

5.

LEDERMAN D, COHEN A, ZMORA E, et al.On the useofhiddenMarkovmodelsininfants'cry
classification[C]//Proceedings of the 22nd Convention on
Electrical and Electronics Engineers in Israel.Tel-Aviv,
Israel: IEEE Rress, 2002: 350-352.

6.

CHANG Chuanyu, LI Jiajing.Application of deeplearning for recognizing infant cries[C]//Proceedings of2016 IEEE International Conference on Consumer
Electronics-Taiwan.Nantou County, China: IEEE Press,
2016: 1-2.

7.

CHANG C Y, CHANG C W, KATHIRAVAN S, et al.
DAG-SVM based infant cry classification system using
sequential forward floating feature selection[J].
Multidimensional Systems and Signal Processing, 2017,

6. Conclusion
In this paper, a child speech emotion recognition model
based on improved LSTM network is proposed
frame-level speech features are used instead of
traditional speech features, the attention mechanism isintroduced into the forgetting gate and the input gate of
the internal structure of the LSTM network model toform the attention gate. The experimental results showthat the recognition rate of this model is significantly
higher than that of traditional LSTM model, and therecognition rate of depth model is higher than that ofshallow model. In CASIA database with other emotions,the training time of this model is shorter than that of
LSTM model, and the recognition rate is higher thanthat of LSTM model and GRU model. The next step isto introduce this model into the fields of speech
recognition, machine translation and lie detection, totest and study the continuous affective Corpus and to
improve the model for calculating attention scores, tofurther improve children’s speech emotion recognition
rate.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

535

Research of Attention-LSTM Model

28(3): 961-976.

8.

9.

nowcasting[EB/OL].(2015-06-13)[2019-10-20].

20. ZHANG Shiliang, JIANG Hui, WEI Si, et al.Feedforward

SCHULLER
B,
STEIDLS,
BATLINER
A.Theinterspeech2009
emotionchallenge[C]//Proceedings of the 10th Annual Conferenceof the International Speech Communication Association
2009.Brighton, UK: IEEE Press, 2009: 1-7.

sequential memory neural networks without recurrent
feedback[EB/OL].(2015-10-09)[2019-10-20].

21. ZHANG Shiliang, LEI Ming, YAN Zhijie, etal.Deep-FSMN for large vocabulary continuous speech
recognition[C]//Proceedings of 2018 IEEE International
Conference on Acoustics, Speech and Signal
Processing.Calgary, Canada: IEEE Press, 2018:5869-5873.

SCHULLER B, STEIDL S, BATLINER A, et al.The
INTERSPEECH
2010
paralinguistic
challenge[C]//Proceedings of the 11th Annual Conferenceof the Inter-national Speech Communication Association
2010.Makuhari, Japan: IEEE Press, 2010: 25-32.

22. TAO Fei, LIU Gang.Advanced LSTM: a study about

10. SCHULLER B, STEIDL S, BATLINER A, et al.The
INTERSPEECH 2016 computational paralinguistics
challenge:
deception,
sincerityandnativelanguage[C]//Proceedings of Interspeech 2016.San
Francisco, USA: ISCA, 2016: 2001-2005.

better time dependency modeling in emotion
recognition[C]//Proceedings of 2018 IEEE International
Conference on Acoustics, Speech and Signal
Processing.Calgary, Canada: IEEE Press, 2018:
2906-2910.

11. HOCHREITER S, SCHMIDHUBER J. Long short-term

23. XIE Yue, LIANG Ruiyu, LIANG Zhenlin, et al. Speech

memory[J]. Neural Computation, 1997, 9(8): 1735-1780.

emotion classification using attention-based LSTM[J].
ACM Transactions on Audio, Speech, and Language
Processing, 2019, 27(11): 1675-1685.

12. HAN Wenjing, LI Haifeng. A brief review on emotional
speech databases[J]. Intelligent
Applications, 2013, 3(1): 5-7.

Computerand24.. TRIGEORGIS G, RINGEVAL F, BRUECKNER R, etal.Adieu features end-to-end speech emotion recognition
using
a
deep
convolutional
recurrent
network[C]//Proceedings of 2016 IEEE International
Conference on Acoustics, Speech and Signal
Processing.Shanghai, China: IEEE Press, 2016:
5200-5204.

13. ATHIWARATKUN

B, STOKES J W.Malware
classification with LSTM and GRU language models anda character-level CNN[C]//Proceedings of 2017 IEEE
International Conference on Acoustics, Speech andSignal Processing.New Orleans, USA: IEEE Press, 2017:2482-2486.

25. TZIRAKIS P, TRIGEORGIS G, NICOLAOU M A, et al.

14. MERITY S, KESKAR N S, SOCHER R.Regularizing
andoptimizing
LSTM
models[EB/OL].(2017-08-07)[2019-10-20].

End-to-end multimodal emotion recognition using deepneural networks[J]. IEEE Journal of Selected Topics in
Signal Processing, 2017, 11(8): 1301-1309.

language

15. LI Wei, MAK B.Derivation of document vectors from

26. BAHDANAU D, HYUN CHO K, BENGIO Y.Neural

adaptation of LSTM language model[C]//Proceedings ofthe 15th Conference of the European Chapter of the
Association for Computational Linguistics.Stroudsburg,
USA: Association for Computational Linguistics, 2017:456-461.

machine translation by jointly learning to align andtranslate[EB/OL].(2014-09-01)[2019-10-20].

27. CHOROWSKI J, BAHDANAU D, CHO K, etal.End-to-end continuous speech recognition using
attention-based
recurrentNN:
first
results[EB/OL].(2014-12-04)[2019-10-20].

16. LIU Yue, ZHAI Donghai, REN Qingning. News textclassification based on CNLSTM model with attention
mechanism[J]. Computer Engineering, 2019, 45(7):303-308.

28. LUONG M T, PHAM H, MANNING C D.Effective
approaches to attention-based neural machinetranslation[EB/OL].(2015-08-17)[2019-10-20].

17. CHO K, VAN M B, GULCEHRE C, et al.Learning phrase

29. VASWANI A, SHAZEER N, PARMAR N, et al.Attention

representations using RNN encoder-decoder for statisticalmachine translation[EB/OL].(2014-06-03)[2019-10-20].

is all you need[C]//Proceedings of Advances in neural
information processing systems 2017.Long Beach, USA:IEEE Press, 2017: 5998-6008.

18. BRITZ D, GOLDIE A, LUONG M T, et al.Massive
explorationofneuralmachinetranslationarchitectures[EB/OL].(2017-03-11)[2019-10-20].

30. LIN Z H, FENG M W, SANTOS C N, et al.A structured
self-attentive
sentenceembedding[EB/OL].(2017-03-09)[2019-10-20].

19. SHI Xingjian, CHEN Zhourong, WANG Hao, etal.Convolutional LSTM network: a machine learning
approachfor
precipitation

31. XIE Yue, LIANG Ruiyu, LIANG Zhenlin, et al.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

536

Tianye Jian, Yizhun Peng, Wanlong Peng, Zhou Yang

Attention-based dense LSTM for speech emotion
recognition[J]. ICE Transactions on Information and
Systems, 2019, 102(7): 1426-1429.

quantization
andcoding[EB/OL].(2015-10-01)[2019-10-20].

37. DAI Dawei, YU Liping, WEI Hui. Parameters sharing in

32. MIRSAMADI S, BARSOUM E, ZHANG C.Automatic

residual neural networks[J]. Neural Processing Letters,
2020, 51(2): 1393-1410.

speech emotion recognition using recurrent neuralnetworks with local attention[C]//Proceedings of 2017
IEEE International Conference on Acoustics, Speech andSignal Processing.New Orleans, USA: IEEE Press, 2017:2227-2231.

38. HOCHREITER S, SCHMIDHUBER J. Long short-term
memory[J]. Neural Computation, 1997, 9(8): 1735-1780.

39. PAN Shifeng, TAO Jianhua, LI Ya.The CASIA audio
emotion recognition method for audio/visual emotion
challenge 2011[C]//Proceedings of Affective Computing
and Intelligent Interaction.Berlin, Germany: Springer
Berlin Heidelberg, 2011: 388-395.

33. GERS F A, SCHMIDHUBER J.Recurrent nets that timeand count[C]//Proceedings of Proceedings of theIEEE-INNS-ENNS International Joint Conference on
Neural Networks.Como, Italy: IEEE Press, 2000:
189-194.

40. JAITLY N, HINTON G.Learning a better representationof speech soundwaves using restricted boltzmann
machines[C]//Proceedings of 2011 IEEE International
Conference on Acoustics, Speech and Signal
Processing.Prague, Czech Republic: IEEE Press, 2011:5884-5887.

34. HINTON G, VINYALS O, DEAN J.Distilling theknowledge
inanetwork[EB/OL].(2015-03-09)[2019-10-20].

huffmanneural

35. JACOB B, KLIGYS S, CHEN B, et al.Quantization andtrainingofneuralnetworks
forefficientinteger-arithmetic-only inference[C]//Proceedings ofIEEE Conference on Computer Vision and Pattern
Recognition.Salt Lake City, USA: IEEE Press, 2018: 1-5.

36. HAN S, MAO H Z, DALLY W J.Deep compression:
compressing deep neural networks with pruning, trained© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

537

A Design and Implement of an Automatic Intelligent Car
Chengzhi Liu , Yizhun Peng*,Jikai Zhao , Dezhi Yang,

College of Electronic Information and Automation,
Tianjin University of Science and Technology, Tianjin, 300222,China;
E-mail: * pengyizhun@tust.edu.cn
www.tust.edu.cn

Abstract
In order to study the situation that some unmanned vehicles need to drive repeatedly on fixed lines, this article
introduces an autonomous smart car based on the STM32 platform. Infrared detectors are used to determine
whether the vehicle is driving on a prescribed route. Ultrasonic detectors can prevent the vehicle from colliding
with other people or vehicles. Inexpensive cars can help express companies to transport packages and letters to
reduce labor costs. Large vehicles can be used to carry large cargo such as containers in docks and other places.
Keywords: Automatic driving, Ultrasonic detectors, Infrared detectors, Parcel shipping

1. Introduction
With the development of online shopping and the impactof the epidemic, contactless delivery of goods is now ofgreat importance. In universities and office buildings,
cargo delivery robots are becoming more and morecommon, but high prices often limit their application
scenarios. Recently, the China Post Bureau released the
May Express Development Index Report. In May 2020,
China’s express delivery development index was 344.2,
an increase of 73.5% year-on-year. Among them, thedevelopment scale index, service quality index,
development capability index and development trend
index were 291.1. 622.4. 202.4 and 94.0, respectively.
Upgrade, showing a development trend of capacity
expansion and quality improvement.

Fig.1.Block diagram of the overall structure

After setting the driving path in the main control of thecar, the car will run independently. The ultrasonic module
can transmit the signal of whether there are obstacles on
the driving path to the car. The infrared module is
responsible for the judgment of the driving path of the car,and the driving direction and speed of the car areautomatically set by the main control.

2. Hardware design
2.1. Overall structure of the car

2.2. Control section© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

538

Chengzhi Liu, Jikai Zhao, Dezhi Yang

STM32 microcontroller is used as the main control. The
single-chip minimum system includes a display, matrix
keyboard, A/D, D/A and other modules, which cansignificantly reduce the design of peripheral circuits andreduce the difficulty of system design.
The STM32 series microcontroller is a 32-bit
microcontroller, based on the ARM Cortex-M3 core
specially designed for embedded applications that require
high performance, low cost, and low power consumption.

Fig.3.The motor drive module

Power module.
In terms of power supply, rechargeable batteries, theworking principle is the mutual conversion of chemicalenergy and electrical energy. When the voltage isinsufficient, it can be charged to keep the voltage stable,
and the voltage can be maintained at a stable value,
making the experiment more accurate. Rechargeable

2.3. Drive sectionStep-down module
The step-down module can reduce the power supply
voltage to the appropriate voltage required by the module,
usually 5v and 3.3v. Therefore LM2596S were chosen
DC-DC high-power DC voltage regulator board.

Fig.2.The step-down module

Fig.4.The DC geared motor

Motor drive module
The TB6612FNG used in this design is a motor drive IC
that can independently control two DC motors in both
directions. It has a high degree of integration, and at thesame time can provide sufficient output capacity, also hasadvantages in operating performance and energy
consumption. Therefore, it can be used as an ideal motor
drive device in an integrated and miniaturized motor
control system.

batteries can be used multiple times, which is beneficial
to protect the environment.In terms of environmental
protection and convenience, the power supply module
uses rechargeable batteries.
DC geared motor
In order to enable the car to carry heavier goods and get
the speed of the car, a DC geared motor with a reduction
ratio of 30 is the better choice.The encoder based on Hall
component can return the current speed of the car to themicrocontroller to control the car.
2.4. Tracking section
TCRT500 infrared integrated transceiver based on
infrared detection method is our choice. The infrared
detection method uses the characteristics of different
reflection intensities of infrared on the surface of objectsof different colors to distinguish the current detection
state. The infrared photoelectric tube can greatly reduce
external interference. The circuit design is relatively© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

539

A Design and Implement

simple, the detection information speed is fast, and themarket price is low.

can automaticallylarge-angle slopes.

avoid obstacles

and

drive

on

Fig.5.The infrared detection

2.5. Ultrasonic obstacle avoidance sensor
The model of ultrasonic obstacle avoidance sensor usedin this car is ULB-1 ultrasonic distance sensor, which has
the characteristics of high resolution, high precision andlow consumption. Not only in the design but also in theinterference noise processing, with anti-noise interference
ability. And for the different size of the target and thechange of the supply voltage, do the sensitivity
compensation. In addition, it also has standard internal
temperature compensation, which makes the measured
distance data more accurate.

Fig.7.The system flow chart

3.2. infrared sensor
TCRT5000 infrared tracking module is our selected
tracking module. The module has three leads: VCC,
GND and OUT, and VCC is connected to a +5V
power supply. After correct connection, it can befound that when the probe detects the driving track,
the indicator light on the rear of the module willremain off, and the output of the OUT terminal willchange to low level, and the OUT terminal willchange to high level. Normally level.
3.3. DC motor driver
The rotation speed and direction of a motor are controlled
by one PWM signal and two control signals. Pin
definition of motor drive is shown in Tab.1

Fig.6.Ultrasonic obstacle avoidance sensor

3. Software design
3.1. Program function description and design ideas
When the car arrives at the loading area, it automaticallystops waiting for loading, and the on-board weightdetection system will automatically detect whether theweight of the goods exceeds the standard. After theloading is completed, it will drive to the unloading areaaccording to the set path. During the driving process, it

Fig.8.Block diagram of the motor driver© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

540

Chengzhi Liu, Jikai Zhao, Dezhi Yang

4.2. Test results
Tab.1.Pin definition of motor drive

After multiple tests and repeated experiments, themaximum load of the car is 7.6 kg, and the load is 5 kg
when the minimum speed is guaranteed. There is acertain probability of taking the wrong path when drivingon a complex route, especially when multiple trajectories
intersect in one place.

Number Name Description
1

PWMA Speed control pin of motor A

3

AIN2 Direction control pin 2 of motor A

4

AIN1 Direction control pin 1 of motor A

5

STBY Drive enable pin (connect to VCC)

6

BIN1 Direction control pin 1 of motor B

7

BIN2 Direction control pin 2 of motor B

8

PWMB Speed control pin of motor B

9

VM

Positive motor power supply

10

VCC

Logic power supply positive

11

GND

Logic power supply negative

12

GND

Motor power negative

13

AO1

A road motor output 1

14

AO2

A road motor output 2

15

BO1

B road motor output 1

16

BO2

B road motor output 2

4. Testing and conclusion

Fig.10.Weight test

4.1. Test plan

References

Detect the maximum mass that can be carried when
the car is given maximum power.
Lay complex lines on the ground to detect thefunctions of infrared tracking modules andalgorithms.
Put many cars on the line to run at the same time todetect the ultrasonic obstacle avoidance function.

1.Research on Application of Automated Warehouse
Transportation Path Planning[D]. Teng Zhipeng. Changchun
University of Technology 2020
2.Razvan Luca, Fritz Troester, Robert Gall and Carmen
Simion,Autonomous parking procedures using ultrasonicsensors,Annals of DAAAM & Proceedings,Jan. 1, 2010 .
3. Wu Guojing, etc. Microcontroller Application Technology
China Electric Power Press, 2004.1
4.Design and Research of Mobile Robot Control System Basedon Automatic Guidance[J]. Li Peiying, Liu Weiliang.
Innovation in Science and Technology. 2020(29)5.Realization of obstacle avoidance technology for smart car[J].
Li Lei, Cao Shengwei. Micromotor. 2011(06)6.Design and implementation of ultrasonic anti-collision speed
measurement and obstacle avoidance instrument[J]. Su Qiumei,
Liu Yarong. Science and Technology Innovation andApplication. 2020(29)Fig.9.Line inspection test

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

541

Target detection in remote sensing image based on deep learning
Lianchen Zhao 1,2, Yizhun Peng 1,2*, Di Li1,2, Yuheng Zhang12

College of Electronic Information and Automation,
Tianjin University of Science and Technology, China
1

Advanced Structural Integrity International Joint Research Centre,
Tianjin University of Science and Technology, China
E-mail: * pengyizhun@tust.edu.cn
www.tust.edu.cn

2

Abstract
For high-resolution optical remote sensing images, there are still many challenges in target detection. In this paper,
deep learning algorithm is used to detect the target in remote sensing image. Improve and optimize the deeplearning target detection algorithm. When the selected data set is used for target detection, the AP value is improved,
which leads to the concept of multi-scale feature fusion feature pyramid and residual network. By improving theselected Yolov3 network model, the detection effect of the two targets of aircraft and ships in remote sensing
images has been significantly improved.
Keywords: Deep learning; Target detection; Residual network; Remote sensing image

1. Introduction

2. One-Stage target detection algorithm

For high-resolution optical remote sensing images, there
are still many challenges in target detection. This paper
introduces the multi-scale target detection in opticalremote sensing images by using Yolov31, and comparesthe detection results of multi-scale targets on the selected
remote sensing data sets by using the improved Yolov3
model.
Target detection is an important direction of computer
vision and digital image processing2. Target detection is
also widely used in robot navigation, real-time vehicle
monitoring, defective product detection in industry,
aerospace ship detection and other target detection.
Through the algorithm of target detection in deeplearning, we can realize the accurate detection of thetarget, which not only has a great reduction in manpower,
but also has a high efficiency improvement, which has
important practical significance. Target detection
algorithms in deep learning can be divided into two
categories: one-stage algorithm and two-stage algorithm3.

One-Stage, a target detection algorithm based on deeplearning, has a faster detection speed when detecting thetarget4, because the target detection algorithm discards
the process of single region recommendation. Thisalgorithm is first proposed from CVPR 2016, and Yolo(you only look once: unified, real time object detection)is an innovative one stage detection.
The method of generating candidate regions first and thendetecting has relatively high accuracy, but it is relatively
slow in terms of detection speed.
Yolo did not remove the candidate regions5, but directly
divided the input network image into 49 7 * 7 grids. Any
existing grid predicted two boundary boxes, so as toachieve the prediction of 98 boundary boxes. It can beroughly understood as a rough selection of 98 candidate
regions on the input image, which cover the entire regionof the image. Therefore, regression prediction is used tocompare the existing 98 candidate boxes to get the final
applicable boundary box.
Yolo network draws lessons from the structure of
GoogleNet6 classification network. Compared with© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

542

Lianchen Zhao, Yizhun Peng, Di Li, Yuheng Zhang

GoogleNet classification network, the 1x1 convolution
layer and 3x3 convolution layers are used to replace theinception module in Yolo. There are 24 convolution
network layers and 2 full connection layers in the wholedetection network. Compared with the previous versionof Yolo, Yolov3 adjusts the network structure, and uses
the Darknet-537 network structure compared with Yolov2.
The specific network structure is shown in Figure 1
below:
Fig. 2. Leaky Relu function graph

In Yolov3, anchor boxes are needed in bounding box
prediction. In Yolov3, K-means clustering algorithm isused to obtain anchors suitable for data sets. The goal of
K-means algorithm is to divide n samples into K clusters
according to the calculation results8, so that the similar
samples in n samples can be divided into the same cluster.
The calculation method used to measure the similarity is
to use the size of Euclidean distance (2).
n
(2)
2
2
2
2
d ( x, y )=Fig. 1. Darknet-53 network structure∑ (x − y )( x1 − y1 ) + ( x2 − y2 ) +  + ( xn − yn ) =

i =1

i

i

The loss function in Yolov3 is shown in formula (3) as
In Figure 1, the main structure of the network structure offollows:
darknet-53 mainly consists of DBL module, Upsample
S
B
module, Shortcut module, Res module and Route module.
Loss λcoord ∑∑ I ijobj [( xi − xˆij ) 2 + ( yi − yˆij ) 2 ] +==i 0=j 0
The main components of DBL are convolution network,
S
B
BN and Leaky relu.
λcoord ∑∑ I ijobj [( wij − wˆ ij ) 2 + ( hij − hˆij ) 2 ] −The size of network input image is 416 * 416 * channels.
(3)=i 0=j 0
S
B
After 5 times of down sampling and 2 times of up∑∑ Iijobj [Cˆi j log(Ci j ) + (1 − Cˆi j ) log(1 − Ci j )] −sampling stitching through convolution layer, three kinds=i 0=j 0of feature maps will be output. There are 53 layers ofSBλnoobj ∑∑ I ijnoobj [Cˆ i j log(Ci j ) + (1 − Cˆi j ) log(1 − Ci j )] −convolutions in the darknet-53 network. Except for the=i 0=j 0last FC, there are 52 convolutions as the main network
S
model structure. For the low-level convolution layer, theI ijobj ∑ [ Pˆi j log( Pi j ) + (1 − Pˆi j ) log(1 − Pi j )]∑c∈classes
field of vision is relatively small, which is responsible for =i 0
I obj
the detection of small targets. For the deep convolution
In the above formula, ij is used to determine whether
layer, the field of vision is relatively large, which is
the j anchor box in the i grid is responsible for the
obj
obj
responsible for the detection of larger targets.
current object. If I ij = 1 is responsible, if I ij = 0 is not.
The activation function adopted by yolov3 is leaky relu.
ˆjCˆ i jCompared with relu, this activation function sets allparameter is the confidence degree, Ci is the real
negative values to zero. Leaky relu function only gives avalue in the training process. The value of this parameter
non-zero slope to all negative values. The mathematicaldepends on whether the bounding box of grid cell isexpression is shown in (1)：responsible for the prediction of an object. If it is
2

2

2

2

2

yi = {xxii if
aiif

xi ≥ 0xi < 0ˆjˆjresponsible for Ci = 1 , otherwise, Ci = 0 .
Central coordinate error:

(1)

S2

B

∑∑ I [( x − xˆ ) + ( y − yˆ ) ]The leaky relu function image is shown in Figure 2:=i 0=j 0

obj
ij

i

i

j 2

i

i

j 2

The formula of wide coordinate error in calculation:© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

543

(4)

Target Detection In Remote
S2B=i 0=j 0

obj
ij

i
j

i
j

2

i
j

i
j

then get the final evaluation result of the training result
according to the test results. Since the data set used in thisexperiment is to integrate part of the data in NWPU
VHR-10 remote sensing data set and part of the data in
DOTA-v1.5 remote sensing data set, and only contains
two types of targets, the detection results are shown in
Figure 4, and the test evaluation is shown in table 1:(5)∑∑ I [( w − wˆ ) + ( h − hˆ ) ]2

Confidence error:
S2

B∑∑ I [Cˆ log(C ) + (1 − Cˆ ) log(1 − C )] −objij

=i 0=j 0

S2

i

j

i

j

i

j

i

j

(6)

B

λnoobj ∑∑ I ijnoobj [Cˆ i j log(Ci j ) + (1 − Cˆ i j ) log(1 − Ci j )]Table 1. Evaluation of test set experiment results
TargetPrecision
RecallAP
category=i 0=j 0

3. Experimental results and analysis before
improving the model

plane

0.64

0.93

83.83%

ship

0.71

0.43

40.02%

3.1. Experimental process
The target detection process is as follows:(1) The experimental data set is preprocessed, and thelarge-scale remote sensing image is clipped;(2) Determine whether the remote sensing image is asingle channel image, and select the appropriate
defogging processing method;(3) Remove the objects that are not interested in the
remote sensing data set, and keep the whole data set only
contain the two kinds of objects of interest in this paper;(4) The processed remote sensing data are input into the
network model for training, and the training model isobtained;(5) After training, the target is detected and classified.

Fig. 4. Test result display diagram

According to the above table 1, the AP value of aircraft is
83.83%, the AP value of ship is 40.02%, and the mAP of
two types is 61.93%. From the recall rate, we can see thatthe recall value for ships is smaller than for aircraft, but
the detection accuracy for these two kinds of targets is
not high. From the above table, it is not difficult to seethat the detection effect is better for large targets such as
aircraft, but poor for targets with small ship scale.

3.2. Analysis of experimental results
The changes of loss value and val_loss value during
training are shown in Figure 3 below:

4. Experimental results and analysis after
improving the model(a)

In this paper, we consider the scale problem to increase
the size of the feature map to retain more semantic
information on the resolution. By increasing the scale ofthe input image, more semantic information can beretained. In this paper, the size of the input image ismodified to 608 × 608 and 672 × 672.
The prior frame was originally proposed because of
Faster R-CNN9. According to the size of the ground_truth
mark box in the training set, the width and height of thefrequently appeared label box are counted, and thesemark boxes are taken as the prior box. In order to makethe prior frame suitable for the remote sensing data set
used in this experiment, therefore, K-means clustering
operations are carried out on the dataset for several times.

(b)Fig. 3. Changes in loss during training

According to the change of loss value in Figure 3 (a), it
can be seen that the loss value changes greatly in the firststage, and gradually decreases and tends to be stable inthe second stage. According to the change of loss ofverification set in Figure 3 (b), it can be seen that thechange of loss value tends to decrease during the training
process.
After training, you will get trained_weights_final.h5 fileis used to test the test set, set the IOU value to 0.5, and© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

544

Lianchen Zhao, Yizhun Peng, Di Li, Yuheng Zhang

We use ten times clustering to get ten groups of priorframes, and get the most suitable prior frame by means of
average value. As shown in Table 2 below:
Table 2. A priori box improvement
Characteristic
13*13
26*26
map
Perception
large
mediumvisionInitial prior(116x90)(30x61)box(156x198)(62x45)(373x326)(59x119)(10x13)(16x30)(33x23)Clustering
mean priorframe(14x27)(21x23)(24x23)(49x37)(65x67)(125x123)(31x40)(35x26)(43x30)As shown in Figure 6, the change trend of the loss value
in the network training process is basically consistent.
Compared with the input image 608 × 608, the change ofthe loss value of the verification set with the input image
scale of 672 × 672 fluctuates, but the overall trend is thatthe loss value is gradually reduced.
Combined with the change of prior box, the model ischanged to input scale of 672 × 672 and 608 × 608. The
number of layers is changed from 52 layers to 56 layers
due to the addition of 1 to the layers repeated 8 times inconvolution layer. With the increase of input scale andthe deepening of network layers, not only more semantic
information is retained, but also more semantic
information can be extracted in feature extraction.

52*52
small

For IOU, IOU is the intersection ratio of detection frame
and real frame. In case of non-intersecting border, thegradient will become 0 and cannot be optimized. But for
GIOU, this situation can be avoided, so GIOU will be
used instead of IOU in this paper. The formula is asfollows (7):C \ ( A ∪ B)
(7)
GIOU
= IOU −C

Table 3. Comparison of improved network experiment resultsmAP
network model
Ship AP
Plane APmAP
Yolov338.60%
86.77%
61.93%
Yolov3_608
42.85%
86.48%
64.69%
Yolov3_672
40.07%
88.94%
64.84%
Yolov3_608_GIOU
40.02%
83.83%
61.93%
Yolov338.60%
86.77%
61.93%
Table 4. The improved network Precision and Recall
comparison
network
Ship
Ship
Plane
Plane
model
Precision Recall Precision Recall
Yolov3
0.814
0.411
0.85
0.92
Yolov3_608
0.82
0.448
0.865
0.918
Yolov3_672
0.806
0.433
0.838
0.939
Yolov3_608_GIOU
0.711
0.438
0.648
0.935

The loss function of GIOU is shown in formula (8)：LGIOU = 1 − GIOU

(8)

The evaluation index of the improved model is the sameas the evaluation index in the previous section. After the
improvement of the model, the loss value of the training
set constructed by the partial data of NWPU VHR-10
data set and DOTA-v1.5 data set is constantly decreasing.

The Yolov3_608_GIOU and Yolov3_672_GIOU
versions in the above table refer to the replacement of
IOU with GIOU based on the Yolov3_608 andYolov3_672 versions. Due to the poor detection effect ofYolov3 on small targets, the recall rate is not very ideal.
It can be seen from the above table that the recall rate ofships with more small targets has been improved inseveral improved versions, and the precision will not begreatly reduced. According to the analysis of the mAPvalue, only the mAP value of the improved network
Yolov3_608_GIOU is lower than that of the initial
network by 0.75%, and the detection effect of the
improved network Yolov3_672_GIOU is the best, andthe map value is 3.26% higher than that of the initial
network.

(a)(b)
Fig. 5. Change of loss value of input image scale 608×608 after
network improvement(a)(b)
Fig. 6. Change of loss value of input image scale 672×672after
network improvement© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

545

Target Detection In Remote

5. Conclusion
This chapter first introduces the characteristic pyramid
and the loss function of Yolov3, and then optimizes the
Yolov3 network. Firstly, K-means clustering method isused to cluster according to the training set, and then the
average value is used as the adjustment of prior frame. In
order to retain more semantic information, adjust thescale of input data, extract more semantic informationand increase the depth of the network, Yolov3 is
optimized according to the data set in this paper. Finally,
IOU is replaced by GIOU. Compared with several
improved networks, the detection effect of the improved
network on the test set is improved. Finally, the detection
effect of the improved network Yolov3_672_GIOU is thebest, and the mAP value has been significantly improved.
References
1.

2.

3.

4.
5.

6.

7.

8.
9.

Redmon J, Divvala S, Girshick R, et al. You Only Look
Once: Unified, Real-Time Object Detection [M]. 2016
IEEE Conference on Computer Vision and Pattern
Recognition. 2016:779-88.
Felzenszwalb P F, Girshick R B, Mcallester D, et al.
ObjectDetection
withDiscriminativelyTrained
Part-Based Models[J]. IEEE Transactions on Software
Engineering, 2010, 32(9):1627-1645.
Chen L-C, Zhu Y, Papandreou G, et al. Encoder-decoderwith atrous separable convolutionfor semantic image
segmentation[C]. Proceedings of the European
Conference on ComputerVision (ECCV). 2018: 801 –818.
Liu Chang, Wang Pengjun, Zhang Meiling, et al Researchon sparse video detection technology based on IOU
analysis [J]. High technology letters, 2019(10).
Zhang R, Yao J, Zhang K, et al. S-cnn-based ship detection
from high-resolution remote sensing images. [J].
International Archives of the Photogrammetry, Remote
Sensing & Spatial Information Sciences, 2016, 41.
Walk S, Majer N, Schindler K, et al. New features andinsights for pedestrian detection[C].IEEE Conference on
Computer Vision and Pattern Recognition, 2010:
1030-1037.
Ok A O, Başeski E. Circular oil tank detection frompanchromatic satellite images: a new automatedapproach[J]. IEEE Geoscience and Remote Sensing
Letters, 2015, 12(6): 1347 –1351.
Li Weijun. Overview of K-means clustering algorithm [J].
Modern computer (Professional Edition), 2014 (8): 31-32
M. Kang, X. Leng, Z. Lin, et al. A modified faster R-CNN
based on CFAR algorithm for SAR ship detection[C].
2017 International Workshop on Remote Sensing withIntelligent Processing (RSIP), Shanghai, China, 2017, 1-4.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

546

第1期

·1·

A design of home smart nursing robot based on Raspberry Pi
Yuqi Zhao, Yizhun Peng*, Jie Liu, Xiaowei Wu, Jikai Zhao
College of Electronic Information and Automation,
Tianjin University of Science and Technology, Tianjin, 300222, China;

E-mail: *pengyizhun@tust.edu.cn
www.tust.edu.cn

Abstract
This product is based on Bluetooth, WIFI and other wireless technologies, with the Raspberry Pi 4 as the core, tomeet the health, safety and entertainment needs of the serviced users, mainly the elderly, while assisting the leisureand entertainment of users when traveling outdoors. A multi-functional escort robot designed for safety and otherissues. It has many methods such as temperature and humidity monitoring, harmful gas monitoring, and noisemonitoring to ensure the safety of users. At the same time, it also has TV projection, audio-visual entertainment
touch screen interaction, mobile phone projection and other entertainment functions.
Keywords: Pension; artificial intelligence; entertainment; convenience; accompany; Internet of things; big data

1.Introduction

2.System overall design ideas
2.1 Overall functional framework
The system includes three components: an indoor body, aphysical sign monitoring bracelet, and an outdoor body.
It combines Bluetooth communication technology,
network communication technology, Internet of Things
technology, GPS positioning technology, and intelligent
path planning technology. Among the three components,
the physical sign monitoring bracelet can monitor theelderly’s blood oxygen, pulse, heart rate and otherphysical sign data and GPS location in real time2, andrealize two-way data exchange through Bluetooth andembedded systems. Through the wireless data network,indoor and outdoor body networking can be realized, andnetwork entertainment resources can be obtained. The
outdoor subject can integrate the camera, sensor, GPScoordinates of the bracelet and other information torealize intelligent path planning.
When the user is indoors, the physical sign monitoringbracelet will send the user’s physical sign information to
the indoor subject via Bluetooth, and calculate the user’s
health information after processing by the host system
algorithm. When the system determines that the user’s
physical signs are abnormal, it will send the user’s

With the rapid increase of the elderly population, theissue of elderly care has become increasingly prominent,
especially for the elderly in empty nests. It has become
the norm for the elderly to live alone at home while theirchildren are at work. At the same time, in recent years,
the nursing industry such as "monthly sister" has not
only experienced frequent problems but also increased
industry prices. The home intelligent nursing robot is
specially designed for the elderly. It has multiple
functions such as health monitoring, smart medicine
cabinet,housekeeping
services,
communicationentertainment, abnormal situation alarm, critical situation
handling, and action interaction. It is very suitable for theelderly living alone1.
This home care intelligent robot called the spouse, justlike its name, is an intelligent robot specially designed
and developed for the elderly living alone. In order torealize the function of accompany the elderly indoors
and outdoors, it is based on wireless technologies such as
Bluetooth and WIFI. To meet the entertainment needs
and health and safety monitoring of the elderly, it canalso solve the problems of the elderly in travel, such asloading, rest, health, etc., in all aspects.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

547

Yuqi Zhao, Yizhun Peng, Jie Liu, Xiaowei Wu, Jikai Zhao

physical signs to the settings through the network The
first contact sends a warning carefully and calls 120
when necessary. At the same time, the indoor main bodycan remind the user to take medicine on time to preventthe elderly from forgetting. The indoor main body canalso monitor whether the living environment is safe andissue an alarm in real time while working, so as to avoid
the loss of life and property caused by accidents such as
fire and gas leakage. The double safety protection of theuser's body and living environment is realized; the indoor
host is equipped with a large LCD screen speciallydesigned for the elderly, which can be used forentertainment by the elderly without excessively
adversely affecting the eyes3.
When the user is outdoors, the physical sign monitoringand GPS system work as usual, and the outdoor subject
will follow the user, and all the user needs to do is act as
usual. The outdoor part is box-shaped with storage spaceinside, and users can use it as a mobile basket. In
addition, the box can be used as a seat for rest after
fatigue. The outdoor part serves as a self-moving storage
space, a movable seat, and can also achieve autonomous
climbing of stairs through wheels with a peculiar
mechanical structure. It greatly facilitates the lives of
elderly people with limited mobility.

Blood pressure and heart rate monitoring: The
photoelectric sensor can collect the pulse wave at our
wrist. By analyzing the pulse wave's rising slope andband time and other characteristic parameters, a specificcalculation formula can be obtained to estimate the bloodpressure value. For example, when the blood pressure ishigh, the slope of the rising band of the ejection period
will increase. This is also the method adopted by mostbracelets on the market, because it is simple andconvenient, and will not cause trouble to users4.
At the same time, the heart rate can be calculated througha certain algorithm. Nowadays, heart rate bracelets
basically use light-sensing heart rate, which is tested by
the principle of light reflection. Generally, they areaffected by ambient light. Some products do not haveheart rate data when facing strong light or completely
black. Generally, green (OSRAM) is used with aphotosensitive photodiode to detect the flow of blood
flowing through the wrist at any time. When the user's
heart beats, more blood will flow through the user's wrist,
and the greater the amount of green light absorbed.
Between the beating of the heart, blood flow is reduced,
resulting in a reduction in the absorption of green light.
The heart rate signal is effectively monitored andextracted through complex and sophisticated algorithms,
most of which use the original NXP heart rate.
Based on the secondary development of the ready-madebracelet, integrated GPS module, easy to coordinate withthe "wife" to complete positioning and tracking, built-in
heartbeat detection, body temperature detection module,
based on the Internet connection with the Raspberry Pi,
collect the physical signs of the elderly at all times, by
the Raspberry Pi Perform data review, and automaticallysend alarm and first aid information to children's mobile
phones when physical signs of dangerous data appear5.

Fig.1 System Block Diagram]

2.2 Vital Sign Monitoring Bracelet Design Scheme
Body temperature detection: Because it is a bracelet tomeasure temperature, and according to the accuracy
requirements, the method of contact temperature
measurement is used to measure body temperature. The
subsequent influence of the external temperature and thedifference between the temperature of the wrist and theunderarm temperature are calculated and compensated.
So as to get a reliable body temperature. The LMT70
temperature sensor can be used as a low-power, highprecision sensor for temperature acquisition, andfeedback to the microprocessor through analog-to-digitalconversion. Further calculations are carried out by fittingcompensation curves and calculation formulas of outdoor
temperature, armpit temperature and wrist temperature indifferent environments

Fig.2 Bracelet diagram

2.3 Interior design plan
The product integrates the HR202 humidity detection© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

548

A design of homemodule and the DS18B20 temperature measurement
module. By connecting the data to the core processor, it
forms an important part of the indoor monitoring andalarm system.
The gas-sensitive material used in the MQ135 gas sensor
is tin dioxide (SnO2) with low conductivity in clean air.
When there is pollutant gas in the environment where thesensor is located, the conductivity of the sensor increases
with the increase in the concentration of pollutant gas in
the air. Using a simple circuit, the change in conductivity
can be converted into an output signal corresponding to
the gas concentration. The MQ135 sensor has highsensitivity to ammonia, sulfide, and benzene vapors, andis ideal for detecting smoke and other harmful gases.
The MAX9814 high-sensitivity microphone sensor
module integrated on the product can collect the sound in
the environment and send it to the core processor, andthen identify the abnormal sound in it, and then canjudge the outdoor noise or the abnormal sound in theroom (water leak, air leak, people Lock picking, etc.) andfurther processing as appropriate.
A unique sensor, the thermal imaging module IFD-X, isinstalled on the product. It monitors the temperaturechanges in the body of the elderly in real time andtransmits the data to the Raspberry Pi for review. If data
fluctuations are found to exceed the warning line,
warning and detection information can be issued For
children, on the other hand, it solves the physicalchanges that cannot be monitored by the bracelet, detects
emergency situations one step in advance, and adds astronger protection to the lives of the elderly.
IFD-X thermal imaging module, Red Eye Camera(hereinafter referred to as "REC" or "IFD-x" or"equipment") is a non-contact thermal imaging
instrument based on infrared array high-precision
temperature sensors and advanced software algorithms.
Infrared imaging of any object within the field, theimaging resolution is up to 512*384 pixels, thetemperature sensitivity is 0.1℃, the absolute accuracy is±1.5℃, and the refresh frequency is up to 64Hz. Comes
with storage and real-time clock, with real-time data
output display, photo storage function, digital interface
including UART and USB, can be directly connected tothe computer and mobile phone, with the upper computer
software or mobile phone APP program, it is very
convenient to use6.
This product is connected with a large-screen TV
projection module, a touch screen interactive module, amobile phone projection module, etc., which can meetvarious entertainment activities and external contacts inthe daily life of the elderly. The details will not be
repeated.

Fig.3 Conceptual drawing of indoor part2.4 Outdoor part design scheme
This product has the ability to exercise autonomously,
and a large-capacity cabinet is reserved in the design,
which can help the elderly transport vegetables when theuser is out shopping, reducing the burden on the user andfreeing hands. The product comes with a buffer moduleand a shock-proof module. Even if fragile dishes (such aseggs) are placed in the product, it will not break undermost roads in the city. At the same time, the product isequipped with an auxiliary device for climbing stairs,
which can assist users to climb stairs and reduce thedifficulty of users carrying dishes upstairs. The materials
are all made of high-quality compressive materials, andthe design has a load-bearing structure. The maximum
load-bearing capacity is 100kg. The user can sit directlyon the product and rest. At the same time, the lid andhandle are designed according to ergonomics. Even if
sitting for a long time, there will be no back pain . At thesame time, it will not exert any pressure on the innerbasket. At the same time, the product also has a radio
function for users to listen to and relax in their leisuretime7.
The content of intelligent path planning to achieveprecise follow is as follows: The positioning method of
our core design is based on visual positioning and sensor
positioning, each has its advantages and disadvantages.
The advantages of sensor positioning are:
1. Able to find the x.y, z coordinates of the target
2. It can be positioned in 360 degrees
3. The positioning target is less affected by
obstacles
Based on visual realization, the method is richer, and avariety of technology combinations can be used. For
example, based on the depth camera to directly recognize
the combination of human bones + face recognition +feature mark recognition. Atlas robots like Boston© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

549

Yuqi Zhao, Yizhun Peng, Jie Liu, Xiaowei Wu, Jikai Zhao

Dynamics are based on visual recognition to achieve
walking and carrying objects. The advantages of visual
recognition targets are: 1. Not only can the x.y.z
coordinates be calculated, but also the 3-dimensional
deflection angle of the object relative to the camera. Can
get richer decision-making information. 2. The visualunit can be used not only to identify targets, but also to
identify most obstacles. In summary, we adopt acomprehensive positioning method of vision and sensing
to obtain the best positioning effect. The sensor can be an
infrared thermal imaging sensor, and the GPS function of
the bracelet is used for positioning to determine thelocation of the owner. At the same time, three different
positions on the product are installed with triangular
8266 modules. If the user chooses to use the mobile
phone on his body, connect the three modules on thesuitcase to this hotspot, and continuously obtain thecurrent rssi value, and use this value to judge the distance
between this module and the mobile phone. The threemodules are sufficient to judge the mobile phone ’ s
Direction, so as to achieve the purpose of following theuser more accurately.
After the human body is recognized, the next step isgenerally to think of how to recognize the obstacle.
Obstacle recognition, commonly used technologies
include depth camera recognition, ultrasonic ranging,and infrared ranging. The advantages of depth camera
and infrared ranging are that they are cheap and fast, butthey cannot identify glass and black objects. Ultrasonicdistance measurement can be used as a supplement. If
the cost and size are not limited, lidar and millimeter
multi-radar can also be considered. We use depth camera
recognition, ultrasonic ranging, and infrared ranging to
avoid obstacles.
Dynamic path planning, this product uses to build a 3D
spatial map and rasterize the map into small grids thatcan or cannot be passed. With the aid of path planningalgorithm, and then follow the target. The difficulty of
the whole scheme is that the targets and robots are
dynamic and need to be adjusted continuously to achieve
optimal path planning. The camera is a more importantpart of the robot, this product uses the openmv highdefinition camera.

Fig.4 Concept map of outdoor part

3.System software design
3.1Software design of the interior partThe software design of the indoor part includes threeparts: the communication between the embedded device
and the mobile phone client, the driving reading of theembedded device to various sensors8, and thecommunication between the embedded device and thephysical sign monitoring bracelet.
1.Drive readings of various sensors by embedded
devices:
The core processor of the embedded system adopts the
stm32f103 series single-chip microcomputer of STM
Company to realize the driving and reading of the sensor.
The initialization of each sensor needs to be configuredby the microcontroller according to different working
principles. Sensor reading generally includes ADC
reading and serial communication (including SPI, IIC,etc.).
2.The communication between the embedded
device and the physical sign monitoring bracelet: senddata via Bluetooth. Bluetooth HC05 is a master-slave
Bluetooth serial port module. When the Bluetooth device
is successfully paired and connected, the internal
communication protocol of the Bluetooth can be ignored
and the Bluetooth can be used as a serial port. When aconnection is established, two devices share a channel,
that is, the same serial port. One device sends data to thechannel, and the other device can receive the data in thechannel. hc-05 supports standard baud rates from4800bps to 1382400bps. The baud rate must be matchedwhen using it. Connect vcc to 3.3 or 5V GND to GND
TX to RX on the microcontroller and RX to TX.
3.Feedback the user's health information to themobile phone:© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

550

A design of home2) MAX30100 is a sensor that can read heart rate andblood oxygen. The communication method is through
IIC. Its working principle is to obtain the ADC value ofthe heart rate through infrared LED light irradiation. It isa non-invasive integrated pulse oximetry and heart ratemonitoring sensor solution9.
3.3 Outdoor software design
There are many commonly used computer vision
solutions, such as binocular vision, TOF-based depth
cameras, and structured light-based depth cameras. The
outdoor part of our current intelligent escort robot mainly
works in an outdoor environment. The active light source
will be greatly affected by conditions such as sunlight.
Therefore, the passive vision solution of binocular vision
is more suitable. Therefore, the vision solution we adopt
is based on dual vision. Visual.

Fig.6 Binocular vision example image

The distance measurement of binocular vision isessentially a triangular distance measurement method.
Because the positions of the two cameras are different,
just like our two eyes, we see different objects. The same
point P seen by the two cameras will have different pixel
positions during imaging. At this time, the distance of
this point can be measured by triangulation. Different
from the structured light method, the points calculated by
the structured light are actively emitted and known anddetermined, while the points calculated by the binocularalgorithm are generally image features captured by thealgorithm, such as SIFT or SURF features. The feature is
calculated as a sparse image. In order to avoid obstacles
well, we need to obtain a dense point cloud image andthe depth information of the entire scene. This depth
information can help us find walkable areas andobstacles in the map scene. The whole output is similar
to the 3D point cloud image output by Lidar.

Fig.5 Feedback health information

3.2Sign detection software design
1) Calculate the heart rate through a certain algorithm.
The heart rate bracelet basically uses light-sensing heartrate. It is tested by the principle of light reflection. It is
generally affected by ambient light. Some products are
not exposed to strong light or all black. Heart rate
bracelets with heart rate data are generally green(OSRAM) with a photosensitive photodiode to detect theflow of blood flowing through the wrist at any time.
When the user's heart beats, more blood will flow
through the user's wrist, and the greater the amount ofgreen light absorbed. Between the beating of the heart,
blood flow is reduced, resulting in a reduction in theabsorption of green light. The heart rate signal iseffectively monitored and extracted through complex andsophisticated algorithms, and the original phase heart
rate of NXP is used.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

551

Yuqi Zhao, Yizhun Peng, Jie Liu, Xiaowei Wu, Jikai Zhao

factors. You can look at this icon for details:

Fig.7 Actual output depth map, different colors represent
different distances

At the same time, we use the vector field histogram(VFH) algorithm to achieve accurate obstacle avoidance
and follow: during its execution, a local map based onpolar coordinates is created for the current surrounding
environment of the mobile robot. This local uses the grid
map representation method , Will be updated by somerecent sensor data. The polar coordinate histogram
generated by the VFH algorithm is shown in the figure:Fig.9 Influencing factors of cost function

4.Concluding remarks
Based on the serious aging problem in our country, theproblem of elderly care has become increasingly
prominent. The research and development of this robot
fills the gap in the market, and to a certain extent
alleviates the inconvenience of living alone for elderly
people. It is a "wife" invented based on the designconcept of serving the society and solving practicalsocial problems. Innovative products of the times that are
truly designed for the elderly. It is of great significanceto solve social problems such as the difficulty ofproviding for the elderly in an aging society, thegenerally low quality of life of the elderly, and theunattended elderly in empty nests. With its
comprehensive and humanized functions as its
advantages, the home care intelligent escort robot will
surely be recognized and supported by the society.

Fig.8 VFH algorithm polar coordinate histogram

In the figure, the x-axis is the angle of the obstacle
perceived by the robot as the center, and the y-axis
indicates the probability p of the obstacle in this direction.
In the actual application process, the histogram will first
identify all gaps that are large enough to allow the robot
to pass, and then calculate the cost function for all thesegaps, and finally select the path with the lowest costfunction to pass.
The cost function is affected by three factors: the target
direction, the current direction of the robot, and thepreviously selected direction. The final generated cost is
the weighted value of these three factors. The robot's
selection preference can be adjusted by adjusting
different weights. The VFH algorithm also has otherextensions and improvements. For example, in the VFH+
algorithm, the limitations of robot kinematics are
considered. Due to the difference in the actual underlying
motion structure, the actual motion capability of themachine is limited. For example, the structure of a carcannot be turned in place at will. The VFH+ algorithm
considers the blocking effect of obstacles on the
trajectory of the robot's actual movement ability, andshields those trajectories that are not occupied by
obstacles but cannot be actually reached because of theirblocking. Our E-patrol robot uses a two-wheel
differential drive motion form, which is very flexible,
and its practical application is less affected by theseAcknowledgements
This research was supported by Student’s Platform for
Innovation and Entrepreneur Training Program, theMinistry of education of China(202010057018 ).
References
1. Zhang Wenhui, Qi Naiming, Yin Hongliang. Adaptive
neural variable structure robot trajectory tracking control[J]. Control and Decision. 2011 (04)2. Yu Ting. Research on multi-robot queue curve movement[D]. Shanghai Jiaotong University 2009
3. Chen Canfeng. The principle and application of Bluetooth
low energy technology [M]. Beijing: Beihang University
Press, 2013.
4. Shen Leixian, Pang Jiayi, Zhang Qingyun, Xu He. Design
and implementation of positioning service platform basedon Beidou navigation[J]. Computer Technology andDevelopment, 2019, (01): 1-4.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

552

A design of home

5. Chen Kaixuan, Xie Haitao. GPRS principle and its
application[J]. Railway Communication Signal, 2003,
39(7): 7-8.
6. Wu Guojing, etc. Microcontroller Application Technology
China Electric Power Press, 2004.1
7. Robot path planning Ren Chunmingdi, Zhang Jianxun1 (1.
School of Information Technology and Science, Nankai
University, Tianjin 300071; 2. Department of Information
Science and Technology, Tianjin University of Financeand Economics, Tianjin 300222)
8. Human Health Index-Ling Yisheng [2] Wang Yongyan,
Sun Yiwen
9. Wu Xiaofang. New Thoughts on Internet of Things and
Big Data[J]. Communication World, 2017, (01): 12.[2017-08-31].

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

553

Design and Simulation of Indoor Tour Guide Robot Based on ROS
Yuheng Zhang 1,2, Yizhun Peng 1,2*, Lianchen zhao1,2

College of Electronic Information and Automation,
Tianjin University of Science and Technology,China
2
Advanced Structural Integrity International Joint Research Centre,
Tianjin University of Science and Technology, China
E-mail: * pengyizhun@tust.edu.cn
www.tust.edu.cn
1

Abstract
Aiming at the problems of the small number of lecturers in museums and other venues and the large demand foraudience consultation, a ros-based indoor guide robot was designed. The robot consists of a mechanical system, amotion control system, and a sensor system. In order to improve the efficiency of development and debugging, andreduce the cost of experiments, it is necessary to test related algorithms in a virtual simulation environment before
the robot enters the actual working state. Experiments have proved that the robot can autonomously guide guests tothe destination and explain according to a preset path; the robot has multiple sensors to sense obstacles, and canautonomously avoid obstacles during the explanation and continue to move; the robot can accurately and efficiently
recognize faces and provide accurate services Fast.
Keywords: Indoor tour guide robot; Autonomously guide; Voice interaction; Face recognition

can interact with the robot at any time if they havequestions or problems.
(iii) Face recognition: The robot takes pictures of theuser who is guided for the first time to collect public
information, continuously enriching the public's personalportraits, establishing more complete personal informationmaterials, and providing accurate and fast services.

1. Introduction
With the rapid development of intelligent technology,
the quality of people's lives continues to improve. Since
the existing "guide" intelligent robots can only realize thefunction of navigating the road when in use, the function
is relatively single, and the existing "guide" intelligent
robots cannot guide tourists and cannot achieve the effectof reminding.1 Aiming at the problems of museums andother venues such as the lack of interpreters and the large
demand for audience consultation, a ros-based indoor tour
guide robot was designed and simulated in gazebo
software. The robot mainly realizes the following 3
functions:(i) Welcome explanation: The robot can guide the guests
to the destination and give an explanation according to apreset path. After the explanation, the robot willautomatically return to the original starting position;(ii) Audio guide: The robot can interact with people byvoice. During the robot's explanation process, the audience

2. The Composition of the Robot
The robot consists of four layers. The first layer is the
chassis movement layer, including two motors, two
driving wheels, one universal wheel, and three infrared
sensors; the second layer is a lidar layer with a lidar; thethird layer It is the driving layer, including the lower
computer, battery, three ultrasonic sensors, a smoke sensor,
and a temperature and humidity sensor; the fourth layer isthe decision-making computing layer, including the upper
computer, monitor, camera, microphone and speaker. The
schematic diagram of the robot is shown in Figure 1.The
upper computer is an industrial computer with high anti-© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

554

Yuheng Zhang, Yizhun Peng, Lianchen zhao

3. Robot Simulation Test

magnetic, dust-proof and impact-proof capabilities.2 Thereis a special power supply in the chassis, and the power
supply has strong anti-interference ability. The processor
is an inter i7 processor with four usb3.0 interfaces And
four usb2.0 interfaces, two network ports. The lower
computer controller mainly realizes the bottom-levelmotor drive, power management and acquisition of various
sensors such as ultrasonic and infrared, and at the sametime feeds back the collected data to the upper computer in
time.
The
lower
computer
controller
uses
STMicroelectronics' STM32F103ZET6 chip as the main
control, the chip flash has 512K, the SRAM is 64KB, andthe pins are 144.3 The chip runs the FreeRTOS system. The
infrared sensor can realize the function of assisting
obstacle avoidance, preventing the robot from falling when
it reaches the road pit.4 Lidar is used to realize the range
finding scan and generate the plane point cloud map
information of the space. These cloud map information canbe used in practical applications such as map surveying,robot positioning and navigation, and object/environment
modeling. The block diagram of the robot system is shownin Figure 2.

3.1. Construction of simulation environment
The simulation environment of the robot is built on thegazebo simulation platform, and the walls are drawn by thebuilding editor tool provided by gazebo. We built theindoor scene of a simulated exhibition hall, including
venues in 16 Chinese cities such as Beijing and Tianjin.
Each venue is 12 meters long and 4 meters wide. The robot
model is built and exported by solidworks software. The
initial position of the robot is in the center of the hall of theexhibition hall. The simulation environment of the robot isshown in Figure 3.

Fig.3.

Robot simulation environment

3.2. Mapping and navigation
Fig.1.

The schematic diagram of the robot

Fig.2.

The block diagram of the robot system

Only after the establishment of the map is completed
and the map is obtained, the robot can be controlled tonavigate. SLAM is simultaneous localization and mapping,which can be described as: the robot starts to move froman unknown position in an unknown environment.5 During
the movement, it locates itself according to the position
estimation and the map, and at the same time builds anincremental map to realize the robot's autonomous
positioning and navigation.6 This design uses the mostcommonly used and mature gmapping algorithm in SLAM,
gmapping integrates the Rao-Blackwellized particle filter© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

555

Design and Simulation ofcomplete the autonomous positioning function, that is, therobot can calculate its position on the map in any state.7
Perform navigation simulation in Rviz, as shown in the
figure below, where the red arrow is the specified target
position, and the green line between the red arrow and therobot is the global path planned by the navigation function
package.

algorithm, the following is the overall framework of
gmapping and navigation.

Fig.4.

The overall framework of gmapping and navigation

The gmapping function package subscribes to therobot's depth information, IMU information and odometry
information, and at the same time completes theconfiguration of some necessary parameters, you cancreate and output a probability-based two-dimensional
grid map. The key to navigation is robot positioning andpath planning. The navigation function package needs tocollect the sensor information of the robot to achieve theeffect of real-time obstacle avoidance. It also needs theodometry information of the robot and the corresponding
TF transformation. Finally, the navigation function
package outputs instructions that can control the
movement of the robot.
Use the keyboard node to control the robot to move in
the Gazebo environment. After the map is created, saveit.This map will be called during the navigation process.
Use the move_base function package for global path
planning and local real-time planning. According to thegiven target location and the constructed global map, theA* algorithm is used to plan the global path, and the
optimal route from the robot to the target location is
calculated as the global route. In practice, obstacles may
appear at any time, so local real-time planning must be
carried out to make it conform to the global optimal pathas much as possible. Use the Dynamic Window
Approaches algorithm to search for multiple paths to avoid
and travel, and select the optimal path according to theevaluation index. Use the amcl function package to

Fig.5.

Robot navigation simulation

3.3. Welcome explanation
The robot can guide the guests to the destination andgive an explanation according to a preset path. After the
explanation, the robot will automatically return to theoriginal starting position. When building a map, record thelocation of each venue as a navigation target point.
Move_base implements path planning through action, andaction uses a client/server architecture. Use thesend_goal() interface to send to the server of move_base.
After receiving the target task, the server will realize thepath planning function according to the current pose of therobot, and output the speed command to control the
movement of the robot. Go to the target location and returnthe result to the client. If the result is successful, it will
send the introduction of the corresponding venue to the
speech synthesis module, and let the robot speak the
introduction. This function module is based on the
tts_sample program in the iFLYTEK SDK, modify themain code file, add the ros interface, subscribe to theintroduction topic, receive the input voice string, and use
the SDK interface in the callback function to convert thestring into Voice, after the voice broadcast is completed,
the robot will navigate to the next target point and continue
to broadcast until the last target point is finished.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

556

Yuheng Zhang, Yizhun Peng, Lianchen zhao

3.4. Audio guideThis function is based on the speech recognition SDK
provided by IFLYTEK and is constructed using the ROS
system. The iat_record_sample program in IFLYTEKSDK is modified and the required ROS interface is added.
This design adds a publisher, a subscriber and a
move_base client. The Subscriber is used to receive thevoice wake-up signal. After receiving the wake-up signal,
the wakeupFlag variable is set, and then the voice dictation
function of the SDK is called in the main loop to recognize
Human voice information, set resultFlag variable after
successful recognition. Then use dataString.find() todetermine the meaning of Chinese voice input, and publish
the corresponding target point pose through thesend_goal() interface. After receiving the target task, theserver of move_base will realize global and local Pathplanning function according to the current pose of therobot, output speed command to control the robot to moveto the target position, and return the result to the client. If
the result is successful, it will send the introduction of thecorresponding venue to the speech synthesis module, andlet the robot speak the introduction.

Fig.6.

Face detection flowchart

3.5. Target face recognition
The specific process of face detection is shown in Figure
6. First, load the face detection classifier that comes with
opencv. In this article, the classifier based on haar feature
values is used. In this article, we mainly use the contour ofthe face and human eye recognition to realize facedetection. The next step is to preprocess the image to berecognized, to grayscale the image, obtain its histogram,and then perform equalization. This step is mainly to makethe feature value of the face in the image better extracted
come out. Calling the detectMultiscal function torecognize the picture is mainly because the input picture isof different sizes, so it needs to be detected in multiple
dimensions.8 By calling the rectangle function, draw therectangular frame of the face profile recognized by the
program. Finally, the drawn rectangular frame of the faceis cut out in the original image.

Fig.7.

Face detection

To let the program know who to recognize, it isnecessary to train the face model. Firstly, the target image
is preprocessed, and the sample is normalized, mainly forthe same size of the image, and then grayscale. Next, label
the sample, this is to tell the training model which number
belongs to whose face. Then start training, mainly using
three
methods,
namelyEigenFaceRecognizer,
FisherFaceRecognizer and LBPHFaceRecognizer. The
EigenFaceRecognizer method is also face recognition
based on PCA transformation. The principle of PCA
transformation is a way to reduce the dimensionality of animage, because in face recognition, the image is generally
processed as a vector, but at the same time, thedimensionality of the vector is too large, and the hugedimensionality is quite difficult for subsequent image
calculations. Therefore, it is necessary to reduce the image
dimension without losing important information as much

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

557

Design and Simulation ofReferences

as possible. The FisherFaceRecognizer method is a facerecognition
basedon
Fishertransform.9
LBPHFaceRecognizer is a face recognition method basedon local binary patterns. After training, save the training
model locally, and then you can start face recognition.

Fig.8.

1. Jianfa Dai, Jingrong Ying, Hongtao Liu. A "Guide"
Intelligent Robot Design, Science Technology andInnovation, 2020(11): 142-143.
2. Wenbin Ni, Guanghua Lu, Yi Zhang. A guide service robot
design , Machinery Manufacturing and Automation, 2016,
45(06): 176-179.
3. Jing He. Design and production of campus tour guide robot,
Electronic Testing, 2018(11): 76-77.
4. Chenxi Guan. Design and research of indoor inspection
robot system based on ROS, Electromechanical Information,
2020(30): 128-129.
5. Cadena C.,Carlone L.,Carrillo H.,et al.Past,Present,and
Future of Simultaneous Localization and Mapping:Toward
the Robust-Perception Age, IEEE Transactions on
Robotics,2016,32(6):1309-1332.
6. Rowekamper J.,Sprunk C.,Tipaldi G D.,et al.On the position
accuracy of mobile robot localization based on particle
filters combined with scan matching, Intelligent robots andsystems,2012:3158-3164.
7. Yanbin G.,Shifei L.,Mohamed A.,et al.INS/GPS/LiDAR
Integrated Navigation System for Urban and Indoor
Environments Using Hybrid Scan Matching Algorithm,
Sensors,2015,15(9):23286-23302.
8. Honglong Chen, Dongyong Liu, Zhichen Ni,et al.The facerecognition experiment platform of quadrotor drone basedon YOLO, Experimental Technology and Management,
2020,37(10):107-111.
9. REDMON J, DIVVALA S, GIRSHICK R, et al. You only
look once:Unified, real-time object detection, in IEEE
Conference on Computer Vision and Pattern
Recognition(CVPR), Las Vegas, Nevada, USA, 2016:779788.

Face training flowchart

After having the above foundation, the final facerecognition can be performed. First, image preprocessing
is performed, and then the trained classifier is loaded toobtain the label value on the input image, and then thecharacter name is obtained through the label value.

Fig.9.

Target face recognition

4. Conclusion
In view of the small number of lecturers in museums andother venues and the large demand for audience
consultation, the overall robot design was completed, andexperiments were carried out on the gazebo simulation
platform. The test proved that the robot can complete thefunctions of welcome explanation, voice guide, and facerecognition. By interacting with robots, visitors canunderstand the situation of each exhibition area. The
robot's way-guide function can also provide moreconvenience for visitors. This solution can improve theconstruction effectiveness and quality of museums andother venues, as well as a smart and personalized service
system.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

558

Research on Path Planning of Manipulator
Nana Wang 1, Yizhun Peng 1,2*, Zhou Yang 1, Yuheng Zhang 1

College of Electronic Information and Automation,
Tianjin University of Science and Technology, China;
12Advanced Structural Integrity International Joint Research Centre,

Tianjin University of Science and Technology, China;

E-mail: * pengyizhun@tust.edu.cn
www.tust.edu.cn

Abstract

In order to further realize the flexibility, intelligence and human-machine friendliness, the robot musthave the ability of flexible motion planning. This paper mainly introduces the general steps of manipulator
path planning and common path planning algorithms. First, the path planning of the manipulator isreviewed from three aspects: environment modeling, path search, and path smoothing. In addition,compared various manipulator path planning algorithms, such as RRT algorithm, ant colony algorithm,
genetic algorithm, etc.Finally, the development trend of path planning technology is summarized andanalyzed.
Keywords: Manipulator, Path planning, RRT algorithm, Ant Colony Algorithm, Genetic Algorithms

1. Introduction

This paper mainly introduces the path planning of 6-DOF
manipulator.

With the progress of the times, the application of robotic
arm technology is becoming more and more popular. For
the manipulator, reasonably planning its own movement
path is the most basic function to complete the task in thereal world, which is a key technology in robot research path planning. Path planning can be divided into two types:
global path planning and local path planning. Global path
planning refers to path planning based on allenvironmental information through certain algorithms
under the condition that all environmental information is
known. Local path planning refers to the situation wherepart or all of the environmental information is
known.Real-time environmental information is collectedby the sensor and the collected environmental information
is analyzed for real-time path planning1. With the rapid
development of the manipulator, the requirement for thepath planning of the manipulator is getting higher andhigher. Therefore, it is necessary to constantly optimize thepath planning algorithm to save the movement time of themanipulator and reduce the joint loss of the manipulator.

2. Path planning of 6-DOF manipulator
Path planning should first establish an environment modelthat is convenient for computer to carry out path planning.
Then, a walking path that enables the predictedperformance function to obtain the optimal value issearched through a certain algorithm. Finally, the pathsearched through the corresponding algorithm is smoothen
to make it a practical and feasible path2.
2.1. Modeling Kinematics of manipulator
2.1.1.

Modeling

Aiming at the problem of how to describe the geometricparameters and coordinate system direction between theadjacent links of the mechanical arm, Denavit andHartenberg came up with the DH method3, in this method,a coordinate system is fixed on each manipulator link, andthe rotation and translation vectors are described by meansof homogeneous transformation. The transformation© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

559

Nana Wang, Yizhun Peng, Zhou Yang, Yuheng Zhang

2.2. Collision detection

relation from connecting rod i-1 to i is shown in Equation(1). DH method is universally applicable. The joints
between the link of the robot can be either rotating joints
or prismatic joints, and there are no limits on the lengthand plane of the joints.

Collision detection technology is an important part of thepath planning of the manipulator. It is mainly used to judge
whether there is a collision between two objects. Accurate
and effective collision detection method is the premise ofeffective obstacle avoidance path planning for the
manipulator. Generally speaking, space obstacles are
irregular geometric shapes. In order to judge the collision
problem of manipulator conveniently, it is necessary to
regularize the obstacles. Some scholars have proposedusing the regular body envelope of obstacles toapproximate modeling. Although this approximation
expands the obstacle domain, it greatly simplifies thedescription of the obstacle domain and effectively
improves the efficiency of planning7. Common obstacleenvelopment models include: Sphere, AABB, OBB, KDOPS, etc.

T = Rot ( xi −1 , α i −1 )Trans ( xi −1 , α i −1 )i −1iRot ( z , θi )Trans ( z , di ).（1）

2.1.2. Positive solution to the kinematic equation of
the manipulator
Forward kinematics is to find out the spatial position andattitude of the end-effector of the mechanical arm relative
to the base coordinate system according to the Angle valueof each joint of the manipulator and the parameter
information of each lever4. The 6-DOF manipulator iscomposed of link connected with six joints, and theposition and attitude of the end-effector relative to the baseare formula (2)5.
0
6T = 01T 21T 23T 34T 45T 56T

Sphere is the smallest Sphere that contains objects8.
This method is not only simple in structure, but also does
not need to be updated in real time as the object rotates andmoves. However, its tightness is poor and large
redundancy space will be generated.So it's less used.

.(2)

2.1.3. Inverse solution to the kinematic equation of
the manipulator
Inverse kinematics refers to solving the Angle value ofeach joint of the manipulator based on the spatial position
and attitude of the end-effector relative to the basecoordinate system. Inverse transformation is often used tosolve inverse kinematics, According to the forward
kinematics equation of the manipulator, the solution
process is formula (3). the Angle of joint 1 is calculated
according to the corresponding matrix operation. Move thenext joint Angle to the left of the equation and repeat until
you have all the solutions.

( 01T ) −1 06T = 21T 23T 34T 45T 56T

Fig. 1. This is a two-dimensional projection of sphere

AABB9-10 is the smallest hexahedron that contains an
object and its sides are parallel to the axis. The AABB box
model was the first to be used.This method is not only
simple in construction but also fast in updating andrelatively high in efficiency, which can be applied to thecollision detection between deformed bodies.However, forsome objects with irregular shape, its envelopment is poor,
resulting in large redundancy space.

.(3)

The inverse kinematics solution of the 6-DOF
manipulator with the last three joints perpendicular to eachother theoretically has a maximum of 8 sets, and thenneeds to be selected according to the actual joint Angle
range6.

Fig. 2. This is a two-dimensional projection of AABB© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

560

Research on Path Planningenvironment representation methods include raster method,cell tree method and polygon representation method.

OBB11-13 is the smallest cuboid that contains an object
and is arbitrary relative to the direction of the axes. OBB
is a more commonly used type of bounding box.The
biggest advantage of this method is its arbitrary orientation,
which makes it possible to surround the object as closely
as possible according to the shape characteristics of thesurrounded object, but its construction is very difficult.

2.4. Path search
The path searching stage is to apply the corresponding
algorithm to find a walking path based on the environment
model so that the predetermined performance function can
get the optimal value. Common path search algorithms
include RRT algorithm, ant colony algorithm, genetic
algorithm, etc.
2.4.1.

RRT algorithm

RRT algorithm is a global path planning algorithm basedon random sampling proposed by Steven M. aValle. Thismethod does not need to be learned, and is suitable for path
planning in dynamic environment. RRT algorithm can bedivided into two categories, one of which is Single
directional rapidly-exploring random tree (Single-RRT),
and the other is Bi-directional rapidly-exploring random
tree (BI-RRT). The single-RRT method randomly selects
a point in the moving space of the manipulator as a nodein the tree, and continuously searches for nodes to expand
the tree until a path is found from the starting point to thetarget point. The idea of BI-RRT algorithm is to generate
two fast search random trees from the starting point andthe target point until the two trees are connected. The
advantage of this algorithm is that the two trees can be
expanded in parallel. The complexity of this algorithm lies
in how to make the two trees cross connected. The
following mainly introduces the single branch search rapid
expansion of random number method, the main process is
as follows19.
(1) The starting point Xstart and target point Xgoal of themanipulator are given in the working environment space,
and the starting point is taken as the target point of thegrowth tree.
(2) The search step of the given algorithm is L0, and arandom tree Trand in the manipulator workspace with N
nodes is generated from the starting point Xstart.
(3) Select any random point Xrand in the manipulator
workspace.
(4) Traverse the random tree Trand, calculate the distancefrom each node of Trand to the random point Xrand, andscreen out the point Xnear closest to the random point
Xrand among the N nodes.
(5) Take the line of Xnear and random point Xrand as thegrowth direction, take Xnear as the starting point, andgenerate a new node Xnew along the growth direction withthe step length of the algorithm as the length.

Fig. 3. This is a two-dimensional projection of OBB

K-dop
is a convex hull containing an object whose
normal vector of all faces is taken from a fixed direction.
It inherits the good compactness of convex hull and has thecharacteristics of simplicity16.
14-15

Fig. 4. This is a two-dimensional projection of K-DOP

2.3. Environmental modeling
The actual working environment of mobile robot is a real
physical space, while the space handled by path planningalgorithm
istheabstract
space
of
the
environment.Environment modeling is a mapping fromphysical space to abstract space.This efficient descriptionof the robot's active space is called environment
modeling17. Environment model is the basis of path
planning algorithm, whether it can fully reflect theenvironment information and establish environment model
effectively is related to the performance of the whole path
planning algorithm.When the environment information is
fully known, the global environment model can beestablished.In practical applications, the working
environment of mobile robots is often uncertain, whichmakes it impossible for robots to build global environment
model in advance, but to build local environment model inreal time according to sensor information18.
Environment modeling is a very important part of robot
path planning.The reasonable environment model isbeneficial to the reduction of search volume and the savingof space-time cost in the path planning process. Common© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

561

Nana Wang, Yizhun Peng, Zhou Yang, Yuheng Zhang(6) Judge whether Xnew has collision with the obstacle. If
there is no collision, add it to the random tree Trand; if
there is collision, discard the node Xnew.
(7) Judge whether the distance between Xnew and Xgoal
is less than the setting neighborhood (DISfinish) of the end
point.If less than, the path search ends, if not less than
repeat steps 3-6 until less than.
The advantage of RRT algorithm is that it has powerfulability to search unknown space, and it is often used forpath navigation in unknown environment, and it isfast.However, the planned path is not smooth enough.
When the sampling domain of RTT algorithm and theoriginal problem are not suitable, the convergence rate ofstandard RRT algorithm will be slow.
2.4.2.

(5)By counting the total length of the path taken by eachant, the shortest path and the length of the shortest path in
this cycle are obtained.
(6)Update the pheromone concentration on each pathaccording to Formula (5).

τ (t + n) = ρ *τ (t ) + ∆τ , ρ ∈ (0,1)
ij

2.4.3.

GA is an iterative search method based on the principleof genetics proposed by John Holland in 1962.Thealgorithm is mainly composed of eight steps, respectively
set up maps, initialization of population, calculating thevalue of individual fitness, choosing the appropriate
individuals into the next generation, crossover, mutation,
update the population, if appear the optimal path or thenumber of iterations is output optimal solution, otherwise,
continue to iteration, the optimal output of the individual
as the optimal solution.
The method requires fewer objective functions andconstraints, and the optimization scope is extended to thewhole independent variable space. After iteration, themethod can approach the global optimal solution withgreater probability.At the same time, genetic algorithm cansolve nonlinear and complex optimal solutions effectively
by using random transformation rules instead ofdeterministic rules. Its greatest advantage is that it is easy
to combine with other algorithms, and it gives full play to
its iterative advantages.However, due to the large storage
space and long operation time of this algorithm, theoperation efficiency is not high and it is easy to converge
too early.

τ（
ij 0）

=const.M ants were placed at the starting point of
the path planning, Sstart, and Sstart was added to the tabu.
(2)If the current grid is not the target grid, select the next
grid according to the probability calculated by the formula
below, and add the selected grid to your own tabu list.The
probability that ant K at node I chooses node J as thetransfer direction is formula (4).

τ (t )η (t ) , j = allowed
(
)
=
t
P
∑
τ (t )η (t )P (t ) = 0, j = otherwise.(4)βs∈allowedk

ij

α

β

ij

is

Genetic Algorithms

22

β

ij

.(5)

(7)Tabu table cleared, cycle times increased by 1.If thenumber of cycles is less than the maximum number ofcycles, go to Step 2; otherwise, skip the entire cycle andoutput the optimal path and the optimal path length.
This algorithm is a distributed system, which has strong
adaptive ability and good global optimization ability, but
it has a large amount of computation, is easy to fall into the
local optimal solution, and does not have real-time ability.

heuristic factor is α , the expected heuristic factor is ,the pheromone volatility coefficientρ, constant Q, and theinitial pheromone on the path between grids is constant

ij

k

k =1

Ant Colony Algorithmαij∆τ ij = ∑ ∆τ ij

ACA20 is a heuristic global path planning algorithm
proposed by Italian scholar Dorigo Mden.The main idea of
this algorithm is that each ant leaves a certain
concentration of pheromone on the path it travels while
foraging, and the concentration of pheromone is inversely
proportional to the length of the path. By adopting thepositive feedback mechanism, the shortest path with a highconcentration of pheromone will be found soon. The basic
steps of the ant colony algorithm are as follows(Takeenvironment modeling as a raster method as an
example)21:(1)Set the initial grid of path planning as Sstart and thetarget grid as Sgoal.The initialization cycle number Nstartis 0, the maximum cycle number is Nmax, the information

k

ij

m

k

2.5. Path smooth
In general, the path planned by path search algorithm
is a continuous line segment composed of somediscrete points, and the shape of the path is not
smooth.This not only reduces the motion stability ofthe mechanical arm, but also causes the impact andkij(3)Repeat execution 2 until all ants have completed thesubsequent grid selection.
(4)Repeat 2,3.Until all ants have moved to the target grid.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

562

Research on Path Planningvibration of the motor and accelerates the wear of themechanical parts.Therefore, path smoothing is very
important to ensure the smooth movement of themanipulator.Quadric B-spline curves are often used tosmooth obstacle avoidance paths so that the
manipulator can work continuously and stably.

efficient, flexible and accurate path planningalgorithms will be born.

References
1. Chen Gu,Study on modular manipulator design and its
path planning method. jiangsu,Jiangsu University of
Science and Technology,2014.
2. Guanglin Zhang,Xiaomei Hu,Summary of path
planning algorithm and its application.The modern
machinery,2011.
3. Zhaoguang Wang,Rrsearch on obstacle avoidance path
planning for 6-DOF manipultor. Southwest Jiaotong
University,2018.
4. Chengxing Zong,Research on obstacle avoidance
motion planning method for spatial multi DOF serial
manipulator.Hefei ,Hefei university of Technology,2017.
5. Guoxiu Cheng,Research of Obstacle avoidance path
planning based on manipulator with six degrees of
freedom.Shenyang, Northeastern University,2012.
6. WeiSheng,The research of planning technology for
obstacle avoidance path based on multi-degree of freedom
industrial manipulator.Jiangsu, Jiangnan university,2012.
7. Qingxuan Jia,Gang Chen,Path planning for space
manipulator to avoid obstacle based on A*algorithun.
journal of mechanical end ineering,2010,46(13).
8. Guangmiao Jiang,Research and application of collision
detection algorithm based on bounding box.University ofElectronic Science and Technology,2012.
9. Xuewen Chen,Wusheng Chou,Research on collision
detection algorithm based on bounding box.Computer
Engineering and applications,2005,41(5):46-50.
10. Shaowei Fan,Research on real-time collision detection
technology. Zhejiang university,2003.
11. Zhilei Ding, Fast collision detection based on OBB
bounding box. Lanzhou University of Technology,2006.
12. Gottschalk S.Collision queries using oriented boundingboxes:The
University of North Carolina at Chapel
Hill,2000 .
13. D.Baraff. Interactive simulation of solid rigid bodies.
IEEE Computer Graphics and Applications, 1995,15(3):6375
14. Kim Y J, Lin M, Manocha D. Fast penetration depth
estimation using rasterization hardware and hierarchicalrefinement.Proc of Symposium on Computational
Geometry.2003:386-387
15. Klosowski J, Held M, Mitchell J S B,etal. Efficientcollision detection using bounding volume hierarchies of k2

3. The conclusion
As the robotic arm is more and more widely used invarious fields, the path planning environment of therobotic arm is more complex and changeable.This
requires that the path planning algorithm should havethe ability to respond quickly to complex environmentchanges.This paper summarizes the basic methods ofpath planning.In the future, path planning technology
will be applied in more and more fields, and more

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

563

Nana Wang, Yizhun Peng, Zhou Yang, Yuheng Zhang

DOPs. IEEE Transaction on Visualization and Computer
Graphics, 1998,4(1):21-37
16. Tao Wang,The Study of robotic path planning andrelated simulation system based on ant algorithm.Xian,Xi an
University of Science and Technology,2009.
17. Lihui Peng,Research on the kinematics and path
planning of manipulator.Hunan,Hunan University ofTechnology,2012.
18. Chao Ma,Research on path Planning of 6R tandem
manipulator in complex space environment.Jinan
University,2017.
19. Jinming Dong,Research on path Planning based on Ant
colony algorithm.Xian,Shaanxi Normal University,2009.
20. Jincai Ma,Research on path planning based on six-DOF
manipulator.South
China
UniversityofTechnology,Guangzhou.2013
21. J Tu，S Yang． Genetic Algorithm Based Path Planning
for a Mobile Robot.Taiwan: Proceedings of IEEE
Intelligent
Conferenceon
Robotics
andAutomation,3003:P1221-P1226.
22. Tao Han,Huaiyu Wu,Research on path planning for
manipulator to avoid real-time obstacle based on genetic
algorithms.Applicatian Research f Camputers,30(5),2013.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

564

Research on White-Line-tracking Walking Technique of NAO Robot
Wanlong Peng1,2, Yizhun Peng1,2*, Yuheng Zhang1,2, Tianye Jian1,2
1
College of Electronic Information and Automation,
Tianjin University of Science and Technology, China
2
Advanced Structural Integrity International Joint Research Centre,
Tianjin University of Science and Technology, China
E-mail: * pengyizhun@tust.edu.cn
www.tust.edu.cn

Abstract
Liner-tracking walking technique is an essential part of intelligent robot technology. This paper
selected NAO robot as the research platform to study the line-tracking walking1 problem. Firstly,
NAO white shell increased the difficulty of white line recognition, this is different from otherrobot. A method based on image preconditioning2 is presented, sorting white edge with cut-point
which is non-differential or derivative is zero, then designed the controller based on the
improved digital incremental PID(proportion integration differentiation)algorithm anddetermined the controller parameters through the concise test method on Simulation. Finally, it
was realized by programming in Python language.
Keywords: Humanoid robot, Edge classification, Image preconditioning, Simulation

1. Introduction

images every 30s,and converts the image
information into reference input signals throughtechnical means; Find the difference between thepotential and the reference signal as thecontroller input; Write the program in python
and carry out the line-tracking simulation
experiment in the software.
2.White line processing
We takes the following steps to processing thewhite line:

In recent years, the research and application of
intelligent humanoid robots have gradually
emerged. Line-tracking robot with certain
autonomous decision-making and perception
capabilities have quietly become an industry
trend. Intelligent robot with line-following
walking function can replace or help humans tocomplete some high-complexity, high-risk andhigh-workload inspections3 under complex
conditions in some harsh environments, to findand eliminate equipment abnormalities in time.
Minimize the loss of personnel and property.
Therefore, the research on intelligent robots withline-following walking function has greatsignificance in the industrial field. The NAO
robot represent by the humanoid robot is a bipedhumanoid robot used for scientific research、
competition and entertainment, with 25 degrees
of freedom and 36 sensors. Possess certain
environmental perception and the ability to makecorresponding decisions. This paper uses the
NAO robot as a platform to study the image
perception processing of robot. The main work
content of this research includes: Propose acolor-segmentation method to increase the speed
and reliability of the line recognition
algorithm4;The robot`s built-in camera collects

Fig1.white line identification flowchart

2.1 Image preprocessing
Through the color segmentation of the image
the points outside the white line will generally
become the fulcrum,so that the image boundary
is easy to find.The original image format
provided by the NAO robot is YUV422,but theoriginal image is generally used directly to avoid
the overhead of format conversion.
The color table is a three-dimensional array,each dimension represents Y, U, and V© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

565

Wanlong Peng, Yizhun Peng, Yuheng Zhang, Tianye Jian

multiple paths that may appear in the outer scanfor the first time, there are also two cases where
there are multiple path options, as shown in
Figure 5 and Figure 6, other points are normal,
There is only one way to traverse。

respectively. The array value stores the color
type, such as 0 for unknown color, 1 for red,
etc.Set the dimension to 64,64 and 64
respectively enough to meet the requirements.
The color table needs to be established bymanually collecting pixels. This is establishedoffline during robot debugging. The establishedcolor table is used in the competition.
The image segmented5 by the color table
generally has some color points that are different
from the actual ones. For example, there aregreen points on the robot, and sometimes whiteor unknown colors may appear on the groundthat should be green. This phenomenon isdifficult to remove, but it can use some noise
processing methods to filter out some salt andpepper noise.
3 White edge recognition
The traversal of white points is an edge
growth process , and the recognition process6 is
as follows: First,scan the image in order, find thefirst white edge point, and record all adjacentpoints of this point. If the image is scanned, turnto enter; Second, take one of the neighboring
points as the starting point, start from this point,
traverse along the edge, and record these edge
points,turn to third, turn to traverse, if theneighboring points are traversed, return first to
perform the next scan;Third,calculate therelative slope information of the current point
and the previous recorded point every few points
to determine whether it is a split point, if it is nota split point, continue to traverse, otherwise go
to last;Lase, when the split point appears,
calculate the slope information of the point
recorded in second, determine whether it is along straight line, a short straight line or an arc,and record the result, and return to second;
Next,The result recorded in the comprehensive
analysis last Information, output results.
3.1 Scan and traverse edge points
Scan the image first to find the white edge
points, and click the grid scan here. So as toensure the effectiveness of scanning and speed
up the algorithm at the same time. Scan theimage from top to bottom in a column. If white
dots and non-white dots appear alternately in thesame row or in the same column of the 4 dots
during scanning, edge dots appear, and then scanrow by row or column by row to locate the edge
point.
There is the problem of whether the path isunique during traversal. In addition to theFig.2 Convex edge

Fig.3 Edge with width 2

The numbers 1 to 8 in Figure 2 are all whiteedges. When traversing to 4, there are two pathsof 5 and 6 in different directions. There are two
ways to solve this convex situation: (1) Use thestack data structure to save Such different
paths are then traversed by depth-first traversal ;(2) Consider this situation during image
preprocessing, and directly filter out the points 4
and 5 that cause this convex edge. Anothersituation where different paths appear is shownin Figure 3. After research, it is found that when
the traversal priority is the same, the traversed
paths will not intersect between the edges. Thissolves the ambiguity path problem that appears
when traversing along the white edge.
3.2 Calculate the split point
As shown in Figure 4, there are several straightlines and curves on the plane, so how to classify
these lines based on the situation of the NAO
stadium? This article uses dividing points todivide them. Intuitively, the intersection of thestraight lines should be divided Based on thisidea, the dividing point in this paper is the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

566

Research on White-Line-tracking

non-derivable point and the point where the

The curve EOF is exactly the deformation ofderivative is 0.

ABC, and these can be treated in the same way.
The curve GH in Figure 4 is a circular arc with alarge curvature. When traversing GH, there is apoint on which the derivative is zero, but whencalculating the slope, it is found that the slopechanges very little and must be within thethreshold range. There is no split point.
3.3 Result classification
When a segmentation point8 appears or theimage boundary is reached during traversal, aseries of points just traversed need to beclassified. These points can be classified intolong straight lines, short straight lines and arcs.
First of all, the number of points can be used tojudge whether the current points are shortstraight lines. The others are long straight lines
or circular arcs. Some simple calculations can beused to judge whether they are long straight lines
or circular arcs. Obviously, the slope of the startpoint and the end point of the long straight line
should not change much, while the arc shouldchange more. The method used in thisexperiment is to compare the slopes of the front
N points and the back N points of a series ofpoints to be classified to judge, that is, use thedifference between slope1 of formula (2) andslope2 of formula (3) to measure , And thenjudge whether formula (4) is true or not, thismethod is simple to calculate without losing
validity.
slope1 = (∑N(2)
i=0(PiPi + 1)/N

Fig.4 Line classification

There is a polygonal line ABC in 4, and its
function is f(x). When accessing from point A topoint C along the polygonal line ABC, calculate
the left and right limits of each point P on thepath. If the left and right limits of P exist and areequal, the formula (1) Is established, thencontinue to the next point. When visiting point B,
it is found that the left limit of point B is equal to
the slope of the straight line AB, and its right
limit is equal to the slope of the straight line BC,
that is, formula (1) is not true, then ABC mustnot be on a straight line, and point B is thedividing point.

Record the path from A to B,

then start from point B and continue to visit
point C in the same way, thus dividing thepolyline ABC into two segments AB and BC.

Slope2 = (∑N
i=0(Pn − i − 1Pn − i)/N

When the straight line and the abscissa are|slope1-slope2|=threshold

perpendicular, the points on the straight line canbe guided but the slope does not exist, and theincluded angle is 90 degrees. It is worth
mentioning that in order to reduce the impact oflarge changes in the slope of adjacent edge
points caused by unsmooth edges7, you cancalculate the angle of the vector every N points
along the edge, which may bring about thephenomenon of "turning back" when traversing
the edge with a small width. In this way, pay
attention to the direction change of the traverse
point when using the angle instead of the slope.

Fig.5 import result

lim 𝑓𝑓(𝑥𝑥) = lim 𝑓𝑓(𝑥𝑥) （1）𝑥𝑥→𝑝𝑝+

𝑥𝑥→𝑝𝑝−© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

567

(3)
(4)

Wanlong Peng, Yizhun Peng, Yuheng Zhang, Tianye Jian

The color table of the experiment in Table 1 isconfigured under light 1, and the white line thatis too far away cannot be recognized basically(when the distance is very far, the recognized
court boundary can be used to give an estimatedvalue), and it can be recognized when the
distance is gradually reduced

Beyond theboundary, there is a good recognition rate around2m.When the distance is too close, although theFig. 6 Classification result

image is very clear, the robot’s field of view is

It can be seen from the classification resultsvery small, and it can be easily recognized whenthat the arcs on the court were originally

there is no interference. When NAO is used toclassified as straight lines. It is necessary toeliminate interference, the robot basically blocks

considerinthe white line, so the formal recognition rate is

combination with the actual situation of the court.

50%. Using the color table configured underIn fact, many recognition algorithms directly

light 1 to split the color of the court under light 2,transform the arc into a short straight line, andthe experimental results are much worse thanthen connect, after comprehensive consideration.

that under light 1, but the close distance isThis can be achieved by the principle of Houghbasically not affected. In fact, most of theline transformation, but the element considered

recognition results are used by the upper layer inin Hough line transformation is a point.And here

the robot positioning algorithm. Most of theis a straight line,Also pay attention to the parallel

positioning methods used are Monte Carlo

features of the lines, and join together shortfiltering or generalized Kalman filtering, so thatstraight lines, long straight lines, and arcs.

the program only needs to identify the image

4. Experimental results and analysis

The main lines and arcs.his method is used for

the

arcs

and

straight

lines

The experiment used is 3. The 3 version of thethe calculation of the two-dimensional array of

NAO robot, the main hardware parameters9:

the same price of the three-dimensional image

CPU:AMD GEODE x86 500MHz; Synchronous

after color segmentation. It also uses grid

Dynamic Random Access Memory (SDRAM):

scanning. The main reason is that the recognition

256MB, Flash: 2G; Camera: 2 COMS cameras,

method is simple to calculate. The white line

providing up to 30 frames/s, resolution of 640 ×

recognition program runs generally in 9ms to

480 images; Open CV library comes with the

12ms, which satisfies the robot competition.

operating system. The criterion of experimental

Requirements for straight line recognition.

judgment is to identify the main straight lines

Compared with simple line recognition, theand arcs. The results are shown in Table 1.

method in this paper pays more attention to theoverall efficiency of robot image processing.The

Table 1. Experimental data

distance

Light 1

image preprocessing in it makes subsequentLight 2

recognition algorithms more effective.

light
Far

away

63%

20%

96%

71%

50%

50%

(more than 4m)
Medium(0.5m to 3m)
Near

(less

than 0.3m)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

568

Research on White-Line-tracking

References
1. Thomas Rofer,Tim laue,.Particle-filter-based
Self-localization using Landmaks and Directed
Lines Robot Soccer World Cup
IX.2005,4020,pp.608-615.
2. Xiangkan,Sun,.Detection and tracking of NAO
based on machine learning[J].Journal of
Changchun university of Science andTechnology(Natural Science
Edition),2016,39(2),pp.116-119.
3. Rong-Jong Wai. Tracking control based onneural network strategy for robot manipulator[J].
Neurocomputing,2002.
4. D.M. Dawson,Z. Qu,F.J. F. Dorsey.Robust
control for the tracking of robot motion[J].
International Journal of Control,1990.
5. Shen Zhang,Research on target recognition andlocalization of NAO robot [D].Anhui University,
2013.pp.20-23.
6. Kun Zhao,Research on the visual navigation
method of intelligent inspection robot for
substation [D].North China Electric Power
University, 2014.pp.43-49.
7. Xuefeng Bai,Bin Yang,Based on NAO robot
target recognition and localization algorithm [J].
Journal of Chengdu University of Information
Technology, 2014, 29(6),pp.625-629.
8. Yanhong Wang，Research on moving target
detection and tracking algorithm based on
OpenCV [D]. Hangzhou: Hangzhou Dianzi
University, 2013。
9. Xin Wang, Hongliang Yu,Research on diesel
engine fault diagnosis based on the fusion ofBayesian and genetic neural network [J]. Ship
engineering. 2012,34(1),pp.32-35.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

569

A Design and Implement of Portable Epidemic Detection Device based on STM32
Yuqi Zhao, Yizhun Peng*, Xiaowei Wu, Yusong Zhou

College of Electronic Information and Automation,
Tianjin University of Science and Technology, Tianjin, 300222, China;
E-mail: * pengyizhun@tust.edu.cn
www.tust.edu.cn

Abstract
The equipment is mainly designed for epidemic detection. It is an intelligent detection box device. It collects andanalyzes data through cameras and sensors. It uses LBP algorithm and LAB color model to enable the detection box
to identify the temperature and mask of the entered personnel. Wearing conditions, preliminary comparison of
epidemic prevention standards for passers-by, whether to wear masks, whether the body temperature exceeds thestandard, and at the same time enter the identity of new personnel and store them in the database, and detect theincoming personnel through face recognition to find out in time Those who are not entered shall take corresponding
measures in time to improve the efficiency of epidemic prevention and control. The device does not need to be held
in hand, can be fixed anywhere, can effectively avoid cross-infection, and supports self-set temperature alarm
thresholds, support voice broadcast reminders, fast and accurate non-contact temperature measurement.
Keywords: Non-contact temperature measurement, face recognition, mask recognition, identity collection

1. Introduction

designed a smart detection box to make detection moreconvenient and quicker and improve the efficiency ofSince the first case of unidentified pneumonia appeared

prevention and control.

in Wuhan on December 8, 2019, to December 1, 2020, a

2. System overview

total of 93,889 new coronary pneumonia patients havebeen diagnosed nationwide, of which 87,707 cases have

The device is a temperature detection box with

been effectively cured, accounting for approximately theidentification. The detection box can realize non-contact

total number of confirmed cases Of 93.41%.

temperature

Since the new crown epidemic, domestic measures haverecognition,

face

recognition,

identity

comparison, mask recognition, voice prompts and otherbeen effectively implemented1. A large number offunctions, data analysis through2 camera and sensor

detection equipment has been used in epidemic

recognition, and human-computer interaction throughprevention and control, and there has been a relatively

OLED display. The main functions that can be achievedlarge market in China and the world. At the same time,

are: 1. Non-contact temperature detection, and at thethe functional requirements of detection equipment havesame time, by setting the alarm threshold, a faster andbecome higher and higher. Non-contact body temperature

more accurate identification of epidemic prevention

detection, mask wearing detection, identity entry, etc. canstandards can be achieved, and then voice broadcast

achieve precise identification and detection. The accuracy

reminders can effectively improve the prevention andof the infrared thermometer is related to four key factors:

control efficiency. 2. Data collection through the camera,

sensor, ADC accuracy, algorithm, and the structure andreal-time data transmission through the serial port, data

production process of the forehead thermometer. So we© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

570

Yuqi Zhao, Yizhun Peng, Xiaowei Wu, Yusong Zhou

analysis through the MCU, and then voice broadcast of

hardware resources, leading to UART, I2C, SPI, PWM,

the mask wearing situation, mainly used for standard

ADC, DAC and GPIO interfaces to facilitate themask recognition in the current epidemic. 3. Throughexpansion of peripheral functions. The USB interface isoled display, perform human-computer interaction, enterusedface data, and store it in the database, so that faceenvironment OpenMVIDE on the computer to assist in

information can be effectively recognized, identity

programming, debugging and updating firmware. The TF

recognition, and prevention and control become morecard slot supports large-capacity TF cards, which can be

efficient. 4. The device does not need to be held in handused to store programs and save photos. The open source

and can be fixed anywhere, which can effectively avoid

environment and rich Python libraries make it very

cross-infection.It also supports self-settable temperature

conducive to the development of visual identity.

alarm thresholds, voice broadcast reminders, and fast andcharacteristics.

integrated

development

measurement.

Infraredtemperature

measurement is to determine the temperature of the

The core controller of the device uses the stm32f1 series
obviousthe

MLX90614 is an infrared thermometer for non-contact
temperature

3.1. Core controller
hasto

The MLX90614 infrared temperature sensor is used.

3. The hardware structure design

whichconnect

3.3. Sensor

accurate non-contact temperature measurement.。

chip,

to

object according to the infrared radiation energy of the

ARM'sobject to be measured. It does not contact the object to behigh-performance "Cortex-M3" core: 1.25DMips/MHz,measured, and has no influence on the temperature

while the ARM7TDMI is only 0.95DMips/MHz,

distribution field of the object to be measured. It has highfirst-class peripherals. 1μs dual 12-bit ADC, 4Mbit/s

temperature resolution, fast response speed, and wide

UART, 18Mbit/s SPI, 18MHz I/O flip speed. Low powertemperature measurement range. Restricted by the upper

consumption, consumes 36mA at 72MHz (all peripherals

limit of temperature measurement, good stability andare in working state), drops to 2 μ A in standby,

other characteristics. The measured target temperature

maximum integration, reset circuit, low voltage detection,

and ambient temperature of MLX90614 are output

voltage regulator, precise RC oscillator, etc3.

through the IIC interface. The sensor is directly
connected to the core board, and is packaged as a

3.2. Image acquisition equipment

temperature measurement structure through a printed

The device uses an openmv4 camera, which is an open

external package. At the same time, the circuit issource, low-cost, and powerful machine vision module.

connected in series with an infrared indicator light toWith STM32F427CPU as the core, it integrates theconfirm and indicate the object under test.

OV7725 camera chip. On the small hardware module, theWe need to read the data from the MLX90614 infrared

core machine vision algorithm is efficiently implemented

temperature sensor, no matter whether the data is inin C language, and the Python programming interface is

EEPROM or RAM, we can read it in the same way. We

provided. The adopted STM32F427 has abundant

have already said that the command byte for operating© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

571

A Design and Implement

is self-luminous. For the same display, the OLED effectshould be better. With current technology, it is difficult toincrease the size of OLEDs, but the resolution can bevery high.
The oled display is as follows, which can effectively
carry out human-computer interaction
3.5. Voice broadcast
The voice broadcast function of the device is realized bysyn6288 module, which broadcasts through the speaker.

Fig.1 MLX90614 infrared temperature sensor

The SYN6288 Chinese speech synthesis chip used by

EEPROM and RAM consists of a 3-bit command and a

Speech Synthesizer Bee is a speech synthesis chip with

5-bit address. We implement the data reading function as

high cost performance, more natural speech synthesis,

follows:

and for high-end applications4. SYN6288 receives the
text to be synthesized through the asynchronous serial
port, and realizes the conversion of text to voice (TTS).
The module is directly connected to the MCU.
4.

Fig.2 Data reading function

System process

4.1. System operation process

3.4. Display

According to the needs of users, the detection mode is

The display uses an OLED display module, OLED, or

selected. The detection mode includes mask recognition,

Organic Light-Emitting Diode (Organic Light-Emitting

identity entry and temperature detection.

Diode), also known as Organic Electroluminesence
Display (OELD). OLED has excellent characteristics
such as self-luminous, no backlight, high contrast, thin
thickness, wide viewing angle, fast response speed, canbe used for flexible panels, wide operating temperature
range, simple structure and manufacturing process, etc. It
is considered to be The next-generation flat panel display
emerging application technology.
OLED

displaytechnology

has

self-luminous

characteristics. It uses very thin organic material coatings

Fig.3 System operation process

and glass substrates. When current passes through, these

When the user enters the mask recognition, the camera

organic materials will emit light. The OLED display

module will automatically turn on to take pictures,

screen has a large viewing angle and can save power.

compare and analyze the photographed image with theThis kind of display device has been used in MP3 players

mask recognition set that has been trained, and get thesince 1988.

result. Mask recognition is based on the neural network

LCDs require backlighting, but OLEDs do not, because it© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

572

Yuqi Zhao, Yizhun Peng, Xiaowei Wu, Yusong Zhou

(DataH:DataL) data, according to the formula:model trained by openmv using EDGE IMPULSE's
online website. Training the neural network model is to

T=(DataH:DataL)*0.02-273.15

perform machine learning. Through machine learning,

To get the final temperature T, we use the bubble

you can determine whether there are people in the photosorting method when collecting, and each temperature

and whether to wear masks and other functions. By

measurement will collect ten times of data. After sorting,establishing two categories, one face and one mask, one

remove the abnormal temperature value and average it as

hundred images are collected each, and the collected data

the final temperature measurement data, so as to avoid

is sent to the EDGE IMPULSE website for model

abnormal temperature And make the final temperature

training. After DSP preprocessing, the following analysis

data more accurate.

is obtained:

Fig.4 DSP preprocessing results

It can be seen that the two colors of the training model
face and mask are clearly distinguished and are not mixed
together. After ten rounds of training, the training dataobtained is as follows:

Fig.6 Temperature measurement flow chart

Fig.5 DSP training data

It can be seen that our mask recognition model has an
accuracy of 95.1%. Run the training set on openmv andthe mask recognition is completed.
4.2. Detection processing
After the user enters the temperature detection, thethreshold value of the temperature measurement device
can be set by key operation5, so that the high temperature
alarm system remains effective. After the MLX90614
non-contact temperature measurement module collects© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

573

A Design and Implement

•

•

•

Enter the identity in advance, save it in the database,and then perform face recognition and check the facerecognition function
There are two cases of wearing masks and notwearing masks by three people, and testing the threegroups wearing masks to test their stability
Detect the on-site deep learning function of thedevice, Provide three unfamiliar faces, learn theirfacial identity within 5 minutes and record them inthe database, and then the temperature measuring
box identifies whether they are the input personnel,
and detects whether they have the function of deeplearning

5.2. Test results
After repeated testing, the device has the above functions,and can identify and detect very stably. It can quickly andaccurately identify human body temperature and watertemperature and broadcast it. It can accurately and stably

Fig.7 MLX90614 send and receive byte flow chart

recognize the wearing of a mask. It can learn on-site,

4.3. System circuit design

automatically enter the identity and recognize it. . The

The system is powered by 5v with energy, voltage

temperature detection efficiency is greatly improved, andthe identification is accurately prevented and controlled.

conversion is connected to the core board through a
voltage stabilizing module, the circuit board adopts a

References

homemade pcb circuit board, integrated oled, MLX90614,

1. Yang Li, Yang Zhen "Principle and Technology of
Infrared Thermal Imaging Temperature Measurement"
Science Press, 2012.6.
2. Li Min, "Design of Wireless Infrared Temperature
Measurement System" Guangdong Vocational College ofAgriculture, Industry and Commerce, 2020.3.
3. Zheng Zaisheng "Research on Automatic Fare Collection
System Based on Face Recognition" China Computer
Room Facilities Engineering Co., Ltd., 2020.9
4. Li Chengyong, Tan Hanzhong, Wang Sha, et al. Intelligent"seeking" car control system based on Open MV[J].
Liquid Crystal Display,2020,08:870-876.
5. Ye Siyan, Zhang Yongning, Li Jin, et al. Design of smart
follow luggage based on UWB positioning and OpenMV recognition[J]. Journal of Chengdu University ofTechnology,2020,03:39-33.

openmv and power interface, which can make the signaltransmission more stable and reduce external signal
interference , To ensure the stability and accuracy of thetest results.
5. Test and conclusion
5.1. Test plan••

The encapsulated temperature detection gun is used
for non-contact detection of different watertemperatures, such as 35.7°C, 41.2°C, and 80°C, andthe results are tested with the threshold 25-48°C set
in advance
Measure human body temperature, floor temperature,desktop temperature without contact within a certain
distance© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

574

Landslide Area Detection from Synthetic Aperture Radar Images Using Convolutional
Adversarial Autoencoder and One-class SVM
Shingo Mabu

Graduate School of Sciences and Technology for Innovation, Yamaguchi University
2-16-1 Tokiwadai, Ube, Yamaguchi 755-8611, Japan
Soichiro Hirata

Graduate School of Sciences and Technology for Innovation, Yamaguchi University
2-16-1 Tokiwadai, Ube, Yamaguchi 755-8611, Japan
Takashi Kuremoto

Graduate School of Sciences and Technology for Innovation, Yamaguchi University
2-16-1 Tokiwadai, Ube, Yamaguchi 755-8611, Japan
E-mail: mabu@yamaguchi-u.ac.jp, wu@yamaguchi-u.ac.jp

Abstract
An anomaly detection model using deep learning for detecting disaster-stricken (landslide) areas in synthetic apertureradar images is proposed. Since it is difficult to obtain a large number of training images, especially disaster area
images, with annotations, we design an anomaly detection model that only uses normal area images for the training,where the proposed model combines a convolutional adversarial autoencoder and one-class SVM. In the experiments,
the ability in detecting normal and abnormal areas is evaluated.
Keywords: anomaly detection, convolutional neural network, adversarial autoencoder, one-class SVM, synthetic
aperture radar, landslidefrom sunlight reflection, and synthetic aperture radar(SAR) images that are obtained by sensors that emit
microwaves to the ground surface of the earth. It is easy
for a human to interpret optical images, but they cannot
be observed at night or in bad weather. On the other hand,
SAR can observe the surface of the earth regardless oftime and weather; thus, SAR images are useful for rapid
rescue activities at night and in bad weather conditions.
However, it is difficult for human eyes to interpret SAR
images, unlike optical images. Therefore, many methods
that analyze SAR images have been proposed to detectdisaster areas rapidly3,4, and machine learning techniques,
especially deep learning, have also been applied tolandslide area detection5.
When deep learning is applied to disaster area detection,a large number of training samples are necessary.
However, shooting conditions of SAR images are

1. Introduction
In Japan, typhoons often approach and pass from July to
September every year because of the effects of westerly
winds and high atmospheric pressures. Landslidedisasters frequently occur due to heavy rains caused bytyphoons, which leads to major accidents related totransportation and human life. Until now, observation of
the disaster areas, e.g., landslides, has been mainly
conducted by aircraft1. However, in recent years, remotesensing using satellite images has attracted attention as amethod for observing a wide area2. In satellite remotesensing, a sensor is mounted on an artificial satellite toobserve the ground surface of the earth, and the obtained
image is analyzed. Therefore, we can see the damages
without going directly to the disaster areas. Typicalsatellite images include optical images that are obtained© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

575

Shingo Mabu, Soichiro Hirata, Takashi Kuremoto

different time by time, e.g., seasons, and it is also difficult
to obtain a sufficient number of training samples withannotation (correct class labels) immediately after adisaster occurs. To overcome this problem, we propose
an anomaly detection system that detects abnormal
images that deviate from the features of normal images.
The proposed method uses only normal images for the
training without using abnormal (disaster) images.
Normal images are relatively easier to obtain comparedwith abnormal images, thus anomaly detection is usefulin a practical situation of disaster area detection. The
proposed method consists of a feature extraction part anda detection part. The feature extraction is realized byconvolutional adversarial autoencoder (CAAE) that is theextension of adversarial autoencoder (AAE)6, where thelayers of AAE are replaced with convolution anddeconvolution layers. CAAE is trained with
unsupervised learning, thus the cost of the annotation is
not necessary. The detection part is realized by one-classSVM (OCSVM)7 that detects outliers that deviate fromthe normal region.
This paper is organized as follows. In section 2, themechanism and features of SAR is introduced. In Section
3, the proposed method that combines CAAE andOCSVM is explained. In section 4, the experimental
conditions and results are described. Finally, section 5 isdevoted to conclusions.

Fig. 1. Microwave emission, Scattering, and backscattering of
SAR10electric characteristics (dielectric constant, etc.) of thescatterer. Therefore, for example, seawater or cars madeof metal strongly reflect microwaves because a current iseasily induced. On the other hand, sand and trees have
low reflectivity because hardly any current is induced. In
each pixel of SAR images, the intensity of the received
microwave is recorded.
3. Materials and methods
3.1. Dataset
A SAR image used in this paper is shown in Fig. 2. It
shows the northern Kyushu area in Japan on July 7, 2017,
taken by Advanced Land Observing Satellite No. 2(ALOS-2). Large-scale landslides due to the torrential
rain from July 5 to 6, 2017, occurred in this area. The
yellow areas in Fig. 2 show the locations of the landslidethat were annotated by the Geospatial Information
Authority of Japan. The size of the image is 6648 ×4360 [pixels] and the resolution per pixel is about 3m2.
In this paper, ROI (region of interest)-based anomaly
detection is carried out, that is, we first divided the whole

2. Synthetic Aperture Radar (SAR)
SAR is an active image radar that synthesizes small
antennas mounted on a platform, such as an aircraft orsatellite, to realize large virtual antennas and generates
high-resolution radar images8,9,10). Because SAR is anactive sensor that emits microwaves, it is possible toobserve the surface of the earth regardless of the presenceor absence of sunlight and clouds. SAR images areapplied to the research fields of agriculture, disaster,oceans, earth science, and so on11,12). SAR emits
microwaves and receives the reflected microwaves fromthe surface of the earth (Fig. 1). When a microwave
emitted from the SAR antenna enters a conductor ordielectric, a current is induced, and the microwave isreemitted from the induced current. This is called
scattering and scattering in the opposite direction of theincident wave is called backscattering. Because
backscattering is the diffuse reflection caused byscattering, backscattering is different from specularreflection.
SAR receives the backscattering and executes image
reproduction. The scattering intensity of the microwaves
strongly depends on the frequency, wavelength andFig. 2. SAR image and landslide areas (yellow areas)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

576

Landslide Area Detection from
Feature extraction

Input 𝐱

Decoder

Encoder

PCA
𝒛𝒛

𝐱�

Anomaly detection
Output

One-Class
SVM

NormalorAbnormalFeature vector 𝒛𝒛 (negative / fake)𝒛𝒛𝒇 = 𝒛𝒛Fig. 3. Examples of 128 × 128 [pixels] ROI images

SAR image into 128 × 128 [pixels] ROI images. Fig. 3
shows examples of normal and abnormal ROI images.
Note that the target area for the detection of normal andabnormal is the central 32 × 32 [pixels] area of each
ROI image. In other words, the surrounding area of thetarget area is used as a piece of supporting information(context) for the detection. The ROI images were split
into training data including normal only, testing data ofnormal, and testing data of abnormal.

Gaussian distribution

Fig. 4. Examples of 128 × 128 [pixels] ROI images

shows the detailed structure of CAAE that extracts
features from ROI images. The upper part of Fig. 5 is a
convolutional autoencoder (CAE) that contains an
encoder and a decoder. The encoder outputs a 128dimensional feature vector 𝒛𝒛, and the decoder outputs thereconstructed ROI images. The encoder consists of some
Block1 and fully-connected layers, where a Block1
contains convolution and max pooling layers along withbatch normalization and rectified linear unit (ReLU). The
decoder consists of some Block2, fully-connected anddeconvolution layers, where a Block2 contains adeconvolution layer along with batch normalization and
ReLU. The loss function (LOSS1 in Fig. 5) is the meansquared error between the input and output images, andthe weights of the encoder and decoder are updated byerror backpropagation with Adam13 to minimize LOSS1.
The encoder and discriminator at the lower part of Fig. 5
are adversarial networks, where the discriminator

The overview of the anomaly detection model designedin this paper is shown in Fig. 4, where CAAE generates
feature fector 𝒛𝒛, 𝒛𝒛 is transformed by principal component
analysis (PCA), and OCSVM classifies 𝒛𝒛 as normal or
abnormal.
3.1.1. Feature extraction using Convolutional
Adversarial Autoencoder (CAAE)
In this paper, the extended model of AAE6, named CAAE,
is used for the feature extraction of SAR images. Fig. 5

Sigmoid

Probability of Positive
Probability of Negative

ReLU

Block 2LOSS2Gaussian distribution

Fig. 5. Detailed structure of CAAE
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

577

Batch
normalization

OutputDeconvolution

Deconvolution

Block 2Block 2Block 2Block 2
Softmax

Fullyconnected

ReLU

Fullyconnected

ReLU

Input

LOSS1

Discriminator
Fullyconnected

Positive(Real)

Block 2

Dropout+ Fully-connected+ ReLU

Feature 𝐳

Fully-connected +ReLU+ Fully-connected

Block 1

Block 1

Block 1

Block 1

Block 1

Block 1ReLU

Max poolingBatchnormalization

Convolution

Block 1

Negative(Fake)

Block 2

Decoder

Encoder (Generator)

Draw samples
from 𝑝 𝐳

Probability of fake / negative

𝒛𝒛𝒓

Generate
random vector (positive / real)

Convolutional Adversarial Autoencoder

3.2. Method

Input

Probability of real / positive

DiscriminatorShingo Mabu, Soichiro Hirata, Takashi Kuremoto𝑧2

𝑧2

Table 1. The number of ROI images in condition 1
Normal area

Training/Testing
Training (normal only)
Testing (normal)
Testing (abnormal)

Normal area

Large ν𝑧2

small νThe number of images
13542
164
164

𝑧1

Fig. 6. Effects of ν in OCSVM

4. Experiments

distinguishes the vectors generated by Gaussian
distribution (regarded as positive or real samples) fromthe feature vector 𝒛𝒛 inputted from the encoder (regarded
as negative or fake samples). The loss function (LOSS2
in Fig. 5) is the softmax cross-entropy based on thepositive and negative probabilities outputted by thediscriminator. The discriminator updates the weights tominimize LOSS2, while the encoder updates the weights
to maximize LOSS2. As a result of the above learning,
the feature vector 𝒛𝒛 captures the features of the input
images and its distribution follows a Gaussian
distribution. The constraint imposed by the adversarial
networks increases the robustness of the feature
extraction executed by the convolutional autoencoder.

4.1. Condition 1
The class labels of ROIs are given by the following rule.
When 60% or more area of the central 32 × 32 area of
each ROI image contains the abnormal area, it is regarded
as abnormal, otherwise, normal. The number of training
data (normal only), that of testing data (normal), and thatof testing data (abnormal) are shown in Table 1. CAAE
is trained with the training data and all the training data
are encoded as the feature vectors. Then, PCA is applied
to the feature vectors to reduce the number of dimensions
from 128 to 40. The feature vectors after the dimensional
reduction by PCA are used to train OCSVM. Finally, thetesting images encoded and transformed by CAAE and
PCA are classified as normal and abnormal.
Table 2 shows the results obtained by the combination ofCAE, PCA, and OCSVM (called conventional method),
and Table 3 shows those obtained by the combination of
CAAE, PCA, and OCSVM (called proposed method).
The best accuracy obtained by the conventional method
is 55.7% (ν=0.5) and that obtained by the proposed
method is 57.9% (ν=0.4). Therefore, we can confirm that
CAAE is better than CAE as a feature extraction method,
however, there is a large room for improvement on theaccuracy.

3.3. Anomaly detection by One-class SVM(OCSVM)
The feature vector 𝒛𝒛 generated by CAAE is inputted to
OCSVM. SVM is originally a supervised-learning-basedclassification algorithm for two classes, which needs
training data with class labels. OCSVM needs only oneclass data for the training, and after the training, OCSVM
regards the data that deviates from the learned class dataas outliers, that is, abnormal. Anomaly detection for thetesting images in this paper is carried out by OCSVM.
OCSVM has a parameter ν that determines the proportionof normal and abnormal areas. For example, when ν is set
at a large value as shown in Fig. 6 (left), the normal area
becomes small. In this case, only the data that is very nearto the center of the normal area is regarded as normal,
which increases the sensitivity of abnormal, but may
increase false positives, that is, the true normal (negative)
is misclassified as abnormal (positive). Conversely,
when ν is small, the normal area becomes large, which
increases the specificity, but may increase false negatives,that is, the true abnormal is regarded as normal.
Therefore, the value of ν should be determined carefully
considering the trade-off between false positive and false
negative.

4.2. Condition 2
In condition 2, the class labels of ROIs are given by thefollowing rule. The labeling rule of abnormal areas is thesame as condition 1, that is, when 60% or more area of
each ROI image contains abnormal, it is regarded asabnormal. In the case of the labeling of normal, ROIs
without containing any abnormal areas are regarded as
normal. In condition 1, the threshold that separates thenormal and abnormal ROIs is 60%, which means thateven if the ROIs contains 59% abnormal areas, theyshould be classified as normal. To clearly separate the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

578

Landslide Area Detection from

Table 4. The number of ROI images in condition 2

Table 2. Results obtained by CAE + PCA + OCSVM(Conventional method) in condition 1

Training/Testing
Training (normal only)
Testing (normal)
Testing (abnormal)

Evaluation
metric [%]

ν
0.2

0.3

0.4

0.5

0.6

0.7

Accuracy

53.4

54.9

55.6

55.7

49.4

47.4

Sensitivity

23.4

32.1

41.1

51.1

56.1

63.8

Specificity

83.4

77.8

70.1

60.3

42.7

31.1

The number of images
7967
164
164

Table 5. Result obtained by CAE + PCA + OCSVM(Conventional method) in condition 2
ν

Evaluation
metric [%]Table 3. Results obtained by CAAE + PCA + OCSVM(Proposed method) in condition 1

Evaluation
metric [%]

ν
0.2

0.3

0.4

0.5

0.6

0.7

Accuracy

55.9

56.6

57.9

57.4

54.4

52.4

Sensitivity

24.5

31.5

37.1

39.4

41.9

47.2

Specificity

87.3

81.6

78.7

75.4

66.9

57.6

0.2

0.3

0.4

0.5

0.6

0.7

Accuracy

61.4

60.4

61.2

59.8

58.7

56.2

Sensitivity

39.5

47.7

56.6

64.3

68.1

73.3

Specificity

83.3

73.1

65.8

55.5

49.3

39.1

Table 6. Result obtained by CAAE + PCA + OCSVM(Proposed method) in condition 2

features of normal and abnormal, condition 2 adopts theabove labeling rule. Table 4 shows the numbers of
training and testing data. The procedure of executing
CAAE, PCA and OCSVM is the same as condition 1.
Table 5 shows the results of the conventional method andTable 6 shows those of the proposed method. The best
accuracy obtained by the conventional method is 61.4%(ν=0.2) and that obtained by the proposed method is
66.0% (ν=0.5); thus we can see that the accuracy
becomes better than condition 1 and the accuracy
obtained by CAAE is better than that by CAE.

Evaluation
metric [%]

ν
0.2

0.3

0.4

0.5

0.6

0.7

Accuracy

64.0

64.8

65.5

66.0

62.7

59.6

Sensitivity

39.6

44.4

50.0

54.8

56.9

60.6

Specificity

88.3

85.1

80.9

77.2

68.5

58.5

abnormal, other method such as segmentaion models
would be useful. In addition, the proposed method shouldbe evaluated on other areas of disasters to confirm thegeneralization ability.

5. Conclusions
In this paper, we proposed an anomaly detection method
for classifying normal (non-disaster) areas and abnormal(disaster/landslide) areas, where the proposed method
consists of CAAE, PCA, and OCSVM. In theexperiments, we evaluated the detection performance inthe two conditions. From the results, it was clarified that
CAAE is better than CAE as a feature extraction method.
The remaining problem is the classification of ROI
images that contain small abnormal areas. Since it isdifficult to make a sharp boundary between normal andAcknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP20H02417.
References
1. T. Lillesand, R. W. Kiefer, J. Chipman, Remote sensing
and image interpretation, John Wiley & Sons, 2014

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

579

Shingo Mabu, Soichiro Hirata, Takashi Kuremoto

2. Y, Liu and L. Wu, Geological disaster recognition on
optical remote sensing images using deep learning,
Procedia Computer Science 91, 2016, pp. 566-575
3. H. Shahabi, M. Hashim, Landslide susceptibility mapping
using gis-based statistical models and remote sensing datain tropical environment, Scientific reports 5, 9899, 2015
4. N. Casagli, V. Tofani, A. Ciampalini, F. Raspini, P. Lu, S.
Morelli, Txt-tool 2.039-3.1: Satellite remote sensing
techniques for landslides detection and mapping,Landslide Dynamics: ISDR-ICL Landslide Interactive
Teaching Tools, Springer, 2018, pp. 235-254
5. P. Liu, Y. Wei, Q. Wang, Y. Chen, J. Xie, Research on
post-earthquake landslide extraction algorithm based on
improved u-net model, Remote Sensing 12(5), 894, 2020
6. A. Makhzani, J. Shlens, N. Jaitly and I. J. Goodfellow,
Adversarial Autoencoders, International Conference on
Learning Representations, 2016
7. L. M. Manevitz and M. Yousef, One-Class SVMs forDocument Classification, Journal of Machine Learning
Research, 2001, pp. 139-154
8. J. P. Fitch, Synthetic aperture radar, Springer Science &Business Media, 2012
9. K. Ouchi, Principles of synthetic aperture radar for
remote sensing, Tokyo Denki University Press, 2009 (inJapanese)
10. S. Mabu, Y. Nakayama and T. Kuremoto, Landslide
Classification from Synthetic Aperture Radar Images
Using Convolutional Neural Network with Multichannel
Information, Journal of Signal Processing, 24(2), 2020, pp.
61-73
11. C. Atzberger, Advances in remote sensing of agriculture:
Context description, existing operational monitoring
systems and major information needs, Remote Sensing,
5(2), 2013, pp.949-981
12. E. C. Barret, Introduction to environmental remote sensing,Routledge, 2013
13. D. Kingma and J. Ba, Adam: A method for stochastic
optimization. arXiv preprint arXiv:1412.6980, 2014

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

580

Anomaly Detection in Time Series Data Using Support Vector Machines
1

Umaporn Yokkampon1, Sakmongkon Chumkamon1, Abbe Mowshowitz2 , Eiji Hayashi1

Department of Computer Science and Systems Engineering, Kyushu Institute of Technology, 680-4 Kawazu,
Iizuka, Fukuoka 820-8502, Japan2
Department of Computer Science, The City College of New York,160 Convent Avenue, New York, NY 10031, USA
E-mail: may@mmcs.mse.kyutech.ac.jp, m-san@mmcs.mse.kyutech.ac.jp,
amowshowitz@ccny.cuny.edu, haya@mse.kyutech.ac.jp
www.kyutech.ac.jp

Abstract
Analysis of large data sets is increasingly important in business and scientific research. One of the challenges in suchanalysis stems from uncertainty in data, which can produce anomalous results. In this paper, we propose a method of
anomaly detection in time series data using a Support Vector Machine. Three different kernels of the Support Vector
Machine are analyzed to predict anomalies in the UCR public data set. Comparison of the three kernels shows thatthe defined parameter values of the RBF kernel are critical for improving the validity and accuracy in anomaly
detection. Our results show that the RBF kernel of the Support Vector Machine can be used to advantage in detecting
anomalies.
Keywords: Anomaly detection, Support Vector Machine, Data mining, Factory automation.

also cause sensor networks to generate erroneous data.
When a sensor malfunctions, it is unable to capture data
correctly and thus may produce anomalies. Abnormalchanges in data sources may also result in anomalies [2].
Anomaly detection typically uses data mining andmachine learning techniques to detect abnormal activities
in systems. Over the past decade, many anomaly
detection techniques have been developed, including
Support Vector Machines (SVM), a supervised machinelearning algorithm that can be used for classification or
to solve regression problems. In practice, the SVM
algorithm is applied with the kernel that transforms an
input data space into the required form. Kernel function
and kernel parameters affect the performance of SVM.
The selection quality of SVM parameters and kernel
functions has an effect on learning and generation
performance. Appropriate kernel function and associatedparameters should be selected to obtain optimal

1. Introduction
Research on anomaly detection is of great interest in
machine learning and data mining. Detecting anomalies
or finding outliers involves identifying abnormal orinconsistent patterns in a dataset. Abnormal data often
results from unauthorized activity. Credit card fraud
offers a well known example. Transactions with a stolen
or fake credit card can produce suspicious data. A fakecard can be made by copying information from an
authorized card and using it to create a new unauthorized
one. Data such as personal identifying information may
be obtained through phishing or from employees who
work in credit card companies [1]. Another source ofabnormal data may derive from unauthorized intrusions
in networks. Abnormal traffic or user actions arecommon signs of intrusions, which may occasion
breaches of sensitive or confidential data. Intrusions may© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

581

Umaporn Yokkampon, Sakmongkon Chumkamon, Abbe Mowshowitz, Eiji Hayashi

In 2015, Yu et al. [7] proposed a prediction model of bus
arrival time based on SVM and a forgetting factor. The
actual time of bus arrival at each time point is predictedby taking account of the time, weather and certain
historical data as input vectors. A k-Nearest Neighbor
Model forMultiple-Time-Step Prediction was
introduced to predict short-term traffic conditions.
Moreover, the Grubbs’ test method was applied toremove outliers from the input data.

classification performance. When an appropriate kernel
function and parameters are selected, the prediction errorof SVM can be minimized.
This paper reports on application of the support vector
machine method to eight real world time series data sets
to detect anomalies using three different kernels foranalysis and prediction. In addition, SVM kernels arecompared for effectiveness based on AUC, Precision,
Recall, and F1-Score criteria.
2. Related Work
Anomaly detection is widely used in many fields, andvarious methods have been proposed over the years.
In 2003, Ma et al. [3] used one class SVMs for prediction
which require a set of vectors as input instead of a timeseries. They convert the time series into a phase-space
using a time-delay embedding process, i.e., by creating
overlapping subsequences from a given long sequence.
These vectors are projected into an orthogonal subspacethat acts as a high pass filter to filter out the low
frequency components and allow only high frequency
ones (anomalies).
In 2005, Kim and Cha [4] tested the effectiveness of
SVM in detecting masquerade activities. Theirexperiments showed that their model could detect thistype of attack with an accuracy of 80.1%., thus
empirically demonstrating that SVM offers an effective
method for masquerade detection.
In 2008, Sugumaran et al. [5] developed fault diagnosticsof roller bearings using a neighborhood score multiclass
SVM in EDM machining. The roller bearing is one of themost widely used elements in a rotary machine. RBF isused as a kernel function. This research used a kernel
based neighborhood score multiclass SVM forclassification and a decision tree for addressing the future
selection process. The study of a multiclass SVM showed
its effectiveness in diagnosing the fault conditions of the
bearing.
In 2012, Caydas and Ekici [6] applied SVM to develop
prediction models for surface roughness in the turning
process of AISI 304 austenitic stainless steel. The
relevant parameters are cutting speed, feed rate and depthof cut using RBF as a kernel function. Three different
SVM models were developed, namely, LS-SVM, spider
SVM and SVM-KM. Spider SVM offered the best
prediction model for surface roughness.

Fig. 1. Structure of anomaly detection in time series data used
SVM. We used eight time series data sets processed by SVM,
and three different kernels based on AUC, Precision, Recall andF1-Score criteria.

3. Support Vector Machine Algorithm
The Support Vector Machine or SVM is one of the mostpopular Supervised Learning algorithms, which is used
for classification as well as regression problems. The
SVM algorithm's goal is to create the best line or decisionboundary that can decompose an n-dimensional space
into sets supporting categorization of new data points.
Hyperplanes define the boundaries in this space.
For a given dataset x with a number i of training data,
SVM finds the maximum margin hyperplane separating
different classes of data [8]:=x=( xi , yi ) , xi ∈  p , yi ∈ {−1,1} , ∀ i 1, 2,..., N (1)where xi is the p -dimensional input vector and yi is
the output value (1 or −1). A decision vector separating
two classes is given by:

wT ⋅ x + b =0(2)

where wT is the optimal weighting vector and b is thebias. For linearly separable training data, margins aredefined as:© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

582

Anomaly Detection in TimewT ⋅ x + b =1 and wT ⋅ x + b = − 1

4.2. Radial Basis Function (RBF) kernel(3)k ( xi , x j ) = exp(− xi − x j 2 (2σ 2 ))The distance between the margins is given by 2 wT .
Hence, the objective function is to minimize wT . In
practice, it is not easy to linearly decompose the training
dataset. Let C be the regularization parameter thatdefines the separation of two classes and the error whenusing a training dataset. The hyperplane is determined byminimizing:NC ∑εi +
i =1

1 2
w
2

The two kernels have their own advantages andlimitations. The linear kernel offers ease of performanceas well as an ability to deal with small and linearly
separable samples. The RBF kernel, on the other hand, isknown as a good mapping function since it can be used
for all kinds of samples, small or large with both high andlow dimensions [9]. The SVM performance for eachkernel will be evaluated in this study to determine theoptimal kernel.

(4)

1,..., N where ti iswith constraints ti y ( xi ) ≥ 1 − ε i , i =the target value and ε i is the set of slack variables.

5. Experiments

Instead of employing a minimization model (4), theproblem may be formulated using Lagrangian dual
multipliers α as:Nmax ∑ α i −
=i 1

1 N N
∑∑ α iα j yi y j ( xi , x j )2=i 1 =j 1

This section introduces the data sets and the evaluation
metric employed. We have compared the three kernels
and evaluated their effectiveness for anomaly detectionin Support Vector Machines.

(5)5.1. Data sets

subject to:n

0 ≤ αi ≤ C ∀ i =1, 2,..., n and ∑ α i yi = 0

Time series data obtained from UCR public data set [10]were used to evaluate effectiveness. The details of thedatasets are given in Table I. All datasets are presented intime series form, and every data point is manually labeled.
For all datasets, we designated the minority class as ananomaly class. Twenty percent of the data was used fortesting.

(6)

i =1

Using the “kernel trick” reduces the complexity of the
optimization problem that now only depends on the input
space instead of the feature space:
The objective function for SVM with nonlinear kernel
has the form:Nmax ∑ α i −
=i 1

1 N N
∑∑ α iα j yi y j k ( xi , x j )2=i 1 =j 1

Table 1. Summary of the datasets(7)

4. Kernels
Kernel methods form a popular class of machine learningtechniques for various tasks. An important featureoffered by kernel methods is the ability to model complex
data through the use of the kernel trick (Schölkopf &Smola, 2002).
In this paper, two types of kernel functions are chosen
and evaluated, namely the linear and radial basis function(RBF). The mathematical formula for the said functions
are as follows:

Datasets

LengthItalyPowerDemand
Wafer
SonyAIBORobotSurface2
ECGFiveDays
TwoLeadECG
MoteStrain
Herring
Strawberry

24
152
65
136
82
84
512
235

Number of
instances
1096
7164
980
884
1162
1272
128
983

Anomaly
Ratio
0.49
0.11
0.38
0.50
0.50
0.46
0.40
0.36

5.2. Performance Evaluation
The accuracy of an anomaly detection method is
evaluated using the area under the curve (AUC) of thereceiver operating characteristic (ROC), Precision (Pre),
Recall (Rec), and F1-Score, defined as follows:

4.1. Linear kernelk ( xi , x j ) = ( xi , x j )

(9)

(8)

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

583

Umaporn Yokkampon, Sakmongkon Chumkamon, Abbe Mowshowitz, Eiji Hayashi

TPTP + FPTPRec =TP + FN

Pre =

F1=

2×

Pre×Rec
Pre + Rec

(10)
(11)
(12)

where TP is the correctly detected anomaly, FP is thefalsely detected anomaly, TN is the correctly assigned
normal, and FN is the falsely assigned normal.
6. Results and Discussion
The efficiency of the following three SVM kernels
are compared:
1. Linear Kernel
2. RBF Kernel (Default parameters value)
3. RBF1 Kernel (We define the parameters C = 20,
γ = 0.02)

Fig. 3. The kernel performance comparison of Linear, RBF
and RBF1 for testing Wafer dataset using ROC.

Fig. 3. shows that the RBF and RBF1 kernels are
more efficient than the Linear kernel, and that the RBF

We performed experiments on accuracy of analysis
and prediction of anomalies for eight time series data setsusing the three different SVM kernels. Accuracy ofanalysis and prediction can be measured by the AUC as
shown in the ROC in Fig. 2 - 9. The blue line is the Linear

Fig. 4. The kernel performance comparison of Linear, RBF
and RBF1 for testing SonyAIBORobotSurface2 dataset using
ROC.

Fig. 2. The kernel performance comparison of Linear, RBF
and RBF1 for testing ItalyPowerDemand dataset using ROC.

Kernel of SVM, the orange line is the RBF kernel, andthe green line is the RBF1 kernel.
Fig. 2. shows that the RBF1 kernel is slightly more
efficient than the Linear and RBF kernels for the
ItalyPowerDemand data set. However, all three kernels
yield almost 100 percent accuracy.
Fig. 5. The kernel performance comparison of Linear, RBF
and RBF1 for testing ECGFiveDays dataset using ROC.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

584

Anomaly Detection in Time

Fig. 7. shows that the RBF1 kernel to be slightly more
efficient than the linear and RBF kernels for the

kernel is almost 100 percent accurate. In particular, theRBF1 kernel gives a ROC value perfectly for the Wafer
data set.
Fig. 4. reveals that RBF1 is slightly more efficientthan the linear and RBF kernels for the
SonyAIBORobotSurface2 data set. In particular, theRBF and RBF1 kernels provide almost 100 percent
accuracy.

Fig. 8. The kernel performance comparison of Linear, RBF
and RBF1 for testing Herring dataset using ROC.

Fig. 6. The kernel performance comparison of Linear, RBF
and RBF1 for testing TwoLeadECG dataset using ROC.

Fig. 5. shows that all three kernels give perfect ROC
values for the ECGFiveDays data set.
Fig. 6. shows that the RBF kernel is almost 100

Fig. 9. The kernel performance comparison of Linear, RBF
and RBF1 for testing Strawberry dataset using ROC.

MoteStrain data set.
Fig. 8. reveals that the Linear kernel gives slightly
more accurate ROC values than does the RBF kernel, but
the RBF1 kernel is the most accurate for the Herring data
set.
Finally, Fig. 9. shows that the Linear kernel is moreaccurate than the RBF kernel. In particular, the RBF1
kernel is almost 100 percent for the Strawberry data set.
The anomaly detection results and comparisons are
summarized in Table II. The results show that SVM with
RBF1 kernel gives the highest accuracy and F1-Score on
all aspects and data sets, except for the Herring data set,
for which the highest of F1-Score is given by the RBF
kernel. All three kernels gave perfect results for AUC,

Fig. 7. The kernel performance comparison of Linear, RBF
and RBF1 for testing MoteStrain dataset using ROC.

percent. Linear and RBF1 kernels give nearly perfect
ROC values for the TwoLeadECG data set.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

585

Umaporn Yokkampon, Sakmongkon Chumkamon, Abbe Mowshowitz, Eiji Hayashi

References

Precision, Recall and F1-Score on the ECGFiveDays data
set.
This shows that the RBF kernel with parametervalues C = 20, γ = 0.02 exhibits good performance inanomaly detection for time series data.

1. E. Hormozi, M. Kazem Akbari, H. Hormozi, Accuracy
evaluation of a credit card fraud detection system on
Hadoop MapReduce, In The 5th conference on
information and knowledge technology, 2013, pp. 35-39.

Table 2. Summary of the kernel performance comparison of Linear, RBF, and RBF1.
Kernel : Linear

Datasets

AUC

Kernel : RBF (Default)

Kernel : RBF ( C = 20, γ = 0.02)

Precision Recall

F1-Score

AUC

Precision Recall F1-Score

AUC

Precision Recall F1-Score

0.9994ItalyPowerDemand
0.7510
Wafer
SonyAIBORobotSurface2 0.9819
1.0000ECGFiveDays
1.0000
TwoLeadECG
0.9540
MoteStrain
0.6667
Herring

0.98180.96870.96461.00000.99150.95520.76470.98180.9946
0.94781.00001.00000.87670.76470.98180.98150.95611.00000.99570.91430.76470.99900.99990.99741.00000.99940.97890.62750.99101.00000.99101.00000.99150.97830.65381.00000.99690.95651.00001.00000.92471.00000.99550.99850.97351.00000.99570.95070.79070.99991.00000.99971.00001.00000.98400.72550.99101.00001.00001.00001.00000.9716
0.76921.00000.99850.98261.00001.00000.9384
0.58820.99550.99920.99121.00001.00000.9547
0.66670.98840.99080.87100.92700.9366

0.63271.00000.77500.9904

0.99110.89520.9407

Strawberry

7. Conclusion

2. Varun Chandola, Arindam Banerjee, and Vipin Kumar,
Anomaly detection: A survey, ACM computing surveys(CSUR). 41(3), 2009.
3. J. Ma and S. Perkins, Time-series novelty detection using
one-class support vector machines, In Proceedings of theInternational Joint Conference on Neural Networks, 2003,
vol. 3, pp. 1741-1745.
4. H.-S. Kim and S.-D. Cha, Empirical evaluation of SVMbased masquerade detection using UNIX commands,
Computers & Security, 24(2), 2005, pp. 160-168.
5. V. Sugumaran, G. R. Sabareesh, K. I. Ramachandran,
Fault diagnostics of roller bearing using kernel basedneighborhood score multi-class support vector machine,
Expert Systems with Applications, 34(4), 2008, pp. 30903098.
6. U. Caydas and S. Ekici, Support vector machines models
for surface roughness prediction in CNC turning of AISI
304 austenitic stainless steel, Journal of intelligent
Manufacturing, 23(3), 2012, pp. 639-650.
7. B. Yu, X. Song, F. Guan, Z. Yang, and B. Yao, k-Nearest
Neighbor Model for Multiple-Time-Step Prediction of
Short-Term Traffic Condition, Journal of Transportation
Engineering, 142(6), 2016.
8. P. Batta, M. Singh, Z. Li, Q. Ding, and L. Trajkovic,
Evaluation of Support Vector Machine Kernels for
detecting network anomalies, In 2018 IEEE International
Symposium on Circuits and Systems, 2018, pp. 1-4.
9. W. Cao, L. Li and X. LV, Kernel function characteristic
analysis based on support vector machine in facerecognition, In 2007 International Conference on Machine
Learning and Cybernetics, 2007, vol. 5, pp. 2869-2873.
10. H. A. Dau, E. Keogh, K. Kamgar, C.-C. M. Yeh, Y. Zhu,
S. Gharghabi, C. A. Ratanamahatana, Yanping, B. Hu, N.
Begum, A. Bagnall, A. Mueen, and G. Batista, “The ucr

In this paper, we presented an analysis of anomaly
detection in time series data using a Support Vector
Machine with three different kernels, namely, Linear,
RBF and RBF1. We evaluated the accuracy of anomaly
detection methods based on AUC, Precision, Recall andF1-Score criteria. The evaluation results show that thekernel with defined parameters can improve accuracy on
all aspects and data sets. This application of the Support
Vector Machine method, with the RBF kernel, can be
efficient for detecting anomalies in time series data. The
results for data set ECGFiveDays show 100 percent
accuracy with all three kernels, and the results for the
TwoLeadECG show almost 100 percent with all threekernels. Moreover, the results indicate a high degree of
accuracy for the three kernels on all the data sets, perhaps
because our data was trained in supervised conditions.
In the future, we intend to implement the variational
autoencoder method for detecting and predicting
anomalies in time series and spectrum data in order tocompare it with the autoencoder method.
Acknowledgements
This research was partially supported by a Japanese
Government Project in collaboration with the Kyushu
Institute of Technology, Yaskawa Electric Corporation,
Kitakyushu Foundation for the Advancement of Industry,
Science and Technology, and the Hayashi Laboratory.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

586

Anomaly Detection in Timetime series classification archive,” November 2019,https://www.cs.ucr.edu/∼eamonn/time series data 2018/.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

587

Relationship Between Tactile Stimuli and Human Body Sway
Masaya Tadocoro, Taro Shibanoki

Ibaraki University, Hitachi, Ibaraki, Japan

Abstract

This paper describes the relationship between tactile stimulation and human body sway. We previously
proposed a body sway mitigation system based on tactile stimulation and revealed that simultaneous
stimulation behind both auricles were significantly improve human balance function, however, themechanism of it were not fully discussed. In this paper, some stimulation patterns were applied toparticipants and COP distribution before/after stimulation was extracted. The results showed that COP
values after stimulation could be inclined to the opposite of the stimulation site. It indicates that tactile
stimuli can control human balance function.
Keywords: balance function, body sway, tactile stimulation, center of pressure

Previous studies have not fully clarified the relationshipbetween tactile stimulation and body sway.
In this paper, we attempt to quantify the influence ofvibration stimulation on body sway by clarifying therelationship between vibratory stimulation near theauricle and body sway deflection. In the experiment,
regular on-off stimulation patterns are applied to near theauricle of the subject, and the possibility of inducing
center of pressure (COP) sway by the stimulation site is
discussed.

1. Introduction
According to a survey by the Tokyo Fire Department,
falls account for about 80 percent of accidents1, andprevention of falls is an urgent issue in Japan, where thepopulation is rapidly aging2.It is said that the decline of
motor function with aging increases the risk of falls by
about five times3, and various balance function
improvement and fall prevention systems have beendeveloped4-7.
Studies focusing on improving balance function canbe divided into two categories: those that use light touchcontact (LTC) 4 and those that use stimulation to thehuman body to improve vestibular and somatosensory
fuctions5-7. In LTC, it has been shown that the balance
function can be improved by touching a fixed point. It
has also been shown that it is possible to decrease andcontrol body sway by applying acoustic5, electrical6 andvibratory7 stimuli. However, because these approaches
may exert a physical burden on the use, it is difficult to
use in a daily life.
To overcome these problems, our research group has
been proposed the body sway mitigation system based ona tactile stimulation and showed that body sway can be
reduced via the application of vibratory stimuli aroundthe pinna8. Here, acoustic stimulation can induce bodysway to the opposite direction of the stimulus5, however,

2. Method
Figure 1 shows the proposed vibratory stimulationbalance function analysis method based on the bodysway mitigation system using tactile stimulation8. The
system consists of a stabilometer, tactile stimulators, amicrocontroller for controlling vibratory stimulation
pattern, and a PC. Stimulation patterns can be changed
from the PC via TCP/IP communication. Figure 2
shows an example of stimulation pattern. The system
can change the on-off interval 𝑇𝑇𝑠𝑠 [s], stimulationinterval 𝑇𝑇on [s], and non-stimulation interval 𝑇𝑇off [s] ofeach tactile stimulator via PWM control. The details of
the proposed vibratory stimulation-balance function
analysis are shown below.
The subjects were asked to maintain an eye-closed
tandem limb stance for T [s]. Tactile stimulators were

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

588

Masaya Tadocoro, Taro Shibanoki

tandem limb stance with eye-closed and the left leg backfor 𝑇𝑇 = 90[s] after a preparation time for 10 [s]. The
COP was measured at a sampling frequency of 100 [Hz]using a Wii Balance Board (Nintendo Co., Ltd.). The
stimulus pattern applied to the subject was steady-state
stimulus with 𝑇𝑇𝑠𝑠 = 10 [s], stimulus intensity 𝐷𝐷max =100 [%] during 𝑇𝑇on = 5 [s], and stimulus intensity
𝐷𝐷min = 0 [%] during 𝑇𝑇off = 5 [s]. This stimulus pattern
was applied to left or right side.
The number of trials was twenty per side, and ten trials
per side were conducted per day with an interval of one
minute. The other ten trials were conducted on different
days.

Fig. 1. Overview of the proposed balance function
analysis system.

3.2. Results and discussion
Figure 3 (a) shows examples of signals measured
during the experiments. Figure 3 (a) shows the resultsfrom the first trial, indicating stimulation patterns and
COPs. The shaded areas represent times during nonstimulations. Figure 3 (b) is an enlargement of COPs
from Fig. 3 (a), and it cannot be confirmed that thestimulus affected to the balance function.
Figure 4 (a) and (b) show histograms of COP of y axis
(frontal plane) with median and mean values for the leftside and right-side stimulus conditions, respectively.
Figure 4(a) of the results for the left-side stimulus
condition shows that the median is smaller than the meanvalue in the stimulus interval, and conversely, the medianis larger than the mean value in the non-stimulus interval.
In the Fig. 4, positive value shows deviation to the left

Fig. 2. An example of the stimulation pattern.
applied near the left and right auricles to provide
vibration stimulation to the subjects. The stimulus
intensity was the maximum stimulus intensity 𝐷𝐷maxduring 𝑇𝑇on and 𝐷𝐷min during 𝑇𝑇off , which was less than orequal to the minimum stimulus intensity perceived by
the subject.
In the experiment, the stimulus was applied to only
one of the left and right sides. The time series waveform
COP{𝑥𝑥,𝑦𝑦} (𝑡𝑡) of each axis of the measured center ofpressure (COP) was smoothed by a second-order
Butterworth digital low-pass filter (cut-off frequency:
𝑓𝑓𝑐𝑐 [Hz]), and deviation from the median of each T [s]was calculated. In this method, filtered COP value
COP{𝑥𝑥,𝑦𝑦} (𝑡𝑡) was divided into M every 𝑇𝑇𝑠𝑠 [s], and theerror between median values med𝑚𝑚 in eachinterval (𝑚𝑚 = 1, 2, … , 𝑀𝑀) and mean values aveon
𝑚𝑚 forstimulation interval and aveoff
fornon-stimulation𝑚𝑚interval were evaluated. This clarifies the bias of bodysway with and without vibratory stimulation.
3. Experiments
3.1. Experimental conditions
In the experiments performed, COP sway during theapplication of vibratory stimulation was measured with
one healthy male (23 [years]). Two motor oscillators(KD18B1) were connected to a Raspberry Pi 3B and a
voltage was applied by pulse width control with a duty
cycle of 0.1 [s]. The subject was asked to maintain© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

589

Relationship Between Tactile Stimuli(a) Left stimulus(b) Right stimulus

Fig. 4. Examples of COP distributions.
distribution. From the result, it was confirmed that the
distribution of COP was deviated to the right during leftside stimulation and to the left during non-stimulation.
The results of the heteroskedasticity t-test confirmed asignificant difference at the level of 0.1 [%] (see Fig. 6(a)). On the other hand, when right stimulation wasapplied, there was a bias to the left side during
stimulation (p < 0.05), and a bias to the right side during
non-stimulation (p<0.05) (see Fig. 6 (b). These results
suggest that COP bias in the left-right direction may be
induced to the opposite side of the stimulated side. It is
possible that COP bias is induced to the opposite side of
the stimulus, resulting in a larger bias toward the stimulus
to maintain body balance equilibrium.
These results indicate that regular unilateral steadystimulation to the auricle induces COP sway on the
opposite side of the stimulus, and that the sway may be
deflected to the stimulus side to resist it. In the future, wewill increase the number of subjects and investigate
whether this tendency appears or not, and also whether it
is possible to induce body sway arbitrarily by changing
the vibration pattern or not.

(a) COP in each direction(b) An enlargement of COPs
Fig.3. Examples of experimental results.

4. Conclusionside, therefore, the subject may tilt the opposite side of
the stimulation.
Figure 5 shows the difference between mean values
during stimulation and non-stimulation and the median in
the interval of 10 [s] for (a) left-side stimulation
condition and (b) right-side stimulation condition,
respectively. The positive component represents the bias
to the right side of COP distribution, and negative
component represents the bias to the left side of COP

In this paper, we examined the effects of regularunilateral steady-state stimulation patterns on COP bias
when they were applied to near pinna. In the experiment,
the subject was asked to maintain tandem limb positions
with closed eyes, and the change of COP deviation withand without stimulation in each stimulation pattern wascompared. It is confirmed that the stimulation may
induce the deviation of COP oscillation in the frontal
plane to the opposite side of the stimulated side. This

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

590

Masaya Tadocoro, Taro Shibanoki(a) Left stimulus(b) Right stimulus

Fig. 6. Comparison of mean and median differencesin COPs.
(a) Left stimulus

Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number 20K20212.
References
1. Tokyo Fire Department: https://www.tfd.metro.tokyo.
lg.jp/lfe/topics/202009/kkhansoudeta.html
2. Statistics Bureau, Japan STATISTICAL HANDBOOK
OF JAPAN 2020
3. Masdeu J.C., Sudarsky L., Wolfson L., “Gait Disorders ofAging: Falls and Therapeutic Strategies”, Philadelphia,
Lippincot -Raven, pp. 309–326, 1997
4. M.A. Riley, T.A. Stoffregen, M.J. Grocki, M.T. Turvey:
Postural stabilization for the control of touching, Human
Movement Science, Vol. 18, No. 6, pp. 795–817, 1999.
5. Miriam S. Welgampola and Brian L. Day, Craniocentric
body-sway responses to 500Hz bone-conducted tones inman, J Physiol 577.1, pp 81-95, 2006
6. Chisato Fujimoto, Yoshuharu Yamamoto, Teru
Kamogashira, Makoto Kinoshita, Naoya Egami, Yukari
Uemura, Fumiharu Togo, Tatsuya Yamasoba and Shinichi
Iwasaki, Noisy galvanic vestibular stimulation induces asustained improvedment in body balance in elderly adults,
SCIENTIFIC REPORTS, 2016
7. W. Nanhoe-Mahabier, J.H. Allum, E.P. Pasman, S.
Overem, B.R. Bloem, The effects of vibrotactile
biofeedback training on trunk sway in Parkinson disease
patients, Parkinsonism and Related Disorders Vol. 18,
pp.1017-1021, 2012
8. Satowa Watanabe, Taro Shibanoki, Koji Shimatani, A
Body Sway Mitigration Method Based on Tactile
Stimulation, Proceedings of the SICE Annual Conference
2016, pp.518-19, 2016(b) Right stimulus
Fig. 5. The difference between the median and eachmean value in the stimulation and non-stimulationintervals.
suggests that the amount of deflection to the stimulus side
may increase to maintain the equilibrium against theinduced COP oscillation.
In future research, we will increase the number ofsubjects and deepen the verification of the experimentconducted in this study. In addition, we plan to change
the pattern of the vibratory stimulation and investigate
the relationship with the body sway.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

591

Analysis of the Consensus Protocol of Heterogeneous Agents with Time-Delays
Jichao Zhao 1, Fengzhi Dai 1,3 *, Yunzhong Song 2
1

College of Electronic Information and Automation, Tianjin University of Science and Technology,
China,

2

School of Electrical Engineering and Automation, Henan Polytechnic University, 454003, China,
3

Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China
China E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

In practical engineering applications, the research on the consensus of heterogeneous multi-agents is ofgreat significance. The consensus of multi-agents mainly includes average consensus, maximum
consensus, and minimum consensus. The paper studies the average consensus of heterogeneous multiagents, including continuous-time consensus protocol, discrete-time consensus protocol, consensus withtime delay, and consensus of switching topology. The knowledge of graph theory is used to describe thesystem, and the results are simulated and analyzed for the consensus of time delay and switching topology
to verify the correctness and effectiveness of the consensus protocol.
Keywords: Heterogeneous Multi-Agent; Consensus Protocol; Switching Topology; Time-Delays

scholars to put into the research work in this field andfurther explore the theory and application of MAS.
MAS has the characteristics of a large number ofsubjects, a wide distribution of perceptions, and complex
communication. There is often information interaction
between subjects. Therefore, it is natural to use graphtheory to describe this information interaction channel.
Consensus protocol is a rule of interaction between
agents, which reflects the process of informationexchange between adjacent agents.
In this paper, the validity of agent consensus protocol
and the relationship between consensus protocol andgraph Laplace matrix are verified by analyzing the stateof each node in MAS with inertial link. The relationshipbetween algebraic connectivity (Fiedler eigenvalue) of
Laplace matrix and convergence rate of MAS is verified
by simulation results.

1. Introduction
A multi-agent system (MAS) is a computing system
consisting of multiple robots interacting in anenvironment. Cooperative robots can be used to performtasks that are too diﬃcult for a single robot to perform
alone. In recent years, as a main branch of distributed
system with the development of computer technology
and network communication technology, MAS has been
widely developed in the fields of distributed sensor
network 1, network congestion control 2 and flocking
movement 3. The consensus control of MAS is a very
important research topic in distributed system.
At present, the research on MAS is still in the initialstage of development on the whole, and there is still a
certain distance from the real application. However, its
extensive application indicates great development
potential, which will definitely attract more experts and© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

592

Jichao Zhao, Fengzhi Dai, Yunzhong Song2. Preliminaries

3. Consensus Protocol

In this paper, the graph theory is used to represent theMAS. A graph 𝐺𝐺 = (𝑉𝑉, 𝐸𝐸, 𝐴𝐴) is used to represent the
topology of the information exchange network between
agents. 𝑉𝑉 = {𝑣𝑣1 , 𝑣𝑣2 , … , 𝑣𝑣𝑛𝑛 } indicates that the graph has
𝑛𝑛 nodes and that the system is composed of 𝑛𝑛 multiagents. 𝐸𝐸 ⊆ {(𝑖𝑖, 𝑗𝑗): 𝑖𝑖, 𝑗𝑗 ∈ 𝑉𝑉} is the set of edges about
𝑒𝑒𝑖𝑖𝑖𝑖 , indicating that there is information exchange between
agent 𝑖𝑖 and 𝑗𝑗, and the information is from 𝑖𝑖 to 𝑗𝑗. 𝐴𝐴 =[𝑎𝑎𝑖𝑖𝑖𝑖 ] is the adjacency matrix of the graph, indicating theweight of information exchange between 𝑖𝑖 and 𝑗𝑗. To
simplify calculation, let 𝑎𝑎𝑖𝑖𝑖𝑖 ∈ {0,1}. Note that 𝑎𝑎𝑖𝑖𝑖𝑖 = 𝑎𝑎𝑗𝑗𝑗𝑗
in the undirected graph, and 𝑎𝑎𝑖𝑖𝑖𝑖 ≠ 𝑎𝑎𝑗𝑗𝑗𝑗 in the directed
graph.
The set of neighbor node 𝑁𝑁𝑖𝑖 = {𝑗𝑗|𝑗𝑗 ∈ 𝑉𝑉: 𝑒𝑒𝑖𝑖𝑖𝑖 ∈ 𝐸𝐸} ,which represents the set composed of all agents that haveinformation exchange with agent 𝑖𝑖. In the graph, it isrepresented as the set composed of nodes connected to
the wired segment of node 𝑖𝑖 . 𝑑𝑑𝑑𝑑𝑑𝑑𝑜𝑜𝑜𝑜𝑜𝑜 (𝑣𝑣𝑖𝑖 ) , the out
degree of node 𝑖𝑖 is the number of edges starting fromnode 𝑖𝑖. 𝑑𝑑𝑑𝑑𝑑𝑑𝑖𝑖𝑖𝑖 (𝑣𝑣𝑖𝑖 ) , the into degree of node 𝑖𝑖 is thenumber of edges go into node 𝑖𝑖.
The matrix 𝐷𝐷 ∈ 𝑁𝑁𝑛𝑛×𝑛𝑛 is the degree matrix, is defined
as follows
𝐷𝐷𝑖𝑖𝑖𝑖 = 𝑑𝑑𝑑𝑑𝑑𝑑𝑜𝑜𝑜𝑜𝑜𝑜 (𝑖𝑖)�𝐷𝐷𝑖𝑖𝑖𝑖 = 0

The individual state of the dynamic MAS is described asfollows:
1
𝐾𝐾𝑖𝑖
𝑥𝑥̇ 𝑖𝑖 (𝑡𝑡) = − 𝑥𝑥𝑖𝑖 (𝑡𝑡) + 𝑢𝑢𝑖𝑖 (𝑡𝑡), 𝑖𝑖 = 1, 2, … , 𝑛𝑛. (1)
𝑇𝑇𝑖𝑖
𝑇𝑇𝑖𝑖

where 𝑇𝑇𝑖𝑖 , 𝐾𝐾𝑖𝑖 denote different coefficients of node 𝑖𝑖, and𝑇𝑇𝑖𝑖 ≠ 𝑇𝑇𝑗𝑗 , 𝐾𝐾𝑖𝑖 ≠ 𝐾𝐾𝑗𝑗 , ∀𝑖𝑖 ≠ 𝑗𝑗;
System to reach consensus, means that the system allindividual states are equal, mathematical representation,𝑥𝑥𝑖𝑖 = 𝑥𝑥𝑗𝑗 , ∀𝑖𝑖 ≠ 𝑗𝑗, 𝑖𝑖 = 1,2, … , 𝑛𝑛 or 𝑢𝑢𝑖𝑖 = 0, 𝑖𝑖 = 1,2, … , 𝑛𝑛.
When there is a directed spanning tree incommunication topology 𝐺𝐺 , information interaction
between agents can be realized to reach consensus.
Therefore, it is assumed that the system topology 𝐺𝐺
contains a directed spanning tree.
In particular, the consensus problem also includes
average consensus, maximum consensus and minimum
consensus. But it should be noted that the systemconsensus is equivalent to the average consistency if thegraph is undirected or directed symmetric. But if thegraph is asymmetric, the consensus of the system is not
equivalent to the consensus of the average 4.
3.1. Consensus Protocol of Continuous TimeFor continuous time systems (1), the sonsensus protocol
is as follows:

The graph laplacian matrix is defined as
𝐿𝐿 = 𝐷𝐷 − 𝐴𝐴𝑢𝑢𝑖𝑖 (𝑡𝑡) = � 𝑎𝑎𝑖𝑖𝑖𝑖 (𝑥𝑥𝑗𝑗 − 𝑥𝑥𝑖𝑖 )𝑗𝑗∈𝑁𝑁𝑖𝑖(2)

The agreement can be achieved if and only if theundirected graph is connected or the directed graph exists
spanning tree. In the case of topological structure
switching, the condition of consensus is that the union ofswitching topological graphs is undirected graphconnected or directed graph exists spanning tree. Specificproof can be found in the literature 5.
Refer to the topology of Fig.1, (2) can be described as:
𝑢𝑢1−2𝑥𝑥1 + 𝑥𝑥2 + 𝑥𝑥3
2 −1 −1 𝑥𝑥1
�𝑢𝑢2 � = � −𝑥𝑥2 + 𝑥𝑥3 � = − � 0
1 −1� �𝑥𝑥2 �𝑢𝑢3
𝑥𝑥1 − 𝑥𝑥3
−1 0
1 𝑥𝑥3

Fig.1. The main global currents

As shown in Fig.1, the system 𝐺𝐺 has three nodes andfour edges: 𝑉𝑉 = {𝑣𝑣1 , 𝑣𝑣2 , 𝑣𝑣3 }, 𝐸𝐸 = {𝑒𝑒12 , 𝑒𝑒23 , 𝑒𝑒31 , 𝑒𝑒13 } ,laplacian matrix 𝐿𝐿 equals
2 0 0
0 1 1
2 −1 −1
𝐿𝐿 = �0 1 0� − �0 0 1� = � 0
1 −1�
0 0 1
1 0 0
−1 0
1

Combined with the graph laplacian matrix, (2) can beconverted into the following form:(3)𝑢𝑢(𝑡𝑡) = −𝐿𝐿 ∗ 𝑥𝑥(𝑡𝑡)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

593

Analysis of the Consensus

3.2. Consensus Protocol of Discrete Timeby software tools, and the system convergence results
were drawn as shown in the Fig.3.

The discrete-time consensus protocol is as follows:𝑥𝑥𝑖𝑖 (𝑘𝑘 + 1) = 𝑥𝑥𝑖𝑖 (𝑘𝑘) + 𝜖𝜖𝑢𝑢𝑖𝑖 (𝑘𝑘)(4)𝜖𝜖 > 0 means step size, the length of time for each step.
3.3. Switching Topology Model

Since the topological network is not fixed, there is anadditional switching signal 𝑘𝑘 is a function of time 𝑡𝑡.
The protocol is as follows:(5)
𝑢𝑢𝑖𝑖 (𝑡𝑡) = −𝐿𝐿𝑘𝑘 ∗ 𝑥𝑥(𝑡𝑡),𝑘𝑘 = 𝑠𝑠(𝑡𝑡)Fiedler eigenvalue is the second smallest eigenvalue ofthe Laplace matrix of the graph, which is positively
correlated with the convergence rate of the graph. If thelarger the Fiedler eigenvalue is, the faster theconvergence rate of the system will be. On the contrary,
the same relationship exists.
3.4. Time-Delays Model

The consensus protocol with time delay is:𝑢𝑢𝑖𝑖 (𝑡𝑡) = � 𝑎𝑎𝑖𝑖𝑖𝑖 (𝑥𝑥𝑗𝑗 (𝑡𝑡 − 𝜏𝜏) − 𝑥𝑥𝑖𝑖 (𝑡𝑡 − 𝜏𝜏))𝑗𝑗∈𝑁𝑁𝑖𝑖

Among them, time delay 𝜏𝜏 has a threshold value of

(6)
𝜋𝜋

2𝜆𝜆𝑛𝑛

Fig.2. Topology graph of the system

In Fig.2, the order of switching topology is Ga-Gb-Gc,
when 𝑡𝑡 = 0, the system is topology A, when 𝑡𝑡 = 1, thesystem is topology B, and then c, and the loop is carriedout in this way.

.

For a network with the same time delay, assume 𝐺𝐺 is
undirected and connected. If and only if at the time delay𝜋𝜋, 𝜆𝜆𝑛𝑛 = 𝜆𝜆𝑚𝑚𝑚𝑚𝑚𝑚 (𝐿𝐿) , the system can𝜏𝜏 meet 𝜏𝜏 <2𝜆𝜆𝑛𝑛

asymptotically achieve consensus 4.

Tab.1. MAS Parameters

𝑖𝑖
1
2
3
4
5
6

4. Simulink Results
Given a MAS topology graph, as shown in Fig.2, thesystem has a total of 6 multi-agents, and the three
topologies 𝐺𝐺𝑎𝑎 , 𝐺𝐺𝑏𝑏 and 𝐺𝐺𝑐𝑐 all have the same number of
agents, but their topological structures are different.
Let the initial state of the system for each agent is 𝑥𝑥𝑖𝑖
as shown in Tab.1. Notice that the average of the initialstate of the system is 𝐴𝐴𝐴𝐴𝐴𝐴(𝑋𝑋0 ) = 5 ≠ 0 . Then, thedynamic equation of each node in the MAS is established.
The dynamic equation parameters 𝑇𝑇𝑖𝑖 and 𝐾𝐾𝑖𝑖 are shownin Tab.1, and the parameters obey the random Gaussian
distribution. The system state differential equation wasestablished, the system differential equation was solved

𝑥𝑥𝑖𝑖 (0)-20.5030.25-13.25
24.75
-7.25
16.00

𝑇𝑇𝑖𝑖
0.81
0.27
0.95
0.79
0.67
0.70

𝐾𝐾𝑖𝑖
0.91
0.55
0.49
0.96
0.76
0.03

Fig.3 shows the input-free convergence graph of theMAS, the fixed topology convergence and the switched
topology convergence of the system. When there is no
input, the state of the system changes with its inertial link.
When input is available, Fiedler eigenvalues of topology
𝐺𝐺𝑎𝑎 , 𝐺𝐺𝑏𝑏 and 𝐺𝐺𝑐𝑐 increase in turn, and the convergencevelocity of the three topology graphs in the graph isobserved in turn, which can verify the positive© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

594

Jichao Zhao, Fengzhi Dai, Yunzhong Songcorrelation between Fiedler eigenvalue
convergence velocity of the system.

and

the

Fig.3. The result of input-free, fixed topology and switched topology

5. Conclusion

2.

In this paper, the consensus protocols of MAS are sortedout and each protocol is simply explained. An example isgiven to illustrate the relationship between multi-agent
system and Laplacian matrix in graph theory. The multiagent consensus protocol is simulated, and the protocol
is analyzed and verified by combining the knowledge ofgraph theory and system convergence graph.

3.

4.

References

5.

1. M. M. I. Khan et al. Efficient and fast convergent
consensus algorithms for faulty nodes tracking in
distributed wireless sensor networks. 3rd International

Conference on Electrical Engineering and Information
Communication Technology, Dhaka, 2016:1-6.
Y. Wang, Z. Ma et al. Avoiding Congestion in Cluster
Consensus of the Second-Order Nonlinear Multiagent
Systems. IEEE Transactions on Neural Networks andLearning Systems, 2018, 29(8): 3490-3498.
C. Bhowmick, L. Behera, et al. Flocking control of multiagent system with leader-follower architecture using
consensus based estimated flocking center. IECON 2016 42nd Annual Conference of the IEEE Industrial
Electronics Society, Florence, 2016:166-171.
Olfati-Saber R, Murray R M. Consensus problems in
networks of agents with switching topology and timedelays. IEEE Transactions on Automatic Control, 2004,
49(9):1520-1533.
Ren W, Beard R W. Distributed consensus in multivehicle cooperative control. London: Springer London,
2008.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

595

Burrows-Wheeler transform acceleration based on CUDA
Chang Sheng, Fengzhi Dai*
Tianjin University of Science and Technology, Tianjin, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
Burrows-Wheeler transform (BWT) is a commonly used transform in compression or text comparison. For example,
in bzip2, BWT is used to preprocess the original data, then the same characters in the original data are close to eachother, which improves the compression rate. Because the prefix tree of the original string can be easily obtained
from the result of the BWT, BWT is also applied to the search and comparison of strings. For instance, thecomparison of DNA sequences uses the BWT algorithm. However, BWT is not a fast algorithm, only tens of
megabytes per second on CPU. This article uses the GPU to sort the original string by the base of the 4-byte key
size radix sort. After radix sort, the part with insufficient length is sorted again to complete the BWT algorithm.
Keywords: BWT, acceleration, CUDA, GPU

1. Introduction

a string of length 6 "aabcg#", and then perform a circular
shift to get 6 strings, then sort, and the last column of thesorted result is the transformed result "g#aabc", as shownin Table 1.

Burrows-Wheeler transform (BWT) is a datacompression algorithm proposed by Burrows andWheeler in 1994 1. It can be used before othercompression algorithms, such as MTF, Huffman, andRLE, so that these zero-order entropy codes can achieve
the effect of high-order entropy coding. Therefore, it canreplace the LZ77 sliding window search algorithm before
the general compression algorithm. Moreover, thanks to
the restoration method of BWT transformation, we canobtain the prefix tree of the original string through theresult of BWT transformation. Through the prefix tree,
the search for the string may be completed faster, and theresult of the BWT transformation takes up less space thanusing the prefix tree. Therefore, BWT transformation isused in bzip2 compression and DNA sequence alignment.
For the implementation of the BWT algorithm, first,
add a terminator to the original string, so that the starting
position can be found when restoring. For example, forthe string "aabcg", add the terminating character '#' to get

Rotateleft
aabcg#
abcg#abcg#aacg#aabg#aabc#aabcgTable 1. BWT Algorithm
first
last
Sort
column column#aabcg#g
aabcg#
a#abcg#aaa
bcg#aabacg#aabcbg#aabcg
c

The last column is the result. The method of restoring
the string is to start with the character '#' in the firstcolumn of Table 1, get the first 'g' in the last column, thenfind the first 'g' in the first column, and repeat thisoperation to get '#gcbaa', then reverse the arrangement to
get the original string.
In the process of performing the BWT algorithm, we
can find that the original algorithm needs to occupy© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

596

Chang Sheng, Fengzhi Dai

use the value corresponding to the sorted key to generate
the BWT result.

O(m*m) space. Of course, this can be turned into O(m)space through the position index. So choosing thesuitable string sorting algorithm becomes the main pointof acceleration. If a complete string comparison is
performed each time, memory access will be
discontinuous, so using 4 bytes as a key, and indexing thecorresponding value, sorting the key-value pairs canincrease the continuity of memory access. For the sorting
algorithm, it is realized by selecting the radix sorting thatis convenient to run on the GPU. Experiments show that
this method is effective.

3. Implementation details
For the above process, the specific implementation can bedivided into the following 4 parts.
3.1. Generate key-value pairs
This saves space and is a step towards shifting to
key-value pair sorting. The key is the first 4 bytes ofm*m strings, and the value is the number of rows
corresponding to m*m strings. So 3 bytes from thecurrent position, plus the current byte, a total of 4 bytes
are keys, and the data from 0 to 100000000-1 is thevalue.
It is worth noting that the memory access must be
performed in a continuous manner as shown in Fig.2,
otherwise the performance will be greatly reduced. The
subsequent algorithms in this article use this method
when this access is optional.

2. Main ideas

Fig.1 is the Implementation process.
Initial array

Key

Value

4-byte radix sort

Sorted key-value pairs
Fig.2 Memory access method
Find the same key
and reorder

3.2. Key-value pairs radix sort
By sorting the key-value pairs, most of the data can be in
the correct order. For the sorting algorithm, radix sortingis selected here. Because the radix sort has a timecomplexity of O(nlogn) when the radix size is determined,
and occupies 2n space, the algorithm is simple andregular, suitable for implementation by GPU, and has
been implemented on GPU.

Index value after sorting

BWT result

Fig.1 Implementation process

3.3. Same key recognition and reorder
Because the 4-byte fixed-length mode is used, and theactual string length is 100M, it needs to be comparedagain when the 4-byte keys are equal. This program
compares all keys. If they are not equal, the value in the
key-value pair is used to index the original data, and thecomparison is performed until the result is different. This

In order to complete the BWT algorithm, first, use 100M
data to generate 4-byte keys and values, Then use theradix sort to sort the generated key-value pairs according
to the key. For the case where the key values are the same
because of the limited length, re-sort them, and finally© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

597

Burrows-Wheeler transform acceleration basedalgorithm sorts multiple data with smaller length at thesame time.
Because the comparison of indefinite length isperformed, the comparison sorting algorithm is used forthe reordering part. Merge sort is considered for parallelsorting of the remaining data. However, because themerge sort has low parallelism in the final stage, andsome data has a large length, the bitonic sort algorithm isused for the part with a length greater than 2000. It has
high parallelism and is suitable for sorting small-length
data.
3.4. Generate BWT results
After the previous step, the sorted keys have beenobtained. According to the corresponding value, thesorted result of the original m*m strings can be obtained.
By formula (1), the result corresponding to the lastcolumn of Table 1 can be obtained.

pos 2 = (pos1 + L - 1)% L

Fig.3 Performance on different graphics cards

4.2. Sensitivity to data
In order to obtain the running performance on different
data, tests were carried out. The test data includes random
data, win7 installation iso format files, molecular
dynamics simulation trajectory trr format files, andhuman genome fasta format files. With 100MByte data
volume, the data of Table 2 was obtained on 2080ti.

(1)

In formula (1), pos2 is the result of the last column,
pos1 is the index value obtained after sorting, L is thelength of the original data, % is the same as that
represented in the C language for taking the remainder.
In this way, the BWT results can be obtained. Because
the data is sorted, the access to the original string
becomes random access, so this step consumes time.

Table 2. Running performance
File type
Speed
Random data
1.01GB/s
Win7 iso0.998GB/s
Trajectory trr
0.549GB/s
Genome fasta
0.202GB/s

4. Experimental results

For genetic data, because there are only 4 kinds ofcharacters per byte, 16 characters are used to generate a
4-byte key, which has an impact on speed.
When there is a certain repetition in the data, thisalgorithm has appropriate sensitivity to the data. It will
increase the amount of reordered data and cause too
many character comparisons during comparison. Using
Manber-Myers multiplication algorithm 2 can reduce data
sensitivity, but will increase the amount of calculation forless repeated data.

In order to determine the difference in performance ofvarious graphics cards, the sensitivity to data and theinternal performance of the algorithm, the following
experiment was carried out.
4.1. Performance on various graphics cards
In order to obtain the performance of this algorithm on
different graphics cards and data volumes, the following
tests were performed with random data. The result isshown in Fig.3.
It can be seen that when the amount of data is small,the concurrency of the GPU is not enough and the speed
is reduced. As the amount of data increases, the speed
begins to rise, and for random data, as the amount of data
further rises, the speed does not decrease.

4.3. Algorithm internal performance
In order to understand the time consumption of each partof the algorithm, a test was performed on 2080ti using
random data with a data volume of 100MByte. The
results are shown in Table 3.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

598

Chang Sheng, Fengzhi Dai

Table 3. Time consumption
Stage
Time
Generate key-value pairs
1.9ms
Key-value pairs radix sort
62ms
Recognition and reorder
5.67msGenerate BWT results
29ms

5. Conclusion
Speed
52.6GB/s
1.61GB/s
17.6GB/s
3.44GB/s

The algorithm sorts 4-byte key-value pairs by radix sort.
It can sort most of the data with less repetitiveness, andthen use the reordering method to sort the remaining data.
Great performance improvement for data with low
repeatability. For data with high repeatability and uneven
distribution, the speed is higher than the CPU. Using themultiplication algorithm can reduce data sensitivity 3, andthis algorithm has more advantages for data with low
repeatability.

Time is mainly used to sort the key-value pairs, which
is an important step in ordering the data. It also takes acertain amount of time to generate the result, because theaccess to the original string becomes random after thesort. In the reordering phase, because the amount ofreordered data on random data is less, and the length of
the reordering comparison is similar, it does not takemuch time.

References
1. Michael Burrows, David J. Wheeler. A block-sorting
lossless data compression algorithm. Technical report,
Digital SRC Research Report, 1994
2. Udi Manber, Gene Myers. Suffix Arrays: A New Method
for On-Line String Searches. Siam Journal on Computing,
1993, 22(5): pp.935-948.
3. Kartsev, Petr F. High performance OpenCL realization of
Burrows-Wheeler transform on GPU.
International
Workshop on OpenCL, Oxford, ENGLAND, May. 2018,
pp. 83-84.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

599

Design of Automatic Water Supply Upper Computer System
Peng Lu1, Fengzhi Dai1,2 *, Tianyi Zhang1
1

Tianjin University of Science and Technology, China;

Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China

2E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
Aiming at the unstable water supply and fluctuating water pressure in the water supply system, an upper computer
system of automatic water supply system based on Siemens PLC and KingView software was proposed. The
pressure sensor in the water supply pipeline is used to detect the pressure in the pipeline, and the liquid level sensor
monitors the liquid level in the tank. The sensor transmits the data to the PLC, and the PLC issues the control
instruction after the computation processing. The KingView software can realize real-time monitoring and fault
alarm of the system. The system can effectively improve the stability of water supply, and can avoid humanentering the dangerous environment to search for unknown faults.
Keywords: KingView, constant pressure water supply, PLC, upper computer system, remote control

1. Introduction

supply. The KingView software and Siemens PLC areused to design the upper computer of the water supply
system and connect it with the lower computer of thewater supply system. Finally, realize the constantpressure water supply, remote monitoring and remote
operation of the water supply system.

With the continuous development of China's urbanization
process, the pressure on the urban water supply system isincreasing. In old urban communities, regional watersupply is basically achieved by the municipal pipe
network through secondary artificial pressurization, water
towers and high-level pools.
However, these water supply methods have serious
imbalances between water pressure and water
consumption. In addition, the monitoring of parameterssuch as the pressure of the pipe network and the runningstate of the motor in the old-fashioned water supply
system is through manual inspection of measuring
instruments. This greatly reduces the operating safety ofthe operator. And when the water supply pressurechanges greatly at different times, it can only be adjusted
manually. This adjustment method is difficult to meet thesystem adjustment requirements in time 1.
Based on the above discussion, this design focuses on
the upper computer system for constant pressure water

2. The Hardware Structure
The core task of this system design is to remotely monitor
and operate the water supply system through the hostcomputer. Use PLC to make the frequency converter
implement cyclic control of multiple pumps 2. So thewhole hardware system is composed of a liquid levelsensor, three water pumps, a pressure sensor, a frequency
converter and PLC.
2.1. SIMATIC S7-300 PLC
This design uses SIMATIC S7-300 PLC. SIMATIC
S7-300 PLC has good versatility. It has the advantages of© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

600

Peng Lu, Fengzhi Dai, Tianyi Zhang

controlled by a microprocessor and has high functional
diversity. MicroMaster430 series inverters are
particularly suitable for driving water pumps and fans.
Has high flexibility and reliability. The following
functions can be realized: bypass function, multi-pumpswitching, energy-saving operation, manual/automatic
switching, broken belt and water shortage detection, etc 4.
The inverter terminal interface is shown in Fig.3.

easy
realizationof
distributed
configuration,
modularization, low user difficulty in mastering, and no
exhaust fan structure. It is very suitable for thesmall-scale control 3.
This design controls the frequency converter through
PLC, and then controls the start and stop of the water
pump through the frequency converter. SIMATIC S7-300
PLC is shown in Fig.1.

Fig.1. SIMATIC S7-300 PLC

The automatic water supply system has a medium
amount of program and does not occupy much storage
space. And the computing speed, communication
resources and programming resources required by thesystem are not much. So this design chooses the CPUmodel of 315-2DP in SIMATIC S7-300 PLC. The
315-2DP used in this design is shown in Fig.2.This CPU
has the following characteristics:• 48 KB working memory.
• 0.3 ms/1000 instructions.
• MPI + DP connection.

Fig.3. Inverter terminal interface

2.3. Sensor module
In this system design, MC20B economical pressure
sensor and MC20C drop-in liquid level sensor are used.
The MC20B economical pressure sensor selected inthis design can modify the range and zero point according
to the needs on site. And it has high anti-interference andoverload performance, and it is not prone to failure.
The MC20C drop-in liquid level sensor uses anisolated diffused silicon sensor to convert the hydrostaticpressure into an electrical signal. Then after temperaturecompensation and linear correction, it is converted into
standard electrical signal output. The liquid level sensor
has the following characteristics:• Measuring range: 0.3M~200M.
• Working temperature: -20 oC ~ 80 oC.
•
Output signal: 4mA~20mA.

Fig.2. The 315-2DP CPU

3. System programming

2.2. Frequency converter

In the system programming, this design consists of PLC
programming and configuration simulation design. The
combination of these two programs can well reflect thecontrol process of the system.

This design uses the MicroMaster430 series inverters,
which are particularly suitable for driving water pumps
and fans. The model used is 6SE6430-2UD27-5CA0. The
power of this type of inverter is 7.5~250KW. It is© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

601

Design of Automatic Water

3.1. PLC programming

3.2. Configuration simulation design

This design uses STEP7 programming software for PLC
programming. The programming method used is the logic
algebra design. It is to edit the PLC ladder diagram
program according to the simplified logical expression 5.
In this design, the PLC program needs to realize thefollowing control actions. The first water pump starts andstops. When the frequency of the second frequency
converter reaches the upper limit, the water pump is
switched between frequency conversion and industrial
frequency. Thirdly, when the pressure fluctuation of thepipeline is small, the water pump shall be alternatedperiodically. The fourth is to detect the liquid level, andthe system can be stopped when the water supply level is
too low.
PLC programming adopts modular programming. The
main program OB1 is composed of seven functions orfunction blocks. Make the main program simple andvisual for easy troubleshooting. The program in function
FC1 is used to process the input analog quantity. The
program in function FC2 is used for boot initialization.
The program in function FC3 is the start-up program of
the water pump motor. The program in function FC4 isthe switch between frequency conversion and industrial
frequency of the water pump motor. The program infunction FC5 is an alarm program. The program infunction FC6 is an analog output program. The programin function FC7 is a water pump alternate working
program.
The main program OB1 part of the program design isshown in Fig.4.

In this design, the design of the configuration screen of
the upper computer system uses the KingView software
produced by Beijing Yakong Company. The software is
widely used in the current industrial field, and it hasvarious functions such as process control design, field
control, and resource management 6.
The configuration screen designed by this systemcovers the main screen, real-time curve, alarm screen,
historical data report, historical curve, and real-time datareport. In the main configuration screen, the operator canstart, stop and emergency stop the system. And can adjust
the pressure of the pipe network. At the same time, it canmonitor the running status of the water pump, the liquid
level of the pool and the pressure of the outlet pipe
network. On the main page, you can also perform a pumpfault test. The main configuration screen is shown in
Fig.5.

Fig.5. Main screen

4. System Simulation
After the system design is completed, the system can be
simulated on the Kingview software.
In the simulation of this design, the main screen of thesystem can realize real-time monitoring of the operationof the automatic water supply system. In the real-time
curve interface, the operator can intuitively monitor thepipe network pressure and pool water level in the systemthrough real-time changes in the curve. It can be seen in
the real-time curve simulation image that by changing thesystem water supply pressure setting value, the system
will automatically adjust the actual water supply pressure
to match the standard value. The real-time curve image isshown in Fig.6.

Fig.4. Main program OB1© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

602

Peng Lu, Fengzhi Dai, Tianyi Zhang

and the water pump. Can read the real-time data andhistorical data of the automatic water supply system.
Monitor the changes in the operation of the water pumpand the occurrence of system failures when the pressureof the water supply system fluctuates. The combinationof KingView and PLC in this design makes the system
program simple, integrated and highly transplantable.
References
1. Xiaoling Mo, Based on PLC the Automatic ConstantPressure Control Water Supply System Design, Applied
Mechanics and Materials, 2014, 12: pp.1587-1590.
2. Tiewei Bu, The Overall Design of Frequency Conversion
Water Supply Control System Based on PLC, Advanced
Materials Research, 2014, 4: pp.1465-1468.
3. Yilin Qin, Siemens S7-300 PLC application technology,
Electronic Industry Press, Beijing, China, 2007.
4. Yong Pan, Jian Wang, Quick commissioning of
MICROMASTER440 inverter, Automation application,
2011, 1: pp.21-23.
5. Xuiling Lu, Several commonly used programmablecontroller ladder diagram design methods, Machine Tool
Electric Apparatus, 2004, 10: pp.43-45.
6. Zhenfang Liu, Based on KingView configuration softwareand PLC water tower water level control system, Digital
Communication World, 2019, 9: pp.85-86.

Fig.6. Real-time curve

In the historical curve interface, the operator canintuitively monitor the changes of the pipe networkpressure and the water level of the pool in the historicaltime through the historical changes of the curve. Aftermany simulations, the historical curve interface is shownin Fig.7.

Fig.7. Historical curve

5. Conclusion
This design uses the KingView software combined with
PLC and frequency converter to form an automatic watersupply control system, replacing the previous valve
control and manual water supply system. It is ensured
that the pressure of the water supply pipe network will
not drop during the time when the residents have morewater. It also ensures that when the residential water islow, there will be no impact on the components of theentire system due to excessive pressure on the pipe
network. It greatly slows down the aging of the watersupply system components and extends the service life of
the system.
The upper computer system in this design realizes themonitoring of the running status of the lower computer© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

603

Analysis of boiler water level system Based on the fuzzy control
Tianyi Zhang1, Fengzhi Dai1,2 *, Peng Lu1
Tianjin University of Science and Technology, China;

1

Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China

2E-mail: * daifz@tust.edu.cn
www.tust.edu.cn
Abstract
Based on the three-stroke water supply system, this paper analyzes the performance characteristics of the boiler
water level control system in reality, so as to reason out the appropriate fuzzy control rules, design fuzzy controller,
and applied to the control system, so that the system for self-adjustment of PID parameters, constitute a fuzzy PID
control system. On this basis, this paper analyzes the performance, advantages and characteristics of two control
systems: the traditional PID control system and the fuzzy PID control system, and simulates the parameters of the
input variables for comparison and analysis.
Keywords: Fuzzy PID, three-pulse, MATLAB, PID control

1. Introduction

The fuzzy control is not necessarily an accurate
mathematical analysis of the controlled object, but rather
a combination of fuzzy control algorithms and PID
control to control non-linearity, make the boiler levelsystem stable and robust.

Boilers are playing a very important role in factory
production. The main task of boiler production is to
produce and transfer heat energy and converting energy
from coal or natural gas to other energy sources such as
electricity. It is a large-area capacity facility used inindustrial production processes. Therefore, boiler safety
is an issue of great concern to us.
Boilers are often subject to a wide variety of objects
and system disturbances. It would be unrealistic to relyon the operator to keep it running accurately in the plant
for long periods of time. Therefore, the study of
automatic control systems for boiler equipment is a
general trend 1.
In the boiler drum water level fuzzy control system, thecore control method is the fuzzy control. Fuzzy control isa control method based on fuzzy logic theory, fuzzy set
theory and fuzzy language variables 2.
The fuzzy PID boiler water level control system isinvestigated and analyzed in terms of model andcharacteristics 3,4. The design, simulation, and analysis
were performed in the Fuzzy Logic Toolbox in
MATLAB.

2. System Control Principle
Generally speaking, the larger the boiler drum capacity,
the larger the volume of steam mixture and steam in the
drum. Sufficient space is required for separation andcirculation, and the moisture content of the generated
steam is much more stringent. In turn, the water level ofthe cartridge is also very strict.
In a double-pulse control system, automatic adjustmentof the water supply valve does not occur in time,
resulting in a delay. Therefore, the water level in thepacket will deviate due to the delay. The tri-pulse control
system introduces the control amount of the water supply
signal based on the level and steam signals of the
double-pulse control system.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

604

Tianyi Zhang, Fengzhi Dai, Peng Lu

2.1. boiler water level system structure
Boiler drum water level control system consists of
Economizer, Drum Boiler, Down tube, Superheater,
Lifter tube, Water Supply and so on.
The boiler obtains water through a water supply
control valve, then passes through an economizer whereit absorbs heat from the burning fuel, regulates it to theamount of water supply, dynamically equilibrates thesteam mixture and liquid water in the steam package byregulating the water supply, then passes through thewater cooling wall around the furnace in the top
connection box, and finally returns to the steam package 3.
Boiler drum water level control system is shown in theFig.1.

Fig.2. Classical PID Control Schematic Block Diagram

The control rule is
1

𝑡𝑡

𝑑𝑑

U(t)=𝐾𝐾𝑃𝑃 �e(t) + 𝑇𝑇 ∫0 𝑒𝑒(𝑡𝑡) 𝑑𝑑𝑡𝑡 + 𝑇𝑇𝐷𝐷 𝑑𝑑𝑒𝑒(𝑡𝑡)�
𝑖𝑖

𝑡𝑡

(1)
When selecting a PID controller, choose a simpler
and more accurate incremental PID controller.
∆𝑈𝑈(𝑘𝑘) = 𝑢𝑢(𝑘𝑘) − 𝑢𝑢(𝑘𝑘 − 1)(2)Only the first three deviations are needed to
calculate ∆ U(k), and the incremental PID control
algorithm is now widely used.
2.3. cascade control systems
Cascade systems form an inner and outer closed loop in
the system configuration. The execution of the control
sequence of the system is the outer ring first, followed by
the inner ring. Cascade Control Systems is shown in
Fig.3.

Fig.1. Boiler drum water level control system

2.2. principles of PID control
Start with the classical PID control, which consists ofproportional, integral, and differential components. The
proportional part of this is to maintain system stability .
As the scale factor increases, the adjustment time
becomes shorter and the resulting system deviation
becomes smaller. However, if the proportionality factor is
too large, oscillations can occur. Therefore, the scale
factor K should be appropriate to achieve the effect ofsmall transition time, small static difference, and highstability.
The function of the integral part is to eliminate
systematic deviations. The larger the integration time, theweaker the system's ability to eliminate deviations andthe longer the transition time. The smaller the integration
time, the greater the ability to eliminate deviations that
may cause the system to oscillate. The role of the
microcomponent is to improve the dynamic performanceof the system. Classical PID Control Schematic Block
Diagram is shown in Fig.2.

Fig.3. Cascade Control Systems

3. Design of Fuzzy Controller
The following introduces the basic structure of the fuzzycontroller, and the fuzzy controller designed in this paper.
3.1. the basic structure
It consists of four parts: Knowledge Base, Fuzzy
Reasoning, Fuzzification of Inputs, and Accuracy of
Outputs.
First, the input quantity is fuzzified and reasoned
about by the knowledge base, and finally the output
quantity is decoupled and refined. Basic structure of the
fuzzy controller is shown in Fig.4.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

605

Analysis of Boiler Water

4. Control System Modeling and Simulation
Design of automatic adjustment of steam packet levelaccording to boiler water level dynamics.
4.1. classical PID control modeling
From empirical data, under unit step perturbation of
feedwater flow rate, floating velocity of boiler water level
V=0.05 mm/s.
The time constant T2 = 15s, so the transfer function isdetermined as G(W) = 0.05/(15S^2+S). Under steam flow
perturbation, the amplification factor K2 = 5, T2 = 15s,
yielding the transmission G(D) = -0.05/S+S/(15S+1).
The boiler water level model is built using the
SIMULINK toolbox in the MATLAB environment, andthe three parameters Kp=10, Ki=0.0045, and Kd=0 are
obtained by the improvised test method 4. Classical PID
control model is shown in Fig.6.

Fig.4. Basic structure of the fuzzy controller

3.2. design process for fuzzy controller
There are the following steps.
• Structural design with two-dimensional fuzzycontroller for maximum efficiency. Controller has
two inputs and three outputs.
• Fuzzing of Inputs. The set of input and output words
is denoted by {NB,NM,NS,ZO,PS,PM,PB} and thesubordinate function is chosen to be an isosceles
triangle. Isosceles triangle affiliation function imageis shown in Fig.5.

Fig.5. Isosceles Triangle Affiliation Function Image••

Fig.6. Classical PID control model

Define fuzzy rules. Make separate tables of the three
output variables and analyze how they change on
their own as E and Ec change. The 49 control rules
are deduced from practical experience. The Kp isshown in the Table 1.
e\ec

NB

NB
NM
NS
ZO
PS
PM
PB

PB
PB
PM
PM
PS
0
0

Table 1. Kp Fuzzy Rules Table
NM
NS
ZO
PS
PM
PB
PB
PM
PS
PS
0
NS

PM
PM
PM
PS
0
NS
NS

PM
PM
PS
0
NS
NM
NM

PS
PS
0
NS
NS
NM
NM

PS
0
NS
NM
NM
NM
NB

4.2. fuzzy PID control modeling
Add a fuzzy logical controller with associated fuzzy rules
to the classical PID control simulation model. Fuzzy PID
control model is shown in Fig.7.

PB
0
0
NM
NM
NM
NB
NB

The rule table reasoning for Ki and Kd is the same asabove.
Clarifying Variables. In order to obtain accurate
output variables of the control system, the weighted
average method (center of gravity method) is used tosolve the fuzzy.

Fig.7. Fuzzy PID control model

4.3. simulation results analysis•

Simulation graphics with only level perturbation© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

606

Tianyi Zhang, Fengzhi Dai, Peng Lu

perturbations are added, the fuzzy PID control system ismore stable.

Simulation diagrams with only level disturbance is shownin Fig.8. The left figure shows the simulation results ofthe classical PID control model and the right figure
shows the simulation results of the fuzzy PID control
model.

5.Conclusion
By analyzing the dynamic and static characteristics of theautomatic drum water level control system, the paper
constructed a conventional PID simulation model and a
fuzzy PID simulation model to validate the results. It canbe concluded that the fuzzy PID control system has asmall amount of overshoot, short oscillation period andshort setting time.
Fuzzy PID control system has greater flexibility androbustness, and better fault tolerance.

Fig.8. Simulation diagrams with only level disturbance•

Add steam disturbance at 500s in addition to level
disturbance. Simulation diagrams for liquid level
disturbance and vapor disturbance is shown in Fig.9.

References

Fig.9. Simulation diagrams for liquid level disturbance andvapor disturbance

2. Jian Yang. Water level modeling and control strategy forlarge marine fuel boilers. Shanghai Jiao Tong University,
2007, pp.42-43.
3. Ping Chen. Study of fuzzy PID control of boiler drum
water level, 2006, pp. 5-10.
4. Na Feng, Yao Wang. MatLab simulation study of boiler
drum water level fuzzy control system. Automation andInstrumentation, 2019, 34(04): 80-84+99.

1. Zhongning Li. Design and analysis of boiler water levelfuzzy control system based on MATLAB, 2008, pp. 5-7.

From the two graphs, the fuzzy PID control system(right) has better overshoot and setting time than theclassical PID control system (left). When multiple© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

607

Development of the Circuit System for Greenhouse Environment Regulation
Yuhui Cheng, Fengzhi Dai *, Chengxu Ji, Peng Lu

Tianjin University of Science and Technology, China;
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
Aiming at the demand of agricultural modernization, this paper proposes a circuit system for greenhouseenvironment regulation. For the greenhouse, due to the requirements of the crops for the stability of the growthenvironment, the cheap single-chip control system can be used to complete this work well. The system can obtainthe current environmental data in the greenhouse through the DHT11 sensor and the light sensor, regulate andcontrol the environment through external equipment to keep the environment stable.
Keywords: STC89C52 microcontroller, DHT11 temperature and humidity sensor, light sensor

1. Introduction

monitoring of temperature, humidity, light intensity in thegreenhouse. The system will be on the sensor datacollected in real time comparison, if the preset thresholdis reached, then the corresponding module will start workin order to achieve the purpose of the environment within
the greenhouse to be regulated.

With the development of automation technology and thegradual reduction of the number of people engaged inagricultural work, the method of agricultural production
has begun to turn to automation and unmanned. Many
new terms are beginning to be known by people, such as
automatic harvesting, automatic cultivation and so on.
On greenhouse, the temperature, humidity and lightintensity control three environmental factors, for crops
grown in greenhouses of great importance. Closely
related to the development and environmental changes ingreenhouse crops, from sowing a seed to germinate andthen pull back strain results, optimal plant growthenvironment in different growth period the demand is not
the same. Most direct impact is the temperature, humidity,
and in light of these factors 1.
Based on the above discussion, this circuit system
mainly focuses on the collection and automatic
adjustment of relevant data in the greenhouse. The
control and calculation module of this system is a single
chip microcomputer. The system to achieve datacollection work by DHT11 temperature and humidity
sensor and an illumination sensor can be real-time

2. The Hardware StructureThis system is mainly responsible for the collection ofenvironmental data in the greenhouse and the control of
the corresponding work modules. The system is mainly
divided into three parts. The first part is the calculationand control part composed of single-chip microcomputers,
the second part is the sensing part composed of sensors,
and the third part is the external equipment controlled by
the relay.
2.1. DHT11 temperature and humidity sensor
DHT11 digital temperature and humidity sensor is atemperature and humidity composite sensor withcalibrated digital signal output. It uses dedicated digitalmodule acquisition technology and temperature and© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

608

Yuhui Cheng, Fengzhi Dai, Chengxu Ji, Peng Lu

2.4. Main control chiphumidity sensing technology to ensure that the producthas extremely high reliability and excellent long-term
stability 2. The DHT11 module is shown in Fig.1.

In this design, the external MCU is the STC89C52RC.The
chip has 8KB flash memory and 512B RAM. And it has
more than 30 general-purpose IO ports, multiple timers,
which fully meet the design of the required pin andmemory requirements. The main control chip is shown in
Fig.4, and it has the following characteristics:• Operating Voltage 3.3V ~ 5.5V.
• Operating temperature range: 0 oC ~ 75oC.

Fig.1. DHT11 module

2.2. BH1750FVI module
The BH1750FVI module (Fig.2) is a digital lightintensity sensor for a two-wire serial bus interface, with abuilt-in 16-bit ADC. The module has the following
characteristics: Wide brightness range: 0LX ~ 65535LX;
Operating Voltage 3.0V ~ 5.0V.

Fig.4. STC89C52RC chip

3. System circuit module design
In the circuit design, this system uses temperature andhumidity data acquisition module, light data acquisition
module and relay control module. These modules play avital role in this system.

Fig.2. BH1750FVI module

2.3. LCD1602 liquid crystal display

3.1. Design of connection circuit for temperatureand humidity module

LCD1602 liquid crystal display is a dot matrix liquid
crystal module used to display letters, numbers, symbols
and other characters. The liquid crystal display consistsof multiple 5X7 or 5X11 dot matrix character bits to form
a dot matrix character display module, which is shown in
Fig.3.

The IO pin of the DHT11 module for data exchange isconnected with a 4.7K pull-up resistor in parallel with
VCC pin to increase the pin level to transmit data. In
addition, a capacitor is connected to the power supply fordecoupling and filtering to improve voltage stability.
Because the module's single-bus design greatly reduces
the occupancy rate of circuit resources in actual use, theoverall circuit looks very simple and clear.
3.2. Design of connection circuit for IlluminancemoduleThe VCC pin and GND pin of the module can beconnected to the positive and negative poles of the
microcontroller. ADD is the address pin to zero. SCL and

Fig.3. STM32F103ZET6 chip

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

609

Development of the Circuit

4.3. Environmental Control Module

SDA are the clock and data lines in the I2C bus
respectively. SCL is connected to the P10 pin of the
microcontroller. The SDA pin is connected to the P11 pinof the microcontroller.

Relay is a circuit control element, which is triggered athigh level. The input terminal is generally provided withthree pins, GND, VCC, and IN. What we use in thissystem is two dynamic close type relay modules,
controlled by the single-chip P7 pin and P8 pin.

The control circuit system of the greenhouse environment
regulation system controls the external equipment
through the relay.
When the environmental data collected by the system
is not in the preset range, it will control the relay to turnon the corresponding external equipment such as water
pump, exhaust fan, fill light, etc. according to the type ofvalue. This design method can replace different types ofexternal equipment according to different environmental
requirements, which has better practicability.

4. Introduction of functional module

5. Conclusion

Through the work of each functional module, thegreenhouse environment control circuit system can wellcomplete the set work.

This design integrates the environmental information in
the greenhouse and manages it uniformly.
From the perspective of operation, it can save a lot of
human resources and avoid some mistakes caused bymanual operation. From the perspective of production,
the system regulates various factors affecting the growthof crops in the greenhouses to keep them in an
appropriate range, and the yield and crop quality ofgreenhouses have been improved to a certain extent.
From the point of view of cost, the hardware cost of thesystem is very low, which is conducive to large-scalelaying.

3.3. Relay module control circuit

4.1. Data collection module
The data acquisition module of this system is composedof DHT11 temperature and humidity sensor and lightsensor. When the system is powered on, each sensor will
read the environmental data in the greenhouse in realtime and send it to the processing unit.
4.2. Airkiss distribution network
The system can adjust the threshold of environmental
parameters by pressing the buttons. The functions of eachbutton are shown in Table 1.

References
1. Yin Liu, Huijuan Lv, The influence of external conditions
on vegetable growth and its control methods, Modern
Agriculture Research, 2011, 2: pp.10-11.
2. Xinxiu Xu, ShiYuan Zhao, The influence of external
conditions on vegetable growth and its control methods,
Modern Information Technology, 2020, 14: pp.57-59.

Table 1. Introduction to key functions
ButtonFunction
Key1
reset
Key2
selection
Key3
value up
Key4
value down

The adjustable parameters of the system are shown in
Table2.
Table 2. System adjustable parameters
Name
Temperature
Humidity
Light

Nun1
upper limit
upper limit
upper limit

Num2
lower limit
lower limit
lower limit© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to24, 2021

610

Design of An Intelligent Car Based on MSP430
Ruming Kang, Fengzhi Dai *Tianjin University of Science and Technology, Tianjin, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
The system takes the MSP430 single chip microcomputer as the control core, uses the reflective photoelectric sensor
TCRT5000 module to track the line, and realizes the automatic tracking. In the experiment, the black-and-white line is used as
the route. The system is driven by the L298N module and controlled by the PWM DC motor. This paper introduces theprinciple of the reflective photoelectric sensor and the circuit diagram of the tracking module. It also explains how to realize
the automatic tracking based on the MSP430 single chip microcomputer. The technology can be used in warehouse, unmanned
production line, intelligent service robot and other fields.
Keywords: MSP430, Sensor, Tracking module, Motor driven

1. Introduction

the board; the marking line turns 90°at the top of theslope, and the turning radius is 20cm; The distance
between the marking line parallel to the top of the slope
is ≥30cm, and the distance from the top of the slope is≤20cm; the total length of the marking line is 1m. The
parking mark is a black line with a width of 1cm and alength of 5cm, perpendicular to the marking line on the
top of the slope. The slope angle of the trolley and the topview of the driving route are shown in Fig.1.

This smart car material comes from the 2020 National
University Student Electronic Design Competition. It
used TI’s MSP430 microcontroller as the control centerof the four-wheeled car, and transmits the signals of eachsensor to the microcontroller for analysis and processing,
thereby controlling the L298N motor drive, controllingthe car and buzzer the device tweets.
The car is required to be able to follow the designatedroute and automatically follow the line on the ramp. The
car must run independently, and no equipment (including
power supply) can be used outside the car. The weight of
the trolley (including battery) is less than 1.5kg, and theoutline dimension on the ground is not more than 25cm×25cm. The ramp is made of blockboard with a length andwidth of about 1m, allowing the natural color and natural
wood grain on the board.
The surface of the wood board is laid with 1cm×1cm
black and white paper strips (hereinafter referred to as themarking line) as route instructions; the starting section of
the marking line is a straight line, parallel to both sides ofFig.1 Schematic diagram of car ramp angle and top view of driving
route© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

611

Ruming Kang, Fengzhi Dai

2. Modularization of the car hardware

2.2 Tracking module

The smart car can be divided into: power module, sensor
detection module, microcontroller module and DC motor
drive module. The smart car can realize functions such as
tracking and parking. It uses the MSP430 single-chip
microcomputer as the main control chip, and uses a DC
motor as the power. The hardware system structure
diagram is shown as in Fig.2.

Component: CRT5000 photoelectric sensor
Principle: The infrared emitting diode of theTCRT5000 sensor continuously emits infrared rays.
When the emitted infrared rays are not reflected back orreflected back but the intensity is not strong enough, theinfrared receiving tube is always in the off state, and theoutput terminal of the module is high at this time.
Indicating that the diode is always off; when the detected
object appears in the detection range, the infrared isreflected back and the intensity is large enough, theinfrared receiving tube is saturated, and the output of themodule is low at this time, indicating that the diode is lit.
The application in the actual program is: two
TCRT5000 sensors are installed on the left and rightsides of the car. (1) When the left and right sensors are in
the white area, that is, the emitted infrared rays are
emitted back, both the left and right motors rotate. (2)When only the left sensor detects the black line, theinfrared is not reflected back, indicating that the car isdriving to the right. At this time, the motor on the rightrotates and the motor on the left stops, and the car showsa left turn correction. (3) When the right sensor detects a
black line, the emitted infrared is not reflected back,indicating that the car is to the left. The motor on the left
rotates and the motor on the right stops, and the car turns
right to correct.
The schematic diagram of the tracking process and thecircuit diagram of the tracking sensor module are shownin Fig.4 and Fig.5.

Fig.2 Hardware system structure diagram

2.1 Motor drive module
The motor adopts a PWM DC motor, which can be used
as an ordinary motor (no coding function is used to makeit have better mechanical speed regulation performance,
and the DC-DC step-down voltage stabilized power
supply module is used to realize the mechanical speed
regulation of the motor), and it is driven by L298N.
The single-chip microcomputer controls the level ofthe L298N enable terminal to realize the rotation of the
motor and realize the forward or steering of the trolley.
The principle of the direction change of the DC motor is:
if turning left, the left wheel will not move, and the rightwheel will turn; if turning right, the opposite; if goingstraight, both left and right wheels will turn. But the
problem that 298 chips are prone to heat is difficult tosolve. The physical picture of L298N is shown in Fig.3.

Fig.3 Physical Picture of L298N Module© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

612

Design of An Intelligent

according to the parking subroutine and makes a parking
action.
If it reflects that the obstacle avoidance and parking
actions are inaccurate or there are loopholes, you canappropriately add line-following sensors on both sides,
and make corresponding modifications in the subroutine
to improve the accuracy of obstacle avoidance andparking. The obstacle avoidance flowchart is shown in
the Fig.6.

Fig.5 Schematic diagram of car ramp angle and top view of driving
route

2.3 Power Module
The power module is divided into two parts: one is thechip power module, which uses a 4.2V lithium battery todrive the chip; the other is the motor drive module, which
provides higher power to drive the motor through L298N.
Motor drive power supply: Use two 4.2V lithium
batteries connected in series to form about 8.4V toprovide power to the L298N chip. Generally, the battery
has high power, is more stable and durable, can be
charged more economically, and has better stability, andis more convenient to use.
Chip drive power supply: Since the power requirement
is not large, but the stability is high, so use the 8.4
voltage regulator circuit to drive (the voltage drop has
been considered, of course, the voltage drop of eachelectronic device is different, and you need to debug
according to your own.

Fig.6 System software design flow chart

4. Conclusion
This article explains the intelligent car control system
based on MSP430 single chip microcomputer chip.
Through the infrared tracking sensor, the car tracking andparking functions are realized, and the expected design
effect is achieved. In the design process, light has anoccasional impact on the tracking. LED lights can beinstalled adjacent to the line-following sensor to enhance
the light and strengthen the recognition of black lines. At
the same time, the tracking algorithm should be
optimized to make the tracking more accurate.

3. System software design
The smart car is based on the modular design concept,
and the software design corresponds to the hardware
system, mainly for tracking subroutine modules andparking subroutine modules. The main program calls
these subroutine modules in logical order.
The single-chip microcomputer collects the signals
collected by the sensors of each module, after processing,and then controls the actions of the trolley through the
motor driver to complete the predetermined target. Whenthe single-side sensor detects the black line, it judges thestatus according to the tracking subroutine and makes
steering adjustment. When both the line-following
sensors detect the black line, it judges the turning state

References
1. Ying Ceng, He Ming Zhao, Zhi Ling Peng, Design ofMultifunctional Intelligent Car Based on MSP430,
Electronic Measurement Technology, 2017 (6): pp. 10-14.
2. Meng ting Chen, Bai yan Hu, Can Huang, Design ofIntelligent Tracking and Obstacle Avoidance Car Based onSingle Chip Computer, Smart Robot, 2016(4): pp.47.
3. Zheng Bing Wan, Design of automatic tracking car based on
LDC1000, Automation technology, 2016(10): pp. 111-157.
4. Qiu Yan Zhang, Jun Xia Zhang,Yang Yang, Design ofMultifunctional Intelligent Car Based on MSP430,
Information Technology, 2018(11): pp.54-58.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

613

Design of a WIFI Video Car
Qianqian Zhang1, Fengzhi Dai 1, 2 *, Jichao Zhao 1, Haokang Wen 1, Hongbo Hao 1
1
2

Tianjin University of Science and Technology, China;

Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China
E-mail: daifz@tust.edu.cn
www.tust.edu.cn

Abstract
This subject is mainly composed of motor, frame, STC89C51 single-chip microcomputer and other auxiliary modules.
Through the android client broadcast video car cameras, wireless video signal acquisition by the client sendcommands to the car at the same time, wireless router transmission instruction by wireless router, single-chip
microcomputer is run for processing, then the motors are driven by single chip microcomputer control to rotation andsteering, so as to realize the control of motor sports and video acquisition.
Keywords: The router OpenWRT, The Socket, WIFI video car, STC89C51, The steering gear

1. Introduction

scheme selection is to complete the function of the system
according to the knowledge and information, according to
the tasks and requirements 1.

WIFI video car can achieve real-time monitoring, throughthe computer, mobile phone, tablet etc to connect wirelessrouter, and through the client sends commands to the WIFI
router serial port on a router by transmitted to MCU. MCU
program parses the command, then operating LN298
driver module so as to achieve the effect of motor control.
It has 128 RAM and 4K of ROM inside.
2. Overall scheme design
In the design and production of circuits, we try to reduceother interference introduced in the measurement toachieve a higher level of performance indicators andfunctions of electronic products. Usually the circuit design
is to take into full account the various factors and thecircuit physical.
The circuit mainly involves the connection of router,
MCU minimum system, LN298 drive module, steering
gear, and camera. Make clear the production task of thesystem, make specific analysis of the production task,
understand the performance of the system, indicators,
content and requirements. The important task of theFig.1. General layout structure

3. Subsystem design
3.1Circuit design steps
Economical and detailed circuit design is half the successof our project. First of all, we need to design a minimum
system circuit. We need to know which components are© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

614

Qianqian Zhang，Fengzhi Dai，Jichao Zhao, Haokang Wen, Hongbo Haoused in a minimum system circuit. In addition, we need toaccurately determine the parameters of various
components, such as clock frequency, resistance value,capacitance parameters and so on 2.
Draw the circuit diagram of the minimum system andsimulate it. After the simulation is successful, the circuit
welding is carried out. The circuit diagrams are shown in
Fig.2 to Fig.4.

adjust the duty ratio of the steering gear, so that we canthrough the command code to control the rotation Angleof steering gear 3.
The second is to use a serial port interrupt, timer is used
to set the baud rate (refers to the speed of data transmission)
to 9600, and the timer 2 should set automatic loading mode
(refers to give it an initial value). The timer need to be set
up only once, after the timer overflow, automatically
return to the initial value.
The 298 drive is based on the high and low level
connected to the controller pin to determine the positive
motor inversion, of course, the drive board is to be incommon with the minimum system, which is used to run
the program to be effective control driver board, so as tocontrol the motor rotation and steering. When the anodealso points to a branch of the steering gear connected to thepower supply, in most cases the steering gear is needed,
the power converter.
3.2 Wireless RouterFig.2. Minimum system simulation circuit

The wireless router here is actually the same as the one weuse in our homes. WIFI is a network protocol, in fact, it isequivalent to a computer, can run the operating system,
also need to install the driver. Router connection is shownin Fig.5.

Fig.3. L298N drive module connection circuit

Fig.5. Logic control circuit
Fig.4. Steering gear connection circuit

3.3 Principle of steering gear

Actually, the whole hardware system is not complicated,the main route as the core, openWRT as the main control
system. For the next bit machine, 51 of the procedures will
be responsible for driving motor and steering gear, we
mainly use the two timers, the first is for steering gear,

The operating principle of steering gear is: in the same
period, there are high and low level, high level is thereference signal, generally 1.5ms, the period is generally
20ms, which is the duty ratio we often mentioned. The© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

615

WIFI video car design

difference in voltage between the high and low levels
allows the small motor inside to rotate.
Control mode of steering gear is: The control of steering
gear generally requires a 20ms time-base pulse, and thehigh level part of the pulse is generally the angle control
pulse part within the range of 0.5ms~2.5ms 4. Take thesteering gear with an angle of 180 degrees as an example,
then the corresponding control relationship is as follows:
0.5ms-- 0 degrees; 1.0 ms-- 45 degrees; 1.5 ms -- 90
degrees; 2.0 ms-- - 135 degrees; 2.5 ms -- 180 degrees;
Duty ratio, cycle and signal line of the steering gear are
shown in Fig.6.

OpenWRT, we should first build Linux development
environment Ubuntu.
4.1 Function implementation
This time, OpenWRT system was used to mainly complete
the data acquisition of the camera, and the image wasformed after MJPG video streaming software was installed.
Through this system to mount the USB interface form of
the camera, we use the Onio 12 megapixel camera. Thissystem can receive the command code sent by the mobile
phone client, and send it to the next computer throughSer2net software in the way of serial port. So as to achieve
the control video car effect.
4.2 Socket programming
Socket is just an interface, it's just at the transport level,
and of course in order to do network communication, we
have to explore the next level, which is the IP level, so
there's a lot of connection between network programmingand IP addresses, port Numbers.
An IP address is the identity of a host on the Internet.
The port number is used to send packets received by thehost to that process. Inet_aton( ) is the function used toconvert a string referred to by STRPTR into a 32-bit
network byte order binary value. The header file is#include< ARPA /inet.h>, inet_addr( ). Same as above,and returns the converted address. The TCP server andclient process is shown in Fig.7.

Fig.6. Coplanar camera

4. The OpenWRT system and construction ofdevelopment environment
OpenWRT is a small Linux system, which was originallyresearched by Apple. However, due to the use of Linuxsource code, it was forced to open the OpenWRT sourcecode to comply with the GNU Project. That allowed us to
use the prototype, which has evolved over time into an
operating system commonly used in the embedded space 5.
Environment setup: OpenWRT is an open source
system based on Linux. We can download the source codefrom its official website. Of course, before developing

Fig.7. TCP processes

The function socket( ) creates a socket, bind( ) binds theaddress and port, connect( ) establishes a connection, andlisten( ) listens for a socket. The function accept( ) accepts
connections, recv( ) receives data, and send( ) sends data,close( ) closes the socket.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

616

Qianqian Zhang，Fengzhi Dai，Jichao Zhao, Haokang Wen, Hongbo Hao

4.3 Description of WIFI video car

5. Liu gang, zhao jianchuan. Linux system transplantation(second edition). Beijing: Tsinghua University press, 2014,
2.

WIFI video car is mainly loaded with router, steering gear,
two independent 5V power supply, minimal system andcamera. The steering gear is used to control the up, down,
left and right swing of the camera. This car has beensuccessfully realized 5.
The mobile client commands are parsed, and the video
collection data is sent to the mobile phone, and the mobile
real-time monitoring system is successfully completed.
WIFI video car is shown in Fig.8.

Fig.8. WiFi video car object

5. Conclusion
Embedded technology is now regarded as an indispensablehigh technology in the field of electronics. All theexperiments in the design and coding are all belong to thecategory of embedded. In science and technology
developed today, embedded single no longer refers to thesingle chip microcomputer, it has to grow, or evenembedded into many pieces, such as embedded hardware
engineer, embedded application engineer, embedded
systems engineers, embedded driver engineer.
Our design involves the application layer and hardware,also used in the system, embedded development. Further,
we should add a voice transmission function, thus realize
the audio/video synchronization.
References
1. Li Chaoqing. Principle and Interface Technology of Single
Chip Microcomputer. Beijing University of Aeronautics andAstronautics Press, 2013.
2. li shengli. Fundamentals of analog electronics technology.
School of electrical and electronic engineering, Wuhan
institute of engineering, 2013, 13.
3. Yuan Zengmin. Basic Course of Analog Electronics
technology.Beijing: Electronics Teaching and ResearchGroup, Tsinghua University, 2014.
4. Song jingbin.Linux network programming (2nd edition).
Beijing: Tsinghua University Press, 2014, 2.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

617

Design of WebGIS transportation and distribution system based on the genetic algorithm
Hongbo Hao1 , Fengzhi Dai 1, 2 *1
2

Tianjin University of Science and Technology, China;

Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
With the concept of "smart logistics" put forward, the transformation of the logistics industry facing information
technology is facing great challenges. At present, China's logistics industry is facing problems such as extensive
resource allocation, low management level and high cost. Aiming at the transportation problem in logistics industry,
this paper designs a solution method of balanced/unbalanced transportation problem based on the Monte Carlosimilarity and genetic algorithm. Two coding methods, Prufer number and matrix are adopted. On this basis,
dynamic mutation rate and random mutation strategy are designed, and Monte Carlo similarity receiving method is
introduced. Finally, from the perspective of system requirements, the WebGIS transportation and distribution
system based on genetic algorithm is designed and developed.
Keywords: smart logistics, balanced/unbalanced, Prufer, WebGIS

1. Introduction

In this paper, a genetic algorithm based on Monte
Carlo similarity is proposed for transportation problem,
and Monte Carlo similarity operator is constructed to
avoid the algorithm falling into local optimal solution.
The research and development system has a certain
practical significance for the promotion and developmentof intelligent logistics 1.
This paper describes some basic theories involved in
the algorithm, such as genetic algorithm, Monte Carlo,
similarity theory and the related technical framework of
WebGIS transportation and distribution system. It also
includes the mathematical model and transformation
relationship of the balanced/unbalanced transportation
problem. An intelligent optimization algorithm based on
the combination of Monte Carlo similarity and genetic
algorithm is designed. The Monte Carlo similarity
receiving method is introduced, and the conclusion thatthe genetic algorithm based on Monte Carlo and adopting
matrix coding is better 2.

Modern logistics industry is based on information
technology and transportation technology. Logistics
industry is composed of goods packaging, goods handling,
goods allocation, goods management and goods storage
and transportation. These elements are developed aroundtransportation.
Therefore, the study of transportation is very importantand it has a strong practical significance. The
transportation problem is mainly to solve the problem of
allocation between producing area and selling place, andits mathematical model belongs to the category of linear
programming.
The traditional algorithm is not suitable forhigh-dimensional transportation problems. Therefore, theintelligent algorithms such as ant colony algorithm andneural network are usually used to solve thehigh-dimensional transportation problems.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

618

Hongbo Hao, Fengzhi Dai

construct a bean container to store singleton objects. The
architecture of spring is shown in Fig.2.

With the continuous enhancement of desktop processor
performance, such as the WebGIS framework represented
by ArcGIS API for JavaScript launched by Esri, thepressure on back-end services has been greatly reduced.
2. Overall scheme design
This paper uses the basic theory of genetic algorithm,
Monte Carlo thought and common similarity algorithm.
Then it introduces the key technology of system designand development. The front end uses bootstrap, Vue and
ArcGIS API for JavaScript framework. The back end isbased on the springboot framework, combined withspring, spring MVC and mybatis framework. MySQL isused as the system storage database.

Fig.2. Spring architecture diagram

2.1. Front end architecture

3. Design and solution of transportation modelThis paper uses bootstrap that is very popular nowadays.
Vue.js.Bootstrap is an open source CSS framework
developed by twitter engineers Jacob Thornton and markOtto to improve code reusability. Bootstrap is a flexible,
elegant, easy to use and extensible front-end
framework.Vue. It is a progressive framework forbuilding the front-end interface. Different from otherframeworks, it can be applied layer by layer from bottom
to top and developed based on MVVM mode Vue.JS
Framework 3. MVVM pattern is Vue.JS, one of thebiggest features of the pattern architecture that is shownin Fig.1.

3.1. Balanced transportation model
The equilibrium transportation problem is described asfollows: suppose a product has m producing areas W1,
W2...Wm and N outlets Q1, Q2...Qn. Among the output of
producing area is Wi, and the sales volume is Qj. The costof transporting a unit of goods from Wi to Qj with acertain route s in a certain period of time is Cij(t, s),which is represented by Cij below. Schematic diagram oforigin and sale of place 4 is shown in Fig.3.

Fig.3. Schematic diagram of origin and sale of place
Fig.1. Vue.JS MVVM mode architecture diagram

3.2. Fitness function design

2.2. Back end architecture

The fitness function of genetic algorithm is used as anindex to evaluate the quality of individuals in a
population. The larger the fitness value is, the better theindividual is. The fitness function is shown below.

The Springboot solves the characteristics of various
configuration, low development efficiency and complex
deployment of back-end development. The Mavendependency is simplified by using start POM, and theconfiguration is completed by annotation. The single
instance servlet improves the running efficiency. It is
adopted, combined with spring, spring MVC and mybatis
framework to complete the development. Therefore,
spring takes IOC and AOP design patterns as the core to(1)
3.3. Solving transportation problems
The algorithm process steps are as follows:© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

619

Design of WebGIS transportation

•
•
•
•
•
•

4. Design of WebGIS distribution system

Set the operation parameters;
Initial population;
The fitness of each chromosome was calculated;
The chromosomes were sorted according to thefitness;
The first 10% individuals with high fitness werereserved;
Monte Carlo similarity method is used to receive thedifference solution between the mutated individual
and the mutated parent 5.

4.1. System Requirements Analysis
The WebGIS transportation and distribution system
abandons the previous purely digital representation. It
combines the GIS system to introduce spatial
relationships into the transportation and distribution
system, and selects production and sales locations in an
intuitive and optional way. The system is mainly basedon the task creation module. It is supplemented by task
list module, OD cost module, distribution plan module,
task analysis module and personal center module.

3.4. Algorithm results and analysis

4.2. System technical route

The performance of GA algorithm based on PRü fer
number coding and matrix coding are compared. It can beseen that the convergence speed and optimal value of
TGA and algorithms are not as good as the improved
ITGA and IMGA algorithms without using strategies.
In the process of crossover and mutation of IMGA
algorithm, the generated offspring meets the constraint
conditions, while the ITGA algorithm is easy to generateinfeasible solutions in the crossover process.
The experimental results show that the final
convergence result of IMGA algorithm is better than thatof ITGA algorithm. In Fig.4, the horizontal axis is thenumber of iterations and the vertical axis is the average ofthe optimal value of each iteration in every 30
experiments. The average value of the optiomal value ofeach algorithm’s iteration is shown in the Fig.4.

The WebGIS transportation and distribution system
adopts the B/S architecture model, which reduces theburden of the client on the server and the cost of system
maintenance and upgrades. The system server level uses
the GIS server in ArcGIS Enterprise and the
Tomcat-based Web server. The system uses the SSM
framework, which is divided into a presentation layer, abusiness processing layer and a data persistence layer 6.
4.3. System database design
The design of the system is mainly based on the task
creation module. The table structure and relationship aredesigned in a task-driven way, including system user
table, task list, expense table, expense detail table,
distribution table, distribution details table, and algorithm
analysis table , sales surface, algorithm iteration record
table, etc.
4.4. System function module design
According to the actual system requirements, the
WebGIS transportation and distribution system isequipped with various module functions of WebGIS,
which are mainly divided into 6 modules: task list
module, task creation module, OD cost module,
distribution plan module, task analysis module andpersonal center module.
4.5. System function module design
The distribution plan module plays the role of query of
calculation results in the WebGIS transportation anddistribution system. This module undertakes the task
creation module and the OD expense module, obtains the
preprocessing data necessary for the algorithm. It calls
the intelligent algorithm according to the task. The
distribution plan module visually allows users to view the
distribution of production and sales. With WebGIS

Fig.4. Average of the optimal value of each algorithm’s
iteration© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

620

Hongbo Hao, Fengzhi Dai

technology, the production and sale area are marked in
the form of a map. The number and shapes are combinedto achieve the purpose of abstract digital visualization.

Industry-University Cooperation and Education Project(201802286009) from Ministry of Education, China.
References

5. Conclusion

1. Elena Rudskaia, Rudskaia Elena, Eremenko Igor.
Conceptual framework for solving problems ofcommercial transportation of goods in the city logistics
system. IOP Conference Series: Materials Science andEngineering, 2020, 918(1): pp. 012038-.
2. Liu Lang. No-core Monte Carlo shell model calculations
with unitary correlation operator method and similarity
renormalization group. Chinese Physics C, 2015, 39(5).
3. Matevž Pustišek, Andrej Kos. Approaches to Front-End
IoT Application Development for the Ethereum
Blockchain. Procedia Computer Science, 2018, 129: pp.
410-419.
4. Meijie Ji. Study on Location Selection and Optimization of
Logistics Center based on Particle Swarm Optimization.
International Core Journal of Engineering, 2020, 6(9).
5. Qi Li, Rui Miao, Jie Zhang, et al. An Intelligent
Recommendation Method for Service Personalized
Customization. IFAC-PapersOnLine, 2019, 52(13): pp.
1543-1548.
6. YONGGANG ZUO, WEIPING ZHU. Aviation Logistics
Information System Based on SSM Framework. 2017 2ndInternational Conference on Computer Engineering,
Information Science and Internet Technology, 2017, pp.
533-538.
7. A. Taivalsaari, T. Mikkonen, D. Ingalls and K. Palacz.
Web Browser as an Application Platform, 2008 34th
Euromicro Conference Software Engineering andAdvanced Applications, Parma, 2008, pp. 293-302.
8. Wiart Jaspar, Wong Elaine. Walsh functions,scrambled-nets, and negative covariance: Applying
symbolic computation to quasi-Monte Carlo integration.
Mathematics and Computers in Simulation, 2021, 182: pp.
277-295.

This article aims at the completeness and ease of use. The
specific operation process of the WebGIS transportationand distribution system is shown in Fig5.

Fig.5. WebGIS distribution system flow chart

This paper takes the transportation and distribution
system of a logistics company as the background. It
combines the Monte Carlo similarity GA transportation
problem algorithm based on matrix coding on the basis of
consideringtheactualtransportation
problemrequirements. The front-end framework uses Bootstrap,
Vue.js and ArcGIS API for JavaScript 7. The back-end
framework adopts SSM and other Web systemdevelopment technologies, and with the help of ArcGIS
platform.
The WebGIS system that meets transportation anddistribution is finally designed and realized. The system
uses Intellij idea as the integrated development
environment. The core algorithms and back-end services
are built in Java language. The front-end development
framework uses ArcGIS API for JavaScript, Bootstrap
and Vue.js, etc., and data storage and management uses
MySQL and ArcGIS DataStore 8.
Acknowledgements
The research is partly supported by the Project of Tianjin
Enterprise Science and Technology Commissioner to
Tianjin Tianke Intelligent and Manufacture Technology
Co., Ltd (19JCTPJC53700). It is also supported by the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

621

A Study of YOLO Algorithm for Target Detection
Haokang Wen 1, Fengzhi Dai 1, 2 *, Yasheng Yuan 1
1
2

Tianjin University of Science and Technology, China;

Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
With the development of deep learning, target detection has become one of the research directions of many scholars.
As one of the more mature algorithms, the YOLO series of algorithms have been widely used in real life.
Combining the development history of the YOLO algorithm, this article focuses on the main framework and maincontent of the current latest YOLOv5 algorithm, and uses the YOLOv5 model to identify and detect footballs. This
article evaluates its detection effect. The test results show that YOLOv5 has a wider application meaning in reallife.
Keywords: target detection, YOLOv5, deep learning, computer vision technology

1. Introduction

2. Object detection algorithm

Computer vision includes target detection, target
segmentation, target tracking, image description, event
detection, and activity recognition. Target detection is thecornerstone of other more complex vision tasks. Its maintask is to use computers to predict: a given image andvideo Object, what is it or where it is.
Currently, target recognition technology is widely usedin the following fields:• Security field: fingerprint recognition, facerecognition, etc.
• Military field: terrain survey, flying object
recognition, etc.
• Traffic field: license plate number recognition,
unmanned driving, traffic sign recognition, etc.
• Medical field: electrocardiogram, B-ultrasound,
health management, etc.
• Life field: smart home, shopping, etc.

With the rapid development of deep learning technology,
since 2012 target detection algorithms have shifted fromtraditional target recognition algorithms based on manualfeatures to target recognition technologies based on deepneural networks. A road map of object detection 1 isshown in the Fig.1.

Fig.1 A road map of object detection

There are currently two mainstream target detection
algorithms based on deep neural networks: one is asingle-stage algorithm represented by YOLO series
algorithms and SSD series algorithms, and the other is a

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

622

Haokang Wen, Fengzhi Dai, Yasheng Yuan

3.2. YOLO v2

two-stage algorithm represented by R-CNN series
algorithms.
The single-stage algorithm treats the target detection
process as a regression problem, and uses a unified deepneural network for feature extraction, target classification
and bounding box regression, achieving end-to-end
reasoning. It has a faster detection speed, but its detection
accuracy is lower than the two-stage algorithm.
The two-stage target detection algorithm uses theregion proposal network to extract the region of interest,
that is, the region containing the target, and then uses a
deep neural network to classify the region of interest andreturn the bounding box, which has higher detection
accuracy. However, the detection speed of the two-stage
target detection algorithm is too slow to meet thereal-time requirements.
Comparing these two methods, the single-stage target
detection algorithm is usually simpler, faster than the
two-stage target detection algorithm, and has moreadvantages in real-time detection. This article introduces
the detection algorithm based on YOLO algorithm.

YOLOv2 uses multi-scale feature maps to detect objects
based on SSD, and proposes pass through layer to link
high-resolution feature maps with low-resolution featuremaps to achieve multi-scale detection.
YOLOv2 applies the anchor mechanism on the basis of
grid constraints. By presetting a priori boxes of different
scales, the detector focuses on detecting objects that are
similar in shape to the a priori box. At the same time, thebatch normalization method is used to accelerate
convergence and avoid overfitting.
3.3. YOLO v3
YOLOv3 has made two improvements on the basis of
YOLOv2. One is to use the residual model to further
deepen the network structure; the other is to use the FPN
architecture to achieve multi-scale detection.
YOLOv3 uses the residual network idea in Res Net todesign Darkent-53 as the backbone network for feature
extraction 3. On the basis of YOLOv2, it draws on themulti-scale idea of FPN and designs 3 different scales.
Three a priori boxes are allocated for each detection
layer.
YOLOv3 predicts the position of the object through themethod of frame regression prediction, which solves the
problem of instability of the linear regression of the priorframe mechanism.
In YOLOv3, each box uses multiple label
classifications to predict a bounding box might containwhich classes 4. YOLOv3 architecture 5 is shown in theFig.2.

3. YOLO algorithm
The YOLO series algorithm is one of the representativesof the one-stage target detection algorithm. Since the first
generation YOLOv1 algorithm was proposed in 2016,
after recent years of development, there are now fivemain versions proposed. The main features of eachversion are introduced below.
3.1. YOLO v1
YOLOv1 proposes to detect the target through grid
division, and detect the target through the position of thetarget center point on the grid, which significantly
improves the detection speed.
Before the YOLO algorithm was proposed, object
detection methods were based on the method of first
generating candidate regions and then detecting.
Although there is a relatively high detection accuracy rate,
the running speed is slow.
YOLO creatively treats the object detection task
directly as a regression problem, combining the candidate
area and the detection phase into one. "You Only Look
Once", YOLO really can let you know what objects are ineach image and where the objects are at a glance 2.

Fig.2 YOLOv3 architecture© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

623

A Study of YOLO

3.4. YOLO v4

generated from the existing training data. Various
advanced data enhancement techniques are used tomaximize the use of data sets to achieve a breakthroughin the performance of the target detection framework.
Through a series of image enhancement technology steps,
the performance of the model can be improved without
increasing the reasoning delay.
YOLOv5 uses the CSPDarknet feature extraction
network to effectively extract image features, and uses
BottleneckCSP instead of shortcut residual connection tostrengthen the description of image features. The Neck
module is mainly used to generate feature pyramids. The
feature pyramid can enhance the model's detection of
objects of different scales, thereby being able to identify
the same object of different sizes and scales.

The backbone network CSPDarknet53 of YOLOv4 is thecore of the algorithm and is used to extract target features.
Drawing lessons from the CSPNet's experience inmaintaining
accuracy,
reducing
computationalbottlenecks and memory costs, YOLOv4 adds CSP toeach large residual block of Darknet53, divides thefeature map of the base layer into two parts, and merges
them through a cross-stage hierarchy. This reduces theamount of calculation and ensures accuracy. The
activation function of CSPDarknet53 uses the Mish
activation function, and the following network uses theleaky_relu function, so that the setting is more accurate intarget detection.
Unlike the YOLOv3 algorithm that uses FPN for
upsampling, YOLOv4 draws on the idea of information
circulation in the PANet network. First, the semantic
information of high-level features is propagated to the
low-level network through up-sampling, and then it isfused with the high-resolution information of theunderlying features to improve the detection effect ofsmall targets. Then increase the information transmission
path from the bottom to the top, and enhance the feature
pyramid through downsampling. Finally, the featuremaps of different layers are fused to make predictions.

4. The test of YOLOv5
There are four network models in YOLOv5: YOLOv5s,
YOLOv5m, YOLOv5l and YOLOV5x. Based on theenvironment that is shown in Table 1, the YOLOv5s
model was built.
Table 1. Computer environment
GPU
Video memory
operating system
CUDA architecture

3.5. YOLO v5
Compared with YOLOv4, YOLOv5 has a higher
accuracy rate and better ability to recognize small objects
6. YOLOv5 is more flexible and faster than YOLOv4,
and has great advantages in the rapid deployment ofmodels.
The YOLOv5 detection network uses CSPDarknet as
the feature extraction network to extract rich information
features from the input image. CSPNet solves thegradient information duplication problem of network
optimization in other large-scale convolutional neuralnetwork frameworks, and integrates the gradient changes
from the beginning to the end into the feature map, thus
reducing the amount of model parameters and FLOPS
values. This not only ensures the speed and accuracy ofinference, but also reduces the scale of the model.
YOLOv5 proposes to use the Fcos algorithm toparticipate in the calculation of the frame selection area,
which greatly improves the detection efficiency. And
through image enhancement, new training samples are

NVIDIA GeForce MX150
2 GB
Windows 10
CUDA 10.2

After training the neural network using YOLOv5s
algorithm, actual tests were performed on football
pictures and videos. As shown in the Fig.3, it is the resultof detecting football.

Fig.3 The result of detecting football© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

624

Haokang Wen, Fengzhi Dai, Yasheng Yuan

References

Experiments show that the algorithm can detectfootball in real time even in complex scenes.
Due to the impact of insufficient training times andinsufficient data sets, there are still some errors. But
overall, this method has good results and can accurately
identify footballs in many complicated scenes.

1. Zou Z, Shi Z, Guo Y, et al. Object detection in 20 years: A
survey, arXiv preprint arXiv: 1905.05055, 2019.
2. Redmon J, Divvala S, Girshick R, et al. You only lookonce: Unified, real-time object detection, Proceedings ofthe IEEE conference on computer vision and patternrecognition. 2016: pp.779-788.
3. Jing, Junfeng, et al. Fabric defect detection using theimproved YOLOv3 model, Journal of Engineered Fibers
and Fabrics, 2020(15): 1558925020908268.
4. Yuan Y, Dai F, Song Y, et al. On Fatigue Driving
Detection System Based on Deep Learning, ChineseIntelligent Systems Conference. Springer, Singapore, 2020:
pp.734-741.
5. Li Y, Zhao Z, Luo Y, et al. Real-Time Pattern-Recognitionof GPR Images with YOLO v3 Implemented by
Tensorflow, Sensors, 2020, 20(22): pp.6476.
6. Yifan Liu, BingHang Lu, Jingyu Peng, et al. Research on
the Use of YOLOv5 Object Detection Algorithm in Mask
Wearing Recognition, World Scientific Research Journal,
2020, 6(11).

5. Conclusion
YOLOv5 is an excellent target detection algorithm.
After years of development, the algorithm has been
continuously improving the network structure to maintainthe advantage of faster detection speed while maintaining
high accuracy. YOLOv5 algorithm has very considerable
prospects in future detection work. In theory, this method
has a wide range of application value in real life.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

625

Research on Recognition and Application of EEG Signal Based on SSVEP-BCI
Di Yin 1, Fengzhi Dai 1, 3*, Mengqi Yin2, Yasheng Yuan1, Yuxuan Zhu1
1

Tianjin University of Science and Technology, China;
2

3

Hebei University of Chinese Medicine, China;

Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China,
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
In recent years, brain-computer interface (BCI) systems based on steady-state visual evoked potentials (SSVEP)
have attracted attention due to their high information transfer rate (ITR) and more and more targets. The current
mainstream algorithms for SSVEP recognition have greatly improved the accuracy and target detection time. This
paper designs a robotic arm application system based on the eCCA-Y method for multi-target recognition. The
phase characteristics of CCA's sine and cosine signals are added to the EEG signal. Compared with mainstreamalgorithms, research shows that this method can improve the SSVEP-based BCI performance. And choose a
six-degree-of-freedom manipulator as the actuator of the brain-computer interface, and use a phase-encoded
stimulation paradigm for multi-target recognition to conduct experiments on the application of the proposed
method.
Keywords: Brain-computer interface, Feature extraction, Multi-target recognition, EEG signal, Steady state visualevoked potential

1. Introduction

multi-channel CCA 3. Wong et al. proposed eCCA 4，
Masaki et al. introduced the task-related component
analysis (TRCA) method to SSVEP identification 5.
We found that no phase information is seen in the sineand cosine reference signal, that is, there is no trace ofphase information in the constructed fitting signal, so analgorithm based on eCCA (eCCA-Y) 6, in the SSVEP
stimulation paradigm is proposed. The phase is added to
the reference signal, and the phase of the reference signal
is obtained by optimizing the parameters. Compared withthe existing two mainstream algorithms, the recognition
rate and ITR have a certain improvement.
In terms of application research, the EEG signal is used
to control the robotic arm equipment to verify theeffectiveness of the algorithm, complete thecommunication establishment of the two subsystems, andthe EEG signal instructions are converted into motion

Patients suffering from diseases such as muscular
dystrophy and spinal injury have physical disabilities but
normal brain functions, but they cannot interact with theoutside world like normal people. Therefore, an output
channel can be created directly by the brain to theexternal environment, without communicating throughperipheral nerves and muscle tissues 1,2. Brain-computerinterface technology is booming and can meet thisdemand. The brain signals issue commands to directly
control external devices, so that patients can restore theirbody functions and live like normal people.
In the research of brain-computer interface, the feature
extraction algorithm of EEG signal is to find a suitablespatial filter, and get the weight vector through projection
and linear combination. Zhang et.al proposed the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

626

Di Yin, Fengzhi Dai, Mengqi Yin, Yasheng Yuan, Yuxuan Zhu

3.2. The experimental data

control instructions for controlling the end effector of therobotic arm.

The data set collected SSVEP-BCI records of 5 healthy
subjects (4 males and 1 female), the average age of thesubjects was 24 years old. Before starting, the two
subjects who have been tested before will be trained withdata. For each subject, a total of 5 blocks were performed,
each with 11 targets. The stimulation paradigm has beenshown in Fig.2.
The multi-target recognition sinusoidal coding method
was used to accurately generate 11 stimulation
frequencies on the LCD display. Just as shown in Fig.2,
the location of the eleven targets on the screen
stimulation frequency corresponds to 9Hz, 9.25Hz, 9.5Hz,
9.75Hz, 10.25Hz, 10.5Hz, 10.75Hz, 11Hz, 11.25Hz,
11.5Hz and 11.75Hz.
Traverse 11 targets in turn, each target stimulus is 5s(including stimulus prompt 0.5s, stimulus flashing 4.5s).
Participants try to avoid blinking each time the stimulus
target blinks. Each time the stimulus target blinks, thesubjects try their best to avoid blinking, so after the endof each group of experiments rest for 2-3 minutes in
between. The experimental arrangement of the test data isshown in Fig.3.

2. Application of EEG Signal Recognition Basedon SSVEP-BCI System
According to the experimental intent of the stimulus
paradigm and the current position of the robotic arm, theparticipants feedback and select the control instructionsof the experimental paradigm. At the same time, the EEG
acquisition equipment performs a series of processing(preprocessing,featureextraction
andfeaturerecognition). Map the recognized instructions to theactual robotic arm commands, and the robotic arm
executes actions according to the received instructions.
As shown in Fig.1, it mainly consists of four parts:
EEG signal acquisition module, signal decoding module,
human-computer interaction module and command
output module.

Fig.3 Experimental arrangement of test data

Fig.1 Application flow chart based on SSVEP-BCI system

3.3. Data preprocessing

3. Experiment

Preprocess the collected EEG data in the EEGLAB
toolbox of MATLAB software, extract 8 channels (Pz,
PO3, POz, PO4, PO6, O1, Oz, and O2), and get the brain
electrical three-dimensional data (channel*points*trial).
In order to make the collected EEG data moreconvincing in the subsequent feature extraction andanalysis process, the number of the experimental groups(block)ismergedintothepreprocessed
three-dimensional EEG data and thus obtain thefour-dimensional data (channel*points*trial*block).
Down sampling is performed from 1000Hz to 250Hz, andeach trial contains 3000 sampling points, which forms
8*750*11*5 EEG data.

3.1. Experimental paradigm
The EEG acquisition system based on SSVEP-BCI
compiles some inherent actions of the robotic arm
through MATLAB's PSYCHTOOLBOX (PTB) toolbox
to form a specific stimulation paradigm. The stimulus
paradigm formed at the end of this experiment is shownin Fig.2.

4. Feature extraction

Fig.2 Manipulator stimulation paradigm based on
SSVEP-BCI control© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

627

Research on Recognition and

The reference signal is:

When the subjects are training data, they can obtain threekinds of multi-channel signals:
1. Test data 𝑋𝑋(𝑡𝑡) ∈ 𝑅𝑅𝑁𝑁𝐶𝐶×𝑁𝑁𝑆𝑆 ×𝑁𝑁𝑡𝑡
2. The average signal of the subject's training data
𝑁𝑁 ×𝑁𝑁𝑋𝑋�𝑘𝑘 𝑐𝑐 𝑠𝑠
3. Constructed sine and cosine reference signal 𝑌𝑌𝑓𝑓𝑓𝑓
Among them, 𝑁𝑁𝐶𝐶 ，𝑁𝑁𝑆𝑆 ，𝑁𝑁𝑡𝑡 represent channel, points,
and test set trial respectively. Any two signals can beused to calculate a spatial filter based on CCA.
Chen et al.7 combined the CCA system with thePearson correlation coefficient of the test training data of
the subjects and proposed a method to extend CCA. Only
three spatial filter forms are taken, namely:
1. Testdata
andaveragetraining
data
𝑊𝑊𝑋𝑋 �𝑋𝑋𝑋𝑋�𝑘𝑘 �, 𝑊𝑊𝑋𝑋�𝑘𝑘 �𝑋𝑋𝑋𝑋�𝑘𝑘 �
2. Test data and reference signal 𝑊𝑊𝑋𝑋 �𝑋𝑋𝑌𝑌𝑓𝑓𝑓𝑓 �
3. Average
training
dataandreference
signal 𝑊𝑊𝑋𝑋�𝑘𝑘 �𝑋𝑋�𝑘𝑘 𝑌𝑌𝑓𝑓𝑓𝑓 �
According to the above three spatial filters, four
correlation coefficient combinations as in formula (1) areselected: rk (1)  r (2) 
k

rk =
=
 rk (3)  rk (4)  ρ ( X T WX ( XY fk ), Y fkT WY fk ( XY fk ))  ρ ( X T WX ( XXˆ k ), Xˆ kT WX ( XXˆ k ))



T
T
ˆ
 ρ ( X WX ( XY fk ), X k WX ( XY fk ))  ρ ( X T WXˆ ( Xˆ k Y fk ), Xˆ kT WXˆ ( Xˆ k Y fk )) kk sin(2π f k t + θ k )  cos(2π f t + θ ) 
k
k



Y fk ,θk =  sin(2π N h f k t + N hθ k ) cos(2π N f t + N θ ) 
h k
h k 


T

（2 ）

According to the experimental intent, the stimulus
paradigm designed by the researcher determines the sizeof 𝜃𝜃𝑘𝑘 , as expressed by formula (3):（3 ）
θ k = θ 0 + ∆θ × (k y − 1) × 5 + (k x − 1) Among them, 𝑘𝑘𝑥𝑥 and 𝑘𝑘𝑦𝑦 represent the row andcolumn indices of the visual stimulation matrix,
respectively. 𝜃𝜃0 represents the initial phase, and ∆𝜃𝜃represents the phase interval.
5. Research on Application of Manipulator Basedon SSVEP-BCI（1）

The correlation coefficient is fused to obtain thecorrelation coefficient at the k-th stimulus frequency, andthen the classification can be completed by selecting thelargest correlation coefficient. The principle analysis
diagram is shown in Fig.4.

The robotic arm application based on SSVEP-BCI is
composed of two systems, namely the BCI EEG
acquisition system and the robotic system. The BCI
subsystem and the robot subsystem communicate throughthe UDP protocol. The BCI subsystem is composed of awireless EEG acquisition system, LCD display and online
processing of EEG data.
The EEG acquisition equipment is responsible forconverting the recorded EEG signals into digital signals
through AD, and through online real-time processing(feature extraction and conversion algorithms) to generatecontrol instructions that can be recognized by the roboticarm. The reason for joining the application of robotic armin this article is to study whether the robotic arm canprovide convenience for the disabled and solve the simple
grasping movements necessary in life.
The steps of the application design in this article are: (1)collect the subject’s EEG data and record it as the
subject’s EEG intent, (2) perform data processing andanalysis on the saved data in the MATLAB software, (3)and finally convert it into the control command of therobotic arm and send it to the controller.
In this experiment, the EEG data of 5 students inschool was collected, feature extraction and recognitionand classification were performed, and finally it wastransformed into instructions that can control the roboticarm and sent to the robotic arm controller.

Fig.4 eCCA-Y schematic diagram© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

628

Di Yin, Fengzhi Dai, Mengqi Yin, Yasheng Yuan, Yuxuan Zhu

Acknowledgements

This experiment analyzes the data of 5 subjects. The
number of harmonics is 2 and the data length is 750
sampling points to compare the classification effects ofthe two algorithms eCCA and eCCA-Y, as shown in
Fig.5.

The research is partly supported by the Project of Tianjin
Enterprise Science and Technology Commissioner to
Tianjin Tianke Intelligent and Manufacture Technology
Co., Ltd (19JCTPJC53700). It is also supported by theIndustry-University Cooperation and Education Project(201802286009) from Ministry of Education, China.
References（a）

1. Chang H C, Lee P L, Lo M T, et al．Independence of
amplitude-frequency and phase calibrations in an
SSVEP-based BCI using stepping delay flickering
sequences．IEEE Transactions on Neural Systems andRehabilitation Engineering, 2011, 20(3) : pp.305-312．
2. Mak J N, Wolpaw J R. Clinical applications ofbrain-computer interfaces: current state and future
prospects. IEEE reviews in biomedical engineering, 2009,
2: pp.187-199．
3. Zhang Y.et al. Multiway canonical correlation analysis forfrequency components recognition in SSVEP-based BCIs.
International conference on neural information processing.
Springer, Berlin, Heidelberg, 2011: pp.287-295.
4. Wong CM, Wan F, Wang B, et. al. Learning acrossmulti-stimulus enhances target recognition methods in
SSVEP-based BCIs. J. Neural Engineering, 2020, 17:016026.
5. Nakanishi M, Wang YJ, Chen XG, Wang YT, Gao XR,
Jung TP. Enhancing detection of ssveps for a high-speed
brain speller using task-related component analysis. IEEE
Transactions on Biomedical Engineering, 2018, 65: pp.
104-112.
6. Yin D, Dai F, Song Y, et al. On Improvement of Target
Recognition eCCA Method Based on SSVEP. ChineseIntelligent Systems Conference. Springer, Singapore, 2020:
pp. 742-749.
7. Chen X, Wang Y, Nakanishi M, et al．High-speed spellingwith a noninvasive brain-computer interface．Proceedingsof the national academy of sciences, 2015, 112(44): pp.
E6058-E6067.

（b）

Fig.5 eCCA-Y schematic diagram

It can be seen from Fig.5(a) that the optimized
eCCA-Y algorithm in this paper has improved
recognition rate. Among them, S5 has the same accuracy
rate for the two algorithms, and the recognition rate of S3
is relative to the other four. The test is low, S2 subjects
have the largest difference in recognition rates between
the two algorithms. In terms of ITR, as shown in Fig.5(b),the eCCA-Y algorithm is also improved, and S5 also
shows the same effect as the recognition accuracy. The
two algorithms are the same in terms of ITR. In addition,
S4 subjects have the largest difference in ITR for the twoalgorithms.
6. Conclusion
Based on the SSVEP-BCI system to carry out application
research experiments on the robotic arm, a multi-target
recognition stimulus paradigm for the robotic arm isdesigned. The UDP protocol is used to realize thecommunication between the robotic arm and thebrain-computer interface system. It is added at different
frequencies. Different phases simplified the control
method, refined the control of the robotic arm movement,
and weakened its complexity.
Two feature recognition algorithms, eCCA andeCCA-Y, are used to process data, verifying the roboticarm and the brain-computer interface. The application
realizes the interactive control of the robotic arm, andadds a stroke to the feature processing algorithm, which
provides experience for the development of the BCI
system.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

629

Research on Bad Driving Detection Based on Behavior Recognition
Yasheng Yuan1, Fengzhi Dai1, 2, *, Di Yin1, Yuxuan Zhu1

Tianjin University of Science and Technology, China;
Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China;
E-mail:*daifz@tust.edu.cn
www.tust.edu.cn
1

2

Abstract
Dangerous driving behavior is considered to be the direct or indirect reason of road accidents. Although artificial
video surveillance is good to prevent bad driving, it wastes too much time and manpower. How to effectively
identify behavior becomes the focus of the research. In recent years, deep learning showed the huge advantage inthe field of computer vision. This paper adopt a number of deep learning network models, mining video integrationof space and time features, introduction of analogy in human visual attention mechanism, improve the model deeply,
using the LSTM to accurate and efficient video behavior analysis technology.
Keywords: deep learning, fatigue testing, convolutional neural network, action recognition

1. Introduction

operation will greatly improve the target detection
model miscalculation, so network have time-level
characteristic behavior identification, will have highresearch significance.

With the maturity and development of internet
technology, especially the development of video
surveillance system, people tend to use more and moremonitoring method to prevent some accidents. Video
monitoring is an important part of prevention measures,the importance and actual efficiency were higher thanhuman security. In the highly security demand ofmodern life, it is an essential part of a high and newtechnology, the action of driving of the driver is thatpeople need to use one of monitoring. Research 1 showsthat bad driving behavior of drivers is an importantcause of traffic accidents. How to effectively monitor
the driver's behavior is the key to safe driving.
Compared with traditional detection technology, deeplearning technology shows the strong feature extraction
ability. For target detection that performs well in deeplearning, although it can more accurately detect thesmoking 2, but for the cryptic action of fatigue, notime-level characteristics of target identification is
obviously weak 3. Especially in the video, the driver's
side head, bowed their heads and normal driving

2. Main research method
Based on the deep learning on the driver's action
recognition classification, through the analysis of the
behavior recognition algorithm and the network model.
Two-stream network is our choice as our main network,on the basis of the Two-stream network add amount
Conv layers, improve the effect of feature extraction
and increase the nonlinear. Adding BN layers andDropout layers to prevent over fitting, and chooses
appropriate activation function to further reduce theamount of calculation. The LSTM (Long Short-Term
Memory) layer is added to further learn the timing
information between behaviors and mine the long timedependence relationship of video sequence.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

630

Yasheng Yuan, Fengzhi Dai, DiYin, Yuxuan Zhu

2.1 Data set processing

respectively. The resulting optical flow diagram isshown in Fig.2.

The training data set for this project contains 13,360
YouTube clips from the UCF101 data set and driving
videos from volunteers. The videos mainly include five
types of movements: human and object interaction, only
body movements, human interaction, playing musicequipment, and various kinds of sports. 22 kinds ofvideos of facial movements from UCF101 data set isselected. About 300 short videos in total, this videos
have bad driving behaviors, such as smoking, talking on
the phone, taking a nap and looking left and right, whichwere collected by ourselves from the vehicle camera
during driving.
First, each video is divided into several short videos
of about 120 frames (10s). After that, each short video is
extracted frame by frame into RGB images, which areused as the input of spatial flow. Then, through the
OpenCV and CUDA, the RGB frames extracted fromthe video are converted into optical flow diagrams 4.
The optical flow is the instantaneous velocity of
pixels moving in space on the observed imaging plane.
The optical flow method uses the change of pixels in thetime domain and the correlation between adjacentframes. The method find the corresponding relationshipbetween the previous frame and the current frame. It is amethod to calculate the motion information of objects
between adjacent frames. The projection of the motion
for the object in the three-dimensional space on thetwo-dimensional imaging plane is shown in Fig.1. The
result is a two-dimensional vector describing the changeof position. In the case of minimal motion interval, we
usually regard it as a two-dimensional vector describing
the instantaneous velocity of the point u = (u, v), whichis called the optical flow vector 5.

Fig.2. Optical flow frame and image frame

2.2 Net selection
The two-stream convolutional network 6 is added bysome network in this paper, which is divided into spatialconvolutional network and time convolutional network.
The spatial-flow Convolutional network operates on asingle video frame. Because some behaviors arestrongly related to specific scenes and objects, it iseffective to recognize behaviors from still images. As
CNN (Convolutional Neural Networks) network isalready a powerful image recognition algorithm, it canbuild a video recognition network based on large-scale
image recognition algorithm. It can also utilize thepre-training network on the existing image classification
data set. Time-flow convolutional network is different
from ordinary CNN network. The input of time-flow
convolutional network is formed by stacking opticalflow displacement field between several consecutive
frames. This input is characterized by the motion
between the video frames, making the recognition
process easier (no network implicit estimation of the
motion is required).

Fig.1. Optical flow method

TVL1 optical flow method in OpenCV to extraction
is used in this paper. Each image obtained an opticalflow diagram extracted from the X and Y directions
Fig.3. Two-stream Convolutional network© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

631

Research on Bad Driving

Layer_2_Conv1_sum
Layer_2_Conv1_expand
BN &Relu
Layer_2_Conv2_1
BN &Relu
Layer_2_Conv2_2
BN &Relu
Layer_2_Conv2_3
Layer_2_Conv2_sum
BN &Relu
Layer_2_Conv3_1
BN &Relu
Layer_2_Conv3_2
BN &Relu
Layer_2_Conv3_3
Layer_2_Conv3_sum
Conv_final
BN &Relu
Pool
Dropout
Global_pool
FC
Loss

The two-stream convolutional network is shown in
Fig.3. The algorithm is divided into two branches ofconvolutional neural network for feature extraction.
First, a frame of RGB image in the video stream was
input. After five layers of convolution and three layersof sampling, it was passed into the full connection layerof the two layers for feature extraction. The extracted
spatial feature was input into the Softmax classifier for
category prediction. Second, the optical stream features
extracted from the video stream were input into the
convolutional neural network with the same feature
extraction branch of pixel space. Finally it passed into
the Softmax classifier for category prediction.
2.3 Identify face areas
The network behavior identification network selected inthis paper is improved based on two-stream, and thenumber of network layers is deepened on this basis.
Before each lower sampling layer, a batchsize
normalization layer, which means BN (Batch
Normalization) & Relu 7, is added to prevent theover-fitting caused by too few training samples and too
deep network. The network structure is shown in Table
1.

1
1

4

1

4

3

4

1
1

4

1

4

3

4

1

8

4
4
0.2
Avg
16

320
320
128
128
320
320
123
128
320
512
---

The BN layer and Dropout mechanism is added,
which is used to a certain extent and avoid network offitting. Relu is selected as the activation function,change the sigmoid disappeared as a result of activation
function gradient (sigmoid close to the saturated zone,change is too slow, derivative tends to zero) to complete
the deep net training. The interdependencies between
parameters are reduced, which can reduce the happen ofover fitting. Through the proper superposition of 1x1
Convolutional layer and 3x3 Convolutional layer, thefeatures are further extracted while the computation is
reduced as much as possible. Finally Global pool isadded to simplify the calculation.

Table 1. Network structure
Layer name
Convolution
Convolution
kernel size
kernel number
Flow_conv_begin
7
64
BN &RELU
4
Conv_begin_pool
3--Layer_1_Conv1_1
1
64
BN &RELU
4
Layer_1_Conv1_2
3
64
BN &Relu
4
Layer_1_Conv1_3
1
256
Layer_1_Conv1_expand1
256
Layer_1_Conv1_sum
1256
BN &Relu
4
Layer_1_Conv2_1
1
64
BN &Relu
4
Layer_1_Conv2_2
3
64
BN &Relu
4
Layer_1_Conv2_3
1
256
Layer_1_Conv2_sum
1256
BN &Relu
4
Layer_2_Conv1_1
1
128
BN &Relu
4
Layer_2_Conv1_2
3
128
BN &Relu
4
Layer_2_Conv1_3
1
320

2.4 Facial image processing
LSTM is an improvement based on the traditional cyclic
neural network (RNN). It is very suitable for processing
video sequences with time-dimension characteristics. In
addition, LSTM network can avoid the gradientdisappearance and gradient explosion problems in thepractical application of RNN network.
In the classical LSTM structure, there are three"gates": (1) Oblivion gate, which is used to help RNN
selectively forget some historical information; (2)Memory gate, which is used to strengthen the memoryof some historical information; (3) Output gate, which isused responsible for comprehensive consideration of© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

632

Yasheng Yuan, Fengzhi Dai, DiYin, Yuxuan Zhu

context information of the input video sequence isobtained through the LSTM layer. Two Softmax layers
and two cross entropy loss layers are deployed after thefeature output layer and the LSTM layer, respectively.
The supervised information of the video category isused to drive the joint training of the LSTM and CONV
networks. The hidden state of the LSTM layer cancapture the time evolution of a specific category in thevideo sequence.

long and short term memory information to generate
output signals.
In this paper, LSTM network (Fig.4) is also used forvideo signal timing modeling.

Fig.4. LSTM network structure

The key of the LSTM is cell state. The unit state iskind of like a conveyor belt. It goes all the way down
the chain in a straight line, with only a few smaller
linear interactions. It is very easy for information toflow unaltered.
State of cell transformation, namely value C
transformation, have two operations. The two operation
determines the forgotten and update of the computed
tomography Ct-1. First of all, Ct-1 through themultiplication of point-wise operation, this step
determines whether the Ct-1 value is forgotten. The
number multiplied by Ct-1 comes from the lower layer,
the bottom of the data through a sigmoid layer, throughthe data values are below (0, 1). In general, when the
input value is greater than 3 or less than -3, sigmoid
value is close to the 1 and 0. That is to say, the value of
C is generally going to be close to Ct-1 unchanged, or 0,
which means that you remember C or you forget Ct-1. Of
course, a number between (0, 1) multiplied by Ct-1
means how many Ct-1 values you need to remember.
Next, the value of Ct-1 encounters an addition, which is achange to the value of Ct-1, such as the addition of new
information, which is generally understood assomething new to be remembered 8.

Fig.5. Two-stream-LD network structure

4. Experiment and Conclusion
All test sets are used to experiment on the model trainedby the initial network, and the normalized confusion
matrix of various behavior recognition rates is shown in
Fig.6.

3. Improved PERCLOS algorithm

Fig.6. The normalized confusion matrix of various behavior
recognition rates

The network structure of this paper is shown in Fig.5.
First, after the image flow and optical flow respectively
pass through the convolutional network, the timing

In the matrix, the vertical axis represents the reallabel of a certain video behavior, and the horizontal axis

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

633

Research on Bad Driving

comprehensive comparison, the method proposed in thispaper is of certain research significance.

represents the predictive label of a certain video
behavior.
The darker the grid matrix color is, the higher therecognition rate is. And the lighter the grid matrix color
is, the lower the recognition rate is. In the darker
diagonals, the data contained represent the correct
recognition rate for a certain type of behavior, while thedata in the other light colored squares represent thewrong recognition rate when a certain type of behavior
is predicted to be other behaviors.
Altogether 25 types of human behaviors wereidentified in this experiment. Among them, there aresimilar frames among the behaviors that need to use
hands in front of the face, such as smoking, making
phone calls and making up, so the error recognition rate
among these people may be relatively high.
Based on the data in the confounding matrix, the dual
flow convolutional neural network designed in thispaper can effectively complete the task of recognizing
and classifying bad driving behaviors.
Finally, the results are fused at the decision level. For
the classification results obtained by multiple classifiersin the classification problem, if each classifier has
different influences on the final prediction results, thenthe fusion can be considered in the way of weighted
sum. We will find that, compared with speech andsubtitle, the visual features contained in the image
sequence are more helpful to determine the accurate
behavior contained in the video. Therefore, we choose
to give a larger weight to the classifier results of thevideo images, and give a smaller weight to theclassification results of other classifiers.
The final result is shown in Table 2, from which canbe seen that the Two-stream did well in UCF101 dataset, this means that the Two-stream network in
extracting time has good extraction effect when the flow
characteristics, but in RGB space flow identification
process is not ideal, but in our network ofTwo-stream-LD, both the space and time flows
recognition effect has a significant improvement.
In the detection of bad driving behaviors, theTwo-stream-LD network can be used to better identify
various actions of the driver in the video. After

Table 2. Final result

References
1. State of the States Report on Drowsy Driving, National
Sleep Foundation, Nov. 2007, U.S.A.
2. Yasheng, Y, Fengzhi, D, Yunzhong, S, et al., On Fatigue
Driving Detection System Based on Deep Learning.
Proceedings of 2020 Chinese Intelligent Systems
Conference (CISC). Lecture Notes in ElectricalEngineering, 2020, pp.734-741.
3. Yasheng, Y., Fengzhi, D., Lingran, A, et al., Research on
fatigue detection method based on deep learning.
Proceedings of 2020 International Conference on
Artificial Life and Robotics, Oita, Japan, 2020: pp.
640–643.
4. Zach C, Pock T, Bischof H. A Duality Based Approachfor Realtime TV-L1 Optical Flow. Proceedings of the
29th DAGM conference on Pattern recognition, 2007: pp.
214–223.
5. Wang L, Xiong Y, Wang Z, et al. Temporal Segment
Networks: Towards Good Practices for Deep Action
Recognition, European Conference on Computer Vision.
Springer, Cham, 2016: pp.20-36.
6. Feichtenhofer C, Pinz A, Wildes R P. Spatiotemporal
Residual Networks for Video Action Recognition. IEEE
Conference on Computer Vision and Pattern Recognition,
2017: pp.3476-3484.
7. He K, Zhang X, Ren S, et al. Deep Residual Learning for
Image Recognition. IEEE Conference on Computer
Vision & Pattern Recognition. IEEE Computer Society,2016: pp.770-778.
8. Donahue J, Hendricks L A, Rohrbach M, et al. Long-term
Recurrent Convolutional Networks for Visual
Recognition and Description. IEEE Transactions on
Pattern Analysis & Machine Intelligence, 2017, 39(4):pp.677-691.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

634

Research on Crop Image Recognition Technology - Take Daylily as an Example
Jichao Zhao, Fengzhi Dai

College of Electronic Information and Automation, Tianjin University of Science and Technology,
Tianjin, China,
China E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract

Digital image recognition technology is the core technology of agricultural robots. The key researchcontent is to judge the maturity of crops through image recognition. In this paper, the daylily is used as
the crop to be identified, and digital image processing technology is used to separate the daylily from thebackground image, and then send the processed information to the picking actuator of the robot to assist
the actuator in picking tasks. This method can not only be used in the recognition of daylily but also canbe used in the recognition of tomatoes, cucumbers, and other crops by adjusting the recognition picturesand recognition parameters. It has high scalability.
Keywords: Crop Picking; Daylily; Digital Image Recognition; Maturityof identification and positioning of agricultural products,
and has a direct impact on the picking effect of the robot.
The visual system is generally composed of camera,
image processing device, distance measuring device andcomputer. At present, the number of commonly usedcameras is 1 or 2, namely monocular or binocular visionsystem. Images of the produce are captured by a camera
before picking, then processed to identify the produce
and its branches and leaves, and finally to determine theexact spatial coordinates of the produce 3-5. Finally,
actuators such as manipulator are driven to pickagricultural products at a certain Angle and strength. Inorder to avoid the influence of natural light, artificial lightsources or optical filters are often needed to reduce theinfluence of surface shadows on the recognition effect ofagricultural products.
In this paper, a crop recognition algorithm is proposed,
which can extract the crops to be recognized from thecrop image collected from the camera of agriculturalrobot. The extraction effect of the algorithm was tested

1. Introduction
Agricultural robot refers to a kind of flexible automationor semi-automation equipment that takes agricultural
products as the operating object, and has some human
information perception and limb action functions. It is akind of flexible automation or semi-automationequipment that integrates a variety of disciplines 1.
Agricultural robots can gradually replace human laborand continuously help agricultural production to reducelabor intensity. At the same time, they can also improvelabor efficiency and help solve the problem of laborscarcity faced by many countries. Agricultural machinery
has been paid more and more attention by developed anddeveloping countries with less agricultural population,and has become one of the focal points of international
agricultural equipment industry technology competition.
Agricultural machinery picking operations, first of all
through the visual system for the identification andpositioning of agricultural products 2. The performance ofthe visual system determines the accuracy and efficiency© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

635

Jichao Zhao, Fengzhi Dai

2.1 HSV Color Spacewith daylily as the main example. The generality of thealgorithm is verified by testing on tomato.

The commonly used color space is RGB color space.
Each pixel of the image contains three channels consistof Red, Green, Blue, and the final pixel color is composedof three colors mixed together. RGB color space is the
cube model. HSV is a kind of color space created basedon the intuitive characteristics of colors. It is also known
as a Hexcone Model. The RGB color model is hardwareoriented, while the HSV color model is user-oriented.
RGB space model and HSV space model are shown in
the Fig.3.

2. Recognition Algorithm
According to their growth characteristics, crops can bedivided into two categories: color changes before andafter maturity, and color does not change. For instance,
daylily and tomato fall into the category of color change,
while cucumbers and beans fall into the category of color
un-change.
Daylily is mainly distinguished from the color of its
flower, which is cyan when immature and light yellow
when mature. As shown in Fig.1, ripe daylilies are on theleft and immature daylilies are on the right.

Fig.3. Color-Space, left is RGB-Space, right is HSV-SpaceIn HSV space, H represents Hue, which is measured at
an Angle ranging from 0° to 360°. The value is calculated
counterclockwise from red, which is 0°, 120° in green,and 240° in blue. Their complementary colors are: yellow
is 60°, cyan is 180° and purple is 300°. S is the Saturation,
which indicates the degree of Saturation of colors. If thewhite component of spectral color is 0, the saturation
value is highest, the color is deep and bright. Generally,
the value range of S is 0% ~ 100%. The larger the value,
the more saturated the color is. V is Value, representing
the brightness of the color, ranging from 0 to 1.
Compared with RGB space, HSV space can intuitively
express the brightness, tone and brightness of colors,
which is convenient for color contrast and emotionaltransmission. Therefore, HSV color space is selected to
identify daylily. In addition, the main parameter that
affects the identification is Hue, and we constantly adjust
the H value to identify the daylily with good maturity.
The effect of daylily extracted through HSV color space
is shown in Fig.5(a).

Fig.1. Daylily, left is ripe and right is immature

Our algorithm is mainly applicable to the extraction ofcrops with different colors before and after maturity. The
algorithm flow chart is shown in Fig.2.

2.2 Noise Filtering
When crops are extracted by color, similar color areasthat do not belong to the crops to be extracted will also
be selected, so further screening is needed. Look at
Fig.5(a), the characteristic of the noise area is relatively

Fig.2. Algorithm Flow

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

636

Research on Crop Image

3.1 Daylily

fragmentary and disconnected, so the noise can befiltered through the filtering algorithm.
Common filtering algorithms include median filteringand mean filtering. In this paper, median filtering
algorithm is used to filter the image noise after testing thetwo effects. The effect of median filtering is shown in
Fig.5(b).

The daylily pictures used in this experiment were taken
by our team members using mobile phones in the daylily
planting base. The daylily is shown in Fig.4.

2.3 Dilate and Erode
Dilate is the operation of finding the local maximum. In
contrast to dilate, erode is the operation of finding a local
minimum. Erode followed by dilate is called open
operation that can remove isolated small points, while thetotal position and shape remain unchanged. After dilate,erode is called close operation that can fill the small holes
and bridge the small gaps, which also does not change theshape and position of the image.
In this paper, we will start with the open operation toremove the noise other than the extraction of daylily.
Then the close operation is used to fuse the small distancearea and fill the blank area inside the image. Finally, themorphologic images are obtained.

Fig.4. Daylilies original image

2.4 Choose Large Connection Domain(a) After HSV color space extraction

After the morphological processing is completed, there
are still large areas in the image that have not been
filtered out, so the image can be further screened by
selecting the large connection domain.
Moreover, since the mature daylily is yellow-green,
there is a clear color difference from the immature daylily.
The color extraction method not only separates the daylily from the branches, but also can judge whether the
day-lily is mature or not. When it is mature, it can bedetected and recognized, while when it is immature, itwill automatically ignore the effect. The extracted daylily
is shown in Fig.5(c).

(b) After morphological processing and noise filtering

3. Experiments
In this experiment, daylily was selected as the identified
crops and were extracted from the pictures by adjusting
the values of the algorithm parameters. Meanwhile, theextraction effect of tomato was also verified. But theparameter value of tomato is different from that of daylily,
which also proves the universality of the algorithm.

(c) After selecting the large connection domain
Fig.5. Extraction results of daylilies© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

637

Jichao Zhao, Fengzhi Dai

4. Conclusions

The parameters used to extract daylilies are shown in
Tab.1 below.

This paper presents an algorithm for crop extraction,
which can extract crops with different colors before andafter ripening. The algorithm was tested on daylily andtomato, and the extraction effect of the algorithm wasdemonstrated.
The validity and generality of the algorithm areverified by experiments. In addition, it can be seen fromthe experiment that this algorithm can be applied to theextraction of other crops only by adjusting the parametervalues in the HSV color space.

Tab.1. Parameters of daylily extraction

Step Name

Param

Value / Range

H[0.10, 0.20]

S[0.26, 1.00]

V[0.50, 1.00]

noise filter

median-erode and dilate

radius5px

large connection
domain>threshold[50, +\infty]

HSV-spaceextraction

Acknowledgements
The research is partly supported by the Project of Tianjin
Enterprise Science and Technology Commissioner to
Tianjin Tianke Intelligent and Manufacture Technology
Co., Ltd (19JCTPJC53700). It is also supported by theIndustry-University Cooperation and Education Project(201802286009) from Ministry of Education, China.

3.2 Tomato
At the same time, we adjusted the parameters of thealgorithm to carry out the experiment of tomato
extraction from Fig.6 (left), the result of which is shownin Fig.6 (right).

References
1. D. Khort, A. Kutyrev, et al. Robotized Platform for
Picking of Strawberry Berries. 2019 IEEE International
Scientific-Practical
Conference
ProblemsofInfocommunications, Science and Technology (PIC S&T),
Kyiv, Ukraine, 2019, pp. 869-872.
2. Patel, Alakh and Jaymin Bhalani. A Review on in Field
Cotton Recognition for Cotton Harvesting Robot based on
Image Processing Technique. International Journal ofComputer Applications 180 (2018): 6-8.
3. K. N. e. Alam Siddiquee, M. S. Islam, et al. Detection,
quantification and classification of ripened tomatoes: acomparative analysis of image processing and machinelearning. IET Image Processing, vol. 14, no. 11, pp. 24422456, 18 9 2020.
4. A. Wang, Y. Xu, et al. Semantic Segmentation of Crop andWeed using an Encoder-Decoder Network and Image
Enhancement Method under Uncontrolled Outdoor
Illumination. IEEE Access, vol. 8, pp. 81724-81734, 2020.
5. M. W. Liu, M. Ozdogan and X. Zhu. Crop TypeClassification by Simultaneous Use of Satellite Images ofDifferent Resolutions. IEEE Transactions on Geoscienceand Remote Sensing, vol. 52, no. 6, pp. 3637-3649, June2014.

Fig.6. Tomatoes extraction

Since the extraction parameters of tomato are the sameas daylily except for values of HSV, only these three
items are shown in Tab.2.
Tab.2. Parameters of tomatoes extraction

Step Name
HSV-spaceextraction

Param

Value / Range

H[0.00, 0.10]

S[0.50, 1.00]

V[0.50, 1.00]© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

638

Design of Daylily Agricultural Picking Robot
Jichao Zhao, Fengzhi Dai *College of Electronic Information and Automation, Tianjin University of Science and Technology,
Tianjin, China
China E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
Agricultural picking activities occupy an important role in daily life, and the research and development of agricultural
picking robots play a more crucial role and increasingly become an advantageous tool to improve agricultural
productivity. Daylily, as a crop for daily consumption, can significantly reduce serum cholesterol and has extremely
high nutritional value. However, the picking conditions of daylily are harsh, and long-term picking is likely to cause
extremely serious harm to the human body, especially the hands. This paper designs an agricultural picking robot thatcan pick daylily automatically, which can greatly free hands and improve labor productivity. Especially in health, it
can significantly reduce the harm to the human body.
Keywords: Picking Robot, Daylily, Agricultural, Robot Design

daylily picker. The production efficiency of this picking
device is 1.45 times higher than the traditional production
method, and the production cost is reduced by 25.2%.
This effectively solves the problem that daylily pickers
are soaked by dew and harms their bodies, significantly
improves the working conditions of daylily pickers,
reduces labor intensity, and has obvious economic andsocial benefits.
This paper designs a daylily picking robot 1. The robot
uses a camera to collect images, determines the positionof the daylily, and uses a parallel robotic arm with cutting
and grabbing functions to pick the daylily and put it into
the designated position. The equipment will have a set ofimage processing algorithms based on machine learningneural networks to determine the horizontal and vertical
positions of daylily 2, 3. The algorithm can autonomously
learn the characteristics of a daylily, such as color,
contour shape, size, etc., and then judge whether thephotographed object is a daylily based on the above
features 4. If it is daylily, then execute the picking

1. Introduction
Daylily, also known as Golden Needle, has beencultivated in China since ancient times. The flower of
daylily has high ornamental value, and its buds are pickedand dried to become vegetables with high nutritional
value. Daylily usually grows buds at dusk every day, theflowers bloom the next morning and wither around noon.
According to daylily’s physiological characteristic,
during the flower picking season, a special person mustbe fixed at 5～8 a.m. for picking activities. Otherwise,
nutrients will be lost after blooming, and both the quality
and the price will be low. The harsh picking conditions
have greatly affected the picking efficiency of daylily,
and long-term picking activities will cause certain harmto the human body. Therefore, agricultural robots that
automatically pick daylily have become an important andurgent need.
Regarding the research of automated daylily picking,in 2012, Shanxi Province, China, developed a riding style© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

639

Jichao Zhao, Fengzhi Dai

2.1. Vision Systemprocedure, if not, then ignore and continue to identify thenext one.

In this design, the vision system needs to meet thefollowing requirements: 1) Can effectively identify thematurity of daylily, selective picking, to ensure thequality of picked daylily; 2) Identify the position
coordinates of daylily in space, including horizontal andvertical coordinates and the distance between the camera
module and daylily, assist the picking of the manipulator;
3) It can judge whether the robot is still in the field byidentifying the environment, and assist the robot inwalking to a certain extent.
Therefore, we choose the RGB-D binocular camera
with depth function as our visual system 5.

Fig.1. Pick daylily

2. Robot Design
The agricultural robot designed in this paper includes avision system, picking structure, mobile structure,
microcontroller, and power system. The functional
structure of the robot is shown in Fig.2.

Fig.3. The main global currents

The camera module has the following characteristics:• Depth range: 0.6-8m• Depth map resolution: 1280*1024 max• Depth filed angle: 58.4*45.5cm• Time delay: 30-45ms• RGB: 1080P• Connection type: USB
2.2. Picking Structure
When using a manipulator to pick daylily, it is important
to ensure that it will not cause secondary damage to thecrop, thus the common rigid manipulator cannot be used.
Here we use an under-actuated flexible manipulator toperform the picking work. When picking, themanipulator relies on the cylinder to provide pressure todrive the fingers to close. This manipulator adopts a
pneumatic drive. Compared with an electric drive, it canfurther adjust the gripping force of the manipulator by
adjusting the pneumatic pressure to protect the daylily
from being damaged 6. The manipulator is shown in Fig.4.

Fig.2. The functional structure of the robot

Through the vision system to identify the maturity of
daylilies and the location information on the branches,the picking structure uses the recognized image
information to separate the ripe daylilies from thebranches. The mobile structure is responsible for the
movement of the robot. The microcontroller is used toprocess the information sent back by the recognition
module, and issue the instruction whether the picking
structure picks, and how the driving mechanism moves.
The power system is responsible for powering the entire
robot.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

640

Design of Daylily Agricultural

2.4. Microcontroller
The microcontroller adopts the design scheme of dual
controller STM32-Raspberry Pi. The two controllers areshown in Fig.6 and Fig.7 respectively.

Fig.4. The manipulator

2.3. Mobile Structure

Fig.6. STM32 main global currents

As is shown in Fig.5, the planting base of daylily is sandy
land, which has a high requirement for the tire of thepicking robot. If the wheel drive, it is easy for the tire toslip. And if it is cloudy and rainy, the ground will become
muddy, and wheeled robots will be unable to move at all.

Fig.7. Raspberry Pi main global currents

STM32 microcontroller is the first choice for
embedded control chips. This microcontroller has thecharacteristics of strong real-time, low power
consumption, high integration, and a rich peripheral
library, which is convenient for development. Here weuse it to connect the mobile structure that drives the robot
and the picking structure.
Raspberry PI internally runs a customized version of
the Linux system, can be installed rich processing
software, software features rich, to capture video andimages can be arbitrarily modified. We use the Raspberry
PI to connect to the vision system to process video
information.

Fig.5. Daylily planting base

We use a crawler drive. The contact area between the
crawler and the ground is large, the conflict force grip isstrong, and it is not easy to sink. The crawler type canwalk on any messy terrain without any influence, and canalso walk normally in any muddy and soft ground.
Whether it is the rainy southern part of China or themonotonous north, whether it is a mountain or a plain,
whether it is sandy or muddy, the crawler drive can makethe robot seem to walk on flat ground.

2.5. Power System
The power system is the power source of the wholesystem and is responsible for providing power for
STM32, Raspberry PI, vision system, etc. The L78 seriesof three-terminal positive regulators (Fig.8) is available
in TO-220, TO-220FP, D²PAK, and DPAK packages andseveral fixed output voltages, making it useful in a wide
range of applications 7.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

641

Jichao Zhao, Fengzhi Dai

importantly, avoid the harm caused to the human body by
the daylily picking.
References
1. Zhao JC, Dai FZ, et al. Device Design Based on TDS
Water Quality Detection, International Conference on
Artificial Life and Robotics, Oita, Japan, 2020, 1: P620623.
2. J. B. Han, K. M. Yang, et al. A Modeling and Simulation
based on the Multibody Dynamics for an Autonomous
Agricultural Robot. 7th International Conference onControl, Mechatronics and Automation, Delft,
Netherlands, 2019, pp. 137-143.
3. U. Rao. Design of automatic cotton picking robot with
Machine vision using Image Processing algorithms.
International Conference on Control, Automation,
Robotics and Embedded Systems, Jabalpur, 2013, pp. 1-5.
4. Y. Yu, K. Zhang, et al. Real-Time Visual Localization of
the Picking Points for a Ridge-Planting Strawberry
Harvesting Robot. IEEE Access, 2020, 8: pp. 116556116568.
5. Z. Xu, R. Jia, Y. Liu, et al. Fast Method of Detecting
Tomatoes in a Complex Scene for Picking Robots. IEEE
Access, 2020, 8: pp. 55289-55299.
6. G. Bao, P. Yao, et al. Flexible pneumatic end-effector for
agricultural robot: Design & experiment. IEEE
International Conference on Robotics and Biomimetics,
Zhuhai, 2015, pp. 2175-2180.
7. A. J. Scarfe, R. C. Flemmer, et al. Development of anautonomous kiwifruit picking robot. 4th International
Conference on Autonomous Robots and Agents,
Wellington, 2009, pp. 380-384.

Fig.8. The L78 serial package

Different systems require different voltages, thisrequires the power system to be able to output different
voltages stably. Therefore, L78 meets our power demand,so the L78 series is adopted as our power chip.
3. Conclusion
In this paper, we designed an agricultural picking robotof daylily according to its growth characteristics. The
robot recognizes the maturity of daylily and analyzes thespatial coordinates through image recognition, and thendrives the manipulator to pick. The robot can effectively
improve people's labor production efficiency, and more

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

642

Development of Intelligent Public Trash Can Based on Machine Vision and Learning
Longyu Gao 1, Fengzhi Dai 1, 2 *, Zhiqing Xiao 1, Jiangyu Wu 1, Zilong Liu 1
1
2

Tianjin University of Science and Technology, China;

Tianjin Tianke Intelligent and Manufacture Technology CO., LTD, China;
E-mail: * daifz@tust.edu.cn
www.tust.edu.cn

Abstract
At present, with the maturity of machine vision technology and the continuous expansion of application fields,
there have been many intelligent trash cans based on machine vision, which can realize certain garbage
identification and automatic classification. However, due to certain technical limitations of machine vision, it is
impossible to identify all garbage. In this paper, a smart public trash can based on machine vision and auxiliary
sensors is proposed. In addition to realizing machine vision to identify and automatically classify garbage, sensors
will also be used to assist in identifying garbage to solve problems such as the same garbage classification ofdifferent shapes. At the same time, enhanced learning will be added to realize the self-learning of the trash can, so
as to achieve the goal of continuously increasing identifiable types.
Keywords: machine vision, machine learning, garbage classification, auxiliary sensors

1. Introduction

to recognize all garbage. For example, it is difficult torecognize the same garbage under different shapes.
In summary, the smart trash can not only needs to be
based on machine vision, but also needs to besupplemented when machine recognition cannot beeffective. Therefore, a smart public trash can based onmachine vision and auxiliary sensors is proposed. In
addition to realizing to recognize garbage andautomatically classify it by machine vision, sensors will
also be installed to assist in identifying garbage to solve
problems such as the same garbage classification ofdifferent shapes. At the same time, enhanced learning
will be added to realize the self-learning of the trash can,so as to achieve the goal of continuously increasing
identifiable types.

With the development of society and the increasing
improvement of people's living standards, society ispaying more and more attention to the problem of
garbage classification, and various regions have also
implemented relevant garbage classification policies.
However, due to various types of garbage and people's
poor awareness of autonomous classification, theclassification effect is difficult to achieve the expected
effect. And as a result, intelligent sorting trash bins
based on various technologies have emerged.
Most the intelligent sorting trash can lies in
automatically identifying the type of trash. At present,with the maturity of machine vision technology and awide range of applications, an intelligent classificationof trash cans based on this technology has been born,which can realize certain garbage identification andautomatic classification. However, due to certain
technical limitations of machine vision, it is impossible

2. Overall Design Scheme of the System
According to actual application scenarios and market
requirements, combined with machine vision, sensor
detection technology, and wireless communication© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

643

Longyu Gao, Fengzhi Dai, Zhiqing Xiao, Jiangyu Wu, Zilong Liu

metal sensor, humidity sensor, VOC smell sensor andcapacitance sensor, communication module, motor, etc.
The hardware control part of the system is introduced
below.

technology, this paper designs a smart public trash canbased on machine vision to realize automatic garbage
sorting and machine learning (continuously increasing
the number of identifiable types), the classification ofdifferent states of the first class of garbage and thereal-time monitoring of garbage filling.
When the smart trash can detects that the rubbish is
input, the trash can uses a visual camera to capture andidentify the dropped rubbish 1. Then compare andanalyze the data in the database to determine the type of
garbage, and then use its own mechanical structure tosort it into the corresponding garbage bin.
For the unrecognizable garbage, the trash can will
work through the interaction of metal sensors, humidity
sensors, VOC odor sensors and capacitive sensors tosort the garbage into the corresponding garbage cans,mainly to solve the problem of garbage occupying theidentification position. At the same time, theunrecognizable garbage is photographed, uploaded tothe server, and re-trained to determine the type of
garbage. When the garbage is recognized again nexttime, the garbage classification will be performed
directly through machine vision. At the same time, eachtrash can will share data, which can greatly improve thelearning speed of the trash can and shorten the learning
cycle. The flow chart of the intelligent public trash canbased on machine vision is shown in Fig.1.

3.1. Main control chip
STM32f407zet6 is selected as the core controller of the
hardware control system, which is mainly used torealize the functions of data acquisition and analysis andexecution of component motion control 2.
The processor of STM32f407zet6 is Arm Cortex-M4
high-performance core, the maximum operating
frequency is 168MHz, and the package form is 1qfp144.
Its peripheral configuration is powerful, supporting
communication interfaces such as SP interface, I2C
interface, USB interface, USART interface, andperipherals such as ADC and timer. STM32f407zet6 isshown in Fig.2.

Fig.2. Main control chip STM32f407zet6

3.2. Metal sensor
The function of material recognition is realized throughthe metal sensor module. The system uses metal sensors
to identify metals. This system uses LJ18A3-8-Z/BX
type metal sensor. The sensor has the advantages of fast
response speed, strong anti-interference ability,
waterproof and corrosion resistance, and very stableperformance.
It is widely used in automated assembly lines,
intelligent hardware and security systems. The operating
voltage of the sensor is 6-36V (DC), and the switch
contains a high-frequency oscillation circuit, a detection
circuit, an amplifying circuit, a de-transmitting circuit
and an output circuit. The metal sensor is shown in
Fig.3.

Fig.1. The Overall flow chart of intelligent public trash binbased on machine vision

3. Design of Hardware Control System
This system selects STM32F103 single-chip
microcomputer as the core controller, and combines
OpenMV and peripheral modules to realize the control
system function. It mainly include: STM32, OpenMV,© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

644

Development of Intelligent Public

Fig.5. Capacitive proximity switch M8M12

3.5. VOC air quality sensor
Fig.3. Metal sensor

The detection of hazardous waste that cannot be
identified by machine vision is mainly achieved throughthe ZP16 digital VOC air quality sensor module. The air
quality module uses advanced chip thick film
semiconductor gas sensors. The gas sensors are resistantto formaldehyde, benzene, carbon monoxide, andammonia, Hydrogen, alcohol, cigarette smoke, flavors
and other organic volatile gases have extremely highsensitivity. After aging, debugging, calibration andadjusting, the module has good consistency andextremely high sensitivity. Extremely high sensitivity,
excellent long-term stability, factory calibration,
convenient and quick to use; with sensor fault
self-diagnosis function, low power consumption andlong life. The VOC air quality sensor is shown in Fig.6.

3.3. Humidity Sensor
Use SHT3x temperature and humidity sensor module tocommunicate with STM32, and turn it on when machine
vision cannot identify the garbage, so as to realize theidentification of wet garbage.
The module applies digital module acquisition
technology and temperature and humidity sensing
technology, and the product has high reliability andlong-term stability. It had the built-in humidity andtemperature sensor elements, analog to digitalconverters, signal processing, calibration data and I2C
host interface. The humidity sensor is shown in Fig.4.

Fig.6. VOC air quality sensor
Fig.4. Humidity Sensor

4. OpenMV

3.4. Capacitive proximity switch

The vision part uses the openMV single-chip
microcomputer to communicate with the STM32. The
OpenMV camera is a small, low-power, low-cost circuitboard that can be scripted by a high-level language
Python (to be precise, MicroPython). We can use an
external terminal to trigger shooting or execute analgorithm, or use the result of the algorithm to control
the IO pin.
OpenMV uses STM32H743II ARM Cortex M7
processor, 480 MHz, 1MB RAM, 2 MB flash. All I/O
pins output 3.3V and are 5V tolerant 3. OpenMV isshown in Fig.7.

The second part of the material identification is mainly
realized by the capacitive proximity switch M8M12. It
can detect any dielectric substance, including
conductors, semiconductors, insulators, and even can beused to detect liquid level and powdered materials. For
non-metallic substances, the movement distance isdetermined by the dielectric constant of the material.
The greater the dielectric constant of the material, thefarther the sensing distance is.
The distance between it and the pallet is controlled by
the motor to realize the classification of different
materials. The capacitive proximity switch M8M12 isshown in Fig.5.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

645

Longyu Gao, Fengzhi Dai, Zhiqing Xiao, Jiangyu Wu, Zilong Liushows that the broad application prospects of theresearch results of this paper in waste classification andenvironmental protection.
In the research process of this article, we found that
there are some problems worthy of further digging. First,
we can make greater improvements in the detection
algorithm to improve the accuracy of small garbage
recognition. Second, we can study the design ofmulti-bin garbage cans or other structures for garbage
collection device to meet the sorting needs of moretarget categories.
Fig.7. OpenMV

Acknowledgements
The research is partly supported by the Project ofTianjinEnterprise
ScienceandTechnology
Commissioner to Tianjin Tianke Intelligent andManufacture Technology Co., Ltd (19JCTPJC53700). It
is also supported by the Industry-University
Cooperation and Education Project (201802286009)
from Ministry of Education, China.

5. Overflow Reminder Module
The overflow detection uses the infrared distancemeasuring sensor module GP2Y0A21YK0F 10-80cm
distance sensor, the integrated combination of PSD(position sensitive detector), IRED (infrared lightemitting diode) and signal processing circuit, thereflectivity of various objects, The environment
temperature and working time are not easily affected by
the distance detection. When the garbage overflows, thereturn value of the module will have a big change.
When the return value meets certain conditions, themain control will transmit the corresponding
information to the management terminal via the WIFI
module. The overflow reminder module is shown in
Fig.8.

References
1. Golnabi, H., A. Asadpour. Design and application ofindustrial machine vision systems. Robotics andComputer-Integrated Manufacturing, 2007, 23(6):630-637.
2. Liu, Jianghai, Yadong Jiang. Design of intelligent trash
can be based on machine vision. 2020 International
Conference on Image, Video Processing and Artificial
Intelligence. Vol. 11584. International Society for
Optics and Photonics, 2020.
3. Vernon, David. Machine vision-Automated visualinspection and robot vision. STIA 92 (1991): 40499.

Fig.8. Overflow reminder module

6. Conclusion
Aiming at the problem of public garbage classification,
this paper proposes a classification method based on the
combination of machine vision and auxiliary sensors;
designing an automatic garbage classification hardware
system to realize the automatic classification andrecycling of common garbage in public places in life.
Tests show that the classification method is effective.
The hardware and software system works stably. It© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

646

The Optimized Intelligent Algorithms on Face Recognition and Tracking for
ROS-based Robots
Yue Chen, Shuhao Tian, Huailin Zhao, Shengyang Lu

School of Electrical and Electronic Engineering, Shanghai Institute of Technology, No. 100, HaiquanRoad, Shanghai 201418, China
E-mail: 405838753@qq.com
https://www.sit.edu.cn/

Abstract

With the development of artificial intelligence, face recognition and tracking
technology have been widely used in many fields such as target positioning, automatic
driving, and human-computer interaction. Recently, a large number of face detection,
recognition and tracking algorithms have emerged, but there are still many
shortcomings in practical applications, such as slow face detection, low detection
accuracy, and face recognition and tracking for ROS robots Algorithms are rare. This
paper improves the traditional Haar-like algorithm and LK optical flow tracking
algorithm, and designs a ROS robot platform based on the improved algorithm. By
comparing the accuracy and timeliness of face detection and tracking between the
improved algorithm and the traditional algorithm, the superiority of this design
algorithm is obtained.
Keyword: Face Tracking and recognition; ROS robots; LK optical flow; Haar-likealgorithm

have developed a variety of computer facerecognition and tracking algorithms. The
ultimate goal is to solve noise interference, target
occlusion of rapid movement and thesurrounding environment More complex issues
and real-time tracking. With the continuous
development of computer technology, a large
number of algorithms for real-time video
tracking have emerged, mainly including three
types of algorithms based on target external
characteristics, target contours, and target local

1.Introduction
Video tracking[1] is an important branch ofcomputer vision. With the increasing impact of
artificial intelligence on human production andlife, video tracking is widely used in many fields
such as target positioning, video tracking,automatic driving, and human-computerinteraction. Recognition and tracking of mobile
faces is an important technology to improve theintelligent level of mobile robots. Researchers© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

647

Yue Chen, Shuhao Tian, Huailin Zhao, Shengyang Lu

overview and analysis of existing face detection
and face recognition algorithms, the existing
methods have achieved certain success in facerecognition and detection in complex
environments and occlusions, but there is stillmuch room for improvement. It is very
important to further improve the accuracy androbustness of face recognition and detection
under complex conditions.
In the process of realizing mobile facerecognition and tracking, there are mainly threeissues involved: 1) Recognizing the face, 2)Marking the key points of the face, 3) Detecting
the key points before and after the face moves,and judging whether it is the same face. Achievecorrect tracking. Therefore, after analyzing andcomparing most face recognition algorithms, thisdesign improves the Haar-like[5] facerecognition algorithm to improve the accuracy
and speed of face recognition. Then use theimproved LK optical flow tracking algorithm[6]to track a specific moving face by detecting the
key points before and after the face moves.
Finally, the Camshift[7] color tracking algorithm
is used for face tracking, and the improved
algorithm is compared to realize the facerecognition and tracking of the ROS robot. The
main innovations of this article are as follows:(i) When marking the key points of the face,
we have improved on the basis of theHarris algorithm for corner detection andadopted a more accurate marking method,which is called the better key point marking
algorithm.
(ii) This paper uses an improved LK opticalflow tracking algorithm to detect the key
points before and after the face moves, so
as to realize the tracking of a specificmoving face. Reduce the impact of thesurrounding environment on face tracking.
(iii) As a control, we also applied the Camshift
color tracking algorithm to research andexperiment, and compared and analyzed
the experimental results of the LK opticalflow tracking algorithm.
(iv) This paper designs and implements the facerecognition and tracking system of the ROS
mobile robot platform, on the ROS robot
platform, which can meet the accuracy andreal-time performance of face recognition
and tracking in complex environments
through real environment tests.

areas. Face detection and tracking is anextremely important research field in video
tracking technology, and it is the basis of many
face
applications.
Therefore,
research,
processing and tracking of faces is a very
meaningful direction in face video surveillance.
However, in practical applications, facedetection technology still has some shortcomings,
such as slow face detection speed, low accuracyof face detection, and face recognition andtracking systems for ROS robots are rare. This
paper designs a set of face recognition andtracking system suitable for ROS mobile robots
to improve the intelligence of ROS robots.
Ebied [2] proposed the application of Principal
Component Analysis (PCA) in face feature
extraction and dimensionality reduction to solve
the problem of reduced face recognition ratecaused by the large dimension of face features.
This method applies mirroring technology to theparity analysis of feature maps. The final
simulation results show that this method
effectively improves the accuracy of facedetection and face recognition. But the algorithm
runs slower and has higher requirements on the
training set. TFCootes[3] and his research teamproposed an Active Shape Model (ASM) basedon statistical parameterization. This methodbuilds a statistical parameterized model based on
the statistical analysis results of the target
contour feature vector, which can reflect Draw
out the law of target shape change. Because of
its simplicity and high efficiency, the algorithm
has been well applied in face detection andtracking. Dejun Tang from the School ofComputer Science and Technology of Dalian
Maritime University[4] studied the image feature
extraction and matching technology in facerecognition, using the improved Singular Value
Decomposition (SVD) method of classestimation space for feature extraction to solve
face recognition In the process, the singularvalue vector is directly used as the face feature,
which leads to the problem of low recognition
rate. In addition, in order to reduce the instabilityof face recognition caused by changes inexternal factors such as illumination, expression,
noise, posture and other factors, a face feature
representation method that combines multi-scale
global features and local features is proposed,
and then rough integration is used. The
simplified algorithm performs feature selection
on the extracted features. Through the above© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

648

The Optimized Intelligent Algorithms

2. Face recognition and tracking algorithm
designdetect the area of the face. In order to calculate
the pixel difference between the feature maps,
OpenCV is used to grayscale the RGB featuremaps detected by the Haar detector andnormalize them. The pixel value of each sliding
window is grayed out. For each pixel i ( x, y ) , theaverage gray value of the pixel value mean andthe square average gray value sqmean arecalculated as follows:

Taking into account the recognition speed,
accuracy and compatibility with the robot
system, the face recognition and tracking design
system is mainly composed of Haar-like feature
recognition, better key point marking and adding,and LK optical flow tracking. We call it the LK
optical flow tracking scheme.

w

2.1 Face recognition based on Haar-like
detectormean =

Haar-like classifier detection target is to use thefeatures of known training samples to comparefeatures with unknown images to achieve target
detection. This article is mainly used in facedetection, generally using two rectangularfeatures, by making the difference between thewhite and black area pixels to meet the detectionneeds. The Haar-like detector is composed ofHaar-like features, Haar integral map, andcascade classifier.
Haar-like features can be roughly divided into
three types: edge features, linear features, central
features and diagonal features. According to theabove-mentioned feature group, the Haar-like
detector consists of black and white rectangles asfeature templates, and its feature value is definedas the difference between the white rectangularpixels and the black rectangular pixels. The
Haar-like detector uses the feature template toarbitrarily change its specifications in the image,process the image, and use the exhaustive
method to calculate the feature value of the facearea and the feature value of the non-face area,
and judge the image gray by the feature value
Degree changes and detect target recognition.
The application effect is shown in Fig.1:

=
x 0=
y 0

w•h
w

sqmean =h∑∑ i( x, y)h∑∑ i ( x, y)

(1)

2

=
x 0=
y 0

w•h

(2)

Then the normalization factor (varNormFactor)
and the normalized haar feature value(normValue) are obtained by normalizing theimage. The calculation formula is as follows:sqmean − mean 2(3)featureValuevar NormaFactor(4)

var NormFactor=

normValue =Where featureValue is the feature value ofeach pixel in the feature map. In order toimprove the effect of face detection, this paper
adopts a cascade classifier called Adaboost ,which is achieved by cascading multiple weakclassifiers to form a strong classifier. The
specific algorithm of the Adaboost cascadeclassifier is as follows:(a) The picture to be input is preprocessed toobtainthecharacteristic
value
corresponding to each pixel of the picture.
(b) Use the Haar classifier to calculate the Haarfeature value ( normValue ) corresponding
to the feature value and compare it with theNo. 1 face feature. If not, go to step(f), if
yes, go to step(c).
(c) Compare the Haar feature value( normValue ) with the No. 2 face feature,
if it does not match, go to step(f), if itmatches, go to step(d).
(d) Compare the Haar feature value( normValue ) with the No. 3 face feature,
if it does not match, go to step(f), if itmatches, go to step(e).
(e) Confirm that it is a human face.

Fig.1 Haar module face recognition application
The Haar-like detector uses a sliding window to© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

649

Yue Chen, Shuhao Tian, Huailin Zhao, Shengyang Lu(f) Confirm that it is not a human face.
The specific algorithm flow chart is shown in
Fig. 2:Start

Input the image to be tested(calculate the feature value ofthe image to be tested)

Fig.3 Corner detection
Use corner detection[9] scoring function for
better key point marking algorithm, the specificscoring function is as follows(5)
R = min(λ1 , λ2 )Manually set two feature values λ1 and λ2 . Whenthe score of the picture to be detected exceeds
the threshold values λ1 min and λ2 min, it will be
judged as a corner point. The specific applicationof ST scoring function is shown in Fig.4.

Does not match
Haar classifier No. 1
the facial features(compared with facefeature No. 1)
Meet the facial
Does not matchfeaturesthe facial features
Haar classifier No. 2
Not a human(compared with face
facefeature No. 2)
Meet the facial
features
Does not match
Haar classifier No. 3
the facial features(compared with facefeature No. 3)
Meet the facial
featureshuman faceEnd

Fig.2 Flow chart of cascade classifier judgment
Fig.4 ST scoring function to judge corner points

According to the results of face detection at eachlevel, the image area similar to the face isscreened to accurately locate the face. The
scalefactor variable parameter is introduced to
set the window size. The smaller the value of
this parameter, the smaller the size of the sliding
window and the more detailed the scan of theface, but it will take more time and memory toscan.

After multiple tests, the appropriate number ofcorner points is selected, and the selection of key
points is determined by the number of corner
points, so that the face can be more accurately
recognized and detected.
2.3 LK optical flow tracking algorithm
After extracting the key point features, the LK
optical flow tracking algorithm [11] is used forreal-time face tracking.
We assume:(1) The brightness is constant;(2) The time is continuous or the motion is"differential motion" and the space is consistent;(3) The adjacent pixels remain adjacent andhave similar motions.
According to the above three assumptions, there
is a picture at time t, the internal coordinates ofthe picture are a point at the position (x, y), andthe pixel gray level is I (x, y, t); according to theassumption (2) after differential motion , That is,after moving (dx, dy) distance after dt, its grayvalue is I (x+dx, y+dy, t+dt); finally, according

2.2 Better key point marking algorithm
The better key point marking algorithm mainly
uses the gray-scale change relationship between
the pixels between the marked corner points. The
corner points are mainly the connecting points ofthe contour lines of the objects in the image, as
shown in Fig.3. Use a fixed window to slide inany direction on the image to be tested, andcompare the degree of pixel grayscale changes in
the window before and after the slide. If there isa large gray scale change, then it is determinedthat the window has corners.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

650

The Optimized Intelligent Algorithms

chart is as follows:(a) Make a data set based on the data points
obtained by searching for the first k frames;(b) Search for k+1 frames, compare anddistinguish the feature points. If it isdetermined to be an abnormal point, go tostep(c). If it is not an abnormal point, go tostep(d);(c) Remove anomalies;(d) Keep feature points;(e) Judge whether it is the last frame, if it is theend of the last frame, if it is not the last
frame, go to step(f);(f) Update the statistics set.
Based on the above algorithm, the detection
accuracy of the LK optical flow algorithm canbe improved, and the impact of this method on
the complex environment can also be minimized.
In order to improve the accuracy of therecognition and tracking algorithm, thisalgorithm is used to improve the LK optical flow
algorithm. Although this algorithm has a certain
increase in the amount of calculation, theaccuracy and robustness have been improved alot. The specific algorithm flow chart is shownin Fig. 5 below:

to assumption (1), the previous t time and t+dt
The gray scale is the same at all times. The
mathematical expression is as follows:
(6)
（x,
I
y, t ）=I ( x+dx, y+dy, t +dt )After differential transformation and equationtransformation, the following formula isobtained:  dI1 dI1   dI  ,   dx   1 
dx
dy



    dx ................   dt  = ......   dy    dI k dI k      dI k dt,  dx dy   dx 
(7)Among them, dI / dt represents the graychanges around t and t + dt at the two moments,
dx / dt and dy / dt , represents the movement
speed of the point in the axis x、y direction, anddI / dx and dI / dy represents the gray changes
in the axis x、y direction.
Although the equation cannot be solved based on
the position information of a single key point,
the position of the key point in the picture after
the elapse of time dt can be obtained by
combining multiple key point information andhypothesis (3).
In the algorithm design, this design takes LK
optical flow tracking as an extension of thebetter key point. Therefore, the LK optical flow
tracking node can be used to subscribe to the key
points that have been obtained in the better key
point node, and the algorithm can be used topass the current key point. And two grayscale
pictures to predict the next set of key points, andperform backward prediction.

Start

Take the first k framesof statistical featurepoints as the data set

Update data set

2.4. Optimal design of LK optical flow tracking
algorithm

N

In the process of key point tracking, since thekey points of the tracking are judged by the ST
scoring function to have low scores, these key
points are automatically discarded by the opticalflow tracker. This phenomenon leads to the lossof faces during the tracking process. And
misjudgment. Therefore, the algorithm isimproved and optimized for the above problems.
In the process of tracking, use the tracked results
to make a statistical set, search for each frame in
the video tracking process, and count, delete theabnormal feature points, update the statistical set,and search for the next frame. The specific flow

Are there any
abnormalities in frame
K+1
N

Y

Save feature points
Remove anomalies

Whether to end
YEnd

Fig.5 Optical flow statistics algorithm flow chart© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

651

Yue Chen, Shuhao Tian, Huailin Zhao, Shengyang Lu

3. Design of ROS robot platform based onface recognition and tracking algorithm
The experimental environment is Ubuntu system,
which is realized by programming under theterminal. The hardware platform is Turtlebot2
robot, Kinect camera and host computer. After
programming the upper computer loaded with
the Ubuntu system and completing the basic facerecognition and tracking algorithm design, thecorresponding Python program was designed
and tested on the ROS robot Turtlebot2.

Fig.7(a) 103th frame

Fig.7(b) 107th frame

Fig.7 Face tracking effect before optimization
In response to the above problems, we modified
and optimized the algorithm. Fig.8 uses thealgorithm proposed in this article. It can be seenfrom Fig.8 that with the increase of time, the sizeand shape of the frame tracked by the face in theprocess of continuous movement remainunchanged, and the tracked feature points arealso relatively increased and successfullycompleted the tracking. It can be seen thatcompared with the traditional algorithms, thealgorithm proposed in this paper greatly reduces
background information, makes full use of target
information, and effectively reduces false featurepoints. Not only that, it also ensures that the
number of feature points makes the tracking
accuracy and robustness greatly improved.

3.1 Feature point tracking test
Process the video information received by
Kinect, recognize the face and track the face inreal time, and finally achieve the purpose of facerecognition and tracking in a complex
environment. Fig. 6 is the hardware platform
built for this experiment.

Fig.8(a) 139th frame

Fig.6 The mobile robot Turtlebot2

Fig.8(b) 160th frame

Fig.8 Face tracking effect after optimization

The system uses the traditional Harr-likealgorithm and LK optical flow algorithm to meet
the basic requirements of face recognition andtracking, but there are some problems. As shownin Fig. 7, when the face recognition and tracking
program runs for a certain period of time, someof the key points will drift from the ROI area(the face recognition area) to other non-target
areas. And as the running time goes by, thenumber of key points in the recognition screen issignificantly reduced, until a few key points areleft in the face area, which cannot allow the
robot to track well.

As shown in Fig.8 above, the improved facerecognition and tracking algorithm in the ROS
robot platform has a greatly improved tracking
effect compared with the previous one.
3.2 Face recognition test
The ROS robot platform designed this time canachieve the tracking test of the human face, thatis, when the human face is moving, the robot
also follows the human face to rotate and detect
and recognize the human face. In order to verify
the accuracy of the face recognition of the robot
in the process of moving, the vertical upward isadopted, and the left and right swings at thesame speed are used to measure the maximum
detectable angle 0° of the robot. As shown in Fig.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

652

The Optimized Intelligent Algorithms

robot is also rotating with the human face. In
order to obtain the tracking range of the robot,
the human rotation angle is used as a reference
for comparative analysis. In this experiment, thetraditional Haar-like+optical flow tracking
algorithm is compared with the improved
Haar-like+optical flow tracking algorithm andthe Camshift algorithm, and the superiority ofthis algorithm is obtained. The comparison isshown in Table 1:

9 below:

Fig.9(a) 0°

Fig.9(b) 15°

Fig.9 Swing tracking process
From the above figure, it can be seen that when
the human face rotates at a certain angle, theTable 1 Algorithm effect comparison table
Algorithm nameFace swing angle

Recognition speed(Frames/sec)

Traditional Haar-like + optical±10°  ±15°92flow tracking±24°  ±26°
Camshift tracking algorithm
72±26°  ±28°
Ours
87
algorithm in both processing accuracy andFrom the above table, it can be found that the
speed.
detection angle of the algorithm in this paper has
B. The algorithm in this paper is applied to thebeen greatly improved compared with the
ROS robot platform, and the accuracy of robot
traditional Haar-like+optical flow tracking
face recognition is greatly improved comparedalgorithm, but the detection speed has beenwith traditional algorithms.
reduced due to the increase in calculation. But
compared to the Camshift tracking algorithm,
5. References
both the accuracy and the detection speed are
improved. The algorithm in this paper has a
1. Schnell Anna Elisabeth,Van den Berghgood effect.
Gert,Vermaercke
Ben,Gijbels
Kim,Bossens
4.Conclusion
This paper improves the traditional Haar-likealgorithm and optical flow tracking, and designs
and optimizes a set of face recognition andtracking system based on ROS mobile robot.
The superiority of the improved algorithm isobtained by comparing with the traditionalalgorithm. The specific work is summarized asfollows:
1. Implement and improve the traditional
Haar-like algorithm and LK optical flow
tracking algorithm.
2. Designed a face tracking platform based on
the ROS system, and compared the improved
algorithm with the traditional algorithm, and got
the following conclusions:A. The processing accuracy of the traditional
Haar-like algorithm+optical flow tracking
algorithm is very different from the algorithm inthis paper, but the speed is faster, and theCamshift algorithm is not as good as the

2.

3.

4.

5.

6.

Christophe,de
Beeck
Hans
Op.
Face
categorization and behavioral templates in rats[J].
Journal of vision,2019,19(14).
Chen Jeng. A new patch-based LBP withadaptive weights for gender classification ofhuman face[J]. Journal of the Chinese Institute ofEngineers,2020,43(5).
Zhang G, Vela P A . Good features to track forvisual SLAM[C]// IEEE Conference on
Computer Vision & Pattern Recognition. IEEE
Computer Society, 2015.
Xiaochen L, Xiaojie L, Yufeng X, et al.
Evaluation of Lucas-Kanade based optical flow
algorithm[C]// 2018 IEEE CSAA Guidance,
Navigation and Control Conference (GNCC).
IEEE, 2018.
Li Wang, Huailin Zhao, Yaoyao Li. Research on
Research on the multi-scale network crowd
density estimation algorithm based on attention
mechanism. The 2019 International Conferenceon Intelligent Informatics and BioMedical
Science (ICIIBMS2019), Nov.21-24, 2019,
Shanghai, China, pp272-278.
Yawei Hou, Huailin Zhao. Handwritten Digit
Recognition Based on Depth Neural Network.
2017 International Conference on Intelligent© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

653

Yue Chen, Shuhao Tian, Huailin Zhao, Shengyang Lu

7.

8.

9.

10.

11.
12.

13.

14.
15.

16.

17.

18.

19.

20.

Informatics
andBioMedical
Science(ICIIBMS2017), Nov.24-26, 2017, Okinawa,
Japan. pp35-38. (EI)
Shunzhou Wang, Huailin Zhao, Weiren Wang,
Huijun Di, Xueming Shu. Improving Deep Crowd
Density Estimation via Pre-Classification of
Density. The 24th International Conference on
Neural Information Processing (ICONIP2017),
Nov.14-18, 2017, Guangzhou, China. pp260-269.
(EI)
8.Muhammad Asif, Nadeem Haider, Qasem
Al‐Mdallal, et al. A Haar wavelet collocation
approach for solving one and two‐dimensional
second‐order linear and nonlinear hyperbolic
telegraph equations. 2020, 36(6):1962-1981
Multimedia; New Multimedia Study Findings
Reported from National Taipei University (Fast
Iris Localization Using Haar-like Features andAdaboost Algorithm). 2020,
Xirui Ma, Yizhou Lin, Zhenhua Nie etal. Structural damage identification based on
unsupervised feature-extraction via Variational
Auto-encoder[J] Measurement, 2020, 160
Guangqiang Li, Lei Yu, Shumin Fei. A Binocular
MSCKF-Based Visual Inertial Odometry System
Using LK Optical Flow. 2020, :1-16.
Multimedia; Investigators at Jiangsu University
Describe Findings in Multimedia (Fast Stereo
Visual Odometry Based On Lk Optical Flow andOrb-slam2). 2020.
Timotej Gašpar, Miha Deniša, Primož Radanovič,
et al. Smart hardware integration with advanced
robot programming technologies for efficientreconfiguration of robot workcells. 2020, 66
고동현, 이재현, 김정인, et al. Integration of
Mobile Robot and Manipulator Using the ROS.
2020, 29(3):182-188.
Iván del Pino, Miguel Á. Muñoz-Bañon, Saúl
Cova-Rocamora, et al. Deeper in BLUE
Development of a roBot for Localization in
Unstructured
Environments.
2020,
98(6):207-225.
Enrique Coronado, Gentiane Venture. Towards
IoT-Aided Human–Robot Interaction Using NEP
and ROS: A Platform-Independent, Accessible
and Distributed Approach. 2020, 20(5)Umberto Maniscalco, Antonio Messina, Pietro
Storniolo.
ASS4HR—
An
Artificial
Somatosensory System for a Humanoid Robot.
The ROS package. 2020, 11
Optical Research; New Data from China Nuclear
Power Engineering Co. Ltd. Illuminate Findings
in Optical Research (High Precision Indoor
Positioning Method Based On Visible Light
Communication Using Improved Camshift
Tracking Algorithm). 2020, :337-.
Optical Research; New Data from China Nuclear
Power Engineering Co. Ltd. Illuminate Findings
in Optical Research (High Precision Indoor
Positioning Method Based On Visible Light
Communication Using Improved Camshift
Tracking Algorithm). 2020,
Wanchao Mao, Hongyun Xie, Zequn Tan, et al.

21.
22.
23.

24.

25.

High precision indoor positioning method basedon visible light communication using improved
Camshift tracking algorithm. 2020, 468
Zhang Xiangnan, Gong Wenwen, He Qifeng, et al.
Camshift tracking method based on correlation
probability graph for model pig. 2020, 2020(1)Gao Chaoqian, Chen Hu, Li Tianping. Target
Tracking Based on Camshift Algorithm and Multi.
2020, 1550(3)양병호, 이명진. Study on the Video Stabilizer
based on a Triplet CNN and Training Dataset
Synthesis. 2020, 25(3):428-438.Ruiheng Zhang,
Lingxiang Wu, Yukun Yang, et al.
Multi-camera multi-player tracking with deep
player identification in sports video. 2020,
1020tabek Khudayberdiev, Muhammad Hassaan
Farooq Butt.
Fire detection in Surveillance Videos using a
combination with PCA and CNN. 2020, 3(3).

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

654

Control of a Novel 5D Hyperchaotic System
Qiang Wei

Army Military Transportation University, China
Hong Niu

College of Electronic Information and Automation,
Tianjin University of Science & Technology, China
E-mail: spots@163.com
www.tust.edu.cn

Abstract
In this paper, a novel five-dimensional (5D) autonomous hyperchaotic system is presented, and the characteristicsof the 5D system are given in brief. For control of the 5D hyperchaotic system, a linear feedback controller isdesigned via the Lyapunov stability theory, so that the 5D system is no longer hyperchaotic but globally
asymptotically converges to the equilibrium point at the origin. The numerical simulation results are given toillustrate the feasibility and effectiveness of the method.
Keywords: the novel 5D hyperchaotic system; hyperchaos control; Lyapunov stability theory; global asymptotic
stability

control of the 5D system would be discussed, and somesimulation results would be given to demonstrate thevalidity of the designed linear feedback controller.

1. Introduction
Hyperchaos was first proposed by Otto Rössler in
1979.1 Since then, many novel hyperchaotic systems
have been formulated. In order to obtain hyperchaos, thesystem need to satisfy the following two important
requisites. Firstly, the minimal dimension of the phasespace that embeds a hyperchaotic attractor should be atleast four, which requires the minimum number ofcoupled first-order autonomous ordinary differential
equations to be four. Secondly, the number of terms in
the coupled equations giving rise to instability should be
at least two, of which at least one should have anonlinear function.2 Therefore, hyperchaos is much
more complicated than chaos, and it has greater
engineering significance and application prospect insignal processing, secure communication and so on.
In this paper, a novel 5D hyperchaotic system, which
has been introduced in Ref. 3, is reviewed. Stability

2. The Novel 5D Hyperchaotic System
The dynamic equations of the novel 5D hyperchaotic
system are formulated as=x a ( y − x ) ,y = ( c − a ) x + cy + w − xz ,z =−bz + xy,v = mw,w =− y − hv,(1)where x, y, z , v, w ∈ R are state variables, and a = 23,
b = 3, c = 18, m = 12 and h = 4.3
Let the initial values of the 5D system (1) be (x0, y0,z0, v0, w0) = (1, 1, 1, 1, 1), then the Lyapunov exponents© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

655

Qiang Wei, Hong Niu

respectively are λ1 = 0.8732 > 0, λ2 = 0.1282 > 0, λ3 =-0.0013 ≈ 0, λ4 = -0.5770 < 0 and λ5 = -8.4231 < 0. It
indicates that the 5D system (1) is hyperchaotic. The

phase portraits of the 5D hyperchaotic system (1) are
shown in Fig. 1.

Fig. 1. Phase portraits of the 5D hyperchaotic system: (a1) x-y; (a2) x-z; (a3) x-v; (a4) x-w; (a5) y-z; (a6) y-v; (a7) y-w; (a8) z-v;(a9) z-w; (a10) v-w© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

656

Control of a Novel

3. Hyperchaos Control of the 5D System

For V < 0 , the parameters k1 , k2 , k3 , k4 and k5
should satisfy that

3.1. Formulation of the controlled system

c
c
> 0,
k1 > − a,2233
k2 − c > 0,
k 2 > c,22⇒⇒
k3 + b > 0,
k3 > −b,hk4 > 0,k4 > 0,m
k5 > 0,k5 > 0,

k1 + a −The controlled system is represented asx = a ( y − x ) + uc1 ,y = ( c − a ) x + cy + w − xz + uc 2 ,z =−bz + xy + uc 3 ,=v mw + uc 4 ,( 2)w =− y − hv + uc 5 ,where

uc = [uc1 uc 2

uc 3

uc 4

uc 5 ]T

As a result, the linear feedback controller uc can be
designed as=[ −k1 x −k2 y −k3 z −k4v −k5 w] ,T

uc = [uc1 uc 2

and k1 , k2 , k3 , k4 , k5 ≥ 0 .

=Theorem 1. Let x = 0 be an equilibrium point forx = f ( x ) , where f : D → R n is a locally Lipschitzn

uc 5 ]

T

0 − v − w] .
T

Let the initial values still be (x0, y0, z0, v0, w0) = (1, 1, 1,
1, 1), then the curves of the state variables of thecontrolled system (2) before and after adding thecontroller uc are shown in Fig. 2 and Fig. 3 respectively.
Comparing Fig. 3 with Fig. 2, it can be found that thestate variables x, y, z, v and w converge to zero
asymptotically and rapidly. It implies that the controlled
system (2) is no longer hyperchaotic but asymptoticallystable at the origin. It illustrates that the linear feedback
controller uc is feasible and effective for hyperchaos
control of the 5D system (2).

=V ( 0 ) 0 and V ( x ) > 0, ∀ x ≠ 0

x → ∞ ⇒ V ( x) → ∞V ( x ) < 0, ∀x ≠ 0then x = 0 is globally asymptotically stable.4
From Theorem 1, take a continuously differentiable
functionV=

uc 4

3.3. Numerical simulation

map from a domain D ⊂ R into R . Let V : R → R
be a continuously differentiable function such thatn[0 −30 y

uc 3

From Theorem 1, the controlled system (2) is
globally asymptotically stable at the origin.

3.2. Design of the linear feedback controllernk1 = 0,k2 = 30,k3 =0,
k4 = 1,k5 = 1.

1 2
h 2
2
2
2
x + y +z + v +w 2m

as a Lyapunov function candidate for the controlled
system (2). Then, the derivative V is given byhV = xx + yy + zz + vv + wwm2=− ( k1 + a ) x + cxy − ( k2 − c ) y 2
h 2
v − k5 w 2
m
c
3 


≤ −  k1 + a −  x 2 −  k2 − c  y 2
2
2 


h
− ( k3 + b ) z 2 − k 4 v 2 − k5 w 2 .
m− ( k3 + b ) z 2 − k 4

Fig. 2. Curves of the state variables of the controlled systembefore adding the controller uc© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

657

Qiang Wei, Hong Niu

Fig. 3. Curves of the state variables of the controlled systemafter adding the controller uc

4. Conclusions
In this paper, a novel 5D hyperchaotic system isreviewed. For hyperchaos control of the 5D system, alinear feedback controller is designed. The numericalsimulation results demonstrate the validity of thecontroller. The study in this paper has some engineeringsignificance.
References
1. O. E. Rössler, An equation for hyperchaos, Physics
Letters A 71(2–3), 1979, pp. 155–157.
2. T. G. Gao, G. R. Chen, Z. Q. Chen and S. J. Cang, The
generation and circuit implementation of a new hyperchaos based upon Lorenz system, Physics Letters A
361(1–2), 2007, pp. 78–86.
3. Q. Wei and H. Niu, Analysis and circuit design of a novel
5D hyperchaotic system, Dynamical Systems and Control
8(2), 2019, pp. 118–128.
4. H. K. Khalil, Nonlinear Systems, 3rd edn., Prentice Hall,
New Jersey, 2002, pp. 112–126.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

658

Design of Multifunctional Vehicle Interior Environment Monitoring System Based on WirelessCommunication
Yuqi Yan *, Jialin Yang , Zhongxu Qin

Wuhan University of Technology, China
E-mail: * 1814696080@qq.com
www.whut.edu.cn

Abstract

The interior environment of the car affects the driver's mental state to a certain extent, so it is necessary
to design a multi-functional interior environment monitoring system. In this paper, STM32F103 single
chip microcomputer is used as the core controller, which integrates multiple sensors to collect various
environmental information in the vehicle, and transmits the data to the monitoring platform throughWiFi wireless communication. When the detected data exceeds the preset threshold, the monitoring
system will generate an alarm to remind the user to carry out relevant operations. The system has theadvantages of low cost, convenient use and high precision, and has certain commercial value andmarket prospect.
Keywords: car, sensors, STM32F103, WiFi

1. Introduction

information in real time. The system is convenient to use
and has a certain market value.

At present, the car is an indispensable means oftransportation in daily life. The internal environment of
the car affects the physical condition of the driver andpassengers to a certain extent. Therefore, it is necessary
to design a multifunctional vehicle interior environment
monitoring system. The multi-function internal
environment monitoring system can collect thetemperature, humidity, CO concentration, PM2.5
concentration and other internal environment informationin the car in real time, and display them on the LCD
screen. When the detected data exceeds the preset
threshold, the monitoring system will generate an alarm
to remind the user to carry out corresponding operation.
The system can also transmit various data information to
the monitoring platform through WiFi wirelesscommunication, and users can view the historical data
information on the monitoring platform. The vehicle
interior environment monitoring system can effectively
monitor and alarm the vehicle interior environment

2. Main control chip and peripheral circuit
The overall hardware structure of the system is composedof main control chip and peripheral modules, as shown in
Fig.1. The multifunctional internal environment
monitoring system selects STM32F103VET6 single-chipcomputer as the core controller of the hardware control
system. The processor of STM32F103VET6 is ARM
Cortex-M3 high performance kernel, with a maximum
operating frequency of 72Mhz, Flash memory of 512 KB,
SRAM of 64KB, and encapsulated in LQFP641. Its
peripheral configuration is strong and supports many
communication interfaces including SPI interface, I2C
interface, USB interface, USART interface, ADC, timer
and other peripherals2. With 80 IO ports, it is easy toconnect peripheral modules for function expansion. The
powerful function ensures that the chip can fully meet the
hardware functional requirements of the whole system.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

659

Yuqi Yan , Jialin Yang , Zhongxu Qin

2.2. JTAG interface circuit design
JTAG (Joint Test Action Group) is an international
standard test protocol. It mainly tests the internal MCU
chips of the monitoring and control system, and simulates
and debugs the hardware control part of the system. As
shown in Fig.3, there are four main lines in the JTAG
interface part: TMS, TCK, TDI, TDO, which are mode
selection, clock, data input and data output lines
respectively. In the design of the monitoring and control
system, the JTAG interface circuit uses 20-pin interface
wiring connection. In order to enhance the anti-jamming
capability of the download side of the system, groundwires are added between each signal line.

Fig.1 Overall architecture of hardware system

2.1. AD converter circuit
The STM32F103 series has three ADC, each of whichhas 16 conversion channels, with a conversion accuracyof 12 bits and a resolution of 1/4096. The ADC
conversion circuit is shown in Fig.2. The dynamic
contacts of the patch sliding rheostat of the AD
conversion circuit in the monitoring and control system
are connected to the ADC channel pins of the MCU chip.
When the user rotates the sliding rheostat adjustment
knob, the magnitude of the dynamic contact voltage in
the AD conversion circuit part of the system will change
accordingly. The range of voltage change is 0~3.3V,
which is also the default ADC voltage collection range ofthe hardware end circuit board of the system.

Fig.3. JTAG Download circuit

3. Peripheral module of hardware system
The peripheral modules of the hardware system mainly
include temperature and humidity sensor, smoke sensor,
interactive display screen, WiFi module and so on.
3.1. Temperature and humidity sensor
DHT11 digital temperature and humidity sensor is atemperature and humidity composite sensor withcalibrated digital signal output. By using special digitalmodule collection and temperature and humidity sensor
technology, the environmental monitoring and control
system has strong reliability and stability for temperatureand humidity detection of the environment. The DHT11

Fig.2. ADC converter circuit© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

660

Design of Multifunctional Vehicle

module has low cost and strong anti-jamming capability,
and it can measure the relative temperature and humidityof the automotive interior environment quickly. DHT11
uses full range calibration, which allows the module to beinterchanged directly without having to be re-calibrated if
a failure occurs. As shown in Fig.4, the DHT11 module
contains a capacitive humidity sensor to measure ambient
humidity and an NTC temperature sensor to measureambient temperature. Besides, a small, high-performance
8bit MCU is connected to the module.
Fig.5. MQ-2 sensor

3.3. ESP8266 module
ESP8266 WIFI wireless communication module isselected as the data transmission bridge between smart
hardware and server in this system, as shown in Fig.6.
The ESP8266 module integrates a low power ESP8266chip developed by LEXIN Information Technology
Company. The chip is highly integrated with a built-in
TCP/IP protocol stack and TR switches, amplifiers,
regulators, power management components, etc. Its
on-chip storage and processing power is strong, and it
requires very few peripheral circuits, so it is easy toembed and develop3.
The ESP8266 module is powered by a 3.3V voltage
and operates in three modes: SoftAP mode, Station modeand SoftAP+ Station coexistence mode. In SoftAP mode,
this module is equivalent to a wireless access point
providing wireless access services, similar to a wirelessrouter. Other terminal devices can connect to set up a
LAN with ESP8266 for data transmission; in Stationmode, this module is equivalent to a wireless terminal,
which can connect to the Internet through a router andtransmit data to the server; in SoftAP+Station coexistencemode, ESP8266 module can establish LAN
communication with other terminal devices as a wirelessaccess point, and can also connect services for Internet
communication4.
The ESP8266 module usually uses TCP protocol and
UDP protocol for data transmission. UDP is aconnectionless, unreliable protocol which does not
require the receiver to confirm before sending the data
and data transmission efficiency is high, but thereliability is low, which makes it unable to determine
whether the data is complete in the transmission process.

Fig.4. DHT11 sensor

3.2. MQ-2 sensor
The physical picture of the MQ-2 is shown in Fig.5.
The MQ-2 gas sensor used in the environmental
monitoring and control system can detect the presence ofliquefied gas, butane, methane or alcohol in theenvironment which are potentially dangerous to humanhealth. The conductivity of gas-sensitive materials
increases with the increase of the concentration of
combustible gas. The current smoke concentration value
is output from the sensor signal by the conditioning
circuit and sent to the MCU for processing. The alarmconcentration of this sensor is 0 when it is not set, and thealarm concentration threshold can be set by the user withan upper limit of 100%. The MQ-2 sensor has a wide
detection radius, maintains high sensitivity and fast
response in a variety of complex environments andexcellent stability in harsh environments.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

661

Yuqi Yan , Jialin Yang , Zhongxu Qin

4.

The TCP protocol is a link-oriented and reliable protocol
which needs to wait for the receiver to confirm before theconnection can be established. It is secure, reliable andthe integrity of data can be guaranteed. The ESP8266
module in this system uses TCP protocol to set the WIFI
application mode of ESP8266 as STA mode through AT
instruction, and chooses TCP connection to realize thefunction of establishing data connection with the server.

Software development environment

This system chooses to develop the STM32F103VET6
controller in Keil5 development environment, andchooses C language as the development language. Keil5
is an integrated development environment with powerfulcompilation and simulation debugging capabilities.
STM32 is developed in two ways: registerdevelopment and library development. This system isdeveloped in the way of Library development, calling
function interface directly to configure registers, which is
fast and readable. Use STMISP downloader to program
the STM32 core controller, and use TCP serial debugging
assistant for functional testing.
The core controller of STM32 needs to collect data
information of each sensor module, and control each
peripheral module to realize the sub-functions of thesystem through internal logical analysis and processing.
5.

Fig.6. ESP8266 module

Conclusion

The multi-functional vehicle interior environment
monitoring system designed in this paper can achieve thefunctions designed in advance after testing. The LCD candisplay the data information collected by each sensor inreal time. When the data exceeds the set threshold, thesystem will give an alarm.Through the serial debugging
assistant, the WiFi communication function is normal,
and the hardware device can connect and send data to thecloud server through WiFi module. The monitoring
system designed in this paper has certain practical value.

3.4. User interactive display module
The physical picture of the display is shown in Fig.7,
which consists of an LCD display panel, a capacitive
touch panel and a PCB substrate. The touch panel isequipped with a touch control chip, which is speciallydesigned to handle the touch signals generated by theuser when he touches the display, and then transmits theinformation through the outgoing signal lines. Because
the MCU of the system cannot control the LCD panel
directly, a LCD controller is needed to deal with theprocess of displaying. MCU only needs to give the data
to the controller to display.

References
1. Ibrahim, Dogan, Applying the ESP8266 processor in IoTapplication, Electronics World, 2017, 123(1976): pp.14-17.
2. Honghua Liao, Hao Fu, Weichuang Yu, et al., Design and
Implement of Pyroelectric Energy Harvester Experimental
Measurement System Based on STM32F103VET6,
Sensors & Transducers, 2016, 96(1): pp.69-74.
3. Fengzhi Dai, Yuxing Ouyang, Yiqiao Qin, et al.,Development of Integral Smart Home Appliances, Journalof Robotics, Networking and Artificial Life, 2018, 4(4):pp.291-294.
4. Craig A, Lindley, Meet the ESP8266, Nuts & volts, 2015,36(10): pp.42-47.

Fig.7. User interactive display module© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

662

Robot Structure and Motion Control Design Based on UG and Proteus
Yuhang Sheng

Tianjin University of Science and Technology, China
E-mail: 1346733865@qq.com
www.tust.edu.cn

Abstract
The paper is to design a six-degree-of-freedom biped robot by the research on the humanoid characteristics of thebiped robot. Our biped robot chooses the steering gear ASMC-03B as its power unit, we restricts the model
parameters by analyzing the function relationship between the steering gear torque and the volume, and regulates
the size of every parts, and create a motion analysis model. In the hardware part, Arduino UNO, which is used as
the main control chip, realize the communication between the main control chip and the servo drive module
PAC9685 through the IIC bus protocol, which saves the main control chip resources and ensures the execution
efficiency. In the simulation part, it is to simulate the steering angle of the steering gear, and output it in the form ofa waveform.
Keywords: Keywords: Biped robot,6-degrees of freedom, Mechanical structure, Motion control

1. Introduction

as cool running and triple jumping on one foot, as shownin Fig.1.

With the rapid development of science and technology intoday’s society, robots are playing an increasingly
important role in human life. The invention of robots is
one of the greatest inventions in the history of humandevelopment. At the same time, it is a very inclusive
cross-disciplinary. Biped robot, as one of the hot topics inthe field of robot research, has been paid much attention,and its research results are more outstanding than othertypes of robots.
Relative to domestic, foreign biped robot research fieldstarted earlier. Among them, Japan is the first country toinvest in biped robot research1. The DOF of biped robot
has experienced the development from PLANAR DOF
ROBOT TO SPATIAL DOF robot, the working mode of
biped robot has developed from “teaching-reproduction”
to autonomous decision-making robot. Recently, the mostinteresting biped robot research abroad is the Atlas Robotdeveloped by Boston Power Company. It has a strong
perception system and decision-making system, when
encountered obstacles to self-judgment to adjust thecenter of gravity, and by updating the gait to plan thelanding site to get away from obstacles. The latest versionof Atlas adds body coordination and extreme sports such.

Fig.1. Atlas biped robot

Compared with foreign countries, domestic research on
biped robot technology started late. In 1985, “863” plan
included the research of intelligent robot in “the outlineof China’s high-tech Research and Development
Program”. Among the universities with outstandingachievements in the field of biped robots are the Harbin
Institute of Technology, the National University of
Defense Technology, and the Beijing Institute ofTechnology. In recent years, the outstandingachievements in the field of biped robot research in China
are the “human-shaped robot Butler” Walker launched by© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

663

Yuhang sheng

3. Design UG model

Shenzhen Youbixuan Co. , Ltd. in 2019, as shown in
Fig.2, its range of motion ability includes all-round freewalking, stable climbing and going up and down stairs,
the use of three-dimensional visual navigation positioning
system to achieve navigation and obstacle avoidance
functions, applications such as remote monitoring,education and smart home.

Computer aided technology (CAD) is widely used in thefield of mechanical design. The application of CAD
design combines computer application with engineering
design. parametric design is the Yenshin of CAD
technology. This article uses the Ug drawing software,
which has the powerful compound modelling function,
the modularization is more centralized. The mechanicalstructure design of biped robot can be divided into twoparts: actuator selection and parameter design. The
steering gear is Asmc-03b, and the Working Torque is
0.38N·M. Parameter design2 refers to the design methodof controlling the size and installation position of theparts through the core parameters, normalizing the size ofthe parts to a reasonable range. Among them, the working
torque of the steering gear is an important index tostandardize the size of the parts. According to thephysical definition of Torque, the Torque is equal to thecross product of the radial vector of the rotating shaft andthe acting force, and the expression of the Torque is asshown in (1).

Fig.2. Humanoid Machine Butler

But, the robot product production cycle is long, in the
production process unpredictable factor is many. Basedon this premise, this paper designs a 6-dof biped robot
which meets the requirements by doing the pioneering
work to the biped robot model.

Bring

2. Mechanical design

M= L×FF = m × g and ρ = 𝑚𝑚⁄𝑉𝑉 into available
ρ × V × g × L < 0.38N･M

（1）

（2）

When the size of the robot is designed, the parametersof the Cross section can be set according to the constraint
conditions, and the size is reasonable.

Before modeling the biped robot, the Mechanical
Structure Diagram of the biped robot should beconstructed, and then the joint and leg structure should be
defined. The mechanical structure of biped robot can be
designed according to the anatomy and physiology ofhuman body, the left and right legs are symmetrically
distributed, and 6 joints are 2 hip joints, 2 knee joints and2 ankle joints. The mechanical design of the biped robot
is shown in Fig.3.

The Mechanical Structure Model of biped robot iscomposed of steering gear, rudder plate, waist u-shaped
support, foot plate, a-shaped parts, leg structure function
support, z-shaped function support and other parts. By
establishing the connection relation of each part and theconstraint relation of the parts, the parts added to theassembly environment can be located precisely. The
biped robot model assembled by constraint command isshown in Fig.4.

Fig.3. mechanical structure of robot© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

664

Robot Structure and Motion

The steering gear circuit is controlled by amicro-controller, which communicates with the steering
gear drive module via IIC bus. The steering gear drive
module outputs PWM wave to control the steering gear.
The hardware circuit diagram is shown in Fig.5.

Fig.4. robot assembly drawing

4. Circuit design of Proteus
Based on the Proteus simulation circuit software, thehardware circuit of biped robot is designed to drive thesteering gear. Proteus circuit design part mainly includes:
Power supply voltage-stabilizing circuit, mcu Minimum
System Circuit, rudder drive circuit, rudder circuit.

Fig. 5. Hardware circuit

5. Simulation experiment of Proteus circuit
The hardware-in-the-loop simulation is illustrated in
Fig.6 with an example of a single steering gear
experiment. The adjustable potentiometer (a) and thedisplay screen (b) are added to make the circuit
simulation visual. By adjusting the potentiometer to
output analog voltage, analog voltage signal transmission
to the microcontroller minimum circuit, SCM internal
voltage value according to register configuration, output
the corresponding PWM wave drive actuator rotation
angle.

The power supply module is composed of three partsof regulated power supply circuit. The components ofeach regulated power supply circuit are composed of 78
series three-terminal regulator, capacitor, polar capacitor
and power supply terminal. The regulated power supply
circuit supplies power to the minimum system circuit,
SERVO0, SERVO1, SERVO2, SERVO3, SERVO4 and
SERVO5 rudder circuit respectively.
Single-chip microcomputer circuit3 is the core of theminimum system circuit of single-chip microcomputer,and is an indispensable part of the design control circuit.
The main control chip takes the ATmega3284 core as the
microprocessor, and completes the tasks of data storage,
computation, peripheral device control and interrupt
processing.
The SERVO drive circuit uses PAC9685 chip, in
which the data transmission pin of SDA and SCL clock
pulse pin communicate with the minimum system circuitof single chip microcomputer through IIC bus, and the
LED pin is used as the output of PWM wave by
connecting the row of pins of 16 pins, A0, A1, A2, A3,
A4, A5 are the chip addresses for the driver chips and areconnected to the enable signal.

Fig.6. Single steering engine circuit diagram© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

665

Yuhang sheng

Development Of UG. International Conference on
Mechanical Engineering & Intelligent Systems. 2015.
4. Marwa F Areed. A keyless Entry System based on Arduino
board with Wi-Fi technology. Measurement, 2019, 139.

By adjusting the adjustable potentiometer to control
the pulse width of the steering gear, the Steering Gear
Angle from -90 ° to 90 ° rotation, as shown in Fig.7.

Fig.7. Control diagram of pulse width and angle

The experiment shows that the pulse width iscontrolled by an adjustable potentiometer, the duty cycleof the actuator is changed from 0 to 100% , and therotation angle of the actuator is displayed by a display
screen, so as to achieve precise control.
6. Conclusion
The structure and Motion Control Circuit of biped robot
are designed in this paper. In the part of mechanicalstructure design, through the analysis of the relationshipbetween the rudder torque and material, the dimensionsof the parts are standardized to reach a reasonable range.
Modeling and assembly of parts using UG software. In
the part of Motion Control Circuit, the Proteus circuit ismainly designed, the Arduino Uno is used as the control
chip, PAC9685 is used as the actuator module, and 78
series three-terminal voltage regulator is used as theregulator module. In the part of hardware circuit design,
the steering gear rotation is simulated by visualprogramming to realize the precise control of the steering
gear.
References
1. Chen C, Shih B, Shih C, et al. Design, modeling andstability control for an actuated dynamic walking planar
bipedal robot. Journal of Vibration and Control, 2013,19(3): 376-384.
2. Yunqiang Li, Guobin Lin. Design of intelligent parking lot
based on Arduino. IOPConference Series: Materials
Science and Engineering, 2019, 490(4).
3. Yuan M, Kang M, Ding Y. Parametric Model Of Excavator
Working Equipment Based On The Advanced© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

666

Design of Brushed Motor Position Loop Control System Based on Incremental PID
Tianyi Zhang *, Peng Lu

Tianjin University of Science and Technology, China;
E-mail: * 2679483816@qq.com
www.tust.edu.cn

Abstract
The motors are an important part of the transmission and control system. With the development of technology, thefocus of motors in practical applications has shifted to precise control of speed, position and torque. This paper
focuses on the control system design of the brush motor position loops, using STM32 microcontroller andincremental PID to achieve accurate control of the motor stopping moment position. The software is designed in C
programming. It is divided into four parts: System Clock Configuration Functions, Principal Functions, System
Tick Timer Interrupt Callback Function, Position closed loop PID Control Design Function. The hardware part isbased on the STM32F103VET6 core board.
Keywords: DC brush motor, STM32, closed-loop control, PID

1. Introduction
2. The Hardware Selection

For many reliable technical reasons, DC-powered brushmotors have been largely replaced by electronicallycontrolled brushless motors. In spite of this, in lessdemanding or cost-sensitive applications, Brushed motors
are still an effective solution.
By combining the Brushed Motor with the Basic Motor
Driver IC, brushed motors and end products allow formany additional operations and better protection. This
paper describes the design of position loop hardware andsoftware control system, for brush motor incremental PID.
The function of this system is to rotate the motor, andchange the stopping position after pressing the key.
For YS-F1Pro demo board hardware, we have obtained
a program on the rotation of hard stone motors. We have
optimized the changes based on this program. The
obtained source program is optimized to be migrated tothe core board containing the STM32F103ZET6 chip.
This is used to drive the motor to rotate according toprogrammed instructions. control intelligent household
device and monitor the information of intelligent
household device.

The entire hardware system consists of three parts: themotor, the hardstone driver board, and the black coreboard containing the STM32F103VET6.
2.1. Driver board selection
For the driver board, choose the motor driver ULN2003
from the YS-F1Pro development board. ULN2003 is ahigh-current drive array. It is mostly used in control
circuits of microcontrollers, smart meters, PLCs, digital
output cards, etc. It can directly drive relays and otherloads. Its input 5V TTL level and output up to
500mA/50V. ULN2003 can be directly connected to TTL
and CMOS circuits at 5V operation, it can directly
process data that would otherwise require a standard
Logical Buffer to process.
The ULN2003 is a family of high-voltage, high-current
Darlington transistor arrays, it features high current gain,high operating voltage, wide temperature range, and highload capacity. It is suitable for all types of systems© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

667

Tianyi Zhang, Peng Lu

design of the required pin and memory requirements. It
has the following characteristics:• Supported peripherals: Timer, ADC, SPI, I2C and
USART.
• Operating Voltage 2.0V ~ 3.6V.
• Operating temperature range: -40 oC ~ 85 oC.
The design of the main control chip is shown in Fig.3.

requiring high speed and high power drives.It can drive
brushed DC motors of up to 600W (60V10A).
The schematic diagram of the driver chip ULN2003 isshown in Fig.1.

Fig.3. STM32F103VET6 chipFig.1. The schematic diagram of the driver chip ULN2003

3. Software Design

2.2. Motor selection
For the motor, We chose the GM37-545 Hall-coded
geared motor, which has a precision reducer for better
coil amplification, and features high power and hightorque. the GM37-545 Hall-coded geared motor is shownin Fig.2.

Software design mainly includes the design and writingof programs, the application of functions in programs,
etc.
3.1. Program Block Diagram
The program block diagram of system is shown in Fig.4.

Fig.2. the GM37-545 Hall-coded geared motor

2.3. Core chip selection
For the core broad, the chip selected is the
STM32F103VET6. which is the Cortex-M3 ARM coreand has a 100-pin. The chip has 128KB flash memoryand 20KB RAM. It can through SWD or JTAG with
KEIL 4 to download the program and online debugging 1.
And it has more than 80 general-purpose IO ports,
multiple timers and serial ports, which fully meet theFig.4. program block diagram

The first step is to configure the Flash interface andsystem ticking timers, keys, serial communication,
encoder initialization, and so on. Then configure the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

668

Design of Brushed Motor

timer pulse output. Finally, set the LCD program to an
infinite loop with the while statement.

4.1. Motor control theory
The drive board adopts H-bridge driving circuit, thedirectional control of brushed DC motor is shown in the
figure, when Q1, Q4 conduction, Q2, Q3 closed, thecurrent direction is Q1 → motor→ Q4, then the motor
is positive, when Q2, Q3 conduction, Q1, Q4 closed, thecurrent direction is Q3 → motor→ Q2, then the motor
is reversed. Motor Forward Rotation Diagram is shown in
Fig.6 .Motor Reverse Diagram is shown in Fig.7.

3.2. System Timer Interrupt Callback Function
This function calculates PID results based on a velocity
loop period of 60ms ,and no change in limiting speed. It
uses PWM duty cycle to determine the current motion
direction. Note that every time a timer occurs, theinterrupt enters the callback function once.
3.3. Position closed loop PID control function
The function is enter the current control amount andoutput the target control amount. First, the deviation is
calculated, and the deviation dead-band is used todetermine if the deviation exists. The “ iIncpid” iscalculated according to the formula for the incremental
PID, and the final error is stored for the next calculation 2.
Main procedures for incremental PID control is shown in
Fig.5.

Fig.6. Motor Forward Rotation

Fig.5. Main procedures for incremental pid control
Fig.7. Motor Reverse

This program uses the PID formula to calculate thedeviation iError：
Deviation = set value - measured value
If -50 < deviation < 50, it can be considered 0.
The PID formula is as follows：
𝑢𝑢𝑘𝑘 =𝐾𝐾𝑃𝑃 *𝑒𝑒𝐾𝐾 +𝐾𝐾𝑖𝑖 ∑𝑘𝑘𝑗𝑗=0 𝑒𝑒𝑗𝑗 +𝐾𝐾𝑑𝑑 （𝑒𝑒𝑘𝑘 -𝑒𝑒𝑘𝑘−1 ）(1)
PID incremental value = current error E[K] - last error
E[K-1] + last error E[K-2](2)∆𝑢𝑢𝑘𝑘 =𝑢𝑢𝑘𝑘 − 𝑢𝑢𝑘𝑘−1 =A𝑒𝑒𝑘𝑘 -B𝑒𝑒𝑘𝑘−1 +C𝑒𝑒𝑘𝑘−2
The last error is stored for the next calculation,
returning an infinite loop of incremental values.

4.2. Experimental procedures
After hardware selection and software design, hardware
and software connections are needed to complete theexperiment.
There are the following steps：• Establishment Project• Create new Usart workgroups and files and linksettings.
• Importing software design programs.
Change the optimization program according toexperimental requirements and schematic diagrams,
design main functions and sub-functions, program
the system clock configuration function, mainfunction, system drop timer interrupt callback

4. Project Experiment© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

669

Tianyi Zhang, Peng Lu

5. Conclusion

function, and position closed-loop PID control
design function.
• Connecting Hardware Devices.
Connect the core board, motor and YS-F1Pro driveboard according to the given pin diagram anddatasheet.
The experimental results that can be obtained：
1. Press KEY1: motor start up
2. Press KEY2: motor stop
3. Press key 3 and add 1 from the motor's stop
position.
4. Press key4 and subtract 1 from the motor stop
position.

This experiment is to write software design for position
ring control of brushed motor (incremental PID) using C
program. The program includes the main function, timing
interrupt callback function, clock configuration function,
position ring design function, etc. Connect hardware
devices to complete the experiment. Use PID function to
calculate the motor stop position.
References
1. Yuxing Ouyang , Fengzhi Dai , Yiqiao Qin , et al. Integral
Design of Intelligent Home Equipment. Proceedings ofInternational Conference on Artificial Life and Robotics,
2017, 22: pp.148-151.
2. Zhiwang Chen, Baohua Liu, et al, Getting Started with
STM32 Embedded Microcontrollers, YanShan University,
Hebei, China, 2014.5.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

670

Design of a Fire alarm system
Yuhui Cheng *, Tianyi Zhang

Tianjin University of Science and Technology, China;
E-mail: * 13642017570@163.com
www.tust.edu.cn

Abstract
This article introduces the design of the fire alarm system and analyzes its working principle. The system uses a
distributed sensor network based on ZigBee to achieve fire alarm and real-time evacuation. Select the infrared
flame sensor and MQ2 smoke sensor as the information source. The core control board is the Arduino Mage2560
board. Through the ESP8266 module and the GPRS module, the multi-channel information transmission function
can be realized. The alarm can be classified according to the fire situation.
Keywords: ZigBee, Hierarchical alarm, crowd evacuation

1. Introduction
With the development of cities, the increasing number ofhigh-rise buildings has led to an increase in fire hazards.
Due to the difficulty of evacuation and fire fighting in
high-rise buildings, once a fire occurs, it will cause
serious losses. In this case, it is particularly important tocall the police in time and evacuate the crowd.1.
At present, most fire alarm devices are independent
alarms. The wiring is cumbersome and the wiring is
prone to aging and wear. Therefore, the probability of
failure or false alarms of such alarm devices is relatively
high. On the other hand, a system that only warns but
does not provide specific fire conditions, nor can it guidepeople to evacuate.
Based on the above discussion2, we designed a new
type of fire alarm system. The system obtains data
through the infrared sensor and MQ2 smoke sensor anduploads it to the client through the ESP8266WIFI module.
The Zigbee protocol is used for wireless communication
between monitoring nodes. When a fire breaks out, thesystem will alarm at different levels, and guide the crowd
based on the location and scale of the fire. The overall
structure of the system is shown in Fig.1.

Fig.1 System structure

2. The Hardware Structure
The hardware of this system mainly has four parts. The
Arduino Mage 2560 board and the Arduino Nano board
constitute the control unit of the system node; the infrared© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

671

Yuhui Cheng, Tianyi Zhang

adopts infrared sensor and smoke sensor as the data
monitoring unit of this system.
The infrared sensor can collect the light signal in therange of 750~1100 nanometers, and is very sensitive tothe flame spectrum. It can judge whether the flame isdetected by converting the light signal into a high andlow-level signal by calculation. The detection angle ofthe sensor is about 60°, and the working voltage is
3.3V~5V.
The infrared sensor is shown in Fig.4.

sensor and the MQ2 smoke sensor on the node constitutethe data monitoring unit; the CC2530 communication
module, ESP8266 WiFi module and GPRS SIM300C
module constitute the communication unit; themulti-directional indicator light and the voice module
constitute the evacuation unit.
2.1. control unit
The system is composed of the main node and a sub-node.
The main node uses the Arduino Mage2560 development
board as the control board, and the sub-nodes aredesigned based on the Arduino Nano development board.
The Arduino Mega 2560 is a microcontroller board
based on the ATmega2560. It has 54 digital input/output
pins (of which 15 can be used as PWM outputs), 16
analog inputs, 4 UARTs (hardware serial ports), a 16
MHz crystal oscillator, a USB connection, a power jack,an ICSP header, and a reset button.
The Arduino Mega 2560 board is shown in Fig.2.

Fig.4. infrared sensor
In the early stage of a fire, a lot of smoke is generated, so thedetection of smoke can also be used as one of the methods offire monitoring. The system uses the MQ2 smoke sensor as thesmoke data collection module. The sensor has a response timewithin 10s and can monitor smoke data from 300PPM to
10000PPM.

The MQ2 smoke sensor is shown in Fig.5.

Fig.2 Arduino Mega 2560 board

The Arduino Nano is a small and complete board
based on the ATmega328.
The Arduino Nano board is shown in Fig.3.

Fig.5. MQ2 smoke sensor

2.3. Communication unit
Aiming at the problem of complicated wiring of wired
communication equipment and prone to failure, thissystem uses wireless communication for data exchange.
For the communication between nodes, this systemselects the CC2530 radio frequency module as thecommunication unit between nodes, which has thefunction of ZigBee wireless networking. This module has
the following characteristics:

Fig.3. Arduino Nano board

2.2. Data monitoring unit
Fire alarm equipment with a single information source isprone to false alarms or untimely alarms. The system© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

672

Design of a FireThe design of the monitoring software is shown in
Fig.7.

With strong adaptability, nodes can be added ordeleted according to the needs of the usage scenario• Effective communication distance: 100m
• Low power consumption
After the information is summarized to the master node,
the data will be sent to the client through the ESP8266
WIFI module to realize remote monitoring.
When a fire occurs, the system may face power
outages and network outages. In order to send the alarm
information to relevant persons in time, the system isequipped with a GPRS SIM300C communication module,
which can carry out a fire alarm in the form of SMS
notification.
•

2.4. Guiding unit
Multi-directional LED indicators and MP3 voice module
constitute the system's guiding unit. Multi-directional
LED indicator, the main body is composed of red andgreen LED lights. The MP3 voice module supports audio
output in MP3 and WAV formats, which can becontrolled through the serial port of the microcontroller.
The design of the sound and light evacuation module isshown in Fig.6.

Fig.7. monitoring software

3.2. Design of Hierarchical alarm function
In order to prevent invalid alarms due to sensor failures
or operating errors, this system has designed ahierarchical alarm function. The discrimination
mechanism will divide the fire situation into two levels
according to the number of alarm nodes and the data ofalarm node.
When the fire is small and easy to extinguish, thesystem will send an alarm message to the corresponding
person in charge to notify him to deal with it. When thefire situation is large or there is a continuous
development trend, it will directly send an alarm message
to the neighboring fire department through the alarm
platform.

Fig.6. Sound and light evacuation module

3.3. Sound and light guiding function

3. Introduction of functional module

In order to effectively guide the crowd, the system has
designed a light guide and voice prompt guide.
The light grooming function is completed by amulti-directional indicator, which will display red in the
direction close to the fire point and green in the direction
far away from the fire point according to the informationof the alarm node.

3.1. Data monitoring function
The system uses infrared sensors and smoke sensors tocollect information and sends it to the client for digital
display. The user can view the monitoring data, working
status, and alarm status of each node through the systeminterface. The client is written in C# language and has
good compatibility and stability.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

673

674

Yuhui Cheng, Tianyi Zhang

each fire detection node to facilitate the evacuation of thecrowd.

The function of the voice prompt is realized by thevoice module. When a fire breaks out, the system will
broadcast an alarm and broadcast the approximate
location of the fire based on the node location
information entered in advance to provide guidance forcrowd evacuation.

References
1. Jianguo Wang, Baoyi Guo, Monitoring system based on
ZigBee and WiFi buildings combining, Journal of Xi’an
Technology University, 2008, 28(5): pp.460-465.
2. Fan Yang, Maolin Chen, Xun Wu, et al., Forest fire alarmsystem based on sensor information fusion technology,
Journal of Huazhong University of Science andTechnology: Natural Science Edition, 2013, 41(2):pp.22-25.

4. Conclusion
This fire alarm system is suitable for high-rise buildings
or crowded indoor shopping malls. After a fire occurs, it
can report to the police in time, and provide a reasonableevacuation route according to the location parameters of© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

675

Design of Intelligent Curtain Control Circuit Based on Single Chip Microcomputer
Qianqian Zhang *, Jichao Zhao, Haokang Wen, Hongbo Hao

Tianjin University of Science and Technology, China;
E-mail: 577972810@qq.com
www.tust.edu.cn

Abstract
Since entering the 21st century, high technology has promoted the development of human beings and artificial
intelligence has been popularized gradually. In this paper, the structure and principle of crystal oscillator circuit,reset circuit and photosensitive sensor circuit are introduced with 89C51 single chip microcomputer as the main
control unit. The whole circuit system is analyzed in this paper. The circuit module of photosensitive sensor can beused to detect the external light intensity and automatically control the curtain. The user can set the temperaturethreshold with the remote control.
Keywords: The steering gear 89C51; Photosensitive; Embedded remote control; Temperature monitoringchange of the external light intensity, and then causes thechange of output voltage. When the light intensity
weakens to the set value at night, the light intensity issensed by the photosensitive sensor, and the stepping
motor is controlled by 51 MCU to close the curtain.
When the light intensity increases to the set value in themorning, the light intensity is sensed by thephotosensitive sensor, and the 51 single-chip
microcomputer controls the stepper motor to open the
curtain.
(2)Temperature monitoring module: the infrared
remote control is used to set the temperature value, andthe temperature monitoring system is used to detect theambient temperature. When the temperature exceeds or islower than the given value, 51 single-chip microcomputer
controls the stepping motor to turn forward to open or
close the curtain 2.
(3)Infrared remote control module: The remote controlof this module is realized through the infrared signal sent
by THE HT6221 chip. When the receiver receives theinfrared signal, the decoding operation is performed first,
and then the code value corresponding to each button on
the LCD is displayed. The software program allows theuser to control the forward and reverse of the motor
through the remote control, so that the curtain can be
opened or closed. Here, another function of infrared
remote control is to set the initial temperature value, andthe button can change the initial temperature value, whichmakes people's life more convenient and comfortable.

1. Overall scheme design
This design starts from people's demand for the systemdesign function, under the circumstance ofcomprehensive consideration of various factors, designs
the overall framework of the automatic control system,and on the basis of the realization of the overall function,
considers the scalability of the system as far as possible 1.

Fig.1. Overall scheme block diagram

2. Each module scheme design(1)Photosensitive control module: This module firstly
changes the resistance value of photoresist through the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

676

Qianqian Zhang, Jichao Zhao, Haokang Wen, Hongbo Hao

3. Subsystem design
3.1Circuit design steps
The crystal oscillator in the circuit is used to generate thereference frequency and optimizes the frequency stability
and anti-interference. The accuracy of the frequency inthe circuit is controlled by the reference frequency. In
addition, it can also produce the clock signal andoscillation current of the microcontroller, the operation of
the single chip cannot do without crystal oscillator, it is
the peripheral hardware that the single chip systemcannot lack, if the circuit is abnormal, the single chipsystem will be paralyzed, unable to run, leading to thefailure of the whole system. All the programs of thesingle chip microcomputer are written into the read-only
memory when they are written.
The single chip microcomputer system must read theprogram after it is started. The time it takes the processor
to read a piece of code is one machine cycle of theprocessor, and this time is the beat for the processor.
Therefore, crystal oscillator circuit is the core peripheral
circuit of processor operation. The 12 MHZ vibration
source is selected in this design. Although the internal
vibration source of the MCU is provided, external
vibration source is selected because of the design
requirement. Figure2 shows the microcontroller crystal
oscillator circuit.

Fig.3.Reset circuit diagram

3.2 clock circuit
This design requires the curtain to be opened or closed
within a specified time, so a timer is needed. To ensurethat the microcontroller is consistent with the external
clock, the DS12887 real-time clock chip is used. The chipdata can be stored for more than 10 years without damage,and the signal on the RESET pin has no influence on the
RAM in the MCU4. The clock circuit diagram is shownin Figure4. RESET is directly connected to the Vcc toensure that DS12887 enters or exits the power off state.
From this pin, it can output a square wave with afrequency of 2HZ-256Hz.
When the interrupt occurs, the microcontroller reads
the input signal once or twice to check if the rolloff state
has been entered and obtains the three-phase current andvoltage at the entire point. MOT grounding, the high 8-bitof DS12887 is set to 7FH, while the low 8-bit isdetermined by the address of the chip (00H-3FH) in eachcell.

Fig.2.Single chip crystal oscillator circuit diagram
Fig.4. Clock circuit diagram

3.2Reset circuit

3.4Power circuit
The operation of the MCU requires 5V voltage, so it isnecessary to design the power circuit for the MCU.
Figure 5 shows the power circuit of the microcontroller.

The processor designed the reset function circuit, which
is used to reset the system when the system wants toreturn to the state of restart. In fact, when the processor
starts to run on power, it is a reset state, so the processor
startup is required to reset. You will also need to use thereset button to make pin 9 high to restart 89C51 in the"On" or "dead" locked state. Figure3 shows the reset
circuit diagram.

Fig.5. Power circuit diagram© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

677

Design of intelligent curtain

3.5 Stepping motor circuit
Stepper motor is a digital servo actuator with simple
structure, reliable operation and convenient control. SCM
is controlled by the rotation Angle and displacement of
the stepper motor 3. This design adopts the model ofthree-phase stepless variable speed motor 130HZ308-450
to control the rotation Angle and displacement. Stepper
motor has high torque, high impact resistance and highprecision. The drive circuit of the stepping motor
operates according to the control signal. Fig.6 and Fig.7
show the block diagram and circuit diagram of the
stepper motor control system.

Fig.8.Constant pulse width modulation subdivision drive circuit

In SCM application system keyboard can input dataand send instructions to the microcontroller. It is the mainmeans of manual intervention by a microcontroller 7.
When the program is wrong or has an error, you can press
the reset key to restore its original working state.
3.6LIQUID crystal display circuit
The connection of liquid crystal module is shown in
Figure9. Liquid crystal is used for display in the system.
This device can display all characters except Chinesecharacters, which can fully meet the design requirements.
The single chip microcomputer can control the contentsof the LIQUID crystal display through the data port andthe sequential interface. Through the operation of thesoftware can display any position of the liquid crystal
specified display content, easy to operate, simple. The
most important is the LCD requirements of the power
parameters and SCM system consistent, so that thegeneral power supply, power supply design does not haveto reduce design work.

Fig.6. Block diagram of stepper motor control system

Fig.7.Step motor control system circuit diagram

The pulse distribution circuit is controlled in apredetermined order to open and close the phases. This
design not only simplifies the circuit and reduces the cost,
but also flexibly changes the control scheme of stepper
motor according to the needs of the system. Software
control pulse is described in the software design section.
In order to operate in the state of large pulse current,
the stepper motor drive circuit works. The photoelectric
coupler is used for separation to avoid interference
between the SCM and the power supply circuit of
stepping motor, and to prevent the main control systemfrom strong electrical interference. In addition, if thedriving circuit fails, the MCU at the higher voltage of thepower amplifier will not be damaged, and the external
resistor of the driving circuit consumes a lot of energy,
which will affect the stability of the circuit 5. Dual
voltage drive circuit, as the name implies, has two supply
voltages. The current displays on high and low voltage
connections, which inevitably results in torque at valley
points, which is not easy for motor normal operation todrop.
This disadvantage overcomes the drive to the chopper
and the efficiency of the stepper motor can be improved.
In this design, constant frequency PWM subdivision
drive circuit is adopted, and the block diagram of thiscircuit is shown in Figure8.

Fig.9.LCD module connection diagram

3.7Photosensitive sensor circuit
The electric curtains are automatically switched on andoff depending on the lighting conditions, so photoelectric
sensors are required. Photodiode is the main componentof photosensitive sensor. It is a photoelectronic device
operating on the basis of photoconductor, also known as
the light tube8. The photodiode has no polarity. It is apurely resistive device. It can be used with a DC offset
voltage or ac voltage. When no light is present, the dark
resistance of the photodiode is large, and the current in
the circuit is small. When there is light irradiation, its
photodiode resistance drops sharply, then the current in
the circuit increases rapidly.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

678

Qianqian Zhang, Jichao Zhao, Haokang Wen, Hongbo Hao

Photodiode has the characteristics of high sensitivity
and good light splitting. In addition, it is small in size,
light in weight and stable in performance. Therefore, it is
widely used in automation technology. Photodiodes arecommonly used for optical measurement, optical control,
and photoelectric conversion (light to electricity).
Typically, photodiodes are sliced to absorb more light.
When it is illuminated by light, the electron-hole pair is
excited to participate in the conduction in thesemiconductor wafer (photosensitive layer), reducing theresistance value and thus increasing the current in thecircuit.

automatic curtain control system are finally realized incollaboration with each module.
The sensor part USES photodiodes, which cancontinuously detect changes in the external light intensity.
After the signal from the bridge circuit enters thecomparator, a signal can be obtained, and the operation of
the stepper motor is controlled by the pulse signal of thesingle-chip microcomputer7.This design of the stepper
motor can execute the commands of a single chip very
well. Stepper motor is a digital servo actuator, which has
the advantages of simple structure, reliable operation,
convenient control and good control performance. The
designed clock circuit is compatible with the timing
function of the single chip microcomputer, and thedetection light intensity of the photoelectric sensor solves
the problem of automatic control.
References

Fig.10.Schematic diagram of optical control circuit

Using the light control principle, the daylight curtain
automatically opens and the dark screen automaticallycloses. The operational amplifier consists of acomparison circuit with two resistors. The two resistors
are divided under the same input to obtain a voltage value,
which is used to compare with the reference voltage. The
reverse input USES a photodiode to collect the lightintensity of the external environment. When the indicatorlight is on, the voltage value of the invert input isconstant. Then the voltage values of the two resistors arecompared, and the compared signals are sent to the MCU
to determine the positive and negative rotation of the
motor through the MCU to control the conversion of highand low voltage at port P0, so as to realize the functionsof automatic opening of daylight curtains and automatic
closing of evening curtains.
In the meantime, we can replace photodiodes withcheaper ones on the market, but the effect is the same.
Photodiodes can change their physical properties
according to the intensity of light. The signal is changed
according to changes in the resistance inside thephotodiode by using the unidirectional conductivity ofthe diode.

1. Li Chaoqing. Principle and Interface Technology of Single
Chip Microcomputer. Beijing University of Aeronautics
and Astronautics Press, 2013.
2. Li shengli. Fundamentals of analog electronics technology.
School of electrical and electronic engineering, wuhan
institute of engineering, 2013, 13.
3. Yuan Zengmin. Basic Course of Analog Electronics
technology. Beijing: Electronics Teaching and ResearchGroup, Tsinghua University, 2014.
4. Song jingbin. Linux network programming (2ndedition).Beijing: Tsinghua university press, 2014, 2.
5. Liu gang, Zhao jianchuan. Linux system transplantation(second edition). Beijing: Tsinghua university press, 2014,2.
6. Hu Hancai. Principle of Single Chip Microcomputer andits Interface Technology. Beijing: Tsinghua University
Press, 2010: 59-62.
7. Liu Zhuqin, Bai Zesheng. (in Chinese) A Digital frequency
meter based on single chip Microcomputer. Modern
Electronic Technology, 2010, (1):15-18.
8. Chen Ken, Yang Xiangdong, Liu Li, Yang Dongchao.
Robotics and application. Beijing: Tsinghua University
Press, 2006.

4. Conclusion
The stepper motor is used as a control component of asingle-chip microcomputer to complete the main task of
the opening and close the curtain; Photoresistors are usedas detection kits to provide external illumination changes
for single-chip microcomputers; Infrared detection circuit
is used to achieve manual control, the main control chipusing 89C51 MCU. In addition, with the help of buttons
and display circuits, the intelligent requirements of the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

679

Research on the Algorithm of Flue Gas Desulfurization System
HongBo Hao
Tianjin University of Science and Technology, China
E-mail: 2844011096@qq.com
www.tust.edu.cn

Abstract
In this paper, the process of flue gas desulfurization and denitrification, which is a nonlinear, time-varying, large
lag and strong coupling complex variable process, is analyzed and studied in depth. Based on the analysis of thecoupling characteristics of the adsorption tower, the control model of the desulfurization and denitrification processof the adsorption tower is established, and the system identification adaptive PID Decoupling control algorithm isadopted. The Simulink toolbox corresponding to Matlab software is used to simulate and verify the effectiveness of
the algorithm. The RBF neural network identifier is used to identify the plant model on-line to improve the adaptive
ability of the controller.
Keywords: flue gas, desulfurization, PID, RBF neural network, Matlab

1. Introduction

desulfurization and denitrification process is based on the
joint action of physical and chemical reactions.
The corresponding research institutions have formed alot of mature theoretical literature on this reaction
mechanism. However, for different projects, the processadopted is different, the diversity of equipment and thedifference of process flow. So far, the desulfurization anddenitrification system have not formed a unified
mathematical model, nor has a complete set of intelligent
control algorithm. In recent years, there are more andmore researches on Modeling and control of flue gas
desulfurization and denitrification system. The research
direction is mainly divided into the following aspects:
process improvement, updating and adjustment ofdesulfurization and denitrification chemical equipment,
optimization of control and prediction algorithm.
The most important part of activated carbon
desulfurization and denitrification system is adsorption
tower. In this paper, the intelligent control combined withneural network and PID control is used to control theadsorption tower, which can overcome the problems ofnonlinearity, large lag and coupling in the complex
industrial process of desulfurization and denitrification.

With the continuous development of the country's social
economy, the law enforcement of environmental
protection has also been strengthened. In order toimprove the regional air quality, the state has vigorously
carried out the work of supporting the improvement of
atmospheric environmental quality. In recent years, theconcentration of inhalable particles in urban air has
decreased significantly, but the situation of air pollution
is still very serious. The power industry, as a major
pollution reduction industry, it is facing great pressure
from air pollution control. Activated carbon integrated
desulfurization and denitrification technology has been
widely used in flue gas purification in recent years.
Desulfurization and denitrification process is a nonlinear,large lag, strong coupling complex industrial processcontrol system, domestic and foreign scholars have done
a lot of research on it, and its control has basicallyrealized digital 1. At present, intelligent algorithm andcontrol technology are combined, and various
optimization algorithms are used to optimize it. The© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

680

Hongbo Hao

In this paper, the control system of sintering flue gas
desulfurization and denitrification in a domestic steel
plant are taken as the research object.

reaction treatment
atmosphere 4.

indirectlydischarged

to

the

2. Overall scheme design

The activated carbon system is mainly composed of threeparts: adsorption tower, desorption tower and activatedcarbon transportation system. The main task of theanalytical column is to separate the decomposable
substances adsorbed in the activated carbon at hightemperature, so that the activated carbon can be fully
recycled. Activated carbon transportation system involves
the addition and transportation of activated carbon inadsorption tower and desorption tower.

3.2. Activated carbon system

First of all, this paper in-depth understanding of activatedcarbon integrated desulfurization and denitrification
technology process, system structure, purification
principle. Through the analysis of desulfurization anddenitrification system, the main variables and internal
coupling relationship are determined. The identification
model is established by recursive factor least square
method. The identification model of desulfurization anddenitrification of adsorption tower under the integrated
process of activated carbon is established.
Secondly, based on the combination of neural network
and PID control, an intelligent decoupling control
algorithm, namely RBF online identification adaptive
PID Decoupling control algorithm are designed 2. The
basic adaptive controller NNC is composed of a single
neuron PID with self-learning and self-adaptive ability.
The object model is identified online by RBF neuralnetwork identifier to improve the adaptive ability of thecontroller. The intelligent decoupling control algorithm issimulated to verify the robustness and feasibility of thedecoupling control scheme combining neural networkwith PID control.
Finally, the network structure of the desulfurizationand denitrification control system is described. The
designed self-tuning PID Decoupling Algorithm Basedon RBF model identification is programmed. The
algorithm is programmed. The control effect of thealgorithm and the integrity of the algorithm function
block are verified by simulation 3.

3.3. Ammonia supply system
The main components of ammonia supply system include
liquid ammonia storage facilities and ammonia injection
facilities. The liquid ammonia from the storage tankenters the evaporator by its own pressure and it is heated
and it is evaporated into ammonia. Ammonia and air
from the dilution fan are mixed and diluted in the air
mixer. Next it was injected into the flue through theinjection system for chemical reaction, and nitrogenoxides are removed.
4. Design of desulfurization system
4.1. Multivariable desulfurization system
The multivariable system structure diagram, which
represents a multivariable system with N inputs and N
outputs 5. It is supplemented by task list module,
distribution plan module, task analysis module andpersonal center module. The desulfurization anddenitrification process control of adsorption tower are atypical dual input and double output multivariable control
system, where r1 and r2 are the set values ofdesulfurization and denitrification, and y1 and y2 are theconversion values of desulfurization and denitrification
respectively. Multivariable system structure diagram isshown in Fig.1.

3. Composition of desulfurization system
The flue gas desulfurization and denitrification system
mainly includes three parts: flue gas system, activatedcarbon system and ammonia supply system.
3.1. Flue gas system
The flue gas system is the main part of the desulfurizationand denitrification process. The sintering flue gas isfirstly heated by the heater at the inlet. Then it was sent tothe adsorption tower through the adsorption tower. It isalso sent to the chimney channel through the booster fan
at the tail, and it is the main exhaust fan. After theFig.1. Multivariable system structure diagram© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to24, 2021

681

Research on the Algorithm

desulfurization rate of the channel, and the identification
curve is shown in the Fig.3.

4.2. Analysis of denitration characteristics
Under the steady-state condition, the reaction conditions
and the whole reaction process of the desulfurization anddenitrification system are relatively stable. The relevant
parameters in each reaction process have no obvious
fluctuation, and the corresponding relationship between
controlled variables is relatively clear.
The variation characteristics of generator power at a
certain time, the load of sintering unit is stable in time
period (0-t). It changes into transition state (t1-t2), andstarts to stabilize at t2. It keeps stable for a period of(t2-t3). Finally it starts to change around t3 andrestabilizes at t4.The actual steady-state diagram ofdesulfurization is shown in the Fig.2.

Fig.3. Comparison of fitting value and real value

4.5. Structure of neural PID control
The working principle of neural PID control is as follows:on the basis of online identification of controlled object
by NNI, the weight coefficient of NNC is adjusted in realtime to make the system adaptive and achieve effective
control 5.
The adaptive PID control algorithm is as follows:
Fig.2. Actual steady state diagram of desulfurization

4.3. Data processing method

A single neuron is used to construct PID controller,
and its structure is shown in the Fig.4.

The actual process data contains a lot of industrial object
information, so the amount of data collection should be ascomprehensive as possible, so as to facilitate the latermodeling and model verification. Data processing mainly
includes data conversion and data error processing.
Conversion includes scale, conversion and weightfunction. Conversion has a direct impact on the processaccuracy, nonlinear mapping ability and numerical
optimization algorithm.
4.4. Model of desulfurization and denitrification

Fig.4. Single neuron PID controller

System identification is to determine the model
equivalent to the measured system according to the input
and output data of the system. The purpose of system
identification is to estimate the model structure andunknown parameters. It can be divided into two
categories according to the model form, one is non
parametric model identification method, and the other isparametric model identification method. This paper isbased on the second model identification method, namelyparameter model identification. The recursive factor leastsquare method is used to identify the activated carbon

4.6. RBF adaptive model design
The basic idea of RBF neural network is that RBF as the"base" of hidden layer node constitutes the hidden layer
space.n the case of over weight connection. The input
vector is directly mapped to the hidden space. After theparameters of the neural network are determined, thenonlinear mapping relationship is determined.
The transformation from the input layer space to the
output layer space is linear, and the output of the network© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

682

Hongbo Hao

is the linear weighted sum of the outputs of the hidden
layer nodes. Neural network weights can be solved byvarious linear optimization algorithms. RBF neuralnetwork is a three-layer feedforward network, which ismainly composed of input layer, hidden layer and output
layer 6. The neuron structure of RBF neural network isshown in the Fig.5.below.

According to the analysis of the above Fig.6, thetwo channels can respond quickly and achieve goodsteady-state results. The steady-state error is small.
PID realizes the adaptive adjustment process.
According to the image analysis, RBF realizes theonline identification and adaptive adjustment of themodel 8.
References
1. Li G, Wu Q, Wang S, et al. The Influence of Flue Gas
Components and Activated Carbon Injection on Mercury
Capture of Municipal Solid Waste Incineration in China.
Chemical Engineering Journal, 2017, 326: 561-569.
2. Shi-He Chen, Hong-Ming He, Xiao-Feng Li, et al.
Application of Fuzzy PID Controller with Self-Adaptive
Algorithm and Non-uniform Grid Scheduling to WFGD.
IFAC Proceedings Volumes, 2009, 42(9):20-25.
3. Zhang Y, Luan T, Yao Y. MtsPSO-PID Neural Network
Decoupling Control in Power Plant Boiler. IFAC
Proceedings Volumes, 2013, 46(20):101-105.
4. Sarah Sühnholz, Frank-Dieter Kopinke, Barbara Weiner.
Hydrothermal Treatment for Regeneration of ActivatedCarbon Loaded with Organic Micropollutants. Science of
the Total Environment, 2018, 644:854-861.
5. Yonggang Wang,Xinfu Pang,Zailin Piao,et al. Intelligent
Decoupling PID Control for the Forc ed-circulation
Evaporation System. Chinese Journal of Chemical
Engineering, 2018, 23(12):2075-2086.
6. Shetty K, Srinath, R, Shivashankar, Shetty, Rashmi P. Useof Radial Basis Function Neural Network for Bearing
Pressure Prediction of Strip Footing on Reinforced
Granular Bed Overlying Weak Soil. World Academy of
Science Engineering & Technology, 2012, 1(2):281-294.
7. Junwei Huang, Qingtai Xiao, Jingjing Liu, et al. Modeling
Heat Transfer Properties in an ORC Direct Contact
Evaporator Using RBF Neural Network Combined with
EMD. Energy, 2019, 173:306-316.
8. Deepti Deshwal, Pardeep Sangwan, Divya Kumar. A
Language Identification System Using Hybrid Features
and Back-Propagation Neural Network. Applied Acoustics,
2020, 164:67-92.

Fig.5. Neural structure of RBF neural network

5. Conclusion
This paper mainly analyzes the common methods andprinciples of decoupling, and uses RBF to identify
adaptive PID online. Decoupling control is studied bycontrol algorithm. In this paper, firstly, the classicalfeedforward decoupling controller is designed according
to the identification model. Then, an adaptive single
neuron PID identification based on RBF model is
proposed 7. The design and Simulation of intelligent
decoupling controller are carried out to verify theeffectiveness and superiority of the scheme.
Step signal is used to trigger the two channels at thesame time, and R1 = 1 and R2 = 1 are set to simulate thetriggering of the two channels. The simulation results areshown in the Fig.6.

Fig.6. Y1 and Y2 output curves© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to24, 2021

683

A Survey of Low Visibility Image Enhancement Based on MSRCR Algorithm
Haokang Wen*, Hongbo Hao

Tianjin University of Science and Technology, China
E-mail: * haokangwen@foxmail.com
www.tust.edu.cn

Abstract
With the development of computer vision systems in the fields of traffic and safety monitoring, image enhancement
has become an important research direction of computer vision. After development, the Retinex algorithm has abetter enhancement effect, and the MSRCR algorithm has now developed into one of the important methods inimage enhancement. This article introduces the development of the Retinex algorithm, and focuses on the mainprocess and steps of the MSRCR algorithm, and uses the algorithm to enhance the low-visibility images of haze andnight in multiple scenes. The actual results show that the MSRCR algorithm has a better enhancement effect andhas a wide range of application values.
Keywords: image enhancement, MSRCR, Retinex algorithm

1. Introduction

algorithms for image enhancement, including global andlocal histogram equalization, wavelet transformenhancement and Retinex based on color constancy
theory. This paper introduces the Retinex algorithm.

Sometimes there will be some low visibility in daily life,
such as the influence of haze caused by air pollution, andthe situation of insufficient illumination at night, which
will make the image acquired by the image acquisition
system produce more obvious degradation phenomenon,such as low color saturation, poor definition of edge
details, blurred image and so on. This will reduce theefficiency of later research and analysis. In order toimprove the quality of image acquisition under low
visibility and reduce the serious impact of fog weather on
outdoor imaging system, many scholars and researchers
have carried out a lot of research on image enhancement
technology.
For the image defogging technology, the multi-scale
information characteristics of the original image areusually taken as the main consideration factor, and theimpurities in the non-sensitive area of the image are
eliminated by compensating the contrast, brightness andcolor saturation of the image scene, so as to improve thevisual effect of the image. At present, there are many

2. Development of Retinex Algorithm
2.1. The basic model of Retinex
Retinex is a commonly used image enhancement method
based on scientific experiments and scientific analysis. It
was proposed by Edwin H. land1. Retinex is a word madeup of two words: retina and cortex. Land's Retinex modelis based on three assumptions:• The color we perceive is the result of the interaction
between light and matter. The water we see iscolorless, but the water film soap film is colorful,which is the result of light interference on the surfaceof the film.
• Each color region is composed of red, green and blue
primary colors of a given wavelength.
• The primary colors determine the color of each unit
area.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

684

Haokang Wen, Hongbo Hao

2.2. SSR algorithm

The basic theory of Retinex theory is that the color ofan object is determined by its ability to reflect long wave(red), medium wave (green) and short wave (blue) light.
Color uniformity is the basis of color uniformity.
Different from the traditional linear and nonlinear
methods, Retinex can achieve a balance in dynamic range
compression, edge enhancement and color constancy.
Therefore, Retinex can enhance various types of images
adaptively.
The main steps are as followed:
According to the image formation model, an image canbe expressed as:
I(x, y) = S(x, y) · R(x, y)(1)(x, y) is the coordinates of the pixels in the image, “S”
represents the incident light, “R” represents the reflection
characteristics of the object, “I” is the reflected light,
which is captured by the camera as an image.
Take the logarithm to separate the incident lightcomponent S and the reflected light component R. I’(x,y)
is the logarithm of I(x,y).
I ′ (x, y) = log[S(x, y) · R(x, y)]= log�S(x, y)� + log�R(x, y)�
(2)
Convolve the original image with a Gaussian template,which is equivalent to low-pass filtering the original
image to obtain a low-pass filtered image D(x,y), whereF(x,y) represents Gaussian filtering function:
D(x, y) = I ′ (x, y) · F(x, y)(3)
Subtract the low-pass filtered image from the original
image to obtain the high-frequency enhanced image
G(x,y).
G(x, y) = I′(x, y) − D(x, y)(4)
In the previous steps, the incident light component S
and the reflected light component R are separated, so theantilog of the resultant high-frequency enhanced image
G(x,y) must be taken to obtain the enhanced image
R(x,y).
R(x, y) = exp�G(x, y)�
(5)
The contrast enhancement of R(x,y) is performed toobtain the final result image. As mentioned above, theflowchart of the Retinex algorithm is shown in Fig.1.

Researchers such as Jobson proposed the single-scale
Retinex (SSR) algorithm2,3. The specific formula is asfollows:
R i (x, y) = logIi (x, y) − log[Ii (x, y) ∗ F(x, y)](6)
In the formula, Ri(x,y) is the output of Retinex in the “i”
color spectrum, Ii(x,y) is the image distribution, that is,the brightness value at the position (x,y). * represents the
convolution operation, F(x,y) is a wraparound functionthat is defined by equation (7).
−(𝑥𝑥2 +𝑦𝑦2 )F(x, y) = K · 𝑒𝑒 𝜎𝜎2(7)Among them, σ is the wrapping scale, K is thenormalization constant. The wrapping function satisfies:(8)∬ F(x, y)dxdy = 1
The stronger the dynamic compression capability of
SSR is, the better the details of the dark part of the image
can be enhanced, but the color distortion of the output
image is more serious.
2.3. MSR algorithm
MSR is developed on the basis of SSR algorithm4. The
specific formula of MSR is as followed:
R i (x, y) = ∑Kk=1 wk {logIi (x, y) − log[Ii (x, y) ∗ F(x, y)]},i = 1, … . , N(9)Among them, i represents the i-th color channel, and(x,y) represents the coordinates of the pixel in the image.
N is the number of color channels in the image. N=1
represents a grayscale image, N=3 represents a color
image, i ∈ (R,G,B). Ii(x,y) represents the i-th color
channel in the input image, Ri(x,y) represents the output
result of the MSR of the i-th channel. F(x,y) is a Gaussianfunction, k represents the number of Gaussian surround
functions or the number of surround scales, wk represents
the weight related to the Gaussian function, ∑Kk=1 wk = 1,
in general, MSR takes three scales of high, medium andlow, K=3. As shown in the Fig.2, it is the flow chart ofMSR algorithm.

Fig.2 Flow chart of MSR algorithm
Fig.1 Flow chart of Retinex algorithm© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

685

A Survey of Low

2.4. Basic theory of MSRCR algorithm

and the details of the enhanced image are more obvious.
In particular, the color of the object in the original image
is grayish white, but after enhancement, the color is more
vivid.

The algorithm with color recovery factor originated from
Retinex theory, which adds a color recovery factor to themulti-scale algorithm5,6.
(x, y) = Ci (x, y)RMSR(x, y)RMSRCRii′
Ci (x, y) = β log[αIi (x, y)]�Ii′ (x, y) = Ii (x, y)/ ∑3i=1 Ii (x, y)(10)Among them, i∈ (R, G, B), Ci is the color recovery
coefficient of the i-th channel, which is the key factor, βis the gain constant, α is the controlled nonlinear
enhancement coefficient, 𝑅𝑅𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀
is the output image of𝑖𝑖
the i-th channel processed by this algorithm.
The MSRCR algorithm is based on MSR algorithm,
adding the adjustment factor Ci which considers the ratioof R, G and B in the original image. Because the color
recovery factor has color compensation for the image
enhanced by MSR algorithm, the image enhanced by
MSRCR can better maintain the original color without
distortion.

Fig.3 Comparison of haze image enhancement effectAs shown in the Fig.4 below, the edge outline anddetails of the object in the original image are very fuzzy
and difficult to distinguish. The enhanced image on theright has better visual effect and clearer object contour
after processing the image at night.

3. Test Result
The main purpose of image enhancement is to suppress
the useless noise in the process of image acquisition andtransmission, highlight the useful information in theimage, make the image conform to the visual effect ofhuman eyes as much as possible, or make the imagetransform into a form that is conducive to computer
recognition and analysis, and improve the subsequent
processing ability and application value of the image.
In this paper, under the Windows 10, based on
MATLAB 2015a software, the MSRCR algorithm is used
to process the fuzzy images in haze and night. as shownin the Table 1 below, it is the computer environment.
Table 1. Computer environment
GPU
Video memory
Operating system
Software version

Fig.4 Comparison of night image enhancement effect

NVIDIA GeForce MX150
2 GB
Windows 10
MATLAB R2015aAs shown in the two figures, it can be seen from thecomparison that, although there are halos in some areasof the images, the overall enhancement effect issatisfactory due to the clearer details and brighter colorsof the objects.

As shown in the Fig.3 below, the image on the left is
the original image, and the image on the right is enhancedby this algorithm. The image in haze weather is processed,© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

686

687

A Survey of Low

References

4. Conclusion

1. Land E H. The retinex theory of color vision. Scientificamerican, 1977, 237(6): pp. 108-129.
2. Jobson D J, Rahman Z, Woodell G A, “Properties andperformance of a center/surround retinex,” IEEE
Transactions on Image Processing, 1997, 6(3): pp.
451-462.
3. Hines G, Rahman Z, Jobson D, et al. Single-scale retinex
using digital signal processors. 2005.
4. Rahman Z, Jobson D J, Woodell G A. Multi-scale retinex
for color image enhancement//Proceedings of 3rd IEEE
International Conference on Image Processing. IEEE,
1996, 3: pp. 1003-1006.
5. Jobson D J, Rahman Z, Woodell G A. Retinex image
processing: Improved fidelity to direct visualobservation//Color and Imaging Conference. Society for
Imaging Science and Technology, 1996, 1996(1): pp.
124-125.
6. Girija M G, Shanavaz K T, Ajith G S. Image dehazing
using MSRCR algorithm and morphology based algorithm:
A Concise Review. Materials Today: Proceedings, 2020,
24: pp. 1890-1897.

The theory and development of Retinex algorithm are
introduced in this paper. Especially the process andmethod of SSR, MSR, MSRCR algorithm are introduced.
At the end of the article, the MSRCR algorithm is used tofor practical test and the experimental results are
remarkable and satisfied.
The test results show that the enhancement effect of
this method is obvious and has certain practicability.
Retinex image enhancement algorithm based on humanvisual system has the advantages of color constancy,
image sharpening and high fidelity, which cannot becompared with traditional enhancement algorithm. The
color information of the enhanced image can be restoredor maintained well.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

688

Design of a Taxi Meter Based on Single Chip Computer
Haokang Wen *, Min Wang

Tianjin University of Science and Technology, China
E-mail: * haokangwen@foxmail.com

Abstract
As a convenient means of transportation, taxis greatly facilitate people's daily travel. Based on such a background,
this article designed a simple taxi meter based on STC89C52 microcontroller. This taximeter uses a single-chip
microcomputer as the core, combines display module, clock module, storage module, key module, and combines
software and hardware to build the required taxi meter model. The taxi meter introduced in this article has thefunctions of displaying time, starting price, and real-time display of the cost according to vehicle mileage.
Theoretically, the meter has higher accuracy and has a wider practical value in daily life.
Keywords: Taxi meter, single chip microcomputer, STC89C52

1. Introduction

memory chip and LCD1602 LCD display, including themain program module, power module, data display part,clock part, storage unit design, key module, etc. The
meter can control the charging status of taxis by pressing
buttons.
Firstly, different mileage corresponds to different
charging modes. Secondly, it has different charging
prices for day and night. At the same time, it also adds a
charging standard of waiting state during driving.
LCD1602 also displays real-time data such as time,
distance and price. In addition, the taximeter designed inthis paper not only meets the charging function, but also
increases the practical function of carpool and special car,so that the taximeter can meet more needs of consumers
and has better practicability.

With the continuous progress of society and the rapid
development of science and technology, people's living
standards are developing rapidly. As one of the modes oftransportation, taxi develops rapidly. People have become
more and more dependent on the convenient, fast andcomfortable services provided by taxis. Taxis have
gradually penetrated into our lives and graduallydeveloped into an indispensable part of our social
services. More and more people prefer taxis to travel.
The taximeter is one of the most important parts oftaxis, which plays an important role. The most importantfeature of taxi meter is to record the mileage calculation
cost. Only by ensuring that its core function is recognizedby passengers, can the industry operate well. At the sametime, the taximeter serves the taxi, and constantly
improving its function to better serve passengers will be
the constant pursuit of the whole taxi industry. This paper
will use STC89C52 MCU to design a simple and accurate
taximeter.

3. Hardware Design
3.1. STC89C52
STC89C52 is the core part of this taximeter. This
microcontroller has a wide range of applications, such asa design of intelligent led lighting systems1, and a designof laboratory fire alarm system2. And it can be also used
in a gain controllable system in RF receiver3. STC89C52
is an 8-bit microcontroller with low power consumption

2. Overall Design Ideas
The taximeter designed in this paper is composed of
STC89C52 single chip microcomputer, AT24C02© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

689

Haokang Wen, Min Wangand high performance produced by STC company.
STC89C52 uses the classic MCS-51 core, on this basis,the chip has done a lot of improvement. In this paper,
STC89C52 MCU is used to solve the mileage recording,billing calculation and other functions.
The main features are as follows:• Working voltage: 3.4V-5.5V.
• Working frequency: 0-40MHz.
• The instruction code of traditional C51 MCU iscompletely compatible.
• 512BS data memory can be integrated.
• The chip contains E2PROM function.
• There are three 16 bits timers / counters compatible
with MCS-51 single chip microcomputer timers.

•
•
•
•

The main performance of LCD1602:
The working voltage of the chip: 4.5-5.5V
Capacity: 16 × 2 characters
The best working voltage: 5.0V
Working current: 2.0mA

3.3. DS1302
In this paper, DS1302 chip is used to realize the functionof real-time display time. It has the characteristics of low
power consumption and high performance. It is a kind ofreal-time clock circuit, which can display the date, time,minute and second accurately. In the daytime and at night,
the charging standard of Taximeter is different, and timeis the most basic way to distinguish day and night. It isvery important to display the time in real time throughDS1302 chip for the charging link in the process ofdriving. As shown in the Fig.3, it is DS1302 chip.

3.2. LCD1602
In many electronic devices, LCD1602 will be used as adisplay module4. For example, the display interface ofmultimeter, calculator and so on uses liquid crystal
display, which will present the required digital data
through the LCD module. As shown in the Fig.1, it is the
LCD display module in this paper.

Fig.3 DS1302 pin diagram
Fig.1 LCD module

•
•
•
•
•
•
•
•

In this paper, LCD1602 display module is used todisplay the mileage, price, time and other data in theprocess of driving. It is the display module of this design
and undertakes the important function of displaying data.
As shown in the Fig.2, the circuit diagram of LCD
display module.

Pins and functions of DS1302:
Pin 1: VCC2 is the power supply pin.
Pin 2: X1 is a 32.768KHz crystal oscillator pin.
Pin 3: X2 is also a 32.768KHz crystal oscillator pin.
Pin 4: GND.
Pin 5: RST is the reset pin.
Pin 6: I / O is the input / output pin of data.
Pin 7: SCLK is the serial clock pin.
Pin 8: VCC1 is the power supply pin.

3.4. Key design
For the taximeter, setting the charging standard is a very
important link. This paper sets the charging standard bypressing the button. In the system designed in this paper,a total of six buttons are designed.
The first is the confirm key. By pressing this key, you
can enter the menu to set the parameters. No matter
which setting you want to change, you must press thisFig.2 LCD circuit diagram© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

690

Design of a Taxi

saved the data of the previous section, once powered on,
it can be recovered, which has great practical significancefor the taximeter.
Pins and functions of AT24C02 are shown below:• Pins 1-3: A0 -A2 are device address selection pins.
• Pin 4: GND, ground.
• Pin 5: SDA, serial data input and output port.
• Pin 6: SCL, serial shift clock control terminal.
• Pin 7: WP, hardware write protect control pin.
• Pin 8: VCC, connected to + 5V voltage.
As shown in the Fig.5, it is AT24C02.

key before entering the page to make corresponding
changes.
The second is the add key, which can be used to add
variables when setting parameters. For example, it can beused to accurately adjust the time and unit price. Whenthe time is slow or the unit price is set low, you can use
this key to speed up the time or increase the unit price.
The third key has two functions. It can be used as aminus key and as a start or pause key for managing
waiting time. Waiting time refers to the time thatpassengers spend when they want to get off and dosomething in the process of driving. There is a charging
standard for waiting time. The minus key can be used toadjust the time and unit price.
The fourth is exit key: Exit menu setting key. Aftereach setting, you can press this key to realize an exit
function.The fifth is the reset key, which is to reset it
after the end of the previous trip, so as to start a new
mileage charging.
The sixth key is used to simulate the increase of
mileage, that is, to simulate and record the driving
process of a taxi. Pressing this key is equivalent to a taxi
driving 0.1km.
As shown in the Fig.4, it is the circuit diagram of the
key part.

Fig.5 AT24C02 pin diagram

4. Simulation
In this paper, software Proteus is used for theoreticalsimulation. Proteus has a rich library of components, andcan easily create components. When searching forcomponents, we can accurately select the required
devices according to little information. It has powerful
function of schematic drawing. The MCU simulation andspice circuit simulation are combined. It supports thesimulation of mainstream SCM system. It provides
software debugging function. As shown in the Fig.6, it isa circuit diagram simulated by Proteus.

Fig.4 Key module circuit diagram

3.5. AT24C02 memory chip
The memory chip AT24C02 of ATMEL firm was choseas the memory cell for losing electricity5. AT24C02
memory chip is responsible for the realization of
power-off protection in this design. It is an electricallyerasable memory chip E2PROM with I2C bus interface.
This kind of chip can save data when power is off.
The I2C bus can be used to communicate with single
chip microcomputer and save the working data. In theprocess of driving, if there is power failure, there is noneed to be nervous, because AT24C02 memory chip has

Fig.6 Simulation diagram© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

691

Haokang Wen, Min Wang

5. Software Design
actual test. In theory, the meter has high precision andwide application value.

In this paper, the design of software can be divided into
several steps: main program design, timing interrupt
service program, mileage counting interrupt service
program, midway waiting interrupt service program,display subroutine service program, etc. these programs
are combined with MCU, key circuit, LCD1602 display
circuit and DS1302 clock circuit involved in hardware
circuit. This paper uses the program to drive the circuit toachieve the design function of the taxi meter, and finallypresents a simple but accurate taxi meter.

References
1. Wang T, Chen T, Hu Y, et al. Design of intelligent LED
lighting systems based on STC89C52 microcomputer.
Optik, 2018, 158: pp.1095-1102.
2. Zeng X. Design of Laboratory Fire Alarm System Basedon MCU. International Core Journal of Engineering, 2020,
6(6): pp. 269-272.
3. Dai F, Wen H. A Design of Controllable Gain
Amplification System Based on Single Chip Computer.
4. Wen H, Dai F, Zhao J, et al. Design of a portable
instrument for measuring heart rate and blood oxygen.
Proceedings of International Conference on Artificial Lifeand Robotics, 2020, 25: pp.682-685.
5. Zhao Y R, Wang D S, Li J D. Design of the Function Unitsof Electronic Password Lock//Applied Mechanics andMaterials. Trans Tech Publications Ltd, 2012, 220: pp.
2008-2011.

6. Conclusion
After the actual test, the function of the taximeter
designed in this paper can be realized and the expected
effect can be achieved. The simple taximeter designed inthis paper can meet various scenes of daily life after© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

692

Design of a High Precision Digital Clock Based on Single Chip Microcomputer
Haokang Wen*, Qiang Zheng

Tianjin University of Science and Technology, China
E-mail: * haokangwen@foxmail.com
www.tust.edu.cn

Abstract
Compared with traditional mechanical clocks, digital clocks have higher accuracy and durability and are widelyused in people's daily lives. This paper designs a digital clock which uses AT89S52 single-chip microcomputer as
the main control chip. It can count hours, minutes, and seconds, and can calibrate the time. It can switch between
24-hour and 12-hour systems. As a smart clock, while displaying the time, it also adds the function of temperature
display. The intelligent digital clock designed in this paper has stable performance in theory and has certain
practical value.
Keywords: digital clock, DS18B20, single-chip microcomputer

1. Introduction
Digital clocks have become electronic necessities inpeople's daily life. For example, they have been widelyused in personal family life, subway transportation
stations, movie theaters, cultural offices and other public
places, which greatly facilitate people's life, study, workand entertainment. Compared with the traditional
mechanical clock, the accuracy and intuitiveness of thedigital clock have been improved. Because it has no
mechanical device, the service life of the digital clock isrelatively long and has been widely promoted.
The general trend of digital clock development in thecurrent era is gradually moving closer to higher accuracy,
smaller size, more functions, and lower power
consumption. In this context, the degree of digitizationand precision of the clock has become the dominant
design direction of the current clock development andprogress. The hardware system includes 5 independent
touch switch buttons and an LCD liquid crystal display,which can display necessary information. According to
the needs of the user, the time can be further calibrated,
selected time, and temperature display at any time. In
addition, it can convert between 12-hour and 24-hourformats.

2. Overall Design
The digital clock designed in this article mainly includes
two parts: hardware and software. The hardware part isthe basis for the completion of the operation of thedesigned system, and the software part effectively andfully supports and utilizes the hardware of the system tomeet the functional requirements of the designed system.
As shown in the Fig.1, it is the schematic diagram.

Fig.1 The schematic diagram© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

693

Haokang Wen, Qiang Zheng

of AT89C51 single-chip microcomputer, and has
extremely high compatibility with Intel’s 80C52 series
chips in terms of pin arrangement, hardware composition,
working characteristics and instruction system. In thesystem designed in this article, the internal clock modeand the button level reset circuit are selected to form thesmallest circuit of the single-chip microcomputer.
The main operating characteristics of AT89S52 are as
follows:• 8k byte independent programmable Flash storage.
• Eight interrupt sources.
• Fully compatible with MCS-51 series single-chip
products.
• Fully static command operation: 0Hz～33Hz.
• Three-level encrypted program memory.

The digital clock designed in this paper uses AT89S52
single-chip microcomputer as the main control chip1, theclock module uses DS12C887 as the clock chip, thetemperature sensor DS18B20 chip as the temperature
module, the LCD1602 as the display module, and the
button control part uses the touch switch button circuit.
In theory, the electronic clock designed in this paper
has high practicability and stable performance. The
software part of this system runs under the Keil μVision4
software platform environment, and implements specificfunctions with a single-chip C language program.
3. Hardware
3.1. AT89S52
AT89S52 is a member of MCS‐51 family2. It is a
low-voltage, high-performance CMOS 8-bit single-chip
microcomputer produced by Atmel in the United States.
It contains 8KB of program memory that can be written
repeatedly and 256 bytes of data memory (RAM). The
device is produced using Atmel's high-density,
non-volatile storage technology, compatible with thestandard MCS-51 instruction system, and is equippedwith a general-purpose 8-bit central processing unit (CPU)and Flash storage unit on-chip. As shown in the Fig.2, itis the AT89S52 chip pin diagram.

3.2. DS12C887
The DS12C887 clock chip is a real-time clock/calendar
chip launched by the Dallas Semiconductor Company of
the United States3. It uses a parallel interface and is madeof CMOS technology. It has a crystal oscillator and alithium battery backup for the clock chip. It is compatible
with the pins of MC146818 and DS12887, which arecommonly used in daily computers, and can be directly
replaced. The clock circuit designed with the DS12C887
clock chip does not require any peripheral circuits anddevices, and has a good microcomputer interface. The
DS12C887 clock chip has the advantages of low power
consumption, relatively simple peripheral interface,extremely high accuracy of time travel, and stable andreliable operation. It is widely used in various real-timeclock systems that require higher precision. As shown in
the Fig.3, the connection circuit diagram of the
DS12C887 clock chip and the microcontroller.

Fig.2 AT89S52 pin diagram

The powerful AT89S52 single-chip microcomputercan be flexibly applied to various control fields.
AT89S52 single-chip microcomputer is an enhanced type

Fig.3 The DS12C887 connection circuit diagram© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

694

Design of a High3.3. DS18B20

display letters, numbers, symbols, etc. It is composed ofseveral 5×7 or 5×11 dot matrix character bits. Each dot
matrix character bit can display a character. There is a dot
pitch between each bit, and there is also a gap between
each line that plays the role of character spacing and line
spacing, because of this, it cannot display graphics. The
1602LCD display means that the displayed content is
16×2, that is, it can display two lines, each line has 16
characters LCD modules.
The characteristics of LCD1602 are as follows:• +5V voltage.
• Built-in reset circuit.
• Provide various control commands, such as: clearing
the screen, blinking characters, blinking cursor,
display shift and other functions.
• There are 80 bytes display data memory DDRAM.
• Built-in character generator CGROM with 192 5×7
dot matrix fonts.
• With user-definable 5×7 character generator
CGRAM.

DS18B20 temperature sensor is an improved intelligent
temperature sensor launched by DALLAS Semiconductor
Corporation of the United States. Compared with thetraditional thermistor and other components, the sensor
can directly read the temperature of the environment
where the sensor is currently located, and can achieve
9-12 digital value readings through simple programming
according to actual performance requirements4. The
temperature of the environment is directly used for datatransmission in a one-wire bus, which greatly improves
the anti-interference characteristics of the sensor itself.
This chip is suitable for temperature measurement inharsh environments, such as temperature measurement
consumer electronic products of various scales. As shownin the Fig.4, it is the interface circuit diagram of
DS18B20 temperature sensor.

4. Software
The entire software system in this paper uses C language
programming. The high-precision digital clock software
designed adopts a modular structure and mainly realizes
the following functions:• LCD function realization.
• Time data collection.
• The realization of 12/24 hours conversion.
• Temperature collection.
• Key recognition processing.

Fig.4 DS18B20 circuit diagram

The performance characteristics of this chip are as
follows:• The unique single-wire interface requires only one
port pin for inter-chip communication.
• The temperature measurement range is -55 ℃ to
125 ℃ , and the maximum resolution can reach
0.0625℃.
• The 3-wire system is connected to the single-chip
microcomputer, which reduces the use of external
hardware equipment.
• It can be powered by the data line, and the voltage
range is 3.0V-5.5V.
• Negative voltage characteristics, when the polarity ofthe power supply is reversed, the thermometer will
not burn out due to heat, but it will not worknormally.

5. Conclusion
This text design uses AT89S52 as the main control
single-chip microcomputer, the clock module chooses
DS12C887 as the clock chip, the temperature modulechooses DS18B20 as the temperature sensor, LCD1602
as the display module, and the button circuit for thesetting part.
The high-precision digital clock based on the
single-chip microcomputer in this design realizes thecounting of seconds, minutes and hours, and can switch
between the 24-hour system and the 12-hour system.
After testing, the system has achieved the goal of aligning

3.4. LCD1602
This LCD1602 character liquid crystal display module isa dot matrix liquid crystal module specially used to© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

695

696

Haokang Wen, Qiang Zheng

The content displayed on the LCD1602 display
includes precise hours, minutes, seconds, and current
ambient temperature.
After actual testing, the digital clock designed in thisarticle has high performance and certain usability.

the minutes and hours at any time, and the timing error isless than 0.01s/day.
As shown in the Fig.5, it is the final test result.

References
1. Wu H, Tang M, Huang G. Design of multi-functional
street light control system based on AT89S52 single-chip
microcomputer//2010 The 2nd International Conference on
Industrial Mechatronics and Automation. IEEE, 2010, 1:
pp.134-137.
2. Munir M M, Suhendi A, Thaha H, et al. An AT89S52
microcontroller‐based single board computer for teachingan instrumentation system course. Computer Applications
in Engineering Education, 2007, 15(2): pp.166-173.
3. Ding Y, Yao Y. Illuminance Test System Based on
MSP430//2010 2nd International Conference on
Information Engineering and Computer Science. IEEE,
2010: pp.1-3.
4. Wen H, Dai F, Zhao J, et al. Design of a portable
instrument for measuring heart rate and blood oxygen.
Proceedings of International Conference on Artificial Lifeand Robotics, 2020, 25: pp.682-685.

Fig.5 Test result
The key functions are as follows:• K1: Digital clock 12/24 hours conversion button.
• K2: Setting button, used to adjust the hours, minutes,
and seconds of the time.
• K3: "+" button, the number increases by 1.
• K4: "-" button, the function of reducing the number
by 1.
• K5: Cancel button, cancel the current operation,
return to the main page (time display page).

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

697

Design of Manchu Image Acquisition System Based on STM32
HongBo Hao 1, *, Fulin Zheng 2
1

Tianjin University of Science and Technology, China;
2

Dalian Minzu University, Liaoning,China
E-mail: 2844011096@qq.com
www.tust.edu.cn

Abstract
This paper designs a Manchu image acquisition system based on STM32 to protect and utilize Manchu ancient
books. The system uses STM32F407 as the core controller, with the help of image sensor, LCD screen, and SD
storage module to complete the function of image acquisition, storage and upload of Manchu ancient books. This
paper describes in detail the hardware circuit design, software programming and overall function debugging of theimage acquisition system. In order to achieve the purpose of digital protection of Manchu ancient books, we cancollect, display, save and upload images in different situations.
Keywords: Manchu, STM32F407, SD storage module, image acquisition system

1. Introduction

For the protection and utilization of Manchu ancient
books to make a certain contribution. The researchmethod of this topic can be applied to more ancient books
protection work. After the digital processing of ancient
books, it can save a lot of paper resources, reduce thegeneration of waste and reduce the felling of trees. It
plays a certain role in protecting the environment. At thesame time, digital preservation can save a lot of humanresources, and the transmission speed is faster, realizing
the coordinated development of economy, environmentand society. Based on the embedded processor platform,
using STM32 core controller and CMOS camera, thispaper designs an image acquisition device to realize theacquisition, storage and display of Manchu images inancient books.

Manchu is also used to write Manchu characters. Manchu
not only created Manchu characters, but also used
Manchu characters to record a large number of customs
historical stories. Therefore, Manchu has a very
important position in the history of Chinese characters.
Manchu ancient books also have important historical andcultural value. With the change of time, the preservationof Manchu ancient books is becoming more and moredifficult and the researchers' research is more and moreinconvenient. In order to facilitate the research ofresearchers, at the same time better preservation ofManchu ancient books, with the development of time, weneed to use more advanced technology to protect ancient
books. Considering the present situation of science andtechnology development, design cost and people's use
mode, the first choice of literature protection method is todigitize ancient books. In this paper, we choose theManchu image acquisition system based on STM32 tocarry out the research. That is to use the ancient books
digital technology to digitize the Manchu ancient books,
so as to achieve the purpose of protecting and using theancient books 1.

2. Overall scheme design
In this paper, STM32F407 chip is used as the core
controller of Manchu image acquisition system based on
STM32. Image acquisition module, image saving module,
image display module and key module are controlled bydifferent IO ports to realize image acquisition, storage© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

698

Hongbo Hao, Fulin Zheng

display. At the same time, the data transmission module
completes the connection with the upper computer.
According to the designed system scheme, the camera
image acquisition module, image storage module, key
control module and TFT liquid crystal display module aredesigned and built. After completing the circuit design,
according to the relevant theory of image acquisition andprocessing and the camera control method, the system
software design and overall debugging are completed.
The function of collecting, displaying, saving anduploading Manchu images in ancient books is realized
through the key control module 2.

of CMOS imaging is getting better and better, and thecost is lower than CCD. To sum up, this design uses
CMOS camera as the image sensor of this study.Throughthe study and research of CMOS image sensor, the circuit
diagram design of image acquisition module is completed
according to the function of different interfaces, as isshown in Fig.2 below.

3. Hardware circuit design
3.1. Design of core controller
In this paper, the core controller is used to receive theimages collected by the image sensor and transmit themto TFTLCD for display. According to the researchrequirements, STM32 series chips are selected.
STM32F407 is selected as the core controller to complete
the design. STM32F407 has a great improvementcompared with STM32F1 series. First of all, the speed of
FSMC is very fast. The screen swiping speed of F407 canreach 3000W pixels per second, while F103 is only about
500W. When using the image sensor with 200W pixels,
F103 has a stuck phenomenon, and F407 can be perfectly
processed.

Fig.2. Image acquisition module circuit

In the interface of capacitive touch screen, CS is thechip selection signal pin, which is effective at low level.
RS is the data command signal pin, 0 is the data and 1 isthe command. WR is the write enable signal, which isvalid at low level. Rd can read the pin effectively. RST isa reset signal, which can control the reset of capacitive
touch screen. CS is the reset signal of capacitive touch
screen. According to the function of each pin, thecapacitor touch screen can be controlled to complete therequired operation 3.

3.2. Touch the capacitor button
This design uses the capacitive touch screen to complete,
so we plan to use a capacitive touch button as the return
key of the system. When the system enters a function
module, touch the button to return to the upper level. The
circuit diagram is shown in the Fig.1 below.

3.4. Storage module
The research content of this paper is Manchu image
acquisition, so the preservation of collected image is very
important. After comparison and research, this paper uses
an 8G SD card as the storage module of Manchu image
acquisition system. The circuit diagram is shown in Fig.3.

Fig.1. Capacitive touch button path diagram

3.3. Image acquisition module
In this paper, through the analysis and comparison of theperformance of a variety of image acquisition sensors, for
CCD sensor, the photoelectric conversion of CMOS
sensor is very direct and the efficiency is higher. With thedevelopment of microelectronic technology, the quality

Fig.3. Circuit diagram of image storage module

As shown in the Fig.3, the SD card has 6communication lines and 3 power lines. The CMD is a

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

699

Design of Manchu Image

command line and a two-way signal line. DAT0, DAT1,
DAT2 and DAT3 are four data lines, which are alsobidirectional signal lines. The transmission rate of eachdata line is 25 Mbit per second. The maximum is that four
data lines read data at the same time, and the maximum
can reach 100 Mbit.

create a system_ Int function 5. The flow chart of thewhole is shown in Fig.5 below.

3.5. Data transmission module
The data transmission of this design is to transmit thecollected image to the upper computer for saving orbrowsing. After comparing the speed and stability ofserial communication, wireless communication and USB
communication, USB communication is selected as thedata transmission mode between Manchu image
acquisition system and upper computer. USB
transmission speed is faster and more stable, whentransferring large files such as pictures, it is faster thanserial transmission and Bluetooth transmission. WiFi
transmission needs to install WiFi module and specifichost computer receiving. USB transmission does not needa specific host computer software. It can be convenientfor data transmission with most computers. The standardof USB is four buses, VCC and GND, and D + and D -.
Because USB cannot be directly connected with STM32,
the circuit diagram of this design is shown in Fig.4 below
4.

Fig.5. Main program flow chart

4.2 Main program flow chart
After the initialization, the initial interface isdisplayed. The initial interface mainly shows the
English name of the research content in this paper
and the detection results of whether each module isnormal. It includes whether the icon image of APP
exists, whether the screen is normal. The maininterface is designed to display three main function
icons, which are image acquisition, image display
and USB transmission. Then the system detects
whether there are touch screen events. If the threeicons click events, it will enter different modules
according to the click events.
5. Conclusion

Fig.4. USB transmission circuit diagram

5.1. System

4. Software design
4.1. Main program flow chart

There are many problems in the testing process of image
acquisition. In the first acquisition, OV7725 image
acquisition sensor is used, which is a 30W pixel sensor. It
is found that the collected images cannot be displayed
clearly. After the teacher's help, the OV2640 image
sensor was re selected to complete the image acquisition.

The program design of this chapter is based on Keilsoftware and C language. Finally, the program design ofManchu image acquisition system is completed. In themain. C main program, the system initialization is to© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

700

Hongbo Hao, Fulin Zheng

It is the acquisition interface of Manchu image
acquisition system, as is shown in the Fig.6 below.

communicate with the upper computer. Complete thesystem function of data transmission.
References
1. Shuang Xu, Min Li, Rui-Rui Zheng, et al. Manchu
character segmentation and recognition method. 2017,20(1): 43-53.
2. M. LI, R. ZHENG, S. XU, F. WANG. Design and
Implementation of Embedded Handwritten Manchu Letter
Recognition System, 2018 International Conference on
Information Systems and Computer Aided Education(ICISCAE), Changchun, China, 2018, pp. 7-11.
3. R. Zheng, M. Li, J. He, J. Bi, B. Wu. Segmentation-FreeMulti-Font Printed Manchu Word Recognition Using Deep
Convolutional Features and Data Augmentation, 2018 11th
International Congress on Image and Signal Processing,
BioMedical Engineering and Informatics (CISP-BMEI),
Beijing, China, 2018.
4. Guang-yuan Zhang, Jing-jiao Li, Rong-wei He, Ai-xia
Wang. An offline recognition method of handwritten
primitive Manchu characters based on strokes. Ninth
International Workshop on Frontiers in Handwriting
Recognition, Kokubunji, Tokyo, Japan, 2004, pp. 432-437.
5. M. Li, R. Zheng, S. Xu, Y. Fu, D. Huang. Manchu Word
Recognition Based on Convolutional Neural Network with
Spatial Pyramid Pooling, 2018 11th International
Congress on Image and Signal Processing, BioMedical
Engineering and Informatics (CISP-BMEI), Beijing, China,
2018.
6. Choi Minseok, Lee Choong Ho. A Study on the
Preprocessing for Manchu-Character Recognition. Journalof the Institute of Convergence Signal Processing, 2013,14(2).

Fig.6. Manchu image acquisition and testing

After image acquisition, the image display module
needs to be tested. First, the main interface of image
display is shown in the Fig. below. After entering themain interface, select the image you want to view andclick. Click backward to enter the mode of automaticplay. Automatically play the first picture you click on.
When playing automatically, click the middle area ofthe screen to pause the play, and click again to resume
the play. Click the top and bottom of the screen tocontrol the up and down movement 6. BMP format
image display is shown in the Fig.7 below.

Fig.7. BMP format image display

This paper compares the advantages and disadvantagesof various data transmission methods, and uses USB data
cable to connect with the upper computer. According to
the USB communication protocol, a kind of analog USB
flash disk is realized. Click the data transmission icon,
the system will enter the state of analog U disk and© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

701

Design of Timing Socket Based on Single Chip Microcomputer
HongBo Hao1 *, Shuailin Chen 2
1

Tianjin University of Science and Technology, China;

2

Qingdao University of Science and Technology, Shandong, China
E-mail: 2844011096@qq.com
www.tust.edu.cn

Abstract
In this paper, the design of timing socket is mainly used for smart timing control of household appliances, and theworking time of timing is set by pressing the key. In the working period, the external connection device and thepower supply are in the conduction state. Outside the set time period, the external connector is disconnected fromthe power supply. This design uses STC89C52 as the driving chip. DS1302 clock module provides accurate timing.
LCD1602 LCD module displays time and operation interface. The relay module controls the connection between
the socket and the power supply. DC power supply module is converted to DC power supply module.
Keywords: Smart timing control, STC89C52, DS1302 clock module, DC power, Socket

1. Introduction

the standby loss of electrical appliances can be solved,
the purpose of saving electricity can be achieved, and thehidden danger of safety can be eliminated 1.
At present, in the market, the technology of time
controller is relatively advanced and complete. Time
controller is widely used in all kinds of electricalappliances. In the field of smart socket, however, it is stillin the stage of development. The smart socket is only
limited to the knob timer, or the time control socket withlarger volume designed by counter chip. This kind ofsocket timing time is single, the function is single, thetiming accuracy is low and it is difficult to really meet theneeds of our daily life. Now we need a powerful time
control socket to meet the market demand and improve
our life. With the development of modern electronicscience and technology, low power consumption andother characteristics. The price of finished products has
decreased, and has been widely used. Accurate display ofreal-time time and timing time, it has the memory
function and the data will not be lost after power off. It
not only achieves the function and reduces the cost, butalso it let our daily life have a profound impact.

Nowadays, with the improvement of living standards andthe acceleration of life rhythm, the sockets on the market
cannot meet our needs because of their single function.
For example, electric vehicle charging is usually about 8hours. If it is too long, it is easy to lose the battery. If it is
too full, it is not enough to charge. If it is charged at night,
you often don't want to go out from home to the garage tocharge the car. In the daytime, they forget to turn on the
power; the water tower at home forgets to pump water,which causes temporary inconvenience to life. The
school's broadcast of getting up is delayed because thestaff on duty overslept. The fish tank in the home has notbeen supplied with oxygen for too long, which causes thefish to die of lack of oxygen. Many fans or news
enthusiasts miss the start-up time and miss the wonderfulball games, so it is not convenient to use it at night.
Considering the above reasons, we urgently need a smart
socket. This socket can supply power to electricalappliances regularly and cut off the power supply ofelectrical appliances outside working hours. In this way,

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

702

Hongbo Hao, Shuailin Chen

2. Overall scheme design
This chapter mainly discusses the overall scheme of thesystem and the design scheme of each module, including
the control core, switch, display module, clock circuit andkeyboard circuit. In this paper, STC89C52 is selected as
the core component of this design. The basic timing
control function is realized by using the flexible
programming design and rich IO port of the single chip
microcomputer and the accuracy of its control. The
overall design of the system mainly realizes the following
functions:(1) Human computer interaction interface: throughindependent buttons to establish a complete set ofhuman-computer interaction interface.
(2) Infrared remote control: the system can set two
timing time through the infrared remote control, and It
can be forced on / off to confirm the start timing function(3) Buzzer prompt: when the power is on and off,
there will be a buzzer prompt.

Fig.1. Minimum system circuit

3.2. Clock circuit module
The system needs to calculate the exact time, so themodule uses an 11.0592MHz crystal oscillator. The clock
circuit is used to generate the clock signal needed by
MCU. In order to ensure the realization of synchronousworking mode, the circuit should work in strict
accordance with the time sequence under the control of
the only clock signal. The clock circuit diagram is shownin Fig.2.

2.1 Power supply scheme
The design power supply is + 5V DC power supply. In
this paper, USB power cord is used for power supply, one
end is connected with DC interface, the other end isconnected to + 5V power supply. The power supply modecan be computer USB, mobile phone charger, power bank,etc. The power supply is supplied by 220 VAC to 5 VDC
regulated power supply.

Fig.2. Clock circuit diagram

2.2. Relay circuit scheme
Relay is a kind of electrical switch, which is actually aswitch device with small current to control large current.
It can not only protect the circuit, but also realize thefunction of safe conversion circuit. There are many kindsof relay, including electromagnetic relay, time relay, andtemperature relay. This paper uses SRD-05VDC-SL-C
relay 2.

3.3. Reset circuit
Reset is the initialization operation of single-chip
microcomputer. When the single-chip microcomputer
starts and runs, it must be reset. The reset circuit includes
two parts: power on reset and manual reset. The reset
signal of MCU is high level reset. The reset circuit
diagram is shown in Fig.3.

3. Hardware circuit design
3.1. Minimum system of single chip microcomputer
The minimum system consists of reset circuit, clock
circuit and STC89C2. Minimum system circuit as isshown in the Fig.1 below. This system uses STC89C52
control chip. The single chip microcomputer not only has
strong function and low power consumption, but also hasstrong anti-interference ability and high cost performance
ratio 3.

Fig.3. Reset circuit diagram© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

703

Design of timing socket

the disconnection of relay and the calibration ofoscillation frequency.

3.4. Socket power circuit
The power on and off is not frequent, so this design uses
the relatively low price relay as the control circuit deviceof socket power supply. An I / O port of the single chip
microcomputer controls the on-off of the relay through a
triode, so as to control the power on and off of the socket.
The control circuit of socket power supply is shown in
Fig.4.

Fig.4. Socket power circuit

3.5. Independent key module
The key K1 is used to set the subtraction operation of thetime value. The key K2 is used to set the addition
operation of the time value. Key K3 can be set separately,
and then select manual mode or automatic mode. The
default mode of MCU is automatic mode. It can also
work with the K4 button to set the position of the variable.
Schematic diagram of key module is shown in Fig.5 4.

The flow chart of the whole system software design isshown in Fig.6.
Fig.6. Software design flow chart

4.2. Design of setting time function
The flow chart of setting timing is shown in Fig.7 below.
After power on, enter the time timing program, and thenjudge whether the key is valid. If it is valid, set the startand end time of timing through four independent keys. If
it is judged that the key is invalid, it will return to wait for
the key program until the end of the setting. Set timing
time program design: set function void key_ with( ) 5.
5. Conclusion

Fig.5. Schematic diagram of key module

5.1. System simulation and testing

4. Software design

According to the designed hardware schematic diagram,the simulation circuit diagram is drawn in Proteus. Thenthe designed program is compiled and saved in. Afterrunning, the LCD can display the calendar time. The
display page shows the year, month, day, 2019-05-17 in
the down line, 15:30:51 in the uplink and w: 5 in theweek. System simulation circuit diagram is shown in
Fig.8 below 6.

4.1. Main program flow chart
In this design, the program is mainly under the control ofsingle-chip microcomputer. In this process, the MCU firstinitializes, including setting the direction of each port, theinitialization of each variable, the initialization of LCD,

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

704

Hongbo Hao, Shuailin Chen

Fig.9. System simulation waveform

References
1. K. Tsai, F. Leu, I. You, Residence Energy Control System
Based on Wireless Smart Socket and IoT, IEEE Access,vol. 4, pp. 2885-2894.
2. C. E. Shannon, A symbolic analysis of relay and switchingcircuits, Electrical Engineering, 57(12): pp. 713-723.
3. M. Wu, X. Wu, Design of the single-chip microcomputer
experiment teaching system for application, 2011
International Conference on Electrical and Control
Engineering, Yichang, 2011, pp. 649-652.
4. Kong, Fan Rong, Wang, Wenxin, Zhu, Jiajun. A Smart
Home Control System Based on Controller Area Network
Bus. 2013, 11(9):1667-1671(5).
5. Sandy Suryo Prayogo, Farid Al Rafi, Yulisdin Mukhlis.
Design and Built IoT Home Panic Button for Smart City.
2019, 1175(1).
6. Hamid Mahmoudi, Mohsen Aleenejad, Reza Ahmadi.
Reconfigurable rapid prototyping platform for powerelectronic circuits and systems for research and educational
purposes. 2018, 11(7): 1314-1320.

Fig.7. Schematic diagram of key module

VSS
VDD
VEE

RS
RW
E

D0
D1
D2
D3
D4
D5
D6
D7

4
5
6

7
8
9
10
11
12
13
14
+5V

+5V

1
2
3

LCD1

LM016L

RV1

P00
P01
P02
P03
P04
P05
P06
P07

P24
P25
P26

10k

1

RP1

2
3
4
5
6
7
8
9
10K

GND

Fig.8. System simulation circuit diagram

5.2. Power simulation and testingMultisim 13.0 is selected for the simulation and testing of
DC voltage stabilizing circuit, which has rich simulation
and analysis ability. This design first according to thehardware design circuit diagram, draws the circuit
diagram in the Simulation Software Multisim, and then it
carries on the operation simulation to the circuit. The
waveform after running the simulation is shown in Fig.9.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

705

Study on the Effect of Physical Fitness Training on Children’s Cognitive Ability
Jianhua Deng*, Lei Ning

Sports Training Speciality, Yanshan University, China
E-mail: * 245615583@qq.com
www.tust.edu.cn

Abstract
With the word physical fitness getting more and more attention in our country, physical fitness has been widely
developed in our country. Therefore, in recent years, the students' physique has received attention from all aspectsof society, and improving the students' physique has also become a new topic for physical education workers. I
try to discuss the effect of physical fitness training on children's cognitive ability from the perspective of physicalfitness training. To provide theoretical basis and reference for intervention of children's cognitive ability training
methods. So as to promote the improvement of children's cognitive ability and provide guarantee. This paper
studies the influence of physical fitness training on children's cognitive ability by using the methods of literature,observation and experiment. The following conclusions are drawn: 1. Before the experiment, 8 children from
Qinhuangdao Aikushao Physical Fitness Center were tested for attention cognitive ability and memory cognitive
ability. Among them, 8 children were deficient in cognitive ability to varying degrees. Compared before and after
the experiment, physical fitness training has an effect on children's attention cognitive ability.3. Compared before
and after the experiment, physical fitness training has an effect on the improvement of children's memory cognitive
ability.
Keywords: Physical fitness of children; Children's cognitive ability; Experimental study

decline of young students seriously affects the healthy
growth of children and even the future of the country
and nation.
Children are in the golden development stage of life,
their cognitive ability will directly affect the level oflearning in the future. Therefore, according to the needsof children's own development and the requirements ofimproving children's basic sports ability, this study
makes an experimental study on whether children's
physical fitness can improve their cognitive ability[3].The word physical fitness comes from English
Physical Fitness, first appeared half a century ago by theAmerican Association for Health, Sports, and Leisure.
After many years of development, health fitness has
become an important standard to measure humanphysical health [4]. Scientific research and relatedscientific theoretical research on health fitness play an

1. Introduction
Children are the motherland's "bud and future ". As
early as 1988, at the 10th World Congress of the World
Future Research Federation, experts from various
countries discussed the results of the report on "the endof the children ". [1] It is generally believed that"children are the basic human resources and the creationof a world conducive to their development should berecognized as limited regional and global measures ".
However, due to the influence of the one-sided
pursuit of the transition rate, the society and the school
have the tendency to pay more attention to intellectual
education and light sports, the students' schoolwork
burden is too heavy, the rest and exercise time isseriously insufficient, and the students' interest in sports
gradually weakens or even disappears [2]. The physical© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

706

Jianhua Deng, Lei Ningability test table, the attention cognitive level of 8
experimental children in Aiku Children's physicalFitness Center of Qinhuangdao City was obtained in
Tab.2.

important role in improving human physical health andimproving human living standard [5].
Physical fitness exercises have a positive effect on the
improvement of children's health and physical fitnesslevel. These interventions aim to increase students'
physical fitness level. However, most of the current
studies on students' physical fitness are carried out
outside the classroom, and there are few studies linking
children's physical fitness with cognitive ability.

Tab.2. The attention cognitive ability before the experiment

2. Object and Methods of Study

In Qinhuangdao Aiku Children's physical Fitness Center,
8 children aged 7-8 were randomly selected for theexperiment. All the children participating in theexperiment needed two months of targeted training to
improve their cognitive ability. In the later stage of thisexperiment, the analysis of physical fitness training onchildren's attention cognitive ability and children's
memory cognitive ability was analyzed. Before theexperiment, in order to ensure the contrast of theexperiment, according to the parents' attention ability
test table and the children's memory ability test table as
is shown in the Tab.1, the following table was obtained,
and the attention ability and memory ability of 7-8 years
old children were analyzed.
D

E

F

G

H

Attention

18

12

Memory

23

30

10

7

13

10

15

14

45

65

58

40

49

52

0

0%

mild
medium

4
3

50%
37.5%

severe1

12.5%

Tab.3.The Content Stage of Attention Cognitive Training
Phase

Foundation

Improvement

StrengtheningTime

1-3 weeks

4-6 weeks

7-8 weeksWarm upWarm upWarm upStretchStretchStretchTraining

Tab.1. Cognitive status of attention and memory
C

Well noted

By understanding the training methods of expert
physical fitness and the knowledge of physical fitness I
have learned, the training contents of children's attention
cognitive ability are screened out. According to thestage principle of children's physical fitness training, thetraining will last for 2 months
Practice is arranged in three stages of training in anadvanced manner to prepare for the accuracy of theexperiment. The experiment is shown in the Tab.3

3.1 Analysis of Children's Attention Cognitive
Ability before Physical Fitness Training
Intervention

B

Percentage

3.2 Interventional training arrangements for
improving children's attention awareness3. Findings and analysis

A

NumberThrough the analysis of the attention cognitive abilityof 7-8-year-old children in Qinhuangdao Children's
Physical Fitness Center, You know, There are 0 children
with good attention skills, 0% of the total; There are
four children with a slight lack of attention, 50 per centof the total population; There are three children withmoderate capacity deficits, 37.5 per cent of the total;
There is one child with a severe lack of attention, 12.5%of the total number.

In this study, 7-8-year-old children in Qinhuangdao
Aiku physical Fitness Center were selected to conduct
physical fitness experiments. Eight 7-8-year-old
children in Qinhuangdao Aiku physical Fitness Center
were tested for attention cognitive ability and memory
cognitive ability before the experiment. After 2 monthsof targeted physical fitness training, attention cognitive
ability and memory cognitive ability of 8 7-8-year-old
children in Qinhuangdao Aiku physical Fitness Center
were tested again.

Name

Name

Numberof groups

Dumu Bridge

PorterCanoe Bridge

S Run

Color identification

S running colors

Relax

Relax

Relax

6-8 groups

6-8 groups

6-8 groups

According to the scoring rules of children's attention© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

707

Study on the Effect

3.3 Analysis of Children's Attention Cognitive
Ability after Physical Fitness Training
Intervention
After two months of experiments, An analysis ofattention cognitive ability of 8 7-8 year old children in
Qinhuangdao Aikukuo Physical Fitness Center, You
know, One child with good attention skills, 12.5 percent of the total; There are five children with a slightlack of attention, 62.5 per cent of the total; There are 2
children with moderate capacity deficits, 25 per cent of
the total population; There are 0 children with severe
disability, It accounts for 0% of the total as shown in
Tab.4.

Fig. 1.Comparison of Attention Cognitive Ability of Children
before and after the Experiment

4. Conclusions

Tab.4. The attention cognitive ability after the experiment
Number of
persons

Percentage

Well
noted

1

12.50%

mild

5

62.50%

medium2

25%

severe00%Through the comparison of 7-8-year-old children in
Qinhuangdao Aiku physical ability Center for 2 months, thephysical fitness training is helpful to the improvement ofchildren's attention cognitive ability.

References
1. Seeger JP; Thalain DH; Noor dam Physical fitnesstraining improves physical fitness and vascular function
in children with type 1diabetes. Diabetes, Obesity &Metabolism. 201.
2. Elbasan, Effects of chest physiotherapy and aerobic
exercise training on physical fitness in young children
with cystic fibrosis. Italian Journal of Pediatrics, 2012,38.
3. Yang Taiji .On the concept and relationship between
physical fitness and physical fitness. Contemporary
sports science and technology, 2018, 8(16): 178-179.
4. Zhu JUANJUAN. Research on the construction ofphysical education curriculum for large class children
from the perspective of physical fitness. Wuhan Instituteof physical education, 2018.
5. Wang Zhen. Experimental research on the influence ofchildren's physical fitness training on 3-6-year-old
children's sensory integration ability. Wuhan sports
College, 2018.

3.4 Comparative Analysis of Children's Attention
Cognitive Ability
Through the comparison before and after the experimentas shown in Fig.1, 8 children with good memory ability
before and after the experiment in Qinhuangdao Aikuo
Children's physical ability Center had 1 person ,3children with mild memory deficiency and 6 people
after the experiment; There were 3 children withmoderate memory deficiency and 1 children with severe
memory deficiency. To sum up, after the experiment,
the whole children's memory cognitive ability levelcomparison experiment came to see, has improved.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

708

Effects of high heels on plantar stress in women
Jianhua Deng*, Lei Ning

Sports Training Speciality, Yanshan University, China
E-mail: * 245615583@qq.com
www.tust.edu.cn

Abstract
OBJECTIVE: to investigate the changes of Plantar pressure and gait in women wearing high-heeled shoes fordifferent years. METHODS: from March 2019 to April 2019, a total of 30 female faculty members in Yanshan
University were surveyed by questionnaires and plantar pressure was measured by static and dynamic tests.
RESULT: 1. The peak value of plantar pressure of the arch of foot increased with the increase of wearing time (P &Lt; 0.05). 2. The plantar pressure and impulse of the second metatarsal and the fourth metatarsal increased with theincrease of wearing time (P & Lt; 0.05). CONCLUSION: 1. Long-term wearing high-heeled shoes will cause footdiscomfort, easy to cause flat foot and Hallux Valgus Lesions; 2. The distribution of plantar pressure and thechange of arch shape and gait may be the main reasons for wearing high-heel shoes.
Keywords: high heels; women; questionnaire; plantar stress test

the medial and lateral areas of the heel gradually
decreased, while the peak pressure in the second
metatarsal and third metatarsal regions increased with
the increase of heel height [3]. Metatarsal pressure isdense, long-term heavy load stimulation is easy to cause
damage, so women should choose the appropriate heightof shoes.

1. Introduction
The human foot is composed of 26 bones, 33 joints anda reticular layered structure composed of 126 ligaments,
muscles and nerves. Its basic function is to bear weight,
buffer, absorb impact force, produce forward driving
force, and coordinate and maintain the balance of thehuman body. When the human body walks, the footbears the reaction of the ground to the human body, up
to 1.5 times its weight.
The study of the characteristics of plantar pressure
distribution can reflect the structure and function of gait,
plantar pressure and the movement of the whole body
posture during walking [1]. The maximum peak pressure
on the plantar was mainly concentrated on the medial
and lateral sides of the heel [2]. With the increase of heel
height, the position of the maximum peak pressurechanged, mainly concentrated in the second metatarsal,
the third metatarsal and the medial side of the heel.
With the increase of heel height, the peak pressure in

2. Research subjects and methods
The subjects were female workers and teachers ofYanshan University, and 30 were selected by random
sampling, of which 17 were administrative posts and 13
were teachers. Finally, 15 people each completed theplantar pressure test as required. All the subjects had noabnormal gait and medical history. By using bodycomposition analyzer and PODOTEK HD seriespressure plate, the body composition results are printed
and the plantar pressure distribution map is obtained [4].
Body composition analyzer was used to observe theheight, weight and BMI of human body, and the mean

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

709

Jianhua Deng, Lei Ning

3.2 Pressure in the foot region

peak force, plantar pressure, impulse and pressure ofeach area of plantar were observed by plantar pressure
plate tester. The characteristics of plantar pressure were
evaluated [5].
As shown in Fig.1, the observable sites 1 and 2 areheel (RF); 3,4 and 5 are arch (MF); The 5th metatarsal(MH5),4th metatarsal (MH4),3rd metatarsal (MH3),2nd
metatarsal (MH2) and 1st metatarsal (MH1); the sites
11,12,13,14,15 were followed by the fifth phalanges(T5), the fourth phalanges (T4), the third phalanges (T3),
the second phalanges (T2), and the first phalanges (T1).

After the completion of the test, the computer software
automatically generates the pressure distribution map ofeach area of the foot as shown in Fig.2.

Fig.2. Distribution of pressure in various regions of the footThe pressure peak data of each part are used to draw
the following statistical as shown in Tab.2 and thechange line diagram as shown in Fig.3.
Tab.2. Statistical tables of mean plantar pressure peak in leftfoot groups (x±s)

Fig.1. Plane planer<2-years

2-5 years

6-10
years

11-20 years

40.88±2.9553.16±10.36

56.96±31.
4756.49±10.04

9.58±0.43

14.79±5.43

22.06±8.0
319.23±15.52

MH1 43.07±6.78

51.69±20.89

40.39±12.4347.83±18.38

MH2 49.13±12.48

50.96±18.69

40.61±10.
1353.94±16.78

MH3 55.94±19.79

59.47±13.04

40.72±14.
9652.97±24

MH4 60.11±26.48

49.24±15.7

39.69±17.
4346.23±14.34

MH5 18.33±2.87

32.3±12.56

20.81±10.125.9±5.24

MFM 4.87±4.43

9.35±3.79

13.84±6.9815.64±13.08

MFL 15.13±16.71

29.38±18.44

21.17±14.
1944.69±14.29

RFM 85.16±41.3

78.8±28.99

95.87±30.69100.85±12.3
4

RFL 71.82±12.93

64.26±5.9672.18±8.
273.4±13.84

3. Research findings and analysis
T1

3.1 Physical fundamentals and differences by
group

T2-5

From Table 1, we can see that there is no significant
difference in height, weight and BMI in each group, andthere is a statistical difference in age group. Because age
differences are inherent differences, the impact on the
study can be excluded. The statistical table shows thateach group has the consistency of height, weight and
BMI value, which can exclude the influence of height,
weight and BMI on the test results of each group.
Tab.1. Basic information on groups (x±s)<2-year

2-5 years

6-10 years

11-20 years

Age

32.6±1.4

32.2±4.8

37.4±1.644.4±10.6

Height160.5±12.5158.9±5.4162.3±10.3

161.3±11.8

Weight54.3±8.754.4±4.761.8±9.2

61.6±12.3

BMI

19.6±3.0

20.0±2.1

21.3±4.5

20.9±5.1© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

710

Effects of high heelschanges significantly
High trends. In the period of 6-10 years of wearing,
the arch of the foot has an obvious trend of low level,
which is easy to cause the occurrence of flat foot.
Compared with lower heels, the second metatarsal
pressure of higher heels was significantly higher thanthat of lower heels. The second metatarsal pressure ofheel height in 5-7 cm compared with 3-5 cm wassignificantly higher than that of heel height in 3-5 cm.

Fig .3 Breakline diagram of plantar peak changes in left footgroups

3.3 Analysis of Pressure and Impulse Test Resultsin Subsole Regions

References
1. A study on biomechanical characteristics of gait andplantar pressure in young women with subgloss. Wearing
different heights of heel walking Shandong Institute ofphysical Education, 2014.
2. Wang Haijun, Dong Lishu, Chen Qinghe, Li Rongwei.
Effects of different heel height on plantar pressure
distribution in obese young women with natural walking
gait. Journal of Hebei normal University of Science andTechnology, 26(03): 63-67.
3. Branthwaite H, Chockalingam N, Grogan S, et al.
Footwear choices made by young women and theirpotential impact on foot health. Journal of Health
Psychology, 2013, 18(11): 1422-1431.
4. Morris PH, White J, Morrison ER, et al. High heels as
supernormal stimuli: How wearing high heels affects
judgements of female attractiveness. Evolution and
Human Behavior, 2013, 34(3): 176-181.
5. Menz HB, Morris ME. Footwear characteristics and footproblems in older people. Gerontology, 2005, 51(5):346-351.

The distribution of plantar pressure peak showed thatthe maximum stress position of plantar was medial heel
> lateral heel > second metatarsal > third metatarsal >lateral arch, first phalanx > fourth metatarsal and fifth
metatarsal medial arch >2-5 phalanges
4. Conclusions
Wearing high heels for a long time can lead to a variety
of foot discomfort, the main part of which is located in
the medial side of the anterior foot (60%) and the archof the foot (25%), the main type of discomfort is
intolerance to long walking and pain.
Wearing high heels for a long time can change thecharacteristics of plantar pressure distribution, and show
different trends under different wearing years.
Wearing high heels for a long time can cause changes in
the arch of the foot, and in the 2-5 year period, the arch

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

711

A Research on Intelligent Classification of Urban Trash Bins Based on Machine Learning
Longyu Gao *, Zilong Liu, Luqi Shen, Songyun Shi, Yongzheng Lv

Tianjin University of Science and Technology, China;
E-mail: * ggafcc@163.com
www.tust.edu.cn

Abstract
Aiming at the problems of inaccurate, insensitive and general performance of the current intelligent garbage sorting
bins, an intelligent garbage sorting bin based on machine vision is proposed. The trash can is mainly divided into
five modules: main control module, machine vision module, classification module, overflow reminder module, and
Wi-Fi Internet of Things module. The trash can uses convolutional neural networks to build an intelligent garbage
classification model and classification algorithm to achieve rapid and accurate garbage classification. This
experiment will be based on the identification of waste bottles, analyze the recognition characteristics of machine
vision, and then propose methods to improve the accuracy of recognition.
Keywords: machine vision, convolutional neural network, intelligent classification

1. Introduction

driving functions. Obviously, this kind of smart trash can
is mainly designed for family living, and many functions
are not applicable in public relations situations.
For example, once a pedestrian just passes by the trash
can and has no intention of throwing trash, the trash canwill identify errors, consume unnecessary energy, andgreatly increase costs. In addition, although there aresome smart sorting trash cans on the market that canachieve recyclable and non-recyclable sorting functions,the use of sensor technology results in low classification
accuracy and not widespread popularity.

At present, our country mainly relies on manual garbage
classification, that is, placing several trash cans withdifferent types of marks. Pedestrians need to judge which
type of garbage they are in, and then put them in thecorrect trash can. Or hand over the trash to the staff
nearby, and let them sort the trash and put it out.
Although manual classification has achieved a certain
effect, it is still not a long-term solution. This not only
consumes a lot of time, manpower and material resources,
but also loses the biggest advantage of the trash can itself,that is, convenience and time saving.
Therefore, the intelligent sorting trash can came into
being. Although smart trash cans have long been
proposed and designed, they are still not widely used inpublic places. The main reason is that the cost is too highand the garbage classification function is basically notavailable; and its main intelligent function is to use theinfrared ranging module to sense whether the pedestrianis close to the trash can, and once it is close, the lid of thetrash can is automatically opened for pedestrians to put
Garbage; In addition, it also has voice prompts and2. Overall design scheme of the system
This article discusses an intelligent garbage sorting bin
based on machine learning. The convolutional neuralnetwork is built through the artificial intelligence learning
architecture, the training set is established in the earlystage, the collected image data and image attribute tags
are input, and the recognition model is obtained throughmultiple convolution operations; during image
recognition, the garbage images collected by the camera
are uploaded, After simple preprocessing, the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

712

Longyu Gao, Zilong Liu, Luqi Shen, Songyun Shi, Yongzheng Lv

identification model determines the attributes andcategories of garbage 1.
That is, according to recognition methods such as
contour recognition, feature recognition, color
recognition, and material recognition, find the common
points between the two, and finally realize the judgment
and classification. Finally, the corresponding signal istransmitted back to the single-chip microcomputer, anddifferent steering gear angles are output according to thetype of the returned signal, and then the flip board and thepush rod are controlled to sort the garbage into the correctgroup. The network classification flowchart is shown in
Figure 1.

Fig.2. flow chart

Fig.1. Classification process

The machine learning technology is applied to thesystem, and self-learning is carried out through theunsupervised learning task of clustering in machinelearning. The specific work is: the system recognizes thecharacteristics of a large amount of garbage, and thenpresupposes. The unique characteristics of the four typesof garbage are divided into four categories, each of whichcollects a large amount of data and saves the data in a
database. When new garbage is encountered, the system
extracts the data and matches it with the data in the
database. Find out which type of data is the same orsimilar to the system, and classify it into one category. If
there is no matching data in the system, the system
automatically infers the new data from the existing data,
and the system judges this by itself. Which category the
new data belongs to, which makes the classification moreaccurate.
The main functions of the smart garbage sorting bin
outlined in this article are: taking pictures of garbage
thrown by pedestrians, intelligently sorting garbage,
sensing whether the garbage bin is full, etc. It is mainly
divided into five modules: main control module, machine
vision module, mechanical module, sensor overflow
module, Wi-Fi Internet of Things module. The main flow
chart of the system operation is shown in Figure 2, andthe structure design of the trash can is shown in Figure 3.

Fig.3. Structural design of trash can

In this experiment, a common beverage bottle is taken
as an example. The beverage bottle data set comes fromonline and real-life photos. There are 4000 pictures intotal, of which 60% of the training samples are 2400; thenumber of verification samples is 20%, which is 800;
testing The sample accounts for 20%, which is 800 sheets.
After completing the training of the garbage classification
model, the model is tested for its recognition function andclassification. According to the recognition results, thegarbage classification model is optimized, and thecollected garbage pictures are used as the input of themodel, and the test is performed 800 times. Comparing
the model output results with the real results, it is foundthat the recognition rate reaches 96%. It can be seen thatall bottles tested can be accurately identified, accurately
and reliably. Later, based on this classification model,
other garbage data training will be carried out.

Fig.4. Recognition accuracy© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

713

A Research on Intelligent

3. Design of hardware control system

the size of the color and grayscale value, and the images
are matched with the garbage training atlas pre-input tothe convolutional neural network. According to thecontour recognition, Recognition methods such as color
recognition and material recognition seek common
features between the two, and then identify the type of
garbage that is thrown in according to the image
algorithm written.
This module also adds the function of machinelearning. First, it compares the various characteristics of
the identified garbage with the existing characteristics ofthe system. If they are the same or similar, they areclassified into one category and the data is stored in thesystem at the same time. Enlarge the system’s database to
make the next recognition more accurate and faster.
Compared with the current smart trash can, if there isnothing in common with the data stored in the system, thesystem will have problems that cannot be recognized.
For the system designed in this article, due to theaddition of machine learning functions, when
encountering such problems, the system will analyze its
own existing data, and then self-judge the characteristicsof the new data collected, thereby Perform accurateclassification.

3.1. Main control module
The main control module is mainly composed of STM32
controller. It is the brain of the entire system. When thepressure sensor senses that there is garbage being
dropped, the STM32 controller controls the operation of
the machine vision module. After the image recognition
module recognizes the image category and feeds it backto the main controller, the STM controller will control themechanical module to turn the corresponding angle torealize the intelligent classification and placement ofgarbage. The STM32 controller is shown in Figure 5.

Fig.5. STM32F103ZET6

3.3. Mechanical module

3.2. Machine Vision Module

After the image recognition module recognizes the typeof garbage and feeds it back to the main control module,
the main control module will control the steering gear toturn the corresponding angle according to theprogrammed algorithm.
Control the forward and backward movement of thefront and rear push rods and the left and right flips of theleft and right flaps to realize the movement of garbage in
two dimensions. Finally realize the garbage classification
to the corresponding trash can. The sorting mechanicalstructure is shown in Figure 7.

The OpenMV module is installed at the garbage disposal
port, and it captures images of garbage on the disposal
platform by taking pictures 3. It is mainly composed of acamera. The camera adopts a USB drive-free 1080P
high-definition camera, which greatly improves therecognition resolution; at the same time, it also has a
160-degree wide angle, which greatly increases theshooting range; in addition, it also has a flash function toensure that it takes pictures in a dim environment.
OpenMV is shown in Figure 6.

Fig.6. OpenMV

The machine vision module mainly analyzes andprocesses the photos taken by the machine vision module.
The photos taken by the camera can be regarded as alarge two-dimensional array. The elements of the arrayare called pixels, and their values are called gray values.
The captured photos are converted into grayscale images,
binary images and true color RGB images according to

Fig. 7. Sorting mechanical structure© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

714

Longyu Gao, Zilong Liu, Luqi Shen, Songyun Shi, Yongzheng Lv

3.4. Sensor overflow module

weight sensing and ultrasonic ranging is used todetermine whether the trash can is saturated, which
greatly improves the accuracy of the judgment.
Compared with the existing smart garbage sorting bin,because of the use of machine learning functions, therecognition is more accurate and sensitive, and thepracticability is higher. The recovery rate of recyclable
garbage is greatly improved, and the loss of manpower,
material resources and financial resources is reduced. ,Has a certain use value. In general, the innovation of thesystem lies in simplifying the types of garbage
classification and adding machine learning functions toimprove the problem of garbage classification to thegreatest benefit.

This module is mainly used to detect whether the garbage
in the trash can is full. In order to make the induction
more accurate, it is divided into two small modules:
weighing module and ultrasonic distance measuring
module. The weighing module is used to sense whether
the weight of the trash can exceeds the theoretical weightvalue. The weight value will be set according to the
maximum allowable weight of different types of trashcans through investigation.
In order to prevent some special garbage thrown bypedestrians from being too heavy, causing the weight of
the garbage can to exceed the theoretical weight value,
but it is still not full, we also designed an ultrasonicranging module at the mouth of the inner wall of thegarbage can.
The working principle is that the ultrasonic transmitteremits ultrasonic waves in a certain direction, and starts
timing at the same time as the transmitting time. The
ultrasonic waves propagate in the air and return
immediately when encountering obstacles on the way.
The ultrasonic receiver stops timing immediately after
receiving the reflected waves. Record the time t, then use
the formula S=vt/2 to calculate the distance between theultrasonic source and the obstacle. When the garbage is
not full, the actual distance measured is the inner wall
diameter of the bucket; when the garbage is full, theactual distance measured is the distance from theultrasonic transmitter to the garbage, so as to determinethat the garbage can is full. The ultrasonic ranging sensor
is shown in Figure 8.

References
1. Golnabi, H., and A. Asadpour. Design and application ofindustrial machine vision systems. Robotics andComputer-Integrated Manufacturing, 2007, 23.6:
630-637.
2.Liu, Jianghai, Yadong Jiang. Design of intelligent trash canbe based on machine vision. 2020 International
Conference on Image, Video Processing and Artificial
Intelligence. Vol. 11584. International Society for Optics
and Photonics, 2020.
3.Vernon, David. Machine vision-Automated visualinspection and robot vision. STIA 92 (1991): 40499.

Fig.8. Ultrasonic distance sensor

Conclusion
Through the design of software and hardware, this paper
designs an intelligent sorting bin based on machine vision.
It mainly uses image recognition methods to classify
garbage. In addition, the dual combination method of© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

715

A design on intelligent public trash can based on machine vision and auxiliary sensors
Longyu Gao*, Leixin Han, Jiangyu Wu, Mingfei Liu, Ruming Kang

Tianjin University of Science and Technology, China;
E-mail: * ggafcc@163.com
www.tust.edu.cn

Abstract
In order to improve the correct rate of front-end recognition in the garbage classification process, the automatic
garbage classification system designed based on machine vision technology has a significant improvement inrecognition accuracy compared to traditional smart garbage cans. But in the case of identifying irregular garbage,
the recognition accuracy is greatly reduced. In order to solve this kind of problem, four types of auxiliary sensors
are added to the trash can, through the mutual cooperation between the sensors, combined with the results ofmachine vision recognition, comprehensive judgment, greatly improving the recognition accuracy of irregular
garbage.
Keywords: machine vision, auxiliary sensors, garbage sorting

1. Introduction

identification and classification. Some smart trash canshave introduced machine vision and machine learning
related technologies. However, due to the wide variety of
garbage and the same type of style and Abundant shapechanges make recognition difficult, and current methods
cannot meet the needs of daily intelligent recognition andclassification 1. Although some related studies canachieve a stable distinction between recyclable andnon-recyclable garbage, they have not yet been able tomeet China's need to distinguish four types of garbage.
In order to meet China's garbage classification
requirements and improve the accuracy of front-end
recognition in the garbage classification process, thisdesign adds four types of sensors to the trash can, throughthe mutual cooperation between the sensors, combinedwith the results of machine vision recognition.
Recognition and judgment have successfully achieved asubstantial increase in the recognition accuracy ofirregular garbage.

With the rapid development of society, the level ofconsumption and production has greatly improved. At thesame time, waste production and environmental
degradation have also increased. According to statistics,
the world produces approximately 10 billion tons ofwaste each year. 400 million tons, if garbage cannot beclassified correctly, the land resources occupied bylong-term stacking will exceed 25 million square meters,
so the task of garbage classification is very difficult.
Garbage sorting can be divided into two parts:
front-end sorting collection and back-end sorting andrecycling. At present, the world’s main front-end methodof garbage classification still relies on manual work, andmainly relies on citizens’ consciousness and staff review.
This method is very inefficient, difficult to solve theproblem of large amounts of untreated garbage, andcannot guarantee the correctness rate. Therefore, in thepast two years, the concept of a smart trash can was
proposed.
The current smart trash cans are mostly based on
infrared sensors, capacitive sensors or LCD screens for© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

716

Longyu Gao, Leixin Han, Jiangyu Wu, Mingfei Liu, Ruming Kang

2. Overall design scheme of the system
The intelligent waste classification system is mainlycomposed of machine vision module, auxiliary sensor
module and mechanical module that realizes automaticclassification control. When the garbage is put into thesystem, the machine vision module starts the camera toobtain the garbage image, and then runs the convolutional
neural network to obtain the garbage category
information. For garbage that cannot be identified,auxiliary sensors are used for identification. Finally,
according to the classification of the garbage type, themechanical structure is controlled to make the garbage
fall into the corresponding bin 2. In the auxiliary sensor
part, we have selected 4 auxiliary sensors for thecommonality of different types of garbage, namely VOC
sensors, metal sensors, capacitive proximity sensors, andhumidity sensors. The result of machine vision
recognition is shown in Figure 1.

Fig.2. VOC sensor

2.2. Metal sensor
The metal sensor is used to identify metals and realize thefunction of material identification. This system uses
LJ18A3-8-Z/BX type metal sensor. The sensor has theadvantagesoffast
response
speed,
stronganti-interference ability, waterproof and corrosion
resistance, and very stable performance. It is widely usedin automated assembly lines, intelligent hardware andsecurity systems. The operating voltage of the sensor is
6-36vdc, and the switch contains a high-frequency
oscillation circuit, a detection circuit, an amplifying
circuit, a de-transmitting circuit and an output circuit.
When power is supplied to the switch, the oscillator in
the high-frequency oscillation circuit generates analternating electromagnetic field on the detection surfaceof the switch. When a metal approaches the detection
surface of the switch, the eddy current inside the metal
absorbs the energy of the alternating electromagnetic
field in the oscillator, so that the oscillator weakens or
stops. The two states of the energy change of theoscillator are converted into a level signal by thedetection circuit, the level signal is amplified by the
amplifying circuit, and then the trigger circuit triggers theoutput transistor circuit to work to generate a switching
signal. Thereby detecting the presence or absence of
metal to achieve the purpose of detecting metal. The
metal sensor is shown in Figure 3.

Fig.1. Machine vision recognition results

2.1. VOC sensor
The detection of hazardous waste that cannot berecognized by machine vision is mainly realized by the
ZP16 digital VOC air quality sensor module. The air
quality module uses advanced chip thick film
semiconductor gas sensors. The gas sensors are resistantto formaldehyde, benzene, carbon monoxide, andammonia. , Hydrogen, alcohol, cigarette smoke, flavors
and other organic volatile gases have extremely highsensitivity. After aging, debugging, calibration andcalibration, the module has good consistency andextremely high sensitivity. Extremely high sensitivity,
excellent long-term stability, factory calibration,
convenient and quick to use, with sensor fault
self-diagnosis function, low power consumption and longlife. The VOC sensor is shown in Figure 2.

Fig.3. Metal sensor© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

717

A design on intelligent

2.3. Capacitive proximity sensor

3. Auxiliary sensor recognition

The second part of the material identification is mainly
realized by the capacitive proximity switch M8M12. It
can detect any dielectric substance, including conductors,
semiconductors, insulators, and can even be used todetect liquid level and powdered materials. For
non-metallic substances, the movement distance depends
on the dielectric constant of the material. The greater the
dielectric constant of the material, the farther the sensing
distance is. Cooperating with motor control, changing the
distance between it and the tray can realize theclassification of different materials. The capacitive
proximity sensor is shown in Figure 4.

First, the VOC sensor detects the harmful gas producedby the corrupted hazardous garbage, and if it is detected,
the garbage is marked as hazardous garbage. If noharmful gas is detected, the metal sensor is used to realize
the function of material identification and distinguishbetween recyclable metals and others. If the metal is notrecyclable, the capacitive proximity sensor is used todetermine whether it is paper or plastic, and the paper andplastic are marked as recyclable. Finally, the humidity
sensor is used to distinguish food waste from otherwastes based on the characteristics of food waste
containing more water, and mark them. The auxiliary
sensor recognition process is shown in Figure 6.

Fig.4. Capacitive proximity sensor

2.4. Humidity Sensor
Use SHT3x temperature and humidity sensor module tocommunicate with STM32, and turn it on when machine
vision can't recognize it, so as to realize the identificationof wet garbage. The module applies digital module
collection technology and temperature and humidity
sensing technology, and the product has high reliability
and long-term stability. Built-in humidity andtemperature sensor elements, analog to digital converters,
signal processing, calibration data and I2C host interface.
The technology of using low-k polymer dielectric forhumidity sensing creates a low-power, monolithic MoS
sensor module with low deviation and hysteresis andlong-term stability. It is suitable for measuring humidity,
dew point and temperature. The module is small in size
and the accuracy can reach ±0.4°C. It can be embedded inmany highly integrated environments. With millisecond
measurement conversion time, there is no need to wait
between starting the measurement and reading the data.
The 2.54mm spacing interface is convenient for
integration testing. The humidity sensor is shown in
Figure 5.

Fig.6. Auxiliary sensor recognition flowchart
In the early stage, a large amount of data training iscarried out on the machine recognition system, so that it
can independently realize the correctness of a certain
amount of garbage and recommend the database based on
this, and then use the visual camera to capture andrecognize the image of the discharged garbage. Compare
and analyze with the data in the database to determine thetype of garbage, and then use its own mechanicalstructure to sort and put it into the corresponding garbage
bin 3.
If there is garbage that the garbage can cannot identify,
the four auxiliary sensors are used to determine the typeof garbage in real time, and the garbage is placed in thebuffer area of the garbage can for manual processing to
avoid misplacement. Mark the judgment result, take aphoto and upload it, send it to the staff for manual review,and finally upload the result to the database to serve thenext identification and judgment. When faced with thisFig.5. Humidity Sensor© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

718

Longyu Gao, Leixin Han, Jiangyu Wu, Mingfei Liu, Ruming Kang

problem of global waste sorting and treatment. Therefore,
this design has excellent application prospects.

kind of garbage next time, the trash can will be able todirectly identify and classify through machine vision.
As the database shares the data between all smart trashcans online, the learning speed of smart trash cans will
increase exponentially. The overall flow chart is shown in
Figure 7.

Conclusion
Aiming at the problem of insufficient accuracy of smart
garbage classification, this paper proposes the use of fourtypes of sensors to assist machine vision and machinelearning recognition to improve the recognition accuracy
and the correct rate of intelligent classification. Realize
the classification and release of four kinds of garbage
with high accuracy, greatly improve the accuracy of
garbage identification, reach the standard of actual use,save a lot of waste of manpower and natural resources,
and the design has the advantages of low cost and easy
iteration. , Provides a new reference for the front-end
collection method of garbage classification.
References

Fig.7. Overall flow chart

1.

4. Prospect analysis
2.

Through the analysis of various existing smart trash cans,the problem of insufficient machine vision recognition
accuracy in the front-end collection part of the garbage
classification process and cannot meet the needs of daily
intelligent recognition and classification is summarized
and studied. This article adopts four types of sensors. The
organic combination of “new garbage” classified byauxiliary sensors is marked and then manually reviewed
and entered into the database, which greatly improves theaccuracy of garbage identification.
At the same time, it combines the needs of Chinese
garbage classification, and expands the scope ofclassification to four types: recyclable, hazardous, foodwaste and other garbage. This classification method isbased on whether it is degradable, harmful to the human
body, and reusable. It has universal applicability to thegarbage classification and treatment methods of various
countries. Therefore, the design has been adjusted tosome extent and can be applied to the global garbage
classification.
In addition, placing the "new garbage" mark classified
by the auxiliary sensor into the buffer area for manual
processing ensures the accuracy of the classified garbage
and ensures that the back-end classification and recycling
can be carried out directly after the front-end collection
part is completed. This saves nearly 80% of humanresources, and greatly improves the efficiency of wastesorting and treatment, reducing land resources wasted due
to long-term garbage stacking, and alleviating the3.

Golnabi, H., and A. Asadpour. Design and application ofindustrial machine vision systems. Robotics andComputer-Integrated Manufacturing, 2007, 23.6:
630-637.
Liu, Jianghai, Yadong Jiang. Design of intelligent trash
can be based on machine vision. 2020 International
Conference on Image, Video Processing and Artificial
Intelligence. Vol. 11584. International Society for Optics
and Photonics, 2020.
Vernon, David. Machine vision-Automated visualinspection and robot vision. STIA 92 (1991): 40499.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

719

A Research on Front-End Garbage Classification Based on Machine Vision
Longyu Gao*, Zhiqing Xiao, Junlong Hao, Luqi Shen, Manqian Hu

Tianjin University of Science and Technology, China;
E-mail: * ggafcc@163.com
www.tust.edu.cn

Abstract
Adding a machine vision recognition module to the traditional smart trash can can effectively improve theefficiency of trash recognition. The intelligent garbage classification model constructed by the convolutional neuralnetwork can accurately identify the types of garbage, with an average accuracy rate of 0.87. Deploy the trainedmodel on openMV and test it on the produced physical trash can. After the system is stable, the average time tocomplete a sorting and recovery is 2s. Experiments show that the system can effectively identify the types ofgarbage and complete garbage classification and recycling.
Keywords: machine vision, intelligent classification, convolutional neural network

1. Introduction

carried out to make trash cans have functions such as
automatic classification, alarming and monitoring ofavailable capacity, but it has not yet been able to meetthe needs of automatic identification and classification.
With the rapid development of artificial intelligence
technology, machine vision technology has graduallymatured, and image classification methods based on
deep learning have gradually diversified. This provides
a new direction for the research on garbage recognitionin this article. Scholars have made a lot of efforts ingarbage recognition and image classification. Research
and achieved certain results. For example, using the
YOLOv4 network for garbage identification, a detection
accuracy of 94.8% for dry and wet garbage can beachieved, and an accuracy rate of over 90.0% can beachieved in a complex detection environment.
This paper proposes an automatic garbage
classification system based on machine vision. The
system can meet the classification requirements ofrecyclables, kitchen waste, other waste and hazardous
waste. The intelligent garbage classification model
constructed by the convolutional neural network canaccurately identify the types of garbage. Withmechanical structures such as the front and rear pushWith the rapid economic development and thecontinuous improvement of people's living standards,
the amount of garbage has increased sharply in the cycleof consumption and production. According to theresearch report, by 2050, the global waste volume will
increase by 70%, from 2.01 billion tons in 2016 to 3.4
billion tons in 2050. The task of sorting waste is very
difficult, and it is already very difficult to study aneffective method for sorting waste. urgent. Scholars athome and abroad have done a lot of analysis on waste
classification, but most of the proposals put forward areinnovations in terminal recycling methods. Front-end
collection still relies on people's consciousness, and theaccuracy of garbage classification is very low. Research
on an effective front-end collection method is of greatsignificance to the status quo of garbage classification.
For the front-end collection of garbage, smart trashcans have begun to appear in foreign markets, such as
Nastar in the United States, ccko (cckoing) in Germany,
Zarton and somatosensory smart trash cans in Japan,
which are mostly designed based on infrared sensors
and LCD screens. Domestic research has also been

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

720

Longyu Gao, Zhiqing Xiao, Junlong Hao, Luqi Shen, Manqian Hu

connected to the central processing module, and thelight sensor is used for sensing and recognizing lightchanges in the barrel; the light source module is used forproviding the light source for the machine vision
module.
The inner wall of the trash can is equipped with an
infrared sensor. When the trash is thrown in, theinfrared sensor emits and receives the infrared raysreflected by the object, which can detect the fillingheight of the trash.

rods and left and right flaps, the entire garbage
classification process can be completed.
2. Overall system design
Using Solidworks software to model the trash can, theoverall structure is shown in Figure 1. The trash can
includes a box body, the box body has a top plate, acamera is installed on the lower surface of the top plate,
and the box body is provided with a plurality of sorting
compartments for collecting different types of garbage.
The sorting compartments are arranged oppositely toform two Row 1; the upper middle of the sorting
compartment is provided with a garbage flipping platethat can be driven left and right to flip, the garbage
flipping plate is a trough-shaped structure that can
accept garbage, and the trough-shaped structure isprovided with the garbage to be accepted Push the pushplate toward the front. The structural design of the trash
can is shown in Figure 1.

2.1. Master chipThis text selects STM32f407zet6 as the core controllerof the hardware control system, which is mainly used torealize the functions of data acquisition and analysis andexecution of component motion control. The
STM32f407zet6 processor is an Arm Cortex-M4
high-performance core with a maximum operating
frequency of 168mHz and a package form of 1qfp144.
Its peripheral configuration is powerful, supporting
communication interfaces such as SP interface, I2C
interface, USB interface, USART interface, andperipherals such as ADC and timer. The development
board used is connected to OpenMV, and OpenMV
obtains garbage pictures, processes the obtained picture
information, and recognizes the types of them. Finally,
return the identified garbage type information to the
STM32f407zet6 development board 2. The development
board controls the movement of the steering gear torealize the movement of the front and rear push rods andthe left and right flaps, and then the garbage is dropped
into the correct box. STM32F103ZET6 microcontroller
is shown in Figure 3.

Fig.1. Structural design of trash can

Fig.2. Roof structure

A light sensor and a light source module are also
installed in the identification barrel, which are located in
the top cover part, which is shown in FIG. 2. To provide
a light source for machine vision at night, the lightsensor and the light source module are respectively

Fig.3. STM32F103ZET6© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

721

A Research on Front-End

2.2. OpenMV

2.4. Classification agency

The vision part uses OpenMV single-chip
microcomputer to communicate with STM32. The
OpenMV camera is a small, low-power, low-cost circuitboard that can be scripted by a high-level language
Python (to be precise, MicroPython). You can use an
external terminal to trigger shooting or execute analgorithm, or use the result of the algorithm to control
the IO pin 3.
OpenMV uses STM32H743II ARM Cortex M7
processor, 480 MHz, 1MB RAM, 2 MB flash. All I/O
pins output 3.3V and are 5V tolerant. The OpenMV
module is shown in Figure 4.

This intelligent sorting trash can realizes garbage
classification based on machine vision by using thecamera. Through the left-right flip of the garbage
dumping plate and the front and back movement of thepush plate, it can realize the movement of the garbage in
two dimensions, and then realize the movement of thegarbage position, and finally realize The correctclassification of garbage is sorted and placed in theclassification cabin below to complete the automaticclassification of garbage. The classification mechanism
module is shown in Figure 6.

Fig.6. Classification mechanism module
Fig.4. OpenMV module

2.5. Overflow reminder module
In order to facilitate the frequent overflow of garbage inthe trash bin, the system is designed to detect the full
bin. By installing an ultrasonic module just above eachsorting bin, periodically detecting the distance L1
between the garbage in the bin and the module, andmaking the difference with the distance L2 from themodule to the bottom of the sorting bin, you can get thecurrent height H of the garbage in the sorting bin, that is,
H = L2 -L1. If H exceeds 90% of the height of thesorting bin, the mark is that the sorting bin is full. Whenthe bin is full for processing, a reminder will be sent tothe corresponding APP through the WiFi module toremind relevant personnel to clean up the garbage in thebin as soon as possible. On the other hand, in order toavoid increasing the burden, the Raspberry Pi should
disable the recognition function, that is, put it aside forprocessing before the full bucket state is removed, andno longer run the neural network to identify the thrown
garbage until the full bucket state is removed.

2.3. DS5160 Servo
The smart trash can passes through the DS5160 servo.
The working voltage of the steering gear is 6V-8.4V,
and the controllable angle is 180°. The DS5160 actuator
works fast and has stable performance. Its control
method is PWM control, pulse width range: 500-2500us,control precision: 3us. The actuator can quickly respondto the motion instructions of the system and sortdifferent types of garbage into the corresponding bins.
The DS5160 servo is shown in Figure 5.

Fig.5. DS5160 Servo© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

722

Longyu Gao, Zhiqing Xiao, Junlong Hao, Luqi Shen, Manqian Hu

Conclusion
This article mainly studies the intelligent classificationof urban trash bins based on machine vision, uses
convolutional neural networks to build an intelligent
trash classification model, and then realizes real-timemonitoring of the filling status of smart trash bins
through the Internet of Things technology to achieve
intelligent control of trash bins .
The convolutional neural network is applied togarbage classification, the design scheme is simple toimplement, and the research results achieve practicalresults. At present, there are very few trash cans that areclassified in the process of centralized garbage
collection on the market. The innovation of this
implementation scheme lies in the combination of new
convolutional neural network technology, which cantransform ordinary trash cans into smart devices throughlow-cost embedded devices. Trash can to realize thefunction of identifying and automatically sorting
garbage.

Fig.4. Capacitive proximity sensor

2.6. Analysis of results
This verification experiment conducted experiments on
6 commonly used types of garbage. For each type of
garbage, 10 representatives with obvious differences inappearance were selected, and 100 throws were made atdifferent angles, sizes, and different degrees of damage.
The result data of mechanical structure classification
was recorded. The specific data records are shown in
Table 1.
It can be seen from Table 1 that this system has an
accuracy rate of over 87% for the identification ofcommon 6 types of garbage in actual use, and from theappearance model of the trash can in this experiment, it
can be seen that the size of the trash can is small andmeets the expected design requirements. It is also
possible to increase the types of garbage identification
by expanding the training set. The trash can realizes thetwo-dimensional movement of trash through the
movement of the front and rear push rods and the left
and right flaps, thereby realizing the correctclassification of trash. The average time for eachclassification is about 2 seconds, which meets theexperimental expectations. The system identification
data table is shown in Table 1.

References
1.Golnabi, H., A. Asadpour. Design and application ofindustrial machine vision systems. Robotics andComputer-Integrated Manufacturing, 2007, 23(6): pp.
630-637.
2.Liu, Jianghai, Yadong Jiang. Design of intelligent trash
can be based on machine vision. 2020 International
Conference on Image, Video Processing and Artificial
Intelligence, Vol. 11584. International Society for Optics
and Photonics, 2020.
3.Vernon, David. Machine vision-Automated visualinspection and robot vision. STIA 92 (1991): 40499.

Table 1. System identification data table
Type of garbage

Canplastic bottle
Milk carton
Paper cup
Paper ballbattery

Recognition correct
times/time

87
92
90
94
93
95

Average recognition rate

87%
92%
90%
94%
93%
95%

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

723

NeuroPhyllotaxis: An Interactive Application for Generative Art Based on EEG Data
Chien-Tung Lin, R.P.C Janaka Rajapakse

Graduate Institute of Studies in Documentary & Film Archiving, Graduate Institute of Animation and Film Art, Tainan
National University of the Arts, No. 66, Daqi, Guantian Dist., Tainan City 72045, Taiwan
Yoshimasa Tokuyama

Department of Media and Image Technology, Tokyo Polytechnic University, 1583, Iiyama, Kanagawa 243-0297, Japan
E-mail: 109130101@tnnua.edu.tw, janakaraja@gmail.com, tokuyama@mega.t-kougei.ac.jp
www.tnnua.edu.tw and ‡www.t-kougei.a.cjp

Abstract
Generative art is produced by procedural techniques. It has obtained a lot of attention since the beginning of computer
graphics. Many works of art are inspired by nature, among which phyllotaxis is as well. It is a combination ofmathematics and the beauty of nature. Not only can it be seen everywhere in nature, but also often appear in manmade objects, becoming part of culture or religion. This paper presents the development of an interactive generativeart application that is created from a phyllotaxis pattern by using the user's EEG data. When people are using it, it
will allow them to more easily relax and achieve the function of art therapy. We tried to use EEG data to make aninteractive installation art which creates phyllotaxis patterns that are projected on the wall. Everyone has a different
state, the generated patterns are also different from person to person, which creates interesting interactive contents.
In addition, sound can also be changed by EEG to become dynamic and real-time contents.
Keywords: EEG, phyllotaxis, generative art, interactive art, installation art

brain waves as an interactive input method to bring a newperspective.
In addition, installation art is becoming more andmore popular as a form of modern art. Compared withother art forms, installation art allows people to be more
involved, use touch, and even play in it. Therefore,
interactive methods are also added to allow people toenjoy, participate in, communicate, and interact with art,
to become interactive installation art. However, most of
the art content is fixed, arranged according to the creator.
So there is not much change, resulting in people notwanting to repeat, come, and participate.
Generative art is produced by computer programs. It
has the characteristics of non-repetition and endlessevolution. It has received a lot of attention since thebeginning of computer graphics. Many works of art are

1. Introduction
With the development of microcomputers and single
chips, the way people interact with computers haschanged. It has come a long way from the early keyboard
and mouse to the development of the multi-touch.
Developers always want to find better ways to let peopleinteract with apps in a more natural way. Such as gestures(Leap Motion1), wii2, voice input, etc. However, these are
all motor functions, that is, different people using thesame operation method will get the same result.
Compared with electroencephalogram (EEG), because
people can't see how the user moves, it just exists in thebrain's operation, so no one can copy the actions ofanother person and produce a personal style. Therefore,
we aim at the state of concentration and meditation in the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

724

Chien-Tung Lin, R.P.C Janaka Rajapakse, Yoshimasa Tokuyama

3. Development and application

inspired by nature, among which phyllotaxis is included.
It is a combination of mathematics and the beauty ofnature. Not only can it be seen everywhere in nature, butalso often appear in man-made objects, becoming part ofculture or religion. For example, Madala Kolam is not
only the expression of art, but also healing people's bodyand mind. And Chakra's representative totem is also like
phyllotaxis. When people are using it, it is easier to relax
the mood and achieve the function of art therapy.
Therefore, we try to use EEG to make interactive
installation art, and use EEG input to create phyllotaxis
patterns to project it on the wall. Because everyone has adifferent state, the graphics drawn are also different fromperson to person, creating interesting interactive works.
In addition, the sound can also be changed by EEG,
becoming a dynamic, real time content. More
importantly, when the user looks at the graphics
generated by his own brain, this can bring aboutcommunication with the heart and then achieve the effectof art therapy3.

3.1. system structure
The development environment is Intel I7 PC on windows
10 system, with the NVIDIA GeForce RTX 2080TI
graphics card. The programming language C# has beenused to develop the application on the Unity SDK. The
EEG headset is NeuroSky MindWave Mobile 2.
Figure 1 and Figure 2 show the EEG headset isconnected to the computer via Bluetooth, and uses theintermediary software ThinkGear Connector5 to send thereceived data to Unity6 via UDP network protocol.

Fig. 1. EEG connecting to Unity

2. Background: EEG Control art
2.1. The use of EEG in art
Since 1965, the use of EEG in artistic creation has been
documented. Most of the content has an establishedpattern. Authors or users use their brains or mental statesas artistic creations, such as focusing or relaxing tocontrol sound and images4.
2.2. Passive control and aesthetics

Fig. 2. System Structure

Another way to use EGG as artistic creation is morepassive. It uses the mental and emotional state of theauthor or participant to produce changes in sound andimages, especially music that affects the emotionalchanges of the user. The changing screen synchronizedwith the music4.
However, this application goes beyond this way of
expression, and focuses more on the communication
between the self and the heart. Through the displayed
graphics and sounds to perceive and explore one’s mental
state, open up internal communication and dialogue toachieve relaxation and meditation. This allows for selfreflection, and ultimately the opportunity to achieve thegoal of art therapy.

3.2. Content generation
The simple model for the florets of sunflower was Vogel‘s formula. As the book “The Algorithmic Beauty ofPlants”7 descript below:(1)• n is the ordering index number of a floret, from thecenter to outer.
• φ is the angle of the nth floret in a polar coordinate
system.
• r is the distance from the center of the capitulum to thenth floret.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

725

NeuroPhyllotaxis: An Interactive• c is a constant scaling.

Table 2. Use value of meditation for color.

In Figure 3, it can be found that changing different
angles will produce different pattern changes, so theangle is used as a variable in the program to generatechangeable patterns.

Range of Meditation
0-14
15-29
30-44
45-59
60-74
75-89
90-100Color
RED
ORANGEYELLOW
GREEN
BLUE
INDIGOVIOLET3.4. Installation
Figure 4 shows the installation for the application whichused a projector to project the generated graphics onto theneutral wall, allowing users to sit in front of the
projection screen and watch. This allows the user to relax
and wear the EEG headset.

Fig. 3. Generating phyllotactic patterns from the Vogel’s
formula. (a) α =137.3 ﾟ, (b) α =137.5 ﾟ, (c) α =137.6 ﾟ

3.3. EEG Data
Different waveform data can be obtained from the signal
from the brain waves, such as alpha wave, beta wave, etc.,
as well as data on concentration and meditation. We used
the level of concentration to generate graphics, and use
the level of meditation as a reference value for color andsound.
The degree of concentration can be divided into avalue of 0-100. We divide it into 7 ranges. The value ofconcentration is low, the angle of the graph is small; and,the higher the value is, the larger the angle as shown in
the Table 1.

Fig. 4. Installation for the application.

Table 1. Use value of attention for angle φ.
Range of Attention
0-14
15-29
30-44
45-59
60-74
75-89
90-1004. User feedback

Angle φ
41
59
73
97
109
137.5
157

A user study was conducted over 50 university students,
between the ages of 16 to 25. The experiment apparent
and the projector was used to project the patterns to theneutral wall. Each subject used the applications for 2 to 5
minims. After that, they were asked to take an online
survey. The questions included what they thought while
using the application and how they felt when used theapplications. Figure 5 shows some patterns made bydifferent users.

Similarly, the degree of meditation divides the value
from 0-100 into 7 equal ranges, which are represented by
7 colors of red, orange, yellow, green, blue, indigo, andviolet. Divided into 7 ranges, the number and color of the
chakras are the same8, and they also form the basic colorof white light. This correspondence is natural andconforms to the human body.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

726

Chien-Tung Lin, R.P.C Janaka Rajapakse, Yoshimasa Tokuyama

meaning of these graphics and whether they canunderstand their emotions or psychological state.
5. Conclusion
The use of EEG to generate phyllotaxis is a very novel
experience for most people. As an installation art, thegenerated graphics have different results according to theconditions of different people, making the installation art
show a variety of effects.
Through some applications, users can also recognize
their own state. Through different thinking patterns,
imagination, and memories, people were able to produce
graphics of different colors. Watching the visual andsound effects of these graphics can trigger psychologicalstates, such as novelty, such as a sense of excitement andcalmness, to achieve an increasingly relaxed state, andachieve the effect of meditation. It is like a dialogue andcommunication with the self, allowing the heart to settle,
reflect, meditate, and think, and find the inner peace andanswer.
After using this program, does it have a positive
impact on the user? Can it improve people’s
concentration? Does it allow people to be more relaxed?
Will it increase people’s creativity and work efficiency?
These can all become further research projects.

Fig. 5. Pattern generated by people.

Acknowledgements

Figure 6 shows when people used the application, 22/50
tried to think about different things to see the changes ingraphics, colors, and sounds. 4/50 tried using different
emotions to see different changes. 21/50 were in the stateof emptying meditation. 3/50 were thinking about themeaning of these graphics and how did such things make.

This research was supported in part by the Ministry of
Science and Technology of Taiwan under the grant (IV)MOST 109-2218-E-002-038-.
References
1. Leap Motion, https://developer.leapmotion.com.
Accessed 6 Dec 2020.
2. Wikipedia, Wii, https://en.wikipedia.org/wiki/Wii.
Accessed 6 Dec 2020.
3. Wikipedia, Art therapy,
https://en.wikipedia.org/wiki/Art_therapy. Accessed 7
Dec 2020.
4. M. Zhang, Y. Huang, S. U. Ahmed, M. Shidujaman,
Mapping Between Mind Cybernetics and Aesthetic
Structure in Real-Time EEG Art. DOI: 10.1007/978-3030-60117-1_26
5. NeuroSky, ThinkGear Connector,
http://developer.neurosky.com/docs/doku.php?id=thinkge
ar_connector_tgc. Accessed 7 Dec 2020.
6. Unity, https://unity.com. Accessed 7 Dec 2020.
7. P. Prusinkiewicz et al, The Algorithmic Beauty of Plants,
Regina, Canada, 1990, pp. 99-109
8. Healthline, chakras,
https://www.healthline.com/health/fitness-exercise/7chakras. Accessed 8 Dec 2020.

Fig. 6. How people thinking while using the device.

Among them, 42/50 found it interesting and novel. In
addition to the artistry brought about by the graphics andcolor changes, almost everyone wanted to know the© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

727

ThoughtMix: Interactive Watercolor Generation and Mixing Based on EEG Data
R.P.C. Janaka Rajapakse

Graduate Institute of Animation and Film Art, Tainan National University of the Arts, No. 66, Daci,
Guantian District, Tainan 72045, Taiwan
Yoshimasa Tokuyama

Department of Media and Image Technology, Tokyo Polytechnic University, 1583, Iiyama, Kanagawa 243-0297, Japan
E-mail: janakaraja@gmail.com, tokuyama@mega.t-kougei.ac.jp,*†
www.tnnua.edu.tw and †www.t-kougei.a.cjp

Abstract
In modern art, new types of installations and artworks use sensor-based inputs as an interactive method to create newforms of media art. Most of those installation artworks have static and pre-captured contents; they tend to be played
as dynamic, and by some concerned as interactive creations. However, most installations are not able to capture thevisitor's rich feedback which could trigger the contents. In order to identify the relation between the cyberneticaesthetics and visitor's brain wave activities, we propose an electroencephalogram (EEG) data-driven approach togenerate watercolor and mixing effects which visualizes the changes in brain wave activities in real-time. Our
proposed application transforms the brain waves into real-time watercolor effects that are displayed by generating
the data about emotion, attention, meditation, and neural mechanisms with EEG data. "ThoughtMix" createswatercolor effects as an immersive experience to influence creativity and art therapy by actually visualizing the colors
the user's mind creates.
Keywords: EEG art, watercolor, thought colors, digital colors, interactive mix, interactive generation

for art and technology that support these interactive
applications [5].
It is important that digital art technology moves
beyond the boundaries of the traditional interaction
techniques to provide a creative space that allows the user
to create and experience therapeutic sensations. The
acceptance of brain-computer interface (BCI) leads many
art researchers to explore the diversity of mind cybernetic
methods and expression forms in electroencephalogram(EEG) art [6]. This paper presents an EEG data-driven
approach to generate real-time watercolor and mixing
effects that visualize the changes in brain wave activities
in order to identify the relation between the cyberneticaesthetics and user’s brain wave activities. The proposed
application "ThoughtMix" transforms the brain waves
into real-time watercolor effects that are displayed bygenerating the data about emotion, attention, meditation,

1. Introduction
Currently, people living in a digitally connected era arein the space of reality and virtuality. Digital technology
is commonly used for human-computer interaction andcontent creations [1]. Interactive art is a method that
stimulates a variety of the user’s senses and activities
inside digitally created contents with a computer, so thatusers may interactively experience environments andspaces that can be difficult to create in traditional
techniques [2]. As computer graphics-based applied art
and technology becomes advanced and as various sensor
hardware interfaces are developed and spread, a variety
of applications that enable the user to interactively
encounter spatial and temporal experiences are being
created [3], [4]. Additionally, there is a growing demand© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

728

R.P.C. Janaka Rajapakse, Yoshimasa Tokuyama

introduced a classification of the perceptive interface
based on the level of self-containment. Wadeson et al.
[8] provided different views on creative control, which
are separated into four categories for participants: passive,
selective, direct, and collaborative. These classification
models described the spatial relationship between theexistences of participants and artwork, which determine
the behavioral characteristics of aesthetic construction. In
this paper, we focus on passive inputs which does not
require the participant to perform any particular task tochange or influence their brain activity explicitly.

and neural mechanisms with EEG data. This interactive
application creates watercolor effects as an immersive
experience to influence creativity and art therapy byactually visualizing the colors the user's mind creates.
The paper is organized as follows: section 2 presents
related work. Section 3 describes our development
framework. Section 4 is devoted to experimental results.
Section 5 discusses the main points presented throughout
the article, drawing conclusions on future research
directions.
2. Background2.2. Generative Art: Color and Mixing

There are several works in literature that cover interactive
art in various forms. Most papers feature contents andcomparison methods between different generativetechniques, providing performance results of theinteractive techniques, and often reports of users’
preferences. Unfortunately, due to the variety of different
art installations and different interfaces used forinteraction and control, it is still difficult to derive visitorgenerated contents. Therefore, further research isnecessary to understand which are the best options for a
specific immersive experience to influence creativity andtherapeutic sensations.

Computer-based generative art is a procedure to create
artforms by executing a set of rules in a methodical
collaboration between an artist and a machine [10].
Galanter [9] defined "Generative art refers to any art
practice where the artist uses a system, such as a set of
natural language rules, a computer program, a machine,or other procedural invention, which is set into motion
with some degree of autonomy contributing to or
resulting in a completed work of art". Shuhei Matsuyama[11] created an installation "dyebirth" that continuallycreates organic patterns through digitally controlled
physical phenomena produced by a mixture of water, ink,
and chemical substances.
Refik Anadol [12] presented an installation artwork
named "Melting Memories" based on measured EEG data,
which offered new insights into the representational
possibilities emerging from the intersection of advanced
technology and contemporary art. “Thinking in Color”
was an interactive experience that used an EEG
brainwave scanner to generate colors, influenced in realtime by the user’s brainwave activity [13]. To generatewatercolor effects, the proposed application in this paper,
named "ThoughtMix" also focused on real-time EEG
data and generative art techniques.

2.1. EEG in Art
When compared to traditional interactive feedback
techniques, using EEG as a biological control isconsidered relatively implicit. Mechanical operation andmotion detection are widely used in traditional
interactive art applications. In these traditional
applications, the explicit macro-communication
generated is based on the participant’s movements andbehaviors. However, EEG control directly uses
electrodes to acquire the activity of neurons and takes
thinking as a controller. EEG artwork is a new form ofcommunication that cannot be observed through body
behavior but the mind.
Prpa and Pasquier [6] defined a structured overview
of the expanding field of BCI art, with utilized
approaches and BCI devices, and proposed a systematicway of categorizing artworks based on their similarities
in the taxonomy. This presented taxonomy encompasses
sixty-one artworks that illustrate the nuances of thecategories in the taxonomy. According to the distance
between input and output control, Fishkin’s [7]3. Development Framework
In the proposed development framework, the EEG-based
interaction approach handles brainwave inputs without
additional equipment. In order to acquire EEG data,
MindWave Mobile 2 EEG headset [14] device waschosen because of its economic affordability, reliability
in the delivery of data related to the user's mental states,ease of use, and the comfort of the user feel wheninteracting with the device. Figure 1 shows how the EEG© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

729

ThoughtMix: Interactive Watercolor Generation

headset integrates with the TouchDesigner [15] to createa development environment. MindWave Mobile 2 EEG
headset was connected to the computer via Bluetooth.
Real-time EEG data acquisition, signal processing, andclassification were used in ThinkGear connector installedon the computer that sent the classification data througha user datagram protocol (UDP) over the local network
to the computer running TouchDesigner software which
generates contents.
3.1. Contents creation

Fig. 2. The Noise-TOP and the Displace-TOP can begenerated in TouchDesigner.

To generate watercolor effects, TouchDesigner was used
as a visual programming environment. The Noise texture
operation (TOP) generates a variety of noise patterns

Theta
Low AlphaHigh Alpha
Low Beta
High Beta
Low Gamma
High Gamma

including Perlin, simplex, sparse, alligator, Hermite, andrandom which often used to procedurally generateorganic surface textures and real-time effects. The Noise
texture operation and the displacement texture operation
nodes were used to create watercolor effects (Figure 2).
In TouchDesigner texture operations, Slope-TOP usedhorizontal luminance and vertical luminance for Red andBlue channels while Green channel was kept neutral.
Blur-TOP used as a filter with 5-pixel size, and LevelTOP used to quantization brightness and contrast.
3.2. Mapping colors
As literature described [6], the direct mapping technique
was used. The classified eight EEG band power values
are mapped to the colors as shown in Table 1. High Alphaand High Beta mapped into translate sampling
parameters of Y-axis and Z-axis in Noise TOP (Simplex
3D GPU) which produce real-time dynamic noise
patterns.

4.

Experimental Results

As a pilot experiment, the ThoughtMix application wastested with graduate students from Tainan National
University of the Arts. Figure 3 shows some examples ofthe participant's experiences and the effects created bytheir own brain waves, the left column shows dynamic
attention and the right column shows stable and enhanced
attention.

Table 1. EEG bands, their frequencies and colors
Frequency range
1-3 HzOrangeYellow
Green
Light BlueBlue
IndigoVioletAccording to the ThinkGear Serial Stream SDK, theacquired eSenses Attention meter value is reported on arelative eSense scale of 1 to 100 which can be categorized
into five different levels. However, this application used
an indirect mapping technique which categorized a value
between 0 to 40 at any given moment in time isconsidered “reduced”. A value between 41 to 100
indicates “increased” levels of the eSense. The two states
represent 0 and 1, which operates the reset parameter of
the Feedback texture operation (Feedback TOP [15])node. The Feedback TOP can be used to create a
feedback effect that gives motion blur without cleaning
the color buffer. When the "reduced" state, Feedback
TOP passes through the image connected to input colors
generated by EEG band values.

Fig. 1. Block Diagram of Brainwave Controlled

EEG bandDelta

4-7 Hz8-9 Hz10-12 Hz
13-17 Hz18-30 Hz
31-40 Hz41-50 Hz

Mapped Colors
Red© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

730

R.P.C. Janaka Rajapakse, Yoshimasa Tokuyama

5. Conclusion

References

The main objective of this application is to provide an
immersive experience to influence creativity and art
therapy by visualizing the watercolor effects the user's
mind creates. Despite the limited user study, the system
successfully generated and incorporated EEG data whichallows the user to identify the relation between the
cybernetic-aesthetics and brain wave activities.

1. S.U. Ahmed, A. Al Mahmud, K. Bergaust (2009)
Aesthetics in Human-Computer Interaction: Views andReviews. In: Jacko J.A. (eds) Human-Computer
Interaction. New Trends. HCI 2009. Lecture Notes in
Computer Science, vol 5610. Springer, Berlin, Heidelberg.
https://doi.org/10.1007/978-3-642-02574-7_63
2. E. Edmonds, G. Turner, L. Candy, Approaches tointeractive art systems. In: Proceedings of the 2nd
International Conference on Computer Graphics andInteractive Techniques in Australasia and South East Asia,
pp. 113–117, 2004.
3. Z. Wang, Theory of Digital Interactive Art Creation.
Shanghai University, Shanghai (2011)
4. J. Strehovec, New media art as research: art-making
beyond the autonomy of art and aesthetics. Technoetic
Arts, Vol. 6, No. 3, pp. 233–250, (2009).
https://doi.org/10.1386/tear.6.3.233_1
5. Suzanne Dikker, Matthias Oostrik, Peter Burr, Diederik
Schoorl, and Matthew Patterson Curry. 2014. Mutual
Wave Machine. Art. Retrieved May 3, 2019 fromhttps://mai.art/content/mutual-wave-machine
6. M. Prpa, P. Pasquier, Brain-Computer Interfaces inContemporary Art: A State of the Art and Taxonomy,
Brain Art. Springer, (2019), https://doi.org/10.1007/9783-030-14323-7_3
7. K. P. Fishkin, A taxonomy for and analysis of tangibleinterfaces. Personal and Ubiquitous Computing, Vol. 8,
No. 5, pp. 347–358 (2004)
8. A. Wadeson, A. Nijholt, C. S. Nam, Artistic braincomputer interfaces: state-of-the-art control mechanisms.
Brain-Comput. Interfaces Vol. 2(2–3), pp. 70–75, (2015).
9. P. Galanter, "What is generative art? Complexity theory as
a context for art theory." In GA2003–6th Generative Art
Conference. 2003
10. A. Dorin, J. McCabe, J. McCormack, G. Monro & M.
Whitelaw (2012) A framework for understanding
generative art, Digital Creativity, Vol. 23, No. 3-4, pp.
239-259, DOI: 10.1080/14626268.2012.709940
11. Shuhei Matsuyama, dyebirth: An installation work,https://nor.tokyo/dyebirth, Accessed on 10th Dec 2020.
12. Refik Anadol. 2018. Melting Memories. Installation Art,
http://refikanadol.com/works/melting-memories/
Accessed on 10th Dec 2020.
13. Thinking in Color, https://www.lbbonline.com/news/thisexperience-uses-brain-waves-to-create-colour, Accessedon 10th Dec 2020.
14. NeuroSky, ThinkGear Connector,
http://developer.neurosky.com/docs/doku.php?id=thinkge
ar_connector_tgc. Accessed on 10th Dec 2020.
15. TouchDesigner, https://derivative.ca/, Accessed on 10th
Dec 2020.

Fig. 3. Some examples of the participant's experiences andthe watercolor effects created by their own brain waves.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

731

HaptWarp: Soft Printable and Motion Sensible Game Controller
Lee Jen Tun

School of Knowledge Science, Japan Advanced Institute of Science and Technology,
1 Chome-1 Asahidai Nomi, Ishikawa 923-1211, Japan*
R.P.C Janaka Rajapakse†Graduate Institute of Animation and Film Art, Tainan National University of the Arts, No. 66, Daqi, Guantian Dist.,
Tainan City 72045, Taiwan †Kazunori Miyata

School of Knowledge Science, Japan Advanced Institute of Science and Technology,
1 Chome-1 Asahidai Nomi, Ishikawa 923-1211, Japan‡
E-mail: *bmpsst511@gmail.com, †janakaraja@gmail.com, ‡miyata@jaist.ac.jp

Abstract
Many interaction techniques have been introduced for controlling virtual reality contents and navigation. Traditional
controllers need to map between two-dimensional inputs and three-dimensional actions while interactive surfaces
enable more natural approaches, they still lack tactile feedback. We present real-virtual bend and twist, a tactile
feedback technology that delivers the twist and bends effect into a virtual environment. The method uses two inertial
measurement unit (IMU) sensors fixed on both sides on the 3D printed soft material bar to acquire object deformation
to simulate the virtual deformation based on the changes in the angle exerted on a physical device. People cancustomize the 3D printed connection shape to apply to different virtual scenarios.
Keywords: Interaction, Game Controller, interactive bend, interactive twist, intuitive interaction

1.

companies has allowed experimentation in control with
one important addition: inertia measurement unit (IMU).
Our work has been applied to add a sensor system to a
twistable and squeezable game controller through 3D
printing, with the goal of giving users an intuitive
operating experience. The software communicates withthese systems and formulates a novel interaction between
the player and the game.
Related work is presented in section 2. Section 3
explains the implementation of our system and theconclusion is stated in section 4 respectively.

Introduction

During the development of haptic game control systems,
careful consideration and devising are required to ensure
system stability. Usually, efforts are focused on hardwareand firmware improvement and as a result, an intuitive
control method is of little concern. VR hand controllers
are not dissimilar. Often a touchpad, trigger, or set ofarrow keys are added seemingly as an afterthought. Thismay be sufficient in controlling systems for most of thegames but for providing an intuitive experience andmaking it easier for users to understand how to interact
in three dimensional virtual worlds, they often perform
inadequately.
Over the past decade, the development of relatively
three-dimensional haptic controllers from several

2. Related Work
Research projects related to intuitive control and haptic
applications aim to create an immersive experience,
which can support the haptic feedback. We will not© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

732

Lee Jen Tun, R.P.C Janaka Rajapakse, Kazunori Miyata

extract and review the entire area, but instead, discussareas of the closest relevance to the work presented here.

integrate well with virtual reality. The aim of the researchin this paper was to develop an inexpensive intuitive
game controller that can simultaneously provide theintuitive manipulation for VR applications.

Spatial user interaction plays an important role in
Virtual Environments (VE). It allows the user tonavigate the entire virtual environment to find a target orinvestigate a place or a virtual object in the world. Inorder to implement spatial interaction, this will often
require 6 degrees of freedom (DOF). Inertial
measurement unit (IMU) sensors play an important role
to implement space manipulation [7]. Tseng et al [5]present the design of the EZ-Manipulator, a new 3D
manipulation interface using smartphones that supports
mobile, fast, and ambiguity-free interaction with 3D
objects. Oh et al [6] propose an interaction method
where users can conveniently manipulate a virtual object
with touch interaction recognized from the inertial
measurement unit (IMU) attached to the index finger’s
nail and head movements tracked by the IMU embeddedin the HMD.

3.

Implementation

The system structure is shown in Figure 1. In the physicalenvironment, people usually do most things by their handmotions. The controller digitalizes the hand motions
through wireless transmission protocol into a virtualenvironment. In the virtual environment, players are able
to interact with the virtual object in an intuitive way by
their daily hand motions.

To enhance the virtual environment, gamecontrollers are usually used in VR. Strasnick et al [3]present a novel design of linkage, electro-mechanicallyactuated physical connections capable of rendering
variable stiffness between two commodity handheld
virtual reality (VR) controllers. Cho et al [4] utilizes aspindle and wheel to present the interaction technique
where users can twist or roll a button ball within his/her
fingers simultaneously with other hand motions. For thehaptic feedback, Heo et al [8] uses a single 6-DOF force
sensor and a vibrotactile actuator to render grain
vibrations to simulate the vibrations produced during
object deformation based on the changes in force or
torque exerted on a device. Tokuyama et al [9] proposeda torque display method for free-form deformation(FFD) with a 6-DOF haptic device which combines
bending and twisting operations to extend the editing
capabilities of the constructed shape deformation
system by means of screw-motions. Tamaki et al [10]achieves this by placing the Photo-reflectors over
specific muscle groups on the forearm to detect hand
gestures as input. For output, electrode stimuli pads areplaced over the same muscles to control the user's hand
movements. Both sensors and electrode stimuli pads
target main muscle groups responsible for controllingthe hand.

Fig. 1. This is the system structure of the research.
Physical Environment include the daily hand motions.
Controller is equipped with a 3D printed soft bar, IMU
sensor and one MCU. Virtual Environment were the twocreated scenarios for the user to interact.
1.1. Intuitive Control
We explored how to integrate technology into the humanexperience by the development process of the gamecontroller. It is interesting when a device becomes sonatural that it is almost like a daily real activity in thevirtual environment. Simultaneously, there are many
advanced approaches in the user interfaces that aim to
make the game controller more natural and intuitive,
however, systems that achieve a truly integrated
experience where the interface becomes part of aperson’s experience of themselves remains rare. At theforefront of this integration of technology and humanity,
sensors and customized 3D printed components were
designed specifically to integrate with the gamecontrollers. A device like Tactical haptics [1] andDrag:on [2], transform the way a person can interact withthe virtual environment. A common thread emerges whenreviewing the game controller technologies thatsuccessfully integrate into the human experience, whichis that it must provide an intuitive experience. In our
project, it quickly became clear that it would be vital forour controller design to deliver a realistic and intuitive

While these approaches effectively provide specificspatial interaction and 3D object manipulation andinteraction with people between real-world and virtualenvironment, most researches need many expensive
peripherals or deliver the haptic feedback from outside
stimuli actuator which unafforded to the user and doesn’t© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

733

HaptWarp: Soft Printable

experience during gaming. As game controllers become
wireless and more related to realistic manipulation, it
becomes possible to embed them into a customized gameapplication.

newer data coming in will replace the previous data undera high frequency.
1.4. Virtual Content
When considering what application can be enhanced by
Haptwarp controller, VR games came to our minds first.
In this game, players can do daily activities from certain
scene objects with the Haptwarp controller as shown in
Figure 3 and Figure 4. While doing different naturalactivities, the player can perceive different tactile
sensations from the controller. For example, while
opening the door in a virtual environment, the user will
receive the visual and haptic feedback of feeling the force
from the 3D printed bar. When the user twists thecontroller, the screwdriver will lock the screw in thevirtual environment. By combining those actions, theplayer needs to explore the virtual world by movingaround himself, using the controller to trigger the game
event, receiving the new clue, and figuring out how toescape from the chamber.

1.2. Hardware Design
The development of the game controller utilized a rapid
prototyping methodology with various features trialled
and refined to produce the final design. In particular, theprototyping involved considerable experimentation withdifferent types of shape and mounting for the case andother electronic components. One of the early test
prototypes is shown in Figure 2. The controller was
printed by the 3D printer with soft material to provide
customized joints for a variety of game scenarios. The
IMU sensor was embed in the controller to deliver thebend and twist angle from the player to the Wemos D1mini which is like an Arduino interface but provides
WIFI with a small size all for a great price. Then Wemos
D1 mini through the UDP protocol transmits the data to
Unity virtual environment.

Fig. 3. Users can bend the controller to simulate theopen-door activity in virtual environment.

Fig. 2. This is prototype CAD of the HaptWarp
controller.
1.3. Wireless Data Transmission
Wireless Communication is the fastest growing and mostvibrant technological area especially integrated withgame controllers. Wireless Communication is a methodof transmitting information from one point to another,
without using any connection like wires, cables, or any
physical medium. Apart from mobility, wirelesscommunication also offers flexibility and ease of use. We
chose the UDP protocol since the situation in our
research is a real-time data transmission case when thecontroller is delivering data that can be lost because

Fig. 4. Users can twist the controller to simulate theactivity of locking a screw.
4.

Conclusion

In summary, this paper has performed an intuitive gamecontroller that can provide the user with the act ofinteracting and controlling virtual objects with sensors.
This system approaches the unique capability ofdelivering the human hand motions and haptic sensations
with wireless communication, as compared to© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

734

Lee Jen Tun, R.P.C Janaka Rajapakse, Kazunori Miyata

conventional techniques. Furthermore, to validate theefficacy of this approach, we developed two interactive
contents in the virtual environment for the user toexperience the intuitive manipulation with the HaptWarp
controller.
In our future work, we will optimize the controller.
For the 3D printed soft bar, we focus on enhancing theimmersive feeling by designing a variety of shapes oflinkage able to bring more daily hand motions that can beapplied to multiple events in the virtual environment. For
the sensors, we focus on setup variety bio-sensor toacquire user body information which will be applied into
virtual environments to activate different content toenhance the immersive experience. Finally, we willconduct further studies to investigate the complementary
effect and the tactile illusions in VR for both partial andfull body experiences.

3D
manipulationinterfaceusing
smartphones. Computational Visual Media 4, 139–147(2018). DOI: https://doi.org/10.1007/s41095-018-0105-0
6. Oh, Ju Y.; Park, Ji H.; Park, Jung-Min. 2019. "VirtualObject Manipulation by Combining Touch and HeadInteractions for Mobile Augmented Reality" Appl. Sci. 9,
no. 14: 2933.
7. Li Qian, Wu Hao, and Wang Peng. 2018. Manipulator
Trajectory Planning and Control Method Based on IMU.
In Proceedings of the 2018 International Conference on
Robotics, Control and Automation Engineering (RCAE2018). Association for Computing Machinery, New York,NY,
USA,
132–136.
DOI:
https://doi.org/10.1145/3303714.3303721
8. Seongkook Heo, Jaeyeon Lee, and Daniel Wigdor. 2019.
PseudoBend: Producing Haptic Illusions of Stretching,
Bending, and Twisting Using Grain Vibrations. In
Proceedings of the 32nd Annual ACM Symposium on User
Interface Software and Technology (UIST '19).
Association for Computing Machinery, New York, NY,USA,
803–813.
DOI:
https://doi.org/10.1145/3332165.3347941
9. Yoshimasa Tokuyama, R. P. C. Janaka Rajapakse, Yuji
Nakazawa and Kouichi. Konno, Torque display method
for free-form deformation with haptic device, In theProceedings of 2009 ICCAS-SICE, Fukuoka, 2009, pp.
3803-3808.
10. Emi Tamaki, Terence Chan, and Ken Iwasaki. 2016.
UnlimitedHand: Input and Output Hand Gestures withLess Calibration Time. In Proceedings of the 29th Annual
Symposium on User Interface Software and Technology(UIST '16 Adjunct). Association for Computing
Machinery, New York, NY, USA, 163–165. DOI:
https://doi.org/10.1145/2984751.2985743

1. Acknowledgements
The authors thank the Ministry of Science andTechnology for financially and technically supporting
this research in part under Grant MOST (IV) MOST 1092218-E-002-038- and the project from National Taiwan
University.
2. References
1. B. Lang, “Hands-on: Tactical haptics’ vive demo is further
proof that VR needs more than rumble,” 2015. [Online].
Available:https://web.archive.org/web/20160312220656/
http://www. roadtovr.com/tactical-haptics-htc-vive-demovirtual-realityfeedback-rumble/
2. André Zenner and Antonio Krüger. 2019. Drag:on: A
Virtual Reality Controller Providing Haptic Feedback
Based on Drag and Weight Shift. In Proceedings of the2019 CHI Conference on Human Factors in Computing
Systems (CHI '19). Association for Computing Machinery,
New York, NY, USA, Paper 211, 1–12. DOI:
https://doi.org/10.1145/3290605.3300441
3. Evan Strasnick, Christian Holz, Eyal Ofek, Mike Sinclair,and Hrvoje Benko. 2018. Haptic Links: Bimanual Haptics
for Virtual Reality Using Variable Stiffness Actuation. In
Proceedings of the 2018 CHI Conference on Human
Factors in Computing Systems(CHI '18). Association for
Computing Machinery, New York, NY, USA, Paper 644,
1–12. DOI: https://doi.org/10.1145/3173574.3174218
4. I. Cho and Z. Wartell, Evaluation of a bimanual
simultaneous 7DOF interaction technique in virtualenvironments, 2015 IEEE Symposium on 3D User
Interfaces (3DUI), Arles, 2015, pp. 133-136, doi:
10.1109/3DUI.2015.7131738
5. Tseng, PH., Hung, SH., Chiang, PY. et al. EZManipulator: Designing a mobile, fast, and ambiguity-free© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

735

Gray Level Co-Occurrence Matrix (GLCM) and Gabor Features Based No-Reference Image
Quality Assessment for Wood Images
Heshalini Rajagopal1,2, Norrima Mokhtar1, Anis Salwa Mohd Khairuddin1, Wan Khairunizam3,
Zuwairie Ibrahim4, Asrul Bin Adam4 and Wan Amirul Bin Wan Mohd Mahiyidin1

Department of Electrical Engineering, Faculty of Engineering, University of Malaya, Malaysia
Department of Electrical and Electronic Engineering, Manipal International University, Malaysia
3
School of Mechatronic Engineering, University of Malaysia Perlis, Malaysia
4
College of Engineering, University of Malaysia Pahang, Malaysia
1heshalini@gmail.com
1

2

Abstract
Image Quality Assessment (IQA) is an imperative element in improving the effectiveness of an automatic woodrecognition system. There is a need to develop a No-Reference-IQA (NR-IQA) system as a distortion free wood
images are impossible to be acquired in the dusty environment in timber factories. Therefore, a Gray Level CoOccurrence Matrix (GLCM) and Gabor features-based NR-IQA, GGNR-IQA algorithm is proposed to evaluate
the quality of wood images. The proposed GGNR-IQA algorithm is compared with a well-known NR-IQA,Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) and Full-Reference-IQA (FR-IQA) algorithms,
Structural Similarity Index (SSIM), Multiscale SSIM (MS-SSIM), Feature SIMilarity (FSIM), Information
Weighted SSIM (IW-SSIM) and Gradient Magnitude Similarity Deviation (GMSD). Results shows that the
GGNR-IQA algorithm outperforms the NR-IQA and FR-IQAs. The GGNR-IQA algorithm is beneficial in woodindustry as a distortion free reference image is not required to pre-process wood images.
Keywords: Wood images, GLCM, Gabor, GGNR-IQA, NR-IQA.
1

cause a checkerboard artefact to the wood images[7]. In
addition, the dusty and dark environment in timber factories
could degrade the quality of the image acquired[8]. Thus, anappropriate image quality assessment (IQA) algorithm isrequired to assess the acquired wood images prior to feeding
it to any automatic wood recognition system.

INTRODUCTION

Wood is extensively used for furniture, building
construction and paper production[1]. There are various types
of wood and each of them has different attributes with regard
to its formation, thickness, colour and texture[2]. Thesevarying characteristics defines their ideal usages andeconomic values[3]. As the price and characteristics of every
wood species differs, misclassification may cause financial
losses. Hence, there is a need to identify different woodspecies accurately.
Conventionally, the recognition of wood species isperformed manually by human subjects[4]. However, thispractice is time and cost consuming to the lumber industry.
Hence, several automatic wood species recognition systems
have been developed[1,2,5,6]. The efficiency of automatic
wood recognition systems can be improved by using
superior quality microscopy images which are commonly
enhanced to improve the rate of successful wood species
recognition. Nevertheless, the image enhancement
processes consume extra computational time, and could

IQA can be categorised into subjective and objective
evaluations. Subjective evaluation is the scores given byhuman subjects based on their judgment on the image
quality. While, objective evaluation is done based on
numerical methods to determine the quality of the images.
Even though, subjective evaluation is the benchmark of IQA,
it is impracticable in an industrial environment as it is timeand cost consuming. Therefore, it is necessary to develop an
objective evaluation procedures that is capable to imitate
subjective IQA evaluation[9].
Objective evaluation can be categorised into FullReference-IQA (FR-IQA), Reduced Reference-IQA (RRIQA) and No-Reference/Blind IQA (NR-IQA)[10,11]. FRIQA uses the reference image fully to assess the images© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

736

Heshalini Rajagopal, Norrima Mokhtar, Anis Salwa Mohd Khairuddin, Wan Khairunizam, Zuwairie Ibrahim, Asrul Bin Adam, Wan Amirul Bin Wan Mohd
Mahiyidin

whereas RR-IQA uses the reference images partially. In
contrast, NR-IQA assesses an image without using areference image. NR-IQA is the most appropriate algorithm
to evaluate the quality of the wood images as it may be
impossible to obtain high quality images in the dusty anddark setting of lumber factories. Therefore, we propose theGray Level Co-Occurrence Matrix (GLCM) and Gaborfeatures-based NR-IQA, GGNR-IQA algorithm to evaluate
wood images.
The GLCM and Gabor features are widely used in woodspecies recognition system[5,12–14]. The proposed GGNRIQA algorithm is compared with a commonly utilized NRIQA, Blind/Referenceless Image Spatial Quality Evaluator(BRISQUE), and FR-IQAs namely, Structural Similarity
Index (SSIM)[15], Multiscale SSIM (MS-SSIM) [15], Feature
Similarity (FSIM)[16], Information Weighted SSIM (IWSSIM)[17] and Gradient Magnitude Similarity Deviation(GMSD)[18]. The performances of the GGNR-IQA,
BRISQUE and FR-IQAs are evaluated by using the Pearson
Linear Correlation Coefficient (PLCC) and Root Mean
Squared Error (RMSE) computed between the human mean
opinion scores (MOS) and the algorithms.
2

efficiently[21].Nine modulations of Gaussian white noisewith standard deviation, 𝜎𝜎𝐺𝐺𝐺𝐺 and motion blur with standard
deviation, 𝜎𝜎𝑀𝑀𝑀𝑀 were added to the reference images, i.e.: 𝜎𝜎𝐺𝐺𝐺𝐺= 10, 20, 30, 40, 50, 60, 70, 80 and 90 for Gaussian white
noise and 𝜎𝜎𝑀𝑀𝑀𝑀 = 2, 4, 6, 8, 10, 12, 14, 16 and 18 for motion
blur.
Fig 1. Ten wood images used as reference images (a)Turraeanthus africanus, (b) Ochroma pyramidale, (c) Tilia
americana, (d) Cordia spp., (e) Juglans cinerea, (f)Vouacapoua americana, (g) Dipterocarpus spp., (h)Swartzia Cubensis, (i) Cordia spp., (j) Cornus florida

METHODS2.1.2 GLCM and Gabor Features

Training and Testing Database

First, Mean Subtracted Contrast Normalized (MSCN),
𝐼𝐼̂(𝑚𝑚, 𝑛𝑛) is calculated from the wood image, I(m,n) using Eq.
(1) [22]:
𝐼𝐼(𝑚𝑚,𝑛𝑛)−µ(𝑚𝑚,𝑛𝑛)𝐼𝐼̂(𝑚𝑚, 𝑛𝑛) =(1)

An SVR model is trained with the GLCM and Gaborfeatures calculated for normalized wood images with thehuman MOS which are obtained from the subjective
evaluation for wood images. The MOS, GLCM and Gaborfeatures are utilized as the training and testing database to
obtain an optimized SVR model𝜎𝜎(𝑚𝑚,𝑛𝑛)+1

where µ(m,n) and 𝜎𝜎(𝑚𝑚, 𝑛𝑛) denote the mean and variance ofwood image, I(m,n), respectively ,m ∈ 1,2, … , 𝑀𝑀, 𝑛𝑛 ∈1,2, … , 𝑁𝑁 are spatial indices while M represents the heightand N represents width of image, I(m,n).
The mean, µ(m,n) and variance, σ(m,n) of the wood image
are computed using Eq. (2) and Eq. (3), respectively [22]:𝐿𝐿(2)𝜇𝜇(𝑚𝑚, 𝑛𝑛) = ∑𝐾𝐾
𝑘𝑘=−𝐾𝐾 ∑𝑙𝑙=−𝐿𝐿 𝑤𝑤𝑘𝑘,𝑙𝑙 𝐼𝐼𝑘𝑘,𝑙𝑙 (𝑚𝑚, 𝑛𝑛)

2.1.1 Wood Images
Ten wood images from various wood genus, as shownin Fig. 1 were chosen. The images were acquired from awood database: https://www.wood-database.com/ [19]. The
ten reference images were distorted by Gaussian white noiseand motion blur. These two types of distortions usuallyoccur in the industrial setting. Generally, the wood images
are exposed to Gaussian white noise due to the poor
illumination and heat in the lumber mill while acquiring thewood images[8,20]. On the other hand, wood images are
exposed to motion blur when there is a relative motion
between the wood slice and camera[6].
These distortions degrade the quality of the wood
images where the features of the pores on the wood texture
may not be discerned. Hence, this may lead tomisclassification of the wood genus as the feature extractor
may not obtain distinctive features from the wood images

𝐿𝐿
𝜎𝜎(𝑚𝑚, 𝑛𝑛) = �∑𝐾𝐾
𝑘𝑘=−𝐾𝐾 ∑𝑙𝑙=−𝐿𝐿 𝑤𝑤𝑘𝑘,𝑙𝑙 �𝐼𝐼𝑘𝑘,𝑙𝑙 (𝑚𝑚, 𝑛𝑛) − 𝜇𝜇(𝑚𝑚, 𝑛𝑛)�2

(3)
where 𝑤𝑤 = �𝑤𝑤𝑘𝑘,𝑙𝑙 |𝑘𝑘 = −𝐾𝐾, … , 𝐾𝐾, 𝑙𝑙 = −𝐿𝐿, … , 𝐿𝐿� is a 2dimension (2D) circularly-symmetric Gaussian weighting
function that is sampled out to three standard deviations andrescaled to unit volume, and 𝐾𝐾 and 𝐿𝐿 represent the window
sizes.
The MSCN coefficients, 𝐼𝐼̂(𝑚𝑚, 𝑛𝑛) highlights the mainfeatures of the wood images such as pores and grains, withfew low-energy residual object boundaries[23]. Therefore,
the MSCN is used to compute the GLCM and Gabor featuresinstead of the image, I(m,n). Next, two types of features© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

737

Gray Level Co-Occurrence Matrix

namely, GLCM and Gabor features were incorporated inthis study.

The computational models of 2D Gabor filters aredefined in Eq. (12) and (13) [25]:ℎ𝑒𝑒 = 𝑔𝑔(𝑥𝑥, 𝑦𝑦)𝑐𝑐𝑐𝑐𝑐𝑐�2𝜋𝜋𝜋𝜋(𝑥𝑥 cos 𝜃𝜃 + 𝑦𝑦 sin 𝜃𝜃)�(12)ℎ𝑜𝑜 = 𝑔𝑔(𝑥𝑥, 𝑦𝑦)𝑠𝑠𝑠𝑠𝑠𝑠�2𝜋𝜋𝜋𝜋(𝑥𝑥 cos 𝜃𝜃 + 𝑦𝑦 sin 𝜃𝜃)�(13)
Where ℎ𝑒𝑒 and ℎ𝑜𝑜 represents the even symmetric and odd
symmetric Gabor filters, respectively and 𝑔𝑔(𝑥𝑥, 𝑦𝑦) represents
the isotropic Gaussian function and is computed as Eq. (14)[25]:

2.1.2.1 GLCM Features
The GLCM depicts second order statistical analysis of
an image by analyzing how often the pairs of pixels which
consist of specific values and spatial relationship take placein an image. The probability, 𝑝𝑝(𝑚𝑚, 𝑛𝑛) is computed using Eq.
(4) [24]:𝑝𝑝(𝑚𝑚, 𝑛𝑛) = {𝐶𝐶(𝑚𝑚, 𝑛𝑛)|(𝑑𝑑, 𝜃𝜃)}
(4)
where 𝑑𝑑 is the inter-pixels displacement distance,𝜃𝜃 denotes
orientation and 𝐶𝐶(𝑚𝑚, 𝑛𝑛) denotes the frequency of gray leveloccurrence in MSCN of the image, 𝐼𝐼̂(𝑚𝑚, 𝑛𝑛). Four statisticaltextures such as contrast, correlation, energy, andhomogeneity were extracted from the GLCM matrix.
Contrast calculates the local variations in the gray-levelco-occurrence matrix and is defined as Eq. (5) [24]:(5)Contrast = ∑𝑚𝑚,𝑛𝑛 |𝑚𝑚 − 𝑛𝑛|2 𝑝𝑝(𝑚𝑚, 𝑛𝑛)
Correlation computes the joint probability occurrence of thespecified pixel pairs and is defined as Eq. (6) [24]:(𝑚𝑚−𝜇𝜇𝜇𝜇)(𝑛𝑛−𝜇𝜇𝜇𝜇)𝑝𝑝(𝑚𝑚,𝑛𝑛)
Correlation = ∑𝑚𝑚,𝑛𝑛(6)𝑔𝑔(𝑥𝑥, 𝑦𝑦) =𝜎𝜎𝑚𝑚 𝜎𝜎𝑛𝑛

2.1.2.2 Gabor Features
The 2D Gabor function which represents the spatialsummation properties of simple cells in the visual cortex and
it is defined as Eq. (9) [25]:Where𝑥𝑥 ′ = 𝑥𝑥 𝑐𝑐𝑐𝑐𝑐𝑐 𝜃𝜃 + 𝑦𝑦 𝑠𝑠𝑠𝑠𝑠𝑠 𝜃𝜃𝑦𝑦 ′ = −𝑥𝑥 𝑠𝑠𝑠𝑠𝑠𝑠 𝜃𝜃 + 𝑦𝑦 𝑐𝑐𝑐𝑐𝑐𝑐 𝜃𝜃� 𝑐𝑐𝑐𝑐𝑐𝑐 �2𝜋𝜋

𝑥𝑥′
𝜆𝜆

2𝜎𝜎 2

�

(14)

The MOS values were obtained from subjective
evaluation participated by 10 students aged between 20-25
years from Manipal International University (MIU),
Malaysia. The evaluation was carried out as per theprocedures suggested in Rec. ITU-R BT.500-11[27] where it
was performed in an office environment using a 21-inch
LED computer screen.
Simultaneous Double Stimulus for Continuous
Evaluation (SDSCE) approach was used in this evaluation[27,28]where the reference and distorted images are shown
side-by-side on the computer screen and each subject
compares the quality of the images displayed on the rightside with its reference image (left side) to evaluate thedisplayed image.
The score given by the human subjects are either
Excellent (5), Good (4), Fair (3), Poor (2) or Bad (1) for each
image displayed. The evaluation process takes 15 to 20
minutes for each subject. The scores obtained from thesubjects were averaged to convert them to the MOS[29].
These MOS values are also used to train SVR.

1+|𝑚𝑚−𝑛𝑛|2𝜎𝜎 2𝑥𝑥′2 +𝑦𝑦′2

2.1.3 MOS

These 4 parameters were computed at four directions, 0°,45°, 90° and 135° and this form 16 GLCM features.

𝑥𝑥′2 +𝛾𝛾2 𝑦𝑦′2𝑒𝑒𝑒𝑒𝑒𝑒 �−

And the spatial frequency response of the Gabor functions,𝑓𝑓 is as shown in Eq. (15) [25]:(15)
𝑓𝑓 = 𝑁𝑁/𝑃𝑃
Where 𝑁𝑁 denotes the size of the kernel and 𝑃𝑃 denotes period
in pixel.
In this study, wavelength, 𝜆𝜆 is in increasing powers of
two starting from 4/√2 up to the hypotenuse length of the
input image[26] and this produces 7 Gabor features. The 7
Gabor features were then computed in four orientations, 0°,45°, 90° and 135°, similar to the GLCM computations. This
forms 28 features Gabor features. In total, the 16 GLCM and
28 Gabor features were combined and this forms 44 features.
These 44 features were calculated using the MSCN
coefficients, 𝐼𝐼̂(𝑚𝑚, 𝑛𝑛) and are used to train SVR.

Energy calculates the sum of squared components in the
GLCM. It is also known as uniformity or the angular second
moment. The energy parameter is computed as Eq. (7) [24]:(7)Energy = ∑𝑚𝑚,𝑛𝑛 𝑝𝑝(𝑚𝑚, 𝑛𝑛)2
Homogeneity calculates the closeness of the
distribution of elements in the GLCM to the GLCM
diagonal and is computed as Eq. (8) [24]:𝑝𝑝(𝑚𝑚,𝑛𝑛)
Homogeneity = ∑𝑚𝑚,𝑛𝑛(8)𝑔𝑔(𝑥𝑥, 𝑦𝑦; 𝜆𝜆, 𝜃𝜃, 𝜓𝜓, 𝜎𝜎, 𝛾𝛾) = 𝑒𝑒𝑒𝑒𝑒𝑒 �−1�2𝜋𝜋𝜎𝜎 2

+ 𝜓𝜓�
(9)
(10)
(11)

𝜆𝜆 denotes the wavelength of the sinusoidal factor, 𝜃𝜃 denotes
the orientation of the normal to the parallel stripes of a Gaborfunction, 𝜓𝜓 represents the phase offset, 𝜎𝜎 represents thestandard deviation of the Gaussian envelope and 𝛾𝛾
represents the spatial aspect ratio.

2.1.4 Support Vector Machine Regression (SVR)∈ −SVR [30] is trained using MOS and 44 GLCM and
Gabor features of wood images in this study. The 44 image© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

738

Heshalini Rajagopal, Norrima Mokhtar, Anis Salwa Mohd Khairuddin, Wan Khairunizam, Zuwairie Ibrahim, Asrul Bin Adam, Wan Amirul Bin Wan Mohd
Mahiyidinfeatures (GLCM and Gabor features) calculated for thewood images are mapped to the MOS values of thecorresponding wood images.
The 44 GLCM and Gabor features and MOS of wood
images were randomly split into training and testing sets
where 80% of the 44 features and MOS values were used tousing ten reference images acquired from ten various wood
genus as shown in Fig. 3.
Fig 3. Reference wood images in the second dataset(a) Julbernardia pellegriniana,, (b) Dalbergia cultrate, (c)Dalbergia retusa, (d) Dalbergia cearensis, (e) Guaiacum
officinale, (f) Swartzia spp., (g) Dalbergia spruceana, (h)Dalbergia sissoo, (i) Swartzia benthamiana and (j)Euxylophora paraensis
These reference images were added with the similar
distortion type (Gaussian white noise and motion blur) andmodulations as the training and testing database. This meansthat the second dataset includes 10 reference images and 180
distorted images.
Relationship Between MOS and Quality of
Image with Different Distortion Modulations
Fig. 4 (a) and (b) shows the correlation between MOS andnine distortion modulations of Gaussian white noise andmotion blur. Lower MOS values show lower image quality
which is caused by higher distortion modulation. On theother hand, higher MOS values represent higher image
quality which is generated by lower distortion modulation.
Based on Fig. 4 (a) and (b), the MOS decreases as the
distortion modulation increases. This means that all thehuman subjects could discern the images distorted with thevarious modulations of Gaussian white noise and motion
blur.

train the SVR model to obtain an SVR model with optimized
parameters and 20% were utilized to evaluate the optimized
SVR model.
The flow diagram of the proposed GGNR-IQA isshown in Fig. 2. The performance of GGNR-IQA wasevaluated using PLCC[31] and RMSE[32] calculated between
1000 iterations were performed on the training and testingof the SVR model to obtain an optimized SVR model. The
cost parameter, C, and width parameter, g, of the optimized
SVR model are 32768 and 0.125, respectively.
Fig 2. Flow Diagram of the Proposed GGNR-IQA
Performance Evaluation
The proposed GGNR-IQA is compared with a wellknown NR-IQA algorithm, BRISQUE and five FR-IQAs[21]: Structural Similarity Index (SSIM)[15], Multiscale SSIM
(MS-SSIM)[15], Feature Similarity (FSIM)[16], Information
Weighted SSIM (IW-SSIM)[17] and Gradient Magnitude
Similarity Deviation (GMSD)[18].
The performance of the GGNR-IQA, BRISQUE and
FR-IQAs is assessed using PLCC and RMSE[28] values
calculated between these algorithms and MOS.
3 RESULTS AND DISCUSSIONSThe efficiency of the GGNR-IQA was further assessedusing a second dataset which was generated from the samewood image database [19]. This second dataset was produced© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

739

Gray Level Co-Occurrence Matrix

Fig. 6. RMSE values between GGNR-IQA, BRISQUE,
FR-IQAs and MOS

Fig 4. Scatter Plot of MOS versus nine distortion
modulations of (a) Gaussian White Noise and (b) Motion
Blur

4 CONCLUSIONSA NR-IQA algorithm, GGNR-IQA was proposed toassess wood images prior to feeding the image to woodspecies classification and recognition system. The proposed
GGNR-IQA algorithm was trained using GLCM, Gaborfeatures and MOS obtained from wood images. The
performance of the GGNR-IQA algorithm was assessed bycomparing the PLCC and RMSE values calculated between
GGNR-IQA, BRISQUE, five FR-IQA algorithms and MOS.
PLCC and RMSE values showed that the GGNR-IQA
algorithm outperforms BRISQUE and all the five FR-IQAs.
This shows that the GGNR-IQA algorithm could assess
the quality of wood images accurately. In addition, theproposed GGNR-IQA algorithm would not require a
distortion free reference image to determine the quality of
the wood images. This is beneficial especially when it is
impossible to obtain a distortion free reference image in the
dusty environment of lumber mill.

Correlation between GGNR-IQA, BRISQUE
and FR-IQAs Algorithms and MOS
The PLCC and RMSE values calculated between
GGNR-IQA, BRISQUE, five FR-IQA algorithms:
Structural Similarity Index (SSIM), Multiscale SSIM (MSSSIM), Feature SIMilarity (FSIM), Information Weighted
SSIM (IW-SSIM), Gradient Magnitude Similarity
Deviation (GMSD) and MOS for Gaussian white noise,motion blur and overall images are shown in Fig. 5 and Fig.
6, respectively. The most suitable IQA for wood images is
expected to have the highest PLCC and lowest RMSE values.
Based on Fig. 5 and Fig. 6, the GGNR-IQA has thehighest PLCC and lowest RMSE values for Gaussian white
noise, motion blur and the overall database compared to the
BRISQUE and FR-IQAs. This shows that the GGNR-IQA
algorithm outperforms BRISQUE and all the five FR-IQAs.

REFERENCES[1] Shivashankar S. Automatic Wood Classification using a
Novel Color Texture Features. Int J Comput Appl.
2018;180(27):34-38.
[2] Barmpoutis P, Dimitropoulos K, Barboutis I, Grammalidis N.
Wood species recognition through multidimensional texture
analysis. Comput Electron Agric. 2018;144:241-248.
doi:10.1016/j.compag.2017.12.011[3] Zamri MIP, Cordova F, Khairuddin ASM, Mokhtar N, Yusof
R. Tree species classification based on image analysis using
Improved-Basic Gray Level Aura Matrix. Comput Electron
Agric. 2018; (124):227-233.
[4] doi:https://doi.org/10.1016/j.compag.2016.04.004[5] Gazo R, Wells L, Krs V, Benes B. Validation of automatedhardwood lumber grading system. Comput Electron Agric.
2018;(June):0-1. doi:10.1016/j.compag.2018.06.041[6] Venkatachalapathy K, Sudhakar P. An Intelligent
Recognition System for Identification of Wood Species. J
ComputSci.
2014;10(7):1231-1237.
doi:10.3844/jcssp.2014.1231.1237[7] Guang-sheng C, Peng Z. Sensors and Actuators A : PhysicalDynamic wood slice recognition using image blur information.
Sensors
Actuators
A
Phys.
2012;176:27-33.
doi:10.1016/j.sna.2011.12.056[8] Xiao B, Tang H, Jiang Y, Li W, Wang G. Brightness andcontrast controllable image enhancement based on histogram
specification.
Neurocomputing.
2018;275:2798-2809.
doi:10.1016/j. neucom. 2017.11.057[9] Ratnasinga J, Ioras F, Swan TT, Yoon CY, Thanasegar G.
Determinants of Occupational Accidents in the Woodworking
Sector: The Case of the Malayasian Wooden FurnitureFig. 5. PLCC values between GGNR-IQA, BRISQUE, FRIQAs and MOS© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

740

Heshalini Rajagopal, Norrima Mokhtar, Anis Salwa Mohd Khairuddin, Wan Khairunizam, Zuwairie Ibrahim, Asrul Bin Adam, Wan Amirul Bin Wan Mohd
Mahiyidin

Industry.
J
Appl
Sci.
2011;11(3):561-566.
doi:10.3923/jas.2011.561.566[10] Wang Z. Applications of objective image quality assessment
methods. Signal Process Mag IEEE. 2011;28(6):137-142.
doi:10.1109/MSP.2011.942295[11] Gulame M, Joshi KR, Kamthe RS. A Full Reference Based
Objective Image Quality Assessment. Int J Adv Electr
Electron Eng. 2013;2(6):13-18.
[12] Chandler DM. Seven Challenges in Image Quality
Assessment: Past, Present, and Future Research. ISRN Signal
Process. 2013;2013:1-53. doi:10.1155/2013/905685[13] Khalid M, Lee ELY, Yusof R, Nadaraj M. Design of an
intelligent wood species recognition system. Int J Simul Syst
Sci Technol. 2008;9(3):9-19.
[14] Tou JY, Tay YH, Lau PY. A Comparative Study for Texture
Classification Techniques on Wood Species Recognition
Problem. In: 2009 Fifth International Conference on Natural
Computation. IEEE; 2009:8-12. doi:10.1109/ICNC.2009.594[15] Bremananth R. ; Nithya B. ; Saipriya R. Wood Species
Recognition Using GLCM and Correlation. In: 2009
International Conference on Advances in Recent
Technologies in Communication and Computing. Vol 51. ;2009:431-444. doi:https:/https:/10.1109/ARTCom.2009.10[16] Wang Z, Simoncelli EP, Bovik AC. Multi-Scale Structural
Similarity for Image Quality Assessment. In: Asilomar
Conference on Signals, Systems and Computers. Vol 2. IEEE;
2003:1398–1402.
[17] Zhang L, Zhang D, Mou X. FSIM: a feature similarity index
for image quality assessment. Image Process IEEE Trans.
2011;20(8):2378-2386.
[18] Wang Z, Li Q. Information content weighting for perceptual
image quality assessment. IEEE Trans Image Process.
2011;20(5):1185-1198. doi:10.1109/TIP.2010.2092435[19] Xue W, Zhang L, Mou X, Bovik AC. Gradient magnitude
similarity deviation: A highly efficient perceptual image
quality index. Image Process IEEE Trans. 2014;23(2):684695.
[20] Meier E. The Wood Database. https://www.wooddatabase.com/. Published 2007. Accessed August 16, 2018.
[21] Rahman T, Haque MR, Rozario LJ, Uddin MS. Gaussian
Noise Reduction in Digital Images Using a Modified Fuzzy
Filter. In: International Conference on Computer andInformation Technology (ICCIT). ; 2014:217-222.
[22] Rajagopal, H., Khairuddin, A.S.M., Mokhtar N et al.
Application of image quality assessment module tomotion‑blurred wood images for wood species identification
system. Wood Sci Technol. 2019:1-15.
[23] Mittal A, Moorthy AK, Bovik AC. No-Reference Image
Quality Assessment in the Spatial Domain. IEEE
Transactions
on
Image
Processing.
doi:10.1109/TIP.2012.2214050[24] Heshalini Rajagopal, Norrima Mokhtar TFTMN, Izam
WKWA. No-reference quality assessment for image- based

assessment of economically important tropical woods. PLoS
One. 2020:1-15. doi:10.1371/journal.pone.0233320[25] Abd Latif, MH, Md. Yusof, H, Sidek, S.N, Rusli N.
Implementation of GLCM Features in Thermal Imaging
Affective State Detection. In: IEEE International Symposium
on Robotics and Intelligent Sensors. Vol 76. ; 2015:308-315.
doi:10.1016/j.procs.2015.12.298[26] Rubiyah Yusof, Nenny Ruthfalydia Rosli MK. Using Gabor
Filters as Image Multiplier for Tropical Wood Species
Recognition System. In: International Conference on,
Computer Modeling and Simulation, International
Conference. ; 2010:289-294. doi:10.1109/UKSIM.2010.61[27] Jain, Anil K. and FF. Unsupervised Texture Segmentation
Using Gabor Filters. Pattern Recognit. 1991;24(12):11671186.
[28] Recommendation ITURBT (2002). 500-11. Methodology for
the Subjective Assessment of the Quality of Television
Pictures.; 2000.
[29] Chow LS, Rajagopal H, Paramesran R. Correlation between
subjective and objective assessment of magnetic resonance(MR) images. Magn Reson Imaging. 2016;34(6):820-831.
doi:10.1016/j.mri.2016.03.006[30] Bindu K, Ganpati A, Sharma AK. A Comparative Study of
Image Compression Algorithms. Int J Res Comput Sci.
2012;2(5):37-42. doi:10.7815/ijorcs.25.2012.046[31] Li Sze Chow HR. Modified-BRISQUE as no reference image
quality assessment for structural MR images. Magn Reson
Imaging. 2017;43:74–87. doi:10.1016/j.mri.2017.07.016[32] Chang, Chih-Chung and Lin C-J. LIBSVM: A library forsupport vector machines. ACM Trans Intell Syst Technol.
2011;2(3):27:1--27:27.
[33] Song X-K. Correlated Data Analysis: Modeling, Analytics,
and Applications. Springer Science & Business Media; 2007.
[34] Chai T, Draxler RR. Root mean square error (RMSE) or mean
absolute error (MAE)? -Arguments against avoiding RMSE
in the literature. Geosci Model Dev. 2014;7(3):1247-1250.
doi:10.5194/gmd-7-1247-2014© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

741

Design and development of automated seeding and irrigation system
Anirban Kumar1, Heshalini Rajagopal1

Department of Electrical and Electronic Engineering, Manipal International University, Malaysia
1anirban_kumar1993@yahoo.com

1

Abstract
Malaysia, has a very conducive environment for agriculture. Six percent of the population are employed tosome of the agriculture activities making agriculture the most curtail industry in Malaysia. Hence, there isa need to develop the agricultural facilities by incorporating latest technological advancements.
Conventional seeding process is time consuming and requires additional labor cost. In this paper, anautomated system is proposed for seeding as well as irrigation process in agriculture which reduces the laborcost. This system aims to increase the efficiency of the seeding process without affecting the nature of soil.
The proposed system is equipped with Arduino MEGA and Arduino UNO which acts as the main control
unit while ultrasonic and soil moisture sensors are used to detect the obstacles and soil moisture level,
respectively. The robot consists of a funnel like arrangement to perform the seeding procedure. The field isequipped with moisture sensors placed at different areas that monitors the moisture level of the soil on aregular interval for irrigation purposes. The proposed system will be of great benefit to the future endeavorof agricultural business as well as it will be able to optimize the seeding and irrigation.
Keywords: Agriculture, Seeding, Irrigation, Arduino MEGA, Ultrasonic sensors, Soil moisture sensors.

1 Introduction

2 Prior Work

Agriculture is the oldest and yet an important economic
activity for any given country. There are distinct available
types of machines for harvesting, ploughing, pesticide
control etc. Most of these machines are manually operated to
execute the required action. Different machines are used toperform separate action which leads to lower profit returns[1]. Seeding is a tedious yet a crucial activity for a farmer.
Seeding of a large farmland is even more tedious, which
demands for big group of personnel for manual seeding. To
minimize the personnel required for seeding, harvesting etc.,
agriculture machineries were developed. Planting of seeds
manually, leads to low spacing efficiency and seedplacement, serious health issues and tedious task [2]. Ideal
solution to overcome the deficiencies discussed above is bydeveloping a machine that carries out more than one action
and by atomization of the developed machine, which willlead to intensification of the yield.
The primary aim of this paper is to design and develop
an efficient seeding and irrigation system. This system
consists of an Arduino MEGA, Arduino UNO, ultrasonicsensors, DC motor, servo motors and moisture sensors. The
scissor lift is designed using Fusion 360 and 3D printed.
Arduino MEGA and UNO holds all the commands, receive
all the parameters from the sensors and send output signals
accordingly. The ultrasonic sensors will be used for theobject detection and moisture and DHT 22 sensor willcontinuously monitor the moisture of the soil, humidity andtemperature of the surroundings on a regular interval andalerts the monitoring system.

M.Aravind Kumar et al. (2018) have proposed a machine forseeding and irrigation to limit the working expense anddecrease the ideal opportunity for planting seed & irrigation
operations by using solar energy to operate the system. Thereis a sharp pointed iron attached to the front of the robot to
dig the soil for seed sowing, once the seed is planted the
supporter at the back end of the robot covers the dug landwith soil. Versatile computerized water system robot was
built to irrigate the plants appropriately [3]. However, thisstudy does not irrigate the planted seed immediately after theseeding process is completed.
Md. Didarul Islam Sujon et al. (2018) have proposed anautonomous prototype robot that will help farmers in the
farmland. It is an Arduino controlled robot that carries out
ploughing, sowing and watering process. The robot carries
out farming using ultrasonic sensors in order to change its
position from one farming strip to another. The robot aims
contribute greatly in developing the farming strategies andreduce farmers cost [4]. However, this robot can carry out
farming for land area of limited land area only.
K.A.Sunitha et al. (2017) have proposed a robotic systemthat localizes the path and can navigate itself without humanaction. For ploughing, this robot is equipped with tentacles
that has saw blades. The locomotion is provided with wheels
covered under conveyor belts. For each rotation every toothon gear will take seeds and will drop them on field. Camera
at the front end tracks the path for every fixed distance andat the minimum distance it takes the path pre-programmed[5]. However, this study has a drawback where the system© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

742

Anirban Kumar, Heshalini Rajagopal

uses camera and image processing for object detection whichmakes the system complex. In addition, the object detection
can be inaccurate if the image captured is subjected to motion
blur due to the dirty camera lens and camera movement.
S. Thawali et al. (2017) have proposed robot capable ofperforming operations like automatic ploughing, seeddispensing and pesticide spraying autonomously and also
provides manual control when required. Initially the robot
digs the entire field simultaneously dispensing seeds side byside. On the field the robot operates on automated mode. For
manual control the robot uses the Remote controller as
control device and helps in the navigation of the robot on thefield [6]. However, there is no obstacle detection system andhence there are chances that this robot will runover plantedplats if there is no human presence to deviate the robot fromthe predefined path.
Thorat Swapnil V et al. (2017) have proposed a sowing
machine which is operated. This machine is capable ofplanting different types and different sizes of seeds and also
the spacing between two seeds can be varied. Seed storagetank is connected at the top of the robot near rear wheels.
Front sensor serves the function of guiding the robot. As any
obstacle comes in front of robot it gives the signal to therobot and diverts the path of robot [7]. However, the systemdoes not monitor the soil or plant conditions so the farmers
still have to invest in systems that monitor those twovariables.

the sensors will be sent to the Android Application and SMS
via GSM module.

Fig. 2: Block Diagram for Soil Moisture, Humidity andTemperature Monitoring System
3.1. Circuit Diagram and Simulation
3.1.1. Arduino DC Motor Control

3 Methodology
Fig 1 shows the block diagram for the motion, seeding,irrigation, obstacle detection and avoidance. The main
control unit of this system is Arduino MEGA which is
powered by a 9V battery. Ultrasonic sensors will give input
to the Arduino MEGA regarding the obstacle detection.
Arduino MEGA will analyse the received inputs and control
the output to the lifting servos. DC motors are to be poweredby a 7.7V Lithium Polymer battery; the motors are used forthe movement of the robot. Arduino MEGA controls the
motor direction and speed (RPM) using L298N module.

Fig. 1: Arduino DC Motor Control
Fig. 3 shows the circuit diagram of Arduino DC motor
control. Only three Arduino pins are required, pin 8 is for thepush button which controls the direction of motor rotation.
Pins 9 and 10 are PWM signal outputs, at a given time there
is only one active PWM, this allows for the control of
direction as well as the speed by varying the duty cycle of
the PWM signal. The active PWM pin decides the motor
direction of rotation (one at a time, the other output is logic0). The speed of the DC motor (both directions) is controlledwith the 10kΩ potentiometer which is connected to analog
channel 0 (A0). The L293D driver has 2 VCCs: VCC1 is+5V and VCC2 is +12V. Pins IN1 and IN2 are the control
pins and their functions are shown in Table 1.

Fig. 1: Block Diagram for Seeding, Irrigation and Obstacle
Avoidance
Fig. 2 shows the block diagram for the soil moisture,humidity and temperature monitoring system. Soil moisture
and DHT 22 sensors will give input to the Arduino UNO
regarding the soil moisture level, temperature and humidityof the surrounding respectively. The input information from

Table 1: IN1 and IN2 function© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

743

Design and development of

3.2 Flow Chart
The various steps undertaken to develop the code tooperate the proposed system is shown in Fig 5. Once thehardware design and assembly of the seeding and irrigation
robot is completed, code is developed to carry out the desired
action. During the coding process mainly loops like “while”,“if” and “for” will be used to achieve the desired movements.

3.1.2. Lifting System
Fig. 4 shows the circuit diagram for the lifting system thatuses ultrasonic sensor, Arduino and motor.

Fig 4: Lifting System
The lift like system will be used to propel the robot
upwards once the ultrasonic sensor detects any obstacle, thesystem will be propelled upwards to a predefined height.
Once the robot reaches the predefined height, the plant willbe irrigated and then the robot will move forward to irrigate
the plants ahead. When all the plants in that segment are
irrigated the robot will be brought down.

Fig. 5: Flowchart for Software Development
4 Results
4.1. Seeding Mechanism
The first objective of the system is to design anautomated robot to perform seeding and irrigation procedure.
The robot takes ten seconds to initialize. After initialization
robot moves forward for two seconds at the speed of 200 rpm
and then comes to a halt to facilitate the seeding andirrigation process, respectively. Once the seeding andirrigation process is completed, the robot moves forward fortwo more times before it turns to the next row. The robot is
programmed to drops the seed at an interval of six seconds
and 7 cm. Fig. 6 shows the seeding process of mung dhal

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

744

Anirban Kumar, Heshalini Rajagopal

4.3. Monitoring System
The second objective of the study is to develop atemperature and soil moisture monitoring system by using
Android Application to enhance the user accessibility. The
soil moisture, humidity and temperature monitoring system
is in place for the landowner or the user to monitor the soil
conditions of the separate segments using the Android App
which was created using the MIT App Inventor. The app isdesigned to display the information regarding the soil
moisture, humidity and temperature level of each land
segment. The layout of the App is shown in Fig. 8.
Fig. 6: Seeding Process of Mung Dhal
4.2. Irrigation Mechanism
Fig. 7 shows that the robot performs irrigation to the pla
nt after the seeding process. For the design of water harvesti
ng systems, it is necessary to assess the water requirement of the crop intended to be grown. For crops like mung dhal
and peas initially 20ml to 25ml water is required for irrigat
ing each seed. Table 2 shows the irrigation details of the rob
ot prototype.
Fig. 8: Monitoring the Soil Condition Using Android App
The monitoring system is beneficial for the user, because
it helps to keep track of the soil moisture, the surrounding
environment and carry out on demand irrigation. It also helps
the farmers to store the data for future weather prediction andplanning on land management based on the data collected.
4.4. Lifting System
Thirdly, this paper aimed to design and develop a lifting
system to avoid obstacles while maneuvering and assist in
irrigation. The robot is equipped with two ultrasonic sensors
to detect obstacles at the front and base of the robot. When
front ultrasonic sensor detects an obstacle at distance of 10
cm or less, the system stops and activates the lifting system
to avoid the obstacles. When the system has fully lift
upwards, the base ultrasonic sensor activates, and the robot
continues to move forward. The base ultrasonic sensor isplaced in order to bring the lifted system back to its originallevel if distance between the lifted system and ground ismore than 20 cm and if there is no obstacle in the front. Fig.
9 shows that the robot is lifted when there is an obstacle infront.

Fig. 7: Robot performing irrigation
Table 2: Irrigation Detail
Irrigation Tank Capacity: 500ml
Irrigation Pump Flow Rate: 1.2-1.6 L/min
Irrigation per seed: 20ml - 30ml
Irrigation type: Drip irrigation© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

745

Design and development of

Designed for Seeding Mechanism," IOP Conference Series:
Materials Science and Engineering, 2017.
[6] S. Thawali, B. Yadav, H. Rumde and R. S. Sewane, "Designand Manufacturing of Robot for Digging and Seeding inAgriculture," International Conference on Ideas, Impact andInnovation in Mechanical Engineering, vol. 5, no. 6, p. 1637
– 1644, 2017.
[7] M. L. K. P. G. V. P. R. N. Thorat Swapnil V, "Design andFabrication of Seed Sowing Machine," International
Research Journal of Engineering and Technology, vol. 4, no.
9, pp. 704-707, 2017.

Fig. 9: Robot Lifts Up
The robot is capable to run for about 4km in a single battery
charge.

5 Conclusions

The proposed system is capable to perform seeding
followed by irrigation procedure. The robot is equipped witheight wheels to facilitate the movement of the robot. The
robot is also equipped with two ultrasonic sensors of any
obstacle at the front and base of the robot to detect obstaclesin the front and under the robot. The robot lifts up before
moving forward when it detected obstacles in the front andcomes back down with the assistance of the base ultrasonicsensor when the obstacle has been passed. This also enables
the system to water the tall plants. Soil moisture and DHT22
sensors are fixed on the land to sense the soil moisture,humidity and temperature. The readings of these parameters
are sent to the user vis SMS every one minute. In addition,
the user can view the readings of the parameters via Android
App. The robot moves forward once and perform the seedingat one point, irrigate the seed and moves to the next point forthe seeding process. The robot dispenses seeds automaticallyat a fixed distance (7 cm).

References[1] A. A. M. M. Amrita Sneha.A, "Agricultural Robot for
Automatic Ploughing and Seeding," IEEE International
Conference on Technological Innovations in ICT for
Agriculture and Rural Development, pp. 17 - 23, 2015.
[2] M. L. K. P. G. V. P. R. N. Thorat Swapnil V, "Design andFabrication of Seed Sowing Machine," International
Research Journal of Engineering and Technology (IRJET),
vol. 04, no. 09, pp. 704 - 707, 2017.
[3] M. Kumar, A. S. Reddy and K. Sagadevan, "Automatic Seed
Sowing & Irrigation Agribot using Arduino," International
Journal of Pure and Applied Mathematics, vol. 119, no. 14,
pp. 1089-1093, 2018.
[4] M. D. I. Sujon, R. Nasir, M. M. I. Habib, M. I. Nomaan, J.
Baidya and M. R. Islam, "Agribot: Arduino Controlled
Autonomous Multi-Purpose Farm Machinery Robot for
Small to Medium Scale Cultivation," International
Conference on Intelligent Autonomous Systems, pp. 155-159,
2018.
[5] K. A. Sunitha, G. S. G. S. Suraj, C. P. N. Sowrya, G. A.
Sriram, D. Shreyas and T. Srinivas, "Agricultural Robot© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

746

Investigation of A Real-Time Driver Eye-Closeness for the Application of Drowsiness
Detection

Muhammad Zubir bin Kamazlan1, Wan Khairunizam1, Abdul Hafiz Halin1, M. Rudzuan M. Nor1, Azian Azamimi Abdullah2,
Norrima Mokhtar3

Depart. of Mechatronic Eng., Faculty of Electrical Engineering Technology, University Malaysia Perlis, Malaysia
Depart. of Biomedical Elec. Eng., Faculty of Electronic Engineering Technology, University Malaysia Perlis, Malaysia
3
Department of Electrical Engineering, Faculty of Engineering, University of Malaya, Malaysia
1khairunizam@unimap.edu.my
1

2

Abstract
The increase in accident and death rates due to drowsiness while driving raises concerns to the community. An
efficient solution is vital to ensure the safety of all drivers on the road. Most previous studies have analyzed
drowsiness using head tilt, yawning, and eye condition. Face detection applied in drowsiness detection fromprevious research not included distances between subject and camera. The features used for eye detection requiredlarge storage and long-term process which are not applicable in a real-time system. This study uses Haar algorithm
and analysis is performed based on the size of the region of interest for face detection. Eye monitoring uses facial
landmark features and the evaluation is dependent on the width of the eye. The percentage of eye closure is used
to describe the eyes as closed. This study only takes into account the normal rate of blinking eyes while driving
because of the long-time constraints required for a person to be in a drowsy state. In this research, the Raspberry
Pi 3B+ and Pi cameras are used as processing and vision devices. The highest accuracy of face detection achieved
based on the ROI area at a distance of 80 cm is 98.33%. The lowest difference between eye width and theintercanthal distance is 0.36%. The overall normal eye blink rate while driving is in the range of the normal eye
blink rate which does not exceed 20 blinks/min as reported by the previous researcher.
Keywords: Face detection, eye detection, drowsiness detection.
1

subjective sleepiness, decreasing efficiency, and the numberof lapses in a visual reaction time task closely associatedwith PERCLOS values [4]. The proportion of time theeyelids cover at least 80% of the pupils is measured by
PERCLOS [5].
PERCLOS is more dependable across drivers than EEG,
blinks, and head position in the research since its algorithm
was more than 90% precise in identifying deteriorated
performance during alertness tasks as validated by Dinges etal. (1998) [6]. The eye conditions could be divided into a
few sections relying on PERCLOS. Heartbeat rate, yawning,and head tilting are other terms that can calculate the levelof drowsiness. However, this research only focuses on eyecloseness to determine the normal eye blink rate whiledriving due to limitations that required a long period for a
person to fall in a drowsy state.

INTRODUCTION

One of the potential applications of smart vehicle
systems is the drowsy driver detection system [1]. Thus, thedevelopment of this system vital in preventing traffic
accidents. This system works to measure the blink rate of thedrivers based on eye-closeness. The eye condition contains
critical information, and it is possible to determine a driver's
level of drowsiness if such visual activity can be assessed[2].
Face detection is a primary component that requires a
Pi camera that acts as a video sensor. The location of the eyeshould be determined before the eye's status can be specified
using suitable image processing.
It has been confirmed in a U.S. Department ofTransportation publication: “Only PERCLOS that arehighly correlated with PVT blunders in and between
subjects even though almost every innovation demonstrated
the ability to diagnose drowsiness by detecting lapses in atleast one subject or a subset of subjects” [3]. Increasing© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

747

Muhammad Zubir bin Kamazlan, Wan Khairunizam, Abdul Hafiz Halin, M. Rudzuan M. Nor, Azian Azamimi Abdullah, Norrima Mokhtar

2

implements numerous machine learning models andsupports functions such as threading and networking. Thislibrary
enablestheuseofshape_predictor_68_face_landmarks.dat to extract facial
features effectively. The system functionality tested in thelaboratory first before tested the real driver in the car.

METHODS
Research Flow Chart
START
Phase 1: To implement an algorithm
that can analyse eye-closenessmovement

Phase 2: To investigate theperformance of the software for realtime blink detection

Setup Raspberry Pi 3B+ with Pi
CameraPerform Test in Car

Perform Test in Lab

No

Yes

SystemFunction

2.3. Experimental Setup
Eight (8) subjects from ages 23 to 25 involved in the faceand eye detection tests while only four subjects with morethan one-year driving experience being tested in normal
blink rate evaluation. The experiments conducted from 10
am to 12 pm with sunny weather conditions. However, thedark-tinted car's window reduced the amount of lightentering the car. Face and eye detection task conducted in a
car at Universiti Malaysia Perlis. Normal Blink Rateexperiment requires the subjects to drive on the highway
from Pauh flyover to Universiti Malaysia Perlis as shown in
Fig 2.

Face detection with different distances

Data collection and analysis

Data Collected
and agreed
Eye Blink Rate

No

Yes

Set distance from previous analysis
Data collection and analysis
Eye detection
No

Data Collected
and verified

Data collection and analysis

Yes
Yes

Progress ReportNo

Data Collected

END

Fig. 1. Research flowchart
Fig. 2. Route used to measure blink rate while driving

Fig. 1 shows the flow chart of the research. Phase #1 is todesign an algorithm that analyze the eye-closenessmovement. The setup of software and hardware is
performed in this phase. The system functionality is tested
before proceeding to phase #2. The aim of phase #2 is toinvestigate the performance of the software for real-time eyeblinking. This phase is separated in three parts, which areface detection, eye detection and eye blink rate. The
experiments are carried out part by part.

2.4. Face Detection
The subjects placed in front of the camera as shown in
Fig 3 with the value of x = 75 cm measured using theultrasonic sensor in the car as the set distance between
the seat and camera. The program executed to detect theface of the driver for one minute because theinvestigation of the system performance for the blinkrate is based on every minute. The size of the rectangle
as shown in Fig.4 on the driver’s face was stored as ameasured value. The process repeated with a value of x
= 80 cm and x = 85 cm because the adjustable car seat
has a 5 cm difference between each position. The
program executed three times for each value of x to
obtain the average value. The average and standard
deviation of the face area of five subjects were
measured and is set as the reference. Three subjects
were tested and analyzed based on these reference value.
The reference value will be used to determine the2.2. Software Setup
Raspberry Pi 3 Model B+ chosen as a computer base to
execute the program. 5 MP Raspberry Pi camera also needed
as a visual device and an ultrasonic sensor used to measure
the distance between subject and pi camera. OpenCVPython library is competent for this system to solve
computer vision problems and obtain the required data. Thislibrary
used
the
Haar
CascadeClassifier(haarcascade_frontalface_default .xml) to detect the face inthe frames. Dlib is a modern C++ open-source library that

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

748

Investigation of A Real-Time

accuracy from the further data obtained. If the data are
not within the standard deviation of the reference value,
the data is not accurate or acceptable.

𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑, 𝑑𝑑 =�(𝑥𝑥2 − 𝑥𝑥1)2 − (𝑦𝑦2 − 𝑦𝑦1)2(2)

𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖ℎ𝑎𝑎𝑎𝑎 𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 = 𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤ℎ (𝑟𝑟𝑟𝑟𝑟𝑟ℎ𝑡𝑡 𝑒𝑒𝑒𝑒𝑒𝑒) =(3)
𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤ℎ( 𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙 𝑒𝑒𝑒𝑒𝑒𝑒)(𝑝𝑝39 ↔ 𝑝𝑝42) = (𝑝𝑝36 ↔ 𝑝𝑝39) = (𝑝𝑝42 ↔ 𝑝𝑝45) (4)The eye-detection proceeded using the 80 cm distance thathas the most accuracy from face detection analysis. Facial
landmarks were the process of tracking vital feature points
on the face and had been used for distinguishing andportraying the distinctive regions of the face. The corners ofeach eye were located using the facial landmark predictor.
The Euclidean distance between the points was determined.
This task executed in one minute and repeated three times
for each subject to determine the ability of the system totrack the eye every minute. [7] stated that intercanthal
distance in the average face is equal to the width of an eye.
Thus, this task validates either the statement in [7] can be
accepted or not.

Fig. 3. Position of the subject in front of the camera
𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 = 𝑤𝑤 × ℎ
𝑃𝑃 =

(1)

𝑘𝑘
× 100
𝑚𝑚

where:
P: Percentage Accuracy of the Area (%)
k: number of data (area) within standard deviation ofreference value
m: total of data sample (60)Fig. 6. Point axis

Fig 4. Face area based on ROI box
2.5. Eye Detection

Fig. 7. Eye corner detection
2.6. Normal Blink Rate
This task performed within five minutes for each subject
because the research analyses the blink rate for every minute.
The Eye Aspect Ratio (EAR) from every subject whiledriving was recorded in this test. PERCLOS is the shortenedform of eyelid closure over time [8]. PERCLOS (P80)applied to determine 80 % the percentage closure of the eyesin the preliminary test. The value of 80% of eye closure set
as threshold. The purpose of this task was to validate thenormal eye blink rate while driving as mentioned in [9] thatthe normal eye blink rate is 15 to 20 blinks/ min and the blink

Fig. 5. Intercanthal distance© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

749

Muhammad Zubir bin Kamazlan, Wan Khairunizam, Abdul Hafiz Halin, M. Rudzuan M. Nor, Azian Azamimi Abdullah, Norrima Mokhtar

Table 2. ROI area for subject#1

rate could decrease when the eyes focus on the work such as
driving [10].

DistancefromsubjecttocameraFig. 8. Eye Aspect Ratio
�|𝑝𝑝2 − 𝑝𝑝6|� + ||𝑝𝑝3 − 𝑝𝑝5||
𝐸𝐸𝐸𝐸𝐸𝐸 =2||𝑝𝑝1 − 𝑝𝑝4||(5)𝑇𝑇ℎ𝑟𝑟𝑟𝑟𝑟𝑟ℎ𝑜𝑜𝑜𝑜𝑜𝑜 = [ max(𝐸𝐸𝐸𝐸𝐸𝐸)− min(𝐸𝐸𝐸𝐸𝐸𝐸) ] x 0.2+ min(𝐸𝐸𝐸𝐸𝐸𝐸)(7)𝐸𝐸𝐸𝐸𝐸𝐸(𝑏𝑏𝑏𝑏𝑏𝑏ℎ 𝑒𝑒𝑒𝑒𝑒𝑒) = [𝐸𝐸𝐸𝐸𝐸𝐸(𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙 𝑒𝑒𝑒𝑒𝑒𝑒)+ 𝐸𝐸𝐸𝐸𝐸𝐸(𝑟𝑟𝑟𝑟𝑟𝑟ℎ𝑡𝑡 𝑒𝑒𝑒𝑒𝑒𝑒)] ÷ 2(6)Percentage
Accuracyof theArea (%)

75 cm

Area
(mm2)
5116

Standard
Deviation
480

Area
(mm2)
4789

Standard
Deviation
122

Note
81.67

80 cm

4376

217

4253

90

90.00

85 cm

3932

216

3838

128

88.33

Table 3. ROI area for subject#2
DistancefromsubjecttocameraFace Detection Analysis
Table 1. ROI box average area target on different distancesfor 5 subjects
Reference Target
Area (mm2)

Obtained

Subject #1 produces the average area of 4789 mm2 for the
distance 75 cm, 4253 mm2 for the distance 80 cm, and 3838
mm2 for the distance 85 cm. The experiments are run threetimes for each distance and only the average with the lowest
standard deviation is chosen. Percentage accuracy for
distance 75 cm which is 81.67 % measured using theequation 2.

3 RESULTS AND DISCUSSIONSDistance

Reference Area

Standard
Deviation

75 cm

5116

480

80 cm

4376

217

85 cm

3932

216

Reference Area

Obtained

Percentage
Accuracyof the Area
(%)

Area
(mm2)

Standard
Deviation

Area
(mm2)

Standard
Deviation

75 cm

5116

480

5437

134

81.67

80 cm

4376

217

4393

81

98.33

85 cm

3932

216

4071

109

83.33

Subject #2 produces the average area of 5437 mm2 for
distance 75 cm, 4393 mm2 for distance 80 cm, and 4071
mm2 for distance 85 cm. The experiments are run three times
for each distance and only the average with the lowest
standard deviation is chosen. Percentage accuracy for
distance 75 cm which is 81.67 %.

Five (5) subjects are involved in assessing the average of thereference area of the ROI box for facial detection based on
various distances. Three additional subjects are tested toassess the precision of the detection of the face. The data inthis table is set as a reference for experiments of othersubjects. If the data obtained in those tests are within thestandard deviation, it is considered acceptable.

Table 4. ROI area for subject#3
Distancefromsubjecttocamera75 cm

Reference Area

Obtained

Area
(mm2)

Standard
Deviation

Area
(mm2)

Standard
Deviation

5116

480

4840

168

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

750

Percentage
Accuracyof the Area
(%)

80.00

Investigation of A Real-Time
80 cm

4376

217

4413

89

96.67

85 cm

3932

216

3743

91

86.67

Fig 9 shows that only subject #1 has the smallest difference
between eye width and intercanthal distance while the otherhas a large gap between both values. The eye width in the
average face is not equal to the intercanthal distance asmentioned in [7] since seven over eight-person have a large
percentage difference between those values.

Subject #3 produces the area of 4840 mm2 for distance 75
cm, 4413 mm2 for distance 80 cm, and 3743 mm2 for
distance 85 cm. The information gathered for the mostrecent subject is obtained value and the reference area is theaim value.

Eye width and Intercanthal distance

The highest percentage accuracy among three subjects is forthe distance of 80 cm and the lowest is at 75 cm distance.
The error in percentage accuracy resulted from the area of
ROI that varies out of the reference target standard
deviation. The errors in the value occurred due to thecomplex background [11] and the lighting condition during
the test [12].The distance with highest accuracy is better tobe applied in further test because the facial landmark point
is depend on the ROI area. Thus, the highest accuracy will
ensure that the position of facial landmark have the lowest
variation.

Width (mm)

20
15
10
5

2

3

4

5

6

7

8

Subject #Average Eye Width (Both Eye)Eye Detection Analysis

Intercanthal DistanceThis task is performed with a distance of 80 cm fromthe camera to the subject because it has the highest accuracyof face detection. Table 5 records the eye width andintercanthal distance for each subject. Percentage ofdifferent measured the inequality between the average widthof both eye and the intercanthal distance. As we can see, thelowest percentage difference is from subject 1 while others
have a large percentage difference between eye width andintercanthal distance. Thus, only subject #1 fulfilled thestatement in [7] that the eye width is equal to intercanthal
distance on the average face. The other subject may have abroad range of measurements between eye width andintercanthal distances due to the low resolution of the vision
device used.

Fig. 9. Eye width and intercanthal distance
Eye Blink Rate Analysis
Table 6. Eye blink for subject #1
Time(minute)
1st
2nd
3rd
4th
5th

Table 5. Eye width and intercanthal distanceSubject

1

Intercanthal
Distance (mm)Percentage ofdifferent#1

Average eyewidth forboth eye
(mm)
9.945

9.909

0.36

#2

9.989

15.771

36.66

#3

9.048

15.614

36.05

#4

10.58

16.695

36.63

#5

10.85

16.968

36.05

#6

9.958

16.99841.42#710.06917.362

42.01

#8

9.045

15.997

43.46

Blink rate
4 blink/min
5 blink/min
6 blink/min
4 blink/min
5 blink/min

The normal eye blink rate for subject #1 for five minutes islisted in Table 6. Fig 10 shows the EAR plotted for subject#1. The threshold of the EAR for this subject 0.1814
determined by the minimum and maximum value of EAR
which are 0.127 and 0.399. The value below the threshold isclassified as a blink. The calculation of the threshold can bereferred to equation 8.0.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

751

Muhammad Zubir bin Kamazlan, Wan Khairunizam, Abdul Hafiz Halin, M. Rudzuan M. Nor, Azian Azamimi Abdullah, Norrima Mokhtar

Table 8. Eye blink for subject #3

EAR against Time
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1

Time(minute)
1st
2nd
3rd
4th
5th

EAR

MAX(0.399)

MIN(0.127)
0.01
0.18
0.35
0.52
1.09
1.26
1.43
2
2.17
2.34
2.51
3.08
3.25
3.42
3.59
4.16
4.33
4.5

The normal eye blink rate for subject 3 for five minutes islisted in Table 11. Fig 12 shows the EAR plotted for subject#3. The EAR threshold for this subject 0.2146 is determinedby the minimum and maximum EAR values of 0.138 and
0.521. The eyes will be considered as close if the eyelids
covered more than 80% or less than 0.2146 of the EAR.

Time (min)EAR

ThresholdFig. 10. The EAR for Subject #1

EAR against Time

Table 7. Eye blink for subject #2
Time(minute)
Blink rate
1st
5 blink/min
2nd
4 blink/min
3rd
6 blink/min
4th
4 blink/min
5th
2 blink/min

0.01
0.19
0.37
0.55
1.13
1.31
1.49
2.07
2.25
2.43
3.01
3.19
3.37
3.55
4.13
4.31
4.49
EAR

Overall eye blink rate for four subjects are not exceeding 20
blinks per minute. It is as mention in [9] and [10] that thenormal eye blink rate for an average person is not more than
20 blinks per minute. There are differences in the blink rate
between subject #3 and subject #1, subject #2 as well as
subject #4. The lower blink rate of subject #1, subject #2,
and subject #4 due to focus state while driving as referred to[9] and [10]. Subject #3 may have a higher blinking rate
because of the cold environment due to the air conditioner
in the car. The rise of the blink rate can increase eyetemperature [13].

0.35
EAR

0.3

0.25
0.2

0.01
0.19
0.37
0.55
1.13
1.31
1.49
2.07
2.25
2.43
3.01
3.19
3.37
3.55
4.13
4.31
4.49

MIN(0.142)

EAR

ThresholdFig 12. EAR for Subject #3

MAX(0.415)

0.4

0.1

MIN(0.138)

Time (min)EAR against Time

0.15

MAX(0.521)

EAR

0.55
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1

Table 7 indicates five minutes of the normal eye blink rate
for subject #2. The EAR for subject #2 is shown in Fig 11.
The minimum and maximum values for EAR are 0.142 and0.415 were used to determine the threshold value of 0.1966.
The threshold is the value where 80% of the eyes are closed.

0.45

Blink rate
9 blink/min
5 blink/min
11 blink/min
10 blink/min
15 blink/min

Time(min)ThresholdFig. 11. The EAR for Subject #2© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

752

Investigation of A Real-Time

4 CONCLUSIONS[10] Z. A. Haq, “Eye-Blink rate detection for fatigue
determination,” no. March, 2018.
[11] V. Singh, V. Shokeen, and B. Singh, “Face Detection By Haar
Cascade Classifier With Simple And Complex Backgrounds
Images Using Opencv Implementation,” Int. J. Adv. Technol.
Eng. Sci., vol. 1, no. 12, pp. 33–38, 2013.
[12] M. Rezaei and R. Klette, Driver Drowsiness Detection. 2017.
[13] D. B. Gurung, K. C. Gokul, and P. R. Adhikary,“Mathematical model of thermal effects of blinking in humaneye,” Int. J. Biomath., vol. 9, no. 1, 2016.

The Eye-Closeness system is a functional system to use
for drowsiness detection applications. The features appliedin this system is dependable to identify the drowsy condition.
This research has proven that most accuracy of facedetection is above 80% and the highest reading achieved is98.33%. The error occurred only as a result of the complex
background and unsteady lighting while performing the test.
Seven over eight-persons have a broad range ofmeasurements between eye width and intercanthal distances
due to the low resolution of the hardware used. Throughout
the whole experiment, the algorithm implemented isfunctional to analyse the eye-closeness movement. System
performance is reliable because all four subjects tested fornormal eye blink rate while driving is within the normal
blink rate that is below 20 blinks/min as indicated in theprevious research by Tereza Soukupov´ [9]. The safety ofthe driver and other road users should be concerned. Hence,
the use of this system can prevent accidents caused bydrowsiness.
REFERENCES[1] E. Vural, M. Cetin, A. Ereil, G. Littlewort, M. Bartlett, and J.
Movellan, “Drowsy driver detection through facial movement
analysis,” Lect. Notes Comput. Sci. (including Subser. Lect.
Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 4796
LNCS, no. November, pp. 6–18, 2007.
[2] P. R. Tabrizi and R. A. Zoroofi, “Drowsiness detection basedon brightness and numeral features of eye image,” IIH-MSP2009 - 2009 5th Int. Conf. Intell. Inf. Hiding Multimed. Signal
Process., pp. 1310–1313, 2009.
[3] D. Sommer and D. Edwards, “PERCLOS : An Alertness
Measure of the Past,” no. October, 2017.
[4] G. Marquart, C. Cabrall, and J. de Winter, “Review of Eyerelated Measures of Drivers’ Mental Workload,” Procedia
Manuf., vol. 3, no. Ahfe, pp. 2854–2861, 2015.
[5] S. Vitabile, A. De Paola, and F. Sorbello, “Bright pupil
detection in an embedded, real-time drowsiness monitoring
system,” Proc. - Int. Conf. Adv. Inf. Netw. Appl. AINA, pp.
661–668, 2010.
[6] A. D. McDonald, J. D. Lee, C. Schwarz, and T. L. Brown, “A
contextual and temporal algorithm for driver drowsinessdetection,” Accid. Anal. Prev., vol. 113, no. January, pp. 25–
37, 2018.
[7] J. van L. Gerhard Goos, Juris Hartmanis, Audio- andVideoBased Biometric Person Authentication. 2001.
[8] J. J. Yan, H. H. Kuo, Y. F. Lin, and T. L. Liao, “Real-time
driver drowsiness detection system based on PERCLOS andgrayscale image processing,” Proc. - 2016 IEEE Int. Symp.
Comput. Consum. Control. IS3C 2016, pp. 243–246, 2016.
[9] M. Perception and C. Technical, “Eye-Blink Detection Using
Facial Landmarks,” 2016.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

753

Towards Establishing Path Planning Strategies For Autonomous UAVs; A Brief SurveySummary on Recent Techniques
Anees ul Husnain1 2, Norrima Binti Mokhtar1, Noraisyah Binti Mohamed Shah1, Mahidzal Bin Dahari1

Department of Electrical Engineering, University of Malaya, Kuala Lumpur, Malaysia
Applied Controls and Robotics Research Laboratory, University of Malaya, Kuala Lumpur, Malaysia(Tel: 60-10-245-5060)
norrimamokhtar@um.edu.my
1

2

Abstract
The extent of autonomy in path planning for a UAV primarily depends upon the capabilities of its
algorithm. The diversity in UAV applications and an abundance of choices in autonomous path
planning algorithms are swelling every day, so the selection of most appropriate algorithm gets
baffling. The past two decades of research on UAVs revealed that seventy percent of it had been
published in the previous three and a half years. Hence, a comprehensive survey study was proposedand conducted to obtain an overview of the recent developments in autonomous path planningapplications and their respective algorithms. This article presents a summary of the survey andsuggests most suitable path planning algorithms for a UAV application.
Keywords: Autonomous UAV, Survey, UAV Path Planning.

mission requirements or to restrain themselves
within the areas of interest. This gives a verysignificant role to the aspect of path planning in
UAV operations, as a global objective, alongside thelocal flight requirements.
Usually, a UAV has a global and local set ofdynamics and parameters alongside a high- and lowlevel control strategies, to be generic. Path planning
is a matter of higher-level UAV control strategy
where the UAV considers global parameters to reachit optimality conditions.
From a broader perspective, path planning canbe categorized into two groups, namely offline andonline path planning. The path planning before thetake-off is offline while modification of flight path
during the flight, as required by the mission
objectives or the environmental dynamics, is online
path planning.
The extent of maximum autonomy that a UAV
can achieve is an arbitrarily unachievable as it

1 Introduction
UAVs can provide highly useful data, and theirrole becomes vital in information collection, aerial
surveillance, industrial or agricultural monitoring,
preventing disasters and so on. However, the mostsignificant element that is common in most of theseapplications to pick out would be the access to realtime data. The role of UAV systems, due to their costeffectiveness and eased of access to deploy has
influentially motivated scientists to opt them as atremendous support to research Nevertheless, thiswould require a higher level of autonomous
behavior among UAVs.
An autonomous flight correctly points towards
an onboard intelligent sensors-based system that is
expected to set and modify the intermediate flightpaths, to meet the mission objectives. An Unmanned
Aerial Vehicle (UAV) may either follow a preprogrammed flight trajectory, a remotely operated
path by a human or an dynamic path calculation
during the flight.[1] At all the times in flight, a UAV
must maintain or observe the parameters that mainly
include stable flight dynamics, energy consumptions,primarily depends upon degree of autonomous
operations that a UAV may conduct or a group,© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

754

Anees ul Husnain, Norrima Binti Mokhtar, Noraisyah Binti Mohamed Shah, Mahidzal Bin Dahariswarm and flock of UAV may conduct. It can also
be inferred with respect to current developments inthis technology that the maximum autonomy of a
UAV is a goal still unclear. However, a certainty inthis regard so far is that a higher level of autonomy
would inevitably require exploiting the onboard
computational
capabilities
thatdemandcomputationally intelligent algorithms. Therefore,
numerous approaches, criteria and innovations havebeen observed in effort to enhance the performanceof autonomous flights. The graph depicted in fig.1.
shows the published literature, indexed by the webof science, for the keyword "UAV" in the past 20
years. It shows that approximately 70% of thearticles, published from 2016 to 2019.[2]Another important factor to mention is the
supporting role of UAVs in multidisciplinary
research. This has contributed momentously in thespans of Geology, Chemistry, Environmental
Sciences, Agriculture, Material Sciences, PhysicalGeography, Water Resources, Transportations,
Mechanics, and many other research areas,elaborated in fig.2.[7]. Optimistically, this hasbrought scientists and engineers much closer toseveral aspects of their research.

planning algorithms. The rest of the part wasconsidered as developments in UAV applications.
Algorithms were assigned labels relating to thecategory of their knowledge streams, as an instance,
bio-inspired algorithms, graph search, etc. The
algorithms in the second group were further labelledconcerning the target application.
3 Needfulness
It was required to find the most suited algorithm fora specific target application and to identify apotential set of UAV applications for a particularalgorithm. After discovering the top-notch
algorithms and applications, an application toalgorithms mapping was presented to complete thepicture.
Some excellent surveys on path planning algorithms
in recent years. In these surveys, the focus was
relatively concise in most of them. A few studies
about planning algorithms considered workspacedimensions,[3] and obstacle avoidance,[4] where theprimarily concern was the nature of workspace andamount of information available to the robot.
Another to mention is a technique-specific pathcomputation,[5] moving obstacles clutteredenvironments.[6]. These dealt with the unexpectedchanges within the workspace and primarily deal
with the lower level control of the UAV.
The considerations for this study were more towards
practicality and direct in approach; first, the mostrecent developments in ongoing autonomous path
planning applications and second, the algorithms
associated with the most focused UAV applications.

2 Methodology
The selection of articles was based on two
separate queries, i.e., autonomous path planning for
UAVs and autonomous UAVs swarm. The search
was conducted on the Web of Science Core
Collection database for the past five years. From theresults, potential groups and subgroups were
established. The research undertaken to improve theperformance of existing algorithms or introducing
innovative ideas for computing algorithms weregrouped as developments in autonomous path

This helped to develop an application to algorithms
mapping out of the recent published literature. thatoffers the reader with appropriate choices ofalgorithms for a specific UAV application.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

755

Towards Establishing Path Planning

Fig. 1. Growth of research on UAVs in the past two decades (X: years, Y: publications)[2]Fig. 2. Contribution of UAVs in Multi-Disciplinary Research Areas (2000-2020)[7]
identifying the focus of UAV’s path planningresearch and its directions. At here, only the survey
findings from the algorithmic developments andmost focused applications have been presented,
respectively below.
It was found that the fusion of two or morealgorithms was 25% of all the algorithms, andEvolutionary algorithms were widely chosen for the

4 Survey Findings
The conducted survey was categorized under
two sections; one, most widely utilized algorithms
and two, most significant UAV applications. The
relation established between these two categories
was found helpful in identifying the usefulness ofalgorithms and promising UAV applications. Hence,© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

756

Anees ul Husnain, Norrima Binti Mokhtar, Noraisyah Binti Mohamed Shah, Mahidzal Bin Dahari

optimization process. However, the utilization of
bio-inspired algorithms and modifications, if
combined, these comprised 37% of the whole pie
chart, depicted in Fig. 3.
From the survey, it was also observed that
Cooperative Mission planning among UAVs has
been the hottest and most focused research area inmobility, and accessibility of the required equipment.
The second most attractive application was Sense &Avoid, followed by extended flight duration andcoverage area, applications. New challenges haveemerged in UAV research for communication
infrastructure among UAVs, a unified identificationof foes and friendly flights, secure communications,and UAV traffic regulatory systems.

Autonomous Path Planning. Their reasons mainly
include cost-effective, precision to conduct lower
atmospheric aerial operations, ease of deployment,

Fig.3. Breakdown of autonomous UAV path planning algorithms (left).and applications (right)
However, there have been some worth
mentioning observations:1.
It is quite disturbing to notice an absenceof Machine Learning techniques2.
there can be observed a huge dominance of
optimization algorithms.
3.
The highest level of autonomy has notreached, and it stands as a goal not clear yet.
4.
Any machine that is supposed to be
intelligent must contain a set of specific smart or
intelligent actions. The intelligent actions come from
intelligent algorithms.
5.
A criteria that can be foreseen at this point
in time can be as follows: The selection ofautonomous UAV's behavior from UAV itself, basedon the randomness of its environment and mission
objectives, ought to be rated as the highest degree of
autonomy and hence may serve as the ultimate testof UAV's autonomy.

5 Reflections
Very high diversity in path planning applications
and available algorithms engendered skepticism
about the selection of a better or most suitedalgorithm for a specific UAV application. Therefore,
autonomous path planning applications and thealgorithms utilized for these have been mapped inFig. 4. Some of the algorithms can be observed forserving different categories of applications, whileothers are dedicatedly developed to solve a
particular path planning problem. It can also be
inferred as the figure illustrates the most focused anddemanded features of a UAV that provides a hint of
futuristic models. The future of UAVs anddevelopments in computational intelligence would
look forward to the inclusion and selection ofdifferent data-process models to attain the highest
possible intelligence capabilities, under the lights oftechniques like reinforcements learning.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

757

Towards Establishing Path Planning

Fig. 4. Applications to Algorithms Mapping for Autonomous Path Planning of UAVs
Acknowledgments:

2 Analyzer, C.A.-S.R., 'UAV' search results analysis:
Publication Years (1999-2019), in Clarivate
Analytics - Web of Science, U.s.r.a.P.Y. (1999-2019),
Editor. 2020, Clarivate Analytics: Online.
3 Yang, L., et al., A literature review of UAV 3D path
planning, in Proceeding of the 11th World Congresson Intelligent Control and Automation. 2015, IEEE:
Shenyang, China.
4 Mohammadreza Radmanesh, et al., Overview ofPath-Planning and Obstacle Avoidance Algorithms
for UAVs: A Comparative Study. Unmanned
Systems, 2018. 6(2): p. 13.
5 Noreen, I., A. Khan, and Z. Habib, Optimal
Path Planning using RRT* based Approaches: A
Survey and Future Directions. International JournalofAdvanced
Computer
ScienceandApplications(IJACSA), 2016. 7: p. 11.
6 Jun, M. and R. D'Andrea, Path Planning for
Unmanned Aerial Vehicles in Uncertain andAdversarial Environments, in Cooperative Control:

The study was carried out as part of the researchon "Autonomous Path Planning of UAVs forEnvironmental Monitoring" at the University of
Malaya, Kuala Lumpur, Malaysia. The work is
supported by University of Malaya, grant number:
GPF062A-2018 for experimental equipment
procurement. The Ph.D. scholar is sponsored by afunded scholarship from the Higher Education
Commission (HEC) of Pakistan. The author would
also like to acknowledge the support from theIslamia University of Bahawalpur, Pakistan.
References:
1 Cabreira, T.M., L.B. Brisolara, and P.R.F. Jr., Survey

on Coverage Path Planning with Unmanned Aerial
Vehicles. Drones, 2019. 3(1).

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

758

Anees ul Husnain, Norrima Binti Mokhtar, Noraisyah Binti Mohamed Shah, Mahidzal Bin Dahari

Models, Applications and Algorithms, S. Butenko, R.
Murphey, and P.M. Pardalos, Editors., Springer,
Boston, MA: United States. p. 95-111.
7 Analyzer, C.A.-S.R., 'UAV' search results analysis:
Research Areas Categories (1999-2019), in
Clarivate Analytics - Web of Science, U.s.r.a.R.A.C.
(1999-2019), Editor. 2020, Clarivate Analytics.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

759

Development of Interactive Robot-Emotion Estimation System Using Speech by 1dCNNYugo Kawachi

Department of Mechanical Information Science and Technology, Kyushu Institute of Technology
680-4, Kawazu, Iizuka-City, Fukuoka, 820-8502, Japan
Eiji Hayashi

Department of Mechanical Information Science and Technology, Kyushu Institute of Technology
680-4, Kawazu, Iizuka-City, Fukuoka, 820-8502, Japan

E-mail: kawachi.yugo846@mail.kyuutech.jp, haya@mse.kyutech.ac.jp
http://www.kyutech.ac.jp/

Abstract
In order for robots to interact smoothly with people, they need to recognize human emotions and express its own
emotions through both verbal and non-verbal communication. In non-verbal communication, we naturally do things
such as estimating emotions from the tone of the other person's voice while talking. In this study, we developed asystem to estimate emotions from features of speech rather than the speaker's words by comparing two different data
sets, and compared what features each data set has and how they differ for each subject.
Keywords: personal robot, emotion estimation, nonverbal communication, 1dCNN

emotions from speech. Therefore, in this study, wedeveloped an emotion estimation system for speechusing a machine learning method called 1 dimensional
convolutional neural network(1dCNN).

1. Introduction
With the expansion of the robotics industry market, thedevelopment of service robots is becoming more andmore popular. These robots are intended to be used in
the home, medical care, welfare, and other places where
people communicate with each other, and it is necessary
for them to behave and talk in a friendly manner. In thisresearch, we are developing an interactive robot that
pursues human-like movements by focusing on nonverbal interactions (cooperative behavior) such as facial
expressions and body language.
In conversations, we can infer the other person's
emotions from the inflection of their speech. We
thought it was necessary to have a function to estimate

2. Emotion Estimation System for Speech Using
1dCNN
In this system, we used a model called emotionclassification-Ravdess1. In addition, user status was
defined 'Positive' and 'Negative' in line with previous
studies.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

760

Yugo Kawachi, Eiji Hayashi

negative. The new dataset contains English
greetings of two words or less, not sentences. The
total number of data is 120, 60 for positive and 60
for negative.

2.1. Structure of 1dCNN
The structure of the 1dCNN for emotion estimation isshown in Fig. 1. In Fig. 1, A is Convolution layer and B
is Pooling layer. The parameters of the Convolution andPooling layers are shown in Table. 1, and the ReLU andsoftmax functions are used as activation functions.

2.4. Accuracy Evaluation

Table. 2 shows the classification results by eachmodel of the results of using 1dCNN with
MFCC and Mel frequency spectrum as input,
and it used author's voice.
DatasetFig. 1 Structure of network

Feature Values
Positive
Negative

1dCNN MAX_pooling 1dCNN
Size of filter
585
stride111Number of filters
128
128
128

Ravdess
MFCC
0%
67%

Spectrum
0%67%

Newly Created Dataset

MFCC Spectrum
92%
89%
93%
90%

Table. 2 Classification Accuracies for each featureFirst of all, focusing on the classification accuracyof each dataset, the classifier using Ravdess wasclassified as negative in both MFCC and Mel
frequency spectrum. On the other hand, theevaluation using the newly created data set
showed that the classifier was able to classify
more unbiasedly than the classifier using
Ravdess. The reason is conceivable that Ravdess
dataset is a sentence. When words are given as
input, it is necessary to focus on the inflection of
the words for classification. However, in the caseof sentences, the recognition rate drops due to the
influence of syllables in between sentences.
Next, classification accuracy for each feature
value in the newly created dataset, we can seethat MFCC has a higher accuracy of about 3%.
However, since there was only one difference inthe number of data that failed to be classified, it
was concluded that there was no difference in theclassification accuracy between the two features.
Based on the above results, the classifier
trained on the newly created data set using
MFCC and Mel frequency spectrum features
respectively was adopted as the emotion
estimation system.

Table. 1 parameters of each layers
2.2. Feature Values
The Mel Frequency Spectrum and Mel
Frequency Cepstrum Coefficient (MFCC) wereextracted as features from speech, and thefeatures were used as input data for training. The
MFCC is a discrete cosine transformed coefficientof the Mel Frequency Spectrum. The discretecosine transform is said to improve theperformance of the feature.
2.3. Leaning data
For training, we used the Ravdess2 dataset and a newly
created dataset.
Ravdess was recorded two types of sentences,"Kids are talking by the door" and "Dogs are
sitting by the door," read by 24 professional actors(12 men and 12 women) in eight different
emotions. The total number of data was 1440. The
eight emotions were Classified “neutral, calm,
happiness, sad, anger, fearful, disgust, surprised”.
In this research, neutral and calm were removed,
joy and surprise classified as positive, andsadness, anger, fear, and disgust classified as
negative. The new dataset was created using theauthor's voice, and as in Ravdess. It was recordedby emotion and classified into positive and© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

761

Development of Interactive Robot)

3. Verification
Test data containing the voices of four
people were input to the emotion estimation
system for classification. Fig. 2 shows theresults of classification using MFCC and Mel
Frequency Spectrum. A and C are the resultsof the system's classification of the positive
emotions, and B and D are the results of thesystem's classification of the negative
emotions. Table. 3 shows the classification
accuracies of each class obtained from theclassification results of this experiment.

Fig. 3 Shape of MFCC
Table. 4 shows the emotion classification accuracy

Fig. 4 Shape of Mel Frequency Spectrum

Fig. 2 Result of classification for each featurefor each subject. First, focusing on subject C, theTable. 3 Classification accuracy of emotion foreach feature

Positive
Negative

MFCC
66.7%
76.6%

accuracy is lower than the others in theclassification using the Mel frequency spectrum.
This is thought to be because the distance
between the subject and the microphone is close
and the volume of the sound is larger. Thissuggests that in the classification using the Mel
frequency spectrum, the factor of the loudness ofthe sound is related to the classification.

spectram
76.9%
78.0%

From the results of this study, it was found thatthe classification accuracy of the model using Mel
Frequency Spectrum was on average 5.9% higher
than the model using MFCC for feature
extraction. In addition, the model using MFCC
showed a bias towards the negative, however it
was not the case when the Mel Frequency
Spectrum was used. The shape of each feature
was output as a graph for comparison.
Fig. 3 shows the MFCC features of subject A's
speech, and Fig. 4 shows the same speech as inFig. 3 with the Mel frequency spectrum. Fromthese figures, It can be seen that in MFCC, there
is not much difference in shape between positive
and negative and it is difficult to judge, whereas
in Mel frequency spectrum, there was a clear
difference in shape.

Feature Value

MFCC
Mel Frequency
SpectrumemotionPositive
Negative
Positive
Negative

A
75.0%
83.3%
57.1%
76.9%

B
75.0%
83.3%
88.9%
90.9%

C
88.9%
90.9%
71.4%
33.3%

D
50.0%
66.7%
88.9%
90.9%

Table. 4 Accuracy of classification for each subject© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

762

Yugo Kawachi, Eiji Hayashi

4. Conclusion
In this study, we created and evaluated aclassification system for emotions using 1dCNN.
As a result, we were able to develop a system toclassify users' emotions with an accuracy of 76.9%.
However, we are able to identify emotions evenwhen we use speakers, which means that asystem in which the loudness and height of thevoice affect the classification is not appropriate.
This means that a system in which the loudness
and height of the voice affect the classification is
not appropriate. As a future prospect, it isnecessary to search for an emotion estimation
system that does not depend on the volume andheight of the voice, and also to construct anemotion estimation system that combines facial
expression and body expression recognition.
5. References
1. Marco Giuseppe de Pinto,https://github.com/marcogdepinto/Emotion-Classification-Ravdess(2019)
2. Livingstone, Steven R., Katlyn Peck, and Frank A. Russo.
"Ravdess: The ryerson audio-visual database of emotional speechand song." Annual meeting of the canadian society for brain,
behaviour and cognitive science. 2012.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

763

Deep Learning Methods for Semantic Segmentation of Dense 3D SLAM MapsPei Yingjian

MIST, Kyushu Institute of Technology, 680-4 Kawazu
Iizuka-shi, Fukuoka 820-8502, Japan
Sakmongkon Chumkamon

MIST, Kyushu Institute of Technology, 680-4 Kawazu
Iizuka-shi, Fukuoka 820-8502, Japan
Eiji Hayashi

MIST, Hayashi Lab, 680-4 Kawazu
Iizuka-shi, Fukuoka 820-8502, Japan
E-mail: yingjian.pei801@mail.kyutech.jp, m-san@mmcs.mse.kyutech.ac.jp, haya@mse.kyutech.ac.jp
www.kyutech.ac.jp

Abstract
Most real-time SLAM systems can only achieve semi-dense mapping, and the robot lacks specific knowledge of the
mapping results, so it can only achieve simple positioning and obstacle avoidance, which may be used as an obstaclein the face of the target object to be grasped, thus affecting the realization of motion planning. The use of semantic
segmentation in dense SLAM maps allows the robot to better understand the map information, distinguish themeaning of different blocks in the map by semantic labels, and achieve fast feature matching and Loop Closure
Detection based on the relationship between semantic labels in the scene. There are many semantic segmentation
datasets based on street scenes and indoor scenes available for use, and these datasets have some common tags. Basedon these training data, we can derive a semantic segmentation model based on RGB images by using the Pytorch
platform for training.
Keywords: 3D SLAM, Semantic Segmentation, Point Cloud, ROS

the work. The advantage of 3D SLAM is that the robot
can get the complete spatial information of the current
environment, and get the abstract modeling of the realenvironment in the virtual environment through theoctomap, and through the Rviz plugin of ROS platform,
the obstacle avoidance motion planning can be realizedwith the octomap as the reference.
For semantic segmentation of dense maps, we caneither use a direct segmentation method on 3D point
cloud data or a semantic segmentation method based on
2D RGB images, and we use the second method due tothe convenience of training data. We use the second
method for the convenience of training data. There aremany semantic segmentation datasets based on street

1.Introduction
Active obstacle avoidance motion planning is an
important element in the autonomous motion planning of
the robot arm. Active obstacle avoidance can not only
avoid collision damage during the operation of the robot,
but also improve the robot's ability to sense theenvironment and avoid causing safety accidents. In active
obstacle avoidance motion planning, one of theconditions is the level of the robot's perception of theworkspace. Effective obstacle avoidance motion
planning can only be achieved if the robot has a priorawareness of the obstacles in the workspace and is able
to update environmental information at any time during© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

764

Pei Yingjian, Sakmongkon Chumkamon, Eiji Hayashi

to bottom, the original RGB image, PSPnet_50,
PSPnet_50 with Bayesian filter on, and the unmodified
ResNet_50 image. From the comparison graphs, it can beseen that Bayesian filter does not contribute much to thesemantic segmentation accuracy of indoor scenes, while
PSPnet, as an improved network based on ResNet,
reflects better results on ade20k dataset. It can be seenfrom the figure that PSPnet accurately identifies thescreen that appear only partially in the scene (green part),
while the original ResNet is more ambiguous.

scenes and indoor scenes available for use, and thesedatasets have some common tags. Based on these training
data, we can derive a semantic segmentation model basedon RGB images by using the PyTorch platform fortraining.
There has been some progress in scene marking
research based on this approach, and our project has now
applied some of this technology to detect target types in
3D point cloud data. Combining this progress withobstacle avoidance motion planning, the robot will beable to accurately distinguish between grasping andobstacle avoidance based on an understanding of themeaning of the scene, and it will be easier to integrate thetwo systems into the same framework.
2.Segmentation Model Training
We refer to the work of Xuan Zhang et al. in selecting thesemantic segmentation model and improve it to someextent based on our system. Our study was mainly trainedusing the ade20k dataset, a training set that includes alarge number of images and labels of common household
items and can be applied to most scenarios. For thelaboratory scenario we will use, we used annotation
software to additionally annotate a portion of the images
and add them to the dataset for training. Figure.1 shows
Example image of our own data.
In terms of neural network selection, we chose
PSPNet as the basis, which is very suitable for semantic
segmentation work such as scene parsing, anddemonstrated a certain degree of accuracy when tested inreal scenes using pre-trained models. We used the
PyTorch platform to initially train the dataset with
PSPNet50. Due to the limited GPU computing resources
and the complex environment during the initial training,
the semantic segmentation effect was not perfect, but aclear segmentation boundary could be demonstrated
under sufficient lighting conditions.

Figure. 1 Comparison
between different models
Table.1 Comparison of segmentation speed (FPS)

Model

2.1. Segmentation Models Comparison

PSPnet_50
PSPnet_50 (Bayesian filter)
BasicResNet_50

In choosing the model, we mainly examined the training
accuracy of different neural network models on theade20k dataset, and the indoor part of the ade20k dataset
was chosen for testing, mainly because our project isbased on indoor scenes, and the part of the dataset chosencannot accurately evaluate the accuracy of some labels,
but it can greatly accelerate the training.
Figure.1 represents the real-time display of 2D
semantic segmentation under different models, from top

FPS (Average)
20
15
23

2.2. Segmentation Speed Comparison
Since SLAM pursues real-time, accuracy and robustness,
the frame rate should be used as the reference value forspeed determination. The comparisons in Table.1 are all
made under Ubuntu 18.04, ROS Melodic environment
using CPU calculations, and the relevant point cloud© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

765

Deep Learning Methods forgeneration module is not activated to save system
resources during the frame rate comparison.
2.3. Conclusion of Model Selection
It can be seen from the table that turning on the Bayesian
filter will have a significant impact on the frame rate, andthe original ResNet is better than PSPnet in terms offluency. Taking all factors into consideration, the PSPnet
semantic segmentation model without bayesian filter has
achieved a good balance between accuracy and speed,
and we will use this scheme in the actual test.
3. Overall System Construction
In this section, we will discuss the setting up of theexperimental environment and camera selection. The
first section will give a general overview of theexperimental environment and the experimental format,
and the second section will present our considerations
when selecting a depth camera.

Figure. 3 Conceptual Frame

3.2. Camera Selection
In terms of camera selection, Intel Realsense D400 series
was chosen for this study. The D435i with IMU moduleand the D435 without IMU module were used in practiceand they both performed consistently in the tests. The
reason for using D435 series is that when building 3D
SLAM maps, it is necessary to provide accurate depth
data and clearer images for feature recognition andreference, and the overall size of the camera has to betaken into account as well, so the camera should not be
too heavy or too big to be mounted on the operation endof the robot arm. The D435i, which comes with an IMU
module, is useful for 3D SLAM mapping and offers more
possibilities for optimization in subsequent development.

3.1. Environment Construction
In order to allow flexible observation of the workspaceby depth cameras, we placed the depth camera for SLAM
at the manipulator of the robot arm, and to compensate
for blind spots in the observation, we also placed aplurality of depth cameras at other locations in theworkspace. In the following sections, we will mainly
demonstrate and illustrate the main camera fixed on the
manipulator.
After our depth camera acquires the semantic
segmentation image, it publishes and projects the 2D
image to the point cloud, and generates Octomap using
the point cloud image.
Figure.2 shows how the system looks like in a rviz
UI in ROS, you can see Semantic Image on the corner
and a octomap view in the main window.
Figure.3 shows the overall view of system frame.

4. Mapping
The platform we use is ROS Melodic based on Ubuntu
18.04, and the graphical interface is ROS Rviz, which isalso used to facilitate the subsequent obstacle avoidance
motion planning.
Due to the relatively small workspace used in thisstudy, some of the parameters of the official launch file
are not applicable to the realities of this study, so we haveadjusted some of the parameters.
Through the actual test, the overall map building
resolution is relatively high and the map building isnearly perfect. The semantic octomap built can be
successfully saved through the relevant functions ofoctomap_server, and can be published directly inoctomap_server with .bt format into Marker Array Topic
under the condition that the actual working scene of therobot remains unchanged.
After completing the scene mapping, the obstacle
avoidance motion planning is performed using the
RRTconnect algorithm in MoveIt! and output directly to

Figure. 2 Rviz scene demo

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

766

Pei Yingjian, Sakmongkon Chumkamon, Eiji Hayashi

the Motoman robot arm. In the Gazebo simulation
environment, the obstacle avoidance planning effect on
octomap is good.
In practice, due to the problem of hand-eye calibrationand coordinate conversion, we can use static coordinate
conversion to fix the coordinates of the camera and robot
model to realize the map building and obstacle avoidance
function in MoveIt! It is also possible to save and publish
octomap at the same time to realize real-time mapbuilding and obstacle avoidance planning in different
Rviz terminals.
Since the actual runtime is based on the Python 2
platform, and the latest version of PyTorch no longer
supports Python 2, we use the older version 0.4.0, which
is only supported by the CPU, and therefore runs with acertain degree of lag and has some shortcomings in termsof time efficiency.

3) We placed more than one depth camera in theworkspace, but we only used one of them in practice, andwe hope that in the future we will be able to integrate thedepth data from all the cameras into a complete map.
Our next plan is to replace the modules with models
trained by other deep learning methods to improve theefficiency of map building. We plan to use Siamese
Neural Network-based semantic feature recognition
matching algorithms and implement the object-levelbuilding, and we refer to the methods used in Ref. 4 andRef. 5 for this work.

5. Conclusion

2.

References1.

After simulation and practical testing, the deep learning
method can be integrated with our system to a certain
extent, which is of great help in the subsequentdevelopment of intelligent control for Motoman robot.
Semantic 3D SLAM's mapping results are in line withexpectations, the obstacle avoidance function works
normally, and the constructed map can be reused, which
eliminates the need to model the scene in a virtualenvironment. This module will continue to be refined anddeveloped as part of the project.

3.

4.
5.

6. Discussion & Future Works
In practical tests, we also found the following problems:
1) Inefficient map building. When using the deeplearning method described in this paper, because thefeature point matching method not use the semantic data
to match the point, and the rate at which the camera
extracts keyframes from the video stream depends on theprocessing speed of the computer and the camera, in thecase where the robot arm moves too fast, the camera willcollect two pictures with almost no similar features, andthus cannot form a closed loop to build a map. In theactual test we have to use slower movement speed to get
the complete map, which affects the overall map building
efficiency.
2) Due to the existence of multiple coordinate nodes in
Motoman's virtual model, multiple coordinate
conversions are required when binding the camera
coordinates, which will result in the camera coordinates
being incorrectly bound to the world coordinate systemor the overhang position. We use a compromise between
opening two terminals to create the final effect, and using
fixed coordinates is also a last resort.

Labbe, Mathieu & Michaud, François. (2013).
Appearance-Based Loop Closure Detection for Online
Large-Scale and Long-Term Operation. Robotics, IEEE
Transactions
on.
29.
734-745.
10.1109/TRO.2013.2242375. (IEEE Xplore)Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017).
Pyramid Scene Parsing Network. 2017 IEEE Conference
on Computer Vision and Pattern Recognition (CVPR),
6230-6239.
Xuan, Zhang and David, Filliat, Real-time voxel based
3D semantic mapping with a hand held RGB-D camera,
2018, GitHub, GitHub repository,\url{https://github.com/floatlazer/semantic_slam
Hu, L., Xu, W., Huang, K., & Kneip, L. (2019). DeepSLAM++: Object-level RGBD SLAM based on classspecific deep shape priors. ArXiv, abs/1907.09691.
Puligilla, Sai Shubodh, Satyajit Tourani, Tushar Vaidya,
Udit Singh Parihar, Ravi Kiran Sarvadevabhatla and K.
Krishna. “Topological Mapping for Manhattan-like
Repetitive Environments.” 2020 IEEE International
Conference on Robotics and Automation (ICRA) (2020):
6268-6274.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

767

Robot Motion Generation by Hand Demonstration
Sakmongkon Chumkamon, Umaporn Yokkampon, Eiji Hayashi, Ryusuke Fujisawa

School of Computer Science and Systems Engineering, Kyushu Institute of Technology, Japan
E-mail: m-san@mmcs.mse.kyuetech.ac.jp, may@mmcs.mse.kyuetech.ac.jp, haya@mse.kyutech.ac.jp,
fujisawa@ces.kyutech.ac.jp

Abstract
Since traditional robot teaching requires time and instruction to the robot motion, we present a systematic framework
based on deep learning and experiment for generating robot motion trajectories from human hand demonstration. In
this system, the worker could teach robot easier rather than assigning the instruction to the robot controller manually.
Therefore, the robot can imitate the action in a new situation instead of directly teaching the robot arm. Ourcontributions include three points 1) the real-time extracting method of hand movement without marker using handdetection in 3D from human 2) the motion generalization of the hand trajectories from human 3) Robot path planningfor grasping and place the object to the target. We also present the experiment conducted by the user movement forreal data and evaluate the system using the manipulator robot. The investigation shows the pick-and-place task of therobot for food by hand demonstration.
Keywords: List four to six keywords which characterize the article.

In robot learning by demonstration, recent researchdevelops the robot to predict and learning the motion
from a human. However, the research develops from thedevice which is attached to the human body to extracting
the information of motion3. Additionally, some researchdevelops the teleoperation to the robot to learn how tomove following the human motion4.
In this paper, we propose and contribute the robot systemthat can generate the robotics motion from human handdemonstration and generalize the hand motion by
Dynamic Movement Primitive (DMP)5. Then we apply
the motion to the robot. Besides, this system uses TracIK for kinematic and OMPL for path planning after DMP
process. Fig. 1 shows the overview of our proposed,
which consists of hand demonstration with grasping
object and perception using RGBD cameras with hand
position estimation then robot motion using themanipulator and visualization using PCs.
For the rest of the paper, section 2 presents the frameworkof robotic motion generation approach by humandemonstration. Section 3 explains the experiments aswell as evaluation of the robotics motion. Finally, we

1. Introduction
In order to push country economic, smart automation in
the industrial sector is critical to improving nowadays.
Since the merit of artificial intelligence research could
facilitate various tasks in industrial. Therefore, we candevelop a factory that could produce products faster withhigher precision. In a smart factory, the robot is essential
to assembly because it is more reliable and accurate thanhuman.
The manipulator robot actually is controlled by joint
position; however, we usually control the robot in
Cartesian space; therefore, we have to calculate theinverse kinematic (IK). One of IK that is popular recently
is Trac-IK and KDL, which can figure the solution morethan traditional methods 1. The robot also needs to plan
the trajectory according to the environments such as
avoiding the collision or moving to the target. The recent
motion planning method is Open Motion Planning
Library which is popular, reliable and fast for solving the
trajectory motion in various movement such as robot arm
motion, vehicle, drone2.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

768

Sakmongkon Chumkamon, Umaporn Yokkampon, Eiji Hayashi, Ryusuke Fujisawa

Fig. 1. The overview system of the Robot Motion Generation by Hand Demonstration.

present the conclusion, discussion and critical finding inthis paper.

2.2. Robot Trajectory Generation and Motion
Since the robot knew the hand motion trajectory, we canthen plan the robot's motion according to the workspaceconstraint and the trajectory. Firstly, we set the homepose to initialize the robot joints to prepare a starting
position to move following the trajectory. When the robot
gets the new trajectory from demonstration, it will planmotion using the Open Motion Planning Library (OMPL)
and Trac-IK inverse kinematic. In this implementation,
we utilize the MoveIt tool with Robotics Operation
System to organize motion planning and collision
protection among the workspace environment. Aftercomplete recording the trajectory, then the robot moves
following the human hand trajectory.

2. Approach
In this section, we explain the overview system including
robot hardware configuration, hand recognition, hand
position and trajectory determining. Finally, the motion
repeating following the human hand demonstration.
2.1. Hand Position Extraction
Firstly, we implement the method to collect the data fromthe demonstration by record human hand motion. We
divide the process into two steps. The First step is handdetection. We introduce the deep learning-based model
which is Single Shot Detection (SSD) for real-time handdetection from a colour image6. SSD could outcome thedetection box of hand area then we apply the human skinfilter to getting only hand region. Later, we determine thehand position by calculating the centroid of the handregion. Therefore, we could get the hand position in 2dimension. Since we use the RGBD camera, we apply
both colour and depth image to map each other. After that,
we map the 2D hand position into depth image to get thehand position in 3-dimension cartesian coordinate.
Finally, we concatenate the point while the human moves
to be the trajectory of hand motion and record to send to
the robot.

2.3. Experiment Setup
We use the industrial robot with 7 degrees of freedom andsoft gripper tool of end-effector for the hardware
configuration system since we aim our robot to grasp
various food. For robot control system is implemented in
the desktop PC with GPU Nvidia RTX 2080Ti. The PC
also operates the SSD for hand detection, archiving the
average hand detection 55 frame per second. In
experiment setup, the robot is mounted on the robot basestation with the camera mounted at the base of the robot.
We then calibrate the camera position translation to therobot position and orientation.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

769

Robot Motion Generation byof the robot then move up. Additionally, Fig. 3 presents
the marker of the hand represent in 3D cartesian
coordinate. After that, the user moves the hand down alittle from the centre of the trajectory. Finally, the user
moves the hand down to the target to the left side of therobot. In hand position estimation, we determine theposition using colour image then we map the colour
image to depth image to get the cartesian coordinate ofhand position. Finally, the trajectory is concatenated by
the point to be the waypoint of hand movement. The
example of the hand trajectory is shown in Fig 4, whichis represented by the marker.

Fig. 2. The Trajectory Data collection procedure. The
participant is working on the demonstration by moving his handfor trajectory collection.

3.2. Robot Repeating the Human Hand Motion
Since the participant made the demonstration of handmotion using grasping the object, which is the chicken
fry. Since our research would like to develop the robot toassist the worker in the food assembly industry, theparticipant picks and places the chicken fry from starting
point to the target point to teach the robot move to thesame motion and target as in Fig. 4 which shows thedesired trajectory. After we finish the desired trajectory,
the robot moves following the trajectory. In Fig 5, therobot starts from T1, which the robot moves to thestarting position. At T2, the robot moves up to the left
side of the robot. At T3, the robot moves the end-effector
slightly down as the human demonstration. At T4, therobot moves up to the left side of robot to the target
chicken fry. At T5, the robot is approaching to the target
to the left side of it. Finally, at T6, the robot could moveto the target as the human demonstration before at thesame place that hand placed the chicken fry.

Fig. 3. Human Demonstration in 3-dimension point cloud withhand position detection represented by the green marker.

3. Experiment and Results
In our experiment, the user performs the demonstration,
which is pick-and-place tasks for food. The robot could
detect the hand position in cartesian coordinate then the
robot reproduces the motion following the user’s hand
trajectory by robot end-effector. Firstly, the robot system
performs hand detection, then extract the hand position
in 3D cartesian coordinate and record the user’s hand
trajectory. After that, the robot performs the motion
planning and move the end-effector following the
trajectory.

4. Conclusion
In this paper, to fill the gap of the related research, we
present and implement the robot framework of robot
motion generation by learning from human handdemonstration and generalize the motion applying toindustrial manipulator. It aims toward natural teachingprocess relating to human learning from experience suchas learning from human hand motion.
In this paper, we contribute the method of hand trajectory
extraction using deep learning based which is single shotdetection to detection hand position. The hand detection
using deep neural networks could perform in real-time
hand position estimation using RGBD camera to collect3.1. Human Demonstration by Moving Hand
The participant makes the motion by his hand in front of
the robot. The trajectory of the hand movement follows
to the Fig. 2, which the participant moves from right side

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

770

Sakmongkon Chumkamon, Umaporn Yokkampon, Eiji Hayashi, Ryusuke Fujisawa

In future works, we would like to implement withconstrain workspace and improve the pose estimation ofhand for more reliable learning from humans.
Acknowledgements
This research was partially supported by the Religion
revitalization project in Kitakyushu by the Cabinet Officeof Japan.

References
1. Beeson, Patrick, and Barrett Ames. TRAC-IK: An opensource library for improved solving of generic inverse
kinematics. In 2015 IEEE-RAS 15th International
Conference on Humanoid Robots (Humanoids), pp. 928935. IEEE, 2015.
2. Sucan, I.A., Moll, M. and Kavraki, L.E., 2012. The open
motion planning library. IEEE Robotics & Automation
Magazine, 19(4), pp.72-82.
3. Wang, Weitian, Rui Li, Yi Chen, and Yunyi Jia. "Humanintention prediction in human-robot collaborative tasks."
In Companion of the 2018 ACM/IEEE international
conference on human-robot interaction, pp. 279-280. 2018.
4. Jha, Abhishek, and Shital S. Chiddarwar. "Robot
programming by demonstration using teleoperation
through imitation." Industrial Robot: An International
Journal (2017).
5. Schaal, Stefan, Jan Peters, Jun Nakanishi, and Auke
Ijspeert. "Learning movement primitives." In Robotics
research. the eleventh international symposium, pp. 561572. Springer, Berlin, Heidelberg, 2005.
6. Liu, Wei, Dragomir Anguelov, Dumitru Erhan, ChristianSzegedy, Scott Reed, Cheng-Yang Fu, and Alexander C.
Berg. "SSD: Single shot multibox detector." In European
conference on computer vision, pp. 21-37. Springer, Cham,
2016

Fig. 4. The overview of the Robot Motion Generation by Hand
Demonstration.

Fig. 5. The capture images of robot motion capture according
to the human demonstration.

the trajectory path of hand from the frame by frame fromhuman demonstrating. The second, we proposed the
motion generalize motion from hand and generate themotion to the robot. Our approach can generate andgeneralized motion from hand demonstration without amarker, making the system more reliable and convenientfrom the real-world tasks. We also successfully
experiment and evaluate the performance of the approach.
© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

771

Medical Telerobotic: IRAPs SHaRE-aGIVeR
Noppadol Pudchuen1, Jiraphan Inthiam2, Wisanu Jitviriya3
Amornphun Phunopas4, Chirdpong Deelertpaiboon5

Center of Innovative Robotics and Advanced Precision Systems: iRAPs,
Faculty of Engineering, King Mongkut’s University of Technology North Bangkok,
1518 Pracharat 1 Road,Bangsue, Bangkok 10800, Thailand
Aran Blattler6

Department of Mechanical Information Science and Technology
Kyushu Institute of Technology,
680-4 Kawazu, Iizuka, Fukuoka 820-0067, Japan
E-mail: noppadol.p@eng.kmutnb.ac.th1, jiraphan.i@eng.kmutnb.ac.th2, wisanu.j@eng.kmutnb.ac.th3,
amornphun.p@eng.kmutnb.ac.th4, chirdpong.d@eng.kmutnb.ac.th5, aran.blattler703@mail.kyutech.jp6
https://iraps.eng.kmutnb.ac.th

Abstract
The Coronavirus disease 2019 (COVID-19) pandemic has affected the global population. In particular, themedical personnel in direct contact with patients have been exposed to high risk. To reduce the spread of
COVID-19 and protect health-care workers and patients, we would like to present the fully automated medicaltelerobot as the IRAPs SHaRE-aGIVeR robot. Our robot is capable of generating both 2D and 3D maps
automatically, delivering medical supplies, food, or medical devices such as blood pressure monitors, pulse
oximeters, and so on. In addition, the user interface system is also vital part. Users are able to connect andcontrol the robot using a computer, a mobile device, or a tablet via the wireless network which is installed inside
the robot. Currently, our robots are being operated on with medical personnel at hospitals, regarding the
feedback from the formal caregivers that can prove our robot's efficiency in reducing the risk of COVID-19
spread.
Keywords: Medical Telerobotic, Perception, Planning, IRAPs SHaRE-aGIVeR.

medical robotics systems are mentioned to control andreduce infection spread to a large population [2].
Considering the current COVID-19 pandemic
situation, robots are well suited for caring for the wellbeing of patients, sharing the workload of health-care
workers in hospitals such as utilizing them for cleaning
and food preparation jobs in infected hazardous areas.
Robots are mainly classified with various applications in

1. Introduction
On 30th January 2020, the COVID-19 pandemic wasdeclared as "Global Emergency" by The World HealthOrganization (WHO) because the virus had spread
rapidly [1]. The COVID-19 pandemic has brought along
several threats, including economic and social disruption.
The group of people at high risk of catching COVID-19
are health-care workers, who spend much time up closewith the patient. Therefore, the technologies involving© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

772

Noppadol Pudchuen, Jiraphan Inthiam, Wisanu Jitviriya, Amornphun Phunopas, Chirdpong Deelertpaiboon, Aran Blattler

CameraMicrophone

RGB-D
CameraHolonomic Wheel
Odometry

Laser
Odometry

User Interface

Robot State

MonitorSpeaker

RTAB-Map

Robot State &(Visual SLAM)

Image, 3D
Information

2D Information

Multi-Function
Tray

Accurate
Robot State
Navigation

RGB-D Camera
Laser Range Scanner
Mobile baseUltrasonicsensor x8

3D&2D
Map
3D & 2D
Cost Map

Ultrasonic
Range Sensor

MotionPlanner

Goal (Position& Orientation)

MotionController

HardwareController

Fig.2 Perception Module and Navigation system

Fig.1 IRAPs SHaRE-aGIVeR robot

2. Overview system of IRAPs SHaRE-aGIVeR

healthcare and related fields. Receptionist robots areused at a hospital’s reception to give information about
various sections of the hospital and guide visitors andpatients [3]. Nurse robots are assigned to assist doctors
in the hospital in the same manner as human nurses.
These robots are commonly used in nursing andhealthcare individuals undergo high stress andexhaustion due to patient load. For example, Robear: arobotics nurse to lift patients in Japan [4], Moxi: nursing
robot placing medicines in bins [5], Paro robot andPepper robot are used to assisting elderly patients inproviding therapeutic assistance. Serving robots areused to deliver food and beverages, dispensing of drugs,
or removing unclean laundry etc., inside the hospital[6],[7] and so on. For new alternative of the health-carerobotics system, we would like to propose the fully
automated medical telerobot as the IRAPs SHaREaGIVeR robot. Our robot is a holonomic movement
platform, including the capability of generating both 2D
and 3D maps automatically, path planning and navigation
system, delivering medical supplies, food, or medical
devices (e.g., blood pressure monitors, pulse oximeters)
to patients.
Our paper is organized as follows: Chapter 2 explains
the IRAPs SHaRE-aGIVeR robot overview system,
which has three systems. In Chapter 3, our results areshown how our robot can operate with medical personnel
at hospitals. In the final section, we conclude the paperand describe our future works.

The robotic platform consists of the aluminum profile
construction weighing over 35 kg and capable of a
maximum speed of 0.8 m/s. The Mecanum wheeled driveis used with the robot's mobile base, which provides good
properties and is popular in modern robotic designs. It
can move sideways or rotate around its axis. The
dimensions of the robot are 80 x 70 x 150 cm (length,
width, height), it has a total weight of 60 kg and can carryan additional 80 kg. The appearance of IRAPs SHaREaGIVeR robot is shown in Fig.1.
2.1. Perception System
The first main core of our system is the perception system.
The perception system provides a common state such as
position, orientation velocity, and acceleration. Besides,
it also provides the overall environment in the form of a
3D map. The system overview of our perception system
is shown in Fig. 2. Each sub-system described as follows:
2.1.1. Holonomic Wheel Odometry
The first essential key for the perception-system isHolonomic wheel odometry. Therefore, we use the byproducts of our driving system by employing the encoder
wheel information was attached to the driving systemwith an algorithm in [8] to estimate the robot state suchas position, orientation, and velocity. However, becauseof our drive system, which we use to be our odometry has
four wheels. The odometry matrix that shows in [8] non-

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

773

Medical Telerobotic: IRAPs

system does not have enough features to track between
two frames.

invertible due to the equation system is an
overdetermined system because we have four odometry
wheels, but we have only three state variables to solve
due to our robot laying on the plane. In order to solve thisproblem, we permutate the configuration of the odometry
wheel such as (1,2,3), (2,3,4), (3,4,1), (4,1,2). Therefore,
the odometry matrix turns to be invertible and has aconfiguration as expressed in Eq. (1).
−1cos (θ a ) sin (θ a ) L  φa  vx 

  
(1)
v 
=  cos (θb ) sin (θb ) L  φb  y ω  a ,b ,c  cos (θ c ) sin (θ c ) L  φc 
When the left side vector is a state of the robot. θ is theangle of velocity vector for each wheel reference to robot
local coordinate. φ is the angular velocity of odometry
wheels, and (a, b, c) is odometry wheel configuration.
However, the robot state vector received from the wheel
odometry still has uncertainty due to it does not have anerror correction system. So, we will employ the following
algorithm to correct the accumulative error.

2.1.3. Visual SLAM
Our last perception system is the visual SLAM system.
The two systems mentioned before still have problems
such as accumulative error and cannot deal with a low
geometry feature frame. Both problems can be solved by
SLAM (Simultaneous Localization and Mapping) or, inother words, it is odometry with a close loop. At present,
the state-of-the-art SLAM has both sensors: lidar sensor
and camera. For our robot, we choose RTAB-Map [12]that is visual or camera SLAM. Because it relies on visualfeatures, so it does not affect by low geometry featureframes like lidar odometry. Besides that, it is also
combined with loop closure detection and non-linear
optimization, which is used to deal with the accumulative
error. Moreover, RTAB-Map supported combining theother types of odometry: wheel odometry and lidar
odometry, to reduce or eliminate accumulative error anduncertainty around an environment. RTAB-Map has twoworking modes. The first one is the mapping mode. We
used it to create a consistent map as the memory of therobot. Another one is localization mode, which fuses all
information from wheel odometry, lidar odometry, andvisual odometry to estimate the robot's accurate statevector references to the created map.

2.1.2. Lidar Odometry and Visual Odometry
As mentioned before, wheel odometry has uncertainty
such as friction force and wheel slippage. In order to deal
with the uncertainty of an environment, we employ thelidar odometry techniques because the core algorithm oflidar odometry relies on external features informationsuch as plane and corner. The core concept for our lidar
odometry includes the following steps:• Point Cloud Filtering: The input point cloud isdownsampled by voxel grid filter and statisticaloutlier removal filter [9] in order to reduce
bandwidth, computational power and remove
the outlier which comes from an environment
such a highly reflective surface and thereflection of sunlight• ICP Registration: In order to estimate the statevector of the robot, we employ iterative-closestpoint
(ICP)
[10]
implementedin
libpointmatcher [11] by registering the new
point cloud to the last keyframe, which called
scan to scan method (S2S).
For now, we received the state vector of the robot fromlidar odometry. Nonetheless, it still has uncertainty likewheel odometry. The lidar odometry cannot deal with along corridor or large flat plane environment. Due to the2.2. Navigation System
The next system is the navigation system, which plans
the trajectory path and controls the robot movement. The
system overview of the navigation system is shown in
Fig.2. Each sub-system described as follows:
2.2.1. 3D and 2D Cost mapping
The first sub-module of the navigation system is 3D and
2D cost mapping used to transform the 2D occupancy
grid map provided from RTAB-Map into costmap used
for planning route. The structure of our cost mapping
module references from [13]. It was also implemented
inside the ROS [14] costmap_2d module. However, we
also developed a new range sensor layer to deal withnoisy ultrasonics using a probabilistic model. The overallsystem of costmap is divided into two parts: global and© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

774

Noppadol Pudchuen, Jiraphan Inthiam, Wisanu Jitviriya, Amornphun Phunopas, Chirdpong Deelertpaiboon, Aran Blattler

Switch HUB &USB
CameraLaser Range
Scanner

Wireless Access
Point

USB Interface

Microphone
and Speaker

USB to TTL

Mini-PC(CPU and GPU)USB Interface

RGB-D
CameraMonitorUSB to TTL

Master MCU
32-Bit 120 MHz

Ultrasonicsensor (x8)Serial Communication

Fig.4 (Left) The medical staff can save position and select goal for the
robot. (Right) Telemedicine using peerCalls localhost.

Speed Control (DC motor x4)

A

B

Incremental
RotaryEncoder
PWM

MCU
DIR
32-Bit 72MHz

DC motor

POWER
H-BRIDGE

Fig.3 Low-level Control System

local costmaps, which both parts will be used in the next
sub-module separately.
2.2.2. Motion Planner and Motion Controller

Fig.5 The robot's working area in memory of the mixed map andimage reconstruction in 3D

The next sub-module inside the navigation system is the
motion planner or trajectory planner module. The motion
planner consisted of two parts. The first one is the global
planner. It is used for planning a trajectory to reach thegoal in the long term. It is called as the global path.
Another one is a local planner. It used to deal withimmediate situations by trying to reach the global path.
We called it the local path. If there is no obstacle the local
and global paths will overlap. The first time we build therobot we apply the Dijkstra's/A*global planner together
with the dynamic window approach (DWA) [15], [16]local planner which both implemented inside ROS
navigation stack. Then, we found that the DWA local
planner was not suited for our robot. Because the robot
uses a holonomic drive system which increases thecomplexity for the DWA local planner by “n” times.
When “n” is a number of samples used to exploring thevelocity space for each axis. For example, the differential
drive robot has sample space for all velocity space: O(n2)because it has two of freedom. Nevertheless, for aholonomic robot, the complexity increases to be O(n3)because it has three degrees of freedom. Besides, it also
has unpredictable behavior because its algorithm is asampling base. The user may be confused about its
behavior and feel uncomfortable. Therefore, we decided
to develop a new local planner which we refer to elastic
band planner [17]. Because it has lower complexity,

suitable for a holonomic drive system, and also has a
predictable behavior. The user can get acquainted in ashort time. In summary, we used Dijkstra's/A* for the
global planner and elastic band for the local planner. The
final step for the Motion Planner and also Navigation
system is Motion Controller. We apply a simple PDcontroller to control both the position and orientation of
the robot for each time step.
2.3. Low-level Control System
The robot contains a powerful Mini-PC (CPU and GPU
units) that can handle harder computational tasks. For
the module interface of Mini-PC that can communicate
with all the sensors and other modules. A detailed schemeof the low-level control system is shown in Fig. 3. The
drive subsystem, four DC motors are equipped withincremental rotary encoders (resolution 12 bits) for
rotational speed measurement. PID controller isimplemented in the control loop of the speed controllerof each DC motor.
3.

Verification

The robot was tested in the Covid-19 state quarantine
from June to September 2020 at the Queen Sirikit Naval
Hospital, Thailand. The medical staff could
autonomously send the robot to deliver food and things© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

775

Medical Telerobotic: IRAPs

to the front of infected people's rooms. When the robot
had reached the goal, the medical team could make avideo call from the staff's room to communicate with thepatients without personal contact, as shown in Fig. 4. The
robot moved along the long corridor to stop at the
selected room on the left or right side in Fig. 5. The robot
has logged data for four months to a google sheet, and we
can summarize the operational performance. The robot
gets the active command 5,448 times in 94 days to go to
the rooms. The robot has an average operating time of 12hours and 58 orders per day. It can reach the goals of 94%
and 6% missing because there is something on the goal's
position. The robot waits for clearing another object at
the destination until a timeout and cancel the order byitself. The robot can continue working from 6 to 8 hours
when starting with a full charge, depending on thenumber of orders.
4.

5.
6.
7.
8.

9.

10.

11.

Conclusions

The robot is a success to use in a real situation of state
quarantine. The medical staff can look after the patientsfrom a distance using the robot for delivery andcommunication. The robot can go to the destination
automatically with obstacle avoidance and is easy tocontrol through the web application by any device.
However, the team is developing more functions to
deploy to the robot by the doctor's advice.

12.

13.

Acknowledgements

14.

This work is supported by King Mongkut’s University ofTechnology North Bangkok, Queen Sirikit Naval
Hospital, Navamindradhiraj University, PTTGC and
Polyfoam Manufacturing Co., LTD.

15.

References
1. WHO. Report of the WHO-China Joint Mission on
Coronavirus Disease 2019 (COVID-19); WHO: Geneva,
Switzerland, 2020.
2. Yang, G. Z. et al., "Combating COVID-19—The role ofrobotics in managing public health and infectious
diseases," ed: Science Robotics, 2020.
3. France-Presse, A. Robot Receptionists Introduced at
Hospitals in Belgium. The Guardian. Available online:
https://www.theguardian.com/technology/2016/jun/14/ro
bot-receptionists-hospitals-belgiu m-pepper humanoid.
4. Hurst, D. Japan Lays Groundwork for Boom in Robot
Carers. The Guardian. Available online:

16.

17.

https://www.theguardian.com/world/2018/feb/06/japanrobots-will-care-for-80-of-elderly-by-2020
Care is a Team Effort. Diligentrobots. Available online:
https://diligentrobots.com.
Panasonic Autonomous Delivery Robots—HOSPI—Aid
Hospital Operations at Changi General Hospital.
TUG-Change Healthcare. Aethon. Available online:
https://aethon.com/mobile-robots-for-healthcare
J. Inthiam and C. Deelertpaiboon, "Self-localization andnavigation of holonomic mobile robot using omnidirectional wheel odometry," TENCON 2014 - 2014 IEEE
Region 10 Conference, Bangkok, 2014, pp. 1-5, doi:
10.1109/TENCON.2014.7022281.
R. B. Rusu and S. Cousins, "3D is here: Point Cloud
Library (PCL)," 2011 IEEE International Conference on
Robotics and Automation, Shanghai, 2011, pp. 1-4, doi:
10.1109/ICRA.2011.5980567.
P. J. Besl and N. D. McKay, "A method for registration of
3-D shapes," in IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 14, no. 2, pp. 239-256, Feb.
1992, doi: 10.1109/34.121791.
Pomerleau, F., Colas, F., Siegwart, R. et al. Comparing
ICP variants on real-world data sets. Autonomous Robots
34, 133–148 (2013). https://doi.org/10.1007/s10514-0139327-2
M. Labbé and F. Michaud, “RTAB-Map as an OpenSource Lidar and Visual SLAM Library for Large-Scaleand Long-Term Online Operation,” in Journal of Field
Robotics, vol. 36, no. 2, pp. 416–446, 2019.
D. V. Lu, D. Hershberger and W. D. Smart, "Layered
costmaps for context-sensitive navigation," 2014
IEEE/RSJ International Conference on Intelligent Robots
and Systems, Chicago, IL, 2014, pp. 709-715, doi:
10.1109/IROS.2014.6942636.
Quigley, Morgan, et al. "ROS: an open-source Robot
Operating System." ICRA workshop on opensource
software. Vol. 3. No. 3.2. 2009.
O. Brock and O. Khatib, "High-speed navigation using the
global dynamic window approach," Proceedings 1999
IEEE International Conference on Robotics andAutomation (Cat. No.99CH36288C), Detroit, MI, USA,
1999,
pp.
341-346
vol.1,
doi:
10.1109/ROBOT.1999.770002.
D. Fox, W. Burgard and S. Thrun, "The dynamic window
approach to collision avoidance," in IEEE Robotics &Automation Magazine, vol. 4, no. 1, pp. 23-33, March
1997, doi: 10.1109/100.580977.
S. Quinlan and O. Khatib, "Elastic bands: connecting path
planning and control," [1993] Proceedings IEEE
International Conference on Robotics and Automation,
Atlanta, GA, USA, 1993, pp. 802-807 vol.2, doi:
10.1109/ROBOT.1993.291936.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

776

Crack Detection in a Concrete Structure Using an Underwater Vehicle
Yuya Nishida*

Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
Naoto Sohara*2

Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
Shinsuke Yasukawa*3

Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
Kazuo Ishii*4

Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
E-mail: y-nishida*1, s-yasukawa*3, ishii*4@brain.kyutech.ac.jp, sohara.naoto395@mail.kyutech.jp*2
https://www.kyutech.ac.jp/

Abstract
In this paper, to realize efficient underwater infrastructure inspection, automatic crack detection by image processing
is proposed. In first process of our method generates enhanced image based on the absorbance from the turbidity
meter and removes background component, and then detects crack from the enhanced image by using decision tree
learning algorithm. This paper explains the algorithm of our method and shows evaluation experiment results.
Keywords: Crack detection, image processing, Underwater vehicle

and dangerous to divers. Several institutes have been
researched underwater infrastructure inspection methodby using a ROV (Remotely Operated Vehicle) and image
processing [2]. However, cracks on the underwater
infrastructures are difficult to automatically detect by theimage processing, because underwater images taken by
the ROV depends on the water turbidity that varies fromthe site.
Our laboratory has been developing inspection methodby using the USV (Unmanned Surface Vehicle) and ROV
to realize efficiency inspection of underwater
infrastructure [3]. This paper proposes crack detection
method by using photo image from a camera and1. Introduction
Almost social infrastructures that were built at the timeof high economic growth have become deteriorated andhave a risk of accidents and traffic restrictions [1].
Japanese Ministry of Land, Infrastructure, Transport andTourism made the road law that the infrastructures suchas a bridge and tunnel should be inspected every five
years, in 2014. However, infrastructure inspection isdifficult with human hands alone because there are about
730,000 bridges and more than 10,000 tunnels in Japan.
In particular, the inspection of underwater infrastructures
is not progressing because the inspection is inefficient© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

777

Yuya Nishida, Naoto Sohara, Shinsuke Yasukawa, Kazuo Ishii

Fig. 1 Underwater infrastructure inspection by the USV and the ROV

absorbance from a turbidity meter installed on the ROV,
and experiment results for the evaluation of our method
are shown.
2. Underwater infrastructure inspection
2.1. Inspection procedure
The underwater infrastructures like valley highway
pillar are located in a lake and a river that are hard forpeople to go. The ROV which its movement is restrictedby the umbilical cable is hard to approach theinfrastructures for them inspection. Our laboratory
developed inspection method that a USV supports a ROV
for underwater infrastructure inspection. Figure 1 shows
operation overview and inspection procedure by using
the USV and the ROV. The USV has four thrusters forhorizontal movement, a GPS, IMU for positioning,
wireless LAN for communication to the operator, andbatteries. Fist, the operator deploys the USV with the
ROV in the water and the USV moves to target
infrastructure by heading and position controls. Second,the ROV dives by a winch mounted on the USV andcaptures the infrastructure images by using a front
camera and a LED light. At this time, the ROV moves tohorizontal direction by the USV thrusters and the ROV
heading is controlled its thrusters. The ROV and the USV
are connected an umbilical cable, and the operator on thequay monitors infrastructure image captured by the front
camera of the ROV during inspection. After theinspection, the USV that stores the ROV returns to thequay near the operator and is recovered by the operator.
Finally, cracks on the infrastructure is detected from

Fig. 2 The ROV equipped a turbidity meter

captured images by offline image processing that is
explained in Chapter 3.
2.2. The ROV system for inspection
The ROV shown in Fig.2 is used for inspection ofunderwater infrastructure. The ROV has a front camera
for crack inspection of the infrastructure and a bottom
camera for scouring inspection of seafloor near the
infrastructure as observation device. Two thrusters areused for that the ROV direction is controlled and theinfrastructure is approached. The transponder withmodem function includes the depth sensor and IMU
sensor, and the USV measures the ROV position by using© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

778

Crack Detection in ait. A turbidity meter on the rear consists of a red LED, an
illuminance sensor and sunshade, and surrounding watergets into the sunshade. The illuminance meter measures
the red LED light intensity I’ that is attenuated by thewater. Relationship between Light intensity I0 which isirradiated by the LED and I’ is expressed by following
the Lambert-Beer law:
𝐼𝐼 ′𝐴𝐴 = −log � �
𝐼𝐼0

(1)

where A denotes the absorbance that depends on the
distance traveled by the light. For distance-independent
values are desirable to represent the underwater lightenvironment, equation (1) is rearranged as following.
1𝐼𝐼 ′𝛽𝛽 = − log � �
𝑑𝑑

𝐼𝐼0

(2)

d denotes the distance the LED and the illuminancesensor, our turbidity meter outputs β that denotes theattenuation coefficient.
3.

Fig. 3 Crack detection procedureby deconvolution operation of H on equation (3). In
preprocessing, this research generates high contrast
image that background component is removed from f byusing median filter.

Crack detection method

3.1. Preprocessing

3.2. Detection process

Cracks on the underwater infrastructure are detectedby image processing from the images captured by the
ROV. The image processing is separated thepreprocessing and the detection process, as shown in
Fig.3. The contrast between the cracks and theinfrastructure surface is important for crack detection,
because the cracks is detected based on the brightness,
not color. First step generates high contrast images byusing the absorbance coefficient from the turbidity.
Second step searches crack position from the images byusing the decision tree learning.
Because light in the lake and the river diffuses by
floating particles such as the sand and the plankton,
degraded images are taken by the ROV during theunderwater infrastructure inspection. The degradedimage g is represented by using raw image f.
𝒈𝒈 = 𝑯𝑯 ∗ 𝒇𝒇

This process uses the decision tree learning ofclassification tree that can obtain complex identification
boundaries by combinations of simple identification
roles. The classification tree is learned by CART
(classification and regression tree) that threshold for theclassification is decided by following Gini coefficient [4].
𝐿𝐿(𝑡𝑡) = ∑𝑗𝑗≠1 𝑃𝑃(𝐶𝐶𝑖𝑖 |𝑡𝑡)𝑃𝑃�𝐶𝐶𝑗𝑗 �𝑡𝑡�(4)

∆𝐿𝐿(𝑡𝑡) = 𝐿𝐿(t) − {𝑝𝑝𝐿𝐿 𝐿𝐿(𝑡𝑡𝐿𝐿 ) + 𝑝𝑝𝑅𝑅 𝐿𝐿(𝑡𝑡𝑅𝑅 )}(5)

P(Ci|t) denotes the probability that node t in the tree
selects data of class i, and P(Cj|t) denotes the probability
that node t in the tree selects data of class j. In the Cart,
the decision tree learning is learned to maximize
following equation expressed by the Gini coefficient.
pL and pR represent the probability of being classified intoleft and right node. L(tL) and L(tR) denote Gini
coefficients left and right nodes. In this research, thedecision tree learning is learned by using equation (5)until that the node in the tree reaches 100 or more.
The input vector to the decision tree learning is 32
demission that consists of the brightness of target pixel(1), each brightness after applying 3 square median filters(3)

where * means the convolution operation. H in equation(3) denotes point spread function and is represented
multiplication of line spread functions in u and v axis ofimage coordinate, and those functions are approximatedby the gaussian function. Undegraded image f is obtained© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

779

Yuya Nishida, Naoto Sohara, Shinsuke Yasukawa, Kazuo Ishii(3), each brightness after applying 2 non-square medianfilters from 8 directions (16), each brightness after
multiscale line enhancement by 11 types Hessian matrix(11) and noise candidate data (1). The brightness of target
pixel is used as input vector, because the decision tree
learning in this research classifies target pixel into crack
and back background pixel. If only the brightness is used
as input vector, false recognition can occur due to thesurface color of the infrastructure. Because the brightness
after applying the gaussian filters is included to input
vector, classification results in the learning are lesssensitive to the background color [4]. Each brightness
after applying by the Hessian matrixes is used as input
vector, because enhanced cracks are extracted by multiscale line enhancement using the Hessian matrixes [5].
The brightness like noise is included to the input vector,
to reduce the effect of the impulse noise.

Fig. 4 Comparison of brightness distribution

4. Experiment4.1. Parameter estimation
As mentioned in chapter 3, the image taken by the ROV
in the turbid water degrades based on the point spreadfunction that is represented multiplication of the line
spread functions approximated by gaussian function.
However, nobody knows how the light is spread by turbid
water. To analyze the relationship of the attenuation
coefficient and the standard deviation for the point spreadfunction, this research performed photographingexperiment in the turbid water using the ROV with theturbidity meter. The clack scale which has lines from0.03mm width to 1.50mm width was used for thephotographing target. The turbid waters from 0 degree to
20 degrees were made by using the Kaolin which is also
used for JIS turbidity standard. Figures 4 shows each
picture and the brightness in the turbid waters of 0 degreeand 25 degrees. At this time, target clack was 0.2mm
width line that is the minimum requirement for the repair.
The turbidity meter outputted 3.6 in 0 degree water and4.64 in 25 degrees water, as attenuation coefficient. The
picture in 25 degrees water was darker overall than the
picture in 0 degree and the clack line were blurry. Figure
5 shows the standard deviations of approximatedgaussian function in each brightness versus absorbancecoefficient from the turbidity meter. This research usedapproximated quadratic function which fits to theFig.5 Absorbance coefficient vs. standard deviation

standard deviations without the one outlier data, to detect
the cracks.
4.2. Crack detection
The cracks on the concrete block was detected by our
method for evaluation. Figure 6 shows the image takenby the ROV in 25 degree water, grayscale image
corrected by using the point spread function, the image
with background components removed and detection
results. As a result of comparing the raw image and thecorrected image, the contrast of the crack edge on thecorrected image was higher than the contrast of raw
image cracks. Brightness unevenness and concrete
pattern were removed by process using median filters.
Although there were some cracks that could not be
detected, our method detected the cracks with few false
positives. Table 1 shows accuracy rage and sensitivity
with and without the correction by the point spread© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

780

Crack Detection in a

Table 1 Accuracy rate and sensitive in each turbidity
Turbidity

2.

3.

Fig.5 Absorbance coefficient vs. standard deviation

function. Although performance of the crack detection
with and without the correction was almost same until 10
degrees water, the accuracy rate and the sensitive were
improved with turbidity from 15 degrees to 20 degrees,
and the sensitivity of our method was greatly increased.
This means that the detection rate was improved by thecorrection using the point spread function.

4.

5.

Accuracy rate

Sensitivity

Withcorrection

No
correction

With
correction

No
correction

0

0.992

0.990

0.583

0.358

5

0.996

0.995

0.729

0.713

10

0.995

0.995

0.655

0.672

15

0994

0.990

0.625

0.586

20

0.994

0.974

0.642

0.425

25

0.994

0.972

0.5851

0.294

urism, https://www.mlit.go.jp/hakusyo/mlit/h25/hakusho/
h26/html/n1131000.html, in Japanese
Sugimoto Hideki, et al., Underwater Structure Diagnosis
System by Remotely Operated Vehicle, Journal of the
Robotics Society of Japan, Vol.34, No.8, pp.505-506,
2016, in Japanese
Takumi Ueda, et al., Inspection system for underwater
structure of bridge pier, Proc. of International Conferenceon
Artificial
LifeandRobotics,
DOI:
10.5954/ICAROB.2019.OS21-2, Beppu, 2019
Naoya Kuramoto, et al., Detection of pavement crack fromimage by decision tree learning, Journal of Japan Societyof Civil Engineers, Vol.71, No.2, pp.823-830, 2015, inJapanese
Yusuke Fujita, et al., Automatic and exact crack extraction
from concrete surfaces using image processing techniques,
Doboku Gakkai Ronbunshuu F, Vol.66, No.3, pp.459-470,
2010, in Japanese

5. Conclusions
This paper explains efficient underwater structure
inspection by using the USV and the ROV, and the crack
detection method in the turbid water is proposed. The
underwater infrastructure is observed by the ROV
equipped the turbidity meter which can measure theabsorbance coefficient of the turbid water. In preprocess,the image taken by the ROV is corrected by using thepoint spread function and high contrast image is
generated by using median filters. Our method detectedcracks on the underwater infrastructure in the turbid
water from the image enhanced by the preprocess, withhigh accuracy rate.
References
1. Road Bureau, Aging of social infrastructure, Current statu
s of aging and issues related to aging countermeasures, Ja
panese Ministry of Land, Infrastructure, Transport and To© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

781

Development of Shock Sensitive Tiny Dummy Robot
for Junior High School Rescue Robot Challenge
Kazuo Kawada, Kesuke Iuchi, Keita Murai and Hiroyuki Y. Suzuki

Graduate School of Humanities Social Sciences, Hiroshima University, 1-1-1 Kagamiyama
Higashi-Hiroshima, Hiroshima 739-8524, Japan
E-mail: kawada@hiroshima-u.ac.jp, hiro-suzuki@hiroshima-u.ac.jp

Abstract

Junior High School Rescue Robot Challenge is an annual activity held by Hiroshima University withsponsorship of a construction machinery company. The theme in 2020 was to convey injured people“tenderly” from top of half-demolished building to ground. We developed shock sensitive tiny dummy
robot, controlled by M5STACK microcomputer since it rigged with accelerometer. The remote (wired)controlled robots developed by junior high school students conveyed the dummy and evaluated theirperformances including “tenderness” of the robots using the dummy.
Keywords: Rescue Dummy, Robot Evaluation, Technology Education, Junior High School

must be done from various viewpoints. Remind that theguideline for junior high school technology education(stated by Ministry of Education, Culture, Sports,
Science and Technology) also claims that we have to deal
with humanities in a context of development oftechnologies. Manufacturing education must be
humanity development education as well. Robots of only
high performances must not be required, but robots
which can consider and cooperate with others will be
expected.
In such context, we abandoned the word “Contest” fromour robot activity and chose “Challenge” as an alternative.
Faculty of Education in Hiroshima University has been
holding “Rescue Robot Challenge” from 2014 for juniorhigh school students4. It formerly had been named“Rescue Robot Contest” and held since 2003 with theaim of deepening their understanding of social issues andthe relationship between people and robots through thecreation of robots with an awareness of tenderness tohuman5.
The basic theme of our “Challenge” is human rescue
from disastrous conditions. Japan is known as an

1. Introduction
Robot contest is an attractive activity for both students
and instructors in junior high school technology
education. Planning, processing, assembling andmanipulating robots by themselves make students foster
ingenuity and creativity in a great deal. Such experiences
can increase interest in today’s science and technology
developments as well.
A number of robot contests1-3 have been held heretofore,in Japan, in which they claimed that those contests havebeen successful to foster the capacities of invention,
logical thinking and creativity in young participants.
Those activities, on the other hand, focused on “winningor losing of a game” for evaluation of the robots.
Introduction of competition is a good way to arouse
incentive in participants’ minds, but it also has side effectof narrowing the view on the subject. Any robots containcomplexity in them, regardless of the level of the product.
They are not machines of single function. We always
have to treat the robots as system. Evaluation of robots
also, therefore, must be systematic. In the other word, it© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

782

Kazuo Kawada, Kesuke Iuchi, Keita Murai, Hiroyuki Y. Suzuki

earthquake-prone country. We have faced, in recent years
for example, the Kumamoto Earthquake in 2016, the
Hokkaido Eastern Iburi Earthquake in 2018, etc. It was
natural to desire a rescue robot. In this paper, we aregoing to describe theme, results and evaluating points of
the “Challenge” in 2020.
2. Challenge Theme in 2020
From the era of the “Contest”, our activity has used thetheme of “quickly rescuing a dummy doll (of 1/8 of realhuman) from a simulated earthquake site and transport itto a safety zone”. Time of rescue has been evaluated as
basic data, of course, but ideas for rescuing methods,
humanitarian standpoint have been also evaluated. In
particular, we have focused on the “tenderness” of therobots to the doll during evacuation.
In 2020’s activity, we developed a small shock-sensitive
dummy robot. The dummy was put on the remote (wired)controlled robots developed by the middle school
students and checks how much shocks and/or vibrations
are given during evacuation.
Figure 1 shows the rescue field used in 2020. The robots
with dummies are set in the starting area on the roof ofthe half-demolished building, then start to rescue
dummies passing through form Zone A to C. They canchoose different rout in Zone B of, i) stairs, ii) a gap withvertical walls and iii) a steep slope. On the ground (ZoneC) there are obstacles. The robots finally arrive at thesafety zone.
A team consists of 2-5 members of Junior high school
students. Every teams firstly give a 4-minute presentation
using screen showing about the characteristics of therobot they developed, then followed by a 3-minute rescue
activity using real field.

Fig.1 Rescue Field.
Display) of the M5STACK turns red and emits a warningsound. Introduction of the dummy equipped withaccelerometer made possible to evaluate more precisely
the degree of tenderness of the robot to the dummy.

Fig.2 Rescue Dummy with M5STACK.

3. Development of Rescue Dummy
For simplicity, we used a small tapper box (83 mm inlength, 83 mm in width, and 45 mm in height) as
dummy’s body (Fig. 2), although there are not protrudesof legs, arms or a head. We chose a microcomputer
module of M5STACK (Fig. 3) as controller since itequipped with accelerometers. Gross weight of thedummy was 81 g.
We set thresholds of the accelerometer of ±1.5g[m/s2]
in two directions of X and Y. When the dummy got two
acceleration of threshold or more, LCD (Liquid Crystal

4.

Result of “Challenge” and Evaluation

The 6th Rescue Robot Challenge in 2020 were held on
Feb. 16th, 2020 (still in 2019 as fiscal year). A total of 11
teams were participated. The points they got aresummarized in Table 1. Performances of the robots areevaluated from two viewpoints of, i) goal points and ii)
tenderness points. Maximum points of each viewpointare 20, so that the maximum total points are 40. For thegoal points, if the robot conveys the dummy to the safety© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

783

Development of Shock Sensitive

Another interesting idea was found in the Robot H (Fig.
5). It used crawlers for traveling the stairs in Zone B,
which result in a speedy transportation of the dummy to
the goal. Although it gave several shocks to dummy, thenumber and degree of the shocks were much less thanother robots which chose the stairs in Zone B. It had
ingeniously attached long arms in front and behind thebody, by which the robot could prevent to stagger by
manipulating the arms.

Fig.3 M5STACK with Accelerometer.
Fig.3 Rescue Robot C.

zone in 3 minutes without dropping the dummy, it gets
20 points. The goal points are decreased according todegree of problems happened during the rescue activity.
For the tenderness points, 20 points are given if nowarning sound is made, 15 points for 1-3 times of
warning, 10 points for 4-6 warnings, 5 points for 7-9
warnings, and 0 points if the warning surpasses 10 times.
Let’s see an example. Rescue robot C is shown in Fig.3.
Robot C was succeeded to transport the dummy to thesafety zone without dropping the dummy, so that it got
goal point of 20. On the other hand, the robot C gave 14
to 19 shocks to the dummy, therefore the tenderness
points were decreased much. This robot was using
magnets to slowly descend the steel slope in zone B.
Their trials were successful during practices. But
unluckily, the magnet fell off from the steel slope in final
performance, resulting in a big shock to the dummy.
In contrast, the rescue robot G (Fig. 4) could transport
the dummy without giving shocks. This robot used apantograph mechanism to descend between the gap of
two walls in Zone B.

Fig.4 Rescue Robot G.

© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

784

Kazuo Kawada, Kesuke Iuchi, Keita Murai, Hiroyuki Y. Suzuki

Table 1. Goal points and tenderness points for each rescue robot.

Rescuerobot
GoalpointsTenderness
pointsTotal
points

A

B

C

D

E

F

G

H

I

J

K

0

0

20

0

0

0

20

20

20

0

0

20

15

5

20

20

0

20

15

15

20

20

20

15

25

20

20

0

40

35

35

20

20

were 25, 18.75, and 25 points, respectively. These results
may indicate that the shortcoming of evaluation fromsingle factor. As we mentioned before, the robots
essentially contain “complexity” in their design,
mechanism, manipulating manner, and so on, therefore
their performances also must be evaluated from various
aspects.
6. Conclusions
We developed a dummy robot for “Rescue Robot
Challenge in 2020”. The dummy was sensible to shocks
during rescue transportation. Introduction of shock
sensible dummy made possible to evaluate quantitatively
the “tenderness” of the robots developed by the middle
school teams. Adding evaluation factors can improve thecorrectness of evaluation of robot performances.

Fig.5 Rescue Robot H.
5. Discussions
The 11 rescue robots can be classified according todescending rout in Zone B. Robots of type 1 chose thegap between two walls. Six robots of A, B, E, G, I, and J
chose this way. Type 2, four robots D, F, H, and K chosethe stairs. Only the robot C was classified to Type 3,
descended a steel slope using a magnet, as we explainedin previous section.
The average goal points of Type 1, Type 2, and Type 3
were 6.67 points, 5 points, and 20 points, respectively,
indicating that the Type 3 had higher manipulability
during transportation in Zone B. On the other hand, themean of tenderness points for Type 1, Type 2 and Type 3
were 18.33, 13.75 and 5 points. Comparison of the points
made clear that the robots of the Type 1 had higher
stability during transportation in Zone B. Nonetheless,there are no considerable differences in averages of thetotal points between Type 1, Type 2, and Type 3, which

References
1. C. Kasuga, An Analysis of Contesting Robots – Micro
Mouse and Sumo Robots, JRM, 10(1), 1998, pp. 22-26.
2. M. Asada, RoboCup-97, JRM, 10(1), 1998, pp. 30-33.
3. T. Yamamoto, T. Ichikawa and S. Fujisawa, A
Consideration on Rescue Robot Contest from Viewpointof Technology Education, JRM, 15(5), 2003, pp. 516-520.
4. K. Okamoto, Y. Shoda, T. Ueno, K. Kawada and M.
Nagamatsu, An Approach to 1st Rescue Robot Challenge
for Junior High-Shool Students -Present Conditions andProblems-, Technology Education, 6, 2016, pp. 8-15. (inJapanese)
5. K. Kawada, M. Nagamatsu and Toru Yamamoto, A
Practice of Rescue Roboto Contest in Junior High School,
J.of JSEE, 58-2, 2010, pp. 33-39. (in Japanese)© The 2021 International Conference on Artificial Life and Robotics (ICAROB2021), January 21 to 24, 2021

785